2022-09-02T04:05:05.335 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-02T04:05:05.337 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} --owner scheduled_cbodley@teuthology --verbose --name cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi --archive /home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440 -- /home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440/orig.config.yaml 2022-09-02T04:05:05.392 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-09-02T04:05:05.493 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440 branch: pr-47848 description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '7005440' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pr-47848 ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: afd422385fab8119a539d289dd1aeaad4c056fe4 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: afd422385fab8119a539d289dd1aeaad4c056fe4 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-qa-rgw-s3select-fixes_related_to_new_csv_parser sha1: 98002085f9003280d82d2f21aa1472b865694c0b owner: scheduled_cbodley@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: afd422385fab8119a539d289dd1aeaad4c056fe4 sleep_before_teardown: 0 suite: rgw suite_branch: wip-qa-rgw-s3select-fixes_related_to_new_csv_parser suite_path: /home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: 98002085f9003280d82d2f21aa1472b865694c0b targets: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDut5FvkIQGrPyM0nkT7N8tYudURf71e+foXRKk+M0U4z/GU7R1IHbYePTvXo0Xe2mDfrabdI/sBTAphzeJNUR4XNStF3M+BZId93/7mt0uJAqS5xbV6NkePILDmGdtrgPQRtdtowRiH1XYTgIQnWFMjqntlrki93aFphwIR9DrcTQWwsnCuXH+yUIpu9mLA51Avg4P38VqnycKGjsK35X2oTWzkXVBdG6ycgSNM2IpvdWGA30FknJoOnXqcDJA9lyoXFDffAo2SapEUC7oOP3MPfipbqb9bvJx4BD7Jt2RuPAPnB7nhMuzZXu5TrXgAPMyG8+zfviSPj8ZSFaYKKCweMu4tPvcx3J8qfSzux+qT3otsh9Npo9zdlFrboVFsacTPN70nzngBW1WM4htz+ZdJQDyCd4f1amTu/V5IHTCezhr39kgZGa4oG1DU6jgyyYq6D/KvQCiw55w6lkKf0dfkSyT8yp0iGf7L9W8su8g3C8v+6vRyPg7AtDZm4L93Fc= smithi160.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiwRVfUW8pFNs8ic5wz3OsS4S3a85p5doefMIekSi2aNJwcSHRdMPcJn4h+vlz5R5tj8i8uNPDyQjNwyBcM/wZJ7x0J1f/mFWwx4Zi0liFV7F+WPEdVQYVlLfDuspvkpjODvwSW92QU/8owPNBNLnutasr0SpaHutkXZjsYn7J0ybzdz8lf/HEriZo/5AYTrQyH1LZhhcIoIyvUOBjCQHC8L0LePd9lj+nZ13l/EX+QhsZRBYJh+31IerMPCP0/gk+a41JTqRjJJefFuzVFcudPBqIUwQxGzyT19IHp0O69MXWCH4kLupGDkdhjbQ7+j+fO6fFqOk+aJiGDwWsAChhuv9XeH9osWOUBsUopb7NMICRj0b9EFTKWzB7Ioyooor+ny7gbW31zy/Q+s/NgPwg/IP5lv1KRZYQk7JXEc8liG4q/N5kMCsybGeenr53SFGwm/zWfeNNKCaejo4WvZjtbd6W9lpwFptv/RvCmPFaQMY3z1KcDFg1EkODzuDis6k= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b58f2c18636eb10faa77ed3614abd00cb85dfc2c timestamp: 2022-09-01_15:28:29 tube: smithi user: cbodley verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-09-02T04:05:05.494 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa; will attempt to use it 2022-09-02T04:05:05.495 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa/tasks 2022-09-02T04:05:05.496 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-09-02T04:05:05.499 INFO:teuthology.task.internal:Checking packages... 2022-09-02T04:05:05.530 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'afd422385fab8119a539d289dd1aeaad4c056fe4' 2022-09-02T04:05:05.531 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-09-02T04:05:05.532 INFO:teuthology.packaging:ref: None 2022-09-02T04:05:05.533 INFO:teuthology.packaging:tag: None 2022-09-02T04:05:05.535 INFO:teuthology.packaging:branch: pr-47848 2022-09-02T04:05:05.536 INFO:teuthology.packaging:sha1: afd422385fab8119a539d289dd1aeaad4c056fe4 2022-09-02T04:05:05.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pr-47848 2022-09-02T04:05:05.706 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14689.gafd42238 2022-09-02T04:05:05.707 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-09-02T04:05:05.714 INFO:teuthology.task.internal:no buildpackages task found 2022-09-02T04:05:05.715 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-09-02T04:05:05.741 INFO:teuthology.task.internal:Saving configuration 2022-09-02T04:05:05.765 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-09-02T04:05:05.773 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-02T04:05:05.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-02 03:59:46.399328', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMRj5AnYfezx6xB8YfGzKEfcqeNpRlDhjrttgP2fezU0UmCidHK5CyDS/8OcDgjGZoIid/WKeaO26SNc958uD8='} 2022-09-02T04:05:05.832 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-02 03:59:46.396981', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+CC8GGV4/XtxMyUPPwEIt4E7HNnDbpnTOmppNPf7KzDyJeDHiA3zh2axZGL3iNu8eJcqWlVmzNOrzYg4iA2Ag='} 2022-09-02T04:05:05.833 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-09-02T04:05:05.842 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-09-02T04:05:05.843 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-09-02T04:05:05.843 INFO:teuthology.run_tasks:Running task console_log... 2022-09-02T04:05:05.946 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6549b048c8>, signals=[15]) 2022-09-02T04:05:05.949 INFO:teuthology.run_tasks:Running task internal.connect... 2022-09-02T04:05:05.962 INFO:teuthology.task.internal:Opening connections... 2022-09-02T04:05:05.963 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2022-09-02T04:05:05.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-02T04:05:06.088 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2022-09-02T04:05:06.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-02T04:05:06.189 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-09-02T04:05:06.197 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2022-09-02T04:05:06.220 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2022-09-02T04:05:06.221 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2022-09-02T04:05:06.283 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2022-09-02T04:05:06.283 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2022-09-02T04:05:06.283 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2022-09-02T04:05:06.283 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2022-09-02T04:05:06.284 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2022-09-02T04:05:06.285 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2022-09-02T04:05:06.285 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 8" 2022-09-02T04:05:06.286 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2022-09-02T04:05:06.286 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-09-02T04:05:06.287 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2022-09-02T04:05:06.288 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-02T04:05:06.288 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-02T04:05:06.289 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-09-02T04:05:06.290 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2022-09-02T04:05:06.321 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2022-09-02T04:05:06.341 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2022-09-02T04:05:06.342 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2022-09-02T04:05:06.399 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2022-09-02T04:05:06.400 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="8" 2022-09-02T04:05:06.400 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2022-09-02T04:05:06.400 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2022-09-02T04:05:06.400 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="8" 2022-09-02T04:05:06.400 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el8" 2022-09-02T04:05:06.401 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 8" 2022-09-02T04:05:06.401 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2022-09-02T04:05:06.402 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-09-02T04:05:06.403 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2022-09-02T04:05:06.403 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-02T04:05:06.403 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-02T04:05:06.403 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-09-02T04:05:06.404 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2022-09-02T04:05:06.428 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-09-02T04:05:06.441 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-09-02T04:05:06.454 INFO:teuthology.task.internal:Checking for old test directory... 2022-09-02T04:05:06.455 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2022-09-02T04:05:06.459 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2022-09-02T04:05:06.479 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-09-02T04:05:06.490 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-09-02T04:05:06.491 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2022-09-02T04:05:06.538 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2022-09-02T04:05:06.568 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-09-02T04:05:06.663 INFO:teuthology.run_tasks:Running task kernel... 2022-09-02T04:05:06.679 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-09-02T04:05:06.680 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-09-02T04:05:06.680 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-02T04:05:06.681 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-02T04:05:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T04:05:06.700 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2022-09-02T04:05:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T04:05:06.702 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2022-09-02T04:05:06.758 INFO:teuthology.orchestra.run.smithi160.stdout:4.18.0-408.el8.x86_64 2022-09-02T04:05:06.759 INFO:teuthology.task.kernel:Running kernel on smithi160: 4.18.0-408.el8.x86_64 2022-09-02T04:05:06.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2022-09-02T04:05:06.761 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-408.el8.x86_64 2022-09-02T04:05:06.762 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-408.el8.x86_64 2022-09-02T04:05:06.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2022-09-02T04:05:09.184 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - AppStream 36 MB/s | 25 MB 00:00 2022-09-02T04:05:10.024 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - AppStream 33 MB/s | 25 MB 00:00 2022-09-02T04:05:14.237 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - BaseOS 40 MB/s | 25 MB 00:00 2022-09-02T04:05:14.990 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - BaseOS 39 MB/s | 25 MB 00:00 2022-09-02T04:05:18.084 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - Extras 131 kB/s | 18 kB 00:00 2022-09-02T04:05:18.510 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - PowerTools 16 MB/s | 5.0 MB 00:00 2022-09-02T04:05:18.864 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - Extras 108 kB/s | 18 kB 00:00 2022-09-02T04:05:19.275 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - PowerTools 17 MB/s | 5.0 MB 00:00 2022-09-02T04:05:19.617 INFO:teuthology.orchestra.run.smithi160.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2022-09-02T04:05:20.379 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2022-09-02T04:05:20.534 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 16 MB/s | 13 MB 00:00 2022-09-02T04:05:21.294 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 16 MB/s | 13 MB 00:00 2022-09-02T04:05:23.601 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 453 kB/s | 24 kB 00:00 2022-09-02T04:05:24.360 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 313 kB/s | 24 kB 00:00 2022-09-02T04:05:26.357 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-09-02T04:05:26.358 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-09-02T04:05:26.484 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:05:26.485 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:05:26.485 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:05:26.602 DEBUG:teuthology.orchestra.run.smithi160:> echo no | sudo yum reinstall kernel || true 2022-09-02T04:05:27.182 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:04 ago on Fri 02 Sep 2022 04:05:23 AM UTC. 2022-09-02T04:05:27.197 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-09-02T04:05:27.197 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-09-02T04:05:27.319 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:05:27.319 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:05:27.320 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:05:27.428 DEBUG:teuthology.orchestra.run.smithi099:> echo no | sudo yum reinstall kernel || true 2022-09-02T04:05:28.033 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:04 ago on Fri 02 Sep 2022 04:05:24 AM UTC. 2022-09-02T04:05:28.194 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:05:28.195 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:05:28.195 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2022-09-02T04:05:28.196 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:05:28.196 INFO:teuthology.orchestra.run.smithi160.stdout:Reinstalling: 2022-09-02T04:05:28.196 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-02T04:05:28.197 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:05:28.197 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:05:28.199 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:05:28.199 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:05:28.201 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 8.5 M 2022-09-02T04:05:28.202 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 0 2022-09-02T04:05:28.213 INFO:teuthology.orchestra.run.smithi160.stderr:Operation aborted. 2022-09-02T04:05:28.264 INFO:teuthology.orchestra.run.smithi160.stdout:Is this ok [y/N]: 2022-09-02T04:05:28.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum reinstall -y kernel || true 2022-09-02T04:05:28.839 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:05 ago on Fri 02 Sep 2022 04:05:23 AM UTC. 2022-09-02T04:05:29.039 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:05:29.039 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:05:29.039 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2022-09-02T04:05:29.040 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:05:29.040 INFO:teuthology.orchestra.run.smithi099.stdout:Reinstalling: 2022-09-02T04:05:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-02T04:05:29.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:05:29.040 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:05:29.041 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:05:29.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:05:29.041 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 8.5 M 2022-09-02T04:05:29.050 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 0 2022-09-02T04:05:29.050 INFO:teuthology.orchestra.run.smithi099.stderr:Operation aborted. 2022-09-02T04:05:29.100 INFO:teuthology.orchestra.run.smithi099.stdout:Is this ok [y/N]: 2022-09-02T04:05:29.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum reinstall -y kernel || true 2022-09-02T04:05:29.663 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:05 ago on Fri 02 Sep 2022 04:05:24 AM UTC. 2022-09-02T04:05:29.838 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:05:29.840 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:05:29.841 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2022-09-02T04:05:29.841 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:05:29.841 INFO:teuthology.orchestra.run.smithi160.stdout:Reinstalling: 2022-09-02T04:05:29.842 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-02T04:05:29.842 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:05:29.843 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:05:29.843 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:05:29.843 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:05:29.844 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 8.5 M 2022-09-02T04:05:29.844 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 0 2022-09-02T04:05:29.844 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:05:30.079 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-4.18.0-408.el8.x86_64.rpm 36 MB/s | 8.5 MB 00:00 2022-09-02T04:05:30.079 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:05:30.080 INFO:teuthology.orchestra.run.smithi160.stdout:Total 36 MB/s | 8.5 MB 00:00 2022-09-02T04:05:30.170 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:05:30.547 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:05:30.547 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:05:30.655 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:05:30.656 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:05:30.657 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2022-09-02T04:05:30.657 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:05:30.657 INFO:teuthology.orchestra.run.smithi099.stdout:Reinstalling: 2022-09-02T04:05:30.657 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-09-02T04:05:30.657 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:05:30.658 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:05:30.658 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:05:30.658 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:05:30.659 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 8.5 M 2022-09-02T04:05:30.661 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 0 2022-09-02T04:05:30.662 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:05:30.703 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:05:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:05:30.892 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-408.el8.x86_64.rpm 36 MB/s | 8.5 MB 00:00 2022-09-02T04:05:30.893 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:05:30.893 INFO:teuthology.orchestra.run.smithi099.stdout:Total 36 MB/s | 8.5 MB 00:00 2022-09-02T04:05:30.986 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:05:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:05:31.201 INFO:teuthology.orchestra.run.smithi160.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-02T04:05:31.202 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-02T04:05:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:05:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:05:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:05:31.558 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:05:31.564 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-09-02T04:05:31.564 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-02T04:05:31.845 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:05:32.019 INFO:teuthology.orchestra.run.smithi099.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-02T04:05:32.020 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-02T04:05:32.048 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-02T04:05:32.048 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:05:32.049 INFO:teuthology.orchestra.run.smithi160.stdout:Reinstalled: 2022-09-02T04:05:32.049 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-4.18.0-408.el8.x86_64 2022-09-02T04:05:32.049 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:05:32.049 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:05:32.137 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel --last | head -n 1 2022-09-02T04:05:32.267 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-4.18.0-408.el8.x86_64 Fri 02 Sep 2022 04:05:31 AM UTC 2022-09-02T04:05:32.267 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-09-02T04:05:32.268 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-02T04:05:32.268 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-09-02T04:05:32.269 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-02T04:05:32.302 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2022-09-02T04:05:32.317 DEBUG:teuthology.parallel:result is None 2022-09-02T04:05:32.419 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-09-02T04:05:32.419 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-09-02T04:05:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-09-02T04:05:32.938 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:05:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Reinstalled: 2022-09-02T04:05:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-4.18.0-408.el8.x86_64 2022-09-02T04:05:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:05:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:05:33.018 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel --last | head -n 1 2022-09-02T04:05:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-408.el8.x86_64 Fri 02 Sep 2022 04:05:31 AM UTC 2022-09-02T04:05:33.150 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-09-02T04:05:33.150 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-09-02T04:05:33.150 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-09-02T04:05:33.151 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-09-02T04:05:33.187 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2022-09-02T04:05:33.200 DEBUG:teuthology.parallel:result is None 2022-09-02T04:05:33.200 INFO:teuthology.run_tasks:Running task internal.base... 2022-09-02T04:05:33.213 INFO:teuthology.task.internal:Creating test directory... 2022-09-02T04:05:33.214 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-02T04:05:33.248 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-02T04:05:33.267 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-09-02T04:05:33.275 INFO:teuthology.run_tasks:Running task internal.archive... 2022-09-02T04:05:33.285 INFO:teuthology.task.internal:Creating archive directory... 2022-09-02T04:05:33.286 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-02T04:05:33.308 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-02T04:05:33.352 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-09-02T04:05:33.361 INFO:teuthology.task.internal:Enabling coredump saving... 2022-09-02T04:05:33.362 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-02T04:05:33.373 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-02T04:05:33.401 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-02T04:05:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-02T04:05:33.419 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-02T04:05:33.429 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-02T04:05:33.431 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-09-02T04:05:33.447 INFO:teuthology.task.internal:Configuring sudo... 2022-09-02T04:05:33.448 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-02T04:05:33.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-02T04:05:33.501 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-09-02T04:05:33.517 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-09-02T04:05:33.518 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-02T04:05:33.522 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-02T04:05:33.560 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-02T04:05:33.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-02T04:05:33.679 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-02T04:05:33.767 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-02T04:05:33.809 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:05:33.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-02T04:05:33.877 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-02T04:05:33.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-02T04:05:33.979 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-02T04:05:34.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-02T04:05:34.115 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:05:34.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-02T04:05:34.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2022-09-02T04:05:34.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2022-09-02T04:05:34.255 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-02T04:05:34.289 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-02T04:05:34.712 INFO:teuthology.run_tasks:Running task internal.timer... 2022-09-02T04:05:34.733 INFO:teuthology.task.internal:Starting timer... 2022-09-02T04:05:34.733 INFO:teuthology.run_tasks:Running task pcp... 2022-09-02T04:05:34.745 INFO:teuthology.run_tasks:Running task selinux... 2022-09-02T04:05:34.763 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-09-02T04:05:34.764 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2022-09-02T04:05:34.853 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2022-09-02T04:05:34.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2022-09-02T04:05:34.917 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2022-09-02T04:05:34.921 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-09-02T04:05:34.921 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2022-09-02T04:05:34.956 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2022-09-02T04:05:34.957 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2022-09-02T04:05:34.991 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2022-09-02T04:05:34.992 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2022-09-02T04:05:34.992 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-02T04:05:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T04:05:35.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-02T04:05:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T04:05:35.067 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-09-02T04:05:35.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2022-09-02T04:05:35.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2022-09-02T04:05:35.142 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-09-02T04:05:35.165 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-09-02T04:05:35.166 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-09-02T04:05:35.214 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-09-02T04:05:35.216 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-09-02T04:12:15.284 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2022-09-02T04:12:15.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2022-09-02T04:12:15.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-02T04:12:15.420 DEBUG:teuthology.orchestra.run.smithi099:> true 2022-09-02T04:12:15.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2022-09-02T04:12:15.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-02T04:12:15.583 DEBUG:teuthology.orchestra.run.smithi160:> true 2022-09-02T04:12:15.651 INFO:teuthology.run_tasks:Running task clock... 2022-09-02T04:12:15.665 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-09-02T04:12:15.666 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-02T04:12:15.668 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-02T04:12:15.671 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-02T04:12:15.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-02T04:12:15.702 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-02T04:12:15.716 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-02T04:12:15.723 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-02T04:12:15.737 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-02T04:12:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2022-09-02T04:12:15.784 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2022-09-02T04:12:15.798 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-02T04:12:15.811 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-02T04:12:15.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2022-09-02T04:12:15.828 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2022-09-02T04:12:15.840 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-02T04:12:15.854 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-02T04:12:15.887 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2022-09-02T04:12:15.891 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-02T04:12:15.892 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2022-09-02T04:12:15.892 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.893 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.893 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.894 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.933 INFO:teuthology.orchestra.run.smithi160.stderr:bash: ntpq: command not found 2022-09-02T04:12:15.937 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-02T04:12:15.937 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2022-09-02T04:12:15.938 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.938 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.938 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.938 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-02T04:12:15.939 INFO:teuthology.run_tasks:Running task install... 2022-09-02T04:12:15.953 DEBUG:teuthology.task.install:project ceph 2022-09-02T04:12:15.954 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4'}} 2022-09-02T04:12:15.955 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4'} 2022-09-02T04:12:15.955 INFO:teuthology.task.install:Using flavor: default 2022-09-02T04:12:15.963 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-09-02T04:12:15.964 INFO:teuthology.task.install:extra packages: [] 2022-09-02T04:12:15.964 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2022-09-02T04:12:16.001 INFO:teuthology.orchestra.run.smithi099.stdout:CentOSStream 2022-09-02T04:12:16.003 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-02T04:12:16.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo lsb_release -is 2022-09-02T04:12:16.005 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'tag': None, 'wait_for_package': False} 2022-09-02T04:12:16.005 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2022-09-02T04:12:16.043 INFO:teuthology.orchestra.run.smithi160.stdout:CentOSStream 2022-09-02T04:12:16.045 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-02T04:12:16.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'tag': None, 'wait_for_package': False} 2022-09-02T04:12:16.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo lsb_release -is 2022-09-02T04:12:16.078 INFO:teuthology.orchestra.run.smithi099.stdout:CentOSStream 2022-09-02T04:12:16.080 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-02T04:12:16.081 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=afd422385fab8119a539d289dd1aeaad4c056fe4 2022-09-02T04:12:16.124 INFO:teuthology.orchestra.run.smithi160.stdout:CentOSStream 2022-09-02T04:12:16.126 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-02T04:12:16.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=afd422385fab8119a539d289dd1aeaad4c056fe4 2022-09-02T04:12:16.242 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/ 2022-09-02T04:12:16.243 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14689.gafd42238 2022-09-02T04:12:16.279 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/ 2022-09-02T04:12:16.280 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14689.gafd42238 2022-09-02T04:12:16.388 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-02T04:12:16.389 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:12:16.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-02T04:12:16.403 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/pr-47848/afd422385fab8119a539d289dd1aeaad4c056fe4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-02T04:12:16.404 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:12:16.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-02T04:12:16.417 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-09-02T04:12:16.418 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/afd422385fab8119a539d289dd1aeaad4c056fe4/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-02T04:12:16.433 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-09-02T04:12:16.434 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/afd422385fab8119a539d289dd1aeaad4c056fe4/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-02T04:12:16.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-02T04:12:16.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-02T04:12:16.563 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-02T04:12:16.579 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-02T04:12:16.633 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2022-09-02T04:12:16.633 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2022-09-02T04:12:16.647 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2022-09-02T04:12:16.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2022-09-02T04:12:16.881 INFO:teuthology.orchestra.run.smithi099.stdout:63 files removed 2022-09-02T04:12:16.900 INFO:teuthology.orchestra.run.smithi160.stdout:63 files removed 2022-09-02T04:12:16.906 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2022-09-02T04:12:16.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2022-09-02T04:12:17.508 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 314 kB/s | 80 kB 00:00 2022-09-02T04:12:17.528 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 316 kB/s | 80 kB 00:00 2022-09-02T04:12:17.800 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 37 kB/s | 9.4 kB 00:00 2022-09-02T04:12:17.803 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 39 kB/s | 9.4 kB 00:00 2022-09-02T04:12:18.060 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 6.5 kB/s | 1.5 kB 00:00 2022-09-02T04:12:18.079 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2022-09-02T04:12:18.903 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - AppStream 30 MB/s | 25 MB 00:00 2022-09-02T04:12:18.948 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - AppStream 29 MB/s | 25 MB 00:00 2022-09-02T04:12:24.119 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - BaseOS 30 MB/s | 25 MB 00:00 2022-09-02T04:12:24.190 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - BaseOS 29 MB/s | 25 MB 00:00 2022-09-02T04:12:27.968 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - Extras 179 kB/s | 18 kB 00:00 2022-09-02T04:12:28.072 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - Extras 154 kB/s | 18 kB 00:00 2022-09-02T04:12:28.387 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - PowerTools 17 MB/s | 5.0 MB 00:00 2022-09-02T04:12:28.466 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-8 - PowerTools 18 MB/s | 5.0 MB 00:00 2022-09-02T04:12:29.469 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2022-09-02T04:12:29.540 INFO:teuthology.orchestra.run.smithi160.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2022-09-02T04:12:30.545 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:00 2022-09-02T04:12:30.624 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:00 2022-09-02T04:12:33.578 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 506 kB/s | 24 kB 00:00 2022-09-02T04:12:33.708 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 520 kB/s | 24 kB 00:00 2022-09-02T04:12:36.685 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:12:36.687 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:12:36.687 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:12:36.688 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:12:36.688 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:12:36.688 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:17.0.0-14689.gafd42238.el8 ceph 17 M 2022-09-02T04:12:36.689 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2022-09-02T04:12:36.689 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.6 M 2022-09-02T04:12:36.689 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.6 M 2022-09-02T04:12:36.689 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:12:36.690 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:17.0.0-14689.gafd42238.el8 ceph 5.2 M 2022-09-02T04:12:36.690 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:17.0.0-14689.gafd42238.el8 ceph 24 M 2022-09-02T04:12:36.690 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:17.0.0-14689.gafd42238.el8 ceph 24 k 2022-09-02T04:12:36.691 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-02T04:12:36.691 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 766 k 2022-09-02T04:12:36.691 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-02T04:12:36.691 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-09-02T04:12:36.691 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 473 k 2022-09-02T04:12:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-09-02T04:12:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 5.7 M 2022-09-02T04:12:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-02T04:12:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-09-02T04:12:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse 2022-09-02T04:12:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: x86_64 2:17.0.0-14689.gafd42238.el8 ceph 45 k 2022-09-02T04:12:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:17.0.0-14689.gafd42238.el8 ceph 109 k 2022-09-02T04:12:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:17.0.0-14689.gafd42238.el8 ceph 190 k 2022-09-02T04:12:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-09-02T04:12:36.694 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:17.0.0-14689.gafd42238.el8 ceph 371 k 2022-09-02T04:12:36.694 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:17.0.0-14689.gafd42238.el8 ceph 357 k 2022-09-02T04:12:36.694 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:17.0.0-14689.gafd42238.el8 ceph 107 k 2022-09-02T04:12:36.694 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-02T04:12:36.695 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:12:36.695 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:12:36.695 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:12:36.695 INFO:teuthology.orchestra.run.smithi099.stdout:Install 21 Packages 2022-09-02T04:12:36.696 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2022-09-02T04:12:36.696 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:12:36.697 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 65 M 2022-09-02T04:12:36.697 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:12:36.816 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:12:36.819 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:12:36.819 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:12:36.819 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:12:36.819 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:12:36.820 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:17.0.0-14689.gafd42238.el8 ceph 17 M 2022-09-02T04:12:36.820 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2022-09-02T04:12:36.820 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.6 M 2022-09-02T04:12:36.820 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.6 M 2022-09-02T04:12:36.821 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:12:36.821 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:17.0.0-14689.gafd42238.el8 ceph 5.2 M 2022-09-02T04:12:36.821 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:17.0.0-14689.gafd42238.el8 ceph 24 M 2022-09-02T04:12:36.821 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:17.0.0-14689.gafd42238.el8 ceph 24 k 2022-09-02T04:12:36.822 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-02T04:12:36.822 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 766 k 2022-09-02T04:12:36.822 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-02T04:12:36.822 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2022-09-02T04:12:36.822 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 473 k 2022-09-02T04:12:36.823 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-09-02T04:12:36.823 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:17.0.0-14689.gafd42238.el8 ceph 5.7 M 2022-09-02T04:12:36.823 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-02T04:12:36.823 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-09-02T04:12:36.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse 2022-09-02T04:12:36.824 INFO:teuthology.orchestra.run.smithi160.stdout: x86_64 2:17.0.0-14689.gafd42238.el8 ceph 45 k 2022-09-02T04:12:36.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:17.0.0-14689.gafd42238.el8 ceph 109 k 2022-09-02T04:12:36.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:17.0.0-14689.gafd42238.el8 ceph 190 k 2022-09-02T04:12:36.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-09-02T04:12:36.825 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:17.0.0-14689.gafd42238.el8 ceph 371 k 2022-09-02T04:12:36.825 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:17.0.0-14689.gafd42238.el8 ceph 357 k 2022-09-02T04:12:36.825 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:17.0.0-14689.gafd42238.el8 ceph 107 k 2022-09-02T04:12:36.825 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-02T04:12:36.826 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:12:36.826 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:12:36.826 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:12:36.826 INFO:teuthology.orchestra.run.smithi160.stdout:Install 21 Packages 2022-09-02T04:12:36.827 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2022-09-02T04:12:36.827 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:12:36.827 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 65 M 2022-09-02T04:12:36.828 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:12:38.017 INFO:teuthology.orchestra.run.smithi160.stdout:(1/23): ceph-base-17.0.0-14689.gafd42238.el8.x8 4.3 MB/s | 5.2 MB 00:01 2022-09-02T04:12:38.093 INFO:teuthology.orchestra.run.smithi160.stdout:(2/23): ceph-selinux-17.0.0-14689.gafd42238.el8 323 kB/s | 24 kB 00:00 2022-09-02T04:12:38.302 INFO:teuthology.orchestra.run.smithi160.stdout:(3/23): libcephfs2-17.0.0-14689.gafd42238.el8.x 3.6 MB/s | 766 kB 00:00 2022-09-02T04:12:38.468 INFO:teuthology.orchestra.run.smithi160.stdout:(4/23): libradosstriper1-17.0.0-14689.gafd42238 2.8 MB/s | 473 kB 00:00 2022-09-02T04:12:38.637 INFO:teuthology.orchestra.run.smithi099.stdout:(1/23): ceph-base-17.0.0-14689.gafd42238.el8.x8 2.7 MB/s | 5.2 MB 00:01 2022-09-02T04:12:38.746 INFO:teuthology.orchestra.run.smithi099.stdout:(2/23): ceph-selinux-17.0.0-14689.gafd42238.el8 224 kB/s | 24 kB 00:00 2022-09-02T04:12:39.660 INFO:teuthology.orchestra.run.smithi160.stdout:(5/23): librgw2-17.0.0-14689.gafd42238.el8.x86_ 4.8 MB/s | 5.7 MB 00:01 2022-09-02T04:12:39.788 INFO:teuthology.orchestra.run.smithi099.stdout:(3/23): libcephfs2-17.0.0-14689.gafd42238.el8.x 735 kB/s | 766 kB 00:01 2022-09-02T04:12:39.868 INFO:teuthology.orchestra.run.smithi160.stdout:(6/23): ceph-radosgw-17.0.0-14689.gafd42238.el8 5.7 MB/s | 17 MB 00:03 2022-09-02T04:12:39.894 INFO:teuthology.orchestra.run.smithi160.stdout:(7/23): python3-ceph-argparse-17.0.0-14689.gafd 191 kB/s | 45 kB 00:00 2022-09-02T04:12:39.935 INFO:teuthology.orchestra.run.smithi160.stdout:(8/23): python3-ceph-common-17.0.0-14689.gafd42 1.6 MB/s | 109 kB 00:00 2022-09-02T04:12:39.969 INFO:teuthology.orchestra.run.smithi160.stdout:(9/23): python3-cephfs-17.0.0-14689.gafd42238.e 2.5 MB/s | 190 kB 00:00 2022-09-02T04:12:40.019 INFO:teuthology.orchestra.run.smithi160.stdout:(10/23): python3-rados-17.0.0-14689.gafd42238.e 4.3 MB/s | 371 kB 00:00 2022-09-02T04:12:40.061 INFO:teuthology.orchestra.run.smithi160.stdout:(11/23): python3-rbd-17.0.0-14689.gafd42238.el8 3.8 MB/s | 357 kB 00:00 2022-09-02T04:12:40.088 INFO:teuthology.orchestra.run.smithi099.stdout:(4/23): libradosstriper1-17.0.0-14689.gafd42238 1.5 MB/s | 473 kB 00:00 2022-09-02T04:12:40.095 INFO:teuthology.orchestra.run.smithi160.stdout:(12/23): python3-rgw-17.0.0-14689.gafd42238.el8 1.4 MB/s | 107 kB 00:00 2022-09-02T04:12:40.137 INFO:teuthology.orchestra.run.smithi160.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2022-09-02T04:12:40.162 INFO:teuthology.orchestra.run.smithi160.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2022-09-02T04:12:40.209 INFO:teuthology.orchestra.run.smithi160.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.0 MB/s | 47 kB 00:00 2022-09-02T04:12:40.238 INFO:teuthology.orchestra.run.smithi160.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2022-09-02T04:12:40.505 INFO:teuthology.orchestra.run.smithi160.stdout:(17/23): ceph-common-17.0.0-14689.gafd42238.el8 6.5 MB/s | 24 MB 00:03 2022-09-02T04:12:40.530 INFO:teuthology.orchestra.run.smithi160.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 201 kB/s | 59 kB 00:00 2022-09-02T04:12:40.580 INFO:teuthology.orchestra.run.smithi160.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 824 kB/s | 306 kB 00:00 2022-09-02T04:12:40.640 INFO:teuthology.orchestra.run.smithi160.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 562 kB/s | 75 kB 00:00 2022-09-02T04:12:40.748 INFO:teuthology.orchestra.run.smithi099.stdout:(5/23): ceph-radosgw-17.0.0-14689.gafd42238.el8 4.3 MB/s | 17 MB 00:04 2022-09-02T04:12:40.781 INFO:teuthology.orchestra.run.smithi160.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2022-09-02T04:12:40.830 INFO:teuthology.orchestra.run.smithi099.stdout:(6/23): python3-ceph-argparse-17.0.0-14689.gafd 545 kB/s | 45 kB 00:00 2022-09-02T04:12:40.905 INFO:teuthology.orchestra.run.smithi099.stdout:(7/23): python3-ceph-common-17.0.0-14689.gafd42 1.4 MB/s | 109 kB 00:00 2022-09-02T04:12:41.031 INFO:teuthology.orchestra.run.smithi099.stdout:(8/23): python3-cephfs-17.0.0-14689.gafd42238.e 1.5 MB/s | 190 kB 00:00 2022-09-02T04:12:41.039 INFO:teuthology.orchestra.run.smithi160.stdout:(22/23): librbd1-17.0.0-14689.gafd42238.el8.x86 8.9 MB/s | 3.6 MB 00:00 2022-09-02T04:12:41.106 INFO:teuthology.orchestra.run.smithi160.stdout:(23/23): librados2-17.0.0-14689.gafd42238.el8.x 6.9 MB/s | 3.6 MB 00:00 2022-09-02T04:12:41.110 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:12:41.110 INFO:teuthology.orchestra.run.smithi160.stdout:Total 15 MB/s | 65 MB 00:04 2022-09-02T04:12:41.114 INFO:teuthology.orchestra.run.smithi099.stdout:(9/23): python3-rados-17.0.0-14689.gafd42238.el 4.4 MB/s | 371 kB 00:00 2022-09-02T04:12:41.152 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:12:41.198 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:12:41.198 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:12:41.199 INFO:teuthology.orchestra.run.smithi099.stdout:(10/23): python3-rbd-17.0.0-14689.gafd42238.el8 4.2 MB/s | 357 kB 00:00 2022-09-02T04:12:41.248 INFO:teuthology.orchestra.run.smithi099.stdout:(11/23): python3-rgw-17.0.0-14689.gafd42238.el8 2.1 MB/s | 107 kB 00:00 2022-09-02T04:12:41.507 INFO:teuthology.orchestra.run.smithi099.stdout:(12/23): ceph-common-17.0.0-14689.gafd42238.el8 4.9 MB/s | 24 MB 00:04 2022-09-02T04:12:41.574 INFO:teuthology.orchestra.run.smithi099.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 MB/s | 354 kB 00:00 2022-09-02T04:12:41.591 INFO:teuthology.orchestra.run.smithi099.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 2.6 MB/s | 44 kB 00:00 2022-09-02T04:12:41.608 INFO:teuthology.orchestra.run.smithi099.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.7 MB/s | 47 kB 00:00 2022-09-02T04:12:41.633 INFO:teuthology.orchestra.run.smithi099.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2022-09-02T04:12:41.692 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:12:41.695 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:12:41.708 INFO:teuthology.orchestra.run.smithi099.stdout:(17/23): librgw2-17.0.0-14689.gafd42238.el8.x86 3.5 MB/s | 5.7 MB 00:01 2022-09-02T04:12:41.842 INFO:teuthology.orchestra.run.smithi099.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2022-09-02T04:12:41.860 INFO:teuthology.orchestra.run.smithi099.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 499 kB/s | 75 kB 00:00 2022-09-02T04:12:41.918 INFO:teuthology.orchestra.run.smithi099.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 990 kB/s | 306 kB 00:00 2022-09-02T04:12:42.101 INFO:teuthology.orchestra.run.smithi099.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2022-09-02T04:12:42.301 INFO:teuthology.orchestra.run.smithi099.stdout:(22/23): librbd1-17.0.0-14689.gafd42238.el8.x86 9.3 MB/s | 3.6 MB 00:00 2022-09-02T04:12:42.468 INFO:teuthology.orchestra.run.smithi099.stdout:(23/23): librados2-17.0.0-14689.gafd42238.el8.x 5.9 MB/s | 3.6 MB 00:00 2022-09-02T04:12:42.472 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:12:42.472 INFO:teuthology.orchestra.run.smithi099.stdout:Total 11 MB/s | 65 MB 00:05 2022-09-02T04:12:42.510 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:12:42.554 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:12:42.555 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:12:42.878 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:12:43.035 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-09-02T04:12:43.070 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:12:43.073 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:12:43.240 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-09-02T04:12:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-02T04:12:43.837 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-02T04:12:44.070 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:17.0.0-14689.gafd42238.el8.x86_64 4/25 2022-09-02T04:12:44.155 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:17.0.0-14689.gafd42238.el8.x86_64 4/25 2022-09-02T04:12:44.322 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:12:44.393 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 5/25 2022-09-02T04:12:44.485 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2022-09-02T04:12:44.587 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 6/25 2022-09-02T04:12:44.709 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2022-09-02T04:12:44.887 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-02T04:12:44.961 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 6/25 2022-09-02T04:12:45.170 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 7/25 2022-09-02T04:12:45.236 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 7/25 2022-09-02T04:12:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2022-09-02T04:12:45.443 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-09-02T04:12:45.504 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:17.0.0-14689.gafd42238.el8.x86_64 4/25 2022-09-02T04:12:45.612 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-02T04:12:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:17.0.0-14689.gafd42238.el8.x86_64 4/25 2022-09-02T04:12:45.681 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-02T04:12:45.805 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:17.0.0-14689.gafd42238.el8 10/25 2022-09-02T04:12:45.877 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 5/25 2022-09-02T04:12:46.030 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 11/25 2022-09-02T04:12:46.071 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 6/25 2022-09-02T04:12:46.262 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 12/25 2022-09-02T04:12:46.429 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_ 13/25 2022-09-02T04:12:46.470 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 6/25 2022-09-02T04:12:46.504 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_ 13/25 2022-09-02T04:12:46.678 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 7/25 2022-09-02T04:12:46.725 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-09-02T04:12:46.731 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 7/25 2022-09-02T04:12:46.960 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-09-02T04:12:47.095 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-02T04:12:47.142 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-09-02T04:12:47.314 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:17.0.0-14689.gafd42238.el8 10/25 2022-09-02T04:12:47.458 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-09-02T04:12:47.539 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 11/25 2022-09-02T04:12:47.637 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 16/25 2022-09-02T04:12:47.692 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 16/25 2022-09-02T04:12:47.796 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 12/25 2022-09-02T04:12:47.919 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 17/25 2022-09-02T04:12:47.995 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_ 13/25 2022-09-02T04:12:48.047 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_ 13/25 2022-09-02T04:12:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-09-02T04:12:48.278 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:17.0.0-14689.gafd42238.el8.x 19/25 2022-09-02T04:12:48.299 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2022-09-02T04:12:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2022-09-02T04:12:49.278 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 16/25 2022-09-02T04:12:49.353 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 16/25 2022-09-02T04:12:49.528 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 17/25 2022-09-02T04:12:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-09-02T04:12:49.903 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:17.0.0-14689.gafd42238.el8.x 19/25 2022-09-02T04:12:51.997 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 20/25 2022-09-02T04:12:52.278 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 20/25 2022-09-02T04:12:52.918 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 20/25 2022-09-02T04:12:53.153 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 21/25 2022-09-02T04:12:53.280 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 21/25 2022-09-02T04:12:53.427 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 22/25 2022-09-02T04:12:53.737 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 20/25 2022-09-02T04:12:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 20/25 2022-09-02T04:12:54.784 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 20/25 2022-09-02T04:12:55.028 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 21/25 2022-09-02T04:12:55.164 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 21/25 2022-09-02T04:12:55.302 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 22/25 2022-09-02T04:13:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 22/25 2022-09-02T04:13:11.769 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2022-09-02T04:13:11.770 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2022-09-02T04:13:11.770 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /dev 2022-09-02T04:13:11.771 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /run 2022-09-02T04:13:11.771 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2022-09-02T04:13:11.772 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2022-09-02T04:13:11.772 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2022-09-02T04:13:11.773 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2022-09-02T04:13:11.773 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /dev 2022-09-02T04:13:11.774 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 23/25 2022-09-02T04:13:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 23/25 2022-09-02T04:13:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-02T04:13:13.735 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-02T04:13:13.736 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-02T04:13:13.879 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 22/25 2022-09-02T04:13:13.879 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2022-09-02T04:13:13.880 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2022-09-02T04:13:13.880 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2022-09-02T04:13:13.880 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /run 2022-09-02T04:13:13.880 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2022-09-02T04:13:13.881 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2022-09-02T04:13:13.881 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2022-09-02T04:13:13.881 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2022-09-02T04:13:13.881 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2022-09-02T04:13:13.881 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-02T04:13:14.402 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 1/25 2022-09-02T04:13:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 2/25 2022-09-02T04:13:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 3/25 2022-09-02T04:13:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 4/25 2022-09-02T04:13:14.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 5/25 2022-09-02T04:13:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_ 6/25 2022-09-02T04:13:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 7/25 2022-09-02T04:13:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14689.gafd42238.el8 8/25 2022-09-02T04:13:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:17.0.0-14689.gafd42238.el8.x 9/25 2022-09-02T04:13:14.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 10/25 2022-09-02T04:13:14.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 11/25 2022-09-02T04:13:14.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 12/25 2022-09-02T04:13:14.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 13/25 2022-09-02T04:13:14.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-09-02T04:13:14.406 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-09-02T04:13:14.406 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-09-02T04:13:14.406 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-09-02T04:13:14.406 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-09-02T04:13:14.407 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-09-02T04:13:14.407 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-09-02T04:13:14.407 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-09-02T04:13:14.407 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:17.0.0-14689.gafd42238.el8.x86_64 22/25 2022-09-02T04:13:14.408 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-09-02T04:13:14.408 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 24/25 2022-09-02T04:13:15.225 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-09-02T04:13:15.225 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:15.226 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2022-09-02T04:13:15.226 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.226 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.226 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:13:15.227 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.227 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.227 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.227 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.228 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-02T04:13:15.228 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.228 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-02T04:13:15.228 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-02T04:13:15.229 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.229 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-02T04:13:15.229 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.229 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-02T04:13:15.229 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-02T04:13:15.230 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.230 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.230 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.230 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-02T04:13:15.231 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.231 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.231 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:15.231 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-02T04:13:15.232 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:15.232 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:15.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2022-09-02T04:13:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 23/25 2022-09-02T04:13:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 23/25 2022-09-02T04:13:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-02T04:13:15.858 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-09-02T04:13:15.858 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-02T04:13:16.048 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:43 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:16.521 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-09-02T04:13:16.522 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 1/25 2022-09-02T04:13:16.522 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 2/25 2022-09-02T04:13:16.523 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 3/25 2022-09-02T04:13:16.523 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 4/25 2022-09-02T04:13:16.523 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 5/25 2022-09-02T04:13:16.524 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_ 6/25 2022-09-02T04:13:16.524 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 7/25 2022-09-02T04:13:16.525 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14689.gafd42238.el8 8/25 2022-09-02T04:13:16.525 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:17.0.0-14689.gafd42238.el8.x 9/25 2022-09-02T04:13:16.526 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 10/25 2022-09-02T04:13:16.526 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 11/25 2022-09-02T04:13:16.527 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 12/25 2022-09-02T04:13:16.527 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 13/25 2022-09-02T04:13:16.527 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-09-02T04:13:16.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-09-02T04:13:16.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-09-02T04:13:16.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-09-02T04:13:16.529 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-09-02T04:13:16.529 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-09-02T04:13:16.529 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-09-02T04:13:16.530 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-09-02T04:13:16.530 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:17.0.0-14689.gafd42238.el8.x86_64 22/25 2022-09-02T04:13:16.530 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-09-02T04:13:16.531 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 24/25 2022-09-02T04:13:17.056 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:13:17.058 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:17.058 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:13:17.059 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:17.059 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:13:17.060 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:17.0.0-14689.gafd42238.el8 ceph 53 M 2022-09-02T04:13:17.060 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:13:17.061 INFO:teuthology.orchestra.run.smithi160.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-09-02T04:13:17.061 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:17.0.0-14689.gafd42238.el8 ceph 137 k 2022-09-02T04:13:17.062 INFO:teuthology.orchestra.run.smithi160.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-09-02T04:13:17.062 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-09-02T04:13:17.063 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-PowerTools 71 k 2022-09-02T04:13:17.063 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:17.064 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:13:17.064 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:17.065 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2022-09-02T04:13:17.065 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:17.066 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 54 M 2022-09-02T04:13:17.066 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 235 M 2022-09-02T04:13:17.067 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:13:17.123 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 3.1 MB/s | 202 kB 00:00 2022-09-02T04:13:17.156 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2022-09-02T04:13:17.182 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2022-09-02T04:13:17.207 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.9 MB/s | 71 kB 00:00 2022-09-02T04:13:17.333 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-09-02T04:13:17.333 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:17.334 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2022-09-02T04:13:17.334 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.335 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.335 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:13:17.336 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.336 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.337 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.337 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.338 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-02T04:13:17.338 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.339 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-02T04:13:17.339 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-02T04:13:17.340 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.340 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-02T04:13:17.341 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.341 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-02T04:13:17.341 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-02T04:13:17.342 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.342 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.343 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.344 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-02T04:13:17.344 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.345 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.345 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:17.345 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-02T04:13:17.345 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:17.346 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:13:17.449 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): libcephsqlite-17.0.0-14689.gafd42238.el8 353 kB/s | 137 kB 00:00 2022-09-02T04:13:17.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2022-09-02T04:13:18.111 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:45 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:19.124 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:13:19.125 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:19.126 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:13:19.126 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:19.126 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:13:19.126 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:17.0.0-14689.gafd42238.el8 ceph 53 M 2022-09-02T04:13:19.127 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:13:19.127 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-09-02T04:13:19.127 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:17.0.0-14689.gafd42238.el8 ceph 137 k 2022-09-02T04:13:19.127 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-09-02T04:13:19.128 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-09-02T04:13:19.128 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-PowerTools 71 k 2022-09-02T04:13:19.128 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:19.128 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:13:19.129 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:19.129 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2022-09-02T04:13:19.129 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:19.130 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 54 M 2022-09-02T04:13:19.130 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 235 M 2022-09-02T04:13:19.130 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:13:19.207 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.5 MB/s | 202 kB 00:00 2022-09-02T04:13:19.241 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2022-09-02T04:13:19.275 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2022-09-02T04:13:19.300 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 71 kB 00:00 2022-09-02T04:13:19.700 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): libcephsqlite-17.0.0-14689.gafd42238.el8 240 kB/s | 137 kB 00:00 2022-09-02T04:13:24.294 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): ceph-test-17.0.0-14689.gafd42238.el8.x86 7.4 MB/s | 53 MB 00:07 2022-09-02T04:13:24.295 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:24.295 INFO:teuthology.orchestra.run.smithi160.stdout:Total 7.5 MB/s | 54 MB 00:07 2022-09-02T04:13:24.335 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:13:24.356 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:13:24.357 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:13:24.660 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:13:24.663 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:13:25.155 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:13:25.342 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-09-02T04:13:25.551 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-09-02T04:13:25.714 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-02T04:13:25.793 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-02T04:13:25.977 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-09-02T04:13:26.114 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 5/6 2022-09-02T04:13:26.567 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-test-17.0.0-14689.gafd42238.el8.x86 7.2 MB/s | 53 MB 00:07 2022-09-02T04:13:26.570 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:26.570 INFO:teuthology.orchestra.run.smithi099.stdout:Total 7.3 MB/s | 54 MB 00:07 2022-09-02T04:13:26.610 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:13:26.631 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:13:26.631 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:13:26.976 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:13:26.978 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:13:27.441 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:13:27.673 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2022-09-02T04:13:27.866 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-09-02T04:13:28.053 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-02T04:13:28.136 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-09-02T04:13:28.349 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-09-02T04:13:28.494 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 5/6 2022-09-02T04:13:31.207 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 5/6 2022-09-02T04:13:31.419 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 6/6 2022-09-02T04:13:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 6/6 2022-09-02T04:13:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 1/6 2022-09-02T04:13:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 2/6 2022-09-02T04:13:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-09-02T04:13:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-09-02T04:13:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-09-02T04:13:32.104 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-09-02T04:13:32.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:32.105 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:13:32.105 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:32.106 INFO:teuthology.orchestra.run.smithi160.stdout: jq-1.6-3.el8.x86_64 2022-09-02T04:13:32.106 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:32.106 INFO:teuthology.orchestra.run.smithi160.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-02T04:13:32.106 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-02T04:13:32.106 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-02T04:13:32.107 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:32.107 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:32.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2022-09-02T04:13:32.901 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:59 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:33.604 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 5/6 2022-09-02T04:13:33.832 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 6/6 2022-09-02T04:13:33.886 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:13:33.889 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================== 2022-09-02T04:13:33.889 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:13:33.889 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================== 2022-09-02T04:13:33.890 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:13:33.890 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:17.0.0-14689.gafd42238.el8 ceph 6.4 k 2022-09-02T04:13:33.890 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:13:33.890 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:17.0.0-14689.gafd42238.el8 ceph 2.3 M 2022-09-02T04:13:33.891 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:17.0.0-14689.gafd42238.el8 ceph 1.5 M 2022-09-02T04:13:33.891 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 233 k 2022-09-02T04:13:33.891 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.9 M 2022-09-02T04:13:33.891 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:17.0.0-14689.gafd42238.el8 ceph 17 M 2022-09-02T04:13:33.892 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-02T04:13:33.892 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-02T04:13:33.893 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-09-02T04:13:33.893 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-02T04:13:33.893 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-02T04:13:33.894 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-09-02T04:13:33.894 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-02T04:13:33.895 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-02T04:13:33.895 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-02T04:13:33.896 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-09-02T04:13:33.896 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-09-02T04:13:33.897 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-02T04:13:33.897 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-02T04:13:33.897 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-02T04:13:33.897 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-09-02T04:13:33.898 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-09-02T04:13:33.898 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-09-02T04:13:33.898 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-02T04:13:33.898 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-02T04:13:33.899 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-02T04:13:33.899 INFO:teuthology.orchestra.run.smithi160.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-02T04:13:33.899 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-02T04:13:33.900 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-02T04:13:33.900 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-02T04:13:33.900 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-09-02T04:13:33.900 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-02T04:13:33.900 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:33.901 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:13:33.901 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================== 2022-09-02T04:13:33.901 INFO:teuthology.orchestra.run.smithi160.stdout:Install 32 Packages 2022-09-02T04:13:33.901 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:33.903 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 28 M 2022-09-02T04:13:33.903 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 110 M 2022-09-02T04:13:33.904 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:13:34.031 INFO:teuthology.orchestra.run.smithi160.stdout:(1/32): ceph-17.0.0-14689.gafd42238.el8.x86_64. 47 kB/s | 6.4 kB 00:00 2022-09-02T04:13:34.086 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 6/6 2022-09-02T04:13:34.087 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 1/6 2022-09-02T04:13:34.087 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 2/6 2022-09-02T04:13:34.087 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-09-02T04:13:34.088 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-09-02T04:13:34.088 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-09-02T04:13:34.223 INFO:teuthology.orchestra.run.smithi160.stdout:(2/32): ceph-mgr-17.0.0-14689.gafd42238.el8.x86 4.7 MB/s | 1.5 MB 00:00 2022-09-02T04:13:34.292 INFO:teuthology.orchestra.run.smithi160.stdout:(3/32): ceph-mds-17.0.0-14689.gafd42238.el8.x86 5.7 MB/s | 2.3 MB 00:00 2022-09-02T04:13:34.357 INFO:teuthology.orchestra.run.smithi160.stdout:(4/32): ceph-mgr-modules-core-17.0.0-14689.gafd 3.4 MB/s | 233 kB 00:00 2022-09-02T04:13:34.432 INFO:teuthology.orchestra.run.smithi160.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2022-09-02T04:13:34.466 INFO:teuthology.orchestra.run.smithi160.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.1 MB/s | 39 kB 00:00 2022-09-02T04:13:34.512 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2022-09-02T04:13:34.513 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:34.513 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:13:34.513 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:34.513 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-3.el8.x86_64 2022-09-02T04:13:34.514 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:34.514 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-02T04:13:34.514 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-02T04:13:34.514 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-02T04:13:34.515 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:34.515 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:13:34.516 INFO:teuthology.orchestra.run.smithi160.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2022-09-02T04:13:34.583 INFO:teuthology.orchestra.run.smithi160.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 808 kB/s | 54 kB 00:00 2022-09-02T04:13:34.642 INFO:teuthology.orchestra.run.smithi160.stdout:(9/32): ceph-mon-17.0.0-14689.gafd42238.el8.x86 6.3 MB/s | 3.9 MB 00:00 2022-09-02T04:13:34.676 INFO:teuthology.orchestra.run.smithi160.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.9 MB/s | 457 kB 00:00 2022-09-02T04:13:34.751 INFO:teuthology.orchestra.run.smithi160.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 7.3 MB/s | 559 kB 00:00 2022-09-02T04:13:34.758 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2022-09-02T04:13:34.784 INFO:teuthology.orchestra.run.smithi160.stdout:(12/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.6 MB/s | 237 kB 00:00 2022-09-02T04:13:34.810 INFO:teuthology.orchestra.run.smithi160.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2022-09-02T04:13:34.918 INFO:teuthology.orchestra.run.smithi160.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 326 kB/s | 44 kB 00:00 2022-09-02T04:13:34.969 INFO:teuthology.orchestra.run.smithi160.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2022-09-02T04:13:35.027 INFO:teuthology.orchestra.run.smithi160.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2022-09-02T04:13:35.053 INFO:teuthology.orchestra.run.smithi160.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2022-09-02T04:13:35.078 INFO:teuthology.orchestra.run.smithi160.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2022-09-02T04:13:35.103 INFO:teuthology.orchestra.run.smithi160.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 349 kB/s | 18 kB 00:00 2022-09-02T04:13:35.254 INFO:teuthology.orchestra.run.smithi160.stdout:(20/32): ceph-osd-17.0.0-14689.gafd42238.el8.x8 16 MB/s | 17 MB 00:01 2022-09-02T04:13:35.279 INFO:teuthology.orchestra.run.smithi160.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 243 kB/s | 49 kB 00:00 2022-09-02T04:13:35.304 INFO:teuthology.orchestra.run.smithi160.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 293 kB/s | 59 kB 00:00 2022-09-02T04:13:35.334 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:02 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:35.354 INFO:teuthology.orchestra.run.smithi160.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 377 kB/s | 19 kB 00:00 2022-09-02T04:13:35.379 INFO:teuthology.orchestra.run.smithi160.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2022-09-02T04:13:35.396 INFO:teuthology.orchestra.run.smithi160.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 571 kB/s | 24 kB 00:00 2022-09-02T04:13:35.422 INFO:teuthology.orchestra.run.smithi160.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 697 kB/s | 29 kB 00:00 2022-09-02T04:13:35.438 INFO:teuthology.orchestra.run.smithi160.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 656 kB/s | 27 kB 00:00 2022-09-02T04:13:35.497 INFO:teuthology.orchestra.run.smithi160.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2022-09-02T04:13:35.522 INFO:teuthology.orchestra.run.smithi160.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.3 MB/s | 241 kB 00:00 2022-09-02T04:13:35.548 INFO:teuthology.orchestra.run.smithi160.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2022-09-02T04:13:35.573 INFO:teuthology.orchestra.run.smithi160.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2022-09-02T04:13:35.590 INFO:teuthology.orchestra.run.smithi160.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 549 kB/s | 23 kB 00:00 2022-09-02T04:13:35.593 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:35.593 INFO:teuthology.orchestra.run.smithi160.stdout:Total 17 MB/s | 28 MB 00:01 2022-09-02T04:13:35.672 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:13:35.709 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:13:35.710 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:13:36.009 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:13:36.012 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:13:36.344 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:13:36.346 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2022-09-02T04:13:36.347 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:13:36.347 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2022-09-02T04:13:36.347 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:13:36.348 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:17.0.0-14689.gafd42238.el8 ceph 6.4 k 2022-09-02T04:13:36.348 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:13:36.348 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:17.0.0-14689.gafd42238.el8 ceph 2.3 M 2022-09-02T04:13:36.349 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:17.0.0-14689.gafd42238.el8 ceph 1.5 M 2022-09-02T04:13:36.349 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 233 k 2022-09-02T04:13:36.349 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.9 M 2022-09-02T04:13:36.349 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:17.0.0-14689.gafd42238.el8 ceph 17 M 2022-09-02T04:13:36.350 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-02T04:13:36.350 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-02T04:13:36.351 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2022-09-02T04:13:36.351 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-02T04:13:36.351 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-02T04:13:36.351 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2022-09-02T04:13:36.352 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-02T04:13:36.352 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-02T04:13:36.352 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-02T04:13:36.353 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-09-02T04:13:36.353 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-09-02T04:13:36.353 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-02T04:13:36.353 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-02T04:13:36.354 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-02T04:13:36.354 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-09-02T04:13:36.355 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2022-09-02T04:13:36.355 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-09-02T04:13:36.356 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-02T04:13:36.356 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-02T04:13:36.357 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-02T04:13:36.357 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-02T04:13:36.358 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-02T04:13:36.358 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-02T04:13:36.359 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-02T04:13:36.359 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-09-02T04:13:36.359 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-02T04:13:36.360 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:36.360 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:13:36.361 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2022-09-02T04:13:36.361 INFO:teuthology.orchestra.run.smithi099.stdout:Install 32 Packages 2022-09-02T04:13:36.362 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:36.363 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 28 M 2022-09-02T04:13:36.364 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 110 M 2022-09-02T04:13:36.364 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:13:36.482 INFO:teuthology.orchestra.run.smithi099.stdout:(1/32): ceph-17.0.0-14689.gafd42238.el8.x86_64. 49 kB/s | 6.4 kB 00:00 2022-09-02T04:13:36.738 INFO:teuthology.orchestra.run.smithi099.stdout:(2/32): ceph-mds-17.0.0-14689.gafd42238.el8.x86 5.9 MB/s | 2.3 MB 00:00 2022-09-02T04:13:36.782 INFO:teuthology.orchestra.run.smithi099.stdout:(3/32): ceph-mgr-17.0.0-14689.gafd42238.el8.x86 3.6 MB/s | 1.5 MB 00:00 2022-09-02T04:13:36.874 INFO:teuthology.orchestra.run.smithi099.stdout:(4/32): ceph-mgr-modules-core-17.0.0-14689.gafd 2.5 MB/s | 233 kB 00:00 2022-09-02T04:13:36.934 INFO:teuthology.orchestra.run.smithi099.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-09-02T04:13:36.958 INFO:teuthology.orchestra.run.smithi099.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-09-02T04:13:36.983 INFO:teuthology.orchestra.run.smithi099.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.1 MB/s | 103 kB 00:00 2022-09-02T04:13:37.009 INFO:teuthology.orchestra.run.smithi099.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-09-02T04:13:37.042 INFO:teuthology.orchestra.run.smithi099.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2022-09-02T04:13:37.101 INFO:teuthology.orchestra.run.smithi099.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 237 kB 00:00 2022-09-02T04:13:37.142 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:13:37.142 INFO:teuthology.orchestra.run.smithi099.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2022-09-02T04:13:37.168 INFO:teuthology.orchestra.run.smithi099.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-09-02T04:13:37.243 INFO:teuthology.orchestra.run.smithi099.stdout:(13/32): ceph-mon-17.0.0-14689.gafd42238.el8.x8 5.1 MB/s | 3.9 MB 00:00 2022-09-02T04:13:37.350 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-09-02T04:13:37.352 INFO:teuthology.orchestra.run.smithi099.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 237 kB/s | 44 kB 00:00 2022-09-02T04:13:37.435 INFO:teuthology.orchestra.run.smithi099.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 965 kB/s | 185 kB 00:00 2022-09-02T04:13:37.586 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-09-02T04:13:37.611 INFO:teuthology.orchestra.run.smithi099.stdout:(16/32): ceph-osd-17.0.0-14689.gafd42238.el8.x8 19 MB/s | 17 MB 00:00 2022-09-02T04:13:37.636 INFO:teuthology.orchestra.run.smithi099.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 608 kB/s | 173 kB 00:00 2022-09-02T04:13:37.678 INFO:teuthology.orchestra.run.smithi099.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2022-09-02T04:13:37.703 INFO:teuthology.orchestra.run.smithi099.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2022-09-02T04:13:37.754 INFO:teuthology.orchestra.run.smithi099.stdout:(20/32): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2022-09-02T04:13:37.787 INFO:teuthology.orchestra.run.smithi099.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 447 kB/s | 49 kB 00:00 2022-09-02T04:13:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-09-02T04:13:37.813 INFO:teuthology.orchestra.run.smithi099.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2022-09-02T04:13:37.871 INFO:teuthology.orchestra.run.smithi099.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 321 kB/s | 19 kB 00:00 2022-09-02T04:13:37.896 INFO:teuthology.orchestra.run.smithi099.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2022-09-02T04:13:37.930 INFO:teuthology.orchestra.run.smithi099.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 410 kB/s | 24 kB 00:00 2022-09-02T04:13:37.955 INFO:teuthology.orchestra.run.smithi099.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2022-09-02T04:13:37.968 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-09-02T04:13:37.981 INFO:teuthology.orchestra.run.smithi099.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 347 kB/s | 29 kB 00:00 2022-09-02T04:13:38.006 INFO:teuthology.orchestra.run.smithi099.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 363 kB/s | 27 kB 00:00 2022-09-02T04:13:38.098 INFO:teuthology.orchestra.run.smithi099.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 925 kB/s | 85 kB 00:00 2022-09-02T04:13:38.131 INFO:teuthology.orchestra.run.smithi099.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.3 MB/s | 241 kB 00:00 2022-09-02T04:13:38.141 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-09-02T04:13:38.157 INFO:teuthology.orchestra.run.smithi099.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2022-09-02T04:13:38.182 INFO:teuthology.orchestra.run.smithi099.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 276 kB/s | 23 kB 00:00 2022-09-02T04:13:38.185 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:38.185 INFO:teuthology.orchestra.run.smithi099.stdout:Total 15 MB/s | 28 MB 00:01 2022-09-02T04:13:38.262 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:13:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-09-02T04:13:38.298 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:13:38.299 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:13:38.456 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-09-02T04:13:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-09-02T04:13:38.606 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:13:38.609 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:13:38.799 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-09-02T04:13:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-09-02T04:13:39.232 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-09-02T04:13:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-09-02T04:13:39.788 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:13:39.837 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-09-02T04:13:40.013 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-09-02T04:13:40.073 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-09-02T04:13:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-09-02T04:13:40.283 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-09-02T04:13:40.460 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-09-02T04:13:40.504 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-09-02T04:13:40.646 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-02T04:13:40.689 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-09-02T04:13:40.912 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-09-02T04:13:40.962 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-09-02T04:13:41.067 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-09-02T04:13:41.151 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-09-02T04:13:41.269 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-09-02T04:13:41.322 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-09-02T04:13:41.422 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-09-02T04:13:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-09-02T04:13:41.672 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-09-02T04:13:41.834 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-09-02T04:13:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-09-02T04:13:42.059 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-09-02T04:13:42.147 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-09-02T04:13:42.263 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-09-02T04:13:42.370 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-09-02T04:13:42.604 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-09-02T04:13:42.810 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-09-02T04:13:42.932 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-09-02T04:13:43.086 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-09-02T04:13:43.257 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-09-02T04:13:43.330 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14689.gafd42238.el8 27/32 2022-09-02T04:13:43.520 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 28/32 2022-09-02T04:13:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-09-02T04:13:43.775 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-02T04:13:44.091 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-09-02T04:13:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-09-02T04:13:44.468 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-09-02T04:13:44.695 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-09-02T04:13:44.920 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-09-02T04:13:45.010 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 28/32 2022-09-02T04:13:45.181 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-09-02T04:13:45.219 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 29/32 2022-09-02T04:13:45.409 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-09-02T04:13:45.605 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 29/32 2022-09-02T04:13:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-09-02T04:13:45.769 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 30/32 2022-09-02T04:13:45.983 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 30/32 2022-09-02T04:13:46.069 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-09-02T04:13:46.153 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 31/32 2022-09-02T04:13:46.180 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 31/32 2022-09-02T04:13:46.293 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:17.0.0-14689.gafd42238.el8.x86_64 32/32 2022-09-02T04:13:46.436 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14689.gafd42238.el8 27/32 2022-09-02T04:13:46.600 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 28/32 2022-09-02T04:13:46.682 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:17.0.0-14689.gafd42238.el8.x86_64 32/32 2022-09-02T04:13:46.682 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:17.0.0-14689.gafd42238.el8.x86_64 1/32 2022-09-02T04:13:46.682 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 2/32 2022-09-02T04:13:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 3/32 2022-09-02T04:13:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 4/32 2022-09-02T04:13:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 5/32 2022-09-02T04:13:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14689.gafd42238.el8 6/32 2022-09-02T04:13:46.684 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-09-02T04:13:46.684 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-09-02T04:13:46.684 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-09-02T04:13:46.684 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-09-02T04:13:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-09-02T04:13:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-09-02T04:13:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-09-02T04:13:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-09-02T04:13:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-09-02T04:13:46.686 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-09-02T04:13:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-02T04:13:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-09-02T04:13:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-09-02T04:13:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-09-02T04:13:46.688 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-09-02T04:13:46.688 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-09-02T04:13:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-09-02T04:13:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-09-02T04:13:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-09-02T04:13:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-09-02T04:13:46.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-09-02T04:13:46.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-09-02T04:13:46.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-09-02T04:13:46.690 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-09-02T04:13:46.691 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-09-02T04:13:48.092 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 28/32 2022-09-02T04:13:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-09-02T04:13:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:48.108 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:13:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:13:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-02T04:13:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-02T04:13:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-02T04:13:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-02T04:13:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-02T04:13:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-02T04:13:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-02T04:13:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-02T04:13:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-02T04:13:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-02T04:13:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-02T04:13:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-02T04:13:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-02T04:13:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-2.6-1.el8.noarch 2022-09-02T04:13:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-02T04:13:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-02T04:13:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-02T04:13:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-02T04:13:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-02T04:13:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-02T04:13:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-02T04:13:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-02T04:13:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-02T04:13:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-02T04:13:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-02T04:13:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-02T04:13:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:48.122 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:48.299 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 29/32 2022-09-02T04:13:48.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2022-09-02T04:13:48.687 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 29/32 2022-09-02T04:13:48.884 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:15 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:49.532 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 30/32 2022-09-02T04:13:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 30/32 2022-09-02T04:13:49.817 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:13:49.887 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:13:49.888 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:13:49.889 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:49.965 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 31/32 2022-09-02T04:13:49.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2022-09-02T04:13:49.986 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 31/32 2022-09-02T04:13:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:17.0.0-14689.gafd42238.el8.x86_64 32/32 2022-09-02T04:13:50.500 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:17.0.0-14689.gafd42238.el8.x86_64 32/32 2022-09-02T04:13:50.500 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:17.0.0-14689.gafd42238.el8.x86_64 1/32 2022-09-02T04:13:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 2/32 2022-09-02T04:13:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 3/32 2022-09-02T04:13:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 4/32 2022-09-02T04:13:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 5/32 2022-09-02T04:13:50.502 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14689.gafd42238.el8 6/32 2022-09-02T04:13:50.502 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-09-02T04:13:50.502 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-09-02T04:13:50.502 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-09-02T04:13:50.503 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-09-02T04:13:50.503 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-09-02T04:13:50.503 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-09-02T04:13:50.503 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-09-02T04:13:50.504 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-09-02T04:13:50.504 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-09-02T04:13:50.504 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-09-02T04:13:50.504 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-09-02T04:13:50.505 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-09-02T04:13:50.505 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-09-02T04:13:50.505 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-09-02T04:13:50.506 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-09-02T04:13:50.506 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-09-02T04:13:50.506 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-09-02T04:13:50.507 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-09-02T04:13:50.507 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-09-02T04:13:50.507 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-09-02T04:13:50.507 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-09-02T04:13:50.508 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-09-02T04:13:50.508 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-09-02T04:13:50.508 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-09-02T04:13:50.509 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-09-02T04:13:50.546 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:17 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-09-02T04:13:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:51.430 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:13:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:13:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-02T04:13:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-02T04:13:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-02T04:13:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-02T04:13:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-02T04:13:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-02T04:13:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-02T04:13:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-02T04:13:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-02T04:13:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-02T04:13:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-02T04:13:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-02T04:13:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-02T04:13:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-2.6-1.el8.noarch 2022-09-02T04:13:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-02T04:13:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-02T04:13:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-02T04:13:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-02T04:13:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-02T04:13:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-02T04:13:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-02T04:13:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-02T04:13:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-02T04:13:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-02T04:13:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-02T04:13:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-02T04:13:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:51.438 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:13:51.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2022-09-02T04:13:51.562 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:13:51.563 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:51.563 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:13:51.564 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:51.564 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:13:51.564 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 81 k 2022-09-02T04:13:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:51.565 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:13:51.565 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:51.565 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:13:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:51.566 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 81 k 2022-09-02T04:13:51.566 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 367 k 2022-09-02T04:13:51.567 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:13:51.736 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-17.0.0-14689.gafd42238.el8.noarch.rpm 468 kB/s | 81 kB 00:00 2022-09-02T04:13:51.737 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:51.737 INFO:teuthology.orchestra.run.smithi160.stdout:Total 465 kB/s | 81 kB 00:00 2022-09-02T04:13:51.738 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:13:51.744 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:13:51.744 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:13:51.745 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:13:51.748 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:13:51.789 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:13:52.124 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:19 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:53.074 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:13:53.149 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:13:53.150 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:13:53.151 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:13:53.235 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2022-09-02T04:13:53.266 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:53.439 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:53.799 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:20 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:54.017 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:13:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:13:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:54.018 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:54.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2022-09-02T04:13:54.809 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:13:54.810 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:54.810 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:13:54.810 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:54.811 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:13:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 81 k 2022-09-02T04:13:54.811 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:54.811 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:13:54.812 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:54.812 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:13:54.812 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:54.812 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 81 k 2022-09-02T04:13:54.813 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 367 k 2022-09-02T04:13:54.813 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:13:54.813 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:21 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:54.993 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-17.0.0-14689.gafd42238.el8.noarch.rpm 445 kB/s | 81 kB 00:00 2022-09-02T04:13:54.994 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:54.994 INFO:teuthology.orchestra.run.smithi099.stdout:Total 442 kB/s | 81 kB 00:00 2022-09-02T04:13:54.994 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:13:55.001 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:13:55.002 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:13:55.003 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:13:55.006 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:13:55.050 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:13:55.824 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:13:55.826 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:55.826 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2022-09-02T04:13:55.827 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:55.827 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:13:55.828 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14689.gafd42238.el8 ceph 165 k 2022-09-02T04:13:55.828 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:55.829 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:13:55.829 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:13:55.830 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:13:55.830 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:55.831 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 165 k 2022-09-02T04:13:55.831 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 503 k 2022-09-02T04:13:55.832 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:13:56.023 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-17.0.0-14689.gafd42 838 kB/s | 165 kB 00:00 2022-09-02T04:13:56.024 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:56.024 INFO:teuthology.orchestra.run.smithi160.stdout:Total 833 kB/s | 165 kB 00:00 2022-09-02T04:13:56.024 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:13:56.035 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:13:56.035 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:13:56.082 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:13:56.085 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:13:56.229 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:13:56.421 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14689.gafd42238 1/1 2022-09-02T04:13:56.528 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:56.743 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:56.808 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14689.gafd42238 1/1 2022-09-02T04:13:56.933 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:57.205 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14689.gafd42238 1/1 2022-09-02T04:13:57.206 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:57.206 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:13:57.207 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:13:57.207 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:13:57.208 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:57.284 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:13:57.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:57.285 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:13:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:13:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:57.285 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:13:57.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2022-09-02T04:13:57.503 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2022-09-02T04:13:57.975 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:24 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:58.083 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:25 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:13:58.918 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:13:58.990 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:13:58.991 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:13:58.992 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:13:59.096 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:13:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:59.097 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2022-09-02T04:13:59.098 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:59.098 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:13:59.098 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14689.gafd42238.el8 ceph 165 k 2022-09-02T04:13:59.099 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:59.099 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:13:59.100 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:13:59.100 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:13:59.100 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:13:59.101 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 165 k 2022-09-02T04:13:59.101 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 503 k 2022-09-02T04:13:59.101 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:13:59.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2022-09-02T04:13:59.360 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-17.0.0-14689.gafd42 631 kB/s | 165 kB 00:00 2022-09-02T04:13:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:13:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:Total 628 kB/s | 165 kB 00:00 2022-09-02T04:13:59.362 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:13:59.371 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:13:59.372 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:13:59.417 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:13:59.420 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:13:59.530 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:13:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14689.gafd42238 1/1 2022-09-02T04:13:59.805 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:26 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14689.gafd42238 1/1 2022-09-02T04:14:00.829 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:00.831 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:00.831 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:14:00.831 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:00.832 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:00.832 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 1.6 M 2022-09-02T04:14:00.832 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:14:00.832 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards 2022-09-02T04:14:00.833 INFO:teuthology.orchestra.run.smithi160.stdout: noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 23 k 2022-09-02T04:14:00.833 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 14 k 2022-09-02T04:14:00.833 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-09-02T04:14:00.834 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-02T04:14:00.834 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-02T04:14:00.834 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:00.835 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:00.835 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:00.835 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2022-09-02T04:14:00.835 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:00.836 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 1.9 M 2022-09-02T04:14:00.836 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 7.8 M 2022-09-02T04:14:00.836 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:01.042 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-grafana-dashboards-17.0.0-14689.gaf 109 kB/s | 23 kB 00:00 2022-09-02T04:14:01.067 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): ceph-prometheus-alerts-17.0.0-14689.gafd 61 kB/s | 14 kB 00:00 2022-09-02T04:14:01.126 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 510 kB/s | 43 kB 00:00 2022-09-02T04:14:01.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14689.gafd42238 1/1 2022-09-02T04:14:01.167 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:01.168 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:01.168 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:01.168 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:01.168 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:01.176 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 310 kB/s | 34 kB 00:00 2022-09-02T04:14:01.318 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2022-09-02T04:14:01.376 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): ceph-mgr-dashboard-17.0.0-14689.gafd4223 3.0 MB/s | 1.6 MB 00:00 2022-09-02T04:14:01.378 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:01.379 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.6 MB/s | 1.9 MB 00:00 2022-09-02T04:14:01.380 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2022-09-02T04:14:01.389 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:01.397 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:01.397 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:01.469 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:01.472 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:01.716 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-09-02T04:14:01.956 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:28 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:02.134 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-09-02T04:14:02.311 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-09-02T04:14:02.456 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14689.gafd42238.el8. 4/6 2022-09-02T04:14:02.784 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14689.gafd42238.el8 5/6 2022-09-02T04:14:02.897 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:02.958 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noar 6/6 2022-09-02T04:14:02.971 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:02.972 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:02.972 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:03.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-dashboard 2022-09-02T04:14:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noar 6/6 2022-09-02T04:14:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14689.gafd42238.el8 1/6 2022-09-02T04:14:03.192 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noar 2/6 2022-09-02T04:14:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14689.gafd42238.el8. 3/6 2022-09-02T04:14:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-09-02T04:14:03.193 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-09-02T04:14:03.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-09-02T04:14:03.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:03.582 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:03.582 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:03.583 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:03.583 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:03.583 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-02T04:14:03.583 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-02T04:14:03.584 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-02T04:14:03.584 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:03.584 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:03.636 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:30 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:03.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2022-09-02T04:14:04.267 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:31 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:04.643 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:04.644 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:04.644 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:14:04.644 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:04.645 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:04.645 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 1.6 M 2022-09-02T04:14:04.645 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:14:04.645 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards 2022-09-02T04:14:04.646 INFO:teuthology.orchestra.run.smithi099.stdout: noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 23 k 2022-09-02T04:14:04.647 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 14 k 2022-09-02T04:14:04.647 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2022-09-02T04:14:04.647 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-02T04:14:04.647 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-02T04:14:04.648 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:04.648 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:04.648 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:04.648 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2022-09-02T04:14:04.649 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:04.649 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.9 M 2022-09-02T04:14:04.650 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 7.8 M 2022-09-02T04:14:04.650 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:04.791 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): ceph-grafana-dashboards-17.0.0-14689.gaf 158 kB/s | 23 kB 00:00 2022-09-02T04:14:04.816 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): ceph-prometheus-alerts-17.0.0-14689.gafd 85 kB/s | 14 kB 00:00 2022-09-02T04:14:04.868 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 565 kB/s | 43 kB 00:00 2022-09-02T04:14:04.917 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2022-09-02T04:14:04.950 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): ceph-mgr-dashboard-17.0.0-14689.gafd4223 5.4 MB/s | 1.6 MB 00:00 2022-09-02T04:14:05.050 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2022-09-02T04:14:05.050 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:05.050 INFO:teuthology.orchestra.run.smithi099.stdout:Total 4.9 MB/s | 1.9 MB 00:00 2022-09-02T04:14:05.054 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:05.063 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:05.063 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:05.150 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:05.153 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:05.284 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:05.285 INFO:teuthology.orchestra.run.smithi160.stdout:===================================================================================================== 2022-09-02T04:14:05.286 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:14:05.286 INFO:teuthology.orchestra.run.smithi160.stdout:===================================================================================================== 2022-09-02T04:14:05.286 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:05.287 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 7.4 M 2022-09-02T04:14:05.287 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:14:05.287 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 8.5.0-15.el8 CentOS-BaseOS 644 k 2022-09-02T04:14:05.288 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 8.5.0-15.el8 CentOS-BaseOS 172 k 2022-09-02T04:14:05.288 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-09-02T04:14:05.288 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-09-02T04:14:05.288 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-09-02T04:14:05.289 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-09-02T04:14:05.289 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-09-02T04:14:05.290 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:05.290 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:05.290 INFO:teuthology.orchestra.run.smithi160.stdout:===================================================================================================== 2022-09-02T04:14:05.290 INFO:teuthology.orchestra.run.smithi160.stdout:Install 8 Packages 2022-09-02T04:14:05.291 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:05.291 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 35 M 2022-09-02T04:14:05.292 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 210 M 2022-09-02T04:14:05.292 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:05.444 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:05.458 INFO:teuthology.orchestra.run.smithi160.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2022-09-02T04:14:05.575 INFO:teuthology.orchestra.run.smithi160.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2022-09-02T04:14:05.601 INFO:teuthology.orchestra.run.smithi160.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 8.9 MB/s | 225 kB 00:00 2022-09-02T04:14:05.667 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-09-02T04:14:05.684 INFO:teuthology.orchestra.run.smithi160.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2022-09-02T04:14:05.734 INFO:teuthology.orchestra.run.smithi160.stdout:(5/8): libgfortran-8.5.0-15.el8.x86_64.rpm 13 MB/s | 644 kB 00:00 2022-09-02T04:14:05.768 INFO:teuthology.orchestra.run.smithi160.stdout:(6/8): libquadmath-8.5.0-15.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2022-09-02T04:14:05.868 INFO:teuthology.orchestra.run.smithi160.stdout:(7/8): ceph-mgr-diskprediction-local-17.0.0-146 13 MB/s | 7.4 MB 00:00 2022-09-02T04:14:05.903 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-09-02T04:14:06.060 INFO:teuthology.orchestra.run.smithi160.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 30 MB/s | 14 MB 00:00 2022-09-02T04:14:06.063 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:06.063 INFO:teuthology.orchestra.run.smithi160.stdout:Total 46 MB/s | 35 MB 00:00 2022-09-02T04:14:06.292 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:06.310 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:06.311 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:06.682 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:06.685 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:06.841 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-09-02T04:14:07.229 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:07.366 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-02T04:14:07.367 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14689.gafd42238.el8. 4/6 2022-09-02T04:14:07.518 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-02T04:14:07.641 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-02T04:14:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14689.gafd42238.el8 5/6 2022-09-02T04:14:07.878 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noar 6/6 2022-09-02T04:14:08.144 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noar 6/6 2022-09-02T04:14:08.144 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14689.gafd42238.el8 1/6 2022-09-02T04:14:08.144 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noar 2/6 2022-09-02T04:14:08.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14689.gafd42238.el8. 3/6 2022-09-02T04:14:08.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-09-02T04:14:08.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-09-02T04:14:08.242 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-02T04:14:08.407 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-09-02T04:14:08.529 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-09-02T04:14:08.530 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:08.530 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:08.530 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:08.530 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:08.531 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:08.531 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-02T04:14:08.531 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-02T04:14:08.532 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-02T04:14:08.532 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:08.532 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:08.646 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-diskprediction-local 2022-09-02T04:14:09.014 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-09-02T04:14:09.174 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-02T04:14:09.213 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:36 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:09.848 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-02T04:14:10.225 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:10.226 INFO:teuthology.orchestra.run.smithi099.stdout:===================================================================================================== 2022-09-02T04:14:10.226 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:14:10.227 INFO:teuthology.orchestra.run.smithi099.stdout:===================================================================================================== 2022-09-02T04:14:10.227 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:10.227 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 7.4 M 2022-09-02T04:14:10.227 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:14:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 8.5.0-15.el8 CentOS-BaseOS 644 k 2022-09-02T04:14:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 8.5.0-15.el8 CentOS-BaseOS 172 k 2022-09-02T04:14:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2022-09-02T04:14:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2022-09-02T04:14:10.228 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-09-02T04:14:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-09-02T04:14:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-09-02T04:14:10.229 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:10.229 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:10.230 INFO:teuthology.orchestra.run.smithi099.stdout:===================================================================================================== 2022-09-02T04:14:10.230 INFO:teuthology.orchestra.run.smithi099.stdout:Install 8 Packages 2022-09-02T04:14:10.230 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:10.232 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 35 M 2022-09-02T04:14:10.232 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 210 M 2022-09-02T04:14:10.233 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:10.333 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-09-02T04:14:10.432 INFO:teuthology.orchestra.run.smithi099.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2022-09-02T04:14:10.507 INFO:teuthology.orchestra.run.smithi099.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2022-09-02T04:14:10.516 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-02T04:14:10.541 INFO:teuthology.orchestra.run.smithi099.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-09-02T04:14:10.616 INFO:teuthology.orchestra.run.smithi099.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2022-09-02T04:14:10.692 INFO:teuthology.orchestra.run.smithi099.stdout:(5/8): libgfortran-8.5.0-15.el8.x86_64.rpm 8.4 MB/s | 644 kB 00:00 2022-09-02T04:14:10.725 INFO:teuthology.orchestra.run.smithi099.stdout:(6/8): libquadmath-8.5.0-15.el8.x86_64.rpm 5.0 MB/s | 172 kB 00:00 2022-09-02T04:14:11.010 INFO:teuthology.orchestra.run.smithi099.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 29 MB/s | 14 MB 00:00 2022-09-02T04:14:11.101 INFO:teuthology.orchestra.run.smithi099.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-146 8.5 MB/s | 7.4 MB 00:00 2022-09-02T04:14:11.103 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:11.104 INFO:teuthology.orchestra.run.smithi099.stdout:Total 40 MB/s | 35 MB 00:00 2022-09-02T04:14:11.332 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:11.351 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:11.351 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:11.710 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:11.713 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:12.268 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:12.277 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-02T04:14:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-02T04:14:12.580 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libquadmath-8.5.0-15.el8.x86_64 1/8 2022-09-02T04:14:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-02T04:14:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-09-02T04:14:13.674 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd422 8/8 2022-09-02T04:14:13.819 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libgfortran-8.5.0-15.el8.x86_64 2/8 2022-09-02T04:14:13.943 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd422 8/8 2022-09-02T04:14:13.943 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd422 1/8 2022-09-02T04:14:13.943 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-09-02T04:14:13.944 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-09-02T04:14:13.944 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-09-02T04:14:13.944 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-09-02T04:14:13.944 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-09-02T04:14:13.945 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-8.5.0-15.el8.x86_64 7/8 2022-09-02T04:14:13.984 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2022-09-02T04:14:14.356 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-8.5.0-15.el8.x86_64 8/8 2022-09-02T04:14:14.356 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:14.357 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:14.357 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-8.5.0-15.el8.x86_64 2022-09-02T04:14:14.358 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-8.5.0-15.el8.x86_64 2022-09-02T04:14:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.15-4.el8.x86_64 2022-09-02T04:14:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-09-02T04:14:14.359 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-02T04:14:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-02T04:14:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-09-02T04:14:14.360 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:14.360 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:14.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2022-09-02T04:14:14.589 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2022-09-02T04:14:14.776 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-02T04:14:15.024 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:42 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:15.455 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2022-09-02T04:14:15.765 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-09-02T04:14:15.967 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-02T04:14:16.043 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:16.044 INFO:teuthology.orchestra.run.smithi160.stdout:========================================================================================= 2022-09-02T04:14:16.045 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:14:16.045 INFO:teuthology.orchestra.run.smithi160.stdout:========================================================================================= 2022-09-02T04:14:16.045 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:16.045 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 47 k 2022-09-02T04:14:16.046 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:14:16.046 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-02T04:14:16.046 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-02T04:14:16.047 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-02T04:14:16.047 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-09-02T04:14:16.047 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-09-02T04:14:16.047 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-02T04:14:16.048 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-09-02T04:14:16.048 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-09-02T04:14:16.048 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-09-02T04:14:16.048 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-09-02T04:14:16.049 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-02T04:14:16.049 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-02T04:14:16.049 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:16.049 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:16.050 INFO:teuthology.orchestra.run.smithi160.stdout:========================================================================================= 2022-09-02T04:14:16.050 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2022-09-02T04:14:16.050 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:16.051 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 1.9 M 2022-09-02T04:14:16.052 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 25 M 2022-09-02T04:14:16.052 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:16.606 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 48 kB/s | 27 kB 00:00 2022-09-02T04:14:16.653 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 34 kB/s | 20 kB 00:00 2022-09-02T04:14:16.653 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.7 MB/s | 126 kB 00:00 2022-09-02T04:14:16.678 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-09-02T04:14:16.704 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2022-09-02T04:14:16.729 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 852 kB/s | 43 kB 00:00 2022-09-02T04:14:16.754 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): ceph-mgr-rook-17.0.0-14689.gafd42238.el 66 kB/s | 47 kB 00:00 2022-09-02T04:14:16.846 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2022-09-02T04:14:16.871 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 114 kB/s | 16 kB 00:00 2022-09-02T04:14:16.896 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 572 kB/s | 81 kB 00:00 2022-09-02T04:14:16.948 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-09-02T04:14:16.972 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 613 kB/s | 61 kB 00:00 2022-09-02T04:14:17.080 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.0 MB/s | 1.2 MB 00:00 2022-09-02T04:14:17.083 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:17.083 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2022-09-02T04:14:17.138 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:17.154 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:17.154 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:17.243 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:17.247 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:17.718 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-09-02T04:14:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:17.959 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-09-02T04:14:18.181 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-09-02T04:14:18.392 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-09-02T04:14:18.588 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-09-02T04:14:18.755 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-09-02T04:14:18.947 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-09-02T04:14:18.994 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-09-02T04:14:19.224 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-09-02T04:14:19.308 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd422 8/8 2022-09-02T04:14:19.410 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-09-02T04:14:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd422 8/8 2022-09-02T04:14:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd422 1/8 2022-09-02T04:14:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2022-09-02T04:14:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2022-09-02T04:14:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-09-02T04:14:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-09-02T04:14:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-09-02T04:14:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-8.5.0-15.el8.x86_64 7/8 2022-09-02T04:14:19.994 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-8.5.0-15.el8.x86_64 8/8 2022-09-02T04:14:19.995 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:19.995 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:19.996 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:19.996 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-8.5.0-15.el8.x86_64 2022-09-02T04:14:19.997 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-8.5.0-15.el8.x86_64 2022-09-02T04:14:19.997 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.15-4.el8.x86_64 2022-09-02T04:14:19.998 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads-0.3.15-4.el8.x86_64 2022-09-02T04:14:19.998 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-02T04:14:19.999 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-02T04:14:19.999 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-09-02T04:14:20.000 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:20.001 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:20.094 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-rook 2022-09-02T04:14:20.414 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-09-02T04:14:20.646 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-09-02T04:14:20.660 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:47 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:20.796 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-09-02T04:14:20.982 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-09-02T04:14:21.122 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 13/13 2022-09-02T04:14:21.401 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 13/13 2022-09-02T04:14:21.401 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 1/13 2022-09-02T04:14:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-09-02T04:14:21.402 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-09-02T04:14:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-09-02T04:14:21.403 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-09-02T04:14:21.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-09-02T04:14:21.404 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-09-02T04:14:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-09-02T04:14:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-09-02T04:14:21.405 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-09-02T04:14:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-09-02T04:14:21.406 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-09-02T04:14:21.686 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:21.687 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================= 2022-09-02T04:14:21.687 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:14:21.688 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================= 2022-09-02T04:14:21.688 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:21.689 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 47 k 2022-09-02T04:14:21.689 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:14:21.689 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-02T04:14:21.690 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-02T04:14:21.690 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-02T04:14:21.691 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-09-02T04:14:21.692 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-09-02T04:14:21.692 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-02T04:14:21.692 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2022-09-02T04:14:21.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-09-02T04:14:21.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-09-02T04:14:21.693 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2022-09-02T04:14:21.694 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-02T04:14:21.694 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-02T04:14:21.695 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:21.695 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:21.695 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================= 2022-09-02T04:14:21.696 INFO:teuthology.orchestra.run.smithi099.stdout:Install 13 Packages 2022-09-02T04:14:21.696 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:21.701 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.9 M 2022-09-02T04:14:21.702 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 25 M 2022-09-02T04:14:21.702 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:21.762 INFO:teuthology.orchestra.run.smithi099.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 377 kB/s | 27 kB 00:00 2022-09-02T04:14:21.838 INFO:teuthology.orchestra.run.smithi099.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 229 kB/s | 20 kB 00:00 2022-09-02T04:14:21.838 INFO:teuthology.orchestra.run.smithi099.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2022-09-02T04:14:21.839 INFO:teuthology.orchestra.run.smithi099.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-09-02T04:14:21.854 INFO:teuthology.orchestra.run.smithi099.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2022-09-02T04:14:21.871 INFO:teuthology.orchestra.run.smithi099.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2022-09-02T04:14:21.877 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-09-02T04:14:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:21.879 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:21.879 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-02T04:14:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-02T04:14:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-02T04:14:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-02T04:14:21.880 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-02T04:14:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-02T04:14:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-02T04:14:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-02T04:14:21.881 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-02T04:14:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-02T04:14:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-02T04:14:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-02T04:14:21.882 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:21.883 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:21.888 INFO:teuthology.orchestra.run.smithi099.stdout:(7/13): ceph-mgr-rook-17.0.0-14689.gafd42238.el 238 kB/s | 47 kB 00:00 2022-09-02T04:14:21.982 INFO:teuthology.orchestra.run.smithi099.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 150 kB/s | 16 kB 00:00 2022-09-02T04:14:21.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2022-09-02T04:14:22.005 INFO:teuthology.orchestra.run.smithi099.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 218 kB/s | 33 kB 00:00 2022-09-02T04:14:22.031 INFO:teuthology.orchestra.run.smithi099.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 571 kB/s | 81 kB 00:00 2022-09-02T04:14:22.081 INFO:teuthology.orchestra.run.smithi099.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2022-09-02T04:14:22.106 INFO:teuthology.orchestra.run.smithi099.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 611 kB/s | 61 kB 00:00 2022-09-02T04:14:22.206 INFO:teuthology.orchestra.run.smithi099.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.2 MB/s | 1.2 MB 00:00 2022-09-02T04:14:22.207 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:22.208 INFO:teuthology.orchestra.run.smithi099.stdout:Total 3.7 MB/s | 1.9 MB 00:00 2022-09-02T04:14:22.258 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:22.273 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:22.274 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:22.354 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:22.357 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:22.552 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:49 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:22.856 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:23.110 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-09-02T04:14:23.307 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-09-02T04:14:23.543 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-09-02T04:14:23.557 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:23.558 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:23.558 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:14:23.558 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:23.558 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:23.559 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 114 k 2022-09-02T04:14:23.559 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:14:23.559 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-02T04:14:23.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-09-02T04:14:23.560 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-09-02T04:14:23.561 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-02T04:14:23.561 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:23.561 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:23.562 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:23.562 INFO:teuthology.orchestra.run.smithi160.stdout:Install 5 Packages 2022-09-02T04:14:23.562 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:23.563 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 5.9 M 2022-09-02T04:14:23.563 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 26 M 2022-09-02T04:14:23.564 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:23.638 INFO:teuthology.orchestra.run.smithi160.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.8 MB/s | 538 kB 00:00 2022-09-02T04:14:23.763 INFO:teuthology.orchestra.run.smithi160.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2022-09-02T04:14:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-09-02T04:14:23.789 INFO:teuthology.orchestra.run.smithi160.stdout:(3/5): ceph-mgr-cephadm-17.0.0-14689.gafd42238. 501 kB/s | 114 kB 00:00 2022-09-02T04:14:23.872 INFO:teuthology.orchestra.run.smithi160.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.0 MB/s | 476 kB 00:00 2022-09-02T04:14:23.889 INFO:teuthology.orchestra.run.smithi160.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 481 kB/s | 60 kB 00:00 2022-09-02T04:14:23.890 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:23.891 INFO:teuthology.orchestra.run.smithi160.stdout:Total 18 MB/s | 5.9 MB 00:00 2022-09-02T04:14:23.941 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-09-02T04:14:23.952 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:23.952 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:24.044 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:24.047 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:24.293 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:24.328 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-09-02T04:14:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-09-02T04:14:24.601 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-09-02T04:14:24.819 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-09-02T04:14:25.528 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-09-02T04:14:25.843 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-09-02T04:14:25.858 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-09-02T04:14:26.083 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-09-02T04:14:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-09-02T04:14:26.255 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 5/5 2022-09-02T04:14:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-09-02T04:14:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-09-02T04:14:26.497 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 5/5 2022-09-02T04:14:26.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/5 2022-09-02T04:14:26.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-09-02T04:14:26.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-09-02T04:14:26.499 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-09-02T04:14:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 13/13 2022-09-02T04:14:26.852 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 13/13 2022-09-02T04:14:26.853 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 1/13 2022-09-02T04:14:26.853 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-09-02T04:14:26.854 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-09-02T04:14:26.854 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-09-02T04:14:26.854 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-09-02T04:14:26.855 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-09-02T04:14:26.855 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-09-02T04:14:26.858 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-09-02T04:14:26.858 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-09-02T04:14:26.859 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-09-02T04:14:26.859 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-09-02T04:14:26.859 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-09-02T04:14:27.264 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-09-02T04:14:27.264 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:27.265 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:27.265 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:27.265 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-02T04:14:27.266 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-02T04:14:27.267 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-02T04:14:27.269 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-02T04:14:27.269 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:27.269 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-09-02T04:14:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:27.297 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-02T04:14:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-02T04:14:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-02T04:14:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-02T04:14:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-02T04:14:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-02T04:14:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-02T04:14:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-02T04:14:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-02T04:14:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-02T04:14:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-02T04:14:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-02T04:14:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:27.301 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:27.394 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-cephadm 2022-09-02T04:14:27.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2022-09-02T04:14:27.961 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:54 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:27.975 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:54 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:28.980 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:28.980 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:28.981 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:14:28.981 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:28.981 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:28.981 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 114 k 2022-09-02T04:14:28.982 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:14:28.982 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-02T04:14:28.982 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-09-02T04:14:28.982 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-09-02T04:14:28.983 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-02T04:14:28.983 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:28.983 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:28.983 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:28.983 INFO:teuthology.orchestra.run.smithi099.stdout:Install 5 Packages 2022-09-02T04:14:28.984 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:28.985 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 5.9 M 2022-09-02T04:14:28.985 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 26 M 2022-09-02T04:14:28.985 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:28.995 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:28.996 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:28.996 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:14:28.997 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:28.997 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:28.997 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:17.0.0-14689.gafd42238.el8 ceph 843 k 2022-09-02T04:14:28.997 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2022-09-02T04:14:28.997 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-09-02T04:14:28.998 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:28.998 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:28.998 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:28.998 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2022-09-02T04:14:28.999 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:28.999 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 926 k 2022-09-02T04:14:29.000 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 2.9 M 2022-09-02T04:14:29.000 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:29.063 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2022-09-02T04:14:29.271 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): ceph-fuse-17.0.0-14689.gafd42238.el8.x86 3.0 MB/s | 843 kB 00:00 2022-09-02T04:14:29.273 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:29.274 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.3 MB/s | 926 kB 00:00 2022-09-02T04:14:29.283 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:29.304 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:29.304 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:29.364 INFO:teuthology.orchestra.run.smithi099.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2022-09-02T04:14:29.377 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:29.380 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:29.535 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:29.564 INFO:teuthology.orchestra.run.smithi099.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 8.2 MB/s | 4.8 MB 00:00 2022-09-02T04:14:29.589 INFO:teuthology.orchestra.run.smithi099.stdout:(3/5): ceph-mgr-cephadm-17.0.0-14689.gafd42238. 189 kB/s | 114 kB 00:00 2022-09-02T04:14:29.697 INFO:teuthology.orchestra.run.smithi099.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 452 kB/s | 60 kB 00:00 2022-09-02T04:14:29.731 INFO:teuthology.orchestra.run.smithi099.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.3 MB/s | 476 kB 00:00 2022-09-02T04:14:29.733 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:29.733 INFO:teuthology.orchestra.run.smithi099.stdout:Total 7.9 MB/s | 5.9 MB 00:00 2022-09-02T04:14:29.784 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:29.795 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:29.796 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:29.853 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-09-02T04:14:29.905 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:29.908 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:30.011 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 2/2 2022-09-02T04:14:30.149 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:30.363 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 2/2 2022-09-02T04:14:30.364 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/2 2022-09-02T04:14:30.478 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-09-02T04:14:30.734 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-09-02T04:14:30.734 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:30.735 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:30.735 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-09-02T04:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:30.736 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:30.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-volume 2022-09-02T04:14:31.419 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-09-02T04:14:31.536 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:58 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:31.808 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-09-02T04:14:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-09-02T04:14:32.230 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 5/5 2022-09-02T04:14:32.504 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 5/5 2022-09-02T04:14:32.505 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 1/5 2022-09-02T04:14:32.506 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-09-02T04:14:32.506 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-09-02T04:14:32.508 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-09-02T04:14:32.549 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:32.550 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:32.551 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2022-09-02T04:14:32.551 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:32.552 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:32.552 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 263 k 2022-09-02T04:14:32.553 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:32.553 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:32.554 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:32.554 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:14:32.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:32.555 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 263 k 2022-09-02T04:14:32.555 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 1.3 M 2022-09-02T04:14:32.556 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:32.816 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-volume-17.0.0-14689.gafd42238.el8.noarch.r 977 kB/s | 263 kB 00:00 2022-09-02T04:14:32.817 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:32.818 INFO:teuthology.orchestra.run.smithi160.stdout:Total 973 kB/s | 263 kB 00:00 2022-09-02T04:14:32.818 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:32.827 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:32.831 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:32.855 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:32.855 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-09-02T04:14:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-02T04:14:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-02T04:14:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-02T04:14:32.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-02T04:14:32.890 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:32.985 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2022-09-02T04:14:33.013 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:33.213 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:14:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:00 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:33.569 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:14:33.994 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:14:33.994 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:33.995 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:33.995 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:33.996 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:33.996 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:34.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2022-09-02T04:14:34.588 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:34.590 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:34.591 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:14:34.591 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:34.591 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:34.592 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:17.0.0-14689.gafd42238.el8 ceph 843 k 2022-09-02T04:14:34.592 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-09-02T04:14:34.593 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2022-09-02T04:14:34.593 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:34.593 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:34.594 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:34.594 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2022-09-02T04:14:34.595 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:34.595 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 926 k 2022-09-02T04:14:34.595 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 2.9 M 2022-09-02T04:14:34.596 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:34.697 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 782 kB/s | 83 kB 00:00 2022-09-02T04:14:34.804 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:01 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:34.955 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-17.0.0-14689.gafd42238.el8.x86 2.3 MB/s | 843 kB 00:00 2022-09-02T04:14:34.956 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:34.957 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.5 MB/s | 926 kB 00:00 2022-09-02T04:14:34.966 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:34.987 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:34.987 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:35.062 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:35.064 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:35.222 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:35.519 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2022-09-02T04:14:35.728 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 2/2 2022-09-02T04:14:35.820 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:35.821 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:35.821 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2022-09-02T04:14:35.822 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:35.822 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:35.822 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:17.0.0-14689.gafd42238.el8 ceph 139 k 2022-09-02T04:14:35.823 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:35.823 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:35.824 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:35.824 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:14:35.825 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:35.826 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 139 k 2022-09-02T04:14:35.827 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 511 k 2022-09-02T04:14:35.827 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:36.029 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-17.0.0-14689.gafd42238.el8.x86_6 675 kB/s | 139 kB 00:00 2022-09-02T04:14:36.030 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:36.031 INFO:teuthology.orchestra.run.smithi160.stdout:Total 671 kB/s | 139 kB 00:00 2022-09-02T04:14:36.033 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:36.034 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:36.034 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:36.082 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:36.089 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:36.090 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 2/2 2022-09-02T04:14:36.091 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/2 2022-09-02T04:14:36.199 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:36.393 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:36.526 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2022-09-02T04:14:36.526 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:36.527 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:36.527 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2022-09-02T04:14:36.527 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:36.528 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:36.644 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:36.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-volume 2022-09-02T04:14:37.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:37.051 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:37.052 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:37.052 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:37.053 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:37.053 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:37.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2022-09-02T04:14:37.359 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:04 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:37.819 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:04 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:38.373 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:38.373 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:38.373 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-09-02T04:14:38.374 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:38.374 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:38.374 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:17.0.0-14689.gafd42238.el8 ceph-noarch 263 k 2022-09-02T04:14:38.374 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:38.375 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:38.375 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:38.375 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:14:38.375 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:38.376 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 263 k 2022-09-02T04:14:38.376 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.3 M 2022-09-02T04:14:38.376 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:38.584 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-volume-17.0.0-14689.gafd42238.el8.noarch.r 1.2 MB/s | 263 kB 00:00 2022-09-02T04:14:38.585 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:38.585 INFO:teuthology.orchestra.run.smithi099.stdout:Total 1.2 MB/s | 263 kB 00:00 2022-09-02T04:14:38.585 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:38.598 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:38.600 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:38.616 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:38.620 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:38.750 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:38.827 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:38.828 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:38.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2022-09-02T04:14:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:14:39.434 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:14:39.474 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:06 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:39.836 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 1/1 2022-09-02T04:14:39.837 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:39.837 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:39.837 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:17.0.0-14689.gafd42238.el8.noarch 2022-09-02T04:14:39.837 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:39.838 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:40.106 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2022-09-02T04:14:40.489 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:40.489 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:40.491 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2022-09-02T04:14:40.491 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:40.492 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:40.492 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:17.0.0-14689.gafd42238.el8 ceph 30 k 2022-09-02T04:14:40.492 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:40.495 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:40.496 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:40.496 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:14:40.497 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:40.497 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 30 k 2022-09-02T04:14:40.498 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 137 k 2022-09-02T04:14:40.498 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:40.647 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-17.0.0-14689.gafd42238.el8.x86_ 205 kB/s | 30 kB 00:00 2022-09-02T04:14:40.647 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:40.648 INFO:teuthology.orchestra.run.smithi160.stdout:Total 203 kB/s | 30 kB 00:00 2022-09-02T04:14:40.648 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:40.649 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:40.649 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:40.682 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:40.686 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:40.691 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:07 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:40.740 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:40.901 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:41.131 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:41.476 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:41.477 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:41.477 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:41.478 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:41.478 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:41.479 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:41.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2022-09-02T04:14:41.703 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:41.704 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:41.704 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2022-09-02T04:14:41.704 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:41.705 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:41.705 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:17.0.0-14689.gafd42238.el8 ceph 139 k 2022-09-02T04:14:41.705 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:41.705 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:41.706 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:41.706 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:14:41.706 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:41.706 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 139 k 2022-09-02T04:14:41.707 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 511 k 2022-09-02T04:14:41.707 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:41.914 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-17.0.0-14689.gafd42238.el8.x86_6 660 kB/s | 139 kB 00:00 2022-09-02T04:14:41.915 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:41.915 INFO:teuthology.orchestra.run.smithi099.stdout:Total 656 kB/s | 139 kB 00:00 2022-09-02T04:14:41.916 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:41.920 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:41.920 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:41.968 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:41.974 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:42.079 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:42.168 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:09 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:42.314 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:42.543 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:42.917 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:42.918 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:42.918 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:42.919 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:42.920 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:42.920 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:43.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2022-09-02T04:14:43.104 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:43.180 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:43.180 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:43.181 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:43.660 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:10 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:43.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2022-09-02T04:14:44.433 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:11 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:44.593 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:44.677 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:44.678 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:44.678 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:44.748 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2022-09-02T04:14:45.326 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:12 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:45.370 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:45.449 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:45.450 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:45.450 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:45.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2022-09-02T04:14:46.105 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:13 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:46.335 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:46.336 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:46.336 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2022-09-02T04:14:46.336 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:46.337 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:46.337 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:17.0.0-14689.gafd42238.el8 ceph 30 k 2022-09-02T04:14:46.338 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:46.338 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:46.339 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:46.339 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:14:46.340 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:46.340 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 30 k 2022-09-02T04:14:46.341 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 137 k 2022-09-02T04:14:46.341 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:46.485 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-17.0.0-14689.gafd42238.el8.x86_ 201 kB/s | 30 kB 00:00 2022-09-02T04:14:46.485 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:46.486 INFO:teuthology.orchestra.run.smithi099.stdout:Total 200 kB/s | 30 kB 00:00 2022-09-02T04:14:46.486 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:46.488 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:46.489 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:46.525 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:46.525 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:46.578 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:46.761 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:46.982 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:47.034 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:47.108 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:47.109 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:47.110 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:47.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2022-09-02T04:14:47.311 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:47.311 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:47.312 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:14:47.312 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:47.312 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:47.313 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:47.405 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2022-09-02T04:14:47.773 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:14 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:47.973 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:14 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:48.711 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:48.787 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:48.787 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:48.788 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:48.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2022-09-02T04:14:48.901 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:48.975 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:48.976 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:48.977 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:49.064 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2022-09-02T04:14:49.443 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:16 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:49.632 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:16 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:50.380 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:50.455 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:50.456 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:50.456 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:50.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2022-09-02T04:14:50.571 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:50.646 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:50.648 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:50.648 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:50.737 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2022-09-02T04:14:51.114 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:18 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:51.311 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:18 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:52.046 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:52.120 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:52.122 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2022-09-02T04:14:52.122 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:52.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2022-09-02T04:14:52.249 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:52.324 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:52.325 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:52.325 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:52.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2022-09-02T04:14:52.774 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:19 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:52.984 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:19 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:53.800 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:53.801 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:53.801 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2022-09-02T04:14:53.802 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:53.803 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:53.803 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:17.0.0-14689.gafd42238.el8 ceph 94 k 2022-09-02T04:14:53.803 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:53.804 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:53.804 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:53.804 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:14:53.804 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:53.805 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 94 k 2022-09-02T04:14:53.805 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 280 k 2022-09-02T04:14:53.805 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:53.917 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:53.978 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-17.0.0-14689.gafd42238.el8.x86_64.rpm 537 kB/s | 94 kB 00:00 2022-09-02T04:14:53.979 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:53.979 INFO:teuthology.orchestra.run.smithi160.stdout:Total 533 kB/s | 94 kB 00:00 2022-09-02T04:14:53.980 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:53.988 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:53.989 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:53.993 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:53.995 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:53.995 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:54.036 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:54.039 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:54.073 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2022-09-02T04:14:54.142 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:54.302 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:54.550 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:54.645 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:21 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:55.061 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:55.062 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:55.062 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:55.063 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:55.063 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:55.063 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:55.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2022-09-02T04:14:55.584 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:55.659 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:55.660 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:55.660 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:55.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2022-09-02T04:14:55.856 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:22 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:56.303 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:23 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:56.887 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:14:56.888 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2022-09-02T04:14:56.888 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:56.889 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:14:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.5 M 2022-09-02T04:14:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:56.889 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:14:56.890 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:14:56.890 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:14:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:56.890 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.5 M 2022-09-02T04:14:56.891 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 16 M 2022-09-02T04:14:56.891 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:14:57.236 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:17.0.0-14689.gafd42238.el8.x86_64 is already installed. 2022-09-02T04:14:57.265 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-17.0.0-14689.gafd42238.el8.x86_64.rp 9.2 MB/s | 3.5 MB 00:00 2022-09-02T04:14:57.267 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:57.267 INFO:teuthology.orchestra.run.smithi160.stdout:Total 9.2 MB/s | 3.5 MB 00:00 2022-09-02T04:14:57.268 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:14:57.278 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:14:57.279 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:14:57.311 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:57.312 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-09-02T04:14:57.313 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:14:57.340 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:14:57.342 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:14:57.527 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2022-09-02T04:14:57.776 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:14:57.961 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:58.093 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:25 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:58.322 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:58.717 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:14:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:14:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:14:58.718 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:14:58.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2022-09-02T04:14:59.111 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:14:59.111 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:59.112 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2022-09-02T04:14:59.112 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:59.113 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:14:59.113 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:17.0.0-14689.gafd42238.el8 ceph 94 k 2022-09-02T04:14:59.114 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:59.114 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:14:59.114 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:14:59.115 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:14:59.115 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:14:59.118 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 94 k 2022-09-02T04:14:59.119 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 280 k 2022-09-02T04:14:59.119 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:14:59.316 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-17.0.0-14689.gafd42238.el8.x86_64.rpm 458 kB/s | 94 kB 00:00 2022-09-02T04:14:59.317 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:14:59.317 INFO:teuthology.orchestra.run.smithi099.stdout:Total 455 kB/s | 94 kB 00:00 2022-09-02T04:14:59.317 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:14:59.328 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:14:59.328 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:14:59.376 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:14:59.378 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:14:59.478 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:14:59.522 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:26 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:14:59.664 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:14:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:00.310 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:00.310 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:00.310 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:15:00.311 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:15:00.311 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:00.311 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:15:00.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2022-09-02T04:15:00.520 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2022-09-02T04:15:00.523 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:15:00.523 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2022-09-02T04:15:00.524 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:15:00.525 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2022-09-02T04:15:00.525 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:17.0.0-14689.gafd42238.el8 ceph 175 k 2022-09-02T04:15:00.526 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:00.526 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2022-09-02T04:15:00.526 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2022-09-02T04:15:00.527 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2022-09-02T04:15:00.527 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:00.528 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 175 k 2022-09-02T04:15:00.528 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 556 k 2022-09-02T04:15:00.529 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2022-09-02T04:15:00.731 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-17.0.0-14689.gafd42238.el8.x86_64.rpm 853 kB/s | 175 kB 00:00 2022-09-02T04:15:00.732 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:15:00.732 INFO:teuthology.orchestra.run.smithi160.stdout:Total 848 kB/s | 175 kB 00:00 2022-09-02T04:15:00.733 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2022-09-02T04:15:00.739 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2022-09-02T04:15:00.739 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2022-09-02T04:15:00.787 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2022-09-02T04:15:00.790 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2022-09-02T04:15:00.916 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2022-09-02T04:15:01.082 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:01.085 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:28 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:15:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:01.720 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:01.729 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:01.730 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2022-09-02T04:15:01.730 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:15:01.730 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:01.731 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2022-09-02T04:15:01.888 DEBUG:teuthology.parallel:result is None 2022-09-02T04:15:02.099 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:15:02.099 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:15:02.099 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2022-09-02T04:15:02.100 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:15:02.100 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:15:02.101 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:17.0.0-14689.gafd42238.el8 ceph 3.5 M 2022-09-02T04:15:02.101 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:02.101 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:15:02.102 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:15:02.102 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:15:02.102 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:02.103 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.5 M 2022-09-02T04:15:02.103 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 16 M 2022-09-02T04:15:02.104 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:15:02.476 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-17.0.0-14689.gafd42238.el8.x86_64.rp 9.2 MB/s | 3.5 MB 00:00 2022-09-02T04:15:02.482 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:15:02.483 INFO:teuthology.orchestra.run.smithi099.stdout:Total 9.1 MB/s | 3.5 MB 00:00 2022-09-02T04:15:02.483 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:15:02.490 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:15:02.491 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:15:02.552 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:15:02.555 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:15:03.023 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:15:03.416 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:03.785 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:04.149 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:04.150 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:04.150 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:15:04.150 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:15:04.150 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:04.151 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:15:04.380 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2022-09-02T04:15:04.964 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:31 ago on Fri 02 Sep 2022 04:12:33 AM UTC. 2022-09-02T04:15:05.970 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-09-02T04:15:05.970 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:15:05.970 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2022-09-02T04:15:05.971 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:15:05.971 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-09-02T04:15:05.971 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:17.0.0-14689.gafd42238.el8 ceph 175 k 2022-09-02T04:15:05.971 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:05.972 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-09-02T04:15:05.972 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2022-09-02T04:15:05.972 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2022-09-02T04:15:05.973 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:05.973 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 175 k 2022-09-02T04:15:05.973 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 556 k 2022-09-02T04:15:05.973 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-09-02T04:15:06.175 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-17.0.0-14689.gafd42238.el8.x86_64.rpm 863 kB/s | 175 kB 00:00 2022-09-02T04:15:06.175 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-09-02T04:15:06.176 INFO:teuthology.orchestra.run.smithi099.stdout:Total 858 kB/s | 175 kB 00:00 2022-09-02T04:15:06.176 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-09-02T04:15:06.185 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-09-02T04:15:06.186 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-09-02T04:15:06.233 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-09-02T04:15:06.236 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-09-02T04:15:06.348 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-09-02T04:15:06.487 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:06.725 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 1/1 2022-09-02T04:15:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:07.100 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-09-02T04:15:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:17.0.0-14689.gafd42238.el8.x86_64 2022-09-02T04:15:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:07.101 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-09-02T04:15:07.270 DEBUG:teuthology.parallel:result is None 2022-09-02T04:15:07.272 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2022-09-02T04:15:07.312 INFO:teuthology.orchestra.run.smithi099.stdout:CentOSStream 2022-09-02T04:15:07.313 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-02T04:15:07.315 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=afd422385fab8119a539d289dd1aeaad4c056fe4 2022-09-02T04:15:07.480 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-02T04:15:07.520 INFO:teuthology.orchestra.run.smithi099.stdout:17.0.0-14689.gafd42238.el8 2022-09-02T04:15:07.521 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14689.gafd42238.el8 2022-09-02T04:15:07.521 INFO:teuthology.task.install:The correct ceph version 17.0.0-14689.gafd42238 is installed. 2022-09-02T04:15:07.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo lsb_release -is 2022-09-02T04:15:07.568 INFO:teuthology.orchestra.run.smithi160.stdout:CentOSStream 2022-09-02T04:15:07.569 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-09-02T04:15:07.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=afd422385fab8119a539d289dd1aeaad4c056fe4 2022-09-02T04:15:07.735 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-02T04:15:07.762 INFO:teuthology.orchestra.run.smithi160.stdout:17.0.0-14689.gafd42238.el8 2022-09-02T04:15:07.763 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14689.gafd42238.el8 2022-09-02T04:15:07.764 INFO:teuthology.task.install:The correct ceph version 17.0.0-14689.gafd42238 is installed. 2022-09-02T04:15:07.764 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-09-02T04:15:07.765 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:07.766 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-02T04:15:07.797 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:07.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-02T04:15:07.831 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-09-02T04:15:07.832 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:07.833 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2022-09-02T04:15:07.865 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-02T04:15:07.932 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:07.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2022-09-02T04:15:07.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-02T04:15:08.035 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-09-02T04:15:08.036 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:08.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-02T04:15:08.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-02T04:15:08.133 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:08.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-02T04:15:08.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-02T04:15:08.235 INFO:teuthology.run_tasks:Running task ceph... 2022-09-02T04:15:08.336 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-09-02T04:15:08.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2022-09-02T04:15:08.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /var/log/ceph 2022-09-02T04:15:08.370 INFO:tasks.ceph:Disabling ceph logrotate... 2022-09-02T04:15:08.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-02T04:15:08.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/logrotate.d/ceph 2022-09-02T04:15:08.440 INFO:tasks.ceph:Creating extra log directories... 2022-09-02T04:15:08.440 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-02T04:15:08.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-09-02T04:15:08.526 INFO:tasks.ceph:Creating ceph cluster c1... 2022-09-02T04:15:08.527 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-02T04:15:08.527 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440', 'branch': 'pr-47848', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7005440', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'pr-47848'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-rgw-s3select-fixes_related_to_new_csv_parser', 'sha1': '98002085f9003280d82d2f21aa1472b865694c0b'}}, 'owner': 'scheduled_cbodley@teuthology', 'priority': 75, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-qa-rgw-s3select-fixes_related_to_new_csv_parser', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '98002085f9003280d82d2f21aa1472b865694c0b', 'targets': {'smithi099.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDut5FvkIQGrPyM0nkT7N8tYudURf71e+foXRKk+M0U4z/GU7R1IHbYePTvXo0Xe2mDfrabdI/sBTAphzeJNUR4XNStF3M+BZId93/7mt0uJAqS5xbV6NkePILDmGdtrgPQRtdtowRiH1XYTgIQnWFMjqntlrki93aFphwIR9DrcTQWwsnCuXH+yUIpu9mLA51Avg4P38VqnycKGjsK35X2oTWzkXVBdG6ycgSNM2IpvdWGA30FknJoOnXqcDJA9lyoXFDffAo2SapEUC7oOP3MPfipbqb9bvJx4BD7Jt2RuPAPnB7nhMuzZXu5TrXgAPMyG8+zfviSPj8ZSFaYKKCweMu4tPvcx3J8qfSzux+qT3otsh9Npo9zdlFrboVFsacTPN70nzngBW1WM4htz+ZdJQDyCd4f1amTu/V5IHTCezhr39kgZGa4oG1DU6jgyyYq6D/KvQCiw55w6lkKf0dfkSyT8yp0iGf7L9W8su8g3C8v+6vRyPg7AtDZm4L93Fc=', 'smithi160.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiwRVfUW8pFNs8ic5wz3OsS4S3a85p5doefMIekSi2aNJwcSHRdMPcJn4h+vlz5R5tj8i8uNPDyQjNwyBcM/wZJ7x0J1f/mFWwx4Zi0liFV7F+WPEdVQYVlLfDuspvkpjODvwSW92QU/8owPNBNLnutasr0SpaHutkXZjsYn7J0ybzdz8lf/HEriZo/5AYTrQyH1LZhhcIoIyvUOBjCQHC8L0LePd9lj+nZ13l/EX+QhsZRBYJh+31IerMPCP0/gk+a41JTqRjJJefFuzVFcudPBqIUwQxGzyT19IHp0O69MXWCH4kLupGDkdhjbQ7+j+fO6fFqOk+aJiGDwWsAChhuv9XeH9osWOUBsUopb7NMICRj0b9EFTKWzB7Ioyooor+ny7gbW31zy/Q+s/NgPwg/IP5lv1KRZYQk7JXEc8liG4q/N5kMCsybGeenr53SFGwm/zWfeNNKCaejo4WvZjtbd6W9lpwFptv/RvCmPFaQMY3z1KcDFg1EkODzuDis6k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-01_15:28:29', 'tube': 'smithi', 'user': 'cbodley', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-09-02T04:15:08.528 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-02T04:15:08.550 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-09-02T04:15:08.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2022-09-02T04:15:08.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2022-09-02T04:15:08.645 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:08.645 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2022-09-02T04:15:08.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-02T04:15:08.707 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-02T04:15:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-02T04:15:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:08.763 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 360378 Links: 1 2022-09-02T04:15:08.763 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:08.763 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:08.763 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-09-02 04:15:03.574468834 +0000 2022-09-02T04:15:08.763 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-09-02 04:11:38.577785829 +0000 2022-09-02T04:15:08.764 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-09-02 04:11:38.577785829 +0000 2022-09-02T04:15:08.764 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-09-02T04:15:08.771 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-09-02T04:15:08.772 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-09-02T04:15:08.772 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000227606 s, 2.2 MB/s 2022-09-02T04:15:08.774 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-02T04:15:08.832 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-02T04:15:08.833 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:08.833 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 358315 Links: 1 2022-09-02T04:15:08.834 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:08.834 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:08.834 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-09-02 04:15:03.574468834 +0000 2022-09-02T04:15:08.835 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-09-02 04:11:39.127773190 +0000 2022-09-02T04:15:08.835 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-09-02 04:11:39.127773190 +0000 2022-09-02T04:15:08.835 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-09-02T04:15:08.843 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-09-02T04:15:08.844 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-09-02T04:15:08.844 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000285368 s, 1.8 MB/s 2022-09-02T04:15:08.846 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-02T04:15:08.912 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-02T04:15:08.912 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 371732 Links: 1 2022-09-02T04:15:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-09-02 04:15:03.574468834 +0000 2022-09-02T04:15:08.914 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-09-02 04:11:39.673760642 +0000 2022-09-02T04:15:08.914 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-09-02 04:11:39.673760642 +0000 2022-09-02T04:15:08.925 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-09-02T04:15:08.937 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-09-02T04:15:08.938 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-09-02T04:15:08.938 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.00020051 s, 2.6 MB/s 2022-09-02T04:15:08.938 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-02T04:15:08.996 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-02T04:15:08.997 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:08.997 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 367535 Links: 1 2022-09-02T04:15:08.997 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:08.998 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:08.998 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-09-02 04:15:03.574468834 +0000 2022-09-02T04:15:08.998 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-09-02 04:11:40.243747543 +0000 2022-09-02T04:15:08.998 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-09-02 04:11:40.243747543 +0000 2022-09-02T04:15:08.999 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-09-02T04:15:09.006 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-09-02T04:15:09.006 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-09-02T04:15:09.007 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000216495 s, 2.4 MB/s 2022-09-02T04:15:09.009 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:09.010 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-02T04:15:09.010 INFO:tasks.ceph:Generating config... 2022-09-02T04:15:09.012 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-02T04:15:09.012 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-02T04:15:09.013 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-02T04:15:09.013 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-02T04:15:09.013 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-02T04:15:09.014 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-02T04:15:09.014 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-02T04:15:09.014 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-02T04:15:09.015 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-02T04:15:09.015 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-02T04:15:09.015 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-02T04:15:09.016 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-02T04:15:09.016 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-02T04:15:09.016 INFO:tasks.ceph:[client] setuser = ceph 2022-09-02T04:15:09.017 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-02T04:15:09.017 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-02T04:15:09.017 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-02T04:15:09.017 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-02T04:15:09.018 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-02T04:15:09.018 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-02T04:15:09.018 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-02T04:15:09.019 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-02T04:15:09.019 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-02T04:15:09.019 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-02T04:15:09.020 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-02T04:15:09.020 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-02T04:15:09.020 INFO:tasks.ceph:Setting up c1.mon.a... 2022-09-02T04:15:09.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2022-09-02T04:15:09.096 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.keyring 2022-09-02T04:15:09.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2022-09-02T04:15:09.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-02T04:15:09.219 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.99')] 2022-09-02T04:15:09.220 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.99'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-02T04:15:09.223 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:09.223 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-02T04:15:09.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.99 --print /home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:09.364 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:15:09.364 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:15:09.366 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:09.366 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid f6bb112b-fcb8-4b48-add2-6218ff82c89e 2022-09-02T04:15:09.366 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = octopus 2022-09-02T04:15:09.367 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2022-09-02T04:15:09.367 INFO:teuthology.orchestra.run.smithi099.stdout:fsid f6bb112b-fcb8-4b48-add2-6218ff82c89e 2022-09-02T04:15:09.367 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2022-09-02T04:15:09.363250+0000 2022-09-02T04:15:09.369 INFO:teuthology.orchestra.run.smithi099.stdout:created 2022-09-02T04:15:09.363250+0000 2022-09-02T04:15:09.369 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 15 (octopus) 2022-09-02T04:15:09.370 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 1 2022-09-02T04:15:09.370 INFO:teuthology.orchestra.run.smithi099.stdout:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2022-09-02T04:15:09.376 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2022-09-02T04:15:09.377 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-02T04:15:09.427 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID f6bb112b-fcb8-4b48-add2-6218ff82c89e... 2022-09-02T04:15:09.429 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-02T04:15:09.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-09-02T04:15:09.511 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2022-09-02T04:15:09.512 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2022-09-02T04:15:09.512 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-02T04:15:09.513 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2022-09-02T04:15:09.513 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.514 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2022-09-02T04:15:09.514 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.514 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2022-09-02T04:15:09.515 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.515 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2022-09-02T04:15:09.516 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2022-09-02T04:15:09.516 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.517 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2022-09-02T04:15:09.517 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2022-09-02T04:15:09.518 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.518 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-02T04:15:09.522 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2022-09-02T04:15:09.522 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.522 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2022-09-02T04:15:09.523 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-02T04:15:09.523 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.524 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2022-09-02T04:15:09.525 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.525 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2022-09-02T04:15:09.527 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2022-09-02T04:15:09.527 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2022-09-02T04:15:09.527 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2022-09-02T04:15:09.528 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2022-09-02T04:15:09.528 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2022-09-02T04:15:09.528 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2022-09-02T04:15:09.529 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-02T04:15:09.529 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-02T04:15:09.529 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2022-09-02T04:15:09.530 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.530 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-02T04:15:09.531 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.531 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2022-09-02T04:15:09.532 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.533 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2022-09-02T04:15:09.533 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.534 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2022-09-02T04:15:09.534 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2022-09-02T04:15:09.534 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2022-09-02T04:15:09.536 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = 172.21.15.99 2022-09-02T04:15:09.536 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = f6bb112b-fcb8-4b48-add2-6218ff82c89e 2022-09-02T04:15:09.538 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.538 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2022-09-02T04:15:09.538 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2022-09-02T04:15:09.539 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.539 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2022-09-02T04:15:09.541 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2022-09-02T04:15:09.541 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.541 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2022-09-02T04:15:09.541 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2022-09-02T04:15:09.542 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.542 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2022-09-02T04:15:09.542 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2022-09-02T04:15:09.543 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2022-09-02T04:15:09.543 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.543 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2022-09-02T04:15:09.543 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2022-09-02T04:15:09.544 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.544 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2022-09-02T04:15:09.544 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2022-09-02T04:15:09.546 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2022-09-02T04:15:09.547 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2022-09-02T04:15:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2022-09-02T04:15:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2022-09-02T04:15:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2022-09-02T04:15:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-09-02T04:15:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2022-09-02T04:15:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: rocksdb delete range threshold = 0 2022-09-02T04:15:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.551 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2022-09-02T04:15:09.552 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-09-02T04:15:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2022-09-02T04:15:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2022-09-02T04:15:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2022-09-02T04:15:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2022-09-02T04:15:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.555 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2022-09-02T04:15:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-09-02T04:15:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2022-09-02T04:15:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2022-09-02T04:15:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2022-09-02T04:15:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2022-09-02T04:15:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2022-09-02T04:15:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2022-09-02T04:15:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2022-09-02T04:15:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-02T04:15:09.559 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2022-09-02T04:15:09.559 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2022-09-02T04:15:09.559 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2022-09-02T04:15:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-02T04:15:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-02T04:15:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2022-09-02T04:15:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2022-09-02T04:15:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2022-09-02T04:15:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:09.562 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2022-09-02T04:15:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2022-09-02T04:15:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2022-09-02T04:15:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2022-09-02T04:15:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-02T04:15:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-02T04:15:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw = 20 2022-09-02T04:15:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw sync = 20 2022-09-02T04:15:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt require ssl = False 2022-09-02T04:15:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms backend = testing 2022-09-02T04:15:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-02T04:15:09.568 INFO:teuthology.orchestra.run.smithi099.stdout: rgw curl low speed time = 300 2022-09-02T04:15:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data log num shards = 4 2022-09-02T04:15:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: rgw md log max shards = 4 2022-09-02T04:15:09.569 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync data inject err probability = 0.1 2022-09-02T04:15:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync log trim interval = 0 2022-09-02T04:15:09.570 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync meta inject err probability = 0.1 2022-09-02T04:15:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync obj etag verify = True 2022-09-02T04:15:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: setgroup = ceph 2022-09-02T04:15:09.571 INFO:teuthology.orchestra.run.smithi099.stdout: setuser = ceph 2022-09-02T04:15:09.572 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2022-09-02T04:15:09.572 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2022-09-02T04:15:09.573 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data notify interval msec = 0 2022-09-02T04:15:09.574 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2022-09-02T04:15:09.574 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10 2022-09-02T04:15:09.575 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2022-09-02T04:15:09.575 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 1000 2022-09-02T04:15:09.576 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2022-09-02T04:15:09.576 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10000 2022-09-02T04:15:09.579 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2022-09-02T04:15:09.579 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2022-09-02T04:15:09.580 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-02T04:15:09.580 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2022-09-02T04:15:09.581 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.581 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2022-09-02T04:15:09.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2022-09-02T04:15:09.583 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.583 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2022-09-02T04:15:09.584 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2022-09-02T04:15:09.585 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.585 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2022-09-02T04:15:09.585 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2022-09-02T04:15:09.586 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.586 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-02T04:15:09.586 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2022-09-02T04:15:09.587 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.587 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2022-09-02T04:15:09.587 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-02T04:15:09.587 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.588 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2022-09-02T04:15:09.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.588 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2022-09-02T04:15:09.588 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2022-09-02T04:15:09.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2022-09-02T04:15:09.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2022-09-02T04:15:09.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2022-09-02T04:15:09.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2022-09-02T04:15:09.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2022-09-02T04:15:09.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-02T04:15:09.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-02T04:15:09.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2022-09-02T04:15:09.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-02T04:15:09.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2022-09-02T04:15:09.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.592 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2022-09-02T04:15:09.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.593 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2022-09-02T04:15:09.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2022-09-02T04:15:09.593 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2022-09-02T04:15:09.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.99 2022-09-02T04:15:09.594 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = f6bb112b-fcb8-4b48-add2-6218ff82c89e 2022-09-02T04:15:09.595 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.595 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2022-09-02T04:15:09.595 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2022-09-02T04:15:09.596 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.596 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2022-09-02T04:15:09.596 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2022-09-02T04:15:09.596 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.597 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2022-09-02T04:15:09.597 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2022-09-02T04:15:09.597 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.597 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2022-09-02T04:15:09.598 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2022-09-02T04:15:09.598 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2022-09-02T04:15:09.598 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.599 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2022-09-02T04:15:09.599 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2022-09-02T04:15:09.599 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.599 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2022-09-02T04:15:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2022-09-02T04:15:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.600 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2022-09-02T04:15:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2022-09-02T04:15:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2022-09-02T04:15:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.601 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2022-09-02T04:15:09.602 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2022-09-02T04:15:09.602 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2022-09-02T04:15:09.602 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2022-09-02T04:15:09.602 INFO:teuthology.orchestra.run.smithi160.stdout: rocksdb delete range threshold = 0 2022-09-02T04:15:09.603 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.603 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2022-09-02T04:15:09.604 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2022-09-02T04:15:09.604 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2022-09-02T04:15:09.604 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2022-09-02T04:15:09.604 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2022-09-02T04:15:09.605 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:09.605 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:09.606 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2022-09-02T04:15:09.606 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.606 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2022-09-02T04:15:09.606 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2022-09-02T04:15:09.607 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2022-09-02T04:15:09.607 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2022-09-02T04:15:09.607 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2022-09-02T04:15:09.607 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2022-09-02T04:15:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2022-09-02T04:15:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2022-09-02T04:15:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2022-09-02T04:15:09.608 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-02T04:15:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2022-09-02T04:15:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2022-09-02T04:15:09.609 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.610 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2022-09-02T04:15:09.610 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-02T04:15:09.610 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-02T04:15:09.610 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.611 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2022-09-02T04:15:09.611 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2022-09-02T04:15:09.611 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.611 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2022-09-02T04:15:09.612 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:09.612 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2022-09-02T04:15:09.612 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2022-09-02T04:15:09.612 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2022-09-02T04:15:09.612 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2022-09-02T04:15:09.613 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-02T04:15:09.613 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-02T04:15:09.613 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw = 20 2022-09-02T04:15:09.613 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw sync = 20 2022-09-02T04:15:09.614 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt require ssl = False 2022-09-02T04:15:09.614 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms backend = testing 2022-09-02T04:15:09.614 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-02T04:15:09.614 INFO:teuthology.orchestra.run.smithi160.stdout: rgw curl low speed time = 300 2022-09-02T04:15:09.615 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data log num shards = 4 2022-09-02T04:15:09.615 INFO:teuthology.orchestra.run.smithi160.stdout: rgw md log max shards = 4 2022-09-02T04:15:09.615 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync data inject err probability = 0.1 2022-09-02T04:15:09.615 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync log trim interval = 0 2022-09-02T04:15:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync meta inject err probability = 0.1 2022-09-02T04:15:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync obj etag verify = True 2022-09-02T04:15:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: setgroup = ceph 2022-09-02T04:15:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: setuser = ceph 2022-09-02T04:15:09.616 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2022-09-02T04:15:09.617 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2022-09-02T04:15:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data notify interval msec = 0 2022-09-02T04:15:09.617 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2022-09-02T04:15:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10 2022-09-02T04:15:09.618 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2022-09-02T04:15:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 1000 2022-09-02T04:15:09.618 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2022-09-02T04:15:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10000 2022-09-02T04:15:09.619 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2022-09-02T04:15:09.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2022-09-02T04:15:09.677 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-02T04:15:09.678 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:09.679 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2022-09-02T04:15:09.700 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:09.701 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2022-09-02T04:15:09.770 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2022-09-02T04:15:09.771 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:09.772 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/c1.keyring 2022-09-02T04:15:09.772 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-02T04:15:09.856 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:09.857 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:09.915 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2022-09-02T04:15:09.916 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:09.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/c1.keyring 2022-09-02T04:15:09.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-09-02T04:15:09.963 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:09.964 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:10.028 INFO:tasks.ceph:Setting up mon nodes... 2022-09-02T04:15:10.029 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-02T04:15:10.030 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2022-09-02T04:15:10.084 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2022-09-02T04:15:10.087 INFO:tasks.ceph:Setting up mds nodes... 2022-09-02T04:15:10.087 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-02T04:15:10.088 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2022-09-02T04:15:10.133 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.client.0.keyring 2022-09-02T04:15:10.145 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2022-09-02T04:15:10.228 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.client.1.keyring 2022-09-02T04:15:10.241 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-02T04:15:10.242 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-02T04:15:10.243 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2022-09-02T04:15:10.309 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:10.310 INFO:tasks.ceph:role: c1.osd.0 2022-09-02T04:15:10.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2022-09-02T04:15:10.311 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-02T04:15:10.384 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-02T04:15:10.385 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-02T04:15:10.385 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-02T04:15:10.385 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-02T04:15:10.385 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-02T04:15:10.386 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-09-02T04:15:10.386 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-02T04:15:10.386 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-02T04:15:10.386 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-02T04:15:10.387 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-02T04:15:10.823 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-09-02T04:15:10.824 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-09-02T04:15:10.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2022-09-02T04:15:11.004 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2022-09-02T04:15:11.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2022-09-02T04:15:11.115 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:11.115 INFO:tasks.ceph:role: c1.osd.1 2022-09-02T04:15:11.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2022-09-02T04:15:11.116 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-02T04:15:11.200 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-02T04:15:11.201 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-02T04:15:11.201 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-02T04:15:11.201 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-02T04:15:11.202 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-02T04:15:11.202 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-09-02T04:15:11.202 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-02T04:15:11.202 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-02T04:15:11.203 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-02T04:15:11.203 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-02T04:15:11.600 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-09-02T04:15:11.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-09-02T04:15:11.604 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2022-09-02T04:15:11.713 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2022-09-02T04:15:11.748 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2022-09-02T04:15:11.820 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:11.821 INFO:tasks.ceph:role: c1.osd.2 2022-09-02T04:15:11.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2022-09-02T04:15:11.822 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-02T04:15:11.895 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-02T04:15:11.896 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-02T04:15:11.896 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-02T04:15:11.896 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-02T04:15:11.896 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-02T04:15:11.897 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-09-02T04:15:11.897 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-02T04:15:11.897 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-02T04:15:11.897 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-02T04:15:11.898 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-02T04:15:12.342 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-09-02T04:15:12.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-09-02T04:15:12.345 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2022-09-02T04:15:12.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2022-09-02T04:15:12.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:12.622 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:12.621+0000 7f9c6efe2440 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2022-09-02T04:15:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:12.624+0000 7f9c6efe2440 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2022-09-02T04:15:12.626 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:12.624+0000 7f9c6efe2440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-02T04:15:12.626 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:12.624+0000 7f9c6efe2440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-02T04:15:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:12.624+0000 7f9c6efe2440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-09-02T04:15:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:12.630+0000 7f9c6efe2440 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2022-09-02T04:15:14.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2022-09-02T04:15:14.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:14.850+0000 7fbeb2e77440 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2022-09-02T04:15:14.859 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:14.850+0000 7fbeb2e77440 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2022-09-02T04:15:14.859 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:14.850+0000 7fbeb2e77440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-02T04:15:14.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:14.850+0000 7fbeb2e77440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-02T04:15:14.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:14.851+0000 7fbeb2e77440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-09-02T04:15:14.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:14.851+0000 7fbeb2e77440 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2022-09-02T04:15:16.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2022-09-02T04:15:16.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:17.085+0000 7f771707b440 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2022-09-02T04:15:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:17.086+0000 7f771707b440 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2022-09-02T04:15:17.090 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:17.086+0000 7f771707b440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-02T04:15:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:17.086+0000 7f771707b440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-02T04:15:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:17.086+0000 7f771707b440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-09-02T04:15:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:15:17.086+0000 7f771707b440 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2022-09-02T04:15:19.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2022-09-02T04:15:19.185 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-02T04:15:19.186 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:19.187 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2022-09-02T04:15:19.256 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:19.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2022-09-02T04:15:19.324 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:19.324 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2022-09-02T04:15:19.394 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:19.395 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2022-09-02T04:15:19.466 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:19.467 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2022-09-02T04:15:19.528 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:19.529 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2022-09-02T04:15:19.589 INFO:tasks.ceph:Adding keys to all mons... 2022-09-02T04:15:19.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/c1.keyring 2022-09-02T04:15:19.657 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2022-09-02T04:15:19.658 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBOgxFjXucBBRAAsekWNs3JhnJn6JlQHfPFAw== 2022-09-02T04:15:19.658 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2022-09-02T04:15:19.658 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBQgxFjFM4aJRAAg6oqdlPYiDmi0YxS8iaSFQ== 2022-09-02T04:15:19.659 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2022-09-02T04:15:19.659 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBSgxFjqWXEMhAAq58A5hurWcSPi8lcCtIAvw== 2022-09-02T04:15:19.659 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2022-09-02T04:15:19.660 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBVgxFj19gtBRAA8U3nzk8g5h0AWVR8HNGvjQ== 2022-09-02T04:15:19.660 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2022-09-02T04:15:19.660 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBOgxFjcVPgBxAAyC7KRLbdeAsfP8tf/JzNQw== 2022-09-02T04:15:19.661 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2022-09-02T04:15:19.661 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBOgxFjXOScDRAAnKuY7O4TkYa2SgPwLG0Qkg== 2022-09-02T04:15:19.663 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-02T04:15:19.758 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-02T04:15:19.812 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-02T04:15:19.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-02T04:15:19.920 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-02T04:15:19.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-02T04:15:20.033 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-02T04:15:20.034 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2022-09-02T04:15:20.063 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2022-09-02T04:15:20.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2022-09-02T04:15:20.189 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/c1.monmap 2022-09-02T04:15:20.251 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2022-09-02T04:15:20.252 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2022-09-02T04:15:20.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2022-09-02T04:15:20.296 INFO:tasks.ceph.c1.mon.a:Started 2022-09-02T04:15:20.297 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2022-09-02T04:15:20.298 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2022-09-02T04:15:20.300 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2022-09-02T04:15:20.302 INFO:tasks.ceph.c1.mgr.x:Started 2022-09-02T04:15:20.303 INFO:tasks.ceph:Setting crush tunables to default 2022-09-02T04:15:20.304 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd crush tunables default 2022-09-02T04:15:20.507 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2022-09-02T04:15:20.518 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2022-09-02T04:15:20.519 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:20.519 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2022-09-02T04:15:20.545 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:20.546 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2022-09-02T04:15:20.616 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:20.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2022-09-02T04:15:20.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new 8026c565-6db1-4c08-b7e3-dcefb08471a6 0 2022-09-02T04:15:20.938 INFO:teuthology.orchestra.run.smithi099.stdout:0 2022-09-02T04:15:20.951 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd 1 2022-09-02T04:15:21.162 INFO:teuthology.orchestra.run.smithi099.stdout:1 2022-09-02T04:15:21.184 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new 39182c5a-9069-45f9-9a4b-c0315d741713 2 2022-09-02T04:15:21.395 INFO:teuthology.orchestra.run.smithi099.stdout:2 2022-09-02T04:15:21.408 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2022-09-02T04:15:21.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2022-09-02T04:15:21.411 INFO:tasks.ceph.c1.osd.0:Started 2022-09-02T04:15:21.411 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2022-09-02T04:15:21.412 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2022-09-02T04:15:21.414 INFO:tasks.ceph.c1.osd.1:Started 2022-09-02T04:15:21.414 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2022-09-02T04:15:21.414 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2022-09-02T04:15:21.416 INFO:tasks.ceph.c1.osd.2:Started 2022-09-02T04:15:21.417 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-02T04:15:21.534 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:21.533+0000 7f836e994080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-02T04:15:21.626 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:21.624+0000 7f836e994080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-02T04:15:21.679 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:21.680 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"fsid":"f6bb112b-fcb8-4b48-add2-6218ff82c89e","created":"2022-09-02T04:15:20.412216+0000","modified":"2022-09-02T04:15:21.393347+0000","last_up_change":"0.000000","last_in_change":"2022-09-02T04:15:21.393347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8026c565-6db1-4c08-b7e3-dcefb08471a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39182c5a-9069-45f9-9a4b-c0315d741713","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:21.698 INFO:tasks.ceph.ceph_manager.c1:[] 2022-09-02T04:15:21.699 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-02T04:15:21.717 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:21.715+0000 7f836e994080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-02T04:15:21.901 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:21.898+0000 7f836e994080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-02T04:15:22.007 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-02T04:15:22.016 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2022-09-02T04:15:22.015+0000 7f31b4b9f440 -1 Falling back to public interface 2022-09-02T04:15:22.025 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2022-09-02T04:15:22.021+0000 7f3360630440 -1 Falling back to public interface 2022-09-02T04:15:22.043 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2022-09-02T04:15:22.040+0000 7fbee743a440 -1 Falling back to public interface 2022-09-02T04:15:22.198 INFO:teuthology.misc.health.smithi099.stdout: 2022-09-02T04:15:22.199 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":5,"fsid":"f6bb112b-fcb8-4b48-add2-6218ff82c89e","created":"2022-09-02T04:15:20.412216+0000","modified":"2022-09-02T04:15:21.393347+0000","last_up_change":"0.000000","last_in_change":"2022-09-02T04:15:21.393347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8026c565-6db1-4c08-b7e3-dcefb08471a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39182c5a-9069-45f9-9a4b-c0315d741713","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:22.209 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-02T04:15:22.686 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:22.682+0000 7f836e994080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-02T04:15:22.821 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:22.819+0000 7f836e994080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-02T04:15:23.040 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.038+0000 7f836e994080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-02T04:15:23.108 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.106+0000 7f836e994080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-02T04:15:23.243 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.241+0000 7f836e994080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-02T04:15:23.504 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.502+0000 7f836e994080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-02T04:15:23.684 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.682+0000 7f836e994080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-02T04:15:23.822 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.821+0000 7f836e994080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-02T04:15:23.935 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:23.934+0000 7f836e994080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-02T04:15:24.073 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:24.071+0000 7f836e994080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-02T04:15:24.150 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:24.148+0000 7f836e994080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-02T04:15:24.365 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:24.364+0000 7f836e994080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-02T04:15:24.441 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:24.440+0000 7f836e994080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-02T04:15:25.037 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.036+0000 7f836e994080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-02T04:15:25.287 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.286+0000 7f836e994080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-02T04:15:25.359 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.358+0000 7f836e994080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-02T04:15:25.440 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.438+0000 7f836e994080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-02T04:15:25.618 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.616+0000 7f836e994080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-02T04:15:25.688 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.686+0000 7f836e994080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-02T04:15:25.835 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:25.834+0000 7f836e994080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-02T04:15:26.855 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2022-09-02T04:15:26.854+0000 7f8328771700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-02T04:15:28.513 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-02T04:15:28.819 INFO:teuthology.misc.health.smithi099.stdout: 2022-09-02T04:15:28.820 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":5,"fsid":"f6bb112b-fcb8-4b48-add2-6218ff82c89e","created":"2022-09-02T04:15:20.412216+0000","modified":"2022-09-02T04:15:21.393347+0000","last_up_change":"0.000000","last_in_change":"2022-09-02T04:15:21.393347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8026c565-6db1-4c08-b7e3-dcefb08471a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"39182c5a-9069-45f9-9a4b-c0315d741713","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:28.831 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-02T04:15:29.784 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2022-09-02T04:15:29.783+0000 7f31b4b9f440 -1 osd.0 0 log_to_monitors true 2022-09-02T04:15:29.790 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2022-09-02T04:15:29.789+0000 7f3360630440 -1 osd.2 0 log_to_monitors true 2022-09-02T04:15:29.826 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2022-09-02T04:15:29.823+0000 7fbee743a440 -1 osd.1 0 log_to_monitors true 2022-09-02T04:15:32.083 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2022-09-02T04:15:32.082+0000 7f31a894c700 -1 osd.0 0 waiting for initial osdmap 2022-09-02T04:15:32.095 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2022-09-02T04:15:32.094+0000 7f31a1528700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-02T04:15:32.096 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2022-09-02T04:15:32.094+0000 7f33543dd700 -1 osd.2 0 waiting for initial osdmap 2022-09-02T04:15:32.105 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2022-09-02T04:15:32.104+0000 7f334cfb9700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-02T04:15:32.108 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2022-09-02T04:15:32.107+0000 7fbedb1e7700 -1 osd.1 0 waiting for initial osdmap 2022-09-02T04:15:32.121 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2022-09-02T04:15:32.120+0000 7fbed3dc3700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-02T04:15:35.144 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-09-02T04:15:35.530 INFO:teuthology.misc.health.smithi099.stdout: 2022-09-02T04:15:35.531 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"f6bb112b-fcb8-4b48-add2-6218ff82c89e","created":"2022-09-02T04:15:20.412216+0000","modified":"2022-09-02T04:15:35.112374+0000","last_up_change":"2022-09-02T04:15:33.105745+0000","last_in_change":"2022-09-02T04:15:21.393347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-02T04:15:33.893514+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8026c565-6db1-4c08-b7e3-dcefb08471a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":104038},{"type":"v1","addr":"172.21.15.99:6801","nonce":104038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":104038},{"type":"v1","addr":"172.21.15.99:6803","nonce":104038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":104038},{"type":"v1","addr":"172.21.15.99:6807","nonce":104038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":104038},{"type":"v1","addr":"172.21.15.99:6805","nonce":104038}]},"public_addr":"172.21.15.99:6801/104038","cluster_addr":"172.21.15.99:6803/104038","heartbeat_back_addr":"172.21.15.99:6807/104038","heartbeat_front_addr":"172.21.15.99:6805/104038","state":["exists","up"]},{"osd":1,"uuid":"f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":104055},{"type":"v1","addr":"172.21.15.99:6817","nonce":104055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":104055},{"type":"v1","addr":"172.21.15.99:6819","nonce":104055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":104055},{"type":"v1","addr":"172.21.15.99:6823","nonce":104055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":104055},{"type":"v1","addr":"172.21.15.99:6821","nonce":104055}]},"public_addr":"172.21.15.99:6817/104055","cluster_addr":"172.21.15.99:6819/104055","heartbeat_back_addr":"172.21.15.99:6823/104055","heartbeat_front_addr":"172.21.15.99:6821/104055","state":["exists","up"]},{"osd":2,"uuid":"39182c5a-9069-45f9-9a4b-c0315d741713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":104039},{"type":"v1","addr":"172.21.15.99:6809","nonce":104039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":104039},{"type":"v1","addr":"172.21.15.99:6811","nonce":104039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":104039},{"type":"v1","addr":"172.21.15.99:6815","nonce":104039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":104039},{"type":"v1","addr":"172.21.15.99:6813","nonce":104039}]},"public_addr":"172.21.15.99:6809/104039","cluster_addr":"172.21.15.99:6811/104039","heartbeat_back_addr":"172.21.15.99:6815/104039","heartbeat_front_addr":"172.21.15.99:6813/104039","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.786570+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.859791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.767524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:35.545 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-02T04:15:35.546 INFO:tasks.ceph:Creating RBD pool 2022-09-02T04:15:35.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd pool create rbd 8 2022-09-02T04:15:36.122 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'rbd' created 2022-09-02T04:15:36.135 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-02T04:15:37.134 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-02T04:15:37.145 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2022-09-02T04:15:37.161 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2022-09-02T04:15:37.162 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2022-09-02T04:15:37.162 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2022-09-02T04:15:37.163 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-02T04:15:37.525 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:37.537 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":103631},{"type":"v1","addr":"172.21.15.99:6825","nonce":103631}]},"active_addr":"172.21.15.99:6825/103631","active_change":"2022-09-02T04:15:25.847826+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":1214939190}]},{"addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":2782659524}]},{"addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":3000217521}]},{"addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":2448156510}]}]}} 2022-09-02T04:15:37.543 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2022-09-02T04:15:37.543 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2022-09-02T04:15:37.544 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-02T04:15:37.851 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:37.852 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":12,"fsid":"f6bb112b-fcb8-4b48-add2-6218ff82c89e","created":"2022-09-02T04:15:20.412216+0000","modified":"2022-09-02T04:15:37.119377+0000","last_up_change":"2022-09-02T04:15:33.105745+0000","last_in_change":"2022-09-02T04:15:21.393347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-02T04:15:33.893514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-02T04:15:35.869733+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8026c565-6db1-4c08-b7e3-dcefb08471a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":104038},{"type":"v1","addr":"172.21.15.99:6801","nonce":104038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":104038},{"type":"v1","addr":"172.21.15.99:6803","nonce":104038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":104038},{"type":"v1","addr":"172.21.15.99:6807","nonce":104038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":104038},{"type":"v1","addr":"172.21.15.99:6805","nonce":104038}]},"public_addr":"172.21.15.99:6801/104038","cluster_addr":"172.21.15.99:6803/104038","heartbeat_back_addr":"172.21.15.99:6807/104038","heartbeat_front_addr":"172.21.15.99:6805/104038","state":["exists","up"]},{"osd":1,"uuid":"f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":104055},{"type":"v1","addr":"172.21.15.99:6817","nonce":104055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":104055},{"type":"v1","addr":"172.21.15.99:6819","nonce":104055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":104055},{"type":"v1","addr":"172.21.15.99:6823","nonce":104055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":104055},{"type":"v1","addr":"172.21.15.99:6821","nonce":104055}]},"public_addr":"172.21.15.99:6817/104055","cluster_addr":"172.21.15.99:6819/104055","heartbeat_back_addr":"172.21.15.99:6823/104055","heartbeat_front_addr":"172.21.15.99:6821/104055","state":["exists","up"]},{"osd":2,"uuid":"39182c5a-9069-45f9-9a4b-c0315d741713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":104039},{"type":"v1","addr":"172.21.15.99:6809","nonce":104039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":104039},{"type":"v1","addr":"172.21.15.99:6811","nonce":104039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":104039},{"type":"v1","addr":"172.21.15.99:6815","nonce":104039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":104039},{"type":"v1","addr":"172.21.15.99:6813","nonce":104039}]},"public_addr":"172.21.15.99:6809/104039","cluster_addr":"172.21.15.99:6811/104039","heartbeat_back_addr":"172.21.15.99:6815/104039","heartbeat_front_addr":"172.21.15.99:6813/104039","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.786570+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.859791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.767524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:37.864 INFO:tasks.ceph.ceph_manager.c1:all up! 2022-09-02T04:15:37.865 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-09-02T04:15:38.179 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:38.179 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"fsid":"f6bb112b-fcb8-4b48-add2-6218ff82c89e","created":"2022-09-02T04:15:20.412216+0000","modified":"2022-09-02T04:15:38.128618+0000","last_up_change":"2022-09-02T04:15:33.105745+0000","last_in_change":"2022-09-02T04:15:21.393347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-02T04:15:33.893514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-02T04:15:35.869733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8026c565-6db1-4c08-b7e3-dcefb08471a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":104038},{"type":"v1","addr":"172.21.15.99:6801","nonce":104038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":104038},{"type":"v1","addr":"172.21.15.99:6803","nonce":104038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":104038},{"type":"v1","addr":"172.21.15.99:6807","nonce":104038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":104038},{"type":"v1","addr":"172.21.15.99:6805","nonce":104038}]},"public_addr":"172.21.15.99:6801/104038","cluster_addr":"172.21.15.99:6803/104038","heartbeat_back_addr":"172.21.15.99:6807/104038","heartbeat_front_addr":"172.21.15.99:6805/104038","state":["exists","up"]},{"osd":1,"uuid":"f8d06faa-3c2a-40e7-b095-f25f3cd3c3cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":104055},{"type":"v1","addr":"172.21.15.99:6817","nonce":104055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":104055},{"type":"v1","addr":"172.21.15.99:6819","nonce":104055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":104055},{"type":"v1","addr":"172.21.15.99:6823","nonce":104055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":104055},{"type":"v1","addr":"172.21.15.99:6821","nonce":104055}]},"public_addr":"172.21.15.99:6817/104055","cluster_addr":"172.21.15.99:6819/104055","heartbeat_back_addr":"172.21.15.99:6823/104055","heartbeat_front_addr":"172.21.15.99:6821/104055","state":["exists","up"]},{"osd":2,"uuid":"39182c5a-9069-45f9-9a4b-c0315d741713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":104039},{"type":"v1","addr":"172.21.15.99:6809","nonce":104039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":104039},{"type":"v1","addr":"172.21.15.99:6811","nonce":104039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":104039},{"type":"v1","addr":"172.21.15.99:6815","nonce":104039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":104039},{"type":"v1","addr":"172.21.15.99:6813","nonce":104039}]},"public_addr":"172.21.15.99:6809/104039","cluster_addr":"172.21.15.99:6811/104039","heartbeat_back_addr":"172.21.15.99:6815/104039","heartbeat_front_addr":"172.21.15.99:6813/104039","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.786570+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.859791+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:15:30.767524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:38.187 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2022-09-02T04:15:38.188 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2022-09-02T04:15:38.188 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2022-09-02T04:15:38.353 INFO:teuthology.orchestra.run.smithi099.stdout:34359738372 2022-09-02T04:15:38.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-02T04:15:38.367 INFO:teuthology.orchestra.run.smithi099.stdout:34359738372 2022-09-02T04:15:38.368 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-02T04:15:38.370 INFO:teuthology.orchestra.run.smithi099.stdout:34359738372 2022-09-02T04:15:38.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-02T04:15:38.679 INFO:teuthology.orchestra.run.smithi099.stdout:34359738371 2022-09-02T04:15:38.694 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2022-09-02T04:15:38.734 INFO:teuthology.orchestra.run.smithi099.stdout:34359738371 2022-09-02T04:15:38.741 INFO:teuthology.orchestra.run.smithi099.stdout:34359738371 2022-09-02T04:15:38.747 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.2 2022-09-02T04:15:38.753 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.1 2022-09-02T04:15:39.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-02T04:15:39.748 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-02T04:15:39.754 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-09-02T04:15:40.050 INFO:teuthology.orchestra.run.smithi099.stdout:34359738371 2022-09-02T04:15:40.064 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2022-09-02T04:15:40.083 INFO:teuthology.orchestra.run.smithi099.stdout:34359738371 2022-09-02T04:15:40.096 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.2 2022-09-02T04:15:40.138 INFO:teuthology.orchestra.run.smithi099.stdout:34359738373 2022-09-02T04:15:40.158 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738373 for osd.1 2022-09-02T04:15:40.159 DEBUG:teuthology.parallel:result is None 2022-09-02T04:15:41.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-09-02T04:15:41.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-09-02T04:15:41.408 INFO:teuthology.orchestra.run.smithi099.stdout:34359738373 2022-09-02T04:15:41.424 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738373 for osd.0 2022-09-02T04:15:41.425 DEBUG:teuthology.parallel:result is None 2022-09-02T04:15:41.445 INFO:teuthology.orchestra.run.smithi099.stdout:34359738373 2022-09-02T04:15:41.459 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738373 for osd.2 2022-09-02T04:15:41.460 DEBUG:teuthology.parallel:result is None 2022-09-02T04:15:41.461 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2022-09-02T04:15:41.462 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-02T04:15:41.799 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:41.801 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2022-09-02T04:15:41.813 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2022-09-02T04:15:39.853167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24928,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547872,"statfs":{"total":322122547200,"available":322097020928,"internally_reserved":0,"allocated":1515520,"data_stored":1269342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133382+0000","last_change":"2022-09-02T04:15:37.131430+0000","last_active":"2022-09-02T04:15:38.133382+0000","last_peered":"2022-09-02T04:15:38.133382+0000","last_clean":"2022-09-02T04:15:38.133382+0000","last_became_active":"2022-09-02T04:15:37.131186+0000","last_became_peered":"2022-09-02T04:15:37.131186+0000","last_unstale":"2022-09-02T04:15:38.133382+0000","last_undegraded":"2022-09-02T04:15:38.133382+0000","last_fullsized":"2022-09-02T04:15:38.133382+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T15:15:31.833796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133315+0000","last_change":"2022-09-02T04:15:37.131467+0000","last_active":"2022-09-02T04:15:38.133315+0000","last_peered":"2022-09-02T04:15:38.133315+0000","last_clean":"2022-09-02T04:15:38.133315+0000","last_became_active":"2022-09-02T04:15:37.131250+0000","last_became_peered":"2022-09-02T04:15:37.131250+0000","last_unstale":"2022-09-02T04:15:38.133315+0000","last_undegraded":"2022-09-02T04:15:38.133315+0000","last_fullsized":"2022-09-02T04:15:38.133315+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T04:24:19.856167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133319+0000","last_change":"2022-09-02T04:15:37.133638+0000","last_active":"2022-09-02T04:15:38.133319+0000","last_peered":"2022-09-02T04:15:38.133319+0000","last_clean":"2022-09-02T04:15:38.133319+0000","last_became_active":"2022-09-02T04:15:37.133478+0000","last_became_peered":"2022-09-02T04:15:37.133478+0000","last_unstale":"2022-09-02T04:15:38.133319+0000","last_undegraded":"2022-09-02T04:15:38.133319+0000","last_fullsized":"2022-09-02T04:15:38.133319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T13:59:45.880238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:15:37.133819+0000","last_change":"2022-09-02T04:15:37.131844+0000","last_active":"2022-09-02T04:15:37.133819+0000","last_peered":"2022-09-02T04:15:37.133819+0000","last_clean":"2022-09-02T04:15:37.133819+0000","last_became_active":"2022-09-02T04:15:37.131479+0000","last_became_peered":"2022-09-02T04:15:37.131479+0000","last_unstale":"2022-09-02T04:15:37.133819+0000","last_undegraded":"2022-09-02T04:15:37.133819+0000","last_fullsized":"2022-09-02T04:15:37.133819+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T08:50:17.313253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133254+0000","last_change":"2022-09-02T04:15:35.117916+0000","last_active":"2022-09-02T04:15:38.133254+0000","last_peered":"2022-09-02T04:15:38.133254+0000","last_clean":"2022-09-02T04:15:38.133254+0000","last_became_active":"2022-09-02T04:15:35.117625+0000","last_became_peered":"2022-09-02T04:15:35.117625+0000","last_unstale":"2022-09-02T04:15:38.133254+0000","last_undegraded":"2022-09-02T04:15:38.133254+0000","last_fullsized":"2022-09-02T04:15:38.133254+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:34.109108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:34.109108+0000","last_clean_scrub_stamp":"2022-09-02T04:15:34.109108+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T14:29:54.496539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133340+0000","last_change":"2022-09-02T04:15:37.134438+0000","last_active":"2022-09-02T04:15:38.133340+0000","last_peered":"2022-09-02T04:15:38.133340+0000","last_clean":"2022-09-02T04:15:38.133340+0000","last_became_active":"2022-09-02T04:15:37.134215+0000","last_became_peered":"2022-09-02T04:15:37.134215+0000","last_unstale":"2022-09-02T04:15:38.133340+0000","last_undegraded":"2022-09-02T04:15:38.133340+0000","last_fullsized":"2022-09-02T04:15:38.133340+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T11:12:52.858483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:15:37.133807+0000","last_change":"2022-09-02T04:15:37.132239+0000","last_active":"2022-09-02T04:15:37.133807+0000","last_peered":"2022-09-02T04:15:37.133807+0000","last_clean":"2022-09-02T04:15:37.133807+0000","last_became_active":"2022-09-02T04:15:37.131859+0000","last_became_peered":"2022-09-02T04:15:37.131859+0000","last_unstale":"2022-09-02T04:15:37.133807+0000","last_undegraded":"2022-09-02T04:15:37.133807+0000","last_fullsized":"2022-09-02T04:15:37.133807+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:06:10.551277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:15:37.134113+0000","last_change":"2022-09-02T04:15:37.132624+0000","last_active":"2022-09-02T04:15:37.134113+0000","last_peered":"2022-09-02T04:15:37.134113+0000","last_clean":"2022-09-02T04:15:37.134113+0000","last_became_active":"2022-09-02T04:15:37.132366+0000","last_became_peered":"2022-09-02T04:15:37.132366+0000","last_unstale":"2022-09-02T04:15:37.134113+0000","last_undegraded":"2022-09-02T04:15:37.134113+0000","last_fullsized":"2022-09-02T04:15:37.134113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T08:53:05.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133432+0000","last_change":"2022-09-02T04:15:37.131973+0000","last_active":"2022-09-02T04:15:38.133432+0000","last_peered":"2022-09-02T04:15:38.133432+0000","last_clean":"2022-09-02T04:15:38.133432+0000","last_became_active":"2022-09-02T04:15:37.131719+0000","last_became_peered":"2022-09-02T04:15:37.131719+0000","last_unstale":"2022-09-02T04:15:38.133432+0000","last_undegraded":"2022-09-02T04:15:38.133432+0000","last_fullsized":"2022-09-02T04:15:38.133432+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:21:26.435346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":28380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":618748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":622214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-02T04:15:41.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-09-02T04:15:42.118 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2022-09-02T04:15:42.132 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2022-09-02T04:15:41.853554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24928,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547872,"statfs":{"total":322122547200,"available":322097020928,"internally_reserved":0,"allocated":1515520,"data_stored":1269342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000387"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133382+0000","last_change":"2022-09-02T04:15:37.131430+0000","last_active":"2022-09-02T04:15:38.133382+0000","last_peered":"2022-09-02T04:15:38.133382+0000","last_clean":"2022-09-02T04:15:38.133382+0000","last_became_active":"2022-09-02T04:15:37.131186+0000","last_became_peered":"2022-09-02T04:15:37.131186+0000","last_unstale":"2022-09-02T04:15:38.133382+0000","last_undegraded":"2022-09-02T04:15:38.133382+0000","last_fullsized":"2022-09-02T04:15:38.133382+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T15:15:31.833796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133315+0000","last_change":"2022-09-02T04:15:37.131467+0000","last_active":"2022-09-02T04:15:38.133315+0000","last_peered":"2022-09-02T04:15:38.133315+0000","last_clean":"2022-09-02T04:15:38.133315+0000","last_became_active":"2022-09-02T04:15:37.131250+0000","last_became_peered":"2022-09-02T04:15:37.131250+0000","last_unstale":"2022-09-02T04:15:38.133315+0000","last_undegraded":"2022-09-02T04:15:38.133315+0000","last_fullsized":"2022-09-02T04:15:38.133315+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T04:24:19.856167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133319+0000","last_change":"2022-09-02T04:15:37.133638+0000","last_active":"2022-09-02T04:15:38.133319+0000","last_peered":"2022-09-02T04:15:38.133319+0000","last_clean":"2022-09-02T04:15:38.133319+0000","last_became_active":"2022-09-02T04:15:37.133478+0000","last_became_peered":"2022-09-02T04:15:37.133478+0000","last_unstale":"2022-09-02T04:15:38.133319+0000","last_undegraded":"2022-09-02T04:15:38.133319+0000","last_fullsized":"2022-09-02T04:15:38.133319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T13:59:45.880238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:15:37.133819+0000","last_change":"2022-09-02T04:15:37.131844+0000","last_active":"2022-09-02T04:15:37.133819+0000","last_peered":"2022-09-02T04:15:37.133819+0000","last_clean":"2022-09-02T04:15:37.133819+0000","last_became_active":"2022-09-02T04:15:37.131479+0000","last_became_peered":"2022-09-02T04:15:37.131479+0000","last_unstale":"2022-09-02T04:15:37.133819+0000","last_undegraded":"2022-09-02T04:15:37.133819+0000","last_fullsized":"2022-09-02T04:15:37.133819+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T08:50:17.313253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133254+0000","last_change":"2022-09-02T04:15:35.117916+0000","last_active":"2022-09-02T04:15:38.133254+0000","last_peered":"2022-09-02T04:15:38.133254+0000","last_clean":"2022-09-02T04:15:38.133254+0000","last_became_active":"2022-09-02T04:15:35.117625+0000","last_became_peered":"2022-09-02T04:15:35.117625+0000","last_unstale":"2022-09-02T04:15:38.133254+0000","last_undegraded":"2022-09-02T04:15:38.133254+0000","last_fullsized":"2022-09-02T04:15:38.133254+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:34.109108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:34.109108+0000","last_clean_scrub_stamp":"2022-09-02T04:15:34.109108+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T14:29:54.496539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133340+0000","last_change":"2022-09-02T04:15:37.134438+0000","last_active":"2022-09-02T04:15:38.133340+0000","last_peered":"2022-09-02T04:15:38.133340+0000","last_clean":"2022-09-02T04:15:38.133340+0000","last_became_active":"2022-09-02T04:15:37.134215+0000","last_became_peered":"2022-09-02T04:15:37.134215+0000","last_unstale":"2022-09-02T04:15:38.133340+0000","last_undegraded":"2022-09-02T04:15:38.133340+0000","last_fullsized":"2022-09-02T04:15:38.133340+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T11:12:52.858483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:15:37.133807+0000","last_change":"2022-09-02T04:15:37.132239+0000","last_active":"2022-09-02T04:15:37.133807+0000","last_peered":"2022-09-02T04:15:37.133807+0000","last_clean":"2022-09-02T04:15:37.133807+0000","last_became_active":"2022-09-02T04:15:37.131859+0000","last_became_peered":"2022-09-02T04:15:37.131859+0000","last_unstale":"2022-09-02T04:15:37.133807+0000","last_undegraded":"2022-09-02T04:15:37.133807+0000","last_fullsized":"2022-09-02T04:15:37.133807+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:06:10.551277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:15:37.134113+0000","last_change":"2022-09-02T04:15:37.132624+0000","last_active":"2022-09-02T04:15:37.134113+0000","last_peered":"2022-09-02T04:15:37.134113+0000","last_clean":"2022-09-02T04:15:37.134113+0000","last_became_active":"2022-09-02T04:15:37.132366+0000","last_became_peered":"2022-09-02T04:15:37.132366+0000","last_unstale":"2022-09-02T04:15:37.134113+0000","last_undegraded":"2022-09-02T04:15:37.134113+0000","last_fullsized":"2022-09-02T04:15:37.134113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T08:53:05.841810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:15:38.133432+0000","last_change":"2022-09-02T04:15:37.131973+0000","last_active":"2022-09-02T04:15:38.133432+0000","last_peered":"2022-09-02T04:15:38.133432+0000","last_clean":"2022-09-02T04:15:38.133432+0000","last_became_active":"2022-09-02T04:15:37.131719+0000","last_became_peered":"2022-09-02T04:15:37.131719+0000","last_unstale":"2022-09-02T04:15:38.133432+0000","last_undegraded":"2022-09-02T04:15:38.133432+0000","last_fullsized":"2022-09-02T04:15:38.133432+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:15:36.116119+0000","last_clean_scrub_stamp":"2022-09-02T04:15:36.116119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:21:26.435346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":28380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":618748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":622214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-02T04:15:42.133 INFO:tasks.ceph.ceph_manager.c1:clean! 2022-09-02T04:15:42.133 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2022-09-02T04:15:42.134 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2022-09-02T04:15:42.134 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2022-09-02T04:15:42.477 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:42.478 INFO:teuthology.orchestra.run.smithi099.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-02T04:15:42.491 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2022-09-02T04:15:42.492 INFO:teuthology.run_tasks:Running task ceph... 2022-09-02T04:15:42.511 INFO:tasks.ceph:Creating ceph cluster c2... 2022-09-02T04:15:42.511 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-09-02T04:15:42.512 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi/7005440', 'branch': 'pr-47848', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7005440', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'cbodley-2022-09-01_15:28:29-rgw-pr-47848-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'pr-47848'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-rgw-s3select-fixes_related_to_new_csv_parser', 'sha1': '98002085f9003280d82d2f21aa1472b865694c0b'}}, 'owner': 'scheduled_cbodley@teuthology', 'priority': 75, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-qa-rgw-s3select-fixes_related_to_new_csv_parser', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '98002085f9003280d82d2f21aa1472b865694c0b', 'targets': {'smithi099.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDut5FvkIQGrPyM0nkT7N8tYudURf71e+foXRKk+M0U4z/GU7R1IHbYePTvXo0Xe2mDfrabdI/sBTAphzeJNUR4XNStF3M+BZId93/7mt0uJAqS5xbV6NkePILDmGdtrgPQRtdtowRiH1XYTgIQnWFMjqntlrki93aFphwIR9DrcTQWwsnCuXH+yUIpu9mLA51Avg4P38VqnycKGjsK35X2oTWzkXVBdG6ycgSNM2IpvdWGA30FknJoOnXqcDJA9lyoXFDffAo2SapEUC7oOP3MPfipbqb9bvJx4BD7Jt2RuPAPnB7nhMuzZXu5TrXgAPMyG8+zfviSPj8ZSFaYKKCweMu4tPvcx3J8qfSzux+qT3otsh9Npo9zdlFrboVFsacTPN70nzngBW1WM4htz+ZdJQDyCd4f1amTu/V5IHTCezhr39kgZGa4oG1DU6jgyyYq6D/KvQCiw55w6lkKf0dfkSyT8yp0iGf7L9W8su8g3C8v+6vRyPg7AtDZm4L93Fc=', 'smithi160.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiwRVfUW8pFNs8ic5wz3OsS4S3a85p5doefMIekSi2aNJwcSHRdMPcJn4h+vlz5R5tj8i8uNPDyQjNwyBcM/wZJ7x0J1f/mFWwx4Zi0liFV7F+WPEdVQYVlLfDuspvkpjODvwSW92QU/8owPNBNLnutasr0SpaHutkXZjsYn7J0ybzdz8lf/HEriZo/5AYTrQyH1LZhhcIoIyvUOBjCQHC8L0LePd9lj+nZ13l/EX+QhsZRBYJh+31IerMPCP0/gk+a41JTqRjJJefFuzVFcudPBqIUwQxGzyT19IHp0O69MXWCH4kLupGDkdhjbQ7+j+fO6fFqOk+aJiGDwWsAChhuv9XeH9osWOUBsUopb7NMICRj0b9EFTKWzB7Ioyooor+ny7gbW31zy/Q+s/NgPwg/IP5lv1KRZYQk7JXEc8liG4q/N5kMCsybGeenr53SFGwm/zWfeNNKCaejo4WvZjtbd6W9lpwFptv/RvCmPFaQMY3z1KcDFg1EkODzuDis6k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': 'afd422385fab8119a539d289dd1aeaad4c056fe4', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b58f2c18636eb10faa77ed3614abd00cb85dfc2c', 'timestamp': '2022-09-01_15:28:29', 'tube': 'smithi', 'user': 'cbodley', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-09-02T04:15:42.513 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-02T04:15:42.517 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-09-02T04:15:42.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2022-09-02T04:15:42.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2022-09-02T04:15:42.610 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:42.610 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2022-09-02T04:15:42.670 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-02T04:15:42.671 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-02T04:15:42.730 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-02T04:15:42.731 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:42.732 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 370360 Links: 1 2022-09-02T04:15:42.732 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:42.733 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:42.734 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2022-09-02 04:14:58.109971831 +0000 2022-09-02T04:15:42.741 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2022-09-02 04:11:38.580098402 +0000 2022-09-02T04:15:42.741 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2022-09-02 04:11:38.580098402 +0000 2022-09-02T04:15:42.747 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2022-09-02T04:15:42.748 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2022-09-02T04:15:42.749 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2022-09-02T04:15:42.749 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000236164 s, 2.2 MB/s 2022-09-02T04:15:42.749 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-02T04:15:42.810 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-02T04:15:42.811 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:42.811 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 367362 Links: 1 2022-09-02T04:15:42.812 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:42.812 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:42.813 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2022-09-02 04:14:58.109971831 +0000 2022-09-02T04:15:42.813 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2022-09-02 04:11:39.133089960 +0000 2022-09-02T04:15:42.814 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2022-09-02 04:11:39.133089960 +0000 2022-09-02T04:15:42.814 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2022-09-02T04:15:42.816 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2022-09-02T04:15:42.817 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2022-09-02T04:15:42.817 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00018862 s, 2.7 MB/s 2022-09-02T04:15:42.823 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-02T04:15:42.885 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-02T04:15:42.886 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:42.887 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 373782 Links: 1 2022-09-02T04:15:42.887 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:42.888 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:42.888 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2022-09-02 04:14:58.109971831 +0000 2022-09-02T04:15:42.889 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2022-09-02 04:11:39.670081762 +0000 2022-09-02T04:15:42.889 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2022-09-02 04:11:39.670081762 +0000 2022-09-02T04:15:42.889 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2022-09-02T04:15:42.898 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2022-09-02T04:15:42.898 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2022-09-02T04:15:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000175338 s, 2.9 MB/s 2022-09-02T04:15:42.899 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-02T04:15:42.956 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-02T04:15:42.957 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-02T04:15:42.957 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 6h/6d Inode: 372834 Links: 1 2022-09-02T04:15:42.958 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-02T04:15:42.958 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2022-09-02T04:15:42.958 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2022-09-02 04:14:58.109971831 +0000 2022-09-02T04:15:42.960 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2022-09-02 04:11:40.220073365 +0000 2022-09-02T04:15:42.960 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2022-09-02 04:11:40.220073365 +0000 2022-09-02T04:15:42.961 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2022-09-02T04:15:42.964 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2022-09-02T04:15:42.965 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2022-09-02T04:15:42.965 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000197351 s, 2.6 MB/s 2022-09-02T04:15:42.969 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:42.969 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-02T04:15:42.971 INFO:tasks.ceph:Generating config... 2022-09-02T04:15:42.972 INFO:tasks.ceph:[client] debug rgw = 20 2022-09-02T04:15:42.972 INFO:tasks.ceph:[client] debug rgw sync = 20 2022-09-02T04:15:42.973 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-09-02T04:15:42.973 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-09-02T04:15:42.973 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-02T04:15:42.974 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-09-02T04:15:42.974 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-09-02T04:15:42.975 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-09-02T04:15:42.975 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2022-09-02T04:15:42.977 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-09-02T04:15:42.977 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-09-02T04:15:42.978 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-09-02T04:15:42.978 INFO:tasks.ceph:[client] setgroup = ceph 2022-09-02T04:15:42.978 INFO:tasks.ceph:[client] setuser = ceph 2022-09-02T04:15:42.979 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2022-09-02T04:15:42.979 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-09-02T04:15:42.979 INFO:tasks.ceph:[mgr] debug ms = 1 2022-09-02T04:15:42.980 INFO:tasks.ceph:[mon] debug mon = 20 2022-09-02T04:15:42.980 INFO:tasks.ceph:[mon] debug ms = 1 2022-09-02T04:15:42.980 INFO:tasks.ceph:[mon] debug paxos = 20 2022-09-02T04:15:42.982 INFO:tasks.ceph:[osd] debug ms = 1 2022-09-02T04:15:42.983 INFO:tasks.ceph:[osd] debug osd = 20 2022-09-02T04:15:42.983 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-09-02T04:15:42.983 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-09-02T04:15:42.984 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-09-02T04:15:42.984 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-09-02T04:15:42.984 INFO:tasks.ceph:Setting up c2.mon.a... 2022-09-02T04:15:42.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2022-09-02T04:15:43.060 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c2.keyring 2022-09-02T04:15:43.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2022-09-02T04:15:43.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-02T04:15:43.229 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.160')] 2022-09-02T04:15:43.231 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.160'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-09-02T04:15:43.234 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:43.235 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-09-02T04:15:43.291 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.160 --print /home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:43.375 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:15:43.376 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:15:43.377 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:43.378 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: generated fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T04:15:43.378 INFO:teuthology.orchestra.run.smithi160.stdout:setting min_mon_release = octopus 2022-09-02T04:15:43.378 INFO:teuthology.orchestra.run.smithi160.stdout:epoch 0 2022-09-02T04:15:43.378 INFO:teuthology.orchestra.run.smithi160.stdout:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T04:15:43.379 INFO:teuthology.orchestra.run.smithi160.stdout:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T04:15:43.379 INFO:teuthology.orchestra.run.smithi160.stdout:created 2022-09-02T04:15:43.374867+0000 2022-09-02T04:15:43.379 INFO:teuthology.orchestra.run.smithi160.stdout:min_mon_release 15 (octopus) 2022-09-02T04:15:43.379 INFO:teuthology.orchestra.run.smithi160.stdout:election_strategy: 1 2022-09-02T04:15:43.379 INFO:teuthology.orchestra.run.smithi160.stdout:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T04:15:43.380 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2022-09-02T04:15:43.380 DEBUG:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-09-02T04:15:43.436 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 4a69b944-3256-4419-904f-7378d4120c1b... 2022-09-02T04:15:43.437 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-02T04:15:43.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-09-02T04:15:43.490 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2022-09-02T04:15:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2022-09-02T04:15:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-02T04:15:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2022-09-02T04:15:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2022-09-02T04:15:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2022-09-02T04:15:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2022-09-02T04:15:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2022-09-02T04:15:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2022-09-02T04:15:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2022-09-02T04:15:43.495 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.495 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-02T04:15:43.496 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2022-09-02T04:15:43.496 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.496 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2022-09-02T04:15:43.496 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-02T04:15:43.497 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.497 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2022-09-02T04:15:43.497 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.497 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2022-09-02T04:15:43.497 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2022-09-02T04:15:43.498 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2022-09-02T04:15:43.498 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2022-09-02T04:15:43.498 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2022-09-02T04:15:43.502 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2022-09-02T04:15:43.502 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2022-09-02T04:15:43.503 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-02T04:15:43.503 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-02T04:15:43.504 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2022-09-02T04:15:43.505 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.505 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-02T04:15:43.506 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.506 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2022-09-02T04:15:43.507 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.507 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2022-09-02T04:15:43.507 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.507 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2022-09-02T04:15:43.507 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2022-09-02T04:15:43.508 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2022-09-02T04:15:43.508 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = 172.21.15.160 2022-09-02T04:15:43.509 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T04:15:43.509 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.509 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2022-09-02T04:15:43.509 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2022-09-02T04:15:43.510 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.510 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2022-09-02T04:15:43.510 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2022-09-02T04:15:43.510 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.511 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2022-09-02T04:15:43.511 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2022-09-02T04:15:43.511 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.511 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2022-09-02T04:15:43.512 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2022-09-02T04:15:43.512 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2022-09-02T04:15:43.512 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.512 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2022-09-02T04:15:43.513 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2022-09-02T04:15:43.513 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.513 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2022-09-02T04:15:43.513 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.513 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2022-09-02T04:15:43.514 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.514 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2022-09-02T04:15:43.514 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.514 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2022-09-02T04:15:43.515 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2022-09-02T04:15:43.515 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.515 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2022-09-02T04:15:43.515 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2022-09-02T04:15:43.516 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-09-02T04:15:43.516 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2022-09-02T04:15:43.516 INFO:teuthology.orchestra.run.smithi099.stdout: rocksdb delete range threshold = 0 2022-09-02T04:15:43.516 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.516 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2022-09-02T04:15:43.517 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-09-02T04:15:43.517 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2022-09-02T04:15:43.517 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2022-09-02T04:15:43.517 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2022-09-02T04:15:43.518 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:43.518 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:43.519 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2022-09-02T04:15:43.519 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.519 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2022-09-02T04:15:43.519 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-09-02T04:15:43.519 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2022-09-02T04:15:43.520 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2022-09-02T04:15:43.520 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2022-09-02T04:15:43.520 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2022-09-02T04:15:43.520 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2022-09-02T04:15:43.521 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:43.521 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2022-09-02T04:15:43.521 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2022-09-02T04:15:43.521 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.522 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-02T04:15:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2022-09-02T04:15:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2022-09-02T04:15:43.527 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.528 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2022-09-02T04:15:43.528 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-02T04:15:43.528 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-02T04:15:43.528 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.528 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2022-09-02T04:15:43.529 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2022-09-02T04:15:43.529 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.529 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2022-09-02T04:15:43.529 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:15:43.530 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2022-09-02T04:15:43.530 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2022-09-02T04:15:43.530 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2022-09-02T04:15:43.530 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2022-09-02T04:15:43.531 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-02T04:15:43.531 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-02T04:15:43.531 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw = 20 2022-09-02T04:15:43.531 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw sync = 20 2022-09-02T04:15:43.532 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt require ssl = False 2022-09-02T04:15:43.532 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms backend = testing 2022-09-02T04:15:43.532 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-02T04:15:43.532 INFO:teuthology.orchestra.run.smithi099.stdout: rgw curl low speed time = 300 2022-09-02T04:15:43.532 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data log num shards = 4 2022-09-02T04:15:43.533 INFO:teuthology.orchestra.run.smithi099.stdout: rgw md log max shards = 4 2022-09-02T04:15:43.533 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync data inject err probability = 0.1 2022-09-02T04:15:43.533 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync log trim interval = 0 2022-09-02T04:15:43.535 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync meta inject err probability = 0.1 2022-09-02T04:15:43.536 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync obj etag verify = True 2022-09-02T04:15:43.536 INFO:teuthology.orchestra.run.smithi099.stdout: setgroup = ceph 2022-09-02T04:15:43.536 INFO:teuthology.orchestra.run.smithi099.stdout: setuser = ceph 2022-09-02T04:15:43.537 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2022-09-02T04:15:43.537 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2022-09-02T04:15:43.538 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data notify interval msec = 0 2022-09-02T04:15:43.538 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2022-09-02T04:15:43.538 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10 2022-09-02T04:15:43.538 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2022-09-02T04:15:43.539 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 1000 2022-09-02T04:15:43.539 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2022-09-02T04:15:43.539 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10000 2022-09-02T04:15:43.542 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2022-09-02T04:15:43.542 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2022-09-02T04:15:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-09-02T04:15:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2022-09-02T04:15:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.543 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2022-09-02T04:15:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2022-09-02T04:15:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.544 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2022-09-02T04:15:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2022-09-02T04:15:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2022-09-02T04:15:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2022-09-02T04:15:43.545 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.547 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2022-09-02T04:15:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2022-09-02T04:15:43.548 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2022-09-02T04:15:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2022-09-02T04:15:43.549 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2022-09-02T04:15:43.550 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2022-09-02T04:15:43.551 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2022-09-02T04:15:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2022-09-02T04:15:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2022-09-02T04:15:43.552 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2022-09-02T04:15:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2022-09-02T04:15:43.553 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2022-09-02T04:15:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-09-02T04:15:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2022-09-02T04:15:43.555 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2022-09-02T04:15:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-09-02T04:15:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.556 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2022-09-02T04:15:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2022-09-02T04:15:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.557 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2022-09-02T04:15:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2022-09-02T04:15:43.558 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2022-09-02T04:15:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.160 2022-09-02T04:15:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T04:15:43.559 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.559 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2022-09-02T04:15:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2022-09-02T04:15:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2022-09-02T04:15:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2022-09-02T04:15:43.560 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2022-09-02T04:15:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2022-09-02T04:15:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.561 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2022-09-02T04:15:43.570 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2022-09-02T04:15:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2022-09-02T04:15:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2022-09-02T04:15:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2022-09-02T04:15:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2022-09-02T04:15:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2022-09-02T04:15:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2022-09-02T04:15:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2022-09-02T04:15:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2022-09-02T04:15:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2022-09-02T04:15:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2022-09-02T04:15:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2022-09-02T04:15:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2022-09-02T04:15:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: rocksdb delete range threshold = 0 2022-09-02T04:15:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.575 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2022-09-02T04:15:43.576 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2022-09-02T04:15:43.576 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2022-09-02T04:15:43.576 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2022-09-02T04:15:43.577 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2022-09-02T04:15:43.578 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:43.578 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:43.579 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2022-09-02T04:15:43.579 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.579 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2022-09-02T04:15:43.579 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2022-09-02T04:15:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2022-09-02T04:15:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2022-09-02T04:15:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2022-09-02T04:15:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2022-09-02T04:15:43.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2022-09-02T04:15:43.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2022-09-02T04:15:43.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2022-09-02T04:15:43.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2022-09-02T04:15:43.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2022-09-02T04:15:43.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.582 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-09-02T04:15:43.583 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2022-09-02T04:15:43.583 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2022-09-02T04:15:43.583 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.583 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2022-09-02T04:15:43.583 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-09-02T04:15:43.584 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-09-02T04:15:43.584 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.584 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2022-09-02T04:15:43.584 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2022-09-02T04:15:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2022-09-02T04:15:43.585 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:43.585 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2022-09-02T04:15:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2022-09-02T04:15:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2022-09-02T04:15:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2022-09-02T04:15:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-09-02T04:15:43.586 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-09-02T04:15:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw = 20 2022-09-02T04:15:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: debug rgw sync = 20 2022-09-02T04:15:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt require ssl = False 2022-09-02T04:15:43.587 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms backend = testing 2022-09-02T04:15:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-09-02T04:15:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: rgw curl low speed time = 300 2022-09-02T04:15:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data log num shards = 4 2022-09-02T04:15:43.588 INFO:teuthology.orchestra.run.smithi160.stdout: rgw md log max shards = 4 2022-09-02T04:15:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync data inject err probability = 0.1 2022-09-02T04:15:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync log trim interval = 0 2022-09-02T04:15:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync meta inject err probability = 0.1 2022-09-02T04:15:43.589 INFO:teuthology.orchestra.run.smithi160.stdout: rgw sync obj etag verify = True 2022-09-02T04:15:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: setgroup = ceph 2022-09-02T04:15:43.590 INFO:teuthology.orchestra.run.smithi160.stdout: setuser = ceph 2022-09-02T04:15:43.590 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2022-09-02T04:15:43.591 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2022-09-02T04:15:43.591 INFO:teuthology.orchestra.run.smithi160.stdout: rgw data notify interval msec = 0 2022-09-02T04:15:43.591 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2022-09-02T04:15:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10 2022-09-02T04:15:43.592 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2022-09-02T04:15:43.592 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 1000 2022-09-02T04:15:43.592 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2022-09-02T04:15:43.593 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_omap_entries_per_request = 10000 2022-09-02T04:15:43.593 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2022-09-02T04:15:43.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2022-09-02T04:15:43.653 INFO:tasks.ceph:Copying monmap to all nodes... 2022-09-02T04:15:43.669 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:43.670 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2022-09-02T04:15:43.692 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:43.693 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2022-09-02T04:15:43.771 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2022-09-02T04:15:43.771 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:43.772 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/c2.keyring 2022-09-02T04:15:43.772 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-02T04:15:43.810 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-09-02T04:15:43.815 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:43.871 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2022-09-02T04:15:43.872 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:43.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/c2.keyring 2022-09-02T04:15:43.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-09-02T04:15:43.912 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:43.913 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:43.972 INFO:tasks.ceph:Setting up mon nodes... 2022-09-02T04:15:43.972 INFO:tasks.ceph:Setting up mgr nodes... 2022-09-02T04:15:43.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2022-09-02T04:15:44.069 INFO:teuthology.orchestra.run.smithi160.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2022-09-02T04:15:44.070 INFO:tasks.ceph:Setting up mds nodes... 2022-09-02T04:15:44.079 INFO:tasks.ceph_client:Setting up client nodes... 2022-09-02T04:15:44.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2022-09-02T04:15:44.129 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c2.client.0.keyring 2022-09-02T04:15:44.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2022-09-02T04:15:44.240 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/c2.client.1.keyring 2022-09-02T04:15:44.252 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-09-02T04:15:44.253 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-09-02T04:15:44.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2022-09-02T04:15:44.322 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:44.323 INFO:tasks.ceph:role: c2.osd.0 2022-09-02T04:15:44.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com 2022-09-02T04:15:44.324 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-09-02T04:15:44.397 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-09-02T04:15:44.398 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-02T04:15:44.398 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-02T04:15:44.399 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-02T04:15:44.399 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-02T04:15:44.399 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2022-09-02T04:15:44.400 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-02T04:15:44.400 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-02T04:15:44.401 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-02T04:15:44.401 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-02T04:15:44.835 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2022-09-02T04:15:44.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2022-09-02T04:15:44.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2022-09-02T04:15:45.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2022-09-02T04:15:45.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2022-09-02T04:15:45.124 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:45.125 INFO:tasks.ceph:role: c2.osd.1 2022-09-02T04:15:45.125 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com 2022-09-02T04:15:45.126 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-09-02T04:15:45.200 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-09-02T04:15:45.201 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-02T04:15:45.201 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-02T04:15:45.201 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-02T04:15:45.204 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-02T04:15:45.204 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2022-09-02T04:15:45.204 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-02T04:15:45.204 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-02T04:15:45.205 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-02T04:15:45.205 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-02T04:15:45.615 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2022-09-02T04:15:45.617 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2022-09-02T04:15:45.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2022-09-02T04:15:45.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2022-09-02T04:15:45.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2022-09-02T04:15:45.848 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-09-02T04:15:45.850 INFO:tasks.ceph:role: c2.osd.2 2022-09-02T04:15:45.851 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com 2022-09-02T04:15:45.852 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-09-02T04:15:45.928 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-09-02T04:15:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2022-09-02T04:15:45.929 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-09-02T04:15:45.930 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-09-02T04:15:45.931 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-09-02T04:15:45.932 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2022-09-02T04:15:45.932 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-09-02T04:15:45.933 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-09-02T04:15:45.933 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-09-02T04:15:45.933 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-09-02T04:15:46.371 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2022-09-02T04:15:46.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2022-09-02T04:15:46.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2022-09-02T04:15:46.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2022-09-02T04:15:46.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:46.657 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:46.654+0000 7fb7d0aaf440 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2022-09-02T04:15:46.659 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:46.657+0000 7fb7d0aaf440 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2022-09-02T04:15:46.660 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:46.657+0000 7fb7d0aaf440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-02T04:15:46.661 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:46.657+0000 7fb7d0aaf440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-02T04:15:46.661 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:46.657+0000 7fb7d0aaf440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-09-02T04:15:46.667 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:46.664+0000 7fb7d0aaf440 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2022-09-02T04:15:48.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2022-09-02T04:15:48.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:48.872 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:48.869+0000 7f5287aad440 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2022-09-02T04:15:48.872 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:48.869+0000 7f5287aad440 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2022-09-02T04:15:48.873 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:48.869+0000 7f5287aad440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-02T04:15:48.873 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:48.869+0000 7f5287aad440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-02T04:15:48.873 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:48.869+0000 7f5287aad440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-09-02T04:15:48.874 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:48.869+0000 7f5287aad440 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2022-09-02T04:15:50.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2022-09-02T04:15:50.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:51.060 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:51.057+0000 7fedf32fb440 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2022-09-02T04:15:51.061 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:51.057+0000 7fedf32fb440 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2022-09-02T04:15:51.061 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:51.058+0000 7fedf32fb440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-02T04:15:51.061 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:51.058+0000 7fedf32fb440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-02T04:15:51.062 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:51.058+0000 7fedf32fb440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-09-02T04:15:51.062 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:15:51.058+0000 7fedf32fb440 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2022-09-02T04:15:53.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2022-09-02T04:15:53.175 INFO:tasks.ceph:Reading keys from all nodes... 2022-09-02T04:15:53.175 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:53.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2022-09-02T04:15:53.246 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:53.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2022-09-02T04:15:53.316 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:53.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2022-09-02T04:15:53.389 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:53.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2022-09-02T04:15:53.459 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:53.460 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2022-09-02T04:15:53.519 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:53.519 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2022-09-02T04:15:53.581 INFO:tasks.ceph:Adding keys to all mons... 2022-09-02T04:15:53.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo tee -a /etc/ceph/c2.keyring 2022-09-02T04:15:53.649 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr.x] 2022-09-02T04:15:53.650 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBwgxFjIg3XAxAAhW0NOVljPho22nrVASG5FA== 2022-09-02T04:15:53.650 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2022-09-02T04:15:53.651 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBygxFjIhQUJxAA9Mnk8iufzSapSqJGgrOGdQ== 2022-09-02T04:15:53.651 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2022-09-02T04:15:53.652 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQB0gxFj0V3gMxAA86PUeyKhjHRolgmgYlE+CA== 2022-09-02T04:15:53.652 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2022-09-02T04:15:53.652 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQB3gxFj7YGDAxAA4VKcuEwC0t7zkVp+RodvRQ== 2022-09-02T04:15:53.653 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2022-09-02T04:15:53.653 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBwgxFjw7KIBxAA6g9ajnFXsNwFCri6pUrvEg== 2022-09-02T04:15:53.653 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2022-09-02T04:15:53.654 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBwgxFjH9tDDhAAlNHYxp1/Cf+pYQI5ZfbNlQ== 2022-09-02T04:15:53.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-09-02T04:15:53.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-02T04:15:53.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-02T04:15:53.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-09-02T04:15:53.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-02T04:15:53.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-09-02T04:15:54.040 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-09-02T04:15:54.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2022-09-02T04:15:54.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2022-09-02T04:15:54.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2022-09-02T04:15:54.206 DEBUG:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/c2.monmap 2022-09-02T04:15:54.265 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2022-09-02T04:15:54.266 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2022-09-02T04:15:54.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2022-09-02T04:15:54.308 INFO:tasks.ceph.c2.mon.a:Started 2022-09-02T04:15:54.309 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2022-09-02T04:15:54.309 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2022-09-02T04:15:54.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2022-09-02T04:15:54.315 INFO:tasks.ceph.c2.mgr.x:Started 2022-09-02T04:15:54.316 INFO:tasks.ceph:Setting crush tunables to default 2022-09-02T04:15:54.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd crush tunables default 2022-09-02T04:15:54.521 INFO:teuthology.orchestra.run.smithi160.stderr:adjusted tunables profile to default 2022-09-02T04:15:54.531 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2022-09-02T04:15:54.532 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:54.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2022-09-02T04:15:54.570 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:54.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2022-09-02T04:15:54.637 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2022-09-02T04:15:54.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2022-09-02T04:15:54.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd new 7f270457-2035-461b-897f-a84686686551 0 2022-09-02T04:15:54.957 INFO:teuthology.orchestra.run.smithi160.stdout:0 2022-09-02T04:15:54.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd new c3149314-2145-4566-af10-a77716cfa77f 1 2022-09-02T04:15:55.188 INFO:teuthology.orchestra.run.smithi160.stdout:1 2022-09-02T04:15:55.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd new 957b11a5-cf86-46ec-8155-41dafe36bd43 2 2022-09-02T04:15:55.416 INFO:teuthology.orchestra.run.smithi160.stdout:2 2022-09-02T04:15:55.429 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2022-09-02T04:15:55.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2022-09-02T04:15:55.433 INFO:tasks.ceph.c2.osd.0:Started 2022-09-02T04:15:55.434 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2022-09-02T04:15:55.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2022-09-02T04:15:55.447 INFO:tasks.ceph.c2.osd.1:Started 2022-09-02T04:15:55.450 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2022-09-02T04:15:55.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2022-09-02T04:15:55.457 INFO:tasks.ceph.c2.osd.2:Started 2022-09-02T04:15:55.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-02T04:15:55.521 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:55.519+0000 7f467b743080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-02T04:15:55.597 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:55.594+0000 7f467b743080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-02T04:15:55.681 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:15:55.682 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":5,"fsid":"4a69b944-3256-4419-904f-7378d4120c1b","created":"2022-09-02T04:15:54.423387+0000","modified":"2022-09-02T04:15:55.414131+0000","last_up_change":"0.000000","last_in_change":"2022-09-02T04:15:55.414131+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7f270457-2035-461b-897f-a84686686551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3149314-2145-4566-af10-a77716cfa77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"957b11a5-cf86-46ec-8155-41dafe36bd43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:55.693 INFO:tasks.ceph.ceph_manager.c2:[] 2022-09-02T04:15:55.694 INFO:tasks.ceph:Waiting for OSDs to come up 2022-09-02T04:15:55.700 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:55.699+0000 7f467b743080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-02T04:15:55.885 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:55.883+0000 7f467b743080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-02T04:15:55.997 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-02T04:15:56.060 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T04:15:56.058+0000 7fa580d45440 -1 Falling back to public interface 2022-09-02T04:15:56.065 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T04:15:56.063+0000 7f7cc4d5c440 -1 Falling back to public interface 2022-09-02T04:15:56.069 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T04:15:56.067+0000 7f8eae092440 -1 Falling back to public interface 2022-09-02T04:15:56.189 INFO:teuthology.misc.health.smithi160.stdout: 2022-09-02T04:15:56.192 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":5,"fsid":"4a69b944-3256-4419-904f-7378d4120c1b","created":"2022-09-02T04:15:54.423387+0000","modified":"2022-09-02T04:15:55.414131+0000","last_up_change":"0.000000","last_in_change":"2022-09-02T04:15:55.414131+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7f270457-2035-461b-897f-a84686686551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3149314-2145-4566-af10-a77716cfa77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"957b11a5-cf86-46ec-8155-41dafe36bd43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:15:56.202 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-02T04:15:56.677 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:56.676+0000 7f467b743080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-02T04:15:56.814 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:56.813+0000 7f467b743080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-02T04:15:57.036 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.035+0000 7f467b743080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-02T04:15:57.106 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.104+0000 7f467b743080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-02T04:15:57.241 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.239+0000 7f467b743080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-02T04:15:57.509 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.500+0000 7f467b743080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-02T04:15:57.683 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.682+0000 7f467b743080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-02T04:15:57.822 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.821+0000 7f467b743080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-02T04:15:57.934 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:57.933+0000 7f467b743080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-02T04:15:58.073 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:58.072+0000 7f467b743080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-02T04:15:58.153 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:58.149+0000 7f467b743080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-02T04:15:58.370 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:58.366+0000 7f467b743080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-02T04:15:58.443 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:58.442+0000 7f467b743080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-02T04:15:59.044 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.043+0000 7f467b743080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-02T04:15:59.322 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.321+0000 7f467b743080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-02T04:15:59.397 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.396+0000 7f467b743080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-02T04:15:59.478 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.477+0000 7f467b743080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-02T04:15:59.663 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.658+0000 7f467b743080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-02T04:15:59.730 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.728+0000 7f467b743080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-02T04:15:59.878 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:15:59.877+0000 7f467b743080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-02T04:16:00.898 INFO:tasks.ceph.c2.mgr.x.smithi160.stderr:2022-09-02T04:16:00.897+0000 7f4635520700 -1 mgr.server handle_report got status from non-daemon mon.a 2022-09-02T04:16:02.508 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-02T04:16:02.815 INFO:teuthology.misc.health.smithi160.stdout: 2022-09-02T04:16:02.815 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":5,"fsid":"4a69b944-3256-4419-904f-7378d4120c1b","created":"2022-09-02T04:15:54.423387+0000","modified":"2022-09-02T04:15:55.414131+0000","last_up_change":"0.000000","last_in_change":"2022-09-02T04:15:55.414131+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7f270457-2035-461b-897f-a84686686551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3149314-2145-4566-af10-a77716cfa77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"957b11a5-cf86-46ec-8155-41dafe36bd43","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:16:02.827 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-09-02T04:16:03.768 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T04:16:03.767+0000 7fa580d45440 -1 osd.1 0 log_to_monitors true 2022-09-02T04:16:03.796 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T04:16:03.795+0000 7f7cc4d5c440 -1 osd.0 0 log_to_monitors true 2022-09-02T04:16:03.807 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T04:16:03.797+0000 7f8eae092440 -1 osd.2 0 log_to_monitors true 2022-09-02T04:16:06.116 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T04:16:06.114+0000 7fa574af2700 -1 osd.1 0 waiting for initial osdmap 2022-09-02T04:16:06.129 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T04:16:06.128+0000 7f7cb8b09700 -1 osd.0 0 waiting for initial osdmap 2022-09-02T04:16:06.229 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T04:16:06.229+0000 7fa56d6ce700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-02T04:16:06.243 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T04:16:06.242+0000 7f7cb16e5700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-02T04:16:06.568 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T04:16:06.566+0000 7f8ea1e3f700 -1 osd.2 0 waiting for initial osdmap 2022-09-02T04:16:06.577 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T04:16:06.576+0000 7f8e9aa1b700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-02T04:16:09.141 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-09-02T04:16:09.548 INFO:teuthology.misc.health.smithi160.stdout: 2022-09-02T04:16:09.549 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":10,"fsid":"4a69b944-3256-4419-904f-7378d4120c1b","created":"2022-09-02T04:15:54.423387+0000","modified":"2022-09-02T04:16:09.133376+0000","last_up_change":"2022-09-02T04:16:07.126672+0000","last_in_change":"2022-09-02T04:15:55.414131+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-02T04:16:07.936845+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7f270457-2035-461b-897f-a84686686551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":104330},{"type":"v1","addr":"172.21.15.160:6809","nonce":104330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":104330},{"type":"v1","addr":"172.21.15.160:6811","nonce":104330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":104330},{"type":"v1","addr":"172.21.15.160:6815","nonce":104330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":104330},{"type":"v1","addr":"172.21.15.160:6813","nonce":104330}]},"public_addr":"172.21.15.160:6809/104330","cluster_addr":"172.21.15.160:6811/104330","heartbeat_back_addr":"172.21.15.160:6815/104330","heartbeat_front_addr":"172.21.15.160:6813/104330","state":["exists","up"]},{"osd":1,"uuid":"c3149314-2145-4566-af10-a77716cfa77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":104331},{"type":"v1","addr":"172.21.15.160:6801","nonce":104331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":104331},{"type":"v1","addr":"172.21.15.160:6803","nonce":104331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":104331},{"type":"v1","addr":"172.21.15.160:6807","nonce":104331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":104331},{"type":"v1","addr":"172.21.15.160:6805","nonce":104331}]},"public_addr":"172.21.15.160:6801/104331","cluster_addr":"172.21.15.160:6803/104331","heartbeat_back_addr":"172.21.15.160:6807/104331","heartbeat_front_addr":"172.21.15.160:6805/104331","state":["exists","up"]},{"osd":2,"uuid":"957b11a5-cf86-46ec-8155-41dafe36bd43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":104360},{"type":"v1","addr":"172.21.15.160:6817","nonce":104360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":104360},{"type":"v1","addr":"172.21.15.160:6819","nonce":104360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":104360},{"type":"v1","addr":"172.21.15.160:6823","nonce":104360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":104360},{"type":"v1","addr":"172.21.15.160:6821","nonce":104360}]},"public_addr":"172.21.15.160:6817/104360","cluster_addr":"172.21.15.160:6819/104360","heartbeat_back_addr":"172.21.15.160:6823/104360","heartbeat_front_addr":"172.21.15.160:6821/104360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.813664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.769301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.812734+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:16:09.564 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-09-02T04:16:09.566 INFO:tasks.ceph:Creating RBD pool 2022-09-02T04:16:09.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd pool create rbd 8 2022-09-02T04:16:10.144 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'rbd' created 2022-09-02T04:16:10.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-09-02T04:16:11.151 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rbd' on pool 'rbd' 2022-09-02T04:16:11.164 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2022-09-02T04:16:11.166 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2022-09-02T04:16:11.166 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2022-09-02T04:16:11.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2022-09-02T04:16:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-09-02T04:16:11.535 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:16:11.543 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":103922},{"type":"v1","addr":"172.21.15.160:6825","nonce":103922}]},"active_addr":"172.21.15.160:6825/103922","active_change":"2022-09-02T04:15:59.890201+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":3210291080}]},{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":1702973752}]},{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":423079293}]},{"addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":3079037609}]}]}} 2022-09-02T04:16:11.546 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2022-09-02T04:16:11.547 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2022-09-02T04:16:11.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-02T04:16:11.870 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:16:11.871 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":12,"fsid":"4a69b944-3256-4419-904f-7378d4120c1b","created":"2022-09-02T04:15:54.423387+0000","modified":"2022-09-02T04:16:11.139429+0000","last_up_change":"2022-09-02T04:16:07.126672+0000","last_in_change":"2022-09-02T04:15:55.414131+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-02T04:16:07.936845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-02T04:16:09.887640+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7f270457-2035-461b-897f-a84686686551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":104330},{"type":"v1","addr":"172.21.15.160:6809","nonce":104330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":104330},{"type":"v1","addr":"172.21.15.160:6811","nonce":104330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":104330},{"type":"v1","addr":"172.21.15.160:6815","nonce":104330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":104330},{"type":"v1","addr":"172.21.15.160:6813","nonce":104330}]},"public_addr":"172.21.15.160:6809/104330","cluster_addr":"172.21.15.160:6811/104330","heartbeat_back_addr":"172.21.15.160:6815/104330","heartbeat_front_addr":"172.21.15.160:6813/104330","state":["exists","up"]},{"osd":1,"uuid":"c3149314-2145-4566-af10-a77716cfa77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":104331},{"type":"v1","addr":"172.21.15.160:6801","nonce":104331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":104331},{"type":"v1","addr":"172.21.15.160:6803","nonce":104331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":104331},{"type":"v1","addr":"172.21.15.160:6807","nonce":104331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":104331},{"type":"v1","addr":"172.21.15.160:6805","nonce":104331}]},"public_addr":"172.21.15.160:6801/104331","cluster_addr":"172.21.15.160:6803/104331","heartbeat_back_addr":"172.21.15.160:6807/104331","heartbeat_front_addr":"172.21.15.160:6805/104331","state":["exists","up"]},{"osd":2,"uuid":"957b11a5-cf86-46ec-8155-41dafe36bd43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":104360},{"type":"v1","addr":"172.21.15.160:6817","nonce":104360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":104360},{"type":"v1","addr":"172.21.15.160:6819","nonce":104360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":104360},{"type":"v1","addr":"172.21.15.160:6823","nonce":104360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":104360},{"type":"v1","addr":"172.21.15.160:6821","nonce":104360}]},"public_addr":"172.21.15.160:6817/104360","cluster_addr":"172.21.15.160:6819/104360","heartbeat_back_addr":"172.21.15.160:6823/104360","heartbeat_front_addr":"172.21.15.160:6821/104360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.813664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.769301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.812734+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:16:11.884 INFO:tasks.ceph.ceph_manager.c2:all up! 2022-09-02T04:16:11.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-09-02T04:16:12.193 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:16:12.194 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":13,"fsid":"4a69b944-3256-4419-904f-7378d4120c1b","created":"2022-09-02T04:15:54.423387+0000","modified":"2022-09-02T04:16:12.144292+0000","last_up_change":"2022-09-02T04:16:07.126672+0000","last_in_change":"2022-09-02T04:15:55.414131+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-02T04:16:07.936845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-09-02T04:16:09.887640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7f270457-2035-461b-897f-a84686686551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":104330},{"type":"v1","addr":"172.21.15.160:6809","nonce":104330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":104330},{"type":"v1","addr":"172.21.15.160:6811","nonce":104330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":104330},{"type":"v1","addr":"172.21.15.160:6815","nonce":104330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":104330},{"type":"v1","addr":"172.21.15.160:6813","nonce":104330}]},"public_addr":"172.21.15.160:6809/104330","cluster_addr":"172.21.15.160:6811/104330","heartbeat_back_addr":"172.21.15.160:6815/104330","heartbeat_front_addr":"172.21.15.160:6813/104330","state":["exists","up"]},{"osd":1,"uuid":"c3149314-2145-4566-af10-a77716cfa77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":104331},{"type":"v1","addr":"172.21.15.160:6801","nonce":104331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":104331},{"type":"v1","addr":"172.21.15.160:6803","nonce":104331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":104331},{"type":"v1","addr":"172.21.15.160:6807","nonce":104331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":104331},{"type":"v1","addr":"172.21.15.160:6805","nonce":104331}]},"public_addr":"172.21.15.160:6801/104331","cluster_addr":"172.21.15.160:6803/104331","heartbeat_back_addr":"172.21.15.160:6807/104331","heartbeat_front_addr":"172.21.15.160:6805/104331","state":["exists","up"]},{"osd":2,"uuid":"957b11a5-cf86-46ec-8155-41dafe36bd43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":104360},{"type":"v1","addr":"172.21.15.160:6817","nonce":104360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":104360},{"type":"v1","addr":"172.21.15.160:6819","nonce":104360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":104360},{"type":"v1","addr":"172.21.15.160:6823","nonce":104360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":104360},{"type":"v1","addr":"172.21.15.160:6821","nonce":104360}]},"public_addr":"172.21.15.160:6817/104360","cluster_addr":"172.21.15.160:6819/104360","heartbeat_back_addr":"172.21.15.160:6823/104360","heartbeat_front_addr":"172.21.15.160:6821/104360","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.813664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.769301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-02T04:16:04.812734+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-02T04:16:12.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2022-09-02T04:16:12.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2022-09-02T04:16:12.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2022-09-02T04:16:12.393 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2022-09-02T04:16:12.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-02T04:16:12.411 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2022-09-02T04:16:12.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-02T04:16:12.414 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2022-09-02T04:16:12.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-02T04:16:12.731 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2022-09-02T04:16:12.756 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.1 2022-09-02T04:16:12.778 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2022-09-02T04:16:12.791 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.0 2022-09-02T04:16:12.795 INFO:teuthology.orchestra.run.smithi160.stdout:34359738371 2022-09-02T04:16:12.809 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2022-09-02T04:16:13.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-02T04:16:13.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-09-02T04:16:13.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-09-02T04:16:14.108 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2022-09-02T04:16:14.120 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.1 2022-09-02T04:16:14.154 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2022-09-02T04:16:14.171 INFO:teuthology.orchestra.run.smithi160.stdout:34359738374 2022-09-02T04:16:14.172 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738373 for osd.2 2022-09-02T04:16:14.173 DEBUG:teuthology.parallel:result is None 2022-09-02T04:16:14.174 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738374 for osd.0 2022-09-02T04:16:14.174 DEBUG:teuthology.parallel:result is None 2022-09-02T04:16:15.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-09-02T04:16:15.445 INFO:teuthology.orchestra.run.smithi160.stdout:34359738374 2022-09-02T04:16:15.459 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738374 for osd.1 2022-09-02T04:16:15.460 DEBUG:teuthology.parallel:result is None 2022-09-02T04:16:15.461 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2022-09-02T04:16:15.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-02T04:16:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:16:15.786 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2022-09-02T04:16:15.788 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2022-09-02T04:16:13.895858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25428,"kb_used_data":1476,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547372,"statfs":{"total":322122547200,"available":322096508928,"internally_reserved":0,"allocated":1511424,"data_stored":1268581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148720+0000","last_change":"2022-09-02T04:16:11.148827+0000","last_active":"2022-09-02T04:16:12.148720+0000","last_peered":"2022-09-02T04:16:12.148720+0000","last_clean":"2022-09-02T04:16:12.148720+0000","last_became_active":"2022-09-02T04:16:11.148569+0000","last_became_peered":"2022-09-02T04:16:11.148569+0000","last_unstale":"2022-09-02T04:16:12.148720+0000","last_undegraded":"2022-09-02T04:16:12.148720+0000","last_fullsized":"2022-09-02T04:16:12.148720+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T06:23:14.811205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148702+0000","last_change":"2022-09-02T04:16:11.148127+0000","last_active":"2022-09-02T04:16:12.148702+0000","last_peered":"2022-09-02T04:16:12.148702+0000","last_clean":"2022-09-02T04:16:12.148702+0000","last_became_active":"2022-09-02T04:16:11.147888+0000","last_became_peered":"2022-09-02T04:16:11.147888+0000","last_unstale":"2022-09-02T04:16:12.148702+0000","last_undegraded":"2022-09-02T04:16:12.148702+0000","last_fullsized":"2022-09-02T04:16:12.148702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T14:44:12.569025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148755+0000","last_change":"2022-09-02T04:16:11.148955+0000","last_active":"2022-09-02T04:16:12.148755+0000","last_peered":"2022-09-02T04:16:12.148755+0000","last_clean":"2022-09-02T04:16:12.148755+0000","last_became_active":"2022-09-02T04:16:11.148682+0000","last_became_peered":"2022-09-02T04:16:11.148682+0000","last_unstale":"2022-09-02T04:16:12.148755+0000","last_undegraded":"2022-09-02T04:16:12.148755+0000","last_fullsized":"2022-09-02T04:16:12.148755+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:18:58.523728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:16:11.150814+0000","last_change":"2022-09-02T04:16:11.149332+0000","last_active":"2022-09-02T04:16:11.150814+0000","last_peered":"2022-09-02T04:16:11.150814+0000","last_clean":"2022-09-02T04:16:11.150814+0000","last_became_active":"2022-09-02T04:16:11.149050+0000","last_became_peered":"2022-09-02T04:16:11.149050+0000","last_unstale":"2022-09-02T04:16:11.150814+0000","last_undegraded":"2022-09-02T04:16:11.150814+0000","last_fullsized":"2022-09-02T04:16:11.150814+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T13:29:57.407772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148629+0000","last_change":"2022-09-02T04:16:09.138937+0000","last_active":"2022-09-02T04:16:12.148629+0000","last_peered":"2022-09-02T04:16:12.148629+0000","last_clean":"2022-09-02T04:16:12.148629+0000","last_became_active":"2022-09-02T04:16:09.138644+0000","last_became_peered":"2022-09-02T04:16:09.138644+0000","last_unstale":"2022-09-02T04:16:12.148629+0000","last_undegraded":"2022-09-02T04:16:12.148629+0000","last_fullsized":"2022-09-02T04:16:12.148629+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:08.130066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:08.130066+0000","last_clean_scrub_stamp":"2022-09-02T04:16:08.130066+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T11:05:14.755495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148707+0000","last_change":"2022-09-02T04:16:11.148009+0000","last_active":"2022-09-02T04:16:12.148707+0000","last_peered":"2022-09-02T04:16:12.148707+0000","last_clean":"2022-09-02T04:16:12.148707+0000","last_became_active":"2022-09-02T04:16:11.147791+0000","last_became_peered":"2022-09-02T04:16:11.147791+0000","last_unstale":"2022-09-02T04:16:12.148707+0000","last_undegraded":"2022-09-02T04:16:12.148707+0000","last_fullsized":"2022-09-02T04:16:12.148707+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T06:16:44.873187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:16:11.148694+0000","last_change":"2022-09-02T04:16:11.147924+0000","last_active":"2022-09-02T04:16:11.148694+0000","last_peered":"2022-09-02T04:16:11.148694+0000","last_clean":"2022-09-02T04:16:11.148694+0000","last_became_active":"2022-09-02T04:16:11.147640+0000","last_became_peered":"2022-09-02T04:16:11.147640+0000","last_unstale":"2022-09-02T04:16:11.148694+0000","last_undegraded":"2022-09-02T04:16:11.148694+0000","last_fullsized":"2022-09-02T04:16:11.148694+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T14:49:38.942733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:16:11.150825+0000","last_change":"2022-09-02T04:16:11.149924+0000","last_active":"2022-09-02T04:16:11.150825+0000","last_peered":"2022-09-02T04:16:11.150825+0000","last_clean":"2022-09-02T04:16:11.150825+0000","last_became_active":"2022-09-02T04:16:11.149584+0000","last_became_peered":"2022-09-02T04:16:11.149584+0000","last_unstale":"2022-09-02T04:16:11.150825+0000","last_undegraded":"2022-09-02T04:16:11.150825+0000","last_fullsized":"2022-09-02T04:16:11.150825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:55:52.155456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148804+0000","last_change":"2022-09-02T04:16:11.149104+0000","last_active":"2022-09-02T04:16:12.148804+0000","last_peered":"2022-09-02T04:16:12.148804+0000","last_clean":"2022-09-02T04:16:12.148804+0000","last_became_active":"2022-09-02T04:16:11.148850+0000","last_became_peered":"2022-09-02T04:16:11.148850+0000","last_unstale":"2022-09-02T04:16:12.148804+0000","last_undegraded":"2022-09-02T04:16:12.148804+0000","last_fullsized":"2022-09-02T04:16:12.148804+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T04:25:09.471077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":28380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9128,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848472,"statfs":{"total":107374182400,"available":107364835328,"internally_reserved":0,"allocated":696320,"data_stored":617987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":622214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-02T04:16:15.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-09-02T04:16:16.098 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:16:16.099 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2022-09-02T04:16:16.113 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2022-09-02T04:16:15.896234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25428,"kb_used_data":1476,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547372,"statfs":{"total":322122547200,"available":322096508928,"internally_reserved":0,"allocated":1511424,"data_stored":1268581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000375"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148720+0000","last_change":"2022-09-02T04:16:11.148827+0000","last_active":"2022-09-02T04:16:12.148720+0000","last_peered":"2022-09-02T04:16:12.148720+0000","last_clean":"2022-09-02T04:16:12.148720+0000","last_became_active":"2022-09-02T04:16:11.148569+0000","last_became_peered":"2022-09-02T04:16:11.148569+0000","last_unstale":"2022-09-02T04:16:12.148720+0000","last_undegraded":"2022-09-02T04:16:12.148720+0000","last_fullsized":"2022-09-02T04:16:12.148720+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T06:23:14.811205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148702+0000","last_change":"2022-09-02T04:16:11.148127+0000","last_active":"2022-09-02T04:16:12.148702+0000","last_peered":"2022-09-02T04:16:12.148702+0000","last_clean":"2022-09-02T04:16:12.148702+0000","last_became_active":"2022-09-02T04:16:11.147888+0000","last_became_peered":"2022-09-02T04:16:11.147888+0000","last_unstale":"2022-09-02T04:16:12.148702+0000","last_undegraded":"2022-09-02T04:16:12.148702+0000","last_fullsized":"2022-09-02T04:16:12.148702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T14:44:12.569025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148755+0000","last_change":"2022-09-02T04:16:11.148955+0000","last_active":"2022-09-02T04:16:12.148755+0000","last_peered":"2022-09-02T04:16:12.148755+0000","last_clean":"2022-09-02T04:16:12.148755+0000","last_became_active":"2022-09-02T04:16:11.148682+0000","last_became_peered":"2022-09-02T04:16:11.148682+0000","last_unstale":"2022-09-02T04:16:12.148755+0000","last_undegraded":"2022-09-02T04:16:12.148755+0000","last_fullsized":"2022-09-02T04:16:12.148755+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:18:58.523728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:16:11.150814+0000","last_change":"2022-09-02T04:16:11.149332+0000","last_active":"2022-09-02T04:16:11.150814+0000","last_peered":"2022-09-02T04:16:11.150814+0000","last_clean":"2022-09-02T04:16:11.150814+0000","last_became_active":"2022-09-02T04:16:11.149050+0000","last_became_peered":"2022-09-02T04:16:11.149050+0000","last_unstale":"2022-09-02T04:16:11.150814+0000","last_undegraded":"2022-09-02T04:16:11.150814+0000","last_fullsized":"2022-09-02T04:16:11.150814+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T13:29:57.407772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148629+0000","last_change":"2022-09-02T04:16:09.138937+0000","last_active":"2022-09-02T04:16:12.148629+0000","last_peered":"2022-09-02T04:16:12.148629+0000","last_clean":"2022-09-02T04:16:12.148629+0000","last_became_active":"2022-09-02T04:16:09.138644+0000","last_became_peered":"2022-09-02T04:16:09.138644+0000","last_unstale":"2022-09-02T04:16:12.148629+0000","last_undegraded":"2022-09-02T04:16:12.148629+0000","last_fullsized":"2022-09-02T04:16:12.148629+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:08.130066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:08.130066+0000","last_clean_scrub_stamp":"2022-09-02T04:16:08.130066+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T11:05:14.755495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148707+0000","last_change":"2022-09-02T04:16:11.148009+0000","last_active":"2022-09-02T04:16:12.148707+0000","last_peered":"2022-09-02T04:16:12.148707+0000","last_clean":"2022-09-02T04:16:12.148707+0000","last_became_active":"2022-09-02T04:16:11.147791+0000","last_became_peered":"2022-09-02T04:16:11.147791+0000","last_unstale":"2022-09-02T04:16:12.148707+0000","last_undegraded":"2022-09-02T04:16:12.148707+0000","last_fullsized":"2022-09-02T04:16:12.148707+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T06:16:44.873187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:16:11.148694+0000","last_change":"2022-09-02T04:16:11.147924+0000","last_active":"2022-09-02T04:16:11.148694+0000","last_peered":"2022-09-02T04:16:11.148694+0000","last_clean":"2022-09-02T04:16:11.148694+0000","last_became_active":"2022-09-02T04:16:11.147640+0000","last_became_peered":"2022-09-02T04:16:11.147640+0000","last_unstale":"2022-09-02T04:16:11.148694+0000","last_undegraded":"2022-09-02T04:16:11.148694+0000","last_fullsized":"2022-09-02T04:16:11.148694+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T14:49:38.942733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2022-09-02T04:16:11.150825+0000","last_change":"2022-09-02T04:16:11.149924+0000","last_active":"2022-09-02T04:16:11.150825+0000","last_peered":"2022-09-02T04:16:11.150825+0000","last_clean":"2022-09-02T04:16:11.150825+0000","last_became_active":"2022-09-02T04:16:11.149584+0000","last_became_peered":"2022-09-02T04:16:11.149584+0000","last_unstale":"2022-09-02T04:16:11.150825+0000","last_undegraded":"2022-09-02T04:16:11.150825+0000","last_fullsized":"2022-09-02T04:16:11.150825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T07:55:52.155456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-09-02T04:16:12.148804+0000","last_change":"2022-09-02T04:16:11.149104+0000","last_active":"2022-09-02T04:16:12.148804+0000","last_peered":"2022-09-02T04:16:12.148804+0000","last_clean":"2022-09-02T04:16:12.148804+0000","last_became_active":"2022-09-02T04:16:11.148850+0000","last_became_peered":"2022-09-02T04:16:11.148850+0000","last_unstale":"2022-09-02T04:16:12.148804+0000","last_undegraded":"2022-09-02T04:16:12.148804+0000","last_fullsized":"2022-09-02T04:16:12.148804+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-02T04:16:10.136424+0000","last_clean_scrub_stamp":"2022-09-02T04:16:10.136424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-03T04:25:09.471077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7664,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849936,"statfs":{"total":107374182400,"available":107366334464,"internally_reserved":0,"allocated":106496,"data_stored":28380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":9128,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848472,"statfs":{"total":107374182400,"available":107364835328,"internally_reserved":0,"allocated":696320,"data_stored":617987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8636,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848964,"statfs":{"total":107374182400,"available":107365339136,"internally_reserved":0,"allocated":708608,"data_stored":622214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-02T04:16:16.114 INFO:tasks.ceph.ceph_manager.c2:clean! 2022-09-02T04:16:16.115 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2022-09-02T04:16:16.115 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2022-09-02T04:16:16.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2022-09-02T04:16:16.460 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T04:16:16.460 INFO:teuthology.orchestra.run.smithi160.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-02T04:16:16.479 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2022-09-02T04:16:16.491 INFO:teuthology.run_tasks:Running task rgw... 2022-09-02T04:16:16.521 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2022-09-02T04:16:16.525 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2022-09-02T04:16:16.525 INFO:tasks.rgw:Creating data pools 2022-09-02T04:16:16.526 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2022-09-02T04:16:16.528 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-02T04:16:17.420 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.data' created 2022-09-02T04:16:17.436 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-02T04:16:18.441 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-02T04:16:18.454 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-02T04:16:19.453 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.index' created 2022-09-02T04:16:19.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-02T04:16:20.458 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-02T04:16:20.475 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2022-09-02T04:16:20.476 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-09-02T04:16:20.795 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-02T04:16:20.798 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-09-02T04:16:21.441 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-02T04:16:21.453 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-09-02T04:16:21.762 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-02T04:16:21.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-09-02T04:16:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-02T04:16:22.456 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2022-09-02T04:16:22.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-02T04:16:23.415 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.data' created 2022-09-02T04:16:23.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-02T04:16:24.440 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-02T04:16:24.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-02T04:16:25.427 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.index' created 2022-09-02T04:16:25.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-02T04:16:26.443 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-02T04:16:26.460 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2022-09-02T04:16:26.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-09-02T04:16:26.772 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.data' already exists 2022-09-02T04:16:26.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-09-02T04:16:27.429 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-09-02T04:16:27.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-09-02T04:16:27.747 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'default.rgw.buckets.index' already exists 2022-09-02T04:16:27.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-09-02T04:16:28.436 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-09-02T04:16:28.446 DEBUG:tasks.rgw:Pools created 2022-09-02T04:16:28.450 INFO:tasks.rgw:Configuring compression type = random 2022-09-02T04:16:28.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2022-09-02T04:16:28.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2022-09-02T04:16:28.452 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2022-09-02T04:16:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:28.543+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.485 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.482+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.486 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.482+0000 7f331ba13600 20 realm 2022-09-02T04:16:30.486 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.482+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.486 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-02T04:16:30.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.488 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.489 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.489 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.483+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.484+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.484+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.484+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.484+0000 7f331ba13600 10 Creating default zonegroup 2022-09-02T04:16:30.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.485+0000 7f331ba13600 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-02T04:16:30.509 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.503+0000 7f331ba13600 10 WARNING: store->list_zones() returned r=-2 2022-09-02T04:16:30.510 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.503+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.510 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.503+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.507+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.507+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.507+0000 7f331ba13600 10 could not read realm id: (2) No such file or directory 2022-09-02T04:16:30.516 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.507+0000 7f331ba13600 10 WARNING: failed to set zone as default, r=-22 2022-09-02T04:16:30.516 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.507+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.516 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.508+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.516 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.513+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.518 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.513+0000 7f331ba13600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:30.519 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.513+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.519 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 10 Using default name default 2022-09-02T04:16:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.521 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:30.521 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 zone default found 2022-09-02T04:16:30.521 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 4 Realm: () 2022-09-02T04:16:30.521 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 4 ZoneGroup: default (578316ef-20a8-4fff-9d6c-83d007bc8317) 2022-09-02T04:16:30.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 4 Zone: default (fd09a65f-bafa-4290-b162-3b15d819be90) 2022-09-02T04:16:30.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-02T04:16:30.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 zonegroup default 2022-09-02T04:16:30.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.514+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:30.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.515+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:30.523 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:30.515+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:32.494 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:32.491+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:32.491+0000 7f331ba13600 20 rados->read ofs=0 len=0 2022-09-02T04:16:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:32.494+0000 7f331ba13600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:32.494+0000 7f331ba13600 20 started sync module instance, tier type = 2022-09-02T04:16:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:32.494+0000 7f331ba13600 20 started zone id=fd09a65f-bafa-4290-b162-3b15d819be90 (name=default) with tier type = 2022-09-02T04:16:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=0 2022-09-02T04:16:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=1 2022-09-02T04:16:34.524 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=2 2022-09-02T04:16:34.524 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=3 2022-09-02T04:16:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=4 2022-09-02T04:16:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=5 2022-09-02T04:16:34.526 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.521+0000 7f331ba13600 20 add_watcher() i=6 2022-09-02T04:16:34.528 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.522+0000 7f331ba13600 20 add_watcher() i=7 2022-09-02T04:16:34.529 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.522+0000 7f331ba13600 2 all 8 watchers are set, enabling cache 2022-09-02T04:16:34.529 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.522+0000 7f331ba13600 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-02T04:16:34.529 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.522+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-02T04:16:34.529 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.522+0000 7f331ba13600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:34.530 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-02T04:16:34.530 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-02T04:16:34.530 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:34.531 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-02T04:16:34.531 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-02T04:16:34.531 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:34.531 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-02T04:16:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-02T04:16:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-02T04:16:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.523+0000 7f331ba13600 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-02T04:16:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.525+0000 7f331ba13600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:34.533 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.532+0000 7f3304194700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:34.534 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.532+0000 7f331ba13600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:34.534 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:34.532+0000 7f331ba13600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.507+0000 7f331ba13600 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-02T04:16:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.507+0000 7f331ba13600 5 note: GC not initialized 2022-09-02T04:16:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.509+0000 7f32ff98b700 20 reqs_thread_entry: start 2022-09-02T04:16:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.509+0000 7f331ba13600 20 init_complete bucket index max shards: 11 2022-09-02T04:16:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.509+0000 7f331ba13600 10 rgw notify: Started notification manager with: 1 workers 2022-09-02T04:16:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.509+0000 7f32fd987700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 2 04:17:07 2022 2022-09-02T04:16:37.513 INFO:teuthology.orchestra.run.smithi099.stderr: 2022-09-02T04:16:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.509+0000 7f32fd165700 20 reqs_thread_entry: start 2022-09-02T04:16:37.535 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:16:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.529+0000 7f331ba13600 20 remove_watcher() i=0 2022-09-02T04:16:37.537 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.529+0000 7f331ba13600 2 removed watcher, disabling cache 2022-09-02T04:16:37.538 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.530+0000 7f331ba13600 20 remove_watcher() i=1 2022-09-02T04:16:37.538 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.533+0000 7f331ba13600 20 remove_watcher() i=2 2022-09-02T04:16:37.538 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.535+0000 7f331ba13600 20 remove_watcher() i=3 2022-09-02T04:16:37.539 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.538+0000 7f331ba13600 20 remove_watcher() i=4 2022-09-02T04:16:37.542 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.541+0000 7f331ba13600 20 remove_watcher() i=5 2022-09-02T04:16:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.543+0000 7f331ba13600 20 remove_watcher() i=6 2022-09-02T04:16:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.545+0000 7f331ba13600 20 remove_watcher() i=7 2022-09-02T04:16:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.545+0000 7f3304194700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:37.551 DEBUG:tasks.util.rgw: json result: [] 2022-09-02T04:16:37.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:37.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:37.553 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-02T04:16:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.613+0000 7fba316da600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.613+0000 7fba316da600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:37.618 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.616+0000 7fba16654700 20 reqs_thread_entry: start 2022-09-02T04:16:37.619 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.616+0000 7fba316da600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-02T04:16:37.619 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.616+0000 7fba316da600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.620 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.617+0000 7fba316da600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:37.620 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.617+0000 7fba316da600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.621 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.618+0000 7fba316da600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:37.628 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 RGWRados::pool_iterate: got zone_info.fd09a65f-bafa-4290-b162-3b15d819be90 2022-09-02T04:16:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 RGWRados::pool_iterate: got zonegroup_info.578316ef-20a8-4fff-9d6c-83d007bc8317 2022-09-02T04:16:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-02T04:16:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-02T04:16:37.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:37.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.625+0000 7fba316da600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.626+0000 7fba316da600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:37.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.626+0000 7fba316da600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.626+0000 7fba316da600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:37.633 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:16:37.633 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fd09a65f-bafa-4290-b162-3b15d819be90", 2022-09-02T04:16:37.633 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default", 2022-09-02T04:16:37.634 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "default.rgw.meta:root", 2022-09-02T04:16:37.634 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "default.rgw.control", 2022-09-02T04:16:37.634 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-02T04:16:37.635 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-02T04:16:37.635 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "default.rgw.log", 2022-09-02T04:16:37.635 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-02T04:16:37.636 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-02T04:16:37.636 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-02T04:16:37.636 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-02T04:16:37.636 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-02T04:16:37.636 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-02T04:16:37.637 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-02T04:16:37.637 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-02T04:16:37.637 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "default.rgw.otp", 2022-09-02T04:16:37.637 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:16:37.638 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2022-09-02T04:16:37.638 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2022-09-02T04:16:37.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:37.638 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:16:37.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:16:37.639 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:16:37.639 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:16:37.639 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-02T04:16:37.640 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:16:37.640 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:16:37.640 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-02T04:16:37.640 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2022-09-02T04:16:37.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:37.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:37.641 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-02T04:16:37.642 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:16:37.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:37.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:37.643 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:16:37.643 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2022-09-02T04:16:37.643 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-02T04:16:37.643 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:16:37.644 DEBUG:tasks.util.rgw: json result: {'id': 'fd09a65f-bafa-4290-b162-3b15d819be90', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-02T04:16:37.644 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2022-09-02T04:16:37.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2022-09-02T04:16:37.645 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2022-09-02T04:16:37.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:37.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:37.696 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.694+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.699 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.695+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.699 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.695+0000 7f733ca26600 20 realm 2022-09-02T04:16:37.700 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.695+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.700 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.695+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.700 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.695+0000 7f733ca26600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-02T04:16:37.701 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.695+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.701 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.696+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.702 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.696+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.702 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.696+0000 7f733ca26600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:37.702 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.696+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.703 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.697+0000 7f733ca26600 20 rados_obj.operate() r=0 bl.length=889 2022-09-02T04:16:37.703 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.697+0000 7f733ca26600 20 searching for the correct realm 2022-09-02T04:16:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 RGWRados::pool_iterate: got zone_info.fd09a65f-bafa-4290-b162-3b15d819be90 2022-09-02T04:16:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 RGWRados::pool_iterate: got zonegroup_info.578316ef-20a8-4fff-9d6c-83d007bc8317 2022-09-02T04:16:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-02T04:16:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-02T04:16:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.704+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 zone default found 2022-09-02T04:16:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 4 Realm: () 2022-09-02T04:16:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 4 ZoneGroup: default (578316ef-20a8-4fff-9d6c-83d007bc8317) 2022-09-02T04:16:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 4 Zone: default (fd09a65f-bafa-4290-b162-3b15d819be90) 2022-09-02T04:16:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-02T04:16:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 zonegroup default 2022-09-02T04:16:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 started sync module instance, tier type = 2022-09-02T04:16:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.705+0000 7f733ca26600 20 started zone id=fd09a65f-bafa-4290-b162-3b15d819be90 (name=default) with tier type = 2022-09-02T04:16:37.727 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=0 2022-09-02T04:16:37.728 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=1 2022-09-02T04:16:37.728 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=2 2022-09-02T04:16:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=3 2022-09-02T04:16:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=4 2022-09-02T04:16:37.730 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=5 2022-09-02T04:16:37.730 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.725+0000 7f733ca26600 20 add_watcher() i=6 2022-09-02T04:16:37.731 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.726+0000 7f733ca26600 20 add_watcher() i=7 2022-09-02T04:16:37.731 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.726+0000 7f733ca26600 2 all 8 watchers are set, enabling cache 2022-09-02T04:16:37.732 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.727+0000 7f73251a7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:37.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.727+0000 7f733ca26600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:37.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.727+0000 7f733ca26600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:37.734 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.731+0000 7f733ca26600 5 note: GC not initialized 2022-09-02T04:16:37.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.732+0000 7f732099e700 20 reqs_thread_entry: start 2022-09-02T04:16:37.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.732+0000 7f733ca26600 20 init_complete bucket index max shards: 11 2022-09-02T04:16:37.737 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.732+0000 7f733ca26600 10 rgw notify: Started notification manager with: 1 workers 2022-09-02T04:16:37.737 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.732+0000 7f731e99a700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 2 04:17:07 2022 2022-09-02T04:16:37.738 INFO:teuthology.orchestra.run.smithi099.stderr: 2022-09-02T04:16:37.738 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.732+0000 7f731e178700 20 reqs_thread_entry: start 2022-09-02T04:16:37.741 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:16:37.742 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.739+0000 7f733ca26600 20 remove_watcher() i=0 2022-09-02T04:16:37.743 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.739+0000 7f733ca26600 2 removed watcher, disabling cache 2022-09-02T04:16:37.743 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.740+0000 7f733ca26600 20 remove_watcher() i=1 2022-09-02T04:16:37.743 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.741+0000 7f733ca26600 20 remove_watcher() i=2 2022-09-02T04:16:37.744 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.742+0000 7f733ca26600 20 remove_watcher() i=3 2022-09-02T04:16:37.744 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.743+0000 7f733ca26600 20 remove_watcher() i=4 2022-09-02T04:16:37.745 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.744+0000 7f733ca26600 20 remove_watcher() i=5 2022-09-02T04:16:37.746 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.744+0000 7f733ca26600 20 remove_watcher() i=6 2022-09-02T04:16:37.747 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.745+0000 7f733ca26600 20 remove_watcher() i=7 2022-09-02T04:16:37.747 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.746+0000 7f73251a7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:37.751 DEBUG:tasks.util.rgw: json result: [] 2022-09-02T04:16:37.751 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:37.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:37.752 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-02T04:16:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:37.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:37.856 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.848+0000 7f138a9de600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:37.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.848+0000 7f138a9de600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:37.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.852+0000 7f136f958700 20 reqs_thread_entry: start 2022-09-02T04:16:37.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.852+0000 7f138a9de600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-02T04:16:37.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.852+0000 7f138a9de600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.859 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.852+0000 7f138a9de600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:37.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.852+0000 7f138a9de600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.853+0000 7f138a9de600 20 rados_obj.operate() r=0 bl.length=889 2022-09-02T04:16:37.866 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 RGWRados::pool_iterate: got zone_info.fd09a65f-bafa-4290-b162-3b15d819be90 2022-09-02T04:16:37.866 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 RGWRados::pool_iterate: got zonegroup_info.578316ef-20a8-4fff-9d6c-83d007bc8317 2022-09-02T04:16:37.867 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-02T04:16:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-02T04:16:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados_obj.operate() r=0 bl.length=889 2022-09-02T04:16:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.871 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:37.871 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados->read ofs=0 len=0 2022-09-02T04:16:37.872 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:37.860+0000 7f138a9de600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:37.877 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:16:37.877 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fd09a65f-bafa-4290-b162-3b15d819be90", 2022-09-02T04:16:37.878 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default", 2022-09-02T04:16:37.882 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "default.rgw.meta:root", 2022-09-02T04:16:37.882 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "default.rgw.control", 2022-09-02T04:16:37.883 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-02T04:16:37.883 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-02T04:16:37.884 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "default.rgw.log", 2022-09-02T04:16:37.884 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-02T04:16:37.886 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-02T04:16:37.887 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-02T04:16:37.887 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-02T04:16:37.887 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-02T04:16:37.887 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-02T04:16:37.888 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-02T04:16:37.888 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-02T04:16:37.888 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "default.rgw.otp", 2022-09-02T04:16:37.888 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:16:37.889 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2022-09-02T04:16:37.889 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2022-09-02T04:16:37.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:37.889 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:16:37.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:16:37.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:16:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:16:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-02T04:16:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:16:37.891 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:16:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-02T04:16:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2022-09-02T04:16:37.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:37.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:37.893 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-02T04:16:37.893 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:16:37.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:37.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:37.894 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:16:37.894 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2022-09-02T04:16:37.895 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-02T04:16:37.895 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:16:37.898 DEBUG:tasks.util.rgw: json result: {'id': 'fd09a65f-bafa-4290-b162-3b15d819be90', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-02T04:16:37.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2022-09-02T04:16:37.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2022-09-02T04:16:37.901 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2022-09-02T04:16:37.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:37.986 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:37.991 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:37.989+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.444 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.442+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.444 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.442+0000 7fe74bd7e600 20 realm 2022-09-02T04:16:39.445 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.442+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.451 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.448+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.451 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.448+0000 7fe74bd7e600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-02T04:16:39.452 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.449+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.453 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.451+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.453 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.451+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.455 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.452+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.455 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.452+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.455 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.453+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.453+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.453+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.453+0000 7fe74bd7e600 10 Creating default zonegroup 2022-09-02T04:16:39.457 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.455+0000 7fe74bd7e600 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-09-02T04:16:39.469 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.465+0000 7fe74bd7e600 10 WARNING: store->list_zones() returned r=-2 2022-09-02T04:16:39.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.465+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.465+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.467+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.468+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.468+0000 7fe74bd7e600 10 could not read realm id: (2) No such file or directory 2022-09-02T04:16:39.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.468+0000 7fe74bd7e600 10 WARNING: failed to set zone as default, r=-22 2022-09-02T04:16:39.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.468+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.468+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.473 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.470+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.473 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.470+0000 7fe74bd7e600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:39.474 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.470+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.474 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:39.474 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 10 Using default name default 2022-09-02T04:16:39.474 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:39.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:39.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 zone default found 2022-09-02T04:16:39.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 4 Realm: () 2022-09-02T04:16:39.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 4 ZoneGroup: default (647a6c6f-5365-436a-ae75-c82569f0611b) 2022-09-02T04:16:39.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 4 Zone: default (a30f6539-fecd-42f7-9e74-a2d9485cbc6d) 2022-09-02T04:16:39.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-02T04:16:39.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 zonegroup default 2022-09-02T04:16:39.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.471+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:39.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.472+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:39.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:39.472+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:41.462 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:41.459+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:41.462 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:41.459+0000 7fe74bd7e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:41.463 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:41.462+0000 7fe74bd7e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:41.464 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:41.462+0000 7fe74bd7e600 20 started sync module instance, tier type = 2022-09-02T04:16:41.464 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:41.462+0000 7fe74bd7e600 20 started zone id=a30f6539-fecd-42f7-9e74-a2d9485cbc6d (name=default) with tier type = 2022-09-02T04:16:43.506 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=0 2022-09-02T04:16:43.506 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=1 2022-09-02T04:16:43.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=2 2022-09-02T04:16:43.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=3 2022-09-02T04:16:43.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=4 2022-09-02T04:16:43.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=5 2022-09-02T04:16:43.508 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.502+0000 7fe74bd7e600 20 add_watcher() i=6 2022-09-02T04:16:43.508 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 add_watcher() i=7 2022-09-02T04:16:43.508 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 2 all 8 watchers are set, enabling cache 2022-09-02T04:16:43.508 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-09-02T04:16:43.509 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-09-02T04:16:43.510 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:43.510 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-09-02T04:16:43.511 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-09-02T04:16:43.512 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:43.512 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-09-02T04:16:43.513 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-09-02T04:16:43.513 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.503+0000 7fe74bd7e600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:43.513 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.504+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-09-02T04:16:43.514 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.504+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-09-02T04:16:43.515 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.504+0000 7fe74bd7e600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:43.516 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.504+0000 7fe74bd7e600 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-09-02T04:16:43.516 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.504+0000 7fe74bd7e600 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-09-02T04:16:43.518 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.505+0000 7fe74bd7e600 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-09-02T04:16:43.518 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.508+0000 7fe7344ff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:43.519 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.508+0000 7fe74bd7e600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:43.519 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:43.508+0000 7fe74bd7e600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:46.483 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.474+0000 7fe74bd7e600 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-09-02T04:16:46.484 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.474+0000 7fe74bd7e600 5 note: GC not initialized 2022-09-02T04:16:46.484 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.475+0000 7fe74bd7e600 20 init_complete bucket index max shards: 11 2022-09-02T04:16:46.485 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.475+0000 7fe74bd7e600 10 rgw notify: Started notification manager with: 1 workers 2022-09-02T04:16:46.485 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.476+0000 7fe72dcf2700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 2 04:17:16 2022 2022-09-02T04:16:46.486 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T04:16:46.486 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.476+0000 7fe72d4d0700 20 reqs_thread_entry: start 2022-09-02T04:16:46.486 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.476+0000 7fe72fcf6700 20 reqs_thread_entry: start 2022-09-02T04:16:46.497 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:16:46.498 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.494+0000 7fe74bd7e600 20 remove_watcher() i=0 2022-09-02T04:16:46.498 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.494+0000 7fe74bd7e600 2 removed watcher, disabling cache 2022-09-02T04:16:46.499 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.496+0000 7fe74bd7e600 20 remove_watcher() i=1 2022-09-02T04:16:46.499 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.496+0000 7fe74bd7e600 20 remove_watcher() i=2 2022-09-02T04:16:46.500 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.498+0000 7fe74bd7e600 20 remove_watcher() i=3 2022-09-02T04:16:46.502 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.499+0000 7fe74bd7e600 20 remove_watcher() i=4 2022-09-02T04:16:46.503 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.501+0000 7fe74bd7e600 20 remove_watcher() i=5 2022-09-02T04:16:46.505 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.504+0000 7fe74bd7e600 20 remove_watcher() i=6 2022-09-02T04:16:46.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.505+0000 7fe74bd7e600 20 remove_watcher() i=7 2022-09-02T04:16:46.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.505+0000 7fe7344ff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:46.512 DEBUG:tasks.util.rgw: json result: [] 2022-09-02T04:16:46.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:46.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:46.514 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-02T04:16:46.568 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:46.568 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:46.580 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.572+0000 7fc72c36e600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:46.580 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.572+0000 7fc72c36e600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:46.581 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.575+0000 7fc7112e8700 20 reqs_thread_entry: start 2022-09-02T04:16:46.581 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.575+0000 7fc72c36e600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-02T04:16:46.594 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.575+0000 7fc72c36e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.594 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.576+0000 7fc72c36e600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:46.594 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.576+0000 7fc72c36e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.595 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.576+0000 7fc72c36e600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:46.610 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 RGWRados::pool_iterate: got zonegroup_info.647a6c6f-5365-436a-ae75-c82569f0611b 2022-09-02T04:16:46.611 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 RGWRados::pool_iterate: got zone_info.a30f6539-fecd-42f7-9e74-a2d9485cbc6d 2022-09-02T04:16:46.611 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-02T04:16:46.612 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-02T04:16:46.617 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.618 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 rados_obj.operate() r=0 bl.length=873 2022-09-02T04:16:46.619 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.619 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.620 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.584+0000 7fc72c36e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.620 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.585+0000 7fc72c36e600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:46.621 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.585+0000 7fc72c36e600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.621 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.585+0000 7fc72c36e600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:46.625 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:16:46.625 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "a30f6539-fecd-42f7-9e74-a2d9485cbc6d", 2022-09-02T04:16:46.625 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default", 2022-09-02T04:16:46.626 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "default.rgw.meta:root", 2022-09-02T04:16:46.626 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "default.rgw.control", 2022-09-02T04:16:46.626 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-02T04:16:46.627 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-02T04:16:46.627 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "default.rgw.log", 2022-09-02T04:16:46.627 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-02T04:16:46.627 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-02T04:16:46.628 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-02T04:16:46.628 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-02T04:16:46.628 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-02T04:16:46.629 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-02T04:16:46.629 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-02T04:16:46.629 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-02T04:16:46.629 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "default.rgw.otp", 2022-09-02T04:16:46.630 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:16:46.630 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "", 2022-09-02T04:16:46.630 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "" 2022-09-02T04:16:46.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:16:46.631 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:16:46.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:16:46.632 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:16:46.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:16:46.632 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-02T04:16:46.632 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:16:46.632 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:16:46.633 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-02T04:16:46.633 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:16:46.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:16:46.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:16:46.634 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-02T04:16:46.634 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:16:46.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:16:46.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:16:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:16:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2022-09-02T04:16:46.636 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-02T04:16:46.636 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:16:46.637 DEBUG:tasks.util.rgw: json result: {'id': 'a30f6539-fecd-42f7-9e74-a2d9485cbc6d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-02T04:16:46.637 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2022-09-02T04:16:46.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2022-09-02T04:16:46.638 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2022-09-02T04:16:46.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:46.693 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:46.696 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.692+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.696 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.696 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 realm 2022-09-02T04:16:46.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-09-02T04:16:46.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.698 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.693+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.698 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.694+0000 7f0216c26600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:46.698 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.694+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.699 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.694+0000 7f0216c26600 20 rados_obj.operate() r=0 bl.length=889 2022-09-02T04:16:46.699 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.694+0000 7f0216c26600 20 searching for the correct realm 2022-09-02T04:16:46.719 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 RGWRados::pool_iterate: got zonegroup_info.647a6c6f-5365-436a-ae75-c82569f0611b 2022-09-02T04:16:46.721 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 RGWRados::pool_iterate: got zone_info.a30f6539-fecd-42f7-9e74-a2d9485cbc6d 2022-09-02T04:16:46.722 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-02T04:16:46.722 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-02T04:16:46.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.724 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:46.724 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.701+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.725 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:46.726 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 zone default found 2022-09-02T04:16:46.727 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 4 Realm: () 2022-09-02T04:16:46.728 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 4 ZoneGroup: default (647a6c6f-5365-436a-ae75-c82569f0611b) 2022-09-02T04:16:46.728 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 4 Zone: default (a30f6539-fecd-42f7-9e74-a2d9485cbc6d) 2022-09-02T04:16:46.729 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 10 cannot find current period zonegroup using local zonegroup configuration 2022-09-02T04:16:46.729 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 zonegroup default 2022-09-02T04:16:46.729 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.730 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.731 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.732 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.732 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 started sync module instance, tier type = 2022-09-02T04:16:46.732 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.702+0000 7f0216c26600 20 started zone id=a30f6539-fecd-42f7-9e74-a2d9485cbc6d (name=default) with tier type = 2022-09-02T04:16:46.736 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=0 2022-09-02T04:16:46.737 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=1 2022-09-02T04:16:46.737 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=2 2022-09-02T04:16:46.738 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=3 2022-09-02T04:16:46.738 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=4 2022-09-02T04:16:46.739 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=5 2022-09-02T04:16:46.739 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.723+0000 7f0216c26600 20 add_watcher() i=6 2022-09-02T04:16:46.745 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.724+0000 7f0216c26600 20 add_watcher() i=7 2022-09-02T04:16:46.745 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.724+0000 7f0216c26600 2 all 8 watchers are set, enabling cache 2022-09-02T04:16:46.745 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.725+0000 7f01ff3a7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:46.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.725+0000 7f0216c26600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:46.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.725+0000 7f0216c26600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:46.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.729+0000 7f0216c26600 5 note: GC not initialized 2022-09-02T04:16:46.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.730+0000 7f01fab9e700 20 reqs_thread_entry: start 2022-09-02T04:16:46.747 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.730+0000 7f0216c26600 20 init_complete bucket index max shards: 11 2022-09-02T04:16:46.747 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.730+0000 7f0216c26600 10 rgw notify: Started notification manager with: 1 workers 2022-09-02T04:16:46.747 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.730+0000 7f01f8b9a700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 2 04:17:17 2022 2022-09-02T04:16:46.747 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T04:16:46.748 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.730+0000 7f01f8378700 20 reqs_thread_entry: start 2022-09-02T04:16:46.748 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:16:46.750 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.738+0000 7f0216c26600 20 remove_watcher() i=0 2022-09-02T04:16:46.751 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.738+0000 7f0216c26600 2 removed watcher, disabling cache 2022-09-02T04:16:46.751 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.739+0000 7f0216c26600 20 remove_watcher() i=1 2022-09-02T04:16:46.751 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.740+0000 7f0216c26600 20 remove_watcher() i=2 2022-09-02T04:16:46.751 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.740+0000 7f0216c26600 20 remove_watcher() i=3 2022-09-02T04:16:46.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.741+0000 7f0216c26600 20 remove_watcher() i=4 2022-09-02T04:16:46.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.742+0000 7f0216c26600 20 remove_watcher() i=5 2022-09-02T04:16:46.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.743+0000 7f0216c26600 20 remove_watcher() i=6 2022-09-02T04:16:46.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.744+0000 7f0216c26600 20 remove_watcher() i=7 2022-09-02T04:16:46.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.744+0000 7f01ff3a7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-09-02T04:16:46.753 DEBUG:tasks.util.rgw: json result: [] 2022-09-02T04:16:46.753 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:46.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-09-02T04:16:46.754 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-09-02T04:16:46.847 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:46.848 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:46.848 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.844+0000 7f1de28f4600 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-09-02T04:16:46.849 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.844+0000 7f1de28f4600 20 check_secure_mon_conn(): mode 1 is insecure 2022-09-02T04:16:46.851 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.847+0000 7f1dc786e700 20 reqs_thread_entry: start 2022-09-02T04:16:46.852 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.847+0000 7f1de28f4600 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-09-02T04:16:46.852 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.847+0000 7f1de28f4600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.852 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.848+0000 7f1de28f4600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:46.853 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.848+0000 7f1de28f4600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.853 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.848+0000 7f1de28f4600 20 rados_obj.operate() r=0 bl.length=889 2022-09-02T04:16:46.859 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.855+0000 7f1de28f4600 20 RGWRados::pool_iterate: got zonegroup_info.647a6c6f-5365-436a-ae75-c82569f0611b 2022-09-02T04:16:46.860 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.855+0000 7f1de28f4600 20 RGWRados::pool_iterate: got zone_info.a30f6539-fecd-42f7-9e74-a2d9485cbc6d 2022-09-02T04:16:46.860 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.855+0000 7f1de28f4600 20 RGWRados::pool_iterate: got zone_names.default 2022-09-02T04:16:46.861 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.855+0000 7f1de28f4600 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-09-02T04:16:46.861 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.855+0000 7f1de28f4600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.862 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados_obj.operate() r=0 bl.length=889 2022-09-02T04:16:46.862 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.864 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados_obj.operate() r=-2 bl.length=0 2022-09-02T04:16:46.865 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.865 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados_obj.operate() r=0 bl.length=46 2022-09-02T04:16:46.865 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados->read ofs=0 len=0 2022-09-02T04:16:46.865 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:16:46.856+0000 7f1de28f4600 20 rados_obj.operate() r=0 bl.length=398 2022-09-02T04:16:46.867 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:16:46.867 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "a30f6539-fecd-42f7-9e74-a2d9485cbc6d", 2022-09-02T04:16:46.867 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default", 2022-09-02T04:16:46.867 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "default.rgw.meta:root", 2022-09-02T04:16:46.868 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "default.rgw.control", 2022-09-02T04:16:46.868 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "default.rgw.log:gc", 2022-09-02T04:16:46.868 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "default.rgw.log:lc", 2022-09-02T04:16:46.868 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "default.rgw.log", 2022-09-02T04:16:46.868 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-09-02T04:16:46.869 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-09-02T04:16:46.869 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "default.rgw.meta:roles", 2022-09-02T04:16:46.869 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-09-02T04:16:46.869 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-09-02T04:16:46.870 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-09-02T04:16:46.870 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-09-02T04:16:46.870 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-09-02T04:16:46.870 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "default.rgw.otp", 2022-09-02T04:16:46.870 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:16:46.871 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "", 2022-09-02T04:16:46.871 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "" 2022-09-02T04:16:46.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:16:46.871 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:16:46.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:16:46.872 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:16:46.873 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:16:46.873 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "default.rgw.buckets.index", 2022-09-02T04:16:46.873 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:16:46.873 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:16:46.874 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "default.rgw.buckets.data", 2022-09-02T04:16:46.874 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:16:46.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:16:46.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:16:46.874 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-09-02T04:16:46.875 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:16:46.875 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:16:46.875 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:16:46.875 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:16:46.876 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2022-09-02T04:16:46.876 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "default.rgw.log:notif" 2022-09-02T04:16:46.876 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:16:46.876 DEBUG:tasks.util.rgw: json result: {'id': 'a30f6539-fecd-42f7-9e74-a2d9485cbc6d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-09-02T04:16:46.877 INFO:tasks.rgw:Starting rgw... 2022-09-02T04:16:46.878 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2022-09-02T04:16:46.878 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-02T04:16:46.878 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-02T04:16:46.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-02T04:16:46.887 INFO:tasks.rgw.c1.client.0:Started 2022-09-02T04:16:46.889 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2022-09-02T04:16:46.889 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-02T04:16:46.890 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-02T04:16:46.890 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-02T04:16:46.895 INFO:tasks.rgw.c1.client.1:Started 2022-09-02T04:16:46.895 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2022-09-02T04:16:46.896 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-02T04:16:46.896 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-02T04:16:46.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-02T04:16:46.910 INFO:tasks.rgw.c2.client.0:Started 2022-09-02T04:16:46.911 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2022-09-02T04:16:46.911 INFO:tasks.rgw:Using beast as radosgw frontend 2022-09-02T04:16:46.911 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-02T04:16:46.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-02T04:16:46.922 INFO:tasks.rgw.c2.client.1:Started 2022-09-02T04:16:46.923 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi099.front.sepia.ceph.com:8000/ 2022-09-02T04:16:46.923 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2022-09-02T04:16:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:16:46.992 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:46.992 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:46.993 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:16:47.995 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2022-09-02T04:16:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:16:48.027 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:48.028 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:48.028 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:16:50.029 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2022-09-02T04:16:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:16:50.066 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:50.066 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:50.066 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:16:54.071 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2022-09-02T04:16:54.103 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:54.103 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:54.104 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-09-02T04:16:54.105 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2022-09-02T04:16:54.105 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi099.front.sepia.ceph.com:8001/ 2022-09-02T04:16:54.106 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001/ 2022-09-02T04:16:54.175 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:54.175 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:54.176 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-09-02T04:16:54.179 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2022-09-02T04:16:54.180 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi160.front.sepia.ceph.com:8000/ 2022-09-02T04:16:54.181 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000/ 2022-09-02T04:16:54.236 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:54.237 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:54.238 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-09-02T04:16:54.239 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2022-09-02T04:16:54.239 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi160.front.sepia.ceph.com:8001/ 2022-09-02T04:16:54.240 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001/ 2022-09-02T04:16:54.309 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:16:54.340 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:16:54.340 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-09-02T04:16:54.341 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2022-09-02T04:16:54.342 INFO:teuthology.run_tasks:Running task rgw-multisite... 2022-09-02T04:16:54.763 INFO:tasks.rgw_multisite:creating realm.. 2022-09-02T04:16:54.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:54.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:54.765 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:16:54.819 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:54.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:54.849 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:16:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:16:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-realm", 2022-09-02T04:16:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "current_period": "3488ede9-0214-4989-ab22-77a221ac7b2c", 2022-09-02T04:16:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2022-09-02T04:16:54.852 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:16:54.855 INFO:tasks.rgw_multisite:creating master zonegroup.. 2022-09-02T04:16:54.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:54.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:54.857 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:16:54.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:54.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:54.920 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:54.918+0000 7f1bb6acc600 -1 old format 2022-09-02T04:16:54.927 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:16:54.928 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:16:54.928 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:16:54.929 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:16:54.929 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:16:54.930 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:16:54.930 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:16:54.931 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:16:54.931 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:16:54.932 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:16:54.932 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "", 2022-09-02T04:16:54.932 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [], 2022-09-02T04:16:54.933 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [], 2022-09-02T04:16:54.934 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2022-09-02T04:16:54.934 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:16:54.935 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:16:54.935 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:16:54.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:54.936 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:16:54.936 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:16:54.936 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:16:54.937 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:16:54.937 INFO:tasks.rgw_multisite:creating master zone.. 2022-09-02T04:16:54.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:54.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:54.939 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --endpoints http://smithi099.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:16:55.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:55.024 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:55.055 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:16:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:16:55.056 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:16:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-02T04:16:55.057 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-02T04:16:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-02T04:16:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-02T04:16:55.058 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-02T04:16:55.059 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-02T04:16:55.059 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-02T04:16:55.059 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-02T04:16:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-02T04:16:55.060 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-02T04:16:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-02T04:16:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-02T04:16:55.061 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-02T04:16:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-02T04:16:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:16:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:16:55.067 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:16:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:16:55.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:16:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:16:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:16:55.069 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-02T04:16:55.070 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:16:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:16:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2022-09-02T04:16:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-02T04:16:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:16:55.072 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:16:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:16:55.073 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-02T04:16:55.074 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:16:55.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2022-09-02T04:16:55.535 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'test-zone1.rgw.buckets.data' created 2022-09-02T04:16:55.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2022-09-02T04:16:56.539 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2022-09-02T04:16:56.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:56.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:56.562 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:16:56.617 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:56.618 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:56.647 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:16:56.648 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:16:56.648 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:16:56.648 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-02T04:16:56.649 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-02T04:16:56.649 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-02T04:16:56.649 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-02T04:16:56.650 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-02T04:16:56.650 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-02T04:16:56.651 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-02T04:16:56.651 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-02T04:16:56.651 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-02T04:16:56.652 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-02T04:16:56.653 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-02T04:16:56.654 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-02T04:16:56.654 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-02T04:16:56.655 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-02T04:16:56.655 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:16:56.655 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:16:56.656 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:16:56.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:56.657 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:16:56.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:16:56.658 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:16:56.658 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:16:56.658 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-02T04:16:56.659 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:16:56.659 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:16:56.660 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-02T04:16:56.660 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2022-09-02T04:16:56.660 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:56.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:16:56.661 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-02T04:16:56.662 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:16:56.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:56.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:16:56.663 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:16:56.665 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2022-09-02T04:16:56.665 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-02T04:16:56.666 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:16:56.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:56.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:16:56.669 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:16:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:16:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:16:56.730 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:16:56.725+0000 7f19598fb600 0 period (3488ede9-0214-4989-ab22-77a221ac7b2c does not have zone b210c8c8-c433-420b-87c1-57ebfc433073 configured 2022-09-02T04:17:03.626 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:17:03.627 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:03.627 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1, 2022-09-02T04:17:03.628 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "3488ede9-0214-4989-ab22-77a221ac7b2c", 2022-09-02T04:17:03.628 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2022-09-02T04:17:03.629 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2022-09-02T04:17:03.629 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:03.629 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2022-09-02T04:17:03.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:03.630 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:17:03.631 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:17:03.631 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:17:03.631 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:17:03.632 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:17:03.632 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:03.633 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:03.633 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:17:03.633 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:17:03.634 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:03.634 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:17:03.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:03.635 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:03.635 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:17:03.635 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:17:03.636 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:03.636 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:03.637 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:17:03.637 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": false, 2022-09-02T04:17:03.638 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:03.638 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:17:03.639 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:17:03.639 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:17:03.640 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:17:03.640 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:17:03.641 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:17:03.641 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:17:03.642 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:03.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:03.643 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:03.643 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:17:03.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:03.644 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:17:03.644 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:17:03.645 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:17:03.645 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:17:03.646 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:03.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:03.647 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:03.647 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:17:03.648 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:03.648 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:17:03.649 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:17:03.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.649 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:17:03.650 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:17:03.650 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:03.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:03.651 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:03.652 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2022-09-02T04:17:03.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:03.653 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:03.653 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2020452060 2022-09-02T04:17:03.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:03.654 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:03.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.655 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:17:03.655 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:03.656 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2022-09-02T04:17:03.656 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:17:03.656 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:17:03.657 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:17:03.657 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:17:03.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:17:03.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:17:03.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.659 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:17:03.659 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:17:03.660 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:17:03.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:17:03.661 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:17:03.661 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:17:03.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.662 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2022-09-02T04:17:03.663 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:17:03.663 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:17:03.664 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:17:03.664 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:17:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:17:03.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2022-09-02T04:17:03.666 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:17:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:17:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:17:03.667 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:17:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:17:03.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2022-09-02T04:17:03.669 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:17:03.670 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:17:03.670 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:17:03.671 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:17:03.671 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:17:03.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:03.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:03.673 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:03.673 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2022-09-02T04:17:03.674 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:17:03.674 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:17:03.677 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-09-02T04:17:03.678 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T04:17:03.679 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2022-09-02T04:17:03.677+0000 7ff411aff700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 109678) UID: 0 2022-09-02T04:17:03.680 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2022-09-02T04:17:03.677+0000 7ff415d6a600 -1 shutting down 2022-09-02T04:17:06.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-09-02T04:17:09.780 INFO:tasks.rgw.c1.client.0:Stopped 2022-09-02T04:17:09.780 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-09-02T04:17:09.781 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-09-02T04:17:09.784 INFO:tasks.rgw.c1.client.0:Started 2022-09-02T04:17:09.785 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000 2022-09-02T04:17:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:17:09.830 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:09.831 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:09.831 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:17:10.833 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000 2022-09-02T04:17:10.866 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:10.867 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:10.868 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T04:17:10.869 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2022-09-02T04:17:10.870 INFO:tasks.rgw_multisite:creating admin user.. 2022-09-02T04:17:10.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:10.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:10.872 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:10.975 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:10.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:11.009 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:17:11.010 INFO:teuthology.orchestra.run.smithi099.stdout: "user_id": "realm-admin", 2022-09-02T04:17:11.010 INFO:teuthology.orchestra.run.smithi099.stdout: "display_name": "Realm Admin", 2022-09-02T04:17:11.010 INFO:teuthology.orchestra.run.smithi099.stdout: "email": "", 2022-09-02T04:17:11.011 INFO:teuthology.orchestra.run.smithi099.stdout: "suspended": 0, 2022-09-02T04:17:11.011 INFO:teuthology.orchestra.run.smithi099.stdout: "max_buckets": 1000, 2022-09-02T04:17:11.011 INFO:teuthology.orchestra.run.smithi099.stdout: "subusers": [], 2022-09-02T04:17:11.011 INFO:teuthology.orchestra.run.smithi099.stdout: "keys": [ 2022-09-02T04:17:11.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:11.012 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "realm-admin", 2022-09-02T04:17:11.012 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:11.012 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:11.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:11.013 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:11.013 INFO:teuthology.orchestra.run.smithi099.stdout: "swift_keys": [], 2022-09-02T04:17:11.014 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": [], 2022-09-02T04:17:11.018 INFO:teuthology.orchestra.run.smithi099.stdout: "op_mask": "read, write, delete", 2022-09-02T04:17:11.019 INFO:teuthology.orchestra.run.smithi099.stdout: "system": true, 2022-09-02T04:17:11.019 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2022-09-02T04:17:11.019 INFO:teuthology.orchestra.run.smithi099.stdout: "default_storage_class": "", 2022-09-02T04:17:11.019 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_tags": [], 2022-09-02T04:17:11.020 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:17:11.020 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:17:11.020 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:17:11.020 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:17:11.021 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:17:11.021 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:17:11.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:11.021 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:17:11.021 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:17:11.022 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:17:11.022 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:17:11.023 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:17:11.023 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:17:11.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:11.024 INFO:teuthology.orchestra.run.smithi099.stdout: "temp_url_keys": [], 2022-09-02T04:17:11.024 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "rgw", 2022-09-02T04:17:11.024 INFO:teuthology.orchestra.run.smithi099.stdout: "mfa_ids": [] 2022-09-02T04:17:11.025 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:17:11.025 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:17:11.026 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-02T04:17:11.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:11.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:11.028 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi099.front.sepia.ceph.com:8000 --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:11.091 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:11.091 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:11.145 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:11.145 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:11.146 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-realm", 2022-09-02T04:17:11.146 INFO:teuthology.orchestra.run.smithi160.stdout: "current_period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:11.146 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2 2022-09-02T04:17:11.146 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:11.147 INFO:tasks.rgw_multisite:creating zone.. 2022-09-02T04:17:11.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:11.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:11.148 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --endpoints http://smithi160.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:11.205 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:11.205 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:11.232 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:11.229+0000 7ff8cb020600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:17:11.232 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:11.229+0000 7ff8cb020600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:17:11.237 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:11.238 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:11.238 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:17:11.238 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:17:11.238 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:17:11.239 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:17:11.239 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:17:11.239 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:17:11.240 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:17:11.240 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:17:11.240 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:17:11.240 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:17:11.241 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:17:11.241 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:17:11.241 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:17:11.241 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:17:11.242 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:17:11.242 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:17:11.242 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:11.242 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:11.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:11.243 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:17:11.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:11.243 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:17:11.243 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:17:11.244 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:17:11.244 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:17:11.244 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:17:11.244 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2022-09-02T04:17:11.248 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:11.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:11.249 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:17:11.249 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:17:11.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:11.250 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:11.250 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:11.250 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:11.250 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:17:11.251 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:11.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2022-09-02T04:17:12.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2022-09-02T04:17:12.529 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'test-zone2.rgw.buckets.data' created 2022-09-02T04:17:12.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2022-09-02T04:17:13.550 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2022-09-02T04:17:13.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:13.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:13.570 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:13.626 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:13.627 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:13.649 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:17:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:17:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:17:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:17:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:17:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:17:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:17:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:17:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:17:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:17:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:17:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:17:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:17:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:17:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:17:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:17:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:17:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:17:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:17:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:17:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:17:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:17:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-02T04:17:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:17:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:17:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:17:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2022-09-02T04:17:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:17:13.667 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:13.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:13.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:13.671 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:13.722 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:13.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:13.731 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:13.729+0000 7f980898c600 0 period (e248e3d2-c4b4-4705-8303-7ccdb6560fe1 does not have zone 8b4ce987-f0fe-4038-9bfd-de8d46ec635c configured 2022-09-02T04:17:20.596 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:17:20.620 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:20.621 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:20.621 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2, 2022-09-02T04:17:20.621 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "3488ede9-0214-4989-ab22-77a221ac7b2c", 2022-09-02T04:17:20.622 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [], 2022-09-02T04:17:20.622 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:17:20.622 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:20.623 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:17:20.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:20.624 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:17:20.624 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:17:20.625 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:17:20.625 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:17:20.626 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:20.626 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:20.627 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:20.627 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:17:20.632 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:17:20.633 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:20.633 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:17:20.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:20.634 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:20.634 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:17:20.634 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:20.635 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:17:20.635 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:20.636 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:17:20.636 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:17:20.636 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:20.637 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:17:20.637 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:17:20.637 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:17:20.637 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:17:20.638 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:17:20.638 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:17:20.638 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:20.638 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:20.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:20.639 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:20.639 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:17:20.639 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:20.640 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:20.640 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:20.640 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:17:20.641 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:17:20.641 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:20.641 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:17:20.641 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:17:20.642 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:17:20.642 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:17:20.642 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:17:20.642 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:17:20.643 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:20.643 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:20.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:20.643 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:20.644 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:17:20.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:20.645 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:17:20.645 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:17:20.645 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:17:20.645 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:17:20.646 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:20.646 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:20.647 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:20.647 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:17:20.647 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:20.648 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:17:20.648 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:17:20.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.649 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:17:20.649 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:20.649 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:20.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:20.650 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:20.650 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:17:20.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:20.651 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:20.652 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:17:20.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:20.653 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:20.653 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:17:20.654 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:20.654 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:20.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.655 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:17:20.656 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:20.656 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:17:20.657 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:17:20.657 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:17:20.658 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:17:20.659 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:17:20.659 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:17:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:17:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:17:20.662 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:17:20.662 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:17:20.663 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:17:20.663 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:17:20.664 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:17:20.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:17:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:17:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:17:20.666 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:17:20.666 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:17:20.667 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:17:20.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.669 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:17:20.669 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:17:20.670 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:17:20.671 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:17:20.672 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:17:20.672 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:17:20.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.675 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:17:20.675 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:17:20.675 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:17:20.676 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:17:20.676 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:17:20.677 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:17:20.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:20.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:20.679 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:20.679 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:17:20.680 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:17:20.681 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:20.685 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-02T04:17:20.686 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T04:17:20.688 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T04:17:20.685+0000 7f7d12407700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 109801) UID: 0 2022-09-02T04:17:20.688 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T04:17:20.685+0000 7f7d16672600 -1 shutting down 2022-09-02T04:17:24.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-02T04:17:26.790 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-02T04:17:26.791 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-02T04:17:26.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-02T04:17:26.800 INFO:tasks.rgw.c2.client.0:Started 2022-09-02T04:17:26.801 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T04:17:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:17:26.830 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:26.830 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:26.831 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:17:27.837 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T04:17:27.868 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:27.869 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:17:27.872 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:17:29.877 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T04:17:29.911 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:29.914 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:29.918 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:17:29.919 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T04:17:30.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-02T04:17:33.922 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T04:17:33.949 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:33.949 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:33.951 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2022-09-02T04:17:33.953 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2022-09-02T04:17:33.956 INFO:tasks.rgw_multisite:creating zone.. 2022-09-02T04:17:33.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:33.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:33.957 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --endpoints http://smithi099.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:34.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:34.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:34.042 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:17:34.041+0000 7fe0a1ad8600 0 failed reading obj info from .rgw.root:zone_info.8b4ce987-f0fe-4038-9bfd-de8d46ec635c: (2) No such file or directory 2022-09-02T04:17:34.043 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:17:34.041+0000 7fe0a1ad8600 0 WARNING: could not read zone params for zone id=8b4ce987-f0fe-4038-9bfd-de8d46ec635c name=test-zone2 2022-09-02T04:17:34.046 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:17:34.047 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:17:34.047 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:17:34.048 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-02T04:17:34.048 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-02T04:17:34.049 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-02T04:17:34.049 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-02T04:17:34.050 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-02T04:17:34.050 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-02T04:17:34.052 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-02T04:17:34.053 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-02T04:17:34.053 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-02T04:17:34.054 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-02T04:17:34.054 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-02T04:17:34.055 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-02T04:17:34.055 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-02T04:17:34.056 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-02T04:17:34.056 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:17:34.056 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:34.057 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:34.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:34.058 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:17:34.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:34.059 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:17:34.059 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:17:34.059 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-02T04:17:34.060 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:17:34.060 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:17:34.061 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2022-09-02T04:17:34.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:34.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:34.062 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-02T04:17:34.063 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:17:34.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:34.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:34.064 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:34.064 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:34.065 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-02T04:17:34.065 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:17:34.071 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2022-09-02T04:17:34.624 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'test-zone3.rgw.buckets.data' created 2022-09-02T04:17:34.634 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2022-09-02T04:17:35.644 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2022-09-02T04:17:35.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:35.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:35.673 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:35.727 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:35.728 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:35.750 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:17:35.750 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:17:35.751 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:17:35.751 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-09-02T04:17:35.752 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "test-zone3.rgw.control", 2022-09-02T04:17:35.752 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-09-02T04:17:35.752 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-09-02T04:17:35.753 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "test-zone3.rgw.log", 2022-09-02T04:17:35.753 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-09-02T04:17:35.753 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-09-02T04:17:35.754 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-09-02T04:17:35.754 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-09-02T04:17:35.754 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-09-02T04:17:35.755 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-09-02T04:17:35.755 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-09-02T04:17:35.755 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-09-02T04:17:35.756 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-09-02T04:17:35.757 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:17:35.757 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:35.758 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:35.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:35.759 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:17:35.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:35.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:17:35.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:17:35.760 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-09-02T04:17:35.760 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:17:35.761 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:17:35.761 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2022-09-02T04:17:35.761 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2022-09-02T04:17:35.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:35.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:35.763 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-09-02T04:17:35.763 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:17:35.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:35.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:35.764 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:35.764 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2022-09-02T04:17:35.765 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-09-02T04:17:35.765 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:17:35.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:35.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:35.769 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:35.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:35.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:35.828 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:17:35.826+0000 7ff5e2b63600 0 period (e248e3d2-c4b4-4705-8303-7ccdb6560fe1 does not have zone 839886eb-91c7-4e3c-80d4-fd4e89474f26 configured 2022-09-02T04:17:42.682 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:17:42.709 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:17:42.710 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:42.710 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3, 2022-09-02T04:17:42.710 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "3488ede9-0214-4989-ab22-77a221ac7b2c", 2022-09-02T04:17:42.710 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2022-09-02T04:17:42.711 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2022-09-02T04:17:42.711 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:42.711 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2022-09-02T04:17:42.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.713 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:17:42.713 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:17:42.714 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:17:42.714 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:17:42.715 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:17:42.715 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:42.716 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.716 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:17:42.716 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:17:42.717 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:42.717 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:17:42.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.719 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:17:42.719 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:17:42.719 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:17:42.720 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:17:42.720 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.720 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:17:42.720 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:17:42.721 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:42.721 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:17:42.721 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:17:42.721 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:17:42.721 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:17:42.722 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:17:42.722 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:17:42.722 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:17:42.722 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:42.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.723 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:42.723 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone2", 2022-09-02T04:17:42.724 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:17:42.724 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:17:42.724 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.724 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:17:42.725 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:17:42.725 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:42.727 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:17:42.727 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:17:42.728 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:17:42.730 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:17:42.730 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:17:42.731 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:17:42.731 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:17:42.732 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:42.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.735 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:42.736 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:17:42.736 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:17:42.736 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:42.737 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.737 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:17:42.738 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:17:42.738 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:42.739 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:17:42.739 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:17:42.740 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:17:42.741 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:17:42.741 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:17:42.741 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:17:42.742 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:17:42.743 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:42.743 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:42.744 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.744 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:17:42.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.745 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:17:42.745 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:17:42.746 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:17:42.746 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:17:42.747 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:42.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:42.747 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.747 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:17:42.748 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:42.748 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:17:42.749 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:17:42.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.751 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:17:42.751 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:17:42.752 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:42.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:42.752 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:17:42.753 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2022-09-02T04:17:42.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.753 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:17:42.754 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3855502858 2022-09-02T04:17:42.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.755 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:42.755 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 805948903 2022-09-02T04:17:42.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:17:42.757 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:42.757 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2020452060 2022-09-02T04:17:42.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:42.759 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:17:42.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.763 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:17:42.764 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:42.764 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2022-09-02T04:17:42.765 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:17:42.765 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:17:42.766 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:17:42.766 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:17:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:17:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:17:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:17:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:17:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:17:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:17:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:17:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:17:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2022-09-02T04:17:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:17:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:17:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:17:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:17:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:17:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2022-09-02T04:17:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:17:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:17:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:17:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:17:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:17:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2022-09-02T04:17:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:17:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:17:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:17:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:17:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:17:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:17:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:17:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2022-09-02T04:17:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:17:42.777 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:17:42.778 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-02T04:17:42.778 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T04:17:42.779 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T04:17:42.778+0000 7f19df245700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 109683) UID: 0 2022-09-02T04:17:42.780 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T04:17:42.778+0000 7f19e34b0600 -1 shutting down 2022-09-02T04:17:46.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-02T04:17:48.886 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-02T04:17:48.886 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-02T04:17:48.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-02T04:17:48.899 INFO:tasks.rgw.c1.client.1:Started 2022-09-02T04:17:48.900 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2022-09-02T04:17:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:17:48.933 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:48.933 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:48.934 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2022-09-02T04:17:49.935 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2022-09-02T04:17:49.970 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:17:49.971 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:17:49.971 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T04:17:49.975 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2022-09-02T04:17:49.976 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-09-02T04:17:49.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:49.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:49.978 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi099.front.sepia.ceph.com:8000 --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --url http://smithi099.front.sepia.ceph.com:8000 --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:50.059 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:50.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:50.079 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T04:17:50.074+0000 7f38ab06e700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.d53aaf3c-e9e8-4189-80a7-dfda98d01df2.control err=-107 2022-09-02T04:17:50.084 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:50.085 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:50.086 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-realm", 2022-09-02T04:17:50.086 INFO:teuthology.orchestra.run.smithi160.stdout: "current_period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:50.086 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2 2022-09-02T04:17:50.087 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:50.088 INFO:tasks.rgw_multisite:creating zone.. 2022-09-02T04:17:50.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:50.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--endpoints', 'http://smithi160.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:50.090 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --endpoints http://smithi160.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:50.154 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:50.155 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:50.174 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:50.172+0000 7ff3b3e83600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:17:50.174 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:50.172+0000 7ff3b3e83600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:17:50.175 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:50.173+0000 7ff3b3e83600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:17:50.175 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:50.173+0000 7ff3b3e83600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:17:50.177 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:17:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:17:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-09-02T04:17:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone4.rgw.control", 2022-09-02T04:17:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-09-02T04:17:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-09-02T04:17:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone4.rgw.log", 2022-09-02T04:17:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-09-02T04:17:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-09-02T04:17:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-09-02T04:17:50.181 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-09-02T04:17:50.181 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-09-02T04:17:50.182 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-09-02T04:17:50.182 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-09-02T04:17:50.182 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-09-02T04:17:50.182 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-09-02T04:17:50.183 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:17:50.183 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:50.183 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:50.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:50.184 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:17:50.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:50.185 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:17:50.185 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:17:50.186 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-09-02T04:17:50.186 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:17:50.187 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:17:50.187 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2022-09-02T04:17:50.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:50.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:50.189 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-09-02T04:17:50.189 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:17:50.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:50.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:50.191 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:50.191 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_config": { 2022-09-02T04:17:50.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event_retention_days": 7, 2022-09-02T04:17:50.192 INFO:teuthology.orchestra.run.smithi160.stdout: "start_with_full_sync": false 2022-09-02T04:17:50.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:50.193 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:50.193 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-09-02T04:17:50.221 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:50.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2022-09-02T04:17:50.605 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'test-zone4.rgw.buckets.data' created 2022-09-02T04:17:50.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2022-09-02T04:17:51.644 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2022-09-02T04:17:51.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:51.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:51.669 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:51.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:51.726 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:51.748 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:51.749 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:17:51.749 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:17:51.750 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-09-02T04:17:51.750 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone4.rgw.control", 2022-09-02T04:17:51.750 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-09-02T04:17:51.751 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-09-02T04:17:51.751 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone4.rgw.log", 2022-09-02T04:17:51.752 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-09-02T04:17:51.752 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-09-02T04:17:51.753 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-09-02T04:17:51.753 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-09-02T04:17:51.754 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-09-02T04:17:51.754 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-09-02T04:17:51.755 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-09-02T04:17:51.755 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-09-02T04:17:51.756 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-09-02T04:17:51.756 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:17:51.757 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:17:51.759 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:17:51.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:51.760 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:17:51.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:51.761 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:17:51.761 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:17:51.761 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-09-02T04:17:51.762 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:17:51.763 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:17:51.763 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2022-09-02T04:17:51.763 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:17:51.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:51.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:51.765 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-09-02T04:17:51.766 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:17:51.766 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:51.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:51.768 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:51.768 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_config": { 2022-09-02T04:17:51.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event_retention_days": 7, 2022-09-02T04:17:51.770 INFO:teuthology.orchestra.run.smithi160.stdout: "start_with_full_sync": false 2022-09-02T04:17:51.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:51.771 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "", 2022-09-02T04:17:51.771 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-09-02T04:17:51.772 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:51.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:51.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:17:51.776 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:17:51.826 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:17:51.827 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:17:51.840 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:17:51.833+0000 7f7b0b6ac600 0 period (e248e3d2-c4b4-4705-8303-7ccdb6560fe1 does not have zone fc4dbd55-e667-4fb2-9ede-75f3100f9d19 configured 2022-09-02T04:17:52.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-02T04:17:58.676 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:17:58.706 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:17:58.706 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:58.707 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4, 2022-09-02T04:17:58.707 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "3488ede9-0214-4989-ab22-77a221ac7b2c", 2022-09-02T04:17:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [], 2022-09-02T04:17:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:17:58.709 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:17:58.709 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:17:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:17:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:17:58.711 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:17:58.711 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:17:58.712 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:58.712 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:17:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:17:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:17:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.715 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:17:58.715 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:17:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:17:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:17:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:17:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:17:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:17:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:17:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:17:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:17:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:17:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:17:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:17:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:17:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:17:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:17:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:17:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:17:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:17:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:17:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:17:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:17:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:17:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:17:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:17:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:17:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:17:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:17:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:17:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:17:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:17:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:17:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:17:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:17:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:17:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:17:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:17:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:17:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:17:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:17:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:17:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:17:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:17:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:17:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:17:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:17:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:17:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:17:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:17:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:17:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:17:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:17:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:17:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:17:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:17:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:17:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:17:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:17:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:17:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:17:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:17:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:17:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:17:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:17:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:17:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:17:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:17:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:17:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:17:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:17:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:17:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:17:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:17:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:17:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:17:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:17:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:17:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:17:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:17:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:17:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:17:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:17:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:17:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:17:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:17:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:17:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:17:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:17:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:17:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:17:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:17:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:17:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:17:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:17:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:17:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:17:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:17:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:17:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:17:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:17:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:17:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:17:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:17:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:17:58.752 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:17:58.752 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2022-09-02T04:17:58.753 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T04:17:58.754 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2022-09-02T04:17:58.752+0000 7f5db0af3700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 109809) UID: 0 2022-09-02T04:17:58.754 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2022-09-02T04:17:58.752+0000 7f5db4d5e600 -1 shutting down 2022-09-02T04:18:04.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2022-09-02T04:18:04.856 INFO:tasks.rgw.c2.client.1:Stopped 2022-09-02T04:18:04.857 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-09-02T04:18:04.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-09-02T04:18:04.888 INFO:tasks.rgw.c2.client.1:Started 2022-09-02T04:18:04.902 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2022-09-02T04:18:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:18:04.947 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:18:04.947 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:18:04.947 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001: Connection refused 2022-09-02T04:18:05.948 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2022-09-02T04:18:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:18:05.983 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:18:05.983 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:18:05.984 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001: Connection refused 2022-09-02T04:18:07.986 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2022-09-02T04:18:08.017 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:18:08.018 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:18:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:18:08.021 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8001: Connection refused 2022-09-02T04:18:10.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2022-09-02T04:18:12.023 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8001 2022-09-02T04:18:12.053 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:18:12.053 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:18:12.054 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2022-09-02T04:18:12.056 INFO:teuthology.orchestra.run.smithi160.stdout:{"Code":"MethodNotAllowed","RequestId":"tx0000067007b8b055d8913-0063118404-10ed-test-zone4","HostId":"10ed-test-zone4-test-zonegroup"} 2022-09-02T04:18:12.057 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2022-09-02T04:18:12.057 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2022-09-02T04:18:12.099 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa/../src/test/rgw/rgw_multi 2022-09-02T04:18:12.164 INFO:tasks.rgw_multisite_tests:creating test user.. 2022-09-02T04:18:12.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:18:12.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:18:12.167 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:18:12.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:12.232 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:12.284 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:18:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: "user_id": "rgw-multisite-test-user", 2022-09-02T04:18:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: "display_name": "Multisite Test User", 2022-09-02T04:18:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "email": "", 2022-09-02T04:18:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "suspended": 0, 2022-09-02T04:18:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "max_buckets": 1000, 2022-09-02T04:18:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "subusers": [], 2022-09-02T04:18:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "keys": [ 2022-09-02T04:18:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:18:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "rgw-multisite-test-user", 2022-09-02T04:18:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "K2P3ZGZ9QHARN05S5UWB", 2022-09-02T04:18:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "88UX3ZsKeHBVcUdSYndmZV6C1jdz2atv19EUkp7X" 2022-09-02T04:18:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:18:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:18:12.293 INFO:teuthology.orchestra.run.smithi099.stdout: "swift_keys": [], 2022-09-02T04:18:12.294 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": [ 2022-09-02T04:18:12.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:18:12.294 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "roles", 2022-09-02T04:18:12.295 INFO:teuthology.orchestra.run.smithi099.stdout: "perm": "*" 2022-09-02T04:18:12.295 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:18:12.296 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:18:12.296 INFO:teuthology.orchestra.run.smithi099.stdout: "op_mask": "read, write, delete", 2022-09-02T04:18:12.297 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2022-09-02T04:18:12.297 INFO:teuthology.orchestra.run.smithi099.stdout: "default_storage_class": "", 2022-09-02T04:18:12.298 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_tags": [], 2022-09-02T04:18:12.298 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:18:12.298 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:18:12.299 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:18:12.299 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:18:12.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:18:12.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:18:12.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:18:12.301 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:18:12.301 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:18:12.302 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:18:12.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:18:12.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:18:12.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:18:12.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:18:12.305 INFO:teuthology.orchestra.run.smithi099.stdout: "temp_url_keys": [], 2022-09-02T04:18:12.306 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "rgw", 2022-09-02T04:18:12.306 INFO:teuthology.orchestra.run.smithi099.stdout: "mfa_ids": [] 2022-09-02T04:18:12.306 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:18:12.307 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T04:18:12.309 INFO:rgw_multi.tests:meta checkpoint 2022-09-02T04:18:12.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:12.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:12.313 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:12.394 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:18:12.408 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:18:12.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:18:12.409 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:18:12.410 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:18:12.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:18:12.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:18:12.412 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:18:12.412 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:18:12.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:18:12.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:18:12.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:18:12.415 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:18:12.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:18:12.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:18:12.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:18:12.417 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:18:12.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:18:12.430 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:18:12.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:18:12.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:12.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:12.435 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:12.491 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:12.492 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:12.523 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:12.519+0000 7f59c2cdd600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:12.523 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:12.521+0000 7f59c2cdd600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:12.538 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:12.538 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:12.538 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:12.539 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:12.540 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:12.540 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:12.540 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:12.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:12.541 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:12.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:12.541 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:12.542 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:12.542 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:12.542 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:12.542 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:12.542 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:12.543 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:12.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:12.543 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:12.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:12.558 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:12.559 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:12.560 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:12.560 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:12.561 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:12.561 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:12.562 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:12.562 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:12.563 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:12.565 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:12.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:12.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:12.575 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:12.576 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:12.576 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:12.576 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:12.577 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:12.577 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:12.577 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:12.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:12.577 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:12.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:12.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:12.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:12.578 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:12.579 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:12.579 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:12.579 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:12.579 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:12.580 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:12.580 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:12.580 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:12.580 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:12.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:12.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:12.581 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:12.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:12.581 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:12.582 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:12.582 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:12.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:12.582 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:12Z" 2022-09-02T04:18:12.583 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:12.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:17.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:17.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:17.591 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:17.651 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:17.652 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:17.696 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:17.695+0000 7fecc3a37600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:17.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:17.696+0000 7fecc3a37600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:17.739 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:17Z" 2022-09-02T04:18:17.769 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:17.771 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:22.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:22.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:22.775 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:22.838 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:22.839 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:22.882 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:22.879+0000 7f3e8597b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:22.882 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:22.880+0000 7f3e8597b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:22.898 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:22.898 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:22.899 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:22.899 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:22.900 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:22.900 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:22.900 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:22.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:22.901 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:22.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:22.901 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:22.902 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:22.902 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:22.902 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:22.902 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:22.903 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:22.903 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:22.903 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:22.903 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:22.903 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:22.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:22.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:22.907 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:22.907 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:22.908 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:22.908 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:22.909 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:22.909 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:22.909 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:22.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:22.911 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:22.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:22.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:22.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:22.911 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:22.912 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:22.912 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:22.912 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:22.913 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:22.913 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:22.914 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:22.914 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:22.914 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:22.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:22.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:22.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:22.916 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:22.916 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:22.917 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:22.917 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:22.917 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:22.917 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:22.918 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:22.918 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:22.918 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:22.918 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:22.920 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:22.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:22.920 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:22Z" 2022-09-02T04:18:22.921 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:22.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:27.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:27.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:27.930 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:27.998 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:27.998 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:28.027 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:28.023+0000 7fa21a227600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:28.028 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:28.024+0000 7fa21a227600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:28.051 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:28.052 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:28.052 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:28.053 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:28.053 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:28.063 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:28.063 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:28.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:28.064 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:28.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:28.065 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:28.071 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:28.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:28.077 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:28.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:28.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:28.078 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:28.081 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:28.085 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:28.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:28.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:28.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:28.093 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:28.094 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:28.097 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:28.102 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:28.103 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:28.103 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:28.103 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:28.104 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:28.104 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:28.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:28.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:28.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:28.119 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:28.120 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:28.120 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:28.121 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:28.121 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:28.121 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:28.132 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:28.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:28.134 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:28.135 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:28.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:28.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:28.144 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:28.145 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:28.146 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:28.147 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:28.148 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:28.149 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:28.149 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:28.150 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:28.150 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:28.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:28.151 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:28.151 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:28.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:28.151 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:28.151 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:28.152 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:28.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:28.153 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:28Z" 2022-09-02T04:18:28.153 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:28.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:33.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:33.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:33.157 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:33.221 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:33.222 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:33.260 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:33.259+0000 7f67be1d7600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:33.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:33.260+0000 7f67be1d7600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:33.285 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:33.285 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:33.285 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:33.286 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:33.288 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:33.290 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:33.292 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:33.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:33.293 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:33.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:33.293 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:33.294 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:33.294 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:33.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:33.295 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:33.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:33.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:33.297 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:33.298 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:33.298 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:33.311 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:33.311 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:33.312 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:33.312 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:33.312 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:33.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:33.313 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:33.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:33.314 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:33.314 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:33.314 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:33.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:33.315 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:33Z" 2022-09-02T04:18:33.316 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:33.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:38.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:38.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:38.321 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:38.384 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:38.387 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:38.438 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:38.436+0000 7feedd87d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:38.440 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:38.438+0000 7feedd87d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:38.453 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:38.453 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:38.454 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:38.454 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:38.454 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:38.455 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:38.455 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:38.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:38.455 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:38.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:38.456 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:38.456 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:38.456 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:38.456 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:38.457 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:38.459 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:38.461 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:38.461 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:38.462 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:38.462 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:38.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:38.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:38.463 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:38.463 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:38.463 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:38.464 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:38.464 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:38.464 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:38.464 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:38.468 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:38.469 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:38.470 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:38.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:38.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:38.471 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:38.471 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:38.472 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:38.472 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:38.473 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:38.488 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:38.488 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:38.488 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:38.489 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:38.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:38.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:38.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:38.492 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:38.492 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:38.493 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:38.493 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:38.493 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:38.493 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:38.494 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:38.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:38.494 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:38.495 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:38.495 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:38.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:38.496 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:38Z" 2022-09-02T04:18:38.496 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:38.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:43.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:43.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:43.504 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:43.566 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:43.567 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:43.605 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:43.604+0000 7f206ab7c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:43.607 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:43.605+0000 7f206ab7c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:43.626 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:43.627 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:43.629 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:43.629 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:43.629 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:43.630 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:43.635 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:43.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:43.636 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:43.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:43.637 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:43.637 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:43.637 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:43.638 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:43.642 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:43.642 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:43.642 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:43.642 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:43.643 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:43.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:43.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:43.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:43.644 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:43.644 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:43.644 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:43.644 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:43.645 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:43.645 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:43.645 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:43.645 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:43.645 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:43.648 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:43.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:43.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:43.658 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:43.658 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:43.658 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:43.659 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:43.659 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:43.659 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:43.659 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:43.660 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:43.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:43.669 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:43.676 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:43Z" 2022-09-02T04:18:43.677 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:43.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:48.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:48.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:48.684 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:48.751 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:48.752 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:48.800 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:48.786+0000 7f56e7ef4600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:48.787+0000 7f56e7ef4600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:48.806 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:48.807 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:48.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:48.809 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:48.811 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:48.813 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:48.814 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:48.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:48.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:48.816 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:48.817 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:48.818 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:48.819 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:48.820 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:48.821 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:48.827 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:48.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:48.829 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:48Z" 2022-09-02T04:18:48.829 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:48.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:53.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:53.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:53.840 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:53.900 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:53.900 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:53.928 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:53.925+0000 7f8db20a2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:53.929 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:53.926+0000 7f8db20a2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:53.951 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:53.952 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:53.953 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:53.954 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:53.955 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:53.956 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:53.957 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:53.961 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:53.961 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:53.962 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:53.962 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:53.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:53.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:53.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:53.963 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:53.964 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:53.964 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:53.964 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:53.965 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:53.965 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:53.965 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:53.966 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:53.966 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:53.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:53.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:53.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:53.968 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:53.968 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:53.969 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:53.969 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:53.969 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:53.970 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:53.971 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:53.978 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:53.979 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:53.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:53.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:53.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:53.981 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:53.981 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:53.982 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:53.989 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:53.990 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:53.991 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:53.991 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:53.992 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:53.992 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:53.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:53.996 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:53.996 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:53Z" 2022-09-02T04:18:53.999 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:54.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:18:59.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:59.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:18:59.003 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:18:59.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:18:59.061 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:18:59.125 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:59.115+0000 7f7ba18ea600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:59.125 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:18:59.116+0000 7f7ba18ea600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:18:59.140 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:18:59.141 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:18:59.150 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:18:59.150 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:18:59.151 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:18:59.151 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:18:59.151 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:59.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:59.152 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:18:59.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:59.153 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:18:59.153 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:59.153 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:59.154 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:59.154 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:59.154 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:18:59.155 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:59.156 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:59.156 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:59.156 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:59.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:59.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:59.157 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:18:59.157 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:59.157 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:59.158 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:59.158 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:59.158 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:59.158 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:59.159 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:59.159 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:59.159 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:59.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:59.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:59.161 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:18:59.164 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:59.165 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:59.165 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:59.165 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:59.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:59.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:59.166 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:59.166 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:59.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:59.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:59.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:18:59.167 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:18:59.167 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:18:59.168 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:18:59.168 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:18:59.168 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:18:59.168 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:18:59.169 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:18:59.169 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:18:59.169 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:18:59.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:59.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:18:59.180 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:18:59.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:59.181 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:18:59.181 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:18:59.181 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:18:59.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:18:59.182 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:18:59Z" 2022-09-02T04:18:59.182 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:18:59.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:04.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:04.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:04.190 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:04.254 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:04.255 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:04.279 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:04.276+0000 7f029118b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:04.280 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:04.278+0000 7f029118b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:04.294 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:04.294 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:04.295 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:04.295 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:04.295 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:04.295 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:04.296 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:04.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:04.296 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:04.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:04.297 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:04.297 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:04.297 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:04.297 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:04.297 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:04.298 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:04.298 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:04.298 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:04.298 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:04.299 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:04.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:04.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:04.299 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:04.300 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:04.300 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:04.300 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:04.300 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:04.300 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:04.301 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:04.301 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:04.301 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:04.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:04.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:04.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:04.303 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:04.303 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:04.303 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:04.303 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:04.303 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:04.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:04.304 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:04.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:04.304 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:04.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:04.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:04.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:04.305 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:04.305 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:04.306 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:04.306 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:04.307 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:04.307 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:04.307 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:04.307 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:04.308 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:04.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:04.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:04.308 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:04.310 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:04.310 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:04.311 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:04.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:04.311 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:04Z" 2022-09-02T04:19:04.311 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:04.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:09.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:09.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:09.316 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:09.385 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:09.386 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:09.417 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:09.414+0000 7fe5c7c89600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:09.418 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:09.416+0000 7fe5c7c89600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:09.440 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:09.441 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:09.441 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:09.442 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:09.442 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:09.443 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:09.443 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:09.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:09.446 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:09.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:09.446 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:09.447 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:09.447 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:09.447 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:09.447 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:09.448 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:09.448 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:09.448 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:09.448 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:09.453 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:09.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:09.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:09.454 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:09.455 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:09.455 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:09.455 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:09.455 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:09.456 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:09.456 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:09.456 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:09.456 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:09.457 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:09.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:09.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:09.457 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:09.458 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:09.458 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:09.458 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:09.459 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:09.459 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:09.459 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:09.459 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:09.460 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:09.460 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:09.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:09.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:09.461 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:09.461 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:09.461 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:09.461 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:09.461 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:09.462 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:09.462 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:09.463 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:09.463 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:09.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:09.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:09.464 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:09.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:09.466 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:09.466 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:09.467 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:09.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:09.467 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:09Z" 2022-09-02T04:19:09.467 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:09.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:14.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:14.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:14.475 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:14.554 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:14.555 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:14.591 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:14.589+0000 7fe404b9c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:14.592 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:14.590+0000 7fe404b9c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:14.608 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:14Z" 2022-09-02T04:19:14.632 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:14.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:19.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:19.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:19.637 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:19.694 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:19.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:19.732 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:19.726+0000 7fc620180600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:19.734 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:19.727+0000 7fc620180600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:19.752 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:19.753 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:19.753 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:19.754 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:19.754 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:19.755 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:19.755 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:19.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:19.756 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:19.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:19.766 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:19.766 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:19.766 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:19.767 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:19.772 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:19.772 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:19.773 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:19.773 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:19.774 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:19.774 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:19.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:19.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:19.776 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:19.776 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:19.777 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:19.777 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:19.778 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:19.779 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:19.779 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:19.780 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:19.780 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:19.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:19.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:19.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:19.782 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:19.783 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:19.783 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:19.784 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:19.784 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:19.785 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:19.785 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:19.786 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:19.786 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:19.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:19.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:19.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:19.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:19.789 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:19.789 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:19.790 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:19.790 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:19.791 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:19.791 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:19.792 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:19.792 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:19.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:19.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:19.793 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:19.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:19.794 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:19.795 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:19.795 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:19.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:19.796 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:19Z" 2022-09-02T04:19:19.797 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:19.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:24.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:24.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:24.813 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:24.874 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:24.874 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:24.926 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:24.921+0000 7f02a5e28600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:24.926 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:24.922+0000 7f02a5e28600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:24.949 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:24.950 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:24.951 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:24.952 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:24.953 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:24.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:24.955 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:24.956 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:24.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:24.958 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:24.959 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:24.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:24.961 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:24.968 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:24.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:24.970 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:24.971 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:24.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:24.972 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:24Z" 2022-09-02T04:19:24.972 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:24.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:29.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:29.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:29.977 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:30.036 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:30.037 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:30.075 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:30.075+0000 7f63820c9600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:30.076 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:30.076+0000 7f63820c9600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:30.091 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:30.091 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:30.092 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:30.092 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:30.093 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:30.093 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:30.094 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:30.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:30.095 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:30.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:30.099 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:30.100 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:30.100 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:30.101 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:30.101 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:30.102 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:30.102 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:30.103 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:30.103 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:30.104 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:30.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:30.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:30.107 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:30.107 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:30.108 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:30.108 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:30.110 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:30.111 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:30.111 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:30.111 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:30.112 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:30.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:30.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:30.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:30.117 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:30.118 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:30.118 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:30.119 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:30.119 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:30.119 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:30.119 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:30.120 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:30.120 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:30.120 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:30.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:30.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:30.122 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:30.123 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:30.123 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:30.124 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:30.124 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:30.125 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:30.126 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:30.126 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:30.127 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:30.127 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:30.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:30.128 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:30.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:30.129 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:30.129 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:30.130 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:30.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:30.131 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:30Z" 2022-09-02T04:19:30.131 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:30.132 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:35.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:35.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:35.135 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:35.193 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:35.194 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:35.231 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:35.229+0000 7f3230390600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:35.240 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:35.230+0000 7f3230390600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:35.254 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:35.255 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:35.255 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:35.255 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:35.255 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:35.256 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:35.256 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:35.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:35.256 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:35.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:35.257 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:35.257 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:35.257 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:35.257 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:35.266 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:35.266 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:35.266 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:35.267 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:35.267 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:35.267 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:35.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:35.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:35.268 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:35.269 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:35.269 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:35.270 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:35.270 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:35.270 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:35.271 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:35.271 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:35.272 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:35.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:35.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:35.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:35.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:35.275 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:35.276 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:35.276 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:35.277 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:35.277 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:35.279 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:35.280 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:35.280 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:35.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:35.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:35.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:35.282 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:35.283 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:35.283 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:35.292 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:35.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:35.294 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:35.294 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:35.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:35.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:35.296 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:35.297 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:35.298 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:35.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:35.299 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:35Z" 2022-09-02T04:19:35.300 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:35.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:40.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:40.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:40.305 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:40.364 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:40.365 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:40.408 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:40.405+0000 7fab20080600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:40.409 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:40.407+0000 7fab20080600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:40.428 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:40.428 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:40.429 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:40.429 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:40.430 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:40.430 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:40.430 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:40.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:40.431 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:40.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:40.432 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:40.433 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:40.433 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:40.433 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:40.434 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:40.434 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:40.434 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:40.435 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:40.435 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:40.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:40.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:40.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:40.437 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:40.437 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:40.438 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:40.438 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:40.439 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:40.439 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:40.440 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:40.440 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:40.441 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:40.441 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:40.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:40.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:40.442 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:40.443 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:40.443 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:40.444 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:40.444 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:40.444 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:40.445 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:40.445 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:40.446 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:40.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:40.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:40.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:40.447 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:40.447 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:40.447 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:40.448 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:40.448 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:40.448 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:40.449 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:40.449 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:40.449 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:40.449 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:40.449 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:40.450 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:40.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:40.450 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:40.450 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:40.451 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:40.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:40.451 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:40Z" 2022-09-02T04:19:40.451 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:40.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:45.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:45.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:45.455 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:45.517 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:45.518 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:45.553 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:45.551+0000 7fae4e71e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:45.554 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:45.553+0000 7fae4e71e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:45.571 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:45.571 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:45.572 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:45.572 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:45.572 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:45.573 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:45.573 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:45.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:45.574 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:45.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:45.575 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:45.576 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:45.576 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:45.576 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:45.577 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:45.577 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:45.577 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:45.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:45.578 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:45.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:45.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:45.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:45.579 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:45.579 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:45.580 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:45.580 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:45.581 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:45.581 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:45.581 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:45.582 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:45.582 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:45.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:45.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:45.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:45.584 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:45.585 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:45.585 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:45.586 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:45.586 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:45.587 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:45.587 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:45.587 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:45.588 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:45.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:45.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:45.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:45.590 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:45.591 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:45.591 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:45.595 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:45.596 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:45.596 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:45.597 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:45.597 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:45.597 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:45.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:45.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:45.599 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:45.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:45.599 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:45.600 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:45.600 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:45.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:45.601 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:45Z" 2022-09-02T04:19:45.601 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:45.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:50.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:50.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:50.612 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:50.672 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:50.672 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:50.713 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:50.709+0000 7ff674b1a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:50.713 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:50.710+0000 7ff674b1a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:50.736 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:50.736 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:50.737 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:50.737 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:50.741 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:50.742 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:50.742 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:50.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:50.743 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:50.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:50.744 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:50.744 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:50.744 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:50.744 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:50.745 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:50.745 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:50.745 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:50.745 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:50.745 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:50.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:50.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:50.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:50.747 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:50.747 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:50.748 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:50.748 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:50.748 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:50.748 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:50.749 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:50.749 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:50.749 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:50.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:50.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:50.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:50.750 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:50.750 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:50.751 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:50.751 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:50.751 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:50.751 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:50.752 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:50.752 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:50.752 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:50.752 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:50.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:50.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:50.753 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:50.753 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:50.753 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:50.755 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:50.755 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:50.755 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:50.755 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:50.756 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:50.756 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:50.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:50.757 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:50.758 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:50.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:50.758 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:50.763 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:50.764 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:50.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:50.765 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:50Z" 2022-09-02T04:19:50.766 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:50.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:19:55.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:55.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:19:55.777 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:19:55.840 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:19:55.840 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:19:55.882 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:55.879+0000 7f4537b02600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:55.883 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:19:55.880+0000 7f4537b02600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:19:55.924 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:19:55.925 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:19:55.925 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:19:55.925 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:19:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:19:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:19:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:19:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:19:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:19:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:55.930 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:19:55.931 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:55.931 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:55.931 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:55.931 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:55.932 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:55.932 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:55.932 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:55.932 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:19:55.934 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:55.934 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:55.935 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:55.935 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:55.935 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:55.935 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:55.936 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:55.936 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:55.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:55.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:19:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:19:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:19:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:19:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:19:55.938 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:19:55.938 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:19:55.938 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:19:55.938 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:19:55.939 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:19:55.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:55.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:19:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:19:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:19:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:19:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:19:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:19:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:19:55Z" 2022-09-02T04:19:55.941 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:19:55.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:20:00.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:00.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:00.949 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:01.019 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:01.019 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:01.053 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:01.046+0000 7f5a72438600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:01.054 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:01.047+0000 7f5a72438600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:01.075 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:01.076 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:01.076 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:01.076 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:01.077 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:01.077 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:01.084 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:01.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:01.084 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:01.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:01.085 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:01.086 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:01.086 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:01.086 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:01.087 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:01.087 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:01.087 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:01.089 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:01.089 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:01.091 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:01.091 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:01.091 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:01.091 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:01.092 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:01.092 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:01.092 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:01.092 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:01.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:01.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:01.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:01.093 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:01.094 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:01.094 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:01.094 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:01.094 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:01.095 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:01.095 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:01.095 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:01.095 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:01.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:01.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:01.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:01.096 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:01.097 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:01.097 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:01.097 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:01.097 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:01.097 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:01.098 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:01.098 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:01.100 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:01.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:01.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:01.102 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:01.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:01.103 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:01.103 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:01.104 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:01.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:01Z" 2022-09-02T04:20:01.105 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:01.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1662092292.280197_71.1 target= 2022-09-02T04:20:06.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.113 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:06.175 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:06.177 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:06.227 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:06.224+0000 7f988765a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.227 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:06.225+0000 7f988765a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.247 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:06.248 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:06.248 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:06.249 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:06.249 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:06.249 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:06.250 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.250 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:06.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.251 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:06.251 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.251 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.252 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:06.253 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.254 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:06.254 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.254 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:06.255 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.256 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:06.257 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.258 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.258 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.260 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.261 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.261 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.261 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.263 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.263 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.265 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:06.266 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.266 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.267 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.281 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.283 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.284 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:06.286 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.287 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:06Z" 2022-09-02T04:20:06.287 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:06.289 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:06.290 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:06.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.291 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:06.399+0000 7f401f5b6600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.402 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:06.401+0000 7f401f5b6600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.425 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:06.427 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:06.427 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:06.428 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:06.428 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:06.429 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:06.429 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.430 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:06.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:06.432 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.432 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.433 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:06.434 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.434 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:06.434 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.435 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:06.435 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:06.447 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.447 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.448 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:06.450 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.451 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:06.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.453 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.453 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:06.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.455 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:06.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:06.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.459 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:06Z" 2022-09-02T04:20:06.459 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:06.462 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:06.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:06.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.464 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:06.520 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:06.521 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:06.548 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:06.546+0000 7ff85620e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.549 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:06.547+0000 7ff85620e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.565 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:06.566 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:06.566 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:06.571 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:06.573 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:06.573 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:06.574 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.574 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:06.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.575 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:06.575 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.575 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.576 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:06.576 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.576 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:06.590 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.590 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:06.590 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.591 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:06.592 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.592 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.592 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.594 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.594 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.594 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.595 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.595 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.596 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:06.596 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.596 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.597 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.597 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.597 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.597 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.602 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.604 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:06.604 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.604 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.605 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.605 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.618 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.619 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.620 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.620 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.622 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:06.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.623 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:06.624 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:06.627 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:06.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.628 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:06Z" 2022-09-02T04:20:06.629 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:06.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:06.629 INFO:rgw_multi.tests:meta checkpoint 2022-09-02T04:20:06.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:06.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:06.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:06.732 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:06.748 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:06.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:06.756 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:06.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.756 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.757 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:06.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.757 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.758 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:06.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.758 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.759 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:06.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.759 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:06.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:06.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.765 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:06.815 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:06.816 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:06.853 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:06.845+0000 7f35a5a11600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.854 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:06.846+0000 7f35a5a11600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.860 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:06.861 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:06.862 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:06.862 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:06.862 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:06.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.864 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:06.866 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.868 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.869 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.871 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:06.876 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.877 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.877 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.878 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.879 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.879 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.880 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.880 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:06.881 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:06.882 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:06.882 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:06.882 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:06.882 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:06.883 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:06.883 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:06.883 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.883 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:06.883 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:06.884 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:06.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.884 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:06.885 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:06.885 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:06.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:06.885 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:06Z" 2022-09-02T04:20:06.886 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:06.889 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:06.889 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:06.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.890 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:06.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:06.968 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:06.966+0000 7fd9538fb600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.969 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:06.966+0000 7fd9538fb600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:06.980 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.983 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:06.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.983 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:06.984 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.984 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.984 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:06.984 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.985 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:06.985 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.985 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:06.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.987 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:06.987 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.987 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.988 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.988 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:06.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:06.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.990 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:06.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.991 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:06.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:06.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:06.994 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:06.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:06.994 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:06.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:06.995 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:06.995 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:06.995 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:06.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:06.996 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:06.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.996 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:06.996 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:06.997 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:06.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:06.997 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:06Z" 2022-09-02T04:20:06.997 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:06.998 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:06.998 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:06.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:06.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:07.000 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:07.050 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:07.051 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:07.076 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:07.075+0000 7f24ac581600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:07.077 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:07.076+0000 7f24ac581600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:07.088 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:07.089 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:07.089 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:07.090 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:07.090 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:07.091 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:07.091 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:07.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:07.093 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:07.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:07.094 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:07.100 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:07.101 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:07.102 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:07.102 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:07.102 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:07.103 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:07.103 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:07.103 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:07.103 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:07.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:07.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:07.105 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:07.105 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:07.106 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:07.106 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:07.106 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:07.106 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:07.107 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:07.107 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:07.107 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:07.107 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:07.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:07.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:07.108 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:07.108 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:07.108 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:07.109 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:07.109 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:07.109 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:07.109 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:07.109 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:07.110 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:07.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:07.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:07.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:07.111 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:07.112 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:07.113 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:07.114 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:07.115 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:07.116 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:07Z" 2022-09-02T04:20:07.116 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:07.117 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:07.118 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_cbodley_ceph_98002085f9003280d82d2f21aa1472b865694c0b/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2022-09-02T04:20:07.206 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.209 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.216 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.218 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.219 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.221 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.222 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.225 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.226 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.227 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.227 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.228 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.229 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.232 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.233 DEBUG:boto:Using access key provided by client. 2022-09-02T04:20:07.234 DEBUG:boto:Using secret key provided by client. 2022-09-02T04:20:07.235 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-1 2022-09-02T04:20:07.241 DEBUG:boto:path=/xljbkf-1/ 2022-09-02T04:20:07.242 DEBUG:boto:auth_path=/xljbkf-1/ 2022-09-02T04:20:07.243 DEBUG:boto:Method: PUT 2022-09-02T04:20:07.243 DEBUG:boto:Path: /xljbkf-1/ 2022-09-02T04:20:07.244 DEBUG:boto:Data: 2022-09-02T04:20:07.244 DEBUG:boto:Headers: {} 2022-09-02T04:20:07.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:07.246 DEBUG:boto:Port: 8000 2022-09-02T04:20:07.246 DEBUG:boto:Params: {} 2022-09-02T04:20:07.247 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:20:07.248 DEBUG:boto:Token: None 2022-09-02T04:20:07.249 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:07 GMT /xljbkf-1/ 2022-09-02T04:20:07.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/H8ygaCYo9gvedXrVhD1PntVy+E= 2022-09-02T04:20:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/H8ygaCYo9gvedXrVhD1PntVy+E=', 'Content-Length': '0'} 2022-09-02T04:20:09.588 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054a019aae36b49d4-0063118477-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:09.589 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-2 2022-09-02T04:20:09.590 DEBUG:boto:path=/xljbkf-2/ 2022-09-02T04:20:09.590 DEBUG:boto:auth_path=/xljbkf-2/ 2022-09-02T04:20:09.591 DEBUG:boto:Method: PUT 2022-09-02T04:20:09.592 DEBUG:boto:Path: /xljbkf-2/ 2022-09-02T04:20:09.593 DEBUG:boto:Data: 2022-09-02T04:20:09.594 DEBUG:boto:Headers: {} 2022-09-02T04:20:09.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:09.596 DEBUG:boto:Port: 8000 2022-09-02T04:20:09.597 DEBUG:boto:Params: {} 2022-09-02T04:20:09.598 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:20:09.599 DEBUG:boto:Token: None 2022-09-02T04:20:09.600 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:09 GMT /xljbkf-2/ 2022-09-02T04:20:09.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VuqQm9MLuJpLhBawU0z/QVqOSIA= 2022-09-02T04:20:09.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VuqQm9MLuJpLhBawU0z/QVqOSIA=', 'Content-Length': '0'} 2022-09-02T04:20:12.590 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000071307e3c889fb2a-0063118479-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:12.591 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-3 2022-09-02T04:20:12.592 DEBUG:boto:path=/xljbkf-3/ 2022-09-02T04:20:12.592 DEBUG:boto:auth_path=/xljbkf-3/ 2022-09-02T04:20:12.599 DEBUG:boto:Method: PUT 2022-09-02T04:20:12.600 DEBUG:boto:Path: /xljbkf-3/ 2022-09-02T04:20:12.601 DEBUG:boto:Data: 2022-09-02T04:20:12.602 DEBUG:boto:Headers: {} 2022-09-02T04:20:12.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:12.603 DEBUG:boto:Port: 8001 2022-09-02T04:20:12.603 DEBUG:boto:Params: {} 2022-09-02T04:20:12.604 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:20:12.604 DEBUG:boto:Token: None 2022-09-02T04:20:12.608 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:12 GMT /xljbkf-3/ 2022-09-02T04:20:12.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OlAIZ+NajuipHopHcthqKIvDlPE= 2022-09-02T04:20:12.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OlAIZ+NajuipHopHcthqKIvDlPE=', 'Content-Length': '0'} 2022-09-02T04:20:12.651 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0c011bea09d032d-006311847c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:12.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:12.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:12.656 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:12.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:12.722 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:12.779 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:12.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:12.781 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:12.781 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:12.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:12.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:12.784 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:12.784 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:12.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:12.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:12.786 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:12.786 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:12.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:12.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:12.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092412.636577_104.1", 2022-09-02T04:20:12.789 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:12.636577Z" 2022-09-02T04:20:12.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:12.789 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:12.791 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:12.791 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:12.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:12.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:12.794 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:12.857 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:12.858 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:12.891 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:12.889+0000 7f95b5728600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:12.892 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:12.890+0000 7f95b5728600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:12.903 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:12.903 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:12.904 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:12.904 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:12.905 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:12.905 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:12.905 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:12.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:12.906 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:12.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:12.906 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:12.906 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:12.907 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:12.907 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:12.907 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:12.907 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:12.908 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:12.909 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:12.909 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:12.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:12.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:12.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:12.912 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:12.912 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:12.913 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:12.913 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:12.914 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:12.915 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:12.915 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:12.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:12.916 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:12.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:12.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:12.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:12.918 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:12.918 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:12.919 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:12.919 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:12.920 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:12.920 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:12.920 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:12.921 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:12.921 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:12.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:12.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:12.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:12.922 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:12.922 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:12.923 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:12.923 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092409.573151_93.1", 2022-09-02T04:20:12.923 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:12.923 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:12.924 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:12.924 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.573151Z", 2022-09-02T04:20:12.924 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:12.924 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:12.925 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:12.925 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:12.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:12.925 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:12.926 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:12.927 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:12.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:12.927 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:12Z" 2022-09-02T04:20:12.929 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:12.930 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092409.573151_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:09.573151Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:12Z" } 2022-09-02T04:20:12.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092409.573151_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:09.573151Z', 'realm_epoch': 2}}] 2022-09-02T04:20:12.932 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:12.932 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092409.573151_93.1'} 2022-09-02T04:20:12.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662092412.636577_104.1 target=1_1662092409.573151_93.1 2022-09-02T04:20:17.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:17.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:17.938 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:17.997 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:17.998 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:18.041 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:18.038+0000 7f4f81da9600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.042 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:18.040+0000 7f4f81da9600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.055 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:18.055 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:18.055 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:18.056 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:18.056 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:18.057 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:18.057 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.061 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:18.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.062 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:18.063 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.064 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.065 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:18.065 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.065 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:18.066 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.066 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:18.067 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.067 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.068 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:18.069 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.071 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:18.071 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.071 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.072 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.072 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.075 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:18.075 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.076 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:18.076 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.077 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.078 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.078 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.080 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:18.082 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.082 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.082 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092412.636577_104.1", 2022-09-02T04:20:18.083 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.083 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.083 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.083 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.636577Z", 2022-09-02T04:20:18.084 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.084 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.084 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.084 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:18.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.085 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:18.085 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:18.085 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:18.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.086 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:18Z" 2022-09-02T04:20:18.087 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:18.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092412.636577_104.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:12.636577Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:18Z" } 2022-09-02T04:20:18.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092412.636577_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:12.636577Z', 'realm_epoch': 2}}] 2022-09-02T04:20:18.089 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:18.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:18.092 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:18.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:18.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.095 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:18.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:18.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:18.206 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:18.205+0000 7f0f4b70e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.207 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:18.206+0000 7f0f4b70e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.221 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092412.636577_104.1", 2022-09-02T04:20:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.636577Z", 2022-09-02T04:20:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:18Z" 2022-09-02T04:20:18.241 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:18.242 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092412.636577_104.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:12.636577Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:18Z" } 2022-09-02T04:20:18.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092412.636577_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:12.636577Z', 'realm_epoch': 2}}] 2022-09-02T04:20:18.243 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:18.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:18.244 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:18.244 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:18.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.246 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:18.303 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:18.304 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:18.324 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:18.322+0000 7f3ea277b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.325 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:18.323+0000 7f3ea277b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.338 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:18.338 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:18.339 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:18.339 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:18.340 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:18.340 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:18.341 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.342 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:18.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.343 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:18.344 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.345 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.345 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:18.346 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.347 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:18.347 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.348 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:18.348 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.350 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:18.350 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.351 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.352 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:18.352 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.353 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.353 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.355 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.355 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.356 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:18.358 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.358 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.359 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:18.359 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.359 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.359 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.360 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.360 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.361 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:18.361 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.362 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.363 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092412.636577_104.1", 2022-09-02T04:20:18.363 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.364 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.364 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.365 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.636577Z", 2022-09-02T04:20:18.366 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.368 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:18.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.369 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:18.369 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:18.369 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:18.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.370 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:18Z" 2022-09-02T04:20:18.371 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:18.372 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092412.636577_104.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:12.636577Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:18Z" } 2022-09-02T04:20:18.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092412.636577_104.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:12.636577Z', 'realm_epoch': 2}}] 2022-09-02T04:20:18.374 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:18.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092412.636577_104.1'} 2022-09-02T04:20:18.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:18.376 DEBUG:boto:path=/xljbkf-1/ 2022-09-02T04:20:18.377 DEBUG:boto:auth_path=/xljbkf-1/ 2022-09-02T04:20:18.377 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.378 DEBUG:boto:Path: /xljbkf-1/ 2022-09-02T04:20:18.378 DEBUG:boto:Data: 2022-09-02T04:20:18.379 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:18.381 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.382 DEBUG:boto:Params: {} 2022-09-02T04:20:18.383 DEBUG:boto:Token: None 2022-09-02T04:20:18.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-1/ 2022-09-02T04:20:18.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfOTP9xn1NSwNTbrho8fdHzQciM= 2022-09-02T04:20:18.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfOTP9xn1NSwNTbrho8fdHzQciM=', 'Content-Length': '0'} 2022-09-02T04:20:18.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000346aecb25e087c29-0063118482-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.390 DEBUG:boto:path=/xljbkf-2/ 2022-09-02T04:20:18.390 DEBUG:boto:auth_path=/xljbkf-2/ 2022-09-02T04:20:18.391 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.391 DEBUG:boto:Path: /xljbkf-2/ 2022-09-02T04:20:18.392 DEBUG:boto:Data: 2022-09-02T04:20:18.392 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:18.398 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.399 DEBUG:boto:Params: {} 2022-09-02T04:20:18.400 DEBUG:boto:Token: None 2022-09-02T04:20:18.401 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-2/ 2022-09-02T04:20:18.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EXQukyRSna3Kd2OriQxAuediIH4= 2022-09-02T04:20:18.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EXQukyRSna3Kd2OriQxAuediIH4=', 'Content-Length': '0'} 2022-09-02T04:20:18.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f50b800a37dc3006-0063118482-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.412 DEBUG:boto:path=/xljbkf-3/ 2022-09-02T04:20:18.413 DEBUG:boto:auth_path=/xljbkf-3/ 2022-09-02T04:20:18.414 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.415 DEBUG:boto:Path: /xljbkf-3/ 2022-09-02T04:20:18.415 DEBUG:boto:Data: 2022-09-02T04:20:18.416 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:18.418 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.419 DEBUG:boto:Params: {} 2022-09-02T04:20:18.420 DEBUG:boto:Token: None 2022-09-02T04:20:18.421 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-3/ 2022-09-02T04:20:18.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tpUMCW7nZxC+DoNbPx8JKipuhg4= 2022-09-02T04:20:18.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tpUMCW7nZxC+DoNbPx8JKipuhg4=', 'Content-Length': '0'} 2022-09-02T04:20:18.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003efee8e826df109f-0063118482-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.439 DEBUG:boto:path=/xljbkf-1/ 2022-09-02T04:20:18.440 DEBUG:boto:auth_path=/xljbkf-1/ 2022-09-02T04:20:18.441 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.441 DEBUG:boto:Path: /xljbkf-1/ 2022-09-02T04:20:18.442 DEBUG:boto:Data: 2022-09-02T04:20:18.443 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:18.445 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.446 DEBUG:boto:Params: {} 2022-09-02T04:20:18.446 DEBUG:boto:Token: None 2022-09-02T04:20:18.447 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-1/ 2022-09-02T04:20:18.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfOTP9xn1NSwNTbrho8fdHzQciM= 2022-09-02T04:20:18.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfOTP9xn1NSwNTbrho8fdHzQciM=', 'Content-Length': '0'} 2022-09-02T04:20:18.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acec9c9c108bd7a8-0063118482-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.454 DEBUG:boto:path=/xljbkf-2/ 2022-09-02T04:20:18.455 DEBUG:boto:auth_path=/xljbkf-2/ 2022-09-02T04:20:18.456 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.457 DEBUG:boto:Path: /xljbkf-2/ 2022-09-02T04:20:18.458 DEBUG:boto:Data: 2022-09-02T04:20:18.459 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:18.461 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.462 DEBUG:boto:Params: {} 2022-09-02T04:20:18.464 DEBUG:boto:Token: None 2022-09-02T04:20:18.465 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-2/ 2022-09-02T04:20:18.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EXQukyRSna3Kd2OriQxAuediIH4= 2022-09-02T04:20:18.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EXQukyRSna3Kd2OriQxAuediIH4=', 'Content-Length': '0'} 2022-09-02T04:20:18.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025567de68d6e01b9-0063118482-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.474 DEBUG:boto:path=/xljbkf-3/ 2022-09-02T04:20:18.474 DEBUG:boto:auth_path=/xljbkf-3/ 2022-09-02T04:20:18.475 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.476 DEBUG:boto:Path: /xljbkf-3/ 2022-09-02T04:20:18.477 DEBUG:boto:Data: 2022-09-02T04:20:18.478 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:18.481 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.483 DEBUG:boto:Params: {} 2022-09-02T04:20:18.485 DEBUG:boto:Token: None 2022-09-02T04:20:18.486 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-3/ 2022-09-02T04:20:18.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tpUMCW7nZxC+DoNbPx8JKipuhg4= 2022-09-02T04:20:18.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tpUMCW7nZxC+DoNbPx8JKipuhg4=', 'Content-Length': '0'} 2022-09-02T04:20:18.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006efd21911d5aa100-0063118482-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.499 DEBUG:boto:path=/xljbkf-1/ 2022-09-02T04:20:18.500 DEBUG:boto:auth_path=/xljbkf-1/ 2022-09-02T04:20:18.501 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.503 DEBUG:boto:Path: /xljbkf-1/ 2022-09-02T04:20:18.504 DEBUG:boto:Data: 2022-09-02T04:20:18.505 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:18.507 DEBUG:boto:Port: 8001 2022-09-02T04:20:18.508 DEBUG:boto:Params: {} 2022-09-02T04:20:18.509 DEBUG:boto:Token: None 2022-09-02T04:20:18.510 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-1/ 2022-09-02T04:20:18.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfOTP9xn1NSwNTbrho8fdHzQciM= 2022-09-02T04:20:18.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfOTP9xn1NSwNTbrho8fdHzQciM=', 'Content-Length': '0'} 2022-09-02T04:20:18.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ddacea1d8fd6392-0063118482-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.518 DEBUG:boto:path=/xljbkf-2/ 2022-09-02T04:20:18.519 DEBUG:boto:auth_path=/xljbkf-2/ 2022-09-02T04:20:18.520 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.521 DEBUG:boto:Path: /xljbkf-2/ 2022-09-02T04:20:18.522 DEBUG:boto:Data: 2022-09-02T04:20:18.522 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:18.525 DEBUG:boto:Port: 8001 2022-09-02T04:20:18.526 DEBUG:boto:Params: {} 2022-09-02T04:20:18.528 DEBUG:boto:Token: None 2022-09-02T04:20:18.529 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-2/ 2022-09-02T04:20:18.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EXQukyRSna3Kd2OriQxAuediIH4= 2022-09-02T04:20:18.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EXQukyRSna3Kd2OriQxAuediIH4=', 'Content-Length': '0'} 2022-09-02T04:20:18.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d06b86d1c515d4fa-0063118482-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.534 DEBUG:boto:path=/xljbkf-3/ 2022-09-02T04:20:18.536 DEBUG:boto:auth_path=/xljbkf-3/ 2022-09-02T04:20:18.537 DEBUG:boto:Method: HEAD 2022-09-02T04:20:18.538 DEBUG:boto:Path: /xljbkf-3/ 2022-09-02T04:20:18.538 DEBUG:boto:Data: 2022-09-02T04:20:18.539 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:18.540 DEBUG:boto:Port: 8001 2022-09-02T04:20:18.541 DEBUG:boto:Params: {} 2022-09-02T04:20:18.542 DEBUG:boto:Token: None 2022-09-02T04:20:18.542 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-3/ 2022-09-02T04:20:18.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tpUMCW7nZxC+DoNbPx8JKipuhg4= 2022-09-02T04:20:18.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tpUMCW7nZxC+DoNbPx8JKipuhg4=', 'Content-Length': '0'} 2022-09-02T04:20:18.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053906d458fab82e6-0063118482-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2022-09-02T04:20:18.551 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-4 2022-09-02T04:20:18.552 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:18.553 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:18.554 DEBUG:boto:Method: PUT 2022-09-02T04:20:18.555 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:18.556 DEBUG:boto:Data: 2022-09-02T04:20:18.557 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:18.559 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.559 DEBUG:boto:Params: {} 2022-09-02T04:20:18.560 DEBUG:boto:Token: None 2022-09-02T04:20:18.561 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-4/ 2022-09-02T04:20:18.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:34lrN/wVSWkuuXShYmTxP6gYaQM= 2022-09-02T04:20:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:34lrN/wVSWkuuXShYmTxP6gYaQM=', 'Content-Length': '0'} 2022-09-02T04:20:18.580 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f0a440a31c91e116-0063118482-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.581 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-5 2022-09-02T04:20:18.582 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:18.582 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:18.583 DEBUG:boto:Method: PUT 2022-09-02T04:20:18.584 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:18.585 DEBUG:boto:Data: 2022-09-02T04:20:18.585 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:18.587 DEBUG:boto:Port: 8000 2022-09-02T04:20:18.588 DEBUG:boto:Params: {} 2022-09-02T04:20:18.589 DEBUG:boto:Token: None 2022-09-02T04:20:18.591 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-5/ 2022-09-02T04:20:18.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4BT4AwNtB52TuIFNQdsEM1UAxac= 2022-09-02T04:20:18.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4BT4AwNtB52TuIFNQdsEM1UAxac=', 'Content-Length': '0'} 2022-09-02T04:20:18.627 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f39d30dbdb79e23-0063118482-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.629 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-6 2022-09-02T04:20:18.630 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:18.631 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:18.631 DEBUG:boto:Method: PUT 2022-09-02T04:20:18.632 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:18.632 DEBUG:boto:Data: 2022-09-02T04:20:18.633 DEBUG:boto:Headers: {} 2022-09-02T04:20:18.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:18.634 DEBUG:boto:Port: 8001 2022-09-02T04:20:18.634 DEBUG:boto:Params: {} 2022-09-02T04:20:18.635 DEBUG:boto:Token: None 2022-09-02T04:20:18.636 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:18 GMT /xljbkf-6/ 2022-09-02T04:20:18.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zbQbNWQhvM/JknnvDq42p0cf6L0= 2022-09-02T04:20:18.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zbQbNWQhvM/JknnvDq42p0cf6L0=', 'Content-Length': '0'} 2022-09-02T04:20:18.689 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0fc5c0e0c112ce6-0063118482-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:18.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.692 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:18.796 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:18.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.800 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:18.803 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:18.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.805 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:18.805 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:18.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:18.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:18.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:18.808 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:18.665018Z" 2022-09-02T04:20:18.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:18.811 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:18.814 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:18.815 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:18.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.818 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:18.876 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:18.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:18.922 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:18.918+0000 7fb760b42600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.922 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:18.919+0000 7fb760b42600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:18.938 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:18.939 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:18.939 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:18.940 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:18.940 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:18.941 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:18.941 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:18.944 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.944 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.945 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:18.946 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.946 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:18.947 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.947 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:18.948 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.948 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.950 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.952 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.952 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.954 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:18.954 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.956 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.956 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:18.956 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.956 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:18.958 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:18.959 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:18.959 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:18.960 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:18.960 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:18.961 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:18.961 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:18.961 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.665018Z", 2022-09-02T04:20:18.962 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:18.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:18.963 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:18.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.963 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:18.964 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:18.964 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:18.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:18.965 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:18Z" 2022-09-02T04:20:18.966 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:18.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092418.665018_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:18.665018Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:18Z" } 2022-09-02T04:20:18.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092418.665018_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:18.665018Z', 'realm_epoch': 2}}] 2022-09-02T04:20:18.969 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:18.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:18.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:18.970 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:18.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:18.972 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:19.022 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:19.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:19.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:19.048+0000 7f16eb563600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:19.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:19.049+0000 7f16eb563600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:19.061 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:19.070 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:19.070 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:19.071 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:19.071 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:19.072 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:19.072 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:19.073 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:19.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:19.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:19.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:19.074 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:19.075 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:19.075 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:19.076 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:19.076 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:19.076 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:19.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:19.246 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:19.246 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:19.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:19.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:19.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:19.247 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:19.248 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:19.248 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:19.248 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:19.249 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:19.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:19.250 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:19.250 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.665018Z", 2022-09-02T04:20:19.251 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:19.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:19.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:19.252 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:19.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:19.252 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:19.252 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:19.253 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:19.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:19.254 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:19Z" 2022-09-02T04:20:19.254 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:19.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092418.665018_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:18.665018Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:19Z" } 2022-09-02T04:20:19.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092418.665018_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:18.665018Z', 'realm_epoch': 2}}] 2022-09-02T04:20:19.257 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:19.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:19.263 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:19.264 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:19.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:19.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:19.266 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:19.318 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:19.319 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:19.345 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:19.343+0000 7fbd0a90d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:19.346 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:19.344+0000 7fbd0a90d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:19.365 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:19.366 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:19.366 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:19.367 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:19.367 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:19.368 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:19.368 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:19.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:19.369 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:19.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:19.371 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:19.371 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:19.372 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:19.372 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:19.373 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:19.373 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:19.374 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:19.374 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:19.375 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:19.376 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:19.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:19.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:19.377 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:19.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:19.378 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:19.379 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:19.379 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:19.379 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:19.380 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:19.380 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:19.381 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:19.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:19.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:19.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:19.383 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:19.383 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:19.384 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:19.384 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:19.385 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:19.385 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:19.386 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:19.386 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:19.387 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:19.387 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:19.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:19.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:19.389 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:19.389 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:19.390 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:19.391 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:19.391 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:19.392 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:19.392 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:19.393 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.665018Z", 2022-09-02T04:20:19.393 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:19.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:19.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:19.395 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:19.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:19.396 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:19.397 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:19.397 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:19.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:19.398 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:19Z" 2022-09-02T04:20:19.399 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:19.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092418.665018_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:18.665018Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:19Z" } 2022-09-02T04:20:19.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092418.665018_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:18.665018Z', 'realm_epoch': 2}}] 2022-09-02T04:20:19.402 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:19.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:19.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:19.404 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.405 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.406 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.407 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.408 DEBUG:boto:Data: 2022-09-02T04:20:19.409 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.411 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.412 DEBUG:boto:Params: {} 2022-09-02T04:20:19.416 DEBUG:boto:Token: None 2022-09-02T04:20:19.417 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A= 2022-09-02T04:20:19.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A=', 'Content-Length': '0'} 2022-09-02T04:20:19.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000824e7ad7538dd859-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.424 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.424 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.425 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.425 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.426 DEBUG:boto:Data: 2022-09-02T04:20:19.427 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.428 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.428 DEBUG:boto:Params: {} 2022-09-02T04:20:19.429 DEBUG:boto:Token: None 2022-09-02T04:20:19.429 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo= 2022-09-02T04:20:19.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo=', 'Content-Length': '0'} 2022-09-02T04:20:19.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a6c643c5ade7b7-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.433 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.434 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.435 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.435 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.436 DEBUG:boto:Data: 2022-09-02T04:20:19.436 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.437 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.438 DEBUG:boto:Params: {} 2022-09-02T04:20:19.439 DEBUG:boto:Token: None 2022-09-02T04:20:19.440 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA= 2022-09-02T04:20:19.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA=', 'Content-Length': '0'} 2022-09-02T04:20:19.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c68e094813e9fde-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.445 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.446 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.447 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.448 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.449 DEBUG:boto:Data: 2022-09-02T04:20:19.450 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.451 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.452 DEBUG:boto:Params: {} 2022-09-02T04:20:19.452 DEBUG:boto:Token: None 2022-09-02T04:20:19.453 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A= 2022-09-02T04:20:19.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A=', 'Content-Length': '0'} 2022-09-02T04:20:19.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043ac09b898b584d7-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.457 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.458 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.458 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.459 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.460 DEBUG:boto:Data: 2022-09-02T04:20:19.461 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.462 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.463 DEBUG:boto:Params: {} 2022-09-02T04:20:19.464 DEBUG:boto:Token: None 2022-09-02T04:20:19.465 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo= 2022-09-02T04:20:19.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo=', 'Content-Length': '0'} 2022-09-02T04:20:19.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082e2e39ef012391c-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.474 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.475 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.476 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.477 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.478 DEBUG:boto:Data: 2022-09-02T04:20:19.479 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.482 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.483 DEBUG:boto:Params: {} 2022-09-02T04:20:19.483 DEBUG:boto:Token: None 2022-09-02T04:20:19.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA= 2022-09-02T04:20:19.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA=', 'Content-Length': '0'} 2022-09-02T04:20:19.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036a76386745153bc-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.499 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.500 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.500 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.501 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.501 DEBUG:boto:Data: 2022-09-02T04:20:19.502 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.504 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.504 DEBUG:boto:Params: {} 2022-09-02T04:20:19.505 DEBUG:boto:Token: None 2022-09-02T04:20:19.506 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A= 2022-09-02T04:20:19.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A=', 'Content-Length': '0'} 2022-09-02T04:20:19.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc7b76c22979b0d7-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.519 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.520 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.521 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.521 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.522 DEBUG:boto:Data: 2022-09-02T04:20:19.523 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.525 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.525 DEBUG:boto:Params: {} 2022-09-02T04:20:19.526 DEBUG:boto:Token: None 2022-09-02T04:20:19.527 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo= 2022-09-02T04:20:19.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo=', 'Content-Length': '0'} 2022-09-02T04:20:19.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e4e76020911d55f-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.533 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.534 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.536 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.537 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.538 DEBUG:boto:Data: 2022-09-02T04:20:19.538 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.539 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.540 DEBUG:boto:Params: {} 2022-09-02T04:20:19.541 DEBUG:boto:Token: None 2022-09-02T04:20:19.541 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA= 2022-09-02T04:20:19.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA=', 'Content-Length': '0'} 2022-09-02T04:20:19.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000250874af88283283-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.547 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.548 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.549 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.550 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.551 DEBUG:boto:Data: 2022-09-02T04:20:19.552 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.553 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.554 DEBUG:boto:Params: {} 2022-09-02T04:20:19.554 DEBUG:boto:Token: None 2022-09-02T04:20:19.555 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BJphGjYWIA8lkA64su2ZBM2a80c= 2022-09-02T04:20:19.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BJphGjYWIA8lkA64su2ZBM2a80c=', 'Content-Length': '0'} 2022-09-02T04:20:19.563 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f4e1ee826676179-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.564 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.565 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.566 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.567 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.568 DEBUG:boto:Data: 2022-09-02T04:20:19.569 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.572 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.574 DEBUG:boto:Params: {} 2022-09-02T04:20:19.575 DEBUG:boto:Token: None 2022-09-02T04:20:19.576 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZEJHphdamFuSdE/mqhOMEtEsMg= 2022-09-02T04:20:19.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZEJHphdamFuSdE/mqhOMEtEsMg=', 'Content-Length': '0'} 2022-09-02T04:20:19.588 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ffce235c6dcc0b9-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.589 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.590 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.593 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.594 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.595 DEBUG:boto:Data: 2022-09-02T04:20:19.596 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.598 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.599 DEBUG:boto:Params: {} 2022-09-02T04:20:19.600 DEBUG:boto:Token: None 2022-09-02T04:20:19.601 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cyx/inuY/V/SvTpeBPtWXL9MBD4= 2022-09-02T04:20:19.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cyx/inuY/V/SvTpeBPtWXL9MBD4=', 'Content-Length': '0'} 2022-09-02T04:20:19.609 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000decf43a9778cc087-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.610 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.610 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.611 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.611 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.612 DEBUG:boto:Data: 2022-09-02T04:20:19.612 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.613 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.614 DEBUG:boto:Params: {} 2022-09-02T04:20:19.616 DEBUG:boto:Token: None 2022-09-02T04:20:19.619 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BJphGjYWIA8lkA64su2ZBM2a80c= 2022-09-02T04:20:19.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BJphGjYWIA8lkA64su2ZBM2a80c=', 'Content-Length': '0'} 2022-09-02T04:20:19.627 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000661b3e2774afdee5-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.629 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.630 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.631 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.631 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.632 DEBUG:boto:Data: 2022-09-02T04:20:19.637 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.638 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.638 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.640 DEBUG:boto:Params: {} 2022-09-02T04:20:19.640 DEBUG:boto:Token: None 2022-09-02T04:20:19.641 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZEJHphdamFuSdE/mqhOMEtEsMg= 2022-09-02T04:20:19.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZEJHphdamFuSdE/mqhOMEtEsMg=', 'Content-Length': '0'} 2022-09-02T04:20:19.690 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002dd37aecf70981cf-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.691 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.692 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.693 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.694 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.694 DEBUG:boto:Data: 2022-09-02T04:20:19.695 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.696 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.696 DEBUG:boto:Params: {} 2022-09-02T04:20:19.697 DEBUG:boto:Token: None 2022-09-02T04:20:19.697 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cyx/inuY/V/SvTpeBPtWXL9MBD4= 2022-09-02T04:20:19.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cyx/inuY/V/SvTpeBPtWXL9MBD4=', 'Content-Length': '0'} 2022-09-02T04:20:19.744 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b0873ddd3bc2bb24-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.746 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.749 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.750 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.750 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.751 DEBUG:boto:Data: 2022-09-02T04:20:19.752 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.754 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.755 DEBUG:boto:Params: {} 2022-09-02T04:20:19.757 DEBUG:boto:Token: None 2022-09-02T04:20:19.760 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BJphGjYWIA8lkA64su2ZBM2a80c= 2022-09-02T04:20:19.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BJphGjYWIA8lkA64su2ZBM2a80c=', 'Content-Length': '0'} 2022-09-02T04:20:19.778 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c62af8c945dffa7-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.783 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.785 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.786 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.788 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.789 DEBUG:boto:Data: 2022-09-02T04:20:19.791 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.793 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.794 DEBUG:boto:Params: {} 2022-09-02T04:20:19.795 DEBUG:boto:Token: None 2022-09-02T04:20:19.796 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZEJHphdamFuSdE/mqhOMEtEsMg= 2022-09-02T04:20:19.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZEJHphdamFuSdE/mqhOMEtEsMg=', 'Content-Length': '0'} 2022-09-02T04:20:19.845 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000029f82b161fc9e80-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.846 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.847 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.848 DEBUG:boto:Method: PUT 2022-09-02T04:20:19.848 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.849 DEBUG:boto:Data: 2022-09-02T04:20:19.849 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.850 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.851 DEBUG:boto:Params: {} 2022-09-02T04:20:19.851 DEBUG:boto:Token: None 2022-09-02T04:20:19.852 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cyx/inuY/V/SvTpeBPtWXL9MBD4= 2022-09-02T04:20:19.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cyx/inuY/V/SvTpeBPtWXL9MBD4=', 'Content-Length': '0'} 2022-09-02T04:20:19.901 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000cf24da9bbc87e6c-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.902 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.903 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.903 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.904 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.904 DEBUG:boto:Data: 2022-09-02T04:20:19.905 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.906 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.907 DEBUG:boto:Params: {} 2022-09-02T04:20:19.907 DEBUG:boto:Token: None 2022-09-02T04:20:19.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A= 2022-09-02T04:20:19.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A=', 'Content-Length': '0'} 2022-09-02T04:20:19.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079acd37b9c2f5ae5-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.912 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.913 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.913 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.914 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.914 DEBUG:boto:Data: 2022-09-02T04:20:19.915 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.916 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.917 DEBUG:boto:Params: {} 2022-09-02T04:20:19.918 DEBUG:boto:Token: None 2022-09-02T04:20:19.919 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo= 2022-09-02T04:20:19.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo=', 'Content-Length': '0'} 2022-09-02T04:20:19.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b56370a851679174-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.924 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.924 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.925 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.925 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.926 DEBUG:boto:Data: 2022-09-02T04:20:19.926 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:19.927 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.928 DEBUG:boto:Params: {} 2022-09-02T04:20:19.929 DEBUG:boto:Token: None 2022-09-02T04:20:19.930 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA= 2022-09-02T04:20:19.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA=', 'Content-Length': '0'} 2022-09-02T04:20:19.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a406d3d0e85fa134-0063118483-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.937 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.938 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.938 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.939 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.939 DEBUG:boto:Data: 2022-09-02T04:20:19.940 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.942 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.942 DEBUG:boto:Params: {} 2022-09-02T04:20:19.943 DEBUG:boto:Token: None 2022-09-02T04:20:19.944 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A= 2022-09-02T04:20:19.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A=', 'Content-Length': '0'} 2022-09-02T04:20:19.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb89fcfe525d3137-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.948 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.949 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.950 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.950 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.951 DEBUG:boto:Data: 2022-09-02T04:20:19.951 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.953 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.953 DEBUG:boto:Params: {} 2022-09-02T04:20:19.954 DEBUG:boto:Token: None 2022-09-02T04:20:19.955 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo= 2022-09-02T04:20:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo=', 'Content-Length': '0'} 2022-09-02T04:20:19.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a82d030fa5a9430-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.959 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.960 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:19.961 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.962 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:19.963 DEBUG:boto:Data: 2022-09-02T04:20:19.964 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:19.965 DEBUG:boto:Port: 8000 2022-09-02T04:20:19.966 DEBUG:boto:Params: {} 2022-09-02T04:20:19.967 DEBUG:boto:Token: None 2022-09-02T04:20:19.967 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-6/ 2022-09-02T04:20:19.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA= 2022-09-02T04:20:19.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wucK4Lce7nM9U5H1HroByCQvwUA=', 'Content-Length': '0'} 2022-09-02T04:20:19.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000625807477930721-0063118483-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.972 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:19.973 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:19.976 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.977 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:19.977 DEBUG:boto:Data: 2022-09-02T04:20:19.978 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.979 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.979 DEBUG:boto:Params: {} 2022-09-02T04:20:19.980 DEBUG:boto:Token: None 2022-09-02T04:20:19.980 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-4/ 2022-09-02T04:20:19.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A= 2022-09-02T04:20:19.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEE9corooXq4Ld4rrxXyearja3A=', 'Content-Length': '0'} 2022-09-02T04:20:19.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8e94f7b17205989-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.987 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:19.988 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:19.989 DEBUG:boto:Method: HEAD 2022-09-02T04:20:19.990 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:19.991 DEBUG:boto:Data: 2022-09-02T04:20:19.992 DEBUG:boto:Headers: {} 2022-09-02T04:20:19.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:19.993 DEBUG:boto:Port: 8001 2022-09-02T04:20:19.993 DEBUG:boto:Params: {} 2022-09-02T04:20:19.994 DEBUG:boto:Token: None 2022-09-02T04:20:19.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:19 GMT /xljbkf-5/ 2022-09-02T04:20:19.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo= 2022-09-02T04:20:19.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BzuMYcmFriWHh3NzAjjfDY2pAIo=', 'Content-Length': '0'} 2022-09-02T04:20:19.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000316b3bca28985c60-0063118483-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:19.999 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:19.999 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:20.000 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.000 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:20.001 DEBUG:boto:Data: 2022-09-02T04:20:20.002 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:20.003 DEBUG:boto:Port: 8001 2022-09-02T04:20:20.003 DEBUG:boto:Params: {} 2022-09-02T04:20:20.004 DEBUG:boto:Token: None 2022-09-02T04:20:20.005 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-6/ 2022-09-02T04:20:20.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE= 2022-09-02T04:20:20.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE=', 'Content-Length': '0'} 2022-09-02T04:20:20.008 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055e7e44dfedd4742-0063118484-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.010 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:20.110 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:20.127 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:20.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:20.129 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:20.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.130 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:20.131 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:20.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.132 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:20.136 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:20.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.137 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:20.138 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:18.665018Z" 2022-09-02T04:20:20.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:20.139 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:20.141 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.142 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:20.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.146 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:20.203 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:20.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:20.240 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:20.234+0000 7fee229d9600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:20.241 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:20.235+0000 7fee229d9600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:20.251 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:20.252 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:20.252 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:20.253 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:20.253 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:20.253 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:20.253 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.254 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:20.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.255 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:20.255 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.255 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.255 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:20.256 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.256 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:20.256 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.257 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:20.257 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.260 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:20.260 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.261 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.261 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:20.262 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.262 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:20.263 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.263 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:20.264 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.264 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.265 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:20.265 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.265 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.265 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:20.266 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.267 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:20.267 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.267 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:20.267 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.269 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:20.269 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.269 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.270 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:20.270 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.270 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:20.271 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.271 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.665018Z", 2022-09-02T04:20:20.271 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.272 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:20.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.274 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:20.274 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:20.275 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:20.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.275 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:20Z" 2022-09-02T04:20:20.276 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:20.277 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092418.665018_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:18.665018Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:20Z" } 2022-09-02T04:20:20.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092418.665018_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:18.665018Z', 'realm_epoch': 2}}] 2022-09-02T04:20:20.278 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.280 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:20.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:20.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.286 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:20.351 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:20.351 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:20.379 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:20.378+0000 7f9d5a8eb600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:20.381 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:20.379+0000 7f9d5a8eb600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:20.394 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:20.395 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:20.395 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:20.395 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:20.396 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:20.396 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:20.396 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:20.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.397 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:20.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.397 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:20.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:20.398 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:20.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:20.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:20.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:20.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:20.401 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:20.401 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:20.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:20.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.402 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:20.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:20.403 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:20.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:20.404 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:20.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:20.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:20.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:20.405 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:20.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:20.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.407 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:20.408 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:20.408 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:20.408 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:20.409 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:20.409 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:20.409 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:20.410 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:20.410 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:20.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:20.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:20.412 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:20.412 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:20.413 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:20.413 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:20.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:20.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:20.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:20.415 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.665018Z", 2022-09-02T04:20:20.415 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:20.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:20.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:20.416 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:20.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.417 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:20.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:20.417 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:20.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:20.418 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:20Z" 2022-09-02T04:20:20.419 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:20.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092418.665018_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:18.665018Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:20Z" } 2022-09-02T04:20:20.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092418.665018_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:18.665018Z', 'realm_epoch': 2}}] 2022-09-02T04:20:20.422 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.423 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:20.424 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:20.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.427 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:20.477 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:20.478 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:20.525 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:20.518+0000 7f2dec32c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:20.525 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:20.519+0000 7f2dec32c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:20.539 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:20.539 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:20.540 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:20.540 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:20.541 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:20.541 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:20.542 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.543 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:20.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.545 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:20.546 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.547 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.548 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:20.548 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.549 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:20.549 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.550 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:20.550 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.553 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:20.553 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.553 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.554 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:20.554 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.555 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:20.555 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.556 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:20.556 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.559 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:20.559 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.559 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.560 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:20.561 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.562 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:20.562 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.563 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:20.563 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.564 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:20.566 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:20.567 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:20.567 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:20.568 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092418.665018_116.1", 2022-09-02T04:20:20.568 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:20.569 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:20.569 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:20.570 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.665018Z", 2022-09-02T04:20:20.570 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:20.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:20.572 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:20.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.573 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:20.573 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:20.574 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:20.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:20.575 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:20Z" 2022-09-02T04:20:20.575 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:20.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092418.665018_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:18.665018Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:20Z" } 2022-09-02T04:20:20.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092418.665018_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:18.665018Z', 'realm_epoch': 2}}] 2022-09-02T04:20:20.580 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092418.665018_116.1'} 2022-09-02T04:20:20.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:20.583 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:20.584 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:20.585 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.586 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:20.587 DEBUG:boto:Data: 2022-09-02T04:20:20.588 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:20.590 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.601 DEBUG:boto:Params: {} 2022-09-02T04:20:20.602 DEBUG:boto:Token: None 2022-09-02T04:20:20.605 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-4/ 2022-09-02T04:20:20.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZvEy7jQ6pFsXwOtY2WATpgiqVUE= 2022-09-02T04:20:20.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZvEy7jQ6pFsXwOtY2WATpgiqVUE=', 'Content-Length': '0'} 2022-09-02T04:20:20.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8659406a81fdde4-0063118484-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.615 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:20.616 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:20.617 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.618 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:20.619 DEBUG:boto:Data: 2022-09-02T04:20:20.620 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:20.622 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.623 DEBUG:boto:Params: {} 2022-09-02T04:20:20.624 DEBUG:boto:Token: None 2022-09-02T04:20:20.625 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-5/ 2022-09-02T04:20:20.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QA/dHv6HTIRttTXplpbLmuoUXWI= 2022-09-02T04:20:20.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QA/dHv6HTIRttTXplpbLmuoUXWI=', 'Content-Length': '0'} 2022-09-02T04:20:20.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0e61a51c1fe0249-0063118484-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.632 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:20.632 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:20.633 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.633 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:20.634 DEBUG:boto:Data: 2022-09-02T04:20:20.635 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:20.636 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.637 DEBUG:boto:Params: {} 2022-09-02T04:20:20.638 DEBUG:boto:Token: None 2022-09-02T04:20:20.639 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-6/ 2022-09-02T04:20:20.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE= 2022-09-02T04:20:20.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE=', 'Content-Length': '0'} 2022-09-02T04:20:20.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bf1200817191146-0063118484-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.645 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:20.646 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:20.646 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.647 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:20.647 DEBUG:boto:Data: 2022-09-02T04:20:20.648 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:20.649 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.650 DEBUG:boto:Params: {} 2022-09-02T04:20:20.650 DEBUG:boto:Token: None 2022-09-02T04:20:20.651 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-4/ 2022-09-02T04:20:20.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZvEy7jQ6pFsXwOtY2WATpgiqVUE= 2022-09-02T04:20:20.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZvEy7jQ6pFsXwOtY2WATpgiqVUE=', 'Content-Length': '0'} 2022-09-02T04:20:20.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cebea9b853f9dd63-0063118484-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.657 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:20.657 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:20.658 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.661 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:20.662 DEBUG:boto:Data: 2022-09-02T04:20:20.663 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:20.665 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.665 DEBUG:boto:Params: {} 2022-09-02T04:20:20.666 DEBUG:boto:Token: None 2022-09-02T04:20:20.667 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-5/ 2022-09-02T04:20:20.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QA/dHv6HTIRttTXplpbLmuoUXWI= 2022-09-02T04:20:20.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QA/dHv6HTIRttTXplpbLmuoUXWI=', 'Content-Length': '0'} 2022-09-02T04:20:20.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008065cefdb65bdd05-0063118484-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.674 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:20.675 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:20.675 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.676 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:20.677 DEBUG:boto:Data: 2022-09-02T04:20:20.678 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:20.679 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.680 DEBUG:boto:Params: {} 2022-09-02T04:20:20.681 DEBUG:boto:Token: None 2022-09-02T04:20:20.682 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-6/ 2022-09-02T04:20:20.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE= 2022-09-02T04:20:20.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE=', 'Content-Length': '0'} 2022-09-02T04:20:20.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5ec0b90e02a594d-0063118484-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.696 DEBUG:boto:path=/xljbkf-4/ 2022-09-02T04:20:20.697 DEBUG:boto:auth_path=/xljbkf-4/ 2022-09-02T04:20:20.698 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.699 DEBUG:boto:Path: /xljbkf-4/ 2022-09-02T04:20:20.699 DEBUG:boto:Data: 2022-09-02T04:20:20.701 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:20.702 DEBUG:boto:Port: 8001 2022-09-02T04:20:20.704 DEBUG:boto:Params: {} 2022-09-02T04:20:20.705 DEBUG:boto:Token: None 2022-09-02T04:20:20.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-4/ 2022-09-02T04:20:20.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZvEy7jQ6pFsXwOtY2WATpgiqVUE= 2022-09-02T04:20:20.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZvEy7jQ6pFsXwOtY2WATpgiqVUE=', 'Content-Length': '0'} 2022-09-02T04:20:20.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6bb7cc883101834-0063118484-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.712 DEBUG:boto:path=/xljbkf-5/ 2022-09-02T04:20:20.713 DEBUG:boto:auth_path=/xljbkf-5/ 2022-09-02T04:20:20.714 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.715 DEBUG:boto:Path: /xljbkf-5/ 2022-09-02T04:20:20.716 DEBUG:boto:Data: 2022-09-02T04:20:20.717 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:20.719 DEBUG:boto:Port: 8001 2022-09-02T04:20:20.720 DEBUG:boto:Params: {} 2022-09-02T04:20:20.721 DEBUG:boto:Token: None 2022-09-02T04:20:20.722 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-5/ 2022-09-02T04:20:20.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QA/dHv6HTIRttTXplpbLmuoUXWI= 2022-09-02T04:20:20.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QA/dHv6HTIRttTXplpbLmuoUXWI=', 'Content-Length': '0'} 2022-09-02T04:20:20.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4d88f4f77f1d1c2-0063118484-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.731 DEBUG:boto:path=/xljbkf-6/ 2022-09-02T04:20:20.732 DEBUG:boto:auth_path=/xljbkf-6/ 2022-09-02T04:20:20.733 DEBUG:boto:Method: HEAD 2022-09-02T04:20:20.733 DEBUG:boto:Path: /xljbkf-6/ 2022-09-02T04:20:20.734 DEBUG:boto:Data: 2022-09-02T04:20:20.734 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:20.735 DEBUG:boto:Port: 8001 2022-09-02T04:20:20.736 DEBUG:boto:Params: {} 2022-09-02T04:20:20.737 DEBUG:boto:Token: None 2022-09-02T04:20:20.737 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-6/ 2022-09-02T04:20:20.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE= 2022-09-02T04:20:20.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kzX2RH5CuLx7CZuOkl+PO1VJmGE=', 'Content-Length': '0'} 2022-09-02T04:20:20.749 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009a16760e08ed091-0063118484-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2022-09-02T04:20:20.753 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-7 2022-09-02T04:20:20.754 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:20.755 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:20.756 DEBUG:boto:Method: PUT 2022-09-02T04:20:20.757 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:20.758 DEBUG:boto:Data: 2022-09-02T04:20:20.758 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:20.760 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.760 DEBUG:boto:Params: {} 2022-09-02T04:20:20.761 DEBUG:boto:Token: None 2022-09-02T04:20:20.762 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-7/ 2022-09-02T04:20:20.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Tltmjy5UF4TWsLc+MeiZpHqjqo= 2022-09-02T04:20:20.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Tltmjy5UF4TWsLc+MeiZpHqjqo=', 'Content-Length': '0'} 2022-09-02T04:20:20.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b336787aad6e2dd-0063118484-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.782 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-8 2022-09-02T04:20:20.783 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:20.784 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:20.784 DEBUG:boto:Method: PUT 2022-09-02T04:20:20.785 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:20.786 DEBUG:boto:Data: 2022-09-02T04:20:20.787 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:20.789 DEBUG:boto:Port: 8000 2022-09-02T04:20:20.789 DEBUG:boto:Params: {} 2022-09-02T04:20:20.790 DEBUG:boto:Token: None 2022-09-02T04:20:20.791 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-8/ 2022-09-02T04:20:20.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J6yUeK5IABIjQTODMKCD8sxji7I= 2022-09-02T04:20:20.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J6yUeK5IABIjQTODMKCD8sxji7I=', 'Content-Length': '0'} 2022-09-02T04:20:20.816 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000aee883138a4be06-0063118484-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.817 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-9 2022-09-02T04:20:20.818 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:20.820 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:20.821 DEBUG:boto:Method: PUT 2022-09-02T04:20:20.822 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:20.823 DEBUG:boto:Data: 2022-09-02T04:20:20.824 DEBUG:boto:Headers: {} 2022-09-02T04:20:20.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:20.826 DEBUG:boto:Port: 8001 2022-09-02T04:20:20.827 DEBUG:boto:Params: {} 2022-09-02T04:20:20.828 DEBUG:boto:Token: None 2022-09-02T04:20:20.829 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:20 GMT /xljbkf-9/ 2022-09-02T04:20:20.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CQCQe22EFfJG0GsnCj4YCeCFkSc= 2022-09-02T04:20:20.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CQCQe22EFfJG0GsnCj4YCeCFkSc=', 'Content-Length': '0'} 2022-09-02T04:20:20.889 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009fdee64ac6496235-0063118484-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:20.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:20.891 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:20.945 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:20.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:21.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:21.011 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:21.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.012 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:21.013 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:21.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:21.017 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:21.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.022 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092420.862622_128.1", 2022-09-02T04:20:21.023 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:20.862622Z" 2022-09-02T04:20:21.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.024 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:21.024 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:21.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.035 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:21.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:21.096 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:21.121 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:21.119+0000 7fcb35431600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:21.125 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:21.120+0000 7fcb35431600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:21.134 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:21.134 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:21.135 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:21.135 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:21.136 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:21.140 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:21.144 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.145 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:21.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.145 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:21.146 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.146 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.147 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:21.147 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.148 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:21.148 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.148 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:21.149 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.150 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:21.151 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.151 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.151 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:21.151 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.152 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:21.154 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.155 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:21.156 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.157 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:21.159 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.159 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.160 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:21.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.161 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:21.161 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.161 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:21.162 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.164 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092420.862622_128.1", 2022-09-02T04:20:21.166 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.167 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:21.167 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.168 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.862622Z", 2022-09-02T04:20:21.168 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.169 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:21.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.171 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:21.171 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:21.171 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:21.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.172 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:21Z" 2022-09-02T04:20:21.172 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:21.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092420.862622_128.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:20.862622Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:21Z" } 2022-09-02T04:20:21.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092420.862622_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:20.862622Z', 'realm_epoch': 2}}] 2022-09-02T04:20:21.175 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.176 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:21.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:21.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.180 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:21.260 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:21.255+0000 7fa8afb90600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:21.260 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:21.256+0000 7fa8afb90600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:21.269 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092420.862622_128.1", 2022-09-02T04:20:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.862622Z", 2022-09-02T04:20:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:21Z" 2022-09-02T04:20:21.294 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:21.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092420.862622_128.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:20.862622Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:21Z" } 2022-09-02T04:20:21.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092420.862622_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:20.862622Z', 'realm_epoch': 2}}] 2022-09-02T04:20:21.298 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.299 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.299 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:21.300 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:21.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.303 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:21.353 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:21.354 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:21.393 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:21.391+0000 7f5d5291c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:21.394 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:21.392+0000 7f5d5291c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:21.410 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:21.411 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:21.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.414 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.415 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:21.416 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.417 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:21.418 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:21.419 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.420 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:21.421 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.422 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:21.424 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:21.425 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092420.862622_128.1", 2022-09-02T04:20:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:21.426 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.862622Z", 2022-09-02T04:20:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.427 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:21.428 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:21.429 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:21Z" 2022-09-02T04:20:21.430 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:21.431 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092420.862622_128.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:20.862622Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:21Z" } 2022-09-02T04:20:21.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092420.862622_128.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:20.862622Z', 'realm_epoch': 2}}] 2022-09-02T04:20:21.432 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092420.862622_128.1'} 2022-09-02T04:20:21.433 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:21.434 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:21.435 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:21.435 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.436 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:21.436 DEBUG:boto:Data: 2022-09-02T04:20:21.437 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:21.438 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.438 DEBUG:boto:Params: {} 2022-09-02T04:20:21.439 DEBUG:boto:Token: None 2022-09-02T04:20:21.440 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-7/ 2022-09-02T04:20:21.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lND7X5uTPWR54WyHLdfssDVZHro= 2022-09-02T04:20:21.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lND7X5uTPWR54WyHLdfssDVZHro=', 'Content-Length': '0'} 2022-09-02T04:20:21.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000daad6bc8744c3240-0063118485-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.444 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:21.445 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:21.445 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.446 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:21.446 DEBUG:boto:Data: 2022-09-02T04:20:21.447 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:21.448 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.449 DEBUG:boto:Params: {} 2022-09-02T04:20:21.449 DEBUG:boto:Token: None 2022-09-02T04:20:21.450 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-8/ 2022-09-02T04:20:21.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/VPiRF7tpdzR0x543dHvXooO528= 2022-09-02T04:20:21.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/VPiRF7tpdzR0x543dHvXooO528=', 'Content-Length': '0'} 2022-09-02T04:20:21.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028446e6ca84a1d76-0063118485-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.456 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:21.457 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:21.459 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.459 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:21.461 DEBUG:boto:Data: 2022-09-02T04:20:21.462 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:21.465 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.465 DEBUG:boto:Params: {} 2022-09-02T04:20:21.466 DEBUG:boto:Token: None 2022-09-02T04:20:21.467 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-9/ 2022-09-02T04:20:21.468 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UfrmGbK72H0VlnJaDkKypmgR/YQ= 2022-09-02T04:20:21.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UfrmGbK72H0VlnJaDkKypmgR/YQ=', 'Content-Length': '0'} 2022-09-02T04:20:21.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b615c44fc5dc1be-0063118485-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.473 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:21.473 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:21.474 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.475 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:21.476 DEBUG:boto:Data: 2022-09-02T04:20:21.476 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:21.478 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.479 DEBUG:boto:Params: {} 2022-09-02T04:20:21.480 DEBUG:boto:Token: None 2022-09-02T04:20:21.481 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-7/ 2022-09-02T04:20:21.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lND7X5uTPWR54WyHLdfssDVZHro= 2022-09-02T04:20:21.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lND7X5uTPWR54WyHLdfssDVZHro=', 'Content-Length': '0'} 2022-09-02T04:20:21.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ce981f799023b77-0063118485-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.488 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:21.489 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:21.490 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.491 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:21.492 DEBUG:boto:Data: 2022-09-02T04:20:21.492 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.493 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:21.494 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.495 DEBUG:boto:Params: {} 2022-09-02T04:20:21.496 DEBUG:boto:Token: None 2022-09-02T04:20:21.497 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-8/ 2022-09-02T04:20:21.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/VPiRF7tpdzR0x543dHvXooO528= 2022-09-02T04:20:21.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/VPiRF7tpdzR0x543dHvXooO528=', 'Content-Length': '0'} 2022-09-02T04:20:21.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b92ee1fa85499c7-0063118485-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.504 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:21.505 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:21.506 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.507 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:21.508 DEBUG:boto:Data: 2022-09-02T04:20:21.509 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:21.511 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.512 DEBUG:boto:Params: {} 2022-09-02T04:20:21.513 DEBUG:boto:Token: None 2022-09-02T04:20:21.514 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-9/ 2022-09-02T04:20:21.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UfrmGbK72H0VlnJaDkKypmgR/YQ= 2022-09-02T04:20:21.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UfrmGbK72H0VlnJaDkKypmgR/YQ=', 'Content-Length': '0'} 2022-09-02T04:20:21.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029731340de1e819a-0063118485-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.520 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:21.521 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:21.522 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.523 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:21.524 DEBUG:boto:Data: 2022-09-02T04:20:21.525 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:21.527 DEBUG:boto:Port: 8001 2022-09-02T04:20:21.528 DEBUG:boto:Params: {} 2022-09-02T04:20:21.529 DEBUG:boto:Token: None 2022-09-02T04:20:21.530 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-7/ 2022-09-02T04:20:21.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lND7X5uTPWR54WyHLdfssDVZHro= 2022-09-02T04:20:21.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lND7X5uTPWR54WyHLdfssDVZHro=', 'Content-Length': '0'} 2022-09-02T04:20:21.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ad8e03e1d70d8b2-0063118485-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.536 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:21.537 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:21.538 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.539 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:21.540 DEBUG:boto:Data: 2022-09-02T04:20:21.541 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:21.542 DEBUG:boto:Port: 8001 2022-09-02T04:20:21.543 DEBUG:boto:Params: {} 2022-09-02T04:20:21.544 DEBUG:boto:Token: None 2022-09-02T04:20:21.545 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-8/ 2022-09-02T04:20:21.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/VPiRF7tpdzR0x543dHvXooO528= 2022-09-02T04:20:21.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/VPiRF7tpdzR0x543dHvXooO528=', 'Content-Length': '0'} 2022-09-02T04:20:21.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fc438c997ca37d0-0063118485-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.550 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:21.551 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:21.551 DEBUG:boto:Method: HEAD 2022-09-02T04:20:21.552 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:21.552 DEBUG:boto:Data: 2022-09-02T04:20:21.553 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:21.554 DEBUG:boto:Port: 8001 2022-09-02T04:20:21.555 DEBUG:boto:Params: {} 2022-09-02T04:20:21.556 DEBUG:boto:Token: None 2022-09-02T04:20:21.557 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-9/ 2022-09-02T04:20:21.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UfrmGbK72H0VlnJaDkKypmgR/YQ= 2022-09-02T04:20:21.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UfrmGbK72H0VlnJaDkKypmgR/YQ=', 'Content-Length': '0'} 2022-09-02T04:20:21.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ba874a3d2635a14-0063118485-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.562 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:21.563 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:21.563 DEBUG:boto:Method: DELETE 2022-09-02T04:20:21.564 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:21.564 DEBUG:boto:Data: 2022-09-02T04:20:21.565 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:21.566 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.566 DEBUG:boto:Params: {} 2022-09-02T04:20:21.567 DEBUG:boto:Token: None 2022-09-02T04:20:21.568 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-7/ 2022-09-02T04:20:21.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VNCbMtWdyZxY69fe9ImEJaXwNN8= 2022-09-02T04:20:21.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VNCbMtWdyZxY69fe9ImEJaXwNN8=', 'Content-Length': '0'} 2022-09-02T04:20:21.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e3b7d282e40550e-0063118485-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.609 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:21.610 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:21.611 DEBUG:boto:Method: DELETE 2022-09-02T04:20:21.612 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:21.612 DEBUG:boto:Data: 2022-09-02T04:20:21.613 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:21.615 DEBUG:boto:Port: 8000 2022-09-02T04:20:21.617 DEBUG:boto:Params: {} 2022-09-02T04:20:21.617 DEBUG:boto:Token: None 2022-09-02T04:20:21.618 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-8/ 2022-09-02T04:20:21.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HwwOeqqke4HKKlgeAtOSdwZ2hRU= 2022-09-02T04:20:21.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HwwOeqqke4HKKlgeAtOSdwZ2hRU=', 'Content-Length': '0'} 2022-09-02T04:20:21.718 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005c01f016166c4efa-0063118485-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.720 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:21.721 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:21.722 DEBUG:boto:Method: DELETE 2022-09-02T04:20:21.723 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:21.725 DEBUG:boto:Data: 2022-09-02T04:20:21.726 DEBUG:boto:Headers: {} 2022-09-02T04:20:21.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:21.727 DEBUG:boto:Port: 8001 2022-09-02T04:20:21.728 DEBUG:boto:Params: {} 2022-09-02T04:20:21.729 DEBUG:boto:Token: None 2022-09-02T04:20:21.730 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:20:21 GMT /xljbkf-9/ 2022-09-02T04:20:21.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:73uY7fGlr8AMjlDd+hgLatAAtG0= 2022-09-02T04:20:21.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:73uY7fGlr8AMjlDd+hgLatAAtG0=', 'Content-Length': '0'} 2022-09-02T04:20:21.807 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007fccaab12c83f724-0063118485-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:20:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:21.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.812 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:21.936 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:21.956 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:21.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.957 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:21.957 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:21.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.958 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:21.959 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:21.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.962 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:21.962 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:21.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:21.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:21.963 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:21.964 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:20:21.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:21.965 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:21.969 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:21.972 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:21.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:21.973 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:22.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:22.026 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:22.071 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:22.065+0000 7f5810162600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:22.072 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:22.066+0000 7f5810162600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:22.079 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:22.079 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:22.079 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:22.080 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:22.081 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:22.081 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:22.084 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:22.084 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:22.085 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:22.085 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:22.086 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:22.087 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:22.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:22.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:22.089 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:22.090 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:22.090 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:22.091 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:22.091 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:22.092 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:22.092 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:22.093 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:22.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:22.095 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:22.096 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:22.097 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:22.097 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:22.098 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:22.098 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:22.099 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:22.099 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:22.099 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:22.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:22.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:22.101 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:22.102 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:22.103 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:22.105 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.580527_129.1", 2022-09-02T04:20:22.106 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:22.107 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:22.107 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:22.108 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.580527Z", 2022-09-02T04:20:22.108 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:22.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:22.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:22.110 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:22.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:22.111 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:22.111 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:22.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:22.113 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:22Z" 2022-09-02T04:20:22.116 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:22.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.580527_129.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.580527Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:22Z" } 2022-09-02T04:20:22.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.580527_129.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.580527Z', 'realm_epoch': 2}}] 2022-09-02T04:20:22.120 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:22.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.580527_129.1'} 2022-09-02T04:20:22.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1662092421.771218_140.1 target=1_1662092421.580527_129.1 2022-09-02T04:20:27.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:27.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:27.126 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:27.181 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:27.183 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:27.233 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:27.224+0000 7f387f6a2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:27.234 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:27.225+0000 7f387f6a2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:27.249 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:27.250 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:27.250 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:27.250 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:27.251 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:27.251 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:27.251 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.252 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:27.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.252 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:27.252 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.253 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.253 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:27.253 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.254 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:27.254 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.254 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:27.255 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.256 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:27.256 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.256 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.256 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:27.257 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.257 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:27.257 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.257 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:27.258 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.260 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:27.260 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.260 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.261 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:27.261 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.261 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:27.266 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.266 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:27.266 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.274 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:27.275 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.276 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:27.279 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:27.280 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:20:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.282 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:27.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.282 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:27.283 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:27.283 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:27.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.283 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:27Z" 2022-09-02T04:20:27.284 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:27.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:27Z" } 2022-09-02T04:20:27.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:20:27.287 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:27.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:27.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:27.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:27.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:27.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:27.290 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:27.348 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:27.383+0000 7fba768a1600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:27.385+0000 7fba768a1600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:27.397 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:27.405 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:27.405 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:27.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:27.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:27.406 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:27.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:27.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:27.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:27.407 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:27.407 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:27.407 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:27.407 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:27.408 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:27.408 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:27.408 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:27.408 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:27.409 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:27.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:27.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:27.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:27.410 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:27.410 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:27.410 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:27.410 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:27.411 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:27.411 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:27.411 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:27.411 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:20:27.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:27.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:27.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:27.412 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:27.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:27.413 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:27.413 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:27.413 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:27.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:27.414 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:27Z" 2022-09-02T04:20:27.414 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:27.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:27Z" } 2022-09-02T04:20:27.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:20:27.417 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:27.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:27.418 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:27.419 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:27.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:27.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:27.420 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:27.472 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:27.473 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:27.517+0000 7ff2a5d3d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:27.522 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:27.518+0000 7ff2a5d3d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:27.536 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:27.537 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:27.537 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:27.537 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:27.538 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:27.538 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:27.539 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.539 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:27.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.540 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:27.541 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.541 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.541 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:27.542 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.542 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:27.542 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:27.543 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.545 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:27.546 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.546 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.547 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:27.547 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.548 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:27.549 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.549 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:27.552 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.555 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:27.556 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.557 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.557 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:27.557 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.558 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:27.558 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.559 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:27.559 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.560 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:27.565 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:27.566 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:27.566 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:27.567 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:27.567 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:27.568 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:27.568 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:27.569 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:20:27.569 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:27.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:27.572 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:27.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.573 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:27.573 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:27.574 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:27.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:27.575 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:27Z" 2022-09-02T04:20:27.576 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:27.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:27Z" } 2022-09-02T04:20:27.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:20:27.579 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:27.580 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:27.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:27.581 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:27.583 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:27.584 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.585 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:27.586 DEBUG:boto:Data: 2022-09-02T04:20:27.587 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.588 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.589 DEBUG:boto:Params: {} 2022-09-02T04:20:27.590 DEBUG:boto:Token: None 2022-09-02T04:20:27.591 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-7/ 2022-09-02T04:20:27.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:liW89ewI18w07KCOGuMDYn/I19U= 2022-09-02T04:20:27.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:liW89ewI18w07KCOGuMDYn/I19U=', 'Content-Length': '0'} 2022-09-02T04:20:27.597 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000006573cf08166ca09e-006311848b-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.599 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:27.600 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:27.600 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.601 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:27.603 DEBUG:boto:Data: 2022-09-02T04:20:27.604 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.606 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.607 DEBUG:boto:Params: {} 2022-09-02T04:20:27.608 DEBUG:boto:Token: None 2022-09-02T04:20:27.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-8/ 2022-09-02T04:20:27.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aboXPjT0I03DBvuE+wnQlDCg+58= 2022-09-02T04:20:27.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aboXPjT0I03DBvuE+wnQlDCg+58=', 'Content-Length': '0'} 2022-09-02T04:20:27.614 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000002a1a564aa5c5bf84-006311848b-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.615 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:27.615 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:27.616 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.618 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:27.618 DEBUG:boto:Data: 2022-09-02T04:20:27.619 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.621 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.622 DEBUG:boto:Params: {} 2022-09-02T04:20:27.623 DEBUG:boto:Token: None 2022-09-02T04:20:27.624 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-9/ 2022-09-02T04:20:27.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iEfH4/chc8l3EY7nf3gbUDjAJFA= 2022-09-02T04:20:27.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iEfH4/chc8l3EY7nf3gbUDjAJFA=', 'Content-Length': '0'} 2022-09-02T04:20:27.628 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000d68910fe2754919d-006311848b-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.629 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:27.630 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:27.631 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.632 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:27.633 DEBUG:boto:Data: 2022-09-02T04:20:27.633 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.634 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:27.635 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.636 DEBUG:boto:Params: {} 2022-09-02T04:20:27.637 DEBUG:boto:Token: None 2022-09-02T04:20:27.639 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-7/ 2022-09-02T04:20:27.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:liW89ewI18w07KCOGuMDYn/I19U= 2022-09-02T04:20:27.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:liW89ewI18w07KCOGuMDYn/I19U=', 'Content-Length': '0'} 2022-09-02T04:20:27.643 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000003a7c4982018b84dd-006311848b-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.644 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:27.645 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:27.646 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.647 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:27.647 DEBUG:boto:Data: 2022-09-02T04:20:27.648 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:27.649 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.650 DEBUG:boto:Params: {} 2022-09-02T04:20:27.651 DEBUG:boto:Token: None 2022-09-02T04:20:27.652 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-8/ 2022-09-02T04:20:27.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aboXPjT0I03DBvuE+wnQlDCg+58= 2022-09-02T04:20:27.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aboXPjT0I03DBvuE+wnQlDCg+58=', 'Content-Length': '0'} 2022-09-02T04:20:27.656 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx0000076f2fbd1b91b02f5-006311848b-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.657 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:27.657 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:27.658 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.659 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:27.660 DEBUG:boto:Data: 2022-09-02T04:20:27.661 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:27.662 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.663 DEBUG:boto:Params: {} 2022-09-02T04:20:27.664 DEBUG:boto:Token: None 2022-09-02T04:20:27.666 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-9/ 2022-09-02T04:20:27.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iEfH4/chc8l3EY7nf3gbUDjAJFA= 2022-09-02T04:20:27.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iEfH4/chc8l3EY7nf3gbUDjAJFA=', 'Content-Length': '0'} 2022-09-02T04:20:27.670 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000acb24b78a82b38a9-006311848b-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.672 DEBUG:boto:path=/xljbkf-7/ 2022-09-02T04:20:27.673 DEBUG:boto:auth_path=/xljbkf-7/ 2022-09-02T04:20:27.673 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.674 DEBUG:boto:Path: /xljbkf-7/ 2022-09-02T04:20:27.675 DEBUG:boto:Data: 2022-09-02T04:20:27.677 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:27.679 DEBUG:boto:Port: 8001 2022-09-02T04:20:27.680 DEBUG:boto:Params: {} 2022-09-02T04:20:27.681 DEBUG:boto:Token: None 2022-09-02T04:20:27.682 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-7/ 2022-09-02T04:20:27.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:liW89ewI18w07KCOGuMDYn/I19U= 2022-09-02T04:20:27.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:liW89ewI18w07KCOGuMDYn/I19U=', 'Content-Length': '0'} 2022-09-02T04:20:27.697 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000bcaeefdb0f5176ae-006311848b-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.698 DEBUG:boto:path=/xljbkf-8/ 2022-09-02T04:20:27.699 DEBUG:boto:auth_path=/xljbkf-8/ 2022-09-02T04:20:27.700 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.701 DEBUG:boto:Path: /xljbkf-8/ 2022-09-02T04:20:27.702 DEBUG:boto:Data: 2022-09-02T04:20:27.703 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:27.706 DEBUG:boto:Port: 8001 2022-09-02T04:20:27.712 DEBUG:boto:Params: {} 2022-09-02T04:20:27.713 DEBUG:boto:Token: None 2022-09-02T04:20:27.714 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-8/ 2022-09-02T04:20:27.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aboXPjT0I03DBvuE+wnQlDCg+58= 2022-09-02T04:20:27.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aboXPjT0I03DBvuE+wnQlDCg+58=', 'Content-Length': '0'} 2022-09-02T04:20:27.717 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx000000f7648eeff83b398-006311848b-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.719 DEBUG:boto:path=/xljbkf-9/ 2022-09-02T04:20:27.720 DEBUG:boto:auth_path=/xljbkf-9/ 2022-09-02T04:20:27.721 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.721 DEBUG:boto:Path: /xljbkf-9/ 2022-09-02T04:20:27.722 DEBUG:boto:Data: 2022-09-02T04:20:27.722 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:27.723 DEBUG:boto:Port: 8001 2022-09-02T04:20:27.724 DEBUG:boto:Params: {} 2022-09-02T04:20:27.725 DEBUG:boto:Token: None 2022-09-02T04:20:27.726 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-9/ 2022-09-02T04:20:27.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iEfH4/chc8l3EY7nf3gbUDjAJFA= 2022-09-02T04:20:27.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iEfH4/chc8l3EY7nf3gbUDjAJFA=', 'Content-Length': '0'} 2022-09-02T04:20:27.731 DEBUG:boto:Response headers: [('Content-Length', '231'), ('x-amz-request-id', 'tx00000a70e5b6f22f2f3a1-006311848b-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2022-09-02T04:20:27.733 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-10 2022-09-02T04:20:27.734 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:20:27.735 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:20:27.736 DEBUG:boto:Method: PUT 2022-09-02T04:20:27.737 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:20:27.739 DEBUG:boto:Data: 2022-09-02T04:20:27.740 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.741 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.742 DEBUG:boto:Params: {} 2022-09-02T04:20:27.742 DEBUG:boto:Token: None 2022-09-02T04:20:27.743 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-10/ 2022-09-02T04:20:27.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zwkiu2cPLN9f6Zn1TNF4KW2nW8Q= 2022-09-02T04:20:27.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zwkiu2cPLN9f6Zn1TNF4KW2nW8Q=', 'Content-Length': '0'} 2022-09-02T04:20:27.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000083e40f9ed3ecf224-006311848b-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.759 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-11 2022-09-02T04:20:27.760 DEBUG:boto:path=/xljbkf-11/ 2022-09-02T04:20:27.762 DEBUG:boto:auth_path=/xljbkf-11/ 2022-09-02T04:20:27.762 DEBUG:boto:Method: PUT 2022-09-02T04:20:27.763 DEBUG:boto:Path: /xljbkf-11/ 2022-09-02T04:20:27.764 DEBUG:boto:Data: 2022-09-02T04:20:27.765 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:27.767 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.768 DEBUG:boto:Params: {} 2022-09-02T04:20:27.769 DEBUG:boto:Token: None 2022-09-02T04:20:27.770 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-11/ 2022-09-02T04:20:27.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E2wiMsiu5+7YvosWopz6/C8un4k= 2022-09-02T04:20:27.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E2wiMsiu5+7YvosWopz6/C8un4k=', 'Content-Length': '0'} 2022-09-02T04:20:27.796 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000059ce811521c0b36e-006311848b-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.797 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-12 2022-09-02T04:20:27.798 DEBUG:boto:path=/xljbkf-12/ 2022-09-02T04:20:27.798 DEBUG:boto:auth_path=/xljbkf-12/ 2022-09-02T04:20:27.799 DEBUG:boto:Method: PUT 2022-09-02T04:20:27.799 DEBUG:boto:Path: /xljbkf-12/ 2022-09-02T04:20:27.802 DEBUG:boto:Data: 2022-09-02T04:20:27.803 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:27.805 DEBUG:boto:Port: 8001 2022-09-02T04:20:27.806 DEBUG:boto:Params: {} 2022-09-02T04:20:27.807 DEBUG:boto:Token: None 2022-09-02T04:20:27.808 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-12/ 2022-09-02T04:20:27.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kEBjNCTRCRix0B2Wvj8YH0maMTo= 2022-09-02T04:20:27.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kEBjNCTRCRix0B2Wvj8YH0maMTo=', 'Content-Length': '0'} 2022-09-02T04:20:27.843 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000365ad6ff32479f68-006311848b-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.844 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:20:27.844 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:20:27.845 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.845 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:20:27.846 DEBUG:boto:Data: 2022-09-02T04:20:27.847 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.848 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.848 DEBUG:boto:Params: {} 2022-09-02T04:20:27.850 DEBUG:boto:Token: None 2022-09-02T04:20:27.854 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-10/ 2022-09-02T04:20:27.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsRxS5CRbmZTTlBSl+yNFz3A5yM= 2022-09-02T04:20:27.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsRxS5CRbmZTTlBSl+yNFz3A5yM=', 'Content-Length': '0'} 2022-09-02T04:20:27.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098a65fcec32106f5-006311848b-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.860 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:20:27.862 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:20:27.863 DEBUG:boto:Method: PUT 2022-09-02T04:20:27.863 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:20:27.864 DEBUG:boto:Data: 2022-09-02T04:20:27.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:27.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.866 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.867 DEBUG:boto:Params: {} 2022-09-02T04:20:27.868 DEBUG:boto:Token: None 2022-09-02T04:20:27.868 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-10/myobj 2022-09-02T04:20:27.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHW+MV/lN/01iu/Ae17o36ELHlg= 2022-09-02T04:20:27.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHW+MV/lN/01iu/Ae17o36ELHlg='} 2022-09-02T04:20:27.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005101542b1877390-006311848b-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.902 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:20:27.903 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:20:27.903 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.904 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:20:27.904 DEBUG:boto:Data: 2022-09-02T04:20:27.904 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.906 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.907 DEBUG:boto:Params: {} 2022-09-02T04:20:27.908 DEBUG:boto:Token: None 2022-09-02T04:20:27.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-10/ 2022-09-02T04:20:27.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsRxS5CRbmZTTlBSl+yNFz3A5yM= 2022-09-02T04:20:27.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsRxS5CRbmZTTlBSl+yNFz3A5yM=', 'Content-Length': '0'} 2022-09-02T04:20:27.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d2ef037f47e77c5-006311848b-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.913 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:20:27.913 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:20:27.914 DEBUG:boto:Method: PUT 2022-09-02T04:20:27.915 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:20:27.915 DEBUG:boto:Data: 2022-09-02T04:20:27.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:27.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.918 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.919 DEBUG:boto:Params: {} 2022-09-02T04:20:27.921 DEBUG:boto:Token: None 2022-09-02T04:20:27.922 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-10/_myobj 2022-09-02T04:20:27.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A6GqWYyD5m/E51gIem+c6NQV1Mk= 2022-09-02T04:20:27.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A6GqWYyD5m/E51gIem+c6NQV1Mk='} 2022-09-02T04:20:27.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043fcf38197baf444-006311848b-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.979 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:20:27.980 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:20:27.980 DEBUG:boto:Method: HEAD 2022-09-02T04:20:27.981 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:20:27.982 DEBUG:boto:Data: 2022-09-02T04:20:27.983 DEBUG:boto:Headers: {} 2022-09-02T04:20:27.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:27.984 DEBUG:boto:Port: 8000 2022-09-02T04:20:27.984 DEBUG:boto:Params: {} 2022-09-02T04:20:27.985 DEBUG:boto:Token: None 2022-09-02T04:20:27.986 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:27 GMT /xljbkf-10/ 2022-09-02T04:20:27.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsRxS5CRbmZTTlBSl+yNFz3A5yM= 2022-09-02T04:20:27.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsRxS5CRbmZTTlBSl+yNFz3A5yM=', 'Content-Length': '0'} 2022-09-02T04:20:27.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0719f8c65bc2a7-006311848b-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:27.991 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:20:27.991 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:20:27.992 DEBUG:boto:Method: PUT 2022-09-02T04:20:27.992 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:20:27.993 DEBUG:boto:Data: 2022-09-02T04:20:27.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:28.002 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.002 DEBUG:boto:Params: {} 2022-09-02T04:20:28.003 DEBUG:boto:Token: None 2022-09-02T04:20:28.004 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-10/%3A 2022-09-02T04:20:28.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QvzoPkHcUauItdUMI1bwEXxdGHI= 2022-09-02T04:20:28.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QvzoPkHcUauItdUMI1bwEXxdGHI='} 2022-09-02T04:20:28.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3f90fa459032d61-006311848c-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.017 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:20:28.017 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:20:28.018 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.018 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:20:28.019 DEBUG:boto:Data: 2022-09-02T04:20:28.020 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:28.029 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.029 DEBUG:boto:Params: {} 2022-09-02T04:20:28.031 DEBUG:boto:Token: None 2022-09-02T04:20:28.032 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-10/ 2022-09-02T04:20:28.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qOWfnVf5CK80BwhUo0ipVhMthWI= 2022-09-02T04:20:28.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qOWfnVf5CK80BwhUo0ipVhMthWI=', 'Content-Length': '0'} 2022-09-02T04:20:28.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a954cd37578bc24-006311848c-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.041 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:20:28.042 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:20:28.043 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.044 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:20:28.045 DEBUG:boto:Data: 2022-09-02T04:20:28.046 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:20:28.047 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.049 DEBUG:boto:Params: {} 2022-09-02T04:20:28.051 DEBUG:boto:Token: None 2022-09-02T04:20:28.052 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-10/%26 2022-09-02T04:20:28.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vG8wz5SSVKmgfRzUr6M4j2sQ7LM= 2022-09-02T04:20:28.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vG8wz5SSVKmgfRzUr6M4j2sQ7LM='} 2022-09-02T04:20:28.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077b1b64789a70a2f-006311848c-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.064 DEBUG:boto:path=/xljbkf-11/ 2022-09-02T04:20:28.065 DEBUG:boto:auth_path=/xljbkf-11/ 2022-09-02T04:20:28.071 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.072 DEBUG:boto:Path: /xljbkf-11/ 2022-09-02T04:20:28.074 DEBUG:boto:Data: 2022-09-02T04:20:28.075 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.076 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.077 DEBUG:boto:Params: {} 2022-09-02T04:20:28.078 DEBUG:boto:Token: None 2022-09-02T04:20:28.078 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/ 2022-09-02T04:20:28.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc= 2022-09-02T04:20:28.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc=', 'Content-Length': '0'} 2022-09-02T04:20:28.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fa200692b0b9419-006311848c-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.085 DEBUG:boto:path=/xljbkf-11/myobj 2022-09-02T04:20:28.087 DEBUG:boto:auth_path=/xljbkf-11/myobj 2022-09-02T04:20:28.087 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.088 DEBUG:boto:Path: /xljbkf-11/myobj 2022-09-02T04:20:28.089 DEBUG:boto:Data: 2022-09-02T04:20:28.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.094 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.095 DEBUG:boto:Params: {} 2022-09-02T04:20:28.096 DEBUG:boto:Token: None 2022-09-02T04:20:28.097 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/myobj 2022-09-02T04:20:28.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hWHoSLzl8qQRLS5MJzUzVw0c9Ww= 2022-09-02T04:20:28.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hWHoSLzl8qQRLS5MJzUzVw0c9Ww='} 2022-09-02T04:20:28.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a897252a13bf246-006311848c-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.121 DEBUG:boto:path=/xljbkf-11/ 2022-09-02T04:20:28.122 DEBUG:boto:auth_path=/xljbkf-11/ 2022-09-02T04:20:28.123 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.124 DEBUG:boto:Path: /xljbkf-11/ 2022-09-02T04:20:28.126 DEBUG:boto:Data: 2022-09-02T04:20:28.126 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.127 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.128 DEBUG:boto:Params: {} 2022-09-02T04:20:28.129 DEBUG:boto:Token: None 2022-09-02T04:20:28.129 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/ 2022-09-02T04:20:28.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc= 2022-09-02T04:20:28.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc=', 'Content-Length': '0'} 2022-09-02T04:20:28.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035225c0271049a05-006311848c-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.141 DEBUG:boto:path=/xljbkf-11/_myobj 2022-09-02T04:20:28.142 DEBUG:boto:auth_path=/xljbkf-11/_myobj 2022-09-02T04:20:28.143 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.143 DEBUG:boto:Path: /xljbkf-11/_myobj 2022-09-02T04:20:28.144 DEBUG:boto:Data: 2022-09-02T04:20:28.146 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.148 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.149 DEBUG:boto:Params: {} 2022-09-02T04:20:28.150 DEBUG:boto:Token: None 2022-09-02T04:20:28.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/_myobj 2022-09-02T04:20:28.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ujiGrSeuVy6cyfpQv8wP1oIYMIo= 2022-09-02T04:20:28.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ujiGrSeuVy6cyfpQv8wP1oIYMIo='} 2022-09-02T04:20:28.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033dd9ec5664fae16-006311848c-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.200 DEBUG:boto:path=/xljbkf-11/ 2022-09-02T04:20:28.201 DEBUG:boto:auth_path=/xljbkf-11/ 2022-09-02T04:20:28.202 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.202 DEBUG:boto:Path: /xljbkf-11/ 2022-09-02T04:20:28.203 DEBUG:boto:Data: 2022-09-02T04:20:28.204 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.205 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.205 DEBUG:boto:Params: {} 2022-09-02T04:20:28.206 DEBUG:boto:Token: None 2022-09-02T04:20:28.207 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/ 2022-09-02T04:20:28.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc= 2022-09-02T04:20:28.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc=', 'Content-Length': '0'} 2022-09-02T04:20:28.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1cb07ec49c21a04-006311848c-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.218 DEBUG:boto:path=/xljbkf-11/%3A 2022-09-02T04:20:28.219 DEBUG:boto:auth_path=/xljbkf-11/%3A 2022-09-02T04:20:28.219 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.222 DEBUG:boto:Path: /xljbkf-11/%3A 2022-09-02T04:20:28.223 DEBUG:boto:Data: 2022-09-02T04:20:28.224 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.228 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.229 DEBUG:boto:Params: {} 2022-09-02T04:20:28.229 DEBUG:boto:Token: None 2022-09-02T04:20:28.230 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/%3A 2022-09-02T04:20:28.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SURDixvwB85Qrlzwo9EY6U3I1JY= 2022-09-02T04:20:28.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SURDixvwB85Qrlzwo9EY6U3I1JY='} 2022-09-02T04:20:28.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d960262e89584466-006311848c-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.251 DEBUG:boto:path=/xljbkf-11/ 2022-09-02T04:20:28.252 DEBUG:boto:auth_path=/xljbkf-11/ 2022-09-02T04:20:28.253 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.254 DEBUG:boto:Path: /xljbkf-11/ 2022-09-02T04:20:28.257 DEBUG:boto:Data: 2022-09-02T04:20:28.258 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.262 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.264 DEBUG:boto:Params: {} 2022-09-02T04:20:28.266 DEBUG:boto:Token: None 2022-09-02T04:20:28.267 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/ 2022-09-02T04:20:28.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc= 2022-09-02T04:20:28.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDXwudKT26tuqeIJYZyWyWYUflc=', 'Content-Length': '0'} 2022-09-02T04:20:28.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2036d7cb60825dd-006311848c-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.275 DEBUG:boto:path=/xljbkf-11/%26 2022-09-02T04:20:28.277 DEBUG:boto:auth_path=/xljbkf-11/%26 2022-09-02T04:20:28.278 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.279 DEBUG:boto:Path: /xljbkf-11/%26 2022-09-02T04:20:28.280 DEBUG:boto:Data: 2022-09-02T04:20:28.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:20:28.285 DEBUG:boto:Port: 8000 2022-09-02T04:20:28.288 DEBUG:boto:Params: {} 2022-09-02T04:20:28.289 DEBUG:boto:Token: None 2022-09-02T04:20:28.290 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-11/%26 2022-09-02T04:20:28.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E2ZuZNsg1V0fipITw6apN21/ww8= 2022-09-02T04:20:28.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E2ZuZNsg1V0fipITw6apN21/ww8='} 2022-09-02T04:20:28.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014d04ab01a6dfddf-006311848c-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.307 DEBUG:boto:path=/xljbkf-12/ 2022-09-02T04:20:28.308 DEBUG:boto:auth_path=/xljbkf-12/ 2022-09-02T04:20:28.311 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.312 DEBUG:boto:Path: /xljbkf-12/ 2022-09-02T04:20:28.313 DEBUG:boto:Data: 2022-09-02T04:20:28.314 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.317 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.318 DEBUG:boto:Params: {} 2022-09-02T04:20:28.319 DEBUG:boto:Token: None 2022-09-02T04:20:28.321 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/ 2022-09-02T04:20:28.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w= 2022-09-02T04:20:28.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w=', 'Content-Length': '0'} 2022-09-02T04:20:28.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e569fe080da543c4-006311848c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.336 DEBUG:boto:path=/xljbkf-12/myobj 2022-09-02T04:20:28.337 DEBUG:boto:auth_path=/xljbkf-12/myobj 2022-09-02T04:20:28.344 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.345 DEBUG:boto:Path: /xljbkf-12/myobj 2022-09-02T04:20:28.348 DEBUG:boto:Data: 2022-09-02T04:20:28.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.352 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.352 DEBUG:boto:Params: {} 2022-09-02T04:20:28.353 DEBUG:boto:Token: None 2022-09-02T04:20:28.354 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/myobj 2022-09-02T04:20:28.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rAD4wzy9knyffP1OjDJ4tguLANA= 2022-09-02T04:20:28.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rAD4wzy9knyffP1OjDJ4tguLANA='} 2022-09-02T04:20:28.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000146d4541d20721ae-006311848c-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.377 DEBUG:boto:path=/xljbkf-12/ 2022-09-02T04:20:28.385 DEBUG:boto:auth_path=/xljbkf-12/ 2022-09-02T04:20:28.387 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.388 DEBUG:boto:Path: /xljbkf-12/ 2022-09-02T04:20:28.389 DEBUG:boto:Data: 2022-09-02T04:20:28.389 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.390 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.391 DEBUG:boto:Params: {} 2022-09-02T04:20:28.392 DEBUG:boto:Token: None 2022-09-02T04:20:28.393 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/ 2022-09-02T04:20:28.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w= 2022-09-02T04:20:28.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w=', 'Content-Length': '0'} 2022-09-02T04:20:28.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dadacdcf3ab7c29-006311848c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.400 DEBUG:boto:path=/xljbkf-12/_myobj 2022-09-02T04:20:28.401 DEBUG:boto:auth_path=/xljbkf-12/_myobj 2022-09-02T04:20:28.402 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.404 DEBUG:boto:Path: /xljbkf-12/_myobj 2022-09-02T04:20:28.406 DEBUG:boto:Data: 2022-09-02T04:20:28.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.409 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.411 DEBUG:boto:Params: {} 2022-09-02T04:20:28.412 DEBUG:boto:Token: None 2022-09-02T04:20:28.413 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/_myobj 2022-09-02T04:20:28.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e1ABcxK3npzf3xM/mFfPIZ90aaM= 2022-09-02T04:20:28.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e1ABcxK3npzf3xM/mFfPIZ90aaM='} 2022-09-02T04:20:28.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dac66d6b475d1478-006311848c-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.428 DEBUG:boto:path=/xljbkf-12/ 2022-09-02T04:20:28.429 DEBUG:boto:auth_path=/xljbkf-12/ 2022-09-02T04:20:28.433 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.435 DEBUG:boto:Path: /xljbkf-12/ 2022-09-02T04:20:28.436 DEBUG:boto:Data: 2022-09-02T04:20:28.438 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.440 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.441 DEBUG:boto:Params: {} 2022-09-02T04:20:28.442 DEBUG:boto:Token: None 2022-09-02T04:20:28.443 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/ 2022-09-02T04:20:28.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w= 2022-09-02T04:20:28.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w=', 'Content-Length': '0'} 2022-09-02T04:20:28.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000def996af2c44d26f-006311848c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.451 DEBUG:boto:path=/xljbkf-12/%3A 2022-09-02T04:20:28.452 DEBUG:boto:auth_path=/xljbkf-12/%3A 2022-09-02T04:20:28.452 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.453 DEBUG:boto:Path: /xljbkf-12/%3A 2022-09-02T04:20:28.453 DEBUG:boto:Data: 2022-09-02T04:20:28.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.455 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.455 DEBUG:boto:Params: {} 2022-09-02T04:20:28.456 DEBUG:boto:Token: None 2022-09-02T04:20:28.459 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/%3A 2022-09-02T04:20:28.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1wwayrbfciBXgBYBDUX0NKmQS3c= 2022-09-02T04:20:28.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1wwayrbfciBXgBYBDUX0NKmQS3c='} 2022-09-02T04:20:28.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082f50cedc0759705-006311848c-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.512 DEBUG:boto:path=/xljbkf-12/ 2022-09-02T04:20:28.513 DEBUG:boto:auth_path=/xljbkf-12/ 2022-09-02T04:20:28.514 DEBUG:boto:Method: HEAD 2022-09-02T04:20:28.515 DEBUG:boto:Path: /xljbkf-12/ 2022-09-02T04:20:28.519 DEBUG:boto:Data: 2022-09-02T04:20:28.520 DEBUG:boto:Headers: {} 2022-09-02T04:20:28.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.522 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.524 DEBUG:boto:Params: {} 2022-09-02T04:20:28.525 DEBUG:boto:Token: None 2022-09-02T04:20:28.526 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/ 2022-09-02T04:20:28.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w= 2022-09-02T04:20:28.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/XBkqjuQv5HVyYzfoyUnlkBFG+w=', 'Content-Length': '0'} 2022-09-02T04:20:28.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c06f628f1fa50036-006311848c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.554 DEBUG:boto:path=/xljbkf-12/%26 2022-09-02T04:20:28.556 DEBUG:boto:auth_path=/xljbkf-12/%26 2022-09-02T04:20:28.557 DEBUG:boto:Method: PUT 2022-09-02T04:20:28.561 DEBUG:boto:Path: /xljbkf-12/%26 2022-09-02T04:20:28.562 DEBUG:boto:Data: 2022-09-02T04:20:28.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:20:28.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:20:28.564 DEBUG:boto:Port: 8001 2022-09-02T04:20:28.565 DEBUG:boto:Params: {} 2022-09-02T04:20:28.565 DEBUG:boto:Token: None 2022-09-02T04:20:28.566 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:20:28 GMT /xljbkf-12/%26 2022-09-02T04:20:28.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:46Z41akDKJRpbPBeqMgsbOS5t7Q= 2022-09-02T04:20:28.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:20:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:46Z41akDKJRpbPBeqMgsbOS5t7Q='} 2022-09-02T04:20:28.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000076ad5e45055c929-006311848c-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:20:28.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.576 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:28.629 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:28.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:28.654 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:20:28.672 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:20:28.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.673 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:28.674 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:20:28.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:28.677 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:20:28.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092427.823736_63.1", 2022-09-02T04:20:28.678 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:27.823736Z" 2022-09-02T04:20:28.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.680 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:28.680 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:20:28.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:28.681 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:20:28.682 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:28.683 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:20:28.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.686 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:28.744 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:28.745 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:28.783 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:28.780+0000 7fa7f9f3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:28.782+0000 7fa7f9f3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:28.794 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:28.795 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:28.798 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:28.798 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:28.799 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:28.799 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:28.802 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:28.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:28.802 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:28.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:28.803 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:28.803 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:28.803 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:28.804 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:28.804 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:28.804 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:28.805 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:28.805 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:28.805 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:28.806 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:28.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:28.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:28.806 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:28.807 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:28.807 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:28.807 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:28.808 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:28.808 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:28.808 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:28.808 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:28.809 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:28.810 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:28.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:28.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:28.812 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:28.812 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:28.813 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:28.813 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092427.823736_63.1", 2022-09-02T04:20:28.813 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:28.814 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:28.814 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:28.814 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.823736Z", 2022-09-02T04:20:28.814 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:28.815 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:28.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:28.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:28.816 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:28.816 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:28.816 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:28.817 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:28.817 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:28.817 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:28.818 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:28.818 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:20:28.818 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:28.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:28.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:28.819 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:28.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:28.820 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:28.820 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:28.820 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:28.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:28.821 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:28Z" 2022-09-02T04:20:28.821 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:28.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092427.823736_63.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:27.823736Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:28Z" } 2022-09-02T04:20:28.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092427.823736_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:27.823736Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:20:28.825 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:28.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:28.826 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:20:28.827 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:20:28.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.829 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:28.879 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:28.879 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:28.921 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:28.921+0000 7fd7ca4f8600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:28.922 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:20:28.921+0000 7fd7ca4f8600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:28.939 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:20:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:20:28.940 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:20:28.941 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:20:28.942 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:20:28.943 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:28.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.945 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:20:28.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.956 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:20:28.957 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:28.958 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:28.958 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:28.959 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:28.960 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:20:28.960 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:28.961 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:28.962 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:28.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:28.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.964 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:20:28.964 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:28.965 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:28.965 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:20:28.965 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:28.966 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:28.966 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:28.967 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:20:28.967 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:28.967 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:20:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092427.823736_63.1", 2022-09-02T04:20:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:28.971 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:28.972 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:28.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.823736Z", 2022-09-02T04:20:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:20:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:20:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:20:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:20:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:20:28.978 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:20:28.978 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:20:28.982 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:20:28.982 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:20:28.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:28.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:20:28.984 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:20:28.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.985 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:20:28.985 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:20:28.986 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:20:28.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:20:28.987 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:20:28Z" 2022-09-02T04:20:28.987 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:20:28.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092427.823736_63.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:27.823736Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:28Z" } 2022-09-02T04:20:28.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092427.823736_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:27.823736Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:20:28.992 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:28.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:28.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:20:28.996 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:20:28.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:28.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:20:29.000 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:20:29.054 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:29.060 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:29.081 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:29.080+0000 7fd7fdc24600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:29.082 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:29.081+0000 7fd7fdc24600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:20:29.099 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:20:29.099 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:20:29.100 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:20:29.100 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:20:29.100 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:20:29.101 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:20:29.101 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:29.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:29.101 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:20:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:20:29.107 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:20:29.108 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:20:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:20:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:29.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:20:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:20:29.112 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:29.113 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:29.113 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:20:29.114 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:29.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:29.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:29.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:29.121 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:20:29.121 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:29.122 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:29.122 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092427.823736_63.1", 2022-09-02T04:20:29.123 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:29.123 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:29.125 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:29.125 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.823736Z", 2022-09-02T04:20:29.126 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:29.126 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:29.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:29.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:20:29.128 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:20:29.129 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:20:29.130 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:20:29.131 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:20:29.131 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:20:29.131 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:20:29.132 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:20:29.132 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:20:29.133 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:20:29.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:29.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:20:29.135 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:20:29.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:29.136 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:20:29.136 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:20:29.137 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:20:29.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:20:29.138 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:20:29Z" 2022-09-02T04:20:29.138 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:20:29.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092427.823736_63.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:27.823736Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:20:29Z" } 2022-09-02T04:20:29.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092427.823736_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:27.823736Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:20:29.145 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:29.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092427.823736_63.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:20:29.149 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:20:29.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:20:29.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:20:29.152 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:20:29.209 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:20:29.209 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:20:29.261 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:29.260+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:20:34.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:34.261+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:20:39.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:39.261+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:20:44.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:44.261+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:20:49.263 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:49.261+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:20:54.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:54.262+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:20:59.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:20:59.263+0000 7efee6caa600 1 waiting to reach incremental sync.. 2022-09-02T04:21:04.267 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:04.264+0000 7efee6caa600 1 bucket sync caught up with source: 2022-09-02T04:21:04.267 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:04.268 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:04.268 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:04.264+0000 7efee6caa600 0 bucket checkpoint complete 2022-09-02T04:21:04.286 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-10 zones={test-zone2, test-zone1} 2022-09-02T04:21:04.287 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.288 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.288 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.289 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:21:04.289 DEBUG:boto:Data: 2022-09-02T04:21:04.290 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.291 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.291 DEBUG:boto:Params: {} 2022-09-02T04:21:04.292 DEBUG:boto:Token: None 2022-09-02T04:21:04.293 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PwlB/yCRHY43/ieasjJ5N+fmdfU= 2022-09-02T04:21:04.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PwlB/yCRHY43/ieasjJ5N+fmdfU=', 'Content-Length': '0'} 2022-09-02T04:21:04.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009873267efdfe85a0-00631184b0-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.298 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.299 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.299 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.300 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:21:04.301 DEBUG:boto:Data: 2022-09-02T04:21:04.302 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.304 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.305 DEBUG:boto:Params: {} 2022-09-02T04:21:04.306 DEBUG:boto:Token: None 2022-09-02T04:21:04.307 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PwlB/yCRHY43/ieasjJ5N+fmdfU= 2022-09-02T04:21:04.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PwlB/yCRHY43/ieasjJ5N+fmdfU=', 'Content-Length': '0'} 2022-09-02T04:21:04.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7868369e2250aac-00631184b0-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.321 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:21:04.321 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.323 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.323 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.324 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.325 DEBUG:boto:Method: GET 2022-09-02T04:21:04.326 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.327 DEBUG:boto:Data: 2022-09-02T04:21:04.327 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.328 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.329 DEBUG:boto:Params: {} 2022-09-02T04:21:04.334 DEBUG:boto:Token: None 2022-09-02T04:21:04.335 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/?versions 2022-09-02T04:21:04.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0= 2022-09-02T04:21:04.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0=', 'Content-Length': '0'} 2022-09-02T04:21:04.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b57c9e293b044c9-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.340 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.343 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-02T04:21:04.344 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-02T04:21:04.345 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-02T04:21:04.345 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:21:04.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:21:04.347 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.348 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.349 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.350 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.351 DEBUG:boto:Method: GET 2022-09-02T04:21:04.352 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.353 DEBUG:boto:Data: 2022-09-02T04:21:04.354 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.355 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.356 DEBUG:boto:Params: {} 2022-09-02T04:21:04.357 DEBUG:boto:Token: None 2022-09-02T04:21:04.358 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/?versions 2022-09-02T04:21:04.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0= 2022-09-02T04:21:04.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0=', 'Content-Length': '0'} 2022-09-02T04:21:04.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008da3a3ec2e31830b-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.365 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.366 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-02T04:21:04.367 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-02T04:21:04.368 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-02T04:21:04.369 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:21:04.371 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.371 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.372 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.378 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.379 DEBUG:boto:Method: GET 2022-09-02T04:21:04.379 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.380 DEBUG:boto:Data: 2022-09-02T04:21:04.380 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.381 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.382 DEBUG:boto:Params: {} 2022-09-02T04:21:04.383 DEBUG:boto:Token: None 2022-09-02T04:21:04.383 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/?versions 2022-09-02T04:21:04.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0= 2022-09-02T04:21:04.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0=', 'Content-Length': '0'} 2022-09-02T04:21:04.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18ac7d2b66233d4-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.390 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.392 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.393 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.394 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.395 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.396 DEBUG:boto:Method: GET 2022-09-02T04:21:04.397 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:04.398 DEBUG:boto:Data: 2022-09-02T04:21:04.399 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.401 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.402 DEBUG:boto:Params: {} 2022-09-02T04:21:04.403 DEBUG:boto:Token: None 2022-09-02T04:21:04.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/?versions 2022-09-02T04:21:04.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0= 2022-09-02T04:21:04.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Doe5EjjZD79T/fygbET2UtSpcv0=', 'Content-Length': '0'} 2022-09-02T04:21:04.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3947243ff652cc2-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.412 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-02T04:21:04.417 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.418 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.419 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.419 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.420 DEBUG:boto:Method: GET 2022-09-02T04:21:04.420 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.421 DEBUG:boto:Data: 2022-09-02T04:21:04.422 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.423 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.423 DEBUG:boto:Params: {} 2022-09-02T04:21:04.424 DEBUG:boto:Token: None 2022-09-02T04:21:04.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0= 2022-09-02T04:21:04.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0=', 'Content-Length': '0'} 2022-09-02T04:21:04.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076f23e2757d5f02b-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.432 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.433 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.435 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.435 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.437 DEBUG:boto:Method: GET 2022-09-02T04:21:04.439 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.439 DEBUG:boto:Data: 2022-09-02T04:21:04.441 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.443 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.444 DEBUG:boto:Params: {} 2022-09-02T04:21:04.444 DEBUG:boto:Token: None 2022-09-02T04:21:04.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0= 2022-09-02T04:21:04.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0=', 'Content-Length': '0'} 2022-09-02T04:21:04.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfbaf28f162c5dd6-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.456 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.457 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.460 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.461 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.462 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.465 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.466 DEBUG:boto:Data: 2022-09-02T04:21:04.467 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.469 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.470 DEBUG:boto:Params: {} 2022-09-02T04:21:04.471 DEBUG:boto:Token: None 2022-09-02T04:21:04.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FqJ2pBkmdW2yK9gPPHwakvy5swU= 2022-09-02T04:21:04.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FqJ2pBkmdW2yK9gPPHwakvy5swU=', 'Content-Length': '0'} 2022-09-02T04:21:04.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2192d53bd7c8eb0-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.484 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.484 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.485 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.485 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.486 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.487 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.487 DEBUG:boto:Data: 2022-09-02T04:21:04.488 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.489 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.489 DEBUG:boto:Params: {} 2022-09-02T04:21:04.490 DEBUG:boto:Token: None 2022-09-02T04:21:04.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FqJ2pBkmdW2yK9gPPHwakvy5swU= 2022-09-02T04:21:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FqJ2pBkmdW2yK9gPPHwakvy5swU=', 'Content-Length': '0'} 2022-09-02T04:21:04.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067f9dc0769d32390-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-02T04:21:04.498 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.498 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.499 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.500 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.500 DEBUG:boto:Method: GET 2022-09-02T04:21:04.501 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.501 DEBUG:boto:Data: 2022-09-02T04:21:04.502 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.504 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.504 DEBUG:boto:Params: {} 2022-09-02T04:21:04.505 DEBUG:boto:Token: None 2022-09-02T04:21:04.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0= 2022-09-02T04:21:04.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0=', 'Content-Length': '0'} 2022-09-02T04:21:04.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000838707701dd8c8d4-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.554 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.555 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.556 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.558 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.559 DEBUG:boto:Method: GET 2022-09-02T04:21:04.560 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.561 DEBUG:boto:Data: 2022-09-02T04:21:04.563 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.568 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.569 DEBUG:boto:Params: {} 2022-09-02T04:21:04.577 DEBUG:boto:Token: None 2022-09-02T04:21:04.578 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:04.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0= 2022-09-02T04:21:04.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fy1OVUrnbYesaseAhLmqo+Mj/d0=', 'Content-Length': '0'} 2022-09-02T04:21:04.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079484d1f0e5eec9b-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.585 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.586 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.587 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.587 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.588 DEBUG:boto:Method: GET 2022-09-02T04:21:04.589 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:04.591 DEBUG:boto:Data: 2022-09-02T04:21:04.592 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.597 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.598 DEBUG:boto:Params: {} 2022-09-02T04:21:04.599 DEBUG:boto:Token: None 2022-09-02T04:21:04.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw= 2022-09-02T04:21:04.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw=', 'Content-Length': '0'} 2022-09-02T04:21:04.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fda6cd2fad5dd80-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.608 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.610 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.611 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.612 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.613 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.615 DEBUG:boto:Method: GET 2022-09-02T04:21:04.618 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:04.619 DEBUG:boto:Data: 2022-09-02T04:21:04.620 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.622 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.623 DEBUG:boto:Params: {} 2022-09-02T04:21:04.624 DEBUG:boto:Token: None 2022-09-02T04:21:04.625 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw= 2022-09-02T04:21:04.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw=', 'Content-Length': '0'} 2022-09-02T04:21:04.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066e3f5ab649d77db-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.631 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.632 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.633 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.634 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.635 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:04.636 DEBUG:boto:Data: 2022-09-02T04:21:04.636 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.638 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.639 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.640 DEBUG:boto:Params: {} 2022-09-02T04:21:04.641 DEBUG:boto:Token: None 2022-09-02T04:21:04.642 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26 2022-09-02T04:21:04.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gWjOVq32dr5rnqAok43smY7tb30= 2022-09-02T04:21:04.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gWjOVq32dr5rnqAok43smY7tb30=', 'Content-Length': '0'} 2022-09-02T04:21:04.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8170649dee32df1-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.648 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.649 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.650 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.651 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:04.651 DEBUG:boto:Data: 2022-09-02T04:21:04.652 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.654 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.655 DEBUG:boto:Params: {} 2022-09-02T04:21:04.656 DEBUG:boto:Token: None 2022-09-02T04:21:04.657 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26 2022-09-02T04:21:04.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gWjOVq32dr5rnqAok43smY7tb30= 2022-09-02T04:21:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gWjOVq32dr5rnqAok43smY7tb30=', 'Content-Length': '0'} 2022-09-02T04:21:04.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce9df91b1026373a-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-02T04:21:04.665 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.666 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.666 DEBUG:boto:Method: GET 2022-09-02T04:21:04.667 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:04.668 DEBUG:boto:Data: 2022-09-02T04:21:04.668 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.669 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.670 DEBUG:boto:Params: {} 2022-09-02T04:21:04.671 DEBUG:boto:Token: None 2022-09-02T04:21:04.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26 2022-09-02T04:21:04.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gNlaKBWtOUeAybFSvdvjRZJOwBE= 2022-09-02T04:21:04.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gNlaKBWtOUeAybFSvdvjRZJOwBE=', 'Content-Length': '0'} 2022-09-02T04:21:04.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f53e50d5c77765e-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.677 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:04.677 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:04.679 DEBUG:boto:Method: GET 2022-09-02T04:21:04.680 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:04.680 DEBUG:boto:Data: 2022-09-02T04:21:04.681 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.684 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.685 DEBUG:boto:Params: {} 2022-09-02T04:21:04.686 DEBUG:boto:Token: None 2022-09-02T04:21:04.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%26 2022-09-02T04:21:04.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gNlaKBWtOUeAybFSvdvjRZJOwBE= 2022-09-02T04:21:04.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gNlaKBWtOUeAybFSvdvjRZJOwBE=', 'Content-Length': '0'} 2022-09-02T04:21:04.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003659e69b36fef67f-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.693 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.694 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.695 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.696 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.698 DEBUG:boto:Method: GET 2022-09-02T04:21:04.699 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:04.700 DEBUG:boto:Data: 2022-09-02T04:21:04.700 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.703 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.704 DEBUG:boto:Params: {} 2022-09-02T04:21:04.705 DEBUG:boto:Token: None 2022-09-02T04:21:04.706 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw= 2022-09-02T04:21:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw=', 'Content-Length': '0'} 2022-09-02T04:21:04.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ebde33a8e54cc4-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.758 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.760 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.761 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.761 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.762 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:04.762 DEBUG:boto:Method: GET 2022-09-02T04:21:04.763 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:04.764 DEBUG:boto:Data: 2022-09-02T04:21:04.764 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.765 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.766 DEBUG:boto:Params: {} 2022-09-02T04:21:04.767 DEBUG:boto:Token: None 2022-09-02T04:21:04.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw= 2022-09-02T04:21:04.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw=', 'Content-Length': '0'} 2022-09-02T04:21:04.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474f913dd453c404-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.773 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-02T04:21:04.774 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.775 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.775 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.776 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.776 DEBUG:boto:Method: GET 2022-09-02T04:21:04.777 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.777 DEBUG:boto:Data: 2022-09-02T04:21:04.778 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.778 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.779 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.779 DEBUG:boto:Params: {} 2022-09-02T04:21:04.780 DEBUG:boto:Token: None 2022-09-02T04:21:04.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00= 2022-09-02T04:21:04.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00=', 'Content-Length': '0'} 2022-09-02T04:21:04.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000170b5db07a3abfce-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.785 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.786 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.786 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.787 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.787 DEBUG:boto:Method: GET 2022-09-02T04:21:04.788 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.788 DEBUG:boto:Data: 2022-09-02T04:21:04.789 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.790 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.790 DEBUG:boto:Params: {} 2022-09-02T04:21:04.791 DEBUG:boto:Token: None 2022-09-02T04:21:04.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00= 2022-09-02T04:21:04.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00=', 'Content-Length': '0'} 2022-09-02T04:21:04.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000019dae1c27aceb66-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.838 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.839 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.840 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.841 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.842 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.843 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.844 DEBUG:boto:Data: 2022-09-02T04:21:04.845 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.847 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.847 DEBUG:boto:Params: {} 2022-09-02T04:21:04.848 DEBUG:boto:Token: None 2022-09-02T04:21:04.849 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g2qY7rES4H9uBTUeo2WjnqyIK0k= 2022-09-02T04:21:04.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g2qY7rES4H9uBTUeo2WjnqyIK0k=', 'Content-Length': '0'} 2022-09-02T04:21:04.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e89b864ad2b0b1c3-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.856 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.857 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.858 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.859 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.860 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.861 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.861 DEBUG:boto:Data: 2022-09-02T04:21:04.863 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.865 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.866 DEBUG:boto:Params: {} 2022-09-02T04:21:04.867 DEBUG:boto:Token: None 2022-09-02T04:21:04.868 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g2qY7rES4H9uBTUeo2WjnqyIK0k= 2022-09-02T04:21:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g2qY7rES4H9uBTUeo2WjnqyIK0k=', 'Content-Length': '0'} 2022-09-02T04:21:04.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af744d573b34a14e-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-02T04:21:04.874 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.875 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.875 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.876 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.876 DEBUG:boto:Method: GET 2022-09-02T04:21:04.877 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.877 DEBUG:boto:Data: 2022-09-02T04:21:04.878 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.879 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.880 DEBUG:boto:Params: {} 2022-09-02T04:21:04.880 DEBUG:boto:Token: None 2022-09-02T04:21:04.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00= 2022-09-02T04:21:04.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00=', 'Content-Length': '0'} 2022-09-02T04:21:04.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000940491a21fda6232-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.886 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.886 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.887 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.887 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.888 DEBUG:boto:Method: GET 2022-09-02T04:21:04.890 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.891 DEBUG:boto:Data: 2022-09-02T04:21:04.892 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.894 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.895 DEBUG:boto:Params: {} 2022-09-02T04:21:04.896 DEBUG:boto:Token: None 2022-09-02T04:21:04.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:04.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00= 2022-09-02T04:21:04.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c5BkHNPN7QMoQzhT9azhvgQlo00=', 'Content-Length': '0'} 2022-09-02T04:21:04.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f44c199f0f1f91ae-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.902 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.903 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.904 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:04.905 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:04.906 DEBUG:boto:Method: GET 2022-09-02T04:21:04.907 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:04.908 DEBUG:boto:Data: 2022-09-02T04:21:04.908 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.909 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.910 DEBUG:boto:Params: {} 2022-09-02T04:21:04.911 DEBUG:boto:Token: None 2022-09-02T04:21:04.911 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw= 2022-09-02T04:21:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw=', 'Content-Length': '0'} 2022-09-02T04:21:04.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d5251f359ea35c-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.920 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.921 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:04.922 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:04.923 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:04.932 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:04.933 DEBUG:boto:Method: GET 2022-09-02T04:21:04.936 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:04.936 DEBUG:boto:Data: 2022-09-02T04:21:04.937 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.938 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.938 DEBUG:boto:Params: {} 2022-09-02T04:21:04.939 DEBUG:boto:Token: None 2022-09-02T04:21:04.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/ 2022-09-02T04:21:04.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw= 2022-09-02T04:21:04.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCS8VyKyHcPyiVXM/pab12Sxjpw=', 'Content-Length': '0'} 2022-09-02T04:21:04.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd778f3d92e943c-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.947 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:04.948 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.949 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.952 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.955 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:04.956 DEBUG:boto:Data: 2022-09-02T04:21:04.958 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.959 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.962 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.963 DEBUG:boto:Params: {} 2022-09-02T04:21:04.965 DEBUG:boto:Token: None 2022-09-02T04:21:04.968 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A 2022-09-02T04:21:04.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yk4IjzbocY+dDsNMVTQ3+uSQFNo= 2022-09-02T04:21:04.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yk4IjzbocY+dDsNMVTQ3+uSQFNo=', 'Content-Length': '0'} 2022-09-02T04:21:04.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009de53458ee0d1a3a-00631184b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.973 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.974 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.974 DEBUG:boto:Method: HEAD 2022-09-02T04:21:04.976 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:04.977 DEBUG:boto:Data: 2022-09-02T04:21:04.977 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:04.978 DEBUG:boto:Port: 8000 2022-09-02T04:21:04.980 DEBUG:boto:Params: {} 2022-09-02T04:21:04.980 DEBUG:boto:Token: None 2022-09-02T04:21:04.981 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:04 GMT /xljbkf-10/%3A 2022-09-02T04:21:04.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yk4IjzbocY+dDsNMVTQ3+uSQFNo= 2022-09-02T04:21:04.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yk4IjzbocY+dDsNMVTQ3+uSQFNo=', 'Content-Length': '0'} 2022-09-02T04:21:04.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b17ff1ff47d4759d-00631184b0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:04.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-02T04:21:04.991 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:04.992 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:04.993 DEBUG:boto:Method: GET 2022-09-02T04:21:04.994 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:04.995 DEBUG:boto:Data: 2022-09-02T04:21:04.996 DEBUG:boto:Headers: {} 2022-09-02T04:21:04.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:04.999 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.000 DEBUG:boto:Params: {} 2022-09-02T04:21:05.001 DEBUG:boto:Token: None 2022-09-02T04:21:05.002 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/%3A 2022-09-02T04:21:05.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lyUrcvgeagyLfZY9z1WDq/2jUic= 2022-09-02T04:21:05.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lyUrcvgeagyLfZY9z1WDq/2jUic=', 'Content-Length': '0'} 2022-09-02T04:21:05.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f04ec11a439829ca-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.050 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:05.051 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:05.052 DEBUG:boto:Method: GET 2022-09-02T04:21:05.052 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:05.053 DEBUG:boto:Data: 2022-09-02T04:21:05.054 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.055 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.056 DEBUG:boto:Params: {} 2022-09-02T04:21:05.058 DEBUG:boto:Token: None 2022-09-02T04:21:05.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/%3A 2022-09-02T04:21:05.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lyUrcvgeagyLfZY9z1WDq/2jUic= 2022-09-02T04:21:05.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lyUrcvgeagyLfZY9z1WDq/2jUic=', 'Content-Length': '0'} 2022-09-02T04:21:05.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a37138d42ee7289-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.065 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.067 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.068 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:05.069 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:05.069 DEBUG:boto:Method: GET 2022-09-02T04:21:05.070 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:05.071 DEBUG:boto:Data: 2022-09-02T04:21:05.072 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.073 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.074 DEBUG:boto:Params: {} 2022-09-02T04:21:05.075 DEBUG:boto:Token: None 2022-09-02T04:21:05.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e602d1a26e1a7e8a-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.080 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.081 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.083 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.084 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:05.085 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:05.086 DEBUG:boto:Method: GET 2022-09-02T04:21:05.087 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:05.088 DEBUG:boto:Data: 2022-09-02T04:21:05.089 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.092 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.093 DEBUG:boto:Params: {} 2022-09-02T04:21:05.094 DEBUG:boto:Token: None 2022-09-02T04:21:05.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088873fe1accb3344-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.106 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-02T04:21:05.109 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.110 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.111 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.112 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.113 DEBUG:boto:Method: GET 2022-09-02T04:21:05.115 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.116 DEBUG:boto:Data: 2022-09-02T04:21:05.117 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.120 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.121 DEBUG:boto:Params: {} 2022-09-02T04:21:05.122 DEBUG:boto:Token: None 2022-09-02T04:21:05.124 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg= 2022-09-02T04:21:05.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg=', 'Content-Length': '0'} 2022-09-02T04:21:05.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008940dd615e0b3b03-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.131 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.132 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.133 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.134 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.135 DEBUG:boto:Method: GET 2022-09-02T04:21:05.136 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.137 DEBUG:boto:Data: 2022-09-02T04:21:05.138 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.140 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.141 DEBUG:boto:Params: {} 2022-09-02T04:21:05.142 DEBUG:boto:Token: None 2022-09-02T04:21:05.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg= 2022-09-02T04:21:05.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg=', 'Content-Length': '0'} 2022-09-02T04:21:05.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c4047deaf5c5280-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.245 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.247 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.248 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.249 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.250 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.251 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.252 DEBUG:boto:Data: 2022-09-02T04:21:05.252 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.255 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.255 DEBUG:boto:Params: {} 2022-09-02T04:21:05.256 DEBUG:boto:Token: None 2022-09-02T04:21:05.257 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o9c4CH3Q2L+ggWCgQZYTyeP4Luo= 2022-09-02T04:21:05.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o9c4CH3Q2L+ggWCgQZYTyeP4Luo=', 'Content-Length': '0'} 2022-09-02T04:21:05.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9f6711ffd2372e6-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.263 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.264 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.265 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.269 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.270 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.272 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.273 DEBUG:boto:Data: 2022-09-02T04:21:05.274 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.276 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.277 DEBUG:boto:Params: {} 2022-09-02T04:21:05.278 DEBUG:boto:Token: None 2022-09-02T04:21:05.279 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o9c4CH3Q2L+ggWCgQZYTyeP4Luo= 2022-09-02T04:21:05.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o9c4CH3Q2L+ggWCgQZYTyeP4Luo=', 'Content-Length': '0'} 2022-09-02T04:21:05.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f028f3a460ac50e-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-02T04:21:05.286 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.287 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.287 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.288 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.288 DEBUG:boto:Method: GET 2022-09-02T04:21:05.289 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.289 DEBUG:boto:Data: 2022-09-02T04:21:05.290 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.291 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.292 DEBUG:boto:Params: {} 2022-09-02T04:21:05.292 DEBUG:boto:Token: None 2022-09-02T04:21:05.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg= 2022-09-02T04:21:05.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg=', 'Content-Length': '0'} 2022-09-02T04:21:05.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c0890149c34d3e3-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.298 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.299 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.299 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.300 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.300 DEBUG:boto:Method: GET 2022-09-02T04:21:05.301 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.302 DEBUG:boto:Data: 2022-09-02T04:21:05.302 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.303 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.304 DEBUG:boto:Params: {} 2022-09-02T04:21:05.305 DEBUG:boto:Token: None 2022-09-02T04:21:05.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:05.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg= 2022-09-02T04:21:05.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jT12hLQkiwpr7qwv7daZooJZZfg=', 'Content-Length': '0'} 2022-09-02T04:21:05.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f585faeb8824d88d-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.312 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.312 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.313 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.313 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.314 DEBUG:boto:Method: GET 2022-09-02T04:21:05.315 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.315 DEBUG:boto:Data: 2022-09-02T04:21:05.316 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.317 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.318 DEBUG:boto:Params: {} 2022-09-02T04:21:05.318 DEBUG:boto:Token: None 2022-09-02T04:21:05.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adfc3dd4de909e39-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.328 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.329 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.330 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.330 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.331 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.331 DEBUG:boto:Method: GET 2022-09-02T04:21:05.332 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.333 DEBUG:boto:Data: 2022-09-02T04:21:05.333 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.334 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.335 DEBUG:boto:Params: {} 2022-09-02T04:21:05.335 DEBUG:boto:Token: None 2022-09-02T04:21:05.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d31b0ceddd72efd3-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.390 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.391 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.392 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.392 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.393 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:05.393 DEBUG:boto:Data: 2022-09-02T04:21:05.394 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.394 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.395 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.398 DEBUG:boto:Params: {} 2022-09-02T04:21:05.399 DEBUG:boto:Token: None 2022-09-02T04:21:05.400 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj 2022-09-02T04:21:05.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8siTZ+nWQjSCvwzB2y/POJsVZKI= 2022-09-02T04:21:05.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8siTZ+nWQjSCvwzB2y/POJsVZKI=', 'Content-Length': '0'} 2022-09-02T04:21:05.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b863a5bc4aa60e3-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.407 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.408 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.409 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.410 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:05.412 DEBUG:boto:Data: 2022-09-02T04:21:05.413 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.415 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.416 DEBUG:boto:Params: {} 2022-09-02T04:21:05.417 DEBUG:boto:Token: None 2022-09-02T04:21:05.417 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj 2022-09-02T04:21:05.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8siTZ+nWQjSCvwzB2y/POJsVZKI= 2022-09-02T04:21:05.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8siTZ+nWQjSCvwzB2y/POJsVZKI=', 'Content-Length': '0'} 2022-09-02T04:21:05.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f86e412abb3ba864-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-02T04:21:05.423 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.423 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.424 DEBUG:boto:Method: GET 2022-09-02T04:21:05.425 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:05.425 DEBUG:boto:Data: 2022-09-02T04:21:05.426 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.427 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.428 DEBUG:boto:Params: {} 2022-09-02T04:21:05.430 DEBUG:boto:Token: None 2022-09-02T04:21:05.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj 2022-09-02T04:21:05.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e8OsqrR4V7TwwiADJyHF78v+A3Q= 2022-09-02T04:21:05.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e8OsqrR4V7TwwiADJyHF78v+A3Q=', 'Content-Length': '0'} 2022-09-02T04:21:05.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000356bd95021617b37-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.436 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:05.437 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:05.438 DEBUG:boto:Method: GET 2022-09-02T04:21:05.438 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:05.439 DEBUG:boto:Data: 2022-09-02T04:21:05.440 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.442 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.443 DEBUG:boto:Params: {} 2022-09-02T04:21:05.444 DEBUG:boto:Token: None 2022-09-02T04:21:05.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/_myobj 2022-09-02T04:21:05.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e8OsqrR4V7TwwiADJyHF78v+A3Q= 2022-09-02T04:21:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e8OsqrR4V7TwwiADJyHF78v+A3Q=', 'Content-Length': '0'} 2022-09-02T04:21:05.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f056074b7ec2f7a-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.451 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.452 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.452 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.453 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.454 DEBUG:boto:Method: GET 2022-09-02T04:21:05.455 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.455 DEBUG:boto:Data: 2022-09-02T04:21:05.456 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.457 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.458 DEBUG:boto:Params: {} 2022-09-02T04:21:05.459 DEBUG:boto:Token: None 2022-09-02T04:21:05.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c2c5a50973fe7f-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.464 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.465 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.466 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.467 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.467 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.468 DEBUG:boto:Method: GET 2022-09-02T04:21:05.468 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:05.469 DEBUG:boto:Data: 2022-09-02T04:21:05.469 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.470 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.471 DEBUG:boto:Params: {} 2022-09-02T04:21:05.471 DEBUG:boto:Token: None 2022-09-02T04:21:05.472 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf90e1ee8383fe4-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.478 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:21:05.481 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.486 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.487 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.488 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.489 DEBUG:boto:Method: GET 2022-09-02T04:21:05.490 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.491 DEBUG:boto:Data: 2022-09-02T04:21:05.491 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.493 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.493 DEBUG:boto:Params: {} 2022-09-02T04:21:05.494 DEBUG:boto:Token: None 2022-09-02T04:21:05.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY= 2022-09-02T04:21:05.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY=', 'Content-Length': '0'} 2022-09-02T04:21:05.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b56223cb26b200eb-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.502 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.503 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.504 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.504 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.505 DEBUG:boto:Method: GET 2022-09-02T04:21:05.506 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.507 DEBUG:boto:Data: 2022-09-02T04:21:05.507 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.509 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.510 DEBUG:boto:Params: {} 2022-09-02T04:21:05.511 DEBUG:boto:Token: None 2022-09-02T04:21:05.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY= 2022-09-02T04:21:05.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY=', 'Content-Length': '0'} 2022-09-02T04:21:05.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000196fba7986b86f5c-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.519 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.519 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.520 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.520 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.521 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.521 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.522 DEBUG:boto:Data: 2022-09-02T04:21:05.522 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.524 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.524 DEBUG:boto:Params: {} 2022-09-02T04:21:05.525 DEBUG:boto:Token: None 2022-09-02T04:21:05.526 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nqD58IiHCzg2rlySB7rSYhmroh0= 2022-09-02T04:21:05.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nqD58IiHCzg2rlySB7rSYhmroh0=', 'Content-Length': '0'} 2022-09-02T04:21:05.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a884b840a0b0f14-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.530 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.531 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.531 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.532 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.532 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.533 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.534 DEBUG:boto:Data: 2022-09-02T04:21:05.535 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.537 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.537 DEBUG:boto:Params: {} 2022-09-02T04:21:05.538 DEBUG:boto:Token: None 2022-09-02T04:21:05.538 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nqD58IiHCzg2rlySB7rSYhmroh0= 2022-09-02T04:21:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nqD58IiHCzg2rlySB7rSYhmroh0=', 'Content-Length': '0'} 2022-09-02T04:21:05.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000155f5db47a84cb3f-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:21:05.543 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.544 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.544 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.545 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.545 DEBUG:boto:Method: GET 2022-09-02T04:21:05.546 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.546 DEBUG:boto:Data: 2022-09-02T04:21:05.547 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.548 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.549 DEBUG:boto:Params: {} 2022-09-02T04:21:05.549 DEBUG:boto:Token: None 2022-09-02T04:21:05.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY= 2022-09-02T04:21:05.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY=', 'Content-Length': '0'} 2022-09-02T04:21:05.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063f1416e5839e28d-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.594 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.595 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.596 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.597 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.597 DEBUG:boto:Method: GET 2022-09-02T04:21:05.598 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.598 DEBUG:boto:Data: 2022-09-02T04:21:05.599 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.600 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.601 DEBUG:boto:Params: {} 2022-09-02T04:21:05.601 DEBUG:boto:Token: None 2022-09-02T04:21:05.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:05.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY= 2022-09-02T04:21:05.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4A4NlOfny3nuEffRVHqWcMKFHDY=', 'Content-Length': '0'} 2022-09-02T04:21:05.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd636ae49e73eb3e-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.607 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.609 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.610 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.611 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.611 DEBUG:boto:Method: GET 2022-09-02T04:21:05.612 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.613 DEBUG:boto:Data: 2022-09-02T04:21:05.614 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.616 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.617 DEBUG:boto:Params: {} 2022-09-02T04:21:05.618 DEBUG:boto:Token: None 2022-09-02T04:21:05.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000572386445dfb3d47-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.625 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.626 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.627 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.627 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.628 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.629 DEBUG:boto:Method: GET 2022-09-02T04:21:05.629 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.630 DEBUG:boto:Data: 2022-09-02T04:21:05.631 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.632 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.632 DEBUG:boto:Params: {} 2022-09-02T04:21:05.633 DEBUG:boto:Token: None 2022-09-02T04:21:05.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058fc5665d9288d5b-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.641 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.643 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.644 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.645 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.646 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:05.647 DEBUG:boto:Data: 2022-09-02T04:21:05.648 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.650 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.652 DEBUG:boto:Params: {} 2022-09-02T04:21:05.652 DEBUG:boto:Token: None 2022-09-02T04:21:05.653 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj 2022-09-02T04:21:05.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:he3/CEtkueP0DYb7+iuv3VvsBHM= 2022-09-02T04:21:05.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:he3/CEtkueP0DYb7+iuv3VvsBHM=', 'Content-Length': '0'} 2022-09-02T04:21:05.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c33a3f733d681a0-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.661 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.663 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.665 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.666 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:05.666 DEBUG:boto:Data: 2022-09-02T04:21:05.667 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.668 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.669 DEBUG:boto:Params: {} 2022-09-02T04:21:05.670 DEBUG:boto:Token: None 2022-09-02T04:21:05.671 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj 2022-09-02T04:21:05.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:he3/CEtkueP0DYb7+iuv3VvsBHM= 2022-09-02T04:21:05.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:he3/CEtkueP0DYb7+iuv3VvsBHM=', 'Content-Length': '0'} 2022-09-02T04:21:05.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e311fd47d7eaa32-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:21:05.679 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.679 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.680 DEBUG:boto:Method: GET 2022-09-02T04:21:05.680 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:05.681 DEBUG:boto:Data: 2022-09-02T04:21:05.681 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.682 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.683 DEBUG:boto:Params: {} 2022-09-02T04:21:05.683 DEBUG:boto:Token: None 2022-09-02T04:21:05.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj 2022-09-02T04:21:05.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dlam5IsOdWEJj9IaetvJZdJYSxM= 2022-09-02T04:21:05.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dlam5IsOdWEJj9IaetvJZdJYSxM=', 'Content-Length': '0'} 2022-09-02T04:21:05.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008699b555903b80d2-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.688 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:05.688 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:05.689 DEBUG:boto:Method: GET 2022-09-02T04:21:05.689 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:05.690 DEBUG:boto:Data: 2022-09-02T04:21:05.691 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.692 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.692 DEBUG:boto:Params: {} 2022-09-02T04:21:05.693 DEBUG:boto:Token: None 2022-09-02T04:21:05.693 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/myobj 2022-09-02T04:21:05.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dlam5IsOdWEJj9IaetvJZdJYSxM= 2022-09-02T04:21:05.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dlam5IsOdWEJj9IaetvJZdJYSxM=', 'Content-Length': '0'} 2022-09-02T04:21:05.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb89ffba33c3dd80-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.738 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.739 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.739 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.740 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.741 DEBUG:boto:Method: GET 2022-09-02T04:21:05.741 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.742 DEBUG:boto:Data: 2022-09-02T04:21:05.743 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:21:05.744 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.744 DEBUG:boto:Params: {} 2022-09-02T04:21:05.745 DEBUG:boto:Token: None 2022-09-02T04:21:05.745 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e42a15bb7d5cdd1-00631184b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.750 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.751 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.752 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.752 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.753 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.753 DEBUG:boto:Method: GET 2022-09-02T04:21:05.754 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:05.754 DEBUG:boto:Data: 2022-09-02T04:21:05.755 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.756 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.756 DEBUG:boto:Params: {} 2022-09-02T04:21:05.757 DEBUG:boto:Token: None 2022-09-02T04:21:05.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo= 2022-09-02T04:21:05.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:23pRLAim+vW108+IV/LQfIfmkyo=', 'Content-Length': '0'} 2022-09-02T04:21:05.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a5ad04676dabbb-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.763 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-10 zones={test-zone2, test-zone1} 2022-09-02T04:21:05.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:21:05.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:21:05.765 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:21:05.830 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:21:05.830 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:21:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:21:05.880+0000 7fe5a830b600 1 bucket sync caught up with source: 2022-09-02T04:21:05.881 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:05.882 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:21:05.880+0000 7fe5a830b600 0 bucket checkpoint complete 2022-09-02T04:21:05.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-10 zones={test-zone3, test-zone1} 2022-09-02T04:21:05.899 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.901 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.901 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.902 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:21:05.903 DEBUG:boto:Data: 2022-09-02T04:21:05.903 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:05.907 DEBUG:boto:Port: 8001 2022-09-02T04:21:05.908 DEBUG:boto:Params: {} 2022-09-02T04:21:05.908 DEBUG:boto:Token: None 2022-09-02T04:21:05.909 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ne/VXmisi8p2kUUzpNHgLZn5UdA= 2022-09-02T04:21:05.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ne/VXmisi8p2kUUzpNHgLZn5UdA=', 'Content-Length': '0'} 2022-09-02T04:21:05.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050c9f33fb1db2851-00631184b1-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.915 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.915 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.916 DEBUG:boto:Method: HEAD 2022-09-02T04:21:05.916 DEBUG:boto:Path: /xljbkf-10/ 2022-09-02T04:21:05.917 DEBUG:boto:Data: 2022-09-02T04:21:05.917 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.918 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.919 DEBUG:boto:Params: {} 2022-09-02T04:21:05.919 DEBUG:boto:Token: None 2022-09-02T04:21:05.920 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/ 2022-09-02T04:21:05.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ne/VXmisi8p2kUUzpNHgLZn5UdA= 2022-09-02T04:21:05.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ne/VXmisi8p2kUUzpNHgLZn5UdA=', 'Content-Length': '0'} 2022-09-02T04:21:05.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003be9660440ac8b8d-00631184b1-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.925 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:21:05.926 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.927 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.928 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.929 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.930 DEBUG:boto:Method: GET 2022-09-02T04:21:05.932 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.933 DEBUG:boto:Data: 2022-09-02T04:21:05.933 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:05.935 DEBUG:boto:Port: 8001 2022-09-02T04:21:05.935 DEBUG:boto:Params: {} 2022-09-02T04:21:05.936 DEBUG:boto:Token: None 2022-09-02T04:21:05.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/?versions 2022-09-02T04:21:05.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU= 2022-09-02T04:21:05.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU=', 'Content-Length': '0'} 2022-09-02T04:21:05.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e135e6ffaeef9a6-00631184b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.942 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.943 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-02T04:21:05.943 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-02T04:21:05.944 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-02T04:21:05.944 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:21:05.945 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:21:05.946 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.947 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.948 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.949 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.950 DEBUG:boto:Method: GET 2022-09-02T04:21:05.951 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.952 DEBUG:boto:Data: 2022-09-02T04:21:05.953 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.955 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.956 DEBUG:boto:Params: {} 2022-09-02T04:21:05.957 DEBUG:boto:Token: None 2022-09-02T04:21:05.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/?versions 2022-09-02T04:21:05.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU= 2022-09-02T04:21:05.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU=', 'Content-Length': '0'} 2022-09-02T04:21:05.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056de2ac2cc1b8511-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.964 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.965 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-09-02T04:21:05.966 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-09-02T04:21:05.966 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-09-02T04:21:05.967 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:21:05.967 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.968 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.968 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.969 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.969 DEBUG:boto:Method: GET 2022-09-02T04:21:05.970 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.971 DEBUG:boto:Data: 2022-09-02T04:21:05.971 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:05.972 DEBUG:boto:Port: 8001 2022-09-02T04:21:05.973 DEBUG:boto:Params: {} 2022-09-02T04:21:05.973 DEBUG:boto:Token: None 2022-09-02T04:21:05.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/?versions 2022-09-02T04:21:05.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU= 2022-09-02T04:21:05.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU=', 'Content-Length': '0'} 2022-09-02T04:21:05.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a605143216d023ef-00631184b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:05.980 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:05.982 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:05.982 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:05.983 DEBUG:boto:path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.984 DEBUG:boto:auth_path=/xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.984 DEBUG:boto:Method: GET 2022-09-02T04:21:05.985 DEBUG:boto:Path: /xljbkf-10/?versions&max-keys=999 2022-09-02T04:21:05.985 DEBUG:boto:Data: 2022-09-02T04:21:05.986 DEBUG:boto:Headers: {} 2022-09-02T04:21:05.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:05.989 DEBUG:boto:Port: 8000 2022-09-02T04:21:05.990 DEBUG:boto:Params: {} 2022-09-02T04:21:05.991 DEBUG:boto:Token: None 2022-09-02T04:21:05.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:05 GMT /xljbkf-10/?versions 2022-09-02T04:21:05.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU= 2022-09-02T04:21:05.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BEZCR3SAds1v3pQIQO+L2KKdUIU=', 'Content-Length': '0'} 2022-09-02T04:21:05.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa9e43304e25c54-00631184b1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.002 DEBUG:boto:b'xljbkf-10999false&nulltrue2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-02T04:21:06.004 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.005 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.007 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.007 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.008 DEBUG:boto:Method: GET 2022-09-02T04:21:06.009 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.009 DEBUG:boto:Data: 2022-09-02T04:21:06.010 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.011 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.012 DEBUG:boto:Params: {} 2022-09-02T04:21:06.013 DEBUG:boto:Token: None 2022-09-02T04:21:06.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.014 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I= 2022-09-02T04:21:06.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I=', 'Content-Length': '0'} 2022-09-02T04:21:06.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4b3f0c7b03c664c-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.066 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.067 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.068 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.069 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.070 DEBUG:boto:Method: GET 2022-09-02T04:21:06.070 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.071 DEBUG:boto:Data: 2022-09-02T04:21:06.072 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.076 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.077 DEBUG:boto:Params: {} 2022-09-02T04:21:06.079 DEBUG:boto:Token: None 2022-09-02T04:21:06.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I= 2022-09-02T04:21:06.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I=', 'Content-Length': '0'} 2022-09-02T04:21:06.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001dba9eef1746dfe0-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.088 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.089 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.091 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.092 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.093 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.094 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.095 DEBUG:boto:Data: 2022-09-02T04:21:06.097 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.099 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.100 DEBUG:boto:Params: {} 2022-09-02T04:21:06.101 DEBUG:boto:Token: None 2022-09-02T04:21:06.104 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yCQYkhmCDUxnf7Yhzxfv+Le+pME= 2022-09-02T04:21:06.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yCQYkhmCDUxnf7Yhzxfv+Le+pME=', 'Content-Length': '0'} 2022-09-02T04:21:06.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ecb523051d6f188-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.108 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.109 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.109 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.110 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.111 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.111 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.113 DEBUG:boto:Data: 2022-09-02T04:21:06.113 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.114 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.115 DEBUG:boto:Params: {} 2022-09-02T04:21:06.115 DEBUG:boto:Token: None 2022-09-02T04:21:06.116 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yCQYkhmCDUxnf7Yhzxfv+Le+pME= 2022-09-02T04:21:06.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yCQYkhmCDUxnf7Yhzxfv+Le+pME=', 'Content-Length': '0'} 2022-09-02T04:21:06.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ff40d69bf6a9812-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-02T04:21:06.123 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.124 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.124 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.125 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.126 DEBUG:boto:Method: GET 2022-09-02T04:21:06.127 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.127 DEBUG:boto:Data: 2022-09-02T04:21:06.128 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.129 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.130 DEBUG:boto:Params: {} 2022-09-02T04:21:06.133 DEBUG:boto:Token: None 2022-09-02T04:21:06.134 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I= 2022-09-02T04:21:06.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I=', 'Content-Length': '0'} 2022-09-02T04:21:06.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013333279bfa15a9c-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.141 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.142 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.142 DEBUG:boto:path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.143 DEBUG:boto:auth_path=/xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.144 DEBUG:boto:Method: GET 2022-09-02T04:21:06.144 DEBUG:boto:Path: /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.145 DEBUG:boto:Data: 2022-09-02T04:21:06.145 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.146 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.147 DEBUG:boto:Params: {} 2022-09-02T04:21:06.148 DEBUG:boto:Token: None 2022-09-02T04:21:06.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26?versionId=null 2022-09-02T04:21:06.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I= 2022-09-02T04:21:06.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwllbM2FInpLGGNvm4bhKjQAO5I=', 'Content-Length': '0'} 2022-09-02T04:21:06.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f3caedf150959fc-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.153 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.154 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.154 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.155 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.155 DEBUG:boto:Method: GET 2022-09-02T04:21:06.156 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:06.157 DEBUG:boto:Data: 2022-09-02T04:21:06.157 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.158 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.159 DEBUG:boto:Params: {} 2022-09-02T04:21:06.160 DEBUG:boto:Token: None 2022-09-02T04:21:06.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5f82d8e9910e520-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.166 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.167 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.168 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.169 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.169 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.171 DEBUG:boto:Method: GET 2022-09-02T04:21:06.171 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:06.173 DEBUG:boto:Data: 2022-09-02T04:21:06.173 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.175 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.175 DEBUG:boto:Params: {} 2022-09-02T04:21:06.176 DEBUG:boto:Token: None 2022-09-02T04:21:06.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ee7bd69450425a-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.183 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.184 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.184 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.185 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.185 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:06.186 DEBUG:boto:Data: 2022-09-02T04:21:06.186 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.187 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.188 DEBUG:boto:Params: {} 2022-09-02T04:21:06.189 DEBUG:boto:Token: None 2022-09-02T04:21:06.189 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26 2022-09-02T04:21:06.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GSfBsWfB028SAbsg4ZC+6hg+DtI= 2022-09-02T04:21:06.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GSfBsWfB028SAbsg4ZC+6hg+DtI=', 'Content-Length': '0'} 2022-09-02T04:21:06.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7477072293433d9-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.194 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.195 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.195 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.196 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:06.197 DEBUG:boto:Data: 2022-09-02T04:21:06.197 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.199 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.200 DEBUG:boto:Params: {} 2022-09-02T04:21:06.201 DEBUG:boto:Token: None 2022-09-02T04:21:06.202 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26 2022-09-02T04:21:06.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GSfBsWfB028SAbsg4ZC+6hg+DtI= 2022-09-02T04:21:06.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GSfBsWfB028SAbsg4ZC+6hg+DtI=', 'Content-Length': '0'} 2022-09-02T04:21:06.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000033f2c3a43e78de-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-09-02T04:21:06.207 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.208 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.209 DEBUG:boto:Method: GET 2022-09-02T04:21:06.209 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:06.210 DEBUG:boto:Data: 2022-09-02T04:21:06.211 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.212 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.212 DEBUG:boto:Params: {} 2022-09-02T04:21:06.214 DEBUG:boto:Token: None 2022-09-02T04:21:06.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26 2022-09-02T04:21:06.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tdH42CspQdkQPpRGZpRUg+C1Uo0= 2022-09-02T04:21:06.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tdH42CspQdkQPpRGZpRUg+C1Uo0=', 'Content-Length': '0'} 2022-09-02T04:21:06.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9f5500428deef45-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.220 DEBUG:boto:path=/xljbkf-10/%26 2022-09-02T04:21:06.222 DEBUG:boto:auth_path=/xljbkf-10/%26 2022-09-02T04:21:06.222 DEBUG:boto:Method: GET 2022-09-02T04:21:06.223 DEBUG:boto:Path: /xljbkf-10/%26 2022-09-02T04:21:06.223 DEBUG:boto:Data: 2022-09-02T04:21:06.224 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.225 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.226 DEBUG:boto:Params: {} 2022-09-02T04:21:06.226 DEBUG:boto:Token: None 2022-09-02T04:21:06.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%26 2022-09-02T04:21:06.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tdH42CspQdkQPpRGZpRUg+C1Uo0= 2022-09-02T04:21:06.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tdH42CspQdkQPpRGZpRUg+C1Uo0=', 'Content-Length': '0'} 2022-09-02T04:21:06.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fe6f6b44903063b-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.231 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.232 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.232 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.233 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.233 DEBUG:boto:Method: GET 2022-09-02T04:21:06.234 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:06.235 DEBUG:boto:Data: 2022-09-02T04:21:06.235 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.236 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.237 DEBUG:boto:Params: {} 2022-09-02T04:21:06.237 DEBUG:boto:Token: None 2022-09-02T04:21:06.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007216eea336c02162-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.243 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.243 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.244 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.245 DEBUG:boto:path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.245 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%26 2022-09-02T04:21:06.246 DEBUG:boto:Method: GET 2022-09-02T04:21:06.246 DEBUG:boto:Path: /xljbkf-10/?prefix=%26 2022-09-02T04:21:06.247 DEBUG:boto:Data: 2022-09-02T04:21:06.247 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.248 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.249 DEBUG:boto:Params: {} 2022-09-02T04:21:06.250 DEBUG:boto:Token: None 2022-09-02T04:21:06.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae815cc323df598d-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.298 DEBUG:boto:b'xljbkf-10&1000false&2022-09-02T04:20:28.057Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-02T04:21:06.301 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.301 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.302 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.303 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.303 DEBUG:boto:Method: GET 2022-09-02T04:21:06.304 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.305 DEBUG:boto:Data: 2022-09-02T04:21:06.306 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.307 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.308 DEBUG:boto:Params: {} 2022-09-02T04:21:06.310 DEBUG:boto:Token: None 2022-09-02T04:21:06.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ= 2022-09-02T04:21:06.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ=', 'Content-Length': '0'} 2022-09-02T04:21:06.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007f5e0419ed4a13f-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.315 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.316 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.316 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.317 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.317 DEBUG:boto:Method: GET 2022-09-02T04:21:06.318 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.319 DEBUG:boto:Data: 2022-09-02T04:21:06.319 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.321 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.322 DEBUG:boto:Params: {} 2022-09-02T04:21:06.323 DEBUG:boto:Token: None 2022-09-02T04:21:06.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ= 2022-09-02T04:21:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ=', 'Content-Length': '0'} 2022-09-02T04:21:06.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012bb29d8b08436f1-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.328 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.328 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.329 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.329 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.331 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.331 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.332 DEBUG:boto:Data: 2022-09-02T04:21:06.333 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.334 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.334 DEBUG:boto:Params: {} 2022-09-02T04:21:06.335 DEBUG:boto:Token: None 2022-09-02T04:21:06.335 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TBefH1286i28LSxTvFp4PxI/sHA= 2022-09-02T04:21:06.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TBefH1286i28LSxTvFp4PxI/sHA=', 'Content-Length': '0'} 2022-09-02T04:21:06.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4b08060acb9d7a2-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.339 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.340 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.341 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.341 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.342 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.342 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.343 DEBUG:boto:Data: 2022-09-02T04:21:06.343 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.344 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.345 DEBUG:boto:Params: {} 2022-09-02T04:21:06.345 DEBUG:boto:Token: None 2022-09-02T04:21:06.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TBefH1286i28LSxTvFp4PxI/sHA= 2022-09-02T04:21:06.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TBefH1286i28LSxTvFp4PxI/sHA=', 'Content-Length': '0'} 2022-09-02T04:21:06.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005172fc2fe24ad004-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-02T04:21:06.350 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.351 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.351 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.352 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.352 DEBUG:boto:Method: GET 2022-09-02T04:21:06.353 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.354 DEBUG:boto:Data: 2022-09-02T04:21:06.354 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.355 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.356 DEBUG:boto:Params: {} 2022-09-02T04:21:06.356 DEBUG:boto:Token: None 2022-09-02T04:21:06.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ= 2022-09-02T04:21:06.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ=', 'Content-Length': '0'} 2022-09-02T04:21:06.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040d6aa5417378162-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.361 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.362 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.362 DEBUG:boto:path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.363 DEBUG:boto:auth_path=/xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.363 DEBUG:boto:Method: GET 2022-09-02T04:21:06.364 DEBUG:boto:Path: /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.365 DEBUG:boto:Data: 2022-09-02T04:21:06.365 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.366 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.367 DEBUG:boto:Params: {} 2022-09-02T04:21:06.367 DEBUG:boto:Token: None 2022-09-02T04:21:06.368 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A?versionId=null 2022-09-02T04:21:06.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ= 2022-09-02T04:21:06.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ujk4ISK1cbD4WFNXutoDJIvvElQ=', 'Content-Length': '0'} 2022-09-02T04:21:06.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6990740f6f513dd-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.373 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.373 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.375 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.376 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.377 DEBUG:boto:Method: GET 2022-09-02T04:21:06.378 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.379 DEBUG:boto:Data: 2022-09-02T04:21:06.380 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.381 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.382 DEBUG:boto:Params: {} 2022-09-02T04:21:06.384 DEBUG:boto:Token: None 2022-09-02T04:21:06.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c8b28e2962669f5-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.390 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.391 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.392 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.393 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.394 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.395 DEBUG:boto:Method: GET 2022-09-02T04:21:06.396 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.396 DEBUG:boto:Data: 2022-09-02T04:21:06.397 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.399 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.400 DEBUG:boto:Params: {} 2022-09-02T04:21:06.401 DEBUG:boto:Token: None 2022-09-02T04:21:06.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80500cf40e03b05-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.409 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.411 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.411 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.412 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.413 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:06.413 DEBUG:boto:Data: 2022-09-02T04:21:06.414 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.415 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.416 DEBUG:boto:Params: {} 2022-09-02T04:21:06.417 DEBUG:boto:Token: None 2022-09-02T04:21:06.419 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A 2022-09-02T04:21:06.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0it5Qddm2n3UI0jhK8W79eWG/HE= 2022-09-02T04:21:06.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0it5Qddm2n3UI0jhK8W79eWG/HE=', 'Content-Length': '0'} 2022-09-02T04:21:06.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdf14c9681f48abc-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.424 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.425 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.426 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.427 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:06.428 DEBUG:boto:Data: 2022-09-02T04:21:06.429 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.431 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.431 DEBUG:boto:Params: {} 2022-09-02T04:21:06.432 DEBUG:boto:Token: None 2022-09-02T04:21:06.432 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A 2022-09-02T04:21:06.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0it5Qddm2n3UI0jhK8W79eWG/HE= 2022-09-02T04:21:06.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0it5Qddm2n3UI0jhK8W79eWG/HE=', 'Content-Length': '0'} 2022-09-02T04:21:06.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba226ee2ecb502bb-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-09-02T04:21:06.440 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.441 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.442 DEBUG:boto:Method: GET 2022-09-02T04:21:06.443 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:06.444 DEBUG:boto:Data: 2022-09-02T04:21:06.445 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.447 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.448 DEBUG:boto:Params: {} 2022-09-02T04:21:06.449 DEBUG:boto:Token: None 2022-09-02T04:21:06.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A 2022-09-02T04:21:06.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nE1i2nKexlJAsqiPcKV/6q9pODM= 2022-09-02T04:21:06.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nE1i2nKexlJAsqiPcKV/6q9pODM=', 'Content-Length': '0'} 2022-09-02T04:21:06.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb63642e2701e068-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.502 DEBUG:boto:path=/xljbkf-10/%3A 2022-09-02T04:21:06.503 DEBUG:boto:auth_path=/xljbkf-10/%3A 2022-09-02T04:21:06.504 DEBUG:boto:Method: GET 2022-09-02T04:21:06.505 DEBUG:boto:Path: /xljbkf-10/%3A 2022-09-02T04:21:06.506 DEBUG:boto:Data: 2022-09-02T04:21:06.507 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.508 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.509 DEBUG:boto:Params: {} 2022-09-02T04:21:06.510 DEBUG:boto:Token: None 2022-09-02T04:21:06.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/%3A 2022-09-02T04:21:06.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nE1i2nKexlJAsqiPcKV/6q9pODM= 2022-09-02T04:21:06.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nE1i2nKexlJAsqiPcKV/6q9pODM=', 'Content-Length': '0'} 2022-09-02T04:21:06.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000247a8d4e41bc6b42-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.515 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.516 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.517 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.518 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.519 DEBUG:boto:Method: GET 2022-09-02T04:21:06.520 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.520 DEBUG:boto:Data: 2022-09-02T04:21:06.521 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.523 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.524 DEBUG:boto:Params: {} 2022-09-02T04:21:06.525 DEBUG:boto:Token: None 2022-09-02T04:21:06.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec58a26bacdae8d-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.533 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.535 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.536 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.537 DEBUG:boto:path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.538 DEBUG:boto:auth_path=/xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.539 DEBUG:boto:Method: GET 2022-09-02T04:21:06.540 DEBUG:boto:Path: /xljbkf-10/?prefix=%3A 2022-09-02T04:21:06.541 DEBUG:boto:Data: 2022-09-02T04:21:06.542 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.543 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.544 DEBUG:boto:Params: {} 2022-09-02T04:21:06.545 DEBUG:boto:Token: None 2022-09-02T04:21:06.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000876b4ac58c490409-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.554 DEBUG:boto:b'xljbkf-10:1000false:2022-09-02T04:20:28.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-02T04:21:06.556 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.556 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.557 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.557 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.558 DEBUG:boto:Method: GET 2022-09-02T04:21:06.558 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.559 DEBUG:boto:Data: 2022-09-02T04:21:06.559 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.561 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.561 DEBUG:boto:Params: {} 2022-09-02T04:21:06.562 DEBUG:boto:Token: None 2022-09-02T04:21:06.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM= 2022-09-02T04:21:06.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM=', 'Content-Length': '0'} 2022-09-02T04:21:06.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad3b8e878d2b976-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.567 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.568 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.568 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.569 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.569 DEBUG:boto:Method: GET 2022-09-02T04:21:06.570 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.570 DEBUG:boto:Data: 2022-09-02T04:21:06.571 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.572 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.573 DEBUG:boto:Params: {} 2022-09-02T04:21:06.573 DEBUG:boto:Token: None 2022-09-02T04:21:06.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM= 2022-09-02T04:21:06.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM=', 'Content-Length': '0'} 2022-09-02T04:21:06.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080297558430a4270-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.581 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.581 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.582 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.583 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.584 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.585 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.586 DEBUG:boto:Data: 2022-09-02T04:21:06.587 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.588 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.590 DEBUG:boto:Params: {} 2022-09-02T04:21:06.591 DEBUG:boto:Token: None 2022-09-02T04:21:06.592 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lDhKIcM0dARwZnDXTYpvLbmrLEs= 2022-09-02T04:21:06.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lDhKIcM0dARwZnDXTYpvLbmrLEs=', 'Content-Length': '0'} 2022-09-02T04:21:06.596 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016fcaf3bf2b6b492-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.597 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.599 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.599 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.600 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.601 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.602 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.603 DEBUG:boto:Data: 2022-09-02T04:21:06.604 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.606 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.607 DEBUG:boto:Params: {} 2022-09-02T04:21:06.608 DEBUG:boto:Token: None 2022-09-02T04:21:06.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lDhKIcM0dARwZnDXTYpvLbmrLEs= 2022-09-02T04:21:06.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lDhKIcM0dARwZnDXTYpvLbmrLEs=', 'Content-Length': '0'} 2022-09-02T04:21:06.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051e5a5afb02c0dd4-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-02T04:21:06.615 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.616 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.617 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.618 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.619 DEBUG:boto:Method: GET 2022-09-02T04:21:06.621 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.622 DEBUG:boto:Data: 2022-09-02T04:21:06.623 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.625 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.626 DEBUG:boto:Params: {} 2022-09-02T04:21:06.627 DEBUG:boto:Token: None 2022-09-02T04:21:06.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM= 2022-09-02T04:21:06.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM=', 'Content-Length': '0'} 2022-09-02T04:21:06.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c24b388227ac335c-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.633 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.634 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.635 DEBUG:boto:path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.635 DEBUG:boto:auth_path=/xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.636 DEBUG:boto:Method: GET 2022-09-02T04:21:06.637 DEBUG:boto:Path: /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.637 DEBUG:boto:Data: 2022-09-02T04:21:06.638 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.639 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.640 DEBUG:boto:Params: {} 2022-09-02T04:21:06.640 DEBUG:boto:Token: None 2022-09-02T04:21:06.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj?versionId=null 2022-09-02T04:21:06.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM= 2022-09-02T04:21:06.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DXhNJu4XfwlWqSw92mxvY8AQ+BM=', 'Content-Length': '0'} 2022-09-02T04:21:06.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7ffc050af5a2667-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.686 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.687 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.687 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.688 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.689 DEBUG:boto:Method: GET 2022-09-02T04:21:06.689 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.690 DEBUG:boto:Data: 2022-09-02T04:21:06.690 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.692 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.692 DEBUG:boto:Params: {} 2022-09-02T04:21:06.693 DEBUG:boto:Token: None 2022-09-02T04:21:06.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ad21c724554691-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.699 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.700 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.701 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.701 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.702 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.703 DEBUG:boto:Method: GET 2022-09-02T04:21:06.703 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.704 DEBUG:boto:Data: 2022-09-02T04:21:06.704 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.705 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.706 DEBUG:boto:Params: {} 2022-09-02T04:21:06.707 DEBUG:boto:Token: None 2022-09-02T04:21:06.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c6e26f075e011c-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.713 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.714 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.715 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.716 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.716 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:06.717 DEBUG:boto:Data: 2022-09-02T04:21:06.717 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.719 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.719 DEBUG:boto:Params: {} 2022-09-02T04:21:06.720 DEBUG:boto:Token: None 2022-09-02T04:21:06.720 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj 2022-09-02T04:21:06.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W51ztH8d5F+y2Z5jvvqjcjm7Ps4= 2022-09-02T04:21:06.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W51ztH8d5F+y2Z5jvvqjcjm7Ps4=', 'Content-Length': '0'} 2022-09-02T04:21:06.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d95bacd4bdf46403-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.725 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.725 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.726 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.727 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:06.727 DEBUG:boto:Data: 2022-09-02T04:21:06.728 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.729 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.729 DEBUG:boto:Params: {} 2022-09-02T04:21:06.730 DEBUG:boto:Token: None 2022-09-02T04:21:06.730 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj 2022-09-02T04:21:06.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W51ztH8d5F+y2Z5jvvqjcjm7Ps4= 2022-09-02T04:21:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W51ztH8d5F+y2Z5jvvqjcjm7Ps4=', 'Content-Length': '0'} 2022-09-02T04:21:06.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1cac29b063444f3-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-09-02T04:21:06.735 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.736 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.737 DEBUG:boto:Method: GET 2022-09-02T04:21:06.737 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:06.738 DEBUG:boto:Data: 2022-09-02T04:21:06.738 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.740 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.740 DEBUG:boto:Params: {} 2022-09-02T04:21:06.741 DEBUG:boto:Token: None 2022-09-02T04:21:06.742 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj 2022-09-02T04:21:06.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VEc+SeWIKqQBSkIjccJ47AWKXCQ= 2022-09-02T04:21:06.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VEc+SeWIKqQBSkIjccJ47AWKXCQ=', 'Content-Length': '0'} 2022-09-02T04:21:06.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad7d05e9202e396-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.786 DEBUG:boto:path=/xljbkf-10/_myobj 2022-09-02T04:21:06.787 DEBUG:boto:auth_path=/xljbkf-10/_myobj 2022-09-02T04:21:06.788 DEBUG:boto:Method: GET 2022-09-02T04:21:06.789 DEBUG:boto:Path: /xljbkf-10/_myobj 2022-09-02T04:21:06.789 DEBUG:boto:Data: 2022-09-02T04:21:06.790 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.791 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.792 DEBUG:boto:Params: {} 2022-09-02T04:21:06.792 DEBUG:boto:Token: None 2022-09-02T04:21:06.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/_myobj 2022-09-02T04:21:06.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VEc+SeWIKqQBSkIjccJ47AWKXCQ= 2022-09-02T04:21:06.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VEc+SeWIKqQBSkIjccJ47AWKXCQ=', 'Content-Length': '0'} 2022-09-02T04:21:06.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae8dda150b9c894e-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.799 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.800 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.800 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.801 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.802 DEBUG:boto:Method: GET 2022-09-02T04:21:06.802 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.805 DEBUG:boto:Data: 2022-09-02T04:21:06.806 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.807 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.808 DEBUG:boto:Params: {} 2022-09-02T04:21:06.809 DEBUG:boto:Token: None 2022-09-02T04:21:06.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6020f6c4a8014cf-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.815 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.817 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.818 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.819 DEBUG:boto:path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.820 DEBUG:boto:auth_path=/xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.820 DEBUG:boto:Method: GET 2022-09-02T04:21:06.821 DEBUG:boto:Path: /xljbkf-10/?prefix=_myobj 2022-09-02T04:21:06.821 DEBUG:boto:Data: 2022-09-02T04:21:06.822 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.824 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.824 DEBUG:boto:Params: {} 2022-09-02T04:21:06.825 DEBUG:boto:Token: None 2022-09-02T04:21:06.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8bebefae2e05fd5-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.834 DEBUG:boto:b'xljbkf-10_myobj1000false_myobj2022-09-02T04:20:27.927Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:21:06.836 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:06.837 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:06.838 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.840 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.841 DEBUG:boto:Method: GET 2022-09-02T04:21:06.842 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.843 DEBUG:boto:Data: 2022-09-02T04:21:06.844 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.845 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.846 DEBUG:boto:Params: {} 2022-09-02T04:21:06.847 DEBUG:boto:Token: None 2022-09-02T04:21:06.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0= 2022-09-02T04:21:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0=', 'Content-Length': '0'} 2022-09-02T04:21:06.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020ff4d5c2e6315bc-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.855 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:06.858 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:06.859 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.860 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.861 DEBUG:boto:Method: GET 2022-09-02T04:21:06.863 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.864 DEBUG:boto:Data: 2022-09-02T04:21:06.864 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.868 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.869 DEBUG:boto:Params: {} 2022-09-02T04:21:06.871 DEBUG:boto:Token: None 2022-09-02T04:21:06.871 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0= 2022-09-02T04:21:06.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0=', 'Content-Length': '0'} 2022-09-02T04:21:06.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000748a1e3add66add6-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.879 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:06.881 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:06.882 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.882 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.883 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.884 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.885 DEBUG:boto:Data: 2022-09-02T04:21:06.886 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.888 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.890 DEBUG:boto:Params: {} 2022-09-02T04:21:06.891 DEBUG:boto:Token: None 2022-09-02T04:21:06.892 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wt89w42st2Z07gvLODliH0vB4N0= 2022-09-02T04:21:06.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wt89w42st2Z07gvLODliH0vB4N0=', 'Content-Length': '0'} 2022-09-02T04:21:06.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d213726cad3fa56-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.899 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:06.900 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:06.901 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.901 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.902 DEBUG:boto:Method: HEAD 2022-09-02T04:21:06.902 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.903 DEBUG:boto:Data: 2022-09-02T04:21:06.903 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.905 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.905 DEBUG:boto:Params: {} 2022-09-02T04:21:06.906 DEBUG:boto:Token: None 2022-09-02T04:21:06.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wt89w42st2Z07gvLODliH0vB4N0= 2022-09-02T04:21:06.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wt89w42st2Z07gvLODliH0vB4N0=', 'Content-Length': '0'} 2022-09-02T04:21:06.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd8f4b92df1b853e-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:21:06.912 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:06.913 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:06.914 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.915 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.916 DEBUG:boto:Method: GET 2022-09-02T04:21:06.917 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.918 DEBUG:boto:Data: 2022-09-02T04:21:06.919 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.931 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.932 DEBUG:boto:Params: {} 2022-09-02T04:21:06.936 DEBUG:boto:Token: None 2022-09-02T04:21:06.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0= 2022-09-02T04:21:06.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0=', 'Content-Length': '0'} 2022-09-02T04:21:06.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b54b07d3ccda7a6-00631184b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.945 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:06.946 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:06.951 DEBUG:boto:path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.953 DEBUG:boto:auth_path=/xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.954 DEBUG:boto:Method: GET 2022-09-02T04:21:06.955 DEBUG:boto:Path: /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.958 DEBUG:boto:Data: 2022-09-02T04:21:06.958 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:06.959 DEBUG:boto:Port: 8000 2022-09-02T04:21:06.960 DEBUG:boto:Params: {} 2022-09-02T04:21:06.961 DEBUG:boto:Token: None 2022-09-02T04:21:06.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/myobj?versionId=null 2022-09-02T04:21:06.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0= 2022-09-02T04:21:06.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGDbB+35wi/gtzuZSlMBGVvvRz0=', 'Content-Length': '0'} 2022-09-02T04:21:06.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c5bc6c4493f780b-00631184b2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.970 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.981 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.981 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:06.982 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:06.983 DEBUG:boto:Method: GET 2022-09-02T04:21:06.983 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:06.984 DEBUG:boto:Data: 2022-09-02T04:21:06.985 DEBUG:boto:Headers: {} 2022-09-02T04:21:06.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:06.986 DEBUG:boto:Port: 8001 2022-09-02T04:21:06.987 DEBUG:boto:Params: {} 2022-09-02T04:21:06.987 DEBUG:boto:Token: None 2022-09-02T04:21:06.988 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:06 GMT /xljbkf-10/ 2022-09-02T04:21:06.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE= 2022-09-02T04:21:06.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRcIXQOXoNN331QhGXZcJz/bSjE=', 'Content-Length': '0'} 2022-09-02T04:21:06.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c39f1e30f881756-00631184b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:06.995 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:06.996 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:06.997 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:06.998 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:06.999 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.000 DEBUG:boto:Method: GET 2022-09-02T04:21:07.002 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.003 DEBUG:boto:Data: 2022-09-02T04:21:07.004 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:07.006 DEBUG:boto:Port: 8000 2022-09-02T04:21:07.008 DEBUG:boto:Params: {} 2022-09-02T04:21:07.009 DEBUG:boto:Token: None 2022-09-02T04:21:07.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/ 2022-09-02T04:21:07.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:856T1h4GQcxzrGZMZdG4B5XvPOY= 2022-09-02T04:21:07.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:856T1h4GQcxzrGZMZdG4B5XvPOY=', 'Content-Length': '0'} 2022-09-02T04:21:07.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be3cb2ec8d1dcfd5-00631184b3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.020 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:07.023 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:07.024 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:07.025 DEBUG:boto:Method: HEAD 2022-09-02T04:21:07.026 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:07.027 DEBUG:boto:Data: 2022-09-02T04:21:07.028 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:07.038 DEBUG:boto:Port: 8001 2022-09-02T04:21:07.039 DEBUG:boto:Params: {} 2022-09-02T04:21:07.040 DEBUG:boto:Token: None 2022-09-02T04:21:07.040 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/myobj 2022-09-02T04:21:07.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0WNJASPV5uOdY6tXQ4m6SN5GpXI= 2022-09-02T04:21:07.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0WNJASPV5uOdY6tXQ4m6SN5GpXI=', 'Content-Length': '0'} 2022-09-02T04:21:07.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c279fffd2545ce1a-00631184b3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.047 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:07.048 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:07.049 DEBUG:boto:Method: HEAD 2022-09-02T04:21:07.050 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:07.051 DEBUG:boto:Data: 2022-09-02T04:21:07.052 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:07.054 DEBUG:boto:Port: 8000 2022-09-02T04:21:07.054 DEBUG:boto:Params: {} 2022-09-02T04:21:07.055 DEBUG:boto:Token: None 2022-09-02T04:21:07.056 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/myobj 2022-09-02T04:21:07.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0WNJASPV5uOdY6tXQ4m6SN5GpXI= 2022-09-02T04:21:07.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0WNJASPV5uOdY6tXQ4m6SN5GpXI=', 'Content-Length': '0'} 2022-09-02T04:21:07.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020727382f231cfcc-00631184b3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:21:07.061 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:07.062 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:07.063 DEBUG:boto:Method: GET 2022-09-02T04:21:07.064 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:07.064 DEBUG:boto:Data: 2022-09-02T04:21:07.065 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:07.067 DEBUG:boto:Port: 8001 2022-09-02T04:21:07.068 DEBUG:boto:Params: {} 2022-09-02T04:21:07.069 DEBUG:boto:Token: None 2022-09-02T04:21:07.071 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/myobj 2022-09-02T04:21:07.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FkNFunndy3J+rf887EGH40K7hbY= 2022-09-02T04:21:07.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FkNFunndy3J+rf887EGH40K7hbY=', 'Content-Length': '0'} 2022-09-02T04:21:07.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034ae35fedffdd20b-00631184b3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.080 DEBUG:boto:path=/xljbkf-10/myobj 2022-09-02T04:21:07.081 DEBUG:boto:auth_path=/xljbkf-10/myobj 2022-09-02T04:21:07.081 DEBUG:boto:Method: GET 2022-09-02T04:21:07.082 DEBUG:boto:Path: /xljbkf-10/myobj 2022-09-02T04:21:07.083 DEBUG:boto:Data: 2022-09-02T04:21:07.084 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:07.086 DEBUG:boto:Port: 8000 2022-09-02T04:21:07.088 DEBUG:boto:Params: {} 2022-09-02T04:21:07.089 DEBUG:boto:Token: None 2022-09-02T04:21:07.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/myobj 2022-09-02T04:21:07.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FkNFunndy3J+rf887EGH40K7hbY= 2022-09-02T04:21:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FkNFunndy3J+rf887EGH40K7hbY=', 'Content-Length': '0'} 2022-09-02T04:21:07.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:20:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b89e5ba0e6acaa19-00631184b3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.139 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:07.141 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:07.142 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.143 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.144 DEBUG:boto:Method: GET 2022-09-02T04:21:07.147 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.148 DEBUG:boto:Data: 2022-09-02T04:21:07.149 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:21:07.152 DEBUG:boto:Port: 8001 2022-09-02T04:21:07.153 DEBUG:boto:Params: {} 2022-09-02T04:21:07.156 DEBUG:boto:Token: None 2022-09-02T04:21:07.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/ 2022-09-02T04:21:07.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:856T1h4GQcxzrGZMZdG4B5XvPOY= 2022-09-02T04:21:07.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:856T1h4GQcxzrGZMZdG4B5XvPOY=', 'Content-Length': '0'} 2022-09-02T04:21:07.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b57fe806eacb6ef-00631184b3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.165 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:07.166 DEBUG:boto:path=/xljbkf-10/ 2022-09-02T04:21:07.167 DEBUG:boto:auth_path=/xljbkf-10/ 2022-09-02T04:21:07.168 DEBUG:boto:path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.169 DEBUG:boto:auth_path=/xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.170 DEBUG:boto:Method: GET 2022-09-02T04:21:07.171 DEBUG:boto:Path: /xljbkf-10/?prefix=myobj 2022-09-02T04:21:07.171 DEBUG:boto:Data: 2022-09-02T04:21:07.173 DEBUG:boto:Headers: {} 2022-09-02T04:21:07.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:21:07.175 DEBUG:boto:Port: 8000 2022-09-02T04:21:07.176 DEBUG:boto:Params: {} 2022-09-02T04:21:07.177 DEBUG:boto:Token: None 2022-09-02T04:21:07.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:21:07 GMT /xljbkf-10/ 2022-09-02T04:21:07.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:856T1h4GQcxzrGZMZdG4B5XvPOY= 2022-09-02T04:21:07.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:21:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:856T1h4GQcxzrGZMZdG4B5XvPOY=', 'Content-Length': '0'} 2022-09-02T04:21:07.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e6234ab295873e5-00631184b3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:21:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:21:07.197 DEBUG:boto:b'xljbkf-10myobj1000falsemyobj2022-09-02T04:20:27.873Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:21:07.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-10 zones={test-zone3, test-zone1} 2022-09-02T04:21:07.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:21:07.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:21:07.203 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:21:07.270 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:21:07.271 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:21:07.308 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:07.305+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:07.309 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , , , , , , , , ] 2022-09-02T04:21:07.310 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:12.308 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:12.306+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:12.310 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:12.310 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:17.310 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:17.308+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:17.310 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:17.310 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:22.312 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:22.309+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:22.313 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:22.313 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:27.323 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:27.311+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:27.324 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:27.325 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:32.315 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:32.312+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:32.316 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:32.316 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:37.315 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:37.314+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:37.316 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:37.316 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:42.317 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:42.316+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:42.317 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:42.318 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:47.318 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:47.316+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:47.319 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , , , ] 2022-09-02T04:21:47.320 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:52.322 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:52.318+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:52.323 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:21:52.323 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:21:57.321 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:21:57.319+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:21:57.322 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:21:57.322 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:02.323 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:02.321+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:02.324 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:02.325 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:07.325 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:07.322+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:07.326 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:07.326 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:12.325 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:12.323+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:12.326 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:12.326 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:17.328 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:17.325+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:17.328 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:17.329 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:22.328 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:22.326+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:22.329 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:22.329 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:27.329 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:27.328+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:27.330 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:27.331 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:32.331 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:32.329+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:32.332 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:32.332 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:37.332 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:37.331+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:37.333 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:37.333 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:42.334 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:42.333+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:42.334 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:42.334 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:47.335 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:47.334+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:47.336 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:47.336 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:52.346 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:52.335+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:52.347 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:52.347 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:22:57.338 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:22:57.336+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:22:57.339 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:22:57.339 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:02.341 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:02.338+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:02.341 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:02.342 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:07.341 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:07.339+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:07.342 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:07.342 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:12.345 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:12.341+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:12.346 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:12.346 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:17.344 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:17.342+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:17.344 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:17.345 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:22.349 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:22.343+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:22.350 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:22.351 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:27.347 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:27.345+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:27.348 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:27.350 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:32.358 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:32.346+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:32.360 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:32.360 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:37.353 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:37.348+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:37.353 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:37.353 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:42.351 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:42.349+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:42.352 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:42.352 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:47.352 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:47.351+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:47.353 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:47.353 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:52.354 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:52.353+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:52.354 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:52.354 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:23:57.355 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:23:57.353+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:23:57.355 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:23:57.356 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:02.357 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:02.355+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:02.357 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:02.358 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:07.358 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:07.356+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:07.358 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:07.359 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:12.359 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:12.357+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:12.360 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:12.360 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:17.361 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:17.359+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:17.362 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:17.362 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:22.362 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:22.360+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:22.363 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:22.363 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:27.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:27.362+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:27.365 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:27.365 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:32.365 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:32.363+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:32.365 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:32.366 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:37.371 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:37.365+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:37.371 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:37.372 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:42.375 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:42.366+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:42.376 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:42.376 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:47.372 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:47.367+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:47.373 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:47.373 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:52.371 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:52.370+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:52.372 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:52.372 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:24:57.373 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:24:57.371+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:24:57.374 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:24:57.375 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:02.373 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:02.371+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:02.374 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:02.374 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:07.373+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:07.377 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:07.377 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:12.378 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:12.374+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:12.379 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:12.379 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:17.378 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:17.376+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:17.378 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:17.379 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:22.382 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:22.377+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:22.382 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:22.383 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:27.380 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:27.379+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:27.381 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:27.381 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:32.382 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:32.380+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:32.383 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:32.383 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:37.383 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:37.381+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:37.384 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:37.384 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:42.388 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:42.383+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:42.389 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:42.389 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:47.386 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:47.384+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:47.387 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:47.387 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:52.388 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:52.386+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:52.391 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:52.391 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:25:57.389 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:25:57.388+0000 7f8b11a74600 1 waiting for incremental sync to catch up: 2022-09-02T04:25:57.390 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.5.5, , 00000000001.7.5, , , , , 00000000001.6.5, , ] 2022-09-02T04:25:57.391 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.5.5, , 00000000001.7.5, , , , 00000000001.9.5, 00000000001.6.5, , ] 2022-09-02T04:26:02.393 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:02.390+0000 7f8b11a74600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T04:26:02.394 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:02.390+0000 7f8b11a74600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T04:26:02.395 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:02.390+0000 7f8b11a74600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T04:26:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T04:26:02.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... FAIL 2022-09-02T04:26:02.455 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-13 2022-09-02T04:26:02.456 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:02.457 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:02.459 DEBUG:boto:Method: PUT 2022-09-02T04:26:02.459 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:02.460 DEBUG:boto:Data: 2022-09-02T04:26:02.461 DEBUG:boto:Headers: {} 2022-09-02T04:26:02.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:02.464 DEBUG:boto:Port: 8000 2022-09-02T04:26:02.465 DEBUG:boto:Params: {} 2022-09-02T04:26:02.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:26:02.469 DEBUG:boto:Token: None 2022-09-02T04:26:02.471 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-13/ 2022-09-02T04:26:02.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vjPpNg+Bf0/Qg01dCOZlyQSLUW4= 2022-09-02T04:26:02.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vjPpNg+Bf0/Qg01dCOZlyQSLUW4=', 'Content-Length': '0'} 2022-09-02T04:26:02.514 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2e5d443321058a5-00631185da-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.515 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-14 2022-09-02T04:26:02.516 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:02.527 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:02.527 DEBUG:boto:Method: PUT 2022-09-02T04:26:02.528 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:26:02.529 DEBUG:boto:Data: 2022-09-02T04:26:02.529 DEBUG:boto:Headers: {} 2022-09-02T04:26:02.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:02.530 DEBUG:boto:Port: 8000 2022-09-02T04:26:02.531 DEBUG:boto:Params: {} 2022-09-02T04:26:02.532 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:26:02.533 DEBUG:boto:Token: None 2022-09-02T04:26:02.535 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-14/ 2022-09-02T04:26:02.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B8reJBLeYHI7emzS2Yv1eJhcsV8= 2022-09-02T04:26:02.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B8reJBLeYHI7emzS2Yv1eJhcsV8=', 'Content-Length': '0'} 2022-09-02T04:26:02.598 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c4c8526ee419b9c6-00631185da-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.599 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-15 2022-09-02T04:26:02.600 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:02.600 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:02.610 DEBUG:boto:Method: PUT 2022-09-02T04:26:02.612 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:26:02.613 DEBUG:boto:Data: 2022-09-02T04:26:02.614 DEBUG:boto:Headers: {} 2022-09-02T04:26:02.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:02.616 DEBUG:boto:Port: 8001 2022-09-02T04:26:02.617 DEBUG:boto:Params: {} 2022-09-02T04:26:02.618 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:26:02.619 DEBUG:boto:Token: None 2022-09-02T04:26:02.620 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-15/ 2022-09-02T04:26:02.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:heuEtTZJ2sQHZjQdfl3mR0YC4QE= 2022-09-02T04:26:02.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:heuEtTZJ2sQHZjQdfl3mR0YC4QE=', 'Content-Length': '0'} 2022-09-02T04:26:02.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000652dadef1f37441c-00631185da-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.680 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:02.681 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:02.689 DEBUG:boto:Method: HEAD 2022-09-02T04:26:02.690 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:02.691 DEBUG:boto:Data: 2022-09-02T04:26:02.691 DEBUG:boto:Headers: {} 2022-09-02T04:26:02.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:02.693 DEBUG:boto:Port: 8000 2022-09-02T04:26:02.694 DEBUG:boto:Params: {} 2022-09-02T04:26:02.695 DEBUG:boto:Token: None 2022-09-02T04:26:02.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-13/ 2022-09-02T04:26:02.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UKML7W1DX5TOdCL+5BUgq/4Hr7k= 2022-09-02T04:26:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UKML7W1DX5TOdCL+5BUgq/4Hr7k=', 'Content-Length': '0'} 2022-09-02T04:26:02.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ea509f0afcb41b0-00631185da-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.710 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:02.710 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:02.711 DEBUG:boto:Method: PUT 2022-09-02T04:26:02.711 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:02.712 DEBUG:boto:Data: 2022-09-02T04:26:02.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:26:02.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:02.714 DEBUG:boto:Port: 8000 2022-09-02T04:26:02.715 DEBUG:boto:Params: {} 2022-09-02T04:26:02.716 DEBUG:boto:Token: None 2022-09-02T04:26:02.716 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-13/myobj 2022-09-02T04:26:02.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkQPvZKnGFDoxXRZZSTvGu5U1rU= 2022-09-02T04:26:02.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkQPvZKnGFDoxXRZZSTvGu5U1rU='} 2022-09-02T04:26:02.732 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ece3b9a681d7a2c0-00631185da-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.733 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:02.734 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:02.734 DEBUG:boto:Method: HEAD 2022-09-02T04:26:02.735 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:26:02.736 DEBUG:boto:Data: 2022-09-02T04:26:02.738 DEBUG:boto:Headers: {} 2022-09-02T04:26:02.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:02.739 DEBUG:boto:Port: 8000 2022-09-02T04:26:02.739 DEBUG:boto:Params: {} 2022-09-02T04:26:02.740 DEBUG:boto:Token: None 2022-09-02T04:26:02.740 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-14/ 2022-09-02T04:26:02.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iH1IQx7bCXLE2Ons7RNUztmoPdY= 2022-09-02T04:26:02.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iH1IQx7bCXLE2Ons7RNUztmoPdY=', 'Content-Length': '0'} 2022-09-02T04:26:02.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049059ff0fc79e029-00631185da-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.745 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:02.746 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:02.747 DEBUG:boto:Method: PUT 2022-09-02T04:26:02.747 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:02.748 DEBUG:boto:Data: 2022-09-02T04:26:02.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:26:02.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:02.749 DEBUG:boto:Port: 8000 2022-09-02T04:26:02.750 DEBUG:boto:Params: {} 2022-09-02T04:26:02.750 DEBUG:boto:Token: None 2022-09-02T04:26:02.751 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-14/myobj 2022-09-02T04:26:02.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4px8HxLG6i3dXRGJb6G1beTDKPk= 2022-09-02T04:26:02.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4px8HxLG6i3dXRGJb6G1beTDKPk='} 2022-09-02T04:26:02.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035789fb62e994866-00631185da-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.764 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:02.765 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:02.765 DEBUG:boto:Method: HEAD 2022-09-02T04:26:02.766 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:26:02.766 DEBUG:boto:Data: 2022-09-02T04:26:02.767 DEBUG:boto:Headers: {} 2022-09-02T04:26:02.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:02.768 DEBUG:boto:Port: 8001 2022-09-02T04:26:02.769 DEBUG:boto:Params: {} 2022-09-02T04:26:02.769 DEBUG:boto:Token: None 2022-09-02T04:26:02.770 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-15/ 2022-09-02T04:26:02.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JkjkXeNxviQZl3oelZOWGl8z0gM= 2022-09-02T04:26:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JkjkXeNxviQZl3oelZOWGl8z0gM=', 'Content-Length': '0'} 2022-09-02T04:26:02.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ea24eada92e8bd7-00631185da-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.789 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:02.791 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:02.791 DEBUG:boto:Method: PUT 2022-09-02T04:26:02.792 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:02.793 DEBUG:boto:Data: 2022-09-02T04:26:02.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T04:26:02.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:02.808 DEBUG:boto:Port: 8001 2022-09-02T04:26:02.808 DEBUG:boto:Params: {} 2022-09-02T04:26:02.809 DEBUG:boto:Token: None 2022-09-02T04:26:02.811 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 04:26:02 GMT /xljbkf-15/myobj 2022-09-02T04:26:02.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLWMGb35Pr8yMrGEsFP9JciQztY= 2022-09-02T04:26:02.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:26:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLWMGb35Pr8yMrGEsFP9JciQztY='} 2022-09-02T04:26:02.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b4f79a8303ad248-00631185da-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:02.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:02.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:02.856 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:26:02.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:02.947 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:26:02.961 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:26:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:26:02.962 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:26:02.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:02.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:02.963 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:26:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:26:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:02.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:02.967 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092762.665862_79.1", 2022-09-02T04:26:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:26:02.665862Z" 2022-09-02T04:26:02.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:02.969 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:26:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:26:02.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:02.971 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:26:02.972 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:02.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:26:02.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:02.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:02.982 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:26:03.039 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:03.040 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:03.092 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:03.087+0000 7f7ef9cae600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:03.093 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:03.088+0000 7f7ef9cae600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:03.109 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:26:03.110 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:26:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:26:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:26:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:26:03.111 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:26:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:26:03.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:26:03.113 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:03.117 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:26:03.118 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:03.119 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:26:03.120 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:03.121 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:26:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:03.122 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:03.123 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:26:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:03.124 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:26:03.125 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:03.130 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:26:03.131 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:26:03.132 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:03.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:26:03.134 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:03.135 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092762.665862_79.1", 2022-09-02T04:26:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:26:03.136 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:03.137 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.665862Z", 2022-09-02T04:26:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:03.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:03.140 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:26:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:03.141 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:03.150 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:26:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:26:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:03.151 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:26:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:03.152 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:26:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:26:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:26:03.153 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:26:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:26:03Z" 2022-09-02T04:26:03.169 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:26:03.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092762.665862_79.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:26:02.665862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:26:03Z" } 2022-09-02T04:26:03.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092762.665862_79.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:26:02.665862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:26:03.172 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:03.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:03.174 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:26:03.175 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:26:03.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:03.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:03.178 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:26:03.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:03.229 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:03.278 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:03.252+0000 7f18dd4fe600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:03.279 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:03.253+0000 7f18dd4fe600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:03.281 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:26:03.281 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:26:03.286 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:26:03.287 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:26:03.288 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:26:03.288 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:26:03.290 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:03.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:03.290 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:26:03.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:03.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:26:03.292 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:03.292 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:03.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:26:03.293 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:03.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:26:03.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:26:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:03.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:03.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:03.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:03.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:26:03.299 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:03.299 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:03.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:26:03.300 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:03.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:26:03.301 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:03.301 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:26:03.302 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:03.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:03.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:03.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:03.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:26:03.304 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:03.304 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:03.305 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092762.659538_76.1", 2022-09-02T04:26:03.305 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:03.307 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:26:03.307 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:03.308 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.659538Z", 2022-09-02T04:26:03.308 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:03.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:03.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:03.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:03.310 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:26:03.310 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:03.311 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:03.311 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:26:03.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:03.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:26:03.313 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:03.313 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:26:03.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:03.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:03.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:03.315 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:26:03.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:26:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:26:03.318 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:26:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:03.319 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:26:03Z" 2022-09-02T04:26:03.319 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:26:03.320 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092762.659538_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:26:02.659538Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:26:03Z" } 2022-09-02T04:26:03.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092762.659538_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:26:02.659538Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:26:03.321 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:03.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.659538_76.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:03.324 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1662092762.665862_79.1 target=1_1662092762.659538_76.1 2022-09-02T04:26:08.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:08.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:08.334 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:26:08.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:08.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:08.435 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:08.434+0000 7fcc2df82600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:08.437 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:08.435+0000 7fcc2df82600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:08.455 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:26:08.456 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:26:08.456 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:26:08.456 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:26:08.457 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:26:08.457 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:26:08.457 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:08.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:08.458 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:26:08.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:08.459 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:26:08.459 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:08.459 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:08.460 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:26:08.460 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:08.460 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:26:08.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:08.462 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:26:08.463 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:08.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:08.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:08.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:08.464 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:26:08.464 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:08.464 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:08.464 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:26:08.465 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:08.465 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:26:08.465 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:08.465 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:26:08.466 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:08.466 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:08.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:08.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:08.467 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:26:08.467 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:08.468 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:08.468 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092762.665862_79.1", 2022-09-02T04:26:08.468 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:08.469 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:26:08.469 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:08.469 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.665862Z", 2022-09-02T04:26:08.469 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:08.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:08.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:08.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:26:08.470 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:26:08.471 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:26:08.471 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:26:08.471 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:26:08.471 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:26:08.472 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:26:08.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:26:08.472 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:26:08.472 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:26:08.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:08.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:26:08.473 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:26:08.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:08.474 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:26:08.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:26:08.474 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:26:08.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:26:08.475 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:26:08Z" 2022-09-02T04:26:08.475 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:26:08.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092762.665862_79.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:26:02.665862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:26:08Z" } 2022-09-02T04:26:08.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092762.665862_79.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:26:02.665862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:26:08.479 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:08.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:08.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:26:08.481 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:26:08.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:08.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:26:08.482 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:26:08.542 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:08.543 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:08.568 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:08.566+0000 7f6977ae6600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:08.569 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:08.567+0000 7f6977ae6600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:26:08.587 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:26:08.588 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:26:08.588 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:26:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:26:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:26:08.589 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:26:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:08.590 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:26:08.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:08.591 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:26:08.591 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:08.591 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:08.592 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:26:08.592 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:08.592 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:26:08.593 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:08.593 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:26:08.593 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:08.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:08.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:08.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:08.595 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:26:08.595 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:08.595 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:08.596 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:26:08.604 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:08.604 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:26:08.605 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:08.605 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:26:08.605 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:08.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:08.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:08.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:08.608 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:26:08.609 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:08.609 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:08.609 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092762.665862_79.1", 2022-09-02T04:26:08.610 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:08.610 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:26:08.611 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:08.611 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.665862Z", 2022-09-02T04:26:08.612 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:08.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:08.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:08.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:26:08.613 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:26:08.614 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:26:08.614 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:26:08.615 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:26:08.615 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:26:08.615 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:26:08.616 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:26:08.617 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:26:08.617 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:26:08.618 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:08.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:26:08.619 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:26:08.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:08.620 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:26:08.621 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:26:08.621 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:26:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:26:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:26:08Z" 2022-09-02T04:26:08.623 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:26:08.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092762.665862_79.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:26:02.665862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:26:08Z" } 2022-09-02T04:26:08.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092762.665862_79.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:26:02.665862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:26:08.626 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:08.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092762.665862_79.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:26:08.630 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:26:08.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:08.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:08.635 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:08.704 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:08.705 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:08.744 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:08.741+0000 7f75059f0600 1 waiting to reach incremental sync.. 2022-09-02T04:26:13.743 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:13.741+0000 7f75059f0600 1 waiting to reach incremental sync.. 2022-09-02T04:26:18.753 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:18.742+0000 7f75059f0600 1 waiting to reach incremental sync.. 2022-09-02T04:26:23.745 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:23.744+0000 7f75059f0600 1 bucket sync caught up with source: 2022-09-02T04:26:23.746 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:23.746 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:23.747 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:23.744+0000 7f75059f0600 0 bucket checkpoint complete 2022-09-02T04:26:23.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-13 zones={test-zone2, test-zone1} 2022-09-02T04:26:23.765 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.767 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.768 DEBUG:boto:Method: HEAD 2022-09-02T04:26:23.768 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:23.769 DEBUG:boto:Data: 2022-09-02T04:26:23.771 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.772 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.773 DEBUG:boto:Params: {} 2022-09-02T04:26:23.773 DEBUG:boto:Token: None 2022-09-02T04:26:23.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/ 2022-09-02T04:26:23.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RZAnmsQFhOdkrbvreeToi6ZPZbQ= 2022-09-02T04:26:23.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RZAnmsQFhOdkrbvreeToi6ZPZbQ=', 'Content-Length': '0'} 2022-09-02T04:26:23.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000373c5b7f5a456d4f-00631185ef-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.780 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.781 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.781 DEBUG:boto:Method: HEAD 2022-09-02T04:26:23.782 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:23.782 DEBUG:boto:Data: 2022-09-02T04:26:23.783 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.784 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.785 DEBUG:boto:Params: {} 2022-09-02T04:26:23.787 DEBUG:boto:Token: None 2022-09-02T04:26:23.788 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/ 2022-09-02T04:26:23.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RZAnmsQFhOdkrbvreeToi6ZPZbQ= 2022-09-02T04:26:23.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RZAnmsQFhOdkrbvreeToi6ZPZbQ=', 'Content-Length': '0'} 2022-09-02T04:26:23.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000133d656dfaa7b62-00631185ef-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:26:23.795 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.796 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.797 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.797 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.798 DEBUG:boto:Method: GET 2022-09-02T04:26:23.798 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.799 DEBUG:boto:Data: 2022-09-02T04:26:23.799 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.801 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.802 DEBUG:boto:Params: {} 2022-09-02T04:26:23.803 DEBUG:boto:Token: None 2022-09-02T04:26:23.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/?versions 2022-09-02T04:26:23.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q= 2022-09-02T04:26:23.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q=', 'Content-Length': '0'} 2022-09-02T04:26:23.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b57357787a7833-00631185ef-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.813 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:23.815 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:23.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:26:23.816 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.817 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.818 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.819 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.819 DEBUG:boto:Method: GET 2022-09-02T04:26:23.820 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.820 DEBUG:boto:Data: 2022-09-02T04:26:23.822 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.824 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.825 DEBUG:boto:Params: {} 2022-09-02T04:26:23.826 DEBUG:boto:Token: None 2022-09-02T04:26:23.827 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/?versions 2022-09-02T04:26:23.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q= 2022-09-02T04:26:23.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q=', 'Content-Length': '0'} 2022-09-02T04:26:23.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a5b76fe6c54fdc0-00631185ef-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.834 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:23.838 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.840 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.843 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.843 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.844 DEBUG:boto:Method: GET 2022-09-02T04:26:23.844 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.845 DEBUG:boto:Data: 2022-09-02T04:26:23.845 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.847 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.847 DEBUG:boto:Params: {} 2022-09-02T04:26:23.848 DEBUG:boto:Token: None 2022-09-02T04:26:23.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/?versions 2022-09-02T04:26:23.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q= 2022-09-02T04:26:23.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q=', 'Content-Length': '0'} 2022-09-02T04:26:23.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a1d862bc85499c-00631185ef-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.854 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:23.856 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.857 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.857 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.858 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.858 DEBUG:boto:Method: GET 2022-09-02T04:26:23.859 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:23.859 DEBUG:boto:Data: 2022-09-02T04:26:23.860 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.861 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.862 DEBUG:boto:Params: {} 2022-09-02T04:26:23.863 DEBUG:boto:Token: None 2022-09-02T04:26:23.864 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/?versions 2022-09-02T04:26:23.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q= 2022-09-02T04:26:23.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hx5Vg8Nm5YwRwHJ4bQiIIXlb09Q=', 'Content-Length': '0'} 2022-09-02T04:26:23.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea5ff80cdbc45f8-00631185ef-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.869 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:23.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:23.870 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:23.871 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:23.872 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.872 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.873 DEBUG:boto:Method: GET 2022-09-02T04:26:23.873 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.874 DEBUG:boto:Data: 2022-09-02T04:26:23.874 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.875 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.876 DEBUG:boto:Params: {} 2022-09-02T04:26:23.877 DEBUG:boto:Token: None 2022-09-02T04:26:23.877 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc= 2022-09-02T04:26:23.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc=', 'Content-Length': '0'} 2022-09-02T04:26:23.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5078ae410448ef7-00631185ef-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.882 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:23.883 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:23.883 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.884 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.885 DEBUG:boto:Method: GET 2022-09-02T04:26:23.885 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.886 DEBUG:boto:Data: 2022-09-02T04:26:23.886 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.887 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.888 DEBUG:boto:Params: {} 2022-09-02T04:26:23.889 DEBUG:boto:Token: None 2022-09-02T04:26:23.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc= 2022-09-02T04:26:23.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc=', 'Content-Length': '0'} 2022-09-02T04:26:23.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccb93273725d125e-00631185ef-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.895 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:23.895 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:23.896 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.896 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.897 DEBUG:boto:Method: HEAD 2022-09-02T04:26:23.897 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.898 DEBUG:boto:Data: 2022-09-02T04:26:23.898 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.899 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.900 DEBUG:boto:Params: {} 2022-09-02T04:26:23.901 DEBUG:boto:Token: None 2022-09-02T04:26:23.901 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L2KP+E2OvEj+wd7w+56JKzQEpvs= 2022-09-02T04:26:23.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L2KP+E2OvEj+wd7w+56JKzQEpvs=', 'Content-Length': '0'} 2022-09-02T04:26:23.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054964843da6d0d04-00631185ef-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.906 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:23.906 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:23.907 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.907 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.908 DEBUG:boto:Method: HEAD 2022-09-02T04:26:23.908 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.909 DEBUG:boto:Data: 2022-09-02T04:26:23.909 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.910 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.911 DEBUG:boto:Params: {} 2022-09-02T04:26:23.912 DEBUG:boto:Token: None 2022-09-02T04:26:23.912 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L2KP+E2OvEj+wd7w+56JKzQEpvs= 2022-09-02T04:26:23.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L2KP+E2OvEj+wd7w+56JKzQEpvs=', 'Content-Length': '0'} 2022-09-02T04:26:23.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cb56389be5fd9fe-00631185ef-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:23.919 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:23.920 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:23.921 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.922 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.923 DEBUG:boto:Method: GET 2022-09-02T04:26:23.924 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.925 DEBUG:boto:Data: 2022-09-02T04:26:23.926 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.927 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.928 DEBUG:boto:Params: {} 2022-09-02T04:26:23.929 DEBUG:boto:Token: None 2022-09-02T04:26:23.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc= 2022-09-02T04:26:23.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc=', 'Content-Length': '0'} 2022-09-02T04:26:23.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005edb23948e8eeff7-00631185ef-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.934 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:23.934 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:23.935 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.936 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.936 DEBUG:boto:Method: GET 2022-09-02T04:26:23.937 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.939 DEBUG:boto:Data: 2022-09-02T04:26:23.940 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.942 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.943 DEBUG:boto:Params: {} 2022-09-02T04:26:23.944 DEBUG:boto:Token: None 2022-09-02T04:26:23.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:23.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc= 2022-09-02T04:26:23.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zVBVS+5cUomgPD4a6qMcA8wx/wc=', 'Content-Length': '0'} 2022-09-02T04:26:23.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce4c3955236275aa-00631185ef-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.954 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.955 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.956 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:23.957 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:23.958 DEBUG:boto:Method: GET 2022-09-02T04:26:23.959 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:23.960 DEBUG:boto:Data: 2022-09-02T04:26:23.961 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.962 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:23.962 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.963 DEBUG:boto:Params: {} 2022-09-02T04:26:23.963 DEBUG:boto:Token: None 2022-09-02T04:26:23.964 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/ 2022-09-02T04:26:23.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DLLJFzJxrrtP26FdMdo0MblHWc8= 2022-09-02T04:26:23.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DLLJFzJxrrtP26FdMdo0MblHWc8=', 'Content-Length': '0'} 2022-09-02T04:26:23.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093abcc7d194f7625-00631185ef-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:23.969 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:23.970 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:23.970 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:23.971 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:23.971 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:23.972 DEBUG:boto:Method: GET 2022-09-02T04:26:23.973 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:23.974 DEBUG:boto:Data: 2022-09-02T04:26:23.975 DEBUG:boto:Headers: {} 2022-09-02T04:26:23.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:23.977 DEBUG:boto:Port: 8000 2022-09-02T04:26:23.977 DEBUG:boto:Params: {} 2022-09-02T04:26:23.983 DEBUG:boto:Token: None 2022-09-02T04:26:23.984 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:23 GMT /xljbkf-13/ 2022-09-02T04:26:23.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DLLJFzJxrrtP26FdMdo0MblHWc8= 2022-09-02T04:26:23.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DLLJFzJxrrtP26FdMdo0MblHWc8=', 'Content-Length': '0'} 2022-09-02T04:26:23.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee66e6da91e45155-00631185ef-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.034 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.035 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.036 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.037 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.038 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.038 DEBUG:boto:Data: 2022-09-02T04:26:24.041 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:24.042 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.043 DEBUG:boto:Params: {} 2022-09-02T04:26:24.043 DEBUG:boto:Token: None 2022-09-02T04:26:24.044 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo= 2022-09-02T04:26:24.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo=', 'Content-Length': '0'} 2022-09-02T04:26:24.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2bfd3d774dd9965-00631185f0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.048 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.049 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.050 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.051 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.051 DEBUG:boto:Data: 2022-09-02T04:26:24.052 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.053 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.053 DEBUG:boto:Params: {} 2022-09-02T04:26:24.054 DEBUG:boto:Token: None 2022-09-02T04:26:24.055 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo= 2022-09-02T04:26:24.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo=', 'Content-Length': '0'} 2022-09-02T04:26:24.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ff28240f4c19e44-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:24.061 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.062 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.063 DEBUG:boto:Method: GET 2022-09-02T04:26:24.064 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.065 DEBUG:boto:Data: 2022-09-02T04:26:24.066 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:24.068 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.069 DEBUG:boto:Params: {} 2022-09-02T04:26:24.070 DEBUG:boto:Token: None 2022-09-02T04:26:24.071 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A= 2022-09-02T04:26:24.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A=', 'Content-Length': '0'} 2022-09-02T04:26:24.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feff2954c7ec4944-00631185f0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.077 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.078 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.079 DEBUG:boto:Method: GET 2022-09-02T04:26:24.080 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.081 DEBUG:boto:Data: 2022-09-02T04:26:24.082 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.084 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.085 DEBUG:boto:Params: {} 2022-09-02T04:26:24.086 DEBUG:boto:Token: None 2022-09-02T04:26:24.087 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A= 2022-09-02T04:26:24.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A=', 'Content-Length': '0'} 2022-09-02T04:26:24.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000252add13ad858b8e-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.094 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.095 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.096 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.097 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.098 DEBUG:boto:Method: GET 2022-09-02T04:26:24.099 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.100 DEBUG:boto:Data: 2022-09-02T04:26:24.101 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.103 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:24.104 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.106 DEBUG:boto:Params: {} 2022-09-02T04:26:24.107 DEBUG:boto:Token: None 2022-09-02T04:26:24.108 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98= 2022-09-02T04:26:24.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98=', 'Content-Length': '0'} 2022-09-02T04:26:24.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004817eeb996153eaa-00631185f0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.116 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.118 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.118 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.119 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.120 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.121 DEBUG:boto:Method: GET 2022-09-02T04:26:24.121 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.122 DEBUG:boto:Data: 2022-09-02T04:26:24.122 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.123 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.124 DEBUG:boto:Params: {} 2022-09-02T04:26:24.125 DEBUG:boto:Token: None 2022-09-02T04:26:24.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98= 2022-09-02T04:26:24.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98=', 'Content-Length': '0'} 2022-09-02T04:26:24.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044416326b0a4a93c-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.132 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.133 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-13 zones={test-zone2, test-zone1} 2022-09-02T04:26:24.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:24.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:24.135 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:24.203 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:24.204 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:24.251+0000 7f62f5ccc600 1 bucket sync caught up with source: 2022-09-02T04:26:24.254 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:24.254 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:24.251+0000 7f62f5ccc600 0 bucket checkpoint complete 2022-09-02T04:26:24.274 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-13 zones={test-zone3, test-zone1} 2022-09-02T04:26:24.275 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.277 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.279 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.280 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:24.281 DEBUG:boto:Data: 2022-09-02T04:26:24.282 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.285 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.286 DEBUG:boto:Params: {} 2022-09-02T04:26:24.287 DEBUG:boto:Token: None 2022-09-02T04:26:24.289 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jkPdB3ZrFvbwXAZmBIGn6j7hCeo= 2022-09-02T04:26:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jkPdB3ZrFvbwXAZmBIGn6j7hCeo=', 'Content-Length': '0'} 2022-09-02T04:26:24.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004454ee37b6748f3a-00631185f0-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.305 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.306 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.307 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.308 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:24.309 DEBUG:boto:Data: 2022-09-02T04:26:24.309 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.311 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.312 DEBUG:boto:Params: {} 2022-09-02T04:26:24.314 DEBUG:boto:Token: None 2022-09-02T04:26:24.314 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jkPdB3ZrFvbwXAZmBIGn6j7hCeo= 2022-09-02T04:26:24.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jkPdB3ZrFvbwXAZmBIGn6j7hCeo=', 'Content-Length': '0'} 2022-09-02T04:26:24.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e943101b3ea43a34-00631185f0-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.325 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:26:24.327 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.329 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.331 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.333 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.334 DEBUG:boto:Method: GET 2022-09-02T04:26:24.337 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.338 DEBUG:boto:Data: 2022-09-02T04:26:24.339 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.341 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.342 DEBUG:boto:Params: {} 2022-09-02T04:26:24.343 DEBUG:boto:Token: None 2022-09-02T04:26:24.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/?versions 2022-09-02T04:26:24.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY= 2022-09-02T04:26:24.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY=', 'Content-Length': '0'} 2022-09-02T04:26:24.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af29f61a9b137405-00631185f0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.367 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.369 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:24.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:26:24.370 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.371 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.372 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.373 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.373 DEBUG:boto:Method: GET 2022-09-02T04:26:24.374 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.375 DEBUG:boto:Data: 2022-09-02T04:26:24.375 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.377 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.378 DEBUG:boto:Params: {} 2022-09-02T04:26:24.379 DEBUG:boto:Token: None 2022-09-02T04:26:24.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/?versions 2022-09-02T04:26:24.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY= 2022-09-02T04:26:24.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY=', 'Content-Length': '0'} 2022-09-02T04:26:24.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c421be8bf6461223-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.387 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.388 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:24.389 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.389 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.390 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.391 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.392 DEBUG:boto:Method: GET 2022-09-02T04:26:24.393 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.394 DEBUG:boto:Data: 2022-09-02T04:26:24.395 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.398 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.399 DEBUG:boto:Params: {} 2022-09-02T04:26:24.400 DEBUG:boto:Token: None 2022-09-02T04:26:24.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/?versions 2022-09-02T04:26:24.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY= 2022-09-02T04:26:24.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY=', 'Content-Length': '0'} 2022-09-02T04:26:24.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23786b9ce01984b-00631185f0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.408 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.410 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.411 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.412 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.413 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.415 DEBUG:boto:Method: GET 2022-09-02T04:26:24.417 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:26:24.418 DEBUG:boto:Data: 2022-09-02T04:26:24.418 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.420 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.420 DEBUG:boto:Params: {} 2022-09-02T04:26:24.421 DEBUG:boto:Token: None 2022-09-02T04:26:24.422 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/?versions 2022-09-02T04:26:24.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY= 2022-09-02T04:26:24.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+GjIlXYoETlJecSSeyOy/P8bHY=', 'Content-Length': '0'} 2022-09-02T04:26:24.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e444a41b9bd53eba-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.470 DEBUG:boto:b'xljbkf-13999falsemyobjnulltrue2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:24.473 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.474 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.475 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.476 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.477 DEBUG:boto:Method: GET 2022-09-02T04:26:24.478 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.478 DEBUG:boto:Data: 2022-09-02T04:26:24.479 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.481 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.482 DEBUG:boto:Params: {} 2022-09-02T04:26:24.483 DEBUG:boto:Token: None 2022-09-02T04:26:24.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U= 2022-09-02T04:26:24.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U=', 'Content-Length': '0'} 2022-09-02T04:26:24.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000669d9aaa4fb564a3-00631185f0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.491 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.492 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.493 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.494 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.495 DEBUG:boto:Method: GET 2022-09-02T04:26:24.496 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.497 DEBUG:boto:Data: 2022-09-02T04:26:24.498 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.499 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.500 DEBUG:boto:Params: {} 2022-09-02T04:26:24.501 DEBUG:boto:Token: None 2022-09-02T04:26:24.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U= 2022-09-02T04:26:24.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U=', 'Content-Length': '0'} 2022-09-02T04:26:24.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d33711b6bbff2d0a-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.507 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.508 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.509 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.509 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.509 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.510 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.511 DEBUG:boto:Data: 2022-09-02T04:26:24.512 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.514 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.515 DEBUG:boto:Params: {} 2022-09-02T04:26:24.516 DEBUG:boto:Token: None 2022-09-02T04:26:24.518 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hrl+BCMXScvrlKYS+XzcnfPIpXE= 2022-09-02T04:26:24.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hrl+BCMXScvrlKYS+XzcnfPIpXE=', 'Content-Length': '0'} 2022-09-02T04:26:24.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092f5b28b3565611c-00631185f0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.525 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.526 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.526 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.527 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.528 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.529 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.530 DEBUG:boto:Data: 2022-09-02T04:26:24.531 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.533 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.534 DEBUG:boto:Params: {} 2022-09-02T04:26:24.535 DEBUG:boto:Token: None 2022-09-02T04:26:24.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hrl+BCMXScvrlKYS+XzcnfPIpXE= 2022-09-02T04:26:24.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hrl+BCMXScvrlKYS+XzcnfPIpXE=', 'Content-Length': '0'} 2022-09-02T04:26:24.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073e19dd3f2fcda63-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:24.544 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.545 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.546 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.547 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.548 DEBUG:boto:Method: GET 2022-09-02T04:26:24.549 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.550 DEBUG:boto:Data: 2022-09-02T04:26:24.551 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.553 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.554 DEBUG:boto:Params: {} 2022-09-02T04:26:24.555 DEBUG:boto:Token: None 2022-09-02T04:26:24.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U= 2022-09-02T04:26:24.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U=', 'Content-Length': '0'} 2022-09-02T04:26:24.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f67ecad02272df19-00631185f0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.565 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.566 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.567 DEBUG:boto:path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.570 DEBUG:boto:auth_path=/xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.570 DEBUG:boto:Method: GET 2022-09-02T04:26:24.571 DEBUG:boto:Path: /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.571 DEBUG:boto:Data: 2022-09-02T04:26:24.572 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.573 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.574 DEBUG:boto:Params: {} 2022-09-02T04:26:24.575 DEBUG:boto:Token: None 2022-09-02T04:26:24.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj?versionId=null 2022-09-02T04:26:24.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U= 2022-09-02T04:26:24.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jnl0HgKWTetyVPifxYdPrqY4L/U=', 'Content-Length': '0'} 2022-09-02T04:26:24.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098b7fca34b7919f4-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.582 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.582 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.583 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.583 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.584 DEBUG:boto:Method: GET 2022-09-02T04:26:24.585 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.585 DEBUG:boto:Data: 2022-09-02T04:26:24.586 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.587 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.587 DEBUG:boto:Params: {} 2022-09-02T04:26:24.588 DEBUG:boto:Token: None 2022-09-02T04:26:24.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98= 2022-09-02T04:26:24.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98=', 'Content-Length': '0'} 2022-09-02T04:26:24.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067422bfeb5a5a539-00631185f0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.596 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.597 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.597 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.598 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.599 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.599 DEBUG:boto:Method: GET 2022-09-02T04:26:24.600 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.600 DEBUG:boto:Data: 2022-09-02T04:26:24.601 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.602 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.604 DEBUG:boto:Params: {} 2022-09-02T04:26:24.606 DEBUG:boto:Token: None 2022-09-02T04:26:24.607 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98= 2022-09-02T04:26:24.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98=', 'Content-Length': '0'} 2022-09-02T04:26:24.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cff64b2d25b769b-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.622 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.623 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.624 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.625 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.626 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.626 DEBUG:boto:Data: 2022-09-02T04:26:24.627 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.629 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.630 DEBUG:boto:Params: {} 2022-09-02T04:26:24.631 DEBUG:boto:Token: None 2022-09-02T04:26:24.632 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo= 2022-09-02T04:26:24.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo=', 'Content-Length': '0'} 2022-09-02T04:26:24.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf3072412e7baecc-00631185f0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.637 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.637 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.638 DEBUG:boto:Method: HEAD 2022-09-02T04:26:24.639 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.639 DEBUG:boto:Data: 2022-09-02T04:26:24.640 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.641 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.641 DEBUG:boto:Params: {} 2022-09-02T04:26:24.642 DEBUG:boto:Token: None 2022-09-02T04:26:24.643 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo= 2022-09-02T04:26:24.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WZzUj6sIjRlEszfwc0fTWFKLcEo=', 'Content-Length': '0'} 2022-09-02T04:26:24.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072dda62da01d0fb7-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:24.649 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.650 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.650 DEBUG:boto:Method: GET 2022-09-02T04:26:24.651 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.652 DEBUG:boto:Data: 2022-09-02T04:26:24.652 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.653 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.654 DEBUG:boto:Params: {} 2022-09-02T04:26:24.655 DEBUG:boto:Token: None 2022-09-02T04:26:24.656 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A= 2022-09-02T04:26:24.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A=', 'Content-Length': '0'} 2022-09-02T04:26:24.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032e090769ed71d3c-00631185f0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.707 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:24.708 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:24.708 DEBUG:boto:Method: GET 2022-09-02T04:26:24.709 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:24.712 DEBUG:boto:Data: 2022-09-02T04:26:24.713 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.714 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.715 DEBUG:boto:Params: {} 2022-09-02T04:26:24.717 DEBUG:boto:Token: None 2022-09-02T04:26:24.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/myobj 2022-09-02T04:26:24.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A= 2022-09-02T04:26:24.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bufzwLxSCJ0dbbiZQQMI8zvZj6A=', 'Content-Length': '0'} 2022-09-02T04:26:24.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dd3f16a33cebf89-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.727 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.728 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.729 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.729 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.730 DEBUG:boto:Method: GET 2022-09-02T04:26:24.731 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.732 DEBUG:boto:Data: 2022-09-02T04:26:24.733 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:24.735 DEBUG:boto:Port: 8001 2022-09-02T04:26:24.737 DEBUG:boto:Params: {} 2022-09-02T04:26:24.738 DEBUG:boto:Token: None 2022-09-02T04:26:24.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98= 2022-09-02T04:26:24.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98=', 'Content-Length': '0'} 2022-09-02T04:26:24.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4882e771f3a349-00631185f0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.745 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.746 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:24.747 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:24.748 DEBUG:boto:path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.748 DEBUG:boto:auth_path=/xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.749 DEBUG:boto:Method: GET 2022-09-02T04:26:24.750 DEBUG:boto:Path: /xljbkf-13/?prefix=myobj 2022-09-02T04:26:24.750 DEBUG:boto:Data: 2022-09-02T04:26:24.751 DEBUG:boto:Headers: {} 2022-09-02T04:26:24.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:24.753 DEBUG:boto:Port: 8000 2022-09-02T04:26:24.754 DEBUG:boto:Params: {} 2022-09-02T04:26:24.754 DEBUG:boto:Token: None 2022-09-02T04:26:24.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:24 GMT /xljbkf-13/ 2022-09-02T04:26:24.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98= 2022-09-02T04:26:24.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKEFigWXGayRecw9xxA9Vfdkp98=', 'Content-Length': '0'} 2022-09-02T04:26:24.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068cea5598a5a4c1e-00631185f0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:24.761 DEBUG:boto:b'xljbkf-13myobj1000falsemyobj2022-09-02T04:26:02.721Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:24.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-13 zones={test-zone3, test-zone1} 2022-09-02T04:26:24.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:24.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:24.764 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:24.832 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:24.833 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:24.888 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:24.884+0000 7f273aca5600 1 bucket sync caught up with source: 2022-09-02T04:26:24.888 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:24.889 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:24.889 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:24.884+0000 7f273aca5600 0 bucket checkpoint complete 2022-09-02T04:26:24.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:24.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:24.906 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:24.968 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:24.968 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:25.014 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:25.013+0000 7f21617ae600 1 bucket sync caught up with source: 2022-09-02T04:26:25.015 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:26:25.015 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:26:25.015 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:25.013+0000 7f21617ae600 0 bucket checkpoint complete 2022-09-02T04:26:25.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-14 zones={test-zone1, test-zone2} 2022-09-02T04:26:25.040 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.040 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.041 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.041 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:26:25.042 DEBUG:boto:Data: 2022-09-02T04:26:25.042 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.043 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.044 DEBUG:boto:Params: {} 2022-09-02T04:26:25.044 DEBUG:boto:Token: None 2022-09-02T04:26:25.045 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4= 2022-09-02T04:26:25.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4=', 'Content-Length': '0'} 2022-09-02T04:26:25.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7861d088da37d93-00631185f1-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.051 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.051 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.052 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.052 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:26:25.053 DEBUG:boto:Data: 2022-09-02T04:26:25.053 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.054 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.055 DEBUG:boto:Params: {} 2022-09-02T04:26:25.056 DEBUG:boto:Token: None 2022-09-02T04:26:25.056 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4= 2022-09-02T04:26:25.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4=', 'Content-Length': '0'} 2022-09-02T04:26:25.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000554ed56d80021840-00631185f1-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.060 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:26:25.061 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.061 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.062 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.063 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.064 DEBUG:boto:Method: GET 2022-09-02T04:26:25.065 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.066 DEBUG:boto:Data: 2022-09-02T04:26:25.068 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.070 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.071 DEBUG:boto:Params: {} 2022-09-02T04:26:25.072 DEBUG:boto:Token: None 2022-09-02T04:26:25.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a0955356d4eb6f9-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.081 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.083 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:25.084 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:26:25.085 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.086 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.087 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.088 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.089 DEBUG:boto:Method: GET 2022-09-02T04:26:25.090 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.091 DEBUG:boto:Data: 2022-09-02T04:26:25.093 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.094 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.095 DEBUG:boto:Params: {} 2022-09-02T04:26:25.096 DEBUG:boto:Token: None 2022-09-02T04:26:25.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c55c0b8726a7a52-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.102 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.103 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:25.104 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.104 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.105 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.106 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.107 DEBUG:boto:Method: GET 2022-09-02T04:26:25.107 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.108 DEBUG:boto:Data: 2022-09-02T04:26:25.108 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.109 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.110 DEBUG:boto:Params: {} 2022-09-02T04:26:25.111 DEBUG:boto:Token: None 2022-09-02T04:26:25.112 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008954b62e0bb88bb4-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.159 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.161 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.163 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.164 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.165 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.166 DEBUG:boto:Method: GET 2022-09-02T04:26:25.167 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.168 DEBUG:boto:Data: 2022-09-02T04:26:25.169 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.171 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.172 DEBUG:boto:Params: {} 2022-09-02T04:26:25.173 DEBUG:boto:Token: None 2022-09-02T04:26:25.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000934b26490852ceba-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.187 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:25.190 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.194 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.196 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.197 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.198 DEBUG:boto:Method: GET 2022-09-02T04:26:25.199 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.200 DEBUG:boto:Data: 2022-09-02T04:26:25.201 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.203 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.204 DEBUG:boto:Params: {} 2022-09-02T04:26:25.205 DEBUG:boto:Token: None 2022-09-02T04:26:25.206 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c5fec6749b5b85c-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.213 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.214 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.214 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.215 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.215 DEBUG:boto:Method: GET 2022-09-02T04:26:25.216 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.217 DEBUG:boto:Data: 2022-09-02T04:26:25.218 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.218 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.220 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.220 DEBUG:boto:Params: {} 2022-09-02T04:26:25.221 DEBUG:boto:Token: None 2022-09-02T04:26:25.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b6538b29746086f-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.229 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.229 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.230 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.231 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.232 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.233 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.234 DEBUG:boto:Data: 2022-09-02T04:26:25.235 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.237 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.237 DEBUG:boto:Params: {} 2022-09-02T04:26:25.238 DEBUG:boto:Token: None 2022-09-02T04:26:25.239 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk= 2022-09-02T04:26:25.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk=', 'Content-Length': '0'} 2022-09-02T04:26:25.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e98388ebcf1e75e-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.246 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.247 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.248 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.249 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.250 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.251 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.252 DEBUG:boto:Data: 2022-09-02T04:26:25.253 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.254 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.256 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.257 DEBUG:boto:Params: {} 2022-09-02T04:26:25.258 DEBUG:boto:Token: None 2022-09-02T04:26:25.261 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk= 2022-09-02T04:26:25.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk=', 'Content-Length': '0'} 2022-09-02T04:26:25.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a24c3457e8eb7b5-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:25.274 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.276 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.277 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.278 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.279 DEBUG:boto:Method: GET 2022-09-02T04:26:25.281 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.283 DEBUG:boto:Data: 2022-09-02T04:26:25.284 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.288 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.289 DEBUG:boto:Params: {} 2022-09-02T04:26:25.291 DEBUG:boto:Token: None 2022-09-02T04:26:25.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f948a3ce3ad5d707-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.313 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.314 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.315 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.315 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.316 DEBUG:boto:Method: GET 2022-09-02T04:26:25.317 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.318 DEBUG:boto:Data: 2022-09-02T04:26:25.318 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.320 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.321 DEBUG:boto:Params: {} 2022-09-02T04:26:25.321 DEBUG:boto:Token: None 2022-09-02T04:26:25.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000145bc74d59849b6f-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.338 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.340 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.341 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.342 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.342 DEBUG:boto:Method: GET 2022-09-02T04:26:25.344 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.346 DEBUG:boto:Data: 2022-09-02T04:26:25.347 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.356 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.357 DEBUG:boto:Params: {} 2022-09-02T04:26:25.359 DEBUG:boto:Token: None 2022-09-02T04:26:25.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI= 2022-09-02T04:26:25.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI=', 'Content-Length': '0'} 2022-09-02T04:26:25.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037af53ee50a20adb-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.374 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.377 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.378 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.379 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.380 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.381 DEBUG:boto:Method: GET 2022-09-02T04:26:25.382 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.383 DEBUG:boto:Data: 2022-09-02T04:26:25.385 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.393 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.395 DEBUG:boto:Params: {} 2022-09-02T04:26:25.397 DEBUG:boto:Token: None 2022-09-02T04:26:25.399 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI= 2022-09-02T04:26:25.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI=', 'Content-Length': '0'} 2022-09-02T04:26:25.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088db6110d217d830-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.429 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.431 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.432 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.433 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.434 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:25.435 DEBUG:boto:Data: 2022-09-02T04:26:25.436 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.441 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.442 DEBUG:boto:Params: {} 2022-09-02T04:26:25.443 DEBUG:boto:Token: None 2022-09-02T04:26:25.444 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj 2022-09-02T04:26:25.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ivephGTL1Kgss/JtBFUwmEGCIXg= 2022-09-02T04:26:25.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ivephGTL1Kgss/JtBFUwmEGCIXg=', 'Content-Length': '0'} 2022-09-02T04:26:25.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e6fe19e8ccbf5d5-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.453 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.454 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.455 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.457 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:25.458 DEBUG:boto:Data: 2022-09-02T04:26:25.459 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.462 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.463 DEBUG:boto:Params: {} 2022-09-02T04:26:25.464 DEBUG:boto:Token: None 2022-09-02T04:26:25.467 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj 2022-09-02T04:26:25.472 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ivephGTL1Kgss/JtBFUwmEGCIXg= 2022-09-02T04:26:25.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ivephGTL1Kgss/JtBFUwmEGCIXg=', 'Content-Length': '0'} 2022-09-02T04:26:25.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6f600a88575d730-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:25.480 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.481 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.482 DEBUG:boto:Method: GET 2022-09-02T04:26:25.484 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:25.486 DEBUG:boto:Data: 2022-09-02T04:26:25.487 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.491 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.493 DEBUG:boto:Params: {} 2022-09-02T04:26:25.495 DEBUG:boto:Token: None 2022-09-02T04:26:25.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj 2022-09-02T04:26:25.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kRBMJeYkG0Mh09ibOQDSUxFz2Qg= 2022-09-02T04:26:25.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kRBMJeYkG0Mh09ibOQDSUxFz2Qg=', 'Content-Length': '0'} 2022-09-02T04:26:25.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022d417bc1974b75d-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.504 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.506 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.507 DEBUG:boto:Method: GET 2022-09-02T04:26:25.509 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:25.511 DEBUG:boto:Data: 2022-09-02T04:26:25.512 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.514 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.515 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.516 DEBUG:boto:Params: {} 2022-09-02T04:26:25.518 DEBUG:boto:Token: None 2022-09-02T04:26:25.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj 2022-09-02T04:26:25.522 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kRBMJeYkG0Mh09ibOQDSUxFz2Qg= 2022-09-02T04:26:25.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kRBMJeYkG0Mh09ibOQDSUxFz2Qg=', 'Content-Length': '0'} 2022-09-02T04:26:25.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb78f6c2173fa936-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.529 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.530 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.531 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.532 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.533 DEBUG:boto:Method: GET 2022-09-02T04:26:25.534 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.535 DEBUG:boto:Data: 2022-09-02T04:26:25.536 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:25.539 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.540 DEBUG:boto:Params: {} 2022-09-02T04:26:25.542 DEBUG:boto:Token: None 2022-09-02T04:26:25.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI= 2022-09-02T04:26:25.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI=', 'Content-Length': '0'} 2022-09-02T04:26:25.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000644b16746a3388a5-00631185f1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.552 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.554 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.555 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.557 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.558 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.560 DEBUG:boto:Method: GET 2022-09-02T04:26:25.561 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:25.563 DEBUG:boto:Data: 2022-09-02T04:26:25.564 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.565 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.566 DEBUG:boto:Params: {} 2022-09-02T04:26:25.567 DEBUG:boto:Token: None 2022-09-02T04:26:25.568 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI= 2022-09-02T04:26:25.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YgLB1x13AGM6lkAmCu/UvaSATLI=', 'Content-Length': '0'} 2022-09-02T04:26:25.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000965c767d475ede8d-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.573 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.574 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-14 zones={test-zone1, test-zone2} 2022-09-02T04:26:25.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:25.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:25.577 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:25.638 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:25.639 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:25.689 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:25.687+0000 7f9cfc5cf600 1 bucket sync caught up with source: 2022-09-02T04:26:25.690 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:26:25.691 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:26:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:25.687+0000 7f9cfc5cf600 0 bucket checkpoint complete 2022-09-02T04:26:25.701 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-14 zones={test-zone3, test-zone2} 2022-09-02T04:26:25.702 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.703 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.705 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.706 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:26:25.707 DEBUG:boto:Data: 2022-09-02T04:26:25.709 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:25.712 DEBUG:boto:Port: 8001 2022-09-02T04:26:25.713 DEBUG:boto:Params: {} 2022-09-02T04:26:25.715 DEBUG:boto:Token: None 2022-09-02T04:26:25.716 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4= 2022-09-02T04:26:25.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4=', 'Content-Length': '0'} 2022-09-02T04:26:25.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052716bc6def157af-00631185f1-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.727 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.728 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.729 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.731 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:26:25.732 DEBUG:boto:Data: 2022-09-02T04:26:25.734 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.737 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.738 DEBUG:boto:Params: {} 2022-09-02T04:26:25.740 DEBUG:boto:Token: None 2022-09-02T04:26:25.742 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/ 2022-09-02T04:26:25.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4= 2022-09-02T04:26:25.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UeNWbNrNizFN2w+Ha44WHAMYDW4=', 'Content-Length': '0'} 2022-09-02T04:26:25.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008108813da69a242c-00631185f1-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:26:25.754 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.755 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.756 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.757 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.758 DEBUG:boto:Method: GET 2022-09-02T04:26:25.759 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.760 DEBUG:boto:Data: 2022-09-02T04:26:25.761 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:25.763 DEBUG:boto:Port: 8001 2022-09-02T04:26:25.764 DEBUG:boto:Params: {} 2022-09-02T04:26:25.765 DEBUG:boto:Token: None 2022-09-02T04:26:25.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f85eea5e5d7b908-00631185f1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.775 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.776 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:25.777 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:26:25.778 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.780 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.781 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.782 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.783 DEBUG:boto:Method: GET 2022-09-02T04:26:25.785 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.787 DEBUG:boto:Data: 2022-09-02T04:26:25.788 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.797 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.798 DEBUG:boto:Params: {} 2022-09-02T04:26:25.799 DEBUG:boto:Token: None 2022-09-02T04:26:25.800 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b47805ad959c32f4-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.806 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.808 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:25.809 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.811 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.812 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.813 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.813 DEBUG:boto:Method: GET 2022-09-02T04:26:25.814 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.815 DEBUG:boto:Data: 2022-09-02T04:26:25.816 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:25.817 DEBUG:boto:Port: 8001 2022-09-02T04:26:25.818 DEBUG:boto:Params: {} 2022-09-02T04:26:25.819 DEBUG:boto:Token: None 2022-09-02T04:26:25.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093e05b3b8d8c34ec-00631185f1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.835 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.838 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:25.839 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:25.840 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.841 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.842 DEBUG:boto:Method: GET 2022-09-02T04:26:25.845 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:26:25.846 DEBUG:boto:Data: 2022-09-02T04:26:25.847 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.849 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.850 DEBUG:boto:Params: {} 2022-09-02T04:26:25.851 DEBUG:boto:Token: None 2022-09-02T04:26:25.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/?versions 2022-09-02T04:26:25.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA= 2022-09-02T04:26:25.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fvqqHj9dRm7lJljNFhUEgDagXxA=', 'Content-Length': '0'} 2022-09-02T04:26:25.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb37c2c3a6fca26-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.857 DEBUG:boto:b'xljbkf-14999falsemyobjnulltrue2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:25.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:25.858 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.859 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.860 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.860 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.861 DEBUG:boto:Method: GET 2022-09-02T04:26:25.861 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.862 DEBUG:boto:Data: 2022-09-02T04:26:25.863 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:25.864 DEBUG:boto:Port: 8001 2022-09-02T04:26:25.864 DEBUG:boto:Params: {} 2022-09-02T04:26:25.865 DEBUG:boto:Token: None 2022-09-02T04:26:25.867 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceac1dd65108c7fc-00631185f1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.872 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.872 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.873 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.880 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.882 DEBUG:boto:Method: GET 2022-09-02T04:26:25.883 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.885 DEBUG:boto:Data: 2022-09-02T04:26:25.885 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.887 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.888 DEBUG:boto:Params: {} 2022-09-02T04:26:25.889 DEBUG:boto:Token: None 2022-09-02T04:26:25.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098ff988504f08dec-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.900 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.901 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.903 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.904 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.905 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.906 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.907 DEBUG:boto:Data: 2022-09-02T04:26:25.908 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:25.910 DEBUG:boto:Port: 8001 2022-09-02T04:26:25.915 DEBUG:boto:Params: {} 2022-09-02T04:26:25.916 DEBUG:boto:Token: None 2022-09-02T04:26:25.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk= 2022-09-02T04:26:25.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk=', 'Content-Length': '0'} 2022-09-02T04:26:25.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9b8fd27f90804f7-00631185f1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.923 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.924 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.925 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.926 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.933 DEBUG:boto:Method: HEAD 2022-09-02T04:26:25.934 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.935 DEBUG:boto:Data: 2022-09-02T04:26:25.936 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.938 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.939 DEBUG:boto:Params: {} 2022-09-02T04:26:25.941 DEBUG:boto:Token: None 2022-09-02T04:26:25.943 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk= 2022-09-02T04:26:25.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5K9V7QdRRh5AD6mTadRyjnHhNk=', 'Content-Length': '0'} 2022-09-02T04:26:25.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0f3507bc887cc9a-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:25.952 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.953 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.955 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.956 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.957 DEBUG:boto:Method: GET 2022-09-02T04:26:25.960 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.961 DEBUG:boto:Data: 2022-09-02T04:26:25.961 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:25.964 DEBUG:boto:Port: 8001 2022-09-02T04:26:25.964 DEBUG:boto:Params: {} 2022-09-02T04:26:25.965 DEBUG:boto:Token: None 2022-09-02T04:26:25.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a24993b38fdc400a-00631185f1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:25.975 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:25.976 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:25.977 DEBUG:boto:path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.978 DEBUG:boto:auth_path=/xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.978 DEBUG:boto:Method: GET 2022-09-02T04:26:25.979 DEBUG:boto:Path: /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.979 DEBUG:boto:Data: 2022-09-02T04:26:25.980 DEBUG:boto:Headers: {} 2022-09-02T04:26:25.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:25.981 DEBUG:boto:Port: 8000 2022-09-02T04:26:25.982 DEBUG:boto:Params: {} 2022-09-02T04:26:25.983 DEBUG:boto:Token: None 2022-09-02T04:26:25.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:25 GMT /xljbkf-14/myobj?versionId=null 2022-09-02T04:26:25.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA= 2022-09-02T04:26:25.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8froMrqJRy6776k7KcEe/+1jedA=', 'Content-Length': '0'} 2022-09-02T04:26:25.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000030e1f6a1c28976c-00631185f1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.030 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:26.031 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:26.032 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.032 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.033 DEBUG:boto:Method: GET 2022-09-02T04:26:26.034 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.034 DEBUG:boto:Data: 2022-09-02T04:26:26.036 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.037 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.038 DEBUG:boto:Params: {} 2022-09-02T04:26:26.038 DEBUG:boto:Token: None 2022-09-02T04:26:26.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/ 2022-09-02T04:26:26.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI= 2022-09-02T04:26:26.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI=', 'Content-Length': '0'} 2022-09-02T04:26:26.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6290583a5b0aa8c-00631185f2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.044 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.045 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:26.046 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:26.047 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.047 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.048 DEBUG:boto:Method: GET 2022-09-02T04:26:26.048 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.049 DEBUG:boto:Data: 2022-09-02T04:26:26.049 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:26.050 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.051 DEBUG:boto:Params: {} 2022-09-02T04:26:26.052 DEBUG:boto:Token: None 2022-09-02T04:26:26.052 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/ 2022-09-02T04:26:26.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI= 2022-09-02T04:26:26.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI=', 'Content-Length': '0'} 2022-09-02T04:26:26.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eec460238209a20-00631185f2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.057 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.058 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:26.059 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:26.059 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.060 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:26.060 DEBUG:boto:Data: 2022-09-02T04:26:26.061 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.062 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.062 DEBUG:boto:Params: {} 2022-09-02T04:26:26.063 DEBUG:boto:Token: None 2022-09-02T04:26:26.063 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/myobj 2022-09-02T04:26:26.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8fnfNfzENQvQQNL1uGlkOtZrTwc= 2022-09-02T04:26:26.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8fnfNfzENQvQQNL1uGlkOtZrTwc=', 'Content-Length': '0'} 2022-09-02T04:26:26.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b691f034741d1522-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.068 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:26.068 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:26.069 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.069 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:26.070 DEBUG:boto:Data: 2022-09-02T04:26:26.070 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:26.072 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.072 DEBUG:boto:Params: {} 2022-09-02T04:26:26.073 DEBUG:boto:Token: None 2022-09-02T04:26:26.073 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/myobj 2022-09-02T04:26:26.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8fnfNfzENQvQQNL1uGlkOtZrTwc= 2022-09-02T04:26:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8fnfNfzENQvQQNL1uGlkOtZrTwc=', 'Content-Length': '0'} 2022-09-02T04:26:26.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000343fdd7248c8537d-00631185f2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:26.078 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:26.079 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:26.079 DEBUG:boto:Method: GET 2022-09-02T04:26:26.080 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:26.080 DEBUG:boto:Data: 2022-09-02T04:26:26.081 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.082 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.083 DEBUG:boto:Params: {} 2022-09-02T04:26:26.083 DEBUG:boto:Token: None 2022-09-02T04:26:26.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/myobj 2022-09-02T04:26:26.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LJmwuzXRudGx1xbHtxj4jH2Sq5Y= 2022-09-02T04:26:26.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LJmwuzXRudGx1xbHtxj4jH2Sq5Y=', 'Content-Length': '0'} 2022-09-02T04:26:26.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d93f82eacc5d8842-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.138 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:26:26.140 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:26:26.141 DEBUG:boto:Method: GET 2022-09-02T04:26:26.142 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:26:26.144 DEBUG:boto:Data: 2022-09-02T04:26:26.145 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:26.147 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.149 DEBUG:boto:Params: {} 2022-09-02T04:26:26.150 DEBUG:boto:Token: None 2022-09-02T04:26:26.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/myobj 2022-09-02T04:26:26.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LJmwuzXRudGx1xbHtxj4jH2Sq5Y= 2022-09-02T04:26:26.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LJmwuzXRudGx1xbHtxj4jH2Sq5Y=', 'Content-Length': '0'} 2022-09-02T04:26:26.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc063a4729a28f49-00631185f2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.158 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:26.159 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:26.160 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.161 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.162 DEBUG:boto:Method: GET 2022-09-02T04:26:26.163 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.164 DEBUG:boto:Data: 2022-09-02T04:26:26.165 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.168 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.169 DEBUG:boto:Params: {} 2022-09-02T04:26:26.170 DEBUG:boto:Token: None 2022-09-02T04:26:26.171 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/ 2022-09-02T04:26:26.172 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI= 2022-09-02T04:26:26.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI=', 'Content-Length': '0'} 2022-09-02T04:26:26.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069db935575df0c0a-00631185f2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.177 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.179 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:26:26.180 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:26:26.181 DEBUG:boto:path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.182 DEBUG:boto:auth_path=/xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.183 DEBUG:boto:Method: GET 2022-09-02T04:26:26.184 DEBUG:boto:Path: /xljbkf-14/?prefix=myobj 2022-09-02T04:26:26.185 DEBUG:boto:Data: 2022-09-02T04:26:26.186 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:26.188 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.189 DEBUG:boto:Params: {} 2022-09-02T04:26:26.190 DEBUG:boto:Token: None 2022-09-02T04:26:26.191 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-14/ 2022-09-02T04:26:26.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI= 2022-09-02T04:26:26.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9wsmIQjZvjQHa9bYHN8tqgAeVGI=', 'Content-Length': '0'} 2022-09-02T04:26:26.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd12b565da3f0453-00631185f2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.198 DEBUG:boto:b'xljbkf-14myobj1000falsemyobj2022-09-02T04:26:02.755Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.200 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-14 zones={test-zone3, test-zone2} 2022-09-02T04:26:26.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:26.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:26.203 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:26.295 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:26.293+0000 7fd374a81600 1 bucket sync caught up with source: 2022-09-02T04:26:26.295 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:26:26.296 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:26:26.296 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:26:26.293+0000 7fd374a81600 0 bucket checkpoint complete 2022-09-02T04:26:26.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-15 zones={test-zone1, test-zone3} 2022-09-02T04:26:26.320 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.320 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.321 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.321 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:26:26.322 DEBUG:boto:Data: 2022-09-02T04:26:26.323 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.324 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.324 DEBUG:boto:Params: {} 2022-09-02T04:26:26.325 DEBUG:boto:Token: None 2022-09-02T04:26:26.325 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ= 2022-09-02T04:26:26.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5987dd9e67e6b02-00631185f2-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.330 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.331 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.331 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.332 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:26:26.333 DEBUG:boto:Data: 2022-09-02T04:26:26.333 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.334 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.335 DEBUG:boto:Params: {} 2022-09-02T04:26:26.335 DEBUG:boto:Token: None 2022-09-02T04:26:26.336 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ= 2022-09-02T04:26:26.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006774aea49d1a5b69-00631185f2-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.341 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:26:26.341 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.342 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.343 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.344 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.344 DEBUG:boto:Method: GET 2022-09-02T04:26:26.345 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.346 DEBUG:boto:Data: 2022-09-02T04:26:26.346 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.348 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.348 DEBUG:boto:Params: {} 2022-09-02T04:26:26.349 DEBUG:boto:Token: None 2022-09-02T04:26:26.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/?versions 2022-09-02T04:26:26.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ= 2022-09-02T04:26:26.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a351d16613d9a98-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.354 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.355 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:26.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:26:26.356 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.357 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.357 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.358 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.358 DEBUG:boto:Method: GET 2022-09-02T04:26:26.359 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.360 DEBUG:boto:Data: 2022-09-02T04:26:26.361 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.362 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.362 DEBUG:boto:Params: {} 2022-09-02T04:26:26.364 DEBUG:boto:Token: None 2022-09-02T04:26:26.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/?versions 2022-09-02T04:26:26.370 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ= 2022-09-02T04:26:26.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5e324a3af76c4e-00631185f2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.374 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.375 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:26.376 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.376 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.377 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.378 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.378 DEBUG:boto:Method: GET 2022-09-02T04:26:26.379 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.380 DEBUG:boto:Data: 2022-09-02T04:26:26.380 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.381 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.389 DEBUG:boto:Params: {} 2022-09-02T04:26:26.390 DEBUG:boto:Token: None 2022-09-02T04:26:26.391 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/?versions 2022-09-02T04:26:26.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ= 2022-09-02T04:26:26.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0b9a67575ea951-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.404 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.405 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.405 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.406 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.406 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.407 DEBUG:boto:Method: GET 2022-09-02T04:26:26.407 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.408 DEBUG:boto:Data: 2022-09-02T04:26:26.408 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.409 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.410 DEBUG:boto:Params: {} 2022-09-02T04:26:26.411 DEBUG:boto:Token: None 2022-09-02T04:26:26.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/?versions 2022-09-02T04:26:26.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ= 2022-09-02T04:26:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f004a559c21ed7f-00631185f2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.415 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:26.417 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.417 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.418 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.419 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.419 DEBUG:boto:Method: GET 2022-09-02T04:26:26.420 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.420 DEBUG:boto:Data: 2022-09-02T04:26:26.421 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.422 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.422 DEBUG:boto:Params: {} 2022-09-02T04:26:26.423 DEBUG:boto:Token: None 2022-09-02T04:26:26.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8= 2022-09-02T04:26:26.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8=', 'Content-Length': '0'} 2022-09-02T04:26:26.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8b8ad6ec6256e49-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.428 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.429 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.429 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.430 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.430 DEBUG:boto:Method: GET 2022-09-02T04:26:26.431 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.431 DEBUG:boto:Data: 2022-09-02T04:26:26.432 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.433 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.433 DEBUG:boto:Params: {} 2022-09-02T04:26:26.434 DEBUG:boto:Token: None 2022-09-02T04:26:26.434 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8= 2022-09-02T04:26:26.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8=', 'Content-Length': '0'} 2022-09-02T04:26:26.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e65a1ba5b2356cc1-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.442 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.444 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.445 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.446 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.447 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.448 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.449 DEBUG:boto:Data: 2022-09-02T04:26:26.450 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.453 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.453 DEBUG:boto:Params: {} 2022-09-02T04:26:26.456 DEBUG:boto:Token: None 2022-09-02T04:26:26.457 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UbIOxInesI+D2oDrrSJh6SeKrQs= 2022-09-02T04:26:26.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UbIOxInesI+D2oDrrSJh6SeKrQs=', 'Content-Length': '0'} 2022-09-02T04:26:26.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2e58543898cc44a-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.463 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.464 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.464 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.465 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.466 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.466 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.467 DEBUG:boto:Data: 2022-09-02T04:26:26.468 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.470 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.472 DEBUG:boto:Params: {} 2022-09-02T04:26:26.473 DEBUG:boto:Token: None 2022-09-02T04:26:26.473 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UbIOxInesI+D2oDrrSJh6SeKrQs= 2022-09-02T04:26:26.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UbIOxInesI+D2oDrrSJh6SeKrQs=', 'Content-Length': '0'} 2022-09-02T04:26:26.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084a179b0f7f513c0-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:26.481 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.482 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.483 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.484 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.485 DEBUG:boto:Method: GET 2022-09-02T04:26:26.487 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.487 DEBUG:boto:Data: 2022-09-02T04:26:26.488 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.490 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.491 DEBUG:boto:Params: {} 2022-09-02T04:26:26.493 DEBUG:boto:Token: None 2022-09-02T04:26:26.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8= 2022-09-02T04:26:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8=', 'Content-Length': '0'} 2022-09-02T04:26:26.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e511303c02aa6271-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.502 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.503 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.504 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.505 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.506 DEBUG:boto:Method: GET 2022-09-02T04:26:26.508 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.510 DEBUG:boto:Data: 2022-09-02T04:26:26.511 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.512 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.512 DEBUG:boto:Params: {} 2022-09-02T04:26:26.513 DEBUG:boto:Token: None 2022-09-02T04:26:26.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:26.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8= 2022-09-02T04:26:26.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7MmZbkdT3th4m7Fc4XfyEbxtVL8=', 'Content-Length': '0'} 2022-09-02T04:26:26.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013ee7461c87c206c-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.558 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.561 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.563 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.563 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.564 DEBUG:boto:Method: GET 2022-09-02T04:26:26.565 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.565 DEBUG:boto:Data: 2022-09-02T04:26:26.566 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.568 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.569 DEBUG:boto:Params: {} 2022-09-02T04:26:26.572 DEBUG:boto:Token: None 2022-09-02T04:26:26.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw= 2022-09-02T04:26:26.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw=', 'Content-Length': '0'} 2022-09-02T04:26:26.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb532b5b553dbbff-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.580 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.582 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.584 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.585 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.585 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.587 DEBUG:boto:Method: GET 2022-09-02T04:26:26.589 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.590 DEBUG:boto:Data: 2022-09-02T04:26:26.592 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.593 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.594 DEBUG:boto:Params: {} 2022-09-02T04:26:26.596 DEBUG:boto:Token: None 2022-09-02T04:26:26.597 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw= 2022-09-02T04:26:26.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw=', 'Content-Length': '0'} 2022-09-02T04:26:26.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7d18c875d8214d-00631185f2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.606 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.607 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.608 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.608 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.609 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:26.609 DEBUG:boto:Data: 2022-09-02T04:26:26.610 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.611 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.612 DEBUG:boto:Params: {} 2022-09-02T04:26:26.613 DEBUG:boto:Token: None 2022-09-02T04:26:26.613 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj 2022-09-02T04:26:26.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YtuZle/sndTOlNHYp63lvFrpT5U= 2022-09-02T04:26:26.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YtuZle/sndTOlNHYp63lvFrpT5U=', 'Content-Length': '0'} 2022-09-02T04:26:26.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093b6416cf62c1390-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.618 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.619 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.619 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.620 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:26.620 DEBUG:boto:Data: 2022-09-02T04:26:26.621 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.624 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.625 DEBUG:boto:Params: {} 2022-09-02T04:26:26.626 DEBUG:boto:Token: None 2022-09-02T04:26:26.627 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj 2022-09-02T04:26:26.628 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YtuZle/sndTOlNHYp63lvFrpT5U= 2022-09-02T04:26:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YtuZle/sndTOlNHYp63lvFrpT5U=', 'Content-Length': '0'} 2022-09-02T04:26:26.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bef9a96175daabc3-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:26.635 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.636 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.637 DEBUG:boto:Method: GET 2022-09-02T04:26:26.637 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:26.638 DEBUG:boto:Data: 2022-09-02T04:26:26.639 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.640 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.640 DEBUG:boto:Params: {} 2022-09-02T04:26:26.641 DEBUG:boto:Token: None 2022-09-02T04:26:26.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj 2022-09-02T04:26:26.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GfWgKTNLMN1Ma0MkN80zG9wrNsU= 2022-09-02T04:26:26.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GfWgKTNLMN1Ma0MkN80zG9wrNsU=', 'Content-Length': '0'} 2022-09-02T04:26:26.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030da68346275b215-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.648 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:26.648 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:26.649 DEBUG:boto:Method: GET 2022-09-02T04:26:26.649 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:26.650 DEBUG:boto:Data: 2022-09-02T04:26:26.651 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.652 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.652 DEBUG:boto:Params: {} 2022-09-02T04:26:26.653 DEBUG:boto:Token: None 2022-09-02T04:26:26.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/myobj 2022-09-02T04:26:26.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GfWgKTNLMN1Ma0MkN80zG9wrNsU= 2022-09-02T04:26:26.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GfWgKTNLMN1Ma0MkN80zG9wrNsU=', 'Content-Length': '0'} 2022-09-02T04:26:26.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010be1007192489d8-00631185f2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.661 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.662 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.663 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.665 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.666 DEBUG:boto:Method: GET 2022-09-02T04:26:26.666 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.667 DEBUG:boto:Data: 2022-09-02T04:26:26.668 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:26.669 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.670 DEBUG:boto:Params: {} 2022-09-02T04:26:26.671 DEBUG:boto:Token: None 2022-09-02T04:26:26.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw= 2022-09-02T04:26:26.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw=', 'Content-Length': '0'} 2022-09-02T04:26:26.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda66c7abba1c066-00631185f2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.726 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.728 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.729 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.730 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.730 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.731 DEBUG:boto:Method: GET 2022-09-02T04:26:26.732 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:26.733 DEBUG:boto:Data: 2022-09-02T04:26:26.733 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.734 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.735 DEBUG:boto:Params: {} 2022-09-02T04:26:26.736 DEBUG:boto:Token: None 2022-09-02T04:26:26.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw= 2022-09-02T04:26:26.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8iWDEdgrauQeOsex6c/mIzqOlw=', 'Content-Length': '0'} 2022-09-02T04:26:26.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020349213b5f2d877-00631185f2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.742 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.743 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-15 zones={test-zone1, test-zone3} 2022-09-02T04:26:26.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:26.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:26.746 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:26.813 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:26.814 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:26.858 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:26.856+0000 7fceb326a600 1 bucket sync caught up with source: 2022-09-02T04:26:26.859 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:26:26.860 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:26:26.860 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:26.856+0000 7fceb326a600 0 bucket checkpoint complete 2022-09-02T04:26:26.879 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-15 zones={test-zone2, test-zone3} 2022-09-02T04:26:26.880 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.881 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.882 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.889 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:26:26.890 DEBUG:boto:Data: 2022-09-02T04:26:26.891 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:26.893 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.894 DEBUG:boto:Params: {} 2022-09-02T04:26:26.895 DEBUG:boto:Token: None 2022-09-02T04:26:26.896 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ= 2022-09-02T04:26:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e14fe5b7064c7c-00631185f2-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.909 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.910 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.913 DEBUG:boto:Method: HEAD 2022-09-02T04:26:26.914 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:26:26.916 DEBUG:boto:Data: 2022-09-02T04:26:26.917 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.920 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.921 DEBUG:boto:Params: {} 2022-09-02T04:26:26.922 DEBUG:boto:Token: None 2022-09-02T04:26:26.924 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/ 2022-09-02T04:26:26.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ= 2022-09-02T04:26:26.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/JD3F1OiJx6P/JIWJJGxotqsorQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f6200ca98bc2b76-00631185f2-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:26:26.934 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.935 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.936 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.937 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.938 DEBUG:boto:Method: GET 2022-09-02T04:26:26.939 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.940 DEBUG:boto:Data: 2022-09-02T04:26:26.940 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:26.941 DEBUG:boto:Port: 8000 2022-09-02T04:26:26.942 DEBUG:boto:Params: {} 2022-09-02T04:26:26.944 DEBUG:boto:Token: None 2022-09-02T04:26:26.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/?versions 2022-09-02T04:26:26.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ= 2022-09-02T04:26:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ=', 'Content-Length': '0'} 2022-09-02T04:26:26.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cafbdbff8e6db0b-00631185f2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:26.960 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:26.962 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:26.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:26:26.963 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:26.964 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:26.965 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.966 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.973 DEBUG:boto:Method: GET 2022-09-02T04:26:26.974 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:26.975 DEBUG:boto:Data: 2022-09-02T04:26:26.977 DEBUG:boto:Headers: {} 2022-09-02T04:26:26.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:26.980 DEBUG:boto:Port: 8001 2022-09-02T04:26:26.981 DEBUG:boto:Params: {} 2022-09-02T04:26:26.982 DEBUG:boto:Token: None 2022-09-02T04:26:26.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:26 GMT /xljbkf-15/?versions 2022-09-02T04:26:26.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ= 2022-09-02T04:26:26.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fSOpRW2qWYUegGX3Zx57xu4eMQ=', 'Content-Length': '0'} 2022-09-02T04:26:27.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbede7ee30773676-00631185f3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.016 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.017 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-09-02T04:26:27.018 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:27.019 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:27.020 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:27.021 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:27.022 DEBUG:boto:Method: GET 2022-09-02T04:26:27.023 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:27.025 DEBUG:boto:Data: 2022-09-02T04:26:27.026 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.028 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.029 DEBUG:boto:Params: {} 2022-09-02T04:26:27.031 DEBUG:boto:Token: None 2022-09-02T04:26:27.032 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/?versions 2022-09-02T04:26:27.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8yfVOGez4zstvgj4P1XsVWbeNuM= 2022-09-02T04:26:27.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8yfVOGez4zstvgj4P1XsVWbeNuM=', 'Content-Length': '0'} 2022-09-02T04:26:27.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086bf0f53b8147e64-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.049 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.051 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:27.052 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:27.053 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:27.055 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:27.056 DEBUG:boto:Method: GET 2022-09-02T04:26:27.056 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:26:27.057 DEBUG:boto:Data: 2022-09-02T04:26:27.058 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.060 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.062 DEBUG:boto:Params: {} 2022-09-02T04:26:27.063 DEBUG:boto:Token: None 2022-09-02T04:26:27.065 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/?versions 2022-09-02T04:26:27.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8yfVOGez4zstvgj4P1XsVWbeNuM= 2022-09-02T04:26:27.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8yfVOGez4zstvgj4P1XsVWbeNuM=', 'Content-Length': '0'} 2022-09-02T04:26:27.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ac5978899df40b-00631185f3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.085 DEBUG:boto:b'xljbkf-15999falsemyobjnulltrue2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:27.088 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.089 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.090 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.091 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.092 DEBUG:boto:Method: GET 2022-09-02T04:26:27.093 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.096 DEBUG:boto:Data: 2022-09-02T04:26:27.097 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.100 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.101 DEBUG:boto:Params: {} 2022-09-02T04:26:27.102 DEBUG:boto:Token: None 2022-09-02T04:26:27.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4= 2022-09-02T04:26:27.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4=', 'Content-Length': '0'} 2022-09-02T04:26:27.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a28d015b2492de8-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.117 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.119 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.120 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.122 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.122 DEBUG:boto:Method: GET 2022-09-02T04:26:27.123 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.123 DEBUG:boto:Data: 2022-09-02T04:26:27.124 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.127 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.129 DEBUG:boto:Params: {} 2022-09-02T04:26:27.130 DEBUG:boto:Token: None 2022-09-02T04:26:27.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4= 2022-09-02T04:26:27.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4=', 'Content-Length': '0'} 2022-09-02T04:26:27.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd0e264163ae6923-00631185f3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.183 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.186 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.187 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.188 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.189 DEBUG:boto:Method: HEAD 2022-09-02T04:26:27.190 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.191 DEBUG:boto:Data: 2022-09-02T04:26:27.192 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.194 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.195 DEBUG:boto:Params: {} 2022-09-02T04:26:27.196 DEBUG:boto:Token: None 2022-09-02T04:26:27.197 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pxMRjn9ez0F4zszHyxExI2Z/Uww= 2022-09-02T04:26:27.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pxMRjn9ez0F4zszHyxExI2Z/Uww=', 'Content-Length': '0'} 2022-09-02T04:26:27.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c05a09c72d0afe50-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.203 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.204 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.204 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.205 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.205 DEBUG:boto:Method: HEAD 2022-09-02T04:26:27.207 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.207 DEBUG:boto:Data: 2022-09-02T04:26:27.208 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.209 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.209 DEBUG:boto:Params: {} 2022-09-02T04:26:27.210 DEBUG:boto:Token: None 2022-09-02T04:26:27.211 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pxMRjn9ez0F4zszHyxExI2Z/Uww= 2022-09-02T04:26:27.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pxMRjn9ez0F4zszHyxExI2Z/Uww=', 'Content-Length': '0'} 2022-09-02T04:26:27.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045961a366ccec53a-00631185f3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:27.217 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.218 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.219 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.220 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.221 DEBUG:boto:Method: GET 2022-09-02T04:26:27.223 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.223 DEBUG:boto:Data: 2022-09-02T04:26:27.224 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.225 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.225 DEBUG:boto:Params: {} 2022-09-02T04:26:27.226 DEBUG:boto:Token: None 2022-09-02T04:26:27.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4= 2022-09-02T04:26:27.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4=', 'Content-Length': '0'} 2022-09-02T04:26:27.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0f73d870bf69052-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.234 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.236 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.237 DEBUG:boto:path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.238 DEBUG:boto:auth_path=/xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.239 DEBUG:boto:Method: GET 2022-09-02T04:26:27.240 DEBUG:boto:Path: /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.241 DEBUG:boto:Data: 2022-09-02T04:26:27.242 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.244 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.245 DEBUG:boto:Params: {} 2022-09-02T04:26:27.245 DEBUG:boto:Token: None 2022-09-02T04:26:27.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj?versionId=null 2022-09-02T04:26:27.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4= 2022-09-02T04:26:27.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qahO3ZdEFSr5Pg5rSHa5dfUi6w4=', 'Content-Length': '0'} 2022-09-02T04:26:27.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006599ac27018568bb-00631185f3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.251 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:27.252 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:27.253 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.254 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.255 DEBUG:boto:Method: GET 2022-09-02T04:26:27.256 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.257 DEBUG:boto:Data: 2022-09-02T04:26:27.258 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.259 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.260 DEBUG:boto:Params: {} 2022-09-02T04:26:27.261 DEBUG:boto:Token: None 2022-09-02T04:26:27.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/ 2022-09-02T04:26:27.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0= 2022-09-02T04:26:27.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0=', 'Content-Length': '0'} 2022-09-02T04:26:27.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4111f9fb0257ead-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.314 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.316 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:27.317 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:27.318 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.319 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.319 DEBUG:boto:Method: GET 2022-09-02T04:26:27.320 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.322 DEBUG:boto:Data: 2022-09-02T04:26:27.322 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.324 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.324 DEBUG:boto:Params: {} 2022-09-02T04:26:27.325 DEBUG:boto:Token: None 2022-09-02T04:26:27.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/ 2022-09-02T04:26:27.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0= 2022-09-02T04:26:27.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0=', 'Content-Length': '0'} 2022-09-02T04:26:27.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c299593eb9021fa9-00631185f3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.332 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.334 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.335 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.337 DEBUG:boto:Method: HEAD 2022-09-02T04:26:27.338 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:27.339 DEBUG:boto:Data: 2022-09-02T04:26:27.340 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.341 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.343 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.343 DEBUG:boto:Params: {} 2022-09-02T04:26:27.345 DEBUG:boto:Token: None 2022-09-02T04:26:27.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj 2022-09-02T04:26:27.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzNH3l1TPzR9iTf1ESTDlADyKiI= 2022-09-02T04:26:27.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzNH3l1TPzR9iTf1ESTDlADyKiI=', 'Content-Length': '0'} 2022-09-02T04:26:27.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea72290eb0060c4e-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.353 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.354 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.356 DEBUG:boto:Method: HEAD 2022-09-02T04:26:27.357 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:27.358 DEBUG:boto:Data: 2022-09-02T04:26:27.359 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.362 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.363 DEBUG:boto:Params: {} 2022-09-02T04:26:27.364 DEBUG:boto:Token: None 2022-09-02T04:26:27.365 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj 2022-09-02T04:26:27.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzNH3l1TPzR9iTf1ESTDlADyKiI= 2022-09-02T04:26:27.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzNH3l1TPzR9iTf1ESTDlADyKiI=', 'Content-Length': '0'} 2022-09-02T04:26:27.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ca9e5fef9cfc0d3-00631185f3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-09-02T04:26:27.372 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.373 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.374 DEBUG:boto:Method: GET 2022-09-02T04:26:27.374 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:27.375 DEBUG:boto:Data: 2022-09-02T04:26:27.376 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.377 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.378 DEBUG:boto:Params: {} 2022-09-02T04:26:27.378 DEBUG:boto:Token: None 2022-09-02T04:26:27.379 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj 2022-09-02T04:26:27.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P27DdVAU3OtS1IwGKJfY/ExiFx4= 2022-09-02T04:26:27.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P27DdVAU3OtS1IwGKJfY/ExiFx4=', 'Content-Length': '0'} 2022-09-02T04:26:27.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a6ebadc4da34330-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.386 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:26:27.387 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:26:27.388 DEBUG:boto:Method: GET 2022-09-02T04:26:27.389 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:26:27.390 DEBUG:boto:Data: 2022-09-02T04:26:27.391 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.393 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.395 DEBUG:boto:Params: {} 2022-09-02T04:26:27.396 DEBUG:boto:Token: None 2022-09-02T04:26:27.397 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/myobj 2022-09-02T04:26:27.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P27DdVAU3OtS1IwGKJfY/ExiFx4= 2022-09-02T04:26:27.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P27DdVAU3OtS1IwGKJfY/ExiFx4=', 'Content-Length': '0'} 2022-09-02T04:26:27.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001669450dd8c478af-00631185f3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.442 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:27.444 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:27.445 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.446 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.447 DEBUG:boto:Method: GET 2022-09-02T04:26:27.447 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.448 DEBUG:boto:Data: 2022-09-02T04:26:27.448 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:26:27.449 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.450 DEBUG:boto:Params: {} 2022-09-02T04:26:27.451 DEBUG:boto:Token: None 2022-09-02T04:26:27.451 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/ 2022-09-02T04:26:27.452 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0= 2022-09-02T04:26:27.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0=', 'Content-Length': '0'} 2022-09-02T04:26:27.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2cf120ba61ae504-00631185f3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.456 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.457 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:26:27.457 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:26:27.459 DEBUG:boto:path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.459 DEBUG:boto:auth_path=/xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.460 DEBUG:boto:Method: GET 2022-09-02T04:26:27.463 DEBUG:boto:Path: /xljbkf-15/?prefix=myobj 2022-09-02T04:26:27.464 DEBUG:boto:Data: 2022-09-02T04:26:27.464 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:26:27.466 DEBUG:boto:Port: 8001 2022-09-02T04:26:27.467 DEBUG:boto:Params: {} 2022-09-02T04:26:27.467 DEBUG:boto:Token: None 2022-09-02T04:26:27.469 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-15/ 2022-09-02T04:26:27.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0= 2022-09-02T04:26:27.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lQ9wdaN122p2ErFDkpm5x004oj0=', 'Content-Length': '0'} 2022-09-02T04:26:27.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec96b59382babda-00631185f3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.477 DEBUG:boto:b'xljbkf-15myobj1000falsemyobj2022-09-02T04:26:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:26:27.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-15 zones={test-zone2, test-zone3} 2022-09-02T04:26:27.478 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:26:27.479 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:26:27.480 DEBUG:boto:Method: HEAD 2022-09-02T04:26:27.480 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:26:27.481 DEBUG:boto:Data: 2022-09-02T04:26:27.482 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:27.483 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.483 DEBUG:boto:Params: {} 2022-09-02T04:26:27.484 DEBUG:boto:Token: None 2022-09-02T04:26:27.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-13/ 2022-09-02T04:26:27.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rpi3qjlt5Ny+Bw9LQcAv0sf0bLI= 2022-09-02T04:26:27.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rpi3qjlt5Ny+Bw9LQcAv0sf0bLI=', 'Content-Length': '0'} 2022-09-02T04:26:27.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d95dfcc3edac3cc-00631185f3-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.490 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:27.491 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:27.492 DEBUG:boto:Method: HEAD 2022-09-02T04:26:27.492 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:27.493 DEBUG:boto:Data: 2022-09-02T04:26:27.493 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:27.494 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.495 DEBUG:boto:Params: {} 2022-09-02T04:26:27.496 DEBUG:boto:Token: None 2022-09-02T04:26:27.497 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-13/myobj 2022-09-02T04:26:27.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OhQE0zZbc70uCFO8WhEMxH5r8e4= 2022-09-02T04:26:27.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OhQE0zZbc70uCFO8WhEMxH5r8e4=', 'Content-Length': '0'} 2022-09-02T04:26:27.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000549f0307d48555b5-00631185f3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.501 DEBUG:boto:path=/xljbkf-13/myobj 2022-09-02T04:26:27.501 DEBUG:boto:auth_path=/xljbkf-13/myobj 2022-09-02T04:26:27.502 DEBUG:boto:Method: DELETE 2022-09-02T04:26:27.503 DEBUG:boto:Path: /xljbkf-13/myobj 2022-09-02T04:26:27.504 DEBUG:boto:Data: 2022-09-02T04:26:27.504 DEBUG:boto:Headers: {} 2022-09-02T04:26:27.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:26:27.506 DEBUG:boto:Port: 8000 2022-09-02T04:26:27.507 DEBUG:boto:Params: {} 2022-09-02T04:26:27.508 DEBUG:boto:Token: None 2022-09-02T04:26:27.509 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:26:27 GMT /xljbkf-13/myobj 2022-09-02T04:26:27.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:70UE7uKgMCm9hcVSmU5YjnNRQjU= 2022-09-02T04:26:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:26:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:70UE7uKgMCm9hcVSmU5YjnNRQjU=', 'Content-Length': '0'} 2022-09-02T04:26:27.522 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d86123c84168a82-00631185f3-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:26:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:26:27.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:27.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:26:27.526 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:26:27.592 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:26:27.593 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:26:27.648 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:27.646+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:27.648 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:27.648 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:26:32.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:32.647+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:32.652 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:32.652 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:26:37.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:37.649+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:37.652 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:37.652 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:26:42.652 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:42.650+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:42.653 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:42.654 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:26:47.657 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:47.653+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:47.658 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:47.658 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:26:52.656 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:52.654+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:52.657 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:52.657 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:26:57.662 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:26:57.656+0000 7f4178fcd600 1 waiting for incremental sync to catch up: 2022-09-02T04:26:57.663 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.2.5, , , , , , , , , ] 2022-09-02T04:26:57.664 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:02.659 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:02.658+0000 7f4178fcd600 1 bucket sync caught up with source: 2022-09-02T04:27:02.660 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:02.660 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:02.660 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:02.658+0000 7f4178fcd600 0 bucket checkpoint complete 2022-09-02T04:27:02.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-13 zones={test-zone2, test-zone1} 2022-09-02T04:27:02.679 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.680 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.680 DEBUG:boto:Method: HEAD 2022-09-02T04:27:02.681 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:27:02.681 DEBUG:boto:Data: 2022-09-02T04:27:02.691 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:02.692 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.693 DEBUG:boto:Params: {} 2022-09-02T04:27:02.694 DEBUG:boto:Token: None 2022-09-02T04:27:02.697 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/ 2022-09-02T04:27:02.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ= 2022-09-02T04:27:02.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ=', 'Content-Length': '0'} 2022-09-02T04:27:02.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000892783785db5028f-0063118616-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.712 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.712 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.713 DEBUG:boto:Method: HEAD 2022-09-02T04:27:02.713 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:27:02.714 DEBUG:boto:Data: 2022-09-02T04:27:02.714 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:02.715 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.716 DEBUG:boto:Params: {} 2022-09-02T04:27:02.717 DEBUG:boto:Token: None 2022-09-02T04:27:02.717 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/ 2022-09-02T04:27:02.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ= 2022-09-02T04:27:02.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ=', 'Content-Length': '0'} 2022-09-02T04:27:02.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000151304b7d2e9dc74-0063118616-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.723 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:27:02.723 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.724 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.724 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.725 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.726 DEBUG:boto:Method: GET 2022-09-02T04:27:02.726 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.727 DEBUG:boto:Data: 2022-09-02T04:27:02.727 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:02.728 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.729 DEBUG:boto:Params: {} 2022-09-02T04:27:02.729 DEBUG:boto:Token: None 2022-09-02T04:27:02.730 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/?versions 2022-09-02T04:27:02.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo= 2022-09-02T04:27:02.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo=', 'Content-Length': '0'} 2022-09-02T04:27:02.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3906daed016d3f-0063118616-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.750 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:02.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:27:02.752 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.753 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.754 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.754 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.755 DEBUG:boto:Method: GET 2022-09-02T04:27:02.755 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.756 DEBUG:boto:Data: 2022-09-02T04:27:02.757 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:02.761 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.762 DEBUG:boto:Params: {} 2022-09-02T04:27:02.763 DEBUG:boto:Token: None 2022-09-02T04:27:02.764 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/?versions 2022-09-02T04:27:02.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo= 2022-09-02T04:27:02.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo=', 'Content-Length': '0'} 2022-09-02T04:27:02.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723eb40d1c44702f-0063118616-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.799 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:02.800 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.801 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.801 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.802 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.802 DEBUG:boto:Method: GET 2022-09-02T04:27:02.803 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.804 DEBUG:boto:Data: 2022-09-02T04:27:02.804 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:02.805 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.806 DEBUG:boto:Params: {} 2022-09-02T04:27:02.806 DEBUG:boto:Token: None 2022-09-02T04:27:02.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/?versions 2022-09-02T04:27:02.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo= 2022-09-02T04:27:02.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo=', 'Content-Length': '0'} 2022-09-02T04:27:02.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae4ffe55526c2da-0063118616-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.812 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:02.813 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.813 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.814 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.815 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.815 DEBUG:boto:Method: GET 2022-09-02T04:27:02.816 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.816 DEBUG:boto:Data: 2022-09-02T04:27:02.817 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:02.818 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.818 DEBUG:boto:Params: {} 2022-09-02T04:27:02.819 DEBUG:boto:Token: None 2022-09-02T04:27:02.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/?versions 2022-09-02T04:27:02.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo= 2022-09-02T04:27:02.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo=', 'Content-Length': '0'} 2022-09-02T04:27:02.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba99cdb5a4bc6534-0063118616-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.826 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:02.827 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-13 zones={test-zone2, test-zone1} 2022-09-02T04:27:02.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:02.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:02.829 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:02.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:02.896 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:02.941 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:02.939+0000 7fb2c006e600 1 bucket sync caught up with source: 2022-09-02T04:27:02.942 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:02.942 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:02.942 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:02.939+0000 7fb2c006e600 0 bucket checkpoint complete 2022-09-02T04:27:02.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-13 zones={test-zone3, test-zone1} 2022-09-02T04:27:02.953 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.955 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.956 DEBUG:boto:Method: HEAD 2022-09-02T04:27:02.957 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:27:02.957 DEBUG:boto:Data: 2022-09-02T04:27:02.958 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:02.959 DEBUG:boto:Port: 8001 2022-09-02T04:27:02.959 DEBUG:boto:Params: {} 2022-09-02T04:27:02.960 DEBUG:boto:Token: None 2022-09-02T04:27:02.961 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/ 2022-09-02T04:27:02.961 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ= 2022-09-02T04:27:02.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ=', 'Content-Length': '0'} 2022-09-02T04:27:02.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070f3050601d341d5-0063118616-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.966 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.967 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.967 DEBUG:boto:Method: HEAD 2022-09-02T04:27:02.968 DEBUG:boto:Path: /xljbkf-13/ 2022-09-02T04:27:02.968 DEBUG:boto:Data: 2022-09-02T04:27:02.969 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:02.970 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.971 DEBUG:boto:Params: {} 2022-09-02T04:27:02.971 DEBUG:boto:Token: None 2022-09-02T04:27:02.972 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/ 2022-09-02T04:27:02.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ= 2022-09-02T04:27:02.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZDyibY0j59qSIhvLRPxlOvVQBxQ=', 'Content-Length': '0'} 2022-09-02T04:27:02.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002972f26dcd48c9f6-0063118616-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.976 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:27:02.977 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.977 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.978 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.978 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.979 DEBUG:boto:Method: GET 2022-09-02T04:27:02.979 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.980 DEBUG:boto:Data: 2022-09-02T04:27:02.980 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:02.981 DEBUG:boto:Port: 8001 2022-09-02T04:27:02.982 DEBUG:boto:Params: {} 2022-09-02T04:27:02.982 DEBUG:boto:Token: None 2022-09-02T04:27:02.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:02 GMT /xljbkf-13/?versions 2022-09-02T04:27:02.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo= 2022-09-02T04:27:02.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GJRvezBrGYmLHRIwOrI4dMX9tNo=', 'Content-Length': '0'} 2022-09-02T04:27:02.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000675208bed831b4c9-0063118616-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:02.990 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:02.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:27:02.992 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:02.994 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:02.994 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.995 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.996 DEBUG:boto:Method: GET 2022-09-02T04:27:02.996 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:02.997 DEBUG:boto:Data: 2022-09-02T04:27:02.998 DEBUG:boto:Headers: {} 2022-09-02T04:27:02.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:02.999 DEBUG:boto:Port: 8000 2022-09-02T04:27:02.999 DEBUG:boto:Params: {} 2022-09-02T04:27:03.000 DEBUG:boto:Token: None 2022-09-02T04:27:03.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:03 GMT /xljbkf-13/?versions 2022-09-02T04:27:03.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qAQqOLcSRcQKMfmnJ+wKawc8oGA= 2022-09-02T04:27:03.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qAQqOLcSRcQKMfmnJ+wKawc8oGA=', 'Content-Length': '0'} 2022-09-02T04:27:03.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086d9560031d980b-0063118617-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:03.006 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:03.007 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:03.008 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:03.008 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:03.009 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:03.010 DEBUG:boto:Method: GET 2022-09-02T04:27:03.010 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:03.011 DEBUG:boto:Data: 2022-09-02T04:27:03.011 DEBUG:boto:Headers: {} 2022-09-02T04:27:03.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:03.013 DEBUG:boto:Port: 8001 2022-09-02T04:27:03.013 DEBUG:boto:Params: {} 2022-09-02T04:27:03.014 DEBUG:boto:Token: None 2022-09-02T04:27:03.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:03 GMT /xljbkf-13/?versions 2022-09-02T04:27:03.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qAQqOLcSRcQKMfmnJ+wKawc8oGA= 2022-09-02T04:27:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qAQqOLcSRcQKMfmnJ+wKawc8oGA=', 'Content-Length': '0'} 2022-09-02T04:27:03.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f90eb9568735465-0063118617-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:03.019 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:03.020 DEBUG:boto:path=/xljbkf-13/ 2022-09-02T04:27:03.020 DEBUG:boto:auth_path=/xljbkf-13/ 2022-09-02T04:27:03.021 DEBUG:boto:path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:03.022 DEBUG:boto:auth_path=/xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:03.022 DEBUG:boto:Method: GET 2022-09-02T04:27:03.023 DEBUG:boto:Path: /xljbkf-13/?versions&max-keys=999 2022-09-02T04:27:03.023 DEBUG:boto:Data: 2022-09-02T04:27:03.024 DEBUG:boto:Headers: {} 2022-09-02T04:27:03.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:03.025 DEBUG:boto:Port: 8000 2022-09-02T04:27:03.026 DEBUG:boto:Params: {} 2022-09-02T04:27:03.026 DEBUG:boto:Token: None 2022-09-02T04:27:03.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:03 GMT /xljbkf-13/?versions 2022-09-02T04:27:03.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qAQqOLcSRcQKMfmnJ+wKawc8oGA= 2022-09-02T04:27:03.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qAQqOLcSRcQKMfmnJ+wKawc8oGA=', 'Content-Length': '0'} 2022-09-02T04:27:03.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db1ac10a06875c1f-0063118617-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:03.032 DEBUG:boto:b'xljbkf-13999false' 2022-09-02T04:27:03.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-13 zones={test-zone3, test-zone1} 2022-09-02T04:27:03.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:03.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:03.039 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:03.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:03.105 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:03.145 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:03.142+0000 7f1dae799600 1 bucket sync caught up with source: 2022-09-02T04:27:03.145 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:03.146 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.4.5, , , , , , , , , ] 2022-09-02T04:27:03.146 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:03.142+0000 7f1dae799600 0 bucket checkpoint complete 2022-09-02T04:27:03.158 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:03.159 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:03.160 DEBUG:boto:Method: HEAD 2022-09-02T04:27:03.160 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:27:03.161 DEBUG:boto:Data: 2022-09-02T04:27:03.164 DEBUG:boto:Headers: {} 2022-09-02T04:27:03.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:03.165 DEBUG:boto:Port: 8000 2022-09-02T04:27:03.166 DEBUG:boto:Params: {} 2022-09-02T04:27:03.167 DEBUG:boto:Token: None 2022-09-02T04:27:03.167 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:03 GMT /xljbkf-14/ 2022-09-02T04:27:03.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZdbhnFEl6rzL8vGYnlURjdRzSMg= 2022-09-02T04:27:03.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZdbhnFEl6rzL8vGYnlURjdRzSMg=', 'Content-Length': '0'} 2022-09-02T04:27:03.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be27216efd6e4752-0063118617-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:03.201 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:27:03.202 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:27:03.203 DEBUG:boto:Method: HEAD 2022-09-02T04:27:03.203 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:27:03.204 DEBUG:boto:Data: 2022-09-02T04:27:03.205 DEBUG:boto:Headers: {} 2022-09-02T04:27:03.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:03.206 DEBUG:boto:Port: 8000 2022-09-02T04:27:03.207 DEBUG:boto:Params: {} 2022-09-02T04:27:03.208 DEBUG:boto:Token: None 2022-09-02T04:27:03.209 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:03 GMT /xljbkf-14/myobj 2022-09-02T04:27:03.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GWSpQlku+SzU5AZMK4F4qBIj/4Y= 2022-09-02T04:27:03.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GWSpQlku+SzU5AZMK4F4qBIj/4Y=', 'Content-Length': '0'} 2022-09-02T04:27:03.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e7b3b951f1e83b2-0063118617-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:27:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:03.213 DEBUG:boto:path=/xljbkf-14/myobj 2022-09-02T04:27:03.214 DEBUG:boto:auth_path=/xljbkf-14/myobj 2022-09-02T04:27:03.214 DEBUG:boto:Method: DELETE 2022-09-02T04:27:03.215 DEBUG:boto:Path: /xljbkf-14/myobj 2022-09-02T04:27:03.215 DEBUG:boto:Data: 2022-09-02T04:27:03.216 DEBUG:boto:Headers: {} 2022-09-02T04:27:03.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:03.217 DEBUG:boto:Port: 8000 2022-09-02T04:27:03.218 DEBUG:boto:Params: {} 2022-09-02T04:27:03.220 DEBUG:boto:Token: None 2022-09-02T04:27:03.220 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:27:03 GMT /xljbkf-14/myobj 2022-09-02T04:27:03.221 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zGiw1Wndbgucb4QQhE1vcH2/hno= 2022-09-02T04:27:03.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zGiw1Wndbgucb4QQhE1vcH2/hno=', 'Content-Length': '0'} 2022-09-02T04:27:03.230 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d1205461feb0689-0063118617-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:27:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:03.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:03.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:03.233 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:03.291 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:03.292 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:03.337 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:03.335+0000 7fd2a5aec600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:03.338 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:27:03.338 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:08.342 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:08.336+0000 7fd2a5aec600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:08.343 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:27:08.343 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:13.338+0000 7fd2a5aec600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:13.341 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:27:13.341 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:18.341 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:18.339+0000 7fd2a5aec600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:18.341 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.40.5, , , , , , , , , ] 2022-09-02T04:27:18.341 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:23.340+0000 7fd2a5aec600 1 bucket sync caught up with source: 2022-09-02T04:27:23.354 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:23.355 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:23.340+0000 7fd2a5aec600 0 bucket checkpoint complete 2022-09-02T04:27:23.359 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-14 zones={test-zone1, test-zone2} 2022-09-02T04:27:23.360 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.361 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.363 DEBUG:boto:Method: HEAD 2022-09-02T04:27:23.363 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:27:23.364 DEBUG:boto:Data: 2022-09-02T04:27:23.364 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:23.366 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.366 DEBUG:boto:Params: {} 2022-09-02T04:27:23.367 DEBUG:boto:Token: None 2022-09-02T04:27:23.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/ 2022-09-02T04:27:23.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0= 2022-09-02T04:27:23.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0=', 'Content-Length': '0'} 2022-09-02T04:27:23.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f9eb7187b943151-006311862b-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.379 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.380 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.380 DEBUG:boto:Method: HEAD 2022-09-02T04:27:23.381 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:27:23.381 DEBUG:boto:Data: 2022-09-02T04:27:23.382 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:23.383 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.384 DEBUG:boto:Params: {} 2022-09-02T04:27:23.384 DEBUG:boto:Token: None 2022-09-02T04:27:23.385 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/ 2022-09-02T04:27:23.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0= 2022-09-02T04:27:23.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0=', 'Content-Length': '0'} 2022-09-02T04:27:23.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000946dd1a21a06e29e-006311862b-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:27:23.390 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.391 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.392 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.392 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.393 DEBUG:boto:Method: GET 2022-09-02T04:27:23.394 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.394 DEBUG:boto:Data: 2022-09-02T04:27:23.395 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:23.396 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.396 DEBUG:boto:Params: {} 2022-09-02T04:27:23.397 DEBUG:boto:Token: None 2022-09-02T04:27:23.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d189d27018acef4-006311862b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.413 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.415 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:27:23.416 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.417 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.418 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.419 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.420 DEBUG:boto:Method: GET 2022-09-02T04:27:23.421 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.422 DEBUG:boto:Data: 2022-09-02T04:27:23.423 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:23.425 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.426 DEBUG:boto:Params: {} 2022-09-02T04:27:23.427 DEBUG:boto:Token: None 2022-09-02T04:27:23.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de47995b642f9b70-006311862b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.435 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.436 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.437 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.437 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.438 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.440 DEBUG:boto:Method: GET 2022-09-02T04:27:23.441 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.442 DEBUG:boto:Data: 2022-09-02T04:27:23.443 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:23.446 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.447 DEBUG:boto:Params: {} 2022-09-02T04:27:23.449 DEBUG:boto:Token: None 2022-09-02T04:27:23.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c53dac31931b2e8-006311862b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.456 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.457 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.457 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.458 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.458 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.459 DEBUG:boto:Method: GET 2022-09-02T04:27:23.460 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.460 DEBUG:boto:Data: 2022-09-02T04:27:23.461 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:23.462 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.463 DEBUG:boto:Params: {} 2022-09-02T04:27:23.463 DEBUG:boto:Token: None 2022-09-02T04:27:23.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d5065c5dd6993c-006311862b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.476 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.477 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-14 zones={test-zone1, test-zone2} 2022-09-02T04:27:23.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:23.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:23.480 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:23.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:23.595+0000 7f69786fb600 1 bucket sync caught up with source: 2022-09-02T04:27:23.597 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:23.598 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.44.5, , , , , , , , , ] 2022-09-02T04:27:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:23.595+0000 7f69786fb600 0 bucket checkpoint complete 2022-09-02T04:27:23.614 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-14 zones={test-zone3, test-zone2} 2022-09-02T04:27:23.615 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.616 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.617 DEBUG:boto:Method: HEAD 2022-09-02T04:27:23.618 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:27:23.619 DEBUG:boto:Data: 2022-09-02T04:27:23.620 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:23.622 DEBUG:boto:Port: 8001 2022-09-02T04:27:23.623 DEBUG:boto:Params: {} 2022-09-02T04:27:23.623 DEBUG:boto:Token: None 2022-09-02T04:27:23.624 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/ 2022-09-02T04:27:23.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0= 2022-09-02T04:27:23.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0=', 'Content-Length': '0'} 2022-09-02T04:27:23.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000749446999300c1e-006311862b-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.629 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.630 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.631 DEBUG:boto:Method: HEAD 2022-09-02T04:27:23.632 DEBUG:boto:Path: /xljbkf-14/ 2022-09-02T04:27:23.633 DEBUG:boto:Data: 2022-09-02T04:27:23.634 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:23.635 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.639 DEBUG:boto:Params: {} 2022-09-02T04:27:23.640 DEBUG:boto:Token: None 2022-09-02T04:27:23.641 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/ 2022-09-02T04:27:23.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0= 2022-09-02T04:27:23.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJmF3uXC/FnNY5lt4xt9Tz/CHI0=', 'Content-Length': '0'} 2022-09-02T04:27:23.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ecc516d54bcf02b-006311862b-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:27:23.657 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.658 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.658 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.659 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.659 DEBUG:boto:Method: GET 2022-09-02T04:27:23.660 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.661 DEBUG:boto:Data: 2022-09-02T04:27:23.662 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:23.664 DEBUG:boto:Port: 8001 2022-09-02T04:27:23.665 DEBUG:boto:Params: {} 2022-09-02T04:27:23.666 DEBUG:boto:Token: None 2022-09-02T04:27:23.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4abcc36a32cb85b-006311862b-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.674 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.675 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:27:23.676 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.678 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.679 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.680 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.684 DEBUG:boto:Method: GET 2022-09-02T04:27:23.685 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.685 DEBUG:boto:Data: 2022-09-02T04:27:23.686 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:23.687 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.687 DEBUG:boto:Params: {} 2022-09-02T04:27:23.688 DEBUG:boto:Token: None 2022-09-02T04:27:23.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047da04a6d567dbec-006311862b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.697 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.699 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.699 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.700 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.700 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.701 DEBUG:boto:Method: GET 2022-09-02T04:27:23.702 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.702 DEBUG:boto:Data: 2022-09-02T04:27:23.703 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:23.704 DEBUG:boto:Port: 8001 2022-09-02T04:27:23.705 DEBUG:boto:Params: {} 2022-09-02T04:27:23.705 DEBUG:boto:Token: None 2022-09-02T04:27:23.706 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a2153ac7b758cf-006311862b-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.710 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.712 DEBUG:boto:path=/xljbkf-14/ 2022-09-02T04:27:23.712 DEBUG:boto:auth_path=/xljbkf-14/ 2022-09-02T04:27:23.713 DEBUG:boto:path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.713 DEBUG:boto:auth_path=/xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.714 DEBUG:boto:Method: GET 2022-09-02T04:27:23.714 DEBUG:boto:Path: /xljbkf-14/?versions&max-keys=999 2022-09-02T04:27:23.715 DEBUG:boto:Data: 2022-09-02T04:27:23.715 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:23.716 DEBUG:boto:Port: 8000 2022-09-02T04:27:23.717 DEBUG:boto:Params: {} 2022-09-02T04:27:23.717 DEBUG:boto:Token: None 2022-09-02T04:27:23.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-14/?versions 2022-09-02T04:27:23.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc= 2022-09-02T04:27:23.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gc3ld7RaJPhckqN8AAaFfJw0Mjc=', 'Content-Length': '0'} 2022-09-02T04:27:23.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd04c902b64d4f2-006311862b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.723 DEBUG:boto:b'xljbkf-14999false' 2022-09-02T04:27:23.723 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-14 zones={test-zone3, test-zone2} 2022-09-02T04:27:23.724 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:23.724 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:23.725 DEBUG:boto:Method: HEAD 2022-09-02T04:27:23.725 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:27:23.726 DEBUG:boto:Data: 2022-09-02T04:27:23.726 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:23.727 DEBUG:boto:Port: 8001 2022-09-02T04:27:23.728 DEBUG:boto:Params: {} 2022-09-02T04:27:23.728 DEBUG:boto:Token: None 2022-09-02T04:27:23.729 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-15/ 2022-09-02T04:27:23.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjJ1lz1mt4Q9ZYoqJRVtfyjokmg= 2022-09-02T04:27:23.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjJ1lz1mt4Q9ZYoqJRVtfyjokmg=', 'Content-Length': '0'} 2022-09-02T04:27:23.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087a470335af4de6a-006311862b-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.733 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:27:23.734 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:27:23.734 DEBUG:boto:Method: HEAD 2022-09-02T04:27:23.735 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:27:23.735 DEBUG:boto:Data: 2022-09-02T04:27:23.736 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:23.737 DEBUG:boto:Port: 8001 2022-09-02T04:27:23.737 DEBUG:boto:Params: {} 2022-09-02T04:27:23.738 DEBUG:boto:Token: None 2022-09-02T04:27:23.738 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-15/myobj 2022-09-02T04:27:23.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kov8BTcylt1GQRvRofJ7ZDMt+ys= 2022-09-02T04:27:23.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kov8BTcylt1GQRvRofJ7ZDMt+ys=', 'Content-Length': '0'} 2022-09-02T04:27:23.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:26:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021ba906837d39bf4-006311862b-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.743 DEBUG:boto:path=/xljbkf-15/myobj 2022-09-02T04:27:23.743 DEBUG:boto:auth_path=/xljbkf-15/myobj 2022-09-02T04:27:23.744 DEBUG:boto:Method: DELETE 2022-09-02T04:27:23.744 DEBUG:boto:Path: /xljbkf-15/myobj 2022-09-02T04:27:23.745 DEBUG:boto:Data: 2022-09-02T04:27:23.745 DEBUG:boto:Headers: {} 2022-09-02T04:27:23.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:23.746 DEBUG:boto:Port: 8001 2022-09-02T04:27:23.747 DEBUG:boto:Params: {} 2022-09-02T04:27:23.747 DEBUG:boto:Token: None 2022-09-02T04:27:23.748 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:27:23 GMT /xljbkf-15/myobj 2022-09-02T04:27:23.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GA5nuul/pzY55891Gyc+b/lM8Pc= 2022-09-02T04:27:23.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GA5nuul/pzY55891Gyc+b/lM8Pc=', 'Content-Length': '0'} 2022-09-02T04:27:23.758 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029fbcae2fb9b7cfa-006311862b-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:27:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:23.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:23.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:23.760 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:23.818 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:23.818 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:23.866 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:23.864+0000 7fee5e281600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:23.866 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:27:23.867 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:28.867 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:28.866+0000 7fee5e281600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:28.868 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:27:28.868 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:33.869 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:33.867+0000 7fee5e281600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:33.870 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:27:33.870 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:38.871 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:38.869+0000 7fee5e281600 1 waiting for incremental sync to catch up: 2022-09-02T04:27:38.876 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.28.5, , , , , , , , , ] 2022-09-02T04:27:38.877 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:43.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:43.870+0000 7fee5e281600 1 bucket sync caught up with source: 2022-09-02T04:27:43.884 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:43.885 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:43.885 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:43.870+0000 7fee5e281600 0 bucket checkpoint complete 2022-09-02T04:27:43.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-15 zones={test-zone1, test-zone3} 2022-09-02T04:27:43.892 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:43.893 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:43.893 DEBUG:boto:Method: HEAD 2022-09-02T04:27:43.894 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:27:43.895 DEBUG:boto:Data: 2022-09-02T04:27:43.896 DEBUG:boto:Headers: {} 2022-09-02T04:27:43.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:43.899 DEBUG:boto:Port: 8000 2022-09-02T04:27:43.900 DEBUG:boto:Params: {} 2022-09-02T04:27:43.901 DEBUG:boto:Token: None 2022-09-02T04:27:43.902 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:43 GMT /xljbkf-15/ 2022-09-02T04:27:43.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tsgEtolwHd6YdQdUqpX4PmCkaOQ= 2022-09-02T04:27:43.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tsgEtolwHd6YdQdUqpX4PmCkaOQ=', 'Content-Length': '0'} 2022-09-02T04:27:43.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042d722e8e90e5bdc-006311863f-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:43.913 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:43.914 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:43.915 DEBUG:boto:Method: HEAD 2022-09-02T04:27:43.916 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:27:43.917 DEBUG:boto:Data: 2022-09-02T04:27:43.918 DEBUG:boto:Headers: {} 2022-09-02T04:27:43.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:43.919 DEBUG:boto:Port: 8001 2022-09-02T04:27:43.921 DEBUG:boto:Params: {} 2022-09-02T04:27:43.922 DEBUG:boto:Token: None 2022-09-02T04:27:43.923 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:43 GMT /xljbkf-15/ 2022-09-02T04:27:43.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tsgEtolwHd6YdQdUqpX4PmCkaOQ= 2022-09-02T04:27:43.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tsgEtolwHd6YdQdUqpX4PmCkaOQ=', 'Content-Length': '0'} 2022-09-02T04:27:43.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006034f612693c0220-006311863f-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:43.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:27:43.935 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:43.935 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:43.936 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.938 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.939 DEBUG:boto:Method: GET 2022-09-02T04:27:43.940 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.941 DEBUG:boto:Data: 2022-09-02T04:27:43.942 DEBUG:boto:Headers: {} 2022-09-02T04:27:43.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:43.943 DEBUG:boto:Port: 8000 2022-09-02T04:27:43.944 DEBUG:boto:Params: {} 2022-09-02T04:27:43.946 DEBUG:boto:Token: None 2022-09-02T04:27:43.947 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:43 GMT /xljbkf-15/?versions 2022-09-02T04:27:43.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qEmV9MyeFhfCLooiBLmsNFDfyZM= 2022-09-02T04:27:43.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qEmV9MyeFhfCLooiBLmsNFDfyZM=', 'Content-Length': '0'} 2022-09-02T04:27:43.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58218b46a433f3a-006311863f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:43.964 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:43.965 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:27:43.966 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:43.967 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:43.968 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.969 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.974 DEBUG:boto:Method: GET 2022-09-02T04:27:43.975 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.975 DEBUG:boto:Data: 2022-09-02T04:27:43.976 DEBUG:boto:Headers: {} 2022-09-02T04:27:43.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:43.977 DEBUG:boto:Port: 8001 2022-09-02T04:27:43.977 DEBUG:boto:Params: {} 2022-09-02T04:27:43.978 DEBUG:boto:Token: None 2022-09-02T04:27:43.980 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:43 GMT /xljbkf-15/?versions 2022-09-02T04:27:43.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qEmV9MyeFhfCLooiBLmsNFDfyZM= 2022-09-02T04:27:43.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qEmV9MyeFhfCLooiBLmsNFDfyZM=', 'Content-Length': '0'} 2022-09-02T04:27:43.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000212f5035cdb9354c-006311863f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:43.989 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:43.991 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:43.993 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:43.995 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.996 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.996 DEBUG:boto:Method: GET 2022-09-02T04:27:43.997 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:43.998 DEBUG:boto:Data: 2022-09-02T04:27:43.998 DEBUG:boto:Headers: {} 2022-09-02T04:27:43.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:44.001 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.003 DEBUG:boto:Params: {} 2022-09-02T04:27:44.003 DEBUG:boto:Token: None 2022-09-02T04:27:44.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/?versions 2022-09-02T04:27:44.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE= 2022-09-02T04:27:44.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE=', 'Content-Length': '0'} 2022-09-02T04:27:44.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001079fc21af4858bb-0063118640-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.009 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:44.010 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.011 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.011 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.012 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.013 DEBUG:boto:Method: GET 2022-09-02T04:27:44.014 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.014 DEBUG:boto:Data: 2022-09-02T04:27:44.015 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:44.016 DEBUG:boto:Port: 8001 2022-09-02T04:27:44.016 DEBUG:boto:Params: {} 2022-09-02T04:27:44.017 DEBUG:boto:Token: None 2022-09-02T04:27:44.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/?versions 2022-09-02T04:27:44.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE= 2022-09-02T04:27:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE=', 'Content-Length': '0'} 2022-09-02T04:27:44.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000234daec4f4e3fb74-0063118640-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.023 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:44.025 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-15 zones={test-zone1, test-zone3} 2022-09-02T04:27:44.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:44.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:44.027 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:44.091 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:44.092 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:44.137 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:44.136+0000 7fe1c532e600 1 bucket sync caught up with source: 2022-09-02T04:27:44.137 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:44.138 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.36.5, , , , , , , , , ] 2022-09-02T04:27:44.138 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:44.136+0000 7fe1c532e600 0 bucket checkpoint complete 2022-09-02T04:27:44.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-15 zones={test-zone2, test-zone3} 2022-09-02T04:27:44.152 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.152 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.153 DEBUG:boto:Method: HEAD 2022-09-02T04:27:44.154 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:27:44.155 DEBUG:boto:Data: 2022-09-02T04:27:44.157 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:44.159 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.160 DEBUG:boto:Params: {} 2022-09-02T04:27:44.161 DEBUG:boto:Token: None 2022-09-02T04:27:44.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/ 2022-09-02T04:27:44.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PMMCLd/VTN0In74yYkt/HLKhvxI= 2022-09-02T04:27:44.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PMMCLd/VTN0In74yYkt/HLKhvxI=', 'Content-Length': '0'} 2022-09-02T04:27:44.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5365b3a1268ca03-0063118640-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.168 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.170 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.172 DEBUG:boto:Method: HEAD 2022-09-02T04:27:44.172 DEBUG:boto:Path: /xljbkf-15/ 2022-09-02T04:27:44.173 DEBUG:boto:Data: 2022-09-02T04:27:44.174 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:44.177 DEBUG:boto:Port: 8001 2022-09-02T04:27:44.178 DEBUG:boto:Params: {} 2022-09-02T04:27:44.178 DEBUG:boto:Token: None 2022-09-02T04:27:44.179 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/ 2022-09-02T04:27:44.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PMMCLd/VTN0In74yYkt/HLKhvxI= 2022-09-02T04:27:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PMMCLd/VTN0In74yYkt/HLKhvxI=', 'Content-Length': '0'} 2022-09-02T04:27:44.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000098e17bef939e5a9-0063118640-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:27:44.184 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.185 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.186 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.187 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.187 DEBUG:boto:Method: GET 2022-09-02T04:27:44.188 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.188 DEBUG:boto:Data: 2022-09-02T04:27:44.189 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:44.191 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.192 DEBUG:boto:Params: {} 2022-09-02T04:27:44.192 DEBUG:boto:Token: None 2022-09-02T04:27:44.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/?versions 2022-09-02T04:27:44.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE= 2022-09-02T04:27:44.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE=', 'Content-Length': '0'} 2022-09-02T04:27:44.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bbfdc1bb31958cd-0063118640-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.199 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:44.201 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:27:44.202 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.203 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.204 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.205 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.206 DEBUG:boto:Method: GET 2022-09-02T04:27:44.207 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.208 DEBUG:boto:Data: 2022-09-02T04:27:44.209 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:44.210 DEBUG:boto:Port: 8001 2022-09-02T04:27:44.211 DEBUG:boto:Params: {} 2022-09-02T04:27:44.212 DEBUG:boto:Token: None 2022-09-02T04:27:44.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/?versions 2022-09-02T04:27:44.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE= 2022-09-02T04:27:44.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE=', 'Content-Length': '0'} 2022-09-02T04:27:44.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66541b129e5ec27-0063118640-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.220 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:44.222 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.225 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.226 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.226 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.227 DEBUG:boto:Method: GET 2022-09-02T04:27:44.228 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.228 DEBUG:boto:Data: 2022-09-02T04:27:44.229 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:44.232 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.232 DEBUG:boto:Params: {} 2022-09-02T04:27:44.233 DEBUG:boto:Token: None 2022-09-02T04:27:44.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/?versions 2022-09-02T04:27:44.235 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE= 2022-09-02T04:27:44.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE=', 'Content-Length': '0'} 2022-09-02T04:27:44.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d656ab3c4eb0e155-0063118640-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.241 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:44.242 DEBUG:boto:path=/xljbkf-15/ 2022-09-02T04:27:44.243 DEBUG:boto:auth_path=/xljbkf-15/ 2022-09-02T04:27:44.244 DEBUG:boto:path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.245 DEBUG:boto:auth_path=/xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.246 DEBUG:boto:Method: GET 2022-09-02T04:27:44.247 DEBUG:boto:Path: /xljbkf-15/?versions&max-keys=999 2022-09-02T04:27:44.247 DEBUG:boto:Data: 2022-09-02T04:27:44.248 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:44.249 DEBUG:boto:Port: 8001 2022-09-02T04:27:44.250 DEBUG:boto:Params: {} 2022-09-02T04:27:44.252 DEBUG:boto:Token: None 2022-09-02T04:27:44.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-15/?versions 2022-09-02T04:27:44.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE= 2022-09-02T04:27:44.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJd+83lexgm7O2IARlK01jJu9eE=', 'Content-Length': '0'} 2022-09-02T04:27:44.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eb575ac3badb7b3-0063118640-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.261 DEBUG:boto:b'xljbkf-15999false' 2022-09-02T04:27:44.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-15 zones={test-zone2, test-zone3} 2022-09-02T04:27:44.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2022-09-02T04:27:44.267 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-16 2022-09-02T04:27:44.268 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:27:44.269 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:27:44.270 DEBUG:boto:Method: PUT 2022-09-02T04:27:44.271 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:27:44.272 DEBUG:boto:Data: 2022-09-02T04:27:44.273 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:44.275 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.276 DEBUG:boto:Params: {} 2022-09-02T04:27:44.277 DEBUG:boto:Token: None 2022-09-02T04:27:44.278 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-16/ 2022-09-02T04:27:44.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z8oCUGRBrB3kpEo4YqINwFuUKOA= 2022-09-02T04:27:44.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z8oCUGRBrB3kpEo4YqINwFuUKOA=', 'Content-Length': '0'} 2022-09-02T04:27:44.301 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007154d9872dd8b283-0063118640-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.303 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-17 2022-09-02T04:27:44.304 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:27:44.305 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:27:44.306 DEBUG:boto:Method: PUT 2022-09-02T04:27:44.307 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:27:44.308 DEBUG:boto:Data: 2022-09-02T04:27:44.309 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:44.311 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.312 DEBUG:boto:Params: {} 2022-09-02T04:27:44.313 DEBUG:boto:Token: None 2022-09-02T04:27:44.314 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-17/ 2022-09-02T04:27:44.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8anLf0oNl4VdZo97zjWpL+Inwg= 2022-09-02T04:27:44.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8anLf0oNl4VdZo97zjWpL+Inwg=', 'Content-Length': '0'} 2022-09-02T04:27:44.346 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ec6dd37c8b1540f-0063118640-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.348 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-18 2022-09-02T04:27:44.348 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:27:44.349 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:27:44.349 DEBUG:boto:Method: PUT 2022-09-02T04:27:44.350 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:27:44.350 DEBUG:boto:Data: 2022-09-02T04:27:44.351 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:44.352 DEBUG:boto:Port: 8001 2022-09-02T04:27:44.353 DEBUG:boto:Params: {} 2022-09-02T04:27:44.353 DEBUG:boto:Token: None 2022-09-02T04:27:44.354 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-18/ 2022-09-02T04:27:44.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zKWT3mqBgC4+mnzAYPH4KvL/v00= 2022-09-02T04:27:44.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zKWT3mqBgC4+mnzAYPH4KvL/v00=', 'Content-Length': '0'} 2022-09-02T04:27:44.394 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007077c310cb71d185-0063118640-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.395 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:27:44.395 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:27:44.396 DEBUG:boto:path=/xljbkf-16/?versioning 2022-09-02T04:27:44.396 DEBUG:boto:auth_path=/xljbkf-16/?versioning 2022-09-02T04:27:44.397 DEBUG:boto:Method: PUT 2022-09-02T04:27:44.397 DEBUG:boto:Path: /xljbkf-16/?versioning 2022-09-02T04:27:44.398 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:27:44.399 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:44.400 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.400 DEBUG:boto:Params: {} 2022-09-02T04:27:44.401 DEBUG:boto:Token: None 2022-09-02T04:27:44.402 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-16/?versioning 2022-09-02T04:27:44.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8aas4yk6svyGJDFkVbgyMEYBKc8= 2022-09-02T04:27:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8aas4yk6svyGJDFkVbgyMEYBKc8=', 'Content-Length': '228'} 2022-09-02T04:27:44.412 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000749a57a42f7842e6-0063118640-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.413 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:27:44.414 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:27:44.414 DEBUG:boto:path=/xljbkf-17/?versioning 2022-09-02T04:27:44.415 DEBUG:boto:auth_path=/xljbkf-17/?versioning 2022-09-02T04:27:44.416 DEBUG:boto:Method: PUT 2022-09-02T04:27:44.416 DEBUG:boto:Path: /xljbkf-17/?versioning 2022-09-02T04:27:44.417 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:27:44.417 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:27:44.418 DEBUG:boto:Port: 8000 2022-09-02T04:27:44.418 DEBUG:boto:Params: {} 2022-09-02T04:27:44.419 DEBUG:boto:Token: None 2022-09-02T04:27:44.420 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-17/?versioning 2022-09-02T04:27:44.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Or6BaY2XndTsOHK8HEdebzX7918= 2022-09-02T04:27:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Or6BaY2XndTsOHK8HEdebzX7918=', 'Content-Length': '228'} 2022-09-02T04:27:44.433 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e423429dfd65be4-0063118640-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.434 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:27:44.435 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:27:44.436 DEBUG:boto:path=/xljbkf-18/?versioning 2022-09-02T04:27:44.436 DEBUG:boto:auth_path=/xljbkf-18/?versioning 2022-09-02T04:27:44.437 DEBUG:boto:Method: PUT 2022-09-02T04:27:44.437 DEBUG:boto:Path: /xljbkf-18/?versioning 2022-09-02T04:27:44.438 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:27:44.438 DEBUG:boto:Headers: {} 2022-09-02T04:27:44.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:27:44.439 DEBUG:boto:Port: 8001 2022-09-02T04:27:44.440 DEBUG:boto:Params: {} 2022-09-02T04:27:44.441 DEBUG:boto:Token: None 2022-09-02T04:27:44.442 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:27:44 GMT /xljbkf-18/?versioning 2022-09-02T04:27:44.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7tjXsetA2HIzZY4SZtwV3Cw1kmM= 2022-09-02T04:27:44.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7tjXsetA2HIzZY4SZtwV3Cw1kmM=', 'Content-Length': '228'} 2022-09-02T04:27:44.469 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000068e68cc2ccadeb90-0063118640-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:44.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:44.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:44.471 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:27:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:44.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:44.569 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:27:44.584 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:27:44.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:44.585 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:27:44.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:27:44.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:44.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:44.587 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:27:44.587 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:27:44.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:44.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:44.589 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092864.462134_106.1", 2022-09-02T04:27:44.589 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:27:44.462134Z" 2022-09-02T04:27:44.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:44.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:44.592 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:27:44.593 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:27:44.593 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:27:44.593 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:27:44.595 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:44.596 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:27:44.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:44.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:44.598 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:27:44.652 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:44.652 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:44.695 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:44.693+0000 7fc458b3f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:44.700 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:44.698+0000 7fc458b3f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:44.731 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:27:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:27:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:27:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:27:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:27:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:27:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:44.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:44.734 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:27:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:27:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:27:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:27:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:27:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:27:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:27:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:27:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:44.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:44.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:27:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092864.365816_97.1", 2022-09-02T04:27:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.365816Z", 2022-09-02T04:27:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:27:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:27:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:27:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:27:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:27:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:27:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:27:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:27:44Z" 2022-09-02T04:27:44.756 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:27:44.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092864.365816_97.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:27:44.365816Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:27:44Z" } 2022-09-02T04:27:44.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092864.365816_97.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:27:44.365816Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:27:44.759 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:44.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.365816_97.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:44.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662092864.462134_106.1 target=1_1662092864.365816_97.1 2022-09-02T04:27:49.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:49.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:49.763 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:27:49.822 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:49.822 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:49.860+0000 7f3c7d291600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:49.863 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:49.862+0000 7f3c7d291600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:49.882 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:27:49.882 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:27:49.883 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:27:49.883 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:27:49.883 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:27:49.884 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:27:49.884 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:49.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:49.884 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:27:49.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:49.885 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:27:49.885 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:49.887 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:49.887 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:27:49.887 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:49.888 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:27:49.888 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:49.888 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:27:49.889 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:49.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:49.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:49.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:49.891 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:27:49.891 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:49.892 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:49.892 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:27:49.893 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:49.893 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:49.893 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:49.894 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:27:49.895 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:49.896 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:49.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:49.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:49.897 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:27:49.898 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:49.898 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:49.899 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092864.462134_106.1", 2022-09-02T04:27:49.900 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:49.900 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:49.901 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:49.901 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.462134Z", 2022-09-02T04:27:49.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:49.902 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:49.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:49.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:49.903 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:27:49.903 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:49.904 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:49.904 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:27:49.904 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:49.904 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:49.905 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:49.905 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:27:49.905 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:49.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:49.906 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:49.906 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:27:49.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:49.906 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:27:49.907 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:27:49.907 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:27:49.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:49.907 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:27:49Z" 2022-09-02T04:27:49.908 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:27:49.909 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092864.462134_106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:27:44.462134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:27:49Z" } 2022-09-02T04:27:49.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092864.462134_106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:27:44.462134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:27:49.911 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:49.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:49.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:27:49.912 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:27:49.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:49.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:49.914 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:27:49.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:49.971 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:50.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:49.999+0000 7f2b5c5f2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:50.001 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:27:50.000+0000 7f2b5c5f2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:50.018 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:27:50.018 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:27:50.019 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:27:50.020 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:27:50.020 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:27:50.021 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:27:50.021 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:27:50.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:50.022 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:27:50.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:50.023 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:27:50.024 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:27:50.024 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:27:50.025 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:27:50.025 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:27:50.026 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:27:50.027 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:27:50.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:27:50.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:27:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:27:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:27:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:27:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:27:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:27:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:27:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:27:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:27:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:27:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:27:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:27:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:27:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:27:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:27:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092864.462134_106.1", 2022-09-02T04:27:50.033 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:27:50.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:27:50.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:27:50.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.462134Z", 2022-09-02T04:27:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:27:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:27:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:50.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:27:50.036 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:27:50.036 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:27:50.036 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:27:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:27:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:27:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:27:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:27:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:27:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:27:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:27:50.039 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:27:50.039 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:27:50.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:27:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:27:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:27:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:27:50.041 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:27:50Z" 2022-09-02T04:27:50.041 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:27:50.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092864.462134_106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:27:44.462134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:27:50Z" } 2022-09-02T04:27:50.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092864.462134_106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:27:44.462134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:27:50.043 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:50.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:50.044 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:27:50.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:27:50.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:50.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:27:50.046 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:27:50.103 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:50.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:50.141 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:50.139+0000 7f92434ac600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:50.146 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:50.140+0000 7f92434ac600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:27:50.158 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:27:50.158 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:27:50.159 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:27:50.159 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:27:50.160 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:27:50.160 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:27:50.161 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:50.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:50.162 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:27:50.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:50.163 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:27:50.163 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:50.164 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:50.164 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:27:50.165 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:50.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:27:50.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:50.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:27:50.167 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:50.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:50.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:50.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:50.169 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:27:50.169 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:50.169 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:50.170 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:27:50.170 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:50.171 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:50.171 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:50.172 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:27:50.172 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:50.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:50.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:50.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:50.173 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:27:50.173 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:50.173 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:50.174 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092864.462134_106.1", 2022-09-02T04:27:50.174 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:50.175 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:50.176 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:50.176 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.462134Z", 2022-09-02T04:27:50.176 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:50.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:50.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:50.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:27:50.177 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:27:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:27:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:27:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:27:50.178 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:27:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:27:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:27:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:27:50.179 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:27:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:27:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:27:50.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:50.181 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:27:50.181 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:27:50.182 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:27:50.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:27:50.183 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:27:50Z" 2022-09-02T04:27:50.183 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:27:50.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662092864.462134_106.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:27:44.462134Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:27:50Z" } 2022-09-02T04:27:50.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662092864.462134_106.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:27:44.462134Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:27:50.186 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:50.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662092864.462134_106.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:27:50.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:27:50.189 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:27:50.190 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:27:50.191 DEBUG:boto:Method: HEAD 2022-09-02T04:27:50.191 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:27:50.192 DEBUG:boto:Data: 2022-09-02T04:27:50.193 DEBUG:boto:Headers: {} 2022-09-02T04:27:50.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:50.195 DEBUG:boto:Port: 8000 2022-09-02T04:27:50.196 DEBUG:boto:Params: {} 2022-09-02T04:27:50.198 DEBUG:boto:Token: None 2022-09-02T04:27:50.199 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:27:50 GMT /xljbkf-16/ 2022-09-02T04:27:50.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:03MOeb+do+mKwqj2B9/x9xTbd1o= 2022-09-02T04:27:50.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:27:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:03MOeb+do+mKwqj2B9/x9xTbd1o=', 'Content-Length': '0'} 2022-09-02T04:27:50.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c08360cb608531a2-0063118646-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:50.205 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:27:50.206 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:27:50.208 DEBUG:boto:Method: PUT 2022-09-02T04:27:50.209 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:27:50.210 DEBUG:boto:Data: 2022-09-02T04:27:50.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:27:50.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:27:50.213 DEBUG:boto:Port: 8000 2022-09-02T04:27:50.213 DEBUG:boto:Params: {} 2022-09-02T04:27:50.214 DEBUG:boto:Token: None 2022-09-02T04:27:50.215 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:27:50 GMT /xljbkf-16/dummy 2022-09-02T04:27:50.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3h5ZpTwXKufGkep1XUYGn9BfhDY= 2022-09-02T04:27:50.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:27:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3h5ZpTwXKufGkep1XUYGn9BfhDY='} 2022-09-02T04:27:50.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-amz-request-id', 'tx00000a146c6f4d64752ca-0063118646-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:27:50.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:50.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:27:50.231 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:27:50.290 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:27:50.290 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:27:50.332 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:50.330+0000 7f64ce250600 1 waiting to reach incremental sync.. 2022-09-02T04:27:55.332 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:27:55.331+0000 7f64ce250600 1 waiting to reach incremental sync.. 2022-09-02T04:28:00.334 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:00.333+0000 7f64ce250600 1 bucket sync caught up with source: 2022-09-02T04:28:00.335 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2022-09-02T04:28:00.336 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-09-02T04:28:00.336 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:00.333+0000 7f64ce250600 0 bucket checkpoint complete 2022-09-02T04:28:00.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:00.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:00.360 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:28:00.430 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:28:00.430 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:28:00.477 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:00.476+0000 7f93fe59b600 1 bucket sync caught up with source: 2022-09-02T04:28:00.477 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2022-09-02T04:28:00.478 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-09-02T04:28:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:00.476+0000 7f93fe59b600 0 bucket checkpoint complete 2022-09-02T04:28:00.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:00.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:00.522 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:28:00.579 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:28:00.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:28:00.619 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:00.618+0000 7f6017b54600 1 bucket sync caught up with source: 2022-09-02T04:28:00.620 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2022-09-02T04:28:00.620 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-09-02T04:28:00.620 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:00.618+0000 7f6017b54600 0 bucket checkpoint complete 2022-09-02T04:28:00.650 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:28:00.651 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:28:00.651 DEBUG:boto:Method: HEAD 2022-09-02T04:28:00.652 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:28:00.652 DEBUG:boto:Data: 2022-09-02T04:28:00.653 DEBUG:boto:Headers: {} 2022-09-02T04:28:00.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:28:00.654 DEBUG:boto:Port: 8000 2022-09-02T04:28:00.655 DEBUG:boto:Params: {} 2022-09-02T04:28:00.655 DEBUG:boto:Token: None 2022-09-02T04:28:00.656 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:28:00 GMT /xljbkf-17/ 2022-09-02T04:28:00.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zC/7fXxHv4Nh7BzJcMkJqnc33sQ= 2022-09-02T04:28:00.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:28:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zC/7fXxHv4Nh7BzJcMkJqnc33sQ=', 'Content-Length': '0'} 2022-09-02T04:28:00.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9622b8452a82d25-0063118650-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:28:00.661 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:28:00.662 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:28:00.662 DEBUG:boto:Method: PUT 2022-09-02T04:28:00.663 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:28:00.663 DEBUG:boto:Data: 2022-09-02T04:28:00.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:28:00.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:28:00.665 DEBUG:boto:Port: 8000 2022-09-02T04:28:00.665 DEBUG:boto:Params: {} 2022-09-02T04:28:00.666 DEBUG:boto:Token: None 2022-09-02T04:28:00.667 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:28:00 GMT /xljbkf-17/dummy 2022-09-02T04:28:00.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MQBdBpt+VkXatKygZw8T/BE8l/g= 2022-09-02T04:28:00.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:28:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MQBdBpt+VkXatKygZw8T/BE8l/g='} 2022-09-02T04:28:00.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-amz-request-id', 'tx00000fe1bade19dfec01d-0063118650-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:28:00.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:00.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:00.698 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:28:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:28:00.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:28:00.792 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:00.791+0000 7f40d1026600 1 waiting to reach incremental sync.. 2022-09-02T04:28:05.792 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:05.791+0000 7f40d1026600 1 waiting to reach incremental sync.. 2022-09-02T04:28:10.793 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:10.792+0000 7f40d1026600 1 waiting to reach incremental sync.. 2022-09-02T04:28:15.799 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:15.792+0000 7f40d1026600 1 waiting to reach incremental sync.. 2022-09-02T04:28:20.795 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:20.794+0000 7f40d1026600 1 bucket sync caught up with source: 2022-09-02T04:28:20.796 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.10, , , , , ] 2022-09-02T04:28:20.796 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.10, , , , , ] 2022-09-02T04:28:20.796 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:20.794+0000 7f40d1026600 0 bucket checkpoint complete 2022-09-02T04:28:20.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:20.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:20.812 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:28:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:28:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:28:20.921 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:20.920+0000 7f4d08faf600 1 bucket sync caught up with source: 2022-09-02T04:28:20.922 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.10, , , , , ] 2022-09-02T04:28:20.922 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.10, , , , , ] 2022-09-02T04:28:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:20.920+0000 7f4d08faf600 0 bucket checkpoint complete 2022-09-02T04:28:20.937 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:28:20.938 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:28:20.938 DEBUG:boto:Method: HEAD 2022-09-02T04:28:20.939 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:28:20.939 DEBUG:boto:Data: 2022-09-02T04:28:20.940 DEBUG:boto:Headers: {} 2022-09-02T04:28:20.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:28:20.941 DEBUG:boto:Port: 8001 2022-09-02T04:28:20.941 DEBUG:boto:Params: {} 2022-09-02T04:28:20.942 DEBUG:boto:Token: None 2022-09-02T04:28:20.943 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:28:20 GMT /xljbkf-18/ 2022-09-02T04:28:20.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GFOjjO0UVbkQtHtqiSH27hqaDeM= 2022-09-02T04:28:20.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:28:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GFOjjO0UVbkQtHtqiSH27hqaDeM=', 'Content-Length': '0'} 2022-09-02T04:28:20.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064b6479363529909-0063118664-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:28:20.949 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:28:20.950 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:28:20.950 DEBUG:boto:Method: PUT 2022-09-02T04:28:20.951 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:28:20.951 DEBUG:boto:Data: 2022-09-02T04:28:20.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:28:20.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:28:20.953 DEBUG:boto:Port: 8001 2022-09-02T04:28:20.954 DEBUG:boto:Params: {} 2022-09-02T04:28:20.955 DEBUG:boto:Token: None 2022-09-02T04:28:20.955 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:28:20 GMT /xljbkf-18/dummy 2022-09-02T04:28:20.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z3CTrNVUIHOIoKgElJMD9Vt4OEQ= 2022-09-02T04:28:20.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:28:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z3CTrNVUIHOIoKgElJMD9Vt4OEQ='} 2022-09-02T04:28:21.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-amz-request-id', 'tx00000a38cff3a6d5f29b6-0063118664-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:28:21.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:21.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:21.010 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:28:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:28:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:28:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:21.097+0000 7fd32865f600 1 waiting to reach incremental sync.. 2022-09-02T04:28:26.100 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:26.098+0000 7fd32865f600 1 waiting to reach incremental sync.. 2022-09-02T04:28:31.100 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:31.098+0000 7fd32865f600 1 waiting to reach incremental sync.. 2022-09-02T04:28:36.100 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:36.099+0000 7fd32865f600 1 waiting to reach incremental sync.. 2022-09-02T04:28:41.102 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:41.101+0000 7fd32865f600 1 bucket sync caught up with source: 2022-09-02T04:28:41.103 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.10, , , , , ] 2022-09-02T04:28:41.103 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.10, , , , , ] 2022-09-02T04:28:41.104 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:28:41.101+0000 7fd32865f600 0 bucket checkpoint complete 2022-09-02T04:28:41.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:41.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:28:41.135 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:28:41.205 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:28:41.206 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:28:41.244 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:41.239+0000 7efc83223600 1 waiting for incremental sync to catch up: 2022-09-02T04:28:41.245 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.27.10, , , , , ] 2022-09-02T04:28:41.246 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.31.9, , , , , ] 2022-09-02T04:28:46.243 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:46.241+0000 7efc83223600 1 waiting for incremental sync to catch up: 2022-09-02T04:28:46.326 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.27.10, , , , , ] 2022-09-02T04:28:46.327 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.31.9, , , , , ] 2022-09-02T04:28:51.245 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:51.242+0000 7efc83223600 1 waiting for incremental sync to catch up: 2022-09-02T04:28:51.246 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.27.10, , , , , ] 2022-09-02T04:28:51.246 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.31.9, , , , , ] 2022-09-02T04:28:56.245 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:28:56.243+0000 7efc83223600 1 waiting for incremental sync to catch up: 2022-09-02T04:28:56.246 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.27.10, , , , , ] 2022-09-02T04:28:56.246 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.31.9, , , , , ] 2022-09-02T04:29:01.247 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:01.245+0000 7efc83223600 1 bucket sync caught up with source: 2022-09-02T04:29:01.248 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.31.9, , , , , ] 2022-09-02T04:29:01.249 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.31.9, , , , , ] 2022-09-02T04:29:01.249 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:01.245+0000 7efc83223600 0 bucket checkpoint complete 2022-09-02T04:29:01.266 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:01.266 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:01.267 DEBUG:boto:Method: HEAD 2022-09-02T04:29:01.268 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:01.268 DEBUG:boto:Data: 2022-09-02T04:29:01.269 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.270 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.271 DEBUG:boto:Params: {} 2022-09-02T04:29:01.273 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:29:01.273 DEBUG:boto:Token: None 2022-09-02T04:29:01.274 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/ 2022-09-02T04:29:01.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BatlVa8C2LCR3wCUGfPq7vx/uIs= 2022-09-02T04:29:01.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BatlVa8C2LCR3wCUGfPq7vx/uIs=', 'Content-Length': '0'} 2022-09-02T04:29:01.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e17b89e16547b775-006311868d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.281 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.281 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.282 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.282 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.283 DEBUG:boto:Data: 2022-09-02T04:29:01.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.285 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.285 DEBUG:boto:Params: {} 2022-09-02T04:29:01.286 DEBUG:boto:Token: None 2022-09-02T04:29:01.286 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9HRrYiymy3qqdmL399Re+GLv7es= 2022-09-02T04:29:01.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9HRrYiymy3qqdmL399Re+GLv7es='} 2022-09-02T04:29:01.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-amz-request-id', 'tx000001f455af587988ed6-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.315 DEBUG:rgw_multi.tests:version1 id=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:01.316 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.317 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.318 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.319 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.320 DEBUG:boto:Data: 2022-09-02T04:29:01.321 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.323 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.324 DEBUG:boto:Params: {} 2022-09-02T04:29:01.325 DEBUG:boto:Token: None 2022-09-02T04:29:01.326 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:agH3CPZ6O9n4RFLdNshxpFZK/9I= 2022-09-02T04:29:01.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:agH3CPZ6O9n4RFLdNshxpFZK/9I='} 2022-09-02T04:29:01.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dK-LrUicldXQQnO9t5AxyEgYBxGjAhj'), ('x-amz-request-id', 'tx000000a2d1c60868fe17b-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.349 DEBUG:rgw_multi.tests:version2 id=dK-LrUicldXQQnO9t5AxyEgYBxGjAhj 2022-09-02T04:29:01.350 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.351 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.352 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=dK-LrUicldXQQnO9t5AxyEgYBxGjAhj 2022-09-02T04:29:01.353 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=dK-LrUicldXQQnO9t5AxyEgYBxGjAhj 2022-09-02T04:29:01.353 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.355 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=dK-LrUicldXQQnO9t5AxyEgYBxGjAhj 2022-09-02T04:29:01.356 DEBUG:boto:Data: 2022-09-02T04:29:01.357 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.359 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.360 DEBUG:boto:Params: {} 2022-09-02T04:29:01.361 DEBUG:boto:Token: None 2022-09-02T04:29:01.362 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone1?versionId=dK-LrUicldXQQnO9t5AxyEgYBxGjAhj 2022-09-02T04:29:01.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2wDV8E/IooK2uKeOCuA3e/8xrkI= 2022-09-02T04:29:01.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2wDV8E/IooK2uKeOCuA3e/8xrkI=', 'Content-Length': '0'} 2022-09-02T04:29:01.377 DEBUG:boto:Response headers: [('x-amz-version-id', 'dK-LrUicldXQQnO9t5AxyEgYBxGjAhj'), ('x-amz-request-id', 'tx00000d4859ac0ccd2e945-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.378 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.381 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.381 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.382 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.382 DEBUG:boto:Data: 2022-09-02T04:29:01.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.384 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.385 DEBUG:boto:Params: {} 2022-09-02T04:29:01.386 DEBUG:boto:Token: None 2022-09-02T04:29:01.386 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r00qcp1QCKvTLv5qqCnPPBDRssI= 2022-09-02T04:29:01.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r00qcp1QCKvTLv5qqCnPPBDRssI='} 2022-09-02T04:29:01.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz'), ('x-amz-request-id', 'tx00000abf185194aef01c8-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.431 DEBUG:rgw_multi.tests:version3 id=1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz 2022-09-02T04:29:01.432 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.432 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:01.432 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz 2022-09-02T04:29:01.433 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz 2022-09-02T04:29:01.434 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.434 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz 2022-09-02T04:29:01.435 DEBUG:boto:Data: 2022-09-02T04:29:01.435 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.436 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.437 DEBUG:boto:Params: {} 2022-09-02T04:29:01.437 DEBUG:boto:Token: None 2022-09-02T04:29:01.439 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone1?versionId=1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz 2022-09-02T04:29:01.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qytumbl0D4cJojQngH2ceaSY1MY= 2022-09-02T04:29:01.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qytumbl0D4cJojQngH2ceaSY1MY=', 'Content-Length': '0'} 2022-09-02T04:29:01.451 DEBUG:boto:Response headers: [('x-amz-version-id', '1ZDFm6adKfgRcrvq5tmqaYmEgMtLCsz'), ('x-amz-request-id', 'tx000002823d4e6704e2a99-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.452 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:01.453 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:01.454 DEBUG:boto:Method: HEAD 2022-09-02T04:29:01.454 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:01.455 DEBUG:boto:Data: 2022-09-02T04:29:01.455 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:01.456 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.457 DEBUG:boto:Params: {} 2022-09-02T04:29:01.458 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:29:01.460 DEBUG:boto:Token: None 2022-09-02T04:29:01.461 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/ 2022-09-02T04:29:01.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BatlVa8C2LCR3wCUGfPq7vx/uIs= 2022-09-02T04:29:01.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BatlVa8C2LCR3wCUGfPq7vx/uIs=', 'Content-Length': '0'} 2022-09-02T04:29:01.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfac9983ba75bdf4-006311868d-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.475 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.476 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.477 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.478 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.483 DEBUG:boto:Data: 2022-09-02T04:29:01.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:01.486 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.486 DEBUG:boto:Params: {} 2022-09-02T04:29:01.488 DEBUG:boto:Token: None 2022-09-02T04:29:01.489 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYBnHDmVTcs2JdTbk5Eydr6/87Y= 2022-09-02T04:29:01.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYBnHDmVTcs2JdTbk5Eydr6/87Y='} 2022-09-02T04:29:01.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-amz-request-id', 'tx000004f2da4c63e844b0b-006311868d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.509 DEBUG:rgw_multi.tests:version1 id=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:01.510 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.511 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.513 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.514 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.514 DEBUG:boto:Data: 2022-09-02T04:29:01.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:01.519 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.520 DEBUG:boto:Params: {} 2022-09-02T04:29:01.521 DEBUG:boto:Token: None 2022-09-02T04:29:01.523 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MlaKcqlofi32vQx82HlCPAD9fdI= 2022-09-02T04:29:01.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MlaKcqlofi32vQx82HlCPAD9fdI='} 2022-09-02T04:29:01.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pXWhOahd2fRbXkBXRH4vleu-Y4lsskW'), ('x-amz-request-id', 'tx00000ba66085697c5ba89-006311868d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.546 DEBUG:rgw_multi.tests:version2 id=pXWhOahd2fRbXkBXRH4vleu-Y4lsskW 2022-09-02T04:29:01.546 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.548 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.549 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=pXWhOahd2fRbXkBXRH4vleu-Y4lsskW 2022-09-02T04:29:01.549 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=pXWhOahd2fRbXkBXRH4vleu-Y4lsskW 2022-09-02T04:29:01.550 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.550 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=pXWhOahd2fRbXkBXRH4vleu-Y4lsskW 2022-09-02T04:29:01.551 DEBUG:boto:Data: 2022-09-02T04:29:01.551 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:01.552 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.553 DEBUG:boto:Params: {} 2022-09-02T04:29:01.553 DEBUG:boto:Token: None 2022-09-02T04:29:01.554 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone2?versionId=pXWhOahd2fRbXkBXRH4vleu-Y4lsskW 2022-09-02T04:29:01.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BrtC45DvUArck7yGbO9c7BAtL3A= 2022-09-02T04:29:01.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BrtC45DvUArck7yGbO9c7BAtL3A=', 'Content-Length': '0'} 2022-09-02T04:29:01.568 DEBUG:boto:Response headers: [('x-amz-version-id', 'pXWhOahd2fRbXkBXRH4vleu-Y4lsskW'), ('x-amz-request-id', 'tx00000af8e19d5f64ebb1f-006311868d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.569 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.570 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.570 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.571 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.572 DEBUG:boto:Data: 2022-09-02T04:29:01.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:01.573 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.574 DEBUG:boto:Params: {} 2022-09-02T04:29:01.575 DEBUG:boto:Token: None 2022-09-02T04:29:01.578 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VSchBWnO6pteG8rQymkd6WZUxfM= 2022-09-02T04:29:01.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VSchBWnO6pteG8rQymkd6WZUxfM='} 2022-09-02T04:29:01.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E8DFmxs9B3mKF8WA0h4b76WyIh40TXH'), ('x-amz-request-id', 'tx0000008d66a404659c136-006311868d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.637 DEBUG:rgw_multi.tests:version3 id=E8DFmxs9B3mKF8WA0h4b76WyIh40TXH 2022-09-02T04:29:01.638 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.639 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:01.640 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=E8DFmxs9B3mKF8WA0h4b76WyIh40TXH 2022-09-02T04:29:01.641 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=E8DFmxs9B3mKF8WA0h4b76WyIh40TXH 2022-09-02T04:29:01.641 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.642 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=E8DFmxs9B3mKF8WA0h4b76WyIh40TXH 2022-09-02T04:29:01.643 DEBUG:boto:Data: 2022-09-02T04:29:01.644 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:01.646 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.647 DEBUG:boto:Params: {} 2022-09-02T04:29:01.648 DEBUG:boto:Token: None 2022-09-02T04:29:01.651 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone2?versionId=E8DFmxs9B3mKF8WA0h4b76WyIh40TXH 2022-09-02T04:29:01.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J+G76GV6frn4wwkJHCoCA2XcT14= 2022-09-02T04:29:01.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J+G76GV6frn4wwkJHCoCA2XcT14=', 'Content-Length': '0'} 2022-09-02T04:29:01.687 DEBUG:boto:Response headers: [('x-amz-version-id', 'E8DFmxs9B3mKF8WA0h4b76WyIh40TXH'), ('x-amz-request-id', 'tx000000af81f6e23ceacb2-006311868d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.688 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:01.689 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:01.689 DEBUG:boto:Method: HEAD 2022-09-02T04:29:01.690 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:01.690 DEBUG:boto:Data: 2022-09-02T04:29:01.691 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:01.692 DEBUG:boto:Port: 8001 2022-09-02T04:29:01.692 DEBUG:boto:Params: {} 2022-09-02T04:29:01.693 DEBUG:boto:Token: None 2022-09-02T04:29:01.693 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/ 2022-09-02T04:29:01.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BatlVa8C2LCR3wCUGfPq7vx/uIs= 2022-09-02T04:29:01.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BatlVa8C2LCR3wCUGfPq7vx/uIs=', 'Content-Length': '0'} 2022-09-02T04:29:01.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e486fd0d229acb8-006311868d-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.705 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.705 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.706 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.707 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.707 DEBUG:boto:Data: 2022-09-02T04:29:01.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:01.709 DEBUG:boto:Port: 8001 2022-09-02T04:29:01.709 DEBUG:boto:Params: {} 2022-09-02T04:29:01.710 DEBUG:boto:Token: None 2022-09-02T04:29:01.710 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5g+FYCQnigDczeMinAZOF+0jG6Q= 2022-09-02T04:29:01.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5g+FYCQnigDczeMinAZOF+0jG6Q='} 2022-09-02T04:29:01.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-amz-request-id', 'tx000007e99488e8871f070-006311868d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.724 DEBUG:rgw_multi.tests:version1 id=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:01.724 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.725 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.725 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.726 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.726 DEBUG:boto:Data: 2022-09-02T04:29:01.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:01.728 DEBUG:boto:Port: 8001 2022-09-02T04:29:01.728 DEBUG:boto:Params: {} 2022-09-02T04:29:01.729 DEBUG:boto:Token: None 2022-09-02T04:29:01.730 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ooKWNxf/bZu+5fMTyLZO/0WeQLE= 2022-09-02T04:29:01.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ooKWNxf/bZu+5fMTyLZO/0WeQLE='} 2022-09-02T04:29:01.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b-WgFhUxW4SeKDGDSOh-L7avlinmRXM'), ('x-amz-request-id', 'tx00000bf6981cbd1edba7f-006311868d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.749 DEBUG:rgw_multi.tests:version2 id=b-WgFhUxW4SeKDGDSOh-L7avlinmRXM 2022-09-02T04:29:01.750 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.750 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.751 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=b-WgFhUxW4SeKDGDSOh-L7avlinmRXM 2022-09-02T04:29:01.751 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=b-WgFhUxW4SeKDGDSOh-L7avlinmRXM 2022-09-02T04:29:01.752 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.752 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=b-WgFhUxW4SeKDGDSOh-L7avlinmRXM 2022-09-02T04:29:01.753 DEBUG:boto:Data: 2022-09-02T04:29:01.753 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:01.754 DEBUG:boto:Port: 8001 2022-09-02T04:29:01.755 DEBUG:boto:Params: {} 2022-09-02T04:29:01.756 DEBUG:boto:Token: None 2022-09-02T04:29:01.756 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone3?versionId=b-WgFhUxW4SeKDGDSOh-L7avlinmRXM 2022-09-02T04:29:01.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:reipEkG04+FfIEQxZoJgzywfCR8= 2022-09-02T04:29:01.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:reipEkG04+FfIEQxZoJgzywfCR8=', 'Content-Length': '0'} 2022-09-02T04:29:01.770 DEBUG:boto:Response headers: [('x-amz-version-id', 'b-WgFhUxW4SeKDGDSOh-L7avlinmRXM'), ('x-amz-request-id', 'tx000008167386439b3c92e-006311868d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.772 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.772 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.773 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.774 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.775 DEBUG:boto:Data: 2022-09-02T04:29:01.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:01.778 DEBUG:boto:Port: 8001 2022-09-02T04:29:01.779 DEBUG:boto:Params: {} 2022-09-02T04:29:01.779 DEBUG:boto:Token: None 2022-09-02T04:29:01.780 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Tygbjq+Xr7dwD5d8L7D6a/vSnJk= 2022-09-02T04:29:01.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Tygbjq+Xr7dwD5d8L7D6a/vSnJk='} 2022-09-02T04:29:01.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CYY.pls5x.GGjUgn2i-woRp4HL7NkyN'), ('x-amz-request-id', 'tx0000092aa4fab8c022c0e-006311868d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.828 DEBUG:rgw_multi.tests:version3 id=CYY.pls5x.GGjUgn2i-woRp4HL7NkyN 2022-09-02T04:29:01.828 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.829 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:01.829 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=CYY.pls5x.GGjUgn2i-woRp4HL7NkyN 2022-09-02T04:29:01.830 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=CYY.pls5x.GGjUgn2i-woRp4HL7NkyN 2022-09-02T04:29:01.830 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.831 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=CYY.pls5x.GGjUgn2i-woRp4HL7NkyN 2022-09-02T04:29:01.831 DEBUG:boto:Data: 2022-09-02T04:29:01.832 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:01.833 DEBUG:boto:Port: 8001 2022-09-02T04:29:01.833 DEBUG:boto:Params: {} 2022-09-02T04:29:01.834 DEBUG:boto:Token: None 2022-09-02T04:29:01.834 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-16/obj-test-zone3?versionId=CYY.pls5x.GGjUgn2i-woRp4HL7NkyN 2022-09-02T04:29:01.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ahw8d1PNUWJvw2bUumdKQELpys= 2022-09-02T04:29:01.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ahw8d1PNUWJvw2bUumdKQELpys=', 'Content-Length': '0'} 2022-09-02T04:29:01.853 DEBUG:boto:Response headers: [('x-amz-version-id', 'CYY.pls5x.GGjUgn2i-woRp4HL7NkyN'), ('x-amz-request-id', 'tx0000015b350a308191006-006311868d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.855 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:29:01.856 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:29:01.857 DEBUG:boto:Method: HEAD 2022-09-02T04:29:01.858 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:29:01.859 DEBUG:boto:Data: 2022-09-02T04:29:01.860 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.862 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.863 DEBUG:boto:Params: {} 2022-09-02T04:29:01.864 DEBUG:boto:Token: None 2022-09-02T04:29:01.865 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-17/ 2022-09-02T04:29:01.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vjgPNwILB5URYm/rLaxOFxTMbJI= 2022-09-02T04:29:01.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vjgPNwILB5URYm/rLaxOFxTMbJI=', 'Content-Length': '0'} 2022-09-02T04:29:01.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b64ab98bee8f0ea4-006311868d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.877 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.878 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.879 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.880 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.881 DEBUG:boto:Data: 2022-09-02T04:29:01.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.886 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.887 DEBUG:boto:Params: {} 2022-09-02T04:29:01.887 DEBUG:boto:Token: None 2022-09-02T04:29:01.888 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mqCyTvcfdhIat3szEaAWzOWyIUU= 2022-09-02T04:29:01.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mqCyTvcfdhIat3szEaAWzOWyIUU='} 2022-09-02T04:29:01.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-amz-request-id', 'tx00000f4e1882431dfd55f-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.904 DEBUG:rgw_multi.tests:version1 id=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:29:01.905 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.905 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.906 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.906 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.907 DEBUG:boto:Data: 2022-09-02T04:29:01.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.910 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.911 DEBUG:boto:Params: {} 2022-09-02T04:29:01.911 DEBUG:boto:Token: None 2022-09-02T04:29:01.912 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7A4P26iv9LirEytnnj3U9chcjg= 2022-09-02T04:29:01.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7A4P26iv9LirEytnnj3U9chcjg='} 2022-09-02T04:29:01.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Xl8vBVDVrOU5K168aFl.hRGELvQdPXz'), ('x-amz-request-id', 'tx000005f4b8be3346d429b-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.964 DEBUG:rgw_multi.tests:version2 id=Xl8vBVDVrOU5K168aFl.hRGELvQdPXz 2022-09-02T04:29:01.964 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.965 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.966 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=Xl8vBVDVrOU5K168aFl.hRGELvQdPXz 2022-09-02T04:29:01.967 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=Xl8vBVDVrOU5K168aFl.hRGELvQdPXz 2022-09-02T04:29:01.967 DEBUG:boto:Method: DELETE 2022-09-02T04:29:01.968 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=Xl8vBVDVrOU5K168aFl.hRGELvQdPXz 2022-09-02T04:29:01.970 DEBUG:boto:Data: 2022-09-02T04:29:01.971 DEBUG:boto:Headers: {} 2022-09-02T04:29:01.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.973 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.974 DEBUG:boto:Params: {} 2022-09-02T04:29:01.975 DEBUG:boto:Token: None 2022-09-02T04:29:01.976 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:01 GMT /xljbkf-17/obj-test-zone1?versionId=Xl8vBVDVrOU5K168aFl.hRGELvQdPXz 2022-09-02T04:29:01.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44+PRKL2c/oQK4HXTJN8NmtY5o0= 2022-09-02T04:29:01.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44+PRKL2c/oQK4HXTJN8NmtY5o0=', 'Content-Length': '0'} 2022-09-02T04:29:01.991 DEBUG:boto:Response headers: [('x-amz-version-id', 'Xl8vBVDVrOU5K168aFl.hRGELvQdPXz'), ('x-amz-request-id', 'tx00000d488d0297214d2fd-006311868d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:01.993 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.993 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.995 DEBUG:boto:Method: PUT 2022-09-02T04:29:01.995 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:29:01.996 DEBUG:boto:Data: 2022-09-02T04:29:01.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:01.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:01.998 DEBUG:boto:Port: 8000 2022-09-02T04:29:01.999 DEBUG:boto:Params: {} 2022-09-02T04:29:01.999 DEBUG:boto:Token: None 2022-09-02T04:29:02.000 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:29:02.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LKF97sskffeRjRahO7lM4Dy7iZc= 2022-09-02T04:29:02.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LKF97sskffeRjRahO7lM4Dy7iZc='} 2022-09-02T04:29:02.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lpMMhw.bZbJdNedM6SxBH10lHnzfmv4'), ('x-amz-request-id', 'tx000009a22a855ca2b1481-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.021 DEBUG:rgw_multi.tests:version3 id=lpMMhw.bZbJdNedM6SxBH10lHnzfmv4 2022-09-02T04:29:02.022 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:02.023 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:29:02.024 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=lpMMhw.bZbJdNedM6SxBH10lHnzfmv4 2022-09-02T04:29:02.025 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=lpMMhw.bZbJdNedM6SxBH10lHnzfmv4 2022-09-02T04:29:02.026 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.026 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=lpMMhw.bZbJdNedM6SxBH10lHnzfmv4 2022-09-02T04:29:02.028 DEBUG:boto:Data: 2022-09-02T04:29:02.029 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.031 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.032 DEBUG:boto:Params: {} 2022-09-02T04:29:02.033 DEBUG:boto:Token: None 2022-09-02T04:29:02.034 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone1?versionId=lpMMhw.bZbJdNedM6SxBH10lHnzfmv4 2022-09-02T04:29:02.035 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/sPaG2/wLYcR1DBPHJzH6ggAyGE= 2022-09-02T04:29:02.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/sPaG2/wLYcR1DBPHJzH6ggAyGE=', 'Content-Length': '0'} 2022-09-02T04:29:02.048 DEBUG:boto:Response headers: [('x-amz-version-id', 'lpMMhw.bZbJdNedM6SxBH10lHnzfmv4'), ('x-amz-request-id', 'tx00000aac734e2cda7ed55-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.051 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:29:02.052 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:29:02.053 DEBUG:boto:Method: HEAD 2022-09-02T04:29:02.054 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:29:02.054 DEBUG:boto:Data: 2022-09-02T04:29:02.055 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.057 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.058 DEBUG:boto:Params: {} 2022-09-02T04:29:02.059 DEBUG:boto:Token: None 2022-09-02T04:29:02.061 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/ 2022-09-02T04:29:02.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uyWYAugXGLcyMZlLtIRkG1VrCG8= 2022-09-02T04:29:02.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uyWYAugXGLcyMZlLtIRkG1VrCG8=', 'Content-Length': '0'} 2022-09-02T04:29:02.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cef6095839ea2036-006311868e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.067 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.068 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.069 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.070 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.071 DEBUG:boto:Data: 2022-09-02T04:29:02.072 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.074 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.075 DEBUG:boto:Params: {} 2022-09-02T04:29:02.077 DEBUG:boto:Token: None 2022-09-02T04:29:02.078 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZL6mjEc3eZ3XFZ1gGu+d6h/hv/c= 2022-09-02T04:29:02.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZL6mjEc3eZ3XFZ1gGu+d6h/hv/c='} 2022-09-02T04:29:02.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-amz-request-id', 'tx00000d84636aff24b4d34-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.104 DEBUG:rgw_multi.tests:version1 id=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:29:02.104 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.105 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.106 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.109 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.109 DEBUG:boto:Data: 2022-09-02T04:29:02.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.113 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.114 DEBUG:boto:Params: {} 2022-09-02T04:29:02.115 DEBUG:boto:Token: None 2022-09-02T04:29:02.115 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SukWERA+US5KVQH/luh5Rz8abAE= 2022-09-02T04:29:02.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SukWERA+US5KVQH/luh5Rz8abAE='} 2022-09-02T04:29:02.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3oQx6FfAotUDlTFJduwdNbuS.rTIUE7'), ('x-amz-request-id', 'tx000009c6ba5a54ce3fe54-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.128 DEBUG:rgw_multi.tests:version2 id=3oQx6FfAotUDlTFJduwdNbuS.rTIUE7 2022-09-02T04:29:02.129 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.129 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.130 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=3oQx6FfAotUDlTFJduwdNbuS.rTIUE7 2022-09-02T04:29:02.130 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=3oQx6FfAotUDlTFJduwdNbuS.rTIUE7 2022-09-02T04:29:02.131 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.131 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=3oQx6FfAotUDlTFJduwdNbuS.rTIUE7 2022-09-02T04:29:02.132 DEBUG:boto:Data: 2022-09-02T04:29:02.133 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.134 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.134 DEBUG:boto:Params: {} 2022-09-02T04:29:02.135 DEBUG:boto:Token: None 2022-09-02T04:29:02.135 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone2?versionId=3oQx6FfAotUDlTFJduwdNbuS.rTIUE7 2022-09-02T04:29:02.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pzHQLOlf2h3QQxsYIDwnBX37Y8U= 2022-09-02T04:29:02.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pzHQLOlf2h3QQxsYIDwnBX37Y8U=', 'Content-Length': '0'} 2022-09-02T04:29:02.150 DEBUG:boto:Response headers: [('x-amz-version-id', '3oQx6FfAotUDlTFJduwdNbuS.rTIUE7'), ('x-amz-request-id', 'tx00000895c8aeb0fd8c15e-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.151 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.152 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.153 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.154 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.155 DEBUG:boto:Data: 2022-09-02T04:29:02.156 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.158 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.159 DEBUG:boto:Params: {} 2022-09-02T04:29:02.160 DEBUG:boto:Token: None 2022-09-02T04:29:02.161 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EhdWL1DM95x+PjjaiV4U/+228vM= 2022-09-02T04:29:02.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EhdWL1DM95x+PjjaiV4U/+228vM='} 2022-09-02T04:29:02.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x'), ('x-amz-request-id', 'tx000005b8fc23387efe1e6-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.208 DEBUG:rgw_multi.tests:version3 id=3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x 2022-09-02T04:29:02.209 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.211 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:29:02.212 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x 2022-09-02T04:29:02.212 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x 2022-09-02T04:29:02.213 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.213 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x 2022-09-02T04:29:02.214 DEBUG:boto:Data: 2022-09-02T04:29:02.215 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.216 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.216 DEBUG:boto:Params: {} 2022-09-02T04:29:02.217 DEBUG:boto:Token: None 2022-09-02T04:29:02.217 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone2?versionId=3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x 2022-09-02T04:29:02.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:itNslcHP56IM+hd87qIfp7CAfgM= 2022-09-02T04:29:02.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:itNslcHP56IM+hd87qIfp7CAfgM=', 'Content-Length': '0'} 2022-09-02T04:29:02.230 DEBUG:boto:Response headers: [('x-amz-version-id', '3l9kN5VoIVlmY3z-AtgAbx.fv7C4M9x'), ('x-amz-request-id', 'tx0000056b177205f43706a-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.232 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:29:02.233 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:29:02.233 DEBUG:boto:Method: HEAD 2022-09-02T04:29:02.234 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:29:02.235 DEBUG:boto:Data: 2022-09-02T04:29:02.236 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.237 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.237 DEBUG:boto:Params: {} 2022-09-02T04:29:02.238 DEBUG:boto:Token: None 2022-09-02T04:29:02.238 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/ 2022-09-02T04:29:02.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uyWYAugXGLcyMZlLtIRkG1VrCG8= 2022-09-02T04:29:02.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uyWYAugXGLcyMZlLtIRkG1VrCG8=', 'Content-Length': '0'} 2022-09-02T04:29:02.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad52fc54111eab41-006311868e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.244 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.245 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.245 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.246 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.247 DEBUG:boto:Data: 2022-09-02T04:29:02.247 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.248 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.248 DEBUG:boto:Params: {} 2022-09-02T04:29:02.249 DEBUG:boto:Token: None 2022-09-02T04:29:02.250 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rIPRwZkcX3DVWbKAqu99jkKVjJs= 2022-09-02T04:29:02.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rIPRwZkcX3DVWbKAqu99jkKVjJs='} 2022-09-02T04:29:02.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-amz-request-id', 'tx00000ddc3cbfdcbff6562-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.263 DEBUG:rgw_multi.tests:version1 id=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:29:02.264 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.265 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.265 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.266 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.266 DEBUG:boto:Data: 2022-09-02T04:29:02.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.268 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.268 DEBUG:boto:Params: {} 2022-09-02T04:29:02.269 DEBUG:boto:Token: None 2022-09-02T04:29:02.269 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wNpu+BT//P5Uq5cRxxnwCZ1w/VQ= 2022-09-02T04:29:02.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wNpu+BT//P5Uq5cRxxnwCZ1w/VQ='} 2022-09-02T04:29:02.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nMkplhWE9u8qmXYucRdqlBjSlkekpqk'), ('x-amz-request-id', 'tx0000089a97e8706c000c5-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.315 DEBUG:rgw_multi.tests:version2 id=nMkplhWE9u8qmXYucRdqlBjSlkekpqk 2022-09-02T04:29:02.316 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.316 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.317 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=nMkplhWE9u8qmXYucRdqlBjSlkekpqk 2022-09-02T04:29:02.317 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=nMkplhWE9u8qmXYucRdqlBjSlkekpqk 2022-09-02T04:29:02.318 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.318 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=nMkplhWE9u8qmXYucRdqlBjSlkekpqk 2022-09-02T04:29:02.319 DEBUG:boto:Data: 2022-09-02T04:29:02.319 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.321 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.322 DEBUG:boto:Params: {} 2022-09-02T04:29:02.323 DEBUG:boto:Token: None 2022-09-02T04:29:02.323 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone3?versionId=nMkplhWE9u8qmXYucRdqlBjSlkekpqk 2022-09-02T04:29:02.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://v7AcnyG2Q0UYkNts+/Q+fpV+8= 2022-09-02T04:29:02.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://v7AcnyG2Q0UYkNts+/Q+fpV+8=', 'Content-Length': '0'} 2022-09-02T04:29:02.336 DEBUG:boto:Response headers: [('x-amz-version-id', 'nMkplhWE9u8qmXYucRdqlBjSlkekpqk'), ('x-amz-request-id', 'tx0000008b5992171dab2c4-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.338 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.339 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.340 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.341 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.342 DEBUG:boto:Data: 2022-09-02T04:29:02.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.345 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.346 DEBUG:boto:Params: {} 2022-09-02T04:29:02.347 DEBUG:boto:Token: None 2022-09-02T04:29:02.348 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7GvQfLXSvmOghVX06aF6nocFp98= 2022-09-02T04:29:02.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7GvQfLXSvmOghVX06aF6nocFp98='} 2022-09-02T04:29:02.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6'), ('x-amz-request-id', 'tx000001c8c5a4392c34297-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.370 DEBUG:rgw_multi.tests:version3 id=oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6 2022-09-02T04:29:02.371 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.372 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:29:02.373 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6 2022-09-02T04:29:02.374 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6 2022-09-02T04:29:02.375 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.377 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6 2022-09-02T04:29:02.378 DEBUG:boto:Data: 2022-09-02T04:29:02.379 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.380 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.382 DEBUG:boto:Params: {} 2022-09-02T04:29:02.383 DEBUG:boto:Token: None 2022-09-02T04:29:02.383 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-17/obj-test-zone3?versionId=oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6 2022-09-02T04:29:02.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YBkck3McSmki2gFkeH1BPETx6Oc= 2022-09-02T04:29:02.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YBkck3McSmki2gFkeH1BPETx6Oc=', 'Content-Length': '0'} 2022-09-02T04:29:02.403 DEBUG:boto:Response headers: [('x-amz-version-id', 'oFuM3d3-e6WT6i5oyoFST1HIhjvqkT6'), ('x-amz-request-id', 'tx00000fc234e4773470bee-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.404 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:29:02.404 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:29:02.405 DEBUG:boto:Method: HEAD 2022-09-02T04:29:02.405 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:29:02.406 DEBUG:boto:Data: 2022-09-02T04:29:02.407 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.409 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.410 DEBUG:boto:Params: {} 2022-09-02T04:29:02.411 DEBUG:boto:Token: None 2022-09-02T04:29:02.412 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/ 2022-09-02T04:29:02.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CP5jOukY9xhyZweghKvWzxfq9qU= 2022-09-02T04:29:02.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CP5jOukY9xhyZweghKvWzxfq9qU=', 'Content-Length': '0'} 2022-09-02T04:29:02.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008962c557418ae89c-006311868e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.418 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.419 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.419 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.420 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.421 DEBUG:boto:Data: 2022-09-02T04:29:02.421 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.422 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.423 DEBUG:boto:Params: {} 2022-09-02T04:29:02.423 DEBUG:boto:Token: None 2022-09-02T04:29:02.424 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q2RaPzz0ADmrA++vJgXsfN24Pjo= 2022-09-02T04:29:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q2RaPzz0ADmrA++vJgXsfN24Pjo='} 2022-09-02T04:29:02.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-amz-request-id', 'tx00000900f6d4509624441-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.440 DEBUG:rgw_multi.tests:version1 id=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:29:02.442 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.443 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.444 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.445 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.446 DEBUG:boto:Data: 2022-09-02T04:29:02.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.448 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.451 DEBUG:boto:Params: {} 2022-09-02T04:29:02.452 DEBUG:boto:Token: None 2022-09-02T04:29:02.454 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GXj3NPu22P/ZoulxjmKi2foYQWA= 2022-09-02T04:29:02.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GXj3NPu22P/ZoulxjmKi2foYQWA='} 2022-09-02T04:29:02.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G'), ('x-amz-request-id', 'tx0000026e1674b1503e7db-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.471 DEBUG:rgw_multi.tests:version2 id=QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G 2022-09-02T04:29:02.471 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.472 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.472 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G 2022-09-02T04:29:02.473 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G 2022-09-02T04:29:02.473 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.474 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G 2022-09-02T04:29:02.475 DEBUG:boto:Data: 2022-09-02T04:29:02.475 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.477 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.477 DEBUG:boto:Params: {} 2022-09-02T04:29:02.478 DEBUG:boto:Token: None 2022-09-02T04:29:02.478 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone1?versionId=QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G 2022-09-02T04:29:02.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rA+hUl13RzOvI/aZ1pB5TM70Ds0= 2022-09-02T04:29:02.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rA+hUl13RzOvI/aZ1pB5TM70Ds0=', 'Content-Length': '0'} 2022-09-02T04:29:02.490 DEBUG:boto:Response headers: [('x-amz-version-id', 'QdU6eEDj.XcPgwBjh2GL6Fbus9IT-3G'), ('x-amz-request-id', 'tx00000747c0e2b271dd927-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.493 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.494 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.495 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.495 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.496 DEBUG:boto:Data: 2022-09-02T04:29:02.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.499 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.500 DEBUG:boto:Params: {} 2022-09-02T04:29:02.501 DEBUG:boto:Token: None 2022-09-02T04:29:02.502 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0tBZMNHKaDEMhP218J3O+P6vyKk= 2022-09-02T04:29:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0tBZMNHKaDEMhP218J3O+P6vyKk='} 2022-09-02T04:29:02.550 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'IqIxftznE7187Jdt7lFi7ibhdF1zioQ'), ('x-amz-request-id', 'tx000001269169598ef5c3f-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.551 DEBUG:rgw_multi.tests:version3 id=IqIxftznE7187Jdt7lFi7ibhdF1zioQ 2022-09-02T04:29:02.552 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.553 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:29:02.554 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=IqIxftznE7187Jdt7lFi7ibhdF1zioQ 2022-09-02T04:29:02.555 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=IqIxftznE7187Jdt7lFi7ibhdF1zioQ 2022-09-02T04:29:02.555 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.556 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=IqIxftznE7187Jdt7lFi7ibhdF1zioQ 2022-09-02T04:29:02.557 DEBUG:boto:Data: 2022-09-02T04:29:02.559 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:02.561 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.561 DEBUG:boto:Params: {} 2022-09-02T04:29:02.562 DEBUG:boto:Token: None 2022-09-02T04:29:02.563 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone1?versionId=IqIxftznE7187Jdt7lFi7ibhdF1zioQ 2022-09-02T04:29:02.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cCFbcFgAiyKqI/zRe87+ZV5Ii7E= 2022-09-02T04:29:02.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cCFbcFgAiyKqI/zRe87+ZV5Ii7E=', 'Content-Length': '0'} 2022-09-02T04:29:02.583 DEBUG:boto:Response headers: [('x-amz-version-id', 'IqIxftznE7187Jdt7lFi7ibhdF1zioQ'), ('x-amz-request-id', 'tx00000ccb48a0e8ed8b627-006311868e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.584 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:29:02.585 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:29:02.585 DEBUG:boto:Method: HEAD 2022-09-02T04:29:02.586 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:29:02.586 DEBUG:boto:Data: 2022-09-02T04:29:02.587 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.588 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.588 DEBUG:boto:Params: {} 2022-09-02T04:29:02.589 DEBUG:boto:Token: None 2022-09-02T04:29:02.590 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/ 2022-09-02T04:29:02.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CP5jOukY9xhyZweghKvWzxfq9qU= 2022-09-02T04:29:02.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CP5jOukY9xhyZweghKvWzxfq9qU=', 'Content-Length': '0'} 2022-09-02T04:29:02.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd3f4165d2c38cea-006311868e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.596 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.596 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.597 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.598 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.598 DEBUG:boto:Data: 2022-09-02T04:29:02.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.600 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.600 DEBUG:boto:Params: {} 2022-09-02T04:29:02.601 DEBUG:boto:Token: None 2022-09-02T04:29:02.601 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cdON+6vXgZ5hzSdwDuChTLxU/3g= 2022-09-02T04:29:02.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cdON+6vXgZ5hzSdwDuChTLxU/3g='} 2022-09-02T04:29:02.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-amz-request-id', 'tx0000027e8f3923dfaca6f-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.619 DEBUG:rgw_multi.tests:version1 id=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:29:02.620 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.621 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.622 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.623 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.624 DEBUG:boto:Data: 2022-09-02T04:29:02.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.626 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.627 DEBUG:boto:Params: {} 2022-09-02T04:29:02.628 DEBUG:boto:Token: None 2022-09-02T04:29:02.629 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zK/o9U7PV9eD5cDbkvpXyFwz9Gc= 2022-09-02T04:29:02.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zK/o9U7PV9eD5cDbkvpXyFwz9Gc='} 2022-09-02T04:29:02.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qWF50uQrXhye5lRFr13xWnxgGdZLW.M'), ('x-amz-request-id', 'tx00000a2501e01ea86249c-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.680 DEBUG:rgw_multi.tests:version2 id=qWF50uQrXhye5lRFr13xWnxgGdZLW.M 2022-09-02T04:29:02.680 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.681 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.681 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=qWF50uQrXhye5lRFr13xWnxgGdZLW.M 2022-09-02T04:29:02.682 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=qWF50uQrXhye5lRFr13xWnxgGdZLW.M 2022-09-02T04:29:02.683 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.684 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=qWF50uQrXhye5lRFr13xWnxgGdZLW.M 2022-09-02T04:29:02.685 DEBUG:boto:Data: 2022-09-02T04:29:02.686 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.687 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.688 DEBUG:boto:Params: {} 2022-09-02T04:29:02.689 DEBUG:boto:Token: None 2022-09-02T04:29:02.692 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone2?versionId=qWF50uQrXhye5lRFr13xWnxgGdZLW.M 2022-09-02T04:29:02.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:crmsodjoX9y1Gbozp/W0mRDt82s= 2022-09-02T04:29:02.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:crmsodjoX9y1Gbozp/W0mRDt82s=', 'Content-Length': '0'} 2022-09-02T04:29:02.706 DEBUG:boto:Response headers: [('x-amz-version-id', 'qWF50uQrXhye5lRFr13xWnxgGdZLW.M'), ('x-amz-request-id', 'tx00000a9b3f23f22fcfe78-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.707 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.708 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.708 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.709 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.709 DEBUG:boto:Data: 2022-09-02T04:29:02.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.712 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.713 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.714 DEBUG:boto:Params: {} 2022-09-02T04:29:02.715 DEBUG:boto:Token: None 2022-09-02T04:29:02.716 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U5sNS37zfV1Dxjz18lD1F5x/Dro= 2022-09-02T04:29:02.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U5sNS37zfV1Dxjz18lD1F5x/Dro='} 2022-09-02T04:29:02.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA'), ('x-amz-request-id', 'tx00000c953bcd58bd93536-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.734 DEBUG:rgw_multi.tests:version3 id=hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA 2022-09-02T04:29:02.736 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.737 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:29:02.740 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA 2022-09-02T04:29:02.741 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA 2022-09-02T04:29:02.742 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.743 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA 2022-09-02T04:29:02.743 DEBUG:boto:Data: 2022-09-02T04:29:02.744 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:02.745 DEBUG:boto:Port: 8000 2022-09-02T04:29:02.746 DEBUG:boto:Params: {} 2022-09-02T04:29:02.746 DEBUG:boto:Token: None 2022-09-02T04:29:02.747 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone2?versionId=hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA 2022-09-02T04:29:02.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2tGBT8Vitz5/AA14tbJUFOvyor4= 2022-09-02T04:29:02.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2tGBT8Vitz5/AA14tbJUFOvyor4=', 'Content-Length': '0'} 2022-09-02T04:29:02.770 DEBUG:boto:Response headers: [('x-amz-version-id', 'hMBCEk-DSCVI2zciZSUuvP8gNf1bqXA'), ('x-amz-request-id', 'tx00000cbe9ca741431432e-006311868e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.772 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:29:02.772 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:29:02.773 DEBUG:boto:Method: HEAD 2022-09-02T04:29:02.774 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:29:02.775 DEBUG:boto:Data: 2022-09-02T04:29:02.776 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.778 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.779 DEBUG:boto:Params: {} 2022-09-02T04:29:02.780 DEBUG:boto:Token: None 2022-09-02T04:29:02.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/ 2022-09-02T04:29:02.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CP5jOukY9xhyZweghKvWzxfq9qU= 2022-09-02T04:29:02.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CP5jOukY9xhyZweghKvWzxfq9qU=', 'Content-Length': '0'} 2022-09-02T04:29:02.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000463750979a3c20e9-006311868e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.790 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.790 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.793 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.794 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.795 DEBUG:boto:Data: 2022-09-02T04:29:02.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.798 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.799 DEBUG:boto:Params: {} 2022-09-02T04:29:02.800 DEBUG:boto:Token: None 2022-09-02T04:29:02.801 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+P4q01TlQ4vazYoqHued/fYml0Q= 2022-09-02T04:29:02.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+P4q01TlQ4vazYoqHued/fYml0Q='} 2022-09-02T04:29:02.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-amz-request-id', 'tx00000c78b2ffc67cd71a8-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.824 DEBUG:rgw_multi.tests:version1 id=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:29:02.825 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.826 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.827 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.828 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.829 DEBUG:boto:Data: 2022-09-02T04:29:02.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.832 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.833 DEBUG:boto:Params: {} 2022-09-02T04:29:02.834 DEBUG:boto:Token: None 2022-09-02T04:29:02.835 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FwEvN9vaB421b5ePk4tGnuz4kNk= 2022-09-02T04:29:02.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FwEvN9vaB421b5ePk4tGnuz4kNk='} 2022-09-02T04:29:02.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz'), ('x-amz-request-id', 'tx00000cdaf5d94b1976de4-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.856 DEBUG:rgw_multi.tests:version2 id=qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz 2022-09-02T04:29:02.857 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.857 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.857 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz 2022-09-02T04:29:02.858 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz 2022-09-02T04:29:02.859 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.859 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz 2022-09-02T04:29:02.860 DEBUG:boto:Data: 2022-09-02T04:29:02.860 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.861 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.862 DEBUG:boto:Params: {} 2022-09-02T04:29:02.863 DEBUG:boto:Token: None 2022-09-02T04:29:02.863 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone3?versionId=qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz 2022-09-02T04:29:02.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hfi3sc8O7rrWY5z1nzn7sTLc2rM= 2022-09-02T04:29:02.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hfi3sc8O7rrWY5z1nzn7sTLc2rM=', 'Content-Length': '0'} 2022-09-02T04:29:02.881 DEBUG:boto:Response headers: [('x-amz-version-id', 'qu.-K0JV8vm7FP0VKj894Ny9ysVxFBz'), ('x-amz-request-id', 'tx00000d71920da09167a23-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.883 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.883 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.884 DEBUG:boto:Method: PUT 2022-09-02T04:29:02.885 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.886 DEBUG:boto:Data: 2022-09-02T04:29:02.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:29:02.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.891 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.891 DEBUG:boto:Params: {} 2022-09-02T04:29:02.893 DEBUG:boto:Token: None 2022-09-02T04:29:02.894 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J1KfzrHsWmj0L5BiWHBsz0c0MsM= 2022-09-02T04:29:02.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J1KfzrHsWmj0L5BiWHBsz0c0MsM='} 2022-09-02T04:29:02.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KoXD4qlVluymXNbUsYV9omFnsElVuPK'), ('x-amz-request-id', 'tx00000b5444a7457d9ea4c-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.944 DEBUG:rgw_multi.tests:version3 id=KoXD4qlVluymXNbUsYV9omFnsElVuPK 2022-09-02T04:29:02.945 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.946 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:29:02.947 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=KoXD4qlVluymXNbUsYV9omFnsElVuPK 2022-09-02T04:29:02.948 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=KoXD4qlVluymXNbUsYV9omFnsElVuPK 2022-09-02T04:29:02.949 DEBUG:boto:Method: DELETE 2022-09-02T04:29:02.950 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=KoXD4qlVluymXNbUsYV9omFnsElVuPK 2022-09-02T04:29:02.951 DEBUG:boto:Data: 2022-09-02T04:29:02.951 DEBUG:boto:Headers: {} 2022-09-02T04:29:02.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:02.952 DEBUG:boto:Port: 8001 2022-09-02T04:29:02.953 DEBUG:boto:Params: {} 2022-09-02T04:29:02.953 DEBUG:boto:Token: None 2022-09-02T04:29:02.954 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:29:02 GMT /xljbkf-18/obj-test-zone3?versionId=KoXD4qlVluymXNbUsYV9omFnsElVuPK 2022-09-02T04:29:02.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kJUeJRdZXFNjuc47iH6YjR0YWQE= 2022-09-02T04:29:02.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kJUeJRdZXFNjuc47iH6YjR0YWQE=', 'Content-Length': '0'} 2022-09-02T04:29:02.972 DEBUG:boto:Response headers: [('x-amz-version-id', 'KoXD4qlVluymXNbUsYV9omFnsElVuPK'), ('x-amz-request-id', 'tx000002b85bdf284dcaf89-006311868e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:02.972 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-16 2022-09-02T04:29:02.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:02.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:02.975 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:03.041 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:03.041 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:03.082 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:03.080+0000 7f1992256600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:03.082 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:03.083 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:08.083+0000 7f1992256600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:08.085 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:08.085 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:13.085 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:13.084+0000 7f1992256600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:13.086 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:13.086 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:18.086 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:18.086+0000 7f1992256600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:18.087 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:18.087 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:23.088 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:23.086+0000 7f1992256600 1 bucket sync caught up with source: 2022-09-02T04:29:23.088 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:23.089 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:23.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:23.086+0000 7f1992256600 0 bucket checkpoint complete 2022-09-02T04:29:23.104 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-16 2022-09-02T04:29:23.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:23.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:23.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:23.172 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:23.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:23.216 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:23.215+0000 7f28eb6fb600 1 bucket sync caught up with source: 2022-09-02T04:29:23.217 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000010.31.5, , 00000000002.4.10, , 00000000005.17.9, , , , , ] 2022-09-02T04:29:23.217 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000010.31.5, , 00000000002.4.10, , 00000000005.17.9, , , , , ] 2022-09-02T04:29:23.217 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:23.215+0000 7f28eb6fb600 0 bucket checkpoint complete 2022-09-02T04:29:23.229 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-16 2022-09-02T04:29:23.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:23.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:23.231 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:23.295 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:23.295 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:23.332 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:23.331+0000 7fe3f82be600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:23.333 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:23.334 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , 00000000002.4.10, , 00000000005.17.9, , , , , ] 2022-09-02T04:29:28.334 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:28.333+0000 7fe3f82be600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:28.335 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , , , 00000000005.17.9, , , , , ] 2022-09-02T04:29:28.335 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , 00000000002.4.10, , 00000000005.17.9, , , , , ] 2022-09-02T04:29:33.335 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:33.334+0000 7fe3f82be600 1 bucket sync caught up with source: 2022-09-02T04:29:33.336 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , 00000000002.4.10, , 00000000005.17.9, , , , , ] 2022-09-02T04:29:33.336 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.31.5, , 00000000002.4.10, , 00000000005.17.9, , , , , ] 2022-09-02T04:29:33.337 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:33.334+0000 7fe3f82be600 0 bucket checkpoint complete 2022-09-02T04:29:33.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-16 2022-09-02T04:29:33.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:33.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:33.356 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:33.421 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:33.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:33.480 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:33.479+0000 7f55e03ba600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:33.480 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , 00000000004.13.11, , 00000000004.42.9, , , , , ] 2022-09-02T04:29:33.481 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.66.10, , 00000000010.25.5, , 00000000004.42.9, , , , , ] 2022-09-02T04:29:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:38.481+0000 7f55e03ba600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:38.482 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , 00000000004.13.11, , 00000000004.42.9, , , , , ] 2022-09-02T04:29:38.482 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.66.10, , 00000000010.25.5, , 00000000004.42.9, , , , , ] 2022-09-02T04:29:43.483 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:43.481+0000 7f55e03ba600 1 bucket sync caught up with source: 2022-09-02T04:29:43.483 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.68.9, , 00000000010.25.5, , 00000000007.48.9, , , , , ] 2022-09-02T04:29:43.483 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.66.10, , 00000000010.25.5, , 00000000004.42.9, , , , , ] 2022-09-02T04:29:43.484 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:43.481+0000 7f55e03ba600 0 bucket checkpoint complete 2022-09-02T04:29:43.498 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-16 2022-09-02T04:29:43.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:43.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:43.501 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:43.562 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:43.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:43.615 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:43.614+0000 7f89f0d3a600 1 bucket sync caught up with source: 2022-09-02T04:29:43.616 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.68.9, , 00000000010.25.5, , 00000000007.48.9, , , , , ] 2022-09-02T04:29:43.616 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.68.9, , 00000000010.25.5, , 00000000007.48.9, , , , , ] 2022-09-02T04:29:43.616 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:43.614+0000 7f89f0d3a600 0 bucket checkpoint complete 2022-09-02T04:29:43.629 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-16 2022-09-02T04:29:43.630 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-16 2022-09-02T04:29:43.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:43.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:43.633 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:43.691 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:43.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:43.723+0000 7f124f349600 1 bucket sync caught up with source: 2022-09-02T04:29:43.725 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.40.9, , 00000000003.7.9, , 00000000014.53.5, , , , , ] 2022-09-02T04:29:43.725 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.40.9, , 00000000003.7.9, , 00000000014.53.5, , , , , ] 2022-09-02T04:29:43.725 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:29:43.723+0000 7f124f349600 0 bucket checkpoint complete 2022-09-02T04:29:43.737 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-16 2022-09-02T04:29:43.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:43.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:43.740 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:43.808 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:43.809 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:43.843 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:43.842+0000 7f602c0b1600 1 bucket sync caught up with source: 2022-09-02T04:29:43.843 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.40.9, , 00000000003.7.9, , 00000000014.53.5, , , , , ] 2022-09-02T04:29:43.844 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.40.9, , 00000000003.7.9, , 00000000014.53.5, , , , , ] 2022-09-02T04:29:43.844 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:43.842+0000 7f602c0b1600 0 bucket checkpoint complete 2022-09-02T04:29:43.856 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-16 2022-09-02T04:29:43.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-16 zones={test-zone2, test-zone1} 2022-09-02T04:29:43.858 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:43.858 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:43.859 DEBUG:boto:Method: HEAD 2022-09-02T04:29:43.859 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:43.860 DEBUG:boto:Data: 2022-09-02T04:29:43.861 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:43.863 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.864 DEBUG:boto:Params: {} 2022-09-02T04:29:43.865 DEBUG:boto:Token: None 2022-09-02T04:29:43.865 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/ 2022-09-02T04:29:43.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7mVFeHQg50CGtJr2LHDbgJWiuz8= 2022-09-02T04:29:43.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7mVFeHQg50CGtJr2LHDbgJWiuz8=', 'Content-Length': '0'} 2022-09-02T04:29:43.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006747ca33173a3dec-00631186b7-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.871 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:43.872 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:43.872 DEBUG:boto:Method: HEAD 2022-09-02T04:29:43.873 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:43.873 DEBUG:boto:Data: 2022-09-02T04:29:43.874 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:43.875 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.875 DEBUG:boto:Params: {} 2022-09-02T04:29:43.876 DEBUG:boto:Token: None 2022-09-02T04:29:43.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/ 2022-09-02T04:29:43.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7mVFeHQg50CGtJr2LHDbgJWiuz8= 2022-09-02T04:29:43.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7mVFeHQg50CGtJr2LHDbgJWiuz8=', 'Content-Length': '0'} 2022-09-02T04:29:43.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028574f1e56b9e558-00631186b7-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.881 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:29:43.882 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:43.882 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:43.883 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.883 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.884 DEBUG:boto:Method: GET 2022-09-02T04:29:43.885 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.885 DEBUG:boto:Data: 2022-09-02T04:29:43.886 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:43.887 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.887 DEBUG:boto:Params: {} 2022-09-02T04:29:43.888 DEBUG:boto:Token: None 2022-09-02T04:29:43.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/?versions 2022-09-02T04:29:43.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA= 2022-09-02T04:29:43.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA=', 'Content-Length': '0'} 2022-09-02T04:29:43.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000232ca32b82671473-00631186b7-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.893 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:43.894 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:29:43.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:29:43.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:29:43.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:29:43.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:29:43.897 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:43.897 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:43.898 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.898 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.899 DEBUG:boto:Method: GET 2022-09-02T04:29:43.900 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.900 DEBUG:boto:Data: 2022-09-02T04:29:43.901 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:43.902 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.902 DEBUG:boto:Params: {} 2022-09-02T04:29:43.903 DEBUG:boto:Token: None 2022-09-02T04:29:43.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/?versions 2022-09-02T04:29:43.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA= 2022-09-02T04:29:43.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA=', 'Content-Length': '0'} 2022-09-02T04:29:43.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee6f187b45a7cb42-00631186b7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.908 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:43.909 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:29:43.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:29:43.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:29:43.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:29:43.911 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:43.911 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:43.912 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.912 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.913 DEBUG:boto:Method: GET 2022-09-02T04:29:43.913 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.914 DEBUG:boto:Data: 2022-09-02T04:29:43.915 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:43.916 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.916 DEBUG:boto:Params: {} 2022-09-02T04:29:43.917 DEBUG:boto:Token: None 2022-09-02T04:29:43.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/?versions 2022-09-02T04:29:43.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA= 2022-09-02T04:29:43.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA=', 'Content-Length': '0'} 2022-09-02T04:29:43.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d82c604f323b64-00631186b7-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.922 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:43.923 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:43.923 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:43.924 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.924 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.925 DEBUG:boto:Method: GET 2022-09-02T04:29:43.925 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:43.926 DEBUG:boto:Data: 2022-09-02T04:29:43.926 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:43.927 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.928 DEBUG:boto:Params: {} 2022-09-02T04:29:43.929 DEBUG:boto:Token: None 2022-09-02T04:29:43.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/?versions 2022-09-02T04:29:43.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA= 2022-09-02T04:29:43.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaPS5q7GE20X5k1sUZNmitCTjFA=', 'Content-Length': '0'} 2022-09-02T04:29:43.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af83897b0b626fc6-00631186b7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.934 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:43.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:43.936 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:43.936 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:43.937 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.938 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.938 DEBUG:boto:Method: GET 2022-09-02T04:29:43.939 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.940 DEBUG:boto:Data: 2022-09-02T04:29:43.941 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:43.942 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.942 DEBUG:boto:Params: {} 2022-09-02T04:29:43.943 DEBUG:boto:Token: None 2022-09-02T04:29:43.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EZMb1abwmoUhzYuwKCPn+H35MwE= 2022-09-02T04:29:43.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EZMb1abwmoUhzYuwKCPn+H35MwE=', 'Content-Length': '0'} 2022-09-02T04:29:43.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e06a7c6f5c787d69-00631186b7-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.948 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:43.949 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:43.949 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.950 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.951 DEBUG:boto:Method: GET 2022-09-02T04:29:43.952 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.953 DEBUG:boto:Data: 2022-09-02T04:29:43.953 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:43.954 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.955 DEBUG:boto:Params: {} 2022-09-02T04:29:43.956 DEBUG:boto:Token: None 2022-09-02T04:29:43.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EZMb1abwmoUhzYuwKCPn+H35MwE= 2022-09-02T04:29:43.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EZMb1abwmoUhzYuwKCPn+H35MwE=', 'Content-Length': '0'} 2022-09-02T04:29:43.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054f196162da443bb-00631186b7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.961 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:43.962 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:43.962 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.963 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.963 DEBUG:boto:Method: HEAD 2022-09-02T04:29:43.964 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.964 DEBUG:boto:Data: 2022-09-02T04:29:43.965 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:43.966 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.967 DEBUG:boto:Params: {} 2022-09-02T04:29:43.967 DEBUG:boto:Token: None 2022-09-02T04:29:43.968 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TtnIxsguuF2j2pfPh4qp8xbH6kk= 2022-09-02T04:29:43.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TtnIxsguuF2j2pfPh4qp8xbH6kk=', 'Content-Length': '0'} 2022-09-02T04:29:43.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e9dcfad9ae579f2-00631186b7-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.972 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:43.973 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:43.973 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.974 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.974 DEBUG:boto:Method: HEAD 2022-09-02T04:29:43.975 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.975 DEBUG:boto:Data: 2022-09-02T04:29:43.976 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:43.977 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.978 DEBUG:boto:Params: {} 2022-09-02T04:29:43.978 DEBUG:boto:Token: None 2022-09-02T04:29:43.979 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TtnIxsguuF2j2pfPh4qp8xbH6kk= 2022-09-02T04:29:43.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TtnIxsguuF2j2pfPh4qp8xbH6kk=', 'Content-Length': '0'} 2022-09-02T04:29:43.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e34f791a5ff50acf-00631186b7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:43.984 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:43.985 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:43.985 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.986 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.986 DEBUG:boto:Method: GET 2022-09-02T04:29:43.987 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.988 DEBUG:boto:Data: 2022-09-02T04:29:43.988 DEBUG:boto:Headers: {} 2022-09-02T04:29:43.989 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:43.989 DEBUG:boto:Port: 8000 2022-09-02T04:29:43.990 DEBUG:boto:Params: {} 2022-09-02T04:29:43.991 DEBUG:boto:Token: None 2022-09-02T04:29:43.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:43 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EZMb1abwmoUhzYuwKCPn+H35MwE= 2022-09-02T04:29:43.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EZMb1abwmoUhzYuwKCPn+H35MwE=', 'Content-Length': '0'} 2022-09-02T04:29:43.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f7107b4eda04868-00631186b7-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:43.997 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:43.997 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:43.998 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.999 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:43.999 DEBUG:boto:Method: GET 2022-09-02T04:29:44.000 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.000 DEBUG:boto:Data: 2022-09-02T04:29:44.001 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.002 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.003 DEBUG:boto:Params: {} 2022-09-02T04:29:44.004 DEBUG:boto:Token: None 2022-09-02T04:29:44.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D98VkS3l3MQi/u+4p2tMuxbbghk= 2022-09-02T04:29:44.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D98VkS3l3MQi/u+4p2tMuxbbghk=', 'Content-Length': '0'} 2022-09-02T04:29:44.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006e7faffb47f9516-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.011 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.011 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.012 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.012 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.013 DEBUG:boto:Method: GET 2022-09-02T04:29:44.013 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.014 DEBUG:boto:Data: 2022-09-02T04:29:44.014 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.015 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.016 DEBUG:boto:Params: {} 2022-09-02T04:29:44.017 DEBUG:boto:Token: None 2022-09-02T04:29:44.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f838fbb4e94a3211-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.022 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.023 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.023 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.024 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.024 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.025 DEBUG:boto:Method: GET 2022-09-02T04:29:44.025 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.026 DEBUG:boto:Data: 2022-09-02T04:29:44.026 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.027 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.028 DEBUG:boto:Params: {} 2022-09-02T04:29:44.028 DEBUG:boto:Token: None 2022-09-02T04:29:44.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008058c298853e98c6-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.074 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.075 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:44.076 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:44.076 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.077 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:29:44.077 DEBUG:boto:Data: 2022-09-02T04:29:44.078 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.079 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.079 DEBUG:boto:Params: {} 2022-09-02T04:29:44.080 DEBUG:boto:Token: None 2022-09-02T04:29:44.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/dummy 2022-09-02T04:29:44.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mnDY2Gvi88bf481bvgbmdLAzrqk= 2022-09-02T04:29:44.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mnDY2Gvi88bf481bvgbmdLAzrqk=', 'Content-Length': '0'} 2022-09-02T04:29:44.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000377a913841bea009-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.086 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:44.087 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:44.088 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.088 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:29:44.089 DEBUG:boto:Data: 2022-09-02T04:29:44.089 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.090 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.091 DEBUG:boto:Params: {} 2022-09-02T04:29:44.091 DEBUG:boto:Token: None 2022-09-02T04:29:44.092 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/dummy 2022-09-02T04:29:44.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mnDY2Gvi88bf481bvgbmdLAzrqk= 2022-09-02T04:29:44.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mnDY2Gvi88bf481bvgbmdLAzrqk=', 'Content-Length': '0'} 2022-09-02T04:29:44.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c082f3cca8b43a1-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:44.100 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:44.100 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:44.101 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.102 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.103 DEBUG:boto:Method: GET 2022-09-02T04:29:44.104 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.104 DEBUG:boto:Data: 2022-09-02T04:29:44.105 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.106 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.106 DEBUG:boto:Params: {} 2022-09-02T04:29:44.107 DEBUG:boto:Token: None 2022-09-02T04:29:44.108 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D98VkS3l3MQi/u+4p2tMuxbbghk= 2022-09-02T04:29:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D98VkS3l3MQi/u+4p2tMuxbbghk=', 'Content-Length': '0'} 2022-09-02T04:29:44.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba04f842507c96b8-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.112 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:44.113 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:44.113 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.114 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.115 DEBUG:boto:Method: GET 2022-09-02T04:29:44.115 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.116 DEBUG:boto:Data: 2022-09-02T04:29:44.116 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.118 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.119 DEBUG:boto:Params: {} 2022-09-02T04:29:44.120 DEBUG:boto:Token: None 2022-09-02T04:29:44.121 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:44.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D98VkS3l3MQi/u+4p2tMuxbbghk= 2022-09-02T04:29:44.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D98VkS3l3MQi/u+4p2tMuxbbghk=', 'Content-Length': '0'} 2022-09-02T04:29:44.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060ee309ae7bac7d3-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.136 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.136 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.137 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.138 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.138 DEBUG:boto:Method: GET 2022-09-02T04:29:44.139 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.139 DEBUG:boto:Data: 2022-09-02T04:29:44.140 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.140 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.141 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.141 DEBUG:boto:Params: {} 2022-09-02T04:29:44.142 DEBUG:boto:Token: None 2022-09-02T04:29:44.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe1c4b93eef3311-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.147 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.148 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.149 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.149 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.150 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.151 DEBUG:boto:Method: GET 2022-09-02T04:29:44.151 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:44.152 DEBUG:boto:Data: 2022-09-02T04:29:44.153 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.154 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.155 DEBUG:boto:Params: {} 2022-09-02T04:29:44.156 DEBUG:boto:Token: None 2022-09-02T04:29:44.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000012ffbb065aa13bd-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.161 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:44.163 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.163 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.164 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.165 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.165 DEBUG:boto:Method: GET 2022-09-02T04:29:44.166 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.166 DEBUG:boto:Data: 2022-09-02T04:29:44.167 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.168 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.168 DEBUG:boto:Params: {} 2022-09-02T04:29:44.169 DEBUG:boto:Token: None 2022-09-02T04:29:44.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE= 2022-09-02T04:29:44.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE=', 'Content-Length': '0'} 2022-09-02T04:29:44.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e83520884f27e17-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.214 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.215 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.215 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.216 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.216 DEBUG:boto:Method: GET 2022-09-02T04:29:44.217 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.218 DEBUG:boto:Data: 2022-09-02T04:29:44.219 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.220 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.220 DEBUG:boto:Params: {} 2022-09-02T04:29:44.221 DEBUG:boto:Token: None 2022-09-02T04:29:44.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE= 2022-09-02T04:29:44.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE=', 'Content-Length': '0'} 2022-09-02T04:29:44.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ad572c77b71ae87-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.226 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.226 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.227 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.227 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.228 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.229 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.229 DEBUG:boto:Data: 2022-09-02T04:29:44.230 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.231 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.231 DEBUG:boto:Params: {} 2022-09-02T04:29:44.232 DEBUG:boto:Token: None 2022-09-02T04:29:44.233 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PTw1Y7dCI7CPJWBG6A4H/OAAQvY= 2022-09-02T04:29:44.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PTw1Y7dCI7CPJWBG6A4H/OAAQvY=', 'Content-Length': '0'} 2022-09-02T04:29:44.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6bcb8b0e513c563-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.238 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.238 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.239 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.239 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.240 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.241 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.241 DEBUG:boto:Data: 2022-09-02T04:29:44.242 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.243 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.243 DEBUG:boto:Params: {} 2022-09-02T04:29:44.244 DEBUG:boto:Token: None 2022-09-02T04:29:44.244 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.245 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PTw1Y7dCI7CPJWBG6A4H/OAAQvY= 2022-09-02T04:29:44.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PTw1Y7dCI7CPJWBG6A4H/OAAQvY=', 'Content-Length': '0'} 2022-09-02T04:29:44.248 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008154e4a8d38ea268-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:44.249 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.249 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.250 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.251 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.251 DEBUG:boto:Method: GET 2022-09-02T04:29:44.252 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.252 DEBUG:boto:Data: 2022-09-02T04:29:44.253 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.254 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.254 DEBUG:boto:Params: {} 2022-09-02T04:29:44.255 DEBUG:boto:Token: None 2022-09-02T04:29:44.256 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE= 2022-09-02T04:29:44.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE=', 'Content-Length': '0'} 2022-09-02T04:29:44.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007792efe2c13d8bc-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.260 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.260 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.261 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.261 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.262 DEBUG:boto:Method: GET 2022-09-02T04:29:44.262 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.263 DEBUG:boto:Data: 2022-09-02T04:29:44.264 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.265 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.265 DEBUG:boto:Params: {} 2022-09-02T04:29:44.266 DEBUG:boto:Token: None 2022-09-02T04:29:44.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE= 2022-09-02T04:29:44.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE=', 'Content-Length': '0'} 2022-09-02T04:29:44.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c1c29e0c9940596-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.271 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.271 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.272 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.272 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.273 DEBUG:boto:Method: GET 2022-09-02T04:29:44.273 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.274 DEBUG:boto:Data: 2022-09-02T04:29:44.275 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.276 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.276 DEBUG:boto:Params: {} 2022-09-02T04:29:44.277 DEBUG:boto:Token: None 2022-09-02T04:29:44.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000942565c2f5e496de-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.282 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.283 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.284 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.284 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.285 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.285 DEBUG:boto:Method: GET 2022-09-02T04:29:44.286 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.287 DEBUG:boto:Data: 2022-09-02T04:29:44.287 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.288 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.289 DEBUG:boto:Params: {} 2022-09-02T04:29:44.289 DEBUG:boto:Token: None 2022-09-02T04:29:44.290 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ae31a88fc703fa-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.295 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.295 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.296 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.297 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.297 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.298 DEBUG:boto:Data: 2022-09-02T04:29:44.298 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.299 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.300 DEBUG:boto:Params: {} 2022-09-02T04:29:44.300 DEBUG:boto:Token: None 2022-09-02T04:29:44.301 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aZ2J7EHv0mEVC0ON8c17SMf9Yoc= 2022-09-02T04:29:44.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aZ2J7EHv0mEVC0ON8c17SMf9Yoc=', 'Content-Length': '0'} 2022-09-02T04:29:44.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad549d7f67108456-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.306 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.306 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.307 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.307 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.308 DEBUG:boto:Data: 2022-09-02T04:29:44.308 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.309 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.310 DEBUG:boto:Params: {} 2022-09-02T04:29:44.311 DEBUG:boto:Token: None 2022-09-02T04:29:44.311 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aZ2J7EHv0mEVC0ON8c17SMf9Yoc= 2022-09-02T04:29:44.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aZ2J7EHv0mEVC0ON8c17SMf9Yoc=', 'Content-Length': '0'} 2022-09-02T04:29:44.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059d5ced1e0a4a172-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:44.317 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.318 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.319 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.319 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.320 DEBUG:boto:Method: GET 2022-09-02T04:29:44.320 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.321 DEBUG:boto:Data: 2022-09-02T04:29:44.321 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.323 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.323 DEBUG:boto:Params: {} 2022-09-02T04:29:44.324 DEBUG:boto:Token: None 2022-09-02T04:29:44.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE= 2022-09-02T04:29:44.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE=', 'Content-Length': '0'} 2022-09-02T04:29:44.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f54be3acbd77f6e-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.330 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.331 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:44.331 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.332 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.332 DEBUG:boto:Method: GET 2022-09-02T04:29:44.333 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.333 DEBUG:boto:Data: 2022-09-02T04:29:44.334 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.335 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.336 DEBUG:boto:Params: {} 2022-09-02T04:29:44.337 DEBUG:boto:Token: None 2022-09-02T04:29:44.337 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:44.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE= 2022-09-02T04:29:44.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TzGgTzVvuIa2mezsfiHiqONl1QE=', 'Content-Length': '0'} 2022-09-02T04:29:44.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000064dbd679a64fe88-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.382 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.383 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.384 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.384 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.385 DEBUG:boto:Method: GET 2022-09-02T04:29:44.386 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.386 DEBUG:boto:Data: 2022-09-02T04:29:44.387 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.388 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.388 DEBUG:boto:Params: {} 2022-09-02T04:29:44.389 DEBUG:boto:Token: None 2022-09-02T04:29:44.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019c2b0611288fb1c-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.396 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.397 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.398 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.399 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.399 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.400 DEBUG:boto:Method: GET 2022-09-02T04:29:44.400 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:44.401 DEBUG:boto:Data: 2022-09-02T04:29:44.402 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.403 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.404 DEBUG:boto:Params: {} 2022-09-02T04:29:44.405 DEBUG:boto:Token: None 2022-09-02T04:29:44.406 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000932652e4a5814607-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.413 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:44.415 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.416 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.417 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.418 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.419 DEBUG:boto:Method: GET 2022-09-02T04:29:44.420 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.421 DEBUG:boto:Data: 2022-09-02T04:29:44.422 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.424 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.425 DEBUG:boto:Params: {} 2022-09-02T04:29:44.426 DEBUG:boto:Token: None 2022-09-02T04:29:44.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc= 2022-09-02T04:29:44.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc=', 'Content-Length': '0'} 2022-09-02T04:29:44.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8551313b15e3e95-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.474 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.475 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.476 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.477 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.479 DEBUG:boto:Method: GET 2022-09-02T04:29:44.479 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.480 DEBUG:boto:Data: 2022-09-02T04:29:44.480 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.481 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.482 DEBUG:boto:Params: {} 2022-09-02T04:29:44.482 DEBUG:boto:Token: None 2022-09-02T04:29:44.483 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc= 2022-09-02T04:29:44.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc=', 'Content-Length': '0'} 2022-09-02T04:29:44.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb393f0e69d47984-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.487 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.488 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.489 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.489 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.490 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.490 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.491 DEBUG:boto:Data: 2022-09-02T04:29:44.491 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.492 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.493 DEBUG:boto:Params: {} 2022-09-02T04:29:44.493 DEBUG:boto:Token: None 2022-09-02T04:29:44.494 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lH0WvAAigEF4rIeBAwcGhgoeFHs= 2022-09-02T04:29:44.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lH0WvAAigEF4rIeBAwcGhgoeFHs=', 'Content-Length': '0'} 2022-09-02T04:29:44.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb03ec9f913ad19f-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.499 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.500 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.501 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.501 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.502 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.503 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.504 DEBUG:boto:Data: 2022-09-02T04:29:44.505 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.507 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.508 DEBUG:boto:Params: {} 2022-09-02T04:29:44.509 DEBUG:boto:Token: None 2022-09-02T04:29:44.510 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lH0WvAAigEF4rIeBAwcGhgoeFHs= 2022-09-02T04:29:44.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lH0WvAAigEF4rIeBAwcGhgoeFHs=', 'Content-Length': '0'} 2022-09-02T04:29:44.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6183e308ad7937d-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:44.516 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.517 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.518 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.518 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.519 DEBUG:boto:Method: GET 2022-09-02T04:29:44.520 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.522 DEBUG:boto:Data: 2022-09-02T04:29:44.523 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.525 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.526 DEBUG:boto:Params: {} 2022-09-02T04:29:44.527 DEBUG:boto:Token: None 2022-09-02T04:29:44.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc= 2022-09-02T04:29:44.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc=', 'Content-Length': '0'} 2022-09-02T04:29:44.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a16cad68c6294c79-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.534 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.535 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.537 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.538 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.538 DEBUG:boto:Method: GET 2022-09-02T04:29:44.539 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.539 DEBUG:boto:Data: 2022-09-02T04:29:44.541 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.542 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.542 DEBUG:boto:Params: {} 2022-09-02T04:29:44.543 DEBUG:boto:Token: None 2022-09-02T04:29:44.544 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc= 2022-09-02T04:29:44.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc=', 'Content-Length': '0'} 2022-09-02T04:29:44.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000766c220590ac67cd-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.551 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.552 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.553 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.553 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.554 DEBUG:boto:Method: GET 2022-09-02T04:29:44.554 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.555 DEBUG:boto:Data: 2022-09-02T04:29:44.556 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.557 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.557 DEBUG:boto:Params: {} 2022-09-02T04:29:44.558 DEBUG:boto:Token: None 2022-09-02T04:29:44.559 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc674fbd53c8c0c-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.565 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.566 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.567 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.568 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.569 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.570 DEBUG:boto:Method: GET 2022-09-02T04:29:44.571 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.572 DEBUG:boto:Data: 2022-09-02T04:29:44.573 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.574 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.575 DEBUG:boto:Params: {} 2022-09-02T04:29:44.575 DEBUG:boto:Token: None 2022-09-02T04:29:44.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc46a140d0f25239-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.582 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.583 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.584 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.585 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.586 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.587 DEBUG:boto:Data: 2022-09-02T04:29:44.588 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.590 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.591 DEBUG:boto:Params: {} 2022-09-02T04:29:44.592 DEBUG:boto:Token: None 2022-09-02T04:29:44.593 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s574VnokEbTAF5dKQ183d3Z2mBE= 2022-09-02T04:29:44.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s574VnokEbTAF5dKQ183d3Z2mBE=', 'Content-Length': '0'} 2022-09-02T04:29:44.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e27dea6de75f869b-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.599 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.599 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.600 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.600 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.601 DEBUG:boto:Data: 2022-09-02T04:29:44.601 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.602 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.603 DEBUG:boto:Params: {} 2022-09-02T04:29:44.603 DEBUG:boto:Token: None 2022-09-02T04:29:44.604 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s574VnokEbTAF5dKQ183d3Z2mBE= 2022-09-02T04:29:44.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s574VnokEbTAF5dKQ183d3Z2mBE=', 'Content-Length': '0'} 2022-09-02T04:29:44.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009397614aa08da1f7-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:44.609 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.610 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.610 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.611 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.611 DEBUG:boto:Method: GET 2022-09-02T04:29:44.612 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.613 DEBUG:boto:Data: 2022-09-02T04:29:44.613 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.614 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.615 DEBUG:boto:Params: {} 2022-09-02T04:29:44.615 DEBUG:boto:Token: None 2022-09-02T04:29:44.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc= 2022-09-02T04:29:44.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc=', 'Content-Length': '0'} 2022-09-02T04:29:44.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a4073d271e67e83-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.621 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.621 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:44.622 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.622 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.623 DEBUG:boto:Method: GET 2022-09-02T04:29:44.623 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.624 DEBUG:boto:Data: 2022-09-02T04:29:44.624 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.626 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.626 DEBUG:boto:Params: {} 2022-09-02T04:29:44.627 DEBUG:boto:Token: None 2022-09-02T04:29:44.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:44.628 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc= 2022-09-02T04:29:44.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b6HOzei8iKaM+5rw5nrCsibPVPc=', 'Content-Length': '0'} 2022-09-02T04:29:44.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fc3e139cae1212e-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.632 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.632 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.633 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.633 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.634 DEBUG:boto:Method: GET 2022-09-02T04:29:44.634 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.635 DEBUG:boto:Data: 2022-09-02T04:29:44.636 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.637 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.637 DEBUG:boto:Params: {} 2022-09-02T04:29:44.638 DEBUG:boto:Token: None 2022-09-02T04:29:44.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c97538f4b80af34e-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.644 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.645 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.646 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.647 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.648 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.649 DEBUG:boto:Method: GET 2022-09-02T04:29:44.650 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:44.651 DEBUG:boto:Data: 2022-09-02T04:29:44.652 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.654 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.655 DEBUG:boto:Params: {} 2022-09-02T04:29:44.656 DEBUG:boto:Token: None 2022-09-02T04:29:44.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe148a2f74aa16fc-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.702 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:44.705 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.705 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.706 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.707 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.708 DEBUG:boto:Method: GET 2022-09-02T04:29:44.708 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.709 DEBUG:boto:Data: 2022-09-02T04:29:44.710 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.710 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.711 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.711 DEBUG:boto:Params: {} 2022-09-02T04:29:44.712 DEBUG:boto:Token: None 2022-09-02T04:29:44.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw= 2022-09-02T04:29:44.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw=', 'Content-Length': '0'} 2022-09-02T04:29:44.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c8036c0d91702dc-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.719 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.720 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.721 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.722 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.723 DEBUG:boto:Method: GET 2022-09-02T04:29:44.724 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.725 DEBUG:boto:Data: 2022-09-02T04:29:44.726 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.728 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.728 DEBUG:boto:Params: {} 2022-09-02T04:29:44.729 DEBUG:boto:Token: None 2022-09-02T04:29:44.730 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw= 2022-09-02T04:29:44.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw=', 'Content-Length': '0'} 2022-09-02T04:29:44.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebd4f8e526f0e53b-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.737 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.738 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.739 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.740 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.741 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.741 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.742 DEBUG:boto:Data: 2022-09-02T04:29:44.743 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.745 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.746 DEBUG:boto:Params: {} 2022-09-02T04:29:44.746 DEBUG:boto:Token: None 2022-09-02T04:29:44.747 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K6grbvnSTUyeIwccBZ13K/doL5k= 2022-09-02T04:29:44.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K6grbvnSTUyeIwccBZ13K/doL5k=', 'Content-Length': '0'} 2022-09-02T04:29:44.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008534a3c77108c258-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.751 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.752 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.752 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.753 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.753 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.754 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.755 DEBUG:boto:Data: 2022-09-02T04:29:44.755 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.757 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.757 DEBUG:boto:Params: {} 2022-09-02T04:29:44.758 DEBUG:boto:Token: None 2022-09-02T04:29:44.759 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K6grbvnSTUyeIwccBZ13K/doL5k= 2022-09-02T04:29:44.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K6grbvnSTUyeIwccBZ13K/doL5k=', 'Content-Length': '0'} 2022-09-02T04:29:44.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022be0b65cac4e2c5-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:44.763 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.764 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.764 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.765 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.766 DEBUG:boto:Method: GET 2022-09-02T04:29:44.766 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.767 DEBUG:boto:Data: 2022-09-02T04:29:44.768 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.769 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.769 DEBUG:boto:Params: {} 2022-09-02T04:29:44.770 DEBUG:boto:Token: None 2022-09-02T04:29:44.771 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw= 2022-09-02T04:29:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw=', 'Content-Length': '0'} 2022-09-02T04:29:44.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000468f91bfdd19c44-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.818 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.819 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.819 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.820 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.821 DEBUG:boto:Method: GET 2022-09-02T04:29:44.821 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.822 DEBUG:boto:Data: 2022-09-02T04:29:44.822 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.823 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.824 DEBUG:boto:Params: {} 2022-09-02T04:29:44.824 DEBUG:boto:Token: None 2022-09-02T04:29:44.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw= 2022-09-02T04:29:44.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw=', 'Content-Length': '0'} 2022-09-02T04:29:44.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ed43a10e30fccc6-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.830 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.831 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.832 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.832 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.833 DEBUG:boto:Method: GET 2022-09-02T04:29:44.833 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.834 DEBUG:boto:Data: 2022-09-02T04:29:44.835 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.836 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.836 DEBUG:boto:Params: {} 2022-09-02T04:29:44.837 DEBUG:boto:Token: None 2022-09-02T04:29:44.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b595d08aa707eefb-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.843 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.845 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.846 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.847 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.847 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.848 DEBUG:boto:Method: GET 2022-09-02T04:29:44.848 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.849 DEBUG:boto:Data: 2022-09-02T04:29:44.851 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.852 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.853 DEBUG:boto:Params: {} 2022-09-02T04:29:44.854 DEBUG:boto:Token: None 2022-09-02T04:29:44.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6174395ca66b80-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.861 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.862 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.862 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.863 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.863 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.864 DEBUG:boto:Data: 2022-09-02T04:29:44.864 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.866 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.867 DEBUG:boto:Params: {} 2022-09-02T04:29:44.868 DEBUG:boto:Token: None 2022-09-02T04:29:44.869 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RuhlyXTyr7gjWyjLi/gD4IFk1Y0= 2022-09-02T04:29:44.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RuhlyXTyr7gjWyjLi/gD4IFk1Y0=', 'Content-Length': '0'} 2022-09-02T04:29:44.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000032190e094d70741-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.875 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.875 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.876 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.876 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.877 DEBUG:boto:Data: 2022-09-02T04:29:44.878 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.879 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.880 DEBUG:boto:Params: {} 2022-09-02T04:29:44.880 DEBUG:boto:Token: None 2022-09-02T04:29:44.881 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RuhlyXTyr7gjWyjLi/gD4IFk1Y0= 2022-09-02T04:29:44.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RuhlyXTyr7gjWyjLi/gD4IFk1Y0=', 'Content-Length': '0'} 2022-09-02T04:29:44.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007044c19c22f4867e-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:44.887 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.888 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.889 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.889 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.890 DEBUG:boto:Method: GET 2022-09-02T04:29:44.890 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.891 DEBUG:boto:Data: 2022-09-02T04:29:44.891 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.894 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.895 DEBUG:boto:Params: {} 2022-09-02T04:29:44.896 DEBUG:boto:Token: None 2022-09-02T04:29:44.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw= 2022-09-02T04:29:44.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw=', 'Content-Length': '0'} 2022-09-02T04:29:44.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009039cdcf450b9c7-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.903 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.904 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:44.904 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.905 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.905 DEBUG:boto:Method: GET 2022-09-02T04:29:44.906 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.906 DEBUG:boto:Data: 2022-09-02T04:29:44.907 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.908 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.909 DEBUG:boto:Params: {} 2022-09-02T04:29:44.909 DEBUG:boto:Token: None 2022-09-02T04:29:44.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:44.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw= 2022-09-02T04:29:44.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/si6SVagQ+p+HVOX42u1vdEWTiw=', 'Content-Length': '0'} 2022-09-02T04:29:44.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a46ab8dda3757e5d-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.914 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.915 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.915 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.916 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.917 DEBUG:boto:Method: GET 2022-09-02T04:29:44.918 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.919 DEBUG:boto:Data: 2022-09-02T04:29:44.920 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:44.921 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.921 DEBUG:boto:Params: {} 2022-09-02T04:29:44.922 DEBUG:boto:Token: None 2022-09-02T04:29:44.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000642a59221fdbd8d0-00631186b8-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.930 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.931 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.931 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.932 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.933 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.934 DEBUG:boto:Method: GET 2022-09-02T04:29:44.935 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:44.936 DEBUG:boto:Data: 2022-09-02T04:29:44.937 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.939 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.940 DEBUG:boto:Params: {} 2022-09-02T04:29:44.941 DEBUG:boto:Token: None 2022-09-02T04:29:44.943 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc= 2022-09-02T04:29:44.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYrzJvGX3RLtZoiLQvmmCjr+zIc=', 'Content-Length': '0'} 2022-09-02T04:29:44.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a14f0a8e134ba50-00631186b8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.949 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-16 zones={test-zone2, test-zone1} 2022-09-02T04:29:44.952 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-16 zones={test-zone3, test-zone1} 2022-09-02T04:29:44.953 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.954 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.955 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.956 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:44.956 DEBUG:boto:Data: 2022-09-02T04:29:44.958 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:44.960 DEBUG:boto:Port: 8001 2022-09-02T04:29:44.961 DEBUG:boto:Params: {} 2022-09-02T04:29:44.962 DEBUG:boto:Token: None 2022-09-02T04:29:44.963 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n20G0OUcGsa7qC6F2lxv1HZTeCc= 2022-09-02T04:29:44.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n20G0OUcGsa7qC6F2lxv1HZTeCc=', 'Content-Length': '0'} 2022-09-02T04:29:44.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ee45d3c89b9ee71-00631186b8-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.968 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.969 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.969 DEBUG:boto:Method: HEAD 2022-09-02T04:29:44.970 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:44.970 DEBUG:boto:Data: 2022-09-02T04:29:44.971 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:44.972 DEBUG:boto:Port: 8000 2022-09-02T04:29:44.972 DEBUG:boto:Params: {} 2022-09-02T04:29:44.973 DEBUG:boto:Token: None 2022-09-02T04:29:44.974 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/ 2022-09-02T04:29:44.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n20G0OUcGsa7qC6F2lxv1HZTeCc= 2022-09-02T04:29:44.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n20G0OUcGsa7qC6F2lxv1HZTeCc=', 'Content-Length': '0'} 2022-09-02T04:29:44.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c8d2ff4405bc22d-00631186b8-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:29:44.980 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.981 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.981 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:44.982 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:44.983 DEBUG:boto:Method: GET 2022-09-02T04:29:44.983 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:44.984 DEBUG:boto:Data: 2022-09-02T04:29:44.984 DEBUG:boto:Headers: {} 2022-09-02T04:29:44.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:44.985 DEBUG:boto:Port: 8001 2022-09-02T04:29:44.986 DEBUG:boto:Params: {} 2022-09-02T04:29:44.987 DEBUG:boto:Token: None 2022-09-02T04:29:44.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:44 GMT /xljbkf-16/?versions 2022-09-02T04:29:44.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YRyF+ra/3YpOPl257d8lK0W4Ow0= 2022-09-02T04:29:44.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YRyF+ra/3YpOPl257d8lK0W4Ow0=', 'Content-Length': '0'} 2022-09-02T04:29:44.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098ccc1f7f27461f6-00631186b8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:44.993 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:44.994 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:29:44.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:29:44.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:29:44.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:29:44.996 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:29:44.997 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:44.997 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:44.998 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:44.998 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:44.999 DEBUG:boto:Method: GET 2022-09-02T04:29:44.999 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.000 DEBUG:boto:Data: 2022-09-02T04:29:45.000 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.001 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.002 DEBUG:boto:Params: {} 2022-09-02T04:29:45.002 DEBUG:boto:Token: None 2022-09-02T04:29:45.003 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/?versions 2022-09-02T04:29:45.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCT/gJ3mDZWFm8F8GNJTAd04xJM= 2022-09-02T04:29:45.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCT/gJ3mDZWFm8F8GNJTAd04xJM=', 'Content-Length': '0'} 2022-09-02T04:29:45.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004beb9903816859fb-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.008 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.008 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:29:45.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:29:45.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:29:45.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:29:45.010 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.011 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.012 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.012 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.013 DEBUG:boto:Method: GET 2022-09-02T04:29:45.013 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.014 DEBUG:boto:Data: 2022-09-02T04:29:45.014 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.015 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.016 DEBUG:boto:Params: {} 2022-09-02T04:29:45.016 DEBUG:boto:Token: None 2022-09-02T04:29:45.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/?versions 2022-09-02T04:29:45.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCT/gJ3mDZWFm8F8GNJTAd04xJM= 2022-09-02T04:29:45.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCT/gJ3mDZWFm8F8GNJTAd04xJM=', 'Content-Length': '0'} 2022-09-02T04:29:45.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be7031fc761ac1d-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.022 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.023 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.023 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.024 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.024 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.025 DEBUG:boto:Method: GET 2022-09-02T04:29:45.025 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:45.026 DEBUG:boto:Data: 2022-09-02T04:29:45.026 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.027 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.028 DEBUG:boto:Params: {} 2022-09-02T04:29:45.028 DEBUG:boto:Token: None 2022-09-02T04:29:45.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/?versions 2022-09-02T04:29:45.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCT/gJ3mDZWFm8F8GNJTAd04xJM= 2022-09-02T04:29:45.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCT/gJ3mDZWFm8F8GNJTAd04xJM=', 'Content-Length': '0'} 2022-09-02T04:29:45.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bbd0c8a0210fe76-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.034 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:45.035 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.036 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.036 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.037 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.037 DEBUG:boto:Method: GET 2022-09-02T04:29:45.038 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.038 DEBUG:boto:Data: 2022-09-02T04:29:45.039 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.040 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.041 DEBUG:boto:Params: {} 2022-09-02T04:29:45.041 DEBUG:boto:Token: None 2022-09-02T04:29:45.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk= 2022-09-02T04:29:45.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk=', 'Content-Length': '0'} 2022-09-02T04:29:45.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef54a02d7c715d47-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.047 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.047 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.048 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.048 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.049 DEBUG:boto:Method: GET 2022-09-02T04:29:45.049 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.050 DEBUG:boto:Data: 2022-09-02T04:29:45.051 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.052 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.052 DEBUG:boto:Params: {} 2022-09-02T04:29:45.053 DEBUG:boto:Token: None 2022-09-02T04:29:45.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk= 2022-09-02T04:29:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk=', 'Content-Length': '0'} 2022-09-02T04:29:45.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000834e0b17643e4bdf-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.058 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.059 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.059 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.060 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.061 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.062 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.063 DEBUG:boto:Data: 2022-09-02T04:29:45.064 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.065 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.066 DEBUG:boto:Params: {} 2022-09-02T04:29:45.067 DEBUG:boto:Token: None 2022-09-02T04:29:45.068 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.069 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eUiDLpN9nldHADOTEuA0dlrERKI= 2022-09-02T04:29:45.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eUiDLpN9nldHADOTEuA0dlrERKI=', 'Content-Length': '0'} 2022-09-02T04:29:45.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000ca5299bac08e811-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.074 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.075 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.076 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.076 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.077 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.077 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.078 DEBUG:boto:Data: 2022-09-02T04:29:45.078 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.079 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.080 DEBUG:boto:Params: {} 2022-09-02T04:29:45.080 DEBUG:boto:Token: None 2022-09-02T04:29:45.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eUiDLpN9nldHADOTEuA0dlrERKI= 2022-09-02T04:29:45.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eUiDLpN9nldHADOTEuA0dlrERKI=', 'Content-Length': '0'} 2022-09-02T04:29:45.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073a2995ea17d6fab-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:45.086 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.087 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.087 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.088 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.088 DEBUG:boto:Method: GET 2022-09-02T04:29:45.089 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.089 DEBUG:boto:Data: 2022-09-02T04:29:45.090 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.091 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.091 DEBUG:boto:Params: {} 2022-09-02T04:29:45.092 DEBUG:boto:Token: None 2022-09-02T04:29:45.093 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk= 2022-09-02T04:29:45.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk=', 'Content-Length': '0'} 2022-09-02T04:29:45.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cf5204f6e65bd7c-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.097 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.097 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.098 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.098 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.099 DEBUG:boto:Method: GET 2022-09-02T04:29:45.099 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.100 DEBUG:boto:Data: 2022-09-02T04:29:45.100 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.101 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.102 DEBUG:boto:Params: {} 2022-09-02T04:29:45.103 DEBUG:boto:Token: None 2022-09-02T04:29:45.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk= 2022-09-02T04:29:45.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk=', 'Content-Length': '0'} 2022-09-02T04:29:45.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba290a3a501bb08d-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.107 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.108 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.108 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.109 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.110 DEBUG:boto:Method: GET 2022-09-02T04:29:45.110 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.111 DEBUG:boto:Data: 2022-09-02T04:29:45.111 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.112 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.113 DEBUG:boto:Params: {} 2022-09-02T04:29:45.113 DEBUG:boto:Token: None 2022-09-02T04:29:45.114 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f89e0aeb3d497bba-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.119 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.120 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.120 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.121 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.121 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.122 DEBUG:boto:Method: GET 2022-09-02T04:29:45.122 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.123 DEBUG:boto:Data: 2022-09-02T04:29:45.123 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.124 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.125 DEBUG:boto:Params: {} 2022-09-02T04:29:45.126 DEBUG:boto:Token: None 2022-09-02T04:29:45.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061895aaffb1df32a-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.174 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.175 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.176 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.176 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.177 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:29:45.177 DEBUG:boto:Data: 2022-09-02T04:29:45.178 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.179 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.179 DEBUG:boto:Params: {} 2022-09-02T04:29:45.180 DEBUG:boto:Token: None 2022-09-02T04:29:45.180 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy 2022-09-02T04:29:45.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dYNC3G2GhOasRK6q6bBTOXsRGD8= 2022-09-02T04:29:45.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dYNC3G2GhOasRK6q6bBTOXsRGD8=', 'Content-Length': '0'} 2022-09-02T04:29:45.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd1a40b318b66424-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.185 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.186 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.186 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.187 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:29:45.187 DEBUG:boto:Data: 2022-09-02T04:29:45.188 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.189 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.189 DEBUG:boto:Params: {} 2022-09-02T04:29:45.190 DEBUG:boto:Token: None 2022-09-02T04:29:45.190 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy 2022-09-02T04:29:45.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dYNC3G2GhOasRK6q6bBTOXsRGD8= 2022-09-02T04:29:45.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dYNC3G2GhOasRK6q6bBTOXsRGD8=', 'Content-Length': '0'} 2022-09-02T04:29:45.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e7d0d7c0e0d5ee2-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:45.197 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.198 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.198 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.199 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.200 DEBUG:boto:Method: GET 2022-09-02T04:29:45.200 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.201 DEBUG:boto:Data: 2022-09-02T04:29:45.201 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.203 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.203 DEBUG:boto:Params: {} 2022-09-02T04:29:45.204 DEBUG:boto:Token: None 2022-09-02T04:29:45.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk= 2022-09-02T04:29:45.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk=', 'Content-Length': '0'} 2022-09-02T04:29:45.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a72e7e261e00a1a8-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.209 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:45.210 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:45.210 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.211 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.211 DEBUG:boto:Method: GET 2022-09-02T04:29:45.212 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.212 DEBUG:boto:Data: 2022-09-02T04:29:45.213 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.215 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.215 DEBUG:boto:Params: {} 2022-09-02T04:29:45.216 DEBUG:boto:Token: None 2022-09-02T04:29:45.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:45.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk= 2022-09-02T04:29:45.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LeHk/ah2fuRYrDvtDHKVtnQUopk=', 'Content-Length': '0'} 2022-09-02T04:29:45.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5177af2dcb6aa2f-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.221 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.222 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.222 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.223 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.223 DEBUG:boto:Method: GET 2022-09-02T04:29:45.224 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.224 DEBUG:boto:Data: 2022-09-02T04:29:45.225 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.226 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.226 DEBUG:boto:Params: {} 2022-09-02T04:29:45.227 DEBUG:boto:Token: None 2022-09-02T04:29:45.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007570e8f1b9d8eca9-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.232 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.233 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.234 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.234 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.235 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.235 DEBUG:boto:Method: GET 2022-09-02T04:29:45.236 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:45.236 DEBUG:boto:Data: 2022-09-02T04:29:45.237 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.238 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.239 DEBUG:boto:Params: {} 2022-09-02T04:29:45.240 DEBUG:boto:Token: None 2022-09-02T04:29:45.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c2078553f244b7-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.245 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:45.247 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.248 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.248 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.249 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.249 DEBUG:boto:Method: GET 2022-09-02T04:29:45.250 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.251 DEBUG:boto:Data: 2022-09-02T04:29:45.251 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.252 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.253 DEBUG:boto:Params: {} 2022-09-02T04:29:45.253 DEBUG:boto:Token: None 2022-09-02T04:29:45.254 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8= 2022-09-02T04:29:45.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8=', 'Content-Length': '0'} 2022-09-02T04:29:45.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009564a03ead18f3fe-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.258 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.259 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.259 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.260 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.260 DEBUG:boto:Method: GET 2022-09-02T04:29:45.261 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.261 DEBUG:boto:Data: 2022-09-02T04:29:45.262 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.263 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.264 DEBUG:boto:Params: {} 2022-09-02T04:29:45.264 DEBUG:boto:Token: None 2022-09-02T04:29:45.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8= 2022-09-02T04:29:45.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8=', 'Content-Length': '0'} 2022-09-02T04:29:45.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6ec4c964b81efd3-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.270 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.270 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.271 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.271 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.272 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.272 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.273 DEBUG:boto:Data: 2022-09-02T04:29:45.273 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.274 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.275 DEBUG:boto:Params: {} 2022-09-02T04:29:45.275 DEBUG:boto:Token: None 2022-09-02T04:29:45.276 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e+gyw+wWgFfvQ1mW836NpyySBN4= 2022-09-02T04:29:45.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e+gyw+wWgFfvQ1mW836NpyySBN4=', 'Content-Length': '0'} 2022-09-02T04:29:45.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ad6e98e7ba534ef-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.280 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.280 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.281 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.281 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.282 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.282 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.283 DEBUG:boto:Data: 2022-09-02T04:29:45.283 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.284 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.285 DEBUG:boto:Params: {} 2022-09-02T04:29:45.285 DEBUG:boto:Token: None 2022-09-02T04:29:45.286 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e+gyw+wWgFfvQ1mW836NpyySBN4= 2022-09-02T04:29:45.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e+gyw+wWgFfvQ1mW836NpyySBN4=', 'Content-Length': '0'} 2022-09-02T04:29:45.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007629166aae5afd60-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:45.290 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.291 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.291 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.292 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.293 DEBUG:boto:Method: GET 2022-09-02T04:29:45.293 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.294 DEBUG:boto:Data: 2022-09-02T04:29:45.294 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.295 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.296 DEBUG:boto:Params: {} 2022-09-02T04:29:45.296 DEBUG:boto:Token: None 2022-09-02T04:29:45.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8= 2022-09-02T04:29:45.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8=', 'Content-Length': '0'} 2022-09-02T04:29:45.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee9f804f18c74abc-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.342 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.344 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.345 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.346 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.347 DEBUG:boto:Method: GET 2022-09-02T04:29:45.348 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.348 DEBUG:boto:Data: 2022-09-02T04:29:45.349 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.352 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.353 DEBUG:boto:Params: {} 2022-09-02T04:29:45.354 DEBUG:boto:Token: None 2022-09-02T04:29:45.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8= 2022-09-02T04:29:45.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8=', 'Content-Length': '0'} 2022-09-02T04:29:45.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c355cb7d5df121fb-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.360 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.361 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.362 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.362 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.363 DEBUG:boto:Method: GET 2022-09-02T04:29:45.364 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.365 DEBUG:boto:Data: 2022-09-02T04:29:45.366 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.367 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.368 DEBUG:boto:Params: {} 2022-09-02T04:29:45.369 DEBUG:boto:Token: None 2022-09-02T04:29:45.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9a834a55dc85ce-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.376 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.378 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.379 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.379 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.380 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.381 DEBUG:boto:Method: GET 2022-09-02T04:29:45.381 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.382 DEBUG:boto:Data: 2022-09-02T04:29:45.382 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.383 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.384 DEBUG:boto:Params: {} 2022-09-02T04:29:45.384 DEBUG:boto:Token: None 2022-09-02T04:29:45.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002467b5ed9813c7ca-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.391 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.392 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.393 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.394 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.395 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.396 DEBUG:boto:Data: 2022-09-02T04:29:45.397 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.399 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.400 DEBUG:boto:Params: {} 2022-09-02T04:29:45.401 DEBUG:boto:Token: None 2022-09-02T04:29:45.402 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aeMpcZuyO1uJbz2TQMNEI1HqO3A= 2022-09-02T04:29:45.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aeMpcZuyO1uJbz2TQMNEI1HqO3A=', 'Content-Length': '0'} 2022-09-02T04:29:45.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c2686e99a2e853b-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.409 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.409 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.410 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.410 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.411 DEBUG:boto:Data: 2022-09-02T04:29:45.411 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.412 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.413 DEBUG:boto:Params: {} 2022-09-02T04:29:45.413 DEBUG:boto:Token: None 2022-09-02T04:29:45.414 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aeMpcZuyO1uJbz2TQMNEI1HqO3A= 2022-09-02T04:29:45.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aeMpcZuyO1uJbz2TQMNEI1HqO3A=', 'Content-Length': '0'} 2022-09-02T04:29:45.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000144be4c1b3ffbc8c-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:45.419 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.419 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.420 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.420 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.421 DEBUG:boto:Method: GET 2022-09-02T04:29:45.421 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.422 DEBUG:boto:Data: 2022-09-02T04:29:45.423 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.424 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.424 DEBUG:boto:Params: {} 2022-09-02T04:29:45.425 DEBUG:boto:Token: None 2022-09-02T04:29:45.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8= 2022-09-02T04:29:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8=', 'Content-Length': '0'} 2022-09-02T04:29:45.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a672985ac4932b-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.474 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.475 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:45.476 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.477 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.478 DEBUG:boto:Method: GET 2022-09-02T04:29:45.479 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.480 DEBUG:boto:Data: 2022-09-02T04:29:45.481 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.482 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.483 DEBUG:boto:Params: {} 2022-09-02T04:29:45.484 DEBUG:boto:Token: None 2022-09-02T04:29:45.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:45.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8= 2022-09-02T04:29:45.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wMgH6/rSl/PTsEtndbiBc/ulfX8=', 'Content-Length': '0'} 2022-09-02T04:29:45.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b84d83dc70e4bd79-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.491 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.492 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.493 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.494 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.495 DEBUG:boto:Method: GET 2022-09-02T04:29:45.496 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.497 DEBUG:boto:Data: 2022-09-02T04:29:45.498 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.500 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.501 DEBUG:boto:Params: {} 2022-09-02T04:29:45.502 DEBUG:boto:Token: None 2022-09-02T04:29:45.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3480450ad0a3c72-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.509 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.510 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.511 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.512 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.512 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.513 DEBUG:boto:Method: GET 2022-09-02T04:29:45.513 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:45.514 DEBUG:boto:Data: 2022-09-02T04:29:45.514 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.515 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.516 DEBUG:boto:Params: {} 2022-09-02T04:29:45.517 DEBUG:boto:Token: None 2022-09-02T04:29:45.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c31590fa6d8b642-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.523 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:45.525 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.526 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.527 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.528 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.529 DEBUG:boto:Method: GET 2022-09-02T04:29:45.530 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.531 DEBUG:boto:Data: 2022-09-02T04:29:45.532 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.534 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.536 DEBUG:boto:Params: {} 2022-09-02T04:29:45.537 DEBUG:boto:Token: None 2022-09-02T04:29:45.537 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs= 2022-09-02T04:29:45.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs=', 'Content-Length': '0'} 2022-09-02T04:29:45.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d1e2d4d3be8d4f4-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.542 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.542 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.543 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.545 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.546 DEBUG:boto:Method: GET 2022-09-02T04:29:45.547 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.548 DEBUG:boto:Data: 2022-09-02T04:29:45.549 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.551 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.552 DEBUG:boto:Params: {} 2022-09-02T04:29:45.553 DEBUG:boto:Token: None 2022-09-02T04:29:45.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs= 2022-09-02T04:29:45.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs=', 'Content-Length': '0'} 2022-09-02T04:29:45.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000894aebb08645cae3-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.559 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.560 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.560 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.561 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.562 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.563 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.564 DEBUG:boto:Data: 2022-09-02T04:29:45.565 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.566 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.567 DEBUG:boto:Params: {} 2022-09-02T04:29:45.568 DEBUG:boto:Token: None 2022-09-02T04:29:45.569 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6zK8JFwsfak5EpAuoZdZMTKKRJM= 2022-09-02T04:29:45.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6zK8JFwsfak5EpAuoZdZMTKKRJM=', 'Content-Length': '0'} 2022-09-02T04:29:45.573 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003690897e3c243555-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.574 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.574 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.575 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.575 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.576 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.576 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.577 DEBUG:boto:Data: 2022-09-02T04:29:45.577 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.579 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.579 DEBUG:boto:Params: {} 2022-09-02T04:29:45.580 DEBUG:boto:Token: None 2022-09-02T04:29:45.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6zK8JFwsfak5EpAuoZdZMTKKRJM= 2022-09-02T04:29:45.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6zK8JFwsfak5EpAuoZdZMTKKRJM=', 'Content-Length': '0'} 2022-09-02T04:29:45.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002234d3d6ebaca5e2-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:45.587 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.588 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.589 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.590 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.591 DEBUG:boto:Method: GET 2022-09-02T04:29:45.592 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.593 DEBUG:boto:Data: 2022-09-02T04:29:45.594 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.596 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.597 DEBUG:boto:Params: {} 2022-09-02T04:29:45.598 DEBUG:boto:Token: None 2022-09-02T04:29:45.599 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs= 2022-09-02T04:29:45.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs=', 'Content-Length': '0'} 2022-09-02T04:29:45.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d86cc3513766988b-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.605 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.606 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.607 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.608 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.609 DEBUG:boto:Method: GET 2022-09-02T04:29:45.610 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.610 DEBUG:boto:Data: 2022-09-02T04:29:45.611 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.612 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.613 DEBUG:boto:Params: {} 2022-09-02T04:29:45.613 DEBUG:boto:Token: None 2022-09-02T04:29:45.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs= 2022-09-02T04:29:45.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs=', 'Content-Length': '0'} 2022-09-02T04:29:45.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034d0ea48ac478bed-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.658 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.659 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.659 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.660 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.660 DEBUG:boto:Method: GET 2022-09-02T04:29:45.661 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.662 DEBUG:boto:Data: 2022-09-02T04:29:45.662 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.663 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.664 DEBUG:boto:Params: {} 2022-09-02T04:29:45.665 DEBUG:boto:Token: None 2022-09-02T04:29:45.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191e6dd1d4fe901e-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.671 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.673 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.674 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.675 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.676 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.676 DEBUG:boto:Method: GET 2022-09-02T04:29:45.677 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.678 DEBUG:boto:Data: 2022-09-02T04:29:45.678 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.680 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.681 DEBUG:boto:Params: {} 2022-09-02T04:29:45.682 DEBUG:boto:Token: None 2022-09-02T04:29:45.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000084347c1512acae9-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.688 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.689 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.690 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.690 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.691 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.691 DEBUG:boto:Data: 2022-09-02T04:29:45.692 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.693 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.694 DEBUG:boto:Params: {} 2022-09-02T04:29:45.694 DEBUG:boto:Token: None 2022-09-02T04:29:45.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bGUb9m1kAwowE+Hc6M/CMSjnbZk= 2022-09-02T04:29:45.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bGUb9m1kAwowE+Hc6M/CMSjnbZk=', 'Content-Length': '0'} 2022-09-02T04:29:45.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba0c15de76b1f9d7-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.701 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.703 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.704 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.705 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.705 DEBUG:boto:Data: 2022-09-02T04:29:45.706 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.709 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.710 DEBUG:boto:Params: {} 2022-09-02T04:29:45.711 DEBUG:boto:Token: None 2022-09-02T04:29:45.711 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bGUb9m1kAwowE+Hc6M/CMSjnbZk= 2022-09-02T04:29:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bGUb9m1kAwowE+Hc6M/CMSjnbZk=', 'Content-Length': '0'} 2022-09-02T04:29:45.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048159a27e4225819-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:45.718 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.719 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.720 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.722 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.723 DEBUG:boto:Method: GET 2022-09-02T04:29:45.724 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.725 DEBUG:boto:Data: 2022-09-02T04:29:45.727 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.729 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.729 DEBUG:boto:Params: {} 2022-09-02T04:29:45.731 DEBUG:boto:Token: None 2022-09-02T04:29:45.732 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs= 2022-09-02T04:29:45.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs=', 'Content-Length': '0'} 2022-09-02T04:29:45.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038e948be53a9403d-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.782 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.783 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:45.784 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.785 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.786 DEBUG:boto:Method: GET 2022-09-02T04:29:45.786 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.787 DEBUG:boto:Data: 2022-09-02T04:29:45.787 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.788 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.789 DEBUG:boto:Params: {} 2022-09-02T04:29:45.790 DEBUG:boto:Token: None 2022-09-02T04:29:45.790 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:45.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs= 2022-09-02T04:29:45.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+tLcb1FnxLAvKcriyWYBrhZHRs=', 'Content-Length': '0'} 2022-09-02T04:29:45.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099a49d3326093487-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.795 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.796 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.796 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.797 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.797 DEBUG:boto:Method: GET 2022-09-02T04:29:45.798 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.799 DEBUG:boto:Data: 2022-09-02T04:29:45.799 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.800 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.801 DEBUG:boto:Params: {} 2022-09-02T04:29:45.801 DEBUG:boto:Token: None 2022-09-02T04:29:45.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000223f963e7043ce58-00631186b9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.807 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.808 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:45.809 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:45.809 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.810 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.810 DEBUG:boto:Method: GET 2022-09-02T04:29:45.811 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:45.811 DEBUG:boto:Data: 2022-09-02T04:29:45.812 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.813 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.814 DEBUG:boto:Params: {} 2022-09-02T04:29:45.815 DEBUG:boto:Token: None 2022-09-02T04:29:45.816 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/ 2022-09-02T04:29:45.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts= 2022-09-02T04:29:45.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+r3/KRrsggZjF82l5GLPOH4Euts=', 'Content-Length': '0'} 2022-09-02T04:29:45.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a44886d899fa79c8-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.823 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:45.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:45.825 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.826 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.827 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.827 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.828 DEBUG:boto:Method: GET 2022-09-02T04:29:45.828 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.829 DEBUG:boto:Data: 2022-09-02T04:29:45.829 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.830 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.832 DEBUG:boto:Params: {} 2022-09-02T04:29:45.833 DEBUG:boto:Token: None 2022-09-02T04:29:45.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk= 2022-09-02T04:29:45.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk=', 'Content-Length': '0'} 2022-09-02T04:29:45.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c1966e4feff3b41-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.841 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.841 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.842 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.842 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.843 DEBUG:boto:Method: GET 2022-09-02T04:29:45.844 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.844 DEBUG:boto:Data: 2022-09-02T04:29:45.845 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.846 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.846 DEBUG:boto:Params: {} 2022-09-02T04:29:45.847 DEBUG:boto:Token: None 2022-09-02T04:29:45.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk= 2022-09-02T04:29:45.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk=', 'Content-Length': '0'} 2022-09-02T04:29:45.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b20a86db5a1011c3-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.898 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.900 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.901 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.902 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.903 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.904 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.905 DEBUG:boto:Data: 2022-09-02T04:29:45.906 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.909 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.909 DEBUG:boto:Params: {} 2022-09-02T04:29:45.910 DEBUG:boto:Token: None 2022-09-02T04:29:45.911 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/XfJENbfGxykESltwWLpZCHKwh4= 2022-09-02T04:29:45.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/XfJENbfGxykESltwWLpZCHKwh4=', 'Content-Length': '0'} 2022-09-02T04:29:45.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcb36b715bdbf8a6-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.915 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.916 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.917 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.917 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.918 DEBUG:boto:Method: HEAD 2022-09-02T04:29:45.919 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.920 DEBUG:boto:Data: 2022-09-02T04:29:45.922 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.923 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.924 DEBUG:boto:Params: {} 2022-09-02T04:29:45.926 DEBUG:boto:Token: None 2022-09-02T04:29:45.927 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/XfJENbfGxykESltwWLpZCHKwh4= 2022-09-02T04:29:45.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/XfJENbfGxykESltwWLpZCHKwh4=', 'Content-Length': '0'} 2022-09-02T04:29:45.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dff97597e54c3c2a-00631186b9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:45.933 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.933 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.934 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.934 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.935 DEBUG:boto:Method: GET 2022-09-02T04:29:45.935 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.936 DEBUG:boto:Data: 2022-09-02T04:29:45.936 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:45.937 DEBUG:boto:Port: 8001 2022-09-02T04:29:45.938 DEBUG:boto:Params: {} 2022-09-02T04:29:45.939 DEBUG:boto:Token: None 2022-09-02T04:29:45.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk= 2022-09-02T04:29:45.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk=', 'Content-Length': '0'} 2022-09-02T04:29:45.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ac13eac1ef0cda5-00631186b9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:45.990 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.991 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:45.991 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.992 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.992 DEBUG:boto:Method: GET 2022-09-02T04:29:45.993 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:45.994 DEBUG:boto:Data: 2022-09-02T04:29:45.996 DEBUG:boto:Headers: {} 2022-09-02T04:29:45.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:45.997 DEBUG:boto:Port: 8000 2022-09-02T04:29:45.998 DEBUG:boto:Params: {} 2022-09-02T04:29:45.999 DEBUG:boto:Token: None 2022-09-02T04:29:45.999 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:45 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk= 2022-09-02T04:29:46.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/+saX5vznt+MN/Vuk8rwfjc5Hvk=', 'Content-Length': '0'} 2022-09-02T04:29:46.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000320d3cafbde25183-00631186ba-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.005 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.006 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.006 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.007 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.007 DEBUG:boto:Method: GET 2022-09-02T04:29:46.008 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.009 DEBUG:boto:Data: 2022-09-02T04:29:46.009 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.010 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.011 DEBUG:boto:Params: {} 2022-09-02T04:29:46.012 DEBUG:boto:Token: None 2022-09-02T04:29:46.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aa0af3a0c1f0a61-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.017 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.018 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.019 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.020 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.020 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.021 DEBUG:boto:Method: GET 2022-09-02T04:29:46.022 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.023 DEBUG:boto:Data: 2022-09-02T04:29:46.024 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:46.025 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.026 DEBUG:boto:Params: {} 2022-09-02T04:29:46.027 DEBUG:boto:Token: None 2022-09-02T04:29:46.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a21791c6ea756a9-00631186ba-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.033 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.034 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.035 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.035 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.036 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.037 DEBUG:boto:Data: 2022-09-02T04:29:46.037 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.038 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.039 DEBUG:boto:Params: {} 2022-09-02T04:29:46.039 DEBUG:boto:Token: None 2022-09-02T04:29:46.040 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vFtvjxQ3d9WCH2c9GbuErZLQuU= 2022-09-02T04:29:46.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vFtvjxQ3d9WCH2c9GbuErZLQuU=', 'Content-Length': '0'} 2022-09-02T04:29:46.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad51ef2d0715e8fa-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.045 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.045 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.046 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.047 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.047 DEBUG:boto:Data: 2022-09-02T04:29:46.048 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:46.049 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.050 DEBUG:boto:Params: {} 2022-09-02T04:29:46.051 DEBUG:boto:Token: None 2022-09-02T04:29:46.051 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vFtvjxQ3d9WCH2c9GbuErZLQuU= 2022-09-02T04:29:46.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vFtvjxQ3d9WCH2c9GbuErZLQuU=', 'Content-Length': '0'} 2022-09-02T04:29:46.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef3d8f7e283cbbf4-00631186ba-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:46.057 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.057 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.058 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.058 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.059 DEBUG:boto:Method: GET 2022-09-02T04:29:46.060 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.060 DEBUG:boto:Data: 2022-09-02T04:29:46.061 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.062 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.063 DEBUG:boto:Params: {} 2022-09-02T04:29:46.063 DEBUG:boto:Token: None 2022-09-02T04:29:46.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pVzMmTuqW8eTkQNev08E8g9eK5M= 2022-09-02T04:29:46.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pVzMmTuqW8eTkQNev08E8g9eK5M=', 'Content-Length': '0'} 2022-09-02T04:29:46.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b226ee3b6b48074-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.068 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.069 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:46.070 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.070 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.072 DEBUG:boto:Method: GET 2022-09-02T04:29:46.072 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.073 DEBUG:boto:Data: 2022-09-02T04:29:46.073 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:46.074 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.075 DEBUG:boto:Params: {} 2022-09-02T04:29:46.076 DEBUG:boto:Token: None 2022-09-02T04:29:46.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:46.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pVzMmTuqW8eTkQNev08E8g9eK5M= 2022-09-02T04:29:46.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pVzMmTuqW8eTkQNev08E8g9eK5M=', 'Content-Length': '0'} 2022-09-02T04:29:46.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa018b070e9a9799-00631186ba-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.080 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.081 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.082 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.082 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.083 DEBUG:boto:Method: GET 2022-09-02T04:29:46.083 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.084 DEBUG:boto:Data: 2022-09-02T04:29:46.084 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.085 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.086 DEBUG:boto:Params: {} 2022-09-02T04:29:46.087 DEBUG:boto:Token: None 2022-09-02T04:29:46.087 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09312c12d1ee0af-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.092 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.093 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.094 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.094 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.095 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.096 DEBUG:boto:Method: GET 2022-09-02T04:29:46.096 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:46.097 DEBUG:boto:Data: 2022-09-02T04:29:46.097 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:29:46.098 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.099 DEBUG:boto:Params: {} 2022-09-02T04:29:46.100 DEBUG:boto:Token: None 2022-09-02T04:29:46.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000574b44ddc000e24f-00631186ba-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.105 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-16 zones={test-zone3, test-zone1} 2022-09-02T04:29:46.106 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-16 zones={test-zone3, test-zone2} 2022-09-02T04:29:46.107 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.107 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.108 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.109 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:46.109 DEBUG:boto:Data: 2022-09-02T04:29:46.110 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.111 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.111 DEBUG:boto:Params: {} 2022-09-02T04:29:46.112 DEBUG:boto:Token: None 2022-09-02T04:29:46.112 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vBSZH+pAhbANrZBjTgN3q4ymPUs= 2022-09-02T04:29:46.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vBSZH+pAhbANrZBjTgN3q4ymPUs=', 'Content-Length': '0'} 2022-09-02T04:29:46.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000694c3c70f2f8ffea-00631186ba-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.117 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.117 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.118 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.118 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:29:46.119 DEBUG:boto:Data: 2022-09-02T04:29:46.119 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.120 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.121 DEBUG:boto:Params: {} 2022-09-02T04:29:46.121 DEBUG:boto:Token: None 2022-09-02T04:29:46.122 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vBSZH+pAhbANrZBjTgN3q4ymPUs= 2022-09-02T04:29:46.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vBSZH+pAhbANrZBjTgN3q4ymPUs=', 'Content-Length': '0'} 2022-09-02T04:29:46.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dc0cb39c1bad387-00631186ba-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:29:46.128 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.128 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.129 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.130 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.130 DEBUG:boto:Method: GET 2022-09-02T04:29:46.131 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.132 DEBUG:boto:Data: 2022-09-02T04:29:46.133 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.134 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.134 DEBUG:boto:Params: {} 2022-09-02T04:29:46.135 DEBUG:boto:Token: None 2022-09-02T04:29:46.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/?versions 2022-09-02T04:29:46.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI= 2022-09-02T04:29:46.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI=', 'Content-Length': '0'} 2022-09-02T04:29:46.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad86aaa812785c0-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.141 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.142 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:29:46.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:29:46.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:29:46.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:29:46.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:29:46.145 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.146 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.146 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.147 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.147 DEBUG:boto:Method: GET 2022-09-02T04:29:46.162 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.163 DEBUG:boto:Data: 2022-09-02T04:29:46.164 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.165 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.166 DEBUG:boto:Params: {} 2022-09-02T04:29:46.166 DEBUG:boto:Token: None 2022-09-02T04:29:46.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/?versions 2022-09-02T04:29:46.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI= 2022-09-02T04:29:46.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI=', 'Content-Length': '0'} 2022-09-02T04:29:46.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000881d3e254f433cda-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.175 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.176 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:29:46.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:29:46.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:29:46.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:29:46.181 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.182 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.183 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.189 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.190 DEBUG:boto:Method: GET 2022-09-02T04:29:46.191 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.192 DEBUG:boto:Data: 2022-09-02T04:29:46.193 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.195 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.196 DEBUG:boto:Params: {} 2022-09-02T04:29:46.197 DEBUG:boto:Token: None 2022-09-02T04:29:46.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/?versions 2022-09-02T04:29:46.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI= 2022-09-02T04:29:46.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI=', 'Content-Length': '0'} 2022-09-02T04:29:46.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a911315d2b3d967d-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.218 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.221 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.223 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.224 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.225 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.226 DEBUG:boto:Method: GET 2022-09-02T04:29:46.227 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:29:46.229 DEBUG:boto:Data: 2022-09-02T04:29:46.230 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.233 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.236 DEBUG:boto:Params: {} 2022-09-02T04:29:46.237 DEBUG:boto:Token: None 2022-09-02T04:29:46.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/?versions 2022-09-02T04:29:46.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI= 2022-09-02T04:29:46.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0KDt05DKE2hn/oRv+hlXb8mTiI=', 'Content-Length': '0'} 2022-09-02T04:29:46.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005509ca6097653f37-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.245 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:46.247 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.248 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.249 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.249 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.250 DEBUG:boto:Method: GET 2022-09-02T04:29:46.251 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.251 DEBUG:boto:Data: 2022-09-02T04:29:46.252 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.253 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.254 DEBUG:boto:Params: {} 2022-09-02T04:29:46.254 DEBUG:boto:Token: None 2022-09-02T04:29:46.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw= 2022-09-02T04:29:46.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw=', 'Content-Length': '0'} 2022-09-02T04:29:46.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055d0340d58d35305-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.260 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.260 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.261 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.261 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.262 DEBUG:boto:Method: GET 2022-09-02T04:29:46.262 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.263 DEBUG:boto:Data: 2022-09-02T04:29:46.263 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.264 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.265 DEBUG:boto:Params: {} 2022-09-02T04:29:46.267 DEBUG:boto:Token: None 2022-09-02T04:29:46.267 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw= 2022-09-02T04:29:46.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw=', 'Content-Length': '0'} 2022-09-02T04:29:46.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087e249759302b363-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.272 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.273 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.274 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.275 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.276 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.277 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.278 DEBUG:boto:Data: 2022-09-02T04:29:46.279 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.281 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.282 DEBUG:boto:Params: {} 2022-09-02T04:29:46.283 DEBUG:boto:Token: None 2022-09-02T04:29:46.284 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LgHCr0XrVZAfbxhh9PxNk+WzGHQ= 2022-09-02T04:29:46.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LgHCr0XrVZAfbxhh9PxNk+WzGHQ=', 'Content-Length': '0'} 2022-09-02T04:29:46.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d592311b0b666e5c-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.289 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.289 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.290 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.290 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.291 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.291 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.292 DEBUG:boto:Data: 2022-09-02T04:29:46.292 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.293 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.294 DEBUG:boto:Params: {} 2022-09-02T04:29:46.295 DEBUG:boto:Token: None 2022-09-02T04:29:46.295 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LgHCr0XrVZAfbxhh9PxNk+WzGHQ= 2022-09-02T04:29:46.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LgHCr0XrVZAfbxhh9PxNk+WzGHQ=', 'Content-Length': '0'} 2022-09-02T04:29:46.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba749bda9178a91a-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:46.300 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.301 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.302 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.302 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.303 DEBUG:boto:Method: GET 2022-09-02T04:29:46.304 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.304 DEBUG:boto:Data: 2022-09-02T04:29:46.305 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.306 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.307 DEBUG:boto:Params: {} 2022-09-02T04:29:46.307 DEBUG:boto:Token: None 2022-09-02T04:29:46.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw= 2022-09-02T04:29:46.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw=', 'Content-Length': '0'} 2022-09-02T04:29:46.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef7a1d2cb63dfc23-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.312 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.313 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.313 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.314 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.314 DEBUG:boto:Method: GET 2022-09-02T04:29:46.316 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.316 DEBUG:boto:Data: 2022-09-02T04:29:46.317 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.318 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.318 DEBUG:boto:Params: {} 2022-09-02T04:29:46.319 DEBUG:boto:Token: None 2022-09-02T04:29:46.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw= 2022-09-02T04:29:46.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw=', 'Content-Length': '0'} 2022-09-02T04:29:46.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c00fc882982e12b7-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.324 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.325 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.326 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.326 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.327 DEBUG:boto:Method: GET 2022-09-02T04:29:46.327 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.328 DEBUG:boto:Data: 2022-09-02T04:29:46.328 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.329 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.330 DEBUG:boto:Params: {} 2022-09-02T04:29:46.331 DEBUG:boto:Token: None 2022-09-02T04:29:46.332 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77fd6f775f89a0c-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.380 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.381 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.382 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.383 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.388 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.388 DEBUG:boto:Method: GET 2022-09-02T04:29:46.389 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.389 DEBUG:boto:Data: 2022-09-02T04:29:46.390 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.391 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.392 DEBUG:boto:Params: {} 2022-09-02T04:29:46.392 DEBUG:boto:Token: None 2022-09-02T04:29:46.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bf865d3d1e99bae-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.399 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.400 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.400 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.401 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.401 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:29:46.402 DEBUG:boto:Data: 2022-09-02T04:29:46.402 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.403 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.404 DEBUG:boto:Params: {} 2022-09-02T04:29:46.404 DEBUG:boto:Token: None 2022-09-02T04:29:46.405 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy 2022-09-02T04:29:46.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7//L+PBuFTVTWSVjtoZuSzmBBaI= 2022-09-02T04:29:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7//L+PBuFTVTWSVjtoZuSzmBBaI=', 'Content-Length': '0'} 2022-09-02T04:29:46.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000493aac27310f09fb-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.410 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.411 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.411 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.412 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:29:46.412 DEBUG:boto:Data: 2022-09-02T04:29:46.413 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.415 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.415 DEBUG:boto:Params: {} 2022-09-02T04:29:46.416 DEBUG:boto:Token: None 2022-09-02T04:29:46.417 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy 2022-09-02T04:29:46.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7//L+PBuFTVTWSVjtoZuSzmBBaI= 2022-09-02T04:29:46.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7//L+PBuFTVTWSVjtoZuSzmBBaI=', 'Content-Length': '0'} 2022-09-02T04:29:46.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e3c7367e487bf632-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:29:46.425 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.425 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.426 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.426 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.427 DEBUG:boto:Method: GET 2022-09-02T04:29:46.428 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.428 DEBUG:boto:Data: 2022-09-02T04:29:46.429 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.430 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.431 DEBUG:boto:Params: {} 2022-09-02T04:29:46.431 DEBUG:boto:Token: None 2022-09-02T04:29:46.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw= 2022-09-02T04:29:46.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw=', 'Content-Length': '0'} 2022-09-02T04:29:46.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000269e23c2960388a1-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.437 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:29:46.438 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:29:46.438 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.439 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.440 DEBUG:boto:Method: GET 2022-09-02T04:29:46.441 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.441 DEBUG:boto:Data: 2022-09-02T04:29:46.442 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.443 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.444 DEBUG:boto:Params: {} 2022-09-02T04:29:46.445 DEBUG:boto:Token: None 2022-09-02T04:29:46.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:29:46.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw= 2022-09-02T04:29:46.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nfSTLVlcs+01auqdzqiusNWrxmw=', 'Content-Length': '0'} 2022-09-02T04:29:46.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009da821549baa8ed4-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.450 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.450 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.451 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.451 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.452 DEBUG:boto:Method: GET 2022-09-02T04:29:46.452 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.453 DEBUG:boto:Data: 2022-09-02T04:29:46.453 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.455 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.455 DEBUG:boto:Params: {} 2022-09-02T04:29:46.456 DEBUG:boto:Token: None 2022-09-02T04:29:46.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa94b9756a1d112-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.462 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.464 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.465 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.466 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.467 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.468 DEBUG:boto:Method: GET 2022-09-02T04:29:46.469 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:29:46.470 DEBUG:boto:Data: 2022-09-02T04:29:46.471 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.472 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.473 DEBUG:boto:Params: {} 2022-09-02T04:29:46.473 DEBUG:boto:Token: None 2022-09-02T04:29:46.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad878df4871228b-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.480 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:46.481 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.482 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.483 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.484 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.484 DEBUG:boto:Method: GET 2022-09-02T04:29:46.485 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.486 DEBUG:boto:Data: 2022-09-02T04:29:46.487 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.488 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.489 DEBUG:boto:Params: {} 2022-09-02T04:29:46.490 DEBUG:boto:Token: None 2022-09-02T04:29:46.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8= 2022-09-02T04:29:46.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8=', 'Content-Length': '0'} 2022-09-02T04:29:46.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000575b4069f2583cd7-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.495 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.495 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.496 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.497 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.498 DEBUG:boto:Method: GET 2022-09-02T04:29:46.498 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.499 DEBUG:boto:Data: 2022-09-02T04:29:46.500 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.501 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.501 DEBUG:boto:Params: {} 2022-09-02T04:29:46.502 DEBUG:boto:Token: None 2022-09-02T04:29:46.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8= 2022-09-02T04:29:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8=', 'Content-Length': '0'} 2022-09-02T04:29:46.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbea0f3f377906dc-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.550 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.551 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.552 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.553 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.553 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.555 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.555 DEBUG:boto:Data: 2022-09-02T04:29:46.556 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.557 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.557 DEBUG:boto:Params: {} 2022-09-02T04:29:46.558 DEBUG:boto:Token: None 2022-09-02T04:29:46.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kh1VvfiyZxzbbHxLRY/2qlU/Q+g= 2022-09-02T04:29:46.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kh1VvfiyZxzbbHxLRY/2qlU/Q+g=', 'Content-Length': '0'} 2022-09-02T04:29:46.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029c69c33b5582f88-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.563 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.564 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.565 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.565 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.566 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.567 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.567 DEBUG:boto:Data: 2022-09-02T04:29:46.568 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.569 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.569 DEBUG:boto:Params: {} 2022-09-02T04:29:46.571 DEBUG:boto:Token: None 2022-09-02T04:29:46.572 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kh1VvfiyZxzbbHxLRY/2qlU/Q+g= 2022-09-02T04:29:46.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kh1VvfiyZxzbbHxLRY/2qlU/Q+g=', 'Content-Length': '0'} 2022-09-02T04:29:46.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000317f0e615316c2c0-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:46.577 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.577 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.578 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.579 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.579 DEBUG:boto:Method: GET 2022-09-02T04:29:46.580 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.580 DEBUG:boto:Data: 2022-09-02T04:29:46.581 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.582 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.583 DEBUG:boto:Params: {} 2022-09-02T04:29:46.583 DEBUG:boto:Token: None 2022-09-02T04:29:46.584 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8= 2022-09-02T04:29:46.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8=', 'Content-Length': '0'} 2022-09-02T04:29:46.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008483ffb5da039b2-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.588 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.589 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.589 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.590 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.590 DEBUG:boto:Method: GET 2022-09-02T04:29:46.591 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.592 DEBUG:boto:Data: 2022-09-02T04:29:46.592 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.593 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.594 DEBUG:boto:Params: {} 2022-09-02T04:29:46.595 DEBUG:boto:Token: None 2022-09-02T04:29:46.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8= 2022-09-02T04:29:46.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8=', 'Content-Length': '0'} 2022-09-02T04:29:46.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a87e349b498fb71-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.600 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.600 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.601 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.601 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.602 DEBUG:boto:Method: GET 2022-09-02T04:29:46.602 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.603 DEBUG:boto:Data: 2022-09-02T04:29:46.604 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.611 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.611 DEBUG:boto:Params: {} 2022-09-02T04:29:46.612 DEBUG:boto:Token: None 2022-09-02T04:29:46.613 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb7da34dfd0f3c0-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.621 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.623 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.624 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.625 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.626 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.627 DEBUG:boto:Method: GET 2022-09-02T04:29:46.628 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.628 DEBUG:boto:Data: 2022-09-02T04:29:46.629 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.630 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.631 DEBUG:boto:Params: {} 2022-09-02T04:29:46.631 DEBUG:boto:Token: None 2022-09-02T04:29:46.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029915eebe798e227-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.638 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.639 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.640 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.641 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.641 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.642 DEBUG:boto:Data: 2022-09-02T04:29:46.642 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.644 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.644 DEBUG:boto:Params: {} 2022-09-02T04:29:46.645 DEBUG:boto:Token: None 2022-09-02T04:29:46.645 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TZCnr7PNJYddPyX8v6gh9t4F5FY= 2022-09-02T04:29:46.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TZCnr7PNJYddPyX8v6gh9t4F5FY=', 'Content-Length': '0'} 2022-09-02T04:29:46.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1b364812a4bd3c0-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.650 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.651 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.651 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.652 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.652 DEBUG:boto:Data: 2022-09-02T04:29:46.653 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.654 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.654 DEBUG:boto:Params: {} 2022-09-02T04:29:46.655 DEBUG:boto:Token: None 2022-09-02T04:29:46.656 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TZCnr7PNJYddPyX8v6gh9t4F5FY= 2022-09-02T04:29:46.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TZCnr7PNJYddPyX8v6gh9t4F5FY=', 'Content-Length': '0'} 2022-09-02T04:29:46.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e81fde7e6814cc4d-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:29:46.661 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.662 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.662 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.663 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.663 DEBUG:boto:Method: GET 2022-09-02T04:29:46.664 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.664 DEBUG:boto:Data: 2022-09-02T04:29:46.665 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.667 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.667 DEBUG:boto:Params: {} 2022-09-02T04:29:46.668 DEBUG:boto:Token: None 2022-09-02T04:29:46.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8= 2022-09-02T04:29:46.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8=', 'Content-Length': '0'} 2022-09-02T04:29:46.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008992bdd4db91373a-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.675 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.676 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:29:46.677 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.678 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.679 DEBUG:boto:Method: GET 2022-09-02T04:29:46.680 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.681 DEBUG:boto:Data: 2022-09-02T04:29:46.681 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.683 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.684 DEBUG:boto:Params: {} 2022-09-02T04:29:46.685 DEBUG:boto:Token: None 2022-09-02T04:29:46.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:29:46.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8= 2022-09-02T04:29:46.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9PNweajUTMg32h5Qsxn0+shaZ8=', 'Content-Length': '0'} 2022-09-02T04:29:46.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcdc43e00e6af36e-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.693 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.694 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.695 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.695 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.696 DEBUG:boto:Method: GET 2022-09-02T04:29:46.697 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.697 DEBUG:boto:Data: 2022-09-02T04:29:46.698 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.699 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.699 DEBUG:boto:Params: {} 2022-09-02T04:29:46.700 DEBUG:boto:Token: None 2022-09-02T04:29:46.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e07f2a89f82ae8-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.706 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.706 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.707 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.708 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.708 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.709 DEBUG:boto:Method: GET 2022-09-02T04:29:46.709 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:29:46.710 DEBUG:boto:Data: 2022-09-02T04:29:46.710 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.711 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.712 DEBUG:boto:Params: {} 2022-09-02T04:29:46.713 DEBUG:boto:Token: None 2022-09-02T04:29:46.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000add07e416ee2c2e9-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.718 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:46.720 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.720 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.721 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.721 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.722 DEBUG:boto:Method: GET 2022-09-02T04:29:46.722 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.723 DEBUG:boto:Data: 2022-09-02T04:29:46.723 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.724 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.725 DEBUG:boto:Params: {} 2022-09-02T04:29:46.725 DEBUG:boto:Token: None 2022-09-02T04:29:46.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs= 2022-09-02T04:29:46.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs=', 'Content-Length': '0'} 2022-09-02T04:29:46.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082b921ce85d2c2d2-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.731 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.732 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.733 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.734 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.735 DEBUG:boto:Method: GET 2022-09-02T04:29:46.736 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.737 DEBUG:boto:Data: 2022-09-02T04:29:46.738 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.740 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.741 DEBUG:boto:Params: {} 2022-09-02T04:29:46.743 DEBUG:boto:Token: None 2022-09-02T04:29:46.744 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.745 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs= 2022-09-02T04:29:46.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs=', 'Content-Length': '0'} 2022-09-02T04:29:46.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c6c31397477fc1b-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.752 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.753 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.754 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.754 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.755 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.755 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.756 DEBUG:boto:Data: 2022-09-02T04:29:46.756 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.757 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.758 DEBUG:boto:Params: {} 2022-09-02T04:29:46.759 DEBUG:boto:Token: None 2022-09-02T04:29:46.760 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C40TyYIAL8KXLCfWugxZqPtWuDo= 2022-09-02T04:29:46.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C40TyYIAL8KXLCfWugxZqPtWuDo=', 'Content-Length': '0'} 2022-09-02T04:29:46.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0f65c8498913e18-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.768 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.769 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.770 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.771 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.772 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.773 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.774 DEBUG:boto:Data: 2022-09-02T04:29:46.775 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.777 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.778 DEBUG:boto:Params: {} 2022-09-02T04:29:46.779 DEBUG:boto:Token: None 2022-09-02T04:29:46.780 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C40TyYIAL8KXLCfWugxZqPtWuDo= 2022-09-02T04:29:46.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C40TyYIAL8KXLCfWugxZqPtWuDo=', 'Content-Length': '0'} 2022-09-02T04:29:46.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000541f23a0403ccac7-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:46.787 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.787 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.788 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.789 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.790 DEBUG:boto:Method: GET 2022-09-02T04:29:46.791 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.792 DEBUG:boto:Data: 2022-09-02T04:29:46.793 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.795 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.796 DEBUG:boto:Params: {} 2022-09-02T04:29:46.797 DEBUG:boto:Token: None 2022-09-02T04:29:46.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs= 2022-09-02T04:29:46.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs=', 'Content-Length': '0'} 2022-09-02T04:29:46.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009523c8ddec240eb1-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.805 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.806 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.807 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.808 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.809 DEBUG:boto:Method: GET 2022-09-02T04:29:46.809 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.810 DEBUG:boto:Data: 2022-09-02T04:29:46.810 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.811 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.812 DEBUG:boto:Params: {} 2022-09-02T04:29:46.813 DEBUG:boto:Token: None 2022-09-02T04:29:46.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs= 2022-09-02T04:29:46.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs=', 'Content-Length': '0'} 2022-09-02T04:29:46.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e7fe370625c099b-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.862 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.863 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.863 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.865 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.866 DEBUG:boto:Method: GET 2022-09-02T04:29:46.867 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.868 DEBUG:boto:Data: 2022-09-02T04:29:46.869 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.871 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.872 DEBUG:boto:Params: {} 2022-09-02T04:29:46.873 DEBUG:boto:Token: None 2022-09-02T04:29:46.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004046d19cd1bcf543-00631186ba-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.880 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.881 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.881 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.882 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.883 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.883 DEBUG:boto:Method: GET 2022-09-02T04:29:46.884 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.884 DEBUG:boto:Data: 2022-09-02T04:29:46.885 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.886 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.887 DEBUG:boto:Params: {} 2022-09-02T04:29:46.887 DEBUG:boto:Token: None 2022-09-02T04:29:46.888 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/ 2022-09-02T04:29:46.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y= 2022-09-02T04:29:46.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZBm1amSaJw/tNlZhfo/x1MAl6Y=', 'Content-Length': '0'} 2022-09-02T04:29:46.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000218af9ead00fe2fc-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.893 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:46.894 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.894 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.895 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.896 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.896 DEBUG:boto:Data: 2022-09-02T04:29:46.897 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.898 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.898 DEBUG:boto:Params: {} 2022-09-02T04:29:46.899 DEBUG:boto:Token: None 2022-09-02T04:29:46.900 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xRn1OTosA6clakItqwmPBlDjktA= 2022-09-02T04:29:46.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xRn1OTosA6clakItqwmPBlDjktA=', 'Content-Length': '0'} 2022-09-02T04:29:46.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000ef678e4731dba7-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.904 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.905 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.905 DEBUG:boto:Method: HEAD 2022-09-02T04:29:46.906 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.906 DEBUG:boto:Data: 2022-09-02T04:29:46.907 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.908 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.909 DEBUG:boto:Params: {} 2022-09-02T04:29:46.909 DEBUG:boto:Token: None 2022-09-02T04:29:46.910 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xRn1OTosA6clakItqwmPBlDjktA= 2022-09-02T04:29:46.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xRn1OTosA6clakItqwmPBlDjktA=', 'Content-Length': '0'} 2022-09-02T04:29:46.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c96b1b90082c21a-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:29:46.917 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.918 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.918 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.920 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.921 DEBUG:boto:Method: GET 2022-09-02T04:29:46.922 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.922 DEBUG:boto:Data: 2022-09-02T04:29:46.923 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:46.925 DEBUG:boto:Port: 8001 2022-09-02T04:29:46.926 DEBUG:boto:Params: {} 2022-09-02T04:29:46.927 DEBUG:boto:Token: None 2022-09-02T04:29:46.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs= 2022-09-02T04:29:46.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs=', 'Content-Length': '0'} 2022-09-02T04:29:46.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a38f7f8db5c2fc5c-00631186ba-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.974 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.975 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:29:46.976 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.977 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.978 DEBUG:boto:Method: GET 2022-09-02T04:29:46.979 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.980 DEBUG:boto:Data: 2022-09-02T04:29:46.981 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:46.983 DEBUG:boto:Port: 8000 2022-09-02T04:29:46.984 DEBUG:boto:Params: {} 2022-09-02T04:29:46.985 DEBUG:boto:Token: None 2022-09-02T04:29:46.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:46 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:29:46.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs= 2022-09-02T04:29:46.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRkzQ24thcxLo/HItPNaXMbC1fs=', 'Content-Length': '0'} 2022-09-02T04:29:46.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008346918e8ee450e3-00631186ba-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:46.993 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:46.994 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:46.995 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.996 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.996 DEBUG:boto:Method: GET 2022-09-02T04:29:46.997 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:46.998 DEBUG:boto:Data: 2022-09-02T04:29:46.999 DEBUG:boto:Headers: {} 2022-09-02T04:29:46.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.000 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.001 DEBUG:boto:Params: {} 2022-09-02T04:29:47.002 DEBUG:boto:Token: None 2022-09-02T04:29:47.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/ 2022-09-02T04:29:47.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA= 2022-09-02T04:29:47.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA=', 'Content-Length': '0'} 2022-09-02T04:29:47.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343388b575db1dce-00631186bb-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.010 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:47.011 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:47.012 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:47.012 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:47.013 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:47.013 DEBUG:boto:Method: GET 2022-09-02T04:29:47.014 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:29:47.014 DEBUG:boto:Data: 2022-09-02T04:29:47.015 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.016 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.016 DEBUG:boto:Params: {} 2022-09-02T04:29:47.017 DEBUG:boto:Token: None 2022-09-02T04:29:47.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/ 2022-09-02T04:29:47.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA= 2022-09-02T04:29:47.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA=', 'Content-Length': '0'} 2022-09-02T04:29:47.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab8a7c9673cbc426-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.024 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:47.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:47.027 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.027 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.028 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.028 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.029 DEBUG:boto:Method: GET 2022-09-02T04:29:47.029 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.030 DEBUG:boto:Data: 2022-09-02T04:29:47.030 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.031 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.032 DEBUG:boto:Params: {} 2022-09-02T04:29:47.033 DEBUG:boto:Token: None 2022-09-02T04:29:47.033 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.034 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4= 2022-09-02T04:29:47.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4=', 'Content-Length': '0'} 2022-09-02T04:29:47.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc31751cd53d1352-00631186bb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.038 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.038 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.039 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.040 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.040 DEBUG:boto:Method: GET 2022-09-02T04:29:47.041 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.041 DEBUG:boto:Data: 2022-09-02T04:29:47.042 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.043 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.043 DEBUG:boto:Params: {} 2022-09-02T04:29:47.044 DEBUG:boto:Token: None 2022-09-02T04:29:47.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4= 2022-09-02T04:29:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4=', 'Content-Length': '0'} 2022-09-02T04:29:47.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d12e9be15146c0a-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.095 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.096 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.097 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.098 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.099 DEBUG:boto:Method: HEAD 2022-09-02T04:29:47.100 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.101 DEBUG:boto:Data: 2022-09-02T04:29:47.102 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.104 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.105 DEBUG:boto:Params: {} 2022-09-02T04:29:47.106 DEBUG:boto:Token: None 2022-09-02T04:29:47.107 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:foYL3ubKIyfBf1pUteVkGw2BzRU= 2022-09-02T04:29:47.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:foYL3ubKIyfBf1pUteVkGw2BzRU=', 'Content-Length': '0'} 2022-09-02T04:29:47.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063dec9755f7ec980-00631186bb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.111 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.112 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.112 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.115 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.116 DEBUG:boto:Method: HEAD 2022-09-02T04:29:47.116 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.117 DEBUG:boto:Data: 2022-09-02T04:29:47.117 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.118 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.119 DEBUG:boto:Params: {} 2022-09-02T04:29:47.120 DEBUG:boto:Token: None 2022-09-02T04:29:47.121 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:foYL3ubKIyfBf1pUteVkGw2BzRU= 2022-09-02T04:29:47.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:foYL3ubKIyfBf1pUteVkGw2BzRU=', 'Content-Length': '0'} 2022-09-02T04:29:47.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042681329f3e51134-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:47.127 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.127 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.128 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.129 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.129 DEBUG:boto:Method: GET 2022-09-02T04:29:47.130 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.131 DEBUG:boto:Data: 2022-09-02T04:29:47.132 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.134 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.134 DEBUG:boto:Params: {} 2022-09-02T04:29:47.135 DEBUG:boto:Token: None 2022-09-02T04:29:47.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4= 2022-09-02T04:29:47.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4=', 'Content-Length': '0'} 2022-09-02T04:29:47.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002691d83247cabf4f-00631186bb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.142 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.143 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.144 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.146 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.147 DEBUG:boto:Method: GET 2022-09-02T04:29:47.148 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.149 DEBUG:boto:Data: 2022-09-02T04:29:47.149 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.150 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.151 DEBUG:boto:Params: {} 2022-09-02T04:29:47.151 DEBUG:boto:Token: None 2022-09-02T04:29:47.152 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4= 2022-09-02T04:29:47.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4=', 'Content-Length': '0'} 2022-09-02T04:29:47.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bec7c3a3dede4a49-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.159 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:47.159 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:47.160 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.160 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.161 DEBUG:boto:Method: GET 2022-09-02T04:29:47.161 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.162 DEBUG:boto:Data: 2022-09-02T04:29:47.163 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.164 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.164 DEBUG:boto:Params: {} 2022-09-02T04:29:47.165 DEBUG:boto:Token: None 2022-09-02T04:29:47.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/ 2022-09-02T04:29:47.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA= 2022-09-02T04:29:47.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA=', 'Content-Length': '0'} 2022-09-02T04:29:47.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006185ee71b9bcad95-00631186bb-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.179 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:47.180 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:47.181 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:47.182 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.183 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.184 DEBUG:boto:Method: GET 2022-09-02T04:29:47.185 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.186 DEBUG:boto:Data: 2022-09-02T04:29:47.187 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.189 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.190 DEBUG:boto:Params: {} 2022-09-02T04:29:47.191 DEBUG:boto:Token: None 2022-09-02T04:29:47.192 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/ 2022-09-02T04:29:47.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA= 2022-09-02T04:29:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA=', 'Content-Length': '0'} 2022-09-02T04:29:47.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a408411aa3b6169a-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.199 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:47.200 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.200 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.201 DEBUG:boto:Method: HEAD 2022-09-02T04:29:47.201 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.202 DEBUG:boto:Data: 2022-09-02T04:29:47.203 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.204 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.204 DEBUG:boto:Params: {} 2022-09-02T04:29:47.205 DEBUG:boto:Token: None 2022-09-02T04:29:47.205 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HdexMKmXkMWrCGn9kICODTw/ApE= 2022-09-02T04:29:47.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HdexMKmXkMWrCGn9kICODTw/ApE=', 'Content-Length': '0'} 2022-09-02T04:29:47.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070829d3fdc3af026-00631186bb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.210 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.210 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.211 DEBUG:boto:Method: HEAD 2022-09-02T04:29:47.211 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.212 DEBUG:boto:Data: 2022-09-02T04:29:47.212 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.213 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.214 DEBUG:boto:Params: {} 2022-09-02T04:29:47.215 DEBUG:boto:Token: None 2022-09-02T04:29:47.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HdexMKmXkMWrCGn9kICODTw/ApE= 2022-09-02T04:29:47.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HdexMKmXkMWrCGn9kICODTw/ApE=', 'Content-Length': '0'} 2022-09-02T04:29:47.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061646f318bf024f3-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:29:47.221 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.221 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.223 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.223 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.224 DEBUG:boto:Method: GET 2022-09-02T04:29:47.225 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.225 DEBUG:boto:Data: 2022-09-02T04:29:47.226 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.227 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.228 DEBUG:boto:Params: {} 2022-09-02T04:29:47.229 DEBUG:boto:Token: None 2022-09-02T04:29:47.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4= 2022-09-02T04:29:47.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4=', 'Content-Length': '0'} 2022-09-02T04:29:47.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005113da669880eee3-00631186bb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.234 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.235 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:29:47.235 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.236 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.236 DEBUG:boto:Method: GET 2022-09-02T04:29:47.237 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.237 DEBUG:boto:Data: 2022-09-02T04:29:47.238 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.239 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.240 DEBUG:boto:Params: {} 2022-09-02T04:29:47.241 DEBUG:boto:Token: None 2022-09-02T04:29:47.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:29:47.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4= 2022-09-02T04:29:47.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NBululIJ3KE9npWhzUthu0Rt6q4=', 'Content-Length': '0'} 2022-09-02T04:29:47.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5882450c3f30bdf-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.294 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:47.295 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:47.297 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.298 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.298 DEBUG:boto:Method: GET 2022-09-02T04:29:47.299 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.300 DEBUG:boto:Data: 2022-09-02T04:29:47.300 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:29:47.302 DEBUG:boto:Port: 8001 2022-09-02T04:29:47.302 DEBUG:boto:Params: {} 2022-09-02T04:29:47.303 DEBUG:boto:Token: None 2022-09-02T04:29:47.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/ 2022-09-02T04:29:47.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA= 2022-09-02T04:29:47.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA=', 'Content-Length': '0'} 2022-09-02T04:29:47.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2564d43167f58aa-00631186bb-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.310 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:47.311 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:29:47.312 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:29:47.312 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.313 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.314 DEBUG:boto:Method: GET 2022-09-02T04:29:47.315 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:29:47.315 DEBUG:boto:Data: 2022-09-02T04:29:47.317 DEBUG:boto:Headers: {} 2022-09-02T04:29:47.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:29:47.319 DEBUG:boto:Port: 8000 2022-09-02T04:29:47.319 DEBUG:boto:Params: {} 2022-09-02T04:29:47.320 DEBUG:boto:Token: None 2022-09-02T04:29:47.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:29:47 GMT /xljbkf-16/ 2022-09-02T04:29:47.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA= 2022-09-02T04:29:47.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:29:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khOQ4BULiD5RTk789OkrEoTMlwA=', 'Content-Length': '0'} 2022-09-02T04:29:47.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000427ccd759537cf42-00631186bb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:29:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:29:47.325 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:29:47.326 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-16 zones={test-zone3, test-zone2} 2022-09-02T04:29:47.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-17 2022-09-02T04:29:47.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:47.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:29:47.328 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:29:47.392 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:29:47.393 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:29:47.453 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:47.452+0000 7f5a305b6600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:47.454 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000006.17.9, , , , , ] 2022-09-02T04:29:47.454 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:29:52.454 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:52.453+0000 7f5a305b6600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:52.455 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000006.17.9, , , , , ] 2022-09-02T04:29:52.456 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:29:57.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:29:57.455+0000 7f5a305b6600 1 waiting for incremental sync to catch up: 2022-09-02T04:29:57.456 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000006.17.9, , , , , ] 2022-09-02T04:29:57.456 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.457 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:02.456+0000 7f5a305b6600 1 bucket sync caught up with source: 2022-09-02T04:30:02.458 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.458 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.459 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:02.456+0000 7f5a305b6600 0 bucket checkpoint complete 2022-09-02T04:30:02.475 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-17 2022-09-02T04:30:02.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.478 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:02.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:02.579 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:02.578+0000 7f51fe4ed600 1 bucket sync caught up with source: 2022-09-02T04:30:02.579 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.580 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:02.578+0000 7f51fe4ed600 0 bucket checkpoint complete 2022-09-02T04:30:02.593 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-17 2022-09-02T04:30:02.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.595 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:02.659 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:02.660 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:02.709 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:02.705+0000 7ff2370a5600 1 bucket sync caught up with source: 2022-09-02T04:30:02.710 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.711 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.26.5, , 00000000003.7.9, , 00000000007.19.9, , , , , ] 2022-09-02T04:30:02.712 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:02.705+0000 7ff2370a5600 0 bucket checkpoint complete 2022-09-02T04:30:02.727 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-17 2022-09-02T04:30:02.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.731 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:02.789 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:02.790 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:02.837 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:02.837+0000 7f51be8b6600 1 bucket sync caught up with source: 2022-09-02T04:30:02.838 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.26.9, , 00000000010.29.5, , 00000000008.48.9, , , , , ] 2022-09-02T04:30:02.838 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.26.9, , 00000000010.29.5, , 00000000008.48.9, , , , , ] 2022-09-02T04:30:02.839 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:02.837+0000 7f51be8b6600 0 bucket checkpoint complete 2022-09-02T04:30:02.855 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-17 2022-09-02T04:30:02.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.857 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:02.913 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:02.914 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:02.949 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:02.948+0000 7f4600893600 1 bucket sync caught up with source: 2022-09-02T04:30:02.950 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.26.9, , 00000000010.29.5, , 00000000008.48.9, , , , , ] 2022-09-02T04:30:02.950 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.26.9, , 00000000010.29.5, , 00000000008.48.9, , , , , ] 2022-09-02T04:30:02.951 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:02.948+0000 7f4600893600 0 bucket checkpoint complete 2022-09-02T04:30:02.962 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-17 2022-09-02T04:30:02.963 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-17 2022-09-02T04:30:02.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:02.965 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:03.021 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:03.022 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:03.056+0000 7f800c8aa600 1 bucket sync caught up with source: 2022-09-02T04:30:03.057 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.19.9, , 00000000003.10.9, , 00000000014.59.5, , , , , ] 2022-09-02T04:30:03.058 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.19.9, , 00000000003.10.9, , 00000000014.59.5, , , , , ] 2022-09-02T04:30:03.058 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:03.056+0000 7f800c8aa600 0 bucket checkpoint complete 2022-09-02T04:30:03.068 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-17 2022-09-02T04:30:03.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:03.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:03.070 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:03.130 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:03.130 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:03.188 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:03.188+0000 7f0f62dfb600 1 bucket sync caught up with source: 2022-09-02T04:30:03.189 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.19.9, , 00000000003.10.9, , 00000000014.59.5, , , , , ] 2022-09-02T04:30:03.189 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.19.9, , 00000000003.10.9, , 00000000014.59.5, , , , , ] 2022-09-02T04:30:03.189 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:03.188+0000 7f0f62dfb600 0 bucket checkpoint complete 2022-09-02T04:30:03.201 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-17 2022-09-02T04:30:03.202 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-17 zones={test-zone2, test-zone1} 2022-09-02T04:30:03.202 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.203 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.204 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.205 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:03.205 DEBUG:boto:Data: 2022-09-02T04:30:03.206 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.207 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.207 DEBUG:boto:Params: {} 2022-09-02T04:30:03.208 DEBUG:boto:Token: None 2022-09-02T04:30:03.208 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hZ/F8SFBnPz/j/hYyr/R00J5UpY= 2022-09-02T04:30:03.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hZ/F8SFBnPz/j/hYyr/R00J5UpY=', 'Content-Length': '0'} 2022-09-02T04:30:03.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1caf56ec04582a5-00631186cb-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.213 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.214 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.214 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.215 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:03.215 DEBUG:boto:Data: 2022-09-02T04:30:03.216 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.217 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.217 DEBUG:boto:Params: {} 2022-09-02T04:30:03.218 DEBUG:boto:Token: None 2022-09-02T04:30:03.219 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hZ/F8SFBnPz/j/hYyr/R00J5UpY= 2022-09-02T04:30:03.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hZ/F8SFBnPz/j/hYyr/R00J5UpY=', 'Content-Length': '0'} 2022-09-02T04:30:03.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c36ac762210e988-00631186cb-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.223 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:30:03.224 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.224 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.225 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.225 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.226 DEBUG:boto:Method: GET 2022-09-02T04:30:03.226 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.227 DEBUG:boto:Data: 2022-09-02T04:30:03.227 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.228 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.229 DEBUG:boto:Params: {} 2022-09-02T04:30:03.230 DEBUG:boto:Token: None 2022-09-02T04:30:03.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/?versions 2022-09-02T04:30:03.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0= 2022-09-02T04:30:03.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0=', 'Content-Length': '0'} 2022-09-02T04:30:03.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c391aa8e12655023-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.235 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.236 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:03.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:03.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:03.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:03.238 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:30:03.239 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.240 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.240 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.241 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.241 DEBUG:boto:Method: GET 2022-09-02T04:30:03.242 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.242 DEBUG:boto:Data: 2022-09-02T04:30:03.243 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.244 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.245 DEBUG:boto:Params: {} 2022-09-02T04:30:03.245 DEBUG:boto:Token: None 2022-09-02T04:30:03.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/?versions 2022-09-02T04:30:03.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0= 2022-09-02T04:30:03.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0=', 'Content-Length': '0'} 2022-09-02T04:30:03.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d107bccbe1dc5dde-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.253 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:03.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:03.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:03.257 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.257 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.258 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.258 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.259 DEBUG:boto:Method: GET 2022-09-02T04:30:03.259 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.260 DEBUG:boto:Data: 2022-09-02T04:30:03.261 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.262 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.262 DEBUG:boto:Params: {} 2022-09-02T04:30:03.263 DEBUG:boto:Token: None 2022-09-02T04:30:03.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/?versions 2022-09-02T04:30:03.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0= 2022-09-02T04:30:03.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0=', 'Content-Length': '0'} 2022-09-02T04:30:03.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d3e62756098183-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.268 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.269 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.269 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.270 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.271 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.271 DEBUG:boto:Method: GET 2022-09-02T04:30:03.272 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:03.273 DEBUG:boto:Data: 2022-09-02T04:30:03.274 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.276 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.277 DEBUG:boto:Params: {} 2022-09-02T04:30:03.278 DEBUG:boto:Token: None 2022-09-02T04:30:03.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/?versions 2022-09-02T04:30:03.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0= 2022-09-02T04:30:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NkPcSi2Y+9yq6gqRlTO0ARRcR0=', 'Content-Length': '0'} 2022-09-02T04:30:03.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ce2a7357552f2e-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.284 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:03.286 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.287 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.287 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.288 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.289 DEBUG:boto:Method: GET 2022-09-02T04:30:03.289 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.290 DEBUG:boto:Data: 2022-09-02T04:30:03.290 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.291 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.292 DEBUG:boto:Params: {} 2022-09-02T04:30:03.293 DEBUG:boto:Token: None 2022-09-02T04:30:03.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE= 2022-09-02T04:30:03.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE=', 'Content-Length': '0'} 2022-09-02T04:30:03.297 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e03c56ddb20254c-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.298 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.299 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.299 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.300 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.300 DEBUG:boto:Method: GET 2022-09-02T04:30:03.301 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.301 DEBUG:boto:Data: 2022-09-02T04:30:03.302 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.303 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.304 DEBUG:boto:Params: {} 2022-09-02T04:30:03.304 DEBUG:boto:Token: None 2022-09-02T04:30:03.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE= 2022-09-02T04:30:03.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE=', 'Content-Length': '0'} 2022-09-02T04:30:03.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e6df343ef446e1b-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.309 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.309 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.310 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.310 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.311 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.311 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.312 DEBUG:boto:Data: 2022-09-02T04:30:03.312 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.313 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.314 DEBUG:boto:Params: {} 2022-09-02T04:30:03.315 DEBUG:boto:Token: None 2022-09-02T04:30:03.316 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LyDXpDj3OjyV5Ko7jbIoPpcj6SA= 2022-09-02T04:30:03.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LyDXpDj3OjyV5Ko7jbIoPpcj6SA=', 'Content-Length': '0'} 2022-09-02T04:30:03.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7f48162cae6e017-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.322 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.323 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.324 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.325 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.326 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.327 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.328 DEBUG:boto:Data: 2022-09-02T04:30:03.329 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.330 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.331 DEBUG:boto:Params: {} 2022-09-02T04:30:03.332 DEBUG:boto:Token: None 2022-09-02T04:30:03.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LyDXpDj3OjyV5Ko7jbIoPpcj6SA= 2022-09-02T04:30:03.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LyDXpDj3OjyV5Ko7jbIoPpcj6SA=', 'Content-Length': '0'} 2022-09-02T04:30:03.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003982c11b889da2bc-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:03.341 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.342 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.343 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.344 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.345 DEBUG:boto:Method: GET 2022-09-02T04:30:03.346 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.346 DEBUG:boto:Data: 2022-09-02T04:30:03.347 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.349 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.350 DEBUG:boto:Params: {} 2022-09-02T04:30:03.352 DEBUG:boto:Token: None 2022-09-02T04:30:03.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE= 2022-09-02T04:30:03.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE=', 'Content-Length': '0'} 2022-09-02T04:30:03.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d9e9edadb24ae65-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.358 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.359 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.359 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.360 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.361 DEBUG:boto:Method: GET 2022-09-02T04:30:03.361 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.362 DEBUG:boto:Data: 2022-09-02T04:30:03.362 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.364 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.364 DEBUG:boto:Params: {} 2022-09-02T04:30:03.365 DEBUG:boto:Token: None 2022-09-02T04:30:03.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE= 2022-09-02T04:30:03.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE=', 'Content-Length': '0'} 2022-09-02T04:30:03.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023236866c6a94877-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.370 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.370 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.371 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.371 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.372 DEBUG:boto:Method: GET 2022-09-02T04:30:03.373 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.373 DEBUG:boto:Data: 2022-09-02T04:30:03.374 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.375 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.375 DEBUG:boto:Params: {} 2022-09-02T04:30:03.376 DEBUG:boto:Token: None 2022-09-02T04:30:03.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008147e2bda7d0d36d-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.422 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.432 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.433 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.433 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.434 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.434 DEBUG:boto:Method: GET 2022-09-02T04:30:03.435 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.435 DEBUG:boto:Data: 2022-09-02T04:30:03.436 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.437 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.438 DEBUG:boto:Params: {} 2022-09-02T04:30:03.438 DEBUG:boto:Token: None 2022-09-02T04:30:03.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecdb941c54472ad4-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.444 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.445 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.445 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.446 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.447 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:30:03.447 DEBUG:boto:Data: 2022-09-02T04:30:03.448 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.449 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.449 DEBUG:boto:Params: {} 2022-09-02T04:30:03.450 DEBUG:boto:Token: None 2022-09-02T04:30:03.450 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy 2022-09-02T04:30:03.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/sBGC49cCkIyRB78Mtr9hGIWqAs= 2022-09-02T04:30:03.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/sBGC49cCkIyRB78Mtr9hGIWqAs=', 'Content-Length': '0'} 2022-09-02T04:30:03.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000326bd518c1381b3e-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.454 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.455 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.455 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.456 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:30:03.456 DEBUG:boto:Data: 2022-09-02T04:30:03.457 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.458 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.458 DEBUG:boto:Params: {} 2022-09-02T04:30:03.459 DEBUG:boto:Token: None 2022-09-02T04:30:03.460 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy 2022-09-02T04:30:03.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/sBGC49cCkIyRB78Mtr9hGIWqAs= 2022-09-02T04:30:03.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/sBGC49cCkIyRB78Mtr9hGIWqAs=', 'Content-Length': '0'} 2022-09-02T04:30:03.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c845c32d970e1191-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:03.464 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.465 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.465 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.466 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.466 DEBUG:boto:Method: GET 2022-09-02T04:30:03.467 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.467 DEBUG:boto:Data: 2022-09-02T04:30:03.468 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.469 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.469 DEBUG:boto:Params: {} 2022-09-02T04:30:03.470 DEBUG:boto:Token: None 2022-09-02T04:30:03.471 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE= 2022-09-02T04:30:03.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE=', 'Content-Length': '0'} 2022-09-02T04:30:03.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099c796b095d049cc-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.475 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:03.475 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:03.476 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.476 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.477 DEBUG:boto:Method: GET 2022-09-02T04:30:03.477 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.478 DEBUG:boto:Data: 2022-09-02T04:30:03.479 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.480 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.480 DEBUG:boto:Params: {} 2022-09-02T04:30:03.481 DEBUG:boto:Token: None 2022-09-02T04:30:03.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:03.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE= 2022-09-02T04:30:03.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sRgOoJHvITY8SdUbSequUeP1BE=', 'Content-Length': '0'} 2022-09-02T04:30:03.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d71cc477e6dd9fd3-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.486 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.486 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.487 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.487 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.488 DEBUG:boto:Method: GET 2022-09-02T04:30:03.488 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.489 DEBUG:boto:Data: 2022-09-02T04:30:03.489 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.491 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.491 DEBUG:boto:Params: {} 2022-09-02T04:30:03.492 DEBUG:boto:Token: None 2022-09-02T04:30:03.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420079c9d2769b0a-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.497 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.497 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.498 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.499 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.499 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.500 DEBUG:boto:Method: GET 2022-09-02T04:30:03.500 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:03.501 DEBUG:boto:Data: 2022-09-02T04:30:03.501 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.502 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.503 DEBUG:boto:Params: {} 2022-09-02T04:30:03.504 DEBUG:boto:Token: None 2022-09-02T04:30:03.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8cf43e65bab0cd3-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.509 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:03.510 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.511 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.511 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.512 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.512 DEBUG:boto:Method: GET 2022-09-02T04:30:03.513 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.514 DEBUG:boto:Data: 2022-09-02T04:30:03.514 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.515 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.516 DEBUG:boto:Params: {} 2022-09-02T04:30:03.516 DEBUG:boto:Token: None 2022-09-02T04:30:03.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg= 2022-09-02T04:30:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg=', 'Content-Length': '0'} 2022-09-02T04:30:03.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c1129f0856ce184-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.521 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.522 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.522 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.523 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.523 DEBUG:boto:Method: GET 2022-09-02T04:30:03.524 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.524 DEBUG:boto:Data: 2022-09-02T04:30:03.525 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.526 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.526 DEBUG:boto:Params: {} 2022-09-02T04:30:03.527 DEBUG:boto:Token: None 2022-09-02T04:30:03.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg= 2022-09-02T04:30:03.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg=', 'Content-Length': '0'} 2022-09-02T04:30:03.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aab32a1906606a8-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.537 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.538 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.539 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.539 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.540 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.541 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.541 DEBUG:boto:Data: 2022-09-02T04:30:03.542 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.544 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.545 DEBUG:boto:Params: {} 2022-09-02T04:30:03.545 DEBUG:boto:Token: None 2022-09-02T04:30:03.546 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zc0bOW/epv5TP5PRLUflvibPe/w= 2022-09-02T04:30:03.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zc0bOW/epv5TP5PRLUflvibPe/w=', 'Content-Length': '0'} 2022-09-02T04:30:03.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041b5cf9587c8cbb6-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.550 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.551 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.551 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.552 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.553 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.554 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.554 DEBUG:boto:Data: 2022-09-02T04:30:03.555 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.556 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.557 DEBUG:boto:Params: {} 2022-09-02T04:30:03.557 DEBUG:boto:Token: None 2022-09-02T04:30:03.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zc0bOW/epv5TP5PRLUflvibPe/w= 2022-09-02T04:30:03.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zc0bOW/epv5TP5PRLUflvibPe/w=', 'Content-Length': '0'} 2022-09-02T04:30:03.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eeb91edbbc6c33a-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:03.568 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.570 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.571 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.572 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.573 DEBUG:boto:Method: GET 2022-09-02T04:30:03.574 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.575 DEBUG:boto:Data: 2022-09-02T04:30:03.577 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.579 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.580 DEBUG:boto:Params: {} 2022-09-02T04:30:03.581 DEBUG:boto:Token: None 2022-09-02T04:30:03.582 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.583 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg= 2022-09-02T04:30:03.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg=', 'Content-Length': '0'} 2022-09-02T04:30:03.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027d7a1eadc307972-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.587 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.588 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.589 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.589 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.590 DEBUG:boto:Method: GET 2022-09-02T04:30:03.590 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.591 DEBUG:boto:Data: 2022-09-02T04:30:03.591 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.592 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.593 DEBUG:boto:Params: {} 2022-09-02T04:30:03.593 DEBUG:boto:Token: None 2022-09-02T04:30:03.594 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.595 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg= 2022-09-02T04:30:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg=', 'Content-Length': '0'} 2022-09-02T04:30:03.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041a12e1d7331754b-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.642 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.643 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.643 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.644 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.645 DEBUG:boto:Method: GET 2022-09-02T04:30:03.645 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.646 DEBUG:boto:Data: 2022-09-02T04:30:03.647 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.648 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.648 DEBUG:boto:Params: {} 2022-09-02T04:30:03.649 DEBUG:boto:Token: None 2022-09-02T04:30:03.650 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fde60374184fb5a-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.654 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.655 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.656 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.656 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.657 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.657 DEBUG:boto:Method: GET 2022-09-02T04:30:03.658 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.658 DEBUG:boto:Data: 2022-09-02T04:30:03.659 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.661 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.661 DEBUG:boto:Params: {} 2022-09-02T04:30:03.662 DEBUG:boto:Token: None 2022-09-02T04:30:03.663 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc24dd4e3feaf8f6-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.667 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.668 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.669 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.670 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.670 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.671 DEBUG:boto:Data: 2022-09-02T04:30:03.671 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.672 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.673 DEBUG:boto:Params: {} 2022-09-02T04:30:03.673 DEBUG:boto:Token: None 2022-09-02T04:30:03.674 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PxEqgVR8INpnUNKMeZh+MsNHdr0= 2022-09-02T04:30:03.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PxEqgVR8INpnUNKMeZh+MsNHdr0=', 'Content-Length': '0'} 2022-09-02T04:30:03.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c86c908bfd5cca8a-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.679 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.679 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.680 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.680 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.681 DEBUG:boto:Data: 2022-09-02T04:30:03.681 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.683 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.683 DEBUG:boto:Params: {} 2022-09-02T04:30:03.684 DEBUG:boto:Token: None 2022-09-02T04:30:03.684 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PxEqgVR8INpnUNKMeZh+MsNHdr0= 2022-09-02T04:30:03.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PxEqgVR8INpnUNKMeZh+MsNHdr0=', 'Content-Length': '0'} 2022-09-02T04:30:03.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d32bf8288c3efd-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:03.691 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.692 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.692 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.693 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.694 DEBUG:boto:Method: GET 2022-09-02T04:30:03.694 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.695 DEBUG:boto:Data: 2022-09-02T04:30:03.695 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.697 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.697 DEBUG:boto:Params: {} 2022-09-02T04:30:03.698 DEBUG:boto:Token: None 2022-09-02T04:30:03.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg= 2022-09-02T04:30:03.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg=', 'Content-Length': '0'} 2022-09-02T04:30:03.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000218e6eb415d2c1bf-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.704 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.705 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:03.705 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.706 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.707 DEBUG:boto:Method: GET 2022-09-02T04:30:03.708 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.709 DEBUG:boto:Data: 2022-09-02T04:30:03.710 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.712 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.712 DEBUG:boto:Params: {} 2022-09-02T04:30:03.713 DEBUG:boto:Token: None 2022-09-02T04:30:03.714 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:03.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg= 2022-09-02T04:30:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLqbCBUJl0sdMROf+rWQ7cWlqmg=', 'Content-Length': '0'} 2022-09-02T04:30:03.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c932656ef29f5e2-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.718 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.719 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.719 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.720 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.720 DEBUG:boto:Method: GET 2022-09-02T04:30:03.721 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.721 DEBUG:boto:Data: 2022-09-02T04:30:03.722 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.723 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.723 DEBUG:boto:Params: {} 2022-09-02T04:30:03.724 DEBUG:boto:Token: None 2022-09-02T04:30:03.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6adeddb0f013eff-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.729 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.730 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.731 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.731 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.732 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.732 DEBUG:boto:Method: GET 2022-09-02T04:30:03.733 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:03.733 DEBUG:boto:Data: 2022-09-02T04:30:03.734 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.735 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.735 DEBUG:boto:Params: {} 2022-09-02T04:30:03.736 DEBUG:boto:Token: None 2022-09-02T04:30:03.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000315eff21e2076c4-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.741 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:03.742 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.743 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.744 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.744 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.745 DEBUG:boto:Method: GET 2022-09-02T04:30:03.745 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.746 DEBUG:boto:Data: 2022-09-02T04:30:03.746 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.747 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.748 DEBUG:boto:Params: {} 2022-09-02T04:30:03.748 DEBUG:boto:Token: None 2022-09-02T04:30:03.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk= 2022-09-02T04:30:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk=', 'Content-Length': '0'} 2022-09-02T04:30:03.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056bb295fd847754f-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.753 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.754 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.754 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.755 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.755 DEBUG:boto:Method: GET 2022-09-02T04:30:03.756 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.756 DEBUG:boto:Data: 2022-09-02T04:30:03.757 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.758 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.758 DEBUG:boto:Params: {} 2022-09-02T04:30:03.759 DEBUG:boto:Token: None 2022-09-02T04:30:03.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk= 2022-09-02T04:30:03.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk=', 'Content-Length': '0'} 2022-09-02T04:30:03.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8d321ef9768bc36-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.763 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.764 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.764 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.765 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.765 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.766 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.766 DEBUG:boto:Data: 2022-09-02T04:30:03.767 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.768 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.768 DEBUG:boto:Params: {} 2022-09-02T04:30:03.769 DEBUG:boto:Token: None 2022-09-02T04:30:03.770 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DiLQYNkAidO5vjwcTi1ZBSAc/KU= 2022-09-02T04:30:03.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DiLQYNkAidO5vjwcTi1ZBSAc/KU=', 'Content-Length': '0'} 2022-09-02T04:30:03.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003767c1f61c3faf3c-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.773 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.774 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.774 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.775 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.775 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.776 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.777 DEBUG:boto:Data: 2022-09-02T04:30:03.777 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.778 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.778 DEBUG:boto:Params: {} 2022-09-02T04:30:03.779 DEBUG:boto:Token: None 2022-09-02T04:30:03.780 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DiLQYNkAidO5vjwcTi1ZBSAc/KU= 2022-09-02T04:30:03.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DiLQYNkAidO5vjwcTi1ZBSAc/KU=', 'Content-Length': '0'} 2022-09-02T04:30:03.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097667afab4906465-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:03.784 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.785 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.785 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.786 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.786 DEBUG:boto:Method: GET 2022-09-02T04:30:03.787 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.788 DEBUG:boto:Data: 2022-09-02T04:30:03.788 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.789 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.790 DEBUG:boto:Params: {} 2022-09-02T04:30:03.790 DEBUG:boto:Token: None 2022-09-02T04:30:03.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk= 2022-09-02T04:30:03.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk=', 'Content-Length': '0'} 2022-09-02T04:30:03.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062f4f54574532456-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.795 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.795 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.796 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.796 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.797 DEBUG:boto:Method: GET 2022-09-02T04:30:03.797 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.798 DEBUG:boto:Data: 2022-09-02T04:30:03.799 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.800 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.800 DEBUG:boto:Params: {} 2022-09-02T04:30:03.801 DEBUG:boto:Token: None 2022-09-02T04:30:03.801 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk= 2022-09-02T04:30:03.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk=', 'Content-Length': '0'} 2022-09-02T04:30:03.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071f09c288f779a52-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.850 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.851 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.852 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.852 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.853 DEBUG:boto:Method: GET 2022-09-02T04:30:03.853 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.854 DEBUG:boto:Data: 2022-09-02T04:30:03.854 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.855 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.856 DEBUG:boto:Params: {} 2022-09-02T04:30:03.857 DEBUG:boto:Token: None 2022-09-02T04:30:03.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe31eeed07ea3d0-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.862 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.863 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.863 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.864 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.864 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.865 DEBUG:boto:Method: GET 2022-09-02T04:30:03.865 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.866 DEBUG:boto:Data: 2022-09-02T04:30:03.866 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.867 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.868 DEBUG:boto:Params: {} 2022-09-02T04:30:03.869 DEBUG:boto:Token: None 2022-09-02T04:30:03.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c8753f15f1326cf-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.874 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.874 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.875 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.875 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.876 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.876 DEBUG:boto:Data: 2022-09-02T04:30:03.877 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.878 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.878 DEBUG:boto:Params: {} 2022-09-02T04:30:03.879 DEBUG:boto:Token: None 2022-09-02T04:30:03.879 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nhi66gugwJwBw6Vz8+Jlqnd/Npw= 2022-09-02T04:30:03.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nhi66gugwJwBw6Vz8+Jlqnd/Npw=', 'Content-Length': '0'} 2022-09-02T04:30:03.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000484c9f7e1bc4e52d-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.884 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.884 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.885 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.885 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.886 DEBUG:boto:Data: 2022-09-02T04:30:03.886 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.887 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.888 DEBUG:boto:Params: {} 2022-09-02T04:30:03.888 DEBUG:boto:Token: None 2022-09-02T04:30:03.889 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nhi66gugwJwBw6Vz8+Jlqnd/Npw= 2022-09-02T04:30:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nhi66gugwJwBw6Vz8+Jlqnd/Npw=', 'Content-Length': '0'} 2022-09-02T04:30:03.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4143a0d2790d8c8-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:03.893 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.894 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.895 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.895 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.896 DEBUG:boto:Method: GET 2022-09-02T04:30:03.896 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.897 DEBUG:boto:Data: 2022-09-02T04:30:03.897 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.898 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.899 DEBUG:boto:Params: {} 2022-09-02T04:30:03.900 DEBUG:boto:Token: None 2022-09-02T04:30:03.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk= 2022-09-02T04:30:03.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk=', 'Content-Length': '0'} 2022-09-02T04:30:03.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000665bf08ba2ac9c0f-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.904 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.905 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:03.905 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.906 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.906 DEBUG:boto:Method: GET 2022-09-02T04:30:03.907 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.908 DEBUG:boto:Data: 2022-09-02T04:30:03.908 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.909 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.910 DEBUG:boto:Params: {} 2022-09-02T04:30:03.911 DEBUG:boto:Token: None 2022-09-02T04:30:03.911 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:03.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk= 2022-09-02T04:30:03.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eDWoJ5A3DLI0EN2CtPDqxvIM5Rk=', 'Content-Length': '0'} 2022-09-02T04:30:03.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4697f9382ca10f8-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.915 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.916 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.916 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.917 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.917 DEBUG:boto:Method: GET 2022-09-02T04:30:03.918 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.918 DEBUG:boto:Data: 2022-09-02T04:30:03.919 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.920 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.920 DEBUG:boto:Params: {} 2022-09-02T04:30:03.921 DEBUG:boto:Token: None 2022-09-02T04:30:03.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad5f52b2e4e23fc4-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.926 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.927 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:03.927 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:03.928 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.928 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.929 DEBUG:boto:Method: GET 2022-09-02T04:30:03.929 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:03.930 DEBUG:boto:Data: 2022-09-02T04:30:03.930 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.931 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.932 DEBUG:boto:Params: {} 2022-09-02T04:30:03.933 DEBUG:boto:Token: None 2022-09-02T04:30:03.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/ 2022-09-02T04:30:03.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws= 2022-09-02T04:30:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hpLNe0bIoPfEvLQ1TDz631OG6ws=', 'Content-Length': '0'} 2022-09-02T04:30:03.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008518a18313c70cee-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.938 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:03.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:03.939 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.940 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.940 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.941 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.941 DEBUG:boto:Method: GET 2022-09-02T04:30:03.942 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.942 DEBUG:boto:Data: 2022-09-02T04:30:03.943 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.944 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.944 DEBUG:boto:Params: {} 2022-09-02T04:30:03.945 DEBUG:boto:Token: None 2022-09-02T04:30:03.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0dI5UDUiUe3bvu21cqcnG+2ePCU= 2022-09-02T04:30:03.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0dI5UDUiUe3bvu21cqcnG+2ePCU=', 'Content-Length': '0'} 2022-09-02T04:30:03.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1a705ca72416183-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.950 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.950 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.951 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.951 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.952 DEBUG:boto:Method: GET 2022-09-02T04:30:03.952 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.953 DEBUG:boto:Data: 2022-09-02T04:30:03.954 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.955 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.955 DEBUG:boto:Params: {} 2022-09-02T04:30:03.956 DEBUG:boto:Token: None 2022-09-02T04:30:03.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0dI5UDUiUe3bvu21cqcnG+2ePCU= 2022-09-02T04:30:03.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0dI5UDUiUe3bvu21cqcnG+2ePCU=', 'Content-Length': '0'} 2022-09-02T04:30:03.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a87ca9dc7efd8f2-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.962 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.963 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.963 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.964 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.964 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.965 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.965 DEBUG:boto:Data: 2022-09-02T04:30:03.966 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.967 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.968 DEBUG:boto:Params: {} 2022-09-02T04:30:03.968 DEBUG:boto:Token: None 2022-09-02T04:30:03.969 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1rveXpzzO82eA7hBI7P/km3ZnIE= 2022-09-02T04:30:03.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1rveXpzzO82eA7hBI7P/km3ZnIE=', 'Content-Length': '0'} 2022-09-02T04:30:03.973 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6ca8f7cfdfd39d1-00631186cb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.973 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.974 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.975 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.976 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.977 DEBUG:boto:Method: HEAD 2022-09-02T04:30:03.978 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.979 DEBUG:boto:Data: 2022-09-02T04:30:03.980 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:03.982 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.983 DEBUG:boto:Params: {} 2022-09-02T04:30:03.984 DEBUG:boto:Token: None 2022-09-02T04:30:03.985 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1rveXpzzO82eA7hBI7P/km3ZnIE= 2022-09-02T04:30:03.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1rveXpzzO82eA7hBI7P/km3ZnIE=', 'Content-Length': '0'} 2022-09-02T04:30:03.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000724259a87b10e1a6-00631186cb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:03.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:03.991 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.992 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:03.992 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.993 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.993 DEBUG:boto:Method: GET 2022-09-02T04:30:03.994 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.995 DEBUG:boto:Data: 2022-09-02T04:30:03.995 DEBUG:boto:Headers: {} 2022-09-02T04:30:03.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:03.996 DEBUG:boto:Port: 8000 2022-09-02T04:30:03.997 DEBUG:boto:Params: {} 2022-09-02T04:30:03.997 DEBUG:boto:Token: None 2022-09-02T04:30:03.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:03 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:03.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0dI5UDUiUe3bvu21cqcnG+2ePCU= 2022-09-02T04:30:04.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0dI5UDUiUe3bvu21cqcnG+2ePCU=', 'Content-Length': '0'} 2022-09-02T04:30:04.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000953c43f37a33a381-00631186cc-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.003 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.003 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.004 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.004 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.005 DEBUG:boto:Method: GET 2022-09-02T04:30:04.005 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.006 DEBUG:boto:Data: 2022-09-02T04:30:04.007 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.008 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.008 DEBUG:boto:Params: {} 2022-09-02T04:30:04.009 DEBUG:boto:Token: None 2022-09-02T04:30:04.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n1vsOfRkb105ub8BDwq2HdqHa9Q= 2022-09-02T04:30:04.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n1vsOfRkb105ub8BDwq2HdqHa9Q=', 'Content-Length': '0'} 2022-09-02T04:30:04.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b708a58055469d3-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.015 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.016 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.016 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.017 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.018 DEBUG:boto:Method: GET 2022-09-02T04:30:04.019 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.019 DEBUG:boto:Data: 2022-09-02T04:30:04.020 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:04.021 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.022 DEBUG:boto:Params: {} 2022-09-02T04:30:04.023 DEBUG:boto:Token: None 2022-09-02T04:30:04.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedef04b616b7d02-00631186cc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.071 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.073 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.074 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.075 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.075 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.076 DEBUG:boto:Method: GET 2022-09-02T04:30:04.077 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.078 DEBUG:boto:Data: 2022-09-02T04:30:04.078 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.080 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.081 DEBUG:boto:Params: {} 2022-09-02T04:30:04.082 DEBUG:boto:Token: None 2022-09-02T04:30:04.083 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.084 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe2379b39d96a46-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.088 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.089 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.090 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.091 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.092 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.093 DEBUG:boto:Data: 2022-09-02T04:30:04.094 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:04.095 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.096 DEBUG:boto:Params: {} 2022-09-02T04:30:04.097 DEBUG:boto:Token: None 2022-09-02T04:30:04.098 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:meOvZ5Bej5zF6Y3jt214zDs7RnM= 2022-09-02T04:30:04.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:meOvZ5Bej5zF6Y3jt214zDs7RnM=', 'Content-Length': '0'} 2022-09-02T04:30:04.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003128e11ea1046409-00631186cc-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.105 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.106 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.107 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.107 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.108 DEBUG:boto:Data: 2022-09-02T04:30:04.110 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.112 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.112 DEBUG:boto:Params: {} 2022-09-02T04:30:04.113 DEBUG:boto:Token: None 2022-09-02T04:30:04.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:meOvZ5Bej5zF6Y3jt214zDs7RnM= 2022-09-02T04:30:04.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:meOvZ5Bej5zF6Y3jt214zDs7RnM=', 'Content-Length': '0'} 2022-09-02T04:30:04.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088b04aed394f4741-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:04.122 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.123 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.124 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.125 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.125 DEBUG:boto:Method: GET 2022-09-02T04:30:04.126 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.127 DEBUG:boto:Data: 2022-09-02T04:30:04.128 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:04.130 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.131 DEBUG:boto:Params: {} 2022-09-02T04:30:04.132 DEBUG:boto:Token: None 2022-09-02T04:30:04.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n1vsOfRkb105ub8BDwq2HdqHa9Q= 2022-09-02T04:30:04.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n1vsOfRkb105ub8BDwq2HdqHa9Q=', 'Content-Length': '0'} 2022-09-02T04:30:04.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa063000505c9b96-00631186cc-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.137 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.138 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:04.139 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.139 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.140 DEBUG:boto:Method: GET 2022-09-02T04:30:04.141 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.142 DEBUG:boto:Data: 2022-09-02T04:30:04.143 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.144 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.145 DEBUG:boto:Params: {} 2022-09-02T04:30:04.146 DEBUG:boto:Token: None 2022-09-02T04:30:04.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:04.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n1vsOfRkb105ub8BDwq2HdqHa9Q= 2022-09-02T04:30:04.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n1vsOfRkb105ub8BDwq2HdqHa9Q=', 'Content-Length': '0'} 2022-09-02T04:30:04.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000085bb38957614ad2-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.194 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.195 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.196 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.197 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.198 DEBUG:boto:Method: GET 2022-09-02T04:30:04.199 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.200 DEBUG:boto:Data: 2022-09-02T04:30:04.201 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:04.203 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.204 DEBUG:boto:Params: {} 2022-09-02T04:30:04.205 DEBUG:boto:Token: None 2022-09-02T04:30:04.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f7ed63422544f1-00631186cc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.221 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.222 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.222 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.223 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.224 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.224 DEBUG:boto:Method: GET 2022-09-02T04:30:04.225 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:04.225 DEBUG:boto:Data: 2022-09-02T04:30:04.226 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.228 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.229 DEBUG:boto:Params: {} 2022-09-02T04:30:04.230 DEBUG:boto:Token: None 2022-09-02T04:30:04.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f709ec307cbee70d-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.239 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.240 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-17 zones={test-zone2, test-zone1} 2022-09-02T04:30:04.244 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-17 zones={test-zone3, test-zone1} 2022-09-02T04:30:04.245 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.246 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.247 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.250 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:04.251 DEBUG:boto:Data: 2022-09-02T04:30:04.252 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.254 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.255 DEBUG:boto:Params: {} 2022-09-02T04:30:04.256 DEBUG:boto:Token: None 2022-09-02T04:30:04.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w5sYTpKV8UOqGxzNzNTONqOS3Is= 2022-09-02T04:30:04.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w5sYTpKV8UOqGxzNzNTONqOS3Is=', 'Content-Length': '0'} 2022-09-02T04:30:04.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693cbe778107e93b-00631186cc-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.265 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.266 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.267 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.268 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:04.269 DEBUG:boto:Data: 2022-09-02T04:30:04.270 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.273 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.275 DEBUG:boto:Params: {} 2022-09-02T04:30:04.276 DEBUG:boto:Token: None 2022-09-02T04:30:04.276 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w5sYTpKV8UOqGxzNzNTONqOS3Is= 2022-09-02T04:30:04.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w5sYTpKV8UOqGxzNzNTONqOS3Is=', 'Content-Length': '0'} 2022-09-02T04:30:04.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b525a6dc21479697-00631186cc-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.282 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:30:04.282 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.283 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.283 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.284 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.284 DEBUG:boto:Method: GET 2022-09-02T04:30:04.285 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.286 DEBUG:boto:Data: 2022-09-02T04:30:04.286 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.287 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.288 DEBUG:boto:Params: {} 2022-09-02T04:30:04.289 DEBUG:boto:Token: None 2022-09-02T04:30:04.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/?versions 2022-09-02T04:30:04.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY= 2022-09-02T04:30:04.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY=', 'Content-Length': '0'} 2022-09-02T04:30:04.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b81b68e93e9e9a08-00631186cc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.294 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.295 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:04.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:04.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:04.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:04.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:30:04.298 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.298 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.299 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.300 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.300 DEBUG:boto:Method: GET 2022-09-02T04:30:04.301 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.301 DEBUG:boto:Data: 2022-09-02T04:30:04.302 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.303 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.304 DEBUG:boto:Params: {} 2022-09-02T04:30:04.305 DEBUG:boto:Token: None 2022-09-02T04:30:04.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/?versions 2022-09-02T04:30:04.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY= 2022-09-02T04:30:04.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY=', 'Content-Length': '0'} 2022-09-02T04:30:04.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aef26e78a0b08e0-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.311 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.312 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:04.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:04.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:04.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:04.314 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.315 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.315 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.316 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.316 DEBUG:boto:Method: GET 2022-09-02T04:30:04.317 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.317 DEBUG:boto:Data: 2022-09-02T04:30:04.318 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.323 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.324 DEBUG:boto:Params: {} 2022-09-02T04:30:04.324 DEBUG:boto:Token: None 2022-09-02T04:30:04.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/?versions 2022-09-02T04:30:04.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY= 2022-09-02T04:30:04.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY=', 'Content-Length': '0'} 2022-09-02T04:30:04.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003af462d8193a1bda-00631186cc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.330 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.331 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.332 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.332 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.333 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.333 DEBUG:boto:Method: GET 2022-09-02T04:30:04.334 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:04.335 DEBUG:boto:Data: 2022-09-02T04:30:04.335 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.336 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.337 DEBUG:boto:Params: {} 2022-09-02T04:30:04.338 DEBUG:boto:Token: None 2022-09-02T04:30:04.339 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/?versions 2022-09-02T04:30:04.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY= 2022-09-02T04:30:04.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/PoF2bLupzaV5C1EQ2hlPG0BITY=', 'Content-Length': '0'} 2022-09-02T04:30:04.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003309e18c4f46cfc3-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.344 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:04.346 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.347 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.348 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.349 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.349 DEBUG:boto:Method: GET 2022-09-02T04:30:04.350 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.351 DEBUG:boto:Data: 2022-09-02T04:30:04.352 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.354 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.354 DEBUG:boto:Params: {} 2022-09-02T04:30:04.355 DEBUG:boto:Token: None 2022-09-02T04:30:04.356 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg= 2022-09-02T04:30:04.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg=', 'Content-Length': '0'} 2022-09-02T04:30:04.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019678f485641e02a-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.361 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.362 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.363 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.363 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.364 DEBUG:boto:Method: GET 2022-09-02T04:30:04.366 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.367 DEBUG:boto:Data: 2022-09-02T04:30:04.368 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.371 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.372 DEBUG:boto:Params: {} 2022-09-02T04:30:04.374 DEBUG:boto:Token: None 2022-09-02T04:30:04.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg= 2022-09-02T04:30:04.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg=', 'Content-Length': '0'} 2022-09-02T04:30:04.381 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076f56e05fa40a1ff-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.383 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.384 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.385 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.386 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.387 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.388 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.389 DEBUG:boto:Data: 2022-09-02T04:30:04.390 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.392 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.393 DEBUG:boto:Params: {} 2022-09-02T04:30:04.394 DEBUG:boto:Token: None 2022-09-02T04:30:04.395 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCt7v+pjQ1vGnORXmrvG0obYfoI= 2022-09-02T04:30:04.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCt7v+pjQ1vGnORXmrvG0obYfoI=', 'Content-Length': '0'} 2022-09-02T04:30:04.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4c97aa8bc04646c-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.399 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.400 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.401 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.402 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.403 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.403 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.404 DEBUG:boto:Data: 2022-09-02T04:30:04.404 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.406 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.407 DEBUG:boto:Params: {} 2022-09-02T04:30:04.408 DEBUG:boto:Token: None 2022-09-02T04:30:04.409 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCt7v+pjQ1vGnORXmrvG0obYfoI= 2022-09-02T04:30:04.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCt7v+pjQ1vGnORXmrvG0obYfoI=', 'Content-Length': '0'} 2022-09-02T04:30:04.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe85863b664d16e8-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:04.417 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.418 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.418 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.419 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.420 DEBUG:boto:Method: GET 2022-09-02T04:30:04.420 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.421 DEBUG:boto:Data: 2022-09-02T04:30:04.421 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.423 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.424 DEBUG:boto:Params: {} 2022-09-02T04:30:04.424 DEBUG:boto:Token: None 2022-09-02T04:30:04.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg= 2022-09-02T04:30:04.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg=', 'Content-Length': '0'} 2022-09-02T04:30:04.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd1d1ff391c7e7b4-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.432 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.433 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.433 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.434 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.435 DEBUG:boto:Method: GET 2022-09-02T04:30:04.435 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.436 DEBUG:boto:Data: 2022-09-02T04:30:04.437 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.438 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.439 DEBUG:boto:Params: {} 2022-09-02T04:30:04.440 DEBUG:boto:Token: None 2022-09-02T04:30:04.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg= 2022-09-02T04:30:04.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg=', 'Content-Length': '0'} 2022-09-02T04:30:04.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0aa2a7aa4f0bfe7-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.447 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.448 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.449 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.450 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.451 DEBUG:boto:Method: GET 2022-09-02T04:30:04.452 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.453 DEBUG:boto:Data: 2022-09-02T04:30:04.453 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.454 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.455 DEBUG:boto:Params: {} 2022-09-02T04:30:04.456 DEBUG:boto:Token: None 2022-09-02T04:30:04.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151a0269e8c147ef-00631186cc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.507 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.509 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.510 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.511 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.512 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.513 DEBUG:boto:Method: GET 2022-09-02T04:30:04.514 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.515 DEBUG:boto:Data: 2022-09-02T04:30:04.516 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.518 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.519 DEBUG:boto:Params: {} 2022-09-02T04:30:04.520 DEBUG:boto:Token: None 2022-09-02T04:30:04.521 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.522 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad64f920cad72a0-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.528 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.530 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.531 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.531 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.532 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:30:04.532 DEBUG:boto:Data: 2022-09-02T04:30:04.533 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.535 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.536 DEBUG:boto:Params: {} 2022-09-02T04:30:04.537 DEBUG:boto:Token: None 2022-09-02T04:30:04.538 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy 2022-09-02T04:30:04.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6SUcks/8rKyLB8Cq5BBlQR8Kf5A= 2022-09-02T04:30:04.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6SUcks/8rKyLB8Cq5BBlQR8Kf5A=', 'Content-Length': '0'} 2022-09-02T04:30:04.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b85cf02859c4ef81-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.546 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.547 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.548 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.549 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:30:04.550 DEBUG:boto:Data: 2022-09-02T04:30:04.550 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.552 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.554 DEBUG:boto:Params: {} 2022-09-02T04:30:04.556 DEBUG:boto:Token: None 2022-09-02T04:30:04.556 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy 2022-09-02T04:30:04.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6SUcks/8rKyLB8Cq5BBlQR8Kf5A= 2022-09-02T04:30:04.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6SUcks/8rKyLB8Cq5BBlQR8Kf5A=', 'Content-Length': '0'} 2022-09-02T04:30:04.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000993decbcfc0d8e5c-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:04.566 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.567 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.568 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.569 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.570 DEBUG:boto:Method: GET 2022-09-02T04:30:04.571 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.572 DEBUG:boto:Data: 2022-09-02T04:30:04.573 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.575 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.576 DEBUG:boto:Params: {} 2022-09-02T04:30:04.577 DEBUG:boto:Token: None 2022-09-02T04:30:04.578 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg= 2022-09-02T04:30:04.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg=', 'Content-Length': '0'} 2022-09-02T04:30:04.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f70796c24a7075b-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.583 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:04.584 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:04.585 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.586 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.586 DEBUG:boto:Method: GET 2022-09-02T04:30:04.587 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.588 DEBUG:boto:Data: 2022-09-02T04:30:04.589 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.591 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.592 DEBUG:boto:Params: {} 2022-09-02T04:30:04.593 DEBUG:boto:Token: None 2022-09-02T04:30:04.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:04.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg= 2022-09-02T04:30:04.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8szJQnk3hsg+CM8A74ntlfpvmg=', 'Content-Length': '0'} 2022-09-02T04:30:04.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000beea5a691c06384-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.601 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.602 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.602 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.603 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.603 DEBUG:boto:Method: GET 2022-09-02T04:30:04.604 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.604 DEBUG:boto:Data: 2022-09-02T04:30:04.605 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.606 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.607 DEBUG:boto:Params: {} 2022-09-02T04:30:04.608 DEBUG:boto:Token: None 2022-09-02T04:30:04.609 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4812aea264edb91-00631186cc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.613 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.614 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.615 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.615 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.616 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.617 DEBUG:boto:Method: GET 2022-09-02T04:30:04.617 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:04.618 DEBUG:boto:Data: 2022-09-02T04:30:04.619 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.620 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.621 DEBUG:boto:Params: {} 2022-09-02T04:30:04.622 DEBUG:boto:Token: None 2022-09-02T04:30:04.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bb04ac6173a4377-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.628 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:04.629 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.630 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.631 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.632 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.633 DEBUG:boto:Method: GET 2022-09-02T04:30:04.634 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.635 DEBUG:boto:Data: 2022-09-02T04:30:04.636 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.637 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.638 DEBUG:boto:Params: {} 2022-09-02T04:30:04.639 DEBUG:boto:Token: None 2022-09-02T04:30:04.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg= 2022-09-02T04:30:04.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg=', 'Content-Length': '0'} 2022-09-02T04:30:04.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040002b4216308f52-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.644 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.645 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.645 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.646 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.646 DEBUG:boto:Method: GET 2022-09-02T04:30:04.647 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.647 DEBUG:boto:Data: 2022-09-02T04:30:04.648 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.649 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.650 DEBUG:boto:Params: {} 2022-09-02T04:30:04.650 DEBUG:boto:Token: None 2022-09-02T04:30:04.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg= 2022-09-02T04:30:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg=', 'Content-Length': '0'} 2022-09-02T04:30:04.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbc5ae91af7c9e1f-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.656 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.656 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.657 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.657 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.658 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.658 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.659 DEBUG:boto:Data: 2022-09-02T04:30:04.660 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.661 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.661 DEBUG:boto:Params: {} 2022-09-02T04:30:04.662 DEBUG:boto:Token: None 2022-09-02T04:30:04.662 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SZsstHZzsng3UH7aFwXaJzrkV2g= 2022-09-02T04:30:04.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SZsstHZzsng3UH7aFwXaJzrkV2g=', 'Content-Length': '0'} 2022-09-02T04:30:04.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de82da3ec784c7f0-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.666 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.666 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.667 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.667 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.668 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.668 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.669 DEBUG:boto:Data: 2022-09-02T04:30:04.669 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.670 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.671 DEBUG:boto:Params: {} 2022-09-02T04:30:04.671 DEBUG:boto:Token: None 2022-09-02T04:30:04.672 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SZsstHZzsng3UH7aFwXaJzrkV2g= 2022-09-02T04:30:04.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SZsstHZzsng3UH7aFwXaJzrkV2g=', 'Content-Length': '0'} 2022-09-02T04:30:04.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000464a6cd5e736468e-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:04.677 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.677 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.678 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.678 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.679 DEBUG:boto:Method: GET 2022-09-02T04:30:04.679 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.680 DEBUG:boto:Data: 2022-09-02T04:30:04.680 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.681 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.682 DEBUG:boto:Params: {} 2022-09-02T04:30:04.682 DEBUG:boto:Token: None 2022-09-02T04:30:04.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg= 2022-09-02T04:30:04.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg=', 'Content-Length': '0'} 2022-09-02T04:30:04.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b243e68fc69f19ab-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.730 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.731 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.732 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.733 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.734 DEBUG:boto:Method: GET 2022-09-02T04:30:04.735 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.736 DEBUG:boto:Data: 2022-09-02T04:30:04.737 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.738 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.739 DEBUG:boto:Params: {} 2022-09-02T04:30:04.740 DEBUG:boto:Token: None 2022-09-02T04:30:04.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg= 2022-09-02T04:30:04.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg=', 'Content-Length': '0'} 2022-09-02T04:30:04.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000424091ea3158436-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.747 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.748 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.748 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.749 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.750 DEBUG:boto:Method: GET 2022-09-02T04:30:04.750 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.751 DEBUG:boto:Data: 2022-09-02T04:30:04.751 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.752 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.753 DEBUG:boto:Params: {} 2022-09-02T04:30:04.753 DEBUG:boto:Token: None 2022-09-02T04:30:04.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d1a18af242cfeac-00631186cc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.759 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.760 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.760 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.761 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.762 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.762 DEBUG:boto:Method: GET 2022-09-02T04:30:04.763 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.763 DEBUG:boto:Data: 2022-09-02T04:30:04.764 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.765 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.765 DEBUG:boto:Params: {} 2022-09-02T04:30:04.766 DEBUG:boto:Token: None 2022-09-02T04:30:04.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ca0a393da2b3fe-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.772 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.774 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.775 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.776 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.777 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.778 DEBUG:boto:Data: 2022-09-02T04:30:04.779 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.781 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.782 DEBUG:boto:Params: {} 2022-09-02T04:30:04.783 DEBUG:boto:Token: None 2022-09-02T04:30:04.784 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3aFn2ceOkYK2St7Z1zaJ+0Ru71I= 2022-09-02T04:30:04.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3aFn2ceOkYK2St7Z1zaJ+0Ru71I=', 'Content-Length': '0'} 2022-09-02T04:30:04.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6e90d25611b6a45-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.790 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.791 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.792 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.793 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.793 DEBUG:boto:Data: 2022-09-02T04:30:04.794 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.795 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.796 DEBUG:boto:Params: {} 2022-09-02T04:30:04.797 DEBUG:boto:Token: None 2022-09-02T04:30:04.798 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3aFn2ceOkYK2St7Z1zaJ+0Ru71I= 2022-09-02T04:30:04.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3aFn2ceOkYK2St7Z1zaJ+0Ru71I=', 'Content-Length': '0'} 2022-09-02T04:30:04.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4115b5c8ec817d7-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:04.806 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.807 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.808 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.809 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.810 DEBUG:boto:Method: GET 2022-09-02T04:30:04.811 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.812 DEBUG:boto:Data: 2022-09-02T04:30:04.814 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.817 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.818 DEBUG:boto:Params: {} 2022-09-02T04:30:04.819 DEBUG:boto:Token: None 2022-09-02T04:30:04.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg= 2022-09-02T04:30:04.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg=', 'Content-Length': '0'} 2022-09-02T04:30:04.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c1e270c2edba27-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.825 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.825 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:04.826 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.826 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.827 DEBUG:boto:Method: GET 2022-09-02T04:30:04.827 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.828 DEBUG:boto:Data: 2022-09-02T04:30:04.829 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.830 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.830 DEBUG:boto:Params: {} 2022-09-02T04:30:04.831 DEBUG:boto:Token: None 2022-09-02T04:30:04.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:04.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg= 2022-09-02T04:30:04.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A7Juljg1EZtWNUffQOFgDMKJsWg=', 'Content-Length': '0'} 2022-09-02T04:30:04.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb83e62c32ec1be5-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.878 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.879 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.879 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.880 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.881 DEBUG:boto:Method: GET 2022-09-02T04:30:04.882 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.882 DEBUG:boto:Data: 2022-09-02T04:30:04.883 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.884 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.885 DEBUG:boto:Params: {} 2022-09-02T04:30:04.885 DEBUG:boto:Token: None 2022-09-02T04:30:04.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000271d251338661be8-00631186cc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.891 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.891 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:04.892 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:04.893 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.893 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.894 DEBUG:boto:Method: GET 2022-09-02T04:30:04.894 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:04.895 DEBUG:boto:Data: 2022-09-02T04:30:04.895 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.896 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.897 DEBUG:boto:Params: {} 2022-09-02T04:30:04.898 DEBUG:boto:Token: None 2022-09-02T04:30:04.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/ 2022-09-02T04:30:04.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY= 2022-09-02T04:30:04.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxYJFSXdUoYhqQTs/2YnnRdpVwY=', 'Content-Length': '0'} 2022-09-02T04:30:04.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000866a966765c88430-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.903 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:04.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:04.904 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.904 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.905 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.906 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.906 DEBUG:boto:Method: GET 2022-09-02T04:30:04.907 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.907 DEBUG:boto:Data: 2022-09-02T04:30:04.908 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.909 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.909 DEBUG:boto:Params: {} 2022-09-02T04:30:04.910 DEBUG:boto:Token: None 2022-09-02T04:30:04.911 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MP91dCOJtyCYWONBPo6z5GjkgK8= 2022-09-02T04:30:04.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MP91dCOJtyCYWONBPo6z5GjkgK8=', 'Content-Length': '0'} 2022-09-02T04:30:04.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002674d17664f498d9-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.958 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.959 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.959 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.960 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.961 DEBUG:boto:Method: GET 2022-09-02T04:30:04.961 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.962 DEBUG:boto:Data: 2022-09-02T04:30:04.962 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.963 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.964 DEBUG:boto:Params: {} 2022-09-02T04:30:04.965 DEBUG:boto:Token: None 2022-09-02T04:30:04.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MP91dCOJtyCYWONBPo6z5GjkgK8= 2022-09-02T04:30:04.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MP91dCOJtyCYWONBPo6z5GjkgK8=', 'Content-Length': '0'} 2022-09-02T04:30:04.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035e7eec8c43c9bb1-00631186cc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.974 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.975 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.976 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.977 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.978 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.978 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.979 DEBUG:boto:Data: 2022-09-02T04:30:04.980 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:04.982 DEBUG:boto:Port: 8001 2022-09-02T04:30:04.983 DEBUG:boto:Params: {} 2022-09-02T04:30:04.983 DEBUG:boto:Token: None 2022-09-02T04:30:04.985 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LUsJR1S8Z0lCLXGtZF8yxScDbZQ= 2022-09-02T04:30:04.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LUsJR1S8Z0lCLXGtZF8yxScDbZQ=', 'Content-Length': '0'} 2022-09-02T04:30:04.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042acb528a6a16231-00631186cc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:04.991 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.992 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:04.992 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.993 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.994 DEBUG:boto:Method: HEAD 2022-09-02T04:30:04.995 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:04.995 DEBUG:boto:Data: 2022-09-02T04:30:04.996 DEBUG:boto:Headers: {} 2022-09-02T04:30:04.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:04.997 DEBUG:boto:Port: 8000 2022-09-02T04:30:04.998 DEBUG:boto:Params: {} 2022-09-02T04:30:04.999 DEBUG:boto:Token: None 2022-09-02T04:30:04.999 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:04 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LUsJR1S8Z0lCLXGtZF8yxScDbZQ= 2022-09-02T04:30:05.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LUsJR1S8Z0lCLXGtZF8yxScDbZQ=', 'Content-Length': '0'} 2022-09-02T04:30:05.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028e67fb3c5fa71a2-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:05.004 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.004 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.005 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.006 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.006 DEBUG:boto:Method: GET 2022-09-02T04:30:05.007 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.008 DEBUG:boto:Data: 2022-09-02T04:30:05.008 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.009 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.010 DEBUG:boto:Params: {} 2022-09-02T04:30:05.011 DEBUG:boto:Token: None 2022-09-02T04:30:05.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA= 2022-09-02T04:30:05.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA=', 'Content-Length': '0'} 2022-09-02T04:30:05.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000128158cf09d5ed77-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.017 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.019 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.020 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.021 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.021 DEBUG:boto:Method: GET 2022-09-02T04:30:05.022 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.023 DEBUG:boto:Data: 2022-09-02T04:30:05.024 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.026 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.026 DEBUG:boto:Params: {} 2022-09-02T04:30:05.028 DEBUG:boto:Token: None 2022-09-02T04:30:05.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA= 2022-09-02T04:30:05.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA=', 'Content-Length': '0'} 2022-09-02T04:30:05.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf0db9e2601a07d8-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.074 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.075 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.075 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.076 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.077 DEBUG:boto:Method: GET 2022-09-02T04:30:05.077 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.078 DEBUG:boto:Data: 2022-09-02T04:30:05.078 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.079 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.080 DEBUG:boto:Params: {} 2022-09-02T04:30:05.080 DEBUG:boto:Token: None 2022-09-02T04:30:05.081 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c622bcba0ca1177d-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.086 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.087 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.087 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.088 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.088 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.089 DEBUG:boto:Method: GET 2022-09-02T04:30:05.089 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.090 DEBUG:boto:Data: 2022-09-02T04:30:05.091 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.092 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.092 DEBUG:boto:Params: {} 2022-09-02T04:30:05.093 DEBUG:boto:Token: None 2022-09-02T04:30:05.093 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c59efb8d7aba744-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.098 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.099 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.099 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.100 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.100 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.101 DEBUG:boto:Data: 2022-09-02T04:30:05.101 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.102 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.103 DEBUG:boto:Params: {} 2022-09-02T04:30:05.104 DEBUG:boto:Token: None 2022-09-02T04:30:05.104 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GWMtG+Yy68yDZr/HMKFaWAIQPhM= 2022-09-02T04:30:05.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GWMtG+Yy68yDZr/HMKFaWAIQPhM=', 'Content-Length': '0'} 2022-09-02T04:30:05.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d730d0562ebf3f2-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.108 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.109 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.109 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.110 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.110 DEBUG:boto:Data: 2022-09-02T04:30:05.111 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.112 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.112 DEBUG:boto:Params: {} 2022-09-02T04:30:05.113 DEBUG:boto:Token: None 2022-09-02T04:30:05.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GWMtG+Yy68yDZr/HMKFaWAIQPhM= 2022-09-02T04:30:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GWMtG+Yy68yDZr/HMKFaWAIQPhM=', 'Content-Length': '0'} 2022-09-02T04:30:05.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d32c98dcdd342d4-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:05.118 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.119 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.119 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.120 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.120 DEBUG:boto:Method: GET 2022-09-02T04:30:05.121 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.121 DEBUG:boto:Data: 2022-09-02T04:30:05.122 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.123 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.123 DEBUG:boto:Params: {} 2022-09-02T04:30:05.124 DEBUG:boto:Token: None 2022-09-02T04:30:05.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA= 2022-09-02T04:30:05.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA=', 'Content-Length': '0'} 2022-09-02T04:30:05.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf72797a0b43f08a-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.129 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.130 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.130 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.131 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.132 DEBUG:boto:Method: GET 2022-09-02T04:30:05.132 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.133 DEBUG:boto:Data: 2022-09-02T04:30:05.133 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.134 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.135 DEBUG:boto:Params: {} 2022-09-02T04:30:05.136 DEBUG:boto:Token: None 2022-09-02T04:30:05.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA= 2022-09-02T04:30:05.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA=', 'Content-Length': '0'} 2022-09-02T04:30:05.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fa12fb4000137b6-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.140 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.141 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.142 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.142 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.143 DEBUG:boto:Method: GET 2022-09-02T04:30:05.143 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.144 DEBUG:boto:Data: 2022-09-02T04:30:05.145 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.146 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.146 DEBUG:boto:Params: {} 2022-09-02T04:30:05.147 DEBUG:boto:Token: None 2022-09-02T04:30:05.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e84d0ea894e6ef3-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.152 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.153 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.153 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.154 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.155 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.155 DEBUG:boto:Method: GET 2022-09-02T04:30:05.156 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:05.156 DEBUG:boto:Data: 2022-09-02T04:30:05.157 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.158 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.159 DEBUG:boto:Params: {} 2022-09-02T04:30:05.160 DEBUG:boto:Token: None 2022-09-02T04:30:05.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c47eedb7c06c333-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.165 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:05.166 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.166 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.167 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.167 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.168 DEBUG:boto:Method: GET 2022-09-02T04:30:05.169 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.170 DEBUG:boto:Data: 2022-09-02T04:30:05.170 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.171 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.172 DEBUG:boto:Params: {} 2022-09-02T04:30:05.173 DEBUG:boto:Token: None 2022-09-02T04:30:05.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU= 2022-09-02T04:30:05.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU=', 'Content-Length': '0'} 2022-09-02T04:30:05.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f92184f2aabc98ec-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.177 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.178 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.178 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.179 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.179 DEBUG:boto:Method: GET 2022-09-02T04:30:05.180 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.181 DEBUG:boto:Data: 2022-09-02T04:30:05.182 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.183 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.184 DEBUG:boto:Params: {} 2022-09-02T04:30:05.185 DEBUG:boto:Token: None 2022-09-02T04:30:05.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU= 2022-09-02T04:30:05.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU=', 'Content-Length': '0'} 2022-09-02T04:30:05.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000390dfb5da5ac80fe-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.189 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.190 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.190 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.191 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.191 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.192 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.193 DEBUG:boto:Data: 2022-09-02T04:30:05.193 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.194 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.195 DEBUG:boto:Params: {} 2022-09-02T04:30:05.195 DEBUG:boto:Token: None 2022-09-02T04:30:05.196 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8hKRlrHJGWcQdBy7KHlKXohiL1U= 2022-09-02T04:30:05.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8hKRlrHJGWcQdBy7KHlKXohiL1U=', 'Content-Length': '0'} 2022-09-02T04:30:05.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8e0ffb693e71914-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.200 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.200 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.201 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.201 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.202 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.202 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.203 DEBUG:boto:Data: 2022-09-02T04:30:05.203 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.204 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.205 DEBUG:boto:Params: {} 2022-09-02T04:30:05.205 DEBUG:boto:Token: None 2022-09-02T04:30:05.206 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8hKRlrHJGWcQdBy7KHlKXohiL1U= 2022-09-02T04:30:05.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8hKRlrHJGWcQdBy7KHlKXohiL1U=', 'Content-Length': '0'} 2022-09-02T04:30:05.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c76fc403d7a5b67-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:05.210 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.211 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.211 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.212 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.212 DEBUG:boto:Method: GET 2022-09-02T04:30:05.213 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.213 DEBUG:boto:Data: 2022-09-02T04:30:05.214 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.215 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.215 DEBUG:boto:Params: {} 2022-09-02T04:30:05.216 DEBUG:boto:Token: None 2022-09-02T04:30:05.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU= 2022-09-02T04:30:05.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU=', 'Content-Length': '0'} 2022-09-02T04:30:05.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b179deef2a518151-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.220 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.221 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.221 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.222 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.223 DEBUG:boto:Method: GET 2022-09-02T04:30:05.223 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.224 DEBUG:boto:Data: 2022-09-02T04:30:05.224 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.225 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.226 DEBUG:boto:Params: {} 2022-09-02T04:30:05.227 DEBUG:boto:Token: None 2022-09-02T04:30:05.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU= 2022-09-02T04:30:05.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU=', 'Content-Length': '0'} 2022-09-02T04:30:05.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ae9c39766a96d8c-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.231 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.232 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.232 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.233 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.234 DEBUG:boto:Method: GET 2022-09-02T04:30:05.234 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.235 DEBUG:boto:Data: 2022-09-02T04:30:05.235 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.236 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.237 DEBUG:boto:Params: {} 2022-09-02T04:30:05.237 DEBUG:boto:Token: None 2022-09-02T04:30:05.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011c809150078b2f5-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.242 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.243 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.244 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.244 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.245 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.245 DEBUG:boto:Method: GET 2022-09-02T04:30:05.246 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.246 DEBUG:boto:Data: 2022-09-02T04:30:05.247 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.248 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.249 DEBUG:boto:Params: {} 2022-09-02T04:30:05.249 DEBUG:boto:Token: None 2022-09-02T04:30:05.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4b39a9564089a61-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.254 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.255 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.256 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.256 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.257 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.257 DEBUG:boto:Data: 2022-09-02T04:30:05.258 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.259 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.259 DEBUG:boto:Params: {} 2022-09-02T04:30:05.260 DEBUG:boto:Token: None 2022-09-02T04:30:05.260 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fxAYH5oqIpIC2b4kYWnvO8f5JWg= 2022-09-02T04:30:05.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fxAYH5oqIpIC2b4kYWnvO8f5JWg=', 'Content-Length': '0'} 2022-09-02T04:30:05.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae63de25838e2768-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.264 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.265 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.266 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.266 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.267 DEBUG:boto:Data: 2022-09-02T04:30:05.267 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.268 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.269 DEBUG:boto:Params: {} 2022-09-02T04:30:05.269 DEBUG:boto:Token: None 2022-09-02T04:30:05.270 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fxAYH5oqIpIC2b4kYWnvO8f5JWg= 2022-09-02T04:30:05.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fxAYH5oqIpIC2b4kYWnvO8f5JWg=', 'Content-Length': '0'} 2022-09-02T04:30:05.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5dced4a7f4d74f3-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:05.275 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.275 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.276 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.276 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.277 DEBUG:boto:Method: GET 2022-09-02T04:30:05.277 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.278 DEBUG:boto:Data: 2022-09-02T04:30:05.278 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.279 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.280 DEBUG:boto:Params: {} 2022-09-02T04:30:05.281 DEBUG:boto:Token: None 2022-09-02T04:30:05.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU= 2022-09-02T04:30:05.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU=', 'Content-Length': '0'} 2022-09-02T04:30:05.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012c1fa8caf48a700-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.286 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.286 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:05.287 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.288 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.288 DEBUG:boto:Method: GET 2022-09-02T04:30:05.289 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.289 DEBUG:boto:Data: 2022-09-02T04:30:05.290 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.291 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.292 DEBUG:boto:Params: {} 2022-09-02T04:30:05.292 DEBUG:boto:Token: None 2022-09-02T04:30:05.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:05.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU= 2022-09-02T04:30:05.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jd4QFzMlEupfKXd6n2m0/5KD/DU=', 'Content-Length': '0'} 2022-09-02T04:30:05.296 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000937db4077578437f-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.298 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.298 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.299 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.300 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.300 DEBUG:boto:Method: GET 2022-09-02T04:30:05.301 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.302 DEBUG:boto:Data: 2022-09-02T04:30:05.303 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.306 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.307 DEBUG:boto:Params: {} 2022-09-02T04:30:05.308 DEBUG:boto:Token: None 2022-09-02T04:30:05.309 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13b2cfe6323adfd-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.314 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.315 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.315 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.316 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.316 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.317 DEBUG:boto:Method: GET 2022-09-02T04:30:05.318 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:05.319 DEBUG:boto:Data: 2022-09-02T04:30:05.320 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:05.322 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.323 DEBUG:boto:Params: {} 2022-09-02T04:30:05.325 DEBUG:boto:Token: None 2022-09-02T04:30:05.326 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19979f75fc9ab67-00631186cd-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.332 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-17 zones={test-zone3, test-zone1} 2022-09-02T04:30:05.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-17 zones={test-zone3, test-zone2} 2022-09-02T04:30:05.335 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.336 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.337 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.338 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:05.339 DEBUG:boto:Data: 2022-09-02T04:30:05.340 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.342 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.343 DEBUG:boto:Params: {} 2022-09-02T04:30:05.344 DEBUG:boto:Token: None 2022-09-02T04:30:05.345 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qhU/RqEiy6XBTf/5/nBxNtccIxA= 2022-09-02T04:30:05.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qhU/RqEiy6XBTf/5/nBxNtccIxA=', 'Content-Length': '0'} 2022-09-02T04:30:05.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a479fcccf7a2b028-00631186cd-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.351 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.351 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.352 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.352 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:05.353 DEBUG:boto:Data: 2022-09-02T04:30:05.353 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.354 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.355 DEBUG:boto:Params: {} 2022-09-02T04:30:05.355 DEBUG:boto:Token: None 2022-09-02T04:30:05.356 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qhU/RqEiy6XBTf/5/nBxNtccIxA= 2022-09-02T04:30:05.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qhU/RqEiy6XBTf/5/nBxNtccIxA=', 'Content-Length': '0'} 2022-09-02T04:30:05.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b72d7d853a64622c-00631186cd-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.361 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:30:05.362 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.362 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.363 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.363 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.364 DEBUG:boto:Method: GET 2022-09-02T04:30:05.365 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.365 DEBUG:boto:Data: 2022-09-02T04:30:05.366 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.367 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.367 DEBUG:boto:Params: {} 2022-09-02T04:30:05.368 DEBUG:boto:Token: None 2022-09-02T04:30:05.369 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/?versions 2022-09-02T04:30:05.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c= 2022-09-02T04:30:05.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c=', 'Content-Length': '0'} 2022-09-02T04:30:05.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ffca886297cde0-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.373 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.374 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:05.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:05.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:05.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:05.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:30:05.376 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.377 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.377 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.378 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.378 DEBUG:boto:Method: GET 2022-09-02T04:30:05.379 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.379 DEBUG:boto:Data: 2022-09-02T04:30:05.380 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.381 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.381 DEBUG:boto:Params: {} 2022-09-02T04:30:05.382 DEBUG:boto:Token: None 2022-09-02T04:30:05.383 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/?versions 2022-09-02T04:30:05.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c= 2022-09-02T04:30:05.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c=', 'Content-Length': '0'} 2022-09-02T04:30:05.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f5e2dde22b43ac5-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.387 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.388 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:05.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:05.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:05.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:05.390 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.391 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.391 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.392 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.392 DEBUG:boto:Method: GET 2022-09-02T04:30:05.393 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.393 DEBUG:boto:Data: 2022-09-02T04:30:05.394 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.395 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.395 DEBUG:boto:Params: {} 2022-09-02T04:30:05.396 DEBUG:boto:Token: None 2022-09-02T04:30:05.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/?versions 2022-09-02T04:30:05.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c= 2022-09-02T04:30:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c=', 'Content-Length': '0'} 2022-09-02T04:30:05.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e620138a6f4943b8-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.401 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.402 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.403 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.403 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.404 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.404 DEBUG:boto:Method: GET 2022-09-02T04:30:05.405 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:30:05.405 DEBUG:boto:Data: 2022-09-02T04:30:05.406 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.407 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.407 DEBUG:boto:Params: {} 2022-09-02T04:30:05.408 DEBUG:boto:Token: None 2022-09-02T04:30:05.408 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/?versions 2022-09-02T04:30:05.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c= 2022-09-02T04:30:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Da0fmP2rzGzRUbJ/Kx5c8TTVu0c=', 'Content-Length': '0'} 2022-09-02T04:30:05.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f813e365e3399df-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.415 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:05.418 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.421 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.422 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.423 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.424 DEBUG:boto:Method: GET 2022-09-02T04:30:05.425 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.426 DEBUG:boto:Data: 2022-09-02T04:30:05.427 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.428 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.429 DEBUG:boto:Params: {} 2022-09-02T04:30:05.430 DEBUG:boto:Token: None 2022-09-02T04:30:05.431 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg= 2022-09-02T04:30:05.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg=', 'Content-Length': '0'} 2022-09-02T04:30:05.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004825f5b047b6794-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.437 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.438 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.439 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.441 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.442 DEBUG:boto:Method: GET 2022-09-02T04:30:05.443 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.444 DEBUG:boto:Data: 2022-09-02T04:30:05.445 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.447 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.447 DEBUG:boto:Params: {} 2022-09-02T04:30:05.448 DEBUG:boto:Token: None 2022-09-02T04:30:05.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg= 2022-09-02T04:30:05.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg=', 'Content-Length': '0'} 2022-09-02T04:30:05.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd08934b07ddca5c-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.454 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.455 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.455 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.456 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.457 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.457 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.458 DEBUG:boto:Data: 2022-09-02T04:30:05.458 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.459 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.460 DEBUG:boto:Params: {} 2022-09-02T04:30:05.460 DEBUG:boto:Token: None 2022-09-02T04:30:05.461 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b1ahLp+W+ZH6vxf5HXV2jkRjPUA= 2022-09-02T04:30:05.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b1ahLp+W+ZH6vxf5HXV2jkRjPUA=', 'Content-Length': '0'} 2022-09-02T04:30:05.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8eef2b89b4a53e5-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.465 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.466 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.466 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.467 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.468 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.469 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.469 DEBUG:boto:Data: 2022-09-02T04:30:05.470 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.471 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.471 DEBUG:boto:Params: {} 2022-09-02T04:30:05.472 DEBUG:boto:Token: None 2022-09-02T04:30:05.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b1ahLp+W+ZH6vxf5HXV2jkRjPUA= 2022-09-02T04:30:05.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b1ahLp+W+ZH6vxf5HXV2jkRjPUA=', 'Content-Length': '0'} 2022-09-02T04:30:05.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000906ea496de373857-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:05.477 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.478 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.478 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.479 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.479 DEBUG:boto:Method: GET 2022-09-02T04:30:05.480 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.481 DEBUG:boto:Data: 2022-09-02T04:30:05.481 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.482 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.483 DEBUG:boto:Params: {} 2022-09-02T04:30:05.484 DEBUG:boto:Token: None 2022-09-02T04:30:05.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg= 2022-09-02T04:30:05.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg=', 'Content-Length': '0'} 2022-09-02T04:30:05.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097bde3ae0868417c-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.492 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.493 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.494 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.495 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.495 DEBUG:boto:Method: GET 2022-09-02T04:30:05.496 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.497 DEBUG:boto:Data: 2022-09-02T04:30:05.497 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.498 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.499 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.500 DEBUG:boto:Params: {} 2022-09-02T04:30:05.501 DEBUG:boto:Token: None 2022-09-02T04:30:05.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg= 2022-09-02T04:30:05.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg=', 'Content-Length': '0'} 2022-09-02T04:30:05.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fea097d34820f4f4-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.510 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.511 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.515 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.516 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.517 DEBUG:boto:Method: GET 2022-09-02T04:30:05.518 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.519 DEBUG:boto:Data: 2022-09-02T04:30:05.520 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.523 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.524 DEBUG:boto:Params: {} 2022-09-02T04:30:05.524 DEBUG:boto:Token: None 2022-09-02T04:30:05.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdcd548d6fa2afdf-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.574 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.576 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.576 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.577 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.578 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.578 DEBUG:boto:Method: GET 2022-09-02T04:30:05.579 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.580 DEBUG:boto:Data: 2022-09-02T04:30:05.580 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.581 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.582 DEBUG:boto:Params: {} 2022-09-02T04:30:05.582 DEBUG:boto:Token: None 2022-09-02T04:30:05.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029386ddb4a877fe1-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.588 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.590 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.591 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.592 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.593 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:30:05.593 DEBUG:boto:Data: 2022-09-02T04:30:05.594 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.596 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.598 DEBUG:boto:Params: {} 2022-09-02T04:30:05.599 DEBUG:boto:Token: None 2022-09-02T04:30:05.602 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy 2022-09-02T04:30:05.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HMZp6s8KIP22ARalzRoc+RyrGI8= 2022-09-02T04:30:05.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HMZp6s8KIP22ARalzRoc+RyrGI8=', 'Content-Length': '0'} 2022-09-02T04:30:05.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008aeaa4ff3a4488be-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.606 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.607 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.607 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.608 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:30:05.608 DEBUG:boto:Data: 2022-09-02T04:30:05.609 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.610 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.611 DEBUG:boto:Params: {} 2022-09-02T04:30:05.611 DEBUG:boto:Token: None 2022-09-02T04:30:05.613 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy 2022-09-02T04:30:05.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HMZp6s8KIP22ARalzRoc+RyrGI8= 2022-09-02T04:30:05.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HMZp6s8KIP22ARalzRoc+RyrGI8=', 'Content-Length': '0'} 2022-09-02T04:30:05.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c4b4ab694ff4a83-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:05.623 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.624 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.625 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.626 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.627 DEBUG:boto:Method: GET 2022-09-02T04:30:05.628 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.629 DEBUG:boto:Data: 2022-09-02T04:30:05.630 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.632 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.633 DEBUG:boto:Params: {} 2022-09-02T04:30:05.635 DEBUG:boto:Token: None 2022-09-02T04:30:05.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg= 2022-09-02T04:30:05.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg=', 'Content-Length': '0'} 2022-09-02T04:30:05.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3674184d26316b8-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.643 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:30:05.644 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:30:05.646 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.648 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.649 DEBUG:boto:Method: GET 2022-09-02T04:30:05.650 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.651 DEBUG:boto:Data: 2022-09-02T04:30:05.652 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.654 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.655 DEBUG:boto:Params: {} 2022-09-02T04:30:05.657 DEBUG:boto:Token: None 2022-09-02T04:30:05.658 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:30:05.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg= 2022-09-02T04:30:05.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/aQD7zif8aqpYVDU2M6R1vMAdg=', 'Content-Length': '0'} 2022-09-02T04:30:05.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c366ff54307be86-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.665 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.666 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.667 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.668 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.668 DEBUG:boto:Method: GET 2022-09-02T04:30:05.669 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.670 DEBUG:boto:Data: 2022-09-02T04:30:05.671 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.672 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.673 DEBUG:boto:Params: {} 2022-09-02T04:30:05.674 DEBUG:boto:Token: None 2022-09-02T04:30:05.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383bb28dcda10406-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.679 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.680 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.681 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.681 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.682 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.682 DEBUG:boto:Method: GET 2022-09-02T04:30:05.683 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:30:05.683 DEBUG:boto:Data: 2022-09-02T04:30:05.684 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.685 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.686 DEBUG:boto:Params: {} 2022-09-02T04:30:05.686 DEBUG:boto:Token: None 2022-09-02T04:30:05.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbae2966df8a7a79-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.691 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:05.693 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.693 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.694 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.694 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.695 DEBUG:boto:Method: GET 2022-09-02T04:30:05.696 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.696 DEBUG:boto:Data: 2022-09-02T04:30:05.697 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.699 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.699 DEBUG:boto:Params: {} 2022-09-02T04:30:05.701 DEBUG:boto:Token: None 2022-09-02T04:30:05.702 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U= 2022-09-02T04:30:05.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U=', 'Content-Length': '0'} 2022-09-02T04:30:05.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072bd3fef026d8605-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.707 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.708 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.708 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.709 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.709 DEBUG:boto:Method: GET 2022-09-02T04:30:05.710 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.710 DEBUG:boto:Data: 2022-09-02T04:30:05.713 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.715 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.716 DEBUG:boto:Params: {} 2022-09-02T04:30:05.717 DEBUG:boto:Token: None 2022-09-02T04:30:05.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U= 2022-09-02T04:30:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U=', 'Content-Length': '0'} 2022-09-02T04:30:05.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079e22d0c78703060-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.762 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.763 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.764 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.764 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.765 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.765 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.766 DEBUG:boto:Data: 2022-09-02T04:30:05.766 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.767 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.768 DEBUG:boto:Params: {} 2022-09-02T04:30:05.768 DEBUG:boto:Token: None 2022-09-02T04:30:05.769 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0RgITViS8kFOl15LZmQqsS7IwBo= 2022-09-02T04:30:05.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0RgITViS8kFOl15LZmQqsS7IwBo=', 'Content-Length': '0'} 2022-09-02T04:30:05.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa4c61a0500bd41e-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.776 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.776 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.777 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.778 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.778 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.779 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.779 DEBUG:boto:Data: 2022-09-02T04:30:05.780 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.781 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.782 DEBUG:boto:Params: {} 2022-09-02T04:30:05.783 DEBUG:boto:Token: None 2022-09-02T04:30:05.783 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0RgITViS8kFOl15LZmQqsS7IwBo= 2022-09-02T04:30:05.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0RgITViS8kFOl15LZmQqsS7IwBo=', 'Content-Length': '0'} 2022-09-02T04:30:05.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065dfa88665e4f978-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:05.789 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.790 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.791 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.791 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.792 DEBUG:boto:Method: GET 2022-09-02T04:30:05.793 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.793 DEBUG:boto:Data: 2022-09-02T04:30:05.794 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.795 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.795 DEBUG:boto:Params: {} 2022-09-02T04:30:05.796 DEBUG:boto:Token: None 2022-09-02T04:30:05.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U= 2022-09-02T04:30:05.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U=', 'Content-Length': '0'} 2022-09-02T04:30:05.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095c4820059563acf-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.801 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.802 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.802 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.803 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.803 DEBUG:boto:Method: GET 2022-09-02T04:30:05.804 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.805 DEBUG:boto:Data: 2022-09-02T04:30:05.805 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.807 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.807 DEBUG:boto:Params: {} 2022-09-02T04:30:05.808 DEBUG:boto:Token: None 2022-09-02T04:30:05.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U= 2022-09-02T04:30:05.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U=', 'Content-Length': '0'} 2022-09-02T04:30:05.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2440d581a834b9a-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.815 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.815 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.816 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.816 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.817 DEBUG:boto:Method: GET 2022-09-02T04:30:05.817 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.818 DEBUG:boto:Data: 2022-09-02T04:30:05.818 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.819 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.820 DEBUG:boto:Params: {} 2022-09-02T04:30:05.821 DEBUG:boto:Token: None 2022-09-02T04:30:05.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9eb8dbde7c5f60-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.830 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.831 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.831 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.832 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.832 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.833 DEBUG:boto:Method: GET 2022-09-02T04:30:05.833 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.834 DEBUG:boto:Data: 2022-09-02T04:30:05.835 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.836 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.836 DEBUG:boto:Params: {} 2022-09-02T04:30:05.837 DEBUG:boto:Token: None 2022-09-02T04:30:05.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048b591a36e0414f0-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.842 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.843 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.844 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.844 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.845 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.845 DEBUG:boto:Data: 2022-09-02T04:30:05.846 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.847 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.847 DEBUG:boto:Params: {} 2022-09-02T04:30:05.848 DEBUG:boto:Token: None 2022-09-02T04:30:05.849 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N1Jpo6zp3824weDBQ199Ls7JvUo= 2022-09-02T04:30:05.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N1Jpo6zp3824weDBQ199Ls7JvUo=', 'Content-Length': '0'} 2022-09-02T04:30:05.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb95d3e8e4c3508c-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.853 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.853 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.854 DEBUG:boto:Method: HEAD 2022-09-02T04:30:05.854 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.855 DEBUG:boto:Data: 2022-09-02T04:30:05.855 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.856 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.857 DEBUG:boto:Params: {} 2022-09-02T04:30:05.857 DEBUG:boto:Token: None 2022-09-02T04:30:05.858 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N1Jpo6zp3824weDBQ199Ls7JvUo= 2022-09-02T04:30:05.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N1Jpo6zp3824weDBQ199Ls7JvUo=', 'Content-Length': '0'} 2022-09-02T04:30:05.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb9bb222872c86cf-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:05.864 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.864 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.865 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.865 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.866 DEBUG:boto:Method: GET 2022-09-02T04:30:05.866 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.867 DEBUG:boto:Data: 2022-09-02T04:30:05.867 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.868 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.869 DEBUG:boto:Params: {} 2022-09-02T04:30:05.870 DEBUG:boto:Token: None 2022-09-02T04:30:05.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U= 2022-09-02T04:30:05.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U=', 'Content-Length': '0'} 2022-09-02T04:30:05.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f64c5e02465836f-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.922 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.923 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:05.924 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.924 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.925 DEBUG:boto:Method: GET 2022-09-02T04:30:05.925 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.926 DEBUG:boto:Data: 2022-09-02T04:30:05.926 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.932 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.932 DEBUG:boto:Params: {} 2022-09-02T04:30:05.933 DEBUG:boto:Token: None 2022-09-02T04:30:05.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:30:05.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U= 2022-09-02T04:30:05.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1vvOTtV/32YoU4SMIHTzcuXV74U=', 'Content-Length': '0'} 2022-09-02T04:30:05.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:01 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045e8265c00bd78e5-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.938 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.939 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.940 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.940 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.941 DEBUG:boto:Method: GET 2022-09-02T04:30:05.941 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.942 DEBUG:boto:Data: 2022-09-02T04:30:05.943 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.944 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.944 DEBUG:boto:Params: {} 2022-09-02T04:30:05.945 DEBUG:boto:Token: None 2022-09-02T04:30:05.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000396f8a15ade7ea92-00631186cd-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.950 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.951 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:05.951 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:05.952 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.952 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.953 DEBUG:boto:Method: GET 2022-09-02T04:30:05.954 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:30:05.954 DEBUG:boto:Data: 2022-09-02T04:30:05.955 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.956 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.956 DEBUG:boto:Params: {} 2022-09-02T04:30:05.957 DEBUG:boto:Token: None 2022-09-02T04:30:05.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/ 2022-09-02T04:30:05.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0= 2022-09-02T04:30:05.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZNdKdQqwO4QRcdZxRYc3X5bUP0=', 'Content-Length': '0'} 2022-09-02T04:30:05.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad6e9b8bbb9417e8-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.962 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:05.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:05.963 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.964 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.965 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.965 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.966 DEBUG:boto:Method: GET 2022-09-02T04:30:05.966 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.967 DEBUG:boto:Data: 2022-09-02T04:30:05.967 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:05.969 DEBUG:boto:Port: 8001 2022-09-02T04:30:05.969 DEBUG:boto:Params: {} 2022-09-02T04:30:05.970 DEBUG:boto:Token: None 2022-09-02T04:30:05.971 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA= 2022-09-02T04:30:05.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA=', 'Content-Length': '0'} 2022-09-02T04:30:05.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019bba223f435a673-00631186cd-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:05.978 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.985 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:05.985 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.986 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.987 DEBUG:boto:Method: GET 2022-09-02T04:30:05.988 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.988 DEBUG:boto:Data: 2022-09-02T04:30:05.989 DEBUG:boto:Headers: {} 2022-09-02T04:30:05.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:05.990 DEBUG:boto:Port: 8000 2022-09-02T04:30:05.991 DEBUG:boto:Params: {} 2022-09-02T04:30:05.991 DEBUG:boto:Token: None 2022-09-02T04:30:05.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:05 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:05.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA= 2022-09-02T04:30:05.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t96kZdlsX+PgS6gEvnn04P/W9wA=', 'Content-Length': '0'} 2022-09-02T04:30:05.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f7759670b3fde49-00631186cd-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.042 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.043 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.043 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.044 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.045 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.045 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.046 DEBUG:boto:Data: 2022-09-02T04:30:06.047 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.048 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.048 DEBUG:boto:Params: {} 2022-09-02T04:30:06.049 DEBUG:boto:Token: None 2022-09-02T04:30:06.049 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7QYEx718s2Cd+faC+dUz99FOSk= 2022-09-02T04:30:06.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7QYEx718s2Cd+faC+dUz99FOSk=', 'Content-Length': '0'} 2022-09-02T04:30:06.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f019f2ccaed720e-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.054 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.055 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.055 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.056 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.057 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.058 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.059 DEBUG:boto:Data: 2022-09-02T04:30:06.060 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.062 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.063 DEBUG:boto:Params: {} 2022-09-02T04:30:06.063 DEBUG:boto:Token: None 2022-09-02T04:30:06.064 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7QYEx718s2Cd+faC+dUz99FOSk= 2022-09-02T04:30:06.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7QYEx718s2Cd+faC+dUz99FOSk=', 'Content-Length': '0'} 2022-09-02T04:30:06.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ad5b3a2fcba545-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:06.071 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.072 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.072 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.073 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.073 DEBUG:boto:Method: GET 2022-09-02T04:30:06.074 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.075 DEBUG:boto:Data: 2022-09-02T04:30:06.076 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.077 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.078 DEBUG:boto:Params: {} 2022-09-02T04:30:06.080 DEBUG:boto:Token: None 2022-09-02T04:30:06.081 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY= 2022-09-02T04:30:06.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY=', 'Content-Length': '0'} 2022-09-02T04:30:06.087 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025375f6597d32d4f-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.088 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.088 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.089 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.089 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.090 DEBUG:boto:Method: GET 2022-09-02T04:30:06.090 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.091 DEBUG:boto:Data: 2022-09-02T04:30:06.091 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.093 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.093 DEBUG:boto:Params: {} 2022-09-02T04:30:06.094 DEBUG:boto:Token: None 2022-09-02T04:30:06.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY= 2022-09-02T04:30:06.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY=', 'Content-Length': '0'} 2022-09-02T04:30:06.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0c211ed47482797-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.099 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.099 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.100 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.100 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.101 DEBUG:boto:Method: GET 2022-09-02T04:30:06.101 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.102 DEBUG:boto:Data: 2022-09-02T04:30:06.103 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.104 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.104 DEBUG:boto:Params: {} 2022-09-02T04:30:06.105 DEBUG:boto:Token: None 2022-09-02T04:30:06.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13b6ff0857a262f-00631186ce-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.110 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.111 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.111 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.112 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.112 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.113 DEBUG:boto:Method: GET 2022-09-02T04:30:06.113 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.114 DEBUG:boto:Data: 2022-09-02T04:30:06.114 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.116 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.116 DEBUG:boto:Params: {} 2022-09-02T04:30:06.117 DEBUG:boto:Token: None 2022-09-02T04:30:06.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000aac77815705305-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.122 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.123 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.124 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.124 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.125 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.125 DEBUG:boto:Data: 2022-09-02T04:30:06.126 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.127 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.127 DEBUG:boto:Params: {} 2022-09-02T04:30:06.128 DEBUG:boto:Token: None 2022-09-02T04:30:06.129 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5mwsALspWlHKkoyn3/6tiMEC/7c= 2022-09-02T04:30:06.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5mwsALspWlHKkoyn3/6tiMEC/7c=', 'Content-Length': '0'} 2022-09-02T04:30:06.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ba0f4bef9c632da-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.133 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.133 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.134 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.134 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.135 DEBUG:boto:Data: 2022-09-02T04:30:06.135 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.136 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.137 DEBUG:boto:Params: {} 2022-09-02T04:30:06.137 DEBUG:boto:Token: None 2022-09-02T04:30:06.138 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5mwsALspWlHKkoyn3/6tiMEC/7c= 2022-09-02T04:30:06.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5mwsALspWlHKkoyn3/6tiMEC/7c=', 'Content-Length': '0'} 2022-09-02T04:30:06.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f726408484164372-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:06.143 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.143 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.144 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.145 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.145 DEBUG:boto:Method: GET 2022-09-02T04:30:06.146 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.146 DEBUG:boto:Data: 2022-09-02T04:30:06.147 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.148 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.148 DEBUG:boto:Params: {} 2022-09-02T04:30:06.149 DEBUG:boto:Token: None 2022-09-02T04:30:06.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY= 2022-09-02T04:30:06.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY=', 'Content-Length': '0'} 2022-09-02T04:30:06.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082a63e94d61665fa-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.154 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.154 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:06.155 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.155 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.156 DEBUG:boto:Method: GET 2022-09-02T04:30:06.156 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.157 DEBUG:boto:Data: 2022-09-02T04:30:06.157 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.158 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.159 DEBUG:boto:Params: {} 2022-09-02T04:30:06.159 DEBUG:boto:Token: None 2022-09-02T04:30:06.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:30:06.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY= 2022-09-02T04:30:06.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3VHRLnwvOE7zy2VgAq5sMWDlOY=', 'Content-Length': '0'} 2022-09-02T04:30:06.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cad6af6ffdd866ec-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.164 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.165 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.165 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.166 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.166 DEBUG:boto:Method: GET 2022-09-02T04:30:06.167 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.167 DEBUG:boto:Data: 2022-09-02T04:30:06.168 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.169 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.170 DEBUG:boto:Params: {} 2022-09-02T04:30:06.170 DEBUG:boto:Token: None 2022-09-02T04:30:06.171 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068271f0720314019-00631186ce-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.175 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.176 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.177 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.177 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.178 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.178 DEBUG:boto:Method: GET 2022-09-02T04:30:06.179 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:30:06.179 DEBUG:boto:Data: 2022-09-02T04:30:06.180 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.181 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.181 DEBUG:boto:Params: {} 2022-09-02T04:30:06.182 DEBUG:boto:Token: None 2022-09-02T04:30:06.182 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa3e29a0255f8dc-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.188 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:06.191 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.191 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.192 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.192 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.193 DEBUG:boto:Method: GET 2022-09-02T04:30:06.194 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.194 DEBUG:boto:Data: 2022-09-02T04:30:06.195 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.196 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.196 DEBUG:boto:Params: {} 2022-09-02T04:30:06.197 DEBUG:boto:Token: None 2022-09-02T04:30:06.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ= 2022-09-02T04:30:06.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ=', 'Content-Length': '0'} 2022-09-02T04:30:06.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023433dae9a00c768-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.246 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.247 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.248 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.249 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.251 DEBUG:boto:Method: GET 2022-09-02T04:30:06.252 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.253 DEBUG:boto:Data: 2022-09-02T04:30:06.254 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.256 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.257 DEBUG:boto:Params: {} 2022-09-02T04:30:06.258 DEBUG:boto:Token: None 2022-09-02T04:30:06.259 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ= 2022-09-02T04:30:06.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ=', 'Content-Length': '0'} 2022-09-02T04:30:06.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5fc6fb288240d14-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.265 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.265 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.266 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.267 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.267 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.268 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.269 DEBUG:boto:Data: 2022-09-02T04:30:06.270 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.272 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.273 DEBUG:boto:Params: {} 2022-09-02T04:30:06.274 DEBUG:boto:Token: None 2022-09-02T04:30:06.275 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RvoaqEij2OlEmOcEdieRoFv9OTI= 2022-09-02T04:30:06.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RvoaqEij2OlEmOcEdieRoFv9OTI=', 'Content-Length': '0'} 2022-09-02T04:30:06.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000137a316ea89e2b11-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.281 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.282 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.283 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.284 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.285 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.286 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.287 DEBUG:boto:Data: 2022-09-02T04:30:06.288 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.289 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.289 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.290 DEBUG:boto:Params: {} 2022-09-02T04:30:06.291 DEBUG:boto:Token: None 2022-09-02T04:30:06.292 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RvoaqEij2OlEmOcEdieRoFv9OTI= 2022-09-02T04:30:06.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RvoaqEij2OlEmOcEdieRoFv9OTI=', 'Content-Length': '0'} 2022-09-02T04:30:06.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003910812bf0e1227-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:06.298 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.299 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.299 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.300 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.301 DEBUG:boto:Method: GET 2022-09-02T04:30:06.301 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.302 DEBUG:boto:Data: 2022-09-02T04:30:06.302 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.303 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.304 DEBUG:boto:Params: {} 2022-09-02T04:30:06.304 DEBUG:boto:Token: None 2022-09-02T04:30:06.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ= 2022-09-02T04:30:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ=', 'Content-Length': '0'} 2022-09-02T04:30:06.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008171853a81ffc2ba-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.309 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.310 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.310 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.311 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.311 DEBUG:boto:Method: GET 2022-09-02T04:30:06.312 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.312 DEBUG:boto:Data: 2022-09-02T04:30:06.313 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.314 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.315 DEBUG:boto:Params: {} 2022-09-02T04:30:06.315 DEBUG:boto:Token: None 2022-09-02T04:30:06.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ= 2022-09-02T04:30:06.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ=', 'Content-Length': '0'} 2022-09-02T04:30:06.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224e0a9d59fe0edd-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.320 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.321 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.321 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.322 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.322 DEBUG:boto:Method: GET 2022-09-02T04:30:06.323 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.323 DEBUG:boto:Data: 2022-09-02T04:30:06.324 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.325 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.326 DEBUG:boto:Params: {} 2022-09-02T04:30:06.326 DEBUG:boto:Token: None 2022-09-02T04:30:06.327 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a667babd2ea8983-00631186ce-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.341 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.343 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.344 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.345 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.346 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.347 DEBUG:boto:Method: GET 2022-09-02T04:30:06.348 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.350 DEBUG:boto:Data: 2022-09-02T04:30:06.350 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.352 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.353 DEBUG:boto:Params: {} 2022-09-02T04:30:06.354 DEBUG:boto:Token: None 2022-09-02T04:30:06.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d4600dfe8b3299-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.366 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.368 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.369 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.370 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.371 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.372 DEBUG:boto:Data: 2022-09-02T04:30:06.373 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.374 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.375 DEBUG:boto:Params: {} 2022-09-02T04:30:06.376 DEBUG:boto:Token: None 2022-09-02T04:30:06.377 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SasXnfgVIHk5d7M+Xpi5HrqOWYg= 2022-09-02T04:30:06.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SasXnfgVIHk5d7M+Xpi5HrqOWYg=', 'Content-Length': '0'} 2022-09-02T04:30:06.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009920875168a83aab-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.383 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.384 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.385 DEBUG:boto:Method: HEAD 2022-09-02T04:30:06.385 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.386 DEBUG:boto:Data: 2022-09-02T04:30:06.387 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.388 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.388 DEBUG:boto:Params: {} 2022-09-02T04:30:06.389 DEBUG:boto:Token: None 2022-09-02T04:30:06.390 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SasXnfgVIHk5d7M+Xpi5HrqOWYg= 2022-09-02T04:30:06.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SasXnfgVIHk5d7M+Xpi5HrqOWYg=', 'Content-Length': '0'} 2022-09-02T04:30:06.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068c12c9821aa5fa1-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:06.400 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.401 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.402 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.403 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.404 DEBUG:boto:Method: GET 2022-09-02T04:30:06.405 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.406 DEBUG:boto:Data: 2022-09-02T04:30:06.407 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.408 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.409 DEBUG:boto:Params: {} 2022-09-02T04:30:06.410 DEBUG:boto:Token: None 2022-09-02T04:30:06.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ= 2022-09-02T04:30:06.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ=', 'Content-Length': '0'} 2022-09-02T04:30:06.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ed3170e6dd27d8d-00631186ce-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.417 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.417 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:06.418 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.418 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.419 DEBUG:boto:Method: GET 2022-09-02T04:30:06.420 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.421 DEBUG:boto:Data: 2022-09-02T04:30:06.422 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.424 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.425 DEBUG:boto:Params: {} 2022-09-02T04:30:06.426 DEBUG:boto:Token: None 2022-09-02T04:30:06.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:30:06.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ= 2022-09-02T04:30:06.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u4WV4Bzh/jJQWQQpIk1+iTWD7zQ=', 'Content-Length': '0'} 2022-09-02T04:30:06.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034445f9d18b1b34a-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.478 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.479 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.480 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.480 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.481 DEBUG:boto:Method: GET 2022-09-02T04:30:06.482 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.483 DEBUG:boto:Data: 2022-09-02T04:30:06.483 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:06.484 DEBUG:boto:Port: 8001 2022-09-02T04:30:06.485 DEBUG:boto:Params: {} 2022-09-02T04:30:06.486 DEBUG:boto:Token: None 2022-09-02T04:30:06.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb67c30b84fc8f3-00631186ce-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.492 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.492 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:06.493 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:06.494 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.495 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.495 DEBUG:boto:Method: GET 2022-09-02T04:30:06.496 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:30:06.496 DEBUG:boto:Data: 2022-09-02T04:30:06.497 DEBUG:boto:Headers: {} 2022-09-02T04:30:06.497 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:06.498 DEBUG:boto:Port: 8000 2022-09-02T04:30:06.499 DEBUG:boto:Params: {} 2022-09-02T04:30:06.501 DEBUG:boto:Token: None 2022-09-02T04:30:06.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:06 GMT /xljbkf-17/ 2022-09-02T04:30:06.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0= 2022-09-02T04:30:06.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzjDdmoGSblg1UCiX2+vuto54c0=', 'Content-Length': '0'} 2022-09-02T04:30:06.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000881c2722402c0cfb-00631186ce-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:06.508 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:06.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-17 zones={test-zone3, test-zone2} 2022-09-02T04:30:06.510 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-18 2022-09-02T04:30:06.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.512 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:06.576 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:06.577 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:06.621 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:06.620+0000 7f3833ef4600 1 bucket sync caught up with source: 2022-09-02T04:30:06.622 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.28.5, , 00000000003.37.9, , 00000000007.25.9, , , , , ] 2022-09-02T04:30:06.622 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.28.5, , 00000000003.37.9, , 00000000007.25.9, , , , , ] 2022-09-02T04:30:06.622 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:06.620+0000 7f3833ef4600 0 bucket checkpoint complete 2022-09-02T04:30:06.634 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-18 2022-09-02T04:30:06.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.638 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:06.701 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:06.702 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:06.738 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:06.737+0000 7f24e03d1600 1 bucket sync caught up with source: 2022-09-02T04:30:06.739 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000010.28.5, , 00000000003.37.9, , 00000000007.25.9, , , , , ] 2022-09-02T04:30:06.739 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000010.28.5, , 00000000003.37.9, , 00000000007.25.9, , , , , ] 2022-09-02T04:30:06.740 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:06.737+0000 7f24e03d1600 0 bucket checkpoint complete 2022-09-02T04:30:06.762 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-18 2022-09-02T04:30:06.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.764 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:06.828 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:06.829 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:06.875 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:06.874+0000 7fb53d1ee600 1 bucket sync caught up with source: 2022-09-02T04:30:06.875 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.28.5, , 00000000003.37.9, , 00000000007.25.9, , , , , ] 2022-09-02T04:30:06.876 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000010.28.5, , 00000000003.37.9, , 00000000007.25.9, , , , , ] 2022-09-02T04:30:06.876 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:06.874+0000 7fb53d1ee600 0 bucket checkpoint complete 2022-09-02T04:30:06.888 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-18 2022-09-02T04:30:06.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:06.890 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:06.954 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:07.013 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:07.013+0000 7f1c84089600 1 bucket sync caught up with source: 2022-09-02T04:30:07.014 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.30.9, , 00000000010.62.5, , 00000000007.41.9, , , , , ] 2022-09-02T04:30:07.014 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.30.9, , 00000000010.62.5, , 00000000007.41.9, , , , , ] 2022-09-02T04:30:07.015 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:07.013+0000 7f1c84089600 0 bucket checkpoint complete 2022-09-02T04:30:07.037 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-18 2022-09-02T04:30:07.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:07.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:07.041 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:07.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:07.099 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:07.144 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:07.143+0000 7ff17d700600 1 bucket sync caught up with source: 2022-09-02T04:30:07.145 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.30.9, , 00000000010.62.5, , 00000000007.41.9, , , , , ] 2022-09-02T04:30:07.146 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.30.9, , 00000000010.62.5, , 00000000007.41.9, , , , , ] 2022-09-02T04:30:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:07.143+0000 7ff17d700600 0 bucket checkpoint complete 2022-09-02T04:30:07.155 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-18 2022-09-02T04:30:07.157 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-18 2022-09-02T04:30:07.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:07.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:07.159 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:07.216 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:07.217 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:07.252 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:07.250+0000 7f4535ffd600 1 bucket sync caught up with source: 2022-09-02T04:30:07.253 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.20.9, , 00000000003.38.9, , 00000000015.53.5, , , , , ] 2022-09-02T04:30:07.254 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.20.9, , 00000000003.38.9, , 00000000015.53.5, , , , , ] 2022-09-02T04:30:07.254 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:07.250+0000 7f4535ffd600 0 bucket checkpoint complete 2022-09-02T04:30:07.267 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-18 2022-09-02T04:30:07.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:07.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:07.271 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:07.331 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:07.332 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:07.365+0000 7f43514a6600 1 bucket sync caught up with source: 2022-09-02T04:30:07.366 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.20.9, , 00000000003.38.9, , 00000000015.53.5, , , , , ] 2022-09-02T04:30:07.367 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.20.9, , 00000000003.38.9, , 00000000015.53.5, , , , , ] 2022-09-02T04:30:07.367 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:07.365+0000 7f43514a6600 0 bucket checkpoint complete 2022-09-02T04:30:07.378 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-18 2022-09-02T04:30:07.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-18 zones={test-zone2, test-zone1} 2022-09-02T04:30:07.380 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.381 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.382 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.383 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:07.384 DEBUG:boto:Data: 2022-09-02T04:30:07.385 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.386 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.387 DEBUG:boto:Params: {} 2022-09-02T04:30:07.388 DEBUG:boto:Token: None 2022-09-02T04:30:07.389 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zG2cz/C5GI5FqCs2hBhpiB+XWo4= 2022-09-02T04:30:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zG2cz/C5GI5FqCs2hBhpiB+XWo4=', 'Content-Length': '0'} 2022-09-02T04:30:07.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018e7006767fb6f9c-00631186cf-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.401 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.403 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.405 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.405 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:07.406 DEBUG:boto:Data: 2022-09-02T04:30:07.406 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.407 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.408 DEBUG:boto:Params: {} 2022-09-02T04:30:07.410 DEBUG:boto:Token: None 2022-09-02T04:30:07.410 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zG2cz/C5GI5FqCs2hBhpiB+XWo4= 2022-09-02T04:30:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zG2cz/C5GI5FqCs2hBhpiB+XWo4=', 'Content-Length': '0'} 2022-09-02T04:30:07.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d42abd1b33232691-00631186cf-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.416 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:30:07.417 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.417 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.418 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.418 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.419 DEBUG:boto:Method: GET 2022-09-02T04:30:07.419 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.420 DEBUG:boto:Data: 2022-09-02T04:30:07.420 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.421 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.422 DEBUG:boto:Params: {} 2022-09-02T04:30:07.423 DEBUG:boto:Token: None 2022-09-02T04:30:07.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/?versions 2022-09-02T04:30:07.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ= 2022-09-02T04:30:07.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ=', 'Content-Length': '0'} 2022-09-02T04:30:07.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a430cae76df19160-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.428 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.429 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:07.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:07.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:07.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:07.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:30:07.431 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.432 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.432 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.433 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.433 DEBUG:boto:Method: GET 2022-09-02T04:30:07.434 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.434 DEBUG:boto:Data: 2022-09-02T04:30:07.437 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.442 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.443 DEBUG:boto:Params: {} 2022-09-02T04:30:07.444 DEBUG:boto:Token: None 2022-09-02T04:30:07.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/?versions 2022-09-02T04:30:07.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ= 2022-09-02T04:30:07.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ=', 'Content-Length': '0'} 2022-09-02T04:30:07.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc10339e271bd569-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.451 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.452 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:07.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:07.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:07.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:07.455 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.455 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.456 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.456 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.457 DEBUG:boto:Method: GET 2022-09-02T04:30:07.457 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.458 DEBUG:boto:Data: 2022-09-02T04:30:07.459 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.461 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.462 DEBUG:boto:Params: {} 2022-09-02T04:30:07.463 DEBUG:boto:Token: None 2022-09-02T04:30:07.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/?versions 2022-09-02T04:30:07.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ= 2022-09-02T04:30:07.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ=', 'Content-Length': '0'} 2022-09-02T04:30:07.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f78aaf6985f9b65-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.471 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.472 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.474 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.475 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.476 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.477 DEBUG:boto:Method: GET 2022-09-02T04:30:07.479 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:07.480 DEBUG:boto:Data: 2022-09-02T04:30:07.481 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.483 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.483 DEBUG:boto:Params: {} 2022-09-02T04:30:07.484 DEBUG:boto:Token: None 2022-09-02T04:30:07.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/?versions 2022-09-02T04:30:07.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ= 2022-09-02T04:30:07.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JTqrQHWz1hZemgk2LDsCC+w3UuQ=', 'Content-Length': '0'} 2022-09-02T04:30:07.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036879a8a274846c4-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.491 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:07.494 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.495 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.496 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.497 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.498 DEBUG:boto:Method: GET 2022-09-02T04:30:07.498 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.499 DEBUG:boto:Data: 2022-09-02T04:30:07.499 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.500 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.500 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.501 DEBUG:boto:Params: {} 2022-09-02T04:30:07.502 DEBUG:boto:Token: None 2022-09-02T04:30:07.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4= 2022-09-02T04:30:07.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4=', 'Content-Length': '0'} 2022-09-02T04:30:07.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b8b8c9dbf4b8b84-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.506 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.507 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.508 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.508 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.509 DEBUG:boto:Method: GET 2022-09-02T04:30:07.509 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.510 DEBUG:boto:Data: 2022-09-02T04:30:07.510 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.511 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.512 DEBUG:boto:Params: {} 2022-09-02T04:30:07.513 DEBUG:boto:Token: None 2022-09-02T04:30:07.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4= 2022-09-02T04:30:07.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4=', 'Content-Length': '0'} 2022-09-02T04:30:07.517 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bcad538b3bfe6c3-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.518 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.518 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.519 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.519 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.520 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.520 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.521 DEBUG:boto:Data: 2022-09-02T04:30:07.521 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.522 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.523 DEBUG:boto:Params: {} 2022-09-02T04:30:07.523 DEBUG:boto:Token: None 2022-09-02T04:30:07.524 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:57gXQfJ4LkQUPrArHq3Q+iW+J+Q= 2022-09-02T04:30:07.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:57gXQfJ4LkQUPrArHq3Q+iW+J+Q=', 'Content-Length': '0'} 2022-09-02T04:30:07.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005775369142cf9add-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.528 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.529 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.530 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.531 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.532 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.533 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.535 DEBUG:boto:Data: 2022-09-02T04:30:07.537 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.538 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.539 DEBUG:boto:Params: {} 2022-09-02T04:30:07.539 DEBUG:boto:Token: None 2022-09-02T04:30:07.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:57gXQfJ4LkQUPrArHq3Q+iW+J+Q= 2022-09-02T04:30:07.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:57gXQfJ4LkQUPrArHq3Q+iW+J+Q=', 'Content-Length': '0'} 2022-09-02T04:30:07.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000200d4a4d90ddc713-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:07.545 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.547 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.548 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.549 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.550 DEBUG:boto:Method: GET 2022-09-02T04:30:07.551 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.552 DEBUG:boto:Data: 2022-09-02T04:30:07.554 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.555 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.556 DEBUG:boto:Params: {} 2022-09-02T04:30:07.557 DEBUG:boto:Token: None 2022-09-02T04:30:07.558 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4= 2022-09-02T04:30:07.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4=', 'Content-Length': '0'} 2022-09-02T04:30:07.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082df90c3bbb634cb-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.562 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.563 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.563 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.564 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.565 DEBUG:boto:Method: GET 2022-09-02T04:30:07.567 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.568 DEBUG:boto:Data: 2022-09-02T04:30:07.569 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.571 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.572 DEBUG:boto:Params: {} 2022-09-02T04:30:07.574 DEBUG:boto:Token: None 2022-09-02T04:30:07.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4= 2022-09-02T04:30:07.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4=', 'Content-Length': '0'} 2022-09-02T04:30:07.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049b389d1ad6d5d40-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.582 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.583 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.584 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.585 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.587 DEBUG:boto:Method: GET 2022-09-02T04:30:07.593 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.593 DEBUG:boto:Data: 2022-09-02T04:30:07.594 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.595 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.596 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.597 DEBUG:boto:Params: {} 2022-09-02T04:30:07.598 DEBUG:boto:Token: None 2022-09-02T04:30:07.599 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000334875e367463629-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.604 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.605 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.606 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.607 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.608 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.608 DEBUG:boto:Method: GET 2022-09-02T04:30:07.609 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.610 DEBUG:boto:Data: 2022-09-02T04:30:07.611 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.612 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.613 DEBUG:boto:Params: {} 2022-09-02T04:30:07.614 DEBUG:boto:Token: None 2022-09-02T04:30:07.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004980c87725d02963-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.622 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.623 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.625 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.626 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.627 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:30:07.628 DEBUG:boto:Data: 2022-09-02T04:30:07.629 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.630 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.631 DEBUG:boto:Params: {} 2022-09-02T04:30:07.633 DEBUG:boto:Token: None 2022-09-02T04:30:07.634 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy 2022-09-02T04:30:07.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6abeRvfLglOlqPGQV2LU0OclPh0= 2022-09-02T04:30:07.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6abeRvfLglOlqPGQV2LU0OclPh0=', 'Content-Length': '0'} 2022-09-02T04:30:07.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079f8a6abff2bf712-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.641 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.642 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.643 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.644 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:30:07.645 DEBUG:boto:Data: 2022-09-02T04:30:07.646 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.647 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.648 DEBUG:boto:Params: {} 2022-09-02T04:30:07.649 DEBUG:boto:Token: None 2022-09-02T04:30:07.650 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy 2022-09-02T04:30:07.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6abeRvfLglOlqPGQV2LU0OclPh0= 2022-09-02T04:30:07.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6abeRvfLglOlqPGQV2LU0OclPh0=', 'Content-Length': '0'} 2022-09-02T04:30:07.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c24fea5b100fcea-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:07.656 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.657 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.658 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.659 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.660 DEBUG:boto:Method: GET 2022-09-02T04:30:07.661 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.662 DEBUG:boto:Data: 2022-09-02T04:30:07.663 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.665 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.666 DEBUG:boto:Params: {} 2022-09-02T04:30:07.667 DEBUG:boto:Token: None 2022-09-02T04:30:07.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4= 2022-09-02T04:30:07.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4=', 'Content-Length': '0'} 2022-09-02T04:30:07.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000037c7271d0895e1e4-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.674 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:07.675 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:07.675 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.676 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.677 DEBUG:boto:Method: GET 2022-09-02T04:30:07.678 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.679 DEBUG:boto:Data: 2022-09-02T04:30:07.680 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.681 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.682 DEBUG:boto:Params: {} 2022-09-02T04:30:07.682 DEBUG:boto:Token: None 2022-09-02T04:30:07.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:07.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4= 2022-09-02T04:30:07.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bhUd0TgtV9PTBgExHRIuzAUvS4=', 'Content-Length': '0'} 2022-09-02T04:30:07.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000635f3f5276f13385-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.687 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.688 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.688 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.689 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.689 DEBUG:boto:Method: GET 2022-09-02T04:30:07.691 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.692 DEBUG:boto:Data: 2022-09-02T04:30:07.692 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.693 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.694 DEBUG:boto:Params: {} 2022-09-02T04:30:07.695 DEBUG:boto:Token: None 2022-09-02T04:30:07.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d09c478fc5cec618-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.700 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.702 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.703 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.704 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.705 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.705 DEBUG:boto:Method: GET 2022-09-02T04:30:07.706 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:07.707 DEBUG:boto:Data: 2022-09-02T04:30:07.707 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.709 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.710 DEBUG:boto:Params: {} 2022-09-02T04:30:07.711 DEBUG:boto:Token: None 2022-09-02T04:30:07.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d92894f28b0396-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.758 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:07.760 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.760 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.761 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.761 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.762 DEBUG:boto:Method: GET 2022-09-02T04:30:07.762 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.763 DEBUG:boto:Data: 2022-09-02T04:30:07.763 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.764 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.765 DEBUG:boto:Params: {} 2022-09-02T04:30:07.766 DEBUG:boto:Token: None 2022-09-02T04:30:07.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8= 2022-09-02T04:30:07.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8=', 'Content-Length': '0'} 2022-09-02T04:30:07.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093a42a5e2f24bc27-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.772 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.773 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.774 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.776 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.777 DEBUG:boto:Method: GET 2022-09-02T04:30:07.778 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.779 DEBUG:boto:Data: 2022-09-02T04:30:07.780 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.782 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.783 DEBUG:boto:Params: {} 2022-09-02T04:30:07.784 DEBUG:boto:Token: None 2022-09-02T04:30:07.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8= 2022-09-02T04:30:07.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8=', 'Content-Length': '0'} 2022-09-02T04:30:07.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000108aeacfde50a2be-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.791 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.792 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.793 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.794 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.795 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.796 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.797 DEBUG:boto:Data: 2022-09-02T04:30:07.798 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.800 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.801 DEBUG:boto:Params: {} 2022-09-02T04:30:07.802 DEBUG:boto:Token: None 2022-09-02T04:30:07.803 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xAeu3qVpH08wap19DMrkSA59JfI= 2022-09-02T04:30:07.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xAeu3qVpH08wap19DMrkSA59JfI=', 'Content-Length': '0'} 2022-09-02T04:30:07.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b81cbe056d8e8821-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.809 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.811 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.812 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.814 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.815 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.816 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.817 DEBUG:boto:Data: 2022-09-02T04:30:07.817 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.819 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.820 DEBUG:boto:Params: {} 2022-09-02T04:30:07.824 DEBUG:boto:Token: None 2022-09-02T04:30:07.825 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xAeu3qVpH08wap19DMrkSA59JfI= 2022-09-02T04:30:07.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xAeu3qVpH08wap19DMrkSA59JfI=', 'Content-Length': '0'} 2022-09-02T04:30:07.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094b43aa5b61e4a0f-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:07.832 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.833 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.834 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.835 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.835 DEBUG:boto:Method: GET 2022-09-02T04:30:07.836 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.836 DEBUG:boto:Data: 2022-09-02T04:30:07.837 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.838 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.839 DEBUG:boto:Params: {} 2022-09-02T04:30:07.840 DEBUG:boto:Token: None 2022-09-02T04:30:07.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8= 2022-09-02T04:30:07.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8=', 'Content-Length': '0'} 2022-09-02T04:30:07.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f834e194cebc44cd-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.846 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.847 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.848 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.849 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.850 DEBUG:boto:Method: GET 2022-09-02T04:30:07.851 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.852 DEBUG:boto:Data: 2022-09-02T04:30:07.853 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.854 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.855 DEBUG:boto:Params: {} 2022-09-02T04:30:07.857 DEBUG:boto:Token: None 2022-09-02T04:30:07.858 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8= 2022-09-02T04:30:07.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8=', 'Content-Length': '0'} 2022-09-02T04:30:07.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000515d01f11ce23cb7-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.862 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.863 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.863 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.864 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.865 DEBUG:boto:Method: GET 2022-09-02T04:30:07.866 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.866 DEBUG:boto:Data: 2022-09-02T04:30:07.867 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.868 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.868 DEBUG:boto:Params: {} 2022-09-02T04:30:07.869 DEBUG:boto:Token: None 2022-09-02T04:30:07.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d1df09c9cc9ff07-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.877 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.878 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.879 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.880 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.881 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.881 DEBUG:boto:Method: GET 2022-09-02T04:30:07.882 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.883 DEBUG:boto:Data: 2022-09-02T04:30:07.883 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.884 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.885 DEBUG:boto:Params: {} 2022-09-02T04:30:07.885 DEBUG:boto:Token: None 2022-09-02T04:30:07.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a9ef9556b8c265a-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.890 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.891 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.892 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.892 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.893 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.893 DEBUG:boto:Data: 2022-09-02T04:30:07.894 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.895 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.895 DEBUG:boto:Params: {} 2022-09-02T04:30:07.896 DEBUG:boto:Token: None 2022-09-02T04:30:07.896 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wDhCYR4mWjRbyMuICoMJQ3NWznQ= 2022-09-02T04:30:07.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wDhCYR4mWjRbyMuICoMJQ3NWznQ=', 'Content-Length': '0'} 2022-09-02T04:30:07.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049b91efe0a6041a3-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.900 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.901 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.901 DEBUG:boto:Method: HEAD 2022-09-02T04:30:07.902 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.903 DEBUG:boto:Data: 2022-09-02T04:30:07.903 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.904 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.905 DEBUG:boto:Params: {} 2022-09-02T04:30:07.905 DEBUG:boto:Token: None 2022-09-02T04:30:07.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wDhCYR4mWjRbyMuICoMJQ3NWznQ= 2022-09-02T04:30:07.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wDhCYR4mWjRbyMuICoMJQ3NWznQ=', 'Content-Length': '0'} 2022-09-02T04:30:07.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4090e232cddcb91-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:07.911 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.911 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.912 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.912 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.913 DEBUG:boto:Method: GET 2022-09-02T04:30:07.914 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.914 DEBUG:boto:Data: 2022-09-02T04:30:07.915 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.916 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.916 DEBUG:boto:Params: {} 2022-09-02T04:30:07.917 DEBUG:boto:Token: None 2022-09-02T04:30:07.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8= 2022-09-02T04:30:07.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8=', 'Content-Length': '0'} 2022-09-02T04:30:07.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8bb9fcc6905a7e2-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.922 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.922 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:07.923 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.923 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.924 DEBUG:boto:Method: GET 2022-09-02T04:30:07.924 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.925 DEBUG:boto:Data: 2022-09-02T04:30:07.925 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.927 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.927 DEBUG:boto:Params: {} 2022-09-02T04:30:07.928 DEBUG:boto:Token: None 2022-09-02T04:30:07.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:07.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8= 2022-09-02T04:30:07.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Upj0maRm+W0Jj59HuUKTS2NaxX8=', 'Content-Length': '0'} 2022-09-02T04:30:07.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f20f1227aea94057-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.932 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.933 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.934 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.934 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.935 DEBUG:boto:Method: GET 2022-09-02T04:30:07.935 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.936 DEBUG:boto:Data: 2022-09-02T04:30:07.936 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.937 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.938 DEBUG:boto:Params: {} 2022-09-02T04:30:07.938 DEBUG:boto:Token: None 2022-09-02T04:30:07.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e350b161801be5fc-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.943 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.944 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:07.945 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:07.945 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.947 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.948 DEBUG:boto:Method: GET 2022-09-02T04:30:07.948 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:07.949 DEBUG:boto:Data: 2022-09-02T04:30:07.949 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:07.950 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.951 DEBUG:boto:Params: {} 2022-09-02T04:30:07.952 DEBUG:boto:Token: None 2022-09-02T04:30:07.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/ 2022-09-02T04:30:07.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A= 2022-09-02T04:30:07.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rYCXhpMUiTwcwU3ISnPE2Uvf/0A=', 'Content-Length': '0'} 2022-09-02T04:30:07.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9ff71c57b47581-00631186cf-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:07.957 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:07.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:07.958 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:07.959 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:07.959 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:07.960 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:07.960 DEBUG:boto:Method: GET 2022-09-02T04:30:07.961 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:07.961 DEBUG:boto:Data: 2022-09-02T04:30:07.962 DEBUG:boto:Headers: {} 2022-09-02T04:30:07.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:07.963 DEBUG:boto:Port: 8000 2022-09-02T04:30:07.964 DEBUG:boto:Params: {} 2022-09-02T04:30:07.964 DEBUG:boto:Token: None 2022-09-02T04:30:07.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:07 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:07.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XbyuYTsx+XIO5fTyqqV7bOe/0qo= 2022-09-02T04:30:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XbyuYTsx+XIO5fTyqqV7bOe/0qo=', 'Content-Length': '0'} 2022-09-02T04:30:07.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000562f42cc18d911f7-00631186cf-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.010 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.011 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.012 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.012 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.013 DEBUG:boto:Method: GET 2022-09-02T04:30:08.013 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.014 DEBUG:boto:Data: 2022-09-02T04:30:08.014 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.016 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.016 DEBUG:boto:Params: {} 2022-09-02T04:30:08.017 DEBUG:boto:Token: None 2022-09-02T04:30:08.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc= 2022-09-02T04:30:08.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc=', 'Content-Length': '0'} 2022-09-02T04:30:08.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eee319cac3dbdb34-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.022 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.022 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.023 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.023 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.024 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.025 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.025 DEBUG:boto:Data: 2022-09-02T04:30:08.026 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.027 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.027 DEBUG:boto:Params: {} 2022-09-02T04:30:08.028 DEBUG:boto:Token: None 2022-09-02T04:30:08.028 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M8Bt+BMRlCu0WAv6M9QfHhZHoIo= 2022-09-02T04:30:08.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M8Bt+BMRlCu0WAv6M9QfHhZHoIo=', 'Content-Length': '0'} 2022-09-02T04:30:08.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017ce479ff047ee51-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.032 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.033 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.033 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.034 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.034 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.035 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.036 DEBUG:boto:Data: 2022-09-02T04:30:08.036 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.037 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.038 DEBUG:boto:Params: {} 2022-09-02T04:30:08.038 DEBUG:boto:Token: None 2022-09-02T04:30:08.039 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M8Bt+BMRlCu0WAv6M9QfHhZHoIo= 2022-09-02T04:30:08.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M8Bt+BMRlCu0WAv6M9QfHhZHoIo=', 'Content-Length': '0'} 2022-09-02T04:30:08.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5fb7e3d6c4f51d8-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:08.043 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.044 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.044 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.045 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.045 DEBUG:boto:Method: GET 2022-09-02T04:30:08.046 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.047 DEBUG:boto:Data: 2022-09-02T04:30:08.047 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.048 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.049 DEBUG:boto:Params: {} 2022-09-02T04:30:08.049 DEBUG:boto:Token: None 2022-09-02T04:30:08.050 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc= 2022-09-02T04:30:08.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc=', 'Content-Length': '0'} 2022-09-02T04:30:08.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c5db99d6914013e-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.054 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.055 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.055 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.056 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.056 DEBUG:boto:Method: GET 2022-09-02T04:30:08.057 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.058 DEBUG:boto:Data: 2022-09-02T04:30:08.058 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.059 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.060 DEBUG:boto:Params: {} 2022-09-02T04:30:08.061 DEBUG:boto:Token: None 2022-09-02T04:30:08.061 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc= 2022-09-02T04:30:08.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc=', 'Content-Length': '0'} 2022-09-02T04:30:08.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed9615fe7aa75ce7-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.066 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.066 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.068 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.069 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.069 DEBUG:boto:Method: GET 2022-09-02T04:30:08.070 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.070 DEBUG:boto:Data: 2022-09-02T04:30:08.071 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.073 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.073 DEBUG:boto:Params: {} 2022-09-02T04:30:08.074 DEBUG:boto:Token: None 2022-09-02T04:30:08.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e675db8a6d6d1410-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.080 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.080 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.081 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.082 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.082 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.083 DEBUG:boto:Method: GET 2022-09-02T04:30:08.083 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.084 DEBUG:boto:Data: 2022-09-02T04:30:08.084 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.085 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.086 DEBUG:boto:Params: {} 2022-09-02T04:30:08.087 DEBUG:boto:Token: None 2022-09-02T04:30:08.087 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba9fa263a25c9b8-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.092 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.093 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.093 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.094 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.094 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.095 DEBUG:boto:Data: 2022-09-02T04:30:08.095 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.096 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.097 DEBUG:boto:Params: {} 2022-09-02T04:30:08.097 DEBUG:boto:Token: None 2022-09-02T04:30:08.098 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eOaIcnXiUohK4jAu4K+NCw+TNTk= 2022-09-02T04:30:08.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eOaIcnXiUohK4jAu4K+NCw+TNTk=', 'Content-Length': '0'} 2022-09-02T04:30:08.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b70faa2feea9c62-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.102 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.102 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.103 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.104 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.104 DEBUG:boto:Data: 2022-09-02T04:30:08.104 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.105 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.106 DEBUG:boto:Params: {} 2022-09-02T04:30:08.107 DEBUG:boto:Token: None 2022-09-02T04:30:08.107 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eOaIcnXiUohK4jAu4K+NCw+TNTk= 2022-09-02T04:30:08.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eOaIcnXiUohK4jAu4K+NCw+TNTk=', 'Content-Length': '0'} 2022-09-02T04:30:08.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001360903c03b8bade-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:08.112 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.112 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.113 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.113 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.114 DEBUG:boto:Method: GET 2022-09-02T04:30:08.115 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.115 DEBUG:boto:Data: 2022-09-02T04:30:08.116 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.117 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.117 DEBUG:boto:Params: {} 2022-09-02T04:30:08.118 DEBUG:boto:Token: None 2022-09-02T04:30:08.119 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc= 2022-09-02T04:30:08.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc=', 'Content-Length': '0'} 2022-09-02T04:30:08.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000173eff9ef5049a74-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.123 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.123 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:08.124 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.124 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.125 DEBUG:boto:Method: GET 2022-09-02T04:30:08.125 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.126 DEBUG:boto:Data: 2022-09-02T04:30:08.127 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.128 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.128 DEBUG:boto:Params: {} 2022-09-02T04:30:08.129 DEBUG:boto:Token: None 2022-09-02T04:30:08.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:08.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc= 2022-09-02T04:30:08.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4EIpzRHhfbO4Cy0yObeszbuxmZc=', 'Content-Length': '0'} 2022-09-02T04:30:08.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ad52133119a77fa-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.174 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.175 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.175 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.176 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.176 DEBUG:boto:Method: GET 2022-09-02T04:30:08.177 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.178 DEBUG:boto:Data: 2022-09-02T04:30:08.178 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.179 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.180 DEBUG:boto:Params: {} 2022-09-02T04:30:08.180 DEBUG:boto:Token: None 2022-09-02T04:30:08.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a6f763e6dbae8cd-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.185 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.186 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.187 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.188 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.188 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.189 DEBUG:boto:Method: GET 2022-09-02T04:30:08.189 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:08.190 DEBUG:boto:Data: 2022-09-02T04:30:08.190 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.191 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.192 DEBUG:boto:Params: {} 2022-09-02T04:30:08.192 DEBUG:boto:Token: None 2022-09-02T04:30:08.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027bda7be09a962fb-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.198 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:08.199 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.200 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.200 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.201 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.202 DEBUG:boto:Method: GET 2022-09-02T04:30:08.203 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.204 DEBUG:boto:Data: 2022-09-02T04:30:08.205 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.207 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.208 DEBUG:boto:Params: {} 2022-09-02T04:30:08.208 DEBUG:boto:Token: None 2022-09-02T04:30:08.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA= 2022-09-02T04:30:08.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA=', 'Content-Length': '0'} 2022-09-02T04:30:08.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f04c905f24062114-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.214 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.215 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.215 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.216 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.217 DEBUG:boto:Method: GET 2022-09-02T04:30:08.217 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.218 DEBUG:boto:Data: 2022-09-02T04:30:08.219 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.220 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.221 DEBUG:boto:Params: {} 2022-09-02T04:30:08.221 DEBUG:boto:Token: None 2022-09-02T04:30:08.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA= 2022-09-02T04:30:08.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA=', 'Content-Length': '0'} 2022-09-02T04:30:08.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b1287bc6ed2680f-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.226 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.227 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.227 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.228 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.228 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.229 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.229 DEBUG:boto:Data: 2022-09-02T04:30:08.230 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.231 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.231 DEBUG:boto:Params: {} 2022-09-02T04:30:08.232 DEBUG:boto:Token: None 2022-09-02T04:30:08.233 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3OxKjC0/B0f2KXOM2ySRVqXLMCI= 2022-09-02T04:30:08.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3OxKjC0/B0f2KXOM2ySRVqXLMCI=', 'Content-Length': '0'} 2022-09-02T04:30:08.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa3cb01f43ad81c4-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.236 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.237 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.238 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.238 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.239 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.239 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.240 DEBUG:boto:Data: 2022-09-02T04:30:08.240 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.241 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.242 DEBUG:boto:Params: {} 2022-09-02T04:30:08.242 DEBUG:boto:Token: None 2022-09-02T04:30:08.243 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3OxKjC0/B0f2KXOM2ySRVqXLMCI= 2022-09-02T04:30:08.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3OxKjC0/B0f2KXOM2ySRVqXLMCI=', 'Content-Length': '0'} 2022-09-02T04:30:08.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce9917c58f6e53fe-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:08.248 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.248 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.249 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.249 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.250 DEBUG:boto:Method: GET 2022-09-02T04:30:08.250 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.251 DEBUG:boto:Data: 2022-09-02T04:30:08.251 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.253 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.253 DEBUG:boto:Params: {} 2022-09-02T04:30:08.254 DEBUG:boto:Token: None 2022-09-02T04:30:08.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA= 2022-09-02T04:30:08.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA=', 'Content-Length': '0'} 2022-09-02T04:30:08.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056088f843feba8a5-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.259 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.259 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.260 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.260 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.261 DEBUG:boto:Method: GET 2022-09-02T04:30:08.262 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.262 DEBUG:boto:Data: 2022-09-02T04:30:08.263 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.264 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.264 DEBUG:boto:Params: {} 2022-09-02T04:30:08.265 DEBUG:boto:Token: None 2022-09-02T04:30:08.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA= 2022-09-02T04:30:08.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA=', 'Content-Length': '0'} 2022-09-02T04:30:08.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b58583604ad522c7-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.270 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.270 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.271 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.271 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.272 DEBUG:boto:Method: GET 2022-09-02T04:30:08.273 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.273 DEBUG:boto:Data: 2022-09-02T04:30:08.274 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.275 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.275 DEBUG:boto:Params: {} 2022-09-02T04:30:08.276 DEBUG:boto:Token: None 2022-09-02T04:30:08.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a509e7c7c67acca4-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.326 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.328 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.328 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.329 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.329 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.330 DEBUG:boto:Method: GET 2022-09-02T04:30:08.330 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.331 DEBUG:boto:Data: 2022-09-02T04:30:08.331 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.333 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.335 DEBUG:boto:Params: {} 2022-09-02T04:30:08.335 DEBUG:boto:Token: None 2022-09-02T04:30:08.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e1fc47656b7bde4-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.341 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.341 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.342 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.343 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.343 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.344 DEBUG:boto:Data: 2022-09-02T04:30:08.345 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.346 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.346 DEBUG:boto:Params: {} 2022-09-02T04:30:08.357 DEBUG:boto:Token: None 2022-09-02T04:30:08.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UG++xqjdeIG9NX9GuGIFTd4xCqc= 2022-09-02T04:30:08.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UG++xqjdeIG9NX9GuGIFTd4xCqc=', 'Content-Length': '0'} 2022-09-02T04:30:08.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fb3b681805964e0-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.362 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.363 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.364 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.365 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.366 DEBUG:boto:Data: 2022-09-02T04:30:08.367 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.369 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.370 DEBUG:boto:Params: {} 2022-09-02T04:30:08.371 DEBUG:boto:Token: None 2022-09-02T04:30:08.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UG++xqjdeIG9NX9GuGIFTd4xCqc= 2022-09-02T04:30:08.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UG++xqjdeIG9NX9GuGIFTd4xCqc=', 'Content-Length': '0'} 2022-09-02T04:30:08.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000020061dc4630214b-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:08.378 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.378 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.379 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.379 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.380 DEBUG:boto:Method: GET 2022-09-02T04:30:08.380 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.381 DEBUG:boto:Data: 2022-09-02T04:30:08.382 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.383 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.383 DEBUG:boto:Params: {} 2022-09-02T04:30:08.384 DEBUG:boto:Token: None 2022-09-02T04:30:08.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA= 2022-09-02T04:30:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA=', 'Content-Length': '0'} 2022-09-02T04:30:08.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033e3a535fd591c21-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.389 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.390 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:08.390 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.391 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.391 DEBUG:boto:Method: GET 2022-09-02T04:30:08.392 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.392 DEBUG:boto:Data: 2022-09-02T04:30:08.393 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.396 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.396 DEBUG:boto:Params: {} 2022-09-02T04:30:08.397 DEBUG:boto:Token: None 2022-09-02T04:30:08.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:08.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA= 2022-09-02T04:30:08.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg3wQQ7AU/Xe+Cb2B5fkECQsaLA=', 'Content-Length': '0'} 2022-09-02T04:30:08.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffdbdce1f79957e8-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.403 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.404 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.404 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.405 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.406 DEBUG:boto:Method: GET 2022-09-02T04:30:08.407 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.407 DEBUG:boto:Data: 2022-09-02T04:30:08.408 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:08.409 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.410 DEBUG:boto:Params: {} 2022-09-02T04:30:08.410 DEBUG:boto:Token: None 2022-09-02T04:30:08.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a082f6bb6b9ab9-00631186d0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.416 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.417 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.418 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.418 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.419 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.420 DEBUG:boto:Method: GET 2022-09-02T04:30:08.420 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:08.421 DEBUG:boto:Data: 2022-09-02T04:30:08.421 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.423 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.423 DEBUG:boto:Params: {} 2022-09-02T04:30:08.424 DEBUG:boto:Token: None 2022-09-02T04:30:08.424 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c29f668fd3a4a12-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.430 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.431 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-18 zones={test-zone2, test-zone1} 2022-09-02T04:30:08.432 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-18 zones={test-zone3, test-zone1} 2022-09-02T04:30:08.432 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.433 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.433 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.434 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:08.434 DEBUG:boto:Data: 2022-09-02T04:30:08.435 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.436 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.436 DEBUG:boto:Params: {} 2022-09-02T04:30:08.437 DEBUG:boto:Token: None 2022-09-02T04:30:08.437 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5VR6TShcGKKNpSfpHCAPfwy35BE= 2022-09-02T04:30:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5VR6TShcGKKNpSfpHCAPfwy35BE=', 'Content-Length': '0'} 2022-09-02T04:30:08.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e5027d5c78e669a-00631186d0-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.442 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.443 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.443 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.444 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:08.444 DEBUG:boto:Data: 2022-09-02T04:30:08.445 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.446 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.446 DEBUG:boto:Params: {} 2022-09-02T04:30:08.448 DEBUG:boto:Token: None 2022-09-02T04:30:08.448 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5VR6TShcGKKNpSfpHCAPfwy35BE= 2022-09-02T04:30:08.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5VR6TShcGKKNpSfpHCAPfwy35BE=', 'Content-Length': '0'} 2022-09-02T04:30:08.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046d9f55a838f8aee-00631186d0-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:30:08.453 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.454 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.454 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.455 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.456 DEBUG:boto:Method: GET 2022-09-02T04:30:08.456 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.457 DEBUG:boto:Data: 2022-09-02T04:30:08.458 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.459 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.459 DEBUG:boto:Params: {} 2022-09-02T04:30:08.460 DEBUG:boto:Token: None 2022-09-02T04:30:08.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/?versions 2022-09-02T04:30:08.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE= 2022-09-02T04:30:08.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE=', 'Content-Length': '0'} 2022-09-02T04:30:08.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012bd200c3211bc56-00631186d0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.466 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.467 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:08.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:08.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:08.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:08.469 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:30:08.470 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.471 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.471 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.472 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.474 DEBUG:boto:Method: GET 2022-09-02T04:30:08.475 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.475 DEBUG:boto:Data: 2022-09-02T04:30:08.476 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.478 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.478 DEBUG:boto:Params: {} 2022-09-02T04:30:08.479 DEBUG:boto:Token: None 2022-09-02T04:30:08.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/?versions 2022-09-02T04:30:08.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE= 2022-09-02T04:30:08.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE=', 'Content-Length': '0'} 2022-09-02T04:30:08.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068352891d321e1b0-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.484 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.485 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:08.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:08.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:08.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:08.488 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.488 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.489 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.489 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.490 DEBUG:boto:Method: GET 2022-09-02T04:30:08.491 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.492 DEBUG:boto:Data: 2022-09-02T04:30:08.492 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.493 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.494 DEBUG:boto:Params: {} 2022-09-02T04:30:08.495 DEBUG:boto:Token: None 2022-09-02T04:30:08.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/?versions 2022-09-02T04:30:08.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE= 2022-09-02T04:30:08.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE=', 'Content-Length': '0'} 2022-09-02T04:30:08.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008022c009d252ae38-00631186d0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.500 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.502 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.502 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.503 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.503 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.504 DEBUG:boto:Method: GET 2022-09-02T04:30:08.504 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:08.505 DEBUG:boto:Data: 2022-09-02T04:30:08.505 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.506 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.507 DEBUG:boto:Params: {} 2022-09-02T04:30:08.508 DEBUG:boto:Token: None 2022-09-02T04:30:08.508 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/?versions 2022-09-02T04:30:08.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE= 2022-09-02T04:30:08.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UPdTQsp/MSbHqanKQ+XLDcGgEuE=', 'Content-Length': '0'} 2022-09-02T04:30:08.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b235c9d009d48a1c-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.514 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:08.515 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.516 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.517 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.517 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.518 DEBUG:boto:Method: GET 2022-09-02T04:30:08.518 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.519 DEBUG:boto:Data: 2022-09-02T04:30:08.519 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.521 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.522 DEBUG:boto:Params: {} 2022-09-02T04:30:08.523 DEBUG:boto:Token: None 2022-09-02T04:30:08.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg= 2022-09-02T04:30:08.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg=', 'Content-Length': '0'} 2022-09-02T04:30:08.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd70b8bcf121f3bb-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.528 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.528 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.529 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.529 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.530 DEBUG:boto:Method: GET 2022-09-02T04:30:08.530 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.531 DEBUG:boto:Data: 2022-09-02T04:30:08.532 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.533 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.533 DEBUG:boto:Params: {} 2022-09-02T04:30:08.534 DEBUG:boto:Token: None 2022-09-02T04:30:08.535 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg= 2022-09-02T04:30:08.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg=', 'Content-Length': '0'} 2022-09-02T04:30:08.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d02657e44a12bb8c-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.540 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.540 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.542 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.543 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.544 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.545 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.546 DEBUG:boto:Data: 2022-09-02T04:30:08.546 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.548 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.548 DEBUG:boto:Params: {} 2022-09-02T04:30:08.549 DEBUG:boto:Token: None 2022-09-02T04:30:08.549 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uBqZ+8Aia1pijTNzSNWyIonQFsY= 2022-09-02T04:30:08.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uBqZ+8Aia1pijTNzSNWyIonQFsY=', 'Content-Length': '0'} 2022-09-02T04:30:08.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009dc6b0a5c7d66f5d-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.555 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.556 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.557 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.558 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.559 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.559 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.560 DEBUG:boto:Data: 2022-09-02T04:30:08.560 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.561 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.562 DEBUG:boto:Params: {} 2022-09-02T04:30:08.563 DEBUG:boto:Token: None 2022-09-02T04:30:08.564 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uBqZ+8Aia1pijTNzSNWyIonQFsY= 2022-09-02T04:30:08.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uBqZ+8Aia1pijTNzSNWyIonQFsY=', 'Content-Length': '0'} 2022-09-02T04:30:08.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b321b7b508c37e40-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:08.571 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.572 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.573 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.573 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.574 DEBUG:boto:Method: GET 2022-09-02T04:30:08.576 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.577 DEBUG:boto:Data: 2022-09-02T04:30:08.578 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.580 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.581 DEBUG:boto:Params: {} 2022-09-02T04:30:08.582 DEBUG:boto:Token: None 2022-09-02T04:30:08.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg= 2022-09-02T04:30:08.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg=', 'Content-Length': '0'} 2022-09-02T04:30:08.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dea7a75ca71e6cc2-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.588 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.589 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.589 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.590 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.591 DEBUG:boto:Method: GET 2022-09-02T04:30:08.591 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.592 DEBUG:boto:Data: 2022-09-02T04:30:08.594 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.597 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.598 DEBUG:boto:Params: {} 2022-09-02T04:30:08.599 DEBUG:boto:Token: None 2022-09-02T04:30:08.600 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg= 2022-09-02T04:30:08.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg=', 'Content-Length': '0'} 2022-09-02T04:30:08.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e880c52f1a3708d-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.609 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.610 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.611 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.611 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.613 DEBUG:boto:Method: GET 2022-09-02T04:30:08.614 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.615 DEBUG:boto:Data: 2022-09-02T04:30:08.616 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.618 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.618 DEBUG:boto:Params: {} 2022-09-02T04:30:08.619 DEBUG:boto:Token: None 2022-09-02T04:30:08.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006419581de76c3ff1-00631186d0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.668 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.670 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.671 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.672 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.673 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.674 DEBUG:boto:Method: GET 2022-09-02T04:30:08.675 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.676 DEBUG:boto:Data: 2022-09-02T04:30:08.677 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.679 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.680 DEBUG:boto:Params: {} 2022-09-02T04:30:08.681 DEBUG:boto:Token: None 2022-09-02T04:30:08.682 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b23f3dbf62e39be1-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.688 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.689 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.690 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.691 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.691 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:30:08.692 DEBUG:boto:Data: 2022-09-02T04:30:08.692 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.693 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.694 DEBUG:boto:Params: {} 2022-09-02T04:30:08.694 DEBUG:boto:Token: None 2022-09-02T04:30:08.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy 2022-09-02T04:30:08.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCAhJqtYDekjQP0ExKdJuzuxRk8= 2022-09-02T04:30:08.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCAhJqtYDekjQP0ExKdJuzuxRk8=', 'Content-Length': '0'} 2022-09-02T04:30:08.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016673856cbab26c7-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.700 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.700 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.701 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.702 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:30:08.702 DEBUG:boto:Data: 2022-09-02T04:30:08.703 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.704 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.704 DEBUG:boto:Params: {} 2022-09-02T04:30:08.705 DEBUG:boto:Token: None 2022-09-02T04:30:08.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy 2022-09-02T04:30:08.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCAhJqtYDekjQP0ExKdJuzuxRk8= 2022-09-02T04:30:08.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCAhJqtYDekjQP0ExKdJuzuxRk8=', 'Content-Length': '0'} 2022-09-02T04:30:08.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a5cff9a35ae4eda-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:08.712 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.713 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.714 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.715 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.715 DEBUG:boto:Method: GET 2022-09-02T04:30:08.717 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.718 DEBUG:boto:Data: 2022-09-02T04:30:08.719 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.721 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.722 DEBUG:boto:Params: {} 2022-09-02T04:30:08.723 DEBUG:boto:Token: None 2022-09-02T04:30:08.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg= 2022-09-02T04:30:08.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg=', 'Content-Length': '0'} 2022-09-02T04:30:08.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad9b2c9596ae22fa-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.730 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:08.731 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:08.731 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.732 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.732 DEBUG:boto:Method: GET 2022-09-02T04:30:08.733 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.733 DEBUG:boto:Data: 2022-09-02T04:30:08.734 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.735 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.735 DEBUG:boto:Params: {} 2022-09-02T04:30:08.736 DEBUG:boto:Token: None 2022-09-02T04:30:08.737 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:08.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg= 2022-09-02T04:30:08.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g8MPENdo7QTXQ3UrQ2yLUyheQg=', 'Content-Length': '0'} 2022-09-02T04:30:08.741 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000402d4c063b005428-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.743 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.743 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.745 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.745 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.746 DEBUG:boto:Method: GET 2022-09-02T04:30:08.747 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.748 DEBUG:boto:Data: 2022-09-02T04:30:08.749 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.750 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.751 DEBUG:boto:Params: {} 2022-09-02T04:30:08.751 DEBUG:boto:Token: None 2022-09-02T04:30:08.752 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123b0aad7b883300-00631186d0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.757 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.758 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.759 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.760 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.760 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.761 DEBUG:boto:Method: GET 2022-09-02T04:30:08.761 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:08.762 DEBUG:boto:Data: 2022-09-02T04:30:08.762 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.763 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.764 DEBUG:boto:Params: {} 2022-09-02T04:30:08.765 DEBUG:boto:Token: None 2022-09-02T04:30:08.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f32553d06539fe-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.770 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:08.772 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.773 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.773 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.774 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.775 DEBUG:boto:Method: GET 2022-09-02T04:30:08.775 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.776 DEBUG:boto:Data: 2022-09-02T04:30:08.777 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.778 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.778 DEBUG:boto:Params: {} 2022-09-02T04:30:08.780 DEBUG:boto:Token: None 2022-09-02T04:30:08.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE= 2022-09-02T04:30:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE=', 'Content-Length': '0'} 2022-09-02T04:30:08.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c3923508597176d-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.785 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.785 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.786 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.787 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.787 DEBUG:boto:Method: GET 2022-09-02T04:30:08.788 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.788 DEBUG:boto:Data: 2022-09-02T04:30:08.789 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.791 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.791 DEBUG:boto:Params: {} 2022-09-02T04:30:08.792 DEBUG:boto:Token: None 2022-09-02T04:30:08.793 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE= 2022-09-02T04:30:08.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE=', 'Content-Length': '0'} 2022-09-02T04:30:08.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fe84f5d79c628ae-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.797 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.798 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.798 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.799 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.799 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.800 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.801 DEBUG:boto:Data: 2022-09-02T04:30:08.801 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.802 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.803 DEBUG:boto:Params: {} 2022-09-02T04:30:08.803 DEBUG:boto:Token: None 2022-09-02T04:30:08.804 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lqrpNVzBmf6HasU597SThBtFDj8= 2022-09-02T04:30:08.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lqrpNVzBmf6HasU597SThBtFDj8=', 'Content-Length': '0'} 2022-09-02T04:30:08.808 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd9a60924e0eb81d-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.809 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.810 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.811 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.812 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.813 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.814 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.815 DEBUG:boto:Data: 2022-09-02T04:30:08.816 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.817 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.818 DEBUG:boto:Params: {} 2022-09-02T04:30:08.819 DEBUG:boto:Token: None 2022-09-02T04:30:08.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lqrpNVzBmf6HasU597SThBtFDj8= 2022-09-02T04:30:08.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lqrpNVzBmf6HasU597SThBtFDj8=', 'Content-Length': '0'} 2022-09-02T04:30:08.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a7607f7ec348f41-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:08.828 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.829 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.830 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.831 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.832 DEBUG:boto:Method: GET 2022-09-02T04:30:08.833 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.834 DEBUG:boto:Data: 2022-09-02T04:30:08.835 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.837 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.838 DEBUG:boto:Params: {} 2022-09-02T04:30:08.839 DEBUG:boto:Token: None 2022-09-02T04:30:08.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE= 2022-09-02T04:30:08.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE=', 'Content-Length': '0'} 2022-09-02T04:30:08.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026c8848e51cba61f-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.845 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.847 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.847 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.848 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.849 DEBUG:boto:Method: GET 2022-09-02T04:30:08.850 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.851 DEBUG:boto:Data: 2022-09-02T04:30:08.852 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.854 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.854 DEBUG:boto:Params: {} 2022-09-02T04:30:08.855 DEBUG:boto:Token: None 2022-09-02T04:30:08.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE= 2022-09-02T04:30:08.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE=', 'Content-Length': '0'} 2022-09-02T04:30:08.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cac75c67a1ed0ce7-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.863 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.864 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.865 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:08.866 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:08.866 DEBUG:boto:Method: GET 2022-09-02T04:30:08.867 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:08.868 DEBUG:boto:Data: 2022-09-02T04:30:08.869 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.870 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.870 DEBUG:boto:Params: {} 2022-09-02T04:30:08.871 DEBUG:boto:Token: None 2022-09-02T04:30:08.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfece3f55323eb3b-00631186d0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.879 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.880 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:08.880 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:08.881 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:08.882 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:08.883 DEBUG:boto:Method: GET 2022-09-02T04:30:08.884 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:08.885 DEBUG:boto:Data: 2022-09-02T04:30:08.886 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.887 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.888 DEBUG:boto:Params: {} 2022-09-02T04:30:08.888 DEBUG:boto:Token: None 2022-09-02T04:30:08.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/ 2022-09-02T04:30:08.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y= 2022-09-02T04:30:08.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMXKyDpG0DfnrbEVQXyv3Hg/+2Y=', 'Content-Length': '0'} 2022-09-02T04:30:08.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f5b2ee559629f3-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.895 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:08.896 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.896 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.897 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.899 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.900 DEBUG:boto:Data: 2022-09-02T04:30:08.901 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.903 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.904 DEBUG:boto:Params: {} 2022-09-02T04:30:08.905 DEBUG:boto:Token: None 2022-09-02T04:30:08.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3aoQ0U9AMuGp+/tD9skhZBQEqh4= 2022-09-02T04:30:08.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3aoQ0U9AMuGp+/tD9skhZBQEqh4=', 'Content-Length': '0'} 2022-09-02T04:30:08.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6c132a510c5caee-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.915 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.916 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.917 DEBUG:boto:Method: HEAD 2022-09-02T04:30:08.918 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.919 DEBUG:boto:Data: 2022-09-02T04:30:08.919 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.921 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.922 DEBUG:boto:Params: {} 2022-09-02T04:30:08.927 DEBUG:boto:Token: None 2022-09-02T04:30:08.928 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3aoQ0U9AMuGp+/tD9skhZBQEqh4= 2022-09-02T04:30:08.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3aoQ0U9AMuGp+/tD9skhZBQEqh4=', 'Content-Length': '0'} 2022-09-02T04:30:08.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000571b7192390e2bcb-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:08.935 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.936 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.937 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.938 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.939 DEBUG:boto:Method: GET 2022-09-02T04:30:08.940 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.942 DEBUG:boto:Data: 2022-09-02T04:30:08.943 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:08.945 DEBUG:boto:Port: 8001 2022-09-02T04:30:08.946 DEBUG:boto:Params: {} 2022-09-02T04:30:08.947 DEBUG:boto:Token: None 2022-09-02T04:30:08.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE= 2022-09-02T04:30:08.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE=', 'Content-Length': '0'} 2022-09-02T04:30:08.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000889ee81fceb98f48-00631186d0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:08.954 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.955 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:08.956 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.957 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.958 DEBUG:boto:Method: GET 2022-09-02T04:30:08.959 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.960 DEBUG:boto:Data: 2022-09-02T04:30:08.961 DEBUG:boto:Headers: {} 2022-09-02T04:30:08.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:08.963 DEBUG:boto:Port: 8000 2022-09-02T04:30:08.964 DEBUG:boto:Params: {} 2022-09-02T04:30:08.965 DEBUG:boto:Token: None 2022-09-02T04:30:08.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:08 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:08.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE= 2022-09-02T04:30:08.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5/qo+D8LNSJ5lreUr8gi83mReE=', 'Content-Length': '0'} 2022-09-02T04:30:08.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d57cac78824eefcd-00631186d0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.014 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.015 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.015 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:09.016 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:09.016 DEBUG:boto:Method: GET 2022-09-02T04:30:09.017 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:09.017 DEBUG:boto:Data: 2022-09-02T04:30:09.018 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.019 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.020 DEBUG:boto:Params: {} 2022-09-02T04:30:09.020 DEBUG:boto:Token: None 2022-09-02T04:30:09.021 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000905d41fd73bc2b66-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.035 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.037 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.038 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.038 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:09.039 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:09.040 DEBUG:boto:Method: GET 2022-09-02T04:30:09.040 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:09.041 DEBUG:boto:Data: 2022-09-02T04:30:09.042 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.043 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.043 DEBUG:boto:Params: {} 2022-09-02T04:30:09.044 DEBUG:boto:Token: None 2022-09-02T04:30:09.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b54d2564e5e4ef-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.051 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:09.053 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.054 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.055 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.056 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.058 DEBUG:boto:Method: GET 2022-09-02T04:30:09.059 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.060 DEBUG:boto:Data: 2022-09-02T04:30:09.061 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.064 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.065 DEBUG:boto:Params: {} 2022-09-02T04:30:09.066 DEBUG:boto:Token: None 2022-09-02T04:30:09.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.069 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg= 2022-09-02T04:30:09.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg=', 'Content-Length': '0'} 2022-09-02T04:30:09.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000657f5d3c83889957-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.122 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.123 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.124 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.125 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.126 DEBUG:boto:Method: GET 2022-09-02T04:30:09.126 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.127 DEBUG:boto:Data: 2022-09-02T04:30:09.128 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.132 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.133 DEBUG:boto:Params: {} 2022-09-02T04:30:09.133 DEBUG:boto:Token: None 2022-09-02T04:30:09.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg= 2022-09-02T04:30:09.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg=', 'Content-Length': '0'} 2022-09-02T04:30:09.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8fac10e4949049f-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.151 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.152 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.152 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.153 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.154 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.154 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.155 DEBUG:boto:Data: 2022-09-02T04:30:09.156 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.157 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.158 DEBUG:boto:Params: {} 2022-09-02T04:30:09.158 DEBUG:boto:Token: None 2022-09-02T04:30:09.159 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DDZ89ZrnUsyXfzHmKcWzPZuFnMg= 2022-09-02T04:30:09.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DDZ89ZrnUsyXfzHmKcWzPZuFnMg=', 'Content-Length': '0'} 2022-09-02T04:30:09.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdf21441485b360d-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.163 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.164 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.165 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.165 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.166 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.166 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.167 DEBUG:boto:Data: 2022-09-02T04:30:09.167 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.168 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.169 DEBUG:boto:Params: {} 2022-09-02T04:30:09.170 DEBUG:boto:Token: None 2022-09-02T04:30:09.171 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DDZ89ZrnUsyXfzHmKcWzPZuFnMg= 2022-09-02T04:30:09.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DDZ89ZrnUsyXfzHmKcWzPZuFnMg=', 'Content-Length': '0'} 2022-09-02T04:30:09.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b0894d394d2389-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:09.179 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.180 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.181 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.181 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.182 DEBUG:boto:Method: GET 2022-09-02T04:30:09.183 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.183 DEBUG:boto:Data: 2022-09-02T04:30:09.184 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.185 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.185 DEBUG:boto:Params: {} 2022-09-02T04:30:09.186 DEBUG:boto:Token: None 2022-09-02T04:30:09.187 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg= 2022-09-02T04:30:09.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg=', 'Content-Length': '0'} 2022-09-02T04:30:09.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c88ec544083a2fdd-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.192 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.192 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.193 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.193 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.194 DEBUG:boto:Method: GET 2022-09-02T04:30:09.195 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.195 DEBUG:boto:Data: 2022-09-02T04:30:09.196 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.197 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.197 DEBUG:boto:Params: {} 2022-09-02T04:30:09.198 DEBUG:boto:Token: None 2022-09-02T04:30:09.199 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg= 2022-09-02T04:30:09.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg=', 'Content-Length': '0'} 2022-09-02T04:30:09.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e6efe30b828fe71-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.246 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.247 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.248 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.248 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.249 DEBUG:boto:Method: GET 2022-09-02T04:30:09.249 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.250 DEBUG:boto:Data: 2022-09-02T04:30:09.250 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.252 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.252 DEBUG:boto:Params: {} 2022-09-02T04:30:09.253 DEBUG:boto:Token: None 2022-09-02T04:30:09.254 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009395e0b85b53c1c2-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.259 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.260 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.260 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.261 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.261 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.262 DEBUG:boto:Method: GET 2022-09-02T04:30:09.263 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.263 DEBUG:boto:Data: 2022-09-02T04:30:09.264 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.265 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.265 DEBUG:boto:Params: {} 2022-09-02T04:30:09.266 DEBUG:boto:Token: None 2022-09-02T04:30:09.267 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec5941ee7e8ddf02-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.271 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.272 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.273 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.273 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.274 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.274 DEBUG:boto:Data: 2022-09-02T04:30:09.275 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.277 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.278 DEBUG:boto:Params: {} 2022-09-02T04:30:09.278 DEBUG:boto:Token: None 2022-09-02T04:30:09.279 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GmC0fXJs8pcHHqZjjL8s4fXffVc= 2022-09-02T04:30:09.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GmC0fXJs8pcHHqZjjL8s4fXffVc=', 'Content-Length': '0'} 2022-09-02T04:30:09.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005746bc1b6cb5d291-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.284 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.285 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.285 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.287 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.287 DEBUG:boto:Data: 2022-09-02T04:30:09.288 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.289 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.290 DEBUG:boto:Params: {} 2022-09-02T04:30:09.291 DEBUG:boto:Token: None 2022-09-02T04:30:09.291 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GmC0fXJs8pcHHqZjjL8s4fXffVc= 2022-09-02T04:30:09.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GmC0fXJs8pcHHqZjjL8s4fXffVc=', 'Content-Length': '0'} 2022-09-02T04:30:09.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000610bbb570e4357ef-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:09.299 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.300 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.301 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.301 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.302 DEBUG:boto:Method: GET 2022-09-02T04:30:09.302 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.303 DEBUG:boto:Data: 2022-09-02T04:30:09.303 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.304 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.305 DEBUG:boto:Params: {} 2022-09-02T04:30:09.305 DEBUG:boto:Token: None 2022-09-02T04:30:09.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg= 2022-09-02T04:30:09.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg=', 'Content-Length': '0'} 2022-09-02T04:30:09.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6e9c9dae1e2e7a6-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.354 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.355 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:09.356 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.356 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.357 DEBUG:boto:Method: GET 2022-09-02T04:30:09.358 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.359 DEBUG:boto:Data: 2022-09-02T04:30:09.359 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.360 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.361 DEBUG:boto:Params: {} 2022-09-02T04:30:09.362 DEBUG:boto:Token: None 2022-09-02T04:30:09.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:09.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg= 2022-09-02T04:30:09.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3e/8drz3auMIikjhLGmxWN7Lvg=', 'Content-Length': '0'} 2022-09-02T04:30:09.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057d8ad5bc559914c-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.370 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.370 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.371 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.372 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.372 DEBUG:boto:Method: GET 2022-09-02T04:30:09.373 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.373 DEBUG:boto:Data: 2022-09-02T04:30:09.374 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.375 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.376 DEBUG:boto:Params: {} 2022-09-02T04:30:09.377 DEBUG:boto:Token: None 2022-09-02T04:30:09.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c5ff927b03d8a42-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.382 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.383 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.384 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.384 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.385 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.385 DEBUG:boto:Method: GET 2022-09-02T04:30:09.386 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:09.387 DEBUG:boto:Data: 2022-09-02T04:30:09.387 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.388 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.389 DEBUG:boto:Params: {} 2022-09-02T04:30:09.390 DEBUG:boto:Token: None 2022-09-02T04:30:09.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc83cb108f1e666-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.442 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:09.444 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.445 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.445 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.446 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.446 DEBUG:boto:Method: GET 2022-09-02T04:30:09.447 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.448 DEBUG:boto:Data: 2022-09-02T04:30:09.448 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.450 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.451 DEBUG:boto:Params: {} 2022-09-02T04:30:09.451 DEBUG:boto:Token: None 2022-09-02T04:30:09.452 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0= 2022-09-02T04:30:09.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0=', 'Content-Length': '0'} 2022-09-02T04:30:09.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b133bfcc341d97eb-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.457 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.458 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.458 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.459 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.460 DEBUG:boto:Method: GET 2022-09-02T04:30:09.460 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.461 DEBUG:boto:Data: 2022-09-02T04:30:09.461 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.463 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.464 DEBUG:boto:Params: {} 2022-09-02T04:30:09.464 DEBUG:boto:Token: None 2022-09-02T04:30:09.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0= 2022-09-02T04:30:09.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0=', 'Content-Length': '0'} 2022-09-02T04:30:09.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9a3a31265c36d0c-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.469 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.470 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.471 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.471 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.472 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.472 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.473 DEBUG:boto:Data: 2022-09-02T04:30:09.473 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.475 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.475 DEBUG:boto:Params: {} 2022-09-02T04:30:09.476 DEBUG:boto:Token: None 2022-09-02T04:30:09.476 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZmeAGLZkG0u45sLvmuvx+Y+f6BU= 2022-09-02T04:30:09.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZmeAGLZkG0u45sLvmuvx+Y+f6BU=', 'Content-Length': '0'} 2022-09-02T04:30:09.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d099532041df0f5a-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.481 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.481 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.482 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.483 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.483 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.484 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.484 DEBUG:boto:Data: 2022-09-02T04:30:09.485 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.486 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.486 DEBUG:boto:Params: {} 2022-09-02T04:30:09.487 DEBUG:boto:Token: None 2022-09-02T04:30:09.487 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZmeAGLZkG0u45sLvmuvx+Y+f6BU= 2022-09-02T04:30:09.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZmeAGLZkG0u45sLvmuvx+Y+f6BU=', 'Content-Length': '0'} 2022-09-02T04:30:09.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dc075f8b600f58e-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:09.492 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.493 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.493 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.494 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.494 DEBUG:boto:Method: GET 2022-09-02T04:30:09.495 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.495 DEBUG:boto:Data: 2022-09-02T04:30:09.496 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.497 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.498 DEBUG:boto:Params: {} 2022-09-02T04:30:09.499 DEBUG:boto:Token: None 2022-09-02T04:30:09.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0= 2022-09-02T04:30:09.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0=', 'Content-Length': '0'} 2022-09-02T04:30:09.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cd32f41263aef1f-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.504 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.504 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.505 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.505 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.506 DEBUG:boto:Method: GET 2022-09-02T04:30:09.507 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.507 DEBUG:boto:Data: 2022-09-02T04:30:09.508 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.509 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.509 DEBUG:boto:Params: {} 2022-09-02T04:30:09.510 DEBUG:boto:Token: None 2022-09-02T04:30:09.511 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0= 2022-09-02T04:30:09.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0=', 'Content-Length': '0'} 2022-09-02T04:30:09.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0a37ad8d721a22a-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.515 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.516 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.516 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.517 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.517 DEBUG:boto:Method: GET 2022-09-02T04:30:09.518 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.518 DEBUG:boto:Data: 2022-09-02T04:30:09.519 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.520 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.521 DEBUG:boto:Params: {} 2022-09-02T04:30:09.522 DEBUG:boto:Token: None 2022-09-02T04:30:09.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c01b19d5673316-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.528 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.530 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.530 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.531 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.532 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.532 DEBUG:boto:Method: GET 2022-09-02T04:30:09.533 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.533 DEBUG:boto:Data: 2022-09-02T04:30:09.534 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.535 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.536 DEBUG:boto:Params: {} 2022-09-02T04:30:09.536 DEBUG:boto:Token: None 2022-09-02T04:30:09.537 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a77a6f06105bd01-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.582 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.583 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.584 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.585 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.585 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.586 DEBUG:boto:Data: 2022-09-02T04:30:09.586 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.587 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.588 DEBUG:boto:Params: {} 2022-09-02T04:30:09.589 DEBUG:boto:Token: None 2022-09-02T04:30:09.589 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QBJCuYfwBV0KvGR+3FxrSqShjbk= 2022-09-02T04:30:09.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QBJCuYfwBV0KvGR+3FxrSqShjbk=', 'Content-Length': '0'} 2022-09-02T04:30:09.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025f18a55119c25c9-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.596 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.597 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.597 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.598 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.598 DEBUG:boto:Data: 2022-09-02T04:30:09.599 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.600 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.601 DEBUG:boto:Params: {} 2022-09-02T04:30:09.601 DEBUG:boto:Token: None 2022-09-02T04:30:09.602 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QBJCuYfwBV0KvGR+3FxrSqShjbk= 2022-09-02T04:30:09.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QBJCuYfwBV0KvGR+3FxrSqShjbk=', 'Content-Length': '0'} 2022-09-02T04:30:09.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000039fb3cbc52b60b0-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:09.607 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.608 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.608 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.609 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.609 DEBUG:boto:Method: GET 2022-09-02T04:30:09.610 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.610 DEBUG:boto:Data: 2022-09-02T04:30:09.611 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.612 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.612 DEBUG:boto:Params: {} 2022-09-02T04:30:09.613 DEBUG:boto:Token: None 2022-09-02T04:30:09.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0= 2022-09-02T04:30:09.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0=', 'Content-Length': '0'} 2022-09-02T04:30:09.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b850e75f2dcb8a5b-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.658 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.659 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:09.659 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.660 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.660 DEBUG:boto:Method: GET 2022-09-02T04:30:09.661 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.662 DEBUG:boto:Data: 2022-09-02T04:30:09.663 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.665 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.666 DEBUG:boto:Params: {} 2022-09-02T04:30:09.667 DEBUG:boto:Token: None 2022-09-02T04:30:09.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:09.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0= 2022-09-02T04:30:09.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WRaCIa+Q7nB7MRWHWYFPEUYx4o0=', 'Content-Length': '0'} 2022-09-02T04:30:09.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004517c91b24ca4045-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.673 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.674 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.674 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.675 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.675 DEBUG:boto:Method: GET 2022-09-02T04:30:09.676 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.677 DEBUG:boto:Data: 2022-09-02T04:30:09.677 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.678 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.679 DEBUG:boto:Params: {} 2022-09-02T04:30:09.679 DEBUG:boto:Token: None 2022-09-02T04:30:09.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030df169049092cb2-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.686 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.687 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.687 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.688 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.688 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.689 DEBUG:boto:Method: GET 2022-09-02T04:30:09.690 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:09.690 DEBUG:boto:Data: 2022-09-02T04:30:09.691 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:09.692 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.693 DEBUG:boto:Params: {} 2022-09-02T04:30:09.693 DEBUG:boto:Token: None 2022-09-02T04:30:09.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000677955e000062521-00631186d1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.699 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.699 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-18 zones={test-zone3, test-zone1} 2022-09-02T04:30:09.700 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-18 zones={test-zone3, test-zone2} 2022-09-02T04:30:09.701 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.701 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.702 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.702 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:09.703 DEBUG:boto:Data: 2022-09-02T04:30:09.703 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.704 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.705 DEBUG:boto:Params: {} 2022-09-02T04:30:09.706 DEBUG:boto:Token: None 2022-09-02T04:30:09.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p3nCM1jbmizlLFeBwDs9Tu2QhKE= 2022-09-02T04:30:09.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p3nCM1jbmizlLFeBwDs9Tu2QhKE=', 'Content-Length': '0'} 2022-09-02T04:30:09.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070dc096db86f7e28-00631186d1-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.711 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.711 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.712 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.712 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:09.713 DEBUG:boto:Data: 2022-09-02T04:30:09.713 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.714 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.715 DEBUG:boto:Params: {} 2022-09-02T04:30:09.716 DEBUG:boto:Token: None 2022-09-02T04:30:09.716 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p3nCM1jbmizlLFeBwDs9Tu2QhKE= 2022-09-02T04:30:09.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p3nCM1jbmizlLFeBwDs9Tu2QhKE=', 'Content-Length': '0'} 2022-09-02T04:30:09.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5f19a1759b3ab2c-00631186d1-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.721 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:30:09.721 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.722 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.723 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.724 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.724 DEBUG:boto:Method: GET 2022-09-02T04:30:09.725 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.726 DEBUG:boto:Data: 2022-09-02T04:30:09.727 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.729 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.730 DEBUG:boto:Params: {} 2022-09-02T04:30:09.732 DEBUG:boto:Token: None 2022-09-02T04:30:09.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/?versions 2022-09-02T04:30:09.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc= 2022-09-02T04:30:09.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc=', 'Content-Length': '0'} 2022-09-02T04:30:09.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7d0aea62bd83f5-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.740 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.742 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:09.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:09.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:09.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:09.745 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:30:09.746 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.747 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.748 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.749 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.750 DEBUG:boto:Method: GET 2022-09-02T04:30:09.751 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.752 DEBUG:boto:Data: 2022-09-02T04:30:09.753 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.755 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.758 DEBUG:boto:Params: {} 2022-09-02T04:30:09.759 DEBUG:boto:Token: None 2022-09-02T04:30:09.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/?versions 2022-09-02T04:30:09.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc= 2022-09-02T04:30:09.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc=', 'Content-Length': '0'} 2022-09-02T04:30:09.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095a39ad5f98bfe5-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.770 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.771 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:30:09.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:30:09.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:30:09.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:30:09.775 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.776 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.777 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.779 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.779 DEBUG:boto:Method: GET 2022-09-02T04:30:09.780 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.781 DEBUG:boto:Data: 2022-09-02T04:30:09.782 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.784 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.785 DEBUG:boto:Params: {} 2022-09-02T04:30:09.786 DEBUG:boto:Token: None 2022-09-02T04:30:09.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/?versions 2022-09-02T04:30:09.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc= 2022-09-02T04:30:09.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc=', 'Content-Length': '0'} 2022-09-02T04:30:09.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004af8188aa049169-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.793 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.794 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.795 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.796 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.796 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.797 DEBUG:boto:Method: GET 2022-09-02T04:30:09.798 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:30:09.798 DEBUG:boto:Data: 2022-09-02T04:30:09.799 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.800 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.801 DEBUG:boto:Params: {} 2022-09-02T04:30:09.801 DEBUG:boto:Token: None 2022-09-02T04:30:09.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/?versions 2022-09-02T04:30:09.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc= 2022-09-02T04:30:09.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xHxYuQ7Hg7s+o3pgNcCx6rCcoGc=', 'Content-Length': '0'} 2022-09-02T04:30:09.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000741c366a0001b7fb-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.811 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:09.813 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.813 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.814 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.814 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.815 DEBUG:boto:Method: GET 2022-09-02T04:30:09.815 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.816 DEBUG:boto:Data: 2022-09-02T04:30:09.817 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.818 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.819 DEBUG:boto:Params: {} 2022-09-02T04:30:09.819 DEBUG:boto:Token: None 2022-09-02T04:30:09.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0= 2022-09-02T04:30:09.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0=', 'Content-Length': '0'} 2022-09-02T04:30:09.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077937cf28993f87e-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.824 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.825 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.825 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.826 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.827 DEBUG:boto:Method: GET 2022-09-02T04:30:09.828 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.828 DEBUG:boto:Data: 2022-09-02T04:30:09.829 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.830 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.830 DEBUG:boto:Params: {} 2022-09-02T04:30:09.831 DEBUG:boto:Token: None 2022-09-02T04:30:09.832 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0= 2022-09-02T04:30:09.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0=', 'Content-Length': '0'} 2022-09-02T04:30:09.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b00ece5e77edbe4-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.836 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.837 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.837 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.838 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.838 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.839 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.839 DEBUG:boto:Data: 2022-09-02T04:30:09.840 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.841 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.841 DEBUG:boto:Params: {} 2022-09-02T04:30:09.843 DEBUG:boto:Token: None 2022-09-02T04:30:09.843 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.844 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p+wDMdG0CHBMDsAcv2zzwjwGGXs= 2022-09-02T04:30:09.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p+wDMdG0CHBMDsAcv2zzwjwGGXs=', 'Content-Length': '0'} 2022-09-02T04:30:09.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffb36813b8222a33-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.848 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.849 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.851 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.853 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.854 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.855 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.855 DEBUG:boto:Data: 2022-09-02T04:30:09.856 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.857 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.857 DEBUG:boto:Params: {} 2022-09-02T04:30:09.858 DEBUG:boto:Token: None 2022-09-02T04:30:09.859 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p+wDMdG0CHBMDsAcv2zzwjwGGXs= 2022-09-02T04:30:09.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p+wDMdG0CHBMDsAcv2zzwjwGGXs=', 'Content-Length': '0'} 2022-09-02T04:30:09.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000efe1c1b05d6316b6-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:09.864 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.865 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.866 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.867 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.868 DEBUG:boto:Method: GET 2022-09-02T04:30:09.869 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.870 DEBUG:boto:Data: 2022-09-02T04:30:09.871 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.873 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.874 DEBUG:boto:Params: {} 2022-09-02T04:30:09.875 DEBUG:boto:Token: None 2022-09-02T04:30:09.876 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0= 2022-09-02T04:30:09.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0=', 'Content-Length': '0'} 2022-09-02T04:30:09.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6c114ed21881c27-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.883 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.884 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.884 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.885 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.886 DEBUG:boto:Method: GET 2022-09-02T04:30:09.887 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.888 DEBUG:boto:Data: 2022-09-02T04:30:09.889 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.892 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.894 DEBUG:boto:Params: {} 2022-09-02T04:30:09.895 DEBUG:boto:Token: None 2022-09-02T04:30:09.896 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0= 2022-09-02T04:30:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0=', 'Content-Length': '0'} 2022-09-02T04:30:09.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbf5a4fc48725899-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.904 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.905 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.907 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:09.907 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:09.909 DEBUG:boto:Method: GET 2022-09-02T04:30:09.910 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:09.911 DEBUG:boto:Data: 2022-09-02T04:30:09.911 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.912 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.913 DEBUG:boto:Params: {} 2022-09-02T04:30:09.915 DEBUG:boto:Token: None 2022-09-02T04:30:09.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c8acef8c80bab17-00631186d1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.926 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.927 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:09.929 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:09.930 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:09.931 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:09.932 DEBUG:boto:Method: GET 2022-09-02T04:30:09.933 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:09.934 DEBUG:boto:Data: 2022-09-02T04:30:09.935 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.937 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.938 DEBUG:boto:Params: {} 2022-09-02T04:30:09.939 DEBUG:boto:Token: None 2022-09-02T04:30:09.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/ 2022-09-02T04:30:09.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s= 2022-09-02T04:30:09.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5WuXpotYR2jTSsawUuX6WGOw+3s=', 'Content-Length': '0'} 2022-09-02T04:30:09.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132b041f0d8b6345-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.948 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:09.949 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.950 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.951 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.951 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:30:09.952 DEBUG:boto:Data: 2022-09-02T04:30:09.952 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.953 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.953 DEBUG:boto:Params: {} 2022-09-02T04:30:09.954 DEBUG:boto:Token: None 2022-09-02T04:30:09.955 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy 2022-09-02T04:30:09.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsA3ChOkTxlDTGZdncL38+bItpo= 2022-09-02T04:30:09.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsA3ChOkTxlDTGZdncL38+bItpo=', 'Content-Length': '0'} 2022-09-02T04:30:09.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002080fce7e6b4360d-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.961 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.961 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.962 DEBUG:boto:Method: HEAD 2022-09-02T04:30:09.962 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:30:09.963 DEBUG:boto:Data: 2022-09-02T04:30:09.963 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.964 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.965 DEBUG:boto:Params: {} 2022-09-02T04:30:09.966 DEBUG:boto:Token: None 2022-09-02T04:30:09.967 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy 2022-09-02T04:30:09.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsA3ChOkTxlDTGZdncL38+bItpo= 2022-09-02T04:30:09.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsA3ChOkTxlDTGZdncL38+bItpo=', 'Content-Length': '0'} 2022-09-02T04:30:09.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068096d55e5ae4fd1-00631186d1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:30:09.972 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.973 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.973 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.974 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.974 DEBUG:boto:Method: GET 2022-09-02T04:30:09.975 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.975 DEBUG:boto:Data: 2022-09-02T04:30:09.976 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:09.979 DEBUG:boto:Port: 8001 2022-09-02T04:30:09.980 DEBUG:boto:Params: {} 2022-09-02T04:30:09.981 DEBUG:boto:Token: None 2022-09-02T04:30:09.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0= 2022-09-02T04:30:09.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0=', 'Content-Length': '0'} 2022-09-02T04:30:09.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a67a72f04838c153-00631186d1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:09.988 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:30:09.989 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:30:09.989 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.990 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.990 DEBUG:boto:Method: GET 2022-09-02T04:30:09.991 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.992 DEBUG:boto:Data: 2022-09-02T04:30:09.992 DEBUG:boto:Headers: {} 2022-09-02T04:30:09.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:09.994 DEBUG:boto:Port: 8000 2022-09-02T04:30:09.995 DEBUG:boto:Params: {} 2022-09-02T04:30:09.996 DEBUG:boto:Token: None 2022-09-02T04:30:09.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:09 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:30:09.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0= 2022-09-02T04:30:09.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EdK8hHdrA7SMHtDBaPII4P5ZJu0=', 'Content-Length': '0'} 2022-09-02T04:30:10.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9ba7b743d176af0-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.001 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.002 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.002 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:10.003 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:10.004 DEBUG:boto:Method: GET 2022-09-02T04:30:10.004 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:10.005 DEBUG:boto:Data: 2022-09-02T04:30:10.006 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.007 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.007 DEBUG:boto:Params: {} 2022-09-02T04:30:10.008 DEBUG:boto:Token: None 2022-09-02T04:30:10.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006577b33d9553f899-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.015 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.016 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.017 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.018 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:10.019 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:30:10.020 DEBUG:boto:Method: GET 2022-09-02T04:30:10.021 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:30:10.021 DEBUG:boto:Data: 2022-09-02T04:30:10.022 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.023 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.024 DEBUG:boto:Params: {} 2022-09-02T04:30:10.024 DEBUG:boto:Token: None 2022-09-02T04:30:10.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d882454b3f25f83-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.031 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:10.032 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.033 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.034 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.034 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.035 DEBUG:boto:Method: GET 2022-09-02T04:30:10.036 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.036 DEBUG:boto:Data: 2022-09-02T04:30:10.037 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.038 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.039 DEBUG:boto:Params: {} 2022-09-02T04:30:10.039 DEBUG:boto:Token: None 2022-09-02T04:30:10.040 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE= 2022-09-02T04:30:10.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE=', 'Content-Length': '0'} 2022-09-02T04:30:10.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0ae1781285d974c-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.046 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.047 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.047 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.048 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.048 DEBUG:boto:Method: GET 2022-09-02T04:30:10.049 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.050 DEBUG:boto:Data: 2022-09-02T04:30:10.051 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.052 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.053 DEBUG:boto:Params: {} 2022-09-02T04:30:10.053 DEBUG:boto:Token: None 2022-09-02T04:30:10.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE= 2022-09-02T04:30:10.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE=', 'Content-Length': '0'} 2022-09-02T04:30:10.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000569388a485930c73-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.061 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.061 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.062 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.063 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.064 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.065 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.066 DEBUG:boto:Data: 2022-09-02T04:30:10.067 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.068 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.068 DEBUG:boto:Params: {} 2022-09-02T04:30:10.069 DEBUG:boto:Token: None 2022-09-02T04:30:10.069 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vmaxM0GDTb/+mftD1LPMJWtGRTY= 2022-09-02T04:30:10.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vmaxM0GDTb/+mftD1LPMJWtGRTY=', 'Content-Length': '0'} 2022-09-02T04:30:10.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026f13cd3b3dc7ae0-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.074 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.074 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.075 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.076 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.076 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.077 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.077 DEBUG:boto:Data: 2022-09-02T04:30:10.078 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.079 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.079 DEBUG:boto:Params: {} 2022-09-02T04:30:10.080 DEBUG:boto:Token: None 2022-09-02T04:30:10.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vmaxM0GDTb/+mftD1LPMJWtGRTY= 2022-09-02T04:30:10.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vmaxM0GDTb/+mftD1LPMJWtGRTY=', 'Content-Length': '0'} 2022-09-02T04:30:10.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000854e46bd96d5bf8e-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:10.085 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.086 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.087 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.088 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.088 DEBUG:boto:Method: GET 2022-09-02T04:30:10.089 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.090 DEBUG:boto:Data: 2022-09-02T04:30:10.090 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.092 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.093 DEBUG:boto:Params: {} 2022-09-02T04:30:10.094 DEBUG:boto:Token: None 2022-09-02T04:30:10.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE= 2022-09-02T04:30:10.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE=', 'Content-Length': '0'} 2022-09-02T04:30:10.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055e74516ad361c5c-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.103 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.104 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.105 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.106 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.107 DEBUG:boto:Method: GET 2022-09-02T04:30:10.108 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.109 DEBUG:boto:Data: 2022-09-02T04:30:10.110 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.111 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.111 DEBUG:boto:Params: {} 2022-09-02T04:30:10.112 DEBUG:boto:Token: None 2022-09-02T04:30:10.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE= 2022-09-02T04:30:10.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE=', 'Content-Length': '0'} 2022-09-02T04:30:10.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004189e7c583785969-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.117 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.118 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.118 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.119 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.119 DEBUG:boto:Method: GET 2022-09-02T04:30:10.120 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.120 DEBUG:boto:Data: 2022-09-02T04:30:10.121 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.122 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.123 DEBUG:boto:Params: {} 2022-09-02T04:30:10.124 DEBUG:boto:Token: None 2022-09-02T04:30:10.124 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6c2e0b9ed7edd2-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.129 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.130 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.131 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.131 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.132 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.132 DEBUG:boto:Method: GET 2022-09-02T04:30:10.133 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.133 DEBUG:boto:Data: 2022-09-02T04:30:10.134 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.135 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.136 DEBUG:boto:Params: {} 2022-09-02T04:30:10.136 DEBUG:boto:Token: None 2022-09-02T04:30:10.137 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3803f12435c1c48-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.142 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.143 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.143 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.144 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.144 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.145 DEBUG:boto:Data: 2022-09-02T04:30:10.146 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.148 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.149 DEBUG:boto:Params: {} 2022-09-02T04:30:10.150 DEBUG:boto:Token: None 2022-09-02T04:30:10.151 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MbKFBZWV4fFmn6sqHTYJreYXVOc= 2022-09-02T04:30:10.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MbKFBZWV4fFmn6sqHTYJreYXVOc=', 'Content-Length': '0'} 2022-09-02T04:30:10.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc296448124ae4a7-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.157 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.158 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.159 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.160 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.161 DEBUG:boto:Data: 2022-09-02T04:30:10.162 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.164 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.165 DEBUG:boto:Params: {} 2022-09-02T04:30:10.166 DEBUG:boto:Token: None 2022-09-02T04:30:10.167 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MbKFBZWV4fFmn6sqHTYJreYXVOc= 2022-09-02T04:30:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MbKFBZWV4fFmn6sqHTYJreYXVOc=', 'Content-Length': '0'} 2022-09-02T04:30:10.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca5b95f45c38d63b-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:30:10.176 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.177 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.177 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.178 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.178 DEBUG:boto:Method: GET 2022-09-02T04:30:10.179 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.179 DEBUG:boto:Data: 2022-09-02T04:30:10.180 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.181 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.182 DEBUG:boto:Params: {} 2022-09-02T04:30:10.182 DEBUG:boto:Token: None 2022-09-02T04:30:10.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE= 2022-09-02T04:30:10.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE=', 'Content-Length': '0'} 2022-09-02T04:30:10.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042a75a7767ac565c-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.187 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.188 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:10.188 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.189 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.190 DEBUG:boto:Method: GET 2022-09-02T04:30:10.190 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.191 DEBUG:boto:Data: 2022-09-02T04:30:10.191 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.192 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.193 DEBUG:boto:Params: {} 2022-09-02T04:30:10.193 DEBUG:boto:Token: None 2022-09-02T04:30:10.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:30:10.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE= 2022-09-02T04:30:10.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giI9iMBXEnHJRWALsmitWCkZgkE=', 'Content-Length': '0'} 2022-09-02T04:30:10.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000243ffd3239106eee-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.198 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.199 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.200 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.200 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.201 DEBUG:boto:Method: GET 2022-09-02T04:30:10.201 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.202 DEBUG:boto:Data: 2022-09-02T04:30:10.202 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.203 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.204 DEBUG:boto:Params: {} 2022-09-02T04:30:10.204 DEBUG:boto:Token: None 2022-09-02T04:30:10.205 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b8525f8f706e12-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.211 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.212 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.212 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.213 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.213 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.214 DEBUG:boto:Method: GET 2022-09-02T04:30:10.215 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:30:10.216 DEBUG:boto:Data: 2022-09-02T04:30:10.216 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.217 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.218 DEBUG:boto:Params: {} 2022-09-02T04:30:10.219 DEBUG:boto:Token: None 2022-09-02T04:30:10.220 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c19bfc7ac94d43a-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.225 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:10.227 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.228 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.229 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.229 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.231 DEBUG:boto:Method: GET 2022-09-02T04:30:10.232 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.232 DEBUG:boto:Data: 2022-09-02T04:30:10.233 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.234 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.235 DEBUG:boto:Params: {} 2022-09-02T04:30:10.235 DEBUG:boto:Token: None 2022-09-02T04:30:10.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM= 2022-09-02T04:30:10.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM=', 'Content-Length': '0'} 2022-09-02T04:30:10.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae169c5de4900cee-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.240 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.241 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.241 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.242 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.243 DEBUG:boto:Method: GET 2022-09-02T04:30:10.243 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.244 DEBUG:boto:Data: 2022-09-02T04:30:10.244 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.245 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.246 DEBUG:boto:Params: {} 2022-09-02T04:30:10.247 DEBUG:boto:Token: None 2022-09-02T04:30:10.247 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM= 2022-09-02T04:30:10.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM=', 'Content-Length': '0'} 2022-09-02T04:30:10.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d39db474def1725f-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.252 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.253 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.253 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.254 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.254 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.255 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.255 DEBUG:boto:Data: 2022-09-02T04:30:10.256 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.257 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.258 DEBUG:boto:Params: {} 2022-09-02T04:30:10.258 DEBUG:boto:Token: None 2022-09-02T04:30:10.259 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VdSEdC4OeoQD2DzU5HaGVdyzfvg= 2022-09-02T04:30:10.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VdSEdC4OeoQD2DzU5HaGVdyzfvg=', 'Content-Length': '0'} 2022-09-02T04:30:10.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f03ba2bcf5878a2f-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.264 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.264 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.265 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.265 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.266 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.267 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.267 DEBUG:boto:Data: 2022-09-02T04:30:10.268 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.269 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.269 DEBUG:boto:Params: {} 2022-09-02T04:30:10.270 DEBUG:boto:Token: None 2022-09-02T04:30:10.271 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VdSEdC4OeoQD2DzU5HaGVdyzfvg= 2022-09-02T04:30:10.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VdSEdC4OeoQD2DzU5HaGVdyzfvg=', 'Content-Length': '0'} 2022-09-02T04:30:10.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000084edf67c31ed496-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:10.275 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.276 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.277 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.278 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.279 DEBUG:boto:Method: GET 2022-09-02T04:30:10.280 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.281 DEBUG:boto:Data: 2022-09-02T04:30:10.281 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.283 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.283 DEBUG:boto:Params: {} 2022-09-02T04:30:10.284 DEBUG:boto:Token: None 2022-09-02T04:30:10.285 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM= 2022-09-02T04:30:10.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM=', 'Content-Length': '0'} 2022-09-02T04:30:10.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090302258cf7fd82b-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.334 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.335 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.336 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.336 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.337 DEBUG:boto:Method: GET 2022-09-02T04:30:10.339 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.340 DEBUG:boto:Data: 2022-09-02T04:30:10.341 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.344 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.345 DEBUG:boto:Params: {} 2022-09-02T04:30:10.345 DEBUG:boto:Token: None 2022-09-02T04:30:10.346 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM= 2022-09-02T04:30:10.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM=', 'Content-Length': '0'} 2022-09-02T04:30:10.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a46c13d0d8e35c-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.350 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.351 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.352 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.353 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.353 DEBUG:boto:Method: GET 2022-09-02T04:30:10.354 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.355 DEBUG:boto:Data: 2022-09-02T04:30:10.356 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.358 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.359 DEBUG:boto:Params: {} 2022-09-02T04:30:10.361 DEBUG:boto:Token: None 2022-09-02T04:30:10.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d09e188d35f056-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.381 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.382 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.383 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.384 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.385 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.386 DEBUG:boto:Method: GET 2022-09-02T04:30:10.387 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.388 DEBUG:boto:Data: 2022-09-02T04:30:10.389 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.391 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.392 DEBUG:boto:Params: {} 2022-09-02T04:30:10.393 DEBUG:boto:Token: None 2022-09-02T04:30:10.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f63a99c929bc6d3-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.401 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.403 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.404 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.405 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.406 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.407 DEBUG:boto:Data: 2022-09-02T04:30:10.408 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.409 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.410 DEBUG:boto:Params: {} 2022-09-02T04:30:10.411 DEBUG:boto:Token: None 2022-09-02T04:30:10.412 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5KLtuRE8d9oaxNvsfqc4doLgz9U= 2022-09-02T04:30:10.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5KLtuRE8d9oaxNvsfqc4doLgz9U=', 'Content-Length': '0'} 2022-09-02T04:30:10.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3499b005d9b8945-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.419 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.419 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.420 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.421 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.422 DEBUG:boto:Data: 2022-09-02T04:30:10.423 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.424 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.425 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.426 DEBUG:boto:Params: {} 2022-09-02T04:30:10.427 DEBUG:boto:Token: None 2022-09-02T04:30:10.428 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5KLtuRE8d9oaxNvsfqc4doLgz9U= 2022-09-02T04:30:10.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5KLtuRE8d9oaxNvsfqc4doLgz9U=', 'Content-Length': '0'} 2022-09-02T04:30:10.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c97d222746bcb0b-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:30:10.435 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.436 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.436 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.437 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.438 DEBUG:boto:Method: GET 2022-09-02T04:30:10.439 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.439 DEBUG:boto:Data: 2022-09-02T04:30:10.440 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.441 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.442 DEBUG:boto:Params: {} 2022-09-02T04:30:10.443 DEBUG:boto:Token: None 2022-09-02T04:30:10.443 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM= 2022-09-02T04:30:10.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM=', 'Content-Length': '0'} 2022-09-02T04:30:10.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb020331963a4f77-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.448 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.448 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:10.449 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.449 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.450 DEBUG:boto:Method: GET 2022-09-02T04:30:10.450 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.451 DEBUG:boto:Data: 2022-09-02T04:30:10.452 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.453 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.453 DEBUG:boto:Params: {} 2022-09-02T04:30:10.454 DEBUG:boto:Token: None 2022-09-02T04:30:10.455 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:30:10.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM= 2022-09-02T04:30:10.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FRs/mjL/UbKdvE9s7FPQ9ow7ukM=', 'Content-Length': '0'} 2022-09-02T04:30:10.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae6a23dabb0ab1b-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.460 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.460 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.461 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.461 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.462 DEBUG:boto:Method: GET 2022-09-02T04:30:10.462 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.463 DEBUG:boto:Data: 2022-09-02T04:30:10.464 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.465 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.465 DEBUG:boto:Params: {} 2022-09-02T04:30:10.466 DEBUG:boto:Token: None 2022-09-02T04:30:10.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000045454bfe7bec3f1-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.471 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.472 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.473 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.473 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.474 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.475 DEBUG:boto:Method: GET 2022-09-02T04:30:10.475 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:30:10.476 DEBUG:boto:Data: 2022-09-02T04:30:10.476 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.477 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.478 DEBUG:boto:Params: {} 2022-09-02T04:30:10.479 DEBUG:boto:Token: None 2022-09-02T04:30:10.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c0a2123a9e30cc2-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.526 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:10.528 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.528 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.529 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.529 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.530 DEBUG:boto:Method: GET 2022-09-02T04:30:10.530 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.531 DEBUG:boto:Data: 2022-09-02T04:30:10.531 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.532 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.533 DEBUG:boto:Params: {} 2022-09-02T04:30:10.534 DEBUG:boto:Token: None 2022-09-02T04:30:10.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38= 2022-09-02T04:30:10.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38=', 'Content-Length': '0'} 2022-09-02T04:30:10.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c8d9e3758e6d38a-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.539 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.539 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.540 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.541 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.541 DEBUG:boto:Method: GET 2022-09-02T04:30:10.542 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.543 DEBUG:boto:Data: 2022-09-02T04:30:10.544 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.546 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.547 DEBUG:boto:Params: {} 2022-09-02T04:30:10.548 DEBUG:boto:Token: None 2022-09-02T04:30:10.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38= 2022-09-02T04:30:10.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38=', 'Content-Length': '0'} 2022-09-02T04:30:10.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0511a5677efa924-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.555 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.556 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.556 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.557 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.557 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.558 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.558 DEBUG:boto:Data: 2022-09-02T04:30:10.559 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.560 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.561 DEBUG:boto:Params: {} 2022-09-02T04:30:10.561 DEBUG:boto:Token: None 2022-09-02T04:30:10.562 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:artw/OiNwGDYBtQshxuv3njArzs= 2022-09-02T04:30:10.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:artw/OiNwGDYBtQshxuv3njArzs=', 'Content-Length': '0'} 2022-09-02T04:30:10.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e27f45609970037-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.566 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.567 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.567 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.568 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.568 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.569 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.569 DEBUG:boto:Data: 2022-09-02T04:30:10.570 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.571 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.572 DEBUG:boto:Params: {} 2022-09-02T04:30:10.572 DEBUG:boto:Token: None 2022-09-02T04:30:10.573 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:artw/OiNwGDYBtQshxuv3njArzs= 2022-09-02T04:30:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:artw/OiNwGDYBtQshxuv3njArzs=', 'Content-Length': '0'} 2022-09-02T04:30:10.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009779c308980220df-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:10.578 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.578 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.579 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.579 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.580 DEBUG:boto:Method: GET 2022-09-02T04:30:10.580 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.581 DEBUG:boto:Data: 2022-09-02T04:30:10.581 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.583 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.583 DEBUG:boto:Params: {} 2022-09-02T04:30:10.584 DEBUG:boto:Token: None 2022-09-02T04:30:10.584 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38= 2022-09-02T04:30:10.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38=', 'Content-Length': '0'} 2022-09-02T04:30:10.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057e85314cbad23d6-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.589 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.589 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.590 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.590 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.591 DEBUG:boto:Method: GET 2022-09-02T04:30:10.592 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.592 DEBUG:boto:Data: 2022-09-02T04:30:10.593 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.594 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.594 DEBUG:boto:Params: {} 2022-09-02T04:30:10.595 DEBUG:boto:Token: None 2022-09-02T04:30:10.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38= 2022-09-02T04:30:10.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38=', 'Content-Length': '0'} 2022-09-02T04:30:10.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000183b8ee1fb554dd5-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.600 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.600 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.601 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.602 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.602 DEBUG:boto:Method: GET 2022-09-02T04:30:10.603 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.603 DEBUG:boto:Data: 2022-09-02T04:30:10.604 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.605 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.606 DEBUG:boto:Params: {} 2022-09-02T04:30:10.606 DEBUG:boto:Token: None 2022-09-02T04:30:10.607 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f416830c9395139e-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.612 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.612 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.613 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.614 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.614 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.615 DEBUG:boto:Method: GET 2022-09-02T04:30:10.615 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.616 DEBUG:boto:Data: 2022-09-02T04:30:10.616 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.618 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.618 DEBUG:boto:Params: {} 2022-09-02T04:30:10.619 DEBUG:boto:Token: None 2022-09-02T04:30:10.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e89bc4a772ac4fa6-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.625 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.625 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.626 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.627 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.627 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.628 DEBUG:boto:Data: 2022-09-02T04:30:10.628 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.629 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.630 DEBUG:boto:Params: {} 2022-09-02T04:30:10.630 DEBUG:boto:Token: None 2022-09-02T04:30:10.631 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GLOjpnIVxxKtvMdxcNdZu68U8oU= 2022-09-02T04:30:10.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GLOjpnIVxxKtvMdxcNdZu68U8oU=', 'Content-Length': '0'} 2022-09-02T04:30:10.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010db9b38bfe95c6b-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.636 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.636 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.637 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.637 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.638 DEBUG:boto:Data: 2022-09-02T04:30:10.638 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.639 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.640 DEBUG:boto:Params: {} 2022-09-02T04:30:10.640 DEBUG:boto:Token: None 2022-09-02T04:30:10.641 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GLOjpnIVxxKtvMdxcNdZu68U8oU= 2022-09-02T04:30:10.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GLOjpnIVxxKtvMdxcNdZu68U8oU=', 'Content-Length': '0'} 2022-09-02T04:30:10.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003986218ec3f29ea3-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:30:10.646 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.647 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.647 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.648 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.648 DEBUG:boto:Method: GET 2022-09-02T04:30:10.649 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.649 DEBUG:boto:Data: 2022-09-02T04:30:10.650 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.651 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.651 DEBUG:boto:Params: {} 2022-09-02T04:30:10.652 DEBUG:boto:Token: None 2022-09-02T04:30:10.653 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38= 2022-09-02T04:30:10.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38=', 'Content-Length': '0'} 2022-09-02T04:30:10.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000404707d8ed8d569b-00631186d2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.702 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.703 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:10.704 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.704 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.705 DEBUG:boto:Method: GET 2022-09-02T04:30:10.705 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.706 DEBUG:boto:Data: 2022-09-02T04:30:10.707 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.708 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.709 DEBUG:boto:Params: {} 2022-09-02T04:30:10.709 DEBUG:boto:Token: None 2022-09-02T04:30:10.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:30:10.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38= 2022-09-02T04:30:10.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGPuhbb/AdTdz6sQfHNj92jHa38=', 'Content-Length': '0'} 2022-09-02T04:30:10.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:29:02 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dad601d244ceaf4e-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.714 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.715 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.715 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.716 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.717 DEBUG:boto:Method: GET 2022-09-02T04:30:10.717 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.718 DEBUG:boto:Data: 2022-09-02T04:30:10.718 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.719 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.720 DEBUG:boto:Params: {} 2022-09-02T04:30:10.721 DEBUG:boto:Token: None 2022-09-02T04:30:10.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2715c6d771d329-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.726 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.727 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.727 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.728 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.728 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.729 DEBUG:boto:Method: GET 2022-09-02T04:30:10.729 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:30:10.730 DEBUG:boto:Data: 2022-09-02T04:30:10.731 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.732 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.732 DEBUG:boto:Params: {} 2022-09-02T04:30:10.733 DEBUG:boto:Token: None 2022-09-02T04:30:10.734 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE= 2022-09-02T04:30:10.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8t5yobtTc+mKfb6k6bPwM1qpBE=', 'Content-Length': '0'} 2022-09-02T04:30:10.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e6cd8e95e778709-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.739 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.740 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-18 zones={test-zone3, test-zone2} 2022-09-02T04:30:10.740 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:30:10.741 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:30:10.741 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.742 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:30:10.743 DEBUG:boto:Data: 2022-09-02T04:30:10.743 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.744 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.745 DEBUG:boto:Params: {} 2022-09-02T04:30:10.746 DEBUG:boto:Token: None 2022-09-02T04:30:10.747 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-16/ 2022-09-02T04:30:10.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:07kVcBBWttAWbYkwyBXMVdiE+Rs= 2022-09-02T04:30:10.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:07kVcBBWttAWbYkwyBXMVdiE+Rs=', 'Content-Length': '0'} 2022-09-02T04:30:10.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2721263f46647ba-00631186d2-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.753 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:30:10.753 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:30:10.754 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.755 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.755 DEBUG:boto:Method: GET 2022-09-02T04:30:10.756 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.756 DEBUG:boto:Data: 2022-09-02T04:30:10.757 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.758 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.758 DEBUG:boto:Params: {} 2022-09-02T04:30:10.759 DEBUG:boto:Token: None 2022-09-02T04:30:10.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-16/?versions 2022-09-02T04:30:10.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:luYEfZujAyTUBR6EJ+KzMpW9r7M= 2022-09-02T04:30:10.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:luYEfZujAyTUBR6EJ+KzMpW9r7M=', 'Content-Length': '0'} 2022-09-02T04:30:10.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f10c2e58ecc0fb-00631186d2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.764 DEBUG:boto:b'xljbkf-16obj-test-zone1999falseobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.765 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:30:10.766 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:30:10.766 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?acl 2022-09-02T04:30:10.767 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?acl 2022-09-02T04:30:10.767 DEBUG:boto:Method: PUT 2022-09-02T04:30:10.768 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?acl 2022-09-02T04:30:10.768 DEBUG:boto:Data: 2022-09-02T04:30:10.769 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:10.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.770 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.770 DEBUG:boto:Params: {} 2022-09-02T04:30:10.771 DEBUG:boto:Token: None 2022-09-02T04:30:10.772 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:10 GMT x-amz-acl:public-read /xljbkf-16/obj-test-zone1?acl 2022-09-02T04:30:10.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3nD68zD9b2Cbh1TUdRzbscNxC4E= 2022-09-02T04:30:10.773 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3nD68zD9b2Cbh1TUdRzbscNxC4E=', 'Content-Length': '0'} 2022-09-02T04:30:10.780 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d60bd1916bab749a-00631186d2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.780 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:30:10.781 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:30:10.781 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.782 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:30:10.783 DEBUG:boto:Data: 2022-09-02T04:30:10.783 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.784 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.785 DEBUG:boto:Params: {} 2022-09-02T04:30:10.785 DEBUG:boto:Token: None 2022-09-02T04:30:10.786 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-16/ 2022-09-02T04:30:10.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:07kVcBBWttAWbYkwyBXMVdiE+Rs= 2022-09-02T04:30:10.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:07kVcBBWttAWbYkwyBXMVdiE+Rs=', 'Content-Length': '0'} 2022-09-02T04:30:10.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da05b28df01175c-00631186d2-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.790 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:30:10.791 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:30:10.791 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:10.792 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:10.793 DEBUG:boto:Method: GET 2022-09-02T04:30:10.793 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:10.794 DEBUG:boto:Data: 2022-09-02T04:30:10.794 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.795 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.796 DEBUG:boto:Params: {} 2022-09-02T04:30:10.797 DEBUG:boto:Token: None 2022-09-02T04:30:10.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-16/?versions 2022-09-02T04:30:10.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:luYEfZujAyTUBR6EJ+KzMpW9r7M= 2022-09-02T04:30:10.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:luYEfZujAyTUBR6EJ+KzMpW9r7M=', 'Content-Length': '0'} 2022-09-02T04:30:10.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b684152a59066b2-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.802 DEBUG:boto:b'xljbkf-16obj-test-zone2999falseobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.802 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:30:10.803 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:30:10.804 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?acl 2022-09-02T04:30:10.804 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?acl 2022-09-02T04:30:10.805 DEBUG:boto:Method: PUT 2022-09-02T04:30:10.805 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?acl 2022-09-02T04:30:10.806 DEBUG:boto:Data: 2022-09-02T04:30:10.806 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:10.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.807 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.808 DEBUG:boto:Params: {} 2022-09-02T04:30:10.808 DEBUG:boto:Token: None 2022-09-02T04:30:10.809 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:10 GMT x-amz-acl:public-read /xljbkf-16/obj-test-zone2?acl 2022-09-02T04:30:10.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rlDYSuaZsCCDikYnYWu0t4sda8g= 2022-09-02T04:30:10.810 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rlDYSuaZsCCDikYnYWu0t4sda8g=', 'Content-Length': '0'} 2022-09-02T04:30:10.818 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd696d20335502f7-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.818 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:30:10.819 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:30:10.819 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.820 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:30:10.820 DEBUG:boto:Data: 2022-09-02T04:30:10.821 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.822 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.822 DEBUG:boto:Params: {} 2022-09-02T04:30:10.823 DEBUG:boto:Token: None 2022-09-02T04:30:10.824 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-16/ 2022-09-02T04:30:10.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:07kVcBBWttAWbYkwyBXMVdiE+Rs= 2022-09-02T04:30:10.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:07kVcBBWttAWbYkwyBXMVdiE+Rs=', 'Content-Length': '0'} 2022-09-02T04:30:10.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d957d310c47eb66-00631186d2-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.828 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:30:10.828 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:30:10.829 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:10.830 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:10.830 DEBUG:boto:Method: GET 2022-09-02T04:30:10.831 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:10.831 DEBUG:boto:Data: 2022-09-02T04:30:10.832 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.833 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.833 DEBUG:boto:Params: {} 2022-09-02T04:30:10.834 DEBUG:boto:Token: None 2022-09-02T04:30:10.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-16/?versions 2022-09-02T04:30:10.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:luYEfZujAyTUBR6EJ+KzMpW9r7M= 2022-09-02T04:30:10.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:luYEfZujAyTUBR6EJ+KzMpW9r7M=', 'Content-Length': '0'} 2022-09-02T04:30:10.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caea836ca90de37d-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.839 DEBUG:boto:b'xljbkf-16obj-test-zone3999falseobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.840 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:30:10.841 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:30:10.841 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?acl 2022-09-02T04:30:10.842 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?acl 2022-09-02T04:30:10.842 DEBUG:boto:Method: PUT 2022-09-02T04:30:10.843 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?acl 2022-09-02T04:30:10.843 DEBUG:boto:Data: 2022-09-02T04:30:10.844 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:10.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.845 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.845 DEBUG:boto:Params: {} 2022-09-02T04:30:10.846 DEBUG:boto:Token: None 2022-09-02T04:30:10.847 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:10 GMT x-amz-acl:public-read /xljbkf-16/obj-test-zone3?acl 2022-09-02T04:30:10.847 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UrEu3BqX/Wlfj/ck/6D/Ofa61Yg= 2022-09-02T04:30:10.848 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UrEu3BqX/Wlfj/ck/6D/Ofa61Yg=', 'Content-Length': '0'} 2022-09-02T04:30:10.862 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004331b89eddd9c4d5-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.863 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:10.863 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:10.864 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.864 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:10.865 DEBUG:boto:Data: 2022-09-02T04:30:10.865 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.866 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.867 DEBUG:boto:Params: {} 2022-09-02T04:30:10.867 DEBUG:boto:Token: None 2022-09-02T04:30:10.868 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-17/ 2022-09-02T04:30:10.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vljdwEWoz5pyUfGULd+wqGQL10= 2022-09-02T04:30:10.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vljdwEWoz5pyUfGULd+wqGQL10=', 'Content-Length': '0'} 2022-09-02T04:30:10.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe8e98780d419394-00631186d2-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.873 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:10.873 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:10.874 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.874 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.875 DEBUG:boto:Method: GET 2022-09-02T04:30:10.875 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.876 DEBUG:boto:Data: 2022-09-02T04:30:10.876 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.877 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.878 DEBUG:boto:Params: {} 2022-09-02T04:30:10.878 DEBUG:boto:Token: None 2022-09-02T04:30:10.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-17/?versions 2022-09-02T04:30:10.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fkk+BacunGqHQOKpEfpKFnmFRLQ= 2022-09-02T04:30:10.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fkk+BacunGqHQOKpEfpKFnmFRLQ=', 'Content-Length': '0'} 2022-09-02T04:30:10.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000035a6ff43f6e7d9e-00631186d2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.884 DEBUG:boto:b'xljbkf-17obj-test-zone1999falseobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.885 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:10.886 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:30:10.886 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?acl 2022-09-02T04:30:10.887 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?acl 2022-09-02T04:30:10.887 DEBUG:boto:Method: PUT 2022-09-02T04:30:10.888 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?acl 2022-09-02T04:30:10.888 DEBUG:boto:Data: 2022-09-02T04:30:10.889 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:10.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.890 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.890 DEBUG:boto:Params: {} 2022-09-02T04:30:10.891 DEBUG:boto:Token: None 2022-09-02T04:30:10.891 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:10 GMT x-amz-acl:public-read /xljbkf-17/obj-test-zone1?acl 2022-09-02T04:30:10.892 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WHwPqWhqK3cPMHXVmS/KgQXOVWw= 2022-09-02T04:30:10.892 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WHwPqWhqK3cPMHXVmS/KgQXOVWw=', 'Content-Length': '0'} 2022-09-02T04:30:10.899 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b79fee0feb28686-00631186d2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.900 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:10.901 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:10.901 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.902 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:10.902 DEBUG:boto:Data: 2022-09-02T04:30:10.903 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.904 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.904 DEBUG:boto:Params: {} 2022-09-02T04:30:10.905 DEBUG:boto:Token: None 2022-09-02T04:30:10.905 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-17/ 2022-09-02T04:30:10.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vljdwEWoz5pyUfGULd+wqGQL10= 2022-09-02T04:30:10.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vljdwEWoz5pyUfGULd+wqGQL10=', 'Content-Length': '0'} 2022-09-02T04:30:10.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000014530af45e9e289-00631186d2-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.910 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:10.911 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:10.911 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:10.912 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:10.912 DEBUG:boto:Method: GET 2022-09-02T04:30:10.913 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:10.913 DEBUG:boto:Data: 2022-09-02T04:30:10.914 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.915 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.916 DEBUG:boto:Params: {} 2022-09-02T04:30:10.916 DEBUG:boto:Token: None 2022-09-02T04:30:10.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-17/?versions 2022-09-02T04:30:10.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fkk+BacunGqHQOKpEfpKFnmFRLQ= 2022-09-02T04:30:10.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fkk+BacunGqHQOKpEfpKFnmFRLQ=', 'Content-Length': '0'} 2022-09-02T04:30:10.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091e0d511b807ecbb-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.922 DEBUG:boto:b'xljbkf-17obj-test-zone2999falseobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.922 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:10.923 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:30:10.923 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?acl 2022-09-02T04:30:10.924 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?acl 2022-09-02T04:30:10.924 DEBUG:boto:Method: PUT 2022-09-02T04:30:10.925 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?acl 2022-09-02T04:30:10.925 DEBUG:boto:Data: 2022-09-02T04:30:10.926 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:10.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:10.927 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.927 DEBUG:boto:Params: {} 2022-09-02T04:30:10.928 DEBUG:boto:Token: None 2022-09-02T04:30:10.929 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:10 GMT x-amz-acl:public-read /xljbkf-17/obj-test-zone2?acl 2022-09-02T04:30:10.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jj7/IILJm0+fsbPaDJKF+tcB0kE= 2022-09-02T04:30:10.930 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jj7/IILJm0+fsbPaDJKF+tcB0kE=', 'Content-Length': '0'} 2022-09-02T04:30:10.942 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cff70d458937cc65-00631186d2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.943 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:10.944 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:10.944 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.945 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:30:10.945 DEBUG:boto:Data: 2022-09-02T04:30:10.946 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.947 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.947 DEBUG:boto:Params: {} 2022-09-02T04:30:10.948 DEBUG:boto:Token: None 2022-09-02T04:30:10.948 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-17/ 2022-09-02T04:30:10.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vljdwEWoz5pyUfGULd+wqGQL10= 2022-09-02T04:30:10.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vljdwEWoz5pyUfGULd+wqGQL10=', 'Content-Length': '0'} 2022-09-02T04:30:10.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006731e7715ddd6bc1-00631186d2-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.953 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:30:10.954 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:30:10.955 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:10.955 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:10.956 DEBUG:boto:Method: GET 2022-09-02T04:30:10.956 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:10.957 DEBUG:boto:Data: 2022-09-02T04:30:10.957 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.958 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.959 DEBUG:boto:Params: {} 2022-09-02T04:30:10.959 DEBUG:boto:Token: None 2022-09-02T04:30:10.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-17/?versions 2022-09-02T04:30:10.961 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fkk+BacunGqHQOKpEfpKFnmFRLQ= 2022-09-02T04:30:10.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fkk+BacunGqHQOKpEfpKFnmFRLQ=', 'Content-Length': '0'} 2022-09-02T04:30:10.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99b265a4b7cd55b-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.965 DEBUG:boto:b'xljbkf-17obj-test-zone3999falseobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:10.966 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:10.966 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:30:10.967 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?acl 2022-09-02T04:30:10.967 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?acl 2022-09-02T04:30:10.968 DEBUG:boto:Method: PUT 2022-09-02T04:30:10.968 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?acl 2022-09-02T04:30:10.969 DEBUG:boto:Data: 2022-09-02T04:30:10.969 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:10.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:10.970 DEBUG:boto:Port: 8001 2022-09-02T04:30:10.971 DEBUG:boto:Params: {} 2022-09-02T04:30:10.971 DEBUG:boto:Token: None 2022-09-02T04:30:10.972 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:10 GMT x-amz-acl:public-read /xljbkf-17/obj-test-zone3?acl 2022-09-02T04:30:10.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uVNXsiErWFaPhMp9KnTdNXNM75g= 2022-09-02T04:30:10.973 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uVNXsiErWFaPhMp9KnTdNXNM75g=', 'Content-Length': '0'} 2022-09-02T04:30:10.980 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e69c41a31ec520cc-00631186d2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.981 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.981 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.982 DEBUG:boto:Method: HEAD 2022-09-02T04:30:10.983 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:10.983 DEBUG:boto:Data: 2022-09-02T04:30:10.984 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.985 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.985 DEBUG:boto:Params: {} 2022-09-02T04:30:10.986 DEBUG:boto:Token: None 2022-09-02T04:30:10.987 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/ 2022-09-02T04:30:10.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MQAAzg80s9IGcnF1SVz43pVmwqM= 2022-09-02T04:30:10.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MQAAzg80s9IGcnF1SVz43pVmwqM=', 'Content-Length': '0'} 2022-09-02T04:30:10.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008082c3d210036010-00631186d2-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:10.991 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:10.992 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:10.992 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.993 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.993 DEBUG:boto:Method: GET 2022-09-02T04:30:10.994 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone1 2022-09-02T04:30:10.995 DEBUG:boto:Data: 2022-09-02T04:30:10.995 DEBUG:boto:Headers: {} 2022-09-02T04:30:10.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:10.996 DEBUG:boto:Port: 8000 2022-09-02T04:30:10.997 DEBUG:boto:Params: {} 2022-09-02T04:30:10.997 DEBUG:boto:Token: None 2022-09-02T04:30:10.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:10 GMT /xljbkf-18/?versions 2022-09-02T04:30:10.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5ifFWvwTjFMuABbszdIYmsVpRU= 2022-09-02T04:30:10.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5ifFWvwTjFMuABbszdIYmsVpRU=', 'Content-Length': '0'} 2022-09-02T04:30:11.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087daf6d0519a1f04-00631186d3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.003 DEBUG:boto:b'xljbkf-18obj-test-zone1999falseobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:11.004 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:11.005 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:30:11.006 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?acl 2022-09-02T04:30:11.007 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?acl 2022-09-02T04:30:11.008 DEBUG:boto:Method: PUT 2022-09-02T04:30:11.009 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?acl 2022-09-02T04:30:11.010 DEBUG:boto:Data: 2022-09-02T04:30:11.011 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:11.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:30:11.013 DEBUG:boto:Port: 8000 2022-09-02T04:30:11.014 DEBUG:boto:Params: {} 2022-09-02T04:30:11.015 DEBUG:boto:Token: None 2022-09-02T04:30:11.016 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:11 GMT x-amz-acl:public-read /xljbkf-18/obj-test-zone1?acl 2022-09-02T04:30:11.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V9nyYOmwRel9MfNTxX9ECqlLhdE= 2022-09-02T04:30:11.018 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V9nyYOmwRel9MfNTxX9ECqlLhdE=', 'Content-Length': '0'} 2022-09-02T04:30:11.026 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a66edd5e5755cf13-00631186d3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.026 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:11.027 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:11.028 DEBUG:boto:Method: HEAD 2022-09-02T04:30:11.029 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:11.029 DEBUG:boto:Data: 2022-09-02T04:30:11.030 DEBUG:boto:Headers: {} 2022-09-02T04:30:11.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:11.031 DEBUG:boto:Port: 8000 2022-09-02T04:30:11.031 DEBUG:boto:Params: {} 2022-09-02T04:30:11.032 DEBUG:boto:Token: None 2022-09-02T04:30:11.033 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:11 GMT /xljbkf-18/ 2022-09-02T04:30:11.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SoXAW/7GFK67dR1p0pkUoDSeubM= 2022-09-02T04:30:11.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SoXAW/7GFK67dR1p0pkUoDSeubM=', 'Content-Length': '0'} 2022-09-02T04:30:11.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000795747403cb25eda-00631186d3-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.038 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:11.039 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:11.040 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:11.040 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:11.041 DEBUG:boto:Method: GET 2022-09-02T04:30:11.041 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone2 2022-09-02T04:30:11.042 DEBUG:boto:Data: 2022-09-02T04:30:11.043 DEBUG:boto:Headers: {} 2022-09-02T04:30:11.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:11.044 DEBUG:boto:Port: 8000 2022-09-02T04:30:11.044 DEBUG:boto:Params: {} 2022-09-02T04:30:11.045 DEBUG:boto:Token: None 2022-09-02T04:30:11.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:11 GMT /xljbkf-18/?versions 2022-09-02T04:30:11.047 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zv5Byg2dtc8G8zPZmN6c1JomU+E= 2022-09-02T04:30:11.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zv5Byg2dtc8G8zPZmN6c1JomU+E=', 'Content-Length': '0'} 2022-09-02T04:30:11.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf4abccf2a3df32-00631186d3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.052 DEBUG:boto:b'xljbkf-18obj-test-zone2999falseobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:11.053 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:11.054 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:30:11.055 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?acl 2022-09-02T04:30:11.056 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?acl 2022-09-02T04:30:11.057 DEBUG:boto:Method: PUT 2022-09-02T04:30:11.058 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?acl 2022-09-02T04:30:11.058 DEBUG:boto:Data: 2022-09-02T04:30:11.059 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:11.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:30:11.060 DEBUG:boto:Port: 8000 2022-09-02T04:30:11.060 DEBUG:boto:Params: {} 2022-09-02T04:30:11.061 DEBUG:boto:Token: None 2022-09-02T04:30:11.061 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:11 GMT x-amz-acl:public-read /xljbkf-18/obj-test-zone2?acl 2022-09-02T04:30:11.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghND9v9NP2GxBbFIw3DqY7eM3xI= 2022-09-02T04:30:11.063 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghND9v9NP2GxBbFIw3DqY7eM3xI=', 'Content-Length': '0'} 2022-09-02T04:30:11.070 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac921e84a7dd5e00-00631186d3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.071 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:11.072 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:11.073 DEBUG:boto:Method: HEAD 2022-09-02T04:30:11.073 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:30:11.074 DEBUG:boto:Data: 2022-09-02T04:30:11.075 DEBUG:boto:Headers: {} 2022-09-02T04:30:11.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:11.077 DEBUG:boto:Port: 8001 2022-09-02T04:30:11.078 DEBUG:boto:Params: {} 2022-09-02T04:30:11.079 DEBUG:boto:Token: None 2022-09-02T04:30:11.080 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:30:11 GMT /xljbkf-18/ 2022-09-02T04:30:11.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SoXAW/7GFK67dR1p0pkUoDSeubM= 2022-09-02T04:30:11.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SoXAW/7GFK67dR1p0pkUoDSeubM=', 'Content-Length': '0'} 2022-09-02T04:30:11.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062f1ff79efb0b35a-00631186d3-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.087 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:30:11.088 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:30:11.089 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:11.090 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:11.091 DEBUG:boto:Method: GET 2022-09-02T04:30:11.092 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999&prefix=obj-test-zone3 2022-09-02T04:30:11.093 DEBUG:boto:Data: 2022-09-02T04:30:11.093 DEBUG:boto:Headers: {} 2022-09-02T04:30:11.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:11.095 DEBUG:boto:Port: 8001 2022-09-02T04:30:11.095 DEBUG:boto:Params: {} 2022-09-02T04:30:11.096 DEBUG:boto:Token: None 2022-09-02T04:30:11.097 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:30:11 GMT /xljbkf-18/?versions 2022-09-02T04:30:11.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zv5Byg2dtc8G8zPZmN6c1JomU+E= 2022-09-02T04:30:11.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zv5Byg2dtc8G8zPZmN6c1JomU+E=', 'Content-Length': '0'} 2022-09-02T04:30:11.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f47043e51b359c0f-00631186d3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.101 DEBUG:boto:b'xljbkf-18obj-test-zone3999falseobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:30:11.102 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:11.103 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:30:11.103 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?acl 2022-09-02T04:30:11.104 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?acl 2022-09-02T04:30:11.104 DEBUG:boto:Method: PUT 2022-09-02T04:30:11.105 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?acl 2022-09-02T04:30:11.105 DEBUG:boto:Data: 2022-09-02T04:30:11.106 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:30:11.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:30:11.107 DEBUG:boto:Port: 8001 2022-09-02T04:30:11.107 DEBUG:boto:Params: {} 2022-09-02T04:30:11.108 DEBUG:boto:Token: None 2022-09-02T04:30:11.108 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:30:11 GMT x-amz-acl:public-read /xljbkf-18/obj-test-zone3?acl 2022-09-02T04:30:11.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yfx7RaaGtsa/y1Xxe6FhIPqZo9k= 2022-09-02T04:30:11.109 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:30:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yfx7RaaGtsa/y1Xxe6FhIPqZo9k=', 'Content-Length': '0'} 2022-09-02T04:30:11.122 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5845fea355b80aa-00631186d3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:30:11.123 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-16 2022-09-02T04:30:11.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:11.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:11.125 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:11.190 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:11.191 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:12.252 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:12.251+0000 7f378f0c6600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:12.254 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:12.254 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:17.252 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:17.252+0000 7f378f0c6600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:17.252 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:17.253 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:22.254 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:22.253+0000 7f378f0c6600 1 bucket sync caught up with source: 2022-09-02T04:30:22.255 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:22.255 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:22.256 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:22.253+0000 7f378f0c6600 0 bucket checkpoint complete 2022-09-02T04:30:22.269 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-16 2022-09-02T04:30:22.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:22.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:22.272 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:22.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:22.339 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:22.397 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:22.397+0000 7fabda4b8600 1 bucket sync caught up with source: 2022-09-02T04:30:22.398 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:22.398 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:22.398 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:22.397+0000 7fabda4b8600 0 bucket checkpoint complete 2022-09-02T04:30:22.415 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-16 2022-09-02T04:30:22.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:22.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:22.418 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:22.482 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:22.483 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:22.521 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:22.520+0000 7fc270900600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:22.522 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:22.522 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:27.523 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:27.522+0000 7fc270900600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:27.523 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000010.31.5, , 00000000003.6.9, , 00000000008.23.9, , , , , ] 2022-09-02T04:30:27.524 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:32.525 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:32.523+0000 7fc270900600 1 bucket sync caught up with source: 2022-09-02T04:30:32.526 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:32.526 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.40.5, , 00000000003.6.9, , 00000000009.26.5, , , , , ] 2022-09-02T04:30:32.527 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:30:32.523+0000 7fc270900600 0 bucket checkpoint complete 2022-09-02T04:30:32.545 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-16 2022-09-02T04:30:32.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:32.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:30:32.548 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:30:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:30:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:30:32.663 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:32.663+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:32.664 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.68.9, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:32.664 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:37.665 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:37.664+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:37.665 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.68.9, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:37.666 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:42.665+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:42.666 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:42.667 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:47.668 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:47.667+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:47.668 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:47.669 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:52.669 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:52.668+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:52.669 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:52.670 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:57.670 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:30:57.670+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:30:57.671 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:30:57.671 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:02.671+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:02.673 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:02.673 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:07.673 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:07.672+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:07.674 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:07.674 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:12.674+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:12.675 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:12.675 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:17.676 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:17.675+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:17.676 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:17.677 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:22.677 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:22.677+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:22.678 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:22.678 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:27.679 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:27.678+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:27.679 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:27.679 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:32.680 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:32.679+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:32.680 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:32.680 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:37.682 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:37.681+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:37.682 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:37.683 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:42.683 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:42.682+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:42.683 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:42.684 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:47.684 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:47.684+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:47.685 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:47.685 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:52.685+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:52.686 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:52.687 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:31:57.686+0000 7f995c88c600 1 waiting for incremental sync to catch up: 2022-09-02T04:31:57.688 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000010.25.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:31:57.688 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:32:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:02.688+0000 7f995c88c600 1 bucket sync caught up with source: 2022-09-02T04:32:02.689 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:32:02.690 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:32:02.690 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:02.688+0000 7f995c88c600 0 bucket checkpoint complete 2022-09-02T04:32:02.708 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-16 2022-09-02T04:32:02.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:02.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:02.711 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:02.772 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:02.772 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:02.811 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:02.810+0000 7f9059c79600 1 bucket sync caught up with source: 2022-09-02T04:32:02.811 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:32:02.811 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.73.5, , 00000000011.28.5, , 00000000008.51.5, , , , , ] 2022-09-02T04:32:02.812 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:02.810+0000 7f9059c79600 0 bucket checkpoint complete 2022-09-02T04:32:02.824 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-16 2022-09-02T04:32:02.825 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-16 2022-09-02T04:32:02.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:02.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:02.827 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:02.885 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:02.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:02.919 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:02.919+0000 7f815853e600 1 bucket sync caught up with source: 2022-09-02T04:32:02.920 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.43.5, , 00000000004.10.5, , 00000000015.56.5, , , , , ] 2022-09-02T04:32:02.921 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.43.5, , 00000000004.10.5, , 00000000015.56.5, , , , , ] 2022-09-02T04:32:02.921 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:02.919+0000 7f815853e600 0 bucket checkpoint complete 2022-09-02T04:32:02.942 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-16 2022-09-02T04:32:02.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:02.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:02.944 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:03.007 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:03.008 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:03.046 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:03.045+0000 7fbba57e7600 1 bucket sync caught up with source: 2022-09-02T04:32:03.046 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000004.43.5, , 00000000004.10.5, , 00000000015.56.5, , , , , ] 2022-09-02T04:32:03.047 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000004.43.5, , 00000000004.10.5, , 00000000015.56.5, , , , , ] 2022-09-02T04:32:03.047 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:03.045+0000 7fbba57e7600 0 bucket checkpoint complete 2022-09-02T04:32:03.070 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-16 2022-09-02T04:32:03.071 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-16 zones={test-zone2, test-zone1} 2022-09-02T04:32:03.072 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.073 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.073 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.074 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:32:03.074 DEBUG:boto:Data: 2022-09-02T04:32:03.075 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.076 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.077 DEBUG:boto:Params: {} 2022-09-02T04:32:03.078 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:32:03.079 DEBUG:boto:Token: None 2022-09-02T04:32:03.079 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FSoyvHUNmR+z8R/T44gRIjMoZuk= 2022-09-02T04:32:03.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FSoyvHUNmR+z8R/T44gRIjMoZuk=', 'Content-Length': '0'} 2022-09-02T04:32:03.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0c25e5c0e7c23af-0063118743-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.086 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.087 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.087 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.088 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:32:03.088 DEBUG:boto:Data: 2022-09-02T04:32:03.089 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.090 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.090 DEBUG:boto:Params: {} 2022-09-02T04:32:03.091 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:32:03.091 DEBUG:boto:Token: None 2022-09-02T04:32:03.092 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FSoyvHUNmR+z8R/T44gRIjMoZuk= 2022-09-02T04:32:03.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FSoyvHUNmR+z8R/T44gRIjMoZuk=', 'Content-Length': '0'} 2022-09-02T04:32:03.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce4da5525e0e6573-0063118743-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.097 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:03.098 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.098 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.099 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.100 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.100 DEBUG:boto:Method: GET 2022-09-02T04:32:03.101 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.101 DEBUG:boto:Data: 2022-09-02T04:32:03.102 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.103 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.103 DEBUG:boto:Params: {} 2022-09-02T04:32:03.104 DEBUG:boto:Token: None 2022-09-02T04:32:03.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/?versions 2022-09-02T04:32:03.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I= 2022-09-02T04:32:03.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I=', 'Content-Length': '0'} 2022-09-02T04:32:03.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048844b55a8cbd831-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.109 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.111 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:03.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:03.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:03.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:03.113 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:03.113 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.114 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.114 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.115 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.115 DEBUG:boto:Method: GET 2022-09-02T04:32:03.116 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.117 DEBUG:boto:Data: 2022-09-02T04:32:03.117 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.118 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.119 DEBUG:boto:Params: {} 2022-09-02T04:32:03.119 DEBUG:boto:Token: None 2022-09-02T04:32:03.120 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/?versions 2022-09-02T04:32:03.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I= 2022-09-02T04:32:03.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I=', 'Content-Length': '0'} 2022-09-02T04:32:03.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f299901ab545d52-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.124 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.125 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:03.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:03.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:03.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:03.127 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.128 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.128 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.129 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.130 DEBUG:boto:Method: GET 2022-09-02T04:32:03.130 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.131 DEBUG:boto:Data: 2022-09-02T04:32:03.131 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.132 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.133 DEBUG:boto:Params: {} 2022-09-02T04:32:03.133 DEBUG:boto:Token: None 2022-09-02T04:32:03.134 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/?versions 2022-09-02T04:32:03.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I= 2022-09-02T04:32:03.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I=', 'Content-Length': '0'} 2022-09-02T04:32:03.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000865855a98960d4d7-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.138 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.140 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.140 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.141 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.142 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.142 DEBUG:boto:Method: GET 2022-09-02T04:32:03.143 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:03.143 DEBUG:boto:Data: 2022-09-02T04:32:03.144 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.145 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.145 DEBUG:boto:Params: {} 2022-09-02T04:32:03.146 DEBUG:boto:Token: None 2022-09-02T04:32:03.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/?versions 2022-09-02T04:32:03.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I= 2022-09-02T04:32:03.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IR1w8jBgn5huIgUITGRSVljBX7I=', 'Content-Length': '0'} 2022-09-02T04:32:03.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f40db1599094b6-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.151 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:03.153 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.153 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.154 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.155 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.155 DEBUG:boto:Method: GET 2022-09-02T04:32:03.156 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.156 DEBUG:boto:Data: 2022-09-02T04:32:03.157 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.158 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.159 DEBUG:boto:Params: {} 2022-09-02T04:32:03.160 DEBUG:boto:Token: None 2022-09-02T04:32:03.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w= 2022-09-02T04:32:03.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w=', 'Content-Length': '0'} 2022-09-02T04:32:03.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c1e04ffbd3d32f6-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.165 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.165 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.166 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.166 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.167 DEBUG:boto:Method: GET 2022-09-02T04:32:03.168 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.168 DEBUG:boto:Data: 2022-09-02T04:32:03.169 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.170 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.170 DEBUG:boto:Params: {} 2022-09-02T04:32:03.171 DEBUG:boto:Token: None 2022-09-02T04:32:03.171 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.172 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w= 2022-09-02T04:32:03.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w=', 'Content-Length': '0'} 2022-09-02T04:32:03.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6df0d2880395691-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.176 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.177 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.177 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.178 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.178 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.179 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.179 DEBUG:boto:Data: 2022-09-02T04:32:03.180 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.181 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.181 DEBUG:boto:Params: {} 2022-09-02T04:32:03.182 DEBUG:boto:Token: None 2022-09-02T04:32:03.183 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uJyyb+Po3I8Z5R4cSTQyaxQJfa0= 2022-09-02T04:32:03.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uJyyb+Po3I8Z5R4cSTQyaxQJfa0=', 'Content-Length': '0'} 2022-09-02T04:32:03.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008891bb028cd4add9-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.187 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.188 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.188 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.189 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.189 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.190 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.190 DEBUG:boto:Data: 2022-09-02T04:32:03.191 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.192 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.192 DEBUG:boto:Params: {} 2022-09-02T04:32:03.193 DEBUG:boto:Token: None 2022-09-02T04:32:03.193 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uJyyb+Po3I8Z5R4cSTQyaxQJfa0= 2022-09-02T04:32:03.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uJyyb+Po3I8Z5R4cSTQyaxQJfa0=', 'Content-Length': '0'} 2022-09-02T04:32:03.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbb4cbb406836fe1-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:03.199 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.200 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.201 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.202 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.202 DEBUG:boto:Method: GET 2022-09-02T04:32:03.203 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.204 DEBUG:boto:Data: 2022-09-02T04:32:03.205 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.206 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.207 DEBUG:boto:Params: {} 2022-09-02T04:32:03.207 DEBUG:boto:Token: None 2022-09-02T04:32:03.208 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w= 2022-09-02T04:32:03.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w=', 'Content-Length': '0'} 2022-09-02T04:32:03.211 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099f802e7eb2ff889-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.212 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.213 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.213 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.214 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.214 DEBUG:boto:Method: GET 2022-09-02T04:32:03.215 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.216 DEBUG:boto:Data: 2022-09-02T04:32:03.217 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.218 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.218 DEBUG:boto:Params: {} 2022-09-02T04:32:03.219 DEBUG:boto:Token: None 2022-09-02T04:32:03.220 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w= 2022-09-02T04:32:03.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w=', 'Content-Length': '0'} 2022-09-02T04:32:03.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b21eadce2fd16b70-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.224 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.224 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.225 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.225 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.226 DEBUG:boto:Method: GET 2022-09-02T04:32:03.226 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.227 DEBUG:boto:Data: 2022-09-02T04:32:03.228 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.229 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.229 DEBUG:boto:Params: {} 2022-09-02T04:32:03.230 DEBUG:boto:Token: None 2022-09-02T04:32:03.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a0b685d3dbbbf4-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.235 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.235 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.236 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.237 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.237 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.238 DEBUG:boto:Method: GET 2022-09-02T04:32:03.238 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.239 DEBUG:boto:Data: 2022-09-02T04:32:03.239 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.240 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.241 DEBUG:boto:Params: {} 2022-09-02T04:32:03.241 DEBUG:boto:Token: None 2022-09-02T04:32:03.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d8a38bd4514d39-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.246 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.247 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.248 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.248 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.249 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:32:03.249 DEBUG:boto:Data: 2022-09-02T04:32:03.250 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.251 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.251 DEBUG:boto:Params: {} 2022-09-02T04:32:03.252 DEBUG:boto:Token: None 2022-09-02T04:32:03.252 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy 2022-09-02T04:32:03.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dM1yb0w6dBoFsRwuHzA0HwBZR48= 2022-09-02T04:32:03.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dM1yb0w6dBoFsRwuHzA0HwBZR48=', 'Content-Length': '0'} 2022-09-02T04:32:03.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a89284cab450bc22-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.256 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.256 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.257 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.257 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:32:03.258 DEBUG:boto:Data: 2022-09-02T04:32:03.258 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.259 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.260 DEBUG:boto:Params: {} 2022-09-02T04:32:03.260 DEBUG:boto:Token: None 2022-09-02T04:32:03.261 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy 2022-09-02T04:32:03.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dM1yb0w6dBoFsRwuHzA0HwBZR48= 2022-09-02T04:32:03.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dM1yb0w6dBoFsRwuHzA0HwBZR48=', 'Content-Length': '0'} 2022-09-02T04:32:03.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b0254823fecfba4-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:03.267 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.267 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.268 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.268 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.269 DEBUG:boto:Method: GET 2022-09-02T04:32:03.269 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.270 DEBUG:boto:Data: 2022-09-02T04:32:03.271 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.272 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.273 DEBUG:boto:Params: {} 2022-09-02T04:32:03.274 DEBUG:boto:Token: None 2022-09-02T04:32:03.275 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w= 2022-09-02T04:32:03.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w=', 'Content-Length': '0'} 2022-09-02T04:32:03.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005769aaa456dee14f-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.280 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:03.281 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:03.281 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.282 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.283 DEBUG:boto:Method: GET 2022-09-02T04:32:03.283 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.284 DEBUG:boto:Data: 2022-09-02T04:32:03.284 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.285 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.286 DEBUG:boto:Params: {} 2022-09-02T04:32:03.286 DEBUG:boto:Token: None 2022-09-02T04:32:03.287 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:03.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w= 2022-09-02T04:32:03.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O4eoh4zJDVwNZ0xVMBvyjOrVq0w=', 'Content-Length': '0'} 2022-09-02T04:32:03.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a34f724d5c7725f5-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.291 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.292 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.293 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.293 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.294 DEBUG:boto:Method: GET 2022-09-02T04:32:03.294 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.295 DEBUG:boto:Data: 2022-09-02T04:32:03.295 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.296 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.297 DEBUG:boto:Params: {} 2022-09-02T04:32:03.298 DEBUG:boto:Token: None 2022-09-02T04:32:03.299 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.300 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b545c2011c29bb-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.305 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.306 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.306 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.307 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.307 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.308 DEBUG:boto:Method: GET 2022-09-02T04:32:03.308 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:03.309 DEBUG:boto:Data: 2022-09-02T04:32:03.309 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.310 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.311 DEBUG:boto:Params: {} 2022-09-02T04:32:03.311 DEBUG:boto:Token: None 2022-09-02T04:32:03.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5bcbc79fd15523-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.316 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:03.318 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.318 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.319 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.319 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.320 DEBUG:boto:Method: GET 2022-09-02T04:32:03.320 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.321 DEBUG:boto:Data: 2022-09-02T04:32:03.321 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.329 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.329 DEBUG:boto:Params: {} 2022-09-02T04:32:03.330 DEBUG:boto:Token: None 2022-09-02T04:32:03.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg= 2022-09-02T04:32:03.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg=', 'Content-Length': '0'} 2022-09-02T04:32:03.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e60ebe938bcc4f40-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.335 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.335 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.336 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.336 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.337 DEBUG:boto:Method: GET 2022-09-02T04:32:03.337 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.338 DEBUG:boto:Data: 2022-09-02T04:32:03.338 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.339 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.340 DEBUG:boto:Params: {} 2022-09-02T04:32:03.341 DEBUG:boto:Token: None 2022-09-02T04:32:03.341 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg= 2022-09-02T04:32:03.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg=', 'Content-Length': '0'} 2022-09-02T04:32:03.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a98f9b137ffc7ce-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.345 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.346 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.346 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.347 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.347 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.348 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.348 DEBUG:boto:Data: 2022-09-02T04:32:03.349 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.350 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.350 DEBUG:boto:Params: {} 2022-09-02T04:32:03.351 DEBUG:boto:Token: None 2022-09-02T04:32:03.351 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o329sEuQH3AQ1KZ56RGTDMwzxvY= 2022-09-02T04:32:03.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o329sEuQH3AQ1KZ56RGTDMwzxvY=', 'Content-Length': '0'} 2022-09-02T04:32:03.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007078f4e9684c039b-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.355 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.356 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.356 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.357 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.357 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.358 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.358 DEBUG:boto:Data: 2022-09-02T04:32:03.359 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.360 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.360 DEBUG:boto:Params: {} 2022-09-02T04:32:03.361 DEBUG:boto:Token: None 2022-09-02T04:32:03.361 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o329sEuQH3AQ1KZ56RGTDMwzxvY= 2022-09-02T04:32:03.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o329sEuQH3AQ1KZ56RGTDMwzxvY=', 'Content-Length': '0'} 2022-09-02T04:32:03.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e794ef1aa76b6cd6-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:03.366 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.366 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.367 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.367 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.368 DEBUG:boto:Method: GET 2022-09-02T04:32:03.368 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.369 DEBUG:boto:Data: 2022-09-02T04:32:03.369 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.370 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.371 DEBUG:boto:Params: {} 2022-09-02T04:32:03.372 DEBUG:boto:Token: None 2022-09-02T04:32:03.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg= 2022-09-02T04:32:03.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg=', 'Content-Length': '0'} 2022-09-02T04:32:03.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003676ea6c1855c261-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.418 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.419 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.419 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.420 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.420 DEBUG:boto:Method: GET 2022-09-02T04:32:03.421 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.421 DEBUG:boto:Data: 2022-09-02T04:32:03.422 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.423 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.423 DEBUG:boto:Params: {} 2022-09-02T04:32:03.424 DEBUG:boto:Token: None 2022-09-02T04:32:03.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg= 2022-09-02T04:32:03.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg=', 'Content-Length': '0'} 2022-09-02T04:32:03.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058d213852cb856cb-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.428 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.429 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.429 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.430 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.431 DEBUG:boto:Method: GET 2022-09-02T04:32:03.431 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.432 DEBUG:boto:Data: 2022-09-02T04:32:03.432 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.433 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.434 DEBUG:boto:Params: {} 2022-09-02T04:32:03.434 DEBUG:boto:Token: None 2022-09-02T04:32:03.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215a2c33c8afc6e4-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.439 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.440 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.440 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.441 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.441 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.442 DEBUG:boto:Method: GET 2022-09-02T04:32:03.442 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.443 DEBUG:boto:Data: 2022-09-02T04:32:03.443 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.445 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.445 DEBUG:boto:Params: {} 2022-09-02T04:32:03.446 DEBUG:boto:Token: None 2022-09-02T04:32:03.446 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009db7c88722bfbafd-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.450 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.451 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.452 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.452 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.453 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.453 DEBUG:boto:Data: 2022-09-02T04:32:03.454 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.455 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.455 DEBUG:boto:Params: {} 2022-09-02T04:32:03.456 DEBUG:boto:Token: None 2022-09-02T04:32:03.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiEkXFKykgGnGSAuM0iA5DGDvj8= 2022-09-02T04:32:03.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiEkXFKykgGnGSAuM0iA5DGDvj8=', 'Content-Length': '0'} 2022-09-02T04:32:03.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000635b516c7311bd05-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.461 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.461 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.462 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.462 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.463 DEBUG:boto:Data: 2022-09-02T04:32:03.463 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.464 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.465 DEBUG:boto:Params: {} 2022-09-02T04:32:03.465 DEBUG:boto:Token: None 2022-09-02T04:32:03.466 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiEkXFKykgGnGSAuM0iA5DGDvj8= 2022-09-02T04:32:03.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiEkXFKykgGnGSAuM0iA5DGDvj8=', 'Content-Length': '0'} 2022-09-02T04:32:03.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da119bbe8bccf0bc-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:03.470 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.471 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.471 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.472 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.472 DEBUG:boto:Method: GET 2022-09-02T04:32:03.473 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.473 DEBUG:boto:Data: 2022-09-02T04:32:03.474 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.475 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.476 DEBUG:boto:Params: {} 2022-09-02T04:32:03.476 DEBUG:boto:Token: None 2022-09-02T04:32:03.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg= 2022-09-02T04:32:03.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg=', 'Content-Length': '0'} 2022-09-02T04:32:03.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b544beff139e79d5-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.480 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.481 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:03.482 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.482 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.483 DEBUG:boto:Method: GET 2022-09-02T04:32:03.483 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.484 DEBUG:boto:Data: 2022-09-02T04:32:03.484 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.485 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.486 DEBUG:boto:Params: {} 2022-09-02T04:32:03.486 DEBUG:boto:Token: None 2022-09-02T04:32:03.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:03.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg= 2022-09-02T04:32:03.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYRR4FwlPf9tGi8pNz8xOckvhmg=', 'Content-Length': '0'} 2022-09-02T04:32:03.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bdd1b94330961d8-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.491 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.491 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.492 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.492 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.493 DEBUG:boto:Method: GET 2022-09-02T04:32:03.493 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.494 DEBUG:boto:Data: 2022-09-02T04:32:03.495 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.496 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.496 DEBUG:boto:Params: {} 2022-09-02T04:32:03.497 DEBUG:boto:Token: None 2022-09-02T04:32:03.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3c46a9639553cb6-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.502 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.502 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.503 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.503 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.504 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.505 DEBUG:boto:Method: GET 2022-09-02T04:32:03.505 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:03.506 DEBUG:boto:Data: 2022-09-02T04:32:03.506 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.507 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.508 DEBUG:boto:Params: {} 2022-09-02T04:32:03.508 DEBUG:boto:Token: None 2022-09-02T04:32:03.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d567a997bbc33953-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.554 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:03.555 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.556 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.556 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.557 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.557 DEBUG:boto:Method: GET 2022-09-02T04:32:03.558 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.558 DEBUG:boto:Data: 2022-09-02T04:32:03.559 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.560 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.561 DEBUG:boto:Params: {} 2022-09-02T04:32:03.561 DEBUG:boto:Token: None 2022-09-02T04:32:03.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc= 2022-09-02T04:32:03.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc=', 'Content-Length': '0'} 2022-09-02T04:32:03.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec445517c34f63c6-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.566 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.566 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.567 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.567 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.568 DEBUG:boto:Method: GET 2022-09-02T04:32:03.568 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.569 DEBUG:boto:Data: 2022-09-02T04:32:03.569 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.570 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.571 DEBUG:boto:Params: {} 2022-09-02T04:32:03.572 DEBUG:boto:Token: None 2022-09-02T04:32:03.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc= 2022-09-02T04:32:03.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc=', 'Content-Length': '0'} 2022-09-02T04:32:03.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b207f32187c09fc-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.576 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.577 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.577 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.578 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.578 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.579 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.579 DEBUG:boto:Data: 2022-09-02T04:32:03.580 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.581 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.581 DEBUG:boto:Params: {} 2022-09-02T04:32:03.582 DEBUG:boto:Token: None 2022-09-02T04:32:03.582 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.583 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0NvR6qKA2rrdIqfLB1vFKSCE2XA= 2022-09-02T04:32:03.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0NvR6qKA2rrdIqfLB1vFKSCE2XA=', 'Content-Length': '0'} 2022-09-02T04:32:03.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e1e8d24371095b9-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.586 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.587 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.587 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.588 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.588 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.589 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.589 DEBUG:boto:Data: 2022-09-02T04:32:03.590 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.591 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.591 DEBUG:boto:Params: {} 2022-09-02T04:32:03.592 DEBUG:boto:Token: None 2022-09-02T04:32:03.593 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0NvR6qKA2rrdIqfLB1vFKSCE2XA= 2022-09-02T04:32:03.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0NvR6qKA2rrdIqfLB1vFKSCE2XA=', 'Content-Length': '0'} 2022-09-02T04:32:03.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000550d71d377237f79-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:03.598 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.598 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.599 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.599 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.600 DEBUG:boto:Method: GET 2022-09-02T04:32:03.600 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.601 DEBUG:boto:Data: 2022-09-02T04:32:03.601 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.602 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.603 DEBUG:boto:Params: {} 2022-09-02T04:32:03.603 DEBUG:boto:Token: None 2022-09-02T04:32:03.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc= 2022-09-02T04:32:03.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc=', 'Content-Length': '0'} 2022-09-02T04:32:03.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb121a4376af82cc-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.608 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.609 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.610 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.610 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.611 DEBUG:boto:Method: GET 2022-09-02T04:32:03.611 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.612 DEBUG:boto:Data: 2022-09-02T04:32:03.612 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.613 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.614 DEBUG:boto:Params: {} 2022-09-02T04:32:03.614 DEBUG:boto:Token: None 2022-09-02T04:32:03.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc= 2022-09-02T04:32:03.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc=', 'Content-Length': '0'} 2022-09-02T04:32:03.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beecea3fec5d4787-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.619 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.620 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.620 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.621 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.621 DEBUG:boto:Method: GET 2022-09-02T04:32:03.622 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.623 DEBUG:boto:Data: 2022-09-02T04:32:03.623 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.624 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.625 DEBUG:boto:Params: {} 2022-09-02T04:32:03.625 DEBUG:boto:Token: None 2022-09-02T04:32:03.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004648733963e8c829-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.630 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.631 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.632 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.632 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.633 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.633 DEBUG:boto:Method: GET 2022-09-02T04:32:03.634 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.635 DEBUG:boto:Data: 2022-09-02T04:32:03.635 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.636 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.637 DEBUG:boto:Params: {} 2022-09-02T04:32:03.638 DEBUG:boto:Token: None 2022-09-02T04:32:03.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7201e793ee4792f-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.643 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.643 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.644 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.645 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.645 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.646 DEBUG:boto:Data: 2022-09-02T04:32:03.646 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.647 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.648 DEBUG:boto:Params: {} 2022-09-02T04:32:03.648 DEBUG:boto:Token: None 2022-09-02T04:32:03.649 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.649 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qCRF44uHcD1B4PTQz+gX11Ykd3g= 2022-09-02T04:32:03.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qCRF44uHcD1B4PTQz+gX11Ykd3g=', 'Content-Length': '0'} 2022-09-02T04:32:03.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec6ae7688b552258-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.653 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.653 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.654 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.654 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.655 DEBUG:boto:Data: 2022-09-02T04:32:03.655 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.656 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.657 DEBUG:boto:Params: {} 2022-09-02T04:32:03.657 DEBUG:boto:Token: None 2022-09-02T04:32:03.658 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qCRF44uHcD1B4PTQz+gX11Ykd3g= 2022-09-02T04:32:03.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qCRF44uHcD1B4PTQz+gX11Ykd3g=', 'Content-Length': '0'} 2022-09-02T04:32:03.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006787db83f29bb298-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:03.663 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.663 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.664 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.664 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.665 DEBUG:boto:Method: GET 2022-09-02T04:32:03.665 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.666 DEBUG:boto:Data: 2022-09-02T04:32:03.666 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.667 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.668 DEBUG:boto:Params: {} 2022-09-02T04:32:03.668 DEBUG:boto:Token: None 2022-09-02T04:32:03.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc= 2022-09-02T04:32:03.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc=', 'Content-Length': '0'} 2022-09-02T04:32:03.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a63fbf5dad5ce13e-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.714 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.715 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:03.715 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.716 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.716 DEBUG:boto:Method: GET 2022-09-02T04:32:03.717 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.717 DEBUG:boto:Data: 2022-09-02T04:32:03.718 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.719 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.720 DEBUG:boto:Params: {} 2022-09-02T04:32:03.720 DEBUG:boto:Token: None 2022-09-02T04:32:03.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:03.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc= 2022-09-02T04:32:03.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XF5Cn3EeWiBdivoe08GlACGx/uc=', 'Content-Length': '0'} 2022-09-02T04:32:03.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059bc120a2d99f92d-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.726 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.727 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.727 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.728 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.729 DEBUG:boto:Method: GET 2022-09-02T04:32:03.730 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.731 DEBUG:boto:Data: 2022-09-02T04:32:03.731 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.734 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.735 DEBUG:boto:Params: {} 2022-09-02T04:32:03.735 DEBUG:boto:Token: None 2022-09-02T04:32:03.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1de9b9ab3fbbd62-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.741 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.742 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.743 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.744 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.744 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.745 DEBUG:boto:Method: GET 2022-09-02T04:32:03.746 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:03.746 DEBUG:boto:Data: 2022-09-02T04:32:03.747 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.748 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.749 DEBUG:boto:Params: {} 2022-09-02T04:32:03.750 DEBUG:boto:Token: None 2022-09-02T04:32:03.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000563a993569580902-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.755 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:03.757 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.757 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.758 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.759 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.760 DEBUG:boto:Method: GET 2022-09-02T04:32:03.760 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.761 DEBUG:boto:Data: 2022-09-02T04:32:03.761 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.762 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.763 DEBUG:boto:Params: {} 2022-09-02T04:32:03.763 DEBUG:boto:Token: None 2022-09-02T04:32:03.764 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw= 2022-09-02T04:32:03.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw=', 'Content-Length': '0'} 2022-09-02T04:32:03.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c060bd2f9cc9540c-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.769 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.770 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.771 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.772 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.773 DEBUG:boto:Method: GET 2022-09-02T04:32:03.773 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.774 DEBUG:boto:Data: 2022-09-02T04:32:03.775 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.776 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.777 DEBUG:boto:Params: {} 2022-09-02T04:32:03.779 DEBUG:boto:Token: None 2022-09-02T04:32:03.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw= 2022-09-02T04:32:03.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw=', 'Content-Length': '0'} 2022-09-02T04:32:03.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eda6b7d50246e879-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.785 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.785 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.786 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.786 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.787 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.788 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.789 DEBUG:boto:Data: 2022-09-02T04:32:03.790 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.797 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.798 DEBUG:boto:Params: {} 2022-09-02T04:32:03.799 DEBUG:boto:Token: None 2022-09-02T04:32:03.800 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aHGSwFmesKjuskauTcW75AZowO0= 2022-09-02T04:32:03.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aHGSwFmesKjuskauTcW75AZowO0=', 'Content-Length': '0'} 2022-09-02T04:32:03.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000972c9e2207d3826f-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.805 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.805 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.806 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.807 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.807 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.808 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.808 DEBUG:boto:Data: 2022-09-02T04:32:03.808 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.809 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.810 DEBUG:boto:Params: {} 2022-09-02T04:32:03.811 DEBUG:boto:Token: None 2022-09-02T04:32:03.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aHGSwFmesKjuskauTcW75AZowO0= 2022-09-02T04:32:03.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aHGSwFmesKjuskauTcW75AZowO0=', 'Content-Length': '0'} 2022-09-02T04:32:03.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8ab537b8d411ecf-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:03.815 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.816 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.817 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.817 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.818 DEBUG:boto:Method: GET 2022-09-02T04:32:03.818 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.819 DEBUG:boto:Data: 2022-09-02T04:32:03.819 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.820 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.821 DEBUG:boto:Params: {} 2022-09-02T04:32:03.822 DEBUG:boto:Token: None 2022-09-02T04:32:03.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw= 2022-09-02T04:32:03.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw=', 'Content-Length': '0'} 2022-09-02T04:32:03.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecb24a3a2788e7b0-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.826 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.827 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.828 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.828 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.829 DEBUG:boto:Method: GET 2022-09-02T04:32:03.830 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.831 DEBUG:boto:Data: 2022-09-02T04:32:03.832 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.833 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.834 DEBUG:boto:Params: {} 2022-09-02T04:32:03.834 DEBUG:boto:Token: None 2022-09-02T04:32:03.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw= 2022-09-02T04:32:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw=', 'Content-Length': '0'} 2022-09-02T04:32:03.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e706be1b3b3e424-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.839 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.840 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.840 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.841 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.841 DEBUG:boto:Method: GET 2022-09-02T04:32:03.842 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.843 DEBUG:boto:Data: 2022-09-02T04:32:03.844 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.845 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.846 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.847 DEBUG:boto:Params: {} 2022-09-02T04:32:03.848 DEBUG:boto:Token: None 2022-09-02T04:32:03.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e8a6ddeca69efa-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.856 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.857 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.858 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.859 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.860 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.861 DEBUG:boto:Method: GET 2022-09-02T04:32:03.862 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.863 DEBUG:boto:Data: 2022-09-02T04:32:03.864 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.866 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.867 DEBUG:boto:Params: {} 2022-09-02T04:32:03.868 DEBUG:boto:Token: None 2022-09-02T04:32:03.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4db587bc5e84cd-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.918 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.920 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.920 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.921 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.922 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.923 DEBUG:boto:Data: 2022-09-02T04:32:03.923 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.924 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.925 DEBUG:boto:Params: {} 2022-09-02T04:32:03.925 DEBUG:boto:Token: None 2022-09-02T04:32:03.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y72kS88j3kOPQVlmLjcA3+LfXmM= 2022-09-02T04:32:03.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y72kS88j3kOPQVlmLjcA3+LfXmM=', 'Content-Length': '0'} 2022-09-02T04:32:03.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fbf18dd76f6eabd-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.930 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.931 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.932 DEBUG:boto:Method: HEAD 2022-09-02T04:32:03.932 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.933 DEBUG:boto:Data: 2022-09-02T04:32:03.934 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.936 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.936 DEBUG:boto:Params: {} 2022-09-02T04:32:03.937 DEBUG:boto:Token: None 2022-09-02T04:32:03.938 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y72kS88j3kOPQVlmLjcA3+LfXmM= 2022-09-02T04:32:03.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y72kS88j3kOPQVlmLjcA3+LfXmM=', 'Content-Length': '0'} 2022-09-02T04:32:03.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1108a3b2587ef75-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:03.943 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.943 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.944 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.945 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.945 DEBUG:boto:Method: GET 2022-09-02T04:32:03.946 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.947 DEBUG:boto:Data: 2022-09-02T04:32:03.948 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.949 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.950 DEBUG:boto:Params: {} 2022-09-02T04:32:03.950 DEBUG:boto:Token: None 2022-09-02T04:32:03.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw= 2022-09-02T04:32:03.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw=', 'Content-Length': '0'} 2022-09-02T04:32:03.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087ee28d2a9bddb51-0063118743-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.956 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.957 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:03.957 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.958 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.959 DEBUG:boto:Method: GET 2022-09-02T04:32:03.959 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.960 DEBUG:boto:Data: 2022-09-02T04:32:03.961 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.962 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.963 DEBUG:boto:Params: {} 2022-09-02T04:32:03.963 DEBUG:boto:Token: None 2022-09-02T04:32:03.964 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:03.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw= 2022-09-02T04:32:03.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F6+WkyO+WnzmjxTF/3p9LNqUdvw=', 'Content-Length': '0'} 2022-09-02T04:32:03.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f86722529463f447-0063118743-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.969 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.969 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.970 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.971 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.971 DEBUG:boto:Method: GET 2022-09-02T04:32:03.972 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.973 DEBUG:boto:Data: 2022-09-02T04:32:03.973 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:03.975 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.975 DEBUG:boto:Params: {} 2022-09-02T04:32:03.976 DEBUG:boto:Token: None 2022-09-02T04:32:03.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29452ab2530be81-0063118743-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.982 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.983 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.984 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:03.984 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.985 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.986 DEBUG:boto:Method: GET 2022-09-02T04:32:03.986 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:03.987 DEBUG:boto:Data: 2022-09-02T04:32:03.987 DEBUG:boto:Headers: {} 2022-09-02T04:32:03.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:03.989 DEBUG:boto:Port: 8000 2022-09-02T04:32:03.989 DEBUG:boto:Params: {} 2022-09-02T04:32:03.990 DEBUG:boto:Token: None 2022-09-02T04:32:03.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:03 GMT /xljbkf-16/ 2022-09-02T04:32:03.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY= 2022-09-02T04:32:03.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TQRSr9aj/lZ4D0/WxyZsYpUfxY=', 'Content-Length': '0'} 2022-09-02T04:32:03.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ce7e2bec4111093-0063118743-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:03.996 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:03.997 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-16 zones={test-zone2, test-zone1} 2022-09-02T04:32:03.997 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-16 zones={test-zone3, test-zone1} 2022-09-02T04:32:03.998 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:03.999 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.000 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.001 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:32:04.002 DEBUG:boto:Data: 2022-09-02T04:32:04.003 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.005 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.006 DEBUG:boto:Params: {} 2022-09-02T04:32:04.007 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:32:04.008 DEBUG:boto:Token: None 2022-09-02T04:32:04.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+lThcLFxiJmbOx8uyxXv67HaRxM= 2022-09-02T04:32:04.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+lThcLFxiJmbOx8uyxXv67HaRxM=', 'Content-Length': '0'} 2022-09-02T04:32:04.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae40c52b83a10100-0063118744-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.015 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.015 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.016 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.017 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:32:04.018 DEBUG:boto:Data: 2022-09-02T04:32:04.019 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.021 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.021 DEBUG:boto:Params: {} 2022-09-02T04:32:04.022 DEBUG:boto:Token: None 2022-09-02T04:32:04.023 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+lThcLFxiJmbOx8uyxXv67HaRxM= 2022-09-02T04:32:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+lThcLFxiJmbOx8uyxXv67HaRxM=', 'Content-Length': '0'} 2022-09-02T04:32:04.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e816993526f22884-0063118744-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:04.028 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.029 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.029 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.030 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.031 DEBUG:boto:Method: GET 2022-09-02T04:32:04.032 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.033 DEBUG:boto:Data: 2022-09-02T04:32:04.033 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.035 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.035 DEBUG:boto:Params: {} 2022-09-02T04:32:04.036 DEBUG:boto:Token: None 2022-09-02T04:32:04.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/?versions 2022-09-02T04:32:04.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY= 2022-09-02T04:32:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY=', 'Content-Length': '0'} 2022-09-02T04:32:04.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb75421590b9de5e-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.042 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.043 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:04.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:04.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:04.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:04.045 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:04.046 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.047 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.047 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.048 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.048 DEBUG:boto:Method: GET 2022-09-02T04:32:04.049 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.050 DEBUG:boto:Data: 2022-09-02T04:32:04.050 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.052 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.053 DEBUG:boto:Params: {} 2022-09-02T04:32:04.054 DEBUG:boto:Token: None 2022-09-02T04:32:04.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/?versions 2022-09-02T04:32:04.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY= 2022-09-02T04:32:04.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY=', 'Content-Length': '0'} 2022-09-02T04:32:04.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c2701a3d91ff1f-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.059 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.060 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:04.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:04.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:04.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:04.063 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.064 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.065 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.065 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.066 DEBUG:boto:Method: GET 2022-09-02T04:32:04.067 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.067 DEBUG:boto:Data: 2022-09-02T04:32:04.068 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.069 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.070 DEBUG:boto:Params: {} 2022-09-02T04:32:04.071 DEBUG:boto:Token: None 2022-09-02T04:32:04.071 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/?versions 2022-09-02T04:32:04.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY= 2022-09-02T04:32:04.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY=', 'Content-Length': '0'} 2022-09-02T04:32:04.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0db33126a824f82-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.076 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.078 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.078 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.079 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.080 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.080 DEBUG:boto:Method: GET 2022-09-02T04:32:04.081 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:04.082 DEBUG:boto:Data: 2022-09-02T04:32:04.082 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.084 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.084 DEBUG:boto:Params: {} 2022-09-02T04:32:04.085 DEBUG:boto:Token: None 2022-09-02T04:32:04.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/?versions 2022-09-02T04:32:04.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY= 2022-09-02T04:32:04.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CLWw9EpuQaiKTXJaNg9EQSLblYY=', 'Content-Length': '0'} 2022-09-02T04:32:04.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e65df216ab4c3f-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.091 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:04.092 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.093 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.093 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.094 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.094 DEBUG:boto:Method: GET 2022-09-02T04:32:04.095 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.096 DEBUG:boto:Data: 2022-09-02T04:32:04.096 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.097 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.098 DEBUG:boto:Params: {} 2022-09-02T04:32:04.099 DEBUG:boto:Token: None 2022-09-02T04:32:04.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc= 2022-09-02T04:32:04.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc=', 'Content-Length': '0'} 2022-09-02T04:32:04.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c675da642cd5a8f6-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.103 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.104 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.105 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.105 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.106 DEBUG:boto:Method: GET 2022-09-02T04:32:04.106 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.107 DEBUG:boto:Data: 2022-09-02T04:32:04.107 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.109 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.109 DEBUG:boto:Params: {} 2022-09-02T04:32:04.110 DEBUG:boto:Token: None 2022-09-02T04:32:04.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc= 2022-09-02T04:32:04.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc=', 'Content-Length': '0'} 2022-09-02T04:32:04.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052e756685817cf57-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.114 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.115 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.115 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.116 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.117 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.117 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.118 DEBUG:boto:Data: 2022-09-02T04:32:04.118 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.119 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.120 DEBUG:boto:Params: {} 2022-09-02T04:32:04.120 DEBUG:boto:Token: None 2022-09-02T04:32:04.121 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L1QVug6clz8fe5Rd0/0upinCeiE= 2022-09-02T04:32:04.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L1QVug6clz8fe5Rd0/0upinCeiE=', 'Content-Length': '0'} 2022-09-02T04:32:04.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060456a83d993bf80-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.124 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.125 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.125 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.126 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.126 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.127 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.127 DEBUG:boto:Data: 2022-09-02T04:32:04.128 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.129 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.129 DEBUG:boto:Params: {} 2022-09-02T04:32:04.130 DEBUG:boto:Token: None 2022-09-02T04:32:04.131 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L1QVug6clz8fe5Rd0/0upinCeiE= 2022-09-02T04:32:04.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L1QVug6clz8fe5Rd0/0upinCeiE=', 'Content-Length': '0'} 2022-09-02T04:32:04.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000101581b9a799be44-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:04.135 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.136 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.137 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.137 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.138 DEBUG:boto:Method: GET 2022-09-02T04:32:04.138 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.139 DEBUG:boto:Data: 2022-09-02T04:32:04.140 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.142 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.143 DEBUG:boto:Params: {} 2022-09-02T04:32:04.144 DEBUG:boto:Token: None 2022-09-02T04:32:04.145 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc= 2022-09-02T04:32:04.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc=', 'Content-Length': '0'} 2022-09-02T04:32:04.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4fccf5fef357cb5-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.149 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.149 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.150 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.151 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.151 DEBUG:boto:Method: GET 2022-09-02T04:32:04.152 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.152 DEBUG:boto:Data: 2022-09-02T04:32:04.153 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.154 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.154 DEBUG:boto:Params: {} 2022-09-02T04:32:04.155 DEBUG:boto:Token: None 2022-09-02T04:32:04.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc= 2022-09-02T04:32:04.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc=', 'Content-Length': '0'} 2022-09-02T04:32:04.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db84248412e5cd75-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.159 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.160 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.160 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.161 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.161 DEBUG:boto:Method: GET 2022-09-02T04:32:04.162 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.162 DEBUG:boto:Data: 2022-09-02T04:32:04.163 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.164 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.165 DEBUG:boto:Params: {} 2022-09-02T04:32:04.165 DEBUG:boto:Token: None 2022-09-02T04:32:04.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b1f757f1038a33d-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.170 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.171 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.172 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.172 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.173 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.173 DEBUG:boto:Method: GET 2022-09-02T04:32:04.174 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.175 DEBUG:boto:Data: 2022-09-02T04:32:04.176 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.178 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.179 DEBUG:boto:Params: {} 2022-09-02T04:32:04.180 DEBUG:boto:Token: None 2022-09-02T04:32:04.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e04f6f3980d626-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.230 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.231 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.232 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.232 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.233 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:32:04.233 DEBUG:boto:Data: 2022-09-02T04:32:04.234 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.235 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.235 DEBUG:boto:Params: {} 2022-09-02T04:32:04.236 DEBUG:boto:Token: None 2022-09-02T04:32:04.237 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy 2022-09-02T04:32:04.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X75b978LL6J8STPhA5zQJYpgUzs= 2022-09-02T04:32:04.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X75b978LL6J8STPhA5zQJYpgUzs=', 'Content-Length': '0'} 2022-09-02T04:32:04.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063bddffcfe82e4af-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.241 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.242 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.243 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.243 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:32:04.244 DEBUG:boto:Data: 2022-09-02T04:32:04.244 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.245 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.246 DEBUG:boto:Params: {} 2022-09-02T04:32:04.246 DEBUG:boto:Token: None 2022-09-02T04:32:04.247 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy 2022-09-02T04:32:04.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X75b978LL6J8STPhA5zQJYpgUzs= 2022-09-02T04:32:04.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X75b978LL6J8STPhA5zQJYpgUzs=', 'Content-Length': '0'} 2022-09-02T04:32:04.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a35c28b1618c04e-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:04.252 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.253 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.253 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.254 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.255 DEBUG:boto:Method: GET 2022-09-02T04:32:04.255 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.256 DEBUG:boto:Data: 2022-09-02T04:32:04.257 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.258 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.258 DEBUG:boto:Params: {} 2022-09-02T04:32:04.259 DEBUG:boto:Token: None 2022-09-02T04:32:04.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc= 2022-09-02T04:32:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc=', 'Content-Length': '0'} 2022-09-02T04:32:04.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfde092314dea0c2-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.264 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:04.265 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:04.265 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.266 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.266 DEBUG:boto:Method: GET 2022-09-02T04:32:04.267 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.268 DEBUG:boto:Data: 2022-09-02T04:32:04.268 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.269 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.270 DEBUG:boto:Params: {} 2022-09-02T04:32:04.270 DEBUG:boto:Token: None 2022-09-02T04:32:04.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:04.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc= 2022-09-02T04:32:04.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DOk8JmeYvsAsxUeiGMMf6UGQTNc=', 'Content-Length': '0'} 2022-09-02T04:32:04.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015a396b0f7667a02-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.275 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.276 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.276 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.277 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.277 DEBUG:boto:Method: GET 2022-09-02T04:32:04.278 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.278 DEBUG:boto:Data: 2022-09-02T04:32:04.279 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.280 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.281 DEBUG:boto:Params: {} 2022-09-02T04:32:04.281 DEBUG:boto:Token: None 2022-09-02T04:32:04.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9eb248f91da56e-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.286 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.287 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.287 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.288 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.288 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.289 DEBUG:boto:Method: GET 2022-09-02T04:32:04.290 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:04.290 DEBUG:boto:Data: 2022-09-02T04:32:04.291 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.292 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.292 DEBUG:boto:Params: {} 2022-09-02T04:32:04.293 DEBUG:boto:Token: None 2022-09-02T04:32:04.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566668bb613397eb-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.338 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:04.340 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.340 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.341 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.341 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.342 DEBUG:boto:Method: GET 2022-09-02T04:32:04.343 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.343 DEBUG:boto:Data: 2022-09-02T04:32:04.344 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.345 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.346 DEBUG:boto:Params: {} 2022-09-02T04:32:04.346 DEBUG:boto:Token: None 2022-09-02T04:32:04.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg= 2022-09-02T04:32:04.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg=', 'Content-Length': '0'} 2022-09-02T04:32:04.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce893a05ec5f3c13-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.352 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.352 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.354 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.354 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.355 DEBUG:boto:Method: GET 2022-09-02T04:32:04.355 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.356 DEBUG:boto:Data: 2022-09-02T04:32:04.356 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.357 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.358 DEBUG:boto:Params: {} 2022-09-02T04:32:04.358 DEBUG:boto:Token: None 2022-09-02T04:32:04.359 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg= 2022-09-02T04:32:04.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg=', 'Content-Length': '0'} 2022-09-02T04:32:04.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059f5372f7c20a519-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.364 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.364 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.365 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.365 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.366 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.366 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.367 DEBUG:boto:Data: 2022-09-02T04:32:04.367 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.369 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.370 DEBUG:boto:Params: {} 2022-09-02T04:32:04.371 DEBUG:boto:Token: None 2022-09-02T04:32:04.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0HIpTmANsh9SduGhDpbSBfNiBs= 2022-09-02T04:32:04.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0HIpTmANsh9SduGhDpbSBfNiBs=', 'Content-Length': '0'} 2022-09-02T04:32:04.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020305bfa0d0c2894-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.378 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.379 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.380 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.380 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.381 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.382 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.382 DEBUG:boto:Data: 2022-09-02T04:32:04.383 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.384 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.384 DEBUG:boto:Params: {} 2022-09-02T04:32:04.385 DEBUG:boto:Token: None 2022-09-02T04:32:04.385 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0HIpTmANsh9SduGhDpbSBfNiBs= 2022-09-02T04:32:04.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0HIpTmANsh9SduGhDpbSBfNiBs=', 'Content-Length': '0'} 2022-09-02T04:32:04.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d70c39316af1e1d-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:04.390 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.391 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.391 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.392 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.392 DEBUG:boto:Method: GET 2022-09-02T04:32:04.393 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.393 DEBUG:boto:Data: 2022-09-02T04:32:04.394 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.395 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.396 DEBUG:boto:Params: {} 2022-09-02T04:32:04.397 DEBUG:boto:Token: None 2022-09-02T04:32:04.397 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg= 2022-09-02T04:32:04.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg=', 'Content-Length': '0'} 2022-09-02T04:32:04.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9afb1a174b1b86d-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.401 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.402 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.403 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.404 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.405 DEBUG:boto:Method: GET 2022-09-02T04:32:04.406 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.407 DEBUG:boto:Data: 2022-09-02T04:32:04.408 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.410 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.411 DEBUG:boto:Params: {} 2022-09-02T04:32:04.412 DEBUG:boto:Token: None 2022-09-02T04:32:04.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg= 2022-09-02T04:32:04.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg=', 'Content-Length': '0'} 2022-09-02T04:32:04.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047698d74a7e6d105-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.418 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.418 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.419 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.419 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.420 DEBUG:boto:Method: GET 2022-09-02T04:32:04.421 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.421 DEBUG:boto:Data: 2022-09-02T04:32:04.422 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.423 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.424 DEBUG:boto:Params: {} 2022-09-02T04:32:04.424 DEBUG:boto:Token: None 2022-09-02T04:32:04.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005942e8de8a828df8-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.430 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.432 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.432 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.433 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.434 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.435 DEBUG:boto:Method: GET 2022-09-02T04:32:04.436 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.436 DEBUG:boto:Data: 2022-09-02T04:32:04.437 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.438 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.438 DEBUG:boto:Params: {} 2022-09-02T04:32:04.439 DEBUG:boto:Token: None 2022-09-02T04:32:04.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000424f5076c11b305d-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.445 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.446 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.446 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.447 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.447 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.448 DEBUG:boto:Data: 2022-09-02T04:32:04.448 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.449 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.450 DEBUG:boto:Params: {} 2022-09-02T04:32:04.450 DEBUG:boto:Token: None 2022-09-02T04:32:04.451 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.452 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z9AGjooVFYOwOnZxpzn/dk1Za10= 2022-09-02T04:32:04.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z9AGjooVFYOwOnZxpzn/dk1Za10=', 'Content-Length': '0'} 2022-09-02T04:32:04.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa36ae792f1915c2-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.455 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.457 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.457 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.458 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.458 DEBUG:boto:Data: 2022-09-02T04:32:04.459 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.460 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.461 DEBUG:boto:Params: {} 2022-09-02T04:32:04.461 DEBUG:boto:Token: None 2022-09-02T04:32:04.462 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z9AGjooVFYOwOnZxpzn/dk1Za10= 2022-09-02T04:32:04.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z9AGjooVFYOwOnZxpzn/dk1Za10=', 'Content-Length': '0'} 2022-09-02T04:32:04.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f24901db6511a69b-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:04.467 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.468 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.468 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.469 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.469 DEBUG:boto:Method: GET 2022-09-02T04:32:04.470 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.471 DEBUG:boto:Data: 2022-09-02T04:32:04.471 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.472 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.473 DEBUG:boto:Params: {} 2022-09-02T04:32:04.474 DEBUG:boto:Token: None 2022-09-02T04:32:04.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg= 2022-09-02T04:32:04.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg=', 'Content-Length': '0'} 2022-09-02T04:32:04.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ac4e55fbe4b2d54-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.522 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.523 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:04.524 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.524 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.525 DEBUG:boto:Method: GET 2022-09-02T04:32:04.526 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.527 DEBUG:boto:Data: 2022-09-02T04:32:04.527 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.528 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.529 DEBUG:boto:Params: {} 2022-09-02T04:32:04.529 DEBUG:boto:Token: None 2022-09-02T04:32:04.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:04.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg= 2022-09-02T04:32:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSc6eLFtoe3fmmvS9J3Zv4jOQHg=', 'Content-Length': '0'} 2022-09-02T04:32:04.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000528a6a2630557c3e-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.536 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.537 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.538 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.539 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.539 DEBUG:boto:Method: GET 2022-09-02T04:32:04.540 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.541 DEBUG:boto:Data: 2022-09-02T04:32:04.542 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.544 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.545 DEBUG:boto:Params: {} 2022-09-02T04:32:04.547 DEBUG:boto:Token: None 2022-09-02T04:32:04.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5641767a6ec8842-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.554 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.555 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.556 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.556 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.557 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.557 DEBUG:boto:Method: GET 2022-09-02T04:32:04.558 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:04.558 DEBUG:boto:Data: 2022-09-02T04:32:04.559 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.560 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.560 DEBUG:boto:Params: {} 2022-09-02T04:32:04.561 DEBUG:boto:Token: None 2022-09-02T04:32:04.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000970f06825aaf0fb6-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.566 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:04.568 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.568 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.569 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.569 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.570 DEBUG:boto:Method: GET 2022-09-02T04:32:04.571 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.571 DEBUG:boto:Data: 2022-09-02T04:32:04.572 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.573 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.573 DEBUG:boto:Params: {} 2022-09-02T04:32:04.574 DEBUG:boto:Token: None 2022-09-02T04:32:04.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs= 2022-09-02T04:32:04.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs=', 'Content-Length': '0'} 2022-09-02T04:32:04.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008393a38e1561438d-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.580 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.581 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.582 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.583 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.584 DEBUG:boto:Method: GET 2022-09-02T04:32:04.585 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.585 DEBUG:boto:Data: 2022-09-02T04:32:04.586 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.587 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.588 DEBUG:boto:Params: {} 2022-09-02T04:32:04.588 DEBUG:boto:Token: None 2022-09-02T04:32:04.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs= 2022-09-02T04:32:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs=', 'Content-Length': '0'} 2022-09-02T04:32:04.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6024e446abb08e5-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.593 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.593 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.594 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.595 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.595 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.596 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.596 DEBUG:boto:Data: 2022-09-02T04:32:04.597 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.598 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.598 DEBUG:boto:Params: {} 2022-09-02T04:32:04.599 DEBUG:boto:Token: None 2022-09-02T04:32:04.599 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ug1qJM4U87nLqP0KfDUTthugvdw= 2022-09-02T04:32:04.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ug1qJM4U87nLqP0KfDUTthugvdw=', 'Content-Length': '0'} 2022-09-02T04:32:04.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcad7bb45076f4fe-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.603 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.604 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.604 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.605 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.605 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.606 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.606 DEBUG:boto:Data: 2022-09-02T04:32:04.607 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.608 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.608 DEBUG:boto:Params: {} 2022-09-02T04:32:04.609 DEBUG:boto:Token: None 2022-09-02T04:32:04.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ug1qJM4U87nLqP0KfDUTthugvdw= 2022-09-02T04:32:04.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ug1qJM4U87nLqP0KfDUTthugvdw=', 'Content-Length': '0'} 2022-09-02T04:32:04.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fba8f43533d1cc3d-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:04.614 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.614 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.615 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.615 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.616 DEBUG:boto:Method: GET 2022-09-02T04:32:04.616 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.617 DEBUG:boto:Data: 2022-09-02T04:32:04.617 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.618 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.619 DEBUG:boto:Params: {} 2022-09-02T04:32:04.620 DEBUG:boto:Token: None 2022-09-02T04:32:04.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs= 2022-09-02T04:32:04.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs=', 'Content-Length': '0'} 2022-09-02T04:32:04.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066b029a0543a99d2-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.666 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.667 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.667 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.668 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.668 DEBUG:boto:Method: GET 2022-09-02T04:32:04.669 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.669 DEBUG:boto:Data: 2022-09-02T04:32:04.670 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.671 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.672 DEBUG:boto:Params: {} 2022-09-02T04:32:04.672 DEBUG:boto:Token: None 2022-09-02T04:32:04.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs= 2022-09-02T04:32:04.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs=', 'Content-Length': '0'} 2022-09-02T04:32:04.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e07f09bd66b8400a-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.677 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.677 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.678 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.678 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.679 DEBUG:boto:Method: GET 2022-09-02T04:32:04.679 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.680 DEBUG:boto:Data: 2022-09-02T04:32:04.680 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.682 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.682 DEBUG:boto:Params: {} 2022-09-02T04:32:04.683 DEBUG:boto:Token: None 2022-09-02T04:32:04.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020463385c0a8d97d-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.688 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.689 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.689 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.690 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.690 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.691 DEBUG:boto:Method: GET 2022-09-02T04:32:04.691 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.692 DEBUG:boto:Data: 2022-09-02T04:32:04.692 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.693 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.694 DEBUG:boto:Params: {} 2022-09-02T04:32:04.695 DEBUG:boto:Token: None 2022-09-02T04:32:04.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d636d3fbfaa833-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.700 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.701 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.701 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.702 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.702 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.703 DEBUG:boto:Data: 2022-09-02T04:32:04.703 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.704 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.705 DEBUG:boto:Params: {} 2022-09-02T04:32:04.706 DEBUG:boto:Token: None 2022-09-02T04:32:04.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i1FumpjpDAL3jjqNW6sbXqkMKDU= 2022-09-02T04:32:04.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i1FumpjpDAL3jjqNW6sbXqkMKDU=', 'Content-Length': '0'} 2022-09-02T04:32:04.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054159f82dee181b6-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.710 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.711 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.711 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.712 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.712 DEBUG:boto:Data: 2022-09-02T04:32:04.713 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.714 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.715 DEBUG:boto:Params: {} 2022-09-02T04:32:04.715 DEBUG:boto:Token: None 2022-09-02T04:32:04.716 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i1FumpjpDAL3jjqNW6sbXqkMKDU= 2022-09-02T04:32:04.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i1FumpjpDAL3jjqNW6sbXqkMKDU=', 'Content-Length': '0'} 2022-09-02T04:32:04.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000086527fe30b4d3ae-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:04.721 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.721 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.722 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.723 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.723 DEBUG:boto:Method: GET 2022-09-02T04:32:04.724 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.724 DEBUG:boto:Data: 2022-09-02T04:32:04.725 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.726 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.727 DEBUG:boto:Params: {} 2022-09-02T04:32:04.727 DEBUG:boto:Token: None 2022-09-02T04:32:04.728 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs= 2022-09-02T04:32:04.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs=', 'Content-Length': '0'} 2022-09-02T04:32:04.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ac66515431395d4-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.732 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.732 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:04.733 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.733 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.734 DEBUG:boto:Method: GET 2022-09-02T04:32:04.735 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.736 DEBUG:boto:Data: 2022-09-02T04:32:04.736 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.737 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.738 DEBUG:boto:Params: {} 2022-09-02T04:32:04.738 DEBUG:boto:Token: None 2022-09-02T04:32:04.739 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:04.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs= 2022-09-02T04:32:04.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:egB/qLNOeE21qFyp7qJTJ8X0lAs=', 'Content-Length': '0'} 2022-09-02T04:32:04.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078310445d35cd1c3-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.744 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.744 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.745 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.745 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.746 DEBUG:boto:Method: GET 2022-09-02T04:32:04.746 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.747 DEBUG:boto:Data: 2022-09-02T04:32:04.747 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.748 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.749 DEBUG:boto:Params: {} 2022-09-02T04:32:04.750 DEBUG:boto:Token: None 2022-09-02T04:32:04.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea61852da78f5dc-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.755 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.756 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.756 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.757 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.757 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.758 DEBUG:boto:Method: GET 2022-09-02T04:32:04.758 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:04.759 DEBUG:boto:Data: 2022-09-02T04:32:04.759 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.761 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.761 DEBUG:boto:Params: {} 2022-09-02T04:32:04.762 DEBUG:boto:Token: None 2022-09-02T04:32:04.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a5d5d4ac5ba6350-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.767 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:04.768 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.769 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.770 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.770 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.771 DEBUG:boto:Method: GET 2022-09-02T04:32:04.771 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.772 DEBUG:boto:Data: 2022-09-02T04:32:04.773 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.774 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.774 DEBUG:boto:Params: {} 2022-09-02T04:32:04.775 DEBUG:boto:Token: None 2022-09-02T04:32:04.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0= 2022-09-02T04:32:04.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0=', 'Content-Length': '0'} 2022-09-02T04:32:04.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065f4dba3452d96f0-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.826 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.827 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.828 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.829 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.830 DEBUG:boto:Method: GET 2022-09-02T04:32:04.831 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.832 DEBUG:boto:Data: 2022-09-02T04:32:04.833 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.835 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.836 DEBUG:boto:Params: {} 2022-09-02T04:32:04.837 DEBUG:boto:Token: None 2022-09-02T04:32:04.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0= 2022-09-02T04:32:04.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0=', 'Content-Length': '0'} 2022-09-02T04:32:04.843 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f462df683ceada85-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.845 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.845 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.846 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.847 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.848 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.849 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.850 DEBUG:boto:Data: 2022-09-02T04:32:04.851 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.853 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.854 DEBUG:boto:Params: {} 2022-09-02T04:32:04.855 DEBUG:boto:Token: None 2022-09-02T04:32:04.856 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4IlVmYpyxwAdJGMNk9XSXMuBbg= 2022-09-02T04:32:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4IlVmYpyxwAdJGMNk9XSXMuBbg=', 'Content-Length': '0'} 2022-09-02T04:32:04.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa78064bd1ee8905-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.862 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.863 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.863 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.864 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.865 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.865 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.866 DEBUG:boto:Data: 2022-09-02T04:32:04.866 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.867 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.868 DEBUG:boto:Params: {} 2022-09-02T04:32:04.869 DEBUG:boto:Token: None 2022-09-02T04:32:04.869 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4IlVmYpyxwAdJGMNk9XSXMuBbg= 2022-09-02T04:32:04.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4IlVmYpyxwAdJGMNk9XSXMuBbg=', 'Content-Length': '0'} 2022-09-02T04:32:04.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a92466288611d7-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:04.874 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.874 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.875 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.875 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.876 DEBUG:boto:Method: GET 2022-09-02T04:32:04.876 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.877 DEBUG:boto:Data: 2022-09-02T04:32:04.877 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.878 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.879 DEBUG:boto:Params: {} 2022-09-02T04:32:04.879 DEBUG:boto:Token: None 2022-09-02T04:32:04.880 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0= 2022-09-02T04:32:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0=', 'Content-Length': '0'} 2022-09-02T04:32:04.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9d038c64340c8aa-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.884 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.884 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.885 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.885 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.886 DEBUG:boto:Method: GET 2022-09-02T04:32:04.887 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.888 DEBUG:boto:Data: 2022-09-02T04:32:04.889 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.891 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.892 DEBUG:boto:Params: {} 2022-09-02T04:32:04.893 DEBUG:boto:Token: None 2022-09-02T04:32:04.894 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:04.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0= 2022-09-02T04:32:04.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udN37h0MCxeVmNKgxJCL/gxr8f0=', 'Content-Length': '0'} 2022-09-02T04:32:04.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b1d96279a644d70-0063118744-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.900 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.901 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.902 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:04.903 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:04.904 DEBUG:boto:Method: GET 2022-09-02T04:32:04.905 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:04.906 DEBUG:boto:Data: 2022-09-02T04:32:04.907 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.909 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.910 DEBUG:boto:Params: {} 2022-09-02T04:32:04.911 DEBUG:boto:Token: None 2022-09-02T04:32:04.912 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007416b0d62b0e5f2d-0063118744-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.919 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.921 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:04.921 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:04.922 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:04.923 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:04.924 DEBUG:boto:Method: GET 2022-09-02T04:32:04.925 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:04.926 DEBUG:boto:Data: 2022-09-02T04:32:04.927 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:04.929 DEBUG:boto:Port: 8000 2022-09-02T04:32:04.930 DEBUG:boto:Params: {} 2022-09-02T04:32:04.931 DEBUG:boto:Token: None 2022-09-02T04:32:04.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/ 2022-09-02T04:32:04.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM= 2022-09-02T04:32:04.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FeelRr3RC5EFwDqMfHbx363iRnM=', 'Content-Length': '0'} 2022-09-02T04:32:04.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cbf73630763c539-0063118744-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.978 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:04.980 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.981 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.982 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.983 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.983 DEBUG:boto:Data: 2022-09-02T04:32:04.984 DEBUG:boto:Headers: {} 2022-09-02T04:32:04.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:04.986 DEBUG:boto:Port: 8001 2022-09-02T04:32:04.987 DEBUG:boto:Params: {} 2022-09-02T04:32:04.988 DEBUG:boto:Token: None 2022-09-02T04:32:04.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:04 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ydKIMq447UQ0iiwuM/boS9ZT5lY= 2022-09-02T04:32:04.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ydKIMq447UQ0iiwuM/boS9ZT5lY=', 'Content-Length': '0'} 2022-09-02T04:32:04.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfe07c8e96a4a477-0063118744-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:04.995 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.996 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.997 DEBUG:boto:Method: HEAD 2022-09-02T04:32:04.998 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:32:04.999 DEBUG:boto:Data: 2022-09-02T04:32:05.000 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:05.002 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.003 DEBUG:boto:Params: {} 2022-09-02T04:32:05.004 DEBUG:boto:Token: None 2022-09-02T04:32:05.005 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:glm3U9esXAiKuMfPGkOwKk2r9Cw= 2022-09-02T04:32:05.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:glm3U9esXAiKuMfPGkOwKk2r9Cw=', 'Content-Length': '0'} 2022-09-02T04:32:05.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f38a369420003bc-0063118745-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:05.012 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.013 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.014 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.015 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.016 DEBUG:boto:Method: GET 2022-09-02T04:32:05.017 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.018 DEBUG:boto:Data: 2022-09-02T04:32:05.018 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.020 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.021 DEBUG:boto:Params: {} 2022-09-02T04:32:05.022 DEBUG:boto:Token: None 2022-09-02T04:32:05.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w= 2022-09-02T04:32:05.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w=', 'Content-Length': '0'} 2022-09-02T04:32:05.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd02e4a23dd85f9a-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.032 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.033 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.033 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.034 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.035 DEBUG:boto:Method: GET 2022-09-02T04:32:05.036 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.037 DEBUG:boto:Data: 2022-09-02T04:32:05.038 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:05.039 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.039 DEBUG:boto:Params: {} 2022-09-02T04:32:05.040 DEBUG:boto:Token: None 2022-09-02T04:32:05.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w= 2022-09-02T04:32:05.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w=', 'Content-Length': '0'} 2022-09-02T04:32:05.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071cd65c200736d0d-0063118745-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.046 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.046 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.047 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.047 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.048 DEBUG:boto:Method: GET 2022-09-02T04:32:05.048 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.049 DEBUG:boto:Data: 2022-09-02T04:32:05.050 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.051 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.052 DEBUG:boto:Params: {} 2022-09-02T04:32:05.053 DEBUG:boto:Token: None 2022-09-02T04:32:05.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077aafdbf1747d933-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.059 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.060 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.061 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.062 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.062 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.063 DEBUG:boto:Method: GET 2022-09-02T04:32:05.064 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.064 DEBUG:boto:Data: 2022-09-02T04:32:05.065 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:05.066 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.066 DEBUG:boto:Params: {} 2022-09-02T04:32:05.067 DEBUG:boto:Token: None 2022-09-02T04:32:05.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db462fffccdd5168-0063118745-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.072 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.073 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-16 zones={test-zone3, test-zone1} 2022-09-02T04:32:05.073 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-16 zones={test-zone3, test-zone2} 2022-09-02T04:32:05.074 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.074 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.075 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.075 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:32:05.076 DEBUG:boto:Data: 2022-09-02T04:32:05.076 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.077 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.078 DEBUG:boto:Params: {} 2022-09-02T04:32:05.078 DEBUG:boto:Token: None 2022-09-02T04:32:05.079 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RoTKW8Is+fFn/RHcsBsYnt7uYeE= 2022-09-02T04:32:05.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RoTKW8Is+fFn/RHcsBsYnt7uYeE=', 'Content-Length': '0'} 2022-09-02T04:32:05.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe6a85c38d41cdc2-0063118745-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.083 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.084 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.084 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.085 DEBUG:boto:Path: /xljbkf-16/ 2022-09-02T04:32:05.085 DEBUG:boto:Data: 2022-09-02T04:32:05.086 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.087 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.087 DEBUG:boto:Params: {} 2022-09-02T04:32:05.088 DEBUG:boto:Token: None 2022-09-02T04:32:05.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RoTKW8Is+fFn/RHcsBsYnt7uYeE= 2022-09-02T04:32:05.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RoTKW8Is+fFn/RHcsBsYnt7uYeE=', 'Content-Length': '0'} 2022-09-02T04:32:05.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055b1039c5a057f3b-0063118745-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:05.094 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.094 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.095 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.095 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.096 DEBUG:boto:Method: GET 2022-09-02T04:32:05.096 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.097 DEBUG:boto:Data: 2022-09-02T04:32:05.097 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.098 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.099 DEBUG:boto:Params: {} 2022-09-02T04:32:05.099 DEBUG:boto:Token: None 2022-09-02T04:32:05.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/?versions 2022-09-02T04:32:05.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI= 2022-09-02T04:32:05.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI=', 'Content-Length': '0'} 2022-09-02T04:32:05.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f647fac20629fa0-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.105 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:05.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:05.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:05.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:05.108 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:05.109 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.110 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.110 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.111 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.111 DEBUG:boto:Method: GET 2022-09-02T04:32:05.112 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.112 DEBUG:boto:Data: 2022-09-02T04:32:05.113 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.114 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.114 DEBUG:boto:Params: {} 2022-09-02T04:32:05.115 DEBUG:boto:Token: None 2022-09-02T04:32:05.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/?versions 2022-09-02T04:32:05.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI= 2022-09-02T04:32:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI=', 'Content-Length': '0'} 2022-09-02T04:32:05.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd1d1bb60f9b356d-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.120 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.121 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:05.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:05.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:05.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:05.123 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.124 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.125 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.125 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.126 DEBUG:boto:Method: GET 2022-09-02T04:32:05.127 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.128 DEBUG:boto:Data: 2022-09-02T04:32:05.128 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.129 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.130 DEBUG:boto:Params: {} 2022-09-02T04:32:05.130 DEBUG:boto:Token: None 2022-09-02T04:32:05.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/?versions 2022-09-02T04:32:05.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI= 2022-09-02T04:32:05.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI=', 'Content-Length': '0'} 2022-09-02T04:32:05.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ea56f8e840f865-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.136 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.138 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.139 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.140 DEBUG:boto:path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.140 DEBUG:boto:auth_path=/xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.141 DEBUG:boto:Method: GET 2022-09-02T04:32:05.143 DEBUG:boto:Path: /xljbkf-16/?versions&max-keys=999 2022-09-02T04:32:05.143 DEBUG:boto:Data: 2022-09-02T04:32:05.144 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.146 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.147 DEBUG:boto:Params: {} 2022-09-02T04:32:05.148 DEBUG:boto:Token: None 2022-09-02T04:32:05.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/?versions 2022-09-02T04:32:05.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI= 2022-09-02T04:32:05.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y5SNS45kW2YQc52LKwWVd3WvbTI=', 'Content-Length': '0'} 2022-09-02T04:32:05.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dd1d3775ddfa40e-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.156 DEBUG:boto:b'xljbkf-16999falsedummyEpPaDytRiBZn9lxmkHc2MMgHmHXmCAUtrue2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4true2022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2UXpYMmzkckungyvLw4PJUPJMqFqQA8Otrue2022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3jM4gXBxJMljLGpwPEfFc7tDmGXhzdPCtrue2022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:05.157 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.158 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.159 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.160 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.161 DEBUG:boto:Method: GET 2022-09-02T04:32:05.161 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.162 DEBUG:boto:Data: 2022-09-02T04:32:05.162 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.164 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.164 DEBUG:boto:Params: {} 2022-09-02T04:32:05.165 DEBUG:boto:Token: None 2022-09-02T04:32:05.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE= 2022-09-02T04:32:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE=', 'Content-Length': '0'} 2022-09-02T04:32:05.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000bf4c76c36ce3721-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.172 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.172 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.173 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.173 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.174 DEBUG:boto:Method: GET 2022-09-02T04:32:05.174 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.175 DEBUG:boto:Data: 2022-09-02T04:32:05.176 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.177 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.177 DEBUG:boto:Params: {} 2022-09-02T04:32:05.178 DEBUG:boto:Token: None 2022-09-02T04:32:05.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE= 2022-09-02T04:32:05.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE=', 'Content-Length': '0'} 2022-09-02T04:32:05.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003669a638ca99f296-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.183 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.183 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.184 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.184 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.185 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.185 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.186 DEBUG:boto:Data: 2022-09-02T04:32:05.186 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.187 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.188 DEBUG:boto:Params: {} 2022-09-02T04:32:05.188 DEBUG:boto:Token: None 2022-09-02T04:32:05.189 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eFfYuwLQizaqaIr7nI1lYi/0BSk= 2022-09-02T04:32:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eFfYuwLQizaqaIr7nI1lYi/0BSk=', 'Content-Length': '0'} 2022-09-02T04:32:05.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3146fa8b90f8d53-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.192 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.193 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.193 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.194 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.194 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.195 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.195 DEBUG:boto:Data: 2022-09-02T04:32:05.196 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.197 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.197 DEBUG:boto:Params: {} 2022-09-02T04:32:05.198 DEBUG:boto:Token: None 2022-09-02T04:32:05.199 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eFfYuwLQizaqaIr7nI1lYi/0BSk= 2022-09-02T04:32:05.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eFfYuwLQizaqaIr7nI1lYi/0BSk=', 'Content-Length': '0'} 2022-09-02T04:32:05.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085120625b05d4b60-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:05.203 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.204 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.204 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.205 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.205 DEBUG:boto:Method: GET 2022-09-02T04:32:05.206 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.206 DEBUG:boto:Data: 2022-09-02T04:32:05.207 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.208 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.209 DEBUG:boto:Params: {} 2022-09-02T04:32:05.209 DEBUG:boto:Token: None 2022-09-02T04:32:05.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE= 2022-09-02T04:32:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE=', 'Content-Length': '0'} 2022-09-02T04:32:05.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2577212acaa2b26-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.214 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.214 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.215 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.215 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.216 DEBUG:boto:Method: GET 2022-09-02T04:32:05.216 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.217 DEBUG:boto:Data: 2022-09-02T04:32:05.217 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.218 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.218 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.219 DEBUG:boto:Params: {} 2022-09-02T04:32:05.220 DEBUG:boto:Token: None 2022-09-02T04:32:05.220 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.221 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE= 2022-09-02T04:32:05.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE=', 'Content-Length': '0'} 2022-09-02T04:32:05.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000497388675a33b65b-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.225 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.225 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.226 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.226 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.227 DEBUG:boto:Method: GET 2022-09-02T04:32:05.227 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.228 DEBUG:boto:Data: 2022-09-02T04:32:05.228 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.229 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.230 DEBUG:boto:Params: {} 2022-09-02T04:32:05.231 DEBUG:boto:Token: None 2022-09-02T04:32:05.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7f784b0168efa6-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.278 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.279 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.280 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.280 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.281 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.282 DEBUG:boto:Method: GET 2022-09-02T04:32:05.283 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.284 DEBUG:boto:Data: 2022-09-02T04:32:05.284 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.286 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.286 DEBUG:boto:Params: {} 2022-09-02T04:32:05.287 DEBUG:boto:Token: None 2022-09-02T04:32:05.288 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46cdd869cd746c2-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.293 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.293 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.294 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.295 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.295 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:32:05.296 DEBUG:boto:Data: 2022-09-02T04:32:05.296 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.297 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.298 DEBUG:boto:Params: {} 2022-09-02T04:32:05.298 DEBUG:boto:Token: None 2022-09-02T04:32:05.299 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy 2022-09-02T04:32:05.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lF8UqJDDRoFcj6X5VbPOfFTW+F4= 2022-09-02T04:32:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lF8UqJDDRoFcj6X5VbPOfFTW+F4=', 'Content-Length': '0'} 2022-09-02T04:32:05.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d387a5d88f17238b-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.303 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.303 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.304 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.304 DEBUG:boto:Path: /xljbkf-16/dummy 2022-09-02T04:32:05.305 DEBUG:boto:Data: 2022-09-02T04:32:05.305 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.306 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.307 DEBUG:boto:Params: {} 2022-09-02T04:32:05.307 DEBUG:boto:Token: None 2022-09-02T04:32:05.308 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy 2022-09-02T04:32:05.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lF8UqJDDRoFcj6X5VbPOfFTW+F4= 2022-09-02T04:32:05.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lF8UqJDDRoFcj6X5VbPOfFTW+F4=', 'Content-Length': '0'} 2022-09-02T04:32:05.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093f3b8477bc23e78-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:05.313 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.313 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.314 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.314 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.315 DEBUG:boto:Method: GET 2022-09-02T04:32:05.315 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.316 DEBUG:boto:Data: 2022-09-02T04:32:05.317 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.318 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.318 DEBUG:boto:Params: {} 2022-09-02T04:32:05.319 DEBUG:boto:Token: None 2022-09-02T04:32:05.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE= 2022-09-02T04:32:05.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE=', 'Content-Length': '0'} 2022-09-02T04:32:05.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a80e4d0e65936b43-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.323 DEBUG:boto:path=/xljbkf-16/dummy 2022-09-02T04:32:05.324 DEBUG:boto:auth_path=/xljbkf-16/dummy 2022-09-02T04:32:05.325 DEBUG:boto:path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.325 DEBUG:boto:auth_path=/xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.326 DEBUG:boto:Method: GET 2022-09-02T04:32:05.327 DEBUG:boto:Path: /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.328 DEBUG:boto:Data: 2022-09-02T04:32:05.328 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.329 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.330 DEBUG:boto:Params: {} 2022-09-02T04:32:05.331 DEBUG:boto:Token: None 2022-09-02T04:32:05.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/dummy?versionId=EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU 2022-09-02T04:32:05.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE= 2022-09-02T04:32:05.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jfpBPHCM1OYx3RY9h98GN7jwHUE=', 'Content-Length': '0'} 2022-09-02T04:32:05.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:27:50 GMT'), ('x-amz-version-id', 'EpPaDytRiBZn9lxmkHc2MMgHmHXmCAU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000accb6e630c4d500c-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.336 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.337 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.343 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.344 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.345 DEBUG:boto:Method: GET 2022-09-02T04:32:05.345 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.346 DEBUG:boto:Data: 2022-09-02T04:32:05.347 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.348 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.348 DEBUG:boto:Params: {} 2022-09-02T04:32:05.349 DEBUG:boto:Token: None 2022-09-02T04:32:05.350 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031708f6563eb19d9-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.394 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.395 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.396 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.397 DEBUG:boto:path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.397 DEBUG:boto:auth_path=/xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.398 DEBUG:boto:Method: GET 2022-09-02T04:32:05.399 DEBUG:boto:Path: /xljbkf-16/?prefix=dummy 2022-09-02T04:32:05.399 DEBUG:boto:Data: 2022-09-02T04:32:05.400 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.401 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.402 DEBUG:boto:Params: {} 2022-09-02T04:32:05.402 DEBUG:boto:Token: None 2022-09-02T04:32:05.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000335c27d70107412-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.408 DEBUG:boto:b'xljbkf-16dummy1000falsedummy2022-09-02T04:27:50.220Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:05.409 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.410 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.411 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.411 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.412 DEBUG:boto:Method: GET 2022-09-02T04:32:05.412 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.413 DEBUG:boto:Data: 2022-09-02T04:32:05.413 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.414 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.415 DEBUG:boto:Params: {} 2022-09-02T04:32:05.415 DEBUG:boto:Token: None 2022-09-02T04:32:05.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ= 2022-09-02T04:32:05.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ=', 'Content-Length': '0'} 2022-09-02T04:32:05.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058017f85b4b1cf9a-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.422 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.423 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.423 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.424 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.425 DEBUG:boto:Method: GET 2022-09-02T04:32:05.426 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.426 DEBUG:boto:Data: 2022-09-02T04:32:05.427 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.428 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.429 DEBUG:boto:Params: {} 2022-09-02T04:32:05.430 DEBUG:boto:Token: None 2022-09-02T04:32:05.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ= 2022-09-02T04:32:05.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ=', 'Content-Length': '0'} 2022-09-02T04:32:05.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063c8a95b06ac33e6-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.436 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.436 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.437 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.438 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.439 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.439 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.440 DEBUG:boto:Data: 2022-09-02T04:32:05.441 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.442 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.443 DEBUG:boto:Params: {} 2022-09-02T04:32:05.443 DEBUG:boto:Token: None 2022-09-02T04:32:05.444 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pSSfD3Cn8pjiLgXUkFzT53oZGlM= 2022-09-02T04:32:05.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pSSfD3Cn8pjiLgXUkFzT53oZGlM=', 'Content-Length': '0'} 2022-09-02T04:32:05.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061462e5d8804eb8c-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.448 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.449 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.449 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.450 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.451 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.451 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.452 DEBUG:boto:Data: 2022-09-02T04:32:05.452 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.453 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.453 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.454 DEBUG:boto:Params: {} 2022-09-02T04:32:05.454 DEBUG:boto:Token: None 2022-09-02T04:32:05.455 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pSSfD3Cn8pjiLgXUkFzT53oZGlM= 2022-09-02T04:32:05.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pSSfD3Cn8pjiLgXUkFzT53oZGlM=', 'Content-Length': '0'} 2022-09-02T04:32:05.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a06f6467e096ae2b-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:05.460 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.460 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.461 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.461 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.462 DEBUG:boto:Method: GET 2022-09-02T04:32:05.463 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.463 DEBUG:boto:Data: 2022-09-02T04:32:05.464 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.465 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.465 DEBUG:boto:Params: {} 2022-09-02T04:32:05.466 DEBUG:boto:Token: None 2022-09-02T04:32:05.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ= 2022-09-02T04:32:05.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ=', 'Content-Length': '0'} 2022-09-02T04:32:05.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000387c41f869adf891-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.470 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.471 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.471 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.472 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.472 DEBUG:boto:Method: GET 2022-09-02T04:32:05.473 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.474 DEBUG:boto:Data: 2022-09-02T04:32:05.474 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.475 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.476 DEBUG:boto:Params: {} 2022-09-02T04:32:05.476 DEBUG:boto:Token: None 2022-09-02T04:32:05.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ= 2022-09-02T04:32:05.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ=', 'Content-Length': '0'} 2022-09-02T04:32:05.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee189655680cfb93-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.526 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.527 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.528 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.528 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.529 DEBUG:boto:Method: GET 2022-09-02T04:32:05.529 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.530 DEBUG:boto:Data: 2022-09-02T04:32:05.530 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.531 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.532 DEBUG:boto:Params: {} 2022-09-02T04:32:05.532 DEBUG:boto:Token: None 2022-09-02T04:32:05.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002559745692648e1c-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.537 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.538 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.539 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.539 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.540 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.540 DEBUG:boto:Method: GET 2022-09-02T04:32:05.541 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.542 DEBUG:boto:Data: 2022-09-02T04:32:05.542 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.543 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.544 DEBUG:boto:Params: {} 2022-09-02T04:32:05.544 DEBUG:boto:Token: None 2022-09-02T04:32:05.545 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63a68847061c217-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.550 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.550 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.551 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.551 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.552 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.552 DEBUG:boto:Data: 2022-09-02T04:32:05.553 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.554 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.554 DEBUG:boto:Params: {} 2022-09-02T04:32:05.555 DEBUG:boto:Token: None 2022-09-02T04:32:05.555 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HClDNpgP4hbwetdzmCisMTYKTow= 2022-09-02T04:32:05.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HClDNpgP4hbwetdzmCisMTYKTow=', 'Content-Length': '0'} 2022-09-02T04:32:05.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000913090ab5213f48b-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.560 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.560 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.561 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.561 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.562 DEBUG:boto:Data: 2022-09-02T04:32:05.562 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.563 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.564 DEBUG:boto:Params: {} 2022-09-02T04:32:05.564 DEBUG:boto:Token: None 2022-09-02T04:32:05.565 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HClDNpgP4hbwetdzmCisMTYKTow= 2022-09-02T04:32:05.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HClDNpgP4hbwetdzmCisMTYKTow=', 'Content-Length': '0'} 2022-09-02T04:32:05.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ec665e88b8021fb-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:05.570 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.570 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.571 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.571 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.572 DEBUG:boto:Method: GET 2022-09-02T04:32:05.572 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.573 DEBUG:boto:Data: 2022-09-02T04:32:05.573 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.575 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.575 DEBUG:boto:Params: {} 2022-09-02T04:32:05.576 DEBUG:boto:Token: None 2022-09-02T04:32:05.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ= 2022-09-02T04:32:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ=', 'Content-Length': '0'} 2022-09-02T04:32:05.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013a83cc6c5bbbc56-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.580 DEBUG:boto:path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.581 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1 2022-09-02T04:32:05.581 DEBUG:boto:path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.582 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.582 DEBUG:boto:Method: GET 2022-09-02T04:32:05.583 DEBUG:boto:Path: /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.583 DEBUG:boto:Data: 2022-09-02T04:32:05.584 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.585 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.585 DEBUG:boto:Params: {} 2022-09-02T04:32:05.586 DEBUG:boto:Token: None 2022-09-02T04:32:05.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone1?versionId=l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4 2022-09-02T04:32:05.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ= 2022-09-02T04:32:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wR18yp/aXHTTUsG66MTjLJNJOnQ=', 'Content-Length': '0'} 2022-09-02T04:32:05.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'l2snnfE9Q5D7r1MA1n0QAuacR0FSTK4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000549f7e9013dfa305-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.591 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.591 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.592 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.592 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.593 DEBUG:boto:Method: GET 2022-09-02T04:32:05.593 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.594 DEBUG:boto:Data: 2022-09-02T04:32:05.594 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.595 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.596 DEBUG:boto:Params: {} 2022-09-02T04:32:05.596 DEBUG:boto:Token: None 2022-09-02T04:32:05.597 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f586919cd604ca-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.601 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.602 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.603 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.603 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.604 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.604 DEBUG:boto:Method: GET 2022-09-02T04:32:05.605 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone1 2022-09-02T04:32:05.606 DEBUG:boto:Data: 2022-09-02T04:32:05.606 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.607 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.608 DEBUG:boto:Params: {} 2022-09-02T04:32:05.608 DEBUG:boto:Token: None 2022-09-02T04:32:05.609 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e24030d03f9df81-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.614 DEBUG:boto:b'xljbkf-16obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.291Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:05.615 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.615 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.616 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.617 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.617 DEBUG:boto:Method: GET 2022-09-02T04:32:05.618 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.618 DEBUG:boto:Data: 2022-09-02T04:32:05.619 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.620 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.620 DEBUG:boto:Params: {} 2022-09-02T04:32:05.621 DEBUG:boto:Token: None 2022-09-02T04:32:05.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto= 2022-09-02T04:32:05.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto=', 'Content-Length': '0'} 2022-09-02T04:32:05.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000760f1ee6dd30e51d-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.625 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.626 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.627 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.627 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.628 DEBUG:boto:Method: GET 2022-09-02T04:32:05.628 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.629 DEBUG:boto:Data: 2022-09-02T04:32:05.629 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.630 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.631 DEBUG:boto:Params: {} 2022-09-02T04:32:05.631 DEBUG:boto:Token: None 2022-09-02T04:32:05.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto= 2022-09-02T04:32:05.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto=', 'Content-Length': '0'} 2022-09-02T04:32:05.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003207feb58da73794-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.636 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.636 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.637 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.637 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.638 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.638 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.639 DEBUG:boto:Data: 2022-09-02T04:32:05.639 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.640 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.641 DEBUG:boto:Params: {} 2022-09-02T04:32:05.641 DEBUG:boto:Token: None 2022-09-02T04:32:05.642 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:roihqoWJWoTXKuxaZ/ieQED5gyI= 2022-09-02T04:32:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:roihqoWJWoTXKuxaZ/ieQED5gyI=', 'Content-Length': '0'} 2022-09-02T04:32:05.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f224d650d85b16d-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.645 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.646 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.647 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.647 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.647 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.648 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.648 DEBUG:boto:Data: 2022-09-02T04:32:05.649 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.650 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.651 DEBUG:boto:Params: {} 2022-09-02T04:32:05.651 DEBUG:boto:Token: None 2022-09-02T04:32:05.652 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:roihqoWJWoTXKuxaZ/ieQED5gyI= 2022-09-02T04:32:05.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:roihqoWJWoTXKuxaZ/ieQED5gyI=', 'Content-Length': '0'} 2022-09-02T04:32:05.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af1083410df2f48c-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:05.656 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.657 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.657 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.658 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.658 DEBUG:boto:Method: GET 2022-09-02T04:32:05.659 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.659 DEBUG:boto:Data: 2022-09-02T04:32:05.660 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.661 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.662 DEBUG:boto:Params: {} 2022-09-02T04:32:05.662 DEBUG:boto:Token: None 2022-09-02T04:32:05.663 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto= 2022-09-02T04:32:05.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto=', 'Content-Length': '0'} 2022-09-02T04:32:05.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091d33272b9dfcf5d-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.667 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.667 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.668 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.668 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.669 DEBUG:boto:Method: GET 2022-09-02T04:32:05.669 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.670 DEBUG:boto:Data: 2022-09-02T04:32:05.670 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.671 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.672 DEBUG:boto:Params: {} 2022-09-02T04:32:05.673 DEBUG:boto:Token: None 2022-09-02T04:32:05.674 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto= 2022-09-02T04:32:05.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto=', 'Content-Length': '0'} 2022-09-02T04:32:05.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bf414265adedb22-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.680 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.681 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.682 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.683 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.684 DEBUG:boto:Method: GET 2022-09-02T04:32:05.685 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.686 DEBUG:boto:Data: 2022-09-02T04:32:05.687 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.688 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.689 DEBUG:boto:Params: {} 2022-09-02T04:32:05.690 DEBUG:boto:Token: None 2022-09-02T04:32:05.691 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070568979c15ee6bd-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.742 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.744 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.745 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.746 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.747 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.748 DEBUG:boto:Method: GET 2022-09-02T04:32:05.749 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.750 DEBUG:boto:Data: 2022-09-02T04:32:05.751 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.753 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.753 DEBUG:boto:Params: {} 2022-09-02T04:32:05.754 DEBUG:boto:Token: None 2022-09-02T04:32:05.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064baf35dc894e117-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.759 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.760 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.760 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.761 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.761 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.762 DEBUG:boto:Data: 2022-09-02T04:32:05.763 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.764 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.764 DEBUG:boto:Params: {} 2022-09-02T04:32:05.765 DEBUG:boto:Token: None 2022-09-02T04:32:05.765 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wYJqAgV8O5EBqzo+vK8HgraXhV4= 2022-09-02T04:32:05.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wYJqAgV8O5EBqzo+vK8HgraXhV4=', 'Content-Length': '0'} 2022-09-02T04:32:05.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fab28c50556d940e-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.769 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.770 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.770 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.771 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.772 DEBUG:boto:Data: 2022-09-02T04:32:05.773 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.775 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.776 DEBUG:boto:Params: {} 2022-09-02T04:32:05.776 DEBUG:boto:Token: None 2022-09-02T04:32:05.777 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wYJqAgV8O5EBqzo+vK8HgraXhV4= 2022-09-02T04:32:05.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wYJqAgV8O5EBqzo+vK8HgraXhV4=', 'Content-Length': '0'} 2022-09-02T04:32:05.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019f136bbeebc629a-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:05.783 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.784 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.784 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.785 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.786 DEBUG:boto:Method: GET 2022-09-02T04:32:05.786 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.787 DEBUG:boto:Data: 2022-09-02T04:32:05.787 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.788 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.789 DEBUG:boto:Params: {} 2022-09-02T04:32:05.790 DEBUG:boto:Token: None 2022-09-02T04:32:05.790 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto= 2022-09-02T04:32:05.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto=', 'Content-Length': '0'} 2022-09-02T04:32:05.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a03cad6bde4ad51f-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.794 DEBUG:boto:path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.795 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2 2022-09-02T04:32:05.795 DEBUG:boto:path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.796 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.796 DEBUG:boto:Method: GET 2022-09-02T04:32:05.797 DEBUG:boto:Path: /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.797 DEBUG:boto:Data: 2022-09-02T04:32:05.798 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.799 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.800 DEBUG:boto:Params: {} 2022-09-02T04:32:05.801 DEBUG:boto:Token: None 2022-09-02T04:32:05.801 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone2?versionId=UXpYMmzkckungyvLw4PJUPJMqFqQA8O 2022-09-02T04:32:05.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto= 2022-09-02T04:32:05.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S2ioqwnUpCvK1T4UKTqo+UHoEto=', 'Content-Length': '0'} 2022-09-02T04:32:05.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'UXpYMmzkckungyvLw4PJUPJMqFqQA8O'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000111e8edb15efaf83-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.850 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.851 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.852 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.853 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.854 DEBUG:boto:Method: GET 2022-09-02T04:32:05.855 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.856 DEBUG:boto:Data: 2022-09-02T04:32:05.857 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.858 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.859 DEBUG:boto:Params: {} 2022-09-02T04:32:05.860 DEBUG:boto:Token: None 2022-09-02T04:32:05.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d14b1099e94c160-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.865 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.866 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.866 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.867 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.867 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.868 DEBUG:boto:Method: GET 2022-09-02T04:32:05.868 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone2 2022-09-02T04:32:05.869 DEBUG:boto:Data: 2022-09-02T04:32:05.869 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.871 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.871 DEBUG:boto:Params: {} 2022-09-02T04:32:05.872 DEBUG:boto:Token: None 2022-09-02T04:32:05.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076760cec26be839e-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.878 DEBUG:boto:b'xljbkf-16obj-test-zone21000falseobj-test-zone22022-09-02T04:29:01.496Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:05.879 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.880 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.881 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.881 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.882 DEBUG:boto:Method: GET 2022-09-02T04:32:05.882 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.883 DEBUG:boto:Data: 2022-09-02T04:32:05.883 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.885 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.885 DEBUG:boto:Params: {} 2022-09-02T04:32:05.886 DEBUG:boto:Token: None 2022-09-02T04:32:05.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w= 2022-09-02T04:32:05.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w=', 'Content-Length': '0'} 2022-09-02T04:32:05.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000192ba3929b51dcf6-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.890 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.891 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.891 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.892 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.892 DEBUG:boto:Method: GET 2022-09-02T04:32:05.893 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.893 DEBUG:boto:Data: 2022-09-02T04:32:05.894 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.895 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.896 DEBUG:boto:Params: {} 2022-09-02T04:32:05.896 DEBUG:boto:Token: None 2022-09-02T04:32:05.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w= 2022-09-02T04:32:05.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w=', 'Content-Length': '0'} 2022-09-02T04:32:05.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f404f13f10e797c-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.903 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.904 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.904 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.905 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.906 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.907 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.907 DEBUG:boto:Data: 2022-09-02T04:32:05.908 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.910 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.911 DEBUG:boto:Params: {} 2022-09-02T04:32:05.911 DEBUG:boto:Token: None 2022-09-02T04:32:05.912 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:259+MHBo83F13a6Dt/8F+K2Gwuk= 2022-09-02T04:32:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:259+MHBo83F13a6Dt/8F+K2Gwuk=', 'Content-Length': '0'} 2022-09-02T04:32:05.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d898c593c9474771-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.917 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.918 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.918 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.919 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.919 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.920 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.920 DEBUG:boto:Data: 2022-09-02T04:32:05.921 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.922 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.922 DEBUG:boto:Params: {} 2022-09-02T04:32:05.923 DEBUG:boto:Token: None 2022-09-02T04:32:05.923 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:259+MHBo83F13a6Dt/8F+K2Gwuk= 2022-09-02T04:32:05.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:259+MHBo83F13a6Dt/8F+K2Gwuk=', 'Content-Length': '0'} 2022-09-02T04:32:05.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c77fcdab1133518d-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:05.929 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.929 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.930 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.930 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.931 DEBUG:boto:Method: GET 2022-09-02T04:32:05.932 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.933 DEBUG:boto:Data: 2022-09-02T04:32:05.933 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.935 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.935 DEBUG:boto:Params: {} 2022-09-02T04:32:05.936 DEBUG:boto:Token: None 2022-09-02T04:32:05.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w= 2022-09-02T04:32:05.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w=', 'Content-Length': '0'} 2022-09-02T04:32:05.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012ef22fd58082ad9-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.940 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.941 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.942 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.943 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.944 DEBUG:boto:Method: GET 2022-09-02T04:32:05.945 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.946 DEBUG:boto:Data: 2022-09-02T04:32:05.947 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.949 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.950 DEBUG:boto:Params: {} 2022-09-02T04:32:05.951 DEBUG:boto:Token: None 2022-09-02T04:32:05.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:05.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w= 2022-09-02T04:32:05.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UUWCaY5xTzSPC+HqxYB2q09Vp1w=', 'Content-Length': '0'} 2022-09-02T04:32:05.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bacacfdb470034bd-0063118745-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.958 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.959 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.959 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.960 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.961 DEBUG:boto:Method: GET 2022-09-02T04:32:05.962 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.963 DEBUG:boto:Data: 2022-09-02T04:32:05.964 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.966 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.967 DEBUG:boto:Params: {} 2022-09-02T04:32:05.968 DEBUG:boto:Token: None 2022-09-02T04:32:05.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd92a5dd57471a2b-0063118745-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.974 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.975 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:05.976 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:05.976 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.977 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.977 DEBUG:boto:Method: GET 2022-09-02T04:32:05.978 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:05.978 DEBUG:boto:Data: 2022-09-02T04:32:05.979 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:05.980 DEBUG:boto:Port: 8000 2022-09-02T04:32:05.981 DEBUG:boto:Params: {} 2022-09-02T04:32:05.982 DEBUG:boto:Token: None 2022-09-02T04:32:05.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/ 2022-09-02T04:32:05.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U= 2022-09-02T04:32:05.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZVttW19Cb33vetTVzMRdFxlDq4U=', 'Content-Length': '0'} 2022-09-02T04:32:05.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f36d7c7a645c76bd-0063118745-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.987 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:05.988 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.989 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.989 DEBUG:boto:Method: HEAD 2022-09-02T04:32:05.990 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.990 DEBUG:boto:Data: 2022-09-02T04:32:05.991 DEBUG:boto:Headers: {} 2022-09-02T04:32:05.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:05.992 DEBUG:boto:Port: 8001 2022-09-02T04:32:05.992 DEBUG:boto:Params: {} 2022-09-02T04:32:05.993 DEBUG:boto:Token: None 2022-09-02T04:32:05.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:05 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:glm3U9esXAiKuMfPGkOwKk2r9Cw= 2022-09-02T04:32:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:glm3U9esXAiKuMfPGkOwKk2r9Cw=', 'Content-Length': '0'} 2022-09-02T04:32:05.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f3c41d55d59772-0063118745-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:05.998 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:05.999 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.000 DEBUG:boto:Method: HEAD 2022-09-02T04:32:06.001 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.002 DEBUG:boto:Data: 2022-09-02T04:32:06.003 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.004 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.005 DEBUG:boto:Params: {} 2022-09-02T04:32:06.005 DEBUG:boto:Token: None 2022-09-02T04:32:06.006 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HDEU+BwgWpu8AQtgOUsWv6om5xs= 2022-09-02T04:32:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HDEU+BwgWpu8AQtgOUsWv6om5xs=', 'Content-Length': '0'} 2022-09-02T04:32:06.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004eaee35dbce7e8d4-0063118746-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:06.011 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.012 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.012 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.013 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.013 DEBUG:boto:Method: GET 2022-09-02T04:32:06.014 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.014 DEBUG:boto:Data: 2022-09-02T04:32:06.015 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:06.016 DEBUG:boto:Port: 8001 2022-09-02T04:32:06.016 DEBUG:boto:Params: {} 2022-09-02T04:32:06.017 DEBUG:boto:Token: None 2022-09-02T04:32:06.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AEqFFZjPVX+l/bdf7yEtqL07NMU= 2022-09-02T04:32:06.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AEqFFZjPVX+l/bdf7yEtqL07NMU=', 'Content-Length': '0'} 2022-09-02T04:32:06.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba31436e07c193f8-0063118746-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.021 DEBUG:boto:path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.022 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3 2022-09-02T04:32:06.023 DEBUG:boto:path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.023 DEBUG:boto:auth_path=/xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.024 DEBUG:boto:Method: GET 2022-09-02T04:32:06.024 DEBUG:boto:Path: /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.025 DEBUG:boto:Data: 2022-09-02T04:32:06.025 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.026 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.027 DEBUG:boto:Params: {} 2022-09-02T04:32:06.027 DEBUG:boto:Token: None 2022-09-02T04:32:06.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-16/obj-test-zone3?versionId=jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC 2022-09-02T04:32:06.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AEqFFZjPVX+l/bdf7yEtqL07NMU= 2022-09-02T04:32:06.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AEqFFZjPVX+l/bdf7yEtqL07NMU=', 'Content-Length': '0'} 2022-09-02T04:32:06.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'jM4gXBxJMljLGpwPEfFc7tDmGXhzdPC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029f07da32fcf686e-0063118746-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.033 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:06.034 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:06.035 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:06.036 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:06.037 DEBUG:boto:Method: GET 2022-09-02T04:32:06.038 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:06.038 DEBUG:boto:Data: 2022-09-02T04:32:06.039 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:06.040 DEBUG:boto:Port: 8001 2022-09-02T04:32:06.041 DEBUG:boto:Params: {} 2022-09-02T04:32:06.041 DEBUG:boto:Token: None 2022-09-02T04:32:06.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-16/ 2022-09-02T04:32:06.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:16p7gc51gef/eIrpWbzi7gVkvuI= 2022-09-02T04:32:06.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:16p7gc51gef/eIrpWbzi7gVkvuI=', 'Content-Length': '0'} 2022-09-02T04:32:06.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000876411ed581bcbec-0063118746-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.046 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:06.047 DEBUG:boto:path=/xljbkf-16/ 2022-09-02T04:32:06.048 DEBUG:boto:auth_path=/xljbkf-16/ 2022-09-02T04:32:06.048 DEBUG:boto:path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:06.049 DEBUG:boto:auth_path=/xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:06.049 DEBUG:boto:Method: GET 2022-09-02T04:32:06.050 DEBUG:boto:Path: /xljbkf-16/?prefix=obj-test-zone3 2022-09-02T04:32:06.050 DEBUG:boto:Data: 2022-09-02T04:32:06.051 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.052 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.053 DEBUG:boto:Params: {} 2022-09-02T04:32:06.054 DEBUG:boto:Token: None 2022-09-02T04:32:06.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-16/ 2022-09-02T04:32:06.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:16p7gc51gef/eIrpWbzi7gVkvuI= 2022-09-02T04:32:06.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:16p7gc51gef/eIrpWbzi7gVkvuI=', 'Content-Length': '0'} 2022-09-02T04:32:06.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d32a8f4bd4d09c-0063118746-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.060 DEBUG:boto:b'xljbkf-16obj-test-zone31000falseobj-test-zone32022-09-02T04:29:01.714Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:06.061 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-16 zones={test-zone3, test-zone2} 2022-09-02T04:32:06.061 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-17 2022-09-02T04:32:06.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.063 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.128 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.128 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.187 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:06.187+0000 7f3f661c6600 1 bucket sync caught up with source: 2022-09-02T04:32:06.188 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.29.5, , 00000000004.10.5, , 00000000008.22.5, , , , , ] 2022-09-02T04:32:06.188 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.29.5, , 00000000004.10.5, , 00000000008.22.5, , , , , ] 2022-09-02T04:32:06.188 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:06.187+0000 7f3f661c6600 0 bucket checkpoint complete 2022-09-02T04:32:06.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-17 2022-09-02T04:32:06.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.213 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.329 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.328+0000 7f4f185bc600 1 bucket sync caught up with source: 2022-09-02T04:32:06.329 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000011.29.5, , 00000000004.10.5, , 00000000008.22.5, , , , , ] 2022-09-02T04:32:06.330 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000011.29.5, , 00000000004.10.5, , 00000000008.22.5, , , , , ] 2022-09-02T04:32:06.330 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.328+0000 7f4f185bc600 0 bucket checkpoint complete 2022-09-02T04:32:06.350 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-17 2022-09-02T04:32:06.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.353 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.411 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.411 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.450 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:06.450+0000 7fe9a6baa600 1 bucket sync caught up with source: 2022-09-02T04:32:06.451 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.29.5, , 00000000004.10.5, , 00000000008.22.5, , , , , ] 2022-09-02T04:32:06.451 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.29.5, , 00000000004.10.5, , 00000000008.22.5, , , , , ] 2022-09-02T04:32:06.451 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:06.450+0000 7fe9a6baa600 0 bucket checkpoint complete 2022-09-02T04:32:06.463 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-17 2022-09-02T04:32:06.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.465 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.557 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.556+0000 7f0990986600 1 bucket sync caught up with source: 2022-09-02T04:32:06.557 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.29.5, , 00000000011.32.5, , 00000000009.51.5, , , , , ] 2022-09-02T04:32:06.557 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.29.5, , 00000000011.32.5, , 00000000009.51.5, , , , , ] 2022-09-02T04:32:06.558 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.556+0000 7f0990986600 0 bucket checkpoint complete 2022-09-02T04:32:06.569 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-17 2022-09-02T04:32:06.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.571 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.664 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.664+0000 7f3cec519600 1 bucket sync caught up with source: 2022-09-02T04:32:06.665 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.29.5, , 00000000011.32.5, , 00000000009.51.5, , , , , ] 2022-09-02T04:32:06.665 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.29.5, , 00000000011.32.5, , 00000000009.51.5, , , , , ] 2022-09-02T04:32:06.665 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.664+0000 7f3cec519600 0 bucket checkpoint complete 2022-09-02T04:32:06.679 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-17 2022-09-02T04:32:06.680 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-17 2022-09-02T04:32:06.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.738 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.738 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.778 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.778+0000 7f06f385e600 1 bucket sync caught up with source: 2022-09-02T04:32:06.779 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.22.5, , 00000000004.13.5, , 00000000015.62.5, , , , , ] 2022-09-02T04:32:06.779 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.22.5, , 00000000004.13.5, , 00000000015.62.5, , , , , ] 2022-09-02T04:32:06.779 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:06.778+0000 7f06f385e600 0 bucket checkpoint complete 2022-09-02T04:32:06.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-17 2022-09-02T04:32:06.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:06.792 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:06.849 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:06.850 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:06.883 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:06.882+0000 7fc56d73d600 1 bucket sync caught up with source: 2022-09-02T04:32:06.884 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000004.22.5, , 00000000004.13.5, , 00000000015.62.5, , , , , ] 2022-09-02T04:32:06.884 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000004.22.5, , 00000000004.13.5, , 00000000015.62.5, , , , , ] 2022-09-02T04:32:06.884 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:06.882+0000 7fc56d73d600 0 bucket checkpoint complete 2022-09-02T04:32:06.899 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-17 2022-09-02T04:32:06.900 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-17 zones={test-zone2, test-zone1} 2022-09-02T04:32:06.901 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:06.901 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:06.902 DEBUG:boto:Method: HEAD 2022-09-02T04:32:06.902 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:32:06.903 DEBUG:boto:Data: 2022-09-02T04:32:06.904 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.905 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.905 DEBUG:boto:Params: {} 2022-09-02T04:32:06.906 DEBUG:boto:Token: None 2022-09-02T04:32:06.907 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/ 2022-09-02T04:32:06.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6dgntpfabpY5Ga+uSVVq7DgK/Ho= 2022-09-02T04:32:06.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6dgntpfabpY5Ga+uSVVq7DgK/Ho=', 'Content-Length': '0'} 2022-09-02T04:32:06.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000341dc20acae670ef-0063118746-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.911 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:06.912 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:06.912 DEBUG:boto:Method: HEAD 2022-09-02T04:32:06.913 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:32:06.914 DEBUG:boto:Data: 2022-09-02T04:32:06.914 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:06.916 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.917 DEBUG:boto:Params: {} 2022-09-02T04:32:06.918 DEBUG:boto:Token: None 2022-09-02T04:32:06.919 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/ 2022-09-02T04:32:06.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6dgntpfabpY5Ga+uSVVq7DgK/Ho= 2022-09-02T04:32:06.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6dgntpfabpY5Ga+uSVVq7DgK/Ho=', 'Content-Length': '0'} 2022-09-02T04:32:06.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc1307aa1a6f44e4-0063118746-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.924 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:06.925 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:06.925 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:06.926 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.926 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.927 DEBUG:boto:Method: GET 2022-09-02T04:32:06.927 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.928 DEBUG:boto:Data: 2022-09-02T04:32:06.928 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.929 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.930 DEBUG:boto:Params: {} 2022-09-02T04:32:06.930 DEBUG:boto:Token: None 2022-09-02T04:32:06.931 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/?versions 2022-09-02T04:32:06.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU= 2022-09-02T04:32:06.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU=', 'Content-Length': '0'} 2022-09-02T04:32:06.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006339977ea72d3416-0063118746-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.936 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:06.938 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:06.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:06.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:06.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:06.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:06.942 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:06.943 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:06.944 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.945 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.946 DEBUG:boto:Method: GET 2022-09-02T04:32:06.946 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.947 DEBUG:boto:Data: 2022-09-02T04:32:06.947 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:06.948 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.949 DEBUG:boto:Params: {} 2022-09-02T04:32:06.950 DEBUG:boto:Token: None 2022-09-02T04:32:06.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/?versions 2022-09-02T04:32:06.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU= 2022-09-02T04:32:06.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU=', 'Content-Length': '0'} 2022-09-02T04:32:06.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061ac7f65010f2ea0-0063118746-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.955 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:06.956 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:06.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:06.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:06.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:06.960 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:06.960 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:06.961 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.962 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.963 DEBUG:boto:Method: GET 2022-09-02T04:32:06.964 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.965 DEBUG:boto:Data: 2022-09-02T04:32:06.966 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.967 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.968 DEBUG:boto:Params: {} 2022-09-02T04:32:06.969 DEBUG:boto:Token: None 2022-09-02T04:32:06.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/?versions 2022-09-02T04:32:06.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU= 2022-09-02T04:32:06.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU=', 'Content-Length': '0'} 2022-09-02T04:32:06.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be6e02f95777746-0063118746-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.974 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:06.975 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:06.976 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:06.976 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.977 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.977 DEBUG:boto:Method: GET 2022-09-02T04:32:06.978 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:06.978 DEBUG:boto:Data: 2022-09-02T04:32:06.979 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:06.980 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.981 DEBUG:boto:Params: {} 2022-09-02T04:32:06.981 DEBUG:boto:Token: None 2022-09-02T04:32:06.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/?versions 2022-09-02T04:32:06.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU= 2022-09-02T04:32:06.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3UnGErxZsGIyKEQkCeEA/5DkZSU=', 'Content-Length': '0'} 2022-09-02T04:32:06.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9cb50c27f1c4f9c-0063118746-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.986 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:06.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:06.988 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:06.989 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:06.989 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:06.990 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:06.990 DEBUG:boto:Method: GET 2022-09-02T04:32:06.991 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:06.991 DEBUG:boto:Data: 2022-09-02T04:32:06.992 DEBUG:boto:Headers: {} 2022-09-02T04:32:06.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:06.993 DEBUG:boto:Port: 8000 2022-09-02T04:32:06.994 DEBUG:boto:Params: {} 2022-09-02T04:32:06.994 DEBUG:boto:Token: None 2022-09-02T04:32:06.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:06 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:06.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KTNkfUqNcZAtG7iiL9pyGjVUMAI= 2022-09-02T04:32:06.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KTNkfUqNcZAtG7iiL9pyGjVUMAI=', 'Content-Length': '0'} 2022-09-02T04:32:06.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000215b89f07c0c9688-0063118746-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:06.999 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.000 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.001 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.001 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.002 DEBUG:boto:Method: GET 2022-09-02T04:32:07.003 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.003 DEBUG:boto:Data: 2022-09-02T04:32:07.004 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.005 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.006 DEBUG:boto:Params: {} 2022-09-02T04:32:07.007 DEBUG:boto:Token: None 2022-09-02T04:32:07.008 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.008 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw= 2022-09-02T04:32:07.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw=', 'Content-Length': '0'} 2022-09-02T04:32:07.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000590627d262299c42-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.012 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.013 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.014 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.014 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.015 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.016 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.016 DEBUG:boto:Data: 2022-09-02T04:32:07.017 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.018 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.019 DEBUG:boto:Params: {} 2022-09-02T04:32:07.020 DEBUG:boto:Token: None 2022-09-02T04:32:07.020 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IDGQzMrxvLIGqiPQfjtRnoTQDwU= 2022-09-02T04:32:07.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IDGQzMrxvLIGqiPQfjtRnoTQDwU=', 'Content-Length': '0'} 2022-09-02T04:32:07.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000435f4b3abd0bfa1c-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.025 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.025 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.026 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.027 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.027 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.028 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.029 DEBUG:boto:Data: 2022-09-02T04:32:07.029 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.030 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.031 DEBUG:boto:Params: {} 2022-09-02T04:32:07.032 DEBUG:boto:Token: None 2022-09-02T04:32:07.032 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IDGQzMrxvLIGqiPQfjtRnoTQDwU= 2022-09-02T04:32:07.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IDGQzMrxvLIGqiPQfjtRnoTQDwU=', 'Content-Length': '0'} 2022-09-02T04:32:07.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af3e53b00cda01d1-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:07.037 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.038 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.039 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.039 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.040 DEBUG:boto:Method: GET 2022-09-02T04:32:07.041 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.042 DEBUG:boto:Data: 2022-09-02T04:32:07.042 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.043 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.044 DEBUG:boto:Params: {} 2022-09-02T04:32:07.045 DEBUG:boto:Token: None 2022-09-02T04:32:07.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw= 2022-09-02T04:32:07.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw=', 'Content-Length': '0'} 2022-09-02T04:32:07.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092b4650ab7dc3834-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.050 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.051 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.051 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.052 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.053 DEBUG:boto:Method: GET 2022-09-02T04:32:07.053 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.055 DEBUG:boto:Data: 2022-09-02T04:32:07.055 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.057 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.057 DEBUG:boto:Params: {} 2022-09-02T04:32:07.058 DEBUG:boto:Token: None 2022-09-02T04:32:07.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw= 2022-09-02T04:32:07.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw=', 'Content-Length': '0'} 2022-09-02T04:32:07.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a907fbb6721fe0d3-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.063 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.064 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.065 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.066 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.067 DEBUG:boto:Method: GET 2022-09-02T04:32:07.067 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.068 DEBUG:boto:Data: 2022-09-02T04:32:07.069 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.069 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.070 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.071 DEBUG:boto:Params: {} 2022-09-02T04:32:07.072 DEBUG:boto:Token: None 2022-09-02T04:32:07.072 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.073 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003555104871ec8875-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.118 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.119 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.120 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.121 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.122 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.122 DEBUG:boto:Method: GET 2022-09-02T04:32:07.123 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.124 DEBUG:boto:Data: 2022-09-02T04:32:07.125 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.126 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.127 DEBUG:boto:Params: {} 2022-09-02T04:32:07.128 DEBUG:boto:Token: None 2022-09-02T04:32:07.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3be8b1db6728bf-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.133 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.134 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.135 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.136 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.136 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:32:07.137 DEBUG:boto:Data: 2022-09-02T04:32:07.137 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.138 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.139 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.139 DEBUG:boto:Params: {} 2022-09-02T04:32:07.140 DEBUG:boto:Token: None 2022-09-02T04:32:07.141 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy 2022-09-02T04:32:07.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:azaC6khC9EaU0NrHzZ3VaftsS6c= 2022-09-02T04:32:07.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:azaC6khC9EaU0NrHzZ3VaftsS6c=', 'Content-Length': '0'} 2022-09-02T04:32:07.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b41104384216e7b9-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.145 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.146 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.147 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.147 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:32:07.148 DEBUG:boto:Data: 2022-09-02T04:32:07.149 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.150 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.151 DEBUG:boto:Params: {} 2022-09-02T04:32:07.152 DEBUG:boto:Token: None 2022-09-02T04:32:07.152 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy 2022-09-02T04:32:07.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:azaC6khC9EaU0NrHzZ3VaftsS6c= 2022-09-02T04:32:07.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:azaC6khC9EaU0NrHzZ3VaftsS6c=', 'Content-Length': '0'} 2022-09-02T04:32:07.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7be24c429ca9c16-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:07.158 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.159 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.160 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.160 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.161 DEBUG:boto:Method: GET 2022-09-02T04:32:07.162 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.162 DEBUG:boto:Data: 2022-09-02T04:32:07.163 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.164 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.165 DEBUG:boto:Params: {} 2022-09-02T04:32:07.166 DEBUG:boto:Token: None 2022-09-02T04:32:07.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw= 2022-09-02T04:32:07.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw=', 'Content-Length': '0'} 2022-09-02T04:32:07.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016679e43fff0e449-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.171 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.172 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.173 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.174 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.176 DEBUG:boto:Method: GET 2022-09-02T04:32:07.177 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.178 DEBUG:boto:Data: 2022-09-02T04:32:07.179 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.181 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.181 DEBUG:boto:Params: {} 2022-09-02T04:32:07.182 DEBUG:boto:Token: None 2022-09-02T04:32:07.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw= 2022-09-02T04:32:07.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw=', 'Content-Length': '0'} 2022-09-02T04:32:07.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca3792d2e435c9e7-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.188 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.189 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.189 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.190 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.191 DEBUG:boto:Method: GET 2022-09-02T04:32:07.191 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.192 DEBUG:boto:Data: 2022-09-02T04:32:07.193 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.195 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.195 DEBUG:boto:Params: {} 2022-09-02T04:32:07.196 DEBUG:boto:Token: None 2022-09-02T04:32:07.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000929f21f55062d367-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.202 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.203 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.203 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.204 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.205 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.205 DEBUG:boto:Method: GET 2022-09-02T04:32:07.206 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:07.206 DEBUG:boto:Data: 2022-09-02T04:32:07.207 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.208 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.209 DEBUG:boto:Params: {} 2022-09-02T04:32:07.209 DEBUG:boto:Token: None 2022-09-02T04:32:07.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab42ca17aebb2e26-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.254 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:07.256 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.256 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.257 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.257 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.258 DEBUG:boto:Method: GET 2022-09-02T04:32:07.259 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.259 DEBUG:boto:Data: 2022-09-02T04:32:07.260 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.261 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.261 DEBUG:boto:Params: {} 2022-09-02T04:32:07.262 DEBUG:boto:Token: None 2022-09-02T04:32:07.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU= 2022-09-02T04:32:07.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU=', 'Content-Length': '0'} 2022-09-02T04:32:07.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c911bdcc335ece6-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.267 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.268 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.268 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.269 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.270 DEBUG:boto:Method: GET 2022-09-02T04:32:07.270 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.271 DEBUG:boto:Data: 2022-09-02T04:32:07.271 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.272 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.273 DEBUG:boto:Params: {} 2022-09-02T04:32:07.274 DEBUG:boto:Token: None 2022-09-02T04:32:07.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU= 2022-09-02T04:32:07.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU=', 'Content-Length': '0'} 2022-09-02T04:32:07.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b568f2b324902c4-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.279 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.279 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.280 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.280 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.281 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.281 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.282 DEBUG:boto:Data: 2022-09-02T04:32:07.282 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.283 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.283 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.284 DEBUG:boto:Params: {} 2022-09-02T04:32:07.284 DEBUG:boto:Token: None 2022-09-02T04:32:07.285 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+DDRCI/FUhGnjdBH/LxEI4nRBTA= 2022-09-02T04:32:07.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+DDRCI/FUhGnjdBH/LxEI4nRBTA=', 'Content-Length': '0'} 2022-09-02T04:32:07.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8813e933864ffb4-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.289 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.289 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.290 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.290 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.291 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.291 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.292 DEBUG:boto:Data: 2022-09-02T04:32:07.292 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.293 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.293 DEBUG:boto:Params: {} 2022-09-02T04:32:07.294 DEBUG:boto:Token: None 2022-09-02T04:32:07.295 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+DDRCI/FUhGnjdBH/LxEI4nRBTA= 2022-09-02T04:32:07.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+DDRCI/FUhGnjdBH/LxEI4nRBTA=', 'Content-Length': '0'} 2022-09-02T04:32:07.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f5b71af9b9db7c1-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:07.299 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.299 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.300 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.300 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.301 DEBUG:boto:Method: GET 2022-09-02T04:32:07.302 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.302 DEBUG:boto:Data: 2022-09-02T04:32:07.303 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.304 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.304 DEBUG:boto:Params: {} 2022-09-02T04:32:07.305 DEBUG:boto:Token: None 2022-09-02T04:32:07.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU= 2022-09-02T04:32:07.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU=', 'Content-Length': '0'} 2022-09-02T04:32:07.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000433c94f3efbcf472-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.350 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.351 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.351 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.352 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.352 DEBUG:boto:Method: GET 2022-09-02T04:32:07.353 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.354 DEBUG:boto:Data: 2022-09-02T04:32:07.354 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.355 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.356 DEBUG:boto:Params: {} 2022-09-02T04:32:07.357 DEBUG:boto:Token: None 2022-09-02T04:32:07.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU= 2022-09-02T04:32:07.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU=', 'Content-Length': '0'} 2022-09-02T04:32:07.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d3d6a1a410d4f56-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.370 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.370 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.371 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.371 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.372 DEBUG:boto:Method: GET 2022-09-02T04:32:07.373 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.373 DEBUG:boto:Data: 2022-09-02T04:32:07.374 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.374 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.375 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.375 DEBUG:boto:Params: {} 2022-09-02T04:32:07.376 DEBUG:boto:Token: None 2022-09-02T04:32:07.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000390be400131ca5cf-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.381 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.382 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.382 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.383 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.383 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.384 DEBUG:boto:Method: GET 2022-09-02T04:32:07.384 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.385 DEBUG:boto:Data: 2022-09-02T04:32:07.385 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.387 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.387 DEBUG:boto:Params: {} 2022-09-02T04:32:07.388 DEBUG:boto:Token: None 2022-09-02T04:32:07.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f1ab605a904ca3-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.393 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.394 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.394 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.395 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.395 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.396 DEBUG:boto:Data: 2022-09-02T04:32:07.396 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.397 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.398 DEBUG:boto:Params: {} 2022-09-02T04:32:07.398 DEBUG:boto:Token: None 2022-09-02T04:32:07.399 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.400 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DZajaKok+TnEp+8FzYY36bUwTfU= 2022-09-02T04:32:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DZajaKok+TnEp+8FzYY36bUwTfU=', 'Content-Length': '0'} 2022-09-02T04:32:07.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7766b9d61502186-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.403 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.404 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.404 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.405 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.405 DEBUG:boto:Data: 2022-09-02T04:32:07.406 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.407 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.407 DEBUG:boto:Params: {} 2022-09-02T04:32:07.408 DEBUG:boto:Token: None 2022-09-02T04:32:07.408 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DZajaKok+TnEp+8FzYY36bUwTfU= 2022-09-02T04:32:07.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DZajaKok+TnEp+8FzYY36bUwTfU=', 'Content-Length': '0'} 2022-09-02T04:32:07.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9967308625e09a2-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:07.413 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.414 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.414 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.415 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.415 DEBUG:boto:Method: GET 2022-09-02T04:32:07.416 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.416 DEBUG:boto:Data: 2022-09-02T04:32:07.417 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.418 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.418 DEBUG:boto:Params: {} 2022-09-02T04:32:07.419 DEBUG:boto:Token: None 2022-09-02T04:32:07.420 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU= 2022-09-02T04:32:07.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU=', 'Content-Length': '0'} 2022-09-02T04:32:07.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4939e1efc98fc7f-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.423 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.424 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:07.424 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.425 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.426 DEBUG:boto:Method: GET 2022-09-02T04:32:07.426 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.427 DEBUG:boto:Data: 2022-09-02T04:32:07.427 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.428 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.429 DEBUG:boto:Params: {} 2022-09-02T04:32:07.429 DEBUG:boto:Token: None 2022-09-02T04:32:07.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:07.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU= 2022-09-02T04:32:07.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gfQK1S70vozblfHchwZosz54XIU=', 'Content-Length': '0'} 2022-09-02T04:32:07.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ed4dcee9696edd5-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.434 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.434 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.435 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.436 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.436 DEBUG:boto:Method: GET 2022-09-02T04:32:07.437 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.437 DEBUG:boto:Data: 2022-09-02T04:32:07.438 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.438 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.439 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.439 DEBUG:boto:Params: {} 2022-09-02T04:32:07.440 DEBUG:boto:Token: None 2022-09-02T04:32:07.440 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.441 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8482dd7746e51b-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.486 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.487 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.488 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.488 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.489 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.489 DEBUG:boto:Method: GET 2022-09-02T04:32:07.490 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:07.490 DEBUG:boto:Data: 2022-09-02T04:32:07.491 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.492 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.492 DEBUG:boto:Params: {} 2022-09-02T04:32:07.493 DEBUG:boto:Token: None 2022-09-02T04:32:07.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ade3042772bf0b-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.498 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:07.499 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.500 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.500 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.501 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.501 DEBUG:boto:Method: GET 2022-09-02T04:32:07.502 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.502 DEBUG:boto:Data: 2022-09-02T04:32:07.503 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.503 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.504 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.504 DEBUG:boto:Params: {} 2022-09-02T04:32:07.505 DEBUG:boto:Token: None 2022-09-02T04:32:07.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto= 2022-09-02T04:32:07.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto=', 'Content-Length': '0'} 2022-09-02T04:32:07.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007caf2718aba4e05d-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.510 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.510 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.511 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.511 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.512 DEBUG:boto:Method: GET 2022-09-02T04:32:07.512 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.513 DEBUG:boto:Data: 2022-09-02T04:32:07.513 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.514 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.515 DEBUG:boto:Params: {} 2022-09-02T04:32:07.516 DEBUG:boto:Token: None 2022-09-02T04:32:07.516 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto= 2022-09-02T04:32:07.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto=', 'Content-Length': '0'} 2022-09-02T04:32:07.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000164fd3c04c6f0fae-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.521 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.522 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.523 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.524 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.525 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.526 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.527 DEBUG:boto:Data: 2022-09-02T04:32:07.528 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.530 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.531 DEBUG:boto:Params: {} 2022-09-02T04:32:07.532 DEBUG:boto:Token: None 2022-09-02T04:32:07.532 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XHqW4ZvnME2n/rLluiQYmdW1S58= 2022-09-02T04:32:07.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XHqW4ZvnME2n/rLluiQYmdW1S58=', 'Content-Length': '0'} 2022-09-02T04:32:07.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021a175b357e74488-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.539 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.539 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.540 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.541 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.542 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.543 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.544 DEBUG:boto:Data: 2022-09-02T04:32:07.545 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.547 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.548 DEBUG:boto:Params: {} 2022-09-02T04:32:07.549 DEBUG:boto:Token: None 2022-09-02T04:32:07.550 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XHqW4ZvnME2n/rLluiQYmdW1S58= 2022-09-02T04:32:07.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XHqW4ZvnME2n/rLluiQYmdW1S58=', 'Content-Length': '0'} 2022-09-02T04:32:07.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a6a2f82f902219c-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:07.554 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.555 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.556 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.556 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.557 DEBUG:boto:Method: GET 2022-09-02T04:32:07.557 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.558 DEBUG:boto:Data: 2022-09-02T04:32:07.558 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.559 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.560 DEBUG:boto:Params: {} 2022-09-02T04:32:07.561 DEBUG:boto:Token: None 2022-09-02T04:32:07.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto= 2022-09-02T04:32:07.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto=', 'Content-Length': '0'} 2022-09-02T04:32:07.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f224174c765eac1f-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.565 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.566 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.566 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.567 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.568 DEBUG:boto:Method: GET 2022-09-02T04:32:07.568 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.569 DEBUG:boto:Data: 2022-09-02T04:32:07.569 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.570 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.571 DEBUG:boto:Params: {} 2022-09-02T04:32:07.571 DEBUG:boto:Token: None 2022-09-02T04:32:07.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto= 2022-09-02T04:32:07.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto=', 'Content-Length': '0'} 2022-09-02T04:32:07.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000206f0ce733b49456-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.576 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.577 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.577 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.578 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.578 DEBUG:boto:Method: GET 2022-09-02T04:32:07.579 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.579 DEBUG:boto:Data: 2022-09-02T04:32:07.580 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.581 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.582 DEBUG:boto:Params: {} 2022-09-02T04:32:07.582 DEBUG:boto:Token: None 2022-09-02T04:32:07.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016af975fe9383aab-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.587 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.588 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.589 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.589 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.590 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.590 DEBUG:boto:Method: GET 2022-09-02T04:32:07.591 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.591 DEBUG:boto:Data: 2022-09-02T04:32:07.592 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.593 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.594 DEBUG:boto:Params: {} 2022-09-02T04:32:07.594 DEBUG:boto:Token: None 2022-09-02T04:32:07.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65016c92ed27af1-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.599 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.600 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.601 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.601 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.602 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.602 DEBUG:boto:Data: 2022-09-02T04:32:07.603 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.604 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.604 DEBUG:boto:Params: {} 2022-09-02T04:32:07.605 DEBUG:boto:Token: None 2022-09-02T04:32:07.605 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:00FAow2pNSxjgbWlJTfaPdkWuTQ= 2022-09-02T04:32:07.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:00FAow2pNSxjgbWlJTfaPdkWuTQ=', 'Content-Length': '0'} 2022-09-02T04:32:07.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076d1b40427be18ec-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.610 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.610 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.611 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.611 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.612 DEBUG:boto:Data: 2022-09-02T04:32:07.612 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.613 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.614 DEBUG:boto:Params: {} 2022-09-02T04:32:07.614 DEBUG:boto:Token: None 2022-09-02T04:32:07.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:00FAow2pNSxjgbWlJTfaPdkWuTQ= 2022-09-02T04:32:07.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:00FAow2pNSxjgbWlJTfaPdkWuTQ=', 'Content-Length': '0'} 2022-09-02T04:32:07.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000315cbd96b9d386ee-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:07.620 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.620 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.621 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.621 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.622 DEBUG:boto:Method: GET 2022-09-02T04:32:07.622 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.623 DEBUG:boto:Data: 2022-09-02T04:32:07.623 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.624 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.625 DEBUG:boto:Params: {} 2022-09-02T04:32:07.625 DEBUG:boto:Token: None 2022-09-02T04:32:07.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto= 2022-09-02T04:32:07.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto=', 'Content-Length': '0'} 2022-09-02T04:32:07.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e9b928354f2fbbd-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.630 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.631 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:07.631 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.632 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.633 DEBUG:boto:Method: GET 2022-09-02T04:32:07.634 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.634 DEBUG:boto:Data: 2022-09-02T04:32:07.635 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.636 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.637 DEBUG:boto:Params: {} 2022-09-02T04:32:07.637 DEBUG:boto:Token: None 2022-09-02T04:32:07.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:07.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto= 2022-09-02T04:32:07.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NiH5e7vxoTsIqFrGuChyJtdAvto=', 'Content-Length': '0'} 2022-09-02T04:32:07.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a9c4a974b5983a6-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.682 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.683 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.683 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.684 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.684 DEBUG:boto:Method: GET 2022-09-02T04:32:07.685 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.685 DEBUG:boto:Data: 2022-09-02T04:32:07.686 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.687 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.687 DEBUG:boto:Params: {} 2022-09-02T04:32:07.688 DEBUG:boto:Token: None 2022-09-02T04:32:07.688 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072868da0755fcf9c-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.693 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.694 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.695 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.695 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.696 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.696 DEBUG:boto:Method: GET 2022-09-02T04:32:07.697 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:07.697 DEBUG:boto:Data: 2022-09-02T04:32:07.698 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.699 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.700 DEBUG:boto:Params: {} 2022-09-02T04:32:07.700 DEBUG:boto:Token: None 2022-09-02T04:32:07.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007932d3c50b10ca81-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.706 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:07.707 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.708 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.708 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.709 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.709 DEBUG:boto:Method: GET 2022-09-02T04:32:07.710 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.710 DEBUG:boto:Data: 2022-09-02T04:32:07.711 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.712 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.712 DEBUG:boto:Params: {} 2022-09-02T04:32:07.713 DEBUG:boto:Token: None 2022-09-02T04:32:07.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk= 2022-09-02T04:32:07.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk=', 'Content-Length': '0'} 2022-09-02T04:32:07.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8ad54367435fc0e-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.717 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.718 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.718 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.719 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.720 DEBUG:boto:Method: GET 2022-09-02T04:32:07.720 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.721 DEBUG:boto:Data: 2022-09-02T04:32:07.721 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.722 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.723 DEBUG:boto:Params: {} 2022-09-02T04:32:07.723 DEBUG:boto:Token: None 2022-09-02T04:32:07.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk= 2022-09-02T04:32:07.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk=', 'Content-Length': '0'} 2022-09-02T04:32:07.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062a0b9bcc182d365-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.728 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.729 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.729 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.730 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.730 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.731 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.731 DEBUG:boto:Data: 2022-09-02T04:32:07.732 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.733 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.733 DEBUG:boto:Params: {} 2022-09-02T04:32:07.734 DEBUG:boto:Token: None 2022-09-02T04:32:07.734 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vBdvan01cnhnUg+rBgUkrYl1az4= 2022-09-02T04:32:07.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vBdvan01cnhnUg+rBgUkrYl1az4=', 'Content-Length': '0'} 2022-09-02T04:32:07.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008aa657a9b2f36d41-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.738 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.739 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.739 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.740 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.740 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.741 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.741 DEBUG:boto:Data: 2022-09-02T04:32:07.742 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.743 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.743 DEBUG:boto:Params: {} 2022-09-02T04:32:07.744 DEBUG:boto:Token: None 2022-09-02T04:32:07.744 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.745 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vBdvan01cnhnUg+rBgUkrYl1az4= 2022-09-02T04:32:07.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vBdvan01cnhnUg+rBgUkrYl1az4=', 'Content-Length': '0'} 2022-09-02T04:32:07.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009edef80d85654c8d-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:07.748 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.749 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.749 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.750 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.750 DEBUG:boto:Method: GET 2022-09-02T04:32:07.751 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.751 DEBUG:boto:Data: 2022-09-02T04:32:07.752 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.753 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.753 DEBUG:boto:Params: {} 2022-09-02T04:32:07.754 DEBUG:boto:Token: None 2022-09-02T04:32:07.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk= 2022-09-02T04:32:07.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk=', 'Content-Length': '0'} 2022-09-02T04:32:07.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc48170d50a66750-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.758 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.759 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.759 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.760 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.760 DEBUG:boto:Method: GET 2022-09-02T04:32:07.761 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.761 DEBUG:boto:Data: 2022-09-02T04:32:07.762 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.763 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.764 DEBUG:boto:Params: {} 2022-09-02T04:32:07.764 DEBUG:boto:Token: None 2022-09-02T04:32:07.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk= 2022-09-02T04:32:07.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk=', 'Content-Length': '0'} 2022-09-02T04:32:07.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa23630869ebccab-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.769 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.769 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.770 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.771 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.771 DEBUG:boto:Method: GET 2022-09-02T04:32:07.772 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.772 DEBUG:boto:Data: 2022-09-02T04:32:07.773 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.774 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.775 DEBUG:boto:Params: {} 2022-09-02T04:32:07.775 DEBUG:boto:Token: None 2022-09-02T04:32:07.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872aa973670d6c13-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.780 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.781 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.782 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.782 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.783 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.783 DEBUG:boto:Method: GET 2022-09-02T04:32:07.784 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.784 DEBUG:boto:Data: 2022-09-02T04:32:07.785 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.786 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.787 DEBUG:boto:Params: {} 2022-09-02T04:32:07.787 DEBUG:boto:Token: None 2022-09-02T04:32:07.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093c47ad3c1a86065-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.792 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.793 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.794 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.794 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.795 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.795 DEBUG:boto:Data: 2022-09-02T04:32:07.796 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.797 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.798 DEBUG:boto:Params: {} 2022-09-02T04:32:07.798 DEBUG:boto:Token: None 2022-09-02T04:32:07.799 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7wigreHPklxVq/22X4tJP15AUtM= 2022-09-02T04:32:07.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7wigreHPklxVq/22X4tJP15AUtM=', 'Content-Length': '0'} 2022-09-02T04:32:07.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005335afa706689738-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.803 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.803 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.804 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.804 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.805 DEBUG:boto:Data: 2022-09-02T04:32:07.805 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.806 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.807 DEBUG:boto:Params: {} 2022-09-02T04:32:07.807 DEBUG:boto:Token: None 2022-09-02T04:32:07.808 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7wigreHPklxVq/22X4tJP15AUtM= 2022-09-02T04:32:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7wigreHPklxVq/22X4tJP15AUtM=', 'Content-Length': '0'} 2022-09-02T04:32:07.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b998e24361fcbd93-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:07.813 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.814 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.814 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.815 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.815 DEBUG:boto:Method: GET 2022-09-02T04:32:07.816 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.816 DEBUG:boto:Data: 2022-09-02T04:32:07.817 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.818 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.818 DEBUG:boto:Params: {} 2022-09-02T04:32:07.819 DEBUG:boto:Token: None 2022-09-02T04:32:07.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk= 2022-09-02T04:32:07.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk=', 'Content-Length': '0'} 2022-09-02T04:32:07.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f33bfae680be45c2-0063118747-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.866 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.867 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:07.868 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.868 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.869 DEBUG:boto:Method: GET 2022-09-02T04:32:07.869 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.870 DEBUG:boto:Data: 2022-09-02T04:32:07.871 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.872 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.872 DEBUG:boto:Params: {} 2022-09-02T04:32:07.873 DEBUG:boto:Token: None 2022-09-02T04:32:07.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:07.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk= 2022-09-02T04:32:07.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nuLdnXFDGLk71CnBN12/4EIgpWk=', 'Content-Length': '0'} 2022-09-02T04:32:07.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0ba49c1099a6af7-0063118747-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.878 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.879 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.879 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.880 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.880 DEBUG:boto:Method: GET 2022-09-02T04:32:07.881 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.882 DEBUG:boto:Data: 2022-09-02T04:32:07.882 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:07.883 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.884 DEBUG:boto:Params: {} 2022-09-02T04:32:07.885 DEBUG:boto:Token: None 2022-09-02T04:32:07.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003958cf02859adc57-0063118747-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.892 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.894 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.895 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.896 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.897 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.897 DEBUG:boto:Method: GET 2022-09-02T04:32:07.898 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:07.899 DEBUG:boto:Data: 2022-09-02T04:32:07.899 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.901 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.901 DEBUG:boto:Params: {} 2022-09-02T04:32:07.902 DEBUG:boto:Token: None 2022-09-02T04:32:07.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs= 2022-09-02T04:32:07.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ds8jkb1xdq9KHlfSIPDOTshYZOs=', 'Content-Length': '0'} 2022-09-02T04:32:07.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbf41196b9b3cdc-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.907 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.908 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-17 zones={test-zone2, test-zone1} 2022-09-02T04:32:07.909 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-17 zones={test-zone3, test-zone1} 2022-09-02T04:32:07.909 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.910 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.910 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.911 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:32:07.911 DEBUG:boto:Data: 2022-09-02T04:32:07.912 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:07.913 DEBUG:boto:Port: 8001 2022-09-02T04:32:07.913 DEBUG:boto:Params: {} 2022-09-02T04:32:07.914 DEBUG:boto:Token: None 2022-09-02T04:32:07.915 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IGFWp4KoM5BsG7n4JDzCCHaon68= 2022-09-02T04:32:07.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IGFWp4KoM5BsG7n4JDzCCHaon68=', 'Content-Length': '0'} 2022-09-02T04:32:07.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d28e9f6ad8d13e65-0063118747-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.919 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.920 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.920 DEBUG:boto:Method: HEAD 2022-09-02T04:32:07.921 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:32:07.921 DEBUG:boto:Data: 2022-09-02T04:32:07.922 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.923 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.923 DEBUG:boto:Params: {} 2022-09-02T04:32:07.924 DEBUG:boto:Token: None 2022-09-02T04:32:07.925 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/ 2022-09-02T04:32:07.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IGFWp4KoM5BsG7n4JDzCCHaon68= 2022-09-02T04:32:07.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IGFWp4KoM5BsG7n4JDzCCHaon68=', 'Content-Length': '0'} 2022-09-02T04:32:07.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025c1bba70d075d4c-0063118747-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.929 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:07.930 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.930 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.931 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.931 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.932 DEBUG:boto:Method: GET 2022-09-02T04:32:07.932 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.933 DEBUG:boto:Data: 2022-09-02T04:32:07.933 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:07.934 DEBUG:boto:Port: 8001 2022-09-02T04:32:07.935 DEBUG:boto:Params: {} 2022-09-02T04:32:07.935 DEBUG:boto:Token: None 2022-09-02T04:32:07.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/?versions 2022-09-02T04:32:07.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak= 2022-09-02T04:32:07.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak=', 'Content-Length': '0'} 2022-09-02T04:32:07.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d69ec1cac7125250-0063118747-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.941 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.942 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:07.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:07.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:07.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:07.944 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:07.944 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.945 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.945 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.946 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.946 DEBUG:boto:Method: GET 2022-09-02T04:32:07.947 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.947 DEBUG:boto:Data: 2022-09-02T04:32:07.948 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.949 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.949 DEBUG:boto:Params: {} 2022-09-02T04:32:07.950 DEBUG:boto:Token: None 2022-09-02T04:32:07.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/?versions 2022-09-02T04:32:07.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak= 2022-09-02T04:32:07.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak=', 'Content-Length': '0'} 2022-09-02T04:32:07.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003770d0bf416f1205-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.955 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.956 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:07.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:07.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:07.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:07.958 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.959 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.959 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.960 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.961 DEBUG:boto:Method: GET 2022-09-02T04:32:07.961 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.962 DEBUG:boto:Data: 2022-09-02T04:32:07.962 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:07.963 DEBUG:boto:Port: 8001 2022-09-02T04:32:07.964 DEBUG:boto:Params: {} 2022-09-02T04:32:07.964 DEBUG:boto:Token: None 2022-09-02T04:32:07.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/?versions 2022-09-02T04:32:07.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak= 2022-09-02T04:32:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak=', 'Content-Length': '0'} 2022-09-02T04:32:07.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ffcc556580e1a07-0063118747-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.969 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.971 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:07.971 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:07.972 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.972 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.973 DEBUG:boto:Method: GET 2022-09-02T04:32:07.973 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:07.974 DEBUG:boto:Data: 2022-09-02T04:32:07.974 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.975 DEBUG:boto:Port: 8000 2022-09-02T04:32:07.976 DEBUG:boto:Params: {} 2022-09-02T04:32:07.976 DEBUG:boto:Token: None 2022-09-02T04:32:07.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/?versions 2022-09-02T04:32:07.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak= 2022-09-02T04:32:07.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jWbjvNtVtFZ1JUqASXuHiFxu+Ak=', 'Content-Length': '0'} 2022-09-02T04:32:07.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f023abeb2b4cbb62-0063118747-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.982 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:07.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:07.983 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.984 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.984 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.985 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.985 DEBUG:boto:Method: GET 2022-09-02T04:32:07.986 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.987 DEBUG:boto:Data: 2022-09-02T04:32:07.987 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:07.988 DEBUG:boto:Port: 8001 2022-09-02T04:32:07.989 DEBUG:boto:Params: {} 2022-09-02T04:32:07.989 DEBUG:boto:Token: None 2022-09-02T04:32:07.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:07 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw= 2022-09-02T04:32:07.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6aRT3ScOVPhRHOIPEsRmFcf45sw=', 'Content-Length': '0'} 2022-09-02T04:32:07.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004d47113d345e887e-0063118747-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:07.994 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:07.995 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:07.996 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.996 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.997 DEBUG:boto:Method: GET 2022-09-02T04:32:07.997 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:07.998 DEBUG:boto:Data: 2022-09-02T04:32:07.998 DEBUG:boto:Headers: {} 2022-09-02T04:32:07.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:07.999 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.000 DEBUG:boto:Params: {} 2022-09-02T04:32:08.001 DEBUG:boto:Token: None 2022-09-02T04:32:08.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050a4f7c1ba78fbbf-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.005 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.006 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.006 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.007 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.007 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.008 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.008 DEBUG:boto:Data: 2022-09-02T04:32:08.009 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.010 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.011 DEBUG:boto:Params: {} 2022-09-02T04:32:08.011 DEBUG:boto:Token: None 2022-09-02T04:32:08.012 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU= 2022-09-02T04:32:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU=', 'Content-Length': '0'} 2022-09-02T04:32:08.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082fd9d37a4c52a72-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.016 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.016 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.017 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.018 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.018 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.019 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.020 DEBUG:boto:Data: 2022-09-02T04:32:08.020 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.022 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.022 DEBUG:boto:Params: {} 2022-09-02T04:32:08.023 DEBUG:boto:Token: None 2022-09-02T04:32:08.023 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU= 2022-09-02T04:32:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU=', 'Content-Length': '0'} 2022-09-02T04:32:08.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bee4abe03213338-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:08.028 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.028 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.029 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.030 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.030 DEBUG:boto:Method: GET 2022-09-02T04:32:08.031 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.031 DEBUG:boto:Data: 2022-09-02T04:32:08.032 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.033 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.034 DEBUG:boto:Params: {} 2022-09-02T04:32:08.034 DEBUG:boto:Token: None 2022-09-02T04:32:08.035 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.035 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084745977536c14cd-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.039 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.040 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.040 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.041 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.041 DEBUG:boto:Method: GET 2022-09-02T04:32:08.042 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.042 DEBUG:boto:Data: 2022-09-02T04:32:08.043 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.044 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.044 DEBUG:boto:Params: {} 2022-09-02T04:32:08.045 DEBUG:boto:Token: None 2022-09-02T04:32:08.046 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d248a85bf741348-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.049 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.050 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.051 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.051 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.052 DEBUG:boto:Method: GET 2022-09-02T04:32:08.052 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.053 DEBUG:boto:Data: 2022-09-02T04:32:08.053 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.054 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.055 DEBUG:boto:Params: {} 2022-09-02T04:32:08.055 DEBUG:boto:Token: None 2022-09-02T04:32:08.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038343edc7b45be2f-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.106 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.107 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.108 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.108 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.109 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.109 DEBUG:boto:Method: GET 2022-09-02T04:32:08.110 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.110 DEBUG:boto:Data: 2022-09-02T04:32:08.111 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.112 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.112 DEBUG:boto:Params: {} 2022-09-02T04:32:08.113 DEBUG:boto:Token: None 2022-09-02T04:32:08.114 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037556d47b59e2ade-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.118 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.119 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.120 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.120 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.121 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:32:08.121 DEBUG:boto:Data: 2022-09-02T04:32:08.122 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.123 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.123 DEBUG:boto:Params: {} 2022-09-02T04:32:08.124 DEBUG:boto:Token: None 2022-09-02T04:32:08.124 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy 2022-09-02T04:32:08.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wbw+l3cXbbFAZkVtndIsq95yWzU= 2022-09-02T04:32:08.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wbw+l3cXbbFAZkVtndIsq95yWzU=', 'Content-Length': '0'} 2022-09-02T04:32:08.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea849edea7a58a81-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.129 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.129 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.130 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.131 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:32:08.132 DEBUG:boto:Data: 2022-09-02T04:32:08.133 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.135 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.136 DEBUG:boto:Params: {} 2022-09-02T04:32:08.136 DEBUG:boto:Token: None 2022-09-02T04:32:08.137 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy 2022-09-02T04:32:08.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wbw+l3cXbbFAZkVtndIsq95yWzU= 2022-09-02T04:32:08.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wbw+l3cXbbFAZkVtndIsq95yWzU=', 'Content-Length': '0'} 2022-09-02T04:32:08.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ab02ef19ef42773-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:08.142 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.143 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.143 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.144 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.144 DEBUG:boto:Method: GET 2022-09-02T04:32:08.145 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.145 DEBUG:boto:Data: 2022-09-02T04:32:08.146 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.148 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.149 DEBUG:boto:Params: {} 2022-09-02T04:32:08.150 DEBUG:boto:Token: None 2022-09-02T04:32:08.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020db13291a2229d9-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.156 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.156 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.157 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.157 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.158 DEBUG:boto:Method: GET 2022-09-02T04:32:08.158 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.159 DEBUG:boto:Data: 2022-09-02T04:32:08.160 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.162 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.163 DEBUG:boto:Params: {} 2022-09-02T04:32:08.164 DEBUG:boto:Token: None 2022-09-02T04:32:08.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005db5b7169339185e-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.170 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.171 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.171 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.172 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.172 DEBUG:boto:Method: GET 2022-09-02T04:32:08.173 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.173 DEBUG:boto:Data: 2022-09-02T04:32:08.174 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.175 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.175 DEBUG:boto:Params: {} 2022-09-02T04:32:08.176 DEBUG:boto:Token: None 2022-09-02T04:32:08.176 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046de00ad1686084b-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.181 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.182 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.183 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.183 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.184 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.184 DEBUG:boto:Method: GET 2022-09-02T04:32:08.185 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.186 DEBUG:boto:Data: 2022-09-02T04:32:08.187 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.188 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.188 DEBUG:boto:Params: {} 2022-09-02T04:32:08.189 DEBUG:boto:Token: None 2022-09-02T04:32:08.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009692463a603aa266-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.194 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:08.195 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.196 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.197 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.197 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.198 DEBUG:boto:Method: GET 2022-09-02T04:32:08.198 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.199 DEBUG:boto:Data: 2022-09-02T04:32:08.199 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.200 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.201 DEBUG:boto:Params: {} 2022-09-02T04:32:08.202 DEBUG:boto:Token: None 2022-09-02T04:32:08.202 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo= 2022-09-02T04:32:08.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo=', 'Content-Length': '0'} 2022-09-02T04:32:08.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e820444d9c5f27b-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.207 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.207 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.208 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.208 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.209 DEBUG:boto:Method: GET 2022-09-02T04:32:08.209 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.210 DEBUG:boto:Data: 2022-09-02T04:32:08.211 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.213 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.214 DEBUG:boto:Params: {} 2022-09-02T04:32:08.215 DEBUG:boto:Token: None 2022-09-02T04:32:08.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo= 2022-09-02T04:32:08.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo=', 'Content-Length': '0'} 2022-09-02T04:32:08.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8cb473a703957be-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.221 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.221 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.222 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.222 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.223 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.223 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.224 DEBUG:boto:Data: 2022-09-02T04:32:08.225 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.226 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.226 DEBUG:boto:Params: {} 2022-09-02T04:32:08.227 DEBUG:boto:Token: None 2022-09-02T04:32:08.227 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NKt3GMIV2LgnRyWnCH9JLt/nH08= 2022-09-02T04:32:08.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NKt3GMIV2LgnRyWnCH9JLt/nH08=', 'Content-Length': '0'} 2022-09-02T04:32:08.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038f6f3d4d629384c-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.231 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.232 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.232 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.233 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.233 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.234 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.234 DEBUG:boto:Data: 2022-09-02T04:32:08.235 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.236 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.236 DEBUG:boto:Params: {} 2022-09-02T04:32:08.237 DEBUG:boto:Token: None 2022-09-02T04:32:08.238 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NKt3GMIV2LgnRyWnCH9JLt/nH08= 2022-09-02T04:32:08.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NKt3GMIV2LgnRyWnCH9JLt/nH08=', 'Content-Length': '0'} 2022-09-02T04:32:08.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe3774865cac5946-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:08.245 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.245 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.246 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.246 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.247 DEBUG:boto:Method: GET 2022-09-02T04:32:08.248 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.248 DEBUG:boto:Data: 2022-09-02T04:32:08.249 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.250 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.250 DEBUG:boto:Params: {} 2022-09-02T04:32:08.251 DEBUG:boto:Token: None 2022-09-02T04:32:08.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo= 2022-09-02T04:32:08.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo=', 'Content-Length': '0'} 2022-09-02T04:32:08.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a345c8a1e7d957d8-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.256 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.256 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.257 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.257 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.258 DEBUG:boto:Method: GET 2022-09-02T04:32:08.258 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.259 DEBUG:boto:Data: 2022-09-02T04:32:08.259 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.260 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.261 DEBUG:boto:Params: {} 2022-09-02T04:32:08.261 DEBUG:boto:Token: None 2022-09-02T04:32:08.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo= 2022-09-02T04:32:08.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo=', 'Content-Length': '0'} 2022-09-02T04:32:08.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000554ff638141d324a-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.266 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.267 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.267 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.268 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.268 DEBUG:boto:Method: GET 2022-09-02T04:32:08.269 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.269 DEBUG:boto:Data: 2022-09-02T04:32:08.270 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.271 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.271 DEBUG:boto:Params: {} 2022-09-02T04:32:08.272 DEBUG:boto:Token: None 2022-09-02T04:32:08.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd59892e0d1848f3-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.318 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.319 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.320 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.320 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.321 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.321 DEBUG:boto:Method: GET 2022-09-02T04:32:08.322 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.323 DEBUG:boto:Data: 2022-09-02T04:32:08.323 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.325 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.326 DEBUG:boto:Params: {} 2022-09-02T04:32:08.327 DEBUG:boto:Token: None 2022-09-02T04:32:08.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a06104020141dad-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.334 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.335 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.335 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.336 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.336 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.337 DEBUG:boto:Data: 2022-09-02T04:32:08.337 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.338 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.339 DEBUG:boto:Params: {} 2022-09-02T04:32:08.340 DEBUG:boto:Token: None 2022-09-02T04:32:08.340 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dvvBphf4t45/Ro7PIE5Qj7muBK8= 2022-09-02T04:32:08.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dvvBphf4t45/Ro7PIE5Qj7muBK8=', 'Content-Length': '0'} 2022-09-02T04:32:08.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a30fb642acd16b2b-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.345 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.345 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.346 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.347 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.348 DEBUG:boto:Data: 2022-09-02T04:32:08.349 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.351 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.352 DEBUG:boto:Params: {} 2022-09-02T04:32:08.353 DEBUG:boto:Token: None 2022-09-02T04:32:08.353 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dvvBphf4t45/Ro7PIE5Qj7muBK8= 2022-09-02T04:32:08.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dvvBphf4t45/Ro7PIE5Qj7muBK8=', 'Content-Length': '0'} 2022-09-02T04:32:08.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045266602234dd01a-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:08.359 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.359 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.360 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.360 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.361 DEBUG:boto:Method: GET 2022-09-02T04:32:08.361 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.362 DEBUG:boto:Data: 2022-09-02T04:32:08.363 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.364 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.364 DEBUG:boto:Params: {} 2022-09-02T04:32:08.365 DEBUG:boto:Token: None 2022-09-02T04:32:08.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo= 2022-09-02T04:32:08.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo=', 'Content-Length': '0'} 2022-09-02T04:32:08.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084432da49bc8b242-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.370 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.371 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:08.371 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.372 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.373 DEBUG:boto:Method: GET 2022-09-02T04:32:08.373 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.374 DEBUG:boto:Data: 2022-09-02T04:32:08.374 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.375 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.376 DEBUG:boto:Params: {} 2022-09-02T04:32:08.377 DEBUG:boto:Token: None 2022-09-02T04:32:08.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:08.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo= 2022-09-02T04:32:08.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m7NeBGwhZ10N4F3UBuk5eCWUjZo=', 'Content-Length': '0'} 2022-09-02T04:32:08.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064edf9aff390eca5-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.382 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.382 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.383 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.383 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.384 DEBUG:boto:Method: GET 2022-09-02T04:32:08.384 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.385 DEBUG:boto:Data: 2022-09-02T04:32:08.386 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.387 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.387 DEBUG:boto:Params: {} 2022-09-02T04:32:08.388 DEBUG:boto:Token: None 2022-09-02T04:32:08.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025fa4de8903928f5-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.393 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.394 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.395 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.395 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.396 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.396 DEBUG:boto:Method: GET 2022-09-02T04:32:08.397 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:08.397 DEBUG:boto:Data: 2022-09-02T04:32:08.398 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.399 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.400 DEBUG:boto:Params: {} 2022-09-02T04:32:08.400 DEBUG:boto:Token: None 2022-09-02T04:32:08.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c8755713fd98f49-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.405 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:08.407 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.407 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.408 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.408 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.409 DEBUG:boto:Method: GET 2022-09-02T04:32:08.409 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.410 DEBUG:boto:Data: 2022-09-02T04:32:08.410 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.411 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.412 DEBUG:boto:Params: {} 2022-09-02T04:32:08.412 DEBUG:boto:Token: None 2022-09-02T04:32:08.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM= 2022-09-02T04:32:08.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM=', 'Content-Length': '0'} 2022-09-02T04:32:08.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec58d0f3c49cc573-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.417 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.418 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.418 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.419 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.420 DEBUG:boto:Method: GET 2022-09-02T04:32:08.420 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.421 DEBUG:boto:Data: 2022-09-02T04:32:08.421 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.422 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.423 DEBUG:boto:Params: {} 2022-09-02T04:32:08.423 DEBUG:boto:Token: None 2022-09-02T04:32:08.424 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM= 2022-09-02T04:32:08.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM=', 'Content-Length': '0'} 2022-09-02T04:32:08.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4eba1db3ff290a8-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.428 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.429 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.429 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.430 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.430 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.431 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.431 DEBUG:boto:Data: 2022-09-02T04:32:08.432 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.433 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.433 DEBUG:boto:Params: {} 2022-09-02T04:32:08.434 DEBUG:boto:Token: None 2022-09-02T04:32:08.434 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qP+xdUaPKyTJ1abTb5zfB/kEbb4= 2022-09-02T04:32:08.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qP+xdUaPKyTJ1abTb5zfB/kEbb4=', 'Content-Length': '0'} 2022-09-02T04:32:08.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ead28f73a064839-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.439 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.439 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.440 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.440 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.441 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.441 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.442 DEBUG:boto:Data: 2022-09-02T04:32:08.442 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.443 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.444 DEBUG:boto:Params: {} 2022-09-02T04:32:08.444 DEBUG:boto:Token: None 2022-09-02T04:32:08.445 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qP+xdUaPKyTJ1abTb5zfB/kEbb4= 2022-09-02T04:32:08.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qP+xdUaPKyTJ1abTb5zfB/kEbb4=', 'Content-Length': '0'} 2022-09-02T04:32:08.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000597cb0472c40ed8e-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:08.449 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.450 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.450 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.451 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.451 DEBUG:boto:Method: GET 2022-09-02T04:32:08.452 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.453 DEBUG:boto:Data: 2022-09-02T04:32:08.453 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.454 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.455 DEBUG:boto:Params: {} 2022-09-02T04:32:08.455 DEBUG:boto:Token: None 2022-09-02T04:32:08.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM= 2022-09-02T04:32:08.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM=', 'Content-Length': '0'} 2022-09-02T04:32:08.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b202ed17f42e94b8-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.460 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.461 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.461 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.462 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.462 DEBUG:boto:Method: GET 2022-09-02T04:32:08.463 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.463 DEBUG:boto:Data: 2022-09-02T04:32:08.464 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.465 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.466 DEBUG:boto:Params: {} 2022-09-02T04:32:08.466 DEBUG:boto:Token: None 2022-09-02T04:32:08.467 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.468 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM= 2022-09-02T04:32:08.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM=', 'Content-Length': '0'} 2022-09-02T04:32:08.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005838f94c280b96b4-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.472 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.472 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.473 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.473 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.474 DEBUG:boto:Method: GET 2022-09-02T04:32:08.475 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.475 DEBUG:boto:Data: 2022-09-02T04:32:08.476 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.477 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.477 DEBUG:boto:Params: {} 2022-09-02T04:32:08.478 DEBUG:boto:Token: None 2022-09-02T04:32:08.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f63d69929adfd50-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.483 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.484 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.484 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.485 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.485 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.486 DEBUG:boto:Method: GET 2022-09-02T04:32:08.486 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.487 DEBUG:boto:Data: 2022-09-02T04:32:08.487 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.488 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.489 DEBUG:boto:Params: {} 2022-09-02T04:32:08.490 DEBUG:boto:Token: None 2022-09-02T04:32:08.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632e1aaac2faa2c2-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.495 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.497 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.497 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.498 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.499 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.500 DEBUG:boto:Data: 2022-09-02T04:32:08.501 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.502 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.502 DEBUG:boto:Params: {} 2022-09-02T04:32:08.503 DEBUG:boto:Token: None 2022-09-02T04:32:08.504 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vXFHezpa0dM3F0KAd1OjpzM/XXs= 2022-09-02T04:32:08.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vXFHezpa0dM3F0KAd1OjpzM/XXs=', 'Content-Length': '0'} 2022-09-02T04:32:08.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009faba287fbcb02a2-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.509 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.509 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.510 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.510 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.511 DEBUG:boto:Data: 2022-09-02T04:32:08.511 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.512 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.513 DEBUG:boto:Params: {} 2022-09-02T04:32:08.514 DEBUG:boto:Token: None 2022-09-02T04:32:08.514 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vXFHezpa0dM3F0KAd1OjpzM/XXs= 2022-09-02T04:32:08.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vXFHezpa0dM3F0KAd1OjpzM/XXs=', 'Content-Length': '0'} 2022-09-02T04:32:08.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000001a237d67b8c32-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:08.520 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.520 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.521 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.521 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.522 DEBUG:boto:Method: GET 2022-09-02T04:32:08.523 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.523 DEBUG:boto:Data: 2022-09-02T04:32:08.524 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.525 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.526 DEBUG:boto:Params: {} 2022-09-02T04:32:08.526 DEBUG:boto:Token: None 2022-09-02T04:32:08.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM= 2022-09-02T04:32:08.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM=', 'Content-Length': '0'} 2022-09-02T04:32:08.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000779007404359dd07-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.531 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.532 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:08.532 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.533 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.533 DEBUG:boto:Method: GET 2022-09-02T04:32:08.534 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.535 DEBUG:boto:Data: 2022-09-02T04:32:08.535 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.536 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.537 DEBUG:boto:Params: {} 2022-09-02T04:32:08.538 DEBUG:boto:Token: None 2022-09-02T04:32:08.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:08.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM= 2022-09-02T04:32:08.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0B4NHHbHaXY29ufd3d28ANCPYM=', 'Content-Length': '0'} 2022-09-02T04:32:08.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082d496299a4cf63c-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.586 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.587 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.588 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.588 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.589 DEBUG:boto:Method: GET 2022-09-02T04:32:08.590 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.590 DEBUG:boto:Data: 2022-09-02T04:32:08.591 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.592 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.592 DEBUG:boto:Params: {} 2022-09-02T04:32:08.593 DEBUG:boto:Token: None 2022-09-02T04:32:08.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a88b4a447f473f5f-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.598 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.599 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.600 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.600 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.601 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.601 DEBUG:boto:Method: GET 2022-09-02T04:32:08.602 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:08.602 DEBUG:boto:Data: 2022-09-02T04:32:08.603 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.604 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.605 DEBUG:boto:Params: {} 2022-09-02T04:32:08.605 DEBUG:boto:Token: None 2022-09-02T04:32:08.606 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8af2cf306acccb-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.612 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:08.614 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.615 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.615 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.616 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.617 DEBUG:boto:Method: GET 2022-09-02T04:32:08.617 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.618 DEBUG:boto:Data: 2022-09-02T04:32:08.618 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.620 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.620 DEBUG:boto:Params: {} 2022-09-02T04:32:08.621 DEBUG:boto:Token: None 2022-09-02T04:32:08.621 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE= 2022-09-02T04:32:08.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE=', 'Content-Length': '0'} 2022-09-02T04:32:08.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea37f40fd6143d5c-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.626 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.627 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.627 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.628 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.629 DEBUG:boto:Method: GET 2022-09-02T04:32:08.629 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.630 DEBUG:boto:Data: 2022-09-02T04:32:08.630 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.632 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.632 DEBUG:boto:Params: {} 2022-09-02T04:32:08.633 DEBUG:boto:Token: None 2022-09-02T04:32:08.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE= 2022-09-02T04:32:08.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE=', 'Content-Length': '0'} 2022-09-02T04:32:08.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e18ba7b5a9341473-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.638 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.639 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.639 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.640 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.640 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.641 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.641 DEBUG:boto:Data: 2022-09-02T04:32:08.642 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.643 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.643 DEBUG:boto:Params: {} 2022-09-02T04:32:08.644 DEBUG:boto:Token: None 2022-09-02T04:32:08.644 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cOeZYqyf2nIhdVjZUwvYxW0XgKM= 2022-09-02T04:32:08.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cOeZYqyf2nIhdVjZUwvYxW0XgKM=', 'Content-Length': '0'} 2022-09-02T04:32:08.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030de58c2e7b226ca-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.648 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.649 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.649 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.649 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.650 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.651 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.651 DEBUG:boto:Data: 2022-09-02T04:32:08.651 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.652 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.653 DEBUG:boto:Params: {} 2022-09-02T04:32:08.654 DEBUG:boto:Token: None 2022-09-02T04:32:08.654 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cOeZYqyf2nIhdVjZUwvYxW0XgKM= 2022-09-02T04:32:08.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cOeZYqyf2nIhdVjZUwvYxW0XgKM=', 'Content-Length': '0'} 2022-09-02T04:32:08.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed6302dac918684a-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:08.659 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.659 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.660 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.660 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.661 DEBUG:boto:Method: GET 2022-09-02T04:32:08.661 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.662 DEBUG:boto:Data: 2022-09-02T04:32:08.662 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.664 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.664 DEBUG:boto:Params: {} 2022-09-02T04:32:08.665 DEBUG:boto:Token: None 2022-09-02T04:32:08.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE= 2022-09-02T04:32:08.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE=', 'Content-Length': '0'} 2022-09-02T04:32:08.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047c58f9157940320-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.669 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.670 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.671 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.671 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.672 DEBUG:boto:Method: GET 2022-09-02T04:32:08.672 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.673 DEBUG:boto:Data: 2022-09-02T04:32:08.673 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.674 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.675 DEBUG:boto:Params: {} 2022-09-02T04:32:08.675 DEBUG:boto:Token: None 2022-09-02T04:32:08.676 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE= 2022-09-02T04:32:08.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE=', 'Content-Length': '0'} 2022-09-02T04:32:08.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095703ee922a5cf2d-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.680 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.681 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.681 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.682 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.682 DEBUG:boto:Method: GET 2022-09-02T04:32:08.683 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.683 DEBUG:boto:Data: 2022-09-02T04:32:08.684 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.685 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.685 DEBUG:boto:Params: {} 2022-09-02T04:32:08.686 DEBUG:boto:Token: None 2022-09-02T04:32:08.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f6eb633fd3f11fc-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.691 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.692 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.692 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.693 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.693 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.694 DEBUG:boto:Method: GET 2022-09-02T04:32:08.694 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.695 DEBUG:boto:Data: 2022-09-02T04:32:08.695 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.696 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.697 DEBUG:boto:Params: {} 2022-09-02T04:32:08.697 DEBUG:boto:Token: None 2022-09-02T04:32:08.698 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b760402d42ac9ef0-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.702 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.703 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.704 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.704 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.705 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.705 DEBUG:boto:Data: 2022-09-02T04:32:08.706 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.707 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.707 DEBUG:boto:Params: {} 2022-09-02T04:32:08.708 DEBUG:boto:Token: None 2022-09-02T04:32:08.708 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NU832ZKatPOKU6INsrtaQm/o35w= 2022-09-02T04:32:08.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NU832ZKatPOKU6INsrtaQm/o35w=', 'Content-Length': '0'} 2022-09-02T04:32:08.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd6c39e642ecd1ae-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.713 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.713 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.714 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.714 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.715 DEBUG:boto:Data: 2022-09-02T04:32:08.715 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.716 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.717 DEBUG:boto:Params: {} 2022-09-02T04:32:08.717 DEBUG:boto:Token: None 2022-09-02T04:32:08.718 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NU832ZKatPOKU6INsrtaQm/o35w= 2022-09-02T04:32:08.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NU832ZKatPOKU6INsrtaQm/o35w=', 'Content-Length': '0'} 2022-09-02T04:32:08.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002596c2dc034b9172-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:08.723 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.723 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.724 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.724 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.725 DEBUG:boto:Method: GET 2022-09-02T04:32:08.725 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.726 DEBUG:boto:Data: 2022-09-02T04:32:08.726 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.728 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.728 DEBUG:boto:Params: {} 2022-09-02T04:32:08.729 DEBUG:boto:Token: None 2022-09-02T04:32:08.729 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE= 2022-09-02T04:32:08.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE=', 'Content-Length': '0'} 2022-09-02T04:32:08.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce22b7ce5a1b311f-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.774 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.775 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:08.775 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.776 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.776 DEBUG:boto:Method: GET 2022-09-02T04:32:08.777 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.777 DEBUG:boto:Data: 2022-09-02T04:32:08.778 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.779 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.780 DEBUG:boto:Params: {} 2022-09-02T04:32:08.780 DEBUG:boto:Token: None 2022-09-02T04:32:08.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:08.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE= 2022-09-02T04:32:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FbPunJTIu5BmJYSv/EQRSsbJ1rE=', 'Content-Length': '0'} 2022-09-02T04:32:08.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c29ed69709bca4c4-0063118748-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.785 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.786 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.786 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.787 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.787 DEBUG:boto:Method: GET 2022-09-02T04:32:08.788 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.788 DEBUG:boto:Data: 2022-09-02T04:32:08.789 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.790 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.790 DEBUG:boto:Params: {} 2022-09-02T04:32:08.791 DEBUG:boto:Token: None 2022-09-02T04:32:08.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc4c251164fe807c-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.801 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.802 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.803 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.803 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.804 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.804 DEBUG:boto:Method: GET 2022-09-02T04:32:08.805 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:08.805 DEBUG:boto:Data: 2022-09-02T04:32:08.806 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:08.807 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.807 DEBUG:boto:Params: {} 2022-09-02T04:32:08.808 DEBUG:boto:Token: None 2022-09-02T04:32:08.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a5a35c0e7c1521-0063118748-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.813 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-17 zones={test-zone3, test-zone1} 2022-09-02T04:32:08.815 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-17 zones={test-zone3, test-zone2} 2022-09-02T04:32:08.815 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.816 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.816 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.817 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:32:08.817 DEBUG:boto:Data: 2022-09-02T04:32:08.818 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.819 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.819 DEBUG:boto:Params: {} 2022-09-02T04:32:08.820 DEBUG:boto:Token: None 2022-09-02T04:32:08.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tGeoze8OF/FWuQXtrHmjXEDXNUg= 2022-09-02T04:32:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tGeoze8OF/FWuQXtrHmjXEDXNUg=', 'Content-Length': '0'} 2022-09-02T04:32:08.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8d4ec9822a1dc7c-0063118748-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.825 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.826 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.827 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.827 DEBUG:boto:Path: /xljbkf-17/ 2022-09-02T04:32:08.828 DEBUG:boto:Data: 2022-09-02T04:32:08.829 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:08.831 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.832 DEBUG:boto:Params: {} 2022-09-02T04:32:08.833 DEBUG:boto:Token: None 2022-09-02T04:32:08.835 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tGeoze8OF/FWuQXtrHmjXEDXNUg= 2022-09-02T04:32:08.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tGeoze8OF/FWuQXtrHmjXEDXNUg=', 'Content-Length': '0'} 2022-09-02T04:32:08.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e345c6ec6714ec8-0063118748-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:08.843 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.844 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.845 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.846 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.847 DEBUG:boto:Method: GET 2022-09-02T04:32:08.848 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.849 DEBUG:boto:Data: 2022-09-02T04:32:08.850 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.852 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.854 DEBUG:boto:Params: {} 2022-09-02T04:32:08.855 DEBUG:boto:Token: None 2022-09-02T04:32:08.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/?versions 2022-09-02T04:32:08.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s= 2022-09-02T04:32:08.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s=', 'Content-Length': '0'} 2022-09-02T04:32:08.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065811591655b1f45-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.860 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.861 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:08.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:08.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:08.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:08.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:08.864 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.864 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.865 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.865 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.866 DEBUG:boto:Method: GET 2022-09-02T04:32:08.867 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.867 DEBUG:boto:Data: 2022-09-02T04:32:08.868 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:08.869 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.869 DEBUG:boto:Params: {} 2022-09-02T04:32:08.870 DEBUG:boto:Token: None 2022-09-02T04:32:08.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/?versions 2022-09-02T04:32:08.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s= 2022-09-02T04:32:08.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s=', 'Content-Length': '0'} 2022-09-02T04:32:08.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f6d55d94889e1ba-0063118748-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.875 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.876 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:08.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:08.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:08.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:08.878 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.879 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.879 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.880 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.880 DEBUG:boto:Method: GET 2022-09-02T04:32:08.881 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.881 DEBUG:boto:Data: 2022-09-02T04:32:08.882 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.883 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.883 DEBUG:boto:Params: {} 2022-09-02T04:32:08.884 DEBUG:boto:Token: None 2022-09-02T04:32:08.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/?versions 2022-09-02T04:32:08.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s= 2022-09-02T04:32:08.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s=', 'Content-Length': '0'} 2022-09-02T04:32:08.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000713ed1c961847d26-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.889 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.890 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.891 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.891 DEBUG:boto:path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.892 DEBUG:boto:auth_path=/xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.893 DEBUG:boto:Method: GET 2022-09-02T04:32:08.894 DEBUG:boto:Path: /xljbkf-17/?versions&max-keys=999 2022-09-02T04:32:08.894 DEBUG:boto:Data: 2022-09-02T04:32:08.895 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:08.896 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.897 DEBUG:boto:Params: {} 2022-09-02T04:32:08.897 DEBUG:boto:Token: None 2022-09-02T04:32:08.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/?versions 2022-09-02T04:32:08.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s= 2022-09-02T04:32:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rEvmNdS33qCjg6k5rk3etNM/h2s=', 'Content-Length': '0'} 2022-09-02T04:32:08.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d0506fdf60a6c06-0063118748-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.903 DEBUG:boto:b'xljbkf-17999falsedummyh.yiVp4UbsPuQWKsGIzCaYXysbVfBnktrue2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone15vtqVAXCf5aJQzZBkqNt0O4aniybcaetrue2022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2zjx2aowzKhVD2rn7aJkKbjMn1yE54FYtrue2022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3Ms19bVrVfyFR.90vg1GDe61MUMjwXLhtrue2022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:08.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:08.905 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.905 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.906 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.906 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.907 DEBUG:boto:Method: GET 2022-09-02T04:32:08.908 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.908 DEBUG:boto:Data: 2022-09-02T04:32:08.909 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.910 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.911 DEBUG:boto:Params: {} 2022-09-02T04:32:08.911 DEBUG:boto:Token: None 2022-09-02T04:32:08.912 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074998aa0c6543562-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.916 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.917 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.917 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.918 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.919 DEBUG:boto:Method: GET 2022-09-02T04:32:08.919 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.920 DEBUG:boto:Data: 2022-09-02T04:32:08.921 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:08.922 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.923 DEBUG:boto:Params: {} 2022-09-02T04:32:08.924 DEBUG:boto:Token: None 2022-09-02T04:32:08.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036d6e614d1dbfc9e-0063118748-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.929 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.929 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.930 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.930 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.931 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.932 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.933 DEBUG:boto:Data: 2022-09-02T04:32:08.934 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.936 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.937 DEBUG:boto:Params: {} 2022-09-02T04:32:08.938 DEBUG:boto:Token: None 2022-09-02T04:32:08.939 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU= 2022-09-02T04:32:08.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU=', 'Content-Length': '0'} 2022-09-02T04:32:08.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d4957fe5f3afa58-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.944 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.945 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.945 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.946 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.947 DEBUG:boto:Method: HEAD 2022-09-02T04:32:08.947 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.948 DEBUG:boto:Data: 2022-09-02T04:32:08.948 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:08.949 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.950 DEBUG:boto:Params: {} 2022-09-02T04:32:08.950 DEBUG:boto:Token: None 2022-09-02T04:32:08.951 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU= 2022-09-02T04:32:08.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xs3ETyYNF5vroi/QXlophMHq5DU=', 'Content-Length': '0'} 2022-09-02T04:32:08.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083e3dea13acd105d-0063118748-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:08.956 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.956 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.957 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.957 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.958 DEBUG:boto:Method: GET 2022-09-02T04:32:08.958 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.959 DEBUG:boto:Data: 2022-09-02T04:32:08.960 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.961 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.961 DEBUG:boto:Params: {} 2022-09-02T04:32:08.962 DEBUG:boto:Token: None 2022-09-02T04:32:08.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000068a63f535278c2d-0063118748-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.967 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:08.968 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:08.968 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.969 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.969 DEBUG:boto:Method: GET 2022-09-02T04:32:08.970 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.970 DEBUG:boto:Data: 2022-09-02T04:32:08.971 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:08.972 DEBUG:boto:Port: 8000 2022-09-02T04:32:08.973 DEBUG:boto:Params: {} 2022-09-02T04:32:08.974 DEBUG:boto:Token: None 2022-09-02T04:32:08.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:08.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo= 2022-09-02T04:32:08.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/yw4ESe3R2ozWBT8HMsdYVBDEoo=', 'Content-Length': '0'} 2022-09-02T04:32:08.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000453ef152b9b046f8-0063118748-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:08.979 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:08.979 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:08.980 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.980 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.981 DEBUG:boto:Method: GET 2022-09-02T04:32:08.981 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:08.982 DEBUG:boto:Data: 2022-09-02T04:32:08.983 DEBUG:boto:Headers: {} 2022-09-02T04:32:08.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:08.984 DEBUG:boto:Port: 8001 2022-09-02T04:32:08.985 DEBUG:boto:Params: {} 2022-09-02T04:32:08.985 DEBUG:boto:Token: None 2022-09-02T04:32:08.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:08 GMT /xljbkf-17/ 2022-09-02T04:32:08.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U= 2022-09-02T04:32:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPOAiaRUrtpSIFzvPbcw+pR0G0U=', 'Content-Length': '0'} 2022-09-02T04:32:08.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f73bfe5daa8570e8-0063118748-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.038 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.039 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.040 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.041 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.041 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.042 DEBUG:boto:Method: GET 2022-09-02T04:32:09.042 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.044 DEBUG:boto:Data: 2022-09-02T04:32:09.045 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.046 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.046 DEBUG:boto:Params: {} 2022-09-02T04:32:09.047 DEBUG:boto:Token: None 2022-09-02T04:32:09.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450a034094903f8e-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.054 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.055 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:09.056 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:09.057 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.058 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:32:09.059 DEBUG:boto:Data: 2022-09-02T04:32:09.060 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.062 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.063 DEBUG:boto:Params: {} 2022-09-02T04:32:09.064 DEBUG:boto:Token: None 2022-09-02T04:32:09.065 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/dummy 2022-09-02T04:32:09.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bG5pJQyy6/bhMQV6+C22yaEYX4k= 2022-09-02T04:32:09.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bG5pJQyy6/bhMQV6+C22yaEYX4k=', 'Content-Length': '0'} 2022-09-02T04:32:09.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059f478df1a86b1f9-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.071 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:09.071 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:09.072 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.072 DEBUG:boto:Path: /xljbkf-17/dummy 2022-09-02T04:32:09.073 DEBUG:boto:Data: 2022-09-02T04:32:09.073 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.074 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.074 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.075 DEBUG:boto:Params: {} 2022-09-02T04:32:09.075 DEBUG:boto:Token: None 2022-09-02T04:32:09.076 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/dummy 2022-09-02T04:32:09.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bG5pJQyy6/bhMQV6+C22yaEYX4k= 2022-09-02T04:32:09.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bG5pJQyy6/bhMQV6+C22yaEYX4k=', 'Content-Length': '0'} 2022-09-02T04:32:09.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cafae1e10e40a85-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:09.081 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:09.082 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:09.082 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.083 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.083 DEBUG:boto:Method: GET 2022-09-02T04:32:09.084 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.084 DEBUG:boto:Data: 2022-09-02T04:32:09.085 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.086 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.087 DEBUG:boto:Params: {} 2022-09-02T04:32:09.088 DEBUG:boto:Token: None 2022-09-02T04:32:09.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T2yv88MVlWfhbVhSM0l90bbomPE= 2022-09-02T04:32:09.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T2yv88MVlWfhbVhSM0l90bbomPE=', 'Content-Length': '0'} 2022-09-02T04:32:09.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000064f240972515a06f-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.095 DEBUG:boto:path=/xljbkf-17/dummy 2022-09-02T04:32:09.096 DEBUG:boto:auth_path=/xljbkf-17/dummy 2022-09-02T04:32:09.096 DEBUG:boto:path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.097 DEBUG:boto:auth_path=/xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.097 DEBUG:boto:Method: GET 2022-09-02T04:32:09.098 DEBUG:boto:Path: /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.098 DEBUG:boto:Data: 2022-09-02T04:32:09.099 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.100 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.101 DEBUG:boto:Params: {} 2022-09-02T04:32:09.102 DEBUG:boto:Token: None 2022-09-02T04:32:09.102 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/dummy?versionId=h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk 2022-09-02T04:32:09.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T2yv88MVlWfhbVhSM0l90bbomPE= 2022-09-02T04:32:09.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T2yv88MVlWfhbVhSM0l90bbomPE=', 'Content-Length': '0'} 2022-09-02T04:32:09.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:00 GMT'), ('x-amz-version-id', 'h.yiVp4UbsPuQWKsGIzCaYXysbVfBnk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f865a119cee03a9-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.107 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.107 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.108 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.108 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.109 DEBUG:boto:Method: GET 2022-09-02T04:32:09.110 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.110 DEBUG:boto:Data: 2022-09-02T04:32:09.111 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.112 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.112 DEBUG:boto:Params: {} 2022-09-02T04:32:09.113 DEBUG:boto:Token: None 2022-09-02T04:32:09.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4ecf7a6ef50c23-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.118 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.119 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.120 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.120 DEBUG:boto:path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.121 DEBUG:boto:auth_path=/xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.121 DEBUG:boto:Method: GET 2022-09-02T04:32:09.122 DEBUG:boto:Path: /xljbkf-17/?prefix=dummy 2022-09-02T04:32:09.122 DEBUG:boto:Data: 2022-09-02T04:32:09.123 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.124 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.125 DEBUG:boto:Params: {} 2022-09-02T04:32:09.125 DEBUG:boto:Token: None 2022-09-02T04:32:09.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464bb4398f1cd3d2-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.130 DEBUG:boto:b'xljbkf-17dummy1000falsedummy2022-09-02T04:28:00.671Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:09.132 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.132 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.133 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.134 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.134 DEBUG:boto:Method: GET 2022-09-02T04:32:09.135 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.136 DEBUG:boto:Data: 2022-09-02T04:32:09.136 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.137 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.138 DEBUG:boto:Params: {} 2022-09-02T04:32:09.138 DEBUG:boto:Token: None 2022-09-02T04:32:09.139 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ= 2022-09-02T04:32:09.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c936be1cad1ee89e-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.143 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.144 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.144 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.145 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.145 DEBUG:boto:Method: GET 2022-09-02T04:32:09.146 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.146 DEBUG:boto:Data: 2022-09-02T04:32:09.147 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.148 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.149 DEBUG:boto:Params: {} 2022-09-02T04:32:09.149 DEBUG:boto:Token: None 2022-09-02T04:32:09.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ= 2022-09-02T04:32:09.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.154 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000193290552879d4b6-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.154 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.155 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.156 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.156 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.157 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.157 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.158 DEBUG:boto:Data: 2022-09-02T04:32:09.158 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.159 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.160 DEBUG:boto:Params: {} 2022-09-02T04:32:09.160 DEBUG:boto:Token: None 2022-09-02T04:32:09.161 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7emJ5J1qySdwxwIurE4wM8ymcnw= 2022-09-02T04:32:09.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7emJ5J1qySdwxwIurE4wM8ymcnw=', 'Content-Length': '0'} 2022-09-02T04:32:09.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cf33016cfe80afe-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.166 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.166 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.167 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.167 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.168 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.168 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.169 DEBUG:boto:Data: 2022-09-02T04:32:09.169 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.170 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.171 DEBUG:boto:Params: {} 2022-09-02T04:32:09.172 DEBUG:boto:Token: None 2022-09-02T04:32:09.172 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7emJ5J1qySdwxwIurE4wM8ymcnw= 2022-09-02T04:32:09.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7emJ5J1qySdwxwIurE4wM8ymcnw=', 'Content-Length': '0'} 2022-09-02T04:32:09.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2144d6d5d813499-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:09.177 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.178 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.178 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.179 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.180 DEBUG:boto:Method: GET 2022-09-02T04:32:09.180 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.181 DEBUG:boto:Data: 2022-09-02T04:32:09.182 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.183 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.184 DEBUG:boto:Params: {} 2022-09-02T04:32:09.184 DEBUG:boto:Token: None 2022-09-02T04:32:09.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ= 2022-09-02T04:32:09.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6f334adb645c1ae-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.190 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.191 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.191 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.192 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.193 DEBUG:boto:Method: GET 2022-09-02T04:32:09.193 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.194 DEBUG:boto:Data: 2022-09-02T04:32:09.195 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.196 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.197 DEBUG:boto:Params: {} 2022-09-02T04:32:09.197 DEBUG:boto:Token: None 2022-09-02T04:32:09.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ= 2022-09-02T04:32:09.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099386d76aa55f21b-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.203 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.204 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.204 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.205 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.206 DEBUG:boto:Method: GET 2022-09-02T04:32:09.206 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.207 DEBUG:boto:Data: 2022-09-02T04:32:09.208 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.210 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.210 DEBUG:boto:Params: {} 2022-09-02T04:32:09.211 DEBUG:boto:Token: None 2022-09-02T04:32:09.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041fecb2ff5a15484-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.218 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.219 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.220 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.220 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.221 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.222 DEBUG:boto:Method: GET 2022-09-02T04:32:09.222 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.223 DEBUG:boto:Data: 2022-09-02T04:32:09.224 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.225 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.225 DEBUG:boto:Params: {} 2022-09-02T04:32:09.226 DEBUG:boto:Token: None 2022-09-02T04:32:09.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011313cb4aa4c3cf5-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.278 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.279 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.280 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.281 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.282 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.282 DEBUG:boto:Data: 2022-09-02T04:32:09.283 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.285 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.285 DEBUG:boto:Params: {} 2022-09-02T04:32:09.286 DEBUG:boto:Token: None 2022-09-02T04:32:09.287 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aZyR9FojlA4XgQqr3awwb/0NvyI= 2022-09-02T04:32:09.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aZyR9FojlA4XgQqr3awwb/0NvyI=', 'Content-Length': '0'} 2022-09-02T04:32:09.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007386485e9bdb058e-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.292 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.294 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.295 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.296 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.297 DEBUG:boto:Data: 2022-09-02T04:32:09.297 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.299 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.300 DEBUG:boto:Params: {} 2022-09-02T04:32:09.302 DEBUG:boto:Token: None 2022-09-02T04:32:09.303 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aZyR9FojlA4XgQqr3awwb/0NvyI= 2022-09-02T04:32:09.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aZyR9FojlA4XgQqr3awwb/0NvyI=', 'Content-Length': '0'} 2022-09-02T04:32:09.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b34e9fa0106bc95c-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:09.310 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.310 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.311 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.311 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.312 DEBUG:boto:Method: GET 2022-09-02T04:32:09.313 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.313 DEBUG:boto:Data: 2022-09-02T04:32:09.314 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.315 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.316 DEBUG:boto:Params: {} 2022-09-02T04:32:09.316 DEBUG:boto:Token: None 2022-09-02T04:32:09.317 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ= 2022-09-02T04:32:09.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb7601d0730dfb6d-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.362 DEBUG:boto:path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.363 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1 2022-09-02T04:32:09.364 DEBUG:boto:path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.364 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.365 DEBUG:boto:Method: GET 2022-09-02T04:32:09.365 DEBUG:boto:Path: /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.366 DEBUG:boto:Data: 2022-09-02T04:32:09.366 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.368 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.368 DEBUG:boto:Params: {} 2022-09-02T04:32:09.369 DEBUG:boto:Token: None 2022-09-02T04:32:09.369 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone1?versionId=5vtqVAXCf5aJQzZBkqNt0O4aniybcae 2022-09-02T04:32:09.370 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ= 2022-09-02T04:32:09.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GebYGLQT4TNjhksUTqLjkx6hTZQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', '5vtqVAXCf5aJQzZBkqNt0O4aniybcae'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a91a52db5a0a633-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.380 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.380 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.381 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.381 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.382 DEBUG:boto:Method: GET 2022-09-02T04:32:09.382 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.383 DEBUG:boto:Data: 2022-09-02T04:32:09.383 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.384 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.385 DEBUG:boto:Params: {} 2022-09-02T04:32:09.385 DEBUG:boto:Token: None 2022-09-02T04:32:09.387 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057c23620bfd7e927-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.392 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.393 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.393 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.394 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.394 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.395 DEBUG:boto:Method: GET 2022-09-02T04:32:09.395 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone1 2022-09-02T04:32:09.396 DEBUG:boto:Data: 2022-09-02T04:32:09.396 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.398 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.398 DEBUG:boto:Params: {} 2022-09-02T04:32:09.399 DEBUG:boto:Token: None 2022-09-02T04:32:09.399 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.400 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a247b9ba9cc8d30e-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.404 DEBUG:boto:b'xljbkf-17obj-test-zone11000falseobj-test-zone12022-09-02T04:29:01.892Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:09.406 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.407 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.407 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.408 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.408 DEBUG:boto:Method: GET 2022-09-02T04:32:09.409 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.409 DEBUG:boto:Data: 2022-09-02T04:32:09.410 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.411 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.411 DEBUG:boto:Params: {} 2022-09-02T04:32:09.412 DEBUG:boto:Token: None 2022-09-02T04:32:09.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ= 2022-09-02T04:32:09.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2bca8b114de0a47-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.417 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.418 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.419 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.419 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.420 DEBUG:boto:Method: GET 2022-09-02T04:32:09.421 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.421 DEBUG:boto:Data: 2022-09-02T04:32:09.422 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.424 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.424 DEBUG:boto:Params: {} 2022-09-02T04:32:09.425 DEBUG:boto:Token: None 2022-09-02T04:32:09.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ= 2022-09-02T04:32:09.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ee02bce03376474-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.470 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.471 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.472 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.473 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.473 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.474 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.475 DEBUG:boto:Data: 2022-09-02T04:32:09.476 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.478 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.479 DEBUG:boto:Params: {} 2022-09-02T04:32:09.480 DEBUG:boto:Token: None 2022-09-02T04:32:09.481 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8cBd8l/bqQinBXi7RlcjmMKy4g= 2022-09-02T04:32:09.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8cBd8l/bqQinBXi7RlcjmMKy4g=', 'Content-Length': '0'} 2022-09-02T04:32:09.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054e5cd84229951d5-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.487 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.488 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.489 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.490 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.491 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.492 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.493 DEBUG:boto:Data: 2022-09-02T04:32:09.494 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.495 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.496 DEBUG:boto:Params: {} 2022-09-02T04:32:09.497 DEBUG:boto:Token: None 2022-09-02T04:32:09.499 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8cBd8l/bqQinBXi7RlcjmMKy4g= 2022-09-02T04:32:09.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8cBd8l/bqQinBXi7RlcjmMKy4g=', 'Content-Length': '0'} 2022-09-02T04:32:09.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ada023272262ff1-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:09.505 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.506 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.506 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.507 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.507 DEBUG:boto:Method: GET 2022-09-02T04:32:09.508 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.509 DEBUG:boto:Data: 2022-09-02T04:32:09.509 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.511 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.511 DEBUG:boto:Params: {} 2022-09-02T04:32:09.513 DEBUG:boto:Token: None 2022-09-02T04:32:09.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ= 2022-09-02T04:32:09.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fea10fa20598ab5-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.520 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.521 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.522 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.522 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.523 DEBUG:boto:Method: GET 2022-09-02T04:32:09.524 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.524 DEBUG:boto:Data: 2022-09-02T04:32:09.525 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.525 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.526 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.526 DEBUG:boto:Params: {} 2022-09-02T04:32:09.527 DEBUG:boto:Token: None 2022-09-02T04:32:09.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ= 2022-09-02T04:32:09.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000371580a56d0c9c23-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.531 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.532 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.533 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.533 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.534 DEBUG:boto:Method: GET 2022-09-02T04:32:09.534 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.535 DEBUG:boto:Data: 2022-09-02T04:32:09.535 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.536 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.537 DEBUG:boto:Params: {} 2022-09-02T04:32:09.537 DEBUG:boto:Token: None 2022-09-02T04:32:09.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8955e186a5969a-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.586 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.588 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.589 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.590 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.590 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.591 DEBUG:boto:Method: GET 2022-09-02T04:32:09.592 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.593 DEBUG:boto:Data: 2022-09-02T04:32:09.593 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.595 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.595 DEBUG:boto:Params: {} 2022-09-02T04:32:09.596 DEBUG:boto:Token: None 2022-09-02T04:32:09.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000becb913a522bbb4e-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.601 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.602 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.603 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.603 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.604 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.604 DEBUG:boto:Data: 2022-09-02T04:32:09.605 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.606 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.606 DEBUG:boto:Params: {} 2022-09-02T04:32:09.607 DEBUG:boto:Token: None 2022-09-02T04:32:09.608 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G/cBlgB2HyIV2z3U50n5yXRv1Y0= 2022-09-02T04:32:09.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G/cBlgB2HyIV2z3U50n5yXRv1Y0=', 'Content-Length': '0'} 2022-09-02T04:32:09.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a52f50a5f5f4f89c-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.612 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.613 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.613 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.614 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.614 DEBUG:boto:Data: 2022-09-02T04:32:09.615 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.616 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.616 DEBUG:boto:Params: {} 2022-09-02T04:32:09.617 DEBUG:boto:Token: None 2022-09-02T04:32:09.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G/cBlgB2HyIV2z3U50n5yXRv1Y0= 2022-09-02T04:32:09.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G/cBlgB2HyIV2z3U50n5yXRv1Y0=', 'Content-Length': '0'} 2022-09-02T04:32:09.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b74f500892e4452c-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:09.623 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.624 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.624 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.625 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.626 DEBUG:boto:Method: GET 2022-09-02T04:32:09.627 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.627 DEBUG:boto:Data: 2022-09-02T04:32:09.628 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.630 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.631 DEBUG:boto:Params: {} 2022-09-02T04:32:09.632 DEBUG:boto:Token: None 2022-09-02T04:32:09.633 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ= 2022-09-02T04:32:09.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bda590208a9f5c42-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.638 DEBUG:boto:path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.639 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2 2022-09-02T04:32:09.640 DEBUG:boto:path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.641 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.642 DEBUG:boto:Method: GET 2022-09-02T04:32:09.643 DEBUG:boto:Path: /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.644 DEBUG:boto:Data: 2022-09-02T04:32:09.645 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.646 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.647 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.648 DEBUG:boto:Params: {} 2022-09-02T04:32:09.648 DEBUG:boto:Token: None 2022-09-02T04:32:09.649 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone2?versionId=zjx2aowzKhVD2rn7aJkKbjMn1yE54FY 2022-09-02T04:32:09.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ= 2022-09-02T04:32:09.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Eu1xwRultOzawS/V9B7G4Dq3zQ=', 'Content-Length': '0'} 2022-09-02T04:32:09.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'zjx2aowzKhVD2rn7aJkKbjMn1yE54FY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b5d1520aab97343-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.653 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.654 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.655 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.655 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.656 DEBUG:boto:Method: GET 2022-09-02T04:32:09.656 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.657 DEBUG:boto:Data: 2022-09-02T04:32:09.658 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.659 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.659 DEBUG:boto:Params: {} 2022-09-02T04:32:09.660 DEBUG:boto:Token: None 2022-09-02T04:32:09.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e249d2f3b04d4f22-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.665 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.666 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.667 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.667 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.668 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.669 DEBUG:boto:Method: GET 2022-09-02T04:32:09.669 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone2 2022-09-02T04:32:09.670 DEBUG:boto:Data: 2022-09-02T04:32:09.671 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.672 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.673 DEBUG:boto:Params: {} 2022-09-02T04:32:09.674 DEBUG:boto:Token: None 2022-09-02T04:32:09.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e359076dc68ce97c-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.680 DEBUG:boto:b'xljbkf-17obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:09.682 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.682 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.683 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.683 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.684 DEBUG:boto:Method: GET 2022-09-02T04:32:09.684 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.685 DEBUG:boto:Data: 2022-09-02T04:32:09.686 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.687 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.688 DEBUG:boto:Params: {} 2022-09-02T04:32:09.688 DEBUG:boto:Token: None 2022-09-02T04:32:09.689 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA= 2022-09-02T04:32:09.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA=', 'Content-Length': '0'} 2022-09-02T04:32:09.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009da2f046720ff51d-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.695 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.695 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.696 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.696 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.697 DEBUG:boto:Method: GET 2022-09-02T04:32:09.697 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.698 DEBUG:boto:Data: 2022-09-02T04:32:09.698 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.699 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.700 DEBUG:boto:Params: {} 2022-09-02T04:32:09.700 DEBUG:boto:Token: None 2022-09-02T04:32:09.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA= 2022-09-02T04:32:09.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA=', 'Content-Length': '0'} 2022-09-02T04:32:09.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007241b12524a4e122-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.706 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.706 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.707 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.707 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.708 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.708 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.709 DEBUG:boto:Data: 2022-09-02T04:32:09.709 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.710 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.711 DEBUG:boto:Params: {} 2022-09-02T04:32:09.711 DEBUG:boto:Token: None 2022-09-02T04:32:09.712 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zoS6ZeU6kgGIQxPrtJ77Q7hDofo= 2022-09-02T04:32:09.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zoS6ZeU6kgGIQxPrtJ77Q7hDofo=', 'Content-Length': '0'} 2022-09-02T04:32:09.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000749bab8dddcb00e6-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.716 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.716 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.717 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.717 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.718 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.718 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.719 DEBUG:boto:Data: 2022-09-02T04:32:09.719 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.720 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.721 DEBUG:boto:Params: {} 2022-09-02T04:32:09.721 DEBUG:boto:Token: None 2022-09-02T04:32:09.722 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zoS6ZeU6kgGIQxPrtJ77Q7hDofo= 2022-09-02T04:32:09.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zoS6ZeU6kgGIQxPrtJ77Q7hDofo=', 'Content-Length': '0'} 2022-09-02T04:32:09.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075b128ad2d084b9e-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:09.726 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.727 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.727 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.728 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.728 DEBUG:boto:Method: GET 2022-09-02T04:32:09.729 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.729 DEBUG:boto:Data: 2022-09-02T04:32:09.730 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.731 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.732 DEBUG:boto:Params: {} 2022-09-02T04:32:09.732 DEBUG:boto:Token: None 2022-09-02T04:32:09.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA= 2022-09-02T04:32:09.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA=', 'Content-Length': '0'} 2022-09-02T04:32:09.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005daadfcf83ed4727-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.737 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.738 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.738 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.739 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.739 DEBUG:boto:Method: GET 2022-09-02T04:32:09.740 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.740 DEBUG:boto:Data: 2022-09-02T04:32:09.741 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.742 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.742 DEBUG:boto:Params: {} 2022-09-02T04:32:09.743 DEBUG:boto:Token: None 2022-09-02T04:32:09.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA= 2022-09-02T04:32:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA=', 'Content-Length': '0'} 2022-09-02T04:32:09.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ac085cca026d40-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.747 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.748 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.748 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.749 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.749 DEBUG:boto:Method: GET 2022-09-02T04:32:09.750 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.750 DEBUG:boto:Data: 2022-09-02T04:32:09.751 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.752 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.752 DEBUG:boto:Params: {} 2022-09-02T04:32:09.753 DEBUG:boto:Token: None 2022-09-02T04:32:09.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f64e22ddae717cf4-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.758 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.759 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.759 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.760 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.760 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.761 DEBUG:boto:Method: GET 2022-09-02T04:32:09.761 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.762 DEBUG:boto:Data: 2022-09-02T04:32:09.762 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.763 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.764 DEBUG:boto:Params: {} 2022-09-02T04:32:09.764 DEBUG:boto:Token: None 2022-09-02T04:32:09.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002057d7a94cc07652-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.810 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.812 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.813 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.814 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.815 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.816 DEBUG:boto:Data: 2022-09-02T04:32:09.817 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.818 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.819 DEBUG:boto:Params: {} 2022-09-02T04:32:09.820 DEBUG:boto:Token: None 2022-09-02T04:32:09.821 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S/Z+lCz7QdRyj4HDel9IHEgKhYY= 2022-09-02T04:32:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S/Z+lCz7QdRyj4HDel9IHEgKhYY=', 'Content-Length': '0'} 2022-09-02T04:32:09.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f172d52b8e5a911-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.828 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.829 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.830 DEBUG:boto:Method: HEAD 2022-09-02T04:32:09.831 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.832 DEBUG:boto:Data: 2022-09-02T04:32:09.833 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.834 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.835 DEBUG:boto:Params: {} 2022-09-02T04:32:09.836 DEBUG:boto:Token: None 2022-09-02T04:32:09.836 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S/Z+lCz7QdRyj4HDel9IHEgKhYY= 2022-09-02T04:32:09.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S/Z+lCz7QdRyj4HDel9IHEgKhYY=', 'Content-Length': '0'} 2022-09-02T04:32:09.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b11842549af72cc4-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:09.841 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.842 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.843 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.843 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.844 DEBUG:boto:Method: GET 2022-09-02T04:32:09.844 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.845 DEBUG:boto:Data: 2022-09-02T04:32:09.845 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.846 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.847 DEBUG:boto:Params: {} 2022-09-02T04:32:09.848 DEBUG:boto:Token: None 2022-09-02T04:32:09.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA= 2022-09-02T04:32:09.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA=', 'Content-Length': '0'} 2022-09-02T04:32:09.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e02647c961d05327-0063118749-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.898 DEBUG:boto:path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.899 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3 2022-09-02T04:32:09.900 DEBUG:boto:path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.901 DEBUG:boto:auth_path=/xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.901 DEBUG:boto:Method: GET 2022-09-02T04:32:09.902 DEBUG:boto:Path: /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.902 DEBUG:boto:Data: 2022-09-02T04:32:09.903 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.904 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.904 DEBUG:boto:Params: {} 2022-09-02T04:32:09.905 DEBUG:boto:Token: None 2022-09-02T04:32:09.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/obj-test-zone3?versionId=Ms19bVrVfyFR.90vg1GDe61MUMjwXLh 2022-09-02T04:32:09.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA= 2022-09-02T04:32:09.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12op1zUugxUme4MQuYgh+YnqpVA=', 'Content-Length': '0'} 2022-09-02T04:32:09.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:10 GMT'), ('x-amz-version-id', 'Ms19bVrVfyFR.90vg1GDe61MUMjwXLh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb8541a1451e1f41-0063118749-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.910 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.911 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.912 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.912 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.913 DEBUG:boto:Method: GET 2022-09-02T04:32:09.913 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.914 DEBUG:boto:Data: 2022-09-02T04:32:09.914 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:09.915 DEBUG:boto:Port: 8001 2022-09-02T04:32:09.916 DEBUG:boto:Params: {} 2022-09-02T04:32:09.917 DEBUG:boto:Token: None 2022-09-02T04:32:09.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d22cf2a5412c3c6a-0063118749-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.922 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.923 DEBUG:boto:path=/xljbkf-17/ 2022-09-02T04:32:09.923 DEBUG:boto:auth_path=/xljbkf-17/ 2022-09-02T04:32:09.924 DEBUG:boto:path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.925 DEBUG:boto:auth_path=/xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.925 DEBUG:boto:Method: GET 2022-09-02T04:32:09.926 DEBUG:boto:Path: /xljbkf-17/?prefix=obj-test-zone3 2022-09-02T04:32:09.927 DEBUG:boto:Data: 2022-09-02T04:32:09.927 DEBUG:boto:Headers: {} 2022-09-02T04:32:09.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:09.928 DEBUG:boto:Port: 8000 2022-09-02T04:32:09.929 DEBUG:boto:Params: {} 2022-09-02T04:32:09.930 DEBUG:boto:Token: None 2022-09-02T04:32:09.930 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:09 GMT /xljbkf-17/ 2022-09-02T04:32:09.931 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE= 2022-09-02T04:32:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RO02jr4r2onm0QRgV8LtAxlN3YE=', 'Content-Length': '0'} 2022-09-02T04:32:09.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e9f7d1f972c3ea-0063118749-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:09.935 DEBUG:boto:b'xljbkf-17obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.253Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:09.936 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-17 zones={test-zone3, test-zone2} 2022-09-02T04:32:09.936 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-18 2022-09-02T04:32:09.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:09.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:09.938 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.002 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.002 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.051 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:10.051+0000 7ffa7bb22600 1 bucket sync caught up with source: 2022-09-02T04:32:10.052 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.31.5, , 00000000004.42.5, , 00000000008.28.5, , , , , ] 2022-09-02T04:32:10.052 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.31.5, , 00000000004.42.5, , 00000000008.28.5, , , , , ] 2022-09-02T04:32:10.052 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:10.051+0000 7ffa7bb22600 0 bucket checkpoint complete 2022-09-02T04:32:10.068 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-18 2022-09-02T04:32:10.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.070 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.171 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.170+0000 7f7482e1e600 1 bucket sync caught up with source: 2022-09-02T04:32:10.171 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000011.31.5, , 00000000004.42.5, , 00000000008.28.5, , , , , ] 2022-09-02T04:32:10.172 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000011.31.5, , 00000000004.42.5, , 00000000008.28.5, , , , , ] 2022-09-02T04:32:10.172 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.170+0000 7f7482e1e600 0 bucket checkpoint complete 2022-09-02T04:32:10.183 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-18 2022-09-02T04:32:10.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.187 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.246 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.247 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.294 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:10.293+0000 7fd8f21b9600 1 bucket sync caught up with source: 2022-09-02T04:32:10.295 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000011.31.5, , 00000000004.42.5, , 00000000008.28.5, , , , , ] 2022-09-02T04:32:10.295 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000011.31.5, , 00000000004.42.5, , 00000000008.28.5, , , , , ] 2022-09-02T04:32:10.296 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:10.293+0000 7fd8f21b9600 0 bucket checkpoint complete 2022-09-02T04:32:10.306 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-18 2022-09-02T04:32:10.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.309 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.368 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.412+0000 7fdad3c75600 1 bucket sync caught up with source: 2022-09-02T04:32:10.413 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.33.5, , 00000000011.71.5, , 00000000008.44.5, , , , , ] 2022-09-02T04:32:10.413 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.33.5, , 00000000011.71.5, , 00000000008.44.5, , , , , ] 2022-09-02T04:32:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.412+0000 7fdad3c75600 0 bucket checkpoint complete 2022-09-02T04:32:10.425 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-18 2022-09-02T04:32:10.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.427 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.522 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.522+0000 7f3d03784600 1 bucket sync caught up with source: 2022-09-02T04:32:10.523 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.33.5, , 00000000011.71.5, , 00000000008.44.5, , , , , ] 2022-09-02T04:32:10.523 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.33.5, , 00000000011.71.5, , 00000000008.44.5, , , , , ] 2022-09-02T04:32:10.524 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.522+0000 7f3d03784600 0 bucket checkpoint complete 2022-09-02T04:32:10.538 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-18 2022-09-02T04:32:10.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-18 2022-09-02T04:32:10.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.541 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.641 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.640+0000 7fc341b20600 1 bucket sync caught up with source: 2022-09-02T04:32:10.641 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000004.23.5, , 00000000004.45.5, , 00000000016.56.5, , , , , ] 2022-09-02T04:32:10.641 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000004.23.5, , 00000000004.45.5, , 00000000016.56.5, , , , , ] 2022-09-02T04:32:10.642 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:10.640+0000 7fc341b20600 0 bucket checkpoint complete 2022-09-02T04:32:10.653 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-18 2022-09-02T04:32:10.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:10.655 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:10.711 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:10.712 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:10.744+0000 7f2e0b8b8600 1 bucket sync caught up with source: 2022-09-02T04:32:10.745 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000004.23.5, , 00000000004.45.5, , 00000000016.56.5, , , , , ] 2022-09-02T04:32:10.745 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000004.23.5, , 00000000004.45.5, , 00000000016.56.5, , , , , ] 2022-09-02T04:32:10.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:10.744+0000 7f2e0b8b8600 0 bucket checkpoint complete 2022-09-02T04:32:10.756 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-18 2022-09-02T04:32:10.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-18 zones={test-zone2, test-zone1} 2022-09-02T04:32:10.757 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.758 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.759 DEBUG:boto:Method: HEAD 2022-09-02T04:32:10.759 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:32:10.760 DEBUG:boto:Data: 2022-09-02T04:32:10.761 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.763 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.764 DEBUG:boto:Params: {} 2022-09-02T04:32:10.765 DEBUG:boto:Token: None 2022-09-02T04:32:10.766 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/ 2022-09-02T04:32:10.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H4KkQJIQc6pBnHXj2KlImVmowEc= 2022-09-02T04:32:10.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H4KkQJIQc6pBnHXj2KlImVmowEc=', 'Content-Length': '0'} 2022-09-02T04:32:10.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a63fdad2bb71b010-006311874a-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.771 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.771 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.772 DEBUG:boto:Method: HEAD 2022-09-02T04:32:10.772 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:32:10.773 DEBUG:boto:Data: 2022-09-02T04:32:10.774 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.775 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.776 DEBUG:boto:Params: {} 2022-09-02T04:32:10.776 DEBUG:boto:Token: None 2022-09-02T04:32:10.777 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/ 2022-09-02T04:32:10.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H4KkQJIQc6pBnHXj2KlImVmowEc= 2022-09-02T04:32:10.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H4KkQJIQc6pBnHXj2KlImVmowEc=', 'Content-Length': '0'} 2022-09-02T04:32:10.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000741a70419717b6c4-006311874a-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.782 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:10.782 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.783 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.783 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.784 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.784 DEBUG:boto:Method: GET 2022-09-02T04:32:10.785 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.785 DEBUG:boto:Data: 2022-09-02T04:32:10.786 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.787 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.788 DEBUG:boto:Params: {} 2022-09-02T04:32:10.788 DEBUG:boto:Token: None 2022-09-02T04:32:10.789 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/?versions 2022-09-02T04:32:10.790 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU= 2022-09-02T04:32:10.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU=', 'Content-Length': '0'} 2022-09-02T04:32:10.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087304d184b6be725-006311874a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.794 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:10.795 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:10.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:10.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:10.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:10.797 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:10.797 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.798 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.799 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.799 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.800 DEBUG:boto:Method: GET 2022-09-02T04:32:10.800 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.801 DEBUG:boto:Data: 2022-09-02T04:32:10.801 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.802 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.803 DEBUG:boto:Params: {} 2022-09-02T04:32:10.804 DEBUG:boto:Token: None 2022-09-02T04:32:10.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/?versions 2022-09-02T04:32:10.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU= 2022-09-02T04:32:10.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU=', 'Content-Length': '0'} 2022-09-02T04:32:10.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a41ec8d531d332c-006311874a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.809 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:10.810 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:10.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:10.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:10.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:10.813 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.813 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.814 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.814 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.815 DEBUG:boto:Method: GET 2022-09-02T04:32:10.816 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.816 DEBUG:boto:Data: 2022-09-02T04:32:10.817 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.818 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.818 DEBUG:boto:Params: {} 2022-09-02T04:32:10.819 DEBUG:boto:Token: None 2022-09-02T04:32:10.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/?versions 2022-09-02T04:32:10.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU= 2022-09-02T04:32:10.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU=', 'Content-Length': '0'} 2022-09-02T04:32:10.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9b8ef1551fb3ef-006311874a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.825 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:10.826 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.826 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.827 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.827 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.828 DEBUG:boto:Method: GET 2022-09-02T04:32:10.828 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:10.829 DEBUG:boto:Data: 2022-09-02T04:32:10.829 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.831 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.831 DEBUG:boto:Params: {} 2022-09-02T04:32:10.832 DEBUG:boto:Token: None 2022-09-02T04:32:10.833 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/?versions 2022-09-02T04:32:10.833 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU= 2022-09-02T04:32:10.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ynxFQjhTmK+2cizRB1CEGRrsiOU=', 'Content-Length': '0'} 2022-09-02T04:32:10.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015a3d6c198417047-006311874a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.838 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:10.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:10.839 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.840 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.840 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.841 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.842 DEBUG:boto:Method: GET 2022-09-02T04:32:10.842 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.843 DEBUG:boto:Data: 2022-09-02T04:32:10.844 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.845 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.845 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.846 DEBUG:boto:Params: {} 2022-09-02T04:32:10.847 DEBUG:boto:Token: None 2022-09-02T04:32:10.847 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.848 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A= 2022-09-02T04:32:10.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A=', 'Content-Length': '0'} 2022-09-02T04:32:10.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd2acec4963269c4-006311874a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.852 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.853 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.853 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.854 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.855 DEBUG:boto:Method: GET 2022-09-02T04:32:10.855 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.856 DEBUG:boto:Data: 2022-09-02T04:32:10.856 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.857 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.858 DEBUG:boto:Params: {} 2022-09-02T04:32:10.859 DEBUG:boto:Token: None 2022-09-02T04:32:10.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A= 2022-09-02T04:32:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A=', 'Content-Length': '0'} 2022-09-02T04:32:10.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000849fae600c65db54-006311874a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.864 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.865 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.866 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.866 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.867 DEBUG:boto:Method: HEAD 2022-09-02T04:32:10.867 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.868 DEBUG:boto:Data: 2022-09-02T04:32:10.868 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.870 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.871 DEBUG:boto:Params: {} 2022-09-02T04:32:10.872 DEBUG:boto:Token: None 2022-09-02T04:32:10.873 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ezkIYJMK+EAWSAUBxP2HA/yjQQI= 2022-09-02T04:32:10.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ezkIYJMK+EAWSAUBxP2HA/yjQQI=', 'Content-Length': '0'} 2022-09-02T04:32:10.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086c348f3e2ca06a8-006311874a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.878 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.879 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.879 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.880 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.880 DEBUG:boto:Method: HEAD 2022-09-02T04:32:10.881 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.881 DEBUG:boto:Data: 2022-09-02T04:32:10.882 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.883 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.884 DEBUG:boto:Params: {} 2022-09-02T04:32:10.884 DEBUG:boto:Token: None 2022-09-02T04:32:10.885 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ezkIYJMK+EAWSAUBxP2HA/yjQQI= 2022-09-02T04:32:10.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ezkIYJMK+EAWSAUBxP2HA/yjQQI=', 'Content-Length': '0'} 2022-09-02T04:32:10.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000edee413b3243d23e-006311874a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:10.891 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.891 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.892 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.892 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.893 DEBUG:boto:Method: GET 2022-09-02T04:32:10.893 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.894 DEBUG:boto:Data: 2022-09-02T04:32:10.895 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.896 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.896 DEBUG:boto:Params: {} 2022-09-02T04:32:10.897 DEBUG:boto:Token: None 2022-09-02T04:32:10.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A= 2022-09-02T04:32:10.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A=', 'Content-Length': '0'} 2022-09-02T04:32:10.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000978c31c08bfe4a22-006311874a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.902 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.902 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.903 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.904 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.905 DEBUG:boto:Method: GET 2022-09-02T04:32:10.905 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.906 DEBUG:boto:Data: 2022-09-02T04:32:10.907 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.908 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.908 DEBUG:boto:Params: {} 2022-09-02T04:32:10.909 DEBUG:boto:Token: None 2022-09-02T04:32:10.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A= 2022-09-02T04:32:10.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A=', 'Content-Length': '0'} 2022-09-02T04:32:10.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e0967f11ee9112e-006311874a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.914 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.914 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.915 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.916 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.917 DEBUG:boto:Method: GET 2022-09-02T04:32:10.918 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.919 DEBUG:boto:Data: 2022-09-02T04:32:10.920 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.921 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.922 DEBUG:boto:Params: {} 2022-09-02T04:32:10.923 DEBUG:boto:Token: None 2022-09-02T04:32:10.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/ 2022-09-02T04:32:10.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:siDAB7VX4OrO8FWDhzgmUtkQHMs= 2022-09-02T04:32:10.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:siDAB7VX4OrO8FWDhzgmUtkQHMs=', 'Content-Length': '0'} 2022-09-02T04:32:10.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000941e90e474fd3306-006311874a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.929 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:10.930 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.931 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.932 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.932 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.933 DEBUG:boto:Method: GET 2022-09-02T04:32:10.933 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.934 DEBUG:boto:Data: 2022-09-02T04:32:10.934 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.935 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.936 DEBUG:boto:Params: {} 2022-09-02T04:32:10.936 DEBUG:boto:Token: None 2022-09-02T04:32:10.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/ 2022-09-02T04:32:10.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:siDAB7VX4OrO8FWDhzgmUtkQHMs= 2022-09-02T04:32:10.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:siDAB7VX4OrO8FWDhzgmUtkQHMs=', 'Content-Length': '0'} 2022-09-02T04:32:10.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f92e1dfaa0afaba-006311874a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.941 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:10.942 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.943 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.943 DEBUG:boto:Method: HEAD 2022-09-02T04:32:10.944 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:32:10.944 DEBUG:boto:Data: 2022-09-02T04:32:10.945 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.946 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.946 DEBUG:boto:Params: {} 2022-09-02T04:32:10.947 DEBUG:boto:Token: None 2022-09-02T04:32:10.948 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy 2022-09-02T04:32:10.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DXdmPT2XYFHLqy7if3bZaOrBKRQ= 2022-09-02T04:32:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DXdmPT2XYFHLqy7if3bZaOrBKRQ=', 'Content-Length': '0'} 2022-09-02T04:32:10.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad9b4415a2bb6dfa-006311874a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.952 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.952 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.953 DEBUG:boto:Method: HEAD 2022-09-02T04:32:10.953 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:32:10.954 DEBUG:boto:Data: 2022-09-02T04:32:10.954 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.955 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.956 DEBUG:boto:Params: {} 2022-09-02T04:32:10.956 DEBUG:boto:Token: None 2022-09-02T04:32:10.957 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy 2022-09-02T04:32:10.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DXdmPT2XYFHLqy7if3bZaOrBKRQ= 2022-09-02T04:32:10.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DXdmPT2XYFHLqy7if3bZaOrBKRQ=', 'Content-Length': '0'} 2022-09-02T04:32:10.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e61f3e2681ece61-006311874a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:10.962 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.963 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.963 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.964 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.964 DEBUG:boto:Method: GET 2022-09-02T04:32:10.965 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.965 DEBUG:boto:Data: 2022-09-02T04:32:10.966 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.967 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.967 DEBUG:boto:Params: {} 2022-09-02T04:32:10.968 DEBUG:boto:Token: None 2022-09-02T04:32:10.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A= 2022-09-02T04:32:10.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A=', 'Content-Length': '0'} 2022-09-02T04:32:10.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000895bfa51c5b0687e-006311874a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.972 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:10.973 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:10.973 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.974 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.975 DEBUG:boto:Method: GET 2022-09-02T04:32:10.976 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.977 DEBUG:boto:Data: 2022-09-02T04:32:10.978 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:10.980 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.981 DEBUG:boto:Params: {} 2022-09-02T04:32:10.982 DEBUG:boto:Token: None 2022-09-02T04:32:10.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:10.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A= 2022-09-02T04:32:10.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zXhejjLIZ0eAF1lSGkptoXt+w7A=', 'Content-Length': '0'} 2022-09-02T04:32:10.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088ae4019ad2deb9c-006311874a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:10.988 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:10.989 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:10.989 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.990 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.990 DEBUG:boto:Method: GET 2022-09-02T04:32:10.991 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:10.991 DEBUG:boto:Data: 2022-09-02T04:32:10.992 DEBUG:boto:Headers: {} 2022-09-02T04:32:10.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:10.993 DEBUG:boto:Port: 8000 2022-09-02T04:32:10.994 DEBUG:boto:Params: {} 2022-09-02T04:32:10.994 DEBUG:boto:Token: None 2022-09-02T04:32:10.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:10 GMT /xljbkf-18/ 2022-09-02T04:32:10.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:siDAB7VX4OrO8FWDhzgmUtkQHMs= 2022-09-02T04:32:10.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:siDAB7VX4OrO8FWDhzgmUtkQHMs=', 'Content-Length': '0'} 2022-09-02T04:32:10.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb6b8ef0176827c-006311874a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.046 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.047 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.048 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.048 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.049 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.049 DEBUG:boto:Method: GET 2022-09-02T04:32:11.050 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.050 DEBUG:boto:Data: 2022-09-02T04:32:11.051 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.052 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.052 DEBUG:boto:Params: {} 2022-09-02T04:32:11.053 DEBUG:boto:Token: None 2022-09-02T04:32:11.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e8867426074502-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.058 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:11.060 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.060 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.061 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.061 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.062 DEBUG:boto:Method: GET 2022-09-02T04:32:11.062 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.063 DEBUG:boto:Data: 2022-09-02T04:32:11.063 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.064 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.065 DEBUG:boto:Params: {} 2022-09-02T04:32:11.065 DEBUG:boto:Token: None 2022-09-02T04:32:11.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k= 2022-09-02T04:32:11.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k=', 'Content-Length': '0'} 2022-09-02T04:32:11.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5d1682116d4ba68-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.070 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.071 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.071 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.072 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.072 DEBUG:boto:Method: GET 2022-09-02T04:32:11.073 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.073 DEBUG:boto:Data: 2022-09-02T04:32:11.074 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.075 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.076 DEBUG:boto:Params: {} 2022-09-02T04:32:11.076 DEBUG:boto:Token: None 2022-09-02T04:32:11.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k= 2022-09-02T04:32:11.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k=', 'Content-Length': '0'} 2022-09-02T04:32:11.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b11ed58efa34ab34-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.081 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.082 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.082 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.083 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.083 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.084 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.084 DEBUG:boto:Data: 2022-09-02T04:32:11.085 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.086 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.086 DEBUG:boto:Params: {} 2022-09-02T04:32:11.087 DEBUG:boto:Token: None 2022-09-02T04:32:11.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0gKMokup4dp0GfAxv/FBOF95DPk= 2022-09-02T04:32:11.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0gKMokup4dp0GfAxv/FBOF95DPk=', 'Content-Length': '0'} 2022-09-02T04:32:11.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5dff2d26c015437-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.091 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.092 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.092 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.093 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.093 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.094 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.094 DEBUG:boto:Data: 2022-09-02T04:32:11.095 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.096 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.096 DEBUG:boto:Params: {} 2022-09-02T04:32:11.097 DEBUG:boto:Token: None 2022-09-02T04:32:11.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0gKMokup4dp0GfAxv/FBOF95DPk= 2022-09-02T04:32:11.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0gKMokup4dp0GfAxv/FBOF95DPk=', 'Content-Length': '0'} 2022-09-02T04:32:11.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b653796ceb46a8b7-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:11.102 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.102 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.103 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.103 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.104 DEBUG:boto:Method: GET 2022-09-02T04:32:11.104 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.105 DEBUG:boto:Data: 2022-09-02T04:32:11.106 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.106 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.107 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.107 DEBUG:boto:Params: {} 2022-09-02T04:32:11.108 DEBUG:boto:Token: None 2022-09-02T04:32:11.109 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k= 2022-09-02T04:32:11.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k=', 'Content-Length': '0'} 2022-09-02T04:32:11.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f572aea9228ac5c0-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.113 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.113 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.114 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.115 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.115 DEBUG:boto:Method: GET 2022-09-02T04:32:11.116 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.116 DEBUG:boto:Data: 2022-09-02T04:32:11.117 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.118 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.118 DEBUG:boto:Params: {} 2022-09-02T04:32:11.119 DEBUG:boto:Token: None 2022-09-02T04:32:11.120 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.120 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k= 2022-09-02T04:32:11.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k=', 'Content-Length': '0'} 2022-09-02T04:32:11.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1ca8c880a78fade-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.123 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.124 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.125 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.125 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.126 DEBUG:boto:Method: GET 2022-09-02T04:32:11.126 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.127 DEBUG:boto:Data: 2022-09-02T04:32:11.127 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.128 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.129 DEBUG:boto:Params: {} 2022-09-02T04:32:11.129 DEBUG:boto:Token: None 2022-09-02T04:32:11.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b70ffdd619a04fa-006311874b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.134 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.135 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.136 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.137 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.137 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.138 DEBUG:boto:Method: GET 2022-09-02T04:32:11.138 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.139 DEBUG:boto:Data: 2022-09-02T04:32:11.139 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.140 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.141 DEBUG:boto:Params: {} 2022-09-02T04:32:11.141 DEBUG:boto:Token: None 2022-09-02T04:32:11.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000601252b6a5a0f70a-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.146 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.147 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.147 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.148 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.148 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.149 DEBUG:boto:Data: 2022-09-02T04:32:11.149 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.150 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.151 DEBUG:boto:Params: {} 2022-09-02T04:32:11.151 DEBUG:boto:Token: None 2022-09-02T04:32:11.152 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mz4gNSKoK3BP6dP5eid3Mvuua5o= 2022-09-02T04:32:11.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mz4gNSKoK3BP6dP5eid3Mvuua5o=', 'Content-Length': '0'} 2022-09-02T04:32:11.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f313d8936bb9514-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.156 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.157 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.157 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.158 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.158 DEBUG:boto:Data: 2022-09-02T04:32:11.159 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.160 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.160 DEBUG:boto:Params: {} 2022-09-02T04:32:11.161 DEBUG:boto:Token: None 2022-09-02T04:32:11.161 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mz4gNSKoK3BP6dP5eid3Mvuua5o= 2022-09-02T04:32:11.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mz4gNSKoK3BP6dP5eid3Mvuua5o=', 'Content-Length': '0'} 2022-09-02T04:32:11.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001956e3d149b26ab3-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:11.166 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.167 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.168 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.168 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.169 DEBUG:boto:Method: GET 2022-09-02T04:32:11.170 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.171 DEBUG:boto:Data: 2022-09-02T04:32:11.172 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.174 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.175 DEBUG:boto:Params: {} 2022-09-02T04:32:11.175 DEBUG:boto:Token: None 2022-09-02T04:32:11.176 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k= 2022-09-02T04:32:11.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k=', 'Content-Length': '0'} 2022-09-02T04:32:11.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d47cb662bd64cc10-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.222 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.223 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:11.223 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.224 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.224 DEBUG:boto:Method: GET 2022-09-02T04:32:11.225 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.225 DEBUG:boto:Data: 2022-09-02T04:32:11.226 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.227 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.228 DEBUG:boto:Params: {} 2022-09-02T04:32:11.228 DEBUG:boto:Token: None 2022-09-02T04:32:11.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:11.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k= 2022-09-02T04:32:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rixXsieXVR1dc6I6A3ZzU++Px/k=', 'Content-Length': '0'} 2022-09-02T04:32:11.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000690d230d16bd7cc1-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.233 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.234 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.234 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.235 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.235 DEBUG:boto:Method: GET 2022-09-02T04:32:11.236 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.236 DEBUG:boto:Data: 2022-09-02T04:32:11.237 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.238 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.238 DEBUG:boto:Params: {} 2022-09-02T04:32:11.239 DEBUG:boto:Token: None 2022-09-02T04:32:11.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffca48c273b8683b-006311874b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.244 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.245 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.245 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.246 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.246 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.247 DEBUG:boto:Method: GET 2022-09-02T04:32:11.247 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:11.248 DEBUG:boto:Data: 2022-09-02T04:32:11.248 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.249 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.250 DEBUG:boto:Params: {} 2022-09-02T04:32:11.251 DEBUG:boto:Token: None 2022-09-02T04:32:11.251 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001217e620f5abbcbf-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.256 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:11.257 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.257 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.258 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.259 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.259 DEBUG:boto:Method: GET 2022-09-02T04:32:11.260 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.260 DEBUG:boto:Data: 2022-09-02T04:32:11.261 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.262 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.262 DEBUG:boto:Params: {} 2022-09-02T04:32:11.263 DEBUG:boto:Token: None 2022-09-02T04:32:11.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4= 2022-09-02T04:32:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4=', 'Content-Length': '0'} 2022-09-02T04:32:11.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dce0317d92253b79-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.267 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.268 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.268 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.269 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.270 DEBUG:boto:Method: GET 2022-09-02T04:32:11.270 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.271 DEBUG:boto:Data: 2022-09-02T04:32:11.271 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.272 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.273 DEBUG:boto:Params: {} 2022-09-02T04:32:11.273 DEBUG:boto:Token: None 2022-09-02T04:32:11.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4= 2022-09-02T04:32:11.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4=', 'Content-Length': '0'} 2022-09-02T04:32:11.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dea6f6954c753c96-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.326 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.327 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.328 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.329 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.330 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.331 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.332 DEBUG:boto:Data: 2022-09-02T04:32:11.333 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.334 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.335 DEBUG:boto:Params: {} 2022-09-02T04:32:11.335 DEBUG:boto:Token: None 2022-09-02T04:32:11.336 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CI3H6h3kd3V+4eyjkuOVvw6Rds4= 2022-09-02T04:32:11.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CI3H6h3kd3V+4eyjkuOVvw6Rds4=', 'Content-Length': '0'} 2022-09-02T04:32:11.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ca2824b4fa7a187-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.340 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.340 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.341 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.341 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.342 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.342 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.343 DEBUG:boto:Data: 2022-09-02T04:32:11.343 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.344 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.345 DEBUG:boto:Params: {} 2022-09-02T04:32:11.345 DEBUG:boto:Token: None 2022-09-02T04:32:11.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CI3H6h3kd3V+4eyjkuOVvw6Rds4= 2022-09-02T04:32:11.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CI3H6h3kd3V+4eyjkuOVvw6Rds4=', 'Content-Length': '0'} 2022-09-02T04:32:11.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031016e0365b64d13-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:11.350 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.351 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.351 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.352 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.352 DEBUG:boto:Method: GET 2022-09-02T04:32:11.353 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.353 DEBUG:boto:Data: 2022-09-02T04:32:11.354 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.355 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.356 DEBUG:boto:Params: {} 2022-09-02T04:32:11.356 DEBUG:boto:Token: None 2022-09-02T04:32:11.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4= 2022-09-02T04:32:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4=', 'Content-Length': '0'} 2022-09-02T04:32:11.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000befd01caad1036c6-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.361 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.361 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.362 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.363 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.363 DEBUG:boto:Method: GET 2022-09-02T04:32:11.364 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.364 DEBUG:boto:Data: 2022-09-02T04:32:11.365 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.366 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.366 DEBUG:boto:Params: {} 2022-09-02T04:32:11.367 DEBUG:boto:Token: None 2022-09-02T04:32:11.368 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4= 2022-09-02T04:32:11.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4=', 'Content-Length': '0'} 2022-09-02T04:32:11.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090f8b0db2a65908c-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.372 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.372 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.373 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.373 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.374 DEBUG:boto:Method: GET 2022-09-02T04:32:11.374 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.375 DEBUG:boto:Data: 2022-09-02T04:32:11.375 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.376 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.377 DEBUG:boto:Params: {} 2022-09-02T04:32:11.378 DEBUG:boto:Token: None 2022-09-02T04:32:11.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000236fcfe4f5b4eed7-006311874b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.384 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.385 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.386 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.387 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.387 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.388 DEBUG:boto:Method: GET 2022-09-02T04:32:11.388 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.389 DEBUG:boto:Data: 2022-09-02T04:32:11.389 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.396 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.396 DEBUG:boto:Params: {} 2022-09-02T04:32:11.397 DEBUG:boto:Token: None 2022-09-02T04:32:11.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfcbd706dff822a8-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.402 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.403 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.404 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.404 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.405 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.405 DEBUG:boto:Data: 2022-09-02T04:32:11.406 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.407 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.407 DEBUG:boto:Params: {} 2022-09-02T04:32:11.408 DEBUG:boto:Token: None 2022-09-02T04:32:11.408 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RFoIwTR+4UeVHIVLjm5MRuj9OSA= 2022-09-02T04:32:11.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RFoIwTR+4UeVHIVLjm5MRuj9OSA=', 'Content-Length': '0'} 2022-09-02T04:32:11.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c59a68df22623f5-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.412 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.413 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.413 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.414 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.414 DEBUG:boto:Data: 2022-09-02T04:32:11.415 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.416 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.416 DEBUG:boto:Params: {} 2022-09-02T04:32:11.417 DEBUG:boto:Token: None 2022-09-02T04:32:11.418 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RFoIwTR+4UeVHIVLjm5MRuj9OSA= 2022-09-02T04:32:11.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RFoIwTR+4UeVHIVLjm5MRuj9OSA=', 'Content-Length': '0'} 2022-09-02T04:32:11.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000612a6eb14cea3e12-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:11.422 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.423 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.423 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.424 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.424 DEBUG:boto:Method: GET 2022-09-02T04:32:11.425 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.425 DEBUG:boto:Data: 2022-09-02T04:32:11.426 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.427 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.428 DEBUG:boto:Params: {} 2022-09-02T04:32:11.428 DEBUG:boto:Token: None 2022-09-02T04:32:11.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4= 2022-09-02T04:32:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4=', 'Content-Length': '0'} 2022-09-02T04:32:11.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010fc0985e1baa5ce-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.474 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.475 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:11.475 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.476 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.477 DEBUG:boto:Method: GET 2022-09-02T04:32:11.477 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.478 DEBUG:boto:Data: 2022-09-02T04:32:11.478 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.479 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.480 DEBUG:boto:Params: {} 2022-09-02T04:32:11.481 DEBUG:boto:Token: None 2022-09-02T04:32:11.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:11.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4= 2022-09-02T04:32:11.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sHkpOXDVZvw6O83lAEdWCrtpRD4=', 'Content-Length': '0'} 2022-09-02T04:32:11.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9e3621b16f0f8aa-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.486 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.487 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.487 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.488 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.488 DEBUG:boto:Method: GET 2022-09-02T04:32:11.489 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.490 DEBUG:boto:Data: 2022-09-02T04:32:11.490 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.491 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.492 DEBUG:boto:Params: {} 2022-09-02T04:32:11.492 DEBUG:boto:Token: None 2022-09-02T04:32:11.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba40a64eebcd4c12-006311874b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.498 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.498 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.499 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.500 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.500 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.501 DEBUG:boto:Method: GET 2022-09-02T04:32:11.501 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:11.502 DEBUG:boto:Data: 2022-09-02T04:32:11.502 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.504 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.504 DEBUG:boto:Params: {} 2022-09-02T04:32:11.505 DEBUG:boto:Token: None 2022-09-02T04:32:11.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0cee9487d324b01-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.510 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:11.512 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.512 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.513 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.513 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.514 DEBUG:boto:Method: GET 2022-09-02T04:32:11.514 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.515 DEBUG:boto:Data: 2022-09-02T04:32:11.515 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.517 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.517 DEBUG:boto:Params: {} 2022-09-02T04:32:11.518 DEBUG:boto:Token: None 2022-09-02T04:32:11.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU= 2022-09-02T04:32:11.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU=', 'Content-Length': '0'} 2022-09-02T04:32:11.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000143b3bfe28d2a7d2-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.523 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.524 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.524 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.525 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.526 DEBUG:boto:Method: GET 2022-09-02T04:32:11.526 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.527 DEBUG:boto:Data: 2022-09-02T04:32:11.528 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.529 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.530 DEBUG:boto:Params: {} 2022-09-02T04:32:11.530 DEBUG:boto:Token: None 2022-09-02T04:32:11.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU= 2022-09-02T04:32:11.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU=', 'Content-Length': '0'} 2022-09-02T04:32:11.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059a2be00b046d1d4-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.536 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.536 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.537 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.537 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.538 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.539 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.539 DEBUG:boto:Data: 2022-09-02T04:32:11.540 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.541 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.541 DEBUG:boto:Params: {} 2022-09-02T04:32:11.542 DEBUG:boto:Token: None 2022-09-02T04:32:11.542 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yEF6WJFp3coojMjBflchnTsW4Q8= 2022-09-02T04:32:11.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yEF6WJFp3coojMjBflchnTsW4Q8=', 'Content-Length': '0'} 2022-09-02T04:32:11.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb3c0c6279b55ced-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.546 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.547 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.548 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.548 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.549 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.549 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.550 DEBUG:boto:Data: 2022-09-02T04:32:11.550 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.552 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.552 DEBUG:boto:Params: {} 2022-09-02T04:32:11.553 DEBUG:boto:Token: None 2022-09-02T04:32:11.553 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yEF6WJFp3coojMjBflchnTsW4Q8= 2022-09-02T04:32:11.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yEF6WJFp3coojMjBflchnTsW4Q8=', 'Content-Length': '0'} 2022-09-02T04:32:11.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fe2f9ac3d618ed9-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:11.558 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.559 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.560 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.560 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.561 DEBUG:boto:Method: GET 2022-09-02T04:32:11.561 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.562 DEBUG:boto:Data: 2022-09-02T04:32:11.563 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.564 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.564 DEBUG:boto:Params: {} 2022-09-02T04:32:11.565 DEBUG:boto:Token: None 2022-09-02T04:32:11.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU= 2022-09-02T04:32:11.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU=', 'Content-Length': '0'} 2022-09-02T04:32:11.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dddbb9959199038-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.570 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.571 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.571 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.572 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.572 DEBUG:boto:Method: GET 2022-09-02T04:32:11.573 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.573 DEBUG:boto:Data: 2022-09-02T04:32:11.574 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.575 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.576 DEBUG:boto:Params: {} 2022-09-02T04:32:11.578 DEBUG:boto:Token: None 2022-09-02T04:32:11.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU= 2022-09-02T04:32:11.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU=', 'Content-Length': '0'} 2022-09-02T04:32:11.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee2a5983840ff0f1-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.583 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.584 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.584 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.585 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.586 DEBUG:boto:Method: GET 2022-09-02T04:32:11.586 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.587 DEBUG:boto:Data: 2022-09-02T04:32:11.587 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.588 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.589 DEBUG:boto:Params: {} 2022-09-02T04:32:11.589 DEBUG:boto:Token: None 2022-09-02T04:32:11.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030baca862ba1d203-006311874b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.595 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.596 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.597 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.598 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.599 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.600 DEBUG:boto:Method: GET 2022-09-02T04:32:11.601 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.602 DEBUG:boto:Data: 2022-09-02T04:32:11.603 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.605 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.605 DEBUG:boto:Params: {} 2022-09-02T04:32:11.607 DEBUG:boto:Token: None 2022-09-02T04:32:11.607 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021fecd73b6ca240d-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.658 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.659 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.660 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.660 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.661 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.662 DEBUG:boto:Data: 2022-09-02T04:32:11.662 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.663 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.664 DEBUG:boto:Params: {} 2022-09-02T04:32:11.664 DEBUG:boto:Token: None 2022-09-02T04:32:11.665 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:km8xVnfaoInhclRB8ktvCmR+7pU= 2022-09-02T04:32:11.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:km8xVnfaoInhclRB8ktvCmR+7pU=', 'Content-Length': '0'} 2022-09-02T04:32:11.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003615ee395b3f03e5-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.669 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.670 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.670 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.671 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.671 DEBUG:boto:Data: 2022-09-02T04:32:11.672 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.673 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.673 DEBUG:boto:Params: {} 2022-09-02T04:32:11.674 DEBUG:boto:Token: None 2022-09-02T04:32:11.675 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:km8xVnfaoInhclRB8ktvCmR+7pU= 2022-09-02T04:32:11.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:km8xVnfaoInhclRB8ktvCmR+7pU=', 'Content-Length': '0'} 2022-09-02T04:32:11.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047274e2e2e5f72e0-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:11.679 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.680 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.680 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.681 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.681 DEBUG:boto:Method: GET 2022-09-02T04:32:11.682 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.683 DEBUG:boto:Data: 2022-09-02T04:32:11.683 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.684 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.685 DEBUG:boto:Params: {} 2022-09-02T04:32:11.685 DEBUG:boto:Token: None 2022-09-02T04:32:11.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU= 2022-09-02T04:32:11.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU=', 'Content-Length': '0'} 2022-09-02T04:32:11.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003569c5cc90ccd4f-006311874b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.690 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.691 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:11.691 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.692 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.692 DEBUG:boto:Method: GET 2022-09-02T04:32:11.693 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.693 DEBUG:boto:Data: 2022-09-02T04:32:11.694 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.695 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.696 DEBUG:boto:Params: {} 2022-09-02T04:32:11.696 DEBUG:boto:Token: None 2022-09-02T04:32:11.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:11.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU= 2022-09-02T04:32:11.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zBRMpKVOZCrgs7dKu9jQXZGmwtU=', 'Content-Length': '0'} 2022-09-02T04:32:11.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000458edd15366ef503-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.701 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.701 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.702 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.703 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.703 DEBUG:boto:Method: GET 2022-09-02T04:32:11.704 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.704 DEBUG:boto:Data: 2022-09-02T04:32:11.705 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:11.706 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.706 DEBUG:boto:Params: {} 2022-09-02T04:32:11.707 DEBUG:boto:Token: None 2022-09-02T04:32:11.707 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000071a494b811b1b42-006311874b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.712 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.713 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.713 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.714 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.714 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.715 DEBUG:boto:Method: GET 2022-09-02T04:32:11.715 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:11.716 DEBUG:boto:Data: 2022-09-02T04:32:11.717 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.719 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.720 DEBUG:boto:Params: {} 2022-09-02T04:32:11.721 DEBUG:boto:Token: None 2022-09-02T04:32:11.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce25caa2eda1600f-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.729 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-18 zones={test-zone2, test-zone1} 2022-09-02T04:32:11.730 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-18 zones={test-zone3, test-zone1} 2022-09-02T04:32:11.730 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.731 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.731 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.732 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:32:11.732 DEBUG:boto:Data: 2022-09-02T04:32:11.733 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.734 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.735 DEBUG:boto:Params: {} 2022-09-02T04:32:11.735 DEBUG:boto:Token: None 2022-09-02T04:32:11.736 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B4KmwoVV6xB1WRkyPBCwS2bHvwU= 2022-09-02T04:32:11.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B4KmwoVV6xB1WRkyPBCwS2bHvwU=', 'Content-Length': '0'} 2022-09-02T04:32:11.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ce180237dc2fd19-006311874b-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.742 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.743 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.744 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.745 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:32:11.745 DEBUG:boto:Data: 2022-09-02T04:32:11.746 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.748 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.748 DEBUG:boto:Params: {} 2022-09-02T04:32:11.749 DEBUG:boto:Token: None 2022-09-02T04:32:11.749 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B4KmwoVV6xB1WRkyPBCwS2bHvwU= 2022-09-02T04:32:11.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B4KmwoVV6xB1WRkyPBCwS2bHvwU=', 'Content-Length': '0'} 2022-09-02T04:32:11.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2e6094eed45e91f-006311874b-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.753 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:11.754 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.755 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.755 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.756 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.756 DEBUG:boto:Method: GET 2022-09-02T04:32:11.757 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.757 DEBUG:boto:Data: 2022-09-02T04:32:11.758 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.759 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.759 DEBUG:boto:Params: {} 2022-09-02T04:32:11.760 DEBUG:boto:Token: None 2022-09-02T04:32:11.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/?versions 2022-09-02T04:32:11.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA= 2022-09-02T04:32:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA=', 'Content-Length': '0'} 2022-09-02T04:32:11.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f83b0263734b6c04-006311874b-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.765 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.767 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:11.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:11.769 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:11.769 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.770 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.770 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.771 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.771 DEBUG:boto:Method: GET 2022-09-02T04:32:11.772 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.773 DEBUG:boto:Data: 2022-09-02T04:32:11.773 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.774 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.775 DEBUG:boto:Params: {} 2022-09-02T04:32:11.775 DEBUG:boto:Token: None 2022-09-02T04:32:11.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/?versions 2022-09-02T04:32:11.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA= 2022-09-02T04:32:11.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA=', 'Content-Length': '0'} 2022-09-02T04:32:11.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e656053c198ef5be-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.781 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.781 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:11.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:11.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:11.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:11.784 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.784 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.785 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.785 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.786 DEBUG:boto:Method: GET 2022-09-02T04:32:11.786 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.787 DEBUG:boto:Data: 2022-09-02T04:32:11.787 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.788 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.789 DEBUG:boto:Params: {} 2022-09-02T04:32:11.790 DEBUG:boto:Token: None 2022-09-02T04:32:11.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/?versions 2022-09-02T04:32:11.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA= 2022-09-02T04:32:11.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA=', 'Content-Length': '0'} 2022-09-02T04:32:11.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bd593f6f6ac3eb-006311874b-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.796 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.796 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.797 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.798 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.798 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.799 DEBUG:boto:Method: GET 2022-09-02T04:32:11.799 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:11.800 DEBUG:boto:Data: 2022-09-02T04:32:11.800 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.801 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.802 DEBUG:boto:Params: {} 2022-09-02T04:32:11.803 DEBUG:boto:Token: None 2022-09-02T04:32:11.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/?versions 2022-09-02T04:32:11.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA= 2022-09-02T04:32:11.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NWDGrvNkYwaHed6o1ihnmLwbGUA=', 'Content-Length': '0'} 2022-09-02T04:32:11.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008295d1b2699f04f0-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.810 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:11.812 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.813 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.813 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.814 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.814 DEBUG:boto:Method: GET 2022-09-02T04:32:11.815 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.815 DEBUG:boto:Data: 2022-09-02T04:32:11.816 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.817 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.818 DEBUG:boto:Params: {} 2022-09-02T04:32:11.818 DEBUG:boto:Token: None 2022-09-02T04:32:11.819 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE= 2022-09-02T04:32:11.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE=', 'Content-Length': '0'} 2022-09-02T04:32:11.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cfeccd57c78ac934-006311874b-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.823 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.824 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.824 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.825 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.825 DEBUG:boto:Method: GET 2022-09-02T04:32:11.826 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.827 DEBUG:boto:Data: 2022-09-02T04:32:11.827 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.828 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.829 DEBUG:boto:Params: {} 2022-09-02T04:32:11.829 DEBUG:boto:Token: None 2022-09-02T04:32:11.830 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE= 2022-09-02T04:32:11.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE=', 'Content-Length': '0'} 2022-09-02T04:32:11.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019a97f56a728b01f-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.834 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.834 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.835 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.835 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.836 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.836 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.837 DEBUG:boto:Data: 2022-09-02T04:32:11.837 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.838 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.839 DEBUG:boto:Params: {} 2022-09-02T04:32:11.840 DEBUG:boto:Token: None 2022-09-02T04:32:11.840 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eW9SYU13T8I8XqTVAPSfhIGGui8= 2022-09-02T04:32:11.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eW9SYU13T8I8XqTVAPSfhIGGui8=', 'Content-Length': '0'} 2022-09-02T04:32:11.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8aca6139ae7bcef-006311874b-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.844 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.844 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.845 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.845 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.846 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.846 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.847 DEBUG:boto:Data: 2022-09-02T04:32:11.847 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.848 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.849 DEBUG:boto:Params: {} 2022-09-02T04:32:11.849 DEBUG:boto:Token: None 2022-09-02T04:32:11.850 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eW9SYU13T8I8XqTVAPSfhIGGui8= 2022-09-02T04:32:11.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eW9SYU13T8I8XqTVAPSfhIGGui8=', 'Content-Length': '0'} 2022-09-02T04:32:11.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010a0ae5687a8fb35-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:11.854 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.855 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.855 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.856 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.856 DEBUG:boto:Method: GET 2022-09-02T04:32:11.857 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.857 DEBUG:boto:Data: 2022-09-02T04:32:11.858 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.859 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.860 DEBUG:boto:Params: {} 2022-09-02T04:32:11.860 DEBUG:boto:Token: None 2022-09-02T04:32:11.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE= 2022-09-02T04:32:11.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE=', 'Content-Length': '0'} 2022-09-02T04:32:11.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005832d9900408f4b0-006311874b-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.865 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.865 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.866 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.867 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.867 DEBUG:boto:Method: GET 2022-09-02T04:32:11.868 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.868 DEBUG:boto:Data: 2022-09-02T04:32:11.869 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.870 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.870 DEBUG:boto:Params: {} 2022-09-02T04:32:11.871 DEBUG:boto:Token: None 2022-09-02T04:32:11.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE= 2022-09-02T04:32:11.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE=', 'Content-Length': '0'} 2022-09-02T04:32:11.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000087dc21f4eb04451-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.876 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.876 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.877 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.877 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.878 DEBUG:boto:Method: GET 2022-09-02T04:32:11.878 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.879 DEBUG:boto:Data: 2022-09-02T04:32:11.880 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.881 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.881 DEBUG:boto:Params: {} 2022-09-02T04:32:11.882 DEBUG:boto:Token: None 2022-09-02T04:32:11.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b80719a0542d53d9-006311874b-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.930 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.931 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.932 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.932 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.933 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.933 DEBUG:boto:Method: GET 2022-09-02T04:32:11.934 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.935 DEBUG:boto:Data: 2022-09-02T04:32:11.935 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.936 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.937 DEBUG:boto:Params: {} 2022-09-02T04:32:11.937 DEBUG:boto:Token: None 2022-09-02T04:32:11.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1070dd848228cdb-006311874b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.942 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.943 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.943 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.944 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.944 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:32:11.945 DEBUG:boto:Data: 2022-09-02T04:32:11.945 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.947 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.947 DEBUG:boto:Params: {} 2022-09-02T04:32:11.948 DEBUG:boto:Token: None 2022-09-02T04:32:11.948 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy 2022-09-02T04:32:11.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QzZDbFla+y8Mb7PKD3EQsEm5TfA= 2022-09-02T04:32:11.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QzZDbFla+y8Mb7PKD3EQsEm5TfA=', 'Content-Length': '0'} 2022-09-02T04:32:11.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f83008d4527456f-006311874b-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.952 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.953 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.953 DEBUG:boto:Method: HEAD 2022-09-02T04:32:11.954 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:32:11.954 DEBUG:boto:Data: 2022-09-02T04:32:11.955 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.956 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.957 DEBUG:boto:Params: {} 2022-09-02T04:32:11.957 DEBUG:boto:Token: None 2022-09-02T04:32:11.958 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy 2022-09-02T04:32:11.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QzZDbFla+y8Mb7PKD3EQsEm5TfA= 2022-09-02T04:32:11.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QzZDbFla+y8Mb7PKD3EQsEm5TfA=', 'Content-Length': '0'} 2022-09-02T04:32:11.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e5b97d1a52bc016-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:11.962 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.963 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.964 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.964 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.965 DEBUG:boto:Method: GET 2022-09-02T04:32:11.965 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.966 DEBUG:boto:Data: 2022-09-02T04:32:11.966 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.967 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.968 DEBUG:boto:Params: {} 2022-09-02T04:32:11.968 DEBUG:boto:Token: None 2022-09-02T04:32:11.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE= 2022-09-02T04:32:11.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE=', 'Content-Length': '0'} 2022-09-02T04:32:11.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3744981e13e916f-006311874b-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.973 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:11.974 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:11.974 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.975 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.975 DEBUG:boto:Method: GET 2022-09-02T04:32:11.976 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.977 DEBUG:boto:Data: 2022-09-02T04:32:11.977 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:11.978 DEBUG:boto:Port: 8000 2022-09-02T04:32:11.979 DEBUG:boto:Params: {} 2022-09-02T04:32:11.979 DEBUG:boto:Token: None 2022-09-02T04:32:11.980 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:11.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE= 2022-09-02T04:32:11.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJzAF0vOELVtC7YHencJHQ9temE=', 'Content-Length': '0'} 2022-09-02T04:32:11.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001576fbb697337deb-006311874b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.984 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.985 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.985 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.986 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.986 DEBUG:boto:Method: GET 2022-09-02T04:32:11.987 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.988 DEBUG:boto:Data: 2022-09-02T04:32:11.988 DEBUG:boto:Headers: {} 2022-09-02T04:32:11.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:11.989 DEBUG:boto:Port: 8001 2022-09-02T04:32:11.990 DEBUG:boto:Params: {} 2022-09-02T04:32:11.990 DEBUG:boto:Token: None 2022-09-02T04:32:11.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:11 GMT /xljbkf-18/ 2022-09-02T04:32:11.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8= 2022-09-02T04:32:11.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TJTZnqMQmau/6H5CeXy2/CS7tR8=', 'Content-Length': '0'} 2022-09-02T04:32:11.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c09cec11fa326d8-006311874b-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:11.995 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:11.996 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:11.996 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:11.997 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.997 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.998 DEBUG:boto:Method: GET 2022-09-02T04:32:11.999 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:11.999 DEBUG:boto:Data: 2022-09-02T04:32:12.000 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.001 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.001 DEBUG:boto:Params: {} 2022-09-02T04:32:12.002 DEBUG:boto:Token: None 2022-09-02T04:32:12.002 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080ae7df2fce5cf21-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.007 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:12.008 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.009 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.009 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.010 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.010 DEBUG:boto:Method: GET 2022-09-02T04:32:12.011 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.011 DEBUG:boto:Data: 2022-09-02T04:32:12.012 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.013 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.014 DEBUG:boto:Params: {} 2022-09-02T04:32:12.014 DEBUG:boto:Token: None 2022-09-02T04:32:12.015 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8= 2022-09-02T04:32:12.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8=', 'Content-Length': '0'} 2022-09-02T04:32:12.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000187318eaf705cfa8-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.019 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.020 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.020 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.021 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.021 DEBUG:boto:Method: GET 2022-09-02T04:32:12.022 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.022 DEBUG:boto:Data: 2022-09-02T04:32:12.023 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.024 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.025 DEBUG:boto:Params: {} 2022-09-02T04:32:12.025 DEBUG:boto:Token: None 2022-09-02T04:32:12.026 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8= 2022-09-02T04:32:12.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8=', 'Content-Length': '0'} 2022-09-02T04:32:12.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040b61bb264d1606c-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.070 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.071 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.071 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.072 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.072 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.073 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.074 DEBUG:boto:Data: 2022-09-02T04:32:12.074 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.075 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.076 DEBUG:boto:Params: {} 2022-09-02T04:32:12.076 DEBUG:boto:Token: None 2022-09-02T04:32:12.077 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+qHnkGd7ulUFfLfG2FWUvX3dcNk= 2022-09-02T04:32:12.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+qHnkGd7ulUFfLfG2FWUvX3dcNk=', 'Content-Length': '0'} 2022-09-02T04:32:12.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7e4377e1060e1f2-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.081 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.082 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.082 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.083 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.083 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.084 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.084 DEBUG:boto:Data: 2022-09-02T04:32:12.085 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.086 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.086 DEBUG:boto:Params: {} 2022-09-02T04:32:12.087 DEBUG:boto:Token: None 2022-09-02T04:32:12.087 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+qHnkGd7ulUFfLfG2FWUvX3dcNk= 2022-09-02T04:32:12.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+qHnkGd7ulUFfLfG2FWUvX3dcNk=', 'Content-Length': '0'} 2022-09-02T04:32:12.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000265be65b1d71d1f2-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:12.092 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.093 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.093 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.094 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.094 DEBUG:boto:Method: GET 2022-09-02T04:32:12.095 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.095 DEBUG:boto:Data: 2022-09-02T04:32:12.096 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.097 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.098 DEBUG:boto:Params: {} 2022-09-02T04:32:12.098 DEBUG:boto:Token: None 2022-09-02T04:32:12.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8= 2022-09-02T04:32:12.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8=', 'Content-Length': '0'} 2022-09-02T04:32:12.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a9c6009965af8f7-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.103 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.104 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.105 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.105 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.106 DEBUG:boto:Method: GET 2022-09-02T04:32:12.106 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.107 DEBUG:boto:Data: 2022-09-02T04:32:12.108 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.109 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.110 DEBUG:boto:Params: {} 2022-09-02T04:32:12.110 DEBUG:boto:Token: None 2022-09-02T04:32:12.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8= 2022-09-02T04:32:12.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8=', 'Content-Length': '0'} 2022-09-02T04:32:12.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f89636e4b9aa837-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.115 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.116 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.116 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.117 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.117 DEBUG:boto:Method: GET 2022-09-02T04:32:12.118 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.119 DEBUG:boto:Data: 2022-09-02T04:32:12.119 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.120 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.121 DEBUG:boto:Params: {} 2022-09-02T04:32:12.121 DEBUG:boto:Token: None 2022-09-02T04:32:12.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb89ea5fa8d8134-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.166 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.168 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.168 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.169 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.169 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.170 DEBUG:boto:Method: GET 2022-09-02T04:32:12.170 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.171 DEBUG:boto:Data: 2022-09-02T04:32:12.172 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.173 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.173 DEBUG:boto:Params: {} 2022-09-02T04:32:12.174 DEBUG:boto:Token: None 2022-09-02T04:32:12.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084583768e5dcd0ff-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.179 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.180 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.180 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.181 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.182 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.182 DEBUG:boto:Data: 2022-09-02T04:32:12.183 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.184 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.184 DEBUG:boto:Params: {} 2022-09-02T04:32:12.185 DEBUG:boto:Token: None 2022-09-02T04:32:12.185 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dyHb1EM0aWgdWzQJUcAJe5yJnJc= 2022-09-02T04:32:12.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dyHb1EM0aWgdWzQJUcAJe5yJnJc=', 'Content-Length': '0'} 2022-09-02T04:32:12.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e5f44d327c39772-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.190 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.191 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.191 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.192 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.192 DEBUG:boto:Data: 2022-09-02T04:32:12.193 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.194 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.194 DEBUG:boto:Params: {} 2022-09-02T04:32:12.195 DEBUG:boto:Token: None 2022-09-02T04:32:12.196 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dyHb1EM0aWgdWzQJUcAJe5yJnJc= 2022-09-02T04:32:12.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dyHb1EM0aWgdWzQJUcAJe5yJnJc=', 'Content-Length': '0'} 2022-09-02T04:32:12.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b430d44512bf70f6-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:12.201 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.201 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.202 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.203 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.203 DEBUG:boto:Method: GET 2022-09-02T04:32:12.204 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.205 DEBUG:boto:Data: 2022-09-02T04:32:12.205 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.207 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.207 DEBUG:boto:Params: {} 2022-09-02T04:32:12.208 DEBUG:boto:Token: None 2022-09-02T04:32:12.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8= 2022-09-02T04:32:12.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8=', 'Content-Length': '0'} 2022-09-02T04:32:12.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d5e5d33a6f89abf-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.214 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.214 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:12.215 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.216 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.216 DEBUG:boto:Method: GET 2022-09-02T04:32:12.217 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.218 DEBUG:boto:Data: 2022-09-02T04:32:12.218 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.220 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.220 DEBUG:boto:Params: {} 2022-09-02T04:32:12.221 DEBUG:boto:Token: None 2022-09-02T04:32:12.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:12.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8= 2022-09-02T04:32:12.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ZEVC0wpt060OXSO9mPsAvb4Wz8=', 'Content-Length': '0'} 2022-09-02T04:32:12.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050fae9afce1df689-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.266 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.267 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.268 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.268 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.269 DEBUG:boto:Method: GET 2022-09-02T04:32:12.269 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.270 DEBUG:boto:Data: 2022-09-02T04:32:12.271 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.272 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.272 DEBUG:boto:Params: {} 2022-09-02T04:32:12.273 DEBUG:boto:Token: None 2022-09-02T04:32:12.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a1864f466b6982e-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.278 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.279 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.280 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.280 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.281 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.281 DEBUG:boto:Method: GET 2022-09-02T04:32:12.282 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:12.282 DEBUG:boto:Data: 2022-09-02T04:32:12.283 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.284 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.285 DEBUG:boto:Params: {} 2022-09-02T04:32:12.285 DEBUG:boto:Token: None 2022-09-02T04:32:12.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a3984f083a8e229-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.290 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:12.292 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.292 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.293 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.294 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.294 DEBUG:boto:Method: GET 2022-09-02T04:32:12.295 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.296 DEBUG:boto:Data: 2022-09-02T04:32:12.296 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.298 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.298 DEBUG:boto:Params: {} 2022-09-02T04:32:12.299 DEBUG:boto:Token: None 2022-09-02T04:32:12.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4= 2022-09-02T04:32:12.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4=', 'Content-Length': '0'} 2022-09-02T04:32:12.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1be4066dbf66858-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.305 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.305 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.306 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.307 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.307 DEBUG:boto:Method: GET 2022-09-02T04:32:12.308 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.309 DEBUG:boto:Data: 2022-09-02T04:32:12.309 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.311 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.312 DEBUG:boto:Params: {} 2022-09-02T04:32:12.312 DEBUG:boto:Token: None 2022-09-02T04:32:12.313 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4= 2022-09-02T04:32:12.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4=', 'Content-Length': '0'} 2022-09-02T04:32:12.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067f95582a4d6321d-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.318 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.318 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.319 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.320 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.320 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.321 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.322 DEBUG:boto:Data: 2022-09-02T04:32:12.322 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.324 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.324 DEBUG:boto:Params: {} 2022-09-02T04:32:12.325 DEBUG:boto:Token: None 2022-09-02T04:32:12.326 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8V4TaZP5C3r5saJsoDUsXKPNfg= 2022-09-02T04:32:12.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8V4TaZP5C3r5saJsoDUsXKPNfg=', 'Content-Length': '0'} 2022-09-02T04:32:12.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d28b27ee1268fd-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.330 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.331 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.331 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.332 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.332 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.333 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.334 DEBUG:boto:Data: 2022-09-02T04:32:12.334 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.336 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.336 DEBUG:boto:Params: {} 2022-09-02T04:32:12.337 DEBUG:boto:Token: None 2022-09-02T04:32:12.338 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8V4TaZP5C3r5saJsoDUsXKPNfg= 2022-09-02T04:32:12.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8V4TaZP5C3r5saJsoDUsXKPNfg=', 'Content-Length': '0'} 2022-09-02T04:32:12.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000782b1ac6cb2b5607-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:12.343 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.343 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.344 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.345 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.346 DEBUG:boto:Method: GET 2022-09-02T04:32:12.346 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.347 DEBUG:boto:Data: 2022-09-02T04:32:12.347 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.349 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.349 DEBUG:boto:Params: {} 2022-09-02T04:32:12.350 DEBUG:boto:Token: None 2022-09-02T04:32:12.350 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4= 2022-09-02T04:32:12.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4=', 'Content-Length': '0'} 2022-09-02T04:32:12.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b9dfdee549edd9b-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.355 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.356 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.357 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.357 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.358 DEBUG:boto:Method: GET 2022-09-02T04:32:12.359 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.359 DEBUG:boto:Data: 2022-09-02T04:32:12.360 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.362 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.362 DEBUG:boto:Params: {} 2022-09-02T04:32:12.363 DEBUG:boto:Token: None 2022-09-02T04:32:12.364 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4= 2022-09-02T04:32:12.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4=', 'Content-Length': '0'} 2022-09-02T04:32:12.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ae44a48d8029a66-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.369 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.370 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.370 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.371 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.371 DEBUG:boto:Method: GET 2022-09-02T04:32:12.372 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.372 DEBUG:boto:Data: 2022-09-02T04:32:12.373 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.374 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.375 DEBUG:boto:Params: {} 2022-09-02T04:32:12.375 DEBUG:boto:Token: None 2022-09-02T04:32:12.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076d177f908f2db18-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.381 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.381 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.382 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.382 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.383 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.383 DEBUG:boto:Method: GET 2022-09-02T04:32:12.384 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.385 DEBUG:boto:Data: 2022-09-02T04:32:12.385 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.386 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.387 DEBUG:boto:Params: {} 2022-09-02T04:32:12.387 DEBUG:boto:Token: None 2022-09-02T04:32:12.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051dc3e82dda8756e-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.392 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.393 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.394 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.394 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.395 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.395 DEBUG:boto:Data: 2022-09-02T04:32:12.396 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.397 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.397 DEBUG:boto:Params: {} 2022-09-02T04:32:12.398 DEBUG:boto:Token: None 2022-09-02T04:32:12.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bmdWLS1C6JRyFjiuROqijlk4f90= 2022-09-02T04:32:12.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bmdWLS1C6JRyFjiuROqijlk4f90=', 'Content-Length': '0'} 2022-09-02T04:32:12.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000688ff6fae501890d-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.402 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.403 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.403 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.404 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.404 DEBUG:boto:Data: 2022-09-02T04:32:12.405 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.406 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.406 DEBUG:boto:Params: {} 2022-09-02T04:32:12.407 DEBUG:boto:Token: None 2022-09-02T04:32:12.408 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bmdWLS1C6JRyFjiuROqijlk4f90= 2022-09-02T04:32:12.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bmdWLS1C6JRyFjiuROqijlk4f90=', 'Content-Length': '0'} 2022-09-02T04:32:12.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000623288793e730af1-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:12.413 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.413 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.414 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.415 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.415 DEBUG:boto:Method: GET 2022-09-02T04:32:12.416 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.417 DEBUG:boto:Data: 2022-09-02T04:32:12.418 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.420 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.421 DEBUG:boto:Params: {} 2022-09-02T04:32:12.422 DEBUG:boto:Token: None 2022-09-02T04:32:12.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4= 2022-09-02T04:32:12.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4=', 'Content-Length': '0'} 2022-09-02T04:32:12.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d97491b3824467ed-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.427 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.428 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:12.429 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.429 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.430 DEBUG:boto:Method: GET 2022-09-02T04:32:12.430 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.431 DEBUG:boto:Data: 2022-09-02T04:32:12.431 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.433 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.434 DEBUG:boto:Params: {} 2022-09-02T04:32:12.434 DEBUG:boto:Token: None 2022-09-02T04:32:12.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:12.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4= 2022-09-02T04:32:12.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1TzQ5FY57x/2fS/QlCVdchKi/A4=', 'Content-Length': '0'} 2022-09-02T04:32:12.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cb987719df1909a-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.486 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.487 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.488 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.488 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.489 DEBUG:boto:Method: GET 2022-09-02T04:32:12.490 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.490 DEBUG:boto:Data: 2022-09-02T04:32:12.491 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.492 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.493 DEBUG:boto:Params: {} 2022-09-02T04:32:12.494 DEBUG:boto:Token: None 2022-09-02T04:32:12.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000824aadc7e085a35a-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.500 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.502 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.502 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.503 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.503 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.504 DEBUG:boto:Method: GET 2022-09-02T04:32:12.504 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:12.505 DEBUG:boto:Data: 2022-09-02T04:32:12.505 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.506 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.507 DEBUG:boto:Params: {} 2022-09-02T04:32:12.508 DEBUG:boto:Token: None 2022-09-02T04:32:12.508 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61bcedd5ee7ec83-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.514 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:12.515 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.516 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.516 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.517 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.517 DEBUG:boto:Method: GET 2022-09-02T04:32:12.518 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.519 DEBUG:boto:Data: 2022-09-02T04:32:12.519 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.520 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.521 DEBUG:boto:Params: {} 2022-09-02T04:32:12.522 DEBUG:boto:Token: None 2022-09-02T04:32:12.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo= 2022-09-02T04:32:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo=', 'Content-Length': '0'} 2022-09-02T04:32:12.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7810b8514352cdc-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.570 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.571 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.571 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.572 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.572 DEBUG:boto:Method: GET 2022-09-02T04:32:12.573 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.574 DEBUG:boto:Data: 2022-09-02T04:32:12.574 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.576 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.576 DEBUG:boto:Params: {} 2022-09-02T04:32:12.577 DEBUG:boto:Token: None 2022-09-02T04:32:12.578 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo= 2022-09-02T04:32:12.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo=', 'Content-Length': '0'} 2022-09-02T04:32:12.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e53c4d7c1f1df9b-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.582 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.583 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.583 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.584 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.584 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.585 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.586 DEBUG:boto:Data: 2022-09-02T04:32:12.587 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.589 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.589 DEBUG:boto:Params: {} 2022-09-02T04:32:12.590 DEBUG:boto:Token: None 2022-09-02T04:32:12.590 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:npLh4m2HdzY5ajbf/LXWMXolpFQ= 2022-09-02T04:32:12.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:npLh4m2HdzY5ajbf/LXWMXolpFQ=', 'Content-Length': '0'} 2022-09-02T04:32:12.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f3de1f072832bc7-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.595 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.595 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.596 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.596 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.597 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.597 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.598 DEBUG:boto:Data: 2022-09-02T04:32:12.599 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.601 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.602 DEBUG:boto:Params: {} 2022-09-02T04:32:12.603 DEBUG:boto:Token: None 2022-09-02T04:32:12.604 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:npLh4m2HdzY5ajbf/LXWMXolpFQ= 2022-09-02T04:32:12.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:npLh4m2HdzY5ajbf/LXWMXolpFQ=', 'Content-Length': '0'} 2022-09-02T04:32:12.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000399426a6749b1dfc-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:12.611 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.612 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.613 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.614 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.615 DEBUG:boto:Method: GET 2022-09-02T04:32:12.615 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.616 DEBUG:boto:Data: 2022-09-02T04:32:12.617 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.619 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.620 DEBUG:boto:Params: {} 2022-09-02T04:32:12.621 DEBUG:boto:Token: None 2022-09-02T04:32:12.623 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.624 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo= 2022-09-02T04:32:12.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo=', 'Content-Length': '0'} 2022-09-02T04:32:12.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc60f964d3aa38bc-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.628 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.629 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.630 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.631 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.632 DEBUG:boto:Method: GET 2022-09-02T04:32:12.632 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.633 DEBUG:boto:Data: 2022-09-02T04:32:12.633 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.634 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.635 DEBUG:boto:Params: {} 2022-09-02T04:32:12.636 DEBUG:boto:Token: None 2022-09-02T04:32:12.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo= 2022-09-02T04:32:12.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo=', 'Content-Length': '0'} 2022-09-02T04:32:12.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096c9a6bd469f917d-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.640 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.641 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.642 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.643 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.643 DEBUG:boto:Method: GET 2022-09-02T04:32:12.644 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.644 DEBUG:boto:Data: 2022-09-02T04:32:12.645 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.646 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.646 DEBUG:boto:Params: {} 2022-09-02T04:32:12.647 DEBUG:boto:Token: None 2022-09-02T04:32:12.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005958ba25bae8a33a-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.652 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.653 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.654 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.655 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.655 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.656 DEBUG:boto:Method: GET 2022-09-02T04:32:12.656 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.657 DEBUG:boto:Data: 2022-09-02T04:32:12.657 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.658 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.659 DEBUG:boto:Params: {} 2022-09-02T04:32:12.659 DEBUG:boto:Token: None 2022-09-02T04:32:12.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000186d6c1b8df3ad27-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.665 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.666 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.667 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.667 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.668 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.668 DEBUG:boto:Data: 2022-09-02T04:32:12.669 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.670 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.670 DEBUG:boto:Params: {} 2022-09-02T04:32:12.671 DEBUG:boto:Token: None 2022-09-02T04:32:12.671 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IOCfbgB2oJ8VCH/CWH5oifdPKlI= 2022-09-02T04:32:12.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IOCfbgB2oJ8VCH/CWH5oifdPKlI=', 'Content-Length': '0'} 2022-09-02T04:32:12.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe9d6f10a95ba538-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.676 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.677 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.678 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.679 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.680 DEBUG:boto:Data: 2022-09-02T04:32:12.681 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.682 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.683 DEBUG:boto:Params: {} 2022-09-02T04:32:12.684 DEBUG:boto:Token: None 2022-09-02T04:32:12.685 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IOCfbgB2oJ8VCH/CWH5oifdPKlI= 2022-09-02T04:32:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IOCfbgB2oJ8VCH/CWH5oifdPKlI=', 'Content-Length': '0'} 2022-09-02T04:32:12.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aabcd2c97a74c711-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:12.690 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.691 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.691 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.692 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.692 DEBUG:boto:Method: GET 2022-09-02T04:32:12.693 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.694 DEBUG:boto:Data: 2022-09-02T04:32:12.695 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.697 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.698 DEBUG:boto:Params: {} 2022-09-02T04:32:12.699 DEBUG:boto:Token: None 2022-09-02T04:32:12.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo= 2022-09-02T04:32:12.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo=', 'Content-Length': '0'} 2022-09-02T04:32:12.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056b09134e95977d6-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.746 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.747 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:12.748 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.748 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.749 DEBUG:boto:Method: GET 2022-09-02T04:32:12.749 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.750 DEBUG:boto:Data: 2022-09-02T04:32:12.751 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.752 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.753 DEBUG:boto:Params: {} 2022-09-02T04:32:12.753 DEBUG:boto:Token: None 2022-09-02T04:32:12.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:12.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo= 2022-09-02T04:32:12.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR4aMkkiQsdryZmB1nlU2qcrcwo=', 'Content-Length': '0'} 2022-09-02T04:32:12.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5d3997956b757e9-006311874c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.758 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.759 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.759 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.760 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.760 DEBUG:boto:Method: GET 2022-09-02T04:32:12.761 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.762 DEBUG:boto:Data: 2022-09-02T04:32:12.762 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.763 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.764 DEBUG:boto:Params: {} 2022-09-02T04:32:12.764 DEBUG:boto:Token: None 2022-09-02T04:32:12.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a409da95f5b6a6cf-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.770 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.771 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.772 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.772 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.773 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.774 DEBUG:boto:Method: GET 2022-09-02T04:32:12.775 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:12.776 DEBUG:boto:Data: 2022-09-02T04:32:12.776 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:12.778 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.779 DEBUG:boto:Params: {} 2022-09-02T04:32:12.780 DEBUG:boto:Token: None 2022-09-02T04:32:12.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b69830aca8e8de3-006311874c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.785 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-18 zones={test-zone3, test-zone1} 2022-09-02T04:32:12.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-18 zones={test-zone3, test-zone2} 2022-09-02T04:32:12.787 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.788 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.788 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.789 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:32:12.790 DEBUG:boto:Data: 2022-09-02T04:32:12.790 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.791 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.792 DEBUG:boto:Params: {} 2022-09-02T04:32:12.792 DEBUG:boto:Token: None 2022-09-02T04:32:12.793 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ua07nNOlvqtGjJSQXJ08xVMoI7Y= 2022-09-02T04:32:12.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ua07nNOlvqtGjJSQXJ08xVMoI7Y=', 'Content-Length': '0'} 2022-09-02T04:32:12.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091b60ed2c838a2ff-006311874c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.797 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.798 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.798 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.799 DEBUG:boto:Path: /xljbkf-18/ 2022-09-02T04:32:12.799 DEBUG:boto:Data: 2022-09-02T04:32:12.800 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.801 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.801 DEBUG:boto:Params: {} 2022-09-02T04:32:12.802 DEBUG:boto:Token: None 2022-09-02T04:32:12.802 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ua07nNOlvqtGjJSQXJ08xVMoI7Y= 2022-09-02T04:32:12.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ua07nNOlvqtGjJSQXJ08xVMoI7Y=', 'Content-Length': '0'} 2022-09-02T04:32:12.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074a0430f38a610da-006311874c-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.807 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:32:12.808 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.808 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.809 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.809 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.810 DEBUG:boto:Method: GET 2022-09-02T04:32:12.810 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.811 DEBUG:boto:Data: 2022-09-02T04:32:12.811 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.812 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.813 DEBUG:boto:Params: {} 2022-09-02T04:32:12.814 DEBUG:boto:Token: None 2022-09-02T04:32:12.814 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/?versions 2022-09-02T04:32:12.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I= 2022-09-02T04:32:12.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I=', 'Content-Length': '0'} 2022-09-02T04:32:12.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8dadd8aceaa89d-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.819 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.820 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:12.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:12.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:12.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:12.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:32:12.822 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.823 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.823 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.824 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.824 DEBUG:boto:Method: GET 2022-09-02T04:32:12.825 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.825 DEBUG:boto:Data: 2022-09-02T04:32:12.826 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.827 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.827 DEBUG:boto:Params: {} 2022-09-02T04:32:12.828 DEBUG:boto:Token: None 2022-09-02T04:32:12.829 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/?versions 2022-09-02T04:32:12.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I= 2022-09-02T04:32:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I=', 'Content-Length': '0'} 2022-09-02T04:32:12.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2e38eab8802537-006311874c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.834 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.835 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:32:12.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-09-02T04:32:12.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-09-02T04:32:12.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-09-02T04:32:12.837 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.837 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.838 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.838 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.839 DEBUG:boto:Method: GET 2022-09-02T04:32:12.839 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.840 DEBUG:boto:Data: 2022-09-02T04:32:12.841 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.842 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.842 DEBUG:boto:Params: {} 2022-09-02T04:32:12.843 DEBUG:boto:Token: None 2022-09-02T04:32:12.843 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/?versions 2022-09-02T04:32:12.844 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I= 2022-09-02T04:32:12.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I=', 'Content-Length': '0'} 2022-09-02T04:32:12.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a82fd816996ab3b5-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.848 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.849 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.850 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.850 DEBUG:boto:path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.851 DEBUG:boto:auth_path=/xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.851 DEBUG:boto:Method: GET 2022-09-02T04:32:12.852 DEBUG:boto:Path: /xljbkf-18/?versions&max-keys=999 2022-09-02T04:32:12.852 DEBUG:boto:Data: 2022-09-02T04:32:12.853 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.854 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.854 DEBUG:boto:Params: {} 2022-09-02T04:32:12.855 DEBUG:boto:Token: None 2022-09-02T04:32:12.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/?versions 2022-09-02T04:32:12.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I= 2022-09-02T04:32:12.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KTkgAuoBf3y5yeGgqDHvojP9a2I=', 'Content-Length': '0'} 2022-09-02T04:32:12.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe3d90e2fef39fe-006311874c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.860 DEBUG:boto:b'xljbkf-18999falsedummy3AuL8Bud26YknQiSqXDZXoImKs4CeDotrue2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1Z4pvll2V-Jil-CmpmYFtckPiB0qXS7atrue2022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2diYkQYJYADn-uQFWAHzXvtwI0dvLeeRtrue2022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone37MmMpy25pxlZVE.tqG5U1pcEXCr8wJytrue2022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:12.862 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:12.862 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:12.863 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.863 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.864 DEBUG:boto:Method: GET 2022-09-02T04:32:12.864 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.865 DEBUG:boto:Data: 2022-09-02T04:32:12.865 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.866 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.867 DEBUG:boto:Params: {} 2022-09-02T04:32:12.868 DEBUG:boto:Token: None 2022-09-02T04:32:12.868 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY= 2022-09-02T04:32:12.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY=', 'Content-Length': '0'} 2022-09-02T04:32:12.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000812d405d7f06c392-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.872 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:12.873 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:12.873 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.874 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.874 DEBUG:boto:Method: GET 2022-09-02T04:32:12.875 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.876 DEBUG:boto:Data: 2022-09-02T04:32:12.876 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.877 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.878 DEBUG:boto:Params: {} 2022-09-02T04:32:12.878 DEBUG:boto:Token: None 2022-09-02T04:32:12.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY= 2022-09-02T04:32:12.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY=', 'Content-Length': '0'} 2022-09-02T04:32:12.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a03f3a39ad14f2ae-006311874c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.883 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:12.884 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:12.884 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.885 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.885 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.886 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.886 DEBUG:boto:Data: 2022-09-02T04:32:12.887 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.888 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.888 DEBUG:boto:Params: {} 2022-09-02T04:32:12.889 DEBUG:boto:Token: None 2022-09-02T04:32:12.890 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zWMZgGgonuuPs2t0n8yBQRD1UTY= 2022-09-02T04:32:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zWMZgGgonuuPs2t0n8yBQRD1UTY=', 'Content-Length': '0'} 2022-09-02T04:32:12.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b449c9ecdfab6ba0-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.894 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:12.894 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:12.895 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.896 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.897 DEBUG:boto:Method: HEAD 2022-09-02T04:32:12.898 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.898 DEBUG:boto:Data: 2022-09-02T04:32:12.899 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.900 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.900 DEBUG:boto:Params: {} 2022-09-02T04:32:12.901 DEBUG:boto:Token: None 2022-09-02T04:32:12.902 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zWMZgGgonuuPs2t0n8yBQRD1UTY= 2022-09-02T04:32:12.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zWMZgGgonuuPs2t0n8yBQRD1UTY=', 'Content-Length': '0'} 2022-09-02T04:32:12.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aedd22e534ca05bf-006311874c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:12.906 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:12.907 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:12.908 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.908 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.909 DEBUG:boto:Method: GET 2022-09-02T04:32:12.910 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.911 DEBUG:boto:Data: 2022-09-02T04:32:12.912 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.913 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.914 DEBUG:boto:Params: {} 2022-09-02T04:32:12.915 DEBUG:boto:Token: None 2022-09-02T04:32:12.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY= 2022-09-02T04:32:12.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY=', 'Content-Length': '0'} 2022-09-02T04:32:12.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000351d73d56cca75fe-006311874c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.923 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:12.924 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:12.925 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.925 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.926 DEBUG:boto:Method: GET 2022-09-02T04:32:12.927 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.929 DEBUG:boto:Data: 2022-09-02T04:32:12.930 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.932 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.933 DEBUG:boto:Params: {} 2022-09-02T04:32:12.934 DEBUG:boto:Token: None 2022-09-02T04:32:12.935 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:12.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY= 2022-09-02T04:32:12.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DFcsBLIQIrLfjOdaxPnrvRMFjTY=', 'Content-Length': '0'} 2022-09-02T04:32:12.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002452377f122ab043-006311874c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.942 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.943 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.944 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:12.945 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:12.946 DEBUG:boto:Method: GET 2022-09-02T04:32:12.947 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:12.948 DEBUG:boto:Data: 2022-09-02T04:32:12.950 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:12.951 DEBUG:boto:Port: 8001 2022-09-02T04:32:12.952 DEBUG:boto:Params: {} 2022-09-02T04:32:12.953 DEBUG:boto:Token: None 2022-09-02T04:32:12.954 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000473d4d3d76401bf2-006311874c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:12.960 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:12.960 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:12.961 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:12.961 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:12.962 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:12.962 DEBUG:boto:Method: GET 2022-09-02T04:32:12.963 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:12.964 DEBUG:boto:Data: 2022-09-02T04:32:12.965 DEBUG:boto:Headers: {} 2022-09-02T04:32:12.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:12.966 DEBUG:boto:Port: 8000 2022-09-02T04:32:12.967 DEBUG:boto:Params: {} 2022-09-02T04:32:12.968 DEBUG:boto:Token: None 2022-09-02T04:32:12.968 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:12 GMT /xljbkf-18/ 2022-09-02T04:32:12.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU= 2022-09-02T04:32:12.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dwg6nfNCDQUR512yehu3McQZkAU=', 'Content-Length': '0'} 2022-09-02T04:32:12.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70e79dd328f1005-006311874c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.018 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.019 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:13.020 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:13.021 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.021 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:32:13.022 DEBUG:boto:Data: 2022-09-02T04:32:13.023 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.024 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.024 DEBUG:boto:Params: {} 2022-09-02T04:32:13.025 DEBUG:boto:Token: None 2022-09-02T04:32:13.026 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/dummy 2022-09-02T04:32:13.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E6rPnn201Lb8a0uv+EjbhNqfdk4= 2022-09-02T04:32:13.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E6rPnn201Lb8a0uv+EjbhNqfdk4=', 'Content-Length': '0'} 2022-09-02T04:32:13.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099ba07d9a1cd3c3e-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.030 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:13.031 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:13.031 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.032 DEBUG:boto:Path: /xljbkf-18/dummy 2022-09-02T04:32:13.032 DEBUG:boto:Data: 2022-09-02T04:32:13.033 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.034 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.035 DEBUG:boto:Params: {} 2022-09-02T04:32:13.035 DEBUG:boto:Token: None 2022-09-02T04:32:13.036 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/dummy 2022-09-02T04:32:13.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E6rPnn201Lb8a0uv+EjbhNqfdk4= 2022-09-02T04:32:13.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E6rPnn201Lb8a0uv+EjbhNqfdk4=', 'Content-Length': '0'} 2022-09-02T04:32:13.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e1c2453bab9ed3a-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:32:13.041 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:13.042 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:13.043 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.043 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.044 DEBUG:boto:Method: GET 2022-09-02T04:32:13.045 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.045 DEBUG:boto:Data: 2022-09-02T04:32:13.046 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.047 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.047 DEBUG:boto:Params: {} 2022-09-02T04:32:13.048 DEBUG:boto:Token: None 2022-09-02T04:32:13.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pPke/3c34HZasDARRM9e1rVBM/k= 2022-09-02T04:32:13.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pPke/3c34HZasDARRM9e1rVBM/k=', 'Content-Length': '0'} 2022-09-02T04:32:13.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b5bdcba0a81ab89-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.055 DEBUG:boto:path=/xljbkf-18/dummy 2022-09-02T04:32:13.056 DEBUG:boto:auth_path=/xljbkf-18/dummy 2022-09-02T04:32:13.057 DEBUG:boto:path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.057 DEBUG:boto:auth_path=/xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.058 DEBUG:boto:Method: GET 2022-09-02T04:32:13.059 DEBUG:boto:Path: /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.060 DEBUG:boto:Data: 2022-09-02T04:32:13.061 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.062 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.062 DEBUG:boto:Params: {} 2022-09-02T04:32:13.063 DEBUG:boto:Token: None 2022-09-02T04:32:13.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/dummy?versionId=3AuL8Bud26YknQiSqXDZXoImKs4CeDo 2022-09-02T04:32:13.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pPke/3c34HZasDARRM9e1rVBM/k= 2022-09-02T04:32:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pPke/3c34HZasDARRM9e1rVBM/k=', 'Content-Length': '0'} 2022-09-02T04:32:13.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:28:20 GMT'), ('x-amz-version-id', '3AuL8Bud26YknQiSqXDZXoImKs4CeDo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001093e3f278654e40-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.068 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.069 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.069 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:13.071 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:13.071 DEBUG:boto:Method: GET 2022-09-02T04:32:13.072 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:13.073 DEBUG:boto:Data: 2022-09-02T04:32:13.073 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.075 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.075 DEBUG:boto:Params: {} 2022-09-02T04:32:13.076 DEBUG:boto:Token: None 2022-09-02T04:32:13.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3e1ae69bba168c-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.082 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.083 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.083 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.084 DEBUG:boto:path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:13.085 DEBUG:boto:auth_path=/xljbkf-18/?prefix=dummy 2022-09-02T04:32:13.085 DEBUG:boto:Method: GET 2022-09-02T04:32:13.086 DEBUG:boto:Path: /xljbkf-18/?prefix=dummy 2022-09-02T04:32:13.086 DEBUG:boto:Data: 2022-09-02T04:32:13.087 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.088 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.089 DEBUG:boto:Params: {} 2022-09-02T04:32:13.089 DEBUG:boto:Token: None 2022-09-02T04:32:13.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb2ddc2472ff90b-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.095 DEBUG:boto:b'xljbkf-18dummy1000falsedummy2022-09-02T04:28:20.959Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:13.097 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.098 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.099 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.099 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.100 DEBUG:boto:Method: GET 2022-09-02T04:32:13.100 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.101 DEBUG:boto:Data: 2022-09-02T04:32:13.101 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.103 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.104 DEBUG:boto:Params: {} 2022-09-02T04:32:13.104 DEBUG:boto:Token: None 2022-09-02T04:32:13.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow= 2022-09-02T04:32:13.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow=', 'Content-Length': '0'} 2022-09-02T04:32:13.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d4959ef603ae39a-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.109 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.110 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.111 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.111 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.112 DEBUG:boto:Method: GET 2022-09-02T04:32:13.112 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.113 DEBUG:boto:Data: 2022-09-02T04:32:13.114 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.115 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.115 DEBUG:boto:Params: {} 2022-09-02T04:32:13.116 DEBUG:boto:Token: None 2022-09-02T04:32:13.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow= 2022-09-02T04:32:13.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow=', 'Content-Length': '0'} 2022-09-02T04:32:13.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031fd4e5b1251ede0-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.122 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.123 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.124 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.125 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.126 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.127 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.128 DEBUG:boto:Data: 2022-09-02T04:32:13.129 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.131 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.131 DEBUG:boto:Params: {} 2022-09-02T04:32:13.132 DEBUG:boto:Token: None 2022-09-02T04:32:13.133 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a1eYW/Tzf4FMhEzPg/YddfBPXck= 2022-09-02T04:32:13.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a1eYW/Tzf4FMhEzPg/YddfBPXck=', 'Content-Length': '0'} 2022-09-02T04:32:13.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a417467ed7cfd191-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.138 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.139 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.140 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.140 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.141 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.142 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.142 DEBUG:boto:Data: 2022-09-02T04:32:13.143 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.144 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.145 DEBUG:boto:Params: {} 2022-09-02T04:32:13.145 DEBUG:boto:Token: None 2022-09-02T04:32:13.146 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a1eYW/Tzf4FMhEzPg/YddfBPXck= 2022-09-02T04:32:13.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a1eYW/Tzf4FMhEzPg/YddfBPXck=', 'Content-Length': '0'} 2022-09-02T04:32:13.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e100b0d964b5e2a7-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:13.151 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.151 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.152 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.153 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.153 DEBUG:boto:Method: GET 2022-09-02T04:32:13.154 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.154 DEBUG:boto:Data: 2022-09-02T04:32:13.155 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.156 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.156 DEBUG:boto:Params: {} 2022-09-02T04:32:13.158 DEBUG:boto:Token: None 2022-09-02T04:32:13.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow= 2022-09-02T04:32:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow=', 'Content-Length': '0'} 2022-09-02T04:32:13.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024568bc4ca8a9cde-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.163 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.164 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.165 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.165 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.166 DEBUG:boto:Method: GET 2022-09-02T04:32:13.166 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.167 DEBUG:boto:Data: 2022-09-02T04:32:13.168 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.169 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.169 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.170 DEBUG:boto:Params: {} 2022-09-02T04:32:13.171 DEBUG:boto:Token: None 2022-09-02T04:32:13.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.172 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow= 2022-09-02T04:32:13.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow=', 'Content-Length': '0'} 2022-09-02T04:32:13.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d96f9477f33dc235-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.176 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.177 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.177 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.178 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.179 DEBUG:boto:Method: GET 2022-09-02T04:32:13.179 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.180 DEBUG:boto:Data: 2022-09-02T04:32:13.180 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.181 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.182 DEBUG:boto:Params: {} 2022-09-02T04:32:13.182 DEBUG:boto:Token: None 2022-09-02T04:32:13.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deb4878a418e8276-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.234 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.235 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.236 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.236 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.237 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.237 DEBUG:boto:Method: GET 2022-09-02T04:32:13.238 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.239 DEBUG:boto:Data: 2022-09-02T04:32:13.239 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.240 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.241 DEBUG:boto:Params: {} 2022-09-02T04:32:13.241 DEBUG:boto:Token: None 2022-09-02T04:32:13.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba6c6049d89c7fcd-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.247 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.247 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.248 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.249 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.250 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.250 DEBUG:boto:Data: 2022-09-02T04:32:13.251 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.252 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.252 DEBUG:boto:Params: {} 2022-09-02T04:32:13.253 DEBUG:boto:Token: None 2022-09-02T04:32:13.254 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BMN9o5W2dYLus01C0u+8A7GOcM4= 2022-09-02T04:32:13.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BMN9o5W2dYLus01C0u+8A7GOcM4=', 'Content-Length': '0'} 2022-09-02T04:32:13.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e46893e089ea5cd1-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.258 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.258 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.259 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.259 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.260 DEBUG:boto:Data: 2022-09-02T04:32:13.261 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.261 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.262 DEBUG:boto:Params: {} 2022-09-02T04:32:13.263 DEBUG:boto:Token: None 2022-09-02T04:32:13.263 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BMN9o5W2dYLus01C0u+8A7GOcM4= 2022-09-02T04:32:13.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BMN9o5W2dYLus01C0u+8A7GOcM4=', 'Content-Length': '0'} 2022-09-02T04:32:13.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd5e8b3a5e96a1f7-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-09-02T04:32:13.269 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.269 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.270 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.270 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.271 DEBUG:boto:Method: GET 2022-09-02T04:32:13.271 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.272 DEBUG:boto:Data: 2022-09-02T04:32:13.273 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.274 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.274 DEBUG:boto:Params: {} 2022-09-02T04:32:13.275 DEBUG:boto:Token: None 2022-09-02T04:32:13.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow= 2022-09-02T04:32:13.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow=', 'Content-Length': '0'} 2022-09-02T04:32:13.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f21a8c83944d9f-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.280 DEBUG:boto:path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.280 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1 2022-09-02T04:32:13.281 DEBUG:boto:path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.282 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.282 DEBUG:boto:Method: GET 2022-09-02T04:32:13.283 DEBUG:boto:Path: /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.283 DEBUG:boto:Data: 2022-09-02T04:32:13.284 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.285 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.285 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.286 DEBUG:boto:Params: {} 2022-09-02T04:32:13.286 DEBUG:boto:Token: None 2022-09-02T04:32:13.287 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone1?versionId=Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a 2022-09-02T04:32:13.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow= 2022-09-02T04:32:13.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lo8TpRgzYDIHkBLTmrECVPDS7ow=', 'Content-Length': '0'} 2022-09-02T04:32:13.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'Z4pvll2V-Jil-CmpmYFtckPiB0qXS7a'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e72fcc22494a2719-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.338 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.339 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.339 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.340 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.340 DEBUG:boto:Method: GET 2022-09-02T04:32:13.341 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.342 DEBUG:boto:Data: 2022-09-02T04:32:13.342 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.343 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.344 DEBUG:boto:Params: {} 2022-09-02T04:32:13.344 DEBUG:boto:Token: None 2022-09-02T04:32:13.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096544c53176eeebb-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.350 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.351 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.351 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.352 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.353 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.353 DEBUG:boto:Method: GET 2022-09-02T04:32:13.354 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone1 2022-09-02T04:32:13.354 DEBUG:boto:Data: 2022-09-02T04:32:13.355 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.356 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.356 DEBUG:boto:Params: {} 2022-09-02T04:32:13.357 DEBUG:boto:Token: None 2022-09-02T04:32:13.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531f03f065fc90e0-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.362 DEBUG:boto:b'xljbkf-18obj-test-zone11000falseobj-test-zone12022-09-02T04:29:02.427Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:13.363 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.364 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.365 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.365 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.366 DEBUG:boto:Method: GET 2022-09-02T04:32:13.366 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.367 DEBUG:boto:Data: 2022-09-02T04:32:13.367 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.368 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.369 DEBUG:boto:Params: {} 2022-09-02T04:32:13.369 DEBUG:boto:Token: None 2022-09-02T04:32:13.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.370 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA= 2022-09-02T04:32:13.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA=', 'Content-Length': '0'} 2022-09-02T04:32:13.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068de8839300a3f49-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.374 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.375 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.375 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.376 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.376 DEBUG:boto:Method: GET 2022-09-02T04:32:13.377 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.377 DEBUG:boto:Data: 2022-09-02T04:32:13.378 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.379 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.379 DEBUG:boto:Params: {} 2022-09-02T04:32:13.380 DEBUG:boto:Token: None 2022-09-02T04:32:13.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA= 2022-09-02T04:32:13.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA=', 'Content-Length': '0'} 2022-09-02T04:32:13.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000019d686d03f039d0-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.385 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.386 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.386 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.387 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.387 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.388 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.388 DEBUG:boto:Data: 2022-09-02T04:32:13.389 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.390 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.390 DEBUG:boto:Params: {} 2022-09-02T04:32:13.391 DEBUG:boto:Token: None 2022-09-02T04:32:13.391 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X/RzSRCh/2rmDfGIB6gzj/uhvKE= 2022-09-02T04:32:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X/RzSRCh/2rmDfGIB6gzj/uhvKE=', 'Content-Length': '0'} 2022-09-02T04:32:13.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000463364ad97613141-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.395 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.396 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.396 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.397 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.397 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.398 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.398 DEBUG:boto:Data: 2022-09-02T04:32:13.399 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.400 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.400 DEBUG:boto:Params: {} 2022-09-02T04:32:13.401 DEBUG:boto:Token: None 2022-09-02T04:32:13.402 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X/RzSRCh/2rmDfGIB6gzj/uhvKE= 2022-09-02T04:32:13.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X/RzSRCh/2rmDfGIB6gzj/uhvKE=', 'Content-Length': '0'} 2022-09-02T04:32:13.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022a8fa56fcf388ff-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:13.411 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.412 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.412 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.413 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.413 DEBUG:boto:Method: GET 2022-09-02T04:32:13.414 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.415 DEBUG:boto:Data: 2022-09-02T04:32:13.415 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.416 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.417 DEBUG:boto:Params: {} 2022-09-02T04:32:13.417 DEBUG:boto:Token: None 2022-09-02T04:32:13.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA= 2022-09-02T04:32:13.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA=', 'Content-Length': '0'} 2022-09-02T04:32:13.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dc2650e13ea0f0e-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.422 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.423 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.424 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.424 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.425 DEBUG:boto:Method: GET 2022-09-02T04:32:13.425 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.426 DEBUG:boto:Data: 2022-09-02T04:32:13.426 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.427 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.428 DEBUG:boto:Params: {} 2022-09-02T04:32:13.428 DEBUG:boto:Token: None 2022-09-02T04:32:13.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA= 2022-09-02T04:32:13.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA=', 'Content-Length': '0'} 2022-09-02T04:32:13.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000617653dcb91fbf73-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.433 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.434 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.434 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.435 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.435 DEBUG:boto:Method: GET 2022-09-02T04:32:13.436 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.436 DEBUG:boto:Data: 2022-09-02T04:32:13.437 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.438 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.439 DEBUG:boto:Params: {} 2022-09-02T04:32:13.439 DEBUG:boto:Token: None 2022-09-02T04:32:13.440 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd50b035a3a65ec-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.444 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.445 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.446 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.446 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.447 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.447 DEBUG:boto:Method: GET 2022-09-02T04:32:13.448 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.449 DEBUG:boto:Data: 2022-09-02T04:32:13.449 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.450 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.451 DEBUG:boto:Params: {} 2022-09-02T04:32:13.451 DEBUG:boto:Token: None 2022-09-02T04:32:13.452 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.452 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f06bf1f0bc1d616-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.457 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.458 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.458 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.459 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.459 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.460 DEBUG:boto:Data: 2022-09-02T04:32:13.460 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.461 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.462 DEBUG:boto:Params: {} 2022-09-02T04:32:13.462 DEBUG:boto:Token: None 2022-09-02T04:32:13.463 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iNvRCYpeiU4kOXOkWI6YAUY3Qp0= 2022-09-02T04:32:13.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iNvRCYpeiU4kOXOkWI6YAUY3Qp0=', 'Content-Length': '0'} 2022-09-02T04:32:13.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a50e6e7418b40006-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.467 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.468 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.468 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.469 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.469 DEBUG:boto:Data: 2022-09-02T04:32:13.470 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.471 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.471 DEBUG:boto:Params: {} 2022-09-02T04:32:13.472 DEBUG:boto:Token: None 2022-09-02T04:32:13.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iNvRCYpeiU4kOXOkWI6YAUY3Qp0= 2022-09-02T04:32:13.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iNvRCYpeiU4kOXOkWI6YAUY3Qp0=', 'Content-Length': '0'} 2022-09-02T04:32:13.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d6108ad25a5ab9-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-09-02T04:32:13.477 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.478 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.478 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.479 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.479 DEBUG:boto:Method: GET 2022-09-02T04:32:13.480 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.480 DEBUG:boto:Data: 2022-09-02T04:32:13.481 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.482 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.483 DEBUG:boto:Params: {} 2022-09-02T04:32:13.483 DEBUG:boto:Token: None 2022-09-02T04:32:13.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA= 2022-09-02T04:32:13.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA=', 'Content-Length': '0'} 2022-09-02T04:32:13.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c59ff4c67a81b014-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.530 DEBUG:boto:path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.531 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2 2022-09-02T04:32:13.532 DEBUG:boto:path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.532 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.533 DEBUG:boto:Method: GET 2022-09-02T04:32:13.533 DEBUG:boto:Path: /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.534 DEBUG:boto:Data: 2022-09-02T04:32:13.535 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.536 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.536 DEBUG:boto:Params: {} 2022-09-02T04:32:13.537 DEBUG:boto:Token: None 2022-09-02T04:32:13.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone2?versionId=diYkQYJYADn-uQFWAHzXvtwI0dvLeeR 2022-09-02T04:32:13.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA= 2022-09-02T04:32:13.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JxiQ7zjlmxvDCHQFSszfiPbNbA=', 'Content-Length': '0'} 2022-09-02T04:32:13.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', 'diYkQYJYADn-uQFWAHzXvtwI0dvLeeR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3c8e00322b4ff0e-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.542 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.543 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.543 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.544 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.544 DEBUG:boto:Method: GET 2022-09-02T04:32:13.545 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.546 DEBUG:boto:Data: 2022-09-02T04:32:13.546 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.547 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.548 DEBUG:boto:Params: {} 2022-09-02T04:32:13.548 DEBUG:boto:Token: None 2022-09-02T04:32:13.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058723a4204d5fe65-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.554 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.555 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.555 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.556 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.557 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.557 DEBUG:boto:Method: GET 2022-09-02T04:32:13.558 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone2 2022-09-02T04:32:13.558 DEBUG:boto:Data: 2022-09-02T04:32:13.559 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.560 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.560 DEBUG:boto:Params: {} 2022-09-02T04:32:13.561 DEBUG:boto:Token: None 2022-09-02T04:32:13.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000989688b5a37b0eab-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.566 DEBUG:boto:b'xljbkf-18obj-test-zone21000falseobj-test-zone22022-09-02T04:29:02.607Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:13.568 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.568 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.569 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.569 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.570 DEBUG:boto:Method: GET 2022-09-02T04:32:13.571 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.571 DEBUG:boto:Data: 2022-09-02T04:32:13.572 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.573 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.573 DEBUG:boto:Params: {} 2022-09-02T04:32:13.574 DEBUG:boto:Token: None 2022-09-02T04:32:13.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY= 2022-09-02T04:32:13.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY=', 'Content-Length': '0'} 2022-09-02T04:32:13.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f67efd743c80340-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.580 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.580 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.581 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.581 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.582 DEBUG:boto:Method: GET 2022-09-02T04:32:13.582 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.583 DEBUG:boto:Data: 2022-09-02T04:32:13.583 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.585 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.586 DEBUG:boto:Params: {} 2022-09-02T04:32:13.586 DEBUG:boto:Token: None 2022-09-02T04:32:13.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY= 2022-09-02T04:32:13.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY=', 'Content-Length': '0'} 2022-09-02T04:32:13.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033fd5871ab7e5dee-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.591 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.591 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.592 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.592 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.593 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.593 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.594 DEBUG:boto:Data: 2022-09-02T04:32:13.594 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.596 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.596 DEBUG:boto:Params: {} 2022-09-02T04:32:13.597 DEBUG:boto:Token: None 2022-09-02T04:32:13.598 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KdBp78EMbLe/JpBIjRNBGT7sBX0= 2022-09-02T04:32:13.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KdBp78EMbLe/JpBIjRNBGT7sBX0=', 'Content-Length': '0'} 2022-09-02T04:32:13.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdd9ac794430d78c-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.602 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.602 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.603 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.603 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.604 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.604 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.605 DEBUG:boto:Data: 2022-09-02T04:32:13.606 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.607 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.608 DEBUG:boto:Params: {} 2022-09-02T04:32:13.608 DEBUG:boto:Token: None 2022-09-02T04:32:13.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KdBp78EMbLe/JpBIjRNBGT7sBX0= 2022-09-02T04:32:13.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KdBp78EMbLe/JpBIjRNBGT7sBX0=', 'Content-Length': '0'} 2022-09-02T04:32:13.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056389809622a2e48-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:13.613 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.614 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.615 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.615 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.616 DEBUG:boto:Method: GET 2022-09-02T04:32:13.616 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.617 DEBUG:boto:Data: 2022-09-02T04:32:13.617 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.618 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.619 DEBUG:boto:Params: {} 2022-09-02T04:32:13.620 DEBUG:boto:Token: None 2022-09-02T04:32:13.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY= 2022-09-02T04:32:13.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY=', 'Content-Length': '0'} 2022-09-02T04:32:13.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035ca66271ec836be-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.625 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.626 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.626 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.627 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.627 DEBUG:boto:Method: GET 2022-09-02T04:32:13.628 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.628 DEBUG:boto:Data: 2022-09-02T04:32:13.629 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.631 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.631 DEBUG:boto:Params: {} 2022-09-02T04:32:13.632 DEBUG:boto:Token: None 2022-09-02T04:32:13.633 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY= 2022-09-02T04:32:13.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY=', 'Content-Length': '0'} 2022-09-02T04:32:13.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c09c0f1f88dcd3a9-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.678 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.679 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.679 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.680 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.680 DEBUG:boto:Method: GET 2022-09-02T04:32:13.681 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.682 DEBUG:boto:Data: 2022-09-02T04:32:13.682 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.683 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.684 DEBUG:boto:Params: {} 2022-09-02T04:32:13.685 DEBUG:boto:Token: None 2022-09-02T04:32:13.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b3da0a5c906741c-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.690 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.691 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.691 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.692 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.692 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.693 DEBUG:boto:Method: GET 2022-09-02T04:32:13.693 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.694 DEBUG:boto:Data: 2022-09-02T04:32:13.695 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.696 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.696 DEBUG:boto:Params: {} 2022-09-02T04:32:13.697 DEBUG:boto:Token: None 2022-09-02T04:32:13.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e20bb5dd58f390-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.702 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.703 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.704 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.704 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.705 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.705 DEBUG:boto:Data: 2022-09-02T04:32:13.706 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.707 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.707 DEBUG:boto:Params: {} 2022-09-02T04:32:13.708 DEBUG:boto:Token: None 2022-09-02T04:32:13.708 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m1llggYYaBl15UW01lPR1/ncLDQ= 2022-09-02T04:32:13.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m1llggYYaBl15UW01lPR1/ncLDQ=', 'Content-Length': '0'} 2022-09-02T04:32:13.712 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048e56c2378a8be21-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.713 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.713 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.714 DEBUG:boto:Method: HEAD 2022-09-02T04:32:13.715 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.715 DEBUG:boto:Data: 2022-09-02T04:32:13.716 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.717 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.717 DEBUG:boto:Params: {} 2022-09-02T04:32:13.718 DEBUG:boto:Token: None 2022-09-02T04:32:13.718 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m1llggYYaBl15UW01lPR1/ncLDQ= 2022-09-02T04:32:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m1llggYYaBl15UW01lPR1/ncLDQ=', 'Content-Length': '0'} 2022-09-02T04:32:13.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076030213a6f0556d-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-09-02T04:32:13.723 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.724 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.725 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.725 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.726 DEBUG:boto:Method: GET 2022-09-02T04:32:13.726 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.727 DEBUG:boto:Data: 2022-09-02T04:32:13.727 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.728 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.729 DEBUG:boto:Params: {} 2022-09-02T04:32:13.730 DEBUG:boto:Token: None 2022-09-02T04:32:13.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY= 2022-09-02T04:32:13.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY=', 'Content-Length': '0'} 2022-09-02T04:32:13.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e47354b60db35b13-006311874d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.736 DEBUG:boto:path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.736 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3 2022-09-02T04:32:13.737 DEBUG:boto:path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.737 DEBUG:boto:auth_path=/xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.738 DEBUG:boto:Method: GET 2022-09-02T04:32:13.739 DEBUG:boto:Path: /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.739 DEBUG:boto:Data: 2022-09-02T04:32:13.740 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.741 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.741 DEBUG:boto:Params: {} 2022-09-02T04:32:13.742 DEBUG:boto:Token: None 2022-09-02T04:32:13.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/obj-test-zone3?versionId=7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy 2022-09-02T04:32:13.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY= 2022-09-02T04:32:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f2SelCqpVNY6zmdw7SHzYs1zeaY=', 'Content-Length': '0'} 2022-09-02T04:32:13.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:30:11 GMT'), ('x-amz-version-id', '7MmMpy25pxlZVE.tqG5U1pcEXCr8wJy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e2d8fb429b558b1-006311874d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.748 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.748 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.749 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.749 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.750 DEBUG:boto:Method: GET 2022-09-02T04:32:13.750 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.751 DEBUG:boto:Data: 2022-09-02T04:32:13.751 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:32:13.753 DEBUG:boto:Port: 8001 2022-09-02T04:32:13.753 DEBUG:boto:Params: {} 2022-09-02T04:32:13.754 DEBUG:boto:Token: None 2022-09-02T04:32:13.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083bb841308cc3928-006311874d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.807 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.808 DEBUG:boto:path=/xljbkf-18/ 2022-09-02T04:32:13.808 DEBUG:boto:auth_path=/xljbkf-18/ 2022-09-02T04:32:13.809 DEBUG:boto:path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.809 DEBUG:boto:auth_path=/xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.810 DEBUG:boto:Method: GET 2022-09-02T04:32:13.811 DEBUG:boto:Path: /xljbkf-18/?prefix=obj-test-zone3 2022-09-02T04:32:13.811 DEBUG:boto:Data: 2022-09-02T04:32:13.812 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:32:13.813 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.813 DEBUG:boto:Params: {} 2022-09-02T04:32:13.814 DEBUG:boto:Token: None 2022-09-02T04:32:13.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-18/ 2022-09-02T04:32:13.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs= 2022-09-02T04:32:13.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSCnn3pc+cj3HKgWOQjXunDw8gs=', 'Content-Length': '0'} 2022-09-02T04:32:13.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000226edb0dae23e104-006311874d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.819 DEBUG:boto:b'xljbkf-18obj-test-zone31000falseobj-test-zone32022-09-02T04:29:02.808Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:32:13.820 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-18 zones={test-zone3, test-zone2} 2022-09-02T04:32:13.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2022-09-02T04:32:13.825 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:32:13.825 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:32:13.826 DEBUG:boto:Method: PUT 2022-09-02T04:32:13.826 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:32:13.827 DEBUG:boto:Data: 2022-09-02T04:32:13.827 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:13.828 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.829 DEBUG:boto:Params: {} 2022-09-02T04:32:13.829 DEBUG:boto:Token: None 2022-09-02T04:32:13.830 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-19/ 2022-09-02T04:32:13.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fzec3/1M6LpC80xhky8O12cE6lY= 2022-09-02T04:32:13.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fzec3/1M6LpC80xhky8O12cE6lY=', 'Content-Length': '0'} 2022-09-02T04:32:13.848 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce231355890bbb25-006311874d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.850 DEBUG:rgw_multi.tests:created bucket=xljbkf-19 2022-09-02T04:32:13.850 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:32:13.851 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:32:13.851 DEBUG:boto:path=/xljbkf-19/?versioning 2022-09-02T04:32:13.852 DEBUG:boto:auth_path=/xljbkf-19/?versioning 2022-09-02T04:32:13.852 DEBUG:boto:Method: PUT 2022-09-02T04:32:13.853 DEBUG:boto:Path: /xljbkf-19/?versioning 2022-09-02T04:32:13.854 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:32:13.855 DEBUG:boto:Headers: {} 2022-09-02T04:32:13.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:13.857 DEBUG:boto:Port: 8000 2022-09-02T04:32:13.857 DEBUG:boto:Params: {} 2022-09-02T04:32:13.859 DEBUG:boto:Token: None 2022-09-02T04:32:13.859 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:32:13 GMT /xljbkf-19/?versioning 2022-09-02T04:32:13.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rAaDL0iEviYThFqCveozH0oYk0M= 2022-09-02T04:32:13.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rAaDL0iEviYThFqCveozH0oYk0M=', 'Content-Length': '228'} 2022-09-02T04:32:13.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d9b90ca2a10e0a6-006311874d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:13.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:13.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:13.869 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:32:13.925 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:13.925 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:13.974 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:32:13.999 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:32:13.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:14.000 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:32:14.000 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:32:14.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:14.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:14.001 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:32:14.001 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:32:14.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:14.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:14.002 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093133.866884_134.1", 2022-09-02T04:32:14.002 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:32:13.866884Z" 2022-09-02T04:32:14.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:14.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:14.003 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:32:14.003 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:32:14.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:32:14.004 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:32:14.005 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:14.005 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:32:14.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:14.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:14.007 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:32:14.064 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:14.065 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:14.103 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:14.102+0000 7f6b76efe600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:14.104 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:14.103+0000 7f6b76efe600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:14.118 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:32:14.119 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:32:14.119 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:32:14.119 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:32:14.120 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:32:14.120 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:32:14.120 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:14.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:14.121 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:32:14.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:14.121 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:32:14.122 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:14.122 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:14.122 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:32:14.123 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:14.123 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:32:14.123 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:14.123 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:32:14.124 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:14.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:14.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:14.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:14.125 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:32:14.125 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:14.126 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:14.126 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:32:14.126 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:32:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:14.129 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:32:14.129 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:14.129 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:14.129 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093133.839577_129.1", 2022-09-02T04:32:14.130 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:14.130 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:14.130 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:14.131 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.839577Z", 2022-09-02T04:32:14.131 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:14.131 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:14.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:14.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:14.133 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:32:14.133 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:14.133 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:14.133 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:32:14.134 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:14.134 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:14.134 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:14.135 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:32:14.135 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:14.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:14.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:14.136 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:32:14.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:14.137 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:32:14.137 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:32:14.137 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:32:14.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:14.138 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:32:14Z" 2022-09-02T04:32:14.138 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:32:14.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093133.839577_129.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:32:13.839577Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:32:14Z" } 2022-09-02T04:32:14.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093133.839577_129.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:32:13.839577Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:32:14.141 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:14.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.839577_129.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:14.142 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662093133.866884_134.1 target=1_1662093133.839577_129.1 2022-09-02T04:32:19.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:19.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:19.145 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:32:19.201 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:19.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:19.240 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:19.241+0000 7f26a0b90600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:19.241 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:19.242+0000 7f26a0b90600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:19.259 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:32:19.259 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:32:19.260 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:32:19.260 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:32:19.260 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:32:19.260 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:32:19.261 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.261 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:32:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:32:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.263 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:32:19.263 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:32:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:32:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:32:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:32:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:32:19.268 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.269 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:32:19.269 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.269 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.270 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093133.866884_134.1", 2022-09-02T04:32:19.270 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.270 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:19.270 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.271 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.866884Z", 2022-09-02T04:32:19.271 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.271 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.272 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:32:19.272 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.272 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.273 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:32:19.273 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.273 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:19.273 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.274 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:32:19.274 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.275 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:32:19.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.275 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:32:19.275 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:32:19.276 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:32:19.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.276 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:32:19Z" 2022-09-02T04:32:19.277 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:32:19.278 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093133.866884_134.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:32:13.866884Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:32:19Z" } 2022-09-02T04:32:19.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093133.866884_134.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:32:13.866884Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:32:19.279 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:19.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:19.280 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:32:19.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:32:19.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:19.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:19.282 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:32:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:19.371 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:19.371+0000 7fccdee3f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:19.372 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:19.372+0000 7fccdee3f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:19.385 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:32:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:32:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:32:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:32:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:32:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:32:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:32:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:32:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:32:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:32:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:32:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:32:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:32:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:32:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:32:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:32:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:32:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:32:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:32:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:32:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:32:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:32:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:32:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:32:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:32:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:32:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:32:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:32:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:32:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:32:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:32:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093133.866884_134.1", 2022-09-02T04:32:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:32:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:32:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:32:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.866884Z", 2022-09-02T04:32:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:32:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:32:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:32:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:32:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:32:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:32:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:32:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:32:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:32:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:32:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:32:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:32:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:32:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:32:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:32:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:32:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:32:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:32:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:32:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:32:19Z" 2022-09-02T04:32:19.403 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:32:19.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093133.866884_134.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:32:13.866884Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:32:19Z" } 2022-09-02T04:32:19.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093133.866884_134.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:32:13.866884Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:32:19.405 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:19.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:19.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:32:19.407 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:32:19.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:19.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:32:19.408 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:32:19.460 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:19.461 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:19.489 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:19.489+0000 7f2fc9ca7600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:19.490 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:19.490+0000 7f2fc9ca7600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:32:19.501 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:32:19.501 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:32:19.502 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:32:19.503 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:32:19.503 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:32:19.503 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:32:19.504 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.505 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:32:19.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.505 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:32:19.506 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.506 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.507 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:32:19.507 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.508 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:32:19.508 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.508 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:32:19.509 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.510 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:32:19.510 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.510 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.510 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:32:19.511 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.511 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:19.511 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.512 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:32:19.512 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.512 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.513 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:32:19.513 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.514 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.514 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093133.866884_134.1", 2022-09-02T04:32:19.514 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.515 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:19.515 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.515 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.866884Z", 2022-09-02T04:32:19.515 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:32:19.517 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:32:19.517 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:32:19.518 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:32:19.518 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:32:19.518 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:32:19.518 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:32:19.519 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:32:19.519 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:32:19.519 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:32:19.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.520 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:32:19.520 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:32:19.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.521 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:32:19.521 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:32:19.521 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:32:19.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:32:19.522 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:32:19Z" 2022-09-02T04:32:19.522 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:32:19.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093133.866884_134.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:32:13.866884Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:32:19Z" } 2022-09-02T04:32:19.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093133.866884_134.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:32:13.866884Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:32:19.525 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:19.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:32:19.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:32:19.527 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:32:19.528 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:32:19.528 DEBUG:boto:Method: HEAD 2022-09-02T04:32:19.529 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:32:19.529 DEBUG:boto:Data: 2022-09-02T04:32:19.530 DEBUG:boto:Headers: {} 2022-09-02T04:32:19.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:19.531 DEBUG:boto:Port: 8000 2022-09-02T04:32:19.531 DEBUG:boto:Params: {} 2022-09-02T04:32:19.532 DEBUG:boto:Token: None 2022-09-02T04:32:19.533 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:32:19 GMT /xljbkf-19/ 2022-09-02T04:32:19.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SadokusNatCB85dwq1L9PjNdhKA= 2022-09-02T04:32:19.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:32:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SadokusNatCB85dwq1L9PjNdhKA=', 'Content-Length': '0'} 2022-09-02T04:32:19.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000745b2e37cb03c3c8-0063118753-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:19.538 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:32:19.538 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:32:19.539 DEBUG:boto:Method: PUT 2022-09-02T04:32:19.540 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:32:19.541 DEBUG:boto:Data: 2022-09-02T04:32:19.542 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:32:19.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:32:19.544 DEBUG:boto:Port: 8000 2022-09-02T04:32:19.545 DEBUG:boto:Params: {} 2022-09-02T04:32:19.546 DEBUG:boto:Token: None 2022-09-02T04:32:19.547 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:32:19 GMT /xljbkf-19/dummy 2022-09-02T04:32:19.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T5m/UtehOq30TtXnEQHGzScXIMc= 2022-09-02T04:32:19.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:32:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T5m/UtehOq30TtXnEQHGzScXIMc='} 2022-09-02T04:32:19.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-amz-request-id', 'tx00000e1e34ea79cf82cc1-0063118753-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:32:19.560 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-19 2022-09-02T04:32:19.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:19.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:19.562 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:19.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:19.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:19.653 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:19.653+0000 7f74a25e9600 1 waiting to reach incremental sync.. 2022-09-02T04:32:24.655 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:24.655+0000 7f74a25e9600 1 bucket sync caught up with source: 2022-09-02T04:32:24.655 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.32.10, , , , , ] 2022-09-02T04:32:24.656 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.32.10, , , , , ] 2022-09-02T04:32:24.656 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:24.655+0000 7f74a25e9600 0 bucket checkpoint complete 2022-09-02T04:32:24.675 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-19 2022-09-02T04:32:24.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:24.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:24.678 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:24.791 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:24.791+0000 7f0e40f40600 1 bucket sync caught up with source: 2022-09-02T04:32:24.792 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.32.10, , , , , ] 2022-09-02T04:32:24.792 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.32.10, , , , , ] 2022-09-02T04:32:24.793 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:24.791+0000 7f0e40f40600 0 bucket checkpoint complete 2022-09-02T04:32:24.803 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-19 2022-09-02T04:32:24.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:24.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:24.805 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:24.868 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:24.869 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:24.912 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:24.912+0000 7fdc09acc600 1 waiting to reach incremental sync.. 2022-09-02T04:32:29.913 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:29.913+0000 7fdc09acc600 1 waiting to reach incremental sync.. 2022-09-02T04:32:34.915 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:34.914+0000 7fdc09acc600 1 bucket sync caught up with source: 2022-09-02T04:32:34.916 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000002.32.10, , , , , ] 2022-09-02T04:32:34.916 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000002.32.10, , , , , ] 2022-09-02T04:32:34.917 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:32:34.914+0000 7fdc09acc600 0 bucket checkpoint complete 2022-09-02T04:32:34.930 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-19 2022-09-02T04:32:34.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:34.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:34.932 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:34.996 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:34.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:35.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:35.038+0000 7fe7f19a5600 1 waiting to reach incremental sync.. 2022-09-02T04:32:40.042 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:40.042+0000 7fe7f19a5600 1 waiting to reach incremental sync.. 2022-09-02T04:32:45.044 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:45.043+0000 7fe7f19a5600 1 bucket sync caught up with source: 2022-09-02T04:32:45.044 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:32:45.045 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:32:45.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:45.043+0000 7fe7f19a5600 0 bucket checkpoint complete 2022-09-02T04:32:45.063 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-19 2022-09-02T04:32:45.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:45.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:32:45.065 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:32:45.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:32:45.127 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:32:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:45.185+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:32:45.186 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:32:45.186 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:32:50.189 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:50.186+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:32:50.189 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:32:50.190 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:32:55.188 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:32:55.188+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:32:55.189 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:32:55.189 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:00.190 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:00.189+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:33:00.190 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:33:00.190 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:05.191 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:05.191+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:33:05.191 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:33:05.192 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:10.192 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:10.192+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:33:10.193 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:33:10.193 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:15.194 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:15.193+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:33:15.194 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:33:15.194 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:20.195 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:20.195+0000 7ff51fb55600 1 waiting for incremental sync to catch up: 2022-09-02T04:33:20.196 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.55.10, , , , , ] 2022-09-02T04:33:20.196 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:25.197 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:25.196+0000 7ff51fb55600 1 bucket sync caught up with source: 2022-09-02T04:33:25.197 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:25.198 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.57.9, , , , , ] 2022-09-02T04:33:25.198 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:25.196+0000 7ff51fb55600 0 bucket checkpoint complete 2022-09-02T04:33:25.224 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-19 2022-09-02T04:33:25.226 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-19 2022-09-02T04:33:25.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:33:25.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:33:25.229 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:33:25.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:33:25.290 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:33:25.327 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:25.326+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:33:30.327 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:30.327+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:33:35.328 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:35.327+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:33:40.328 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:40.328+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:33:45.329 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:45.329+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:33:50.330 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:50.329+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:33:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:33:55.330+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:00.331 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:00.330+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:05.331 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:05.331+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:10.337 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:10.331+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:15.333 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:15.332+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:20.333 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:20.333+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:25.333 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:25.333+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:30.334 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:30.334+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:35.334 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:35.334+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:40.335 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:40.335+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:45.336 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:45.335+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:50.336 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:50.336+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:34:55.337 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:34:55.337+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:00.338 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:00.337+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:05.338 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:05.338+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:10.339 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:10.339+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:15.340 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:15.340+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:20.340 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:20.341+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:25.341 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:25.341+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:30.342+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:35.342 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:35.342+0000 7f705512a600 1 waiting to reach incremental sync.. 2022-09-02T04:35:40.344 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:40.344+0000 7f705512a600 1 bucket sync caught up with source: 2022-09-02T04:35:40.345 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , , , ] 2022-09-02T04:35:40.345 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , , , ] 2022-09-02T04:35:40.345 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:35:40.344+0000 7f705512a600 0 bucket checkpoint complete 2022-09-02T04:35:40.369 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-19 2022-09-02T04:35:40.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:35:40.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:35:40.373 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:35:40.438 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:35:40.439 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:35:40.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:35:40.477+0000 7fd441c45600 1 bucket sync caught up with source: 2022-09-02T04:35:40.476 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , , , ] 2022-09-02T04:35:40.477 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , , , ] 2022-09-02T04:35:40.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:35:40.477+0000 7fd441c45600 0 bucket checkpoint complete 2022-09-02T04:35:40.489 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-19 2022-09-02T04:35:40.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-19 zones={test-zone2, test-zone1} 2022-09-02T04:35:40.491 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.491 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.492 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.493 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:40.493 DEBUG:boto:Data: 2022-09-02T04:35:40.494 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.495 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.495 DEBUG:boto:Params: {} 2022-09-02T04:35:40.496 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:35:40.497 DEBUG:boto:Token: None 2022-09-02T04:35:40.498 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs= 2022-09-02T04:35:40.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs=', 'Content-Length': '0'} 2022-09-02T04:35:40.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fb0830f3c17959b-006311881c-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.507 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.508 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.509 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.509 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:40.510 DEBUG:boto:Data: 2022-09-02T04:35:40.510 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.512 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.512 DEBUG:boto:Params: {} 2022-09-02T04:35:40.513 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:35:40.513 DEBUG:boto:Token: None 2022-09-02T04:35:40.514 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs= 2022-09-02T04:35:40.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs=', 'Content-Length': '0'} 2022-09-02T04:35:40.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b48b3bb1efce1dd5-006311881c-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:35:40.522 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.522 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.523 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.523 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.524 DEBUG:boto:Method: GET 2022-09-02T04:35:40.524 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.525 DEBUG:boto:Data: 2022-09-02T04:35:40.525 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.526 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.527 DEBUG:boto:Params: {} 2022-09-02T04:35:40.528 DEBUG:boto:Token: None 2022-09-02T04:35:40.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005088ac01b787d622-006311881c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.534 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.535 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:35:40.536 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:35:40.537 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.537 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.538 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.538 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.539 DEBUG:boto:Method: GET 2022-09-02T04:35:40.539 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.540 DEBUG:boto:Data: 2022-09-02T04:35:40.540 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.541 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.542 DEBUG:boto:Params: {} 2022-09-02T04:35:40.542 DEBUG:boto:Token: None 2022-09-02T04:35:40.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c802ab4fae729120-006311881c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.547 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.548 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:35:40.549 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.549 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.550 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.551 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.551 DEBUG:boto:Method: GET 2022-09-02T04:35:40.552 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.552 DEBUG:boto:Data: 2022-09-02T04:35:40.553 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.554 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.554 DEBUG:boto:Params: {} 2022-09-02T04:35:40.555 DEBUG:boto:Token: None 2022-09-02T04:35:40.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87b188a5127c88c-006311881c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.560 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.561 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.561 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.562 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.562 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.563 DEBUG:boto:Method: GET 2022-09-02T04:35:40.563 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.564 DEBUG:boto:Data: 2022-09-02T04:35:40.564 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.565 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.566 DEBUG:boto:Params: {} 2022-09-02T04:35:40.566 DEBUG:boto:Token: None 2022-09-02T04:35:40.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee834ccc22445dc1-006311881c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.571 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:40.573 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.573 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.574 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.574 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.575 DEBUG:boto:Method: GET 2022-09-02T04:35:40.575 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.576 DEBUG:boto:Data: 2022-09-02T04:35:40.577 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.578 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.578 DEBUG:boto:Params: {} 2022-09-02T04:35:40.579 DEBUG:boto:Token: None 2022-09-02T04:35:40.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ef91e0aee64bec9-006311881c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.584 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.584 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.585 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.585 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.586 DEBUG:boto:Method: GET 2022-09-02T04:35:40.586 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.587 DEBUG:boto:Data: 2022-09-02T04:35:40.587 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.588 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.589 DEBUG:boto:Params: {} 2022-09-02T04:35:40.590 DEBUG:boto:Token: None 2022-09-02T04:35:40.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ccf843b24a821e8-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.594 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.595 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.595 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.596 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.597 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.597 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.598 DEBUG:boto:Data: 2022-09-02T04:35:40.598 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.599 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.600 DEBUG:boto:Params: {} 2022-09-02T04:35:40.600 DEBUG:boto:Token: None 2022-09-02T04:35:40.601 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM= 2022-09-02T04:35:40.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM=', 'Content-Length': '0'} 2022-09-02T04:35:40.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000839cd4b8cc32395f-006311881c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.605 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.605 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.606 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.606 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.607 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.607 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.608 DEBUG:boto:Data: 2022-09-02T04:35:40.608 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.609 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.610 DEBUG:boto:Params: {} 2022-09-02T04:35:40.610 DEBUG:boto:Token: None 2022-09-02T04:35:40.611 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM= 2022-09-02T04:35:40.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM=', 'Content-Length': '0'} 2022-09-02T04:35:40.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a400bad947d5c323-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:40.615 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.616 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.616 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.617 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.618 DEBUG:boto:Method: GET 2022-09-02T04:35:40.619 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.620 DEBUG:boto:Data: 2022-09-02T04:35:40.621 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.622 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.623 DEBUG:boto:Params: {} 2022-09-02T04:35:40.624 DEBUG:boto:Token: None 2022-09-02T04:35:40.625 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006580ca02be9b5c94-006311881c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.629 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.629 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.630 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.631 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.631 DEBUG:boto:Method: GET 2022-09-02T04:35:40.632 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.633 DEBUG:boto:Data: 2022-09-02T04:35:40.634 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.635 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.636 DEBUG:boto:Params: {} 2022-09-02T04:35:40.637 DEBUG:boto:Token: None 2022-09-02T04:35:40.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049bba59b55fd8207-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.645 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.645 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.646 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.647 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.647 DEBUG:boto:Method: GET 2022-09-02T04:35:40.648 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.649 DEBUG:boto:Data: 2022-09-02T04:35:40.649 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.651 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.651 DEBUG:boto:Params: {} 2022-09-02T04:35:40.652 DEBUG:boto:Token: None 2022-09-02T04:35:40.653 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk= 2022-09-02T04:35:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk=', 'Content-Length': '0'} 2022-09-02T04:35:40.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc4ef658f6e622c8-006311881c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.659 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.660 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.661 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.662 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.662 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.663 DEBUG:boto:Method: GET 2022-09-02T04:35:40.664 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.664 DEBUG:boto:Data: 2022-09-02T04:35:40.665 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.666 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.666 DEBUG:boto:Params: {} 2022-09-02T04:35:40.667 DEBUG:boto:Token: None 2022-09-02T04:35:40.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk= 2022-09-02T04:35:40.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk=', 'Content-Length': '0'} 2022-09-02T04:35:40.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e223fcc9c8e73f2-006311881c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.714 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.716 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.717 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.718 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.719 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:35:40.719 DEBUG:boto:Data: 2022-09-02T04:35:40.720 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.721 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.722 DEBUG:boto:Params: {} 2022-09-02T04:35:40.722 DEBUG:boto:Token: None 2022-09-02T04:35:40.723 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy 2022-09-02T04:35:40.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LMpiPvvBGuGeX5w/OAbX5ehXxNw= 2022-09-02T04:35:40.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LMpiPvvBGuGeX5w/OAbX5ehXxNw=', 'Content-Length': '0'} 2022-09-02T04:35:40.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000149adf3db50198b5-006311881c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.728 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.728 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.729 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.729 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:35:40.730 DEBUG:boto:Data: 2022-09-02T04:35:40.731 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.733 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.733 DEBUG:boto:Params: {} 2022-09-02T04:35:40.734 DEBUG:boto:Token: None 2022-09-02T04:35:40.735 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy 2022-09-02T04:35:40.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LMpiPvvBGuGeX5w/OAbX5ehXxNw= 2022-09-02T04:35:40.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LMpiPvvBGuGeX5w/OAbX5ehXxNw=', 'Content-Length': '0'} 2022-09-02T04:35:40.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e305cc51173e3d6a-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:40.742 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.743 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.744 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.744 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.745 DEBUG:boto:Method: GET 2022-09-02T04:35:40.746 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.747 DEBUG:boto:Data: 2022-09-02T04:35:40.748 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.749 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.750 DEBUG:boto:Params: {} 2022-09-02T04:35:40.750 DEBUG:boto:Token: None 2022-09-02T04:35:40.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005989ddf86e71b5c5-006311881c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.756 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.757 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.758 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.759 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.759 DEBUG:boto:Method: GET 2022-09-02T04:35:40.760 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.761 DEBUG:boto:Data: 2022-09-02T04:35:40.762 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.764 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.765 DEBUG:boto:Params: {} 2022-09-02T04:35:40.766 DEBUG:boto:Token: None 2022-09-02T04:35:40.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000daa780bbeb4edb1d-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.774 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.775 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.776 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.777 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.778 DEBUG:boto:Method: GET 2022-09-02T04:35:40.779 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.780 DEBUG:boto:Data: 2022-09-02T04:35:40.781 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:40.782 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.783 DEBUG:boto:Params: {} 2022-09-02T04:35:40.783 DEBUG:boto:Token: None 2022-09-02T04:35:40.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk= 2022-09-02T04:35:40.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk=', 'Content-Length': '0'} 2022-09-02T04:35:40.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a23cf732183d76a-006311881c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.830 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.831 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.832 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.833 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.834 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.835 DEBUG:boto:Method: GET 2022-09-02T04:35:40.835 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:40.836 DEBUG:boto:Data: 2022-09-02T04:35:40.837 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.838 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.838 DEBUG:boto:Params: {} 2022-09-02T04:35:40.839 DEBUG:boto:Token: None 2022-09-02T04:35:40.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk= 2022-09-02T04:35:40.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+MC9s2/cbBM12UmYFZlWLyGhZSk=', 'Content-Length': '0'} 2022-09-02T04:35:40.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f84d0d2f342b4bf-006311881c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.844 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-19 zones={test-zone2, test-zone1} 2022-09-02T04:35:40.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-19 zones={test-zone3, test-zone1} 2022-09-02T04:35:40.847 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.847 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.848 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.848 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:40.849 DEBUG:boto:Data: 2022-09-02T04:35:40.849 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:40.850 DEBUG:boto:Port: 8001 2022-09-02T04:35:40.851 DEBUG:boto:Params: {} 2022-09-02T04:35:40.852 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:35:40.853 DEBUG:boto:Token: None 2022-09-02T04:35:40.853 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs= 2022-09-02T04:35:40.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs=', 'Content-Length': '0'} 2022-09-02T04:35:40.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000372efbb5fe63a082-006311881c-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.861 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.861 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.862 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.862 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:40.863 DEBUG:boto:Data: 2022-09-02T04:35:40.863 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.864 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.865 DEBUG:boto:Params: {} 2022-09-02T04:35:40.865 DEBUG:boto:Token: None 2022-09-02T04:35:40.866 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/ 2022-09-02T04:35:40.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs= 2022-09-02T04:35:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mjHMbl/MPg89vjeLtJ0S0hmSKFs=', 'Content-Length': '0'} 2022-09-02T04:35:40.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de52f38ab6a9443-006311881c-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:35:40.871 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.872 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.872 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.873 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.874 DEBUG:boto:Method: GET 2022-09-02T04:35:40.874 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.875 DEBUG:boto:Data: 2022-09-02T04:35:40.875 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:40.876 DEBUG:boto:Port: 8001 2022-09-02T04:35:40.877 DEBUG:boto:Params: {} 2022-09-02T04:35:40.878 DEBUG:boto:Token: None 2022-09-02T04:35:40.878 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.879 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000874f185196511264-006311881c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.883 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.884 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:35:40.884 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:35:40.885 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.885 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.886 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.887 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.887 DEBUG:boto:Method: GET 2022-09-02T04:35:40.888 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.888 DEBUG:boto:Data: 2022-09-02T04:35:40.889 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.890 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.891 DEBUG:boto:Params: {} 2022-09-02T04:35:40.891 DEBUG:boto:Token: None 2022-09-02T04:35:40.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edcd3b9742d544b4-006311881c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.896 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.897 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:35:40.898 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.898 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.899 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.899 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.900 DEBUG:boto:Method: GET 2022-09-02T04:35:40.901 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.901 DEBUG:boto:Data: 2022-09-02T04:35:40.902 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:40.903 DEBUG:boto:Port: 8001 2022-09-02T04:35:40.904 DEBUG:boto:Params: {} 2022-09-02T04:35:40.904 DEBUG:boto:Token: None 2022-09-02T04:35:40.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aba0a900d2f6cbf-006311881c-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.911 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.912 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.913 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.914 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.915 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.915 DEBUG:boto:Method: GET 2022-09-02T04:35:40.916 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:40.917 DEBUG:boto:Data: 2022-09-02T04:35:40.918 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.919 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.919 DEBUG:boto:Params: {} 2022-09-02T04:35:40.920 DEBUG:boto:Token: None 2022-09-02T04:35:40.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/?versions 2022-09-02T04:35:40.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds= 2022-09-02T04:35:40.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:51IKwI1WJnfwxx1Xm2qt7sUM7ds=', 'Content-Length': '0'} 2022-09-02T04:35:40.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000285fd9ce382a0c41-006311881c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.925 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:40.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:40.926 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.927 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.928 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.928 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.929 DEBUG:boto:Method: GET 2022-09-02T04:35:40.930 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.930 DEBUG:boto:Data: 2022-09-02T04:35:40.931 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:40.932 DEBUG:boto:Port: 8001 2022-09-02T04:35:40.933 DEBUG:boto:Params: {} 2022-09-02T04:35:40.933 DEBUG:boto:Token: None 2022-09-02T04:35:40.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b08eec1e2c3ae0d-006311881c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.938 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.939 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.939 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.940 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.940 DEBUG:boto:Method: GET 2022-09-02T04:35:40.941 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.942 DEBUG:boto:Data: 2022-09-02T04:35:40.942 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.943 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.944 DEBUG:boto:Params: {} 2022-09-02T04:35:40.945 DEBUG:boto:Token: None 2022-09-02T04:35:40.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa8003134a965891-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.949 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.950 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.950 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.951 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.951 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.952 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.952 DEBUG:boto:Data: 2022-09-02T04:35:40.953 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:40.954 DEBUG:boto:Port: 8001 2022-09-02T04:35:40.954 DEBUG:boto:Params: {} 2022-09-02T04:35:40.955 DEBUG:boto:Token: None 2022-09-02T04:35:40.956 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM= 2022-09-02T04:35:40.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM=', 'Content-Length': '0'} 2022-09-02T04:35:40.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d4b3c4e279ba3ba3-006311881c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.960 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.961 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.961 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.962 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.962 DEBUG:boto:Method: HEAD 2022-09-02T04:35:40.963 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.963 DEBUG:boto:Data: 2022-09-02T04:35:40.964 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.965 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.966 DEBUG:boto:Params: {} 2022-09-02T04:35:40.966 DEBUG:boto:Token: None 2022-09-02T04:35:40.967 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM= 2022-09-02T04:35:40.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j0l0p43er2WbkrSK6BhGOX+txBM=', 'Content-Length': '0'} 2022-09-02T04:35:40.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef41b56ec8da06a3-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:40.971 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.972 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.973 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.973 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.974 DEBUG:boto:Method: GET 2022-09-02T04:35:40.974 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.975 DEBUG:boto:Data: 2022-09-02T04:35:40.976 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:40.977 DEBUG:boto:Port: 8001 2022-09-02T04:35:40.977 DEBUG:boto:Params: {} 2022-09-02T04:35:40.978 DEBUG:boto:Token: None 2022-09-02T04:35:40.978 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2ce7c9bd977a449-006311881c-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.983 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:40.983 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:40.984 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.984 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.985 DEBUG:boto:Method: GET 2022-09-02T04:35:40.986 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.986 DEBUG:boto:Data: 2022-09-02T04:35:40.987 DEBUG:boto:Headers: {} 2022-09-02T04:35:40.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:40.988 DEBUG:boto:Port: 8000 2022-09-02T04:35:40.989 DEBUG:boto:Params: {} 2022-09-02T04:35:40.990 DEBUG:boto:Token: None 2022-09-02T04:35:40.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:40 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:40.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE= 2022-09-02T04:35:40.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FBnirvbDkk95Rc7V/U5NC813VlE=', 'Content-Length': '0'} 2022-09-02T04:35:40.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002588f147ed0e409c-006311881c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:40.997 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:40.998 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:40.999 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.000 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.001 DEBUG:boto:Method: GET 2022-09-02T04:35:41.002 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.003 DEBUG:boto:Data: 2022-09-02T04:35:41.004 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.005 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.006 DEBUG:boto:Params: {} 2022-09-02T04:35:41.006 DEBUG:boto:Token: None 2022-09-02T04:35:41.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.008 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad7da319f0ca32a9-006311881d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.012 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.013 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.014 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.014 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.015 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.015 DEBUG:boto:Method: GET 2022-09-02T04:35:41.016 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.017 DEBUG:boto:Data: 2022-09-02T04:35:41.017 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.018 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.019 DEBUG:boto:Params: {} 2022-09-02T04:35:41.019 DEBUG:boto:Token: None 2022-09-02T04:35:41.020 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee70352657af12a-006311881d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.066 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.067 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.068 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.069 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.069 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:35:41.070 DEBUG:boto:Data: 2022-09-02T04:35:41.071 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.072 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.072 DEBUG:boto:Params: {} 2022-09-02T04:35:41.073 DEBUG:boto:Token: None 2022-09-02T04:35:41.074 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy 2022-09-02T04:35:41.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI= 2022-09-02T04:35:41.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI=', 'Content-Length': '0'} 2022-09-02T04:35:41.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044b5c0c2de2e8ccb-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.078 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.079 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.080 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.080 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:35:41.081 DEBUG:boto:Data: 2022-09-02T04:35:41.082 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.083 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.084 DEBUG:boto:Params: {} 2022-09-02T04:35:41.085 DEBUG:boto:Token: None 2022-09-02T04:35:41.086 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy 2022-09-02T04:35:41.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI= 2022-09-02T04:35:41.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI=', 'Content-Length': '0'} 2022-09-02T04:35:41.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013c2126b058669d8-006311881d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:41.094 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.094 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.095 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.095 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.096 DEBUG:boto:Method: GET 2022-09-02T04:35:41.097 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.097 DEBUG:boto:Data: 2022-09-02T04:35:41.098 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.099 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.099 DEBUG:boto:Params: {} 2022-09-02T04:35:41.100 DEBUG:boto:Token: None 2022-09-02T04:35:41.101 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ec6a36d0b0eb48f-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.105 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.106 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.107 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.108 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.109 DEBUG:boto:Method: GET 2022-09-02T04:35:41.109 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.110 DEBUG:boto:Data: 2022-09-02T04:35:41.111 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.112 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.113 DEBUG:boto:Params: {} 2022-09-02T04:35:41.114 DEBUG:boto:Token: None 2022-09-02T04:35:41.114 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002422d97f475f8412-006311881d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.120 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.120 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.121 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.122 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.122 DEBUG:boto:Method: GET 2022-09-02T04:35:41.123 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.124 DEBUG:boto:Data: 2022-09-02T04:35:41.124 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.125 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.126 DEBUG:boto:Params: {} 2022-09-02T04:35:41.126 DEBUG:boto:Token: None 2022-09-02T04:35:41.128 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f7f462ffbb6a4e5-006311881d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.134 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.135 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.136 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.137 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.138 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.139 DEBUG:boto:Method: GET 2022-09-02T04:35:41.140 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.141 DEBUG:boto:Data: 2022-09-02T04:35:41.142 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.144 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.145 DEBUG:boto:Params: {} 2022-09-02T04:35:41.146 DEBUG:boto:Token: None 2022-09-02T04:35:41.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d93bc9d063a328d-006311881d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.153 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.154 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-19 zones={test-zone3, test-zone1} 2022-09-02T04:35:41.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-19 zones={test-zone3, test-zone2} 2022-09-02T04:35:41.155 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.156 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.156 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.157 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.157 DEBUG:boto:Data: 2022-09-02T04:35:41.158 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.159 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.161 DEBUG:boto:Params: {} 2022-09-02T04:35:41.161 DEBUG:boto:Token: None 2022-09-02T04:35:41.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbef104b28752ce1-006311881d-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.168 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.169 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.170 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.171 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.172 DEBUG:boto:Data: 2022-09-02T04:35:41.173 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.174 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.175 DEBUG:boto:Params: {} 2022-09-02T04:35:41.176 DEBUG:boto:Token: None 2022-09-02T04:35:41.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077a995125e4f51d5-006311881d-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:35:41.182 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.183 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.183 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.184 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.184 DEBUG:boto:Method: GET 2022-09-02T04:35:41.185 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.186 DEBUG:boto:Data: 2022-09-02T04:35:41.186 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.187 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.188 DEBUG:boto:Params: {} 2022-09-02T04:35:41.189 DEBUG:boto:Token: None 2022-09-02T04:35:41.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/?versions 2022-09-02T04:35:41.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8= 2022-09-02T04:35:41.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8=', 'Content-Length': '0'} 2022-09-02T04:35:41.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008671b8c18e6cc1e1-006311881d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.234 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:35:41.236 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:35:41.237 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.238 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.238 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.239 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.241 DEBUG:boto:Method: GET 2022-09-02T04:35:41.242 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.243 DEBUG:boto:Data: 2022-09-02T04:35:41.244 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.246 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.247 DEBUG:boto:Params: {} 2022-09-02T04:35:41.248 DEBUG:boto:Token: None 2022-09-02T04:35:41.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/?versions 2022-09-02T04:35:41.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8= 2022-09-02T04:35:41.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8=', 'Content-Length': '0'} 2022-09-02T04:35:41.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7c28a5bab0ce2de-006311881d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.257 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.259 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:35:41.259 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.260 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.261 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.262 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.263 DEBUG:boto:Method: GET 2022-09-02T04:35:41.264 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.266 DEBUG:boto:Data: 2022-09-02T04:35:41.267 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.268 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.270 DEBUG:boto:Params: {} 2022-09-02T04:35:41.271 DEBUG:boto:Token: None 2022-09-02T04:35:41.272 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/?versions 2022-09-02T04:35:41.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8= 2022-09-02T04:35:41.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8=', 'Content-Length': '0'} 2022-09-02T04:35:41.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052591b5866c18c25-006311881d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.279 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.280 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.281 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.282 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.283 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.284 DEBUG:boto:Method: GET 2022-09-02T04:35:41.284 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:35:41.285 DEBUG:boto:Data: 2022-09-02T04:35:41.285 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.286 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.287 DEBUG:boto:Params: {} 2022-09-02T04:35:41.287 DEBUG:boto:Token: None 2022-09-02T04:35:41.288 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/?versions 2022-09-02T04:35:41.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8= 2022-09-02T04:35:41.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pRDAvQVrEKfSii9ISSEb6R+n4F8=', 'Content-Length': '0'} 2022-09-02T04:35:41.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037339aeedd33d201-006311881d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.293 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:41.294 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.295 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.295 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.296 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.296 DEBUG:boto:Method: GET 2022-09-02T04:35:41.297 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.297 DEBUG:boto:Data: 2022-09-02T04:35:41.298 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.299 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.299 DEBUG:boto:Params: {} 2022-09-02T04:35:41.300 DEBUG:boto:Token: None 2022-09-02T04:35:41.301 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f9d6960106dee02-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.305 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.305 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.306 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.306 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.307 DEBUG:boto:Method: GET 2022-09-02T04:35:41.307 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.308 DEBUG:boto:Data: 2022-09-02T04:35:41.308 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.309 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.310 DEBUG:boto:Params: {} 2022-09-02T04:35:41.311 DEBUG:boto:Token: None 2022-09-02T04:35:41.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000695000a6e177598a-006311881d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.315 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.316 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.316 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.317 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.318 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.318 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.319 DEBUG:boto:Data: 2022-09-02T04:35:41.319 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.320 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.321 DEBUG:boto:Params: {} 2022-09-02T04:35:41.321 DEBUG:boto:Token: None 2022-09-02T04:35:41.322 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWCahu5nXvPP1Npqu/QBzadmAus= 2022-09-02T04:35:41.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWCahu5nXvPP1Npqu/QBzadmAus=', 'Content-Length': '0'} 2022-09-02T04:35:41.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7ae5ffc33886c07-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.326 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.327 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.327 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.328 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.328 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.329 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.329 DEBUG:boto:Data: 2022-09-02T04:35:41.330 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.331 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.331 DEBUG:boto:Params: {} 2022-09-02T04:35:41.332 DEBUG:boto:Token: None 2022-09-02T04:35:41.332 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWCahu5nXvPP1Npqu/QBzadmAus= 2022-09-02T04:35:41.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWCahu5nXvPP1Npqu/QBzadmAus=', 'Content-Length': '0'} 2022-09-02T04:35:41.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000249ca9643ff7cbc1-006311881d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:41.337 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.337 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.338 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.339 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.339 DEBUG:boto:Method: GET 2022-09-02T04:35:41.340 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.340 DEBUG:boto:Data: 2022-09-02T04:35:41.341 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.342 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.342 DEBUG:boto:Params: {} 2022-09-02T04:35:41.343 DEBUG:boto:Token: None 2022-09-02T04:35:41.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063bfcabc7a30be8c-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.348 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.348 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.349 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.349 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.350 DEBUG:boto:Method: GET 2022-09-02T04:35:41.350 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.351 DEBUG:boto:Data: 2022-09-02T04:35:41.352 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.354 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.355 DEBUG:boto:Params: {} 2022-09-02T04:35:41.356 DEBUG:boto:Token: None 2022-09-02T04:35:41.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb39f28a45e0cf5d-006311881d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.363 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.364 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.364 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.365 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.365 DEBUG:boto:Method: GET 2022-09-02T04:35:41.366 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.366 DEBUG:boto:Data: 2022-09-02T04:35:41.367 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.368 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.368 DEBUG:boto:Params: {} 2022-09-02T04:35:41.369 DEBUG:boto:Token: None 2022-09-02T04:35:41.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.370 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000029c127f1e949a09-006311881d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.374 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.375 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.376 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.376 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.377 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.377 DEBUG:boto:Method: GET 2022-09-02T04:35:41.378 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.378 DEBUG:boto:Data: 2022-09-02T04:35:41.379 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.380 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.380 DEBUG:boto:Params: {} 2022-09-02T04:35:41.381 DEBUG:boto:Token: None 2022-09-02T04:35:41.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f69fc6428ca725d4-006311881d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.426 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.427 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.428 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.428 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.429 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:35:41.429 DEBUG:boto:Data: 2022-09-02T04:35:41.430 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.431 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.431 DEBUG:boto:Params: {} 2022-09-02T04:35:41.432 DEBUG:boto:Token: None 2022-09-02T04:35:41.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy 2022-09-02T04:35:41.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI= 2022-09-02T04:35:41.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI=', 'Content-Length': '0'} 2022-09-02T04:35:41.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000501688a02e6eb6fe-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.438 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.439 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.440 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.440 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:35:41.441 DEBUG:boto:Data: 2022-09-02T04:35:41.441 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.443 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.443 DEBUG:boto:Params: {} 2022-09-02T04:35:41.444 DEBUG:boto:Token: None 2022-09-02T04:35:41.444 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy 2022-09-02T04:35:41.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI= 2022-09-02T04:35:41.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4x0JkSD5mSk7JVsA5PqK5y7lECI=', 'Content-Length': '0'} 2022-09-02T04:35:41.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e44fe40be0cd8a25-006311881d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:35:41.449 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.450 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.451 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.451 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.452 DEBUG:boto:Method: GET 2022-09-02T04:35:41.453 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.453 DEBUG:boto:Data: 2022-09-02T04:35:41.454 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.455 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.456 DEBUG:boto:Params: {} 2022-09-02T04:35:41.456 DEBUG:boto:Token: None 2022-09-02T04:35:41.457 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002733d532db15239a-006311881d-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.461 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:35:41.462 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:35:41.462 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.463 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.464 DEBUG:boto:Method: GET 2022-09-02T04:35:41.464 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.465 DEBUG:boto:Data: 2022-09-02T04:35:41.465 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.466 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.466 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.467 DEBUG:boto:Params: {} 2022-09-02T04:35:41.468 DEBUG:boto:Token: None 2022-09-02T04:35:41.468 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:35:41.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0= 2022-09-02T04:35:41.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK22Plf4Rpys/NLCNROxfnx7eO0=', 'Content-Length': '0'} 2022-09-02T04:35:41.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f19294e5d6a8455-006311881d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.474 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.474 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.475 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.475 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.476 DEBUG:boto:Method: GET 2022-09-02T04:35:41.477 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.477 DEBUG:boto:Data: 2022-09-02T04:35:41.478 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.480 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.481 DEBUG:boto:Params: {} 2022-09-02T04:35:41.482 DEBUG:boto:Token: None 2022-09-02T04:35:41.483 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bafee2a97bad0e7e-006311881d-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.490 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.491 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.492 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.492 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.493 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.493 DEBUG:boto:Method: GET 2022-09-02T04:35:41.494 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:35:41.495 DEBUG:boto:Data: 2022-09-02T04:35:41.495 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.496 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.497 DEBUG:boto:Params: {} 2022-09-02T04:35:41.497 DEBUG:boto:Token: None 2022-09-02T04:35:41.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18= 2022-09-02T04:35:41.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Sc27twdx4tiDzGLX7Li2jO7lw18=', 'Content-Length': '0'} 2022-09-02T04:35:41.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db194cd0c7a298a4-006311881d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.503 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:35:41.504 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-19 zones={test-zone3, test-zone2} 2022-09-02T04:35:41.505 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.505 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.506 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.506 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.507 DEBUG:boto:Data: 2022-09-02T04:35:41.507 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.508 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.509 DEBUG:boto:Params: {} 2022-09-02T04:35:41.510 DEBUG:boto:Token: None 2022-09-02T04:35:41.511 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000761b497e9a80bb74-006311881d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.517 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.518 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.519 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.519 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.520 DEBUG:boto:Data: 2022-09-02T04:35:41.520 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.521 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.522 DEBUG:boto:Params: {} 2022-09-02T04:35:41.523 DEBUG:boto:Token: None 2022-09-02T04:35:41.524 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-amz-request-id', 'tx000002e97b7189a8d29ab-006311881d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.546 DEBUG:rgw_multi.tests:zone=test-zone1 version=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:35:41.547 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.549 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.550 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.551 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.553 DEBUG:boto:Data: 2022-09-02T04:35:41.554 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.555 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.555 DEBUG:boto:Params: {} 2022-09-02T04:35:41.556 DEBUG:boto:Token: None 2022-09-02T04:35:41.557 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac454bc7c1d1bc0f-006311881d-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.562 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.562 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.563 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.563 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.564 DEBUG:boto:Data: 2022-09-02T04:35:41.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.565 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.566 DEBUG:boto:Params: {} 2022-09-02T04:35:41.566 DEBUG:boto:Token: None 2022-09-02T04:35:41.567 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-amz-request-id', 'tx0000031a5bf41aac883a7-006311881d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.582 DEBUG:rgw_multi.tests:zone=test-zone2 version=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:35:41.583 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.584 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.585 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.585 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.586 DEBUG:boto:Data: 2022-09-02T04:35:41.586 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.588 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.589 DEBUG:boto:Params: {} 2022-09-02T04:35:41.589 DEBUG:boto:Token: None 2022-09-02T04:35:41.590 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015323a0054335d09-006311881d-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.595 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.595 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.596 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.596 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.597 DEBUG:boto:Data: 2022-09-02T04:35:41.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.598 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.599 DEBUG:boto:Params: {} 2022-09-02T04:35:41.600 DEBUG:boto:Token: None 2022-09-02T04:35:41.601 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-amz-request-id', 'tx0000019e43ba5c67f8c31-006311881d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.621 DEBUG:rgw_multi.tests:zone=test-zone3 version=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:35:41.622 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.623 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.623 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.624 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.624 DEBUG:boto:Data: 2022-09-02T04:35:41.625 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.626 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.627 DEBUG:boto:Params: {} 2022-09-02T04:35:41.628 DEBUG:boto:Token: None 2022-09-02T04:35:41.629 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f12c1c2b49443e5-006311881d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.634 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.635 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.635 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.636 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.636 DEBUG:boto:Data: 2022-09-02T04:35:41.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.638 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.639 DEBUG:boto:Params: {} 2022-09-02T04:35:41.639 DEBUG:boto:Token: None 2022-09-02T04:35:41.640 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-amz-request-id', 'tx0000087b9f47e63dfbed4-006311881d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.656 DEBUG:rgw_multi.tests:zone=test-zone1 version=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:35:41.657 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.657 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.658 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.658 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.659 DEBUG:boto:Data: 2022-09-02T04:35:41.659 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.660 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.661 DEBUG:boto:Params: {} 2022-09-02T04:35:41.661 DEBUG:boto:Token: None 2022-09-02T04:35:41.662 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061d6a4d49e660f86-006311881d-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.670 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.671 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.672 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.673 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.674 DEBUG:boto:Data: 2022-09-02T04:35:41.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.676 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.677 DEBUG:boto:Params: {} 2022-09-02T04:35:41.678 DEBUG:boto:Token: None 2022-09-02T04:35:41.679 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-amz-request-id', 'tx000009cef9acdf97c9610-006311881d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.696 DEBUG:rgw_multi.tests:zone=test-zone2 version=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:35:41.697 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.697 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.698 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.699 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.699 DEBUG:boto:Data: 2022-09-02T04:35:41.700 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.701 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.701 DEBUG:boto:Params: {} 2022-09-02T04:35:41.702 DEBUG:boto:Token: None 2022-09-02T04:35:41.703 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000680d727bf57a3ed1-006311881d-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.708 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.709 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.709 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.710 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.711 DEBUG:boto:Data: 2022-09-02T04:35:41.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.714 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.715 DEBUG:boto:Params: {} 2022-09-02T04:35:41.716 DEBUG:boto:Token: None 2022-09-02T04:35:41.717 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-amz-request-id', 'tx00000f03b97d5eff02e69-006311881d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.731 DEBUG:rgw_multi.tests:zone=test-zone3 version=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:35:41.732 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.732 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.733 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.733 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.734 DEBUG:boto:Data: 2022-09-02T04:35:41.735 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.736 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.736 DEBUG:boto:Params: {} 2022-09-02T04:35:41.737 DEBUG:boto:Token: None 2022-09-02T04:35:41.738 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b025910ba41791b1-006311881d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.742 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.742 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.743 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.744 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.744 DEBUG:boto:Data: 2022-09-02T04:35:41.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.746 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.746 DEBUG:boto:Params: {} 2022-09-02T04:35:41.747 DEBUG:boto:Token: None 2022-09-02T04:35:41.748 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-amz-request-id', 'tx0000022245821f79b8625-006311881d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.760 DEBUG:rgw_multi.tests:zone=test-zone1 version=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:35:41.761 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.761 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.762 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.762 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.763 DEBUG:boto:Data: 2022-09-02T04:35:41.763 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.764 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.765 DEBUG:boto:Params: {} 2022-09-02T04:35:41.766 DEBUG:boto:Token: None 2022-09-02T04:35:41.766 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f75478d9b85c18f-006311881d-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.772 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.773 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.773 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.774 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.774 DEBUG:boto:Data: 2022-09-02T04:35:41.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.775 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.776 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.776 DEBUG:boto:Params: {} 2022-09-02T04:35:41.777 DEBUG:boto:Token: None 2022-09-02T04:35:41.778 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-amz-request-id', 'tx00000593f4e9231830ffd-006311881d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.823 DEBUG:rgw_multi.tests:zone=test-zone2 version=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:35:41.823 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.824 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.824 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.825 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.826 DEBUG:boto:Data: 2022-09-02T04:35:41.826 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.827 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.828 DEBUG:boto:Params: {} 2022-09-02T04:35:41.828 DEBUG:boto:Token: None 2022-09-02T04:35:41.829 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d8f0ba0de4e7829-006311881d-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.833 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.834 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.834 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.835 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.835 DEBUG:boto:Data: 2022-09-02T04:35:41.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.837 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.837 DEBUG:boto:Params: {} 2022-09-02T04:35:41.838 DEBUG:boto:Token: None 2022-09-02T04:35:41.838 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-amz-request-id', 'tx000003ddd7130b2ef8deb-006311881d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.850 DEBUG:rgw_multi.tests:zone=test-zone3 version=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:35:41.851 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.851 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.852 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.852 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.853 DEBUG:boto:Data: 2022-09-02T04:35:41.853 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.854 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.855 DEBUG:boto:Params: {} 2022-09-02T04:35:41.855 DEBUG:boto:Token: None 2022-09-02T04:35:41.856 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b76ebec4b32f24-006311881d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.861 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.862 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.863 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.864 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.865 DEBUG:boto:Data: 2022-09-02T04:35:41.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.869 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.870 DEBUG:boto:Params: {} 2022-09-02T04:35:41.871 DEBUG:boto:Token: None 2022-09-02T04:35:41.873 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-amz-request-id', 'tx000002fd837e9f91d9c03-006311881d-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.892 DEBUG:rgw_multi.tests:zone=test-zone1 version=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:35:41.893 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.894 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.896 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.898 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.899 DEBUG:boto:Data: 2022-09-02T04:35:41.900 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.902 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.902 DEBUG:boto:Params: {} 2022-09-02T04:35:41.903 DEBUG:boto:Token: None 2022-09-02T04:35:41.904 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ea753291bedbe98-006311881d-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.911 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.912 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.913 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.913 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.914 DEBUG:boto:Data: 2022-09-02T04:35:41.914 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:41.915 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.916 DEBUG:boto:Params: {} 2022-09-02T04:35:41.916 DEBUG:boto:Token: None 2022-09-02T04:35:41.917 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-amz-request-id', 'tx00000eb488b29a23239f8-006311881d-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.934 DEBUG:rgw_multi.tests:zone=test-zone2 version=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:35:41.935 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.936 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.937 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.938 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.939 DEBUG:boto:Data: 2022-09-02T04:35:41.939 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.941 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.944 DEBUG:boto:Params: {} 2022-09-02T04:35:41.946 DEBUG:boto:Token: None 2022-09-02T04:35:41.947 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:41.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c2ea73ec512563d-006311881d-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.954 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:41.955 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:41.955 DEBUG:boto:Method: PUT 2022-09-02T04:35:41.956 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:41.956 DEBUG:boto:Data: 2022-09-02T04:35:41.957 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:41.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:41.958 DEBUG:boto:Port: 8001 2022-09-02T04:35:41.959 DEBUG:boto:Params: {} 2022-09-02T04:35:41.960 DEBUG:boto:Token: None 2022-09-02T04:35:41.960 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/obj 2022-09-02T04:35:41.961 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM= 2022-09-02T04:35:41.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JY7nYCOw6HVWYHl1PJYHlNDLlaM='} 2022-09-02T04:35:41.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-amz-request-id', 'tx00000437411ee64bb4494-006311881d-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:41.984 DEBUG:rgw_multi.tests:zone=test-zone3 version=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:35:41.985 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:41.986 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:41.987 DEBUG:boto:Method: HEAD 2022-09-02T04:35:41.988 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:41.989 DEBUG:boto:Data: 2022-09-02T04:35:41.990 DEBUG:boto:Headers: {} 2022-09-02T04:35:41.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:41.992 DEBUG:boto:Port: 8000 2022-09-02T04:35:41.992 DEBUG:boto:Params: {} 2022-09-02T04:35:41.994 DEBUG:boto:Token: None 2022-09-02T04:35:41.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:41 GMT /xljbkf-19/ 2022-09-02T04:35:41.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw= 2022-09-02T04:35:41.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6pJQjKB2MFoxoFpFNL73VdFxhhw=', 'Content-Length': '0'} 2022-09-02T04:35:42.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f50fec9bfcbc8a2-006311881d-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.001 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.002 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.004 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.005 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.005 DEBUG:boto:Data: 2022-09-02T04:35:42.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.008 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.009 DEBUG:boto:Params: {} 2022-09-02T04:35:42.010 DEBUG:boto:Token: None 2022-09-02T04:35:42.011 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-amz-request-id', 'tx00000966da7cc98928c74-006311881e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.036 DEBUG:rgw_multi.tests:zone=test-zone1 version=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:35:42.036 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.037 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.038 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.039 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.039 DEBUG:boto:Data: 2022-09-02T04:35:42.041 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.043 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.044 DEBUG:boto:Params: {} 2022-09-02T04:35:42.045 DEBUG:boto:Token: None 2022-09-02T04:35:42.047 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000186c0839577027d2-006311881e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.052 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.052 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.053 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.053 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.054 DEBUG:boto:Data: 2022-09-02T04:35:42.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.056 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.058 DEBUG:boto:Params: {} 2022-09-02T04:35:42.059 DEBUG:boto:Token: None 2022-09-02T04:35:42.059 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-amz-request-id', 'tx0000089bf31bfa7bbb8b0-006311881e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.077 DEBUG:rgw_multi.tests:zone=test-zone2 version=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:35:42.078 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.078 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.079 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.080 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.081 DEBUG:boto:Data: 2022-09-02T04:35:42.082 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.084 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.085 DEBUG:boto:Params: {} 2022-09-02T04:35:42.086 DEBUG:boto:Token: None 2022-09-02T04:35:42.087 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ba1182cebf16897-006311881e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.093 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.094 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.094 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.095 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.095 DEBUG:boto:Data: 2022-09-02T04:35:42.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.097 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.097 DEBUG:boto:Params: {} 2022-09-02T04:35:42.098 DEBUG:boto:Token: None 2022-09-02T04:35:42.098 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-amz-request-id', 'tx000008a290fce72f79673-006311881e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.121 DEBUG:rgw_multi.tests:zone=test-zone3 version=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:35:42.122 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.123 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.124 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.125 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.126 DEBUG:boto:Data: 2022-09-02T04:35:42.127 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.128 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.129 DEBUG:boto:Params: {} 2022-09-02T04:35:42.130 DEBUG:boto:Token: None 2022-09-02T04:35:42.131 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2241f71fa8c3776-006311881e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.135 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.136 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.136 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.137 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.137 DEBUG:boto:Data: 2022-09-02T04:35:42.138 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.139 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.140 DEBUG:boto:Params: {} 2022-09-02T04:35:42.141 DEBUG:boto:Token: None 2022-09-02T04:35:42.142 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-amz-request-id', 'tx0000003b3713964cf2aeb-006311881e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.191 DEBUG:rgw_multi.tests:zone=test-zone1 version=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:35:42.192 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.192 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.193 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.193 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.194 DEBUG:boto:Data: 2022-09-02T04:35:42.194 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.195 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.195 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.196 DEBUG:boto:Params: {} 2022-09-02T04:35:42.197 DEBUG:boto:Token: None 2022-09-02T04:35:42.198 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eb3b1005384ccc9-006311881e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.202 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.203 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.203 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.204 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.205 DEBUG:boto:Data: 2022-09-02T04:35:42.205 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.206 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.207 DEBUG:boto:Params: {} 2022-09-02T04:35:42.207 DEBUG:boto:Token: None 2022-09-02T04:35:42.208 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-amz-request-id', 'tx00000f0bba13319bd7ace-006311881e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.221 DEBUG:rgw_multi.tests:zone=test-zone2 version=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:35:42.222 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.222 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.223 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.223 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.224 DEBUG:boto:Data: 2022-09-02T04:35:42.224 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.225 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.226 DEBUG:boto:Params: {} 2022-09-02T04:35:42.227 DEBUG:boto:Token: None 2022-09-02T04:35:42.227 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000932c0f1296753559-006311881e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.231 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.232 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.233 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.233 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.233 DEBUG:boto:Data: 2022-09-02T04:35:42.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.235 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.236 DEBUG:boto:Params: {} 2022-09-02T04:35:42.236 DEBUG:boto:Token: None 2022-09-02T04:35:42.237 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-amz-request-id', 'tx00000be9c1f32fd1aee72-006311881e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.283 DEBUG:rgw_multi.tests:zone=test-zone3 version=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:35:42.284 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.284 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.285 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.285 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.286 DEBUG:boto:Data: 2022-09-02T04:35:42.286 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.287 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.288 DEBUG:boto:Params: {} 2022-09-02T04:35:42.288 DEBUG:boto:Token: None 2022-09-02T04:35:42.289 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce45358f0a6d0c67-006311881e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.293 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.293 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.294 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.295 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.295 DEBUG:boto:Data: 2022-09-02T04:35:42.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.297 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.297 DEBUG:boto:Params: {} 2022-09-02T04:35:42.298 DEBUG:boto:Token: None 2022-09-02T04:35:42.298 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-amz-request-id', 'tx000002d9831f80f910bc0-006311881e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.311 DEBUG:rgw_multi.tests:zone=test-zone1 version=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:35:42.312 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.312 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.313 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.313 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.314 DEBUG:boto:Data: 2022-09-02T04:35:42.314 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.315 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.316 DEBUG:boto:Params: {} 2022-09-02T04:35:42.316 DEBUG:boto:Token: None 2022-09-02T04:35:42.317 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000061ecc04074d4d15-006311881e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.322 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.322 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.323 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.323 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.324 DEBUG:boto:Data: 2022-09-02T04:35:42.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.325 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.325 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.326 DEBUG:boto:Params: {} 2022-09-02T04:35:42.326 DEBUG:boto:Token: None 2022-09-02T04:35:42.327 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-amz-request-id', 'tx00000a97b64a2a22cf981-006311881e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.341 DEBUG:rgw_multi.tests:zone=test-zone2 version=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:35:42.342 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.342 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.343 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.343 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.344 DEBUG:boto:Data: 2022-09-02T04:35:42.345 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.346 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.346 DEBUG:boto:Params: {} 2022-09-02T04:35:42.347 DEBUG:boto:Token: None 2022-09-02T04:35:42.348 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cac21898eeb4c7b1-006311881e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.355 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.356 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.357 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.358 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.359 DEBUG:boto:Data: 2022-09-02T04:35:42.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.363 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.364 DEBUG:boto:Params: {} 2022-09-02T04:35:42.365 DEBUG:boto:Token: None 2022-09-02T04:35:42.365 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-amz-request-id', 'tx00000675d6004a921c111-006311881e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.380 DEBUG:rgw_multi.tests:zone=test-zone3 version=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:35:42.381 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.382 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.382 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.383 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.383 DEBUG:boto:Data: 2022-09-02T04:35:42.384 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.385 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.386 DEBUG:boto:Params: {} 2022-09-02T04:35:42.388 DEBUG:boto:Token: None 2022-09-02T04:35:42.388 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a82ca82976f89230-006311881e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.393 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.394 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.395 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.396 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.397 DEBUG:boto:Data: 2022-09-02T04:35:42.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.400 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.401 DEBUG:boto:Params: {} 2022-09-02T04:35:42.401 DEBUG:boto:Token: None 2022-09-02T04:35:42.402 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-amz-request-id', 'tx000002cecc51a254df8e4-006311881e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.417 DEBUG:rgw_multi.tests:zone=test-zone1 version=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:35:42.418 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.419 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.420 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.420 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.421 DEBUG:boto:Data: 2022-09-02T04:35:42.421 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.422 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.423 DEBUG:boto:Params: {} 2022-09-02T04:35:42.423 DEBUG:boto:Token: None 2022-09-02T04:35:42.424 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001887c8db9308bcc2-006311881e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.429 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.430 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.431 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.432 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.432 DEBUG:boto:Data: 2022-09-02T04:35:42.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.434 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.434 DEBUG:boto:Params: {} 2022-09-02T04:35:42.435 DEBUG:boto:Token: None 2022-09-02T04:35:42.436 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-amz-request-id', 'tx000003bf9fa43c153e479-006311881e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.450 DEBUG:rgw_multi.tests:zone=test-zone2 version=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:35:42.451 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.452 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.452 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.453 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.453 DEBUG:boto:Data: 2022-09-02T04:35:42.454 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.455 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.455 DEBUG:boto:Params: {} 2022-09-02T04:35:42.456 DEBUG:boto:Token: None 2022-09-02T04:35:42.457 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008e8ba0b2a79b2f7-006311881e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.462 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.463 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.466 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.467 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.467 DEBUG:boto:Data: 2022-09-02T04:35:42.468 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.471 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.472 DEBUG:boto:Params: {} 2022-09-02T04:35:42.474 DEBUG:boto:Token: None 2022-09-02T04:35:42.475 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-amz-request-id', 'tx00000bcaef0168f6ffa61-006311881e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.493 DEBUG:rgw_multi.tests:zone=test-zone3 version=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:35:42.494 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.494 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.495 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.496 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.496 DEBUG:boto:Data: 2022-09-02T04:35:42.497 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.498 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.499 DEBUG:boto:Params: {} 2022-09-02T04:35:42.499 DEBUG:boto:Token: None 2022-09-02T04:35:42.500 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000037b49ea9876a281-006311881e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.505 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.506 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.507 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.507 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.508 DEBUG:boto:Data: 2022-09-02T04:35:42.508 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.509 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.510 DEBUG:boto:Params: {} 2022-09-02T04:35:42.511 DEBUG:boto:Token: None 2022-09-02T04:35:42.512 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-amz-request-id', 'tx0000036c8841cf191c2df-006311881e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.534 DEBUG:rgw_multi.tests:zone=test-zone1 version=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:35:42.535 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.536 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.537 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.537 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.538 DEBUG:boto:Data: 2022-09-02T04:35:42.538 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.539 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.540 DEBUG:boto:Params: {} 2022-09-02T04:35:42.541 DEBUG:boto:Token: None 2022-09-02T04:35:42.541 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000388c45a2a02f476b-006311881e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.547 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.548 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.550 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.550 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.551 DEBUG:boto:Data: 2022-09-02T04:35:42.552 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.554 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.555 DEBUG:boto:Params: {} 2022-09-02T04:35:42.556 DEBUG:boto:Token: None 2022-09-02T04:35:42.557 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-amz-request-id', 'tx00000354723224d8b896b-006311881e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.571 DEBUG:rgw_multi.tests:zone=test-zone2 version=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:35:42.572 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.573 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.573 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.574 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.575 DEBUG:boto:Data: 2022-09-02T04:35:42.575 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.576 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.577 DEBUG:boto:Params: {} 2022-09-02T04:35:42.577 DEBUG:boto:Token: None 2022-09-02T04:35:42.578 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ee8173a92fafdf1-006311881e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.582 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.583 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.583 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.584 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.584 DEBUG:boto:Data: 2022-09-02T04:35:42.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.586 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.586 DEBUG:boto:Params: {} 2022-09-02T04:35:42.587 DEBUG:boto:Token: None 2022-09-02T04:35:42.588 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-amz-request-id', 'tx00000c42b4413585ac032-006311881e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.612 DEBUG:rgw_multi.tests:zone=test-zone3 version=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:35:42.612 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.613 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.613 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.614 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.615 DEBUG:boto:Data: 2022-09-02T04:35:42.615 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.616 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.617 DEBUG:boto:Params: {} 2022-09-02T04:35:42.617 DEBUG:boto:Token: None 2022-09-02T04:35:42.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f36cc0ad840c467a-006311881e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.623 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.623 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.624 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.625 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.626 DEBUG:boto:Data: 2022-09-02T04:35:42.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:35:42.628 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.628 DEBUG:boto:Params: {} 2022-09-02T04:35:42.629 DEBUG:boto:Token: None 2022-09-02T04:35:42.629 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.648 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-amz-request-id', 'tx000009899579a9429ca3a-006311881e-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.649 DEBUG:rgw_multi.tests:zone=test-zone1 version=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:35:42.650 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.651 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.652 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.653 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.653 DEBUG:boto:Data: 2022-09-02T04:35:42.654 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.656 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.657 DEBUG:boto:Params: {} 2022-09-02T04:35:42.658 DEBUG:boto:Token: None 2022-09-02T04:35:42.659 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.664 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebe347137d154a3b-006311881e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.665 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.666 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.666 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.667 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.667 DEBUG:boto:Data: 2022-09-02T04:35:42.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:35:42.669 DEBUG:boto:Port: 8000 2022-09-02T04:35:42.669 DEBUG:boto:Params: {} 2022-09-02T04:35:42.670 DEBUG:boto:Token: None 2022-09-02T04:35:42.671 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-amz-request-id', 'tx00000fa071013e348b461-006311881e-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.688 DEBUG:rgw_multi.tests:zone=test-zone2 version=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:35:42.689 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:35:42.689 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:35:42.690 DEBUG:boto:Method: HEAD 2022-09-02T04:35:42.690 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:35:42.691 DEBUG:boto:Data: 2022-09-02T04:35:42.691 DEBUG:boto:Headers: {} 2022-09-02T04:35:42.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.692 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.693 DEBUG:boto:Params: {} 2022-09-02T04:35:42.693 DEBUG:boto:Token: None 2022-09-02T04:35:42.694 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/ 2022-09-02T04:35:42.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M= 2022-09-02T04:35:42.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6yqEi2N1SMQYBIqwqDEz+OXca8M=', 'Content-Length': '0'} 2022-09-02T04:35:42.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000563d2324b6a976c0-006311881e-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.698 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:35:42.699 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:35:42.699 DEBUG:boto:Method: PUT 2022-09-02T04:35:42.700 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:35:42.700 DEBUG:boto:Data: 2022-09-02T04:35:42.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2022-09-02T04:35:42.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:35:42.702 DEBUG:boto:Port: 8001 2022-09-02T04:35:42.702 DEBUG:boto:Params: {} 2022-09-02T04:35:42.703 DEBUG:boto:Token: None 2022-09-02T04:35:42.703 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 02 Sep 2022 04:35:42 GMT /xljbkf-19/obj 2022-09-02T04:35:42.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10= 2022-09-02T04:35:42.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:35:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqH2Jt3heIUWChDoV1o6S0/ym10='} 2022-09-02T04:35:42.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-amz-request-id', 'tx00000e51de19e051aced2-006311881e-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:35:42.725 DEBUG:rgw_multi.tests:zone=test-zone3 version=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:35:42.725 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-19 2022-09-02T04:35:42.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:35:42.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:35:42.727 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:35:42.792 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:35:42.792 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:35:42.834 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:35:42.834+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:35:42.834 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , , , ] 2022-09-02T04:35:42.835 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:35:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:35:47.835+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:35:47.836 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , , , ] 2022-09-02T04:35:47.837 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:35:52.837 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:35:52.836+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:35:52.838 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , , , ] 2022-09-02T04:35:52.838 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:35:57.838 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:35:57.838+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:35:57.839 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , , , ] 2022-09-02T04:35:57.839 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:36:02.841 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:02.839+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:02.841 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000005.21.5, , ] 2022-09-02T04:36:02.842 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:36:07.841 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:07.840+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:07.841 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000005.21.5, , ] 2022-09-02T04:36:07.842 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:36:12.842 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:12.842+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:12.843 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000005.21.5, , ] 2022-09-02T04:36:12.843 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:36:17.843 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:17.843+0000 7f3c161a6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:17.844 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000005.21.5, , ] 2022-09-02T04:36:17.844 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:36:22.845 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:22.845+0000 7f3c161a6600 1 bucket sync caught up with source: 2022-09-02T04:36:22.845 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000024.62.5, , ] 2022-09-02T04:36:22.846 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000020.50.11, , ] 2022-09-02T04:36:22.846 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:22.845+0000 7f3c161a6600 0 bucket checkpoint complete 2022-09-02T04:36:22.869 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-19 2022-09-02T04:36:22.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:36:22.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:36:22.872 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:36:22.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:36:22.938 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:36:23.991 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:23.991+0000 7f82d99ec600 1 bucket sync caught up with source: 2022-09-02T04:36:23.992 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000042.105.5, , ] 2022-09-02T04:36:23.992 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000042.105.5, , ] 2022-09-02T04:36:23.992 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:23.991+0000 7f82d99ec600 0 bucket checkpoint complete 2022-09-02T04:36:24.014 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-19 2022-09-02T04:36:24.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:36:24.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:36:24.016 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:36:24.080 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:36:24.081 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:36:25.123 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:25.123+0000 7f89ba83c600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:25.124 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000033.87.5, , ] 2022-09-02T04:36:25.125 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000042.105.5, , ] 2022-09-02T04:36:30.125 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:30.125+0000 7f89ba83c600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:30.126 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000033.87.5, , ] 2022-09-02T04:36:30.126 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000042.105.5, , ] 2022-09-02T04:36:35.126 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:35.127+0000 7f89ba83c600 1 bucket sync caught up with source: 2022-09-02T04:36:35.127 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000005.38.9, , , 00000000042.105.5, , ] 2022-09-02T04:36:35.128 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000005.38.9, , , 00000000042.105.5, , ] 2022-09-02T04:36:35.128 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:36:35.127+0000 7f89ba83c600 0 bucket checkpoint complete 2022-09-02T04:36:35.152 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-19 2022-09-02T04:36:35.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:36:35.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:36:35.155 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:36:35.220 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:36:35.220 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:36:35.273 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:35.273+0000 7fee05fc6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:35.274 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000019.78.5, , ] 2022-09-02T04:36:35.274 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000042.141.8, , ] 2022-09-02T04:36:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:40.275+0000 7fee05fc6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:40.275 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000019.78.5, , ] 2022-09-02T04:36:40.275 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000042.141.8, , ] 2022-09-02T04:36:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:45.276+0000 7fee05fc6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:45.277 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000019.78.5, , ] 2022-09-02T04:36:45.277 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000042.141.8, , ] 2022-09-02T04:36:50.278 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:50.278+0000 7fee05fc6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:50.278 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000019.78.5, , ] 2022-09-02T04:36:50.278 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000042.141.8, , ] 2022-09-02T04:36:55.279 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:36:55.279+0000 7fee05fc6600 1 waiting for incremental sync to catch up: 2022-09-02T04:36:55.280 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000019.78.5, , ] 2022-09-02T04:36:55.280 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000042.141.8, , ] 2022-09-02T04:37:00.281 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:00.280+0000 7fee05fc6600 1 bucket sync caught up with source: 2022-09-02T04:37:00.282 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000043.143.9, , ] 2022-09-02T04:37:00.282 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000042.141.8, , ] 2022-09-02T04:37:00.283 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:00.280+0000 7fee05fc6600 0 bucket checkpoint complete 2022-09-02T04:37:00.301 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-19 2022-09-02T04:37:00.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:37:00.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:37:00.304 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:37:00.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:37:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:37:00.406 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:00.406+0000 7f0b9a7aa600 1 bucket sync caught up with source: 2022-09-02T04:37:00.407 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.59.9, , , 00000000043.143.9, , ] 2022-09-02T04:37:00.407 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.59.9, , , 00000000043.143.9, , ] 2022-09-02T04:37:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:00.406+0000 7f0b9a7aa600 0 bucket checkpoint complete 2022-09-02T04:37:00.421 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-19 2022-09-02T04:37:00.422 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-19 2022-09-02T04:37:00.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:37:00.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:37:00.425 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:37:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:37:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:37:00.530 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:00.530+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:00.531 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:00.531 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:05.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:05.532+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:05.533 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:05.533 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:10.533 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:10.533+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:10.534 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:10.534 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:15.535 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:15.535+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:15.535 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:15.536 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:20.536 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:20.536+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:20.537 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:20.537 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:25.537 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:25.537+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:25.538 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:25.538 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:30.538+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:30.539 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:30.539 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:35.539 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:35.539+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:35.540 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:35.540 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:40.541 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:40.541+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:40.541 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:40.542 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:45.542 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:45.542+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:45.543 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:45.543 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:50.543+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:50.544 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:50.544 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:37:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:37:55.545+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:37:55.545 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:37:55.546 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:00.546 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:38:00.546+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:00.547 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:38:00.547 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:38:05.548+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:05.548 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:38:05.549 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:10.549 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:38:10.549+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:10.550 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:38:10.551 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:15.550 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:38:15.551+0000 7f70049a5600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:15.551 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000020.54.11, , ] 2022-09-02T04:38:15.551 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:38:20.552+0000 7f70049a5600 1 bucket sync caught up with source: 2022-09-02T04:38:20.552 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:20.553 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:38:20.552+0000 7f70049a5600 0 bucket checkpoint complete 2022-09-02T04:38:20.571 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-19 2022-09-02T04:38:20.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:38:20.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:38:20.574 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:38:20.639 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:38:20.640 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:38:20.692 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:20.691+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:20.693 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:20.693 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:25.693 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:25.693+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:25.693 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:25.694 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:30.695 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:30.694+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:30.695 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:30.696 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:35.696 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:35.696+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:35.696 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:35.697 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:40.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:40.697+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:40.698 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:40.698 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:45.699 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:45.700+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:45.700 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:45.700 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:50.703 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:50.701+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:50.704 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:50.704 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:38:55.702 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:38:55.702+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:38:55.703 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:38:55.703 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:00.703 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:00.703+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:00.704 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:00.704 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:05.705 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:05.705+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:05.705 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:05.706 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:10.706 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:10.706+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:10.707 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:10.707 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:15.709 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:15.707+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:15.709 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:15.710 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:20.709 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:20.709+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:20.709 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:20.709 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:25.710 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:25.710+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:25.711 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:25.711 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:30.712 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:30.712+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:30.712 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:30.712 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:35.713 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:35.713+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:35.714 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:35.714 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:40.715 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:40.714+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:40.715 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:40.716 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:45.716 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:45.716+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:45.716 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:45.717 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:50.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:50.718+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:50.723 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:50.724 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:39:55.727 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:39:55.719+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:39:55.727 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:39:55.728 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:00.720 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:00.720+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:00.721 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:00.721 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:05.722 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:05.721+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:05.723 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:05.723 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:10.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:10.723+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:10.723 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:10.724 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:15.724 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:15.724+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:15.725 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:15.725 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:20.726 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:20.725+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:20.726 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:20.727 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:25.727 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:25.727+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:25.728 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:25.728 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:30.728 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:30.728+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:30.729 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:30.730 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:35.730 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:35.730+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:35.730 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:35.731 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:40.731 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:40.731+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:40.732 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:40.733 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:45.733 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:45.733+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:45.733 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:45.733 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:50.734 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:50.734+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:50.735 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:50.736 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:40:55.736 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:40:55.736+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:40:55.736 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:40:55.737 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:00.738 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:00.738+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:00.739 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:00.739 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:05.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:05.738+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:05.747 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:05.747 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:10.740 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:10.740+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:10.741 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:10.741 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:15.742 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:15.742+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:15.742 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:15.742 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:20.743 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:20.743+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:20.744 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:20.744 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:25.745 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:25.745+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:25.745 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:25.746 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:30.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:30.746+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:30.747 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:30.747 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:35.749 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:35.748+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:35.749 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:35.750 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:40.750 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:40.749+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:40.751 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:40.751 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:45.751 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:45.751+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:45.751 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:45.752 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:50.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:50.752+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:50.753 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:50.754 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:41:55.757 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:41:55.755+0000 7f05dec3b600 1 waiting for incremental sync to catch up: 2022-09-02T04:41:55.760 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000034.92.5, , ] 2022-09-02T04:41:55.760 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:42:00.755 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:00.756+0000 7f05dec3b600 1 bucket sync caught up with source: 2022-09-02T04:42:00.756 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:42:00.756 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000045.122.9, , ] 2022-09-02T04:42:00.756 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:00.756+0000 7f05dec3b600 0 bucket checkpoint complete 2022-09-02T04:42:00.773 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-19 2022-09-02T04:42:00.775 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-19 zones={test-zone2, test-zone1} 2022-09-02T04:42:00.777 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:00.778 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:00.779 DEBUG:boto:Method: HEAD 2022-09-02T04:42:00.780 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:42:00.781 DEBUG:boto:Data: 2022-09-02T04:42:00.782 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:00.783 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.784 DEBUG:boto:Params: {} 2022-09-02T04:42:00.785 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:42:00.786 DEBUG:boto:Token: None 2022-09-02T04:42:00.787 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/ 2022-09-02T04:42:00.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mUYCwYUz4rPlOoCKrIaQSjZTtQg= 2022-09-02T04:42:00.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mUYCwYUz4rPlOoCKrIaQSjZTtQg=', 'Content-Length': '0'} 2022-09-02T04:42:00.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e49a493f081825d5-0063118998-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.795 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:00.796 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:00.796 DEBUG:boto:Method: HEAD 2022-09-02T04:42:00.797 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:42:00.797 DEBUG:boto:Data: 2022-09-02T04:42:00.798 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:00.799 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.799 DEBUG:boto:Params: {} 2022-09-02T04:42:00.800 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:42:00.800 DEBUG:boto:Token: None 2022-09-02T04:42:00.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/ 2022-09-02T04:42:00.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mUYCwYUz4rPlOoCKrIaQSjZTtQg= 2022-09-02T04:42:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mUYCwYUz4rPlOoCKrIaQSjZTtQg=', 'Content-Length': '0'} 2022-09-02T04:42:00.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068d068bef1d9b7f7-0063118998-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.808 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:42:00.809 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:00.809 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:00.810 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.810 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.811 DEBUG:boto:Method: GET 2022-09-02T04:42:00.811 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.812 DEBUG:boto:Data: 2022-09-02T04:42:00.812 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.813 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:00.813 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.814 DEBUG:boto:Params: {} 2022-09-02T04:42:00.815 DEBUG:boto:Token: None 2022-09-02T04:42:00.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/?versions 2022-09-02T04:42:00.816 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q= 2022-09-02T04:42:00.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q=', 'Content-Length': '0'} 2022-09-02T04:42:00.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d37bebd98b85e2c9-0063118998-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.821 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:00.825 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:42:00.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.841 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:42:00.842 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:00.843 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:00.843 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.844 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.844 DEBUG:boto:Method: GET 2022-09-02T04:42:00.845 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.845 DEBUG:boto:Data: 2022-09-02T04:42:00.846 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:00.847 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.847 DEBUG:boto:Params: {} 2022-09-02T04:42:00.848 DEBUG:boto:Token: None 2022-09-02T04:42:00.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/?versions 2022-09-02T04:42:00.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q= 2022-09-02T04:42:00.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q=', 'Content-Length': '0'} 2022-09-02T04:42:00.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ff48af6bffe609-0063118998-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.854 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:00.856 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:42:00.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:00.872 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:00.872 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:00.873 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.874 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.874 DEBUG:boto:Method: GET 2022-09-02T04:42:00.875 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.875 DEBUG:boto:Data: 2022-09-02T04:42:00.876 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:00.877 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.878 DEBUG:boto:Params: {} 2022-09-02T04:42:00.878 DEBUG:boto:Token: None 2022-09-02T04:42:00.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/?versions 2022-09-02T04:42:00.879 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q= 2022-09-02T04:42:00.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q=', 'Content-Length': '0'} 2022-09-02T04:42:00.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c51d74490d8991-0063118998-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.884 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:00.886 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:00.887 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:00.887 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.888 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.888 DEBUG:boto:Method: GET 2022-09-02T04:42:00.889 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:00.889 DEBUG:boto:Data: 2022-09-02T04:42:00.890 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:00.891 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.892 DEBUG:boto:Params: {} 2022-09-02T04:42:00.892 DEBUG:boto:Token: None 2022-09-02T04:42:00.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/?versions 2022-09-02T04:42:00.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q= 2022-09-02T04:42:00.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ppAiTNWWw8Am05VFlT5/w21I+6Q=', 'Content-Length': '0'} 2022-09-02T04:42:00.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375e43babde90125-0063118998-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.898 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:00.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:00.901 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:00.901 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:00.902 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.902 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.903 DEBUG:boto:Method: GET 2022-09-02T04:42:00.904 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.904 DEBUG:boto:Data: 2022-09-02T04:42:00.905 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:00.906 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.906 DEBUG:boto:Params: {} 2022-09-02T04:42:00.907 DEBUG:boto:Token: None 2022-09-02T04:42:00.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhRhpjFHqrZ+6cvjuuClM3oQ3A4= 2022-09-02T04:42:00.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhRhpjFHqrZ+6cvjuuClM3oQ3A4=', 'Content-Length': '0'} 2022-09-02T04:42:00.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dcf66d6cf93ddba5-0063118998-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.913 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:00.914 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:00.915 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.916 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.917 DEBUG:boto:Method: GET 2022-09-02T04:42:00.918 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.919 DEBUG:boto:Data: 2022-09-02T04:42:00.920 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:00.923 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.924 DEBUG:boto:Params: {} 2022-09-02T04:42:00.926 DEBUG:boto:Token: None 2022-09-02T04:42:00.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhRhpjFHqrZ+6cvjuuClM3oQ3A4= 2022-09-02T04:42:00.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhRhpjFHqrZ+6cvjuuClM3oQ3A4=', 'Content-Length': '0'} 2022-09-02T04:42:00.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c893edbbad88794-0063118998-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.936 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:00.937 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:00.938 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.939 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.939 DEBUG:boto:Method: HEAD 2022-09-02T04:42:00.940 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.941 DEBUG:boto:Data: 2022-09-02T04:42:00.941 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.942 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:00.943 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.944 DEBUG:boto:Params: {} 2022-09-02T04:42:00.945 DEBUG:boto:Token: None 2022-09-02T04:42:00.946 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VYc5uMFxZvAFxEfGNnPgBepw9DU= 2022-09-02T04:42:00.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VYc5uMFxZvAFxEfGNnPgBepw9DU=', 'Content-Length': '0'} 2022-09-02T04:42:00.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000522b558d6d4f7e48-0063118998-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.959 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:00.960 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:00.961 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.962 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.963 DEBUG:boto:Method: HEAD 2022-09-02T04:42:00.964 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.965 DEBUG:boto:Data: 2022-09-02T04:42:00.966 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:00.968 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.969 DEBUG:boto:Params: {} 2022-09-02T04:42:00.970 DEBUG:boto:Token: None 2022-09-02T04:42:00.971 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VYc5uMFxZvAFxEfGNnPgBepw9DU= 2022-09-02T04:42:00.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VYc5uMFxZvAFxEfGNnPgBepw9DU=', 'Content-Length': '0'} 2022-09-02T04:42:00.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000565f5a59c2cd8b35-0063118998-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:00.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:00.978 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:00.979 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:00.980 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.981 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.982 DEBUG:boto:Method: GET 2022-09-02T04:42:00.983 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.984 DEBUG:boto:Data: 2022-09-02T04:42:00.985 DEBUG:boto:Headers: {} 2022-09-02T04:42:00.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:00.987 DEBUG:boto:Port: 8000 2022-09-02T04:42:00.988 DEBUG:boto:Params: {} 2022-09-02T04:42:00.989 DEBUG:boto:Token: None 2022-09-02T04:42:00.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:00 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:00.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhRhpjFHqrZ+6cvjuuClM3oQ3A4= 2022-09-02T04:42:00.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhRhpjFHqrZ+6cvjuuClM3oQ3A4=', 'Content-Length': '0'} 2022-09-02T04:42:01.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026b6b0812d21b34b-0063118998-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.005 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:01.006 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:01.007 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.007 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.008 DEBUG:boto:Method: GET 2022-09-02T04:42:01.009 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.010 DEBUG:boto:Data: 2022-09-02T04:42:01.011 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.013 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.014 DEBUG:boto:Params: {} 2022-09-02T04:42:01.015 DEBUG:boto:Token: None 2022-09-02T04:42:01.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X7L1TyW1KlbdU+TJ3DvsbiKcvSs= 2022-09-02T04:42:01.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X7L1TyW1KlbdU+TJ3DvsbiKcvSs=', 'Content-Length': '0'} 2022-09-02T04:42:01.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039866317658e6998-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.023 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.023 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.024 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.024 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.025 DEBUG:boto:Method: GET 2022-09-02T04:42:01.025 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.026 DEBUG:boto:Data: 2022-09-02T04:42:01.026 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.027 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.028 DEBUG:boto:Params: {} 2022-09-02T04:42:01.029 DEBUG:boto:Token: None 2022-09-02T04:42:01.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006096f9aead6a2342-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.074 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.075 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.076 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.077 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.077 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.078 DEBUG:boto:Method: GET 2022-09-02T04:42:01.078 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.079 DEBUG:boto:Data: 2022-09-02T04:42:01.080 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.081 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.081 DEBUG:boto:Params: {} 2022-09-02T04:42:01.082 DEBUG:boto:Token: None 2022-09-02T04:42:01.082 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.083 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d2272c7ecb2db87-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.092 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.093 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:01.094 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:01.095 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.096 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:42:01.097 DEBUG:boto:Data: 2022-09-02T04:42:01.097 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.100 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.101 DEBUG:boto:Params: {} 2022-09-02T04:42:01.102 DEBUG:boto:Token: None 2022-09-02T04:42:01.103 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/dummy 2022-09-02T04:42:01.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJL8d7/RGJCQ+q/pUVuOjrWaiSo= 2022-09-02T04:42:01.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJL8d7/RGJCQ+q/pUVuOjrWaiSo=', 'Content-Length': '0'} 2022-09-02T04:42:01.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4549a2d6b5da668-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.109 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:01.109 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:01.110 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.111 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:42:01.112 DEBUG:boto:Data: 2022-09-02T04:42:01.113 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.115 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.116 DEBUG:boto:Params: {} 2022-09-02T04:42:01.117 DEBUG:boto:Token: None 2022-09-02T04:42:01.117 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/dummy 2022-09-02T04:42:01.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJL8d7/RGJCQ+q/pUVuOjrWaiSo= 2022-09-02T04:42:01.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJL8d7/RGJCQ+q/pUVuOjrWaiSo=', 'Content-Length': '0'} 2022-09-02T04:42:01.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2fa4a1630466750-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:01.124 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:01.125 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:01.125 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.126 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.127 DEBUG:boto:Method: GET 2022-09-02T04:42:01.127 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.128 DEBUG:boto:Data: 2022-09-02T04:42:01.128 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.129 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.130 DEBUG:boto:Params: {} 2022-09-02T04:42:01.130 DEBUG:boto:Token: None 2022-09-02T04:42:01.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X7L1TyW1KlbdU+TJ3DvsbiKcvSs= 2022-09-02T04:42:01.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X7L1TyW1KlbdU+TJ3DvsbiKcvSs=', 'Content-Length': '0'} 2022-09-02T04:42:01.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb06d3f26e721ecf-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.135 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:01.136 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:01.136 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.137 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.137 DEBUG:boto:Method: GET 2022-09-02T04:42:01.138 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.139 DEBUG:boto:Data: 2022-09-02T04:42:01.139 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.140 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.141 DEBUG:boto:Params: {} 2022-09-02T04:42:01.141 DEBUG:boto:Token: None 2022-09-02T04:42:01.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:01.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X7L1TyW1KlbdU+TJ3DvsbiKcvSs= 2022-09-02T04:42:01.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X7L1TyW1KlbdU+TJ3DvsbiKcvSs=', 'Content-Length': '0'} 2022-09-02T04:42:01.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021e46d1bafdd4282-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.146 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.147 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.147 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.148 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.148 DEBUG:boto:Method: GET 2022-09-02T04:42:01.149 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.149 DEBUG:boto:Data: 2022-09-02T04:42:01.150 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.151 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.151 DEBUG:boto:Params: {} 2022-09-02T04:42:01.152 DEBUG:boto:Token: None 2022-09-02T04:42:01.153 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffa1150952584ce-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.157 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.158 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.158 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.159 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.159 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.160 DEBUG:boto:Method: GET 2022-09-02T04:42:01.161 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:01.161 DEBUG:boto:Data: 2022-09-02T04:42:01.162 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.163 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.163 DEBUG:boto:Params: {} 2022-09-02T04:42:01.164 DEBUG:boto:Token: None 2022-09-02T04:42:01.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb89f2e84d52de7-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.170 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.172 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.173 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.174 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.175 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.176 DEBUG:boto:Method: GET 2022-09-02T04:42:01.177 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.178 DEBUG:boto:Data: 2022-09-02T04:42:01.179 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.180 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.182 DEBUG:boto:Params: {} 2022-09-02T04:42:01.182 DEBUG:boto:Token: None 2022-09-02T04:42:01.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c80f5f56323dee8e-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.190 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.191 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.192 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.193 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.193 DEBUG:boto:Method: GET 2022-09-02T04:42:01.194 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.195 DEBUG:boto:Data: 2022-09-02T04:42:01.195 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.197 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.197 DEBUG:boto:Params: {} 2022-09-02T04:42:01.198 DEBUG:boto:Token: None 2022-09-02T04:42:01.199 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000698c74c06259f5a7-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.204 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.204 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.205 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.206 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.207 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.208 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.209 DEBUG:boto:Data: 2022-09-02T04:42:01.210 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.210 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.211 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.213 DEBUG:boto:Params: {} 2022-09-02T04:42:01.214 DEBUG:boto:Token: None 2022-09-02T04:42:01.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52L3+bXCAG/8N599OSb8DT5sciQ= 2022-09-02T04:42:01.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52L3+bXCAG/8N599OSb8DT5sciQ=', 'Content-Length': '0'} 2022-09-02T04:42:01.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d787b0eabceac67-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.219 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.219 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.220 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.220 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.221 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.221 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.222 DEBUG:boto:Data: 2022-09-02T04:42:01.222 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.223 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.224 DEBUG:boto:Params: {} 2022-09-02T04:42:01.225 DEBUG:boto:Token: None 2022-09-02T04:42:01.225 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52L3+bXCAG/8N599OSb8DT5sciQ= 2022-09-02T04:42:01.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52L3+bXCAG/8N599OSb8DT5sciQ=', 'Content-Length': '0'} 2022-09-02T04:42:01.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bf9b5c15e30ecc7-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.230 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.231 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.231 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.232 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.233 DEBUG:boto:Method: GET 2022-09-02T04:42:01.233 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.234 DEBUG:boto:Data: 2022-09-02T04:42:01.234 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.235 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.236 DEBUG:boto:Params: {} 2022-09-02T04:42:01.236 DEBUG:boto:Token: None 2022-09-02T04:42:01.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a85c3e3bde852994-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.286 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.287 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.287 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.288 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.288 DEBUG:boto:Method: GET 2022-09-02T04:42:01.289 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.289 DEBUG:boto:Data: 2022-09-02T04:42:01.290 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.291 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.292 DEBUG:boto:Params: {} 2022-09-02T04:42:01.293 DEBUG:boto:Token: None 2022-09-02T04:42:01.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf9dffa45d807880-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.298 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.298 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.299 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.299 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.300 DEBUG:boto:Method: GET 2022-09-02T04:42:01.301 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.301 DEBUG:boto:Data: 2022-09-02T04:42:01.302 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.303 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.304 DEBUG:boto:Params: {} 2022-09-02T04:42:01.305 DEBUG:boto:Token: None 2022-09-02T04:42:01.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e79c66f3fbedc0d-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.311 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.311 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.312 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.313 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.313 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.314 DEBUG:boto:Method: GET 2022-09-02T04:42:01.314 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.315 DEBUG:boto:Data: 2022-09-02T04:42:01.315 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.317 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.317 DEBUG:boto:Params: {} 2022-09-02T04:42:01.318 DEBUG:boto:Token: None 2022-09-02T04:42:01.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac9469d0d50f47a0-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.323 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.324 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.325 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.325 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.326 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:01.326 DEBUG:boto:Data: 2022-09-02T04:42:01.327 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.328 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.328 DEBUG:boto:Params: {} 2022-09-02T04:42:01.329 DEBUG:boto:Token: None 2022-09-02T04:42:01.329 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj 2022-09-02T04:42:01.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo= 2022-09-02T04:42:01.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo=', 'Content-Length': '0'} 2022-09-02T04:42:01.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000216093c39188668b-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.335 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.336 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.336 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.337 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:01.338 DEBUG:boto:Data: 2022-09-02T04:42:01.338 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.339 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.340 DEBUG:boto:Params: {} 2022-09-02T04:42:01.340 DEBUG:boto:Token: None 2022-09-02T04:42:01.341 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj 2022-09-02T04:42:01.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo= 2022-09-02T04:42:01.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo=', 'Content-Length': '0'} 2022-09-02T04:42:01.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c899e68e3d7aad4c-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.347 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.347 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.348 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.348 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.349 DEBUG:boto:Method: GET 2022-09-02T04:42:01.350 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.350 DEBUG:boto:Data: 2022-09-02T04:42:01.351 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.352 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.352 DEBUG:boto:Params: {} 2022-09-02T04:42:01.353 DEBUG:boto:Token: None 2022-09-02T04:42:01.354 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c68303c76cd18a47-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.398 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.399 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.400 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.400 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.401 DEBUG:boto:Method: GET 2022-09-02T04:42:01.401 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.402 DEBUG:boto:Data: 2022-09-02T04:42:01.403 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.404 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.404 DEBUG:boto:Params: {} 2022-09-02T04:42:01.405 DEBUG:boto:Token: None 2022-09-02T04:42:01.406 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcec59754e1c6f90-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.410 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.411 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.412 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.412 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.413 DEBUG:boto:Method: GET 2022-09-02T04:42:01.413 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.414 DEBUG:boto:Data: 2022-09-02T04:42:01.415 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.416 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.416 DEBUG:boto:Params: {} 2022-09-02T04:42:01.417 DEBUG:boto:Token: None 2022-09-02T04:42:01.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000156b61fa242d97e2-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.427 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.428 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.428 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.429 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.429 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.430 DEBUG:boto:Method: GET 2022-09-02T04:42:01.431 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.431 DEBUG:boto:Data: 2022-09-02T04:42:01.432 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.433 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.433 DEBUG:boto:Params: {} 2022-09-02T04:42:01.434 DEBUG:boto:Token: None 2022-09-02T04:42:01.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097000ccc2c92bf0b-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.439 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.441 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.441 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.442 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.442 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.443 DEBUG:boto:Method: GET 2022-09-02T04:42:01.443 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.444 DEBUG:boto:Data: 2022-09-02T04:42:01.444 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.445 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.446 DEBUG:boto:Params: {} 2022-09-02T04:42:01.447 DEBUG:boto:Token: None 2022-09-02T04:42:01.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4= 2022-09-02T04:42:01.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4=', 'Content-Length': '0'} 2022-09-02T04:42:01.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba470790b16a2f2d-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.452 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.452 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.453 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.453 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.454 DEBUG:boto:Method: GET 2022-09-02T04:42:01.454 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.455 DEBUG:boto:Data: 2022-09-02T04:42:01.455 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.457 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.457 DEBUG:boto:Params: {} 2022-09-02T04:42:01.458 DEBUG:boto:Token: None 2022-09-02T04:42:01.458 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4= 2022-09-02T04:42:01.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4=', 'Content-Length': '0'} 2022-09-02T04:42:01.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050add55a466e5503-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.463 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.463 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.464 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.464 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.465 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.465 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.466 DEBUG:boto:Data: 2022-09-02T04:42:01.466 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.467 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.468 DEBUG:boto:Params: {} 2022-09-02T04:42:01.469 DEBUG:boto:Token: None 2022-09-02T04:42:01.469 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dn3rF3Z9j9ygmUkvjWiPOUbiPXY= 2022-09-02T04:42:01.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dn3rF3Z9j9ygmUkvjWiPOUbiPXY=', 'Content-Length': '0'} 2022-09-02T04:42:01.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bc58e98df1443d5-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.473 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.474 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.474 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.475 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.475 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.476 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.476 DEBUG:boto:Data: 2022-09-02T04:42:01.477 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.478 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.478 DEBUG:boto:Params: {} 2022-09-02T04:42:01.479 DEBUG:boto:Token: None 2022-09-02T04:42:01.479 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dn3rF3Z9j9ygmUkvjWiPOUbiPXY= 2022-09-02T04:42:01.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dn3rF3Z9j9ygmUkvjWiPOUbiPXY=', 'Content-Length': '0'} 2022-09-02T04:42:01.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fa90c7ac84140bd-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.485 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.485 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.486 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.487 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.487 DEBUG:boto:Method: GET 2022-09-02T04:42:01.488 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.488 DEBUG:boto:Data: 2022-09-02T04:42:01.489 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.490 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.491 DEBUG:boto:Params: {} 2022-09-02T04:42:01.491 DEBUG:boto:Token: None 2022-09-02T04:42:01.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4= 2022-09-02T04:42:01.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4=', 'Content-Length': '0'} 2022-09-02T04:42:01.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000052758ef9fe28aaa-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.538 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.539 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.540 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.541 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.542 DEBUG:boto:Method: GET 2022-09-02T04:42:01.543 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.544 DEBUG:boto:Data: 2022-09-02T04:42:01.545 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.547 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.548 DEBUG:boto:Params: {} 2022-09-02T04:42:01.549 DEBUG:boto:Token: None 2022-09-02T04:42:01.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:01.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4= 2022-09-02T04:42:01.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J9H83jAScrozPvqf2AIHLz1Nfu4=', 'Content-Length': '0'} 2022-09-02T04:42:01.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006aeebd9814094bdd-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.557 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.558 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.559 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.560 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.561 DEBUG:boto:Method: GET 2022-09-02T04:42:01.562 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.562 DEBUG:boto:Data: 2022-09-02T04:42:01.563 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.565 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.566 DEBUG:boto:Params: {} 2022-09-02T04:42:01.567 DEBUG:boto:Token: None 2022-09-02T04:42:01.568 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5f660a4ece5218-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.577 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.578 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.578 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.579 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.579 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.580 DEBUG:boto:Method: GET 2022-09-02T04:42:01.580 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.581 DEBUG:boto:Data: 2022-09-02T04:42:01.582 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.583 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.583 DEBUG:boto:Params: {} 2022-09-02T04:42:01.584 DEBUG:boto:Token: None 2022-09-02T04:42:01.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ca70868c19b67b-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.591 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.592 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.592 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.593 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.594 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:01.594 DEBUG:boto:Data: 2022-09-02T04:42:01.595 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.597 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.597 DEBUG:boto:Params: {} 2022-09-02T04:42:01.598 DEBUG:boto:Token: None 2022-09-02T04:42:01.599 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj 2022-09-02T04:42:01.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo= 2022-09-02T04:42:01.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo=', 'Content-Length': '0'} 2022-09-02T04:42:01.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b69205a6ff1cdfbb-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.606 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.607 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.608 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.608 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:01.609 DEBUG:boto:Data: 2022-09-02T04:42:01.609 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.610 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.611 DEBUG:boto:Params: {} 2022-09-02T04:42:01.611 DEBUG:boto:Token: None 2022-09-02T04:42:01.612 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj 2022-09-02T04:42:01.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo= 2022-09-02T04:42:01.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo=', 'Content-Length': '0'} 2022-09-02T04:42:01.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000860a11f7e61d2d12-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.621 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.622 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.623 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.624 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.625 DEBUG:boto:Method: GET 2022-09-02T04:42:01.626 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.626 DEBUG:boto:Data: 2022-09-02T04:42:01.627 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.629 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.630 DEBUG:boto:Params: {} 2022-09-02T04:42:01.631 DEBUG:boto:Token: None 2022-09-02T04:42:01.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069eadf9c4e532bfa-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.682 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.683 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.683 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.684 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.685 DEBUG:boto:Method: GET 2022-09-02T04:42:01.685 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.686 DEBUG:boto:Data: 2022-09-02T04:42:01.686 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.687 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.688 DEBUG:boto:Params: {} 2022-09-02T04:42:01.688 DEBUG:boto:Token: None 2022-09-02T04:42:01.689 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7bf3040d2f8b3fd-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.693 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.694 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.694 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.695 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.695 DEBUG:boto:Method: GET 2022-09-02T04:42:01.696 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.696 DEBUG:boto:Data: 2022-09-02T04:42:01.697 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.698 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.698 DEBUG:boto:Params: {} 2022-09-02T04:42:01.699 DEBUG:boto:Token: None 2022-09-02T04:42:01.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0108b237bfe769c-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.705 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.705 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.706 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.706 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.707 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.708 DEBUG:boto:Method: GET 2022-09-02T04:42:01.708 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.709 DEBUG:boto:Data: 2022-09-02T04:42:01.709 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.710 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.711 DEBUG:boto:Params: {} 2022-09-02T04:42:01.711 DEBUG:boto:Token: None 2022-09-02T04:42:01.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000295240b208e27722-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.717 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.718 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.719 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.719 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.720 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.720 DEBUG:boto:Method: GET 2022-09-02T04:42:01.721 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.721 DEBUG:boto:Data: 2022-09-02T04:42:01.722 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.723 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.723 DEBUG:boto:Params: {} 2022-09-02T04:42:01.724 DEBUG:boto:Token: None 2022-09-02T04:42:01.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY= 2022-09-02T04:42:01.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY=', 'Content-Length': '0'} 2022-09-02T04:42:01.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000055c92166b840527-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.729 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.730 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.730 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.731 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.732 DEBUG:boto:Method: GET 2022-09-02T04:42:01.732 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.733 DEBUG:boto:Data: 2022-09-02T04:42:01.734 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.735 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.735 DEBUG:boto:Params: {} 2022-09-02T04:42:01.736 DEBUG:boto:Token: None 2022-09-02T04:42:01.737 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY= 2022-09-02T04:42:01.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY=', 'Content-Length': '0'} 2022-09-02T04:42:01.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd366605c7f20321-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.741 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.742 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.742 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.743 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.743 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.744 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.744 DEBUG:boto:Data: 2022-09-02T04:42:01.745 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.746 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.746 DEBUG:boto:Params: {} 2022-09-02T04:42:01.747 DEBUG:boto:Token: None 2022-09-02T04:42:01.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tn9c39bXWbKPoht3TgSTKx6WeRA= 2022-09-02T04:42:01.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tn9c39bXWbKPoht3TgSTKx6WeRA=', 'Content-Length': '0'} 2022-09-02T04:42:01.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c463081cf1ca8950-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.752 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.753 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.753 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.754 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.754 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.755 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.756 DEBUG:boto:Data: 2022-09-02T04:42:01.756 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.758 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.758 DEBUG:boto:Params: {} 2022-09-02T04:42:01.759 DEBUG:boto:Token: None 2022-09-02T04:42:01.760 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tn9c39bXWbKPoht3TgSTKx6WeRA= 2022-09-02T04:42:01.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tn9c39bXWbKPoht3TgSTKx6WeRA=', 'Content-Length': '0'} 2022-09-02T04:42:01.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065276c22e9bc9978-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.765 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.765 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.766 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.766 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.767 DEBUG:boto:Method: GET 2022-09-02T04:42:01.768 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.768 DEBUG:boto:Data: 2022-09-02T04:42:01.769 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.769 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.770 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.770 DEBUG:boto:Params: {} 2022-09-02T04:42:01.771 DEBUG:boto:Token: None 2022-09-02T04:42:01.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY= 2022-09-02T04:42:01.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY=', 'Content-Length': '0'} 2022-09-02T04:42:01.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad7bd639846e1842-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.776 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.777 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.777 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.778 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.778 DEBUG:boto:Method: GET 2022-09-02T04:42:01.779 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.780 DEBUG:boto:Data: 2022-09-02T04:42:01.780 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.781 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.782 DEBUG:boto:Params: {} 2022-09-02T04:42:01.782 DEBUG:boto:Token: None 2022-09-02T04:42:01.783 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:01.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY= 2022-09-02T04:42:01.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pYSy+KJH5PZUPpumrvO9ANEV7JY=', 'Content-Length': '0'} 2022-09-02T04:42:01.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a793048cae9558-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.787 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.788 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.789 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.789 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.790 DEBUG:boto:Method: GET 2022-09-02T04:42:01.790 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.791 DEBUG:boto:Data: 2022-09-02T04:42:01.791 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.792 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.793 DEBUG:boto:Params: {} 2022-09-02T04:42:01.794 DEBUG:boto:Token: None 2022-09-02T04:42:01.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001519bf6c46663228-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.799 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.800 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.800 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.801 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.802 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.802 DEBUG:boto:Method: GET 2022-09-02T04:42:01.803 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.803 DEBUG:boto:Data: 2022-09-02T04:42:01.804 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.805 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.805 DEBUG:boto:Params: {} 2022-09-02T04:42:01.806 DEBUG:boto:Token: None 2022-09-02T04:42:01.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000291082c7d1b29195-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.854 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.855 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.856 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.856 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.857 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:01.857 DEBUG:boto:Data: 2022-09-02T04:42:01.858 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.859 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.859 DEBUG:boto:Params: {} 2022-09-02T04:42:01.860 DEBUG:boto:Token: None 2022-09-02T04:42:01.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj 2022-09-02T04:42:01.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo= 2022-09-02T04:42:01.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo=', 'Content-Length': '0'} 2022-09-02T04:42:01.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e21f95bd86df237-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.868 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.869 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.870 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.871 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:01.871 DEBUG:boto:Data: 2022-09-02T04:42:01.872 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.873 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.874 DEBUG:boto:Params: {} 2022-09-02T04:42:01.875 DEBUG:boto:Token: None 2022-09-02T04:42:01.876 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj 2022-09-02T04:42:01.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo= 2022-09-02T04:42:01.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gt5Ix9ET1Ym4mk7AfmvrTEKrgo=', 'Content-Length': '0'} 2022-09-02T04:42:01.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325ab14df6f2e17e-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.883 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.883 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.884 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.884 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.885 DEBUG:boto:Method: GET 2022-09-02T04:42:01.886 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.886 DEBUG:boto:Data: 2022-09-02T04:42:01.887 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.888 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.888 DEBUG:boto:Params: {} 2022-09-02T04:42:01.889 DEBUG:boto:Token: None 2022-09-02T04:42:01.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9dfd9c0e7261459-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.894 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.894 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.895 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.895 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.896 DEBUG:boto:Method: GET 2022-09-02T04:42:01.897 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.897 DEBUG:boto:Data: 2022-09-02T04:42:01.898 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.899 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.899 DEBUG:boto:Params: {} 2022-09-02T04:42:01.900 DEBUG:boto:Token: None 2022-09-02T04:42:01.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:01.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw= 2022-09-02T04:42:01.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/KIv+h4IlWZ9aV486tCH2kxJIw=', 'Content-Length': '0'} 2022-09-02T04:42:01.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c923acfb688b2c-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.905 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.905 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.906 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.907 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.907 DEBUG:boto:Method: GET 2022-09-02T04:42:01.908 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.908 DEBUG:boto:Data: 2022-09-02T04:42:01.909 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.910 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.910 DEBUG:boto:Params: {} 2022-09-02T04:42:01.911 DEBUG:boto:Token: None 2022-09-02T04:42:01.911 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf7e34288f2f3287-0063118999-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.916 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.917 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:01.918 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:01.918 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.919 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:01.919 DEBUG:boto:Method: GET 2022-09-02T04:42:01.920 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:01.920 DEBUG:boto:Data: 2022-09-02T04:42:01.921 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.922 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.923 DEBUG:boto:Params: {} 2022-09-02T04:42:01.923 DEBUG:boto:Token: None 2022-09-02T04:42:01.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/ 2022-09-02T04:42:01.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY= 2022-09-02T04:42:01.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/ueAqvWJwlleZuDKUhXk1C8lIY=', 'Content-Length': '0'} 2022-09-02T04:42:01.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c201028d189c3b3e-0063118999-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.929 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:01.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.930 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.931 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.931 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.932 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.932 DEBUG:boto:Method: GET 2022-09-02T04:42:01.933 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.933 DEBUG:boto:Data: 2022-09-02T04:42:01.934 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.935 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.936 DEBUG:boto:Params: {} 2022-09-02T04:42:01.936 DEBUG:boto:Token: None 2022-09-02T04:42:01.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk= 2022-09-02T04:42:01.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk=', 'Content-Length': '0'} 2022-09-02T04:42:01.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000466484b9c8fa98eb-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.941 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.942 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.942 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.943 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.943 DEBUG:boto:Method: GET 2022-09-02T04:42:01.944 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.945 DEBUG:boto:Data: 2022-09-02T04:42:01.945 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.946 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.947 DEBUG:boto:Params: {} 2022-09-02T04:42:01.947 DEBUG:boto:Token: None 2022-09-02T04:42:01.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk= 2022-09-02T04:42:01.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk=', 'Content-Length': '0'} 2022-09-02T04:42:01.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022e44236e3ddae56-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.952 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.953 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.953 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.954 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.954 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.955 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.955 DEBUG:boto:Data: 2022-09-02T04:42:01.956 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.957 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.957 DEBUG:boto:Params: {} 2022-09-02T04:42:01.958 DEBUG:boto:Token: None 2022-09-02T04:42:01.959 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LPp4yXRxVKSRES+8RUcpDflzfwM= 2022-09-02T04:42:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LPp4yXRxVKSRES+8RUcpDflzfwM=', 'Content-Length': '0'} 2022-09-02T04:42:01.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cb885af19c18d78-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.963 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.963 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.964 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.964 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.965 DEBUG:boto:Method: HEAD 2022-09-02T04:42:01.965 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.966 DEBUG:boto:Data: 2022-09-02T04:42:01.967 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.968 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.969 DEBUG:boto:Params: {} 2022-09-02T04:42:01.970 DEBUG:boto:Token: None 2022-09-02T04:42:01.971 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LPp4yXRxVKSRES+8RUcpDflzfwM= 2022-09-02T04:42:01.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LPp4yXRxVKSRES+8RUcpDflzfwM=', 'Content-Length': '0'} 2022-09-02T04:42:01.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbcbd50516f031e9-0063118999-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:01.977 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.979 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.979 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.980 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.980 DEBUG:boto:Method: GET 2022-09-02T04:42:01.981 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.982 DEBUG:boto:Data: 2022-09-02T04:42:01.982 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:01.983 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.984 DEBUG:boto:Params: {} 2022-09-02T04:42:01.985 DEBUG:boto:Token: None 2022-09-02T04:42:01.985 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk= 2022-09-02T04:42:01.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk=', 'Content-Length': '0'} 2022-09-02T04:42:01.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c8a164ba954175d-0063118999-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:01.990 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:01.990 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:01.991 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.991 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.992 DEBUG:boto:Method: GET 2022-09-02T04:42:01.993 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.993 DEBUG:boto:Data: 2022-09-02T04:42:01.994 DEBUG:boto:Headers: {} 2022-09-02T04:42:01.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:01.995 DEBUG:boto:Port: 8000 2022-09-02T04:42:01.996 DEBUG:boto:Params: {} 2022-09-02T04:42:01.996 DEBUG:boto:Token: None 2022-09-02T04:42:01.997 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:01 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:01.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk= 2022-09-02T04:42:01.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e9y27ubOG745VkVKfNzgRXcwNIk=', 'Content-Length': '0'} 2022-09-02T04:42:02.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000622c71e19f20fca0-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.002 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.003 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.003 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.004 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.005 DEBUG:boto:Method: GET 2022-09-02T04:42:02.005 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.006 DEBUG:boto:Data: 2022-09-02T04:42:02.006 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.007 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.008 DEBUG:boto:Params: {} 2022-09-02T04:42:02.008 DEBUG:boto:Token: None 2022-09-02T04:42:02.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e619d1ffb9f03083-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.014 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.015 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.016 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.016 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.017 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.018 DEBUG:boto:Method: GET 2022-09-02T04:42:02.018 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.019 DEBUG:boto:Data: 2022-09-02T04:42:02.020 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.021 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.021 DEBUG:boto:Params: {} 2022-09-02T04:42:02.022 DEBUG:boto:Token: None 2022-09-02T04:42:02.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000419b631b7aa30257-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.028 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.029 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.029 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.030 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.030 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.031 DEBUG:boto:Data: 2022-09-02T04:42:02.031 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.032 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.032 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.033 DEBUG:boto:Params: {} 2022-09-02T04:42:02.034 DEBUG:boto:Token: None 2022-09-02T04:42:02.034 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.035 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce598ae077f66a8b-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.040 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.041 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.041 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.042 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.042 DEBUG:boto:Data: 2022-09-02T04:42:02.043 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.044 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.044 DEBUG:boto:Params: {} 2022-09-02T04:42:02.045 DEBUG:boto:Token: None 2022-09-02T04:42:02.046 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b93f11f6526f1ff5-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.052 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.053 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.054 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.054 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.055 DEBUG:boto:Method: GET 2022-09-02T04:42:02.055 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.056 DEBUG:boto:Data: 2022-09-02T04:42:02.057 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.058 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.059 DEBUG:boto:Params: {} 2022-09-02T04:42:02.060 DEBUG:boto:Token: None 2022-09-02T04:42:02.060 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026cb0b599d80bf75-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.106 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.107 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.108 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.108 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.109 DEBUG:boto:Method: GET 2022-09-02T04:42:02.109 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.110 DEBUG:boto:Data: 2022-09-02T04:42:02.110 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.111 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.112 DEBUG:boto:Params: {} 2022-09-02T04:42:02.113 DEBUG:boto:Token: None 2022-09-02T04:42:02.114 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a16ebafad3b2eeaf-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.119 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.120 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.121 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.122 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.123 DEBUG:boto:Method: GET 2022-09-02T04:42:02.124 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.125 DEBUG:boto:Data: 2022-09-02T04:42:02.125 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.127 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.127 DEBUG:boto:Params: {} 2022-09-02T04:42:02.128 DEBUG:boto:Token: None 2022-09-02T04:42:02.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca037f7e2f43b41f-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.134 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.136 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.137 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.138 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.139 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.139 DEBUG:boto:Method: GET 2022-09-02T04:42:02.140 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.141 DEBUG:boto:Data: 2022-09-02T04:42:02.142 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.143 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.143 DEBUG:boto:Params: {} 2022-09-02T04:42:02.144 DEBUG:boto:Token: None 2022-09-02T04:42:02.145 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215d644def534c6b-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.150 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.151 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.152 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.153 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.153 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.154 DEBUG:boto:Method: GET 2022-09-02T04:42:02.154 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.155 DEBUG:boto:Data: 2022-09-02T04:42:02.155 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.157 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.157 DEBUG:boto:Params: {} 2022-09-02T04:42:02.158 DEBUG:boto:Token: None 2022-09-02T04:42:02.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM= 2022-09-02T04:42:02.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM=', 'Content-Length': '0'} 2022-09-02T04:42:02.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d082d0b136b4dd9-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.165 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.166 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.167 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.167 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.168 DEBUG:boto:Method: GET 2022-09-02T04:42:02.168 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.169 DEBUG:boto:Data: 2022-09-02T04:42:02.169 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.170 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.171 DEBUG:boto:Params: {} 2022-09-02T04:42:02.171 DEBUG:boto:Token: None 2022-09-02T04:42:02.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM= 2022-09-02T04:42:02.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM=', 'Content-Length': '0'} 2022-09-02T04:42:02.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c68aadc3db04a126-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.176 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.177 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.177 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.178 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.178 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.179 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.179 DEBUG:boto:Data: 2022-09-02T04:42:02.180 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.181 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.181 DEBUG:boto:Params: {} 2022-09-02T04:42:02.182 DEBUG:boto:Token: None 2022-09-02T04:42:02.183 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ifqX09YSPMhnqYIOHVV7Ni9chKE= 2022-09-02T04:42:02.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ifqX09YSPMhnqYIOHVV7Ni9chKE=', 'Content-Length': '0'} 2022-09-02T04:42:02.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070fbc7487fc37097-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.187 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.187 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.188 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.188 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.189 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.189 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.190 DEBUG:boto:Data: 2022-09-02T04:42:02.190 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.191 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.192 DEBUG:boto:Params: {} 2022-09-02T04:42:02.192 DEBUG:boto:Token: None 2022-09-02T04:42:02.193 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ifqX09YSPMhnqYIOHVV7Ni9chKE= 2022-09-02T04:42:02.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ifqX09YSPMhnqYIOHVV7Ni9chKE=', 'Content-Length': '0'} 2022-09-02T04:42:02.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e693b0a23a17b379-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.198 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.198 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.199 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.199 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.200 DEBUG:boto:Method: GET 2022-09-02T04:42:02.201 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.201 DEBUG:boto:Data: 2022-09-02T04:42:02.202 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.203 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.204 DEBUG:boto:Params: {} 2022-09-02T04:42:02.204 DEBUG:boto:Token: None 2022-09-02T04:42:02.205 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM= 2022-09-02T04:42:02.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM=', 'Content-Length': '0'} 2022-09-02T04:42:02.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7651390bc3eaad0-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.209 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.210 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.210 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.211 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.211 DEBUG:boto:Method: GET 2022-09-02T04:42:02.212 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.213 DEBUG:boto:Data: 2022-09-02T04:42:02.213 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.214 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.215 DEBUG:boto:Params: {} 2022-09-02T04:42:02.215 DEBUG:boto:Token: None 2022-09-02T04:42:02.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:02.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM= 2022-09-02T04:42:02.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YYAXYf5LQPka/1WZKmfmgEhBCoM=', 'Content-Length': '0'} 2022-09-02T04:42:02.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000634501430199546f-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.220 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.221 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.221 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.222 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.222 DEBUG:boto:Method: GET 2022-09-02T04:42:02.223 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.223 DEBUG:boto:Data: 2022-09-02T04:42:02.224 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.225 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.225 DEBUG:boto:Params: {} 2022-09-02T04:42:02.226 DEBUG:boto:Token: None 2022-09-02T04:42:02.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f363cec46dd10b7-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.231 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.232 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.233 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.233 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.234 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.234 DEBUG:boto:Method: GET 2022-09-02T04:42:02.235 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.235 DEBUG:boto:Data: 2022-09-02T04:42:02.236 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.237 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.237 DEBUG:boto:Params: {} 2022-09-02T04:42:02.238 DEBUG:boto:Token: None 2022-09-02T04:42:02.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000967aa1b5c0354681-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.243 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.244 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.245 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.246 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.246 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.247 DEBUG:boto:Data: 2022-09-02T04:42:02.247 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.248 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.249 DEBUG:boto:Params: {} 2022-09-02T04:42:02.249 DEBUG:boto:Token: None 2022-09-02T04:42:02.250 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081deafbc7e4e92be-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.255 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.256 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.257 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.257 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.258 DEBUG:boto:Data: 2022-09-02T04:42:02.258 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.259 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.259 DEBUG:boto:Params: {} 2022-09-02T04:42:02.260 DEBUG:boto:Token: None 2022-09-02T04:42:02.261 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee5a385590ebcfd6-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.268 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.269 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.270 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.271 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.272 DEBUG:boto:Method: GET 2022-09-02T04:42:02.273 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.273 DEBUG:boto:Data: 2022-09-02T04:42:02.275 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.277 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.278 DEBUG:boto:Params: {} 2022-09-02T04:42:02.279 DEBUG:boto:Token: None 2022-09-02T04:42:02.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe33399b314d1054-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.329 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.330 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.331 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.332 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.333 DEBUG:boto:Method: GET 2022-09-02T04:42:02.334 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.335 DEBUG:boto:Data: 2022-09-02T04:42:02.336 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.338 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.339 DEBUG:boto:Params: {} 2022-09-02T04:42:02.340 DEBUG:boto:Token: None 2022-09-02T04:42:02.341 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afd1d9b63513d224-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.347 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.348 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.348 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.349 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.350 DEBUG:boto:Method: GET 2022-09-02T04:42:02.351 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.351 DEBUG:boto:Data: 2022-09-02T04:42:02.352 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.353 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.353 DEBUG:boto:Params: {} 2022-09-02T04:42:02.354 DEBUG:boto:Token: None 2022-09-02T04:42:02.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a350acc4fdbd74a7-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.360 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.361 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.361 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.362 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.372 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.378 DEBUG:boto:Method: GET 2022-09-02T04:42:02.379 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.380 DEBUG:boto:Data: 2022-09-02T04:42:02.381 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.383 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.384 DEBUG:boto:Params: {} 2022-09-02T04:42:02.385 DEBUG:boto:Token: None 2022-09-02T04:42:02.386 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081817fd5b3db8498-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.393 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.395 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.396 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.397 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.397 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.398 DEBUG:boto:Method: GET 2022-09-02T04:42:02.399 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.400 DEBUG:boto:Data: 2022-09-02T04:42:02.401 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.402 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.403 DEBUG:boto:Params: {} 2022-09-02T04:42:02.404 DEBUG:boto:Token: None 2022-09-02T04:42:02.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0= 2022-09-02T04:42:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0=', 'Content-Length': '0'} 2022-09-02T04:42:02.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023855f67045ff081-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.411 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.412 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.412 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.413 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.414 DEBUG:boto:Method: GET 2022-09-02T04:42:02.414 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.415 DEBUG:boto:Data: 2022-09-02T04:42:02.415 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.416 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.417 DEBUG:boto:Params: {} 2022-09-02T04:42:02.418 DEBUG:boto:Token: None 2022-09-02T04:42:02.420 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.422 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0= 2022-09-02T04:42:02.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0=', 'Content-Length': '0'} 2022-09-02T04:42:02.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ccd99bb891ac87c-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.428 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.429 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.430 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.431 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.432 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.433 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.434 DEBUG:boto:Data: 2022-09-02T04:42:02.434 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.435 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.436 DEBUG:boto:Params: {} 2022-09-02T04:42:02.437 DEBUG:boto:Token: None 2022-09-02T04:42:02.437 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSSEDEdW1TjkkMguPUk6D52EYhQ= 2022-09-02T04:42:02.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSSEDEdW1TjkkMguPUk6D52EYhQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3d752c18b99a5ae-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.443 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.444 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.445 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.446 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.447 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.448 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.448 DEBUG:boto:Data: 2022-09-02T04:42:02.449 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.451 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.452 DEBUG:boto:Params: {} 2022-09-02T04:42:02.453 DEBUG:boto:Token: None 2022-09-02T04:42:02.454 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSSEDEdW1TjkkMguPUk6D52EYhQ= 2022-09-02T04:42:02.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSSEDEdW1TjkkMguPUk6D52EYhQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028668c7727862c22-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.462 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.463 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.464 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.465 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.466 DEBUG:boto:Method: GET 2022-09-02T04:42:02.467 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.468 DEBUG:boto:Data: 2022-09-02T04:42:02.469 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.471 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.472 DEBUG:boto:Params: {} 2022-09-02T04:42:02.473 DEBUG:boto:Token: None 2022-09-02T04:42:02.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0= 2022-09-02T04:42:02.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0=', 'Content-Length': '0'} 2022-09-02T04:42:02.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c7896ddd00f849d-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.478 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.478 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.479 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.479 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.480 DEBUG:boto:Method: GET 2022-09-02T04:42:02.480 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.481 DEBUG:boto:Data: 2022-09-02T04:42:02.482 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.483 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.484 DEBUG:boto:Params: {} 2022-09-02T04:42:02.484 DEBUG:boto:Token: None 2022-09-02T04:42:02.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:02.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0= 2022-09-02T04:42:02.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JqxQ7z2+s9ks7RAidqiLc4Eq2d0=', 'Content-Length': '0'} 2022-09-02T04:42:02.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fccec200e13525a-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.530 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.531 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.532 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.533 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.534 DEBUG:boto:Method: GET 2022-09-02T04:42:02.535 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.536 DEBUG:boto:Data: 2022-09-02T04:42:02.537 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.538 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.539 DEBUG:boto:Params: {} 2022-09-02T04:42:02.540 DEBUG:boto:Token: None 2022-09-02T04:42:02.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e34bbb6f74eb198-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.547 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.548 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.549 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.550 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.551 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.552 DEBUG:boto:Method: GET 2022-09-02T04:42:02.553 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.554 DEBUG:boto:Data: 2022-09-02T04:42:02.555 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.557 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.558 DEBUG:boto:Params: {} 2022-09-02T04:42:02.560 DEBUG:boto:Token: None 2022-09-02T04:42:02.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb67eb043965844-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.567 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.569 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.570 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.571 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.572 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.572 DEBUG:boto:Data: 2022-09-02T04:42:02.573 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.575 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.575 DEBUG:boto:Params: {} 2022-09-02T04:42:02.576 DEBUG:boto:Token: None 2022-09-02T04:42:02.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002439691e5963651f-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.583 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.583 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.584 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.584 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.585 DEBUG:boto:Data: 2022-09-02T04:42:02.585 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.586 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.587 DEBUG:boto:Params: {} 2022-09-02T04:42:02.587 DEBUG:boto:Token: None 2022-09-02T04:42:02.588 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004645a6cff47ff6cd-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.594 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.595 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.595 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.596 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.596 DEBUG:boto:Method: GET 2022-09-02T04:42:02.597 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.597 DEBUG:boto:Data: 2022-09-02T04:42:02.598 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.599 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.599 DEBUG:boto:Params: {} 2022-09-02T04:42:02.600 DEBUG:boto:Token: None 2022-09-02T04:42:02.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005915ba905ec1413e-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.646 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.647 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.647 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.648 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.648 DEBUG:boto:Method: GET 2022-09-02T04:42:02.649 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.649 DEBUG:boto:Data: 2022-09-02T04:42:02.650 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.651 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.652 DEBUG:boto:Params: {} 2022-09-02T04:42:02.653 DEBUG:boto:Token: None 2022-09-02T04:42:02.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000673b8ae04f4876e1-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.658 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.659 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.659 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.660 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.660 DEBUG:boto:Method: GET 2022-09-02T04:42:02.661 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.661 DEBUG:boto:Data: 2022-09-02T04:42:02.662 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.663 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.663 DEBUG:boto:Params: {} 2022-09-02T04:42:02.664 DEBUG:boto:Token: None 2022-09-02T04:42:02.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011139d237914e9ed-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.670 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.670 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.671 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.671 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.672 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.673 DEBUG:boto:Method: GET 2022-09-02T04:42:02.673 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.674 DEBUG:boto:Data: 2022-09-02T04:42:02.674 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.675 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.676 DEBUG:boto:Params: {} 2022-09-02T04:42:02.676 DEBUG:boto:Token: None 2022-09-02T04:42:02.677 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.678 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77d38a69e33639f-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.682 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.684 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.684 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.685 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.685 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.686 DEBUG:boto:Method: GET 2022-09-02T04:42:02.686 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.687 DEBUG:boto:Data: 2022-09-02T04:42:02.688 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.689 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.689 DEBUG:boto:Params: {} 2022-09-02T04:42:02.690 DEBUG:boto:Token: None 2022-09-02T04:42:02.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g= 2022-09-02T04:42:02.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g=', 'Content-Length': '0'} 2022-09-02T04:42:02.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028bad9d5e37c26ef-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.696 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.696 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.697 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.697 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.698 DEBUG:boto:Method: GET 2022-09-02T04:42:02.698 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.699 DEBUG:boto:Data: 2022-09-02T04:42:02.699 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.701 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.702 DEBUG:boto:Params: {} 2022-09-02T04:42:02.703 DEBUG:boto:Token: None 2022-09-02T04:42:02.703 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g= 2022-09-02T04:42:02.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g=', 'Content-Length': '0'} 2022-09-02T04:42:02.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bbd282dd03e8bfe-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.708 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.709 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.709 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.710 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.711 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.712 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.712 DEBUG:boto:Data: 2022-09-02T04:42:02.713 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.715 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.716 DEBUG:boto:Params: {} 2022-09-02T04:42:02.717 DEBUG:boto:Token: None 2022-09-02T04:42:02.718 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wQCSVrmb0cQWB/siE4yZmX7f0Ww= 2022-09-02T04:42:02.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wQCSVrmb0cQWB/siE4yZmX7f0Ww=', 'Content-Length': '0'} 2022-09-02T04:42:02.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000063d7498406dc3fc-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.723 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.724 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.725 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.726 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.727 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.728 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.729 DEBUG:boto:Data: 2022-09-02T04:42:02.730 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.731 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.733 DEBUG:boto:Params: {} 2022-09-02T04:42:02.734 DEBUG:boto:Token: None 2022-09-02T04:42:02.736 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wQCSVrmb0cQWB/siE4yZmX7f0Ww= 2022-09-02T04:42:02.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wQCSVrmb0cQWB/siE4yZmX7f0Ww=', 'Content-Length': '0'} 2022-09-02T04:42:02.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098b1d52fa381980b-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.745 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.747 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.748 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.749 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.750 DEBUG:boto:Method: GET 2022-09-02T04:42:02.750 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.751 DEBUG:boto:Data: 2022-09-02T04:42:02.751 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.752 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.753 DEBUG:boto:Params: {} 2022-09-02T04:42:02.754 DEBUG:boto:Token: None 2022-09-02T04:42:02.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g= 2022-09-02T04:42:02.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g=', 'Content-Length': '0'} 2022-09-02T04:42:02.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b5e847ea71f5560-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.806 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.807 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.808 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.809 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.810 DEBUG:boto:Method: GET 2022-09-02T04:42:02.811 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.812 DEBUG:boto:Data: 2022-09-02T04:42:02.813 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.815 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.816 DEBUG:boto:Params: {} 2022-09-02T04:42:02.817 DEBUG:boto:Token: None 2022-09-02T04:42:02.818 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:02.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g= 2022-09-02T04:42:02.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N4SUMVmVMtHfTfsUNi0Bu9Y9F/g=', 'Content-Length': '0'} 2022-09-02T04:42:02.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000802f895fe10be321-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.826 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.827 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.829 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.829 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.830 DEBUG:boto:Method: GET 2022-09-02T04:42:02.831 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.832 DEBUG:boto:Data: 2022-09-02T04:42:02.833 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.836 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.837 DEBUG:boto:Params: {} 2022-09-02T04:42:02.839 DEBUG:boto:Token: None 2022-09-02T04:42:02.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c939506481385d-006311899a-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.850 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.851 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.853 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.854 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.855 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.856 DEBUG:boto:Method: GET 2022-09-02T04:42:02.857 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.858 DEBUG:boto:Data: 2022-09-02T04:42:02.859 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.861 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.862 DEBUG:boto:Params: {} 2022-09-02T04:42:02.863 DEBUG:boto:Token: None 2022-09-02T04:42:02.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/ 2022-09-02T04:42:02.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ= 2022-09-02T04:42:02.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B3nndXDoT/JtojuI4v7R0G0MPiQ=', 'Content-Length': '0'} 2022-09-02T04:42:02.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7cb3cc8e0dddaf0-006311899a-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.873 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:02.874 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.875 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.876 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.877 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.878 DEBUG:boto:Data: 2022-09-02T04:42:02.879 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.881 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.882 DEBUG:boto:Params: {} 2022-09-02T04:42:02.883 DEBUG:boto:Token: None 2022-09-02T04:42:02.884 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.891 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014bbcdba98903e53-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.892 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.895 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.896 DEBUG:boto:Method: HEAD 2022-09-02T04:42:02.897 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:02.897 DEBUG:boto:Data: 2022-09-02T04:42:02.898 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.900 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.901 DEBUG:boto:Params: {} 2022-09-02T04:42:02.903 DEBUG:boto:Token: None 2022-09-02T04:42:02.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj 2022-09-02T04:42:02.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0= 2022-09-02T04:42:02.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y7wJuCxIY0CvVq0FUbU3EJydTe0=', 'Content-Length': '0'} 2022-09-02T04:42:02.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056c580c325fca29d-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:02.918 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.918 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.919 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.920 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.920 DEBUG:boto:Method: GET 2022-09-02T04:42:02.921 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.922 DEBUG:boto:Data: 2022-09-02T04:42:02.923 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:02.925 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.925 DEBUG:boto:Params: {} 2022-09-02T04:42:02.926 DEBUG:boto:Token: None 2022-09-02T04:42:02.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000904b62fc4974320a-006311899a-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.980 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:02.981 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:02.981 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.983 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.984 DEBUG:boto:Method: GET 2022-09-02T04:42:02.985 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.985 DEBUG:boto:Data: 2022-09-02T04:42:02.986 DEBUG:boto:Headers: {} 2022-09-02T04:42:02.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:02.987 DEBUG:boto:Port: 8000 2022-09-02T04:42:02.987 DEBUG:boto:Params: {} 2022-09-02T04:42:02.988 DEBUG:boto:Token: None 2022-09-02T04:42:02.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:02 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:02.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk= 2022-09-02T04:42:02.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9BrfziarccXDGiHjA96cKenPUk=', 'Content-Length': '0'} 2022-09-02T04:42:02.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dec451ac5a8e565a-006311899a-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:02.996 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:02.997 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:02.997 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.998 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:02.998 DEBUG:boto:Method: GET 2022-09-02T04:42:02.999 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:02.999 DEBUG:boto:Data: 2022-09-02T04:42:03.000 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.002 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.002 DEBUG:boto:Params: {} 2022-09-02T04:42:03.003 DEBUG:boto:Token: None 2022-09-02T04:42:03.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b1a0df72ab01aae-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.010 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.011 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.012 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.012 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.013 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.013 DEBUG:boto:Method: GET 2022-09-02T04:42:03.014 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.015 DEBUG:boto:Data: 2022-09-02T04:42:03.015 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.016 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.017 DEBUG:boto:Params: {} 2022-09-02T04:42:03.017 DEBUG:boto:Token: None 2022-09-02T04:42:03.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d50b82527928630-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.023 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.024 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.025 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.026 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.026 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.027 DEBUG:boto:Method: GET 2022-09-02T04:42:03.027 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.028 DEBUG:boto:Data: 2022-09-02T04:42:03.028 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.029 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.030 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.030 DEBUG:boto:Params: {} 2022-09-02T04:42:03.031 DEBUG:boto:Token: None 2022-09-02T04:42:03.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A= 2022-09-02T04:42:03.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A=', 'Content-Length': '0'} 2022-09-02T04:42:03.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000552e700ce2c468ba-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.037 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.038 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.039 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.039 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.040 DEBUG:boto:Method: GET 2022-09-02T04:42:03.040 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.041 DEBUG:boto:Data: 2022-09-02T04:42:03.041 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.043 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.043 DEBUG:boto:Params: {} 2022-09-02T04:42:03.044 DEBUG:boto:Token: None 2022-09-02T04:42:03.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A= 2022-09-02T04:42:03.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A=', 'Content-Length': '0'} 2022-09-02T04:42:03.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074880024c764b978-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.049 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.050 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.050 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.051 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.051 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.052 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.052 DEBUG:boto:Data: 2022-09-02T04:42:03.053 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.054 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.054 DEBUG:boto:Params: {} 2022-09-02T04:42:03.055 DEBUG:boto:Token: None 2022-09-02T04:42:03.055 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ieleSJRZpm6YzwURa/rZ2NHn4oA= 2022-09-02T04:42:03.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ieleSJRZpm6YzwURa/rZ2NHn4oA=', 'Content-Length': '0'} 2022-09-02T04:42:03.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfa3804ccfedf324-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.062 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.063 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.063 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.064 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.065 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.065 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.066 DEBUG:boto:Data: 2022-09-02T04:42:03.067 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.068 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.069 DEBUG:boto:Params: {} 2022-09-02T04:42:03.070 DEBUG:boto:Token: None 2022-09-02T04:42:03.070 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ieleSJRZpm6YzwURa/rZ2NHn4oA= 2022-09-02T04:42:03.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ieleSJRZpm6YzwURa/rZ2NHn4oA=', 'Content-Length': '0'} 2022-09-02T04:42:03.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c66921052655b1c-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.075 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.076 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.076 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.077 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.077 DEBUG:boto:Method: GET 2022-09-02T04:42:03.078 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.078 DEBUG:boto:Data: 2022-09-02T04:42:03.079 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.080 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.080 DEBUG:boto:Params: {} 2022-09-02T04:42:03.081 DEBUG:boto:Token: None 2022-09-02T04:42:03.082 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.083 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A= 2022-09-02T04:42:03.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A=', 'Content-Length': '0'} 2022-09-02T04:42:03.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098d51b84bab16bc1-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.087 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.088 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.089 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.089 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.090 DEBUG:boto:Method: GET 2022-09-02T04:42:03.091 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.091 DEBUG:boto:Data: 2022-09-02T04:42:03.092 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.093 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.093 DEBUG:boto:Params: {} 2022-09-02T04:42:03.094 DEBUG:boto:Token: None 2022-09-02T04:42:03.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:03.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A= 2022-09-02T04:42:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I6UkizM44nLjTLDHlmfCac/Pl3A=', 'Content-Length': '0'} 2022-09-02T04:42:03.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0bb205a37a81a12-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.099 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.100 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.101 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.101 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.102 DEBUG:boto:Method: GET 2022-09-02T04:42:03.103 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.103 DEBUG:boto:Data: 2022-09-02T04:42:03.104 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.105 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.105 DEBUG:boto:Params: {} 2022-09-02T04:42:03.106 DEBUG:boto:Token: None 2022-09-02T04:42:03.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a755ec6d56da7b9-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.158 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.160 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.160 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.161 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.162 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.163 DEBUG:boto:Method: GET 2022-09-02T04:42:03.163 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.164 DEBUG:boto:Data: 2022-09-02T04:42:03.165 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.166 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.167 DEBUG:boto:Params: {} 2022-09-02T04:42:03.168 DEBUG:boto:Token: None 2022-09-02T04:42:03.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048b34ffd45c4e74c-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.175 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.176 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.176 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.177 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.177 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.179 DEBUG:boto:Data: 2022-09-02T04:42:03.179 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.180 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.180 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.181 DEBUG:boto:Params: {} 2022-09-02T04:42:03.181 DEBUG:boto:Token: None 2022-09-02T04:42:03.182 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008556c9e8f46aecaa-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.188 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.188 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.189 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.190 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.190 DEBUG:boto:Data: 2022-09-02T04:42:03.191 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.192 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.192 DEBUG:boto:Params: {} 2022-09-02T04:42:03.193 DEBUG:boto:Token: None 2022-09-02T04:42:03.193 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000081f27724a33d91-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.201 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.202 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.203 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.204 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.205 DEBUG:boto:Method: GET 2022-09-02T04:42:03.206 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.207 DEBUG:boto:Data: 2022-09-02T04:42:03.208 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.210 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.211 DEBUG:boto:Params: {} 2022-09-02T04:42:03.212 DEBUG:boto:Token: None 2022-09-02T04:42:03.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8= 2022-09-02T04:42:03.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8=', 'Content-Length': '0'} 2022-09-02T04:42:03.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5d0fa559e85789c-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.217 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.218 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.218 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.219 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.220 DEBUG:boto:Method: GET 2022-09-02T04:42:03.221 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.221 DEBUG:boto:Data: 2022-09-02T04:42:03.222 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.223 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.223 DEBUG:boto:Params: {} 2022-09-02T04:42:03.224 DEBUG:boto:Token: None 2022-09-02T04:42:03.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8= 2022-09-02T04:42:03.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8=', 'Content-Length': '0'} 2022-09-02T04:42:03.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000049bb52d82b8ee07-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.270 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.271 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.271 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.272 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.272 DEBUG:boto:Method: GET 2022-09-02T04:42:03.273 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.273 DEBUG:boto:Data: 2022-09-02T04:42:03.274 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.275 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.276 DEBUG:boto:Params: {} 2022-09-02T04:42:03.276 DEBUG:boto:Token: None 2022-09-02T04:42:03.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000838548a238b43746-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.282 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.284 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.285 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.286 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.287 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.288 DEBUG:boto:Method: GET 2022-09-02T04:42:03.288 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.289 DEBUG:boto:Data: 2022-09-02T04:42:03.290 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.291 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.292 DEBUG:boto:Params: {} 2022-09-02T04:42:03.293 DEBUG:boto:Token: None 2022-09-02T04:42:03.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ff0d54738f85f8-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.301 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.302 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.303 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.303 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.304 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.305 DEBUG:boto:Method: GET 2022-09-02T04:42:03.305 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.306 DEBUG:boto:Data: 2022-09-02T04:42:03.307 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.308 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.309 DEBUG:boto:Params: {} 2022-09-02T04:42:03.309 DEBUG:boto:Token: None 2022-09-02T04:42:03.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms= 2022-09-02T04:42:03.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms=', 'Content-Length': '0'} 2022-09-02T04:42:03.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5ad1d1a79f2b6e2-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.316 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.318 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.319 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.320 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.321 DEBUG:boto:Method: GET 2022-09-02T04:42:03.322 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.323 DEBUG:boto:Data: 2022-09-02T04:42:03.324 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.325 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.326 DEBUG:boto:Params: {} 2022-09-02T04:42:03.327 DEBUG:boto:Token: None 2022-09-02T04:42:03.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms= 2022-09-02T04:42:03.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms=', 'Content-Length': '0'} 2022-09-02T04:42:03.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ff50f1f8394d637-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.335 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.336 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.337 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.338 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.339 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.340 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.340 DEBUG:boto:Data: 2022-09-02T04:42:03.341 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.343 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.344 DEBUG:boto:Params: {} 2022-09-02T04:42:03.345 DEBUG:boto:Token: None 2022-09-02T04:42:03.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/Mhr9OtEysAtBbeAnnKjgvJdfE= 2022-09-02T04:42:03.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/Mhr9OtEysAtBbeAnnKjgvJdfE=', 'Content-Length': '0'} 2022-09-02T04:42:03.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5dab2c45ab4aff7-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.351 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.352 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.353 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.354 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.354 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.355 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.355 DEBUG:boto:Data: 2022-09-02T04:42:03.356 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.357 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.358 DEBUG:boto:Params: {} 2022-09-02T04:42:03.358 DEBUG:boto:Token: None 2022-09-02T04:42:03.359 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/Mhr9OtEysAtBbeAnnKjgvJdfE= 2022-09-02T04:42:03.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/Mhr9OtEysAtBbeAnnKjgvJdfE=', 'Content-Length': '0'} 2022-09-02T04:42:03.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035cfbe35dc63581a-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.364 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.365 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.366 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.366 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.367 DEBUG:boto:Method: GET 2022-09-02T04:42:03.368 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.368 DEBUG:boto:Data: 2022-09-02T04:42:03.369 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.370 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.370 DEBUG:boto:Params: {} 2022-09-02T04:42:03.371 DEBUG:boto:Token: None 2022-09-02T04:42:03.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms= 2022-09-02T04:42:03.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms=', 'Content-Length': '0'} 2022-09-02T04:42:03.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b27986974703328d-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.427 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.428 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.429 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.430 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.431 DEBUG:boto:Method: GET 2022-09-02T04:42:03.432 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.433 DEBUG:boto:Data: 2022-09-02T04:42:03.434 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.436 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.437 DEBUG:boto:Params: {} 2022-09-02T04:42:03.438 DEBUG:boto:Token: None 2022-09-02T04:42:03.440 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:03.441 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms= 2022-09-02T04:42:03.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:67FQw5GidnDg5bRkcMuOij0aPms=', 'Content-Length': '0'} 2022-09-02T04:42:03.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a9cf28221774e95-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.447 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.448 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.449 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.450 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.451 DEBUG:boto:Method: GET 2022-09-02T04:42:03.452 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.453 DEBUG:boto:Data: 2022-09-02T04:42:03.454 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.456 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.457 DEBUG:boto:Params: {} 2022-09-02T04:42:03.459 DEBUG:boto:Token: None 2022-09-02T04:42:03.460 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009744580bb2d1fe2c-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.466 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.467 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.468 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.469 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.469 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.470 DEBUG:boto:Method: GET 2022-09-02T04:42:03.470 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.471 DEBUG:boto:Data: 2022-09-02T04:42:03.471 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.472 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.473 DEBUG:boto:Params: {} 2022-09-02T04:42:03.474 DEBUG:boto:Token: None 2022-09-02T04:42:03.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e59fd3c12a0557-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.480 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.481 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.481 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.482 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.482 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.483 DEBUG:boto:Data: 2022-09-02T04:42:03.483 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.484 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.485 DEBUG:boto:Params: {} 2022-09-02T04:42:03.486 DEBUG:boto:Token: None 2022-09-02T04:42:03.486 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa2e3c437a37c8c4-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.492 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.492 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.493 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.493 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.495 DEBUG:boto:Data: 2022-09-02T04:42:03.495 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.496 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.497 DEBUG:boto:Params: {} 2022-09-02T04:42:03.498 DEBUG:boto:Token: None 2022-09-02T04:42:03.498 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d77d55bb6d1ce69-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.505 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.505 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.506 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.507 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.508 DEBUG:boto:Method: GET 2022-09-02T04:42:03.510 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.510 DEBUG:boto:Data: 2022-09-02T04:42:03.511 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.512 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.512 DEBUG:boto:Params: {} 2022-09-02T04:42:03.513 DEBUG:boto:Token: None 2022-09-02T04:42:03.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8= 2022-09-02T04:42:03.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8=', 'Content-Length': '0'} 2022-09-02T04:42:03.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072c1e4ed6433731b-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.518 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.519 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.519 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.520 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.520 DEBUG:boto:Method: GET 2022-09-02T04:42:03.521 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.521 DEBUG:boto:Data: 2022-09-02T04:42:03.522 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.523 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.523 DEBUG:boto:Params: {} 2022-09-02T04:42:03.524 DEBUG:boto:Token: None 2022-09-02T04:42:03.524 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8= 2022-09-02T04:42:03.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8=', 'Content-Length': '0'} 2022-09-02T04:42:03.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b145a39460d90973-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.529 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.529 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.530 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.530 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.531 DEBUG:boto:Method: GET 2022-09-02T04:42:03.531 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.532 DEBUG:boto:Data: 2022-09-02T04:42:03.532 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.533 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.534 DEBUG:boto:Params: {} 2022-09-02T04:42:03.535 DEBUG:boto:Token: None 2022-09-02T04:42:03.535 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b10926b85b785d9d-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.540 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.541 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.541 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.542 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.542 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.543 DEBUG:boto:Method: GET 2022-09-02T04:42:03.543 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.544 DEBUG:boto:Data: 2022-09-02T04:42:03.544 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.545 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.546 DEBUG:boto:Params: {} 2022-09-02T04:42:03.546 DEBUG:boto:Token: None 2022-09-02T04:42:03.547 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb7f54c6db4c873-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.552 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.554 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.554 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.555 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.555 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.556 DEBUG:boto:Method: GET 2022-09-02T04:42:03.556 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.557 DEBUG:boto:Data: 2022-09-02T04:42:03.557 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.559 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.559 DEBUG:boto:Params: {} 2022-09-02T04:42:03.560 DEBUG:boto:Token: None 2022-09-02T04:42:03.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg= 2022-09-02T04:42:03.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg=', 'Content-Length': '0'} 2022-09-02T04:42:03.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cfb209a82483430-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.565 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.566 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.566 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.567 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.567 DEBUG:boto:Method: GET 2022-09-02T04:42:03.568 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.568 DEBUG:boto:Data: 2022-09-02T04:42:03.569 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.570 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.571 DEBUG:boto:Params: {} 2022-09-02T04:42:03.571 DEBUG:boto:Token: None 2022-09-02T04:42:03.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg= 2022-09-02T04:42:03.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg=', 'Content-Length': '0'} 2022-09-02T04:42:03.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2d32362bc26a83c-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.619 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.620 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.620 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.621 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.621 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.622 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.622 DEBUG:boto:Data: 2022-09-02T04:42:03.623 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.624 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.625 DEBUG:boto:Params: {} 2022-09-02T04:42:03.625 DEBUG:boto:Token: None 2022-09-02T04:42:03.626 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QWRuBv+4cRVphLSQaI9tAmwt4X8= 2022-09-02T04:42:03.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QWRuBv+4cRVphLSQaI9tAmwt4X8=', 'Content-Length': '0'} 2022-09-02T04:42:03.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083a9485baef80f8a-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.630 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.631 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.631 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.632 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.632 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.633 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.633 DEBUG:boto:Data: 2022-09-02T04:42:03.634 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.635 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.636 DEBUG:boto:Params: {} 2022-09-02T04:42:03.636 DEBUG:boto:Token: None 2022-09-02T04:42:03.637 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QWRuBv+4cRVphLSQaI9tAmwt4X8= 2022-09-02T04:42:03.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QWRuBv+4cRVphLSQaI9tAmwt4X8=', 'Content-Length': '0'} 2022-09-02T04:42:03.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037474df97783f29e-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.641 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.642 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.642 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.643 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.643 DEBUG:boto:Method: GET 2022-09-02T04:42:03.644 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.645 DEBUG:boto:Data: 2022-09-02T04:42:03.645 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.646 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.646 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.647 DEBUG:boto:Params: {} 2022-09-02T04:42:03.647 DEBUG:boto:Token: None 2022-09-02T04:42:03.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg= 2022-09-02T04:42:03.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg=', 'Content-Length': '0'} 2022-09-02T04:42:03.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5607270287421ca-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.654 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.655 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.655 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.656 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.656 DEBUG:boto:Method: GET 2022-09-02T04:42:03.657 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.657 DEBUG:boto:Data: 2022-09-02T04:42:03.658 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.659 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.660 DEBUG:boto:Params: {} 2022-09-02T04:42:03.660 DEBUG:boto:Token: None 2022-09-02T04:42:03.661 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:03.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg= 2022-09-02T04:42:03.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UznhwWk0pJjjvssE9JRdwNGS5kg=', 'Content-Length': '0'} 2022-09-02T04:42:03.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a453a1694bc81918-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.665 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.666 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.667 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.667 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.668 DEBUG:boto:Method: GET 2022-09-02T04:42:03.668 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.669 DEBUG:boto:Data: 2022-09-02T04:42:03.669 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.670 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.671 DEBUG:boto:Params: {} 2022-09-02T04:42:03.672 DEBUG:boto:Token: None 2022-09-02T04:42:03.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724648d7ff9953e5-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.677 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.678 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.679 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.679 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.680 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.681 DEBUG:boto:Method: GET 2022-09-02T04:42:03.681 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.682 DEBUG:boto:Data: 2022-09-02T04:42:03.682 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.683 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.684 DEBUG:boto:Params: {} 2022-09-02T04:42:03.685 DEBUG:boto:Token: None 2022-09-02T04:42:03.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a28f484e4f916b4-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.690 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.691 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.692 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.692 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.693 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.693 DEBUG:boto:Data: 2022-09-02T04:42:03.694 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.694 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.695 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.695 DEBUG:boto:Params: {} 2022-09-02T04:42:03.696 DEBUG:boto:Token: None 2022-09-02T04:42:03.697 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094463cd24d6faffb-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.705 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.705 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.706 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.707 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.707 DEBUG:boto:Data: 2022-09-02T04:42:03.708 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.709 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.709 DEBUG:boto:Params: {} 2022-09-02T04:42:03.710 DEBUG:boto:Token: None 2022-09-02T04:42:03.710 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f1d79ca93da846f-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.717 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.717 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.718 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.719 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.719 DEBUG:boto:Method: GET 2022-09-02T04:42:03.720 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.721 DEBUG:boto:Data: 2022-09-02T04:42:03.721 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.722 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.723 DEBUG:boto:Params: {} 2022-09-02T04:42:03.724 DEBUG:boto:Token: None 2022-09-02T04:42:03.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8= 2022-09-02T04:42:03.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8=', 'Content-Length': '0'} 2022-09-02T04:42:03.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006717b8d2f07117fb-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.729 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.729 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.730 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.731 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.731 DEBUG:boto:Method: GET 2022-09-02T04:42:03.732 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.732 DEBUG:boto:Data: 2022-09-02T04:42:03.733 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.734 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.735 DEBUG:boto:Params: {} 2022-09-02T04:42:03.735 DEBUG:boto:Token: None 2022-09-02T04:42:03.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:03.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8= 2022-09-02T04:42:03.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FgoHvxrOWfCvA3X3mYE87+lcVB8=', 'Content-Length': '0'} 2022-09-02T04:42:03.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030ed900b3561f9a9-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.741 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.742 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.742 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.743 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.744 DEBUG:boto:Method: GET 2022-09-02T04:42:03.744 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.745 DEBUG:boto:Data: 2022-09-02T04:42:03.745 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.748 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.748 DEBUG:boto:Params: {} 2022-09-02T04:42:03.749 DEBUG:boto:Token: None 2022-09-02T04:42:03.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d424c5262731ed-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.755 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.756 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.756 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.757 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.757 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.758 DEBUG:boto:Method: GET 2022-09-02T04:42:03.759 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.760 DEBUG:boto:Data: 2022-09-02T04:42:03.761 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.763 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.764 DEBUG:boto:Params: {} 2022-09-02T04:42:03.766 DEBUG:boto:Token: None 2022-09-02T04:42:03.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9f4484441c5d22-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.773 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.775 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.776 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.777 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.778 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.779 DEBUG:boto:Method: GET 2022-09-02T04:42:03.780 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.780 DEBUG:boto:Data: 2022-09-02T04:42:03.781 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.783 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.783 DEBUG:boto:Params: {} 2022-09-02T04:42:03.784 DEBUG:boto:Token: None 2022-09-02T04:42:03.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM= 2022-09-02T04:42:03.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM=', 'Content-Length': '0'} 2022-09-02T04:42:03.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3ef423d72116408-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.841 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.842 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.843 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.844 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.845 DEBUG:boto:Method: GET 2022-09-02T04:42:03.846 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.847 DEBUG:boto:Data: 2022-09-02T04:42:03.848 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.849 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.850 DEBUG:boto:Params: {} 2022-09-02T04:42:03.851 DEBUG:boto:Token: None 2022-09-02T04:42:03.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM= 2022-09-02T04:42:03.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM=', 'Content-Length': '0'} 2022-09-02T04:42:03.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009267b243623473ba-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.859 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.860 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.861 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.862 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.863 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.863 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.864 DEBUG:boto:Data: 2022-09-02T04:42:03.865 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.867 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.868 DEBUG:boto:Params: {} 2022-09-02T04:42:03.869 DEBUG:boto:Token: None 2022-09-02T04:42:03.871 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zLGSmKOusyzcwYlyIi3x96TVcUo= 2022-09-02T04:42:03.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zLGSmKOusyzcwYlyIi3x96TVcUo=', 'Content-Length': '0'} 2022-09-02T04:42:03.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2ee8d842b632f10-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.876 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.877 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.877 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.878 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.878 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.879 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.879 DEBUG:boto:Data: 2022-09-02T04:42:03.880 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.881 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.882 DEBUG:boto:Params: {} 2022-09-02T04:42:03.882 DEBUG:boto:Token: None 2022-09-02T04:42:03.884 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zLGSmKOusyzcwYlyIi3x96TVcUo= 2022-09-02T04:42:03.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zLGSmKOusyzcwYlyIi3x96TVcUo=', 'Content-Length': '0'} 2022-09-02T04:42:03.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1263461c1dc0d3a-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:03.889 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.889 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.890 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.891 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.891 DEBUG:boto:Method: GET 2022-09-02T04:42:03.892 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.892 DEBUG:boto:Data: 2022-09-02T04:42:03.893 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.894 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.895 DEBUG:boto:Params: {} 2022-09-02T04:42:03.895 DEBUG:boto:Token: None 2022-09-02T04:42:03.896 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM= 2022-09-02T04:42:03.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM=', 'Content-Length': '0'} 2022-09-02T04:42:03.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d50d7c117c82b38-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.901 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.901 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.902 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.902 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.903 DEBUG:boto:Method: GET 2022-09-02T04:42:03.903 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.904 DEBUG:boto:Data: 2022-09-02T04:42:03.905 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.906 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.906 DEBUG:boto:Params: {} 2022-09-02T04:42:03.907 DEBUG:boto:Token: None 2022-09-02T04:42:03.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:03.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM= 2022-09-02T04:42:03.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:scN00nIqROQd69/xhsT1yWYzjlM=', 'Content-Length': '0'} 2022-09-02T04:42:03.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000040e09985af7d3be-006311899b-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.912 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.912 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.913 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.913 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.914 DEBUG:boto:Method: GET 2022-09-02T04:42:03.914 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.915 DEBUG:boto:Data: 2022-09-02T04:42:03.915 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.916 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.917 DEBUG:boto:Params: {} 2022-09-02T04:42:03.918 DEBUG:boto:Token: None 2022-09-02T04:42:03.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f509c128d2ca154-006311899b-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.924 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.925 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:03.926 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:03.926 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.927 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:03.927 DEBUG:boto:Method: GET 2022-09-02T04:42:03.928 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:03.928 DEBUG:boto:Data: 2022-09-02T04:42:03.929 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:03.930 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.930 DEBUG:boto:Params: {} 2022-09-02T04:42:03.931 DEBUG:boto:Token: None 2022-09-02T04:42:03.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/ 2022-09-02T04:42:03.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo= 2022-09-02T04:42:03.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Ul6xaZu5ZnEd4MhsfsAsoYtpqo=', 'Content-Length': '0'} 2022-09-02T04:42:03.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d6d51247252cb9-006311899b-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.982 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:03.984 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:03.985 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:03.986 DEBUG:boto:Method: HEAD 2022-09-02T04:42:03.986 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:03.987 DEBUG:boto:Data: 2022-09-02T04:42:03.987 DEBUG:boto:Headers: {} 2022-09-02T04:42:03.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:03.988 DEBUG:boto:Port: 8000 2022-09-02T04:42:03.989 DEBUG:boto:Params: {} 2022-09-02T04:42:03.990 DEBUG:boto:Token: None 2022-09-02T04:42:03.991 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:03 GMT /xljbkf-19/obj 2022-09-02T04:42:03.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c= 2022-09-02T04:42:03.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NxK/GW1LRBBkTHAN32At08qH85c=', 'Content-Length': '0'} 2022-09-02T04:42:03.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004528f4e5f329ca05-006311899b-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:03.999 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.000 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.001 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.001 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.002 DEBUG:boto:Data: 2022-09-02T04:42:04.003 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.004 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.004 DEBUG:boto:Params: {} 2022-09-02T04:42:04.005 DEBUG:boto:Token: None 2022-09-02T04:42:04.006 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da14dcd1b70abae3-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.013 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.013 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.014 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.014 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.015 DEBUG:boto:Method: GET 2022-09-02T04:42:04.015 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.016 DEBUG:boto:Data: 2022-09-02T04:42:04.016 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.017 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.018 DEBUG:boto:Params: {} 2022-09-02T04:42:04.019 DEBUG:boto:Token: None 2022-09-02T04:42:04.019 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a95360709c89688-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.023 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.024 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.024 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.025 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.026 DEBUG:boto:Method: GET 2022-09-02T04:42:04.026 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.027 DEBUG:boto:Data: 2022-09-02T04:42:04.027 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.028 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.029 DEBUG:boto:Params: {} 2022-09-02T04:42:04.029 DEBUG:boto:Token: None 2022-09-02T04:42:04.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000247834c18dbeed23-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.034 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.035 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.035 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.036 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.036 DEBUG:boto:Method: GET 2022-09-02T04:42:04.037 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.037 DEBUG:boto:Data: 2022-09-02T04:42:04.038 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.039 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.040 DEBUG:boto:Params: {} 2022-09-02T04:42:04.040 DEBUG:boto:Token: None 2022-09-02T04:42:04.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c140886be3d22e-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.046 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.046 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.047 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.048 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.048 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.049 DEBUG:boto:Method: GET 2022-09-02T04:42:04.049 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.050 DEBUG:boto:Data: 2022-09-02T04:42:04.050 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.052 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.052 DEBUG:boto:Params: {} 2022-09-02T04:42:04.053 DEBUG:boto:Token: None 2022-09-02T04:42:04.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a842f352e601df1-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.058 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.060 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.060 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.061 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.061 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.062 DEBUG:boto:Method: GET 2022-09-02T04:42:04.063 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.063 DEBUG:boto:Data: 2022-09-02T04:42:04.064 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.065 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.065 DEBUG:boto:Params: {} 2022-09-02T04:42:04.066 DEBUG:boto:Token: None 2022-09-02T04:42:04.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk= 2022-09-02T04:42:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk=', 'Content-Length': '0'} 2022-09-02T04:42:04.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bf5119f5cde11b5-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.071 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.072 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.072 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.073 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.074 DEBUG:boto:Method: GET 2022-09-02T04:42:04.074 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.075 DEBUG:boto:Data: 2022-09-02T04:42:04.075 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.077 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.077 DEBUG:boto:Params: {} 2022-09-02T04:42:04.078 DEBUG:boto:Token: None 2022-09-02T04:42:04.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk= 2022-09-02T04:42:04.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk=', 'Content-Length': '0'} 2022-09-02T04:42:04.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcd4351eac8906d7-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.083 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.083 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.084 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.084 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.085 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.085 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.086 DEBUG:boto:Data: 2022-09-02T04:42:04.086 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.088 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.088 DEBUG:boto:Params: {} 2022-09-02T04:42:04.089 DEBUG:boto:Token: None 2022-09-02T04:42:04.089 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:11gHjykECiebOlaWXXtyBUSPkEo= 2022-09-02T04:42:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:11gHjykECiebOlaWXXtyBUSPkEo=', 'Content-Length': '0'} 2022-09-02T04:42:04.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7abb883d7126047-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.093 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.094 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.094 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.095 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.095 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.096 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.096 DEBUG:boto:Data: 2022-09-02T04:42:04.097 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.098 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.098 DEBUG:boto:Params: {} 2022-09-02T04:42:04.099 DEBUG:boto:Token: None 2022-09-02T04:42:04.099 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:11gHjykECiebOlaWXXtyBUSPkEo= 2022-09-02T04:42:04.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:11gHjykECiebOlaWXXtyBUSPkEo=', 'Content-Length': '0'} 2022-09-02T04:42:04.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027124b92a7f67b38-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.104 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.105 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.105 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.106 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.106 DEBUG:boto:Method: GET 2022-09-02T04:42:04.107 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.107 DEBUG:boto:Data: 2022-09-02T04:42:04.108 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.109 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.110 DEBUG:boto:Params: {} 2022-09-02T04:42:04.111 DEBUG:boto:Token: None 2022-09-02T04:42:04.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk= 2022-09-02T04:42:04.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk=', 'Content-Length': '0'} 2022-09-02T04:42:04.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dd1b8fe904b6df4-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.158 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.159 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.159 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.160 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.160 DEBUG:boto:Method: GET 2022-09-02T04:42:04.161 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.161 DEBUG:boto:Data: 2022-09-02T04:42:04.162 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.163 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.163 DEBUG:boto:Params: {} 2022-09-02T04:42:04.164 DEBUG:boto:Token: None 2022-09-02T04:42:04.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:04.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk= 2022-09-02T04:42:04.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iG54umdMJa/znDOG2EyUNsAKatk=', 'Content-Length': '0'} 2022-09-02T04:42:04.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dc4cebdd0b063b3-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.178 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.179 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.179 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.180 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.180 DEBUG:boto:Method: GET 2022-09-02T04:42:04.181 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.182 DEBUG:boto:Data: 2022-09-02T04:42:04.182 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.183 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.184 DEBUG:boto:Params: {} 2022-09-02T04:42:04.184 DEBUG:boto:Token: None 2022-09-02T04:42:04.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b8ae1cde7b40a39-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.190 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.191 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.192 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.192 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.193 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.193 DEBUG:boto:Method: GET 2022-09-02T04:42:04.195 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.195 DEBUG:boto:Data: 2022-09-02T04:42:04.196 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.197 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.198 DEBUG:boto:Params: {} 2022-09-02T04:42:04.199 DEBUG:boto:Token: None 2022-09-02T04:42:04.199 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd4143685a711ae-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.204 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.205 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.205 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.206 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.206 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.207 DEBUG:boto:Data: 2022-09-02T04:42:04.207 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.208 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.209 DEBUG:boto:Params: {} 2022-09-02T04:42:04.210 DEBUG:boto:Token: None 2022-09-02T04:42:04.210 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b7ef56f0a4c7597-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.216 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.216 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.217 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.217 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.218 DEBUG:boto:Data: 2022-09-02T04:42:04.218 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.220 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.220 DEBUG:boto:Params: {} 2022-09-02T04:42:04.221 DEBUG:boto:Token: None 2022-09-02T04:42:04.222 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab1e7b4a8fe97fbf-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.228 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.229 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.229 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.230 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.231 DEBUG:boto:Method: GET 2022-09-02T04:42:04.231 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.232 DEBUG:boto:Data: 2022-09-02T04:42:04.232 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.233 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.234 DEBUG:boto:Params: {} 2022-09-02T04:42:04.235 DEBUG:boto:Token: None 2022-09-02T04:42:04.235 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003599112cd209b848-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.240 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.240 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.241 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.241 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.242 DEBUG:boto:Method: GET 2022-09-02T04:42:04.242 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.243 DEBUG:boto:Data: 2022-09-02T04:42:04.243 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.245 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.245 DEBUG:boto:Params: {} 2022-09-02T04:42:04.246 DEBUG:boto:Token: None 2022-09-02T04:42:04.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000127b21a6d9d083e1-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.251 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.251 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.252 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.252 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.253 DEBUG:boto:Method: GET 2022-09-02T04:42:04.254 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.254 DEBUG:boto:Data: 2022-09-02T04:42:04.255 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.256 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.256 DEBUG:boto:Params: {} 2022-09-02T04:42:04.257 DEBUG:boto:Token: None 2022-09-02T04:42:04.258 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff53a96a1de45052-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.310 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.311 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.312 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.313 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.313 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.314 DEBUG:boto:Method: GET 2022-09-02T04:42:04.315 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.315 DEBUG:boto:Data: 2022-09-02T04:42:04.316 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.317 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.318 DEBUG:boto:Params: {} 2022-09-02T04:42:04.318 DEBUG:boto:Token: None 2022-09-02T04:42:04.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d38c2c8fe168f51-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.324 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.325 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.326 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.326 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.327 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.327 DEBUG:boto:Method: GET 2022-09-02T04:42:04.328 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.328 DEBUG:boto:Data: 2022-09-02T04:42:04.329 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.330 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.331 DEBUG:boto:Params: {} 2022-09-02T04:42:04.331 DEBUG:boto:Token: None 2022-09-02T04:42:04.332 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY= 2022-09-02T04:42:04.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY=', 'Content-Length': '0'} 2022-09-02T04:42:04.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cdc50dcf512d2e9-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.336 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.337 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.337 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.338 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.338 DEBUG:boto:Method: GET 2022-09-02T04:42:04.339 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.339 DEBUG:boto:Data: 2022-09-02T04:42:04.340 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.341 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.342 DEBUG:boto:Params: {} 2022-09-02T04:42:04.342 DEBUG:boto:Token: None 2022-09-02T04:42:04.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY= 2022-09-02T04:42:04.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY=', 'Content-Length': '0'} 2022-09-02T04:42:04.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005650399e00ade0cd-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.391 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.392 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.393 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.393 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.394 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.394 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.395 DEBUG:boto:Data: 2022-09-02T04:42:04.395 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.396 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.397 DEBUG:boto:Params: {} 2022-09-02T04:42:04.398 DEBUG:boto:Token: None 2022-09-02T04:42:04.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZVOMI//RPUbIi5AXYExVfh1mfc= 2022-09-02T04:42:04.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZVOMI//RPUbIi5AXYExVfh1mfc=', 'Content-Length': '0'} 2022-09-02T04:42:04.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dfa12f7a200d146-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.405 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.406 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.407 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.408 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.409 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.409 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.410 DEBUG:boto:Data: 2022-09-02T04:42:04.411 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.413 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.414 DEBUG:boto:Params: {} 2022-09-02T04:42:04.415 DEBUG:boto:Token: None 2022-09-02T04:42:04.415 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZVOMI//RPUbIi5AXYExVfh1mfc= 2022-09-02T04:42:04.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZVOMI//RPUbIi5AXYExVfh1mfc=', 'Content-Length': '0'} 2022-09-02T04:42:04.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048115cf7379894ea-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.426 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.426 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.427 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.427 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.428 DEBUG:boto:Method: GET 2022-09-02T04:42:04.429 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.429 DEBUG:boto:Data: 2022-09-02T04:42:04.430 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.431 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.432 DEBUG:boto:Params: {} 2022-09-02T04:42:04.432 DEBUG:boto:Token: None 2022-09-02T04:42:04.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY= 2022-09-02T04:42:04.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY=', 'Content-Length': '0'} 2022-09-02T04:42:04.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000488952cd3d122741-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.438 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.439 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.439 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.440 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.440 DEBUG:boto:Method: GET 2022-09-02T04:42:04.441 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.441 DEBUG:boto:Data: 2022-09-02T04:42:04.442 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.443 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.444 DEBUG:boto:Params: {} 2022-09-02T04:42:04.445 DEBUG:boto:Token: None 2022-09-02T04:42:04.446 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:04.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY= 2022-09-02T04:42:04.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W/YPbYMsFzbs2zZX5IFg8QMa7vY=', 'Content-Length': '0'} 2022-09-02T04:42:04.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099f8d320d5f95404-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.450 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.450 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.451 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.451 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.452 DEBUG:boto:Method: GET 2022-09-02T04:42:04.453 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.453 DEBUG:boto:Data: 2022-09-02T04:42:04.454 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.455 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.457 DEBUG:boto:Params: {} 2022-09-02T04:42:04.458 DEBUG:boto:Token: None 2022-09-02T04:42:04.458 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019acec1368ca941f-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.468 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.470 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.471 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.471 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.472 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.474 DEBUG:boto:Method: GET 2022-09-02T04:42:04.474 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.475 DEBUG:boto:Data: 2022-09-02T04:42:04.477 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.479 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.480 DEBUG:boto:Params: {} 2022-09-02T04:42:04.482 DEBUG:boto:Token: None 2022-09-02T04:42:04.483 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b3a224dde5c9498-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.489 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.490 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.491 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.491 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.492 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.493 DEBUG:boto:Data: 2022-09-02T04:42:04.493 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.494 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.495 DEBUG:boto:Params: {} 2022-09-02T04:42:04.495 DEBUG:boto:Token: None 2022-09-02T04:42:04.496 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de697730c27dfbca-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.502 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.503 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.504 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.504 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.505 DEBUG:boto:Data: 2022-09-02T04:42:04.505 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.507 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.507 DEBUG:boto:Params: {} 2022-09-02T04:42:04.508 DEBUG:boto:Token: None 2022-09-02T04:42:04.509 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d0d8fb736778587-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.517 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.518 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.520 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.521 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.522 DEBUG:boto:Method: GET 2022-09-02T04:42:04.523 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.524 DEBUG:boto:Data: 2022-09-02T04:42:04.525 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.527 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.528 DEBUG:boto:Params: {} 2022-09-02T04:42:04.529 DEBUG:boto:Token: None 2022-09-02T04:42:04.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cebb007f73559a0-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.582 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.583 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.583 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.584 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.584 DEBUG:boto:Method: GET 2022-09-02T04:42:04.585 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.586 DEBUG:boto:Data: 2022-09-02T04:42:04.590 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.591 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.592 DEBUG:boto:Params: {} 2022-09-02T04:42:04.592 DEBUG:boto:Token: None 2022-09-02T04:42:04.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efb2e40d546c9c68-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.598 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.599 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.599 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.600 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.600 DEBUG:boto:Method: GET 2022-09-02T04:42:04.601 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.601 DEBUG:boto:Data: 2022-09-02T04:42:04.602 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.603 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.604 DEBUG:boto:Params: {} 2022-09-02T04:42:04.604 DEBUG:boto:Token: None 2022-09-02T04:42:04.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a1880a914f0e79-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.611 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.611 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.612 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.613 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.613 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.614 DEBUG:boto:Method: GET 2022-09-02T04:42:04.614 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.615 DEBUG:boto:Data: 2022-09-02T04:42:04.615 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.617 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.617 DEBUG:boto:Params: {} 2022-09-02T04:42:04.618 DEBUG:boto:Token: None 2022-09-02T04:42:04.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a33959ca878948a6-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.624 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.625 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.626 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.626 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.627 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.627 DEBUG:boto:Method: GET 2022-09-02T04:42:04.628 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.629 DEBUG:boto:Data: 2022-09-02T04:42:04.629 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.630 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.631 DEBUG:boto:Params: {} 2022-09-02T04:42:04.631 DEBUG:boto:Token: None 2022-09-02T04:42:04.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4= 2022-09-02T04:42:04.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4=', 'Content-Length': '0'} 2022-09-02T04:42:04.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009287d4465ce5d1b9-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.636 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.637 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.637 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.638 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.638 DEBUG:boto:Method: GET 2022-09-02T04:42:04.639 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.640 DEBUG:boto:Data: 2022-09-02T04:42:04.640 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.641 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.642 DEBUG:boto:Params: {} 2022-09-02T04:42:04.642 DEBUG:boto:Token: None 2022-09-02T04:42:04.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4= 2022-09-02T04:42:04.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4=', 'Content-Length': '0'} 2022-09-02T04:42:04.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089ea4045bc9c7d98-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.648 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.648 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.649 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.649 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.650 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.650 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.651 DEBUG:boto:Data: 2022-09-02T04:42:04.651 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.652 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.653 DEBUG:boto:Params: {} 2022-09-02T04:42:04.654 DEBUG:boto:Token: None 2022-09-02T04:42:04.655 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gp1J4yFxlLYhdiki6ReD1x1JWGg= 2022-09-02T04:42:04.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gp1J4yFxlLYhdiki6ReD1x1JWGg=', 'Content-Length': '0'} 2022-09-02T04:42:04.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e25af525f9d3c1a-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.661 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.662 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.662 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.663 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.664 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.665 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.666 DEBUG:boto:Data: 2022-09-02T04:42:04.667 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.669 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.670 DEBUG:boto:Params: {} 2022-09-02T04:42:04.671 DEBUG:boto:Token: None 2022-09-02T04:42:04.671 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gp1J4yFxlLYhdiki6ReD1x1JWGg= 2022-09-02T04:42:04.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gp1J4yFxlLYhdiki6ReD1x1JWGg=', 'Content-Length': '0'} 2022-09-02T04:42:04.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023d4802f1905c2a2-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.677 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.678 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.679 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.679 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.684 DEBUG:boto:Method: GET 2022-09-02T04:42:04.685 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.686 DEBUG:boto:Data: 2022-09-02T04:42:04.687 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.689 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.690 DEBUG:boto:Params: {} 2022-09-02T04:42:04.691 DEBUG:boto:Token: None 2022-09-02T04:42:04.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4= 2022-09-02T04:42:04.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4=', 'Content-Length': '0'} 2022-09-02T04:42:04.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000102ec3d11bb8dfa9-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.699 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.699 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.700 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.700 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.701 DEBUG:boto:Method: GET 2022-09-02T04:42:04.702 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.702 DEBUG:boto:Data: 2022-09-02T04:42:04.703 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.704 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.704 DEBUG:boto:Params: {} 2022-09-02T04:42:04.705 DEBUG:boto:Token: None 2022-09-02T04:42:04.706 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:04.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4= 2022-09-02T04:42:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Zn24eYzfh+Yn3f1IjIGljQFyJ4=', 'Content-Length': '0'} 2022-09-02T04:42:04.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000210a9b4d7a89f071-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.750 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.751 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.752 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.752 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.753 DEBUG:boto:Method: GET 2022-09-02T04:42:04.754 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.754 DEBUG:boto:Data: 2022-09-02T04:42:04.755 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.756 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.756 DEBUG:boto:Params: {} 2022-09-02T04:42:04.757 DEBUG:boto:Token: None 2022-09-02T04:42:04.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003834a86fb16abd53-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.764 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.764 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.765 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.765 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.766 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.767 DEBUG:boto:Method: GET 2022-09-02T04:42:04.767 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.768 DEBUG:boto:Data: 2022-09-02T04:42:04.768 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.769 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.770 DEBUG:boto:Params: {} 2022-09-02T04:42:04.771 DEBUG:boto:Token: None 2022-09-02T04:42:04.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f3984fc95b5051-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.776 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.777 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.778 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.779 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.779 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.780 DEBUG:boto:Data: 2022-09-02T04:42:04.781 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.782 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.782 DEBUG:boto:Params: {} 2022-09-02T04:42:04.783 DEBUG:boto:Token: None 2022-09-02T04:42:04.783 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000098ea6a0c2809758-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.789 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.790 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.790 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.791 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:04.791 DEBUG:boto:Data: 2022-09-02T04:42:04.792 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.793 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.793 DEBUG:boto:Params: {} 2022-09-02T04:42:04.794 DEBUG:boto:Token: None 2022-09-02T04:42:04.795 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj 2022-09-02T04:42:04.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0= 2022-09-02T04:42:04.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4llHmfDSQ4RL7gXzgemEA/xkVP0=', 'Content-Length': '0'} 2022-09-02T04:42:04.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5e8f2b9298d8bd8-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.803 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.803 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.804 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.805 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.805 DEBUG:boto:Method: GET 2022-09-02T04:42:04.806 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.806 DEBUG:boto:Data: 2022-09-02T04:42:04.807 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.808 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.809 DEBUG:boto:Params: {} 2022-09-02T04:42:04.810 DEBUG:boto:Token: None 2022-09-02T04:42:04.810 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c086effd33d32c1a-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.859 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.860 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.860 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.861 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.861 DEBUG:boto:Method: GET 2022-09-02T04:42:04.862 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.862 DEBUG:boto:Data: 2022-09-02T04:42:04.863 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.864 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.865 DEBUG:boto:Params: {} 2022-09-02T04:42:04.865 DEBUG:boto:Token: None 2022-09-02T04:42:04.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:04.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE= 2022-09-02T04:42:04.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WSwEcJPYemiQlbnKkFSx5JgTMuE=', 'Content-Length': '0'} 2022-09-02T04:42:04.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b473dd1773ac2bf-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.870 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.871 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.871 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.872 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.873 DEBUG:boto:Method: GET 2022-09-02T04:42:04.873 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.874 DEBUG:boto:Data: 2022-09-02T04:42:04.874 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.875 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.876 DEBUG:boto:Params: {} 2022-09-02T04:42:04.876 DEBUG:boto:Token: None 2022-09-02T04:42:04.877 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2bbfa175b541c69-006311899c-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.882 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.883 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:04.883 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:04.884 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.884 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:04.885 DEBUG:boto:Method: GET 2022-09-02T04:42:04.886 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:04.886 DEBUG:boto:Data: 2022-09-02T04:42:04.887 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.888 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.888 DEBUG:boto:Params: {} 2022-09-02T04:42:04.889 DEBUG:boto:Token: None 2022-09-02T04:42:04.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/ 2022-09-02T04:42:04.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q= 2022-09-02T04:42:04.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nM05wv5RiUqtl52gNsJPagLPn+Q=', 'Content-Length': '0'} 2022-09-02T04:42:04.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c431ed0ec2c04b6c-006311899c-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.895 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:04.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:04.896 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.897 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.897 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.898 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.899 DEBUG:boto:Method: GET 2022-09-02T04:42:04.899 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.900 DEBUG:boto:Data: 2022-09-02T04:42:04.900 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.901 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.902 DEBUG:boto:Params: {} 2022-09-02T04:42:04.902 DEBUG:boto:Token: None 2022-09-02T04:42:04.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cxoYkxVb5c37md/aePm50iWFqVQ= 2022-09-02T04:42:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cxoYkxVb5c37md/aePm50iWFqVQ=', 'Content-Length': '0'} 2022-09-02T04:42:04.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f457741161d6c59-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.908 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.909 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.909 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.910 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.911 DEBUG:boto:Method: GET 2022-09-02T04:42:04.912 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.913 DEBUG:boto:Data: 2022-09-02T04:42:04.913 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.914 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.915 DEBUG:boto:Params: {} 2022-09-02T04:42:04.915 DEBUG:boto:Token: None 2022-09-02T04:42:04.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cxoYkxVb5c37md/aePm50iWFqVQ= 2022-09-02T04:42:04.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cxoYkxVb5c37md/aePm50iWFqVQ=', 'Content-Length': '0'} 2022-09-02T04:42:04.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000073beb8840d8f661-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.966 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.967 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.968 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.969 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.970 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.971 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.971 DEBUG:boto:Data: 2022-09-02T04:42:04.972 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:04.973 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.974 DEBUG:boto:Params: {} 2022-09-02T04:42:04.974 DEBUG:boto:Token: None 2022-09-02T04:42:04.975 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aY5qMslMUHMQUtbouGoOS5Lzmtw= 2022-09-02T04:42:04.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aY5qMslMUHMQUtbouGoOS5Lzmtw=', 'Content-Length': '0'} 2022-09-02T04:42:04.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b1e94d2fd00620d-006311899c-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.979 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:04.980 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:04.980 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.981 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.981 DEBUG:boto:Method: HEAD 2022-09-02T04:42:04.982 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.983 DEBUG:boto:Data: 2022-09-02T04:42:04.983 DEBUG:boto:Headers: {} 2022-09-02T04:42:04.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:04.984 DEBUG:boto:Port: 8000 2022-09-02T04:42:04.985 DEBUG:boto:Params: {} 2022-09-02T04:42:04.985 DEBUG:boto:Token: None 2022-09-02T04:42:04.986 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:04 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:04.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aY5qMslMUHMQUtbouGoOS5Lzmtw= 2022-09-02T04:42:04.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aY5qMslMUHMQUtbouGoOS5Lzmtw=', 'Content-Length': '0'} 2022-09-02T04:42:04.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000252c6a60794a204-006311899c-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:04.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.000 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.001 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.002 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.003 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.004 DEBUG:boto:Method: GET 2022-09-02T04:42:05.005 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.005 DEBUG:boto:Data: 2022-09-02T04:42:05.006 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.008 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.009 DEBUG:boto:Params: {} 2022-09-02T04:42:05.010 DEBUG:boto:Token: None 2022-09-02T04:42:05.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jlpdAg9FAR7rFsi1nMkFmFXwLVM= 2022-09-02T04:42:05.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jlpdAg9FAR7rFsi1nMkFmFXwLVM=', 'Content-Length': '0'} 2022-09-02T04:42:05.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009dcd553dafff20c-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.019 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.020 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.021 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.022 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.023 DEBUG:boto:Method: GET 2022-09-02T04:42:05.024 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.025 DEBUG:boto:Data: 2022-09-02T04:42:05.026 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.027 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.028 DEBUG:boto:Params: {} 2022-09-02T04:42:05.030 DEBUG:boto:Token: None 2022-09-02T04:42:05.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:05.032 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jlpdAg9FAR7rFsi1nMkFmFXwLVM= 2022-09-02T04:42:05.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jlpdAg9FAR7rFsi1nMkFmFXwLVM=', 'Content-Length': '0'} 2022-09-02T04:42:05.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1370cfd8592d074-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.037 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.038 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.039 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.040 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.041 DEBUG:boto:Method: GET 2022-09-02T04:42:05.042 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.043 DEBUG:boto:Data: 2022-09-02T04:42:05.044 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.045 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.046 DEBUG:boto:Params: {} 2022-09-02T04:42:05.047 DEBUG:boto:Token: None 2022-09-02T04:42:05.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000430bfd0b83f55331-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.055 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.057 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.058 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.059 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.060 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.061 DEBUG:boto:Method: GET 2022-09-02T04:42:05.062 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.063 DEBUG:boto:Data: 2022-09-02T04:42:05.064 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.067 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.068 DEBUG:boto:Params: {} 2022-09-02T04:42:05.069 DEBUG:boto:Token: None 2022-09-02T04:42:05.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bdca03093bfcfce-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.078 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.079 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.080 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.082 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.083 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.083 DEBUG:boto:Data: 2022-09-02T04:42:05.084 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.086 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.088 DEBUG:boto:Params: {} 2022-09-02T04:42:05.089 DEBUG:boto:Token: None 2022-09-02T04:42:05.089 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edb4267be5c859cd-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.100 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.101 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.102 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.103 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.104 DEBUG:boto:Data: 2022-09-02T04:42:05.105 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.107 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.108 DEBUG:boto:Params: {} 2022-09-02T04:42:05.109 DEBUG:boto:Token: None 2022-09-02T04:42:05.110 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052851fef79d424f8-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.119 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.120 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.121 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.122 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.123 DEBUG:boto:Method: GET 2022-09-02T04:42:05.124 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.125 DEBUG:boto:Data: 2022-09-02T04:42:05.126 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.128 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.129 DEBUG:boto:Params: {} 2022-09-02T04:42:05.131 DEBUG:boto:Token: None 2022-09-02T04:42:05.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0= 2022-09-02T04:42:05.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0=', 'Content-Length': '0'} 2022-09-02T04:42:05.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000908fc42ddaf49f1f-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.138 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.139 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.140 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.141 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.142 DEBUG:boto:Method: GET 2022-09-02T04:42:05.143 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.143 DEBUG:boto:Data: 2022-09-02T04:42:05.144 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.145 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.146 DEBUG:boto:Params: {} 2022-09-02T04:42:05.146 DEBUG:boto:Token: None 2022-09-02T04:42:05.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0= 2022-09-02T04:42:05.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0=', 'Content-Length': '0'} 2022-09-02T04:42:05.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060a4d9ddcb24fde6-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.152 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.152 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.153 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.153 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.154 DEBUG:boto:Method: GET 2022-09-02T04:42:05.155 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.156 DEBUG:boto:Data: 2022-09-02T04:42:05.157 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.159 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.160 DEBUG:boto:Params: {} 2022-09-02T04:42:05.161 DEBUG:boto:Token: None 2022-09-02T04:42:05.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046476d8cddb5decb-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.169 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.170 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.171 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.171 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.172 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.172 DEBUG:boto:Method: GET 2022-09-02T04:42:05.173 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.173 DEBUG:boto:Data: 2022-09-02T04:42:05.174 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.176 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.177 DEBUG:boto:Params: {} 2022-09-02T04:42:05.177 DEBUG:boto:Token: None 2022-09-02T04:42:05.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b3e2866aed11d8-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.184 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.186 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.186 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.187 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.187 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.188 DEBUG:boto:Method: GET 2022-09-02T04:42:05.189 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.189 DEBUG:boto:Data: 2022-09-02T04:42:05.190 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.191 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.192 DEBUG:boto:Params: {} 2022-09-02T04:42:05.193 DEBUG:boto:Token: None 2022-09-02T04:42:05.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI= 2022-09-02T04:42:05.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI=', 'Content-Length': '0'} 2022-09-02T04:42:05.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d8e28e68a07d0ce-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.198 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.199 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.200 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.201 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.202 DEBUG:boto:Method: GET 2022-09-02T04:42:05.203 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.204 DEBUG:boto:Data: 2022-09-02T04:42:05.205 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.207 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.207 DEBUG:boto:Params: {} 2022-09-02T04:42:05.208 DEBUG:boto:Token: None 2022-09-02T04:42:05.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI= 2022-09-02T04:42:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI=', 'Content-Length': '0'} 2022-09-02T04:42:05.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e034de4ddd21a39-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.262 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.263 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.263 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.264 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.264 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.265 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.265 DEBUG:boto:Data: 2022-09-02T04:42:05.266 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.267 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.268 DEBUG:boto:Params: {} 2022-09-02T04:42:05.269 DEBUG:boto:Token: None 2022-09-02T04:42:05.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v2seF0FePwDPQ6BarXNFBmwGjoY= 2022-09-02T04:42:05.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v2seF0FePwDPQ6BarXNFBmwGjoY=', 'Content-Length': '0'} 2022-09-02T04:42:05.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000331a040b9597c589-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.275 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.275 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.276 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.276 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.277 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.277 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.278 DEBUG:boto:Data: 2022-09-02T04:42:05.278 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.279 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.280 DEBUG:boto:Params: {} 2022-09-02T04:42:05.280 DEBUG:boto:Token: None 2022-09-02T04:42:05.281 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v2seF0FePwDPQ6BarXNFBmwGjoY= 2022-09-02T04:42:05.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v2seF0FePwDPQ6BarXNFBmwGjoY=', 'Content-Length': '0'} 2022-09-02T04:42:05.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f019f2e0aee9976-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.286 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.286 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.287 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.287 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.288 DEBUG:boto:Method: GET 2022-09-02T04:42:05.288 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.289 DEBUG:boto:Data: 2022-09-02T04:42:05.289 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.291 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.291 DEBUG:boto:Params: {} 2022-09-02T04:42:05.292 DEBUG:boto:Token: None 2022-09-02T04:42:05.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI= 2022-09-02T04:42:05.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI=', 'Content-Length': '0'} 2022-09-02T04:42:05.296 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070abafe16ca2f77f-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.338 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.339 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.340 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.340 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.341 DEBUG:boto:Method: GET 2022-09-02T04:42:05.341 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.342 DEBUG:boto:Data: 2022-09-02T04:42:05.343 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.344 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.344 DEBUG:boto:Params: {} 2022-09-02T04:42:05.345 DEBUG:boto:Token: None 2022-09-02T04:42:05.346 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:05.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI= 2022-09-02T04:42:05.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UgodeokmQYuOCvUbOPiIpyu2FyI=', 'Content-Length': '0'} 2022-09-02T04:42:05.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd6f05dd921c2244-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.351 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.351 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.352 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.352 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.353 DEBUG:boto:Method: GET 2022-09-02T04:42:05.353 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.354 DEBUG:boto:Data: 2022-09-02T04:42:05.354 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.355 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.356 DEBUG:boto:Params: {} 2022-09-02T04:42:05.357 DEBUG:boto:Token: None 2022-09-02T04:42:05.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ead306c9def4c6-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.362 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.363 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.364 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.364 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.365 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.366 DEBUG:boto:Method: GET 2022-09-02T04:42:05.367 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.368 DEBUG:boto:Data: 2022-09-02T04:42:05.368 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.370 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.370 DEBUG:boto:Params: {} 2022-09-02T04:42:05.371 DEBUG:boto:Token: None 2022-09-02T04:42:05.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca03adb247276af-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.380 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.381 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.382 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.382 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.383 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.384 DEBUG:boto:Data: 2022-09-02T04:42:05.384 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.385 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.386 DEBUG:boto:Params: {} 2022-09-02T04:42:05.386 DEBUG:boto:Token: None 2022-09-02T04:42:05.387 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000664cfc797774d8cc-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.393 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.393 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.394 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.394 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.395 DEBUG:boto:Data: 2022-09-02T04:42:05.395 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.396 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.397 DEBUG:boto:Params: {} 2022-09-02T04:42:05.397 DEBUG:boto:Token: None 2022-09-02T04:42:05.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038200a6e9e9a4c4e-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.405 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.406 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.408 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.409 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.410 DEBUG:boto:Method: GET 2022-09-02T04:42:05.410 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.411 DEBUG:boto:Data: 2022-09-02T04:42:05.412 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.414 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.416 DEBUG:boto:Params: {} 2022-09-02T04:42:05.417 DEBUG:boto:Token: None 2022-09-02T04:42:05.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0= 2022-09-02T04:42:05.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0=', 'Content-Length': '0'} 2022-09-02T04:42:05.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a71597f559030464-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.423 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.424 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.424 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.425 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.426 DEBUG:boto:Method: GET 2022-09-02T04:42:05.426 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.427 DEBUG:boto:Data: 2022-09-02T04:42:05.427 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.428 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.429 DEBUG:boto:Params: {} 2022-09-02T04:42:05.430 DEBUG:boto:Token: None 2022-09-02T04:42:05.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0= 2022-09-02T04:42:05.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0=', 'Content-Length': '0'} 2022-09-02T04:42:05.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000459d7797b3b8664b-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.474 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.476 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.477 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.477 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.479 DEBUG:boto:Method: GET 2022-09-02T04:42:05.479 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.481 DEBUG:boto:Data: 2022-09-02T04:42:05.482 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.484 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.485 DEBUG:boto:Params: {} 2022-09-02T04:42:05.486 DEBUG:boto:Token: None 2022-09-02T04:42:05.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dda1c60dd92895e-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.494 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.495 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.496 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.497 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.498 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.499 DEBUG:boto:Method: GET 2022-09-02T04:42:05.500 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.501 DEBUG:boto:Data: 2022-09-02T04:42:05.502 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.504 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.505 DEBUG:boto:Params: {} 2022-09-02T04:42:05.507 DEBUG:boto:Token: None 2022-09-02T04:42:05.508 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e63e1898d48ce3e2-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.515 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.518 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.519 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.520 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.521 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.522 DEBUG:boto:Method: GET 2022-09-02T04:42:05.523 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.524 DEBUG:boto:Data: 2022-09-02T04:42:05.525 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.526 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.526 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.527 DEBUG:boto:Params: {} 2022-09-02T04:42:05.529 DEBUG:boto:Token: None 2022-09-02T04:42:05.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw= 2022-09-02T04:42:05.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw=', 'Content-Length': '0'} 2022-09-02T04:42:05.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f312bde3da9a5b5f-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.536 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.537 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.537 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.538 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.539 DEBUG:boto:Method: GET 2022-09-02T04:42:05.539 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.540 DEBUG:boto:Data: 2022-09-02T04:42:05.540 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.542 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.542 DEBUG:boto:Params: {} 2022-09-02T04:42:05.543 DEBUG:boto:Token: None 2022-09-02T04:42:05.544 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw= 2022-09-02T04:42:05.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw=', 'Content-Length': '0'} 2022-09-02T04:42:05.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daa935694b41ddc6-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.549 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.549 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.550 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.550 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.551 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.551 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.552 DEBUG:boto:Data: 2022-09-02T04:42:05.553 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.554 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.555 DEBUG:boto:Params: {} 2022-09-02T04:42:05.556 DEBUG:boto:Token: None 2022-09-02T04:42:05.557 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W0qZznYVKs2Rpxpw8x3+9X8Mlzk= 2022-09-02T04:42:05.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W0qZznYVKs2Rpxpw8x3+9X8Mlzk=', 'Content-Length': '0'} 2022-09-02T04:42:05.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009df098bd45b78cd3-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.563 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.564 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.564 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.565 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.566 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.567 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.568 DEBUG:boto:Data: 2022-09-02T04:42:05.568 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.569 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.570 DEBUG:boto:Params: {} 2022-09-02T04:42:05.571 DEBUG:boto:Token: None 2022-09-02T04:42:05.571 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W0qZznYVKs2Rpxpw8x3+9X8Mlzk= 2022-09-02T04:42:05.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W0qZznYVKs2Rpxpw8x3+9X8Mlzk=', 'Content-Length': '0'} 2022-09-02T04:42:05.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abddb1f7a029d585-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.576 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.577 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.577 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.578 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.579 DEBUG:boto:Method: GET 2022-09-02T04:42:05.579 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.580 DEBUG:boto:Data: 2022-09-02T04:42:05.580 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.581 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.581 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.582 DEBUG:boto:Params: {} 2022-09-02T04:42:05.583 DEBUG:boto:Token: None 2022-09-02T04:42:05.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw= 2022-09-02T04:42:05.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw=', 'Content-Length': '0'} 2022-09-02T04:42:05.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a2432cf74772a88-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.630 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.631 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.632 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.632 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.633 DEBUG:boto:Method: GET 2022-09-02T04:42:05.633 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.634 DEBUG:boto:Data: 2022-09-02T04:42:05.635 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.636 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.636 DEBUG:boto:Params: {} 2022-09-02T04:42:05.637 DEBUG:boto:Token: None 2022-09-02T04:42:05.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:05.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw= 2022-09-02T04:42:05.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUEBm15JBM6prgbt58V2GJlouKw=', 'Content-Length': '0'} 2022-09-02T04:42:05.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e1bd7a8f3683262-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.644 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.645 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.645 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.646 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.647 DEBUG:boto:Method: GET 2022-09-02T04:42:05.647 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.648 DEBUG:boto:Data: 2022-09-02T04:42:05.648 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.650 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.651 DEBUG:boto:Params: {} 2022-09-02T04:42:05.651 DEBUG:boto:Token: None 2022-09-02T04:42:05.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5fc1037175fab6-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.658 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.659 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.660 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.661 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.662 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.662 DEBUG:boto:Method: GET 2022-09-02T04:42:05.663 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.663 DEBUG:boto:Data: 2022-09-02T04:42:05.664 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.665 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.666 DEBUG:boto:Params: {} 2022-09-02T04:42:05.667 DEBUG:boto:Token: None 2022-09-02T04:42:05.667 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097122b35f75f861c-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.672 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.673 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.673 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.674 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.674 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.675 DEBUG:boto:Data: 2022-09-02T04:42:05.675 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.676 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.677 DEBUG:boto:Params: {} 2022-09-02T04:42:05.677 DEBUG:boto:Token: None 2022-09-02T04:42:05.678 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c45c7de2172a633b-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.685 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.686 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.687 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.688 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.689 DEBUG:boto:Data: 2022-09-02T04:42:05.690 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.692 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.693 DEBUG:boto:Params: {} 2022-09-02T04:42:05.695 DEBUG:boto:Token: None 2022-09-02T04:42:05.696 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001b57affdc668fc1-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.703 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.704 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.704 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.705 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.705 DEBUG:boto:Method: GET 2022-09-02T04:42:05.706 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.706 DEBUG:boto:Data: 2022-09-02T04:42:05.707 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.708 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.709 DEBUG:boto:Params: {} 2022-09-02T04:42:05.709 DEBUG:boto:Token: None 2022-09-02T04:42:05.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0= 2022-09-02T04:42:05.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0=', 'Content-Length': '0'} 2022-09-02T04:42:05.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cc6093cc7fa58cd-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.762 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.763 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.764 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.765 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.766 DEBUG:boto:Method: GET 2022-09-02T04:42:05.767 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.767 DEBUG:boto:Data: 2022-09-02T04:42:05.768 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.770 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.771 DEBUG:boto:Params: {} 2022-09-02T04:42:05.772 DEBUG:boto:Token: None 2022-09-02T04:42:05.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:05.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0= 2022-09-02T04:42:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbS2/r0YPjJTJIocxg0bGspmMO0=', 'Content-Length': '0'} 2022-09-02T04:42:05.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000600431fee6cd59cb-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.779 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.780 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.781 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.782 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.783 DEBUG:boto:Method: GET 2022-09-02T04:42:05.784 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.785 DEBUG:boto:Data: 2022-09-02T04:42:05.786 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.787 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.788 DEBUG:boto:Params: {} 2022-09-02T04:42:05.789 DEBUG:boto:Token: None 2022-09-02T04:42:05.790 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb342dd3d08c8f16-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.795 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.796 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.797 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.797 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.798 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.798 DEBUG:boto:Method: GET 2022-09-02T04:42:05.799 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.800 DEBUG:boto:Data: 2022-09-02T04:42:05.800 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.801 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.802 DEBUG:boto:Params: {} 2022-09-02T04:42:05.802 DEBUG:boto:Token: None 2022-09-02T04:42:05.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d864663feeea3e-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.814 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.817 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.818 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.819 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.819 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.820 DEBUG:boto:Method: GET 2022-09-02T04:42:05.820 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.821 DEBUG:boto:Data: 2022-09-02T04:42:05.821 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.823 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.824 DEBUG:boto:Params: {} 2022-09-02T04:42:05.825 DEBUG:boto:Token: None 2022-09-02T04:42:05.827 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4= 2022-09-02T04:42:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4=', 'Content-Length': '0'} 2022-09-02T04:42:05.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fad2597fa49e326a-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.834 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.835 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.836 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.837 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.838 DEBUG:boto:Method: GET 2022-09-02T04:42:05.839 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.842 DEBUG:boto:Data: 2022-09-02T04:42:05.842 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.843 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.844 DEBUG:boto:Params: {} 2022-09-02T04:42:05.844 DEBUG:boto:Token: None 2022-09-02T04:42:05.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4= 2022-09-02T04:42:05.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4=', 'Content-Length': '0'} 2022-09-02T04:42:05.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099a3e0adc55a656f-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.894 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.895 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.896 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.896 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.897 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.897 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.898 DEBUG:boto:Data: 2022-09-02T04:42:05.898 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.900 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.901 DEBUG:boto:Params: {} 2022-09-02T04:42:05.901 DEBUG:boto:Token: None 2022-09-02T04:42:05.902 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gBw5J1QjcVd9bWruXGICVH53Kpo= 2022-09-02T04:42:05.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gBw5J1QjcVd9bWruXGICVH53Kpo=', 'Content-Length': '0'} 2022-09-02T04:42:05.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017c46674df33d6d9-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.907 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.907 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.908 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.908 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.909 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.909 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.910 DEBUG:boto:Data: 2022-09-02T04:42:05.910 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.911 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.912 DEBUG:boto:Params: {} 2022-09-02T04:42:05.912 DEBUG:boto:Token: None 2022-09-02T04:42:05.913 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gBw5J1QjcVd9bWruXGICVH53Kpo= 2022-09-02T04:42:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gBw5J1QjcVd9bWruXGICVH53Kpo=', 'Content-Length': '0'} 2022-09-02T04:42:05.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3d36450e773dc2e-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.918 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.919 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.919 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.920 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.921 DEBUG:boto:Method: GET 2022-09-02T04:42:05.921 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.922 DEBUG:boto:Data: 2022-09-02T04:42:05.922 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.923 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.924 DEBUG:boto:Params: {} 2022-09-02T04:42:05.925 DEBUG:boto:Token: None 2022-09-02T04:42:05.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4= 2022-09-02T04:42:05.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4=', 'Content-Length': '0'} 2022-09-02T04:42:05.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5ec9b7429f454f9-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.930 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.930 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.931 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.931 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.932 DEBUG:boto:Method: GET 2022-09-02T04:42:05.933 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.933 DEBUG:boto:Data: 2022-09-02T04:42:05.934 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.935 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.935 DEBUG:boto:Params: {} 2022-09-02T04:42:05.936 DEBUG:boto:Token: None 2022-09-02T04:42:05.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:05.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4= 2022-09-02T04:42:05.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QB2L/jaUIe3qcjfECvzW+2gcLa4=', 'Content-Length': '0'} 2022-09-02T04:42:05.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bda658bdf181b0a4-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.941 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.942 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.942 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.943 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.943 DEBUG:boto:Method: GET 2022-09-02T04:42:05.944 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.944 DEBUG:boto:Data: 2022-09-02T04:42:05.945 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.946 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.947 DEBUG:boto:Params: {} 2022-09-02T04:42:05.947 DEBUG:boto:Token: None 2022-09-02T04:42:05.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cb14bb6c500500b-006311899d-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.954 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.954 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:05.955 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:05.956 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.956 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:05.957 DEBUG:boto:Method: GET 2022-09-02T04:42:05.957 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:05.958 DEBUG:boto:Data: 2022-09-02T04:42:05.958 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.959 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.960 DEBUG:boto:Params: {} 2022-09-02T04:42:05.960 DEBUG:boto:Token: None 2022-09-02T04:42:05.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/ 2022-09-02T04:42:05.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow= 2022-09-02T04:42:05.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZwuchUtKshKzOgK6nqZlVdw4Ow=', 'Content-Length': '0'} 2022-09-02T04:42:05.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e63e04f04ad595-006311899d-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.967 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:05.968 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.969 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.970 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.971 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.973 DEBUG:boto:Data: 2022-09-02T04:42:05.974 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:05.975 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.976 DEBUG:boto:Params: {} 2022-09-02T04:42:05.977 DEBUG:boto:Token: None 2022-09-02T04:42:05.978 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f16e6fac4d155725-006311899d-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.986 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:05.986 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:05.987 DEBUG:boto:Method: HEAD 2022-09-02T04:42:05.987 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:05.988 DEBUG:boto:Data: 2022-09-02T04:42:05.988 DEBUG:boto:Headers: {} 2022-09-02T04:42:05.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:05.989 DEBUG:boto:Port: 8000 2022-09-02T04:42:05.990 DEBUG:boto:Params: {} 2022-09-02T04:42:05.990 DEBUG:boto:Token: None 2022-09-02T04:42:05.991 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:05 GMT /xljbkf-19/obj 2022-09-02T04:42:05.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms= 2022-09-02T04:42:05.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o0ejoum2A8KaxDjlEJxMZ1TYEms=', 'Content-Length': '0'} 2022-09-02T04:42:05.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d919e04aff5b36e-006311899d-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:05.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:05.999 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.000 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.000 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.001 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.002 DEBUG:boto:Method: GET 2022-09-02T04:42:06.002 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.003 DEBUG:boto:Data: 2022-09-02T04:42:06.004 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.005 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.006 DEBUG:boto:Params: {} 2022-09-02T04:42:06.007 DEBUG:boto:Token: None 2022-09-02T04:42:06.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.013 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd279a51916db2fb-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.058 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.059 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.060 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.060 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.061 DEBUG:boto:Method: GET 2022-09-02T04:42:06.061 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.062 DEBUG:boto:Data: 2022-09-02T04:42:06.063 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.064 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.064 DEBUG:boto:Params: {} 2022-09-02T04:42:06.065 DEBUG:boto:Token: None 2022-09-02T04:42:06.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071c3fb362c34fc8b-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.070 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.071 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.071 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.072 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.073 DEBUG:boto:Method: GET 2022-09-02T04:42:06.073 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.074 DEBUG:boto:Data: 2022-09-02T04:42:06.074 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.075 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.076 DEBUG:boto:Params: {} 2022-09-02T04:42:06.076 DEBUG:boto:Token: None 2022-09-02T04:42:06.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e832bf7716a8fdb5-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.082 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.083 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.083 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.084 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.085 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.085 DEBUG:boto:Method: GET 2022-09-02T04:42:06.086 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.086 DEBUG:boto:Data: 2022-09-02T04:42:06.087 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.088 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.089 DEBUG:boto:Params: {} 2022-09-02T04:42:06.089 DEBUG:boto:Token: None 2022-09-02T04:42:06.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089d286aa11c6440d-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.098 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.099 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.100 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.100 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.101 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.101 DEBUG:boto:Method: GET 2022-09-02T04:42:06.102 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.102 DEBUG:boto:Data: 2022-09-02T04:42:06.103 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.104 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.105 DEBUG:boto:Params: {} 2022-09-02T04:42:06.105 DEBUG:boto:Token: None 2022-09-02T04:42:06.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA= 2022-09-02T04:42:06.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA=', 'Content-Length': '0'} 2022-09-02T04:42:06.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000659975a2e58fd8ed-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.110 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.111 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.111 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.112 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.112 DEBUG:boto:Method: GET 2022-09-02T04:42:06.113 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.113 DEBUG:boto:Data: 2022-09-02T04:42:06.114 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.115 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.116 DEBUG:boto:Params: {} 2022-09-02T04:42:06.116 DEBUG:boto:Token: None 2022-09-02T04:42:06.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA= 2022-09-02T04:42:06.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA=', 'Content-Length': '0'} 2022-09-02T04:42:06.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038127708a840c06a-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.121 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.122 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.122 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.123 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.123 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.124 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.125 DEBUG:boto:Data: 2022-09-02T04:42:06.125 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.126 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.127 DEBUG:boto:Params: {} 2022-09-02T04:42:06.127 DEBUG:boto:Token: None 2022-09-02T04:42:06.128 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+ndwy0V3BcGPX7bXsVpzd/1/P0= 2022-09-02T04:42:06.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+ndwy0V3BcGPX7bXsVpzd/1/P0=', 'Content-Length': '0'} 2022-09-02T04:42:06.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ac6ba1deeb08a61-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.131 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.132 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.132 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.133 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.133 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.134 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.134 DEBUG:boto:Data: 2022-09-02T04:42:06.135 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.136 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.136 DEBUG:boto:Params: {} 2022-09-02T04:42:06.137 DEBUG:boto:Token: None 2022-09-02T04:42:06.138 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.138 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+ndwy0V3BcGPX7bXsVpzd/1/P0= 2022-09-02T04:42:06.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+ndwy0V3BcGPX7bXsVpzd/1/P0=', 'Content-Length': '0'} 2022-09-02T04:42:06.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0dd46cdbb3558b0-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.142 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.142 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.143 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.143 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.144 DEBUG:boto:Method: GET 2022-09-02T04:42:06.144 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.145 DEBUG:boto:Data: 2022-09-02T04:42:06.146 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.147 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.147 DEBUG:boto:Params: {} 2022-09-02T04:42:06.148 DEBUG:boto:Token: None 2022-09-02T04:42:06.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA= 2022-09-02T04:42:06.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA=', 'Content-Length': '0'} 2022-09-02T04:42:06.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000891360e68a324e46-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.153 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.153 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.154 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.155 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.156 DEBUG:boto:Method: GET 2022-09-02T04:42:06.157 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.158 DEBUG:boto:Data: 2022-09-02T04:42:06.159 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.160 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.161 DEBUG:boto:Params: {} 2022-09-02T04:42:06.161 DEBUG:boto:Token: None 2022-09-02T04:42:06.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:06.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA= 2022-09-02T04:42:06.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eOUKYlycIpiNQ4lIPYY6x9wsumA=', 'Content-Length': '0'} 2022-09-02T04:42:06.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005897a09bb305a290-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.166 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.167 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.167 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.168 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.168 DEBUG:boto:Method: GET 2022-09-02T04:42:06.169 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.169 DEBUG:boto:Data: 2022-09-02T04:42:06.170 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.171 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.171 DEBUG:boto:Params: {} 2022-09-02T04:42:06.172 DEBUG:boto:Token: None 2022-09-02T04:42:06.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000e247f68c3c334f-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.177 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.178 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.178 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.179 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.180 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.180 DEBUG:boto:Method: GET 2022-09-02T04:42:06.181 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.181 DEBUG:boto:Data: 2022-09-02T04:42:06.182 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.183 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.183 DEBUG:boto:Params: {} 2022-09-02T04:42:06.184 DEBUG:boto:Token: None 2022-09-02T04:42:06.184 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c08f58c303dec75f-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.190 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.190 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.191 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.191 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.192 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:06.192 DEBUG:boto:Data: 2022-09-02T04:42:06.193 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.194 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.194 DEBUG:boto:Params: {} 2022-09-02T04:42:06.195 DEBUG:boto:Token: None 2022-09-02T04:42:06.196 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj 2022-09-02T04:42:06.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA= 2022-09-02T04:42:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA=', 'Content-Length': '0'} 2022-09-02T04:42:06.200 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f61a88e3b7bc0c7b-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.201 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.201 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.202 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.202 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:06.203 DEBUG:boto:Data: 2022-09-02T04:42:06.203 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.204 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.205 DEBUG:boto:Params: {} 2022-09-02T04:42:06.205 DEBUG:boto:Token: None 2022-09-02T04:42:06.206 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj 2022-09-02T04:42:06.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA= 2022-09-02T04:42:06.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA=', 'Content-Length': '0'} 2022-09-02T04:42:06.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000127108312bddc5ba-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.212 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.212 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.213 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.213 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.214 DEBUG:boto:Method: GET 2022-09-02T04:42:06.215 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.215 DEBUG:boto:Data: 2022-09-02T04:42:06.216 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.217 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.217 DEBUG:boto:Params: {} 2022-09-02T04:42:06.218 DEBUG:boto:Token: None 2022-09-02T04:42:06.218 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6d813f28cae380c-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.266 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.266 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.267 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.268 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.268 DEBUG:boto:Method: GET 2022-09-02T04:42:06.269 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.269 DEBUG:boto:Data: 2022-09-02T04:42:06.270 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.271 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.271 DEBUG:boto:Params: {} 2022-09-02T04:42:06.272 DEBUG:boto:Token: None 2022-09-02T04:42:06.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e7befa3bb7598eb-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.277 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.277 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.278 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.278 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.279 DEBUG:boto:Method: GET 2022-09-02T04:42:06.279 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.280 DEBUG:boto:Data: 2022-09-02T04:42:06.280 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.281 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.282 DEBUG:boto:Params: {} 2022-09-02T04:42:06.282 DEBUG:boto:Token: None 2022-09-02T04:42:06.283 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.284 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000731d1105a61e7a68-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.287 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.288 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.288 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.289 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.289 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.290 DEBUG:boto:Method: GET 2022-09-02T04:42:06.291 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.291 DEBUG:boto:Data: 2022-09-02T04:42:06.292 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.293 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.293 DEBUG:boto:Params: {} 2022-09-02T04:42:06.294 DEBUG:boto:Token: None 2022-09-02T04:42:06.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008486dd0852194fca-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.300 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.302 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.302 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.303 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.304 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.305 DEBUG:boto:Method: GET 2022-09-02T04:42:06.306 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.306 DEBUG:boto:Data: 2022-09-02T04:42:06.307 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.309 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.310 DEBUG:boto:Params: {} 2022-09-02T04:42:06.311 DEBUG:boto:Token: None 2022-09-02T04:42:06.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI= 2022-09-02T04:42:06.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI=', 'Content-Length': '0'} 2022-09-02T04:42:06.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f46a99bb337f2f8b-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.318 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.319 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.319 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.320 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.321 DEBUG:boto:Method: GET 2022-09-02T04:42:06.321 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.322 DEBUG:boto:Data: 2022-09-02T04:42:06.322 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.323 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.324 DEBUG:boto:Params: {} 2022-09-02T04:42:06.324 DEBUG:boto:Token: None 2022-09-02T04:42:06.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI= 2022-09-02T04:42:06.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI=', 'Content-Length': '0'} 2022-09-02T04:42:06.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c960b32fb1a87947-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.329 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.330 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.331 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.331 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.332 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.332 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.334 DEBUG:boto:Data: 2022-09-02T04:42:06.334 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.335 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.336 DEBUG:boto:Params: {} 2022-09-02T04:42:06.336 DEBUG:boto:Token: None 2022-09-02T04:42:06.337 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tUdk9o25qHnptS4ErxirrIoDz0M= 2022-09-02T04:42:06.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tUdk9o25qHnptS4ErxirrIoDz0M=', 'Content-Length': '0'} 2022-09-02T04:42:06.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094e81d0751f59ef3-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.347 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.347 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.348 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.348 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.349 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.349 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.350 DEBUG:boto:Data: 2022-09-02T04:42:06.350 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.351 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.352 DEBUG:boto:Params: {} 2022-09-02T04:42:06.352 DEBUG:boto:Token: None 2022-09-02T04:42:06.353 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tUdk9o25qHnptS4ErxirrIoDz0M= 2022-09-02T04:42:06.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tUdk9o25qHnptS4ErxirrIoDz0M=', 'Content-Length': '0'} 2022-09-02T04:42:06.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cc598234dfa80b3-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.360 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.361 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.361 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.362 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.363 DEBUG:boto:Method: GET 2022-09-02T04:42:06.363 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.364 DEBUG:boto:Data: 2022-09-02T04:42:06.365 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.367 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.368 DEBUG:boto:Params: {} 2022-09-02T04:42:06.369 DEBUG:boto:Token: None 2022-09-02T04:42:06.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI= 2022-09-02T04:42:06.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI=', 'Content-Length': '0'} 2022-09-02T04:42:06.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ee99a43597772f5-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.415 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.416 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.417 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.419 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.421 DEBUG:boto:Method: GET 2022-09-02T04:42:06.422 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.423 DEBUG:boto:Data: 2022-09-02T04:42:06.424 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.425 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.425 DEBUG:boto:Params: {} 2022-09-02T04:42:06.427 DEBUG:boto:Token: None 2022-09-02T04:42:06.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:06.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI= 2022-09-02T04:42:06.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zazNoh4IhI0uvWNBnS7tLcBIZNI=', 'Content-Length': '0'} 2022-09-02T04:42:06.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000726213c9407d9f8-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.434 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.434 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.435 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.436 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.437 DEBUG:boto:Method: GET 2022-09-02T04:42:06.437 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.438 DEBUG:boto:Data: 2022-09-02T04:42:06.439 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.440 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.440 DEBUG:boto:Params: {} 2022-09-02T04:42:06.441 DEBUG:boto:Token: None 2022-09-02T04:42:06.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c345189ecd98e78-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.446 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.447 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.447 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.448 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.448 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.449 DEBUG:boto:Method: GET 2022-09-02T04:42:06.450 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.450 DEBUG:boto:Data: 2022-09-02T04:42:06.451 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.452 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.452 DEBUG:boto:Params: {} 2022-09-02T04:42:06.453 DEBUG:boto:Token: None 2022-09-02T04:42:06.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a07777cb84d8b6c-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.498 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.500 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.501 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.502 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.503 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:06.505 DEBUG:boto:Data: 2022-09-02T04:42:06.506 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.508 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.509 DEBUG:boto:Params: {} 2022-09-02T04:42:06.509 DEBUG:boto:Token: None 2022-09-02T04:42:06.510 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj 2022-09-02T04:42:06.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA= 2022-09-02T04:42:06.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA=', 'Content-Length': '0'} 2022-09-02T04:42:06.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4c8c754800da925-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.518 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.519 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.520 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.521 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:06.522 DEBUG:boto:Data: 2022-09-02T04:42:06.523 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.527 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.527 DEBUG:boto:Params: {} 2022-09-02T04:42:06.528 DEBUG:boto:Token: None 2022-09-02T04:42:06.528 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj 2022-09-02T04:42:06.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA= 2022-09-02T04:42:06.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA=', 'Content-Length': '0'} 2022-09-02T04:42:06.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d650be9644e18753-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.537 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.538 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.539 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.540 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.541 DEBUG:boto:Method: GET 2022-09-02T04:42:06.542 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.543 DEBUG:boto:Data: 2022-09-02T04:42:06.544 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.545 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.546 DEBUG:boto:Params: {} 2022-09-02T04:42:06.547 DEBUG:boto:Token: None 2022-09-02T04:42:06.547 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002228d6ffc5d38ab0-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.553 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.553 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.554 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.555 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.556 DEBUG:boto:Method: GET 2022-09-02T04:42:06.557 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.558 DEBUG:boto:Data: 2022-09-02T04:42:06.559 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.561 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.562 DEBUG:boto:Params: {} 2022-09-02T04:42:06.563 DEBUG:boto:Token: None 2022-09-02T04:42:06.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009df56056b93ccbb7-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.577 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.579 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.580 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.581 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.582 DEBUG:boto:Method: GET 2022-09-02T04:42:06.583 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.584 DEBUG:boto:Data: 2022-09-02T04:42:06.585 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.587 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.588 DEBUG:boto:Params: {} 2022-09-02T04:42:06.589 DEBUG:boto:Token: None 2022-09-02T04:42:06.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c06dfe776905c944-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.594 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.596 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.596 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.597 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.597 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.598 DEBUG:boto:Method: GET 2022-09-02T04:42:06.598 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.599 DEBUG:boto:Data: 2022-09-02T04:42:06.600 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.601 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.601 DEBUG:boto:Params: {} 2022-09-02T04:42:06.602 DEBUG:boto:Token: None 2022-09-02T04:42:06.603 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e4611691b5ef7a-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.608 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.611 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.612 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.614 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.615 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.616 DEBUG:boto:Method: GET 2022-09-02T04:42:06.618 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.618 DEBUG:boto:Data: 2022-09-02T04:42:06.619 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.620 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.620 DEBUG:boto:Params: {} 2022-09-02T04:42:06.621 DEBUG:boto:Token: None 2022-09-02T04:42:06.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU= 2022-09-02T04:42:06.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU=', 'Content-Length': '0'} 2022-09-02T04:42:06.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000885dc573e727c6f3-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.627 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.628 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.628 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.629 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.630 DEBUG:boto:Method: GET 2022-09-02T04:42:06.630 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.631 DEBUG:boto:Data: 2022-09-02T04:42:06.631 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.633 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.634 DEBUG:boto:Params: {} 2022-09-02T04:42:06.634 DEBUG:boto:Token: None 2022-09-02T04:42:06.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU= 2022-09-02T04:42:06.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU=', 'Content-Length': '0'} 2022-09-02T04:42:06.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c978faa727dc13-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.683 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.684 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.685 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.686 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.687 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.688 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.689 DEBUG:boto:Data: 2022-09-02T04:42:06.690 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.692 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.693 DEBUG:boto:Params: {} 2022-09-02T04:42:06.694 DEBUG:boto:Token: None 2022-09-02T04:42:06.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zf81fsR0TNYMCBfCvyxBOjH/smA= 2022-09-02T04:42:06.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zf81fsR0TNYMCBfCvyxBOjH/smA=', 'Content-Length': '0'} 2022-09-02T04:42:06.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba318a232c04e4f0-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.702 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.702 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.703 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.703 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.704 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.704 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.705 DEBUG:boto:Data: 2022-09-02T04:42:06.706 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.708 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.708 DEBUG:boto:Params: {} 2022-09-02T04:42:06.709 DEBUG:boto:Token: None 2022-09-02T04:42:06.709 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zf81fsR0TNYMCBfCvyxBOjH/smA= 2022-09-02T04:42:06.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zf81fsR0TNYMCBfCvyxBOjH/smA=', 'Content-Length': '0'} 2022-09-02T04:42:06.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc0d9eb9542b9c34-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.715 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.716 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.720 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.720 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.721 DEBUG:boto:Method: GET 2022-09-02T04:42:06.721 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.722 DEBUG:boto:Data: 2022-09-02T04:42:06.722 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.724 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.725 DEBUG:boto:Params: {} 2022-09-02T04:42:06.725 DEBUG:boto:Token: None 2022-09-02T04:42:06.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU= 2022-09-02T04:42:06.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU=', 'Content-Length': '0'} 2022-09-02T04:42:06.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f983ddbe9a9c73d-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.735 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.737 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.738 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.738 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.739 DEBUG:boto:Method: GET 2022-09-02T04:42:06.739 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.740 DEBUG:boto:Data: 2022-09-02T04:42:06.740 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.742 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.742 DEBUG:boto:Params: {} 2022-09-02T04:42:06.743 DEBUG:boto:Token: None 2022-09-02T04:42:06.744 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:06.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU= 2022-09-02T04:42:06.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ue/GmuZ+g7N0uqExDxbOVTTqcbU=', 'Content-Length': '0'} 2022-09-02T04:42:06.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060dd819a595af01d-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.749 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.749 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.750 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.750 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.751 DEBUG:boto:Method: GET 2022-09-02T04:42:06.752 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.752 DEBUG:boto:Data: 2022-09-02T04:42:06.753 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.753 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.754 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.756 DEBUG:boto:Params: {} 2022-09-02T04:42:06.756 DEBUG:boto:Token: None 2022-09-02T04:42:06.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000551999aab30a4a51-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.762 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.763 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.763 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.764 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.764 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.765 DEBUG:boto:Method: GET 2022-09-02T04:42:06.765 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.766 DEBUG:boto:Data: 2022-09-02T04:42:06.767 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.768 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.768 DEBUG:boto:Params: {} 2022-09-02T04:42:06.769 DEBUG:boto:Token: None 2022-09-02T04:42:06.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000048cbd8f55af3ea6-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.778 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.778 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.779 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.780 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.780 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:06.781 DEBUG:boto:Data: 2022-09-02T04:42:06.781 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.782 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.783 DEBUG:boto:Params: {} 2022-09-02T04:42:06.784 DEBUG:boto:Token: None 2022-09-02T04:42:06.784 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj 2022-09-02T04:42:06.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA= 2022-09-02T04:42:06.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA=', 'Content-Length': '0'} 2022-09-02T04:42:06.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000219546159803e875-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.794 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.795 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.796 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.796 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:06.797 DEBUG:boto:Data: 2022-09-02T04:42:06.797 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.801 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.802 DEBUG:boto:Params: {} 2022-09-02T04:42:06.803 DEBUG:boto:Token: None 2022-09-02T04:42:06.803 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj 2022-09-02T04:42:06.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA= 2022-09-02T04:42:06.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Tv4bp9e4vaL2ySLbnUAbIbIQwA=', 'Content-Length': '0'} 2022-09-02T04:42:06.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000495f17a3de971343-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.812 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.813 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.814 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.815 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.816 DEBUG:boto:Method: GET 2022-09-02T04:42:06.817 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.819 DEBUG:boto:Data: 2022-09-02T04:42:06.820 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.822 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.823 DEBUG:boto:Params: {} 2022-09-02T04:42:06.824 DEBUG:boto:Token: None 2022-09-02T04:42:06.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003062c7b608c09189-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.832 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.833 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.834 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.835 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.836 DEBUG:boto:Method: GET 2022-09-02T04:42:06.837 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.837 DEBUG:boto:Data: 2022-09-02T04:42:06.838 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.840 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.841 DEBUG:boto:Params: {} 2022-09-02T04:42:06.842 DEBUG:boto:Token: None 2022-09-02T04:42:06.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:06.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw= 2022-09-02T04:42:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/pVlGLc7k86AKwd+6n8XAh5wftw=', 'Content-Length': '0'} 2022-09-02T04:42:06.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a252c8ebc8c14a43-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.852 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.853 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.855 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.857 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.858 DEBUG:boto:Method: GET 2022-09-02T04:42:06.860 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.860 DEBUG:boto:Data: 2022-09-02T04:42:06.861 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.864 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.865 DEBUG:boto:Params: {} 2022-09-02T04:42:06.866 DEBUG:boto:Token: None 2022-09-02T04:42:06.867 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138c6ca68230f154-006311899e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.873 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.874 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:06.875 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:06.876 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.877 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:06.877 DEBUG:boto:Method: GET 2022-09-02T04:42:06.878 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:06.878 DEBUG:boto:Data: 2022-09-02T04:42:06.879 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.880 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.881 DEBUG:boto:Params: {} 2022-09-02T04:42:06.881 DEBUG:boto:Token: None 2022-09-02T04:42:06.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/ 2022-09-02T04:42:06.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ= 2022-09-02T04:42:06.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YnhmUD4vTM6DKuoGB7I1qRbC8IQ=', 'Content-Length': '0'} 2022-09-02T04:42:06.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f93d9d2c96e883-006311899e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.887 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:06.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.890 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.891 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.892 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.892 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.893 DEBUG:boto:Method: GET 2022-09-02T04:42:06.896 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.897 DEBUG:boto:Data: 2022-09-02T04:42:06.897 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.898 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.899 DEBUG:boto:Params: {} 2022-09-02T04:42:06.899 DEBUG:boto:Token: None 2022-09-02T04:42:06.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mk6AFcbPWw0hltbarjjT1DjKyy0= 2022-09-02T04:42:06.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mk6AFcbPWw0hltbarjjT1DjKyy0=', 'Content-Length': '0'} 2022-09-02T04:42:06.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f94a4955130c7d1-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.905 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.907 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.907 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.908 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.909 DEBUG:boto:Method: GET 2022-09-02T04:42:06.909 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.911 DEBUG:boto:Data: 2022-09-02T04:42:06.911 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.912 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.913 DEBUG:boto:Params: {} 2022-09-02T04:42:06.913 DEBUG:boto:Token: None 2022-09-02T04:42:06.914 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mk6AFcbPWw0hltbarjjT1DjKyy0= 2022-09-02T04:42:06.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mk6AFcbPWw0hltbarjjT1DjKyy0=', 'Content-Length': '0'} 2022-09-02T04:42:06.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de477827d328a80d-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.958 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.960 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.961 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.961 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.962 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.963 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.964 DEBUG:boto:Data: 2022-09-02T04:42:06.965 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.967 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.968 DEBUG:boto:Params: {} 2022-09-02T04:42:06.969 DEBUG:boto:Token: None 2022-09-02T04:42:06.970 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xxcTCYTzRRHE00kppfbxLcwtFWk= 2022-09-02T04:42:06.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xxcTCYTzRRHE00kppfbxLcwtFWk=', 'Content-Length': '0'} 2022-09-02T04:42:06.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000398df0c0453ad77a-006311899e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.977 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.979 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.980 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.981 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.982 DEBUG:boto:Method: HEAD 2022-09-02T04:42:06.983 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.984 DEBUG:boto:Data: 2022-09-02T04:42:06.985 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:06.987 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.987 DEBUG:boto:Params: {} 2022-09-02T04:42:06.988 DEBUG:boto:Token: None 2022-09-02T04:42:06.988 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:06 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xxcTCYTzRRHE00kppfbxLcwtFWk= 2022-09-02T04:42:06.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xxcTCYTzRRHE00kppfbxLcwtFWk=', 'Content-Length': '0'} 2022-09-02T04:42:06.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a3fcbab7bab67b6-006311899e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:06.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:06.993 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:06.994 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:06.995 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.995 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.996 DEBUG:boto:Method: GET 2022-09-02T04:42:06.997 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:06.997 DEBUG:boto:Data: 2022-09-02T04:42:06.998 DEBUG:boto:Headers: {} 2022-09-02T04:42:06.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:06.999 DEBUG:boto:Port: 8000 2022-09-02T04:42:06.999 DEBUG:boto:Params: {} 2022-09-02T04:42:07.000 DEBUG:boto:Token: None 2022-09-02T04:42:07.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:07.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VSWaZ+3DtIO5T05vY9vfC0Nh9Qg= 2022-09-02T04:42:07.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VSWaZ+3DtIO5T05vY9vfC0Nh9Qg=', 'Content-Length': '0'} 2022-09-02T04:42:07.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055d55f77ff895f53-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.050 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.051 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.051 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:07.052 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:07.052 DEBUG:boto:Method: GET 2022-09-02T04:42:07.053 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:07.054 DEBUG:boto:Data: 2022-09-02T04:42:07.054 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.055 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.056 DEBUG:boto:Params: {} 2022-09-02T04:42:07.057 DEBUG:boto:Token: None 2022-09-02T04:42:07.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:07.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VSWaZ+3DtIO5T05vY9vfC0Nh9Qg= 2022-09-02T04:42:07.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VSWaZ+3DtIO5T05vY9vfC0Nh9Qg=', 'Content-Length': '0'} 2022-09-02T04:42:07.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000812cd04ac58ca513-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.062 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.062 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.063 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.063 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.064 DEBUG:boto:Method: GET 2022-09-02T04:42:07.065 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.065 DEBUG:boto:Data: 2022-09-02T04:42:07.066 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.068 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.068 DEBUG:boto:Params: {} 2022-09-02T04:42:07.070 DEBUG:boto:Token: None 2022-09-02T04:42:07.071 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea78f6e72d53566a-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.077 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.078 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.079 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.079 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.080 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.081 DEBUG:boto:Method: GET 2022-09-02T04:42:07.081 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.082 DEBUG:boto:Data: 2022-09-02T04:42:07.082 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.084 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.084 DEBUG:boto:Params: {} 2022-09-02T04:42:07.085 DEBUG:boto:Token: None 2022-09-02T04:42:07.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48636f9bcf14d29-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.091 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.092 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.093 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.093 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.094 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.094 DEBUG:boto:Data: 2022-09-02T04:42:07.095 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.096 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.096 DEBUG:boto:Params: {} 2022-09-02T04:42:07.097 DEBUG:boto:Token: None 2022-09-02T04:42:07.098 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f6b80e178090456-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.104 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.105 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.106 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.107 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.109 DEBUG:boto:Data: 2022-09-02T04:42:07.109 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.111 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.112 DEBUG:boto:Params: {} 2022-09-02T04:42:07.114 DEBUG:boto:Token: None 2022-09-02T04:42:07.115 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050b7769536b7df75-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.121 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.122 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.122 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.123 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.123 DEBUG:boto:Method: GET 2022-09-02T04:42:07.124 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.125 DEBUG:boto:Data: 2022-09-02T04:42:07.126 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.128 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.129 DEBUG:boto:Params: {} 2022-09-02T04:42:07.130 DEBUG:boto:Token: None 2022-09-02T04:42:07.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000803772346f79ee16-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.137 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.137 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.138 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.139 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.140 DEBUG:boto:Method: GET 2022-09-02T04:42:07.141 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.142 DEBUG:boto:Data: 2022-09-02T04:42:07.143 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.144 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.144 DEBUG:boto:Params: {} 2022-09-02T04:42:07.145 DEBUG:boto:Token: None 2022-09-02T04:42:07.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efa84ef66712beb7-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.194 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.195 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.196 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.197 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.197 DEBUG:boto:Method: GET 2022-09-02T04:42:07.198 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.198 DEBUG:boto:Data: 2022-09-02T04:42:07.199 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.200 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.201 DEBUG:boto:Params: {} 2022-09-02T04:42:07.201 DEBUG:boto:Token: None 2022-09-02T04:42:07.202 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000558c95e02b5692be-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.207 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.208 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.209 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.209 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.210 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.210 DEBUG:boto:Method: GET 2022-09-02T04:42:07.211 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.211 DEBUG:boto:Data: 2022-09-02T04:42:07.212 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.213 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.214 DEBUG:boto:Params: {} 2022-09-02T04:42:07.214 DEBUG:boto:Token: None 2022-09-02T04:42:07.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9e790bf3c175a1-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.223 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.225 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.225 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.226 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.226 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.227 DEBUG:boto:Method: GET 2022-09-02T04:42:07.227 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.228 DEBUG:boto:Data: 2022-09-02T04:42:07.229 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.230 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.231 DEBUG:boto:Params: {} 2022-09-02T04:42:07.231 DEBUG:boto:Token: None 2022-09-02T04:42:07.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs= 2022-09-02T04:42:07.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs=', 'Content-Length': '0'} 2022-09-02T04:42:07.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001400121351458a4d-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.278 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.279 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.280 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.280 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.281 DEBUG:boto:Method: GET 2022-09-02T04:42:07.282 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.282 DEBUG:boto:Data: 2022-09-02T04:42:07.283 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.284 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.285 DEBUG:boto:Params: {} 2022-09-02T04:42:07.285 DEBUG:boto:Token: None 2022-09-02T04:42:07.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs= 2022-09-02T04:42:07.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs=', 'Content-Length': '0'} 2022-09-02T04:42:07.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037f865a649d25256-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.291 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.291 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.292 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.292 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.293 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.293 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.294 DEBUG:boto:Data: 2022-09-02T04:42:07.295 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.296 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.297 DEBUG:boto:Params: {} 2022-09-02T04:42:07.297 DEBUG:boto:Token: None 2022-09-02T04:42:07.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ElS/L6oE+BeC53xyfvd9QQH0JUM= 2022-09-02T04:42:07.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ElS/L6oE+BeC53xyfvd9QQH0JUM=', 'Content-Length': '0'} 2022-09-02T04:42:07.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000144a14b1c60167ea-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.303 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.304 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.304 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.305 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.305 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.306 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.307 DEBUG:boto:Data: 2022-09-02T04:42:07.307 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.308 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.309 DEBUG:boto:Params: {} 2022-09-02T04:42:07.309 DEBUG:boto:Token: None 2022-09-02T04:42:07.310 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ElS/L6oE+BeC53xyfvd9QQH0JUM= 2022-09-02T04:42:07.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ElS/L6oE+BeC53xyfvd9QQH0JUM=', 'Content-Length': '0'} 2022-09-02T04:42:07.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c4ee0a04bbfd10d-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.315 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.315 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.316 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.316 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.317 DEBUG:boto:Method: GET 2022-09-02T04:42:07.318 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.318 DEBUG:boto:Data: 2022-09-02T04:42:07.319 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.320 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.320 DEBUG:boto:Params: {} 2022-09-02T04:42:07.321 DEBUG:boto:Token: None 2022-09-02T04:42:07.321 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs= 2022-09-02T04:42:07.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs=', 'Content-Length': '0'} 2022-09-02T04:42:07.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038b6ddeaf86c2466-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.326 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.326 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.327 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.328 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.328 DEBUG:boto:Method: GET 2022-09-02T04:42:07.329 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.329 DEBUG:boto:Data: 2022-09-02T04:42:07.330 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.331 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.331 DEBUG:boto:Params: {} 2022-09-02T04:42:07.332 DEBUG:boto:Token: None 2022-09-02T04:42:07.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:07.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs= 2022-09-02T04:42:07.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:juBIaHONeARZ9mZy7qFtZ0UtXFs=', 'Content-Length': '0'} 2022-09-02T04:42:07.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b6a46e32efef3f9-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.337 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.338 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.338 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.339 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.339 DEBUG:boto:Method: GET 2022-09-02T04:42:07.340 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.341 DEBUG:boto:Data: 2022-09-02T04:42:07.341 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.342 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.343 DEBUG:boto:Params: {} 2022-09-02T04:42:07.343 DEBUG:boto:Token: None 2022-09-02T04:42:07.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0aa707d48d885c6-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.349 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.350 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.351 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.351 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.352 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.352 DEBUG:boto:Method: GET 2022-09-02T04:42:07.353 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.353 DEBUG:boto:Data: 2022-09-02T04:42:07.354 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.355 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.355 DEBUG:boto:Params: {} 2022-09-02T04:42:07.356 DEBUG:boto:Token: None 2022-09-02T04:42:07.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d63d499f4a7987e-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.361 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.362 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.363 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.363 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.364 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.364 DEBUG:boto:Data: 2022-09-02T04:42:07.365 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.366 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.367 DEBUG:boto:Params: {} 2022-09-02T04:42:07.367 DEBUG:boto:Token: None 2022-09-02T04:42:07.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000487d58639877c836-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.374 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.374 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.375 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.375 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.376 DEBUG:boto:Data: 2022-09-02T04:42:07.376 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.377 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.378 DEBUG:boto:Params: {} 2022-09-02T04:42:07.378 DEBUG:boto:Token: None 2022-09-02T04:42:07.379 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2d2323ac0797aab-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.385 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.386 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.386 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.387 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.387 DEBUG:boto:Method: GET 2022-09-02T04:42:07.388 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.388 DEBUG:boto:Data: 2022-09-02T04:42:07.389 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.390 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.391 DEBUG:boto:Params: {} 2022-09-02T04:42:07.391 DEBUG:boto:Token: None 2022-09-02T04:42:07.392 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d40f7e96ba4546a8-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.396 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.397 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.397 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.398 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.399 DEBUG:boto:Method: GET 2022-09-02T04:42:07.399 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.400 DEBUG:boto:Data: 2022-09-02T04:42:07.400 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.401 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.402 DEBUG:boto:Params: {} 2022-09-02T04:42:07.402 DEBUG:boto:Token: None 2022-09-02T04:42:07.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2893c2cf5798264-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.409 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.410 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.411 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.413 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.414 DEBUG:boto:Method: GET 2022-09-02T04:42:07.415 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.415 DEBUG:boto:Data: 2022-09-02T04:42:07.416 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.417 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.418 DEBUG:boto:Params: {} 2022-09-02T04:42:07.418 DEBUG:boto:Token: None 2022-09-02T04:42:07.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9ffa17c339025c-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.424 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.425 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.425 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.426 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.426 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.427 DEBUG:boto:Method: GET 2022-09-02T04:42:07.427 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.428 DEBUG:boto:Data: 2022-09-02T04:42:07.429 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.430 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.430 DEBUG:boto:Params: {} 2022-09-02T04:42:07.431 DEBUG:boto:Token: None 2022-09-02T04:42:07.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7bdf8ad332e28d9-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.478 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.480 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.480 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.481 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.481 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.482 DEBUG:boto:Method: GET 2022-09-02T04:42:07.483 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.483 DEBUG:boto:Data: 2022-09-02T04:42:07.484 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.485 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.486 DEBUG:boto:Params: {} 2022-09-02T04:42:07.486 DEBUG:boto:Token: None 2022-09-02T04:42:07.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA= 2022-09-02T04:42:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA=', 'Content-Length': '0'} 2022-09-02T04:42:07.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fced438c8634839-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.492 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.493 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.494 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.495 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.495 DEBUG:boto:Method: GET 2022-09-02T04:42:07.496 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.497 DEBUG:boto:Data: 2022-09-02T04:42:07.497 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.498 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.499 DEBUG:boto:Params: {} 2022-09-02T04:42:07.500 DEBUG:boto:Token: None 2022-09-02T04:42:07.500 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.501 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA= 2022-09-02T04:42:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA=', 'Content-Length': '0'} 2022-09-02T04:42:07.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6f065bac58f966d-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.506 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.507 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.507 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.508 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.509 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.509 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.510 DEBUG:boto:Data: 2022-09-02T04:42:07.510 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.511 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.512 DEBUG:boto:Params: {} 2022-09-02T04:42:07.513 DEBUG:boto:Token: None 2022-09-02T04:42:07.514 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q/jjyFX8q9KLhAcicA/E/RDQBFQ= 2022-09-02T04:42:07.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q/jjyFX8q9KLhAcicA/E/RDQBFQ=', 'Content-Length': '0'} 2022-09-02T04:42:07.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000589f350a162ad463-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.519 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.520 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.520 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.521 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.521 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.522 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.523 DEBUG:boto:Data: 2022-09-02T04:42:07.523 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.524 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.525 DEBUG:boto:Params: {} 2022-09-02T04:42:07.526 DEBUG:boto:Token: None 2022-09-02T04:42:07.526 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q/jjyFX8q9KLhAcicA/E/RDQBFQ= 2022-09-02T04:42:07.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q/jjyFX8q9KLhAcicA/E/RDQBFQ=', 'Content-Length': '0'} 2022-09-02T04:42:07.530 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fa3db0c9e756b46-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.531 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.532 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.532 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.533 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.534 DEBUG:boto:Method: GET 2022-09-02T04:42:07.534 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.535 DEBUG:boto:Data: 2022-09-02T04:42:07.535 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.536 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.537 DEBUG:boto:Params: {} 2022-09-02T04:42:07.537 DEBUG:boto:Token: None 2022-09-02T04:42:07.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA= 2022-09-02T04:42:07.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA=', 'Content-Length': '0'} 2022-09-02T04:42:07.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000730e84712ff9d9ec-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.544 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.544 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.545 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.545 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.546 DEBUG:boto:Method: GET 2022-09-02T04:42:07.547 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.547 DEBUG:boto:Data: 2022-09-02T04:42:07.548 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.549 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.549 DEBUG:boto:Params: {} 2022-09-02T04:42:07.550 DEBUG:boto:Token: None 2022-09-02T04:42:07.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:07.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA= 2022-09-02T04:42:07.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/L5NaOFYViGRxgfuYb/mu3clCA=', 'Content-Length': '0'} 2022-09-02T04:42:07.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1cdefe0674a4e07-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.556 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.556 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.557 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.558 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.558 DEBUG:boto:Method: GET 2022-09-02T04:42:07.559 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.559 DEBUG:boto:Data: 2022-09-02T04:42:07.560 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.561 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.561 DEBUG:boto:Params: {} 2022-09-02T04:42:07.562 DEBUG:boto:Token: None 2022-09-02T04:42:07.563 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051cf58b840576958-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.567 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.569 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.569 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.570 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.570 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.571 DEBUG:boto:Method: GET 2022-09-02T04:42:07.571 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.572 DEBUG:boto:Data: 2022-09-02T04:42:07.572 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.574 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.574 DEBUG:boto:Params: {} 2022-09-02T04:42:07.575 DEBUG:boto:Token: None 2022-09-02T04:42:07.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca70b8ac3318c5db-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.580 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.581 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.581 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.582 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.582 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.583 DEBUG:boto:Data: 2022-09-02T04:42:07.583 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.584 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.585 DEBUG:boto:Params: {} 2022-09-02T04:42:07.585 DEBUG:boto:Token: None 2022-09-02T04:42:07.586 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009253198ed21bf1f8-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.591 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.592 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.593 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.593 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.594 DEBUG:boto:Data: 2022-09-02T04:42:07.595 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.596 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.596 DEBUG:boto:Params: {} 2022-09-02T04:42:07.597 DEBUG:boto:Token: None 2022-09-02T04:42:07.598 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005aa4ad4b78491de5-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.605 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.606 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.607 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.607 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.608 DEBUG:boto:Method: GET 2022-09-02T04:42:07.608 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.609 DEBUG:boto:Data: 2022-09-02T04:42:07.609 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.610 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.611 DEBUG:boto:Params: {} 2022-09-02T04:42:07.611 DEBUG:boto:Token: None 2022-09-02T04:42:07.612 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025b2f9807b698a9c-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.662 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.663 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.663 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.664 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.664 DEBUG:boto:Method: GET 2022-09-02T04:42:07.665 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.666 DEBUG:boto:Data: 2022-09-02T04:42:07.666 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.667 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.668 DEBUG:boto:Params: {} 2022-09-02T04:42:07.668 DEBUG:boto:Token: None 2022-09-02T04:42:07.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000879e7aec4cb8a2de-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.674 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.677 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.678 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.679 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.680 DEBUG:boto:Method: GET 2022-09-02T04:42:07.681 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.682 DEBUG:boto:Data: 2022-09-02T04:42:07.683 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.683 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.684 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.686 DEBUG:boto:Params: {} 2022-09-02T04:42:07.687 DEBUG:boto:Token: None 2022-09-02T04:42:07.688 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f25837c9353cf1-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.695 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.696 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.697 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.698 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.699 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.700 DEBUG:boto:Method: GET 2022-09-02T04:42:07.701 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.702 DEBUG:boto:Data: 2022-09-02T04:42:07.703 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.705 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.706 DEBUG:boto:Params: {} 2022-09-02T04:42:07.707 DEBUG:boto:Token: None 2022-09-02T04:42:07.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a02b538e799af5-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.713 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.715 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.716 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.717 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.718 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.719 DEBUG:boto:Method: GET 2022-09-02T04:42:07.720 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.721 DEBUG:boto:Data: 2022-09-02T04:42:07.721 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.723 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.723 DEBUG:boto:Params: {} 2022-09-02T04:42:07.724 DEBUG:boto:Token: None 2022-09-02T04:42:07.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as= 2022-09-02T04:42:07.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as=', 'Content-Length': '0'} 2022-09-02T04:42:07.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa5d6e438e7125f3-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.730 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.731 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.732 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.732 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.733 DEBUG:boto:Method: GET 2022-09-02T04:42:07.734 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.735 DEBUG:boto:Data: 2022-09-02T04:42:07.736 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.737 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.738 DEBUG:boto:Params: {} 2022-09-02T04:42:07.739 DEBUG:boto:Token: None 2022-09-02T04:42:07.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as= 2022-09-02T04:42:07.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as=', 'Content-Length': '0'} 2022-09-02T04:42:07.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e192caa9523d18c-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.747 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.747 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.748 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.749 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.750 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.751 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.751 DEBUG:boto:Data: 2022-09-02T04:42:07.752 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.753 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.753 DEBUG:boto:Params: {} 2022-09-02T04:42:07.754 DEBUG:boto:Token: None 2022-09-02T04:42:07.755 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F193o6FHCB/zE+dFljRcpKNlX/4= 2022-09-02T04:42:07.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F193o6FHCB/zE+dFljRcpKNlX/4=', 'Content-Length': '0'} 2022-09-02T04:42:07.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000615c790ef6af15cc-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.761 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.762 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.762 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.763 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.764 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.765 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.766 DEBUG:boto:Data: 2022-09-02T04:42:07.767 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.769 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.770 DEBUG:boto:Params: {} 2022-09-02T04:42:07.771 DEBUG:boto:Token: None 2022-09-02T04:42:07.772 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F193o6FHCB/zE+dFljRcpKNlX/4= 2022-09-02T04:42:07.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F193o6FHCB/zE+dFljRcpKNlX/4=', 'Content-Length': '0'} 2022-09-02T04:42:07.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5af00c89200a199-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.777 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.778 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.779 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.779 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.780 DEBUG:boto:Method: GET 2022-09-02T04:42:07.780 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.781 DEBUG:boto:Data: 2022-09-02T04:42:07.782 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.783 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.783 DEBUG:boto:Params: {} 2022-09-02T04:42:07.784 DEBUG:boto:Token: None 2022-09-02T04:42:07.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as= 2022-09-02T04:42:07.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as=', 'Content-Length': '0'} 2022-09-02T04:42:07.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e876ca2dfbd938d-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.834 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.836 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.836 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.837 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.838 DEBUG:boto:Method: GET 2022-09-02T04:42:07.841 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.842 DEBUG:boto:Data: 2022-09-02T04:42:07.843 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.845 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.846 DEBUG:boto:Params: {} 2022-09-02T04:42:07.848 DEBUG:boto:Token: None 2022-09-02T04:42:07.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:07.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as= 2022-09-02T04:42:07.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:arwDxVRxwfTe9ZucJcMY5uB76as=', 'Content-Length': '0'} 2022-09-02T04:42:07.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b225eeda68339e95-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.856 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.856 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.857 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.862 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.863 DEBUG:boto:Method: GET 2022-09-02T04:42:07.864 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.865 DEBUG:boto:Data: 2022-09-02T04:42:07.866 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.868 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.868 DEBUG:boto:Params: {} 2022-09-02T04:42:07.869 DEBUG:boto:Token: None 2022-09-02T04:42:07.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f056a1d64ddf542a-006311899f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.875 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.876 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:07.876 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:07.877 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.877 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:07.878 DEBUG:boto:Method: GET 2022-09-02T04:42:07.879 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:07.879 DEBUG:boto:Data: 2022-09-02T04:42:07.880 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.881 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.881 DEBUG:boto:Params: {} 2022-09-02T04:42:07.882 DEBUG:boto:Token: None 2022-09-02T04:42:07.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/ 2022-09-02T04:42:07.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY= 2022-09-02T04:42:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvqSWeSLUZDEOwzukEy1rdBYxUY=', 'Content-Length': '0'} 2022-09-02T04:42:07.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da28aee1e154906-006311899f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.930 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:07.932 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.933 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.934 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.935 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.935 DEBUG:boto:Data: 2022-09-02T04:42:07.936 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.938 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.939 DEBUG:boto:Params: {} 2022-09-02T04:42:07.939 DEBUG:boto:Token: None 2022-09-02T04:42:07.940 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4f16ab3f7664e88-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.950 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.950 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.951 DEBUG:boto:Method: HEAD 2022-09-02T04:42:07.951 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:07.952 DEBUG:boto:Data: 2022-09-02T04:42:07.952 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:07.953 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.954 DEBUG:boto:Params: {} 2022-09-02T04:42:07.955 DEBUG:boto:Token: None 2022-09-02T04:42:07.955 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj 2022-09-02T04:42:07.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw= 2022-09-02T04:42:07.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znkwpA33+uZ9gh7hiEts73Ue3zw=', 'Content-Length': '0'} 2022-09-02T04:42:07.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006abe2083259a2499-006311899f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:07.968 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.968 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.969 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.969 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.970 DEBUG:boto:Method: GET 2022-09-02T04:42:07.971 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.971 DEBUG:boto:Data: 2022-09-02T04:42:07.972 DEBUG:boto:Headers: {} 2022-09-02T04:42:07.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:07.973 DEBUG:boto:Port: 8000 2022-09-02T04:42:07.974 DEBUG:boto:Params: {} 2022-09-02T04:42:07.974 DEBUG:boto:Token: None 2022-09-02T04:42:07.975 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:07 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE= 2022-09-02T04:42:07.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oFjvzByuBisBc5Dr/K3936+sdGE=', 'Content-Length': '0'} 2022-09-02T04:42:07.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d1eb83fc32b7906-006311899f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:07.992 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:07.993 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:07.994 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:07.995 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.002 DEBUG:boto:Method: GET 2022-09-02T04:42:08.003 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.003 DEBUG:boto:Data: 2022-09-02T04:42:08.004 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.006 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.008 DEBUG:boto:Params: {} 2022-09-02T04:42:08.009 DEBUG:boto:Token: None 2022-09-02T04:42:08.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033203438c208fb37-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.016 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.016 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.017 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.018 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.019 DEBUG:boto:Method: GET 2022-09-02T04:42:08.020 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.021 DEBUG:boto:Data: 2022-09-02T04:42:08.022 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.024 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.025 DEBUG:boto:Params: {} 2022-09-02T04:42:08.026 DEBUG:boto:Token: None 2022-09-02T04:42:08.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7cc39eca2b0d5d2-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.033 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.034 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.035 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.036 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.036 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.037 DEBUG:boto:Method: GET 2022-09-02T04:42:08.038 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.039 DEBUG:boto:Data: 2022-09-02T04:42:08.039 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.041 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.041 DEBUG:boto:Params: {} 2022-09-02T04:42:08.042 DEBUG:boto:Token: None 2022-09-02T04:42:08.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe933893ec85ee0b-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.047 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.049 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.049 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.050 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.050 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.051 DEBUG:boto:Method: GET 2022-09-02T04:42:08.053 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.053 DEBUG:boto:Data: 2022-09-02T04:42:08.054 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.055 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.055 DEBUG:boto:Params: {} 2022-09-02T04:42:08.056 DEBUG:boto:Token: None 2022-09-02T04:42:08.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk= 2022-09-02T04:42:08.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk=', 'Content-Length': '0'} 2022-09-02T04:42:08.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a600c1765f53aa6-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.061 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.062 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.062 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.063 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.064 DEBUG:boto:Method: GET 2022-09-02T04:42:08.064 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.065 DEBUG:boto:Data: 2022-09-02T04:42:08.065 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.066 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.067 DEBUG:boto:Params: {} 2022-09-02T04:42:08.068 DEBUG:boto:Token: None 2022-09-02T04:42:08.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.069 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk= 2022-09-02T04:42:08.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk=', 'Content-Length': '0'} 2022-09-02T04:42:08.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3be84a6b2f736f5-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.110 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.112 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.113 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.113 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.114 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.115 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.116 DEBUG:boto:Data: 2022-09-02T04:42:08.117 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.119 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.121 DEBUG:boto:Params: {} 2022-09-02T04:42:08.123 DEBUG:boto:Token: None 2022-09-02T04:42:08.124 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BKoQqbnb1MCTAobLyQi8HZuGhNk= 2022-09-02T04:42:08.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BKoQqbnb1MCTAobLyQi8HZuGhNk=', 'Content-Length': '0'} 2022-09-02T04:42:08.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a56399d44684aa1d-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.131 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.133 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.134 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.135 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.137 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.138 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.139 DEBUG:boto:Data: 2022-09-02T04:42:08.140 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.142 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.143 DEBUG:boto:Params: {} 2022-09-02T04:42:08.144 DEBUG:boto:Token: None 2022-09-02T04:42:08.145 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BKoQqbnb1MCTAobLyQi8HZuGhNk= 2022-09-02T04:42:08.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BKoQqbnb1MCTAobLyQi8HZuGhNk=', 'Content-Length': '0'} 2022-09-02T04:42:08.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ca807808469ae28-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.152 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.153 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.154 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.155 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.156 DEBUG:boto:Method: GET 2022-09-02T04:42:08.157 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.158 DEBUG:boto:Data: 2022-09-02T04:42:08.159 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.161 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.162 DEBUG:boto:Params: {} 2022-09-02T04:42:08.163 DEBUG:boto:Token: None 2022-09-02T04:42:08.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.164 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk= 2022-09-02T04:42:08.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk=', 'Content-Length': '0'} 2022-09-02T04:42:08.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6ac2e10695b916d-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.171 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.172 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.173 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.175 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.176 DEBUG:boto:Method: GET 2022-09-02T04:42:08.177 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.178 DEBUG:boto:Data: 2022-09-02T04:42:08.179 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.181 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.183 DEBUG:boto:Params: {} 2022-09-02T04:42:08.183 DEBUG:boto:Token: None 2022-09-02T04:42:08.184 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:08.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk= 2022-09-02T04:42:08.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0TprDpLWtCmFNov6GsEWkW5uQqk=', 'Content-Length': '0'} 2022-09-02T04:42:08.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000829f9c656f2cf41-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.190 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.191 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.192 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.193 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.195 DEBUG:boto:Method: GET 2022-09-02T04:42:08.195 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.196 DEBUG:boto:Data: 2022-09-02T04:42:08.197 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.198 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.199 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.199 DEBUG:boto:Params: {} 2022-09-02T04:42:08.200 DEBUG:boto:Token: None 2022-09-02T04:42:08.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b626254b2c1f325a-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.207 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.209 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.210 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.211 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.212 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.213 DEBUG:boto:Method: GET 2022-09-02T04:42:08.214 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.215 DEBUG:boto:Data: 2022-09-02T04:42:08.216 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.218 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.220 DEBUG:boto:Params: {} 2022-09-02T04:42:08.221 DEBUG:boto:Token: None 2022-09-02T04:42:08.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aad297b1249b4a1-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.229 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.230 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.231 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.231 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.233 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:08.233 DEBUG:boto:Data: 2022-09-02T04:42:08.234 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.238 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.239 DEBUG:boto:Params: {} 2022-09-02T04:42:08.239 DEBUG:boto:Token: None 2022-09-02T04:42:08.240 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj 2022-09-02T04:42:08.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w= 2022-09-02T04:42:08.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w=', 'Content-Length': '0'} 2022-09-02T04:42:08.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043bf0168d2f02dca-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.246 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.246 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.247 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.247 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:08.248 DEBUG:boto:Data: 2022-09-02T04:42:08.249 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.250 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.250 DEBUG:boto:Params: {} 2022-09-02T04:42:08.251 DEBUG:boto:Token: None 2022-09-02T04:42:08.252 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj 2022-09-02T04:42:08.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w= 2022-09-02T04:42:08.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w=', 'Content-Length': '0'} 2022-09-02T04:42:08.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a82671dcc76378e7-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.258 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.259 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.259 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.260 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.260 DEBUG:boto:Method: GET 2022-09-02T04:42:08.261 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.261 DEBUG:boto:Data: 2022-09-02T04:42:08.262 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.263 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.264 DEBUG:boto:Params: {} 2022-09-02T04:42:08.265 DEBUG:boto:Token: None 2022-09-02T04:42:08.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ec7cb8e22eb23ea-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.272 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.273 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.274 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.274 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.277 DEBUG:boto:Method: GET 2022-09-02T04:42:08.277 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.278 DEBUG:boto:Data: 2022-09-02T04:42:08.279 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.280 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.280 DEBUG:boto:Params: {} 2022-09-02T04:42:08.281 DEBUG:boto:Token: None 2022-09-02T04:42:08.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab0c4d7fbd3961fe-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.287 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.288 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.289 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.289 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.290 DEBUG:boto:Method: GET 2022-09-02T04:42:08.290 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.291 DEBUG:boto:Data: 2022-09-02T04:42:08.291 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.292 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.293 DEBUG:boto:Params: {} 2022-09-02T04:42:08.294 DEBUG:boto:Token: None 2022-09-02T04:42:08.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074140490cc214439-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.342 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.343 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.344 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.345 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.345 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.346 DEBUG:boto:Method: GET 2022-09-02T04:42:08.346 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.347 DEBUG:boto:Data: 2022-09-02T04:42:08.348 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.350 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.351 DEBUG:boto:Params: {} 2022-09-02T04:42:08.352 DEBUG:boto:Token: None 2022-09-02T04:42:08.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024cf347b2bc0c59b-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.359 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.360 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.361 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.361 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.362 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.362 DEBUG:boto:Method: GET 2022-09-02T04:42:08.363 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.364 DEBUG:boto:Data: 2022-09-02T04:42:08.364 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.365 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.366 DEBUG:boto:Params: {} 2022-09-02T04:42:08.366 DEBUG:boto:Token: None 2022-09-02T04:42:08.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE= 2022-09-02T04:42:08.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE=', 'Content-Length': '0'} 2022-09-02T04:42:08.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e348569cb73eda01-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.372 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.372 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.373 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.373 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.374 DEBUG:boto:Method: GET 2022-09-02T04:42:08.374 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.375 DEBUG:boto:Data: 2022-09-02T04:42:08.375 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.376 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.377 DEBUG:boto:Params: {} 2022-09-02T04:42:08.378 DEBUG:boto:Token: None 2022-09-02T04:42:08.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE= 2022-09-02T04:42:08.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE=', 'Content-Length': '0'} 2022-09-02T04:42:08.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e17335f32002fa0-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.422 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.423 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.423 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.424 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.424 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.425 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.426 DEBUG:boto:Data: 2022-09-02T04:42:08.426 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.428 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.429 DEBUG:boto:Params: {} 2022-09-02T04:42:08.430 DEBUG:boto:Token: None 2022-09-02T04:42:08.431 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qrpcy38J5T1rZc42A+Rg1cwLrjo= 2022-09-02T04:42:08.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qrpcy38J5T1rZc42A+Rg1cwLrjo=', 'Content-Length': '0'} 2022-09-02T04:42:08.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000629d81079a8fd010-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.439 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.439 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.440 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.440 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.441 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.441 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.442 DEBUG:boto:Data: 2022-09-02T04:42:08.442 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.443 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.444 DEBUG:boto:Params: {} 2022-09-02T04:42:08.444 DEBUG:boto:Token: None 2022-09-02T04:42:08.445 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qrpcy38J5T1rZc42A+Rg1cwLrjo= 2022-09-02T04:42:08.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qrpcy38J5T1rZc42A+Rg1cwLrjo=', 'Content-Length': '0'} 2022-09-02T04:42:08.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000661851dd74edb37e-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.450 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.450 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.451 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.451 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.452 DEBUG:boto:Method: GET 2022-09-02T04:42:08.452 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.453 DEBUG:boto:Data: 2022-09-02T04:42:08.453 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.454 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.455 DEBUG:boto:Params: {} 2022-09-02T04:42:08.456 DEBUG:boto:Token: None 2022-09-02T04:42:08.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE= 2022-09-02T04:42:08.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE=', 'Content-Length': '0'} 2022-09-02T04:42:08.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000469ecd6aad062f51-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.461 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.461 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.462 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.462 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.463 DEBUG:boto:Method: GET 2022-09-02T04:42:08.464 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.464 DEBUG:boto:Data: 2022-09-02T04:42:08.465 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.467 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.467 DEBUG:boto:Params: {} 2022-09-02T04:42:08.468 DEBUG:boto:Token: None 2022-09-02T04:42:08.468 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:08.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE= 2022-09-02T04:42:08.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2OCvZY+Wu7Y/92IkPQ7TIzxJAE=', 'Content-Length': '0'} 2022-09-02T04:42:08.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008380b744e0ff59ba-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.473 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.474 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.474 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.475 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.476 DEBUG:boto:Method: GET 2022-09-02T04:42:08.477 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.478 DEBUG:boto:Data: 2022-09-02T04:42:08.479 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.480 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.480 DEBUG:boto:Params: {} 2022-09-02T04:42:08.481 DEBUG:boto:Token: None 2022-09-02T04:42:08.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fc64dea8931d71d-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.488 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.489 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.490 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.490 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.491 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.491 DEBUG:boto:Method: GET 2022-09-02T04:42:08.492 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.492 DEBUG:boto:Data: 2022-09-02T04:42:08.493 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.494 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.494 DEBUG:boto:Params: {} 2022-09-02T04:42:08.495 DEBUG:boto:Token: None 2022-09-02T04:42:08.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a42957899bc15fd2-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.501 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.502 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.502 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.503 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.504 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:08.505 DEBUG:boto:Data: 2022-09-02T04:42:08.505 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.506 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.507 DEBUG:boto:Params: {} 2022-09-02T04:42:08.509 DEBUG:boto:Token: None 2022-09-02T04:42:08.511 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj 2022-09-02T04:42:08.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w= 2022-09-02T04:42:08.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w=', 'Content-Length': '0'} 2022-09-02T04:42:08.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdddf659f73d1a8d-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.520 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.521 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.522 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.523 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:08.524 DEBUG:boto:Data: 2022-09-02T04:42:08.525 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.526 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.527 DEBUG:boto:Params: {} 2022-09-02T04:42:08.528 DEBUG:boto:Token: None 2022-09-02T04:42:08.529 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj 2022-09-02T04:42:08.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w= 2022-09-02T04:42:08.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w=', 'Content-Length': '0'} 2022-09-02T04:42:08.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb70854a62bf0833-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.539 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.540 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.541 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.542 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.543 DEBUG:boto:Method: GET 2022-09-02T04:42:08.543 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.544 DEBUG:boto:Data: 2022-09-02T04:42:08.545 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.546 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.547 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.548 DEBUG:boto:Params: {} 2022-09-02T04:42:08.549 DEBUG:boto:Token: None 2022-09-02T04:42:08.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f266d8bc32a6a51e-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.555 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.556 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.557 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.558 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.558 DEBUG:boto:Method: GET 2022-09-02T04:42:08.559 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.559 DEBUG:boto:Data: 2022-09-02T04:42:08.560 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.561 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.562 DEBUG:boto:Params: {} 2022-09-02T04:42:08.562 DEBUG:boto:Token: None 2022-09-02T04:42:08.563 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5f53f671859d732-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.569 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.569 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.570 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.571 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.571 DEBUG:boto:Method: GET 2022-09-02T04:42:08.572 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.572 DEBUG:boto:Data: 2022-09-02T04:42:08.573 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.574 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.575 DEBUG:boto:Params: {} 2022-09-02T04:42:08.576 DEBUG:boto:Token: None 2022-09-02T04:42:08.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c042a70faec48ead-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.624 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.625 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.626 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.626 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.627 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.627 DEBUG:boto:Method: GET 2022-09-02T04:42:08.628 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.629 DEBUG:boto:Data: 2022-09-02T04:42:08.629 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.631 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.631 DEBUG:boto:Params: {} 2022-09-02T04:42:08.632 DEBUG:boto:Token: None 2022-09-02T04:42:08.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c84b176752b1d6b5-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.637 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.639 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.639 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.640 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.640 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.641 DEBUG:boto:Method: GET 2022-09-02T04:42:08.641 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.642 DEBUG:boto:Data: 2022-09-02T04:42:08.643 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.644 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.644 DEBUG:boto:Params: {} 2022-09-02T04:42:08.645 DEBUG:boto:Token: None 2022-09-02T04:42:08.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI= 2022-09-02T04:42:08.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI=', 'Content-Length': '0'} 2022-09-02T04:42:08.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038c11e66ba623c49-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.650 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.651 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.651 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.652 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.652 DEBUG:boto:Method: GET 2022-09-02T04:42:08.653 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.653 DEBUG:boto:Data: 2022-09-02T04:42:08.654 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.655 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.655 DEBUG:boto:Params: {} 2022-09-02T04:42:08.656 DEBUG:boto:Token: None 2022-09-02T04:42:08.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI= 2022-09-02T04:42:08.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI=', 'Content-Length': '0'} 2022-09-02T04:42:08.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000132435f110123ff2-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.661 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.662 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.662 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.663 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.663 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.664 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.664 DEBUG:boto:Data: 2022-09-02T04:42:08.665 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.666 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.666 DEBUG:boto:Params: {} 2022-09-02T04:42:08.667 DEBUG:boto:Token: None 2022-09-02T04:42:08.668 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:04sSr3zhOAL54HU/0xwtCYrwrUs= 2022-09-02T04:42:08.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:04sSr3zhOAL54HU/0xwtCYrwrUs=', 'Content-Length': '0'} 2022-09-02T04:42:08.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b0cd49fc2856ec6-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.672 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.673 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.673 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.674 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.675 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.675 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.676 DEBUG:boto:Data: 2022-09-02T04:42:08.676 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.677 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.678 DEBUG:boto:Params: {} 2022-09-02T04:42:08.679 DEBUG:boto:Token: None 2022-09-02T04:42:08.679 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:04sSr3zhOAL54HU/0xwtCYrwrUs= 2022-09-02T04:42:08.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:04sSr3zhOAL54HU/0xwtCYrwrUs=', 'Content-Length': '0'} 2022-09-02T04:42:08.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049fb0d85da7ecd37-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.684 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.684 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.685 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.686 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.686 DEBUG:boto:Method: GET 2022-09-02T04:42:08.687 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.687 DEBUG:boto:Data: 2022-09-02T04:42:08.688 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.689 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.689 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.690 DEBUG:boto:Params: {} 2022-09-02T04:42:08.690 DEBUG:boto:Token: None 2022-09-02T04:42:08.691 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI= 2022-09-02T04:42:08.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI=', 'Content-Length': '0'} 2022-09-02T04:42:08.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af86caaeb6b0a32c-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.695 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.696 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.696 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.697 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.698 DEBUG:boto:Method: GET 2022-09-02T04:42:08.698 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.699 DEBUG:boto:Data: 2022-09-02T04:42:08.699 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.700 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.701 DEBUG:boto:Params: {} 2022-09-02T04:42:08.701 DEBUG:boto:Token: None 2022-09-02T04:42:08.702 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:08.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI= 2022-09-02T04:42:08.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jsRMSkbZIdSuYTj1EIIA2MFzUeI=', 'Content-Length': '0'} 2022-09-02T04:42:08.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b41cbdc8c56f7d0a-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.750 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.751 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.752 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.752 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.753 DEBUG:boto:Method: GET 2022-09-02T04:42:08.753 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.754 DEBUG:boto:Data: 2022-09-02T04:42:08.755 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.756 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.756 DEBUG:boto:Params: {} 2022-09-02T04:42:08.757 DEBUG:boto:Token: None 2022-09-02T04:42:08.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088ca3c8e8774c48b-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.762 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.763 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.764 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.765 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.765 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.766 DEBUG:boto:Method: GET 2022-09-02T04:42:08.766 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.767 DEBUG:boto:Data: 2022-09-02T04:42:08.767 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.768 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.769 DEBUG:boto:Params: {} 2022-09-02T04:42:08.769 DEBUG:boto:Token: None 2022-09-02T04:42:08.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fad87de34b1ed06-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.775 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.776 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.776 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.777 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.777 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:08.778 DEBUG:boto:Data: 2022-09-02T04:42:08.778 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.779 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.780 DEBUG:boto:Params: {} 2022-09-02T04:42:08.780 DEBUG:boto:Token: None 2022-09-02T04:42:08.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj 2022-09-02T04:42:08.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w= 2022-09-02T04:42:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w=', 'Content-Length': '0'} 2022-09-02T04:42:08.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6152c6057d7b1e3-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.787 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.787 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.788 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.788 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:08.789 DEBUG:boto:Data: 2022-09-02T04:42:08.789 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.790 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.791 DEBUG:boto:Params: {} 2022-09-02T04:42:08.791 DEBUG:boto:Token: None 2022-09-02T04:42:08.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj 2022-09-02T04:42:08.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w= 2022-09-02T04:42:08.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLd0sCtolElJ0dsHYVKzrfZyd2w=', 'Content-Length': '0'} 2022-09-02T04:42:08.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a31af11e6f2ec129-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.799 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.799 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.800 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.800 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.801 DEBUG:boto:Method: GET 2022-09-02T04:42:08.801 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.802 DEBUG:boto:Data: 2022-09-02T04:42:08.803 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.804 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.804 DEBUG:boto:Params: {} 2022-09-02T04:42:08.805 DEBUG:boto:Token: None 2022-09-02T04:42:08.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d590c29acf7724e-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.854 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.855 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.856 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.856 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.857 DEBUG:boto:Method: GET 2022-09-02T04:42:08.857 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.858 DEBUG:boto:Data: 2022-09-02T04:42:08.859 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.860 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.860 DEBUG:boto:Params: {} 2022-09-02T04:42:08.861 DEBUG:boto:Token: None 2022-09-02T04:42:08.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:08.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE= 2022-09-02T04:42:08.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QdnuA0k2OhN9ATLKhyWa2qZ+RJE=', 'Content-Length': '0'} 2022-09-02T04:42:08.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078c358c05ee3ed42-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.866 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.867 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.867 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.868 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.868 DEBUG:boto:Method: GET 2022-09-02T04:42:08.869 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.870 DEBUG:boto:Data: 2022-09-02T04:42:08.870 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.871 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.871 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.872 DEBUG:boto:Params: {} 2022-09-02T04:42:08.872 DEBUG:boto:Token: None 2022-09-02T04:42:08.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000494072ef2772d7d1-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.878 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.879 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:08.880 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:08.880 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.881 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:08.882 DEBUG:boto:Method: GET 2022-09-02T04:42:08.882 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:08.883 DEBUG:boto:Data: 2022-09-02T04:42:08.883 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.884 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.885 DEBUG:boto:Params: {} 2022-09-02T04:42:08.885 DEBUG:boto:Token: None 2022-09-02T04:42:08.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/ 2022-09-02T04:42:08.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo= 2022-09-02T04:42:08.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xarT8Z46soHV5ZCOODtjMdZcoKo=', 'Content-Length': '0'} 2022-09-02T04:42:08.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a48d3ee3d815f96-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.891 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:08.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.893 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.894 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.895 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.895 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.896 DEBUG:boto:Method: GET 2022-09-02T04:42:08.896 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.897 DEBUG:boto:Data: 2022-09-02T04:42:08.898 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.900 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.901 DEBUG:boto:Params: {} 2022-09-02T04:42:08.903 DEBUG:boto:Token: None 2022-09-02T04:42:08.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j7/JWAPunaK4qm2PiGhOLRbJpko= 2022-09-02T04:42:08.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j7/JWAPunaK4qm2PiGhOLRbJpko=', 'Content-Length': '0'} 2022-09-02T04:42:08.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c06fa872437c5cd4-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.915 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.916 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.916 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.917 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.919 DEBUG:boto:Method: GET 2022-09-02T04:42:08.920 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.920 DEBUG:boto:Data: 2022-09-02T04:42:08.921 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.923 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.924 DEBUG:boto:Params: {} 2022-09-02T04:42:08.925 DEBUG:boto:Token: None 2022-09-02T04:42:08.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j7/JWAPunaK4qm2PiGhOLRbJpko= 2022-09-02T04:42:08.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j7/JWAPunaK4qm2PiGhOLRbJpko=', 'Content-Length': '0'} 2022-09-02T04:42:08.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000858527f7bbf3888a-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.974 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.975 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.975 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.976 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.976 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.977 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.977 DEBUG:boto:Data: 2022-09-02T04:42:08.978 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:08.979 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.980 DEBUG:boto:Params: {} 2022-09-02T04:42:08.980 DEBUG:boto:Token: None 2022-09-02T04:42:08.981 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c8q/EyNelCF4KDDAjKe4eXI/SDM= 2022-09-02T04:42:08.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c8q/EyNelCF4KDDAjKe4eXI/SDM=', 'Content-Length': '0'} 2022-09-02T04:42:08.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f4b1c773b492bdb-00631189a0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.985 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.985 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:08.986 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.987 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.987 DEBUG:boto:Method: HEAD 2022-09-02T04:42:08.988 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.988 DEBUG:boto:Data: 2022-09-02T04:42:08.989 DEBUG:boto:Headers: {} 2022-09-02T04:42:08.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:08.990 DEBUG:boto:Port: 8000 2022-09-02T04:42:08.990 DEBUG:boto:Params: {} 2022-09-02T04:42:08.991 DEBUG:boto:Token: None 2022-09-02T04:42:08.991 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:08 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:08.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c8q/EyNelCF4KDDAjKe4eXI/SDM= 2022-09-02T04:42:08.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c8q/EyNelCF4KDDAjKe4eXI/SDM=', 'Content-Length': '0'} 2022-09-02T04:42:08.996 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc9c883a9ef860b6-00631189a0-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:08.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:08.998 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:08.999 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.002 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.003 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.004 DEBUG:boto:Method: GET 2022-09-02T04:42:09.005 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.006 DEBUG:boto:Data: 2022-09-02T04:42:09.007 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.008 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.009 DEBUG:boto:Params: {} 2022-09-02T04:42:09.012 DEBUG:boto:Token: None 2022-09-02T04:42:09.013 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JfXIjAcidhH64hLlhNwKNJaZLRo= 2022-09-02T04:42:09.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JfXIjAcidhH64hLlhNwKNJaZLRo=', 'Content-Length': '0'} 2022-09-02T04:42:09.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b7fd7093e001e3b-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.019 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.020 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.021 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.021 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.022 DEBUG:boto:Method: GET 2022-09-02T04:42:09.022 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.023 DEBUG:boto:Data: 2022-09-02T04:42:09.023 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.025 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.025 DEBUG:boto:Params: {} 2022-09-02T04:42:09.026 DEBUG:boto:Token: None 2022-09-02T04:42:09.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:09.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JfXIjAcidhH64hLlhNwKNJaZLRo= 2022-09-02T04:42:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JfXIjAcidhH64hLlhNwKNJaZLRo=', 'Content-Length': '0'} 2022-09-02T04:42:09.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fae1d1e813b43889-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.031 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.032 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.032 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.033 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.033 DEBUG:boto:Method: GET 2022-09-02T04:42:09.034 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.035 DEBUG:boto:Data: 2022-09-02T04:42:09.035 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.036 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.037 DEBUG:boto:Params: {} 2022-09-02T04:42:09.037 DEBUG:boto:Token: None 2022-09-02T04:42:09.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000508e4b5419af1b81-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.043 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.043 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.044 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.045 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.045 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.046 DEBUG:boto:Method: GET 2022-09-02T04:42:09.046 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.047 DEBUG:boto:Data: 2022-09-02T04:42:09.047 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.048 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.049 DEBUG:boto:Params: {} 2022-09-02T04:42:09.049 DEBUG:boto:Token: None 2022-09-02T04:42:09.050 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be2c394c9a1dc784-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.098 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.100 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.101 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.101 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.102 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:09.103 DEBUG:boto:Data: 2022-09-02T04:42:09.103 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.104 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.105 DEBUG:boto:Params: {} 2022-09-02T04:42:09.105 DEBUG:boto:Token: None 2022-09-02T04:42:09.106 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj 2022-09-02T04:42:09.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0= 2022-09-02T04:42:09.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0=', 'Content-Length': '0'} 2022-09-02T04:42:09.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a33b987e2e45430b-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.112 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.113 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.113 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.114 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:09.114 DEBUG:boto:Data: 2022-09-02T04:42:09.115 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.116 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.116 DEBUG:boto:Params: {} 2022-09-02T04:42:09.117 DEBUG:boto:Token: None 2022-09-02T04:42:09.117 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj 2022-09-02T04:42:09.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0= 2022-09-02T04:42:09.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0=', 'Content-Length': '0'} 2022-09-02T04:42:09.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b56b4b505899b7b-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:09.124 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.124 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.125 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.125 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.126 DEBUG:boto:Method: GET 2022-09-02T04:42:09.126 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.127 DEBUG:boto:Data: 2022-09-02T04:42:09.128 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.129 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.129 DEBUG:boto:Params: {} 2022-09-02T04:42:09.130 DEBUG:boto:Token: None 2022-09-02T04:42:09.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU= 2022-09-02T04:42:09.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU=', 'Content-Length': '0'} 2022-09-02T04:42:09.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eef24f8210be48cf-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.177 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.177 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.178 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.178 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.179 DEBUG:boto:Method: GET 2022-09-02T04:42:09.179 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.180 DEBUG:boto:Data: 2022-09-02T04:42:09.180 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.182 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.182 DEBUG:boto:Params: {} 2022-09-02T04:42:09.183 DEBUG:boto:Token: None 2022-09-02T04:42:09.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU= 2022-09-02T04:42:09.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU=', 'Content-Length': '0'} 2022-09-02T04:42:09.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000531a35ceef1dd6ac-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.188 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.188 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.189 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.189 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.190 DEBUG:boto:Method: GET 2022-09-02T04:42:09.191 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.191 DEBUG:boto:Data: 2022-09-02T04:42:09.192 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.193 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.193 DEBUG:boto:Params: {} 2022-09-02T04:42:09.194 DEBUG:boto:Token: None 2022-09-02T04:42:09.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001979e1293b73ae47-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.202 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.203 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.204 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.204 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.205 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.205 DEBUG:boto:Method: GET 2022-09-02T04:42:09.206 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.207 DEBUG:boto:Data: 2022-09-02T04:42:09.207 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.208 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.209 DEBUG:boto:Params: {} 2022-09-02T04:42:09.209 DEBUG:boto:Token: None 2022-09-02T04:42:09.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a7d17efeb697aaa-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.215 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:09.217 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.218 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.218 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.219 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.219 DEBUG:boto:Method: GET 2022-09-02T04:42:09.220 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.221 DEBUG:boto:Data: 2022-09-02T04:42:09.222 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.223 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.224 DEBUG:boto:Params: {} 2022-09-02T04:42:09.225 DEBUG:boto:Token: None 2022-09-02T04:42:09.226 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8= 2022-09-02T04:42:09.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8=', 'Content-Length': '0'} 2022-09-02T04:42:09.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005401991e19d1f96f-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.233 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.234 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.235 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.237 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.238 DEBUG:boto:Method: GET 2022-09-02T04:42:09.239 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.240 DEBUG:boto:Data: 2022-09-02T04:42:09.242 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.244 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.245 DEBUG:boto:Params: {} 2022-09-02T04:42:09.246 DEBUG:boto:Token: None 2022-09-02T04:42:09.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8= 2022-09-02T04:42:09.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8=', 'Content-Length': '0'} 2022-09-02T04:42:09.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc73404793388d5e-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.294 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.295 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.296 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.297 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.298 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.298 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.299 DEBUG:boto:Data: 2022-09-02T04:42:09.300 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.301 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.302 DEBUG:boto:Params: {} 2022-09-02T04:42:09.302 DEBUG:boto:Token: None 2022-09-02T04:42:09.303 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IAZ0+oEVo5W237nIYs5h8kLcmrM= 2022-09-02T04:42:09.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IAZ0+oEVo5W237nIYs5h8kLcmrM=', 'Content-Length': '0'} 2022-09-02T04:42:09.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000101ac277d93ff0db-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.308 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.309 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.310 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.311 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.312 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.312 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.313 DEBUG:boto:Data: 2022-09-02T04:42:09.313 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.314 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.315 DEBUG:boto:Params: {} 2022-09-02T04:42:09.315 DEBUG:boto:Token: None 2022-09-02T04:42:09.316 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IAZ0+oEVo5W237nIYs5h8kLcmrM= 2022-09-02T04:42:09.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IAZ0+oEVo5W237nIYs5h8kLcmrM=', 'Content-Length': '0'} 2022-09-02T04:42:09.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000957617e4225ef8c1-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:09.323 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.324 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.324 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.325 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.326 DEBUG:boto:Method: GET 2022-09-02T04:42:09.327 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.328 DEBUG:boto:Data: 2022-09-02T04:42:09.329 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.331 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.332 DEBUG:boto:Params: {} 2022-09-02T04:42:09.333 DEBUG:boto:Token: None 2022-09-02T04:42:09.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8= 2022-09-02T04:42:09.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8=', 'Content-Length': '0'} 2022-09-02T04:42:09.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000423c26116cdefe72-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.386 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.387 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.387 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.388 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.389 DEBUG:boto:Method: GET 2022-09-02T04:42:09.390 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.391 DEBUG:boto:Data: 2022-09-02T04:42:09.392 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.394 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.395 DEBUG:boto:Params: {} 2022-09-02T04:42:09.396 DEBUG:boto:Token: None 2022-09-02T04:42:09.397 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:09.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8= 2022-09-02T04:42:09.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qK1pRBFZetu0SyGWWE9g9tor/n8=', 'Content-Length': '0'} 2022-09-02T04:42:09.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b49641cd19ffba-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.405 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.406 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.407 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.408 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.408 DEBUG:boto:Method: GET 2022-09-02T04:42:09.409 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.410 DEBUG:boto:Data: 2022-09-02T04:42:09.411 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.413 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.414 DEBUG:boto:Params: {} 2022-09-02T04:42:09.415 DEBUG:boto:Token: None 2022-09-02T04:42:09.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aafc18736d64e876-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.423 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.426 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.427 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.428 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.428 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.429 DEBUG:boto:Method: GET 2022-09-02T04:42:09.430 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.432 DEBUG:boto:Data: 2022-09-02T04:42:09.433 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.436 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.437 DEBUG:boto:Params: {} 2022-09-02T04:42:09.438 DEBUG:boto:Token: None 2022-09-02T04:42:09.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499800da8e6160be-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.444 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.446 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.449 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.450 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.451 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:09.452 DEBUG:boto:Data: 2022-09-02T04:42:09.453 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.455 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.455 DEBUG:boto:Params: {} 2022-09-02T04:42:09.456 DEBUG:boto:Token: None 2022-09-02T04:42:09.457 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj 2022-09-02T04:42:09.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0= 2022-09-02T04:42:09.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0=', 'Content-Length': '0'} 2022-09-02T04:42:09.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097b46e1dae8395d3-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.468 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.469 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.470 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.472 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:09.473 DEBUG:boto:Data: 2022-09-02T04:42:09.473 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.476 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.477 DEBUG:boto:Params: {} 2022-09-02T04:42:09.478 DEBUG:boto:Token: None 2022-09-02T04:42:09.479 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj 2022-09-02T04:42:09.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0= 2022-09-02T04:42:09.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DHnVhPd8qvY+jxSE7txxO738Je0=', 'Content-Length': '0'} 2022-09-02T04:42:09.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093b47fdd8f9db193-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:09.490 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.491 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.492 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.493 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.493 DEBUG:boto:Method: GET 2022-09-02T04:42:09.495 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.497 DEBUG:boto:Data: 2022-09-02T04:42:09.498 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.500 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.501 DEBUG:boto:Params: {} 2022-09-02T04:42:09.502 DEBUG:boto:Token: None 2022-09-02T04:42:09.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU= 2022-09-02T04:42:09.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU=', 'Content-Length': '0'} 2022-09-02T04:42:09.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e748ee3d2d04a62-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.509 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:09.510 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:09.512 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.513 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.515 DEBUG:boto:Method: GET 2022-09-02T04:42:09.516 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.516 DEBUG:boto:Data: 2022-09-02T04:42:09.517 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.518 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.519 DEBUG:boto:Params: {} 2022-09-02T04:42:09.520 DEBUG:boto:Token: None 2022-09-02T04:42:09.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:09.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU= 2022-09-02T04:42:09.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nPo6Igxc2OnUl1hcgArqUc2H0uU=', 'Content-Length': '0'} 2022-09-02T04:42:09.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d3f739e6c82ed8f-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.527 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.528 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.529 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.529 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.530 DEBUG:boto:Method: GET 2022-09-02T04:42:09.530 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.531 DEBUG:boto:Data: 2022-09-02T04:42:09.531 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:09.532 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.533 DEBUG:boto:Params: {} 2022-09-02T04:42:09.534 DEBUG:boto:Token: None 2022-09-02T04:42:09.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2d764904abb75d8-00631189a1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.539 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.540 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.541 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.542 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.543 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:09.544 DEBUG:boto:Method: GET 2022-09-02T04:42:09.545 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:09.546 DEBUG:boto:Data: 2022-09-02T04:42:09.547 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.549 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.549 DEBUG:boto:Params: {} 2022-09-02T04:42:09.551 DEBUG:boto:Token: None 2022-09-02T04:42:09.552 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd448021436dcfe-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.558 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.559 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-19 zones={test-zone2, test-zone1} 2022-09-02T04:42:09.560 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-19 zones={test-zone3, test-zone1} 2022-09-02T04:42:09.560 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.561 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.561 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.562 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:42:09.562 DEBUG:boto:Data: 2022-09-02T04:42:09.563 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.564 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.564 DEBUG:boto:Params: {} 2022-09-02T04:42:09.565 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:42:09.566 DEBUG:boto:Token: None 2022-09-02T04:42:09.566 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hQwih0+YFr9TSvGo4J7zy7bb7IQ= 2022-09-02T04:42:09.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hQwih0+YFr9TSvGo4J7zy7bb7IQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c48f6619b2b5e9fa-00631189a1-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.577 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.578 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.579 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.580 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:42:09.581 DEBUG:boto:Data: 2022-09-02T04:42:09.582 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.583 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.584 DEBUG:boto:Params: {} 2022-09-02T04:42:09.585 DEBUG:boto:Token: None 2022-09-02T04:42:09.585 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hQwih0+YFr9TSvGo4J7zy7bb7IQ= 2022-09-02T04:42:09.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hQwih0+YFr9TSvGo4J7zy7bb7IQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000158d575c8e8fedfe-00631189a1-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:42:09.591 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.592 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.593 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.594 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.595 DEBUG:boto:Method: GET 2022-09-02T04:42:09.596 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.597 DEBUG:boto:Data: 2022-09-02T04:42:09.598 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.600 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.600 DEBUG:boto:Params: {} 2022-09-02T04:42:09.601 DEBUG:boto:Token: None 2022-09-02T04:42:09.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/?versions 2022-09-02T04:42:09.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10= 2022-09-02T04:42:09.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10=', 'Content-Length': '0'} 2022-09-02T04:42:09.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf8f2c22e33384f-00631189a1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.607 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.609 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:42:09.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.629 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:42:09.630 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.630 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.631 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.632 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.633 DEBUG:boto:Method: GET 2022-09-02T04:42:09.634 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.635 DEBUG:boto:Data: 2022-09-02T04:42:09.636 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.638 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.639 DEBUG:boto:Params: {} 2022-09-02T04:42:09.640 DEBUG:boto:Token: None 2022-09-02T04:42:09.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/?versions 2022-09-02T04:42:09.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10= 2022-09-02T04:42:09.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10=', 'Content-Length': '0'} 2022-09-02T04:42:09.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000470a796d652b719e-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.649 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.652 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:42:09.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:09.676 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.677 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.678 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.679 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.680 DEBUG:boto:Method: GET 2022-09-02T04:42:09.680 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.681 DEBUG:boto:Data: 2022-09-02T04:42:09.682 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.684 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.689 DEBUG:boto:Params: {} 2022-09-02T04:42:09.690 DEBUG:boto:Token: None 2022-09-02T04:42:09.691 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/?versions 2022-09-02T04:42:09.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10= 2022-09-02T04:42:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10=', 'Content-Length': '0'} 2022-09-02T04:42:09.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab62eb072914159-00631189a1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.699 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.703 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.705 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.706 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.708 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.709 DEBUG:boto:Method: GET 2022-09-02T04:42:09.712 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:09.713 DEBUG:boto:Data: 2022-09-02T04:42:09.715 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.716 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.718 DEBUG:boto:Params: {} 2022-09-02T04:42:09.719 DEBUG:boto:Token: None 2022-09-02T04:42:09.720 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/?versions 2022-09-02T04:42:09.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10= 2022-09-02T04:42:09.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5mUa2tuZFhWTrOkhkA2JhPwk10=', 'Content-Length': '0'} 2022-09-02T04:42:09.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1af6f31629fe061-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.732 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:09.737 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.738 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.739 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.740 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.741 DEBUG:boto:Method: GET 2022-09-02T04:42:09.742 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.743 DEBUG:boto:Data: 2022-09-02T04:42:09.743 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.745 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.746 DEBUG:boto:Params: {} 2022-09-02T04:42:09.747 DEBUG:boto:Token: None 2022-09-02T04:42:09.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE= 2022-09-02T04:42:09.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE=', 'Content-Length': '0'} 2022-09-02T04:42:09.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ccacacc206549882-00631189a1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.753 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.754 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.754 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.755 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.755 DEBUG:boto:Method: GET 2022-09-02T04:42:09.757 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.758 DEBUG:boto:Data: 2022-09-02T04:42:09.758 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.760 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.760 DEBUG:boto:Params: {} 2022-09-02T04:42:09.761 DEBUG:boto:Token: None 2022-09-02T04:42:09.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE= 2022-09-02T04:42:09.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE=', 'Content-Length': '0'} 2022-09-02T04:42:09.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd8ad92df748b49e-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.768 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.769 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.770 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.771 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.772 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.773 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.774 DEBUG:boto:Data: 2022-09-02T04:42:09.775 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.776 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.776 DEBUG:boto:Params: {} 2022-09-02T04:42:09.777 DEBUG:boto:Token: None 2022-09-02T04:42:09.777 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/7Bp5ncCpUzqdZZEsbjAnCm3eA8= 2022-09-02T04:42:09.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/7Bp5ncCpUzqdZZEsbjAnCm3eA8=', 'Content-Length': '0'} 2022-09-02T04:42:09.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f41dcab64f5bc8ca-00631189a1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.781 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.782 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.783 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.783 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.784 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.784 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.785 DEBUG:boto:Data: 2022-09-02T04:42:09.789 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.791 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.792 DEBUG:boto:Params: {} 2022-09-02T04:42:09.793 DEBUG:boto:Token: None 2022-09-02T04:42:09.794 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/7Bp5ncCpUzqdZZEsbjAnCm3eA8= 2022-09-02T04:42:09.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/7Bp5ncCpUzqdZZEsbjAnCm3eA8=', 'Content-Length': '0'} 2022-09-02T04:42:09.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000848c1743006855de-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:09.800 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.801 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.802 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.803 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.804 DEBUG:boto:Method: GET 2022-09-02T04:42:09.805 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.806 DEBUG:boto:Data: 2022-09-02T04:42:09.807 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.809 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.810 DEBUG:boto:Params: {} 2022-09-02T04:42:09.811 DEBUG:boto:Token: None 2022-09-02T04:42:09.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE= 2022-09-02T04:42:09.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE=', 'Content-Length': '0'} 2022-09-02T04:42:09.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a56efdc59dfea60-00631189a1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.820 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.821 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.822 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.823 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.824 DEBUG:boto:Method: GET 2022-09-02T04:42:09.825 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.826 DEBUG:boto:Data: 2022-09-02T04:42:09.827 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.828 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.829 DEBUG:boto:Params: {} 2022-09-02T04:42:09.830 DEBUG:boto:Token: None 2022-09-02T04:42:09.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE= 2022-09-02T04:42:09.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE=', 'Content-Length': '0'} 2022-09-02T04:42:09.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5368fef6d708482-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.835 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.835 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.836 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.837 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.837 DEBUG:boto:Method: GET 2022-09-02T04:42:09.838 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.839 DEBUG:boto:Data: 2022-09-02T04:42:09.839 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.841 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.842 DEBUG:boto:Params: {} 2022-09-02T04:42:09.843 DEBUG:boto:Token: None 2022-09-02T04:42:09.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7fcc7e0295cc45f-00631189a1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.851 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.853 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.854 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.855 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.856 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.856 DEBUG:boto:Method: GET 2022-09-02T04:42:09.857 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.858 DEBUG:boto:Data: 2022-09-02T04:42:09.859 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.861 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.862 DEBUG:boto:Params: {} 2022-09-02T04:42:09.863 DEBUG:boto:Token: None 2022-09-02T04:42:09.864 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/ 2022-09-02T04:42:09.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ= 2022-09-02T04:42:09.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/S7IqOFufUvRGJtyVjqOpN7EhQ=', 'Content-Length': '0'} 2022-09-02T04:42:09.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552d4efcce85aa19-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.914 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:09.916 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.917 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.918 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.919 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:42:09.920 DEBUG:boto:Data: 2022-09-02T04:42:09.921 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.923 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.924 DEBUG:boto:Params: {} 2022-09-02T04:42:09.925 DEBUG:boto:Token: None 2022-09-02T04:42:09.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy 2022-09-02T04:42:09.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VUtMrUjmCf2PVgjdOQ4k2EDOhUM= 2022-09-02T04:42:09.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VUtMrUjmCf2PVgjdOQ4k2EDOhUM=', 'Content-Length': '0'} 2022-09-02T04:42:09.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c0089811e86cb5f-00631189a1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.933 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.933 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.935 DEBUG:boto:Method: HEAD 2022-09-02T04:42:09.936 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:42:09.937 DEBUG:boto:Data: 2022-09-02T04:42:09.938 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.939 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.940 DEBUG:boto:Params: {} 2022-09-02T04:42:09.942 DEBUG:boto:Token: None 2022-09-02T04:42:09.942 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy 2022-09-02T04:42:09.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VUtMrUjmCf2PVgjdOQ4k2EDOhUM= 2022-09-02T04:42:09.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VUtMrUjmCf2PVgjdOQ4k2EDOhUM=', 'Content-Length': '0'} 2022-09-02T04:42:09.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f3a9104df23cbb7-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:09.949 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.950 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.950 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.951 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.952 DEBUG:boto:Method: GET 2022-09-02T04:42:09.952 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.953 DEBUG:boto:Data: 2022-09-02T04:42:09.953 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.955 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.956 DEBUG:boto:Params: {} 2022-09-02T04:42:09.957 DEBUG:boto:Token: None 2022-09-02T04:42:09.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE= 2022-09-02T04:42:09.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE=', 'Content-Length': '0'} 2022-09-02T04:42:09.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e4c059e28dc293b-00631189a1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.969 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:09.970 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:09.972 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.972 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.973 DEBUG:boto:Method: GET 2022-09-02T04:42:09.974 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.975 DEBUG:boto:Data: 2022-09-02T04:42:09.976 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:09.978 DEBUG:boto:Port: 8000 2022-09-02T04:42:09.979 DEBUG:boto:Params: {} 2022-09-02T04:42:09.980 DEBUG:boto:Token: None 2022-09-02T04:42:09.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:09 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:09.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE= 2022-09-02T04:42:09.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bNIdiQXJYKh9wtCo6KyBI8joGbE=', 'Content-Length': '0'} 2022-09-02T04:42:09.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000037e35c57fffa760f-00631189a1-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:09.988 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:09.989 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:09.990 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.992 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.993 DEBUG:boto:Method: GET 2022-09-02T04:42:09.993 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:09.994 DEBUG:boto:Data: 2022-09-02T04:42:09.995 DEBUG:boto:Headers: {} 2022-09-02T04:42:09.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:09.997 DEBUG:boto:Port: 8001 2022-09-02T04:42:09.998 DEBUG:boto:Params: {} 2022-09-02T04:42:09.999 DEBUG:boto:Token: None 2022-09-02T04:42:10.000 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4ea04c0ac173b9d-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.046 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.048 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.048 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.049 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:10.050 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:10.050 DEBUG:boto:Method: GET 2022-09-02T04:42:10.051 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:10.051 DEBUG:boto:Data: 2022-09-02T04:42:10.052 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.053 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.054 DEBUG:boto:Params: {} 2022-09-02T04:42:10.055 DEBUG:boto:Token: None 2022-09-02T04:42:10.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002002b85da521c7a8-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.060 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.062 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.062 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.063 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.063 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.064 DEBUG:boto:Method: GET 2022-09-02T04:42:10.065 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.065 DEBUG:boto:Data: 2022-09-02T04:42:10.066 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.067 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.067 DEBUG:boto:Params: {} 2022-09-02T04:42:10.068 DEBUG:boto:Token: None 2022-09-02T04:42:10.069 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b553fe6b657b54c2-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.076 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.077 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.078 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.078 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.079 DEBUG:boto:Method: GET 2022-09-02T04:42:10.080 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.081 DEBUG:boto:Data: 2022-09-02T04:42:10.082 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.083 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.083 DEBUG:boto:Params: {} 2022-09-02T04:42:10.085 DEBUG:boto:Token: None 2022-09-02T04:42:10.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3fb9e4888be4462-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.092 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.093 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.094 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.095 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.095 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.096 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.097 DEBUG:boto:Data: 2022-09-02T04:42:10.098 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.100 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.101 DEBUG:boto:Params: {} 2022-09-02T04:42:10.102 DEBUG:boto:Token: None 2022-09-02T04:42:10.103 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rO+KMDw1R9TsyCuyKjaEsCQDm+8= 2022-09-02T04:42:10.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rO+KMDw1R9TsyCuyKjaEsCQDm+8=', 'Content-Length': '0'} 2022-09-02T04:42:10.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079683a60ee0247d0-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.107 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.108 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.108 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.109 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.110 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.110 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.111 DEBUG:boto:Data: 2022-09-02T04:42:10.111 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.112 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.113 DEBUG:boto:Params: {} 2022-09-02T04:42:10.113 DEBUG:boto:Token: None 2022-09-02T04:42:10.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rO+KMDw1R9TsyCuyKjaEsCQDm+8= 2022-09-02T04:42:10.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rO+KMDw1R9TsyCuyKjaEsCQDm+8=', 'Content-Length': '0'} 2022-09-02T04:42:10.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c824acda1f10e2cc-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.119 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.120 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.120 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.121 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.121 DEBUG:boto:Method: GET 2022-09-02T04:42:10.122 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.122 DEBUG:boto:Data: 2022-09-02T04:42:10.123 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.124 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.124 DEBUG:boto:Params: {} 2022-09-02T04:42:10.125 DEBUG:boto:Token: None 2022-09-02T04:42:10.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d5136fd689ef3ae-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.170 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.171 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.171 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.172 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.173 DEBUG:boto:Method: GET 2022-09-02T04:42:10.173 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.174 DEBUG:boto:Data: 2022-09-02T04:42:10.174 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.175 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.176 DEBUG:boto:Params: {} 2022-09-02T04:42:10.176 DEBUG:boto:Token: None 2022-09-02T04:42:10.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062a3b302c7edb3a4-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.181 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.182 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.183 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.183 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.184 DEBUG:boto:Method: GET 2022-09-02T04:42:10.184 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.185 DEBUG:boto:Data: 2022-09-02T04:42:10.185 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.187 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.187 DEBUG:boto:Params: {} 2022-09-02T04:42:10.188 DEBUG:boto:Token: None 2022-09-02T04:42:10.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb07f4c7029259bd-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.194 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.194 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.195 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.196 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.196 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.197 DEBUG:boto:Method: GET 2022-09-02T04:42:10.197 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.198 DEBUG:boto:Data: 2022-09-02T04:42:10.198 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.199 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.200 DEBUG:boto:Params: {} 2022-09-02T04:42:10.201 DEBUG:boto:Token: None 2022-09-02T04:42:10.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e965ce5c100a7c22-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.206 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.207 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.207 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.208 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.209 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:10.209 DEBUG:boto:Data: 2022-09-02T04:42:10.209 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.211 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.211 DEBUG:boto:Params: {} 2022-09-02T04:42:10.212 DEBUG:boto:Token: None 2022-09-02T04:42:10.212 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj 2022-09-02T04:42:10.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8= 2022-09-02T04:42:10.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8=', 'Content-Length': '0'} 2022-09-02T04:42:10.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ebbb6c410cada7b-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.218 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.219 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.219 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.220 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:10.220 DEBUG:boto:Data: 2022-09-02T04:42:10.221 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.222 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.223 DEBUG:boto:Params: {} 2022-09-02T04:42:10.224 DEBUG:boto:Token: None 2022-09-02T04:42:10.224 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj 2022-09-02T04:42:10.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8= 2022-09-02T04:42:10.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8=', 'Content-Length': '0'} 2022-09-02T04:42:10.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000493396e20f209502-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.232 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.232 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.233 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.233 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.234 DEBUG:boto:Method: GET 2022-09-02T04:42:10.234 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.235 DEBUG:boto:Data: 2022-09-02T04:42:10.236 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.237 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.237 DEBUG:boto:Params: {} 2022-09-02T04:42:10.238 DEBUG:boto:Token: None 2022-09-02T04:42:10.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c5619c3801444ca-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.248 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.248 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.249 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.249 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.250 DEBUG:boto:Method: GET 2022-09-02T04:42:10.250 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.251 DEBUG:boto:Data: 2022-09-02T04:42:10.251 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.253 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.254 DEBUG:boto:Params: {} 2022-09-02T04:42:10.254 DEBUG:boto:Token: None 2022-09-02T04:42:10.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1b553b4c842e6db-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.260 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.260 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.261 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.261 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.262 DEBUG:boto:Method: GET 2022-09-02T04:42:10.262 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.263 DEBUG:boto:Data: 2022-09-02T04:42:10.263 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.264 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.265 DEBUG:boto:Params: {} 2022-09-02T04:42:10.266 DEBUG:boto:Token: None 2022-09-02T04:42:10.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000245d22562b7f1ac-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.271 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.272 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.273 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.273 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.274 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.274 DEBUG:boto:Method: GET 2022-09-02T04:42:10.275 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.275 DEBUG:boto:Data: 2022-09-02T04:42:10.276 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.277 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.277 DEBUG:boto:Params: {} 2022-09-02T04:42:10.278 DEBUG:boto:Token: None 2022-09-02T04:42:10.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3a4598127523b74-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.284 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.286 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.286 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.287 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.287 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.288 DEBUG:boto:Method: GET 2022-09-02T04:42:10.288 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.289 DEBUG:boto:Data: 2022-09-02T04:42:10.289 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.290 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.291 DEBUG:boto:Params: {} 2022-09-02T04:42:10.292 DEBUG:boto:Token: None 2022-09-02T04:42:10.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE= 2022-09-02T04:42:10.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE=', 'Content-Length': '0'} 2022-09-02T04:42:10.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb4c28fe1d00e260-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.338 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.339 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.341 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.341 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.342 DEBUG:boto:Method: GET 2022-09-02T04:42:10.343 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.344 DEBUG:boto:Data: 2022-09-02T04:42:10.345 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.347 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.347 DEBUG:boto:Params: {} 2022-09-02T04:42:10.348 DEBUG:boto:Token: None 2022-09-02T04:42:10.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE= 2022-09-02T04:42:10.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE=', 'Content-Length': '0'} 2022-09-02T04:42:10.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbd9f8d07757ff5b-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.354 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.354 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.355 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.356 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.356 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.357 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.357 DEBUG:boto:Data: 2022-09-02T04:42:10.358 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.359 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.360 DEBUG:boto:Params: {} 2022-09-02T04:42:10.360 DEBUG:boto:Token: None 2022-09-02T04:42:10.361 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P/PZwO7kYQTjxGbBPo4xYRoNNxI= 2022-09-02T04:42:10.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P/PZwO7kYQTjxGbBPo4xYRoNNxI=', 'Content-Length': '0'} 2022-09-02T04:42:10.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b05c1b241cabfa7-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.365 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.366 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.366 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.367 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.368 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.368 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.369 DEBUG:boto:Data: 2022-09-02T04:42:10.369 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.370 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.371 DEBUG:boto:Params: {} 2022-09-02T04:42:10.371 DEBUG:boto:Token: None 2022-09-02T04:42:10.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P/PZwO7kYQTjxGbBPo4xYRoNNxI= 2022-09-02T04:42:10.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P/PZwO7kYQTjxGbBPo4xYRoNNxI=', 'Content-Length': '0'} 2022-09-02T04:42:10.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4dc531a663909ce-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.377 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.377 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.378 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.378 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.379 DEBUG:boto:Method: GET 2022-09-02T04:42:10.380 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.380 DEBUG:boto:Data: 2022-09-02T04:42:10.381 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.382 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.382 DEBUG:boto:Params: {} 2022-09-02T04:42:10.383 DEBUG:boto:Token: None 2022-09-02T04:42:10.383 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE= 2022-09-02T04:42:10.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE=', 'Content-Length': '0'} 2022-09-02T04:42:10.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d3af9138081495b-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.388 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.388 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.389 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.389 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.390 DEBUG:boto:Method: GET 2022-09-02T04:42:10.391 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.391 DEBUG:boto:Data: 2022-09-02T04:42:10.392 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.393 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.393 DEBUG:boto:Params: {} 2022-09-02T04:42:10.394 DEBUG:boto:Token: None 2022-09-02T04:42:10.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:10.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE= 2022-09-02T04:42:10.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/m8XcFZp1mv8yP0M+CKMWQJXOHE=', 'Content-Length': '0'} 2022-09-02T04:42:10.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000100f5b6500356c51-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.401 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.402 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.403 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.403 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.404 DEBUG:boto:Method: GET 2022-09-02T04:42:10.405 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.406 DEBUG:boto:Data: 2022-09-02T04:42:10.407 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.410 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.410 DEBUG:boto:Params: {} 2022-09-02T04:42:10.411 DEBUG:boto:Token: None 2022-09-02T04:42:10.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f9356ce8608463-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.416 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.418 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.418 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.419 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.420 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.420 DEBUG:boto:Method: GET 2022-09-02T04:42:10.421 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.421 DEBUG:boto:Data: 2022-09-02T04:42:10.422 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.423 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.424 DEBUG:boto:Params: {} 2022-09-02T04:42:10.424 DEBUG:boto:Token: None 2022-09-02T04:42:10.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2d738c93448ffea-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.435 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.437 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.438 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.439 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.439 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:10.440 DEBUG:boto:Data: 2022-09-02T04:42:10.441 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.444 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.445 DEBUG:boto:Params: {} 2022-09-02T04:42:10.446 DEBUG:boto:Token: None 2022-09-02T04:42:10.447 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj 2022-09-02T04:42:10.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8= 2022-09-02T04:42:10.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8=', 'Content-Length': '0'} 2022-09-02T04:42:10.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001df4740dc00d166c-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.453 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.454 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.455 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.456 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:10.457 DEBUG:boto:Data: 2022-09-02T04:42:10.457 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.459 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.460 DEBUG:boto:Params: {} 2022-09-02T04:42:10.461 DEBUG:boto:Token: None 2022-09-02T04:42:10.462 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj 2022-09-02T04:42:10.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8= 2022-09-02T04:42:10.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8=', 'Content-Length': '0'} 2022-09-02T04:42:10.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004619220b1fb2509-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.470 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.471 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.471 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.472 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.473 DEBUG:boto:Method: GET 2022-09-02T04:42:10.473 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.474 DEBUG:boto:Data: 2022-09-02T04:42:10.474 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.475 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.476 DEBUG:boto:Params: {} 2022-09-02T04:42:10.477 DEBUG:boto:Token: None 2022-09-02T04:42:10.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000caeb014be99ba499-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.522 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.523 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.523 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.524 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.525 DEBUG:boto:Method: GET 2022-09-02T04:42:10.525 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.526 DEBUG:boto:Data: 2022-09-02T04:42:10.526 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.527 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.528 DEBUG:boto:Params: {} 2022-09-02T04:42:10.529 DEBUG:boto:Token: None 2022-09-02T04:42:10.529 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eee475fdf66edc99-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.534 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.535 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.535 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.536 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.536 DEBUG:boto:Method: GET 2022-09-02T04:42:10.537 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.537 DEBUG:boto:Data: 2022-09-02T04:42:10.538 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.539 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.540 DEBUG:boto:Params: {} 2022-09-02T04:42:10.540 DEBUG:boto:Token: None 2022-09-02T04:42:10.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004761d8dd344316b2-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.546 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.547 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.547 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.548 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.549 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.549 DEBUG:boto:Method: GET 2022-09-02T04:42:10.550 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.550 DEBUG:boto:Data: 2022-09-02T04:42:10.551 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.552 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.553 DEBUG:boto:Params: {} 2022-09-02T04:42:10.553 DEBUG:boto:Token: None 2022-09-02T04:42:10.554 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc091bc183d9df17-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.558 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.560 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.560 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.561 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.561 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.562 DEBUG:boto:Method: GET 2022-09-02T04:42:10.563 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.563 DEBUG:boto:Data: 2022-09-02T04:42:10.564 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.565 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.565 DEBUG:boto:Params: {} 2022-09-02T04:42:10.566 DEBUG:boto:Token: None 2022-09-02T04:42:10.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M= 2022-09-02T04:42:10.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M=', 'Content-Length': '0'} 2022-09-02T04:42:10.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000910c8353d4d4529b-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.572 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.573 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.574 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.575 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.576 DEBUG:boto:Method: GET 2022-09-02T04:42:10.576 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.577 DEBUG:boto:Data: 2022-09-02T04:42:10.578 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.580 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.581 DEBUG:boto:Params: {} 2022-09-02T04:42:10.582 DEBUG:boto:Token: None 2022-09-02T04:42:10.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M= 2022-09-02T04:42:10.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M=', 'Content-Length': '0'} 2022-09-02T04:42:10.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8fe7a011201886a-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.595 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.595 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.596 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.597 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.597 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.598 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.598 DEBUG:boto:Data: 2022-09-02T04:42:10.599 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.600 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.600 DEBUG:boto:Params: {} 2022-09-02T04:42:10.601 DEBUG:boto:Token: None 2022-09-02T04:42:10.602 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uigfbC7vZ+B7UswAz2OzEMPQfJs= 2022-09-02T04:42:10.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uigfbC7vZ+B7UswAz2OzEMPQfJs=', 'Content-Length': '0'} 2022-09-02T04:42:10.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000022115ed8bc922fc-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.606 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.607 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.608 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.608 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.609 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.609 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.610 DEBUG:boto:Data: 2022-09-02T04:42:10.610 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.611 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.612 DEBUG:boto:Params: {} 2022-09-02T04:42:10.613 DEBUG:boto:Token: None 2022-09-02T04:42:10.614 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uigfbC7vZ+B7UswAz2OzEMPQfJs= 2022-09-02T04:42:10.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uigfbC7vZ+B7UswAz2OzEMPQfJs=', 'Content-Length': '0'} 2022-09-02T04:42:10.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0032e22818ac5cc-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.619 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.619 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.620 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.621 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.622 DEBUG:boto:Method: GET 2022-09-02T04:42:10.623 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.623 DEBUG:boto:Data: 2022-09-02T04:42:10.624 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.625 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.625 DEBUG:boto:Params: {} 2022-09-02T04:42:10.626 DEBUG:boto:Token: None 2022-09-02T04:42:10.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M= 2022-09-02T04:42:10.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M=', 'Content-Length': '0'} 2022-09-02T04:42:10.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f53bd1f7ffc0675e-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.631 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.632 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.633 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.634 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.634 DEBUG:boto:Method: GET 2022-09-02T04:42:10.635 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.636 DEBUG:boto:Data: 2022-09-02T04:42:10.637 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.638 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.639 DEBUG:boto:Params: {} 2022-09-02T04:42:10.640 DEBUG:boto:Token: None 2022-09-02T04:42:10.640 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:10.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M= 2022-09-02T04:42:10.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LM03zofVTk6SWusudjLVTyIP16M=', 'Content-Length': '0'} 2022-09-02T04:42:10.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb8b527ba17e6b78-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.645 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.646 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.647 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.647 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.648 DEBUG:boto:Method: GET 2022-09-02T04:42:10.649 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.649 DEBUG:boto:Data: 2022-09-02T04:42:10.650 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.651 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.651 DEBUG:boto:Params: {} 2022-09-02T04:42:10.652 DEBUG:boto:Token: None 2022-09-02T04:42:10.653 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029ef565f186242c5-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.701 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.703 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.704 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.705 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.706 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.707 DEBUG:boto:Method: GET 2022-09-02T04:42:10.708 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.708 DEBUG:boto:Data: 2022-09-02T04:42:10.709 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.710 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.711 DEBUG:boto:Params: {} 2022-09-02T04:42:10.712 DEBUG:boto:Token: None 2022-09-02T04:42:10.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073b38be7d27fe7fc-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.718 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.719 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.720 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.721 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.722 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:10.722 DEBUG:boto:Data: 2022-09-02T04:42:10.723 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.724 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.725 DEBUG:boto:Params: {} 2022-09-02T04:42:10.725 DEBUG:boto:Token: None 2022-09-02T04:42:10.726 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj 2022-09-02T04:42:10.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8= 2022-09-02T04:42:10.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8=', 'Content-Length': '0'} 2022-09-02T04:42:10.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2af3cd092c9ae5d-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.732 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.732 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.734 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.734 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:10.735 DEBUG:boto:Data: 2022-09-02T04:42:10.736 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.738 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.739 DEBUG:boto:Params: {} 2022-09-02T04:42:10.740 DEBUG:boto:Token: None 2022-09-02T04:42:10.741 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj 2022-09-02T04:42:10.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8= 2022-09-02T04:42:10.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKQJacNs1KlTxqywNIkxoPIbLj8=', 'Content-Length': '0'} 2022-09-02T04:42:10.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000def80510de4c9f1c-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.751 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.752 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.752 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.753 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.754 DEBUG:boto:Method: GET 2022-09-02T04:42:10.755 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.756 DEBUG:boto:Data: 2022-09-02T04:42:10.757 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.759 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.760 DEBUG:boto:Params: {} 2022-09-02T04:42:10.761 DEBUG:boto:Token: None 2022-09-02T04:42:10.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aef77f96b9d7a5a9-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.770 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.771 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.772 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.773 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.774 DEBUG:boto:Method: GET 2022-09-02T04:42:10.775 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.776 DEBUG:boto:Data: 2022-09-02T04:42:10.777 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.779 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.780 DEBUG:boto:Params: {} 2022-09-02T04:42:10.781 DEBUG:boto:Token: None 2022-09-02T04:42:10.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:10.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU= 2022-09-02T04:42:10.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XGWMtHqpw19jnOjQSAx9k0OkmeU=', 'Content-Length': '0'} 2022-09-02T04:42:10.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000e957732b3362d3-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.788 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.788 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.789 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.789 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.790 DEBUG:boto:Method: GET 2022-09-02T04:42:10.791 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.791 DEBUG:boto:Data: 2022-09-02T04:42:10.792 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.793 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.793 DEBUG:boto:Params: {} 2022-09-02T04:42:10.794 DEBUG:boto:Token: None 2022-09-02T04:42:10.795 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f842f2340552cbd-00631189a2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.801 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.802 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.802 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.803 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.803 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.804 DEBUG:boto:Method: GET 2022-09-02T04:42:10.805 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.805 DEBUG:boto:Data: 2022-09-02T04:42:10.806 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.807 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.807 DEBUG:boto:Params: {} 2022-09-02T04:42:10.808 DEBUG:boto:Token: None 2022-09-02T04:42:10.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:10.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec5e9b9ebd3953b-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.815 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:10.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.816 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.817 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.817 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.818 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.819 DEBUG:boto:Method: GET 2022-09-02T04:42:10.819 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.820 DEBUG:boto:Data: 2022-09-02T04:42:10.820 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.821 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.822 DEBUG:boto:Params: {} 2022-09-02T04:42:10.822 DEBUG:boto:Token: None 2022-09-02T04:42:10.823 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM= 2022-09-02T04:42:10.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM=', 'Content-Length': '0'} 2022-09-02T04:42:10.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000118886006e095147-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.874 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.875 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.875 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.876 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.876 DEBUG:boto:Method: GET 2022-09-02T04:42:10.877 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.878 DEBUG:boto:Data: 2022-09-02T04:42:10.878 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.880 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.881 DEBUG:boto:Params: {} 2022-09-02T04:42:10.881 DEBUG:boto:Token: None 2022-09-02T04:42:10.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM= 2022-09-02T04:42:10.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM=', 'Content-Length': '0'} 2022-09-02T04:42:10.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8de6aa440dfcbde-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.886 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.887 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.887 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.888 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.888 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.889 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.889 DEBUG:boto:Data: 2022-09-02T04:42:10.890 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.891 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.891 DEBUG:boto:Params: {} 2022-09-02T04:42:10.892 DEBUG:boto:Token: None 2022-09-02T04:42:10.893 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rHJEXPFzB8J9NuUb+dFDHJeuo/U= 2022-09-02T04:42:10.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rHJEXPFzB8J9NuUb+dFDHJeuo/U=', 'Content-Length': '0'} 2022-09-02T04:42:10.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003484f29e3695f67c-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.898 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.899 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.900 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.901 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.902 DEBUG:boto:Method: HEAD 2022-09-02T04:42:10.903 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.904 DEBUG:boto:Data: 2022-09-02T04:42:10.904 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.907 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.907 DEBUG:boto:Params: {} 2022-09-02T04:42:10.908 DEBUG:boto:Token: None 2022-09-02T04:42:10.910 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rHJEXPFzB8J9NuUb+dFDHJeuo/U= 2022-09-02T04:42:10.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rHJEXPFzB8J9NuUb+dFDHJeuo/U=', 'Content-Length': '0'} 2022-09-02T04:42:10.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005aad78067a9168fc-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:10.917 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.918 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.919 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.919 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.920 DEBUG:boto:Method: GET 2022-09-02T04:42:10.920 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.921 DEBUG:boto:Data: 2022-09-02T04:42:10.921 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.922 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.923 DEBUG:boto:Params: {} 2022-09-02T04:42:10.924 DEBUG:boto:Token: None 2022-09-02T04:42:10.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM= 2022-09-02T04:42:10.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM=', 'Content-Length': '0'} 2022-09-02T04:42:10.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002174ee6501cb3dcf-00631189a2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.929 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:10.929 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:10.930 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.930 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.931 DEBUG:boto:Method: GET 2022-09-02T04:42:10.931 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.932 DEBUG:boto:Data: 2022-09-02T04:42:10.933 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:10.934 DEBUG:boto:Port: 8000 2022-09-02T04:42:10.934 DEBUG:boto:Params: {} 2022-09-02T04:42:10.935 DEBUG:boto:Token: None 2022-09-02T04:42:10.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:10.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM= 2022-09-02T04:42:10.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uHQD3f80DeIRVZRx7C3PSFS2gM=', 'Content-Length': '0'} 2022-09-02T04:42:10.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ebd1c866844dc9f-00631189a2-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:10.986 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:10.987 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:10.988 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.989 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:10.989 DEBUG:boto:Method: GET 2022-09-02T04:42:10.990 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:10.991 DEBUG:boto:Data: 2022-09-02T04:42:10.991 DEBUG:boto:Headers: {} 2022-09-02T04:42:10.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:10.993 DEBUG:boto:Port: 8001 2022-09-02T04:42:10.993 DEBUG:boto:Params: {} 2022-09-02T04:42:10.994 DEBUG:boto:Token: None 2022-09-02T04:42:10.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:10 GMT /xljbkf-19/ 2022-09-02T04:42:10.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc= 2022-09-02T04:42:10.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tKl3vD3cxdqqE/995uWdaeFLfhc=', 'Content-Length': '0'} 2022-09-02T04:42:11.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000032233106804377c-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.002 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.004 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.004 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.005 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.006 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.007 DEBUG:boto:Method: GET 2022-09-02T04:42:11.008 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.009 DEBUG:boto:Data: 2022-09-02T04:42:11.009 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.010 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.011 DEBUG:boto:Params: {} 2022-09-02T04:42:11.012 DEBUG:boto:Token: None 2022-09-02T04:42:11.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420991cefc625d2b-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.017 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.018 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.019 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.019 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.020 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.021 DEBUG:boto:Data: 2022-09-02T04:42:11.022 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.023 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.023 DEBUG:boto:Params: {} 2022-09-02T04:42:11.024 DEBUG:boto:Token: None 2022-09-02T04:42:11.025 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c172eecbbb6e803d-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.032 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.033 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.033 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.034 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.035 DEBUG:boto:Data: 2022-09-02T04:42:11.036 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.037 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.038 DEBUG:boto:Params: {} 2022-09-02T04:42:11.039 DEBUG:boto:Token: None 2022-09-02T04:42:11.040 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072d8b7d998bb2f6a-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.047 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.048 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.049 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.049 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.050 DEBUG:boto:Method: GET 2022-09-02T04:42:11.051 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.052 DEBUG:boto:Data: 2022-09-02T04:42:11.053 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.054 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.055 DEBUG:boto:Params: {} 2022-09-02T04:42:11.056 DEBUG:boto:Token: None 2022-09-02T04:42:11.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e50a13b2dd01c62a-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.061 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.061 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.062 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.063 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.064 DEBUG:boto:Method: GET 2022-09-02T04:42:11.065 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.066 DEBUG:boto:Data: 2022-09-02T04:42:11.067 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.069 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.071 DEBUG:boto:Params: {} 2022-09-02T04:42:11.072 DEBUG:boto:Token: None 2022-09-02T04:42:11.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000950e60f2f1c6c3ad-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.079 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.080 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.080 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.081 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.081 DEBUG:boto:Method: GET 2022-09-02T04:42:11.082 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.082 DEBUG:boto:Data: 2022-09-02T04:42:11.083 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.084 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.084 DEBUG:boto:Params: {} 2022-09-02T04:42:11.085 DEBUG:boto:Token: None 2022-09-02T04:42:11.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031f44093eda71d51-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.091 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.092 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.092 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.093 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.094 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.095 DEBUG:boto:Method: GET 2022-09-02T04:42:11.096 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.097 DEBUG:boto:Data: 2022-09-02T04:42:11.098 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.099 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.100 DEBUG:boto:Params: {} 2022-09-02T04:42:11.101 DEBUG:boto:Token: None 2022-09-02T04:42:11.101 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fef6387db23185f-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.150 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.152 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.153 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.153 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.154 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.155 DEBUG:boto:Method: GET 2022-09-02T04:42:11.156 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.157 DEBUG:boto:Data: 2022-09-02T04:42:11.158 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.159 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.160 DEBUG:boto:Params: {} 2022-09-02T04:42:11.161 DEBUG:boto:Token: None 2022-09-02T04:42:11.161 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8= 2022-09-02T04:42:11.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8=', 'Content-Length': '0'} 2022-09-02T04:42:11.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007c580ca4ffee202-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.166 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.167 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.168 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.169 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.170 DEBUG:boto:Method: GET 2022-09-02T04:42:11.170 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.171 DEBUG:boto:Data: 2022-09-02T04:42:11.172 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.173 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.173 DEBUG:boto:Params: {} 2022-09-02T04:42:11.174 DEBUG:boto:Token: None 2022-09-02T04:42:11.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8= 2022-09-02T04:42:11.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8=', 'Content-Length': '0'} 2022-09-02T04:42:11.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cbab580d8dbe698-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.179 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.180 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.181 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.182 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.183 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.184 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.185 DEBUG:boto:Data: 2022-09-02T04:42:11.186 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.187 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.188 DEBUG:boto:Params: {} 2022-09-02T04:42:11.189 DEBUG:boto:Token: None 2022-09-02T04:42:11.190 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MfwogJsayo5yfXllaXjzonIFiY8= 2022-09-02T04:42:11.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MfwogJsayo5yfXllaXjzonIFiY8=', 'Content-Length': '0'} 2022-09-02T04:42:11.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9cf651016c5f17-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.194 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.195 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.195 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.196 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.196 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.197 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.198 DEBUG:boto:Data: 2022-09-02T04:42:11.198 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.199 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.200 DEBUG:boto:Params: {} 2022-09-02T04:42:11.200 DEBUG:boto:Token: None 2022-09-02T04:42:11.201 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MfwogJsayo5yfXllaXjzonIFiY8= 2022-09-02T04:42:11.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MfwogJsayo5yfXllaXjzonIFiY8=', 'Content-Length': '0'} 2022-09-02T04:42:11.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac7a578d762d7582-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.206 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.206 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.207 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.208 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.208 DEBUG:boto:Method: GET 2022-09-02T04:42:11.209 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.209 DEBUG:boto:Data: 2022-09-02T04:42:11.210 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.211 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.212 DEBUG:boto:Params: {} 2022-09-02T04:42:11.213 DEBUG:boto:Token: None 2022-09-02T04:42:11.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8= 2022-09-02T04:42:11.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8=', 'Content-Length': '0'} 2022-09-02T04:42:11.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005506cad10dc0f139-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.218 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.219 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.220 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.221 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.221 DEBUG:boto:Method: GET 2022-09-02T04:42:11.222 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.222 DEBUG:boto:Data: 2022-09-02T04:42:11.223 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.224 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.225 DEBUG:boto:Params: {} 2022-09-02T04:42:11.226 DEBUG:boto:Token: None 2022-09-02T04:42:11.226 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:11.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8= 2022-09-02T04:42:11.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UAVMc72WE//XPUQBWpEy4iKB3n8=', 'Content-Length': '0'} 2022-09-02T04:42:11.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000693a3952b62b7d86-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.232 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.233 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.233 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.234 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.235 DEBUG:boto:Method: GET 2022-09-02T04:42:11.235 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.236 DEBUG:boto:Data: 2022-09-02T04:42:11.236 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.237 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.238 DEBUG:boto:Params: {} 2022-09-02T04:42:11.239 DEBUG:boto:Token: None 2022-09-02T04:42:11.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cdabae2a4dc3bce-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.246 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.247 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.247 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.248 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.248 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.249 DEBUG:boto:Method: GET 2022-09-02T04:42:11.249 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.250 DEBUG:boto:Data: 2022-09-02T04:42:11.250 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.252 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.252 DEBUG:boto:Params: {} 2022-09-02T04:42:11.253 DEBUG:boto:Token: None 2022-09-02T04:42:11.254 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000248b45bb54f181d5-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.259 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.260 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.261 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.261 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.262 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.262 DEBUG:boto:Data: 2022-09-02T04:42:11.263 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.264 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.265 DEBUG:boto:Params: {} 2022-09-02T04:42:11.266 DEBUG:boto:Token: None 2022-09-02T04:42:11.266 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040f37bc55eb8df29-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.272 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.273 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.274 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.274 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.275 DEBUG:boto:Data: 2022-09-02T04:42:11.275 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.277 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.277 DEBUG:boto:Params: {} 2022-09-02T04:42:11.278 DEBUG:boto:Token: None 2022-09-02T04:42:11.279 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b60eb2a5a5ad1c45-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.285 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.286 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.287 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.288 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.288 DEBUG:boto:Method: GET 2022-09-02T04:42:11.289 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.289 DEBUG:boto:Data: 2022-09-02T04:42:11.290 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.291 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.292 DEBUG:boto:Params: {} 2022-09-02T04:42:11.293 DEBUG:boto:Token: None 2022-09-02T04:42:11.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006afbbe36bf1c953b-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.300 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.301 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.302 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.303 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.303 DEBUG:boto:Method: GET 2022-09-02T04:42:11.304 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.305 DEBUG:boto:Data: 2022-09-02T04:42:11.306 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.307 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.307 DEBUG:boto:Params: {} 2022-09-02T04:42:11.308 DEBUG:boto:Token: None 2022-09-02T04:42:11.309 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a65a213a368c88-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.355 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.355 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.356 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.357 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.357 DEBUG:boto:Method: GET 2022-09-02T04:42:11.358 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.359 DEBUG:boto:Data: 2022-09-02T04:42:11.360 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.362 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.363 DEBUG:boto:Params: {} 2022-09-02T04:42:11.364 DEBUG:boto:Token: None 2022-09-02T04:42:11.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2bbc81421acc5e4-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.370 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.371 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.371 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.373 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.373 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.374 DEBUG:boto:Method: GET 2022-09-02T04:42:11.374 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.375 DEBUG:boto:Data: 2022-09-02T04:42:11.376 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.377 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.378 DEBUG:boto:Params: {} 2022-09-02T04:42:11.378 DEBUG:boto:Token: None 2022-09-02T04:42:11.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bafce8cdf83f8289-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.385 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.387 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.387 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.388 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.388 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.389 DEBUG:boto:Method: GET 2022-09-02T04:42:11.391 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.391 DEBUG:boto:Data: 2022-09-02T04:42:11.392 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.393 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.393 DEBUG:boto:Params: {} 2022-09-02T04:42:11.394 DEBUG:boto:Token: None 2022-09-02T04:42:11.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ= 2022-09-02T04:42:11.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ=', 'Content-Length': '0'} 2022-09-02T04:42:11.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e70f5299a1cc39c-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.399 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.400 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.400 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.401 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.402 DEBUG:boto:Method: GET 2022-09-02T04:42:11.402 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.403 DEBUG:boto:Data: 2022-09-02T04:42:11.404 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.405 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.405 DEBUG:boto:Params: {} 2022-09-02T04:42:11.406 DEBUG:boto:Token: None 2022-09-02T04:42:11.406 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ= 2022-09-02T04:42:11.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ=', 'Content-Length': '0'} 2022-09-02T04:42:11.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f2a6b4437287783-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.412 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.412 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.413 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.413 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.414 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.414 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.415 DEBUG:boto:Data: 2022-09-02T04:42:11.415 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.416 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.417 DEBUG:boto:Params: {} 2022-09-02T04:42:11.418 DEBUG:boto:Token: None 2022-09-02T04:42:11.418 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aqNEooxt34jA36GRSOWLr8Dgi3s= 2022-09-02T04:42:11.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aqNEooxt34jA36GRSOWLr8Dgi3s=', 'Content-Length': '0'} 2022-09-02T04:42:11.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000233f7cdb2d3ecfc1-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.422 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.423 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.423 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.424 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.424 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.425 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.425 DEBUG:boto:Data: 2022-09-02T04:42:11.426 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.427 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.427 DEBUG:boto:Params: {} 2022-09-02T04:42:11.428 DEBUG:boto:Token: None 2022-09-02T04:42:11.429 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aqNEooxt34jA36GRSOWLr8Dgi3s= 2022-09-02T04:42:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aqNEooxt34jA36GRSOWLr8Dgi3s=', 'Content-Length': '0'} 2022-09-02T04:42:11.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e1ce26f3c3945bd-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.433 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.434 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.434 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.435 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.435 DEBUG:boto:Method: GET 2022-09-02T04:42:11.436 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.437 DEBUG:boto:Data: 2022-09-02T04:42:11.437 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.438 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.439 DEBUG:boto:Params: {} 2022-09-02T04:42:11.439 DEBUG:boto:Token: None 2022-09-02T04:42:11.440 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.441 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ= 2022-09-02T04:42:11.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ=', 'Content-Length': '0'} 2022-09-02T04:42:11.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb6d5dfa04b7dbe3-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.486 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.487 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.487 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.488 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.489 DEBUG:boto:Method: GET 2022-09-02T04:42:11.489 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.490 DEBUG:boto:Data: 2022-09-02T04:42:11.490 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.491 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.492 DEBUG:boto:Params: {} 2022-09-02T04:42:11.493 DEBUG:boto:Token: None 2022-09-02T04:42:11.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:11.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ= 2022-09-02T04:42:11.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DR+i8HL4s6gE8qn56uGi6/iJpcQ=', 'Content-Length': '0'} 2022-09-02T04:42:11.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c20e3ed9626224e-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.498 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.499 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.499 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.500 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.500 DEBUG:boto:Method: GET 2022-09-02T04:42:11.501 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.501 DEBUG:boto:Data: 2022-09-02T04:42:11.502 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.503 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.504 DEBUG:boto:Params: {} 2022-09-02T04:42:11.505 DEBUG:boto:Token: None 2022-09-02T04:42:11.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ea72faa49d1baa-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.510 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.511 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.512 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.512 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.513 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.514 DEBUG:boto:Method: GET 2022-09-02T04:42:11.514 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.515 DEBUG:boto:Data: 2022-09-02T04:42:11.515 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.517 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.517 DEBUG:boto:Params: {} 2022-09-02T04:42:11.518 DEBUG:boto:Token: None 2022-09-02T04:42:11.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000610cba2190ee9df9-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.524 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.524 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.525 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.526 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.526 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.527 DEBUG:boto:Data: 2022-09-02T04:42:11.528 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.529 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.529 DEBUG:boto:Params: {} 2022-09-02T04:42:11.530 DEBUG:boto:Token: None 2022-09-02T04:42:11.530 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa0a37d5818a36d3-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.537 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.538 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.539 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.540 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.541 DEBUG:boto:Data: 2022-09-02T04:42:11.543 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.544 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.545 DEBUG:boto:Params: {} 2022-09-02T04:42:11.546 DEBUG:boto:Token: None 2022-09-02T04:42:11.547 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acaa8de62fec74c8-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.553 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.554 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.554 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.555 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.555 DEBUG:boto:Method: GET 2022-09-02T04:42:11.556 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.556 DEBUG:boto:Data: 2022-09-02T04:42:11.557 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.558 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.559 DEBUG:boto:Params: {} 2022-09-02T04:42:11.560 DEBUG:boto:Token: None 2022-09-02T04:42:11.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4d084aa582d13fa-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.565 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.566 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.566 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.567 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.567 DEBUG:boto:Method: GET 2022-09-02T04:42:11.568 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.569 DEBUG:boto:Data: 2022-09-02T04:42:11.570 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.571 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.572 DEBUG:boto:Params: {} 2022-09-02T04:42:11.573 DEBUG:boto:Token: None 2022-09-02T04:42:11.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c884e6b2f87ba6e9-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.622 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.624 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.625 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.626 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.627 DEBUG:boto:Method: GET 2022-09-02T04:42:11.628 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.629 DEBUG:boto:Data: 2022-09-02T04:42:11.630 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.632 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.633 DEBUG:boto:Params: {} 2022-09-02T04:42:11.633 DEBUG:boto:Token: None 2022-09-02T04:42:11.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039dd545bfc26353e-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.639 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.640 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.640 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.641 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.642 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.642 DEBUG:boto:Method: GET 2022-09-02T04:42:11.643 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.643 DEBUG:boto:Data: 2022-09-02T04:42:11.644 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.645 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.646 DEBUG:boto:Params: {} 2022-09-02T04:42:11.646 DEBUG:boto:Token: None 2022-09-02T04:42:11.647 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9ecb3305e59daa-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.652 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.654 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.654 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.655 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.655 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.656 DEBUG:boto:Method: GET 2022-09-02T04:42:11.656 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.657 DEBUG:boto:Data: 2022-09-02T04:42:11.657 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.658 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.659 DEBUG:boto:Params: {} 2022-09-02T04:42:11.660 DEBUG:boto:Token: None 2022-09-02T04:42:11.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU= 2022-09-02T04:42:11.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU=', 'Content-Length': '0'} 2022-09-02T04:42:11.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d64fcda1494e8180-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.706 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.708 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.709 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.710 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.711 DEBUG:boto:Method: GET 2022-09-02T04:42:11.712 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.713 DEBUG:boto:Data: 2022-09-02T04:42:11.714 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.716 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.717 DEBUG:boto:Params: {} 2022-09-02T04:42:11.718 DEBUG:boto:Token: None 2022-09-02T04:42:11.719 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU= 2022-09-02T04:42:11.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU=', 'Content-Length': '0'} 2022-09-02T04:42:11.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d981e8335962e8fb-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.725 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.726 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.727 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.728 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.729 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.730 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.730 DEBUG:boto:Data: 2022-09-02T04:42:11.731 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.733 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.734 DEBUG:boto:Params: {} 2022-09-02T04:42:11.735 DEBUG:boto:Token: None 2022-09-02T04:42:11.735 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ABmcQjyHFRxR0scAzm39Pb5Tz4= 2022-09-02T04:42:11.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ABmcQjyHFRxR0scAzm39Pb5Tz4=', 'Content-Length': '0'} 2022-09-02T04:42:11.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003624b038ecc8d64f-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.739 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.740 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.740 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.741 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.741 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.742 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.742 DEBUG:boto:Data: 2022-09-02T04:42:11.743 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.744 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.744 DEBUG:boto:Params: {} 2022-09-02T04:42:11.745 DEBUG:boto:Token: None 2022-09-02T04:42:11.746 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ABmcQjyHFRxR0scAzm39Pb5Tz4= 2022-09-02T04:42:11.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ABmcQjyHFRxR0scAzm39Pb5Tz4=', 'Content-Length': '0'} 2022-09-02T04:42:11.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015e011627baac60f-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.750 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.751 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.751 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.752 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.752 DEBUG:boto:Method: GET 2022-09-02T04:42:11.753 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.753 DEBUG:boto:Data: 2022-09-02T04:42:11.754 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.755 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.755 DEBUG:boto:Params: {} 2022-09-02T04:42:11.756 DEBUG:boto:Token: None 2022-09-02T04:42:11.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU= 2022-09-02T04:42:11.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU=', 'Content-Length': '0'} 2022-09-02T04:42:11.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a11b7f2a2407f295-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.761 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.761 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.762 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.763 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.763 DEBUG:boto:Method: GET 2022-09-02T04:42:11.764 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.764 DEBUG:boto:Data: 2022-09-02T04:42:11.765 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.766 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.766 DEBUG:boto:Params: {} 2022-09-02T04:42:11.767 DEBUG:boto:Token: None 2022-09-02T04:42:11.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:11.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU= 2022-09-02T04:42:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LHUq03+WxkB8MaAJwV43aUXkTKU=', 'Content-Length': '0'} 2022-09-02T04:42:11.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c077e732fe068c2-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.772 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.773 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.773 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.774 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.774 DEBUG:boto:Method: GET 2022-09-02T04:42:11.775 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.775 DEBUG:boto:Data: 2022-09-02T04:42:11.776 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.777 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.778 DEBUG:boto:Params: {} 2022-09-02T04:42:11.778 DEBUG:boto:Token: None 2022-09-02T04:42:11.779 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18e26d670f96ec2-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.784 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.785 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.785 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.786 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.786 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.787 DEBUG:boto:Method: GET 2022-09-02T04:42:11.788 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.788 DEBUG:boto:Data: 2022-09-02T04:42:11.789 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.790 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.790 DEBUG:boto:Params: {} 2022-09-02T04:42:11.791 DEBUG:boto:Token: None 2022-09-02T04:42:11.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df5ee2d01f946fb-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.796 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.797 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.798 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.798 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.799 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.799 DEBUG:boto:Data: 2022-09-02T04:42:11.800 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.801 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.801 DEBUG:boto:Params: {} 2022-09-02T04:42:11.802 DEBUG:boto:Token: None 2022-09-02T04:42:11.802 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d26093bcbb87d3ea-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.807 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.808 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.808 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.809 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:11.809 DEBUG:boto:Data: 2022-09-02T04:42:11.810 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.811 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.811 DEBUG:boto:Params: {} 2022-09-02T04:42:11.812 DEBUG:boto:Token: None 2022-09-02T04:42:11.813 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj 2022-09-02T04:42:11.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY= 2022-09-02T04:42:11.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pjj8MPtManW/CwqNHET1fTfzyBY=', 'Content-Length': '0'} 2022-09-02T04:42:11.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef075a625b644f4a-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.819 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.820 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.821 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.821 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.822 DEBUG:boto:Method: GET 2022-09-02T04:42:11.822 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.823 DEBUG:boto:Data: 2022-09-02T04:42:11.824 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.825 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.826 DEBUG:boto:Params: {} 2022-09-02T04:42:11.827 DEBUG:boto:Token: None 2022-09-02T04:42:11.827 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3e7983a63da6413-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.876 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.876 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.877 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.878 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.878 DEBUG:boto:Method: GET 2022-09-02T04:42:11.879 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.879 DEBUG:boto:Data: 2022-09-02T04:42:11.880 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.881 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.882 DEBUG:boto:Params: {} 2022-09-02T04:42:11.883 DEBUG:boto:Token: None 2022-09-02T04:42:11.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:11.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0= 2022-09-02T04:42:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0/RR5W3K4K7RQzHKYJPw53LKo0=', 'Content-Length': '0'} 2022-09-02T04:42:11.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a09fa4a820b073-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.888 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.888 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.889 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.889 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.890 DEBUG:boto:Method: GET 2022-09-02T04:42:11.891 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.892 DEBUG:boto:Data: 2022-09-02T04:42:11.892 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.893 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.894 DEBUG:boto:Params: {} 2022-09-02T04:42:11.894 DEBUG:boto:Token: None 2022-09-02T04:42:11.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fc84437cacd147f-00631189a3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.900 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.901 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:11.902 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:11.902 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.903 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:11.904 DEBUG:boto:Method: GET 2022-09-02T04:42:11.904 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:11.905 DEBUG:boto:Data: 2022-09-02T04:42:11.906 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.907 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.908 DEBUG:boto:Params: {} 2022-09-02T04:42:11.908 DEBUG:boto:Token: None 2022-09-02T04:42:11.909 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/ 2022-09-02T04:42:11.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0= 2022-09-02T04:42:11.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4D7EijDncjF9Td6ZJpPWy07Qnf0=', 'Content-Length': '0'} 2022-09-02T04:42:11.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed9733c50d60587-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.914 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:11.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.916 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.916 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.917 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.917 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.918 DEBUG:boto:Method: GET 2022-09-02T04:42:11.918 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.919 DEBUG:boto:Data: 2022-09-02T04:42:11.920 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.921 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.922 DEBUG:boto:Params: {} 2022-09-02T04:42:11.922 DEBUG:boto:Token: None 2022-09-02T04:42:11.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLs3x33Zt1rBE/6VGsMANoTisWc= 2022-09-02T04:42:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLs3x33Zt1rBE/6VGsMANoTisWc=', 'Content-Length': '0'} 2022-09-02T04:42:11.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f621b8c0676a705-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.928 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.928 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.929 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.929 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.930 DEBUG:boto:Method: GET 2022-09-02T04:42:11.931 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.931 DEBUG:boto:Data: 2022-09-02T04:42:11.932 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.933 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.933 DEBUG:boto:Params: {} 2022-09-02T04:42:11.934 DEBUG:boto:Token: None 2022-09-02T04:42:11.935 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLs3x33Zt1rBE/6VGsMANoTisWc= 2022-09-02T04:42:11.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLs3x33Zt1rBE/6VGsMANoTisWc=', 'Content-Length': '0'} 2022-09-02T04:42:11.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038853805822dbf5c-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.939 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.940 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.940 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.941 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.942 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.943 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.943 DEBUG:boto:Data: 2022-09-02T04:42:11.944 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.945 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.945 DEBUG:boto:Params: {} 2022-09-02T04:42:11.946 DEBUG:boto:Token: None 2022-09-02T04:42:11.947 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/zbT/nMW/GVE8QQ3vLjuv8aVG4= 2022-09-02T04:42:11.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/zbT/nMW/GVE8QQ3vLjuv8aVG4=', 'Content-Length': '0'} 2022-09-02T04:42:11.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000350cec25f2946eb2-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.953 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.954 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.955 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.955 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.956 DEBUG:boto:Method: HEAD 2022-09-02T04:42:11.957 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.958 DEBUG:boto:Data: 2022-09-02T04:42:11.959 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:11.960 DEBUG:boto:Port: 8000 2022-09-02T04:42:11.960 DEBUG:boto:Params: {} 2022-09-02T04:42:11.961 DEBUG:boto:Token: None 2022-09-02T04:42:11.962 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/zbT/nMW/GVE8QQ3vLjuv8aVG4= 2022-09-02T04:42:11.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/zbT/nMW/GVE8QQ3vLjuv8aVG4=', 'Content-Length': '0'} 2022-09-02T04:42:11.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076b31f2f78be143a-00631189a3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:11.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:11.971 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:11.972 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:11.973 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.974 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.975 DEBUG:boto:Method: GET 2022-09-02T04:42:11.975 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.976 DEBUG:boto:Data: 2022-09-02T04:42:11.977 DEBUG:boto:Headers: {} 2022-09-02T04:42:11.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:11.979 DEBUG:boto:Port: 8001 2022-09-02T04:42:11.980 DEBUG:boto:Params: {} 2022-09-02T04:42:11.981 DEBUG:boto:Token: None 2022-09-02T04:42:11.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:11 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:11.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLs3x33Zt1rBE/6VGsMANoTisWc= 2022-09-02T04:42:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLs3x33Zt1rBE/6VGsMANoTisWc=', 'Content-Length': '0'} 2022-09-02T04:42:11.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fc6b60cde9aaf98-00631189a3-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.026 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.027 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.028 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:12.029 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:12.030 DEBUG:boto:Method: GET 2022-09-02T04:42:12.031 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:12.032 DEBUG:boto:Data: 2022-09-02T04:42:12.033 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.035 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.036 DEBUG:boto:Params: {} 2022-09-02T04:42:12.037 DEBUG:boto:Token: None 2022-09-02T04:42:12.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:12.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C9Mh2BDwR3oCbZ8rfbpE8PLzGIs= 2022-09-02T04:42:12.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C9Mh2BDwR3oCbZ8rfbpE8PLzGIs=', 'Content-Length': '0'} 2022-09-02T04:42:12.043 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078558cfe8fe2c0bd-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.045 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.046 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.047 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.048 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.049 DEBUG:boto:Method: GET 2022-09-02T04:42:12.050 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.050 DEBUG:boto:Data: 2022-09-02T04:42:12.051 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.052 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.053 DEBUG:boto:Params: {} 2022-09-02T04:42:12.053 DEBUG:boto:Token: None 2022-09-02T04:42:12.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaf7c41b34fb3d5c-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.059 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.060 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.061 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.062 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.062 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.063 DEBUG:boto:Method: GET 2022-09-02T04:42:12.064 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.065 DEBUG:boto:Data: 2022-09-02T04:42:12.066 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.067 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.068 DEBUG:boto:Params: {} 2022-09-02T04:42:12.069 DEBUG:boto:Token: None 2022-09-02T04:42:12.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001750a48f018517e6-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.076 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.077 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.077 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.078 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.079 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.079 DEBUG:boto:Data: 2022-09-02T04:42:12.080 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.081 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.081 DEBUG:boto:Params: {} 2022-09-02T04:42:12.082 DEBUG:boto:Token: None 2022-09-02T04:42:12.082 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.083 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008bea1261391519a-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.088 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.089 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.090 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.090 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.091 DEBUG:boto:Data: 2022-09-02T04:42:12.091 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.092 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.093 DEBUG:boto:Params: {} 2022-09-02T04:42:12.093 DEBUG:boto:Token: None 2022-09-02T04:42:12.094 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8af1e541f271e9e-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.100 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.101 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.102 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.102 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.103 DEBUG:boto:Method: GET 2022-09-02T04:42:12.103 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.104 DEBUG:boto:Data: 2022-09-02T04:42:12.104 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.106 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.107 DEBUG:boto:Params: {} 2022-09-02T04:42:12.107 DEBUG:boto:Token: None 2022-09-02T04:42:12.108 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d00b3a4be21723db-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.154 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.155 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.156 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.156 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.157 DEBUG:boto:Method: GET 2022-09-02T04:42:12.158 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.159 DEBUG:boto:Data: 2022-09-02T04:42:12.160 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.162 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.163 DEBUG:boto:Params: {} 2022-09-02T04:42:12.164 DEBUG:boto:Token: None 2022-09-02T04:42:12.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034992e79971054de-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.171 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.172 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.173 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.174 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.175 DEBUG:boto:Method: GET 2022-09-02T04:42:12.176 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.177 DEBUG:boto:Data: 2022-09-02T04:42:12.177 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.179 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.179 DEBUG:boto:Params: {} 2022-09-02T04:42:12.180 DEBUG:boto:Token: None 2022-09-02T04:42:12.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c18b36dc31bbbe-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.185 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.186 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.187 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.188 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.188 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.189 DEBUG:boto:Method: GET 2022-09-02T04:42:12.189 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.190 DEBUG:boto:Data: 2022-09-02T04:42:12.191 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.192 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.193 DEBUG:boto:Params: {} 2022-09-02T04:42:12.194 DEBUG:boto:Token: None 2022-09-02T04:42:12.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdc30ed72afe6739-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.201 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.204 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.204 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.205 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.206 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.207 DEBUG:boto:Method: GET 2022-09-02T04:42:12.207 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.208 DEBUG:boto:Data: 2022-09-02T04:42:12.209 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.210 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.210 DEBUG:boto:Params: {} 2022-09-02T04:42:12.211 DEBUG:boto:Token: None 2022-09-02T04:42:12.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74= 2022-09-02T04:42:12.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74=', 'Content-Length': '0'} 2022-09-02T04:42:12.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e2d6d40980b4f66-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.219 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.219 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.220 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.221 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.221 DEBUG:boto:Method: GET 2022-09-02T04:42:12.222 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.223 DEBUG:boto:Data: 2022-09-02T04:42:12.224 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.226 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.227 DEBUG:boto:Params: {} 2022-09-02T04:42:12.228 DEBUG:boto:Token: None 2022-09-02T04:42:12.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74= 2022-09-02T04:42:12.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74=', 'Content-Length': '0'} 2022-09-02T04:42:12.234 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcdede7c2449db01-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.235 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.236 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.237 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.238 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.239 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.240 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.241 DEBUG:boto:Data: 2022-09-02T04:42:12.242 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.244 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.245 DEBUG:boto:Params: {} 2022-09-02T04:42:12.246 DEBUG:boto:Token: None 2022-09-02T04:42:12.247 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTE7D5VKvovnGyzZQxm9IXISGKk= 2022-09-02T04:42:12.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTE7D5VKvovnGyzZQxm9IXISGKk=', 'Content-Length': '0'} 2022-09-02T04:42:12.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022020a19dfc29e0c-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.255 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.256 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.257 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.258 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.259 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.259 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.260 DEBUG:boto:Data: 2022-09-02T04:42:12.261 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.262 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.263 DEBUG:boto:Params: {} 2022-09-02T04:42:12.263 DEBUG:boto:Token: None 2022-09-02T04:42:12.264 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTE7D5VKvovnGyzZQxm9IXISGKk= 2022-09-02T04:42:12.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTE7D5VKvovnGyzZQxm9IXISGKk=', 'Content-Length': '0'} 2022-09-02T04:42:12.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe2f4b0cad172f9f-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.270 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.271 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.272 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.272 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.273 DEBUG:boto:Method: GET 2022-09-02T04:42:12.274 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.275 DEBUG:boto:Data: 2022-09-02T04:42:12.276 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.277 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.278 DEBUG:boto:Params: {} 2022-09-02T04:42:12.279 DEBUG:boto:Token: None 2022-09-02T04:42:12.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74= 2022-09-02T04:42:12.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74=', 'Content-Length': '0'} 2022-09-02T04:42:12.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014f89e3048d0e2ae-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.286 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.286 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.287 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.288 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.288 DEBUG:boto:Method: GET 2022-09-02T04:42:12.289 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.290 DEBUG:boto:Data: 2022-09-02T04:42:12.290 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.291 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.292 DEBUG:boto:Params: {} 2022-09-02T04:42:12.293 DEBUG:boto:Token: None 2022-09-02T04:42:12.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:12.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74= 2022-09-02T04:42:12.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JQ4T3ZJoj7Wru3uGn917B7Y3+74=', 'Content-Length': '0'} 2022-09-02T04:42:12.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062f085a6adec835b-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.299 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.299 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.300 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.301 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.302 DEBUG:boto:Method: GET 2022-09-02T04:42:12.303 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.304 DEBUG:boto:Data: 2022-09-02T04:42:12.305 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.306 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.307 DEBUG:boto:Params: {} 2022-09-02T04:42:12.308 DEBUG:boto:Token: None 2022-09-02T04:42:12.309 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80a98d2ff47e624-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.314 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.314 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.315 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.316 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.316 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.317 DEBUG:boto:Method: GET 2022-09-02T04:42:12.317 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.318 DEBUG:boto:Data: 2022-09-02T04:42:12.318 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.319 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.320 DEBUG:boto:Params: {} 2022-09-02T04:42:12.321 DEBUG:boto:Token: None 2022-09-02T04:42:12.321 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc13bddd77e008a2-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.326 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.327 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.328 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.329 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.330 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.331 DEBUG:boto:Data: 2022-09-02T04:42:12.331 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.332 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.333 DEBUG:boto:Params: {} 2022-09-02T04:42:12.334 DEBUG:boto:Token: None 2022-09-02T04:42:12.335 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db77da38c553b115-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.342 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.343 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.344 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.344 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.345 DEBUG:boto:Data: 2022-09-02T04:42:12.346 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.347 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.348 DEBUG:boto:Params: {} 2022-09-02T04:42:12.349 DEBUG:boto:Token: None 2022-09-02T04:42:12.349 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d459f35b69a1e4a-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.357 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.357 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.358 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.358 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.359 DEBUG:boto:Method: GET 2022-09-02T04:42:12.360 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.361 DEBUG:boto:Data: 2022-09-02T04:42:12.362 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.364 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.365 DEBUG:boto:Params: {} 2022-09-02T04:42:12.366 DEBUG:boto:Token: None 2022-09-02T04:42:12.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065b1f5b3762d90f1-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.374 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.375 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.376 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.377 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.378 DEBUG:boto:Method: GET 2022-09-02T04:42:12.379 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.380 DEBUG:boto:Data: 2022-09-02T04:42:12.381 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.382 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.382 DEBUG:boto:Params: {} 2022-09-02T04:42:12.383 DEBUG:boto:Token: None 2022-09-02T04:42:12.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c46825ae117f69c-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.388 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.389 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.389 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.390 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.391 DEBUG:boto:Method: GET 2022-09-02T04:42:12.392 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.392 DEBUG:boto:Data: 2022-09-02T04:42:12.393 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.394 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.395 DEBUG:boto:Params: {} 2022-09-02T04:42:12.395 DEBUG:boto:Token: None 2022-09-02T04:42:12.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4da7c1974e96f6-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.402 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.404 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.405 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.405 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.406 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.407 DEBUG:boto:Method: GET 2022-09-02T04:42:12.408 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.409 DEBUG:boto:Data: 2022-09-02T04:42:12.410 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.412 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.413 DEBUG:boto:Params: {} 2022-09-02T04:42:12.414 DEBUG:boto:Token: None 2022-09-02T04:42:12.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bde0e358f77d6f9-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.420 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.422 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.422 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.423 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.424 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.424 DEBUG:boto:Method: GET 2022-09-02T04:42:12.425 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.425 DEBUG:boto:Data: 2022-09-02T04:42:12.426 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.428 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.429 DEBUG:boto:Params: {} 2022-09-02T04:42:12.430 DEBUG:boto:Token: None 2022-09-02T04:42:12.431 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM= 2022-09-02T04:42:12.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM=', 'Content-Length': '0'} 2022-09-02T04:42:12.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d094f6668542fa0-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.436 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.437 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.438 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.439 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.439 DEBUG:boto:Method: GET 2022-09-02T04:42:12.440 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.441 DEBUG:boto:Data: 2022-09-02T04:42:12.451 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.453 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.454 DEBUG:boto:Params: {} 2022-09-02T04:42:12.455 DEBUG:boto:Token: None 2022-09-02T04:42:12.455 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM= 2022-09-02T04:42:12.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM=', 'Content-Length': '0'} 2022-09-02T04:42:12.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000243c0589302f5d10-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.460 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.461 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.461 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.462 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.463 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.464 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.464 DEBUG:boto:Data: 2022-09-02T04:42:12.465 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.467 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.468 DEBUG:boto:Params: {} 2022-09-02T04:42:12.469 DEBUG:boto:Token: None 2022-09-02T04:42:12.469 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m78l5JXIw9n1ggEWW1+6NGOFbUw= 2022-09-02T04:42:12.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m78l5JXIw9n1ggEWW1+6NGOFbUw=', 'Content-Length': '0'} 2022-09-02T04:42:12.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c523f2ac619a221-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.474 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.475 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.476 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.477 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.478 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.479 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.479 DEBUG:boto:Data: 2022-09-02T04:42:12.480 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.482 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.483 DEBUG:boto:Params: {} 2022-09-02T04:42:12.483 DEBUG:boto:Token: None 2022-09-02T04:42:12.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m78l5JXIw9n1ggEWW1+6NGOFbUw= 2022-09-02T04:42:12.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m78l5JXIw9n1ggEWW1+6NGOFbUw=', 'Content-Length': '0'} 2022-09-02T04:42:12.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006df53c8446dd882-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.490 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.490 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.491 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.491 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.492 DEBUG:boto:Method: GET 2022-09-02T04:42:12.493 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.493 DEBUG:boto:Data: 2022-09-02T04:42:12.494 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.495 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.495 DEBUG:boto:Params: {} 2022-09-02T04:42:12.496 DEBUG:boto:Token: None 2022-09-02T04:42:12.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM= 2022-09-02T04:42:12.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM=', 'Content-Length': '0'} 2022-09-02T04:42:12.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057915d4a32c37774-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.503 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.504 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.505 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.506 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.507 DEBUG:boto:Method: GET 2022-09-02T04:42:12.508 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.509 DEBUG:boto:Data: 2022-09-02T04:42:12.510 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.512 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.513 DEBUG:boto:Params: {} 2022-09-02T04:42:12.514 DEBUG:boto:Token: None 2022-09-02T04:42:12.515 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:12.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM= 2022-09-02T04:42:12.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WN9+W2zOxKcK5XVDFzW3V/pFQM=', 'Content-Length': '0'} 2022-09-02T04:42:12.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000315097f1ff0d77d5-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.519 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.520 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.521 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.521 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.522 DEBUG:boto:Method: GET 2022-09-02T04:42:12.522 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.523 DEBUG:boto:Data: 2022-09-02T04:42:12.523 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.525 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.525 DEBUG:boto:Params: {} 2022-09-02T04:42:12.526 DEBUG:boto:Token: None 2022-09-02T04:42:12.526 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed57b99a7f6f063-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.533 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.535 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.536 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.537 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.538 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.539 DEBUG:boto:Method: GET 2022-09-02T04:42:12.540 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.541 DEBUG:boto:Data: 2022-09-02T04:42:12.542 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.544 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.546 DEBUG:boto:Params: {} 2022-09-02T04:42:12.547 DEBUG:boto:Token: None 2022-09-02T04:42:12.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002509a6d9252bef03-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.594 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.595 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.596 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.596 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.597 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.597 DEBUG:boto:Data: 2022-09-02T04:42:12.598 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.599 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.600 DEBUG:boto:Params: {} 2022-09-02T04:42:12.600 DEBUG:boto:Token: None 2022-09-02T04:42:12.601 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001abce07f617a806e-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.609 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.610 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.610 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.611 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.611 DEBUG:boto:Data: 2022-09-02T04:42:12.612 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.613 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.613 DEBUG:boto:Params: {} 2022-09-02T04:42:12.614 DEBUG:boto:Token: None 2022-09-02T04:42:12.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e0bfe6a92a90edf-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.622 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.623 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.623 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.624 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.625 DEBUG:boto:Method: GET 2022-09-02T04:42:12.626 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.627 DEBUG:boto:Data: 2022-09-02T04:42:12.628 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.630 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.631 DEBUG:boto:Params: {} 2022-09-02T04:42:12.632 DEBUG:boto:Token: None 2022-09-02T04:42:12.633 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae640fd41d08cf59-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.639 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.639 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.640 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.640 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.641 DEBUG:boto:Method: GET 2022-09-02T04:42:12.641 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.642 DEBUG:boto:Data: 2022-09-02T04:42:12.642 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.644 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.644 DEBUG:boto:Params: {} 2022-09-02T04:42:12.645 DEBUG:boto:Token: None 2022-09-02T04:42:12.645 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000929810aa307e310c-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.650 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.650 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.651 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.651 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.652 DEBUG:boto:Method: GET 2022-09-02T04:42:12.652 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.653 DEBUG:boto:Data: 2022-09-02T04:42:12.653 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.655 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.655 DEBUG:boto:Params: {} 2022-09-02T04:42:12.656 DEBUG:boto:Token: None 2022-09-02T04:42:12.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a874a305e125ebb7-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.662 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.663 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.663 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.664 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.664 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.665 DEBUG:boto:Method: GET 2022-09-02T04:42:12.665 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.666 DEBUG:boto:Data: 2022-09-02T04:42:12.667 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.668 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.669 DEBUG:boto:Params: {} 2022-09-02T04:42:12.670 DEBUG:boto:Token: None 2022-09-02T04:42:12.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34770bdc15a43f2-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.676 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.678 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.678 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.679 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.680 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.680 DEBUG:boto:Method: GET 2022-09-02T04:42:12.681 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.681 DEBUG:boto:Data: 2022-09-02T04:42:12.682 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.683 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.684 DEBUG:boto:Params: {} 2022-09-02T04:42:12.685 DEBUG:boto:Token: None 2022-09-02T04:42:12.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ= 2022-09-02T04:42:12.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ=', 'Content-Length': '0'} 2022-09-02T04:42:12.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000508ffbc9f7787f9f-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.690 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.691 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.691 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.692 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.692 DEBUG:boto:Method: GET 2022-09-02T04:42:12.693 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.694 DEBUG:boto:Data: 2022-09-02T04:42:12.694 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.695 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.696 DEBUG:boto:Params: {} 2022-09-02T04:42:12.697 DEBUG:boto:Token: None 2022-09-02T04:42:12.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ= 2022-09-02T04:42:12.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ=', 'Content-Length': '0'} 2022-09-02T04:42:12.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c482dacbc3e30ac-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.742 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.744 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.744 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.745 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.746 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.746 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.747 DEBUG:boto:Data: 2022-09-02T04:42:12.748 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.749 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.750 DEBUG:boto:Params: {} 2022-09-02T04:42:12.751 DEBUG:boto:Token: None 2022-09-02T04:42:12.751 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xPzXHmhaNWCwhw3W+7NxSVkTg04= 2022-09-02T04:42:12.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xPzXHmhaNWCwhw3W+7NxSVkTg04=', 'Content-Length': '0'} 2022-09-02T04:42:12.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010691d8d0882043d-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.756 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.757 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.758 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.758 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.759 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.760 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.760 DEBUG:boto:Data: 2022-09-02T04:42:12.761 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.763 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.764 DEBUG:boto:Params: {} 2022-09-02T04:42:12.765 DEBUG:boto:Token: None 2022-09-02T04:42:12.766 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xPzXHmhaNWCwhw3W+7NxSVkTg04= 2022-09-02T04:42:12.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xPzXHmhaNWCwhw3W+7NxSVkTg04=', 'Content-Length': '0'} 2022-09-02T04:42:12.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000759485f0fd7acb83-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.773 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.774 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.774 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.775 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.776 DEBUG:boto:Method: GET 2022-09-02T04:42:12.776 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.777 DEBUG:boto:Data: 2022-09-02T04:42:12.777 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.779 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.780 DEBUG:boto:Params: {} 2022-09-02T04:42:12.780 DEBUG:boto:Token: None 2022-09-02T04:42:12.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ= 2022-09-02T04:42:12.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ=', 'Content-Length': '0'} 2022-09-02T04:42:12.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0ed3b6d09feb72d-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.826 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.827 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.828 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.829 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.830 DEBUG:boto:Method: GET 2022-09-02T04:42:12.831 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.832 DEBUG:boto:Data: 2022-09-02T04:42:12.833 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.836 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.837 DEBUG:boto:Params: {} 2022-09-02T04:42:12.838 DEBUG:boto:Token: None 2022-09-02T04:42:12.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:12.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ= 2022-09-02T04:42:12.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZlyUB6z6MtoLeL33UA0s4zagXGQ=', 'Content-Length': '0'} 2022-09-02T04:42:12.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ec72b2050a6874d-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.845 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.845 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.846 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.846 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.847 DEBUG:boto:Method: GET 2022-09-02T04:42:12.847 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.848 DEBUG:boto:Data: 2022-09-02T04:42:12.848 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.849 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.850 DEBUG:boto:Params: {} 2022-09-02T04:42:12.851 DEBUG:boto:Token: None 2022-09-02T04:42:12.851 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f47d3dd2c2fb364-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.856 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.857 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.858 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.858 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.859 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.859 DEBUG:boto:Method: GET 2022-09-02T04:42:12.860 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.861 DEBUG:boto:Data: 2022-09-02T04:42:12.861 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.862 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.863 DEBUG:boto:Params: {} 2022-09-02T04:42:12.863 DEBUG:boto:Token: None 2022-09-02T04:42:12.864 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000624ca008200c0762-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.870 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.871 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.872 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.873 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.873 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.874 DEBUG:boto:Data: 2022-09-02T04:42:12.875 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.876 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.876 DEBUG:boto:Params: {} 2022-09-02T04:42:12.877 DEBUG:boto:Token: None 2022-09-02T04:42:12.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3e871bd1367f112-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.883 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.884 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.884 DEBUG:boto:Method: HEAD 2022-09-02T04:42:12.885 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:12.885 DEBUG:boto:Data: 2022-09-02T04:42:12.886 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.887 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.888 DEBUG:boto:Params: {} 2022-09-02T04:42:12.888 DEBUG:boto:Token: None 2022-09-02T04:42:12.889 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj 2022-09-02T04:42:12.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk= 2022-09-02T04:42:12.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qklXk4Qi09krDSulrCXdYyFZoWk=', 'Content-Length': '0'} 2022-09-02T04:42:12.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fb81bc6816f4f1f-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.895 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.896 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.896 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.897 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.897 DEBUG:boto:Method: GET 2022-09-02T04:42:12.898 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.899 DEBUG:boto:Data: 2022-09-02T04:42:12.900 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.901 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.902 DEBUG:boto:Params: {} 2022-09-02T04:42:12.903 DEBUG:boto:Token: None 2022-09-02T04:42:12.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a25c0a6e87f4432b-00631189a4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.952 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.953 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.954 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.955 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.956 DEBUG:boto:Method: GET 2022-09-02T04:42:12.956 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.957 DEBUG:boto:Data: 2022-09-02T04:42:12.958 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.960 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.961 DEBUG:boto:Params: {} 2022-09-02T04:42:12.962 DEBUG:boto:Token: None 2022-09-02T04:42:12.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:12.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts= 2022-09-02T04:42:12.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rY3x0C7c6C0I1VDF7UJlgljQZts=', 'Content-Length': '0'} 2022-09-02T04:42:12.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f45d520163f445b9-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.968 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.969 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.969 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.970 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.970 DEBUG:boto:Method: GET 2022-09-02T04:42:12.971 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.971 DEBUG:boto:Data: 2022-09-02T04:42:12.972 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:12.973 DEBUG:boto:Port: 8001 2022-09-02T04:42:12.974 DEBUG:boto:Params: {} 2022-09-02T04:42:12.975 DEBUG:boto:Token: None 2022-09-02T04:42:12.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c90f76fb0879e64-00631189a4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.982 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.983 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:12.984 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:12.984 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.985 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:12.985 DEBUG:boto:Method: GET 2022-09-02T04:42:12.986 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:12.987 DEBUG:boto:Data: 2022-09-02T04:42:12.987 DEBUG:boto:Headers: {} 2022-09-02T04:42:12.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:12.988 DEBUG:boto:Port: 8000 2022-09-02T04:42:12.989 DEBUG:boto:Params: {} 2022-09-02T04:42:12.989 DEBUG:boto:Token: None 2022-09-02T04:42:12.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:12 GMT /xljbkf-19/ 2022-09-02T04:42:12.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU= 2022-09-02T04:42:12.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s/f+zEKzwEnCTZSukhWVOiwF+MU=', 'Content-Length': '0'} 2022-09-02T04:42:12.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d50a0a53b9a52c-00631189a4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:12.995 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:12.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:12.997 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:12.998 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:12.998 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:12.999 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:12.999 DEBUG:boto:Method: GET 2022-09-02T04:42:13.000 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.001 DEBUG:boto:Data: 2022-09-02T04:42:13.001 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.003 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.003 DEBUG:boto:Params: {} 2022-09-02T04:42:13.004 DEBUG:boto:Token: None 2022-09-02T04:42:13.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE= 2022-09-02T04:42:13.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE=', 'Content-Length': '0'} 2022-09-02T04:42:13.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003205fe1086be98e0-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.012 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.012 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.013 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.014 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.015 DEBUG:boto:Method: GET 2022-09-02T04:42:13.016 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.018 DEBUG:boto:Data: 2022-09-02T04:42:13.019 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.021 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.022 DEBUG:boto:Params: {} 2022-09-02T04:42:13.023 DEBUG:boto:Token: None 2022-09-02T04:42:13.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE= 2022-09-02T04:42:13.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE=', 'Content-Length': '0'} 2022-09-02T04:42:13.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d521830ec1c3fa71-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.031 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.032 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.032 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.033 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.033 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.034 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.035 DEBUG:boto:Data: 2022-09-02T04:42:13.036 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.038 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.039 DEBUG:boto:Params: {} 2022-09-02T04:42:13.040 DEBUG:boto:Token: None 2022-09-02T04:42:13.041 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3kxZ18plT2xUPER5FEQddoQJYA= 2022-09-02T04:42:13.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3kxZ18plT2xUPER5FEQddoQJYA=', 'Content-Length': '0'} 2022-09-02T04:42:13.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002946f8eabfe4a0eb-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.047 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.048 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.048 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.049 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.049 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.050 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.050 DEBUG:boto:Data: 2022-09-02T04:42:13.051 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.052 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.052 DEBUG:boto:Params: {} 2022-09-02T04:42:13.053 DEBUG:boto:Token: None 2022-09-02T04:42:13.054 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3kxZ18plT2xUPER5FEQddoQJYA= 2022-09-02T04:42:13.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3kxZ18plT2xUPER5FEQddoQJYA=', 'Content-Length': '0'} 2022-09-02T04:42:13.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007743a51d970d959a-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.058 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.059 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.060 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.060 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.061 DEBUG:boto:Method: GET 2022-09-02T04:42:13.061 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.062 DEBUG:boto:Data: 2022-09-02T04:42:13.062 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.063 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.064 DEBUG:boto:Params: {} 2022-09-02T04:42:13.064 DEBUG:boto:Token: None 2022-09-02T04:42:13.065 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE= 2022-09-02T04:42:13.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE=', 'Content-Length': '0'} 2022-09-02T04:42:13.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009de368e6826cb39f-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.070 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.070 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.071 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.071 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.072 DEBUG:boto:Method: GET 2022-09-02T04:42:13.072 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.073 DEBUG:boto:Data: 2022-09-02T04:42:13.074 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.075 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.075 DEBUG:boto:Params: {} 2022-09-02T04:42:13.076 DEBUG:boto:Token: None 2022-09-02T04:42:13.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:13.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE= 2022-09-02T04:42:13.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ThyjZ5cfSxLFqa8i+TPPm27SVoE=', 'Content-Length': '0'} 2022-09-02T04:42:13.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c41ae526f30c8bea-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.130 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.131 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.132 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.133 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.135 DEBUG:boto:Method: GET 2022-09-02T04:42:13.135 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.136 DEBUG:boto:Data: 2022-09-02T04:42:13.137 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.139 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.140 DEBUG:boto:Params: {} 2022-09-02T04:42:13.141 DEBUG:boto:Token: None 2022-09-02T04:42:13.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7b05fbeb9fce1c2-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.154 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.156 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.157 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.158 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.159 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.160 DEBUG:boto:Method: GET 2022-09-02T04:42:13.160 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.162 DEBUG:boto:Data: 2022-09-02T04:42:13.163 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.165 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.166 DEBUG:boto:Params: {} 2022-09-02T04:42:13.166 DEBUG:boto:Token: None 2022-09-02T04:42:13.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d30cbf068786b34-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.172 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.173 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.174 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.175 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.176 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.178 DEBUG:boto:Data: 2022-09-02T04:42:13.179 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.181 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.182 DEBUG:boto:Params: {} 2022-09-02T04:42:13.182 DEBUG:boto:Token: None 2022-09-02T04:42:13.183 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj 2022-09-02T04:42:13.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE= 2022-09-02T04:42:13.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE=', 'Content-Length': '0'} 2022-09-02T04:42:13.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec16d6536e4a72f0-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.190 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.191 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.192 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.193 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.193 DEBUG:boto:Data: 2022-09-02T04:42:13.194 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.195 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.196 DEBUG:boto:Params: {} 2022-09-02T04:42:13.197 DEBUG:boto:Token: None 2022-09-02T04:42:13.197 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj 2022-09-02T04:42:13.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE= 2022-09-02T04:42:13.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE=', 'Content-Length': '0'} 2022-09-02T04:42:13.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c7e32bd709a13b-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.204 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.205 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.206 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.206 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.207 DEBUG:boto:Method: GET 2022-09-02T04:42:13.207 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.208 DEBUG:boto:Data: 2022-09-02T04:42:13.209 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.210 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.211 DEBUG:boto:Params: {} 2022-09-02T04:42:13.213 DEBUG:boto:Token: None 2022-09-02T04:42:13.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw= 2022-09-02T04:42:13.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw=', 'Content-Length': '0'} 2022-09-02T04:42:13.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000044479e2c62a68ff-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.262 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.263 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.263 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.264 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.265 DEBUG:boto:Method: GET 2022-09-02T04:42:13.265 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.266 DEBUG:boto:Data: 2022-09-02T04:42:13.266 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.267 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.268 DEBUG:boto:Params: {} 2022-09-02T04:42:13.269 DEBUG:boto:Token: None 2022-09-02T04:42:13.269 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw= 2022-09-02T04:42:13.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw=', 'Content-Length': '0'} 2022-09-02T04:42:13.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071e46df8d04ff20b-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.274 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.274 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.275 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.276 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.276 DEBUG:boto:Method: GET 2022-09-02T04:42:13.277 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.277 DEBUG:boto:Data: 2022-09-02T04:42:13.278 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.279 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.279 DEBUG:boto:Params: {} 2022-09-02T04:42:13.280 DEBUG:boto:Token: None 2022-09-02T04:42:13.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63b301236b8c915-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.286 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.288 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.289 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.290 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.291 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.292 DEBUG:boto:Method: GET 2022-09-02T04:42:13.293 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.294 DEBUG:boto:Data: 2022-09-02T04:42:13.295 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.296 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.297 DEBUG:boto:Params: {} 2022-09-02T04:42:13.297 DEBUG:boto:Token: None 2022-09-02T04:42:13.298 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf7af7bc6654774c-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.304 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.306 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.307 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.308 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.309 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.310 DEBUG:boto:Method: GET 2022-09-02T04:42:13.311 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.312 DEBUG:boto:Data: 2022-09-02T04:42:13.313 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.315 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.315 DEBUG:boto:Params: {} 2022-09-02T04:42:13.316 DEBUG:boto:Token: None 2022-09-02T04:42:13.317 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8= 2022-09-02T04:42:13.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8=', 'Content-Length': '0'} 2022-09-02T04:42:13.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3fca52b4f90cd7a-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.321 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.322 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.322 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.323 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.323 DEBUG:boto:Method: GET 2022-09-02T04:42:13.324 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.324 DEBUG:boto:Data: 2022-09-02T04:42:13.325 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.326 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.326 DEBUG:boto:Params: {} 2022-09-02T04:42:13.327 DEBUG:boto:Token: None 2022-09-02T04:42:13.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8= 2022-09-02T04:42:13.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8=', 'Content-Length': '0'} 2022-09-02T04:42:13.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdceef6ae2585986-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.332 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.333 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.333 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.334 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.334 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.335 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.336 DEBUG:boto:Data: 2022-09-02T04:42:13.336 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.337 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.338 DEBUG:boto:Params: {} 2022-09-02T04:42:13.338 DEBUG:boto:Token: None 2022-09-02T04:42:13.339 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.340 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+s/mhNhDY3jHFQiluFZxxZltJp4= 2022-09-02T04:42:13.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+s/mhNhDY3jHFQiluFZxxZltJp4=', 'Content-Length': '0'} 2022-09-02T04:42:13.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000603813298205716f-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.343 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.344 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.344 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.345 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.345 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.346 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.346 DEBUG:boto:Data: 2022-09-02T04:42:13.347 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.348 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.349 DEBUG:boto:Params: {} 2022-09-02T04:42:13.349 DEBUG:boto:Token: None 2022-09-02T04:42:13.350 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+s/mhNhDY3jHFQiluFZxxZltJp4= 2022-09-02T04:42:13.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+s/mhNhDY3jHFQiluFZxxZltJp4=', 'Content-Length': '0'} 2022-09-02T04:42:13.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054b18a7c55311af5-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.355 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.356 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.356 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.357 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.357 DEBUG:boto:Method: GET 2022-09-02T04:42:13.358 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.358 DEBUG:boto:Data: 2022-09-02T04:42:13.359 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.361 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.361 DEBUG:boto:Params: {} 2022-09-02T04:42:13.362 DEBUG:boto:Token: None 2022-09-02T04:42:13.363 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8= 2022-09-02T04:42:13.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8=', 'Content-Length': '0'} 2022-09-02T04:42:13.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000245b57b040471176-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.370 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.371 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.372 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.373 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.373 DEBUG:boto:Method: GET 2022-09-02T04:42:13.374 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.375 DEBUG:boto:Data: 2022-09-02T04:42:13.375 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.376 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.377 DEBUG:boto:Params: {} 2022-09-02T04:42:13.377 DEBUG:boto:Token: None 2022-09-02T04:42:13.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:13.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8= 2022-09-02T04:42:13.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EKPz8kmUOEKsYpOYaFOi7Hr13Z8=', 'Content-Length': '0'} 2022-09-02T04:42:13.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2bd86258670ab75-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.382 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.383 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.383 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.384 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.384 DEBUG:boto:Method: GET 2022-09-02T04:42:13.385 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.385 DEBUG:boto:Data: 2022-09-02T04:42:13.386 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.387 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.388 DEBUG:boto:Params: {} 2022-09-02T04:42:13.388 DEBUG:boto:Token: None 2022-09-02T04:42:13.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003316d39b72e440b-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.394 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.395 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.395 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.396 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.396 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.397 DEBUG:boto:Method: GET 2022-09-02T04:42:13.397 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.398 DEBUG:boto:Data: 2022-09-02T04:42:13.399 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.400 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.401 DEBUG:boto:Params: {} 2022-09-02T04:42:13.401 DEBUG:boto:Token: None 2022-09-02T04:42:13.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007896a9e6f48015b5-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.407 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.408 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.409 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.409 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.410 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.410 DEBUG:boto:Data: 2022-09-02T04:42:13.411 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.412 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.412 DEBUG:boto:Params: {} 2022-09-02T04:42:13.413 DEBUG:boto:Token: None 2022-09-02T04:42:13.413 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj 2022-09-02T04:42:13.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE= 2022-09-02T04:42:13.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE=', 'Content-Length': '0'} 2022-09-02T04:42:13.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044692e1f19027eba-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.419 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.420 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.420 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.421 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.421 DEBUG:boto:Data: 2022-09-02T04:42:13.422 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.423 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.423 DEBUG:boto:Params: {} 2022-09-02T04:42:13.424 DEBUG:boto:Token: None 2022-09-02T04:42:13.425 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj 2022-09-02T04:42:13.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE= 2022-09-02T04:42:13.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE=', 'Content-Length': '0'} 2022-09-02T04:42:13.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000719affd627ba32a1-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.431 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.431 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.432 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.432 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.433 DEBUG:boto:Method: GET 2022-09-02T04:42:13.433 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.434 DEBUG:boto:Data: 2022-09-02T04:42:13.435 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.436 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.436 DEBUG:boto:Params: {} 2022-09-02T04:42:13.437 DEBUG:boto:Token: None 2022-09-02T04:42:13.438 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw= 2022-09-02T04:42:13.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw=', 'Content-Length': '0'} 2022-09-02T04:42:13.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b4d9f42dcf11916-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.482 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.483 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.483 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.484 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.485 DEBUG:boto:Method: GET 2022-09-02T04:42:13.485 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.486 DEBUG:boto:Data: 2022-09-02T04:42:13.487 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.488 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.489 DEBUG:boto:Params: {} 2022-09-02T04:42:13.489 DEBUG:boto:Token: None 2022-09-02T04:42:13.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw= 2022-09-02T04:42:13.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw=', 'Content-Length': '0'} 2022-09-02T04:42:13.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ae854ceea903726-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.494 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.495 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.495 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.496 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.497 DEBUG:boto:Method: GET 2022-09-02T04:42:13.497 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.498 DEBUG:boto:Data: 2022-09-02T04:42:13.498 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.500 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.500 DEBUG:boto:Params: {} 2022-09-02T04:42:13.501 DEBUG:boto:Token: None 2022-09-02T04:42:13.501 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb5225352ee5d5c-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.507 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.508 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.508 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.509 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.509 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.510 DEBUG:boto:Method: GET 2022-09-02T04:42:13.510 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.511 DEBUG:boto:Data: 2022-09-02T04:42:13.512 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.513 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.513 DEBUG:boto:Params: {} 2022-09-02T04:42:13.514 DEBUG:boto:Token: None 2022-09-02T04:42:13.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f939c1c29c11fd-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.520 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.521 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.522 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.523 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.523 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.524 DEBUG:boto:Method: GET 2022-09-02T04:42:13.524 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.525 DEBUG:boto:Data: 2022-09-02T04:42:13.525 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.527 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.527 DEBUG:boto:Params: {} 2022-09-02T04:42:13.529 DEBUG:boto:Token: None 2022-09-02T04:42:13.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4= 2022-09-02T04:42:13.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4=', 'Content-Length': '0'} 2022-09-02T04:42:13.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d41435742535398e-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.535 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.536 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.536 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.537 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.538 DEBUG:boto:Method: GET 2022-09-02T04:42:13.538 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.539 DEBUG:boto:Data: 2022-09-02T04:42:13.539 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.541 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.541 DEBUG:boto:Params: {} 2022-09-02T04:42:13.542 DEBUG:boto:Token: None 2022-09-02T04:42:13.542 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4= 2022-09-02T04:42:13.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4=', 'Content-Length': '0'} 2022-09-02T04:42:13.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002928c71443a9754b-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.548 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.549 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.550 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.551 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.552 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.553 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.554 DEBUG:boto:Data: 2022-09-02T04:42:13.555 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.556 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.557 DEBUG:boto:Params: {} 2022-09-02T04:42:13.557 DEBUG:boto:Token: None 2022-09-02T04:42:13.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYS1ZOopjDgZs41GIFwrh1OgUUg= 2022-09-02T04:42:13.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYS1ZOopjDgZs41GIFwrh1OgUUg=', 'Content-Length': '0'} 2022-09-02T04:42:13.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003927f1bb763f909f-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.563 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.564 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.564 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.565 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.566 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.567 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.568 DEBUG:boto:Data: 2022-09-02T04:42:13.569 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.571 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.572 DEBUG:boto:Params: {} 2022-09-02T04:42:13.573 DEBUG:boto:Token: None 2022-09-02T04:42:13.574 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYS1ZOopjDgZs41GIFwrh1OgUUg= 2022-09-02T04:42:13.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYS1ZOopjDgZs41GIFwrh1OgUUg=', 'Content-Length': '0'} 2022-09-02T04:42:13.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a67d30a23016b36-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.582 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.583 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.584 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.585 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.586 DEBUG:boto:Method: GET 2022-09-02T04:42:13.587 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.588 DEBUG:boto:Data: 2022-09-02T04:42:13.589 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.590 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.591 DEBUG:boto:Params: {} 2022-09-02T04:42:13.592 DEBUG:boto:Token: None 2022-09-02T04:42:13.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4= 2022-09-02T04:42:13.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4=', 'Content-Length': '0'} 2022-09-02T04:42:13.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083df05703e42352e-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.648 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.649 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.650 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.651 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.652 DEBUG:boto:Method: GET 2022-09-02T04:42:13.653 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.654 DEBUG:boto:Data: 2022-09-02T04:42:13.655 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.657 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.657 DEBUG:boto:Params: {} 2022-09-02T04:42:13.658 DEBUG:boto:Token: None 2022-09-02T04:42:13.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:13.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4= 2022-09-02T04:42:13.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lWxcx1dWDi2PqmugmCRI57J6cf4=', 'Content-Length': '0'} 2022-09-02T04:42:13.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6122d64ae7f3a44-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.666 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.667 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.668 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.669 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.670 DEBUG:boto:Method: GET 2022-09-02T04:42:13.670 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.671 DEBUG:boto:Data: 2022-09-02T04:42:13.671 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.673 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.674 DEBUG:boto:Params: {} 2022-09-02T04:42:13.675 DEBUG:boto:Token: None 2022-09-02T04:42:13.676 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b323a6fd06cf464-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.682 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.683 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.684 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.685 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.685 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.686 DEBUG:boto:Method: GET 2022-09-02T04:42:13.687 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.688 DEBUG:boto:Data: 2022-09-02T04:42:13.688 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.690 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.691 DEBUG:boto:Params: {} 2022-09-02T04:42:13.691 DEBUG:boto:Token: None 2022-09-02T04:42:13.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cebbda8863246980-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.697 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.699 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.699 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.700 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.700 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.701 DEBUG:boto:Data: 2022-09-02T04:42:13.702 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.703 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.703 DEBUG:boto:Params: {} 2022-09-02T04:42:13.704 DEBUG:boto:Token: None 2022-09-02T04:42:13.705 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj 2022-09-02T04:42:13.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE= 2022-09-02T04:42:13.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE=', 'Content-Length': '0'} 2022-09-02T04:42:13.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe50e91beab08d17-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.711 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.711 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.712 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.713 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.714 DEBUG:boto:Data: 2022-09-02T04:42:13.715 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.717 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.718 DEBUG:boto:Params: {} 2022-09-02T04:42:13.719 DEBUG:boto:Token: None 2022-09-02T04:42:13.719 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj 2022-09-02T04:42:13.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE= 2022-09-02T04:42:13.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HSDcU9n/FAXyd5pPUbqBgn+dEHE=', 'Content-Length': '0'} 2022-09-02T04:42:13.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001191219e582826fc-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.725 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.726 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.726 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.727 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.728 DEBUG:boto:Method: GET 2022-09-02T04:42:13.728 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.729 DEBUG:boto:Data: 2022-09-02T04:42:13.729 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.730 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.731 DEBUG:boto:Params: {} 2022-09-02T04:42:13.731 DEBUG:boto:Token: None 2022-09-02T04:42:13.732 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw= 2022-09-02T04:42:13.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw=', 'Content-Length': '0'} 2022-09-02T04:42:13.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004263490e7f16d462-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.736 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.737 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.737 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.738 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.739 DEBUG:boto:Method: GET 2022-09-02T04:42:13.739 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.740 DEBUG:boto:Data: 2022-09-02T04:42:13.740 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.741 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.742 DEBUG:boto:Params: {} 2022-09-02T04:42:13.742 DEBUG:boto:Token: None 2022-09-02T04:42:13.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:13.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw= 2022-09-02T04:42:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4qyrZM0fOAE2qhRwlU1E0KheTrw=', 'Content-Length': '0'} 2022-09-02T04:42:13.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036ca89a67afcb920-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.794 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.796 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.796 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.798 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.799 DEBUG:boto:Method: GET 2022-09-02T04:42:13.800 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.801 DEBUG:boto:Data: 2022-09-02T04:42:13.802 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.804 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.805 DEBUG:boto:Params: {} 2022-09-02T04:42:13.806 DEBUG:boto:Token: None 2022-09-02T04:42:13.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b88e986c5155cfb-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.813 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.814 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.815 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.816 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.816 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.817 DEBUG:boto:Method: GET 2022-09-02T04:42:13.817 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.818 DEBUG:boto:Data: 2022-09-02T04:42:13.819 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.820 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.820 DEBUG:boto:Params: {} 2022-09-02T04:42:13.821 DEBUG:boto:Token: None 2022-09-02T04:42:13.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007427f0a303b0b781-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.827 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.830 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.831 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.832 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.833 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.835 DEBUG:boto:Method: GET 2022-09-02T04:42:13.836 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.836 DEBUG:boto:Data: 2022-09-02T04:42:13.837 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.838 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.839 DEBUG:boto:Params: {} 2022-09-02T04:42:13.839 DEBUG:boto:Token: None 2022-09-02T04:42:13.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M= 2022-09-02T04:42:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M=', 'Content-Length': '0'} 2022-09-02T04:42:13.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b20ab71372caa9f3-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.845 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.846 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.846 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.847 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.847 DEBUG:boto:Method: GET 2022-09-02T04:42:13.848 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.849 DEBUG:boto:Data: 2022-09-02T04:42:13.849 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.850 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.851 DEBUG:boto:Params: {} 2022-09-02T04:42:13.851 DEBUG:boto:Token: None 2022-09-02T04:42:13.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M= 2022-09-02T04:42:13.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M=', 'Content-Length': '0'} 2022-09-02T04:42:13.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000017f3bf68a76bbf0-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.857 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.857 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.858 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.858 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.859 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.859 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.860 DEBUG:boto:Data: 2022-09-02T04:42:13.860 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.861 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.862 DEBUG:boto:Params: {} 2022-09-02T04:42:13.863 DEBUG:boto:Token: None 2022-09-02T04:42:13.863 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a6fPDLjtN9jvpZrDtu2FgU33jfE= 2022-09-02T04:42:13.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a6fPDLjtN9jvpZrDtu2FgU33jfE=', 'Content-Length': '0'} 2022-09-02T04:42:13.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059b044aba0194050-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.868 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.869 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.869 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.870 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.871 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.872 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.873 DEBUG:boto:Data: 2022-09-02T04:42:13.874 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.875 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.876 DEBUG:boto:Params: {} 2022-09-02T04:42:13.877 DEBUG:boto:Token: None 2022-09-02T04:42:13.878 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a6fPDLjtN9jvpZrDtu2FgU33jfE= 2022-09-02T04:42:13.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a6fPDLjtN9jvpZrDtu2FgU33jfE=', 'Content-Length': '0'} 2022-09-02T04:42:13.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a01dbb357f66edbe-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:13.884 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.885 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.886 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.887 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.888 DEBUG:boto:Method: GET 2022-09-02T04:42:13.889 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.890 DEBUG:boto:Data: 2022-09-02T04:42:13.890 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.893 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.893 DEBUG:boto:Params: {} 2022-09-02T04:42:13.894 DEBUG:boto:Token: None 2022-09-02T04:42:13.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M= 2022-09-02T04:42:13.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M=', 'Content-Length': '0'} 2022-09-02T04:42:13.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f8125277184923-00631189a5-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.899 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.901 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.901 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.902 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.902 DEBUG:boto:Method: GET 2022-09-02T04:42:13.903 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.904 DEBUG:boto:Data: 2022-09-02T04:42:13.905 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.906 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.906 DEBUG:boto:Params: {} 2022-09-02T04:42:13.907 DEBUG:boto:Token: None 2022-09-02T04:42:13.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:13.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M= 2022-09-02T04:42:13.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y8M9I9qiA0esNjRCBJf3jllcr6M=', 'Content-Length': '0'} 2022-09-02T04:42:13.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9dc6247a0025eb6-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.912 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.913 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.913 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.914 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.914 DEBUG:boto:Method: GET 2022-09-02T04:42:13.915 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.915 DEBUG:boto:Data: 2022-09-02T04:42:13.916 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:13.917 DEBUG:boto:Port: 8001 2022-09-02T04:42:13.918 DEBUG:boto:Params: {} 2022-09-02T04:42:13.918 DEBUG:boto:Token: None 2022-09-02T04:42:13.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c39c1ac56cd5e513-00631189a5-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.979 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.980 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:13.982 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:13.983 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.984 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:13.985 DEBUG:boto:Method: GET 2022-09-02T04:42:13.985 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:13.986 DEBUG:boto:Data: 2022-09-02T04:42:13.987 DEBUG:boto:Headers: {} 2022-09-02T04:42:13.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:13.988 DEBUG:boto:Port: 8000 2022-09-02T04:42:13.988 DEBUG:boto:Params: {} 2022-09-02T04:42:13.989 DEBUG:boto:Token: None 2022-09-02T04:42:13.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:13 GMT /xljbkf-19/ 2022-09-02T04:42:13.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ= 2022-09-02T04:42:13.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gl3hnnm971BLBHyez8DaRaNyceQ=', 'Content-Length': '0'} 2022-09-02T04:42:13.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602cb797aee1432f-00631189a5-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:13.996 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:13.997 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:13.998 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:13.998 DEBUG:boto:Method: HEAD 2022-09-02T04:42:13.999 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:13.999 DEBUG:boto:Data: 2022-09-02T04:42:14.000 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.001 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.002 DEBUG:boto:Params: {} 2022-09-02T04:42:14.002 DEBUG:boto:Token: None 2022-09-02T04:42:14.003 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c2de78ea2c4817c-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.010 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.011 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.012 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.013 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.014 DEBUG:boto:Data: 2022-09-02T04:42:14.015 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.017 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.018 DEBUG:boto:Params: {} 2022-09-02T04:42:14.019 DEBUG:boto:Token: None 2022-09-02T04:42:14.020 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c0916a7813a0a63-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.030 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.031 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.032 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.032 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.033 DEBUG:boto:Method: GET 2022-09-02T04:42:14.034 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.035 DEBUG:boto:Data: 2022-09-02T04:42:14.041 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.045 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.046 DEBUG:boto:Params: {} 2022-09-02T04:42:14.047 DEBUG:boto:Token: None 2022-09-02T04:42:14.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058241ea57620ed58-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.056 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.057 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.058 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.058 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.059 DEBUG:boto:Method: GET 2022-09-02T04:42:14.060 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.061 DEBUG:boto:Data: 2022-09-02T04:42:14.062 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.066 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.067 DEBUG:boto:Params: {} 2022-09-02T04:42:14.068 DEBUG:boto:Token: None 2022-09-02T04:42:14.069 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bd1a95c94475322-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.086 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.086 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.087 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.088 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.089 DEBUG:boto:Method: GET 2022-09-02T04:42:14.089 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.090 DEBUG:boto:Data: 2022-09-02T04:42:14.091 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.092 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.094 DEBUG:boto:Params: {} 2022-09-02T04:42:14.095 DEBUG:boto:Token: None 2022-09-02T04:42:14.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d23449f5874f2c-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.101 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.102 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.102 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.103 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.104 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.104 DEBUG:boto:Method: GET 2022-09-02T04:42:14.105 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.105 DEBUG:boto:Data: 2022-09-02T04:42:14.106 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.107 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.107 DEBUG:boto:Params: {} 2022-09-02T04:42:14.109 DEBUG:boto:Token: None 2022-09-02T04:42:14.109 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003295653c11fdae86-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.124 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.127 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.128 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.129 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.131 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.132 DEBUG:boto:Method: GET 2022-09-02T04:42:14.133 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.134 DEBUG:boto:Data: 2022-09-02T04:42:14.135 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.137 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.138 DEBUG:boto:Params: {} 2022-09-02T04:42:14.139 DEBUG:boto:Token: None 2022-09-02T04:42:14.140 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk= 2022-09-02T04:42:14.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk=', 'Content-Length': '0'} 2022-09-02T04:42:14.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078b31878094c3ebc-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.147 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.148 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.148 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.149 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.149 DEBUG:boto:Method: GET 2022-09-02T04:42:14.150 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.150 DEBUG:boto:Data: 2022-09-02T04:42:14.151 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.152 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.153 DEBUG:boto:Params: {} 2022-09-02T04:42:14.153 DEBUG:boto:Token: None 2022-09-02T04:42:14.154 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.155 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk= 2022-09-02T04:42:14.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk=', 'Content-Length': '0'} 2022-09-02T04:42:14.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1eef875783b5a4a-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.160 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.160 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.161 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.162 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.162 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.163 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.164 DEBUG:boto:Data: 2022-09-02T04:42:14.164 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.166 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.166 DEBUG:boto:Params: {} 2022-09-02T04:42:14.167 DEBUG:boto:Token: None 2022-09-02T04:42:14.168 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:94/Cswpuf9d9POWJT4yDxaoWd98= 2022-09-02T04:42:14.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:94/Cswpuf9d9POWJT4yDxaoWd98=', 'Content-Length': '0'} 2022-09-02T04:42:14.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c570a4ebb7b50b-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.174 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.175 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.176 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.177 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.177 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.178 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.179 DEBUG:boto:Data: 2022-09-02T04:42:14.180 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.181 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.182 DEBUG:boto:Params: {} 2022-09-02T04:42:14.183 DEBUG:boto:Token: None 2022-09-02T04:42:14.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:94/Cswpuf9d9POWJT4yDxaoWd98= 2022-09-02T04:42:14.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:94/Cswpuf9d9POWJT4yDxaoWd98=', 'Content-Length': '0'} 2022-09-02T04:42:14.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b854852e0897d38f-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.191 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.192 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.193 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.194 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.195 DEBUG:boto:Method: GET 2022-09-02T04:42:14.196 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.197 DEBUG:boto:Data: 2022-09-02T04:42:14.198 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.200 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.201 DEBUG:boto:Params: {} 2022-09-02T04:42:14.202 DEBUG:boto:Token: None 2022-09-02T04:42:14.203 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk= 2022-09-02T04:42:14.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk=', 'Content-Length': '0'} 2022-09-02T04:42:14.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000413d76b54f8438d5-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.210 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.211 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.212 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.213 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.214 DEBUG:boto:Method: GET 2022-09-02T04:42:14.215 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.216 DEBUG:boto:Data: 2022-09-02T04:42:14.217 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.219 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.220 DEBUG:boto:Params: {} 2022-09-02T04:42:14.220 DEBUG:boto:Token: None 2022-09-02T04:42:14.221 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:14.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk= 2022-09-02T04:42:14.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cjouS9PAJUFJGQCXeTjKbF05Kyk=', 'Content-Length': '0'} 2022-09-02T04:42:14.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d736539c02806c55-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.226 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.226 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.227 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.228 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.229 DEBUG:boto:Method: GET 2022-09-02T04:42:14.229 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.230 DEBUG:boto:Data: 2022-09-02T04:42:14.230 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.231 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.232 DEBUG:boto:Params: {} 2022-09-02T04:42:14.233 DEBUG:boto:Token: None 2022-09-02T04:42:14.233 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079b44dcb71ef6467-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.239 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.240 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.241 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.242 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.243 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.244 DEBUG:boto:Method: GET 2022-09-02T04:42:14.245 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.246 DEBUG:boto:Data: 2022-09-02T04:42:14.247 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.248 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.249 DEBUG:boto:Params: {} 2022-09-02T04:42:14.250 DEBUG:boto:Token: None 2022-09-02T04:42:14.251 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000850eadd4f9349298-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.258 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.259 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.260 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.260 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.261 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.261 DEBUG:boto:Data: 2022-09-02T04:42:14.262 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.263 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.263 DEBUG:boto:Params: {} 2022-09-02T04:42:14.264 DEBUG:boto:Token: None 2022-09-02T04:42:14.265 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040076ae24dd7d399-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.270 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.271 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.271 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.272 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.272 DEBUG:boto:Data: 2022-09-02T04:42:14.273 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.274 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.274 DEBUG:boto:Params: {} 2022-09-02T04:42:14.275 DEBUG:boto:Token: None 2022-09-02T04:42:14.276 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d9dc9826dbbbfe7-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.285 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.286 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.287 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.287 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.288 DEBUG:boto:Method: GET 2022-09-02T04:42:14.289 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.290 DEBUG:boto:Data: 2022-09-02T04:42:14.291 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.292 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.293 DEBUG:boto:Params: {} 2022-09-02T04:42:14.294 DEBUG:boto:Token: None 2022-09-02T04:42:14.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000660262b6b5359d21-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.299 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.300 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.300 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.301 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.301 DEBUG:boto:Method: GET 2022-09-02T04:42:14.302 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.303 DEBUG:boto:Data: 2022-09-02T04:42:14.303 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.304 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.305 DEBUG:boto:Params: {} 2022-09-02T04:42:14.306 DEBUG:boto:Token: None 2022-09-02T04:42:14.307 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d02f735475ce9e8-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.354 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.355 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.355 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.356 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.357 DEBUG:boto:Method: GET 2022-09-02T04:42:14.357 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.358 DEBUG:boto:Data: 2022-09-02T04:42:14.359 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.360 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.361 DEBUG:boto:Params: {} 2022-09-02T04:42:14.362 DEBUG:boto:Token: None 2022-09-02T04:42:14.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004944bb7159ca5369-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.367 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.368 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.369 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.369 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.370 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.370 DEBUG:boto:Method: GET 2022-09-02T04:42:14.371 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.372 DEBUG:boto:Data: 2022-09-02T04:42:14.373 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.374 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.375 DEBUG:boto:Params: {} 2022-09-02T04:42:14.375 DEBUG:boto:Token: None 2022-09-02T04:42:14.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002491f23641224e6a-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.381 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.382 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.383 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.383 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.384 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.384 DEBUG:boto:Method: GET 2022-09-02T04:42:14.385 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.385 DEBUG:boto:Data: 2022-09-02T04:42:14.386 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.387 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.388 DEBUG:boto:Params: {} 2022-09-02T04:42:14.388 DEBUG:boto:Token: None 2022-09-02T04:42:14.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg= 2022-09-02T04:42:14.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg=', 'Content-Length': '0'} 2022-09-02T04:42:14.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010c33488bc4f5bcc-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.438 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.439 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.440 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.440 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.441 DEBUG:boto:Method: GET 2022-09-02T04:42:14.441 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.442 DEBUG:boto:Data: 2022-09-02T04:42:14.443 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.444 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.444 DEBUG:boto:Params: {} 2022-09-02T04:42:14.445 DEBUG:boto:Token: None 2022-09-02T04:42:14.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg= 2022-09-02T04:42:14.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg=', 'Content-Length': '0'} 2022-09-02T04:42:14.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e469a87c17c5e445-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.450 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.450 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.451 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.451 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.452 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.452 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.453 DEBUG:boto:Data: 2022-09-02T04:42:14.453 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.454 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.455 DEBUG:boto:Params: {} 2022-09-02T04:42:14.455 DEBUG:boto:Token: None 2022-09-02T04:42:14.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3byXNttgfnfpNP+rQ93NP65VYJY= 2022-09-02T04:42:14.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3byXNttgfnfpNP+rQ93NP65VYJY=', 'Content-Length': '0'} 2022-09-02T04:42:14.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5d5421df07de27f-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.466 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.467 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.467 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.468 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.468 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.469 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.469 DEBUG:boto:Data: 2022-09-02T04:42:14.470 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.471 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.471 DEBUG:boto:Params: {} 2022-09-02T04:42:14.472 DEBUG:boto:Token: None 2022-09-02T04:42:14.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3byXNttgfnfpNP+rQ93NP65VYJY= 2022-09-02T04:42:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3byXNttgfnfpNP+rQ93NP65VYJY=', 'Content-Length': '0'} 2022-09-02T04:42:14.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000704fc5d0494a9b37-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.478 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.478 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.479 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.479 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.480 DEBUG:boto:Method: GET 2022-09-02T04:42:14.481 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.481 DEBUG:boto:Data: 2022-09-02T04:42:14.482 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.483 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.484 DEBUG:boto:Params: {} 2022-09-02T04:42:14.484 DEBUG:boto:Token: None 2022-09-02T04:42:14.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg= 2022-09-02T04:42:14.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg=', 'Content-Length': '0'} 2022-09-02T04:42:14.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c35f0d7c1d89236-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.489 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.490 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.490 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.491 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.491 DEBUG:boto:Method: GET 2022-09-02T04:42:14.492 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.492 DEBUG:boto:Data: 2022-09-02T04:42:14.493 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.494 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.494 DEBUG:boto:Params: {} 2022-09-02T04:42:14.495 DEBUG:boto:Token: None 2022-09-02T04:42:14.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:14.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg= 2022-09-02T04:42:14.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hv7H627K2oApwzNg1j5l73x33Zg=', 'Content-Length': '0'} 2022-09-02T04:42:14.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef22566532266f9e-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.500 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.500 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.501 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.501 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.502 DEBUG:boto:Method: GET 2022-09-02T04:42:14.503 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.503 DEBUG:boto:Data: 2022-09-02T04:42:14.504 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.505 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.505 DEBUG:boto:Params: {} 2022-09-02T04:42:14.506 DEBUG:boto:Token: None 2022-09-02T04:42:14.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b38729c80f8fd5f-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.511 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.512 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.512 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.513 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.513 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.514 DEBUG:boto:Method: GET 2022-09-02T04:42:14.515 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.515 DEBUG:boto:Data: 2022-09-02T04:42:14.516 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.517 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.517 DEBUG:boto:Params: {} 2022-09-02T04:42:14.518 DEBUG:boto:Token: None 2022-09-02T04:42:14.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da4e5a1732e726d-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.523 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.524 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.524 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.525 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.525 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.526 DEBUG:boto:Data: 2022-09-02T04:42:14.526 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.527 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.528 DEBUG:boto:Params: {} 2022-09-02T04:42:14.528 DEBUG:boto:Token: None 2022-09-02T04:42:14.529 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf68662440750593-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.534 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.535 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.535 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.536 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.536 DEBUG:boto:Data: 2022-09-02T04:42:14.537 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.538 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.539 DEBUG:boto:Params: {} 2022-09-02T04:42:14.539 DEBUG:boto:Token: None 2022-09-02T04:42:14.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cfc3a6b32cea76d-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.546 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.547 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.547 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.548 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.548 DEBUG:boto:Method: GET 2022-09-02T04:42:14.549 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.549 DEBUG:boto:Data: 2022-09-02T04:42:14.550 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.551 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.551 DEBUG:boto:Params: {} 2022-09-02T04:42:14.552 DEBUG:boto:Token: None 2022-09-02T04:42:14.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cd3d57afb7b6ed6-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.557 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.557 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.558 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.559 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.559 DEBUG:boto:Method: GET 2022-09-02T04:42:14.560 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.560 DEBUG:boto:Data: 2022-09-02T04:42:14.561 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.562 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.562 DEBUG:boto:Params: {} 2022-09-02T04:42:14.563 DEBUG:boto:Token: None 2022-09-02T04:42:14.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051dce0da30e9e794-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.610 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.611 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.612 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.613 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.614 DEBUG:boto:Method: GET 2022-09-02T04:42:14.615 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.616 DEBUG:boto:Data: 2022-09-02T04:42:14.617 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.619 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.620 DEBUG:boto:Params: {} 2022-09-02T04:42:14.621 DEBUG:boto:Token: None 2022-09-02T04:42:14.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca1096ace603199-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.626 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.628 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.628 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.629 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.629 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.630 DEBUG:boto:Method: GET 2022-09-02T04:42:14.630 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.631 DEBUG:boto:Data: 2022-09-02T04:42:14.632 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.633 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.633 DEBUG:boto:Params: {} 2022-09-02T04:42:14.634 DEBUG:boto:Token: None 2022-09-02T04:42:14.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423d8381201450d4-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.639 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.641 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.641 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.642 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.643 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.643 DEBUG:boto:Method: GET 2022-09-02T04:42:14.644 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.644 DEBUG:boto:Data: 2022-09-02T04:42:14.645 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.646 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.646 DEBUG:boto:Params: {} 2022-09-02T04:42:14.647 DEBUG:boto:Token: None 2022-09-02T04:42:14.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU= 2022-09-02T04:42:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU=', 'Content-Length': '0'} 2022-09-02T04:42:14.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e02e3db476fa0cb-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.653 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.653 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.654 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.655 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.656 DEBUG:boto:Method: GET 2022-09-02T04:42:14.657 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.658 DEBUG:boto:Data: 2022-09-02T04:42:14.659 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.661 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.662 DEBUG:boto:Params: {} 2022-09-02T04:42:14.663 DEBUG:boto:Token: None 2022-09-02T04:42:14.663 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.664 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU= 2022-09-02T04:42:14.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU=', 'Content-Length': '0'} 2022-09-02T04:42:14.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bf62a58ccd48987-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.668 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.668 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.669 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.669 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.670 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.671 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.672 DEBUG:boto:Data: 2022-09-02T04:42:14.672 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.674 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.674 DEBUG:boto:Params: {} 2022-09-02T04:42:14.675 DEBUG:boto:Token: None 2022-09-02T04:42:14.676 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CL/uGnJp8Q3gPxp6moeGWRQ422c= 2022-09-02T04:42:14.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CL/uGnJp8Q3gPxp6moeGWRQ422c=', 'Content-Length': '0'} 2022-09-02T04:42:14.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028a367c8dd58466f-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.681 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.682 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.682 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.683 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.683 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.684 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.684 DEBUG:boto:Data: 2022-09-02T04:42:14.685 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.686 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.686 DEBUG:boto:Params: {} 2022-09-02T04:42:14.687 DEBUG:boto:Token: None 2022-09-02T04:42:14.687 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CL/uGnJp8Q3gPxp6moeGWRQ422c= 2022-09-02T04:42:14.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CL/uGnJp8Q3gPxp6moeGWRQ422c=', 'Content-Length': '0'} 2022-09-02T04:42:14.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe464244f43c4de6-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.698 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.698 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.699 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.699 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.700 DEBUG:boto:Method: GET 2022-09-02T04:42:14.700 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.701 DEBUG:boto:Data: 2022-09-02T04:42:14.702 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.703 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.703 DEBUG:boto:Params: {} 2022-09-02T04:42:14.704 DEBUG:boto:Token: None 2022-09-02T04:42:14.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU= 2022-09-02T04:42:14.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU=', 'Content-Length': '0'} 2022-09-02T04:42:14.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087fc685d78605a8e-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.709 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.709 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.710 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.710 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.711 DEBUG:boto:Method: GET 2022-09-02T04:42:14.711 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.712 DEBUG:boto:Data: 2022-09-02T04:42:14.712 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.713 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.714 DEBUG:boto:Params: {} 2022-09-02T04:42:14.714 DEBUG:boto:Token: None 2022-09-02T04:42:14.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:14.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU= 2022-09-02T04:42:14.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwSZHRWuZdFsAIx5/oH3kbBNauU=', 'Content-Length': '0'} 2022-09-02T04:42:14.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aad3da8fca175864-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.766 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.767 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.768 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.768 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.769 DEBUG:boto:Method: GET 2022-09-02T04:42:14.770 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.771 DEBUG:boto:Data: 2022-09-02T04:42:14.772 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.773 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.774 DEBUG:boto:Params: {} 2022-09-02T04:42:14.774 DEBUG:boto:Token: None 2022-09-02T04:42:14.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000114a683312d48b20-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.780 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.781 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.782 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.782 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.783 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.783 DEBUG:boto:Method: GET 2022-09-02T04:42:14.784 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.784 DEBUG:boto:Data: 2022-09-02T04:42:14.785 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.786 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.787 DEBUG:boto:Params: {} 2022-09-02T04:42:14.787 DEBUG:boto:Token: None 2022-09-02T04:42:14.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e034301785680d-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.793 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.794 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.794 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.795 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.795 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.796 DEBUG:boto:Data: 2022-09-02T04:42:14.796 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.797 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.798 DEBUG:boto:Params: {} 2022-09-02T04:42:14.799 DEBUG:boto:Token: None 2022-09-02T04:42:14.799 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000597d8097971614ae-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.804 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.805 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.806 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.806 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:14.807 DEBUG:boto:Data: 2022-09-02T04:42:14.807 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.808 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.809 DEBUG:boto:Params: {} 2022-09-02T04:42:14.809 DEBUG:boto:Token: None 2022-09-02T04:42:14.810 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj 2022-09-02T04:42:14.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg= 2022-09-02T04:42:14.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hTAbx2ChvO8bHUfrrcf5FQaTjBg=', 'Content-Length': '0'} 2022-09-02T04:42:14.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bf9d258ccdc7233-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.816 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.816 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.817 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.817 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.818 DEBUG:boto:Method: GET 2022-09-02T04:42:14.818 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.819 DEBUG:boto:Data: 2022-09-02T04:42:14.820 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.821 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.821 DEBUG:boto:Params: {} 2022-09-02T04:42:14.822 DEBUG:boto:Token: None 2022-09-02T04:42:14.823 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022f19d643d62520b-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.874 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.875 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.876 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.876 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.877 DEBUG:boto:Method: GET 2022-09-02T04:42:14.877 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.878 DEBUG:boto:Data: 2022-09-02T04:42:14.879 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.880 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.880 DEBUG:boto:Params: {} 2022-09-02T04:42:14.881 DEBUG:boto:Token: None 2022-09-02T04:42:14.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:14.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU= 2022-09-02T04:42:14.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T9EyQWUluu5UwUDSoFw7tgocMAU=', 'Content-Length': '0'} 2022-09-02T04:42:14.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070eb6c5a9c815dba-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.886 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.887 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.887 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.888 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.888 DEBUG:boto:Method: GET 2022-09-02T04:42:14.889 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.889 DEBUG:boto:Data: 2022-09-02T04:42:14.890 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.891 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.892 DEBUG:boto:Params: {} 2022-09-02T04:42:14.892 DEBUG:boto:Token: None 2022-09-02T04:42:14.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019a84018837303a-00631189a6-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.898 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.899 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:14.899 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:14.900 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.900 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:14.901 DEBUG:boto:Method: GET 2022-09-02T04:42:14.902 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:14.902 DEBUG:boto:Data: 2022-09-02T04:42:14.903 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.904 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.904 DEBUG:boto:Params: {} 2022-09-02T04:42:14.905 DEBUG:boto:Token: None 2022-09-02T04:42:14.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/ 2022-09-02T04:42:14.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs= 2022-09-02T04:42:14.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xP8sN5dNf+Cyt+Ye0buLNRjMEKs=', 'Content-Length': '0'} 2022-09-02T04:42:14.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505997a892826938-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.911 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:14.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.912 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.913 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.913 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.914 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.914 DEBUG:boto:Method: GET 2022-09-02T04:42:14.915 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.915 DEBUG:boto:Data: 2022-09-02T04:42:14.916 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.917 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.917 DEBUG:boto:Params: {} 2022-09-02T04:42:14.918 DEBUG:boto:Token: None 2022-09-02T04:42:14.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg= 2022-09-02T04:42:14.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg=', 'Content-Length': '0'} 2022-09-02T04:42:14.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cf46ca572417e48-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.923 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.924 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.924 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.925 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.926 DEBUG:boto:Method: GET 2022-09-02T04:42:14.926 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.927 DEBUG:boto:Data: 2022-09-02T04:42:14.927 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.928 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.929 DEBUG:boto:Params: {} 2022-09-02T04:42:14.929 DEBUG:boto:Token: None 2022-09-02T04:42:14.930 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.931 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg= 2022-09-02T04:42:14.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg=', 'Content-Length': '0'} 2022-09-02T04:42:14.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9eac75a41e3697e-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.934 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.935 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.935 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.936 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.936 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.937 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.937 DEBUG:boto:Data: 2022-09-02T04:42:14.938 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.939 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.939 DEBUG:boto:Params: {} 2022-09-02T04:42:14.940 DEBUG:boto:Token: None 2022-09-02T04:42:14.941 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0ztgR04WjU4P8Lu31wV2ZMO4kyY= 2022-09-02T04:42:14.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0ztgR04WjU4P8Lu31wV2ZMO4kyY=', 'Content-Length': '0'} 2022-09-02T04:42:14.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bec7dd586c0d15f6-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.945 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.945 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.946 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.946 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.947 DEBUG:boto:Method: HEAD 2022-09-02T04:42:14.947 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.948 DEBUG:boto:Data: 2022-09-02T04:42:14.948 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.949 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.950 DEBUG:boto:Params: {} 2022-09-02T04:42:14.951 DEBUG:boto:Token: None 2022-09-02T04:42:14.951 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0ztgR04WjU4P8Lu31wV2ZMO4kyY= 2022-09-02T04:42:14.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0ztgR04WjU4P8Lu31wV2ZMO4kyY=', 'Content-Length': '0'} 2022-09-02T04:42:14.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a1318294ef718c-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:14.956 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.957 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.958 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.959 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.960 DEBUG:boto:Method: GET 2022-09-02T04:42:14.961 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.962 DEBUG:boto:Data: 2022-09-02T04:42:14.963 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:14.964 DEBUG:boto:Port: 8001 2022-09-02T04:42:14.965 DEBUG:boto:Params: {} 2022-09-02T04:42:14.965 DEBUG:boto:Token: None 2022-09-02T04:42:14.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg= 2022-09-02T04:42:14.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg=', 'Content-Length': '0'} 2022-09-02T04:42:14.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007318b7546af6fa3a-00631189a6-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:14.973 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:14.974 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:14.975 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.976 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.977 DEBUG:boto:Method: GET 2022-09-02T04:42:14.978 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.979 DEBUG:boto:Data: 2022-09-02T04:42:14.980 DEBUG:boto:Headers: {} 2022-09-02T04:42:14.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:14.982 DEBUG:boto:Port: 8000 2022-09-02T04:42:14.983 DEBUG:boto:Params: {} 2022-09-02T04:42:14.984 DEBUG:boto:Token: None 2022-09-02T04:42:14.985 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:14 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:14.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg= 2022-09-02T04:42:14.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:edrA+IY2SMUk/nbldqyw9mCuomg=', 'Content-Length': '0'} 2022-09-02T04:42:14.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048b69bc8f6823448-00631189a6-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.032 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.033 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.034 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.035 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.036 DEBUG:boto:Method: GET 2022-09-02T04:42:15.037 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.037 DEBUG:boto:Data: 2022-09-02T04:42:15.038 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.039 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.040 DEBUG:boto:Params: {} 2022-09-02T04:42:15.041 DEBUG:boto:Token: None 2022-09-02T04:42:15.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c626e87c303a0395-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.048 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.049 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.050 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.051 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.051 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.052 DEBUG:boto:Method: GET 2022-09-02T04:42:15.052 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.053 DEBUG:boto:Data: 2022-09-02T04:42:15.053 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.055 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.055 DEBUG:boto:Params: {} 2022-09-02T04:42:15.056 DEBUG:boto:Token: None 2022-09-02T04:42:15.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd013d1ce4a4e427-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.062 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.063 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.063 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.064 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.065 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.065 DEBUG:boto:Data: 2022-09-02T04:42:15.066 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.067 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.068 DEBUG:boto:Params: {} 2022-09-02T04:42:15.069 DEBUG:boto:Token: None 2022-09-02T04:42:15.069 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fbb16a068aa14af-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.077 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.078 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.079 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.079 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.080 DEBUG:boto:Data: 2022-09-02T04:42:15.081 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.083 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.083 DEBUG:boto:Params: {} 2022-09-02T04:42:15.084 DEBUG:boto:Token: None 2022-09-02T04:42:15.085 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecbbe19892c4048b-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.092 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.093 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.094 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.094 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.095 DEBUG:boto:Method: GET 2022-09-02T04:42:15.096 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.097 DEBUG:boto:Data: 2022-09-02T04:42:15.097 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.099 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.099 DEBUG:boto:Params: {} 2022-09-02T04:42:15.100 DEBUG:boto:Token: None 2022-09-02T04:42:15.101 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa7d28dcf41ab9f4-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.146 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.147 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.148 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.148 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.149 DEBUG:boto:Method: GET 2022-09-02T04:42:15.150 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.151 DEBUG:boto:Data: 2022-09-02T04:42:15.152 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.154 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.155 DEBUG:boto:Params: {} 2022-09-02T04:42:15.156 DEBUG:boto:Token: None 2022-09-02T04:42:15.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097ae1caef1043eb6-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.163 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.163 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.164 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.164 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.165 DEBUG:boto:Method: GET 2022-09-02T04:42:15.165 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.166 DEBUG:boto:Data: 2022-09-02T04:42:15.167 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.168 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.168 DEBUG:boto:Params: {} 2022-09-02T04:42:15.169 DEBUG:boto:Token: None 2022-09-02T04:42:15.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af27dbe2870d37d3-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.174 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.175 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.176 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.177 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.177 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.178 DEBUG:boto:Method: GET 2022-09-02T04:42:15.179 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.180 DEBUG:boto:Data: 2022-09-02T04:42:15.180 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.182 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.183 DEBUG:boto:Params: {} 2022-09-02T04:42:15.184 DEBUG:boto:Token: None 2022-09-02T04:42:15.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000603382962ab7eaa-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.191 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.193 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.193 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.194 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.195 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.195 DEBUG:boto:Method: GET 2022-09-02T04:42:15.196 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.197 DEBUG:boto:Data: 2022-09-02T04:42:15.197 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.199 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.200 DEBUG:boto:Params: {} 2022-09-02T04:42:15.201 DEBUG:boto:Token: None 2022-09-02T04:42:15.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA= 2022-09-02T04:42:15.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA=', 'Content-Length': '0'} 2022-09-02T04:42:15.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e64b25710601daf-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.206 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.207 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.207 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.208 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.208 DEBUG:boto:Method: GET 2022-09-02T04:42:15.209 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.210 DEBUG:boto:Data: 2022-09-02T04:42:15.210 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.212 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.213 DEBUG:boto:Params: {} 2022-09-02T04:42:15.214 DEBUG:boto:Token: None 2022-09-02T04:42:15.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA= 2022-09-02T04:42:15.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA=', 'Content-Length': '0'} 2022-09-02T04:42:15.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097456b7640111a76-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.223 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.224 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.225 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.225 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.226 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.227 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.227 DEBUG:boto:Data: 2022-09-02T04:42:15.228 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.229 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.229 DEBUG:boto:Params: {} 2022-09-02T04:42:15.230 DEBUG:boto:Token: None 2022-09-02T04:42:15.231 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4tyl/GGxgbF1huaysSphE/JPGiA= 2022-09-02T04:42:15.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4tyl/GGxgbF1huaysSphE/JPGiA=', 'Content-Length': '0'} 2022-09-02T04:42:15.235 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc146a7ef0692788-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.236 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.236 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.237 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.237 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.238 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.238 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.239 DEBUG:boto:Data: 2022-09-02T04:42:15.240 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.241 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.241 DEBUG:boto:Params: {} 2022-09-02T04:42:15.242 DEBUG:boto:Token: None 2022-09-02T04:42:15.243 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4tyl/GGxgbF1huaysSphE/JPGiA= 2022-09-02T04:42:15.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4tyl/GGxgbF1huaysSphE/JPGiA=', 'Content-Length': '0'} 2022-09-02T04:42:15.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dff5f1a817c48a6c-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.248 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.249 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.249 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.250 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.250 DEBUG:boto:Method: GET 2022-09-02T04:42:15.251 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.251 DEBUG:boto:Data: 2022-09-02T04:42:15.252 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.253 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.254 DEBUG:boto:Params: {} 2022-09-02T04:42:15.254 DEBUG:boto:Token: None 2022-09-02T04:42:15.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA= 2022-09-02T04:42:15.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA=', 'Content-Length': '0'} 2022-09-02T04:42:15.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027d37cd23c0236d9-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.259 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.260 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.260 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.261 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.261 DEBUG:boto:Method: GET 2022-09-02T04:42:15.262 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.262 DEBUG:boto:Data: 2022-09-02T04:42:15.263 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.264 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.265 DEBUG:boto:Params: {} 2022-09-02T04:42:15.265 DEBUG:boto:Token: None 2022-09-02T04:42:15.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:15.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA= 2022-09-02T04:42:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z+tHO3Yrqa333LXb7hCtQ3G37nA=', 'Content-Length': '0'} 2022-09-02T04:42:15.269 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b31db351795b4ef9-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.310 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.311 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.311 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.312 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.313 DEBUG:boto:Method: GET 2022-09-02T04:42:15.313 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.314 DEBUG:boto:Data: 2022-09-02T04:42:15.315 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.316 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.316 DEBUG:boto:Params: {} 2022-09-02T04:42:15.317 DEBUG:boto:Token: None 2022-09-02T04:42:15.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f97fa8abf23a97c6-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.323 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.325 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.326 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.326 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.327 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.328 DEBUG:boto:Method: GET 2022-09-02T04:42:15.329 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.330 DEBUG:boto:Data: 2022-09-02T04:42:15.331 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.332 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.333 DEBUG:boto:Params: {} 2022-09-02T04:42:15.333 DEBUG:boto:Token: None 2022-09-02T04:42:15.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428ebebfc8e06cb8-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.339 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.341 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.341 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.342 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.342 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.343 DEBUG:boto:Data: 2022-09-02T04:42:15.343 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.344 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.345 DEBUG:boto:Params: {} 2022-09-02T04:42:15.346 DEBUG:boto:Token: None 2022-09-02T04:42:15.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e23880fe4e5e530-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.352 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.352 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.353 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.353 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.354 DEBUG:boto:Data: 2022-09-02T04:42:15.355 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.356 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.357 DEBUG:boto:Params: {} 2022-09-02T04:42:15.358 DEBUG:boto:Token: None 2022-09-02T04:42:15.359 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078152ec70d111df0-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.366 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.367 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.368 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.368 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.369 DEBUG:boto:Method: GET 2022-09-02T04:42:15.369 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.370 DEBUG:boto:Data: 2022-09-02T04:42:15.370 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.372 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.372 DEBUG:boto:Params: {} 2022-09-02T04:42:15.373 DEBUG:boto:Token: None 2022-09-02T04:42:15.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055f76c6a7601350f-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.418 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.419 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.420 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.420 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.421 DEBUG:boto:Method: GET 2022-09-02T04:42:15.421 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.422 DEBUG:boto:Data: 2022-09-02T04:42:15.423 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.424 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.425 DEBUG:boto:Params: {} 2022-09-02T04:42:15.426 DEBUG:boto:Token: None 2022-09-02T04:42:15.426 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000264e3fc61d416bd2-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.432 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.433 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.434 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.435 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.436 DEBUG:boto:Method: GET 2022-09-02T04:42:15.437 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.438 DEBUG:boto:Data: 2022-09-02T04:42:15.439 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.442 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.443 DEBUG:boto:Params: {} 2022-09-02T04:42:15.444 DEBUG:boto:Token: None 2022-09-02T04:42:15.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc611de626663f4-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.452 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.453 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.454 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.455 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.456 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.456 DEBUG:boto:Method: GET 2022-09-02T04:42:15.457 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.457 DEBUG:boto:Data: 2022-09-02T04:42:15.458 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.460 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.461 DEBUG:boto:Params: {} 2022-09-02T04:42:15.462 DEBUG:boto:Token: None 2022-09-02T04:42:15.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590974cc9381ca89-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.514 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.516 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.516 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.517 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.517 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.518 DEBUG:boto:Method: GET 2022-09-02T04:42:15.519 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.519 DEBUG:boto:Data: 2022-09-02T04:42:15.520 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.522 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.524 DEBUG:boto:Params: {} 2022-09-02T04:42:15.525 DEBUG:boto:Token: None 2022-09-02T04:42:15.526 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA= 2022-09-02T04:42:15.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA=', 'Content-Length': '0'} 2022-09-02T04:42:15.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000132e87f2db04fa91-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.534 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.535 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.536 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.538 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.539 DEBUG:boto:Method: GET 2022-09-02T04:42:15.539 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.540 DEBUG:boto:Data: 2022-09-02T04:42:15.540 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.542 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.543 DEBUG:boto:Params: {} 2022-09-02T04:42:15.544 DEBUG:boto:Token: None 2022-09-02T04:42:15.545 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA= 2022-09-02T04:42:15.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA=', 'Content-Length': '0'} 2022-09-02T04:42:15.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1a498291373ba41-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.550 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.551 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.552 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.553 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.554 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.554 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.555 DEBUG:boto:Data: 2022-09-02T04:42:15.555 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.556 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.557 DEBUG:boto:Params: {} 2022-09-02T04:42:15.558 DEBUG:boto:Token: None 2022-09-02T04:42:15.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RvE2uf1ChJ11xoXfcz35p4NAAtk= 2022-09-02T04:42:15.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RvE2uf1ChJ11xoXfcz35p4NAAtk=', 'Content-Length': '0'} 2022-09-02T04:42:15.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000189c7b473051941-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.563 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.564 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.565 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.566 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.566 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.567 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.568 DEBUG:boto:Data: 2022-09-02T04:42:15.568 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.570 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.571 DEBUG:boto:Params: {} 2022-09-02T04:42:15.572 DEBUG:boto:Token: None 2022-09-02T04:42:15.573 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RvE2uf1ChJ11xoXfcz35p4NAAtk= 2022-09-02T04:42:15.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RvE2uf1ChJ11xoXfcz35p4NAAtk=', 'Content-Length': '0'} 2022-09-02T04:42:15.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cde5833ff6869e1-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.581 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.582 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.583 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.584 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.584 DEBUG:boto:Method: GET 2022-09-02T04:42:15.585 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.585 DEBUG:boto:Data: 2022-09-02T04:42:15.586 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.588 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.588 DEBUG:boto:Params: {} 2022-09-02T04:42:15.589 DEBUG:boto:Token: None 2022-09-02T04:42:15.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA= 2022-09-02T04:42:15.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA=', 'Content-Length': '0'} 2022-09-02T04:42:15.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9b35334116d0869-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.596 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.597 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.598 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.598 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.599 DEBUG:boto:Method: GET 2022-09-02T04:42:15.600 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.600 DEBUG:boto:Data: 2022-09-02T04:42:15.601 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.602 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.602 DEBUG:boto:Params: {} 2022-09-02T04:42:15.603 DEBUG:boto:Token: None 2022-09-02T04:42:15.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:15.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA= 2022-09-02T04:42:15.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZBnCRa4u4Wtq0eB/xRl49rmRUeA=', 'Content-Length': '0'} 2022-09-02T04:42:15.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000126290b9408d16a1-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.609 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.609 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.610 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.611 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.612 DEBUG:boto:Method: GET 2022-09-02T04:42:15.613 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.614 DEBUG:boto:Data: 2022-09-02T04:42:15.615 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.616 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.617 DEBUG:boto:Params: {} 2022-09-02T04:42:15.618 DEBUG:boto:Token: None 2022-09-02T04:42:15.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000245189b0390b1c8-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.625 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.626 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.627 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.627 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.628 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.628 DEBUG:boto:Method: GET 2022-09-02T04:42:15.629 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.630 DEBUG:boto:Data: 2022-09-02T04:42:15.630 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.631 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.632 DEBUG:boto:Params: {} 2022-09-02T04:42:15.632 DEBUG:boto:Token: None 2022-09-02T04:42:15.633 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e46e30396a8ad7eb-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.638 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.640 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.641 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.642 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.642 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.643 DEBUG:boto:Data: 2022-09-02T04:42:15.644 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.645 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.646 DEBUG:boto:Params: {} 2022-09-02T04:42:15.647 DEBUG:boto:Token: None 2022-09-02T04:42:15.648 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.649 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5dba450f696c0bc-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.655 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.656 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.656 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.657 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.657 DEBUG:boto:Data: 2022-09-02T04:42:15.658 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.659 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.660 DEBUG:boto:Params: {} 2022-09-02T04:42:15.660 DEBUG:boto:Token: None 2022-09-02T04:42:15.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1b90e70f1e47117-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.670 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.671 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.672 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.672 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.673 DEBUG:boto:Method: GET 2022-09-02T04:42:15.674 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.674 DEBUG:boto:Data: 2022-09-02T04:42:15.675 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.676 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.677 DEBUG:boto:Params: {} 2022-09-02T04:42:15.677 DEBUG:boto:Token: None 2022-09-02T04:42:15.678 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003726cbad4b716485-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.683 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.683 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.684 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.685 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.685 DEBUG:boto:Method: GET 2022-09-02T04:42:15.686 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.687 DEBUG:boto:Data: 2022-09-02T04:42:15.687 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.688 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.689 DEBUG:boto:Params: {} 2022-09-02T04:42:15.689 DEBUG:boto:Token: None 2022-09-02T04:42:15.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016d0c7d8ee9e28c7-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.734 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.735 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.735 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.736 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.736 DEBUG:boto:Method: GET 2022-09-02T04:42:15.737 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.737 DEBUG:boto:Data: 2022-09-02T04:42:15.738 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.739 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.740 DEBUG:boto:Params: {} 2022-09-02T04:42:15.740 DEBUG:boto:Token: None 2022-09-02T04:42:15.741 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c9df2a578d4acd-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.749 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.750 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.751 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.752 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.752 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.753 DEBUG:boto:Method: GET 2022-09-02T04:42:15.754 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.755 DEBUG:boto:Data: 2022-09-02T04:42:15.756 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.757 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.758 DEBUG:boto:Params: {} 2022-09-02T04:42:15.759 DEBUG:boto:Token: None 2022-09-02T04:42:15.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081df927b7297994f-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.767 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.769 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.771 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.772 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.772 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.773 DEBUG:boto:Method: GET 2022-09-02T04:42:15.774 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.775 DEBUG:boto:Data: 2022-09-02T04:42:15.776 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.777 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.778 DEBUG:boto:Params: {} 2022-09-02T04:42:15.779 DEBUG:boto:Token: None 2022-09-02T04:42:15.779 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg= 2022-09-02T04:42:15.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg=', 'Content-Length': '0'} 2022-09-02T04:42:15.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099b1bce3610b6d74-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.786 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.787 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.788 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.789 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.790 DEBUG:boto:Method: GET 2022-09-02T04:42:15.791 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.792 DEBUG:boto:Data: 2022-09-02T04:42:15.793 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.796 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.797 DEBUG:boto:Params: {} 2022-09-02T04:42:15.797 DEBUG:boto:Token: None 2022-09-02T04:42:15.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg= 2022-09-02T04:42:15.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg=', 'Content-Length': '0'} 2022-09-02T04:42:15.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e262d1b001fa471-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.804 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.804 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.805 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.805 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.806 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.806 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.807 DEBUG:boto:Data: 2022-09-02T04:42:15.807 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.808 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.809 DEBUG:boto:Params: {} 2022-09-02T04:42:15.809 DEBUG:boto:Token: None 2022-09-02T04:42:15.810 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DRLIeo2iH7mqIRTuPfg5S5bneKA= 2022-09-02T04:42:15.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DRLIeo2iH7mqIRTuPfg5S5bneKA=', 'Content-Length': '0'} 2022-09-02T04:42:15.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086bdb88b88491093-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.814 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.815 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.815 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.816 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.816 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.817 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.817 DEBUG:boto:Data: 2022-09-02T04:42:15.817 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.819 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.819 DEBUG:boto:Params: {} 2022-09-02T04:42:15.820 DEBUG:boto:Token: None 2022-09-02T04:42:15.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DRLIeo2iH7mqIRTuPfg5S5bneKA= 2022-09-02T04:42:15.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DRLIeo2iH7mqIRTuPfg5S5bneKA=', 'Content-Length': '0'} 2022-09-02T04:42:15.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d21f6634ce641257-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.825 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.826 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.826 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.827 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.827 DEBUG:boto:Method: GET 2022-09-02T04:42:15.828 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.829 DEBUG:boto:Data: 2022-09-02T04:42:15.829 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.831 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.832 DEBUG:boto:Params: {} 2022-09-02T04:42:15.832 DEBUG:boto:Token: None 2022-09-02T04:42:15.833 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg= 2022-09-02T04:42:15.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg=', 'Content-Length': '0'} 2022-09-02T04:42:15.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f3dce2e2860cd3b-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.837 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.838 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.839 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.839 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.840 DEBUG:boto:Method: GET 2022-09-02T04:42:15.840 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.841 DEBUG:boto:Data: 2022-09-02T04:42:15.841 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.842 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.843 DEBUG:boto:Params: {} 2022-09-02T04:42:15.844 DEBUG:boto:Token: None 2022-09-02T04:42:15.844 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:15.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg= 2022-09-02T04:42:15.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ra8AIf+Tw/+zGVPbW40UBMHllOg=', 'Content-Length': '0'} 2022-09-02T04:42:15.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b37289a17ba3052-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.894 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.895 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.896 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.897 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.897 DEBUG:boto:Method: GET 2022-09-02T04:42:15.898 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.898 DEBUG:boto:Data: 2022-09-02T04:42:15.899 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.900 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.901 DEBUG:boto:Params: {} 2022-09-02T04:42:15.901 DEBUG:boto:Token: None 2022-09-02T04:42:15.902 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2dcdd1bb9010958-00631189a7-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.907 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.908 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:15.908 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:15.909 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.910 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:15.911 DEBUG:boto:Method: GET 2022-09-02T04:42:15.912 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:15.913 DEBUG:boto:Data: 2022-09-02T04:42:15.914 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.916 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.917 DEBUG:boto:Params: {} 2022-09-02T04:42:15.918 DEBUG:boto:Token: None 2022-09-02T04:42:15.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/ 2022-09-02T04:42:15.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk= 2022-09-02T04:42:15.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VkAZjp3vPAo3coD33FiL7sF13mk=', 'Content-Length': '0'} 2022-09-02T04:42:15.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f4414ec2064dca6-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.925 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:15.926 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.926 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.927 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.927 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.928 DEBUG:boto:Data: 2022-09-02T04:42:15.928 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.929 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.930 DEBUG:boto:Params: {} 2022-09-02T04:42:15.930 DEBUG:boto:Token: None 2022-09-02T04:42:15.931 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f04492890ad23f30-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.937 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.938 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.939 DEBUG:boto:Method: HEAD 2022-09-02T04:42:15.940 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:15.941 DEBUG:boto:Data: 2022-09-02T04:42:15.942 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:15.943 DEBUG:boto:Port: 8000 2022-09-02T04:42:15.944 DEBUG:boto:Params: {} 2022-09-02T04:42:15.944 DEBUG:boto:Token: None 2022-09-02T04:42:15.945 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj 2022-09-02T04:42:15.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY= 2022-09-02T04:42:15.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiiQg96xVfhH2sBoC7tD/z89BrY=', 'Content-Length': '0'} 2022-09-02T04:42:15.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000951b311103fa4666-00631189a7-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:15.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:15.951 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:15.952 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:15.952 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.953 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.953 DEBUG:boto:Method: GET 2022-09-02T04:42:15.954 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.955 DEBUG:boto:Data: 2022-09-02T04:42:15.955 DEBUG:boto:Headers: {} 2022-09-02T04:42:15.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:15.957 DEBUG:boto:Port: 8001 2022-09-02T04:42:15.958 DEBUG:boto:Params: {} 2022-09-02T04:42:15.959 DEBUG:boto:Token: None 2022-09-02T04:42:15.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:15 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:15.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ= 2022-09-02T04:42:15.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FmGbxzqivJ/qKmxANxB/K1tTHQ=', 'Content-Length': '0'} 2022-09-02T04:42:15.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001050f8a626a5245e-00631189a7-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.006 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.007 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.008 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.009 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.010 DEBUG:boto:Method: GET 2022-09-02T04:42:16.011 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.012 DEBUG:boto:Data: 2022-09-02T04:42:16.013 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.015 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.016 DEBUG:boto:Params: {} 2022-09-02T04:42:16.017 DEBUG:boto:Token: None 2022-09-02T04:42:16.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006877627b49ea643-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.024 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.025 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.026 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.027 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.028 DEBUG:boto:Method: GET 2022-09-02T04:42:16.029 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.030 DEBUG:boto:Data: 2022-09-02T04:42:16.031 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.032 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.033 DEBUG:boto:Params: {} 2022-09-02T04:42:16.034 DEBUG:boto:Token: None 2022-09-02T04:42:16.035 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000b6fa855716bbd3-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.043 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.044 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.045 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.045 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.046 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.047 DEBUG:boto:Method: GET 2022-09-02T04:42:16.047 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.048 DEBUG:boto:Data: 2022-09-02T04:42:16.049 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.050 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.051 DEBUG:boto:Params: {} 2022-09-02T04:42:16.052 DEBUG:boto:Token: None 2022-09-02T04:42:16.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab778eb9bec2276-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.106 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.109 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.110 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.111 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.112 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.113 DEBUG:boto:Method: GET 2022-09-02T04:42:16.113 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.114 DEBUG:boto:Data: 2022-09-02T04:42:16.114 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.116 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.117 DEBUG:boto:Params: {} 2022-09-02T04:42:16.118 DEBUG:boto:Token: None 2022-09-02T04:42:16.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y= 2022-09-02T04:42:16.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y=', 'Content-Length': '0'} 2022-09-02T04:42:16.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046c1b19091db9471-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.125 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.126 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.127 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.127 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.128 DEBUG:boto:Method: GET 2022-09-02T04:42:16.129 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.130 DEBUG:boto:Data: 2022-09-02T04:42:16.131 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.132 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.133 DEBUG:boto:Params: {} 2022-09-02T04:42:16.134 DEBUG:boto:Token: None 2022-09-02T04:42:16.134 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y= 2022-09-02T04:42:16.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y=', 'Content-Length': '0'} 2022-09-02T04:42:16.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006edb0f2324130311-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.139 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.140 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.140 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.141 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.141 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.142 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.143 DEBUG:boto:Data: 2022-09-02T04:42:16.144 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.145 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.145 DEBUG:boto:Params: {} 2022-09-02T04:42:16.146 DEBUG:boto:Token: None 2022-09-02T04:42:16.147 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fxjwCHJqx22h3B3VvS/Qbdj69F8= 2022-09-02T04:42:16.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fxjwCHJqx22h3B3VvS/Qbdj69F8=', 'Content-Length': '0'} 2022-09-02T04:42:16.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0dcfe94d62aeabd-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.152 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.152 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.153 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.153 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.154 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.155 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.156 DEBUG:boto:Data: 2022-09-02T04:42:16.156 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.158 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.158 DEBUG:boto:Params: {} 2022-09-02T04:42:16.159 DEBUG:boto:Token: None 2022-09-02T04:42:16.161 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fxjwCHJqx22h3B3VvS/Qbdj69F8= 2022-09-02T04:42:16.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fxjwCHJqx22h3B3VvS/Qbdj69F8=', 'Content-Length': '0'} 2022-09-02T04:42:16.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007531414914a6fa8d-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.168 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.169 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.169 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.170 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.171 DEBUG:boto:Method: GET 2022-09-02T04:42:16.171 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.172 DEBUG:boto:Data: 2022-09-02T04:42:16.172 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.173 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.174 DEBUG:boto:Params: {} 2022-09-02T04:42:16.175 DEBUG:boto:Token: None 2022-09-02T04:42:16.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y= 2022-09-02T04:42:16.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y=', 'Content-Length': '0'} 2022-09-02T04:42:16.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef7b9a41f9ae24dc-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.222 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.223 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.224 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.225 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.226 DEBUG:boto:Method: GET 2022-09-02T04:42:16.228 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.229 DEBUG:boto:Data: 2022-09-02T04:42:16.229 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.230 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.231 DEBUG:boto:Params: {} 2022-09-02T04:42:16.232 DEBUG:boto:Token: None 2022-09-02T04:42:16.233 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:16.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y= 2022-09-02T04:42:16.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L7Wal/tpgeTxLNL063MpSbGFa7Y=', 'Content-Length': '0'} 2022-09-02T04:42:16.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000008c43916f1efd1f-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.240 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.241 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.243 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.243 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.244 DEBUG:boto:Method: GET 2022-09-02T04:42:16.245 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.246 DEBUG:boto:Data: 2022-09-02T04:42:16.247 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.250 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.251 DEBUG:boto:Params: {} 2022-09-02T04:42:16.252 DEBUG:boto:Token: None 2022-09-02T04:42:16.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016bea9afa3f23435-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.259 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.261 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.262 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.262 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.263 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.263 DEBUG:boto:Method: GET 2022-09-02T04:42:16.264 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.264 DEBUG:boto:Data: 2022-09-02T04:42:16.265 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.266 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.266 DEBUG:boto:Params: {} 2022-09-02T04:42:16.267 DEBUG:boto:Token: None 2022-09-02T04:42:16.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000568c08352caf1f56-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.314 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.316 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.317 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.318 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.319 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:16.320 DEBUG:boto:Data: 2022-09-02T04:42:16.320 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.322 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.323 DEBUG:boto:Params: {} 2022-09-02T04:42:16.324 DEBUG:boto:Token: None 2022-09-02T04:42:16.325 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj 2022-09-02T04:42:16.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0= 2022-09-02T04:42:16.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0=', 'Content-Length': '0'} 2022-09-02T04:42:16.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f6765965cdfb274-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.332 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.332 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.333 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.334 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:16.334 DEBUG:boto:Data: 2022-09-02T04:42:16.335 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.337 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.338 DEBUG:boto:Params: {} 2022-09-02T04:42:16.339 DEBUG:boto:Token: None 2022-09-02T04:42:16.340 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj 2022-09-02T04:42:16.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0= 2022-09-02T04:42:16.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0=', 'Content-Length': '0'} 2022-09-02T04:42:16.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c6afc0175e1fbbb-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.347 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.348 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.349 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.350 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.350 DEBUG:boto:Method: GET 2022-09-02T04:42:16.351 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.352 DEBUG:boto:Data: 2022-09-02T04:42:16.352 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.353 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.354 DEBUG:boto:Params: {} 2022-09-02T04:42:16.355 DEBUG:boto:Token: None 2022-09-02T04:42:16.356 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c893ee102d985eda-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.362 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.363 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.364 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.365 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.366 DEBUG:boto:Method: GET 2022-09-02T04:42:16.367 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.367 DEBUG:boto:Data: 2022-09-02T04:42:16.368 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.369 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.369 DEBUG:boto:Params: {} 2022-09-02T04:42:16.370 DEBUG:boto:Token: None 2022-09-02T04:42:16.371 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.375 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bca1523513da1b1-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.377 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.378 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.378 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.379 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.380 DEBUG:boto:Method: GET 2022-09-02T04:42:16.381 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.382 DEBUG:boto:Data: 2022-09-02T04:42:16.383 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.385 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.386 DEBUG:boto:Params: {} 2022-09-02T04:42:16.387 DEBUG:boto:Token: None 2022-09-02T04:42:16.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008998d2a8f9fbb888-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.398 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.399 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.401 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.402 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.403 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.404 DEBUG:boto:Method: GET 2022-09-02T04:42:16.405 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.406 DEBUG:boto:Data: 2022-09-02T04:42:16.407 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.409 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.410 DEBUG:boto:Params: {} 2022-09-02T04:42:16.411 DEBUG:boto:Token: None 2022-09-02T04:42:16.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097eb3019439ce52-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.419 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.421 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.421 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.422 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.423 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.425 DEBUG:boto:Method: GET 2022-09-02T04:42:16.425 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.426 DEBUG:boto:Data: 2022-09-02T04:42:16.427 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.429 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.430 DEBUG:boto:Params: {} 2022-09-02T04:42:16.431 DEBUG:boto:Token: None 2022-09-02T04:42:16.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE= 2022-09-02T04:42:16.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE=', 'Content-Length': '0'} 2022-09-02T04:42:16.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f0500f766f13f19-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.438 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.438 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.439 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.439 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.440 DEBUG:boto:Method: GET 2022-09-02T04:42:16.440 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.441 DEBUG:boto:Data: 2022-09-02T04:42:16.442 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.443 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.444 DEBUG:boto:Params: {} 2022-09-02T04:42:16.444 DEBUG:boto:Token: None 2022-09-02T04:42:16.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE= 2022-09-02T04:42:16.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE=', 'Content-Length': '0'} 2022-09-02T04:42:16.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030e2ef9bc236cc89-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.449 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.450 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.451 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.451 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.452 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.453 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.453 DEBUG:boto:Data: 2022-09-02T04:42:16.454 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.455 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.455 DEBUG:boto:Params: {} 2022-09-02T04:42:16.456 DEBUG:boto:Token: None 2022-09-02T04:42:16.457 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AzwNVnRsLdFSMLK0LuFderbDacc= 2022-09-02T04:42:16.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AzwNVnRsLdFSMLK0LuFderbDacc=', 'Content-Length': '0'} 2022-09-02T04:42:16.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c470748154230a45-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.462 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.463 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.463 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.464 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.465 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.466 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.467 DEBUG:boto:Data: 2022-09-02T04:42:16.467 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.469 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.470 DEBUG:boto:Params: {} 2022-09-02T04:42:16.471 DEBUG:boto:Token: None 2022-09-02T04:42:16.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AzwNVnRsLdFSMLK0LuFderbDacc= 2022-09-02T04:42:16.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AzwNVnRsLdFSMLK0LuFderbDacc=', 'Content-Length': '0'} 2022-09-02T04:42:16.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e47ace6ce328012-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.484 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.485 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.486 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.486 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.487 DEBUG:boto:Method: GET 2022-09-02T04:42:16.487 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.488 DEBUG:boto:Data: 2022-09-02T04:42:16.490 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.492 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.492 DEBUG:boto:Params: {} 2022-09-02T04:42:16.494 DEBUG:boto:Token: None 2022-09-02T04:42:16.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE= 2022-09-02T04:42:16.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE=', 'Content-Length': '0'} 2022-09-02T04:42:16.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081b04de1b4d33840-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.501 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.503 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.504 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.505 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.506 DEBUG:boto:Method: GET 2022-09-02T04:42:16.507 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.507 DEBUG:boto:Data: 2022-09-02T04:42:16.509 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.511 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.511 DEBUG:boto:Params: {} 2022-09-02T04:42:16.512 DEBUG:boto:Token: None 2022-09-02T04:42:16.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:16.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE= 2022-09-02T04:42:16.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dyv8+L6T78C0duVHJVmngDSXCHE=', 'Content-Length': '0'} 2022-09-02T04:42:16.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dce25da72f5f3bfa-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.518 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.519 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.520 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.521 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.522 DEBUG:boto:Method: GET 2022-09-02T04:42:16.523 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.524 DEBUG:boto:Data: 2022-09-02T04:42:16.525 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.527 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.528 DEBUG:boto:Params: {} 2022-09-02T04:42:16.529 DEBUG:boto:Token: None 2022-09-02T04:42:16.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d81ffc94d82cf6-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.535 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.536 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.537 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.538 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.538 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.539 DEBUG:boto:Method: GET 2022-09-02T04:42:16.539 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.540 DEBUG:boto:Data: 2022-09-02T04:42:16.540 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.541 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.542 DEBUG:boto:Params: {} 2022-09-02T04:42:16.543 DEBUG:boto:Token: None 2022-09-02T04:42:16.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa9be6371804d09-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.548 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.549 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.550 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.550 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.551 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:16.552 DEBUG:boto:Data: 2022-09-02T04:42:16.552 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.553 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.554 DEBUG:boto:Params: {} 2022-09-02T04:42:16.554 DEBUG:boto:Token: None 2022-09-02T04:42:16.555 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj 2022-09-02T04:42:16.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0= 2022-09-02T04:42:16.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0=', 'Content-Length': '0'} 2022-09-02T04:42:16.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055573ea45047a72f-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.561 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.561 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.562 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.563 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:16.563 DEBUG:boto:Data: 2022-09-02T04:42:16.564 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.565 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.565 DEBUG:boto:Params: {} 2022-09-02T04:42:16.566 DEBUG:boto:Token: None 2022-09-02T04:42:16.567 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj 2022-09-02T04:42:16.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0= 2022-09-02T04:42:16.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0=', 'Content-Length': '0'} 2022-09-02T04:42:16.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000125693989a8e080a-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.573 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.573 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.574 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.575 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.576 DEBUG:boto:Method: GET 2022-09-02T04:42:16.577 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.578 DEBUG:boto:Data: 2022-09-02T04:42:16.579 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.581 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.581 DEBUG:boto:Params: {} 2022-09-02T04:42:16.582 DEBUG:boto:Token: None 2022-09-02T04:42:16.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bd3bb1503f189ea-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.630 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.631 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.632 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.633 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.634 DEBUG:boto:Method: GET 2022-09-02T04:42:16.635 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.636 DEBUG:boto:Data: 2022-09-02T04:42:16.637 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.638 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.639 DEBUG:boto:Params: {} 2022-09-02T04:42:16.640 DEBUG:boto:Token: None 2022-09-02T04:42:16.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f5b9637ce586597-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.649 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.650 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.651 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.652 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.653 DEBUG:boto:Method: GET 2022-09-02T04:42:16.654 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.655 DEBUG:boto:Data: 2022-09-02T04:42:16.656 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.659 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.660 DEBUG:boto:Params: {} 2022-09-02T04:42:16.661 DEBUG:boto:Token: None 2022-09-02T04:42:16.662 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a7766882031a38-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.669 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.670 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.671 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.673 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.674 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.675 DEBUG:boto:Method: GET 2022-09-02T04:42:16.676 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.677 DEBUG:boto:Data: 2022-09-02T04:42:16.678 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.680 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.681 DEBUG:boto:Params: {} 2022-09-02T04:42:16.682 DEBUG:boto:Token: None 2022-09-02T04:42:16.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d1417bc0ff56600-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.689 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.691 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.691 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.692 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.692 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.693 DEBUG:boto:Method: GET 2022-09-02T04:42:16.694 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.694 DEBUG:boto:Data: 2022-09-02T04:42:16.695 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.696 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.696 DEBUG:boto:Params: {} 2022-09-02T04:42:16.697 DEBUG:boto:Token: None 2022-09-02T04:42:16.698 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik= 2022-09-02T04:42:16.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik=', 'Content-Length': '0'} 2022-09-02T04:42:16.702 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c4b36d5aacfc7b8-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.703 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.704 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.705 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.706 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.707 DEBUG:boto:Method: GET 2022-09-02T04:42:16.708 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.709 DEBUG:boto:Data: 2022-09-02T04:42:16.710 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.712 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.713 DEBUG:boto:Params: {} 2022-09-02T04:42:16.714 DEBUG:boto:Token: None 2022-09-02T04:42:16.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik= 2022-09-02T04:42:16.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik=', 'Content-Length': '0'} 2022-09-02T04:42:16.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b53f40f5c2ae5d9-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.720 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.720 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.721 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.722 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.722 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.723 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.723 DEBUG:boto:Data: 2022-09-02T04:42:16.724 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.725 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.726 DEBUG:boto:Params: {} 2022-09-02T04:42:16.727 DEBUG:boto:Token: None 2022-09-02T04:42:16.727 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wf5rFKppB7b6FpoqxVosJO6m8yk= 2022-09-02T04:42:16.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wf5rFKppB7b6FpoqxVosJO6m8yk=', 'Content-Length': '0'} 2022-09-02T04:42:16.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad0377ccc9510544-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.733 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.733 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.734 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.734 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.735 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.735 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.736 DEBUG:boto:Data: 2022-09-02T04:42:16.736 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.737 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.738 DEBUG:boto:Params: {} 2022-09-02T04:42:16.738 DEBUG:boto:Token: None 2022-09-02T04:42:16.739 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wf5rFKppB7b6FpoqxVosJO6m8yk= 2022-09-02T04:42:16.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wf5rFKppB7b6FpoqxVosJO6m8yk=', 'Content-Length': '0'} 2022-09-02T04:42:16.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1601cebd497914a-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.744 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.744 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.745 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.745 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.746 DEBUG:boto:Method: GET 2022-09-02T04:42:16.746 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.747 DEBUG:boto:Data: 2022-09-02T04:42:16.747 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.748 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.749 DEBUG:boto:Params: {} 2022-09-02T04:42:16.750 DEBUG:boto:Token: None 2022-09-02T04:42:16.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik= 2022-09-02T04:42:16.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik=', 'Content-Length': '0'} 2022-09-02T04:42:16.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2e2421f5d40b4d1-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.754 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.755 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.755 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.756 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.756 DEBUG:boto:Method: GET 2022-09-02T04:42:16.757 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.758 DEBUG:boto:Data: 2022-09-02T04:42:16.758 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.759 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.760 DEBUG:boto:Params: {} 2022-09-02T04:42:16.760 DEBUG:boto:Token: None 2022-09-02T04:42:16.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:16.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik= 2022-09-02T04:42:16.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oGR41bV4QGfHDZadDFklJQppSik=', 'Content-Length': '0'} 2022-09-02T04:42:16.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fceb174ee8e5ff66-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.765 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.766 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.766 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.767 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.767 DEBUG:boto:Method: GET 2022-09-02T04:42:16.768 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.768 DEBUG:boto:Data: 2022-09-02T04:42:16.769 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.770 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.771 DEBUG:boto:Params: {} 2022-09-02T04:42:16.771 DEBUG:boto:Token: None 2022-09-02T04:42:16.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec054564ec20c4a-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.777 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.777 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.778 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.779 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.779 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.780 DEBUG:boto:Method: GET 2022-09-02T04:42:16.780 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.781 DEBUG:boto:Data: 2022-09-02T04:42:16.781 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.782 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.783 DEBUG:boto:Params: {} 2022-09-02T04:42:16.784 DEBUG:boto:Token: None 2022-09-02T04:42:16.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005535ffad43730ea7-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.790 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.791 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.792 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.792 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.793 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:16.793 DEBUG:boto:Data: 2022-09-02T04:42:16.794 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.795 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.796 DEBUG:boto:Params: {} 2022-09-02T04:42:16.797 DEBUG:boto:Token: None 2022-09-02T04:42:16.798 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj 2022-09-02T04:42:16.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0= 2022-09-02T04:42:16.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0=', 'Content-Length': '0'} 2022-09-02T04:42:16.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f587caf6f620de62-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.806 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.806 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.807 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.807 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:16.808 DEBUG:boto:Data: 2022-09-02T04:42:16.808 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.809 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.810 DEBUG:boto:Params: {} 2022-09-02T04:42:16.811 DEBUG:boto:Token: None 2022-09-02T04:42:16.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj 2022-09-02T04:42:16.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0= 2022-09-02T04:42:16.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dimKztBvjBxWbMPLSWe1w7Vv3f0=', 'Content-Length': '0'} 2022-09-02T04:42:16.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009a159f0d9aa9e31-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.817 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.817 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.818 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.819 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.819 DEBUG:boto:Method: GET 2022-09-02T04:42:16.820 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.820 DEBUG:boto:Data: 2022-09-02T04:42:16.821 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.822 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.822 DEBUG:boto:Params: {} 2022-09-02T04:42:16.823 DEBUG:boto:Token: None 2022-09-02T04:42:16.824 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067526d750bd8350b-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.828 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.829 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.829 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.830 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.830 DEBUG:boto:Method: GET 2022-09-02T04:42:16.831 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.831 DEBUG:boto:Data: 2022-09-02T04:42:16.832 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.833 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.833 DEBUG:boto:Params: {} 2022-09-02T04:42:16.834 DEBUG:boto:Token: None 2022-09-02T04:42:16.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:16.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI= 2022-09-02T04:42:16.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jQGCMwnms47KdzCSxgshtaoVcNI=', 'Content-Length': '0'} 2022-09-02T04:42:16.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009251eb2d3708a548-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.839 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.839 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.840 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.840 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.841 DEBUG:boto:Method: GET 2022-09-02T04:42:16.841 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.842 DEBUG:boto:Data: 2022-09-02T04:42:16.843 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.844 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.844 DEBUG:boto:Params: {} 2022-09-02T04:42:16.845 DEBUG:boto:Token: None 2022-09-02T04:42:16.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007376e14535a56d45-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.850 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.851 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.852 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.852 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.853 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.853 DEBUG:boto:Method: GET 2022-09-02T04:42:16.854 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.854 DEBUG:boto:Data: 2022-09-02T04:42:16.855 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.856 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.856 DEBUG:boto:Params: {} 2022-09-02T04:42:16.857 DEBUG:boto:Token: None 2022-09-02T04:42:16.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d44f6393ff875c-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.906 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.908 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.909 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.909 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.910 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.911 DEBUG:boto:Method: GET 2022-09-02T04:42:16.911 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.912 DEBUG:boto:Data: 2022-09-02T04:42:16.912 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.914 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.915 DEBUG:boto:Params: {} 2022-09-02T04:42:16.915 DEBUG:boto:Token: None 2022-09-02T04:42:16.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI= 2022-09-02T04:42:16.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI=', 'Content-Length': '0'} 2022-09-02T04:42:16.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002badef7621d14a35-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.922 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.922 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.923 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.924 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.924 DEBUG:boto:Method: GET 2022-09-02T04:42:16.925 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.925 DEBUG:boto:Data: 2022-09-02T04:42:16.926 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.927 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.927 DEBUG:boto:Params: {} 2022-09-02T04:42:16.928 DEBUG:boto:Token: None 2022-09-02T04:42:16.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI= 2022-09-02T04:42:16.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI=', 'Content-Length': '0'} 2022-09-02T04:42:16.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3f79345d5f07c16-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.933 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.934 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.934 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.935 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.935 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.936 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.936 DEBUG:boto:Data: 2022-09-02T04:42:16.937 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.938 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.939 DEBUG:boto:Params: {} 2022-09-02T04:42:16.939 DEBUG:boto:Token: None 2022-09-02T04:42:16.940 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AdBr1wAu3tJMR60aNiyPTxjglqs= 2022-09-02T04:42:16.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AdBr1wAu3tJMR60aNiyPTxjglqs=', 'Content-Length': '0'} 2022-09-02T04:42:16.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007159d480bbb72ccd-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.944 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.945 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.946 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.946 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.947 DEBUG:boto:Method: HEAD 2022-09-02T04:42:16.947 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.948 DEBUG:boto:Data: 2022-09-02T04:42:16.948 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.950 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.950 DEBUG:boto:Params: {} 2022-09-02T04:42:16.951 DEBUG:boto:Token: None 2022-09-02T04:42:16.952 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AdBr1wAu3tJMR60aNiyPTxjglqs= 2022-09-02T04:42:16.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AdBr1wAu3tJMR60aNiyPTxjglqs=', 'Content-Length': '0'} 2022-09-02T04:42:16.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b79ded9ae4611fa-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:16.956 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.957 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.958 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.958 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.959 DEBUG:boto:Method: GET 2022-09-02T04:42:16.959 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.960 DEBUG:boto:Data: 2022-09-02T04:42:16.960 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.961 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.962 DEBUG:boto:Params: {} 2022-09-02T04:42:16.963 DEBUG:boto:Token: None 2022-09-02T04:42:16.964 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI= 2022-09-02T04:42:16.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI=', 'Content-Length': '0'} 2022-09-02T04:42:16.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000852246eb06351508-00631189a8-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.969 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:16.969 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:16.970 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.970 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.971 DEBUG:boto:Method: GET 2022-09-02T04:42:16.971 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.972 DEBUG:boto:Data: 2022-09-02T04:42:16.973 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:16.975 DEBUG:boto:Port: 8000 2022-09-02T04:42:16.976 DEBUG:boto:Params: {} 2022-09-02T04:42:16.977 DEBUG:boto:Token: None 2022-09-02T04:42:16.978 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:16.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI= 2022-09-02T04:42:16.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7RaC/MX9XUhv3xUegVmJ0oNE0FI=', 'Content-Length': '0'} 2022-09-02T04:42:16.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000314d68105ca13c23-00631189a8-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.982 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.983 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.984 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.984 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.985 DEBUG:boto:Method: GET 2022-09-02T04:42:16.985 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.986 DEBUG:boto:Data: 2022-09-02T04:42:16.987 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:16.988 DEBUG:boto:Port: 8001 2022-09-02T04:42:16.988 DEBUG:boto:Params: {} 2022-09-02T04:42:16.989 DEBUG:boto:Token: None 2022-09-02T04:42:16.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:16 GMT /xljbkf-19/ 2022-09-02T04:42:16.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528= 2022-09-02T04:42:16.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r5GMmrxd7gKAsbrBYt+ldQ5e528=', 'Content-Length': '0'} 2022-09-02T04:42:16.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4a8d7371f1cf4a-00631189a8-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:16.994 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:16.995 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:16.996 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:16.996 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.997 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:16.997 DEBUG:boto:Method: GET 2022-09-02T04:42:16.998 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:16.998 DEBUG:boto:Data: 2022-09-02T04:42:16.999 DEBUG:boto:Headers: {} 2022-09-02T04:42:16.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.000 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.000 DEBUG:boto:Params: {} 2022-09-02T04:42:17.001 DEBUG:boto:Token: None 2022-09-02T04:42:17.002 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084e222aaa11d98a1-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.006 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.007 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.008 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.008 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.009 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.009 DEBUG:boto:Data: 2022-09-02T04:42:17.010 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.011 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.011 DEBUG:boto:Params: {} 2022-09-02T04:42:17.012 DEBUG:boto:Token: None 2022-09-02T04:42:17.013 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048a9946a47f875c4-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.018 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.019 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.019 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.020 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.020 DEBUG:boto:Data: 2022-09-02T04:42:17.021 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.022 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.022 DEBUG:boto:Params: {} 2022-09-02T04:42:17.023 DEBUG:boto:Token: None 2022-09-02T04:42:17.023 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002924b6fbae4044b2-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.030 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.031 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.031 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.032 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.033 DEBUG:boto:Method: GET 2022-09-02T04:42:17.033 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.034 DEBUG:boto:Data: 2022-09-02T04:42:17.034 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.036 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.036 DEBUG:boto:Params: {} 2022-09-02T04:42:17.037 DEBUG:boto:Token: None 2022-09-02T04:42:17.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a17d6fd9dea36f7-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.041 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.042 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.043 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.043 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.044 DEBUG:boto:Method: GET 2022-09-02T04:42:17.044 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.045 DEBUG:boto:Data: 2022-09-02T04:42:17.045 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.046 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.047 DEBUG:boto:Params: {} 2022-09-02T04:42:17.047 DEBUG:boto:Token: None 2022-09-02T04:42:17.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2ca26c8804b8a38-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.094 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.095 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.096 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.097 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.098 DEBUG:boto:Method: GET 2022-09-02T04:42:17.099 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.100 DEBUG:boto:Data: 2022-09-02T04:42:17.100 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.102 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.103 DEBUG:boto:Params: {} 2022-09-02T04:42:17.104 DEBUG:boto:Token: None 2022-09-02T04:42:17.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f331ace8bbb480-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.110 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.111 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.111 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.112 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.112 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.113 DEBUG:boto:Method: GET 2022-09-02T04:42:17.113 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.114 DEBUG:boto:Data: 2022-09-02T04:42:17.114 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.115 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.116 DEBUG:boto:Params: {} 2022-09-02T04:42:17.116 DEBUG:boto:Token: None 2022-09-02T04:42:17.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1d54aa42cdcd6b7-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.122 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.123 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.124 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.124 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.125 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.125 DEBUG:boto:Method: GET 2022-09-02T04:42:17.126 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.127 DEBUG:boto:Data: 2022-09-02T04:42:17.127 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.128 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.129 DEBUG:boto:Params: {} 2022-09-02T04:42:17.129 DEBUG:boto:Token: None 2022-09-02T04:42:17.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4= 2022-09-02T04:42:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4=', 'Content-Length': '0'} 2022-09-02T04:42:17.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009abd3be090599aa8-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.134 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.135 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.135 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.136 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.136 DEBUG:boto:Method: GET 2022-09-02T04:42:17.137 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.137 DEBUG:boto:Data: 2022-09-02T04:42:17.138 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.139 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.139 DEBUG:boto:Params: {} 2022-09-02T04:42:17.140 DEBUG:boto:Token: None 2022-09-02T04:42:17.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4= 2022-09-02T04:42:17.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4=', 'Content-Length': '0'} 2022-09-02T04:42:17.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9168e7b8b7233d5-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.145 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.146 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.146 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.147 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.147 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.148 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.148 DEBUG:boto:Data: 2022-09-02T04:42:17.149 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.150 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.150 DEBUG:boto:Params: {} 2022-09-02T04:42:17.151 DEBUG:boto:Token: None 2022-09-02T04:42:17.152 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xJH28dOpOkKgNYYn3pCXYgA28AQ= 2022-09-02T04:42:17.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xJH28dOpOkKgNYYn3pCXYgA28AQ=', 'Content-Length': '0'} 2022-09-02T04:42:17.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a16027b4da34d33-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.156 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.156 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.157 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.157 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.158 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.158 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.159 DEBUG:boto:Data: 2022-09-02T04:42:17.159 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.160 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.161 DEBUG:boto:Params: {} 2022-09-02T04:42:17.161 DEBUG:boto:Token: None 2022-09-02T04:42:17.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xJH28dOpOkKgNYYn3pCXYgA28AQ= 2022-09-02T04:42:17.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xJH28dOpOkKgNYYn3pCXYgA28AQ=', 'Content-Length': '0'} 2022-09-02T04:42:17.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdb80687028cce68-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.167 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.167 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.168 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.168 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.169 DEBUG:boto:Method: GET 2022-09-02T04:42:17.169 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.170 DEBUG:boto:Data: 2022-09-02T04:42:17.171 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.172 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.172 DEBUG:boto:Params: {} 2022-09-02T04:42:17.173 DEBUG:boto:Token: None 2022-09-02T04:42:17.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4= 2022-09-02T04:42:17.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4=', 'Content-Length': '0'} 2022-09-02T04:42:17.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e663d991b14c24c-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.226 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.227 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.228 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.229 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.229 DEBUG:boto:Method: GET 2022-09-02T04:42:17.230 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.231 DEBUG:boto:Data: 2022-09-02T04:42:17.232 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.234 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.235 DEBUG:boto:Params: {} 2022-09-02T04:42:17.236 DEBUG:boto:Token: None 2022-09-02T04:42:17.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:17.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4= 2022-09-02T04:42:17.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j9lbPKQHPpR/aPYbmClnXE3Gqh4=', 'Content-Length': '0'} 2022-09-02T04:42:17.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfb69ba5fc725588-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.243 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.244 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.245 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.246 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.247 DEBUG:boto:Method: GET 2022-09-02T04:42:17.248 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.248 DEBUG:boto:Data: 2022-09-02T04:42:17.249 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.250 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.250 DEBUG:boto:Params: {} 2022-09-02T04:42:17.251 DEBUG:boto:Token: None 2022-09-02T04:42:17.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ddf8ee5a8bf7130-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.256 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.257 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.258 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.259 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.259 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.260 DEBUG:boto:Method: GET 2022-09-02T04:42:17.260 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.261 DEBUG:boto:Data: 2022-09-02T04:42:17.261 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.262 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.263 DEBUG:boto:Params: {} 2022-09-02T04:42:17.264 DEBUG:boto:Token: None 2022-09-02T04:42:17.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f86a8578512136d-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.271 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.272 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.273 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.274 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.275 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.275 DEBUG:boto:Data: 2022-09-02T04:42:17.276 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.277 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.277 DEBUG:boto:Params: {} 2022-09-02T04:42:17.278 DEBUG:boto:Token: None 2022-09-02T04:42:17.279 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000738a65e6b8286888-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.284 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.285 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.285 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.286 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.287 DEBUG:boto:Data: 2022-09-02T04:42:17.287 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.288 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.289 DEBUG:boto:Params: {} 2022-09-02T04:42:17.290 DEBUG:boto:Token: None 2022-09-02T04:42:17.290 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8692b943fb6b2d6-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.296 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.297 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.298 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.298 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.299 DEBUG:boto:Method: GET 2022-09-02T04:42:17.300 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.300 DEBUG:boto:Data: 2022-09-02T04:42:17.301 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.303 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.304 DEBUG:boto:Params: {} 2022-09-02T04:42:17.305 DEBUG:boto:Token: None 2022-09-02T04:42:17.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c2c4ce7502b1d27-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.313 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.313 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.314 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.315 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.315 DEBUG:boto:Method: GET 2022-09-02T04:42:17.316 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.316 DEBUG:boto:Data: 2022-09-02T04:42:17.317 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.318 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.318 DEBUG:boto:Params: {} 2022-09-02T04:42:17.319 DEBUG:boto:Token: None 2022-09-02T04:42:17.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082a90e00470a5d90-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.324 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.324 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.325 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.326 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.326 DEBUG:boto:Method: GET 2022-09-02T04:42:17.327 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.327 DEBUG:boto:Data: 2022-09-02T04:42:17.328 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.330 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.330 DEBUG:boto:Params: {} 2022-09-02T04:42:17.331 DEBUG:boto:Token: None 2022-09-02T04:42:17.332 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af118aae150be10f-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.337 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.338 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.339 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.339 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.340 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.340 DEBUG:boto:Method: GET 2022-09-02T04:42:17.341 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.342 DEBUG:boto:Data: 2022-09-02T04:42:17.342 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.343 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.344 DEBUG:boto:Params: {} 2022-09-02T04:42:17.344 DEBUG:boto:Token: None 2022-09-02T04:42:17.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4f78621d933a72d-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.390 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.393 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.394 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.394 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.395 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.395 DEBUG:boto:Method: GET 2022-09-02T04:42:17.396 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.396 DEBUG:boto:Data: 2022-09-02T04:42:17.397 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.398 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.399 DEBUG:boto:Params: {} 2022-09-02T04:42:17.399 DEBUG:boto:Token: None 2022-09-02T04:42:17.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w= 2022-09-02T04:42:17.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w=', 'Content-Length': '0'} 2022-09-02T04:42:17.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da4c621630494e69-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.405 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.405 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.406 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.407 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.407 DEBUG:boto:Method: GET 2022-09-02T04:42:17.408 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.408 DEBUG:boto:Data: 2022-09-02T04:42:17.409 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.410 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.411 DEBUG:boto:Params: {} 2022-09-02T04:42:17.411 DEBUG:boto:Token: None 2022-09-02T04:42:17.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w= 2022-09-02T04:42:17.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w=', 'Content-Length': '0'} 2022-09-02T04:42:17.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003e8094e5dd5e033-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.419 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.420 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.421 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.421 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.422 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.423 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.423 DEBUG:boto:Data: 2022-09-02T04:42:17.424 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.426 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.427 DEBUG:boto:Params: {} 2022-09-02T04:42:17.428 DEBUG:boto:Token: None 2022-09-02T04:42:17.429 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KoNKQdWT4ym42jIO1ZPBW6fP1mc= 2022-09-02T04:42:17.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KoNKQdWT4ym42jIO1ZPBW6fP1mc=', 'Content-Length': '0'} 2022-09-02T04:42:17.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6543d542fbe6e14-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.435 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.436 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.437 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.438 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.439 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.440 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.440 DEBUG:boto:Data: 2022-09-02T04:42:17.441 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.443 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.444 DEBUG:boto:Params: {} 2022-09-02T04:42:17.445 DEBUG:boto:Token: None 2022-09-02T04:42:17.446 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KoNKQdWT4ym42jIO1ZPBW6fP1mc= 2022-09-02T04:42:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KoNKQdWT4ym42jIO1ZPBW6fP1mc=', 'Content-Length': '0'} 2022-09-02T04:42:17.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b112c3c5c9a7edf-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.454 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.455 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.455 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.456 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.457 DEBUG:boto:Method: GET 2022-09-02T04:42:17.457 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.458 DEBUG:boto:Data: 2022-09-02T04:42:17.458 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.460 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.460 DEBUG:boto:Params: {} 2022-09-02T04:42:17.461 DEBUG:boto:Token: None 2022-09-02T04:42:17.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w= 2022-09-02T04:42:17.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w=', 'Content-Length': '0'} 2022-09-02T04:42:17.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000ae3905598a4cca-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.506 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.507 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.507 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.508 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.508 DEBUG:boto:Method: GET 2022-09-02T04:42:17.509 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.509 DEBUG:boto:Data: 2022-09-02T04:42:17.510 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.511 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.512 DEBUG:boto:Params: {} 2022-09-02T04:42:17.512 DEBUG:boto:Token: None 2022-09-02T04:42:17.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:17.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w= 2022-09-02T04:42:17.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTc78lIhobJ8uDQltg7A8RmfP+w=', 'Content-Length': '0'} 2022-09-02T04:42:17.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011a3f3f37f3a72a1-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.518 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.518 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.519 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.519 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.520 DEBUG:boto:Method: GET 2022-09-02T04:42:17.521 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.522 DEBUG:boto:Data: 2022-09-02T04:42:17.523 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.525 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.526 DEBUG:boto:Params: {} 2022-09-02T04:42:17.527 DEBUG:boto:Token: None 2022-09-02T04:42:17.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b395053eaac544f5-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.534 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.535 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.536 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.537 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.538 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.539 DEBUG:boto:Method: GET 2022-09-02T04:42:17.540 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.540 DEBUG:boto:Data: 2022-09-02T04:42:17.541 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.543 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.544 DEBUG:boto:Params: {} 2022-09-02T04:42:17.546 DEBUG:boto:Token: None 2022-09-02T04:42:17.547 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043330a368afc22c7-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.553 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.554 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.555 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.556 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.557 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.557 DEBUG:boto:Data: 2022-09-02T04:42:17.558 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.559 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.559 DEBUG:boto:Params: {} 2022-09-02T04:42:17.560 DEBUG:boto:Token: None 2022-09-02T04:42:17.561 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046b1647ab5b30a04-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.569 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.569 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.570 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.570 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.571 DEBUG:boto:Data: 2022-09-02T04:42:17.571 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.572 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.573 DEBUG:boto:Params: {} 2022-09-02T04:42:17.573 DEBUG:boto:Token: None 2022-09-02T04:42:17.574 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daf75c86dfdd451d-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.580 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.581 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.581 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.582 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.582 DEBUG:boto:Method: GET 2022-09-02T04:42:17.583 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.583 DEBUG:boto:Data: 2022-09-02T04:42:17.584 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.585 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.586 DEBUG:boto:Params: {} 2022-09-02T04:42:17.586 DEBUG:boto:Token: None 2022-09-02T04:42:17.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077223a74f3d08743-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.592 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.593 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.594 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.595 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.596 DEBUG:boto:Method: GET 2022-09-02T04:42:17.597 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.598 DEBUG:boto:Data: 2022-09-02T04:42:17.598 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.600 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.600 DEBUG:boto:Params: {} 2022-09-02T04:42:17.601 DEBUG:boto:Token: None 2022-09-02T04:42:17.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac579ee6a7a6a018-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.654 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.655 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.655 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.656 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.656 DEBUG:boto:Method: GET 2022-09-02T04:42:17.657 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.657 DEBUG:boto:Data: 2022-09-02T04:42:17.658 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.659 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.659 DEBUG:boto:Params: {} 2022-09-02T04:42:17.660 DEBUG:boto:Token: None 2022-09-02T04:42:17.661 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af781a312e216656-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.666 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.667 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.667 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.668 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.668 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.669 DEBUG:boto:Method: GET 2022-09-02T04:42:17.669 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.670 DEBUG:boto:Data: 2022-09-02T04:42:17.670 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.671 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.672 DEBUG:boto:Params: {} 2022-09-02T04:42:17.673 DEBUG:boto:Token: None 2022-09-02T04:42:17.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a7cb2a277990e1-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.678 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.679 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.680 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.680 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.681 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.682 DEBUG:boto:Method: GET 2022-09-02T04:42:17.682 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.683 DEBUG:boto:Data: 2022-09-02T04:42:17.683 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.684 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.685 DEBUG:boto:Params: {} 2022-09-02T04:42:17.685 DEBUG:boto:Token: None 2022-09-02T04:42:17.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo= 2022-09-02T04:42:17.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo=', 'Content-Length': '0'} 2022-09-02T04:42:17.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab8632a140b5c2aa-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.691 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.691 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.692 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.692 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.693 DEBUG:boto:Method: GET 2022-09-02T04:42:17.693 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.694 DEBUG:boto:Data: 2022-09-02T04:42:17.695 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.696 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.696 DEBUG:boto:Params: {} 2022-09-02T04:42:17.697 DEBUG:boto:Token: None 2022-09-02T04:42:17.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo= 2022-09-02T04:42:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo=', 'Content-Length': '0'} 2022-09-02T04:42:17.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a787b0745c87168f-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.701 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.702 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.703 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.703 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.704 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.704 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.705 DEBUG:boto:Data: 2022-09-02T04:42:17.705 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.706 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.707 DEBUG:boto:Params: {} 2022-09-02T04:42:17.707 DEBUG:boto:Token: None 2022-09-02T04:42:17.708 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jxDPBEDxnBdo7oMdnE/gH3tFKuM= 2022-09-02T04:42:17.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jxDPBEDxnBdo7oMdnE/gH3tFKuM=', 'Content-Length': '0'} 2022-09-02T04:42:17.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8cc69abe52c4a0d-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.712 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.712 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.713 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.713 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.714 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.715 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.715 DEBUG:boto:Data: 2022-09-02T04:42:17.716 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.717 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.717 DEBUG:boto:Params: {} 2022-09-02T04:42:17.718 DEBUG:boto:Token: None 2022-09-02T04:42:17.718 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jxDPBEDxnBdo7oMdnE/gH3tFKuM= 2022-09-02T04:42:17.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jxDPBEDxnBdo7oMdnE/gH3tFKuM=', 'Content-Length': '0'} 2022-09-02T04:42:17.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca8ab1df051fe7b4-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.723 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.723 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.724 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.724 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.725 DEBUG:boto:Method: GET 2022-09-02T04:42:17.725 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.726 DEBUG:boto:Data: 2022-09-02T04:42:17.726 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.728 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.728 DEBUG:boto:Params: {} 2022-09-02T04:42:17.729 DEBUG:boto:Token: None 2022-09-02T04:42:17.729 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo= 2022-09-02T04:42:17.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo=', 'Content-Length': '0'} 2022-09-02T04:42:17.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000710867b579819726-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.733 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.734 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.734 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.735 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.735 DEBUG:boto:Method: GET 2022-09-02T04:42:17.736 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.736 DEBUG:boto:Data: 2022-09-02T04:42:17.737 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.738 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.739 DEBUG:boto:Params: {} 2022-09-02T04:42:17.739 DEBUG:boto:Token: None 2022-09-02T04:42:17.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:17.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo= 2022-09-02T04:42:17.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7eZFxYkdbpxMVNAXpj6QLA5jOOo=', 'Content-Length': '0'} 2022-09-02T04:42:17.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c7b6c89843973c3-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.744 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.745 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.745 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.746 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.746 DEBUG:boto:Method: GET 2022-09-02T04:42:17.747 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.747 DEBUG:boto:Data: 2022-09-02T04:42:17.748 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.749 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.749 DEBUG:boto:Params: {} 2022-09-02T04:42:17.750 DEBUG:boto:Token: None 2022-09-02T04:42:17.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde0283b1764e3a7-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.755 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.756 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.757 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.757 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.758 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.758 DEBUG:boto:Method: GET 2022-09-02T04:42:17.759 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.759 DEBUG:boto:Data: 2022-09-02T04:42:17.760 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.761 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.762 DEBUG:boto:Params: {} 2022-09-02T04:42:17.762 DEBUG:boto:Token: None 2022-09-02T04:42:17.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000522a01f0bf243d8a-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.768 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.769 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.769 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.770 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.770 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.771 DEBUG:boto:Data: 2022-09-02T04:42:17.772 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.773 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.773 DEBUG:boto:Params: {} 2022-09-02T04:42:17.774 DEBUG:boto:Token: None 2022-09-02T04:42:17.774 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bd4943d1350198a-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.781 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.782 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.783 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.784 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:17.785 DEBUG:boto:Data: 2022-09-02T04:42:17.786 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.788 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.788 DEBUG:boto:Params: {} 2022-09-02T04:42:17.789 DEBUG:boto:Token: None 2022-09-02T04:42:17.790 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj 2022-09-02T04:42:17.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48= 2022-09-02T04:42:17.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWrJLS6V9XoGvyPSkcdO0vBtN48=', 'Content-Length': '0'} 2022-09-02T04:42:17.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057d80283edbba6db-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.799 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.800 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.801 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.802 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.803 DEBUG:boto:Method: GET 2022-09-02T04:42:17.804 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.804 DEBUG:boto:Data: 2022-09-02T04:42:17.805 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.806 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.807 DEBUG:boto:Params: {} 2022-09-02T04:42:17.807 DEBUG:boto:Token: None 2022-09-02T04:42:17.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070096f4b36af47d7-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.854 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.855 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.856 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.856 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.857 DEBUG:boto:Method: GET 2022-09-02T04:42:17.858 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.859 DEBUG:boto:Data: 2022-09-02T04:42:17.859 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.860 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.861 DEBUG:boto:Params: {} 2022-09-02T04:42:17.862 DEBUG:boto:Token: None 2022-09-02T04:42:17.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:17.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw= 2022-09-02T04:42:17.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ymmznRaIy/07GNtyexDmTubtWw=', 'Content-Length': '0'} 2022-09-02T04:42:17.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092a977489d08f66f-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.867 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.868 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.869 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.869 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.870 DEBUG:boto:Method: GET 2022-09-02T04:42:17.870 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.871 DEBUG:boto:Data: 2022-09-02T04:42:17.871 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.873 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.874 DEBUG:boto:Params: {} 2022-09-02T04:42:17.875 DEBUG:boto:Token: None 2022-09-02T04:42:17.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df73238a82a1d0d-00631189a9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.880 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.881 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:17.881 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:17.882 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.883 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:17.883 DEBUG:boto:Method: GET 2022-09-02T04:42:17.884 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:17.884 DEBUG:boto:Data: 2022-09-02T04:42:17.885 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.886 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.886 DEBUG:boto:Params: {} 2022-09-02T04:42:17.887 DEBUG:boto:Token: None 2022-09-02T04:42:17.888 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/ 2022-09-02T04:42:17.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc= 2022-09-02T04:42:17.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i8sTNMWIGI/qf4+KcPL3YQjT2Vc=', 'Content-Length': '0'} 2022-09-02T04:42:17.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb22e1ecf1dad9e-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.934 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:17.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.936 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.937 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.938 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.939 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.940 DEBUG:boto:Method: GET 2022-09-02T04:42:17.940 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.941 DEBUG:boto:Data: 2022-09-02T04:42:17.942 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.944 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.945 DEBUG:boto:Params: {} 2022-09-02T04:42:17.946 DEBUG:boto:Token: None 2022-09-02T04:42:17.947 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+x507i0bg9z9ItFz6sY/oqx+lQg= 2022-09-02T04:42:17.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+x507i0bg9z9ItFz6sY/oqx+lQg=', 'Content-Length': '0'} 2022-09-02T04:42:17.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6d34fc7faeb3088-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.954 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.955 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.956 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.957 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.958 DEBUG:boto:Method: GET 2022-09-02T04:42:17.958 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.959 DEBUG:boto:Data: 2022-09-02T04:42:17.959 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.961 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.961 DEBUG:boto:Params: {} 2022-09-02T04:42:17.962 DEBUG:boto:Token: None 2022-09-02T04:42:17.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+x507i0bg9z9ItFz6sY/oqx+lQg= 2022-09-02T04:42:17.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+x507i0bg9z9ItFz6sY/oqx+lQg=', 'Content-Length': '0'} 2022-09-02T04:42:17.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a09004119bb8d371-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.969 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.969 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.970 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.970 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.971 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.971 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.972 DEBUG:boto:Data: 2022-09-02T04:42:17.973 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.974 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.974 DEBUG:boto:Params: {} 2022-09-02T04:42:17.975 DEBUG:boto:Token: None 2022-09-02T04:42:17.976 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xgivc2yNTysRMlVQT+B1ESlIFyk= 2022-09-02T04:42:17.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xgivc2yNTysRMlVQT+B1ESlIFyk=', 'Content-Length': '0'} 2022-09-02T04:42:17.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006375b49e7b74ff94-00631189a9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.980 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.980 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.981 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.981 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.982 DEBUG:boto:Method: HEAD 2022-09-02T04:42:17.982 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.983 DEBUG:boto:Data: 2022-09-02T04:42:17.983 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:17.984 DEBUG:boto:Port: 8000 2022-09-02T04:42:17.985 DEBUG:boto:Params: {} 2022-09-02T04:42:17.986 DEBUG:boto:Token: None 2022-09-02T04:42:17.986 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xgivc2yNTysRMlVQT+B1ESlIFyk= 2022-09-02T04:42:17.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xgivc2yNTysRMlVQT+B1ESlIFyk=', 'Content-Length': '0'} 2022-09-02T04:42:17.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022dff511af2572a0-00631189a9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:17.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:17.991 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:17.992 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:17.993 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.993 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.994 DEBUG:boto:Method: GET 2022-09-02T04:42:17.994 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.995 DEBUG:boto:Data: 2022-09-02T04:42:17.995 DEBUG:boto:Headers: {} 2022-09-02T04:42:17.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:17.996 DEBUG:boto:Port: 8001 2022-09-02T04:42:17.997 DEBUG:boto:Params: {} 2022-09-02T04:42:17.997 DEBUG:boto:Token: None 2022-09-02T04:42:17.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:17 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:17.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+x507i0bg9z9ItFz6sY/oqx+lQg= 2022-09-02T04:42:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+x507i0bg9z9ItFz6sY/oqx+lQg=', 'Content-Length': '0'} 2022-09-02T04:42:18.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050a95a41fa253669-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.004 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.005 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.005 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:18.006 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:18.006 DEBUG:boto:Method: GET 2022-09-02T04:42:18.007 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:18.008 DEBUG:boto:Data: 2022-09-02T04:42:18.008 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:18.009 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.010 DEBUG:boto:Params: {} 2022-09-02T04:42:18.010 DEBUG:boto:Token: None 2022-09-02T04:42:18.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:18.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:026UOnRedDlWogBNTuK42t5Bwco= 2022-09-02T04:42:18.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:026UOnRedDlWogBNTuK42t5Bwco=', 'Content-Length': '0'} 2022-09-02T04:42:18.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fc3d303c3965cfa-00631189aa-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.015 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.016 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.017 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.017 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.018 DEBUG:boto:Method: GET 2022-09-02T04:42:18.019 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.019 DEBUG:boto:Data: 2022-09-02T04:42:18.021 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.023 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.023 DEBUG:boto:Params: {} 2022-09-02T04:42:18.024 DEBUG:boto:Token: None 2022-09-02T04:42:18.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd753c7341d66a01-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.030 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.032 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.033 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.033 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.034 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.035 DEBUG:boto:Method: GET 2022-09-02T04:42:18.036 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.036 DEBUG:boto:Data: 2022-09-02T04:42:18.037 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:18.038 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.039 DEBUG:boto:Params: {} 2022-09-02T04:42:18.039 DEBUG:boto:Token: None 2022-09-02T04:42:18.040 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039da98be155a1f8f-00631189aa-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.090 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.091 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.092 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.092 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.093 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:18.093 DEBUG:boto:Data: 2022-09-02T04:42:18.094 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.095 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.096 DEBUG:boto:Params: {} 2022-09-02T04:42:18.097 DEBUG:boto:Token: None 2022-09-02T04:42:18.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj 2022-09-02T04:42:18.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4= 2022-09-02T04:42:18.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4=', 'Content-Length': '0'} 2022-09-02T04:42:18.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079dfc39abbc8a10f-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.105 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.106 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.107 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.108 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:18.109 DEBUG:boto:Data: 2022-09-02T04:42:18.110 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:18.112 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.113 DEBUG:boto:Params: {} 2022-09-02T04:42:18.114 DEBUG:boto:Token: None 2022-09-02T04:42:18.115 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj 2022-09-02T04:42:18.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4= 2022-09-02T04:42:18.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4=', 'Content-Length': '0'} 2022-09-02T04:42:18.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000785d853426342262-00631189aa-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:18.131 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.132 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.133 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.134 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.135 DEBUG:boto:Method: GET 2022-09-02T04:42:18.136 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.137 DEBUG:boto:Data: 2022-09-02T04:42:18.138 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.140 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.140 DEBUG:boto:Params: {} 2022-09-02T04:42:18.142 DEBUG:boto:Token: None 2022-09-02T04:42:18.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.148 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0a452ff05fb2cad-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.149 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.149 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.150 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.151 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.151 DEBUG:boto:Method: GET 2022-09-02T04:42:18.152 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.152 DEBUG:boto:Data: 2022-09-02T04:42:18.153 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:18.154 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.154 DEBUG:boto:Params: {} 2022-09-02T04:42:18.155 DEBUG:boto:Token: None 2022-09-02T04:42:18.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e79fd163bc5c30fb-00631189aa-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.162 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.163 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.164 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.165 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.166 DEBUG:boto:Method: GET 2022-09-02T04:42:18.167 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.168 DEBUG:boto:Data: 2022-09-02T04:42:18.169 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.171 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.171 DEBUG:boto:Params: {} 2022-09-02T04:42:18.172 DEBUG:boto:Token: None 2022-09-02T04:42:18.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047abe2e7accd5d87-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.179 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.180 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.181 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.182 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.183 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.184 DEBUG:boto:Method: GET 2022-09-02T04:42:18.185 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.185 DEBUG:boto:Data: 2022-09-02T04:42:18.186 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:18.188 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.189 DEBUG:boto:Params: {} 2022-09-02T04:42:18.190 DEBUG:boto:Token: None 2022-09-02T04:42:18.192 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.193 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9888cee819a9a88-00631189aa-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.198 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-19 zones={test-zone3, test-zone1} 2022-09-02T04:42:18.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-19 zones={test-zone3, test-zone2} 2022-09-02T04:42:18.200 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.201 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.201 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.202 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:42:18.202 DEBUG:boto:Data: 2022-09-02T04:42:18.203 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.204 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.204 DEBUG:boto:Params: {} 2022-09-02T04:42:18.205 DEBUG:boto:Token: None 2022-09-02T04:42:18.205 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RUalH8qRrAwsFePrquyJGBa9IpU= 2022-09-02T04:42:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RUalH8qRrAwsFePrquyJGBa9IpU=', 'Content-Length': '0'} 2022-09-02T04:42:18.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025801df2665c27b6-00631189aa-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.210 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.210 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.211 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.211 DEBUG:boto:Path: /xljbkf-19/ 2022-09-02T04:42:18.212 DEBUG:boto:Data: 2022-09-02T04:42:18.212 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.213 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.214 DEBUG:boto:Params: {} 2022-09-02T04:42:18.215 DEBUG:boto:Token: None 2022-09-02T04:42:18.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RUalH8qRrAwsFePrquyJGBa9IpU= 2022-09-02T04:42:18.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RUalH8qRrAwsFePrquyJGBa9IpU=', 'Content-Length': '0'} 2022-09-02T04:42:18.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d6ccba98dabf182-00631189aa-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.221 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:42:18.222 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.223 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.223 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.224 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.225 DEBUG:boto:Method: GET 2022-09-02T04:42:18.226 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.226 DEBUG:boto:Data: 2022-09-02T04:42:18.227 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.228 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.229 DEBUG:boto:Params: {} 2022-09-02T04:42:18.229 DEBUG:boto:Token: None 2022-09-02T04:42:18.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/?versions 2022-09-02T04:42:18.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc= 2022-09-02T04:42:18.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc=', 'Content-Length': '0'} 2022-09-02T04:42:18.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ca6e01acd03233-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.236 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.240 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:42:18.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.264 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:42:18.264 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.265 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.266 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.266 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.267 DEBUG:boto:Method: GET 2022-09-02T04:42:18.267 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.268 DEBUG:boto:Data: 2022-09-02T04:42:18.269 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.270 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.271 DEBUG:boto:Params: {} 2022-09-02T04:42:18.272 DEBUG:boto:Token: None 2022-09-02T04:42:18.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/?versions 2022-09-02T04:42:18.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc= 2022-09-02T04:42:18.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc=', 'Content-Length': '0'} 2022-09-02T04:42:18.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e989093ca16ddff-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.278 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.281 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-09-02T04:42:18.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:42:18.304 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.304 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.305 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.306 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.306 DEBUG:boto:Method: GET 2022-09-02T04:42:18.307 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.308 DEBUG:boto:Data: 2022-09-02T04:42:18.309 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.311 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.312 DEBUG:boto:Params: {} 2022-09-02T04:42:18.313 DEBUG:boto:Token: None 2022-09-02T04:42:18.314 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/?versions 2022-09-02T04:42:18.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc= 2022-09-02T04:42:18.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc=', 'Content-Length': '0'} 2022-09-02T04:42:18.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000896f7150c1354b48-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.321 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.324 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.324 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.325 DEBUG:boto:path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.325 DEBUG:boto:auth_path=/xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.326 DEBUG:boto:Method: GET 2022-09-02T04:42:18.327 DEBUG:boto:Path: /xljbkf-19/?versions&max-keys=999 2022-09-02T04:42:18.327 DEBUG:boto:Data: 2022-09-02T04:42:18.328 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.329 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.330 DEBUG:boto:Params: {} 2022-09-02T04:42:18.330 DEBUG:boto:Token: None 2022-09-02T04:42:18.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/?versions 2022-09-02T04:42:18.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc= 2022-09-02T04:42:18.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6U8yNyWYIuTtlUwyWLALpP9XHgc=', 'Content-Length': '0'} 2022-09-02T04:42:18.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050707f8bd1cadb93-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.338 DEBUG:boto:b'xljbkf-19999falsedummyRwQu6rF61PdI33CPZvqpcgXWC2GkhBgtrue2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGztrue2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje7qaPJG4fppaX5ghVZyre00NLz23VBnfalse2022-09-02T04:35:42.634Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsZN5xldKFUsj8CX0kacXkmK9a2VT2KDfalse2022-09-02T04:35:42.708Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-drxKfNi9vIRApncRBumUSOKO9YTEEzfalse2022-09-02T04:35:42.593Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMqL.50kcX0KhN7mJPr1ee83fjf8a7Icfalse2022-09-02T04:35:42.518Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjYFP884gcYoRnzZqz3gcCDRzzvE3Z5nYfalse2022-09-02T04:35:42.563Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZffalse2022-09-02T04:35:42.409Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNJMY0IMH4LrgnBeOYLOR9RN0EOHamj4false2022-09-02T04:35:42.481Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzgvcmovaY0vlO0ghuQ9TKaLCsjtQTuPfalse2022-09-02T04:35:42.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4TKoLa.-emUUJiNYIKXx8QDFZGXqr3lfalse2022-09-02T04:35:42.332Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMPSTvqfocHNpjqSSe1XHwMrhDdV5dDafalse2022-09-02T04:35:42.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjriISQQdNpXBqmhJe5KXTuGBBXBg640Rfalse2022-09-02T04:35:42.303Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjC.Anncl.p5MU1JnCw8HH1LhxTImrIT2false2022-09-02T04:35:42.241Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWdeoY5zneE2FAv88Q0QypGUzSkECcddfalse2022-09-02T04:35:42.148Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgA.6Quiy8aYXdSNGParYlZMUxQBATsLfalse2022-09-02T04:35:42.213Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7Yc35uuFZ1bDipSO-0afZ1kQQqX6kESfalse2022-09-02T04:35:42.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMK9VPPFy6IG1KOcSvpN3jERnLiRbAyGfalse2022-09-02T04:35:42.066Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzAZYkI9foiNudwZegqrWCiM-9J1OYVUfalse2022-09-02T04:35:42.018Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeBfalse2022-09-02T04:35:41.879Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgFfalse2022-09-02T04:35:41.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUB2I40j-SHaYLw4bCdkcpcE--BvSAl4false2022-09-02T04:35:41.922Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCsNSL980h52zVI65V9bT5gZak-3oUO6false2022-09-02T04:35:41.783Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLjNxkl08kx14W4bqr69fYD4nB2y0HPMfalse2022-09-02T04:35:41.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobju0TBmiuoo9kWSjCI8dtEPp.2cyghZt4false2022-09-02T04:35:41.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQfalse2022-09-02T04:35:41.685Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7v56mxcaqqipIwpQ7G9zjnh3lbrVkDmfalse2022-09-02T04:35:41.722Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFdBlPONqZDnEX45Z7KNq2vsHOx6AhIzfalse2022-09-02T04:35:41.646Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDoe5jdwZtcMR7HzAlJy83bxJ3p.mKxjfalse2022-09-02T04:35:41.529Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrGGeHMhO34Ab7pKILSilubA2BPH4WTfalse2022-09-02T04:35:41.572Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjt9jRBTjPWPur8r4C2pn1Lomi1F28VA0false2022-09-02T04:35:41.606Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:18.343 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.344 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.344 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.345 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.345 DEBUG:boto:Method: GET 2022-09-02T04:42:18.346 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.346 DEBUG:boto:Data: 2022-09-02T04:42:18.347 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.348 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.349 DEBUG:boto:Params: {} 2022-09-02T04:42:18.349 DEBUG:boto:Token: None 2022-09-02T04:42:18.350 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ= 2022-09-02T04:42:18.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1a4254ad88bf2ad-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.355 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.355 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.356 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.356 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.357 DEBUG:boto:Method: GET 2022-09-02T04:42:18.357 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.358 DEBUG:boto:Data: 2022-09-02T04:42:18.359 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.360 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.360 DEBUG:boto:Params: {} 2022-09-02T04:42:18.361 DEBUG:boto:Token: None 2022-09-02T04:42:18.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ= 2022-09-02T04:42:18.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ea764fe451230a1-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.366 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.366 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.367 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.367 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.368 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.368 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.369 DEBUG:boto:Data: 2022-09-02T04:42:18.369 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.370 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.371 DEBUG:boto:Params: {} 2022-09-02T04:42:18.371 DEBUG:boto:Token: None 2022-09-02T04:42:18.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HRQbLNlGLAHDr82w8fadrbS5OoE= 2022-09-02T04:42:18.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HRQbLNlGLAHDr82w8fadrbS5OoE=', 'Content-Length': '0'} 2022-09-02T04:42:18.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1579dd9991cb077-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.376 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.376 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.377 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.377 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.378 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.379 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.379 DEBUG:boto:Data: 2022-09-02T04:42:18.380 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.381 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.381 DEBUG:boto:Params: {} 2022-09-02T04:42:18.382 DEBUG:boto:Token: None 2022-09-02T04:42:18.382 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HRQbLNlGLAHDr82w8fadrbS5OoE= 2022-09-02T04:42:18.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HRQbLNlGLAHDr82w8fadrbS5OoE=', 'Content-Length': '0'} 2022-09-02T04:42:18.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd7ad80b4213776f-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:18.387 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.388 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.388 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.389 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.389 DEBUG:boto:Method: GET 2022-09-02T04:42:18.390 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.390 DEBUG:boto:Data: 2022-09-02T04:42:18.391 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.392 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.392 DEBUG:boto:Params: {} 2022-09-02T04:42:18.393 DEBUG:boto:Token: None 2022-09-02T04:42:18.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ= 2022-09-02T04:42:18.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbfdba1ced981586-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.398 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.399 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.400 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.400 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.401 DEBUG:boto:Method: GET 2022-09-02T04:42:18.401 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.402 DEBUG:boto:Data: 2022-09-02T04:42:18.402 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.403 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.404 DEBUG:boto:Params: {} 2022-09-02T04:42:18.405 DEBUG:boto:Token: None 2022-09-02T04:42:18.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ= 2022-09-02T04:42:18.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073b7eb279080eedb-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.410 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.410 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.411 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.411 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.412 DEBUG:boto:Method: GET 2022-09-02T04:42:18.412 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.413 DEBUG:boto:Data: 2022-09-02T04:42:18.413 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.415 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.416 DEBUG:boto:Params: {} 2022-09-02T04:42:18.416 DEBUG:boto:Token: None 2022-09-02T04:42:18.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033908c09df74e19-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.465 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.466 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.466 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.467 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.467 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.468 DEBUG:boto:Method: GET 2022-09-02T04:42:18.468 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.469 DEBUG:boto:Data: 2022-09-02T04:42:18.470 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.471 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.472 DEBUG:boto:Params: {} 2022-09-02T04:42:18.473 DEBUG:boto:Token: None 2022-09-02T04:42:18.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b684e23d4231a6af-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.479 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.481 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.482 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.483 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.483 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:42:18.484 DEBUG:boto:Data: 2022-09-02T04:42:18.484 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.485 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.486 DEBUG:boto:Params: {} 2022-09-02T04:42:18.486 DEBUG:boto:Token: None 2022-09-02T04:42:18.487 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy 2022-09-02T04:42:18.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9TFwKMDyDnyq+AOik+GY4+wSXdo= 2022-09-02T04:42:18.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9TFwKMDyDnyq+AOik+GY4+wSXdo=', 'Content-Length': '0'} 2022-09-02T04:42:18.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086c113bf40ecc7f9-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.501 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.501 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.502 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.503 DEBUG:boto:Path: /xljbkf-19/dummy 2022-09-02T04:42:18.504 DEBUG:boto:Data: 2022-09-02T04:42:18.504 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.505 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.505 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.506 DEBUG:boto:Params: {} 2022-09-02T04:42:18.507 DEBUG:boto:Token: None 2022-09-02T04:42:18.507 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy 2022-09-02T04:42:18.508 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9TFwKMDyDnyq+AOik+GY4+wSXdo= 2022-09-02T04:42:18.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9TFwKMDyDnyq+AOik+GY4+wSXdo=', 'Content-Length': '0'} 2022-09-02T04:42:18.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012239c0f77275f3c-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-09-02T04:42:18.512 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.513 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.514 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.514 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.515 DEBUG:boto:Method: GET 2022-09-02T04:42:18.515 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.516 DEBUG:boto:Data: 2022-09-02T04:42:18.516 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.518 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.518 DEBUG:boto:Params: {} 2022-09-02T04:42:18.519 DEBUG:boto:Token: None 2022-09-02T04:42:18.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ= 2022-09-02T04:42:18.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b51c560bda3bca6e-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.524 DEBUG:boto:path=/xljbkf-19/dummy 2022-09-02T04:42:18.524 DEBUG:boto:auth_path=/xljbkf-19/dummy 2022-09-02T04:42:18.525 DEBUG:boto:path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.525 DEBUG:boto:auth_path=/xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.526 DEBUG:boto:Method: GET 2022-09-02T04:42:18.527 DEBUG:boto:Path: /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.527 DEBUG:boto:Data: 2022-09-02T04:42:18.528 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.529 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.529 DEBUG:boto:Params: {} 2022-09-02T04:42:18.530 DEBUG:boto:Token: None 2022-09-02T04:42:18.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/dummy?versionId=RwQu6rF61PdI33CPZvqpcgXWC2GkhBg 2022-09-02T04:42:18.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ= 2022-09-02T04:42:18.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7amdb/rrwbYM1i+gqHeq2/LI0aQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:32:19 GMT'), ('x-amz-version-id', 'RwQu6rF61PdI33CPZvqpcgXWC2GkhBg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005979066fcb488ebf-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.535 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.535 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.536 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.536 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.537 DEBUG:boto:Method: GET 2022-09-02T04:42:18.538 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.538 DEBUG:boto:Data: 2022-09-02T04:42:18.539 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.540 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.540 DEBUG:boto:Params: {} 2022-09-02T04:42:18.541 DEBUG:boto:Token: None 2022-09-02T04:42:18.542 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012112c9782c8be79-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.546 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.547 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.548 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.548 DEBUG:boto:path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.549 DEBUG:boto:auth_path=/xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.549 DEBUG:boto:Method: GET 2022-09-02T04:42:18.550 DEBUG:boto:Path: /xljbkf-19/?prefix=dummy 2022-09-02T04:42:18.550 DEBUG:boto:Data: 2022-09-02T04:42:18.551 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.552 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.553 DEBUG:boto:Params: {} 2022-09-02T04:42:18.553 DEBUG:boto:Token: None 2022-09-02T04:42:18.554 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e5fa00bb68bcede-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.598 DEBUG:boto:b'xljbkf-19dummy1000falsedummy2022-09-02T04:32:19.551Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:18.600 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.600 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.601 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.602 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.602 DEBUG:boto:Method: GET 2022-09-02T04:42:18.603 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.603 DEBUG:boto:Data: 2022-09-02T04:42:18.604 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.605 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.606 DEBUG:boto:Params: {} 2022-09-02T04:42:18.607 DEBUG:boto:Token: None 2022-09-02T04:42:18.608 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061a1b3bb77b85170-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.614 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.615 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.616 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.617 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.618 DEBUG:boto:Method: GET 2022-09-02T04:42:18.619 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.619 DEBUG:boto:Data: 2022-09-02T04:42:18.620 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.621 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.621 DEBUG:boto:Params: {} 2022-09-02T04:42:18.622 DEBUG:boto:Token: None 2022-09-02T04:42:18.623 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3c70851b0ccd13d-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.627 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.628 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.628 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.629 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.630 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.630 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.631 DEBUG:boto:Data: 2022-09-02T04:42:18.631 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.632 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.633 DEBUG:boto:Params: {} 2022-09-02T04:42:18.633 DEBUG:boto:Token: None 2022-09-02T04:42:18.634 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W6Md5B9XFDqQ6T5ttHDv44xiNFM= 2022-09-02T04:42:18.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W6Md5B9XFDqQ6T5ttHDv44xiNFM=', 'Content-Length': '0'} 2022-09-02T04:42:18.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020d82edf1e7a4e72-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.640 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.641 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.642 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.642 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.643 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.644 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.645 DEBUG:boto:Data: 2022-09-02T04:42:18.646 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.647 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.648 DEBUG:boto:Params: {} 2022-09-02T04:42:18.649 DEBUG:boto:Token: None 2022-09-02T04:42:18.649 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W6Md5B9XFDqQ6T5ttHDv44xiNFM= 2022-09-02T04:42:18.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W6Md5B9XFDqQ6T5ttHDv44xiNFM=', 'Content-Length': '0'} 2022-09-02T04:42:18.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b42e645252c1fd5b-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:18.655 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.656 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.657 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.658 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.659 DEBUG:boto:Method: GET 2022-09-02T04:42:18.660 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.660 DEBUG:boto:Data: 2022-09-02T04:42:18.661 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.663 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.664 DEBUG:boto:Params: {} 2022-09-02T04:42:18.665 DEBUG:boto:Token: None 2022-09-02T04:42:18.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000036265aa5477a642-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.714 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.715 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.715 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.716 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.716 DEBUG:boto:Method: GET 2022-09-02T04:42:18.717 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.718 DEBUG:boto:Data: 2022-09-02T04:42:18.718 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.719 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.720 DEBUG:boto:Params: {} 2022-09-02T04:42:18.721 DEBUG:boto:Token: None 2022-09-02T04:42:18.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3b98e3007690320-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.726 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.727 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.727 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.728 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.728 DEBUG:boto:Method: GET 2022-09-02T04:42:18.729 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.729 DEBUG:boto:Data: 2022-09-02T04:42:18.730 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.731 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.732 DEBUG:boto:Params: {} 2022-09-02T04:42:18.732 DEBUG:boto:Token: None 2022-09-02T04:42:18.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e95210a870543bc-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.738 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.739 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.740 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.740 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.741 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.742 DEBUG:boto:Method: GET 2022-09-02T04:42:18.742 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.743 DEBUG:boto:Data: 2022-09-02T04:42:18.743 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.745 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.745 DEBUG:boto:Params: {} 2022-09-02T04:42:18.746 DEBUG:boto:Token: None 2022-09-02T04:42:18.747 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c656bf55b8ff6b25-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.753 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.755 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.756 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.757 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.758 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:18.759 DEBUG:boto:Data: 2022-09-02T04:42:18.760 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.761 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.762 DEBUG:boto:Params: {} 2022-09-02T04:42:18.763 DEBUG:boto:Token: None 2022-09-02T04:42:18.765 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj 2022-09-02T04:42:18.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4= 2022-09-02T04:42:18.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4=', 'Content-Length': '0'} 2022-09-02T04:42:18.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000244474958372dc36-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.773 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.774 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.775 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.776 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:18.776 DEBUG:boto:Data: 2022-09-02T04:42:18.777 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.778 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.779 DEBUG:boto:Params: {} 2022-09-02T04:42:18.780 DEBUG:boto:Token: None 2022-09-02T04:42:18.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj 2022-09-02T04:42:18.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4= 2022-09-02T04:42:18.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s4ShqjWNDPnFSyuDSbr2cWoIHK4=', 'Content-Length': '0'} 2022-09-02T04:42:18.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1718cd2c5e625ba-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:18.789 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.790 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.791 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.792 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.792 DEBUG:boto:Method: GET 2022-09-02T04:42:18.793 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.793 DEBUG:boto:Data: 2022-09-02T04:42:18.794 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.795 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.796 DEBUG:boto:Params: {} 2022-09-02T04:42:18.797 DEBUG:boto:Token: None 2022-09-02T04:42:18.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096cf01fc05dced13-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.802 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.803 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.803 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.804 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.804 DEBUG:boto:Method: GET 2022-09-02T04:42:18.805 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.805 DEBUG:boto:Data: 2022-09-02T04:42:18.806 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.807 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.807 DEBUG:boto:Params: {} 2022-09-02T04:42:18.808 DEBUG:boto:Token: None 2022-09-02T04:42:18.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:18.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos= 2022-09-02T04:42:18.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pM0WF304XtYNCeiVRNGuj/QwCos=', 'Content-Length': '0'} 2022-09-02T04:42:18.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a0f36fcf18b112a-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.854 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.855 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.856 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.856 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.857 DEBUG:boto:Method: GET 2022-09-02T04:42:18.857 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.858 DEBUG:boto:Data: 2022-09-02T04:42:18.858 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.859 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.860 DEBUG:boto:Params: {} 2022-09-02T04:42:18.860 DEBUG:boto:Token: None 2022-09-02T04:42:18.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070973a14ee7144cd-00631189aa-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.866 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.867 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:18.868 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:18.868 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.869 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:18.869 DEBUG:boto:Method: GET 2022-09-02T04:42:18.870 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:18.871 DEBUG:boto:Data: 2022-09-02T04:42:18.871 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.873 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.873 DEBUG:boto:Params: {} 2022-09-02T04:42:18.874 DEBUG:boto:Token: None 2022-09-02T04:42:18.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/ 2022-09-02T04:42:18.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw= 2022-09-02T04:42:18.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsUS/zxlBW/ahkYmLctHVnceepw=', 'Content-Length': '0'} 2022-09-02T04:42:18.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6d877767fb7eaad-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.880 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:18.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:18.882 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.882 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.883 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.883 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.884 DEBUG:boto:Method: GET 2022-09-02T04:42:18.884 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.885 DEBUG:boto:Data: 2022-09-02T04:42:18.885 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.887 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.887 DEBUG:boto:Params: {} 2022-09-02T04:42:18.888 DEBUG:boto:Token: None 2022-09-02T04:42:18.888 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y= 2022-09-02T04:42:18.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y=', 'Content-Length': '0'} 2022-09-02T04:42:18.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fad4931948bc654b-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.893 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.894 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.895 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.895 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.896 DEBUG:boto:Method: GET 2022-09-02T04:42:18.897 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.898 DEBUG:boto:Data: 2022-09-02T04:42:18.898 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.899 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.900 DEBUG:boto:Params: {} 2022-09-02T04:42:18.901 DEBUG:boto:Token: None 2022-09-02T04:42:18.901 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y= 2022-09-02T04:42:18.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y=', 'Content-Length': '0'} 2022-09-02T04:42:18.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff0d9d0e3151a972-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.908 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.909 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.910 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.910 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.911 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.911 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.912 DEBUG:boto:Data: 2022-09-02T04:42:18.913 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.914 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.915 DEBUG:boto:Params: {} 2022-09-02T04:42:18.915 DEBUG:boto:Token: None 2022-09-02T04:42:18.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:baOLqJw9r8+pmrBmaGVaFsUDoQQ= 2022-09-02T04:42:18.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:baOLqJw9r8+pmrBmaGVaFsUDoQQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029d6251face687d0-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.922 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.923 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.924 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.925 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.926 DEBUG:boto:Method: HEAD 2022-09-02T04:42:18.926 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.927 DEBUG:boto:Data: 2022-09-02T04:42:18.927 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.929 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.930 DEBUG:boto:Params: {} 2022-09-02T04:42:18.931 DEBUG:boto:Token: None 2022-09-02T04:42:18.932 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:baOLqJw9r8+pmrBmaGVaFsUDoQQ= 2022-09-02T04:42:18.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:baOLqJw9r8+pmrBmaGVaFsUDoQQ=', 'Content-Length': '0'} 2022-09-02T04:42:18.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ce6087af0d5b36f-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:18.938 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.939 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.939 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.940 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.940 DEBUG:boto:Method: GET 2022-09-02T04:42:18.941 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.941 DEBUG:boto:Data: 2022-09-02T04:42:18.942 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:18.943 DEBUG:boto:Port: 8001 2022-09-02T04:42:18.944 DEBUG:boto:Params: {} 2022-09-02T04:42:18.944 DEBUG:boto:Token: None 2022-09-02T04:42:18.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y= 2022-09-02T04:42:18.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y=', 'Content-Length': '0'} 2022-09-02T04:42:18.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea07bb3a1b0e8ef4-00631189aa-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:18.950 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:18.950 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:18.951 DEBUG:boto:path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.951 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.952 DEBUG:boto:Method: GET 2022-09-02T04:42:18.952 DEBUG:boto:Path: /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.953 DEBUG:boto:Data: 2022-09-02T04:42:18.954 DEBUG:boto:Headers: {} 2022-09-02T04:42:18.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:18.955 DEBUG:boto:Port: 8000 2022-09-02T04:42:18.955 DEBUG:boto:Params: {} 2022-09-02T04:42:18.956 DEBUG:boto:Token: None 2022-09-02T04:42:18.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:18 GMT /xljbkf-19/obj?versionId=e7qaPJG4fppaX5ghVZyre00NLz23VBn 2022-09-02T04:42:18.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y= 2022-09-02T04:42:18.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xNTbglE96vKAJOF9EVP7rs/jT7Y=', 'Content-Length': '0'} 2022-09-02T04:42:18.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'e7qaPJG4fppaX5ghVZyre00NLz23VBn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d941abd56812074-00631189aa-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.002 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.003 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.003 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.004 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.004 DEBUG:boto:Method: GET 2022-09-02T04:42:19.005 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.006 DEBUG:boto:Data: 2022-09-02T04:42:19.006 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.008 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.008 DEBUG:boto:Params: {} 2022-09-02T04:42:19.009 DEBUG:boto:Token: None 2022-09-02T04:42:19.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d38ceca943da8efb-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.015 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.016 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.017 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.017 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.018 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.019 DEBUG:boto:Method: GET 2022-09-02T04:42:19.020 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.020 DEBUG:boto:Data: 2022-09-02T04:42:19.021 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.022 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.023 DEBUG:boto:Params: {} 2022-09-02T04:42:19.023 DEBUG:boto:Token: None 2022-09-02T04:42:19.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fa4c5063675128d-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.029 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.030 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.031 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.031 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.032 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.032 DEBUG:boto:Data: 2022-09-02T04:42:19.033 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.034 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.035 DEBUG:boto:Params: {} 2022-09-02T04:42:19.036 DEBUG:boto:Token: None 2022-09-02T04:42:19.037 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b185f8f3df2195d0-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.043 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.044 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.045 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.046 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.047 DEBUG:boto:Data: 2022-09-02T04:42:19.048 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.050 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.051 DEBUG:boto:Params: {} 2022-09-02T04:42:19.052 DEBUG:boto:Token: None 2022-09-02T04:42:19.053 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a29d5a20869d62e2-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.062 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.063 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.064 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.065 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.066 DEBUG:boto:Method: GET 2022-09-02T04:42:19.067 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.068 DEBUG:boto:Data: 2022-09-02T04:42:19.069 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.071 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.072 DEBUG:boto:Params: {} 2022-09-02T04:42:19.073 DEBUG:boto:Token: None 2022-09-02T04:42:19.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dd6a49af647460f-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.129 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.130 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.131 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.132 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.133 DEBUG:boto:Method: GET 2022-09-02T04:42:19.133 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.134 DEBUG:boto:Data: 2022-09-02T04:42:19.134 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.136 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.137 DEBUG:boto:Params: {} 2022-09-02T04:42:19.137 DEBUG:boto:Token: None 2022-09-02T04:42:19.138 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.141 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b675d26988bd9757-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.142 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.143 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.143 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.144 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.144 DEBUG:boto:Method: GET 2022-09-02T04:42:19.145 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.146 DEBUG:boto:Data: 2022-09-02T04:42:19.146 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.148 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.148 DEBUG:boto:Params: {} 2022-09-02T04:42:19.149 DEBUG:boto:Token: None 2022-09-02T04:42:19.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd6db4db254bf37-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.154 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.155 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.156 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.156 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.157 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.157 DEBUG:boto:Method: GET 2022-09-02T04:42:19.158 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.158 DEBUG:boto:Data: 2022-09-02T04:42:19.160 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.161 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.161 DEBUG:boto:Params: {} 2022-09-02T04:42:19.162 DEBUG:boto:Token: None 2022-09-02T04:42:19.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ffccce8dbef3b48-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.167 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.169 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.170 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.170 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.171 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.172 DEBUG:boto:Method: GET 2022-09-02T04:42:19.172 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.173 DEBUG:boto:Data: 2022-09-02T04:42:19.173 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.174 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.175 DEBUG:boto:Params: {} 2022-09-02T04:42:19.175 DEBUG:boto:Token: None 2022-09-02T04:42:19.176 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8= 2022-09-02T04:42:19.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8=', 'Content-Length': '0'} 2022-09-02T04:42:19.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d80e2c65b4d47597-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.181 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.181 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.182 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.182 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.183 DEBUG:boto:Method: GET 2022-09-02T04:42:19.183 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.184 DEBUG:boto:Data: 2022-09-02T04:42:19.184 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.185 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.186 DEBUG:boto:Params: {} 2022-09-02T04:42:19.187 DEBUG:boto:Token: None 2022-09-02T04:42:19.187 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8= 2022-09-02T04:42:19.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8=', 'Content-Length': '0'} 2022-09-02T04:42:19.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046982ba24651d1d5-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.192 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.192 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.193 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.193 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.194 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.195 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.195 DEBUG:boto:Data: 2022-09-02T04:42:19.196 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.197 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.197 DEBUG:boto:Params: {} 2022-09-02T04:42:19.198 DEBUG:boto:Token: None 2022-09-02T04:42:19.199 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kYDRNIzsngKu3rGXrhbBzkFerWg= 2022-09-02T04:42:19.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kYDRNIzsngKu3rGXrhbBzkFerWg=', 'Content-Length': '0'} 2022-09-02T04:42:19.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7396b18d869895b-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.203 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.203 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.204 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.204 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.205 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.205 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.206 DEBUG:boto:Data: 2022-09-02T04:42:19.207 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.208 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.209 DEBUG:boto:Params: {} 2022-09-02T04:42:19.209 DEBUG:boto:Token: None 2022-09-02T04:42:19.210 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kYDRNIzsngKu3rGXrhbBzkFerWg= 2022-09-02T04:42:19.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kYDRNIzsngKu3rGXrhbBzkFerWg=', 'Content-Length': '0'} 2022-09-02T04:42:19.213 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2429702fa89ea16-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.215 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.215 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.216 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.217 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.218 DEBUG:boto:Method: GET 2022-09-02T04:42:19.218 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.219 DEBUG:boto:Data: 2022-09-02T04:42:19.220 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.222 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.222 DEBUG:boto:Params: {} 2022-09-02T04:42:19.223 DEBUG:boto:Token: None 2022-09-02T04:42:19.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8= 2022-09-02T04:42:19.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8=', 'Content-Length': '0'} 2022-09-02T04:42:19.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d17b7661bb47c5b1-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.229 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.229 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.230 DEBUG:boto:path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.231 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.231 DEBUG:boto:Method: GET 2022-09-02T04:42:19.232 DEBUG:boto:Path: /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.233 DEBUG:boto:Data: 2022-09-02T04:42:19.233 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.234 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.235 DEBUG:boto:Params: {} 2022-09-02T04:42:19.236 DEBUG:boto:Token: None 2022-09-02T04:42:19.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=sZN5xldKFUsj8CX0kacXkmK9a2VT2KD 2022-09-02T04:42:19.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8= 2022-09-02T04:42:19.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lnFe6msiMTkitPi/0X7uopmFNa8=', 'Content-Length': '0'} 2022-09-02T04:42:19.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'sZN5xldKFUsj8CX0kacXkmK9a2VT2KD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000194dc82a9dea3909-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.282 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.283 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.284 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.285 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.286 DEBUG:boto:Method: GET 2022-09-02T04:42:19.287 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.287 DEBUG:boto:Data: 2022-09-02T04:42:19.288 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.289 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.289 DEBUG:boto:Params: {} 2022-09-02T04:42:19.290 DEBUG:boto:Token: None 2022-09-02T04:42:19.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc9925a582c0035-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.295 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.296 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.297 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.297 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.298 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.299 DEBUG:boto:Method: GET 2022-09-02T04:42:19.299 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.300 DEBUG:boto:Data: 2022-09-02T04:42:19.300 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.301 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.302 DEBUG:boto:Params: {} 2022-09-02T04:42:19.302 DEBUG:boto:Token: None 2022-09-02T04:42:19.303 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f45f8dcd8946f34-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.309 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.311 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.312 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.313 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.314 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.314 DEBUG:boto:Data: 2022-09-02T04:42:19.315 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.316 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.317 DEBUG:boto:Params: {} 2022-09-02T04:42:19.317 DEBUG:boto:Token: None 2022-09-02T04:42:19.318 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d337294c483cadc-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.326 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.327 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.327 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.328 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.329 DEBUG:boto:Data: 2022-09-02T04:42:19.329 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.331 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.332 DEBUG:boto:Params: {} 2022-09-02T04:42:19.333 DEBUG:boto:Token: None 2022-09-02T04:42:19.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000178814a96fe90f48-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.341 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.342 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.342 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.343 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.344 DEBUG:boto:Method: GET 2022-09-02T04:42:19.344 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.345 DEBUG:boto:Data: 2022-09-02T04:42:19.346 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.347 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.348 DEBUG:boto:Params: {} 2022-09-02T04:42:19.348 DEBUG:boto:Token: None 2022-09-02T04:42:19.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009be6a9ea99a2ca63-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.394 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.395 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.396 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.396 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.397 DEBUG:boto:Method: GET 2022-09-02T04:42:19.397 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.398 DEBUG:boto:Data: 2022-09-02T04:42:19.398 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.400 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.400 DEBUG:boto:Params: {} 2022-09-02T04:42:19.401 DEBUG:boto:Token: None 2022-09-02T04:42:19.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f68677e6124fe1d-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.407 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.408 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.409 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.410 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.411 DEBUG:boto:Method: GET 2022-09-02T04:42:19.412 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.412 DEBUG:boto:Data: 2022-09-02T04:42:19.413 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.415 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.416 DEBUG:boto:Params: {} 2022-09-02T04:42:19.417 DEBUG:boto:Token: None 2022-09-02T04:42:19.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000752832ddefa80e96-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.425 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.426 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.427 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.428 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.428 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.429 DEBUG:boto:Method: GET 2022-09-02T04:42:19.429 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.430 DEBUG:boto:Data: 2022-09-02T04:42:19.431 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.432 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.432 DEBUG:boto:Params: {} 2022-09-02T04:42:19.433 DEBUG:boto:Token: None 2022-09-02T04:42:19.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d32b06a40ed8bd3-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.438 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.439 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.440 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.441 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.441 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.442 DEBUG:boto:Method: GET 2022-09-02T04:42:19.442 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.443 DEBUG:boto:Data: 2022-09-02T04:42:19.443 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.444 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.445 DEBUG:boto:Params: {} 2022-09-02T04:42:19.445 DEBUG:boto:Token: None 2022-09-02T04:42:19.446 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk= 2022-09-02T04:42:19.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk=', 'Content-Length': '0'} 2022-09-02T04:42:19.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002601ad32d81d9260-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.455 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.455 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.456 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.456 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.457 DEBUG:boto:Method: GET 2022-09-02T04:42:19.458 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.458 DEBUG:boto:Data: 2022-09-02T04:42:19.459 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.460 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.460 DEBUG:boto:Params: {} 2022-09-02T04:42:19.461 DEBUG:boto:Token: None 2022-09-02T04:42:19.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk= 2022-09-02T04:42:19.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk=', 'Content-Length': '0'} 2022-09-02T04:42:19.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7d96704049230a3-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.466 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.467 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.467 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.468 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.468 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.469 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.469 DEBUG:boto:Data: 2022-09-02T04:42:19.470 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.471 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.471 DEBUG:boto:Params: {} 2022-09-02T04:42:19.472 DEBUG:boto:Token: None 2022-09-02T04:42:19.473 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+SgLbH4XksQRYdnHTM6/aNAFHYw= 2022-09-02T04:42:19.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+SgLbH4XksQRYdnHTM6/aNAFHYw=', 'Content-Length': '0'} 2022-09-02T04:42:19.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000910cc56376b55996-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.479 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.480 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.480 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.481 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.482 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.483 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.484 DEBUG:boto:Data: 2022-09-02T04:42:19.485 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.486 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.487 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.488 DEBUG:boto:Params: {} 2022-09-02T04:42:19.489 DEBUG:boto:Token: None 2022-09-02T04:42:19.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+SgLbH4XksQRYdnHTM6/aNAFHYw= 2022-09-02T04:42:19.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+SgLbH4XksQRYdnHTM6/aNAFHYw=', 'Content-Length': '0'} 2022-09-02T04:42:19.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025508edd82654e19-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.496 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.496 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.497 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.498 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.498 DEBUG:boto:Method: GET 2022-09-02T04:42:19.499 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.499 DEBUG:boto:Data: 2022-09-02T04:42:19.500 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.501 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.502 DEBUG:boto:Params: {} 2022-09-02T04:42:19.502 DEBUG:boto:Token: None 2022-09-02T04:42:19.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk= 2022-09-02T04:42:19.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk=', 'Content-Length': '0'} 2022-09-02T04:42:19.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af8125abd19ea8ee-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.507 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.508 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.508 DEBUG:boto:path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.509 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.510 DEBUG:boto:Method: GET 2022-09-02T04:42:19.510 DEBUG:boto:Path: /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.511 DEBUG:boto:Data: 2022-09-02T04:42:19.512 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.513 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.514 DEBUG:boto:Params: {} 2022-09-02T04:42:19.514 DEBUG:boto:Token: None 2022-09-02T04:42:19.515 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=-drxKfNi9vIRApncRBumUSOKO9YTEEz 2022-09-02T04:42:19.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk= 2022-09-02T04:42:19.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dRsB1TVuCMDu+dGBhvZdO3yKeDk=', 'Content-Length': '0'} 2022-09-02T04:42:19.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '-drxKfNi9vIRApncRBumUSOKO9YTEEz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000219a50032d8d4fa1-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.562 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.563 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.564 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.565 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.566 DEBUG:boto:Method: GET 2022-09-02T04:42:19.567 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.568 DEBUG:boto:Data: 2022-09-02T04:42:19.569 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.571 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.572 DEBUG:boto:Params: {} 2022-09-02T04:42:19.572 DEBUG:boto:Token: None 2022-09-02T04:42:19.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554b2cf69fbe80d6-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.578 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.579 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.580 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.581 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.582 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.583 DEBUG:boto:Method: GET 2022-09-02T04:42:19.584 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.585 DEBUG:boto:Data: 2022-09-02T04:42:19.586 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.588 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.589 DEBUG:boto:Params: {} 2022-09-02T04:42:19.590 DEBUG:boto:Token: None 2022-09-02T04:42:19.591 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa6f88f17754fd53-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.600 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.602 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.603 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.604 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.605 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.606 DEBUG:boto:Data: 2022-09-02T04:42:19.607 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.609 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.610 DEBUG:boto:Params: {} 2022-09-02T04:42:19.611 DEBUG:boto:Token: None 2022-09-02T04:42:19.612 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000720efaf322c0a07a-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.622 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.623 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.624 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.625 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.625 DEBUG:boto:Data: 2022-09-02T04:42:19.626 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.628 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.629 DEBUG:boto:Params: {} 2022-09-02T04:42:19.629 DEBUG:boto:Token: None 2022-09-02T04:42:19.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000861f86f7672fbd83-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.636 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.637 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.638 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.638 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.639 DEBUG:boto:Method: GET 2022-09-02T04:42:19.640 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.640 DEBUG:boto:Data: 2022-09-02T04:42:19.641 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.642 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.642 DEBUG:boto:Params: {} 2022-09-02T04:42:19.643 DEBUG:boto:Token: None 2022-09-02T04:42:19.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002410194d299a5f8b-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.648 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.648 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.649 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.650 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.651 DEBUG:boto:Method: GET 2022-09-02T04:42:19.652 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.653 DEBUG:boto:Data: 2022-09-02T04:42:19.655 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.657 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.658 DEBUG:boto:Params: {} 2022-09-02T04:42:19.659 DEBUG:boto:Token: None 2022-09-02T04:42:19.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fd3bac12bc57324-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.706 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.707 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.708 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.709 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.710 DEBUG:boto:Method: GET 2022-09-02T04:42:19.710 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.711 DEBUG:boto:Data: 2022-09-02T04:42:19.712 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.713 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.714 DEBUG:boto:Params: {} 2022-09-02T04:42:19.715 DEBUG:boto:Token: None 2022-09-02T04:42:19.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc1f65f6725cce1-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.725 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.726 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.726 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.727 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.728 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.729 DEBUG:boto:Method: GET 2022-09-02T04:42:19.730 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.731 DEBUG:boto:Data: 2022-09-02T04:42:19.732 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.734 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.735 DEBUG:boto:Params: {} 2022-09-02T04:42:19.736 DEBUG:boto:Token: None 2022-09-02T04:42:19.738 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c7d8472b710ea2e-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.743 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.744 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.745 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.745 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.746 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.747 DEBUG:boto:Method: GET 2022-09-02T04:42:19.747 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.748 DEBUG:boto:Data: 2022-09-02T04:42:19.748 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.750 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.751 DEBUG:boto:Params: {} 2022-09-02T04:42:19.752 DEBUG:boto:Token: None 2022-09-02T04:42:19.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI= 2022-09-02T04:42:19.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI=', 'Content-Length': '0'} 2022-09-02T04:42:19.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088d5d33878e74fbb-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.802 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.803 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.804 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.805 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.806 DEBUG:boto:Method: GET 2022-09-02T04:42:19.807 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.808 DEBUG:boto:Data: 2022-09-02T04:42:19.809 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.812 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.812 DEBUG:boto:Params: {} 2022-09-02T04:42:19.814 DEBUG:boto:Token: None 2022-09-02T04:42:19.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.816 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI= 2022-09-02T04:42:19.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI=', 'Content-Length': '0'} 2022-09-02T04:42:19.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9acd2ec3be71b92-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.821 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.822 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.823 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.824 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.824 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.825 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.825 DEBUG:boto:Data: 2022-09-02T04:42:19.826 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.827 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.828 DEBUG:boto:Params: {} 2022-09-02T04:42:19.828 DEBUG:boto:Token: None 2022-09-02T04:42:19.829 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M/RHL3UPuIPvRGaexNY/wp18Qlc= 2022-09-02T04:42:19.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M/RHL3UPuIPvRGaexNY/wp18Qlc=', 'Content-Length': '0'} 2022-09-02T04:42:19.833 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005485c05ee01b95c1-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.834 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.834 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.835 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.835 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.836 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.836 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.837 DEBUG:boto:Data: 2022-09-02T04:42:19.837 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.838 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.839 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.839 DEBUG:boto:Params: {} 2022-09-02T04:42:19.840 DEBUG:boto:Token: None 2022-09-02T04:42:19.840 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M/RHL3UPuIPvRGaexNY/wp18Qlc= 2022-09-02T04:42:19.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M/RHL3UPuIPvRGaexNY/wp18Qlc=', 'Content-Length': '0'} 2022-09-02T04:42:19.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8408c302a67cfa5-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.845 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.846 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.846 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.847 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.848 DEBUG:boto:Method: GET 2022-09-02T04:42:19.848 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.849 DEBUG:boto:Data: 2022-09-02T04:42:19.850 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.852 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.853 DEBUG:boto:Params: {} 2022-09-02T04:42:19.854 DEBUG:boto:Token: None 2022-09-02T04:42:19.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI= 2022-09-02T04:42:19.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI=', 'Content-Length': '0'} 2022-09-02T04:42:19.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000815e0a820491dfd1-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.861 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.861 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.862 DEBUG:boto:path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.863 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.863 DEBUG:boto:Method: GET 2022-09-02T04:42:19.864 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.864 DEBUG:boto:Data: 2022-09-02T04:42:19.865 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.866 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.866 DEBUG:boto:Params: {} 2022-09-02T04:42:19.867 DEBUG:boto:Token: None 2022-09-02T04:42:19.868 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic 2022-09-02T04:42:19.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI= 2022-09-02T04:42:19.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xwsRvB5TLAETLirpwPxdcbq5IAI=', 'Content-Length': '0'} 2022-09-02T04:42:19.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MqL.50kcX0KhN7mJPr1ee83fjf8a7Ic'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000350fccaf6c4fb2f4-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.872 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.873 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.873 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.874 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.874 DEBUG:boto:Method: GET 2022-09-02T04:42:19.875 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.876 DEBUG:boto:Data: 2022-09-02T04:42:19.876 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.877 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.878 DEBUG:boto:Params: {} 2022-09-02T04:42:19.878 DEBUG:boto:Token: None 2022-09-02T04:42:19.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb644a967d552d1-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.884 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.885 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.885 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.886 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.886 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.887 DEBUG:boto:Method: GET 2022-09-02T04:42:19.887 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.888 DEBUG:boto:Data: 2022-09-02T04:42:19.889 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.890 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.890 DEBUG:boto:Params: {} 2022-09-02T04:42:19.891 DEBUG:boto:Token: None 2022-09-02T04:42:19.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.892 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beab357eb9e0447a-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.896 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.897 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.898 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.898 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.899 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.899 DEBUG:boto:Data: 2022-09-02T04:42:19.900 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.901 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.901 DEBUG:boto:Params: {} 2022-09-02T04:42:19.902 DEBUG:boto:Token: None 2022-09-02T04:42:19.903 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ca023d318a1afc7-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.908 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.909 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.909 DEBUG:boto:Method: HEAD 2022-09-02T04:42:19.910 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:19.910 DEBUG:boto:Data: 2022-09-02T04:42:19.911 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.912 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.912 DEBUG:boto:Params: {} 2022-09-02T04:42:19.913 DEBUG:boto:Token: None 2022-09-02T04:42:19.914 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj 2022-09-02T04:42:19.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA= 2022-09-02T04:42:19.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCPKGvH3bKB0ghL1G6Kmn+BRUpA=', 'Content-Length': '0'} 2022-09-02T04:42:19.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbedcce48299fc19-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:19.920 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.921 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.921 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.922 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.922 DEBUG:boto:Method: GET 2022-09-02T04:42:19.923 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.923 DEBUG:boto:Data: 2022-09-02T04:42:19.924 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.925 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.926 DEBUG:boto:Params: {} 2022-09-02T04:42:19.926 DEBUG:boto:Token: None 2022-09-02T04:42:19.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005971bbc18df5acc6-00631189ab-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.974 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:19.975 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:19.975 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.976 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.977 DEBUG:boto:Method: GET 2022-09-02T04:42:19.977 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.978 DEBUG:boto:Data: 2022-09-02T04:42:19.978 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:19.979 DEBUG:boto:Port: 8000 2022-09-02T04:42:19.980 DEBUG:boto:Params: {} 2022-09-02T04:42:19.981 DEBUG:boto:Token: None 2022-09-02T04:42:19.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:19.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA= 2022-09-02T04:42:19.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X0tms1RFl20GKdKeO7HR2pTZ1JA=', 'Content-Length': '0'} 2022-09-02T04:42:19.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005683109c93644212-00631189ab-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.986 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:19.986 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:19.987 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.987 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:19.988 DEBUG:boto:Method: GET 2022-09-02T04:42:19.988 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:19.989 DEBUG:boto:Data: 2022-09-02T04:42:19.989 DEBUG:boto:Headers: {} 2022-09-02T04:42:19.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:19.991 DEBUG:boto:Port: 8001 2022-09-02T04:42:19.991 DEBUG:boto:Params: {} 2022-09-02T04:42:19.992 DEBUG:boto:Token: None 2022-09-02T04:42:19.993 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:19 GMT /xljbkf-19/ 2022-09-02T04:42:19.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY= 2022-09-02T04:42:19.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KrByjypHWYgAddcQyWQFlRfSQsY=', 'Content-Length': '0'} 2022-09-02T04:42:19.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a31479f59e5695ee-00631189ab-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:19.998 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:19.999 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.000 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.000 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.001 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.002 DEBUG:boto:Method: GET 2022-09-02T04:42:20.003 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.004 DEBUG:boto:Data: 2022-09-02T04:42:20.004 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.006 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.007 DEBUG:boto:Params: {} 2022-09-02T04:42:20.007 DEBUG:boto:Token: None 2022-09-02T04:42:20.008 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.008 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf30db2d445e33e-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.013 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.014 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.015 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.016 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.016 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.017 DEBUG:boto:Method: GET 2022-09-02T04:42:20.017 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.018 DEBUG:boto:Data: 2022-09-02T04:42:20.018 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.019 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.020 DEBUG:boto:Params: {} 2022-09-02T04:42:20.021 DEBUG:boto:Token: None 2022-09-02T04:42:20.021 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.022 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA= 2022-09-02T04:42:20.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA=', 'Content-Length': '0'} 2022-09-02T04:42:20.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c386ea088ed8115-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.026 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.026 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.027 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.027 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.028 DEBUG:boto:Method: GET 2022-09-02T04:42:20.028 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.029 DEBUG:boto:Data: 2022-09-02T04:42:20.029 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.030 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.031 DEBUG:boto:Params: {} 2022-09-02T04:42:20.032 DEBUG:boto:Token: None 2022-09-02T04:42:20.032 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA= 2022-09-02T04:42:20.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA=', 'Content-Length': '0'} 2022-09-02T04:42:20.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000aab58a7c3e5a382-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.036 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.037 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.037 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.038 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.038 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.039 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.039 DEBUG:boto:Data: 2022-09-02T04:42:20.040 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.041 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.041 DEBUG:boto:Params: {} 2022-09-02T04:42:20.042 DEBUG:boto:Token: None 2022-09-02T04:42:20.042 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EiQp4jiR9yYtLUBAZASdot85RNM= 2022-09-02T04:42:20.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EiQp4jiR9yYtLUBAZASdot85RNM=', 'Content-Length': '0'} 2022-09-02T04:42:20.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e22c399c824776de-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.046 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.047 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.047 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.048 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.048 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.049 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.049 DEBUG:boto:Data: 2022-09-02T04:42:20.050 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.051 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.051 DEBUG:boto:Params: {} 2022-09-02T04:42:20.052 DEBUG:boto:Token: None 2022-09-02T04:42:20.053 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EiQp4jiR9yYtLUBAZASdot85RNM= 2022-09-02T04:42:20.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EiQp4jiR9yYtLUBAZASdot85RNM=', 'Content-Length': '0'} 2022-09-02T04:42:20.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003de2ea818198d0b5-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.058 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.058 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.059 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.059 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.060 DEBUG:boto:Method: GET 2022-09-02T04:42:20.061 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.061 DEBUG:boto:Data: 2022-09-02T04:42:20.062 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.063 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.063 DEBUG:boto:Params: {} 2022-09-02T04:42:20.064 DEBUG:boto:Token: None 2022-09-02T04:42:20.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA= 2022-09-02T04:42:20.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA=', 'Content-Length': '0'} 2022-09-02T04:42:20.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f2d16fdd5ce6a1d-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.069 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.069 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.070 DEBUG:boto:path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.070 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.071 DEBUG:boto:Method: GET 2022-09-02T04:42:20.072 DEBUG:boto:Path: /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.072 DEBUG:boto:Data: 2022-09-02T04:42:20.073 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.074 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.074 DEBUG:boto:Params: {} 2022-09-02T04:42:20.075 DEBUG:boto:Token: None 2022-09-02T04:42:20.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY 2022-09-02T04:42:20.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA= 2022-09-02T04:42:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3xIeYZukAscnZUPlSBHlHEd3NxA=', 'Content-Length': '0'} 2022-09-02T04:42:20.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'YFP884gcYoRnzZqz3gcCDRzzvE3Z5nY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016d2132a9888a2d7-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.083 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.084 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.084 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.085 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.085 DEBUG:boto:Method: GET 2022-09-02T04:42:20.086 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.086 DEBUG:boto:Data: 2022-09-02T04:42:20.087 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.088 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.088 DEBUG:boto:Params: {} 2022-09-02T04:42:20.089 DEBUG:boto:Token: None 2022-09-02T04:42:20.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e1e5ec8bf73334-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.095 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.096 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.096 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.097 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.097 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.098 DEBUG:boto:Method: GET 2022-09-02T04:42:20.099 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.099 DEBUG:boto:Data: 2022-09-02T04:42:20.100 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.101 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.101 DEBUG:boto:Params: {} 2022-09-02T04:42:20.102 DEBUG:boto:Token: None 2022-09-02T04:42:20.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882f2fa483074a48-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.150 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.152 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.152 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.153 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.154 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.155 DEBUG:boto:Data: 2022-09-02T04:42:20.155 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.157 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.157 DEBUG:boto:Params: {} 2022-09-02T04:42:20.158 DEBUG:boto:Token: None 2022-09-02T04:42:20.159 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000536ce90fd8bfeacb-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.170 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.171 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.172 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.173 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.174 DEBUG:boto:Data: 2022-09-02T04:42:20.175 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.176 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.177 DEBUG:boto:Params: {} 2022-09-02T04:42:20.178 DEBUG:boto:Token: None 2022-09-02T04:42:20.179 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008da9bdc8cf3113f4-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.186 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.187 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.187 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.188 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.188 DEBUG:boto:Method: GET 2022-09-02T04:42:20.189 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.190 DEBUG:boto:Data: 2022-09-02T04:42:20.190 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.191 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.192 DEBUG:boto:Params: {} 2022-09-02T04:42:20.192 DEBUG:boto:Token: None 2022-09-02T04:42:20.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077fe48cdcec438df-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.197 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.198 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.199 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.199 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.200 DEBUG:boto:Method: GET 2022-09-02T04:42:20.201 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.202 DEBUG:boto:Data: 2022-09-02T04:42:20.203 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.205 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.205 DEBUG:boto:Params: {} 2022-09-02T04:42:20.206 DEBUG:boto:Token: None 2022-09-02T04:42:20.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005107a11b53a6fe29-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.211 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.212 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.212 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.213 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.213 DEBUG:boto:Method: GET 2022-09-02T04:42:20.214 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.215 DEBUG:boto:Data: 2022-09-02T04:42:20.215 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.216 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.217 DEBUG:boto:Params: {} 2022-09-02T04:42:20.218 DEBUG:boto:Token: None 2022-09-02T04:42:20.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c616a56259305000-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.226 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.227 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.228 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.229 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.230 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.231 DEBUG:boto:Method: GET 2022-09-02T04:42:20.232 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.232 DEBUG:boto:Data: 2022-09-02T04:42:20.233 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.234 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.235 DEBUG:boto:Params: {} 2022-09-02T04:42:20.235 DEBUG:boto:Token: None 2022-09-02T04:42:20.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087b48c089ddd37e-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.287 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.289 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.290 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.290 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.291 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.291 DEBUG:boto:Method: GET 2022-09-02T04:42:20.292 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.292 DEBUG:boto:Data: 2022-09-02T04:42:20.293 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.294 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.294 DEBUG:boto:Params: {} 2022-09-02T04:42:20.295 DEBUG:boto:Token: None 2022-09-02T04:42:20.296 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE= 2022-09-02T04:42:20.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE=', 'Content-Length': '0'} 2022-09-02T04:42:20.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056a4276f8ba2d432-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.301 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.301 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.302 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.303 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.303 DEBUG:boto:Method: GET 2022-09-02T04:42:20.304 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.304 DEBUG:boto:Data: 2022-09-02T04:42:20.305 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.305 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.306 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.306 DEBUG:boto:Params: {} 2022-09-02T04:42:20.307 DEBUG:boto:Token: None 2022-09-02T04:42:20.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE= 2022-09-02T04:42:20.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE=', 'Content-Length': '0'} 2022-09-02T04:42:20.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83959481947778f-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.312 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.312 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.313 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.313 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.314 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.315 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.315 DEBUG:boto:Data: 2022-09-02T04:42:20.316 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.317 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.317 DEBUG:boto:Params: {} 2022-09-02T04:42:20.318 DEBUG:boto:Token: None 2022-09-02T04:42:20.318 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XbtMTRYOEsayaCaECgagbiqUQVU= 2022-09-02T04:42:20.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XbtMTRYOEsayaCaECgagbiqUQVU=', 'Content-Length': '0'} 2022-09-02T04:42:20.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dfd317a1b841873-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.323 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.324 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.324 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.325 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.325 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.326 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.327 DEBUG:boto:Data: 2022-09-02T04:42:20.328 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.330 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.331 DEBUG:boto:Params: {} 2022-09-02T04:42:20.331 DEBUG:boto:Token: None 2022-09-02T04:42:20.332 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XbtMTRYOEsayaCaECgagbiqUQVU= 2022-09-02T04:42:20.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XbtMTRYOEsayaCaECgagbiqUQVU=', 'Content-Length': '0'} 2022-09-02T04:42:20.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf584e997fc21654-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.337 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.338 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.339 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.339 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.340 DEBUG:boto:Method: GET 2022-09-02T04:42:20.340 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.341 DEBUG:boto:Data: 2022-09-02T04:42:20.342 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.343 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.344 DEBUG:boto:Params: {} 2022-09-02T04:42:20.344 DEBUG:boto:Token: None 2022-09-02T04:42:20.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE= 2022-09-02T04:42:20.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE=', 'Content-Length': '0'} 2022-09-02T04:42:20.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beb4903c060377cb-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.390 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.391 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.392 DEBUG:boto:path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.393 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.394 DEBUG:boto:Method: GET 2022-09-02T04:42:20.395 DEBUG:boto:Path: /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.396 DEBUG:boto:Data: 2022-09-02T04:42:20.397 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.398 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.399 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.399 DEBUG:boto:Params: {} 2022-09-02T04:42:20.400 DEBUG:boto:Token: None 2022-09-02T04:42:20.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf 2022-09-02T04:42:20.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE= 2022-09-02T04:42:20.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+N3KVpDWUe/CMl3rPt6xTIofZE=', 'Content-Length': '0'} 2022-09-02T04:42:20.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '.Z8Tsv6gjbvHKCELKBxNBi3B2s1DaZf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da64ef05c7809a3d-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.405 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.407 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.407 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.408 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.408 DEBUG:boto:Method: GET 2022-09-02T04:42:20.409 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.410 DEBUG:boto:Data: 2022-09-02T04:42:20.411 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.413 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.414 DEBUG:boto:Params: {} 2022-09-02T04:42:20.415 DEBUG:boto:Token: None 2022-09-02T04:42:20.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d266111a0c44a7d1-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.422 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.423 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.423 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.424 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.424 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.425 DEBUG:boto:Method: GET 2022-09-02T04:42:20.425 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.426 DEBUG:boto:Data: 2022-09-02T04:42:20.426 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.428 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.428 DEBUG:boto:Params: {} 2022-09-02T04:42:20.429 DEBUG:boto:Token: None 2022-09-02T04:42:20.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000995a7cdda2101b04-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.434 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.435 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.436 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.436 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.437 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.437 DEBUG:boto:Data: 2022-09-02T04:42:20.438 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.439 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.440 DEBUG:boto:Params: {} 2022-09-02T04:42:20.441 DEBUG:boto:Token: None 2022-09-02T04:42:20.442 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009edf25ba1cd1c5ea-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.447 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.448 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.449 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.449 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.450 DEBUG:boto:Data: 2022-09-02T04:42:20.450 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.452 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.452 DEBUG:boto:Params: {} 2022-09-02T04:42:20.453 DEBUG:boto:Token: None 2022-09-02T04:42:20.453 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a82d1225af41d4-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.460 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.462 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.463 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.463 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.464 DEBUG:boto:Method: GET 2022-09-02T04:42:20.465 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.466 DEBUG:boto:Data: 2022-09-02T04:42:20.468 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.470 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.471 DEBUG:boto:Params: {} 2022-09-02T04:42:20.472 DEBUG:boto:Token: None 2022-09-02T04:42:20.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a6884ab9e1bc10d-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.478 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.478 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.479 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.480 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.480 DEBUG:boto:Method: GET 2022-09-02T04:42:20.481 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.482 DEBUG:boto:Data: 2022-09-02T04:42:20.483 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.484 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.485 DEBUG:boto:Params: {} 2022-09-02T04:42:20.485 DEBUG:boto:Token: None 2022-09-02T04:42:20.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0e7dcda490c694f-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.491 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.491 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.492 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.492 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.493 DEBUG:boto:Method: GET 2022-09-02T04:42:20.493 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.494 DEBUG:boto:Data: 2022-09-02T04:42:20.494 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.496 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.496 DEBUG:boto:Params: {} 2022-09-02T04:42:20.497 DEBUG:boto:Token: None 2022-09-02T04:42:20.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f3a6096e66a094-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.502 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.503 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.504 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.504 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.505 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.505 DEBUG:boto:Method: GET 2022-09-02T04:42:20.506 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.516 DEBUG:boto:Data: 2022-09-02T04:42:20.516 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.517 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.518 DEBUG:boto:Params: {} 2022-09-02T04:42:20.518 DEBUG:boto:Token: None 2022-09-02T04:42:20.519 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a04b86d87bd8340b-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.527 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.528 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.529 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.529 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.530 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.531 DEBUG:boto:Method: GET 2022-09-02T04:42:20.531 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.532 DEBUG:boto:Data: 2022-09-02T04:42:20.532 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.533 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.534 DEBUG:boto:Params: {} 2022-09-02T04:42:20.535 DEBUG:boto:Token: None 2022-09-02T04:42:20.535 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw= 2022-09-02T04:42:20.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw=', 'Content-Length': '0'} 2022-09-02T04:42:20.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bcc7611298780e9-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.540 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.541 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.541 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.542 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.543 DEBUG:boto:Method: GET 2022-09-02T04:42:20.544 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.544 DEBUG:boto:Data: 2022-09-02T04:42:20.545 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.546 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.546 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.547 DEBUG:boto:Params: {} 2022-09-02T04:42:20.548 DEBUG:boto:Token: None 2022-09-02T04:42:20.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw= 2022-09-02T04:42:20.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw=', 'Content-Length': '0'} 2022-09-02T04:42:20.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e9aa4135aabc5f4-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.598 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.599 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.600 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.601 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.602 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.603 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.604 DEBUG:boto:Data: 2022-09-02T04:42:20.605 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.606 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.607 DEBUG:boto:Params: {} 2022-09-02T04:42:20.608 DEBUG:boto:Token: None 2022-09-02T04:42:20.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+Z3fDz92eKemavhYEkPKpB+0BFk= 2022-09-02T04:42:20.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+Z3fDz92eKemavhYEkPKpB+0BFk=', 'Content-Length': '0'} 2022-09-02T04:42:20.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9aed1ea01f626e1-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.614 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.615 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.615 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.616 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.616 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.617 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.617 DEBUG:boto:Data: 2022-09-02T04:42:20.618 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.619 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.620 DEBUG:boto:Params: {} 2022-09-02T04:42:20.620 DEBUG:boto:Token: None 2022-09-02T04:42:20.621 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+Z3fDz92eKemavhYEkPKpB+0BFk= 2022-09-02T04:42:20.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+Z3fDz92eKemavhYEkPKpB+0BFk=', 'Content-Length': '0'} 2022-09-02T04:42:20.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3bc544713941b32-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.626 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.626 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.627 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.627 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.628 DEBUG:boto:Method: GET 2022-09-02T04:42:20.628 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.629 DEBUG:boto:Data: 2022-09-02T04:42:20.630 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.631 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.631 DEBUG:boto:Params: {} 2022-09-02T04:42:20.632 DEBUG:boto:Token: None 2022-09-02T04:42:20.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw= 2022-09-02T04:42:20.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw=', 'Content-Length': '0'} 2022-09-02T04:42:20.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001035c5c533bf4c38-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.637 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.638 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.639 DEBUG:boto:path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.640 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.641 DEBUG:boto:Method: GET 2022-09-02T04:42:20.642 DEBUG:boto:Path: /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.643 DEBUG:boto:Data: 2022-09-02T04:42:20.644 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.646 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.646 DEBUG:boto:Params: {} 2022-09-02T04:42:20.647 DEBUG:boto:Token: None 2022-09-02T04:42:20.647 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4 2022-09-02T04:42:20.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw= 2022-09-02T04:42:20.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daRaWSbUJCQZuo/gFoCLP0LnObw=', 'Content-Length': '0'} 2022-09-02T04:42:20.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'NJMY0IMH4LrgnBeOYLOR9RN0EOHamj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb8c8c2cc25eaba3-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.652 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.652 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.653 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.653 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.654 DEBUG:boto:Method: GET 2022-09-02T04:42:20.654 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.655 DEBUG:boto:Data: 2022-09-02T04:42:20.655 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.656 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.657 DEBUG:boto:Params: {} 2022-09-02T04:42:20.658 DEBUG:boto:Token: None 2022-09-02T04:42:20.658 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfcd3bfaf707997b-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.664 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.665 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.665 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.666 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.667 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.668 DEBUG:boto:Method: GET 2022-09-02T04:42:20.669 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.669 DEBUG:boto:Data: 2022-09-02T04:42:20.670 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.671 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.671 DEBUG:boto:Params: {} 2022-09-02T04:42:20.672 DEBUG:boto:Token: None 2022-09-02T04:42:20.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048e320ba4b67f51f-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.677 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.678 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.679 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.679 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.680 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.680 DEBUG:boto:Data: 2022-09-02T04:42:20.681 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.682 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.683 DEBUG:boto:Params: {} 2022-09-02T04:42:20.683 DEBUG:boto:Token: None 2022-09-02T04:42:20.684 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e227b86d32ded09e-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.689 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.690 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.690 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.691 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.692 DEBUG:boto:Data: 2022-09-02T04:42:20.692 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.694 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.695 DEBUG:boto:Params: {} 2022-09-02T04:42:20.696 DEBUG:boto:Token: None 2022-09-02T04:42:20.697 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8163ba076f41e17-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.705 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.706 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.707 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.707 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.708 DEBUG:boto:Method: GET 2022-09-02T04:42:20.709 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.710 DEBUG:boto:Data: 2022-09-02T04:42:20.711 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.712 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.713 DEBUG:boto:Params: {} 2022-09-02T04:42:20.713 DEBUG:boto:Token: None 2022-09-02T04:42:20.714 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.719 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce183b16f9413d89-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.762 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.763 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.763 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.764 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.765 DEBUG:boto:Method: GET 2022-09-02T04:42:20.765 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.766 DEBUG:boto:Data: 2022-09-02T04:42:20.767 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.768 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.769 DEBUG:boto:Params: {} 2022-09-02T04:42:20.770 DEBUG:boto:Token: None 2022-09-02T04:42:20.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000825fc390b0c3ca8a-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.778 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.779 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.780 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.781 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.782 DEBUG:boto:Method: GET 2022-09-02T04:42:20.783 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.783 DEBUG:boto:Data: 2022-09-02T04:42:20.784 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.786 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.787 DEBUG:boto:Params: {} 2022-09-02T04:42:20.788 DEBUG:boto:Token: None 2022-09-02T04:42:20.789 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000890fcc9eb4489972-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.794 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.795 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.796 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.796 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.797 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.798 DEBUG:boto:Method: GET 2022-09-02T04:42:20.798 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.799 DEBUG:boto:Data: 2022-09-02T04:42:20.799 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.801 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.802 DEBUG:boto:Params: {} 2022-09-02T04:42:20.802 DEBUG:boto:Token: None 2022-09-02T04:42:20.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7bc7c2e100a38f3-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.809 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.810 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.811 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.812 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.813 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.814 DEBUG:boto:Method: GET 2022-09-02T04:42:20.816 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.817 DEBUG:boto:Data: 2022-09-02T04:42:20.818 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.821 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.823 DEBUG:boto:Params: {} 2022-09-02T04:42:20.824 DEBUG:boto:Token: None 2022-09-02T04:42:20.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys= 2022-09-02T04:42:20.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys=', 'Content-Length': '0'} 2022-09-02T04:42:20.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004059135b2ea871e7-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.830 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.831 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.832 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.833 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.834 DEBUG:boto:Method: GET 2022-09-02T04:42:20.834 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.835 DEBUG:boto:Data: 2022-09-02T04:42:20.836 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.837 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.838 DEBUG:boto:Params: {} 2022-09-02T04:42:20.839 DEBUG:boto:Token: None 2022-09-02T04:42:20.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys= 2022-09-02T04:42:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys=', 'Content-Length': '0'} 2022-09-02T04:42:20.843 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000180610978c3eac95-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.844 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.845 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.845 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.846 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.847 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.847 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.848 DEBUG:boto:Data: 2022-09-02T04:42:20.848 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.850 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.850 DEBUG:boto:Params: {} 2022-09-02T04:42:20.851 DEBUG:boto:Token: None 2022-09-02T04:42:20.852 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XBlhZ5ByN6sEJmH68632qFjFOew= 2022-09-02T04:42:20.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XBlhZ5ByN6sEJmH68632qFjFOew=', 'Content-Length': '0'} 2022-09-02T04:42:20.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071066c61f5533681-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.856 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.857 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.857 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.858 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.858 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.859 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.859 DEBUG:boto:Data: 2022-09-02T04:42:20.860 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.861 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.861 DEBUG:boto:Params: {} 2022-09-02T04:42:20.862 DEBUG:boto:Token: None 2022-09-02T04:42:20.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XBlhZ5ByN6sEJmH68632qFjFOew= 2022-09-02T04:42:20.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XBlhZ5ByN6sEJmH68632qFjFOew=', 'Content-Length': '0'} 2022-09-02T04:42:20.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000664a84feb39a5e3b-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.867 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.867 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.868 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.868 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.869 DEBUG:boto:Method: GET 2022-09-02T04:42:20.869 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.870 DEBUG:boto:Data: 2022-09-02T04:42:20.870 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.871 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.872 DEBUG:boto:Params: {} 2022-09-02T04:42:20.873 DEBUG:boto:Token: None 2022-09-02T04:42:20.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys= 2022-09-02T04:42:20.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys=', 'Content-Length': '0'} 2022-09-02T04:42:20.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000054e366b0e74c305-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.877 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.878 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.878 DEBUG:boto:path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.879 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.879 DEBUG:boto:Method: GET 2022-09-02T04:42:20.881 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.881 DEBUG:boto:Data: 2022-09-02T04:42:20.882 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.883 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.883 DEBUG:boto:Params: {} 2022-09-02T04:42:20.884 DEBUG:boto:Token: None 2022-09-02T04:42:20.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP 2022-09-02T04:42:20.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys= 2022-09-02T04:42:20.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R+0k+WzH5ig0OlwrHmahW77L7ys=', 'Content-Length': '0'} 2022-09-02T04:42:20.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zgvcmovaY0vlO0ghuQ9TKaLCsjtQTuP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fa30c5eb476a2f9-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.889 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.889 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.890 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.890 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.891 DEBUG:boto:Method: GET 2022-09-02T04:42:20.892 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.892 DEBUG:boto:Data: 2022-09-02T04:42:20.893 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.894 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.894 DEBUG:boto:Params: {} 2022-09-02T04:42:20.895 DEBUG:boto:Token: None 2022-09-02T04:42:20.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277c7b3e1d0d05b8-00631189ac-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.900 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.901 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:20.901 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:20.902 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.902 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:20.903 DEBUG:boto:Method: GET 2022-09-02T04:42:20.903 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:20.904 DEBUG:boto:Data: 2022-09-02T04:42:20.904 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.905 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.906 DEBUG:boto:Params: {} 2022-09-02T04:42:20.907 DEBUG:boto:Token: None 2022-09-02T04:42:20.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/ 2022-09-02T04:42:20.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ= 2022-09-02T04:42:20.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AR4sTemMKpFdndDDm6j3OPiZ5lQ=', 'Content-Length': '0'} 2022-09-02T04:42:20.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835aedaeffeb93dd-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.912 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:20.912 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.913 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.913 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.914 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.915 DEBUG:boto:Data: 2022-09-02T04:42:20.915 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.916 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.916 DEBUG:boto:Params: {} 2022-09-02T04:42:20.917 DEBUG:boto:Token: None 2022-09-02T04:42:20.918 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084e528752df3e4ad-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.923 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.924 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.924 DEBUG:boto:Method: HEAD 2022-09-02T04:42:20.925 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:20.925 DEBUG:boto:Data: 2022-09-02T04:42:20.926 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.927 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.927 DEBUG:boto:Params: {} 2022-09-02T04:42:20.928 DEBUG:boto:Token: None 2022-09-02T04:42:20.928 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj 2022-09-02T04:42:20.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8= 2022-09-02T04:42:20.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr/tP9Bm/Khqp3rR8IJOdTCS0u8=', 'Content-Length': '0'} 2022-09-02T04:42:20.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006488e7d51e36b1ef-00631189ac-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:20.936 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.937 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.938 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.939 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.940 DEBUG:boto:Method: GET 2022-09-02T04:42:20.941 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.942 DEBUG:boto:Data: 2022-09-02T04:42:20.943 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:20.944 DEBUG:boto:Port: 8001 2022-09-02T04:42:20.944 DEBUG:boto:Params: {} 2022-09-02T04:42:20.945 DEBUG:boto:Token: None 2022-09-02T04:42:20.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:20 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY= 2022-09-02T04:42:20.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XCZPACOX7AFsBkRVo9c5yliM5jY=', 'Content-Length': '0'} 2022-09-02T04:42:20.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c48e85ddbfb936c-00631189ac-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:20.991 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:20.992 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:20.993 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.994 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.994 DEBUG:boto:Method: GET 2022-09-02T04:42:20.995 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:20.996 DEBUG:boto:Data: 2022-09-02T04:42:20.996 DEBUG:boto:Headers: {} 2022-09-02T04:42:20.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:20.998 DEBUG:boto:Port: 8000 2022-09-02T04:42:20.999 DEBUG:boto:Params: {} 2022-09-02T04:42:21.000 DEBUG:boto:Token: None 2022-09-02T04:42:21.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e5501fb6497fe9d-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.008 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.009 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.010 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.010 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.011 DEBUG:boto:Method: GET 2022-09-02T04:42:21.011 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.012 DEBUG:boto:Data: 2022-09-02T04:42:21.012 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.013 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.014 DEBUG:boto:Params: {} 2022-09-02T04:42:21.015 DEBUG:boto:Token: None 2022-09-02T04:42:21.015 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001585562573a06f10-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.022 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.024 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.024 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.025 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.026 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.026 DEBUG:boto:Method: GET 2022-09-02T04:42:21.027 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.027 DEBUG:boto:Data: 2022-09-02T04:42:21.028 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.029 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.029 DEBUG:boto:Params: {} 2022-09-02T04:42:21.030 DEBUG:boto:Token: None 2022-09-02T04:42:21.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee18c1917e44fab-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.035 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.037 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.037 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.038 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.039 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.039 DEBUG:boto:Method: GET 2022-09-02T04:42:21.040 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.040 DEBUG:boto:Data: 2022-09-02T04:42:21.041 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.042 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.043 DEBUG:boto:Params: {} 2022-09-02T04:42:21.043 DEBUG:boto:Token: None 2022-09-02T04:42:21.044 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs= 2022-09-02T04:42:21.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs=', 'Content-Length': '0'} 2022-09-02T04:42:21.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045bfa7bdbc3dc36a-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.048 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.049 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.049 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.050 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.051 DEBUG:boto:Method: GET 2022-09-02T04:42:21.051 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.052 DEBUG:boto:Data: 2022-09-02T04:42:21.052 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.053 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.054 DEBUG:boto:Params: {} 2022-09-02T04:42:21.054 DEBUG:boto:Token: None 2022-09-02T04:42:21.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs= 2022-09-02T04:42:21.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs=', 'Content-Length': '0'} 2022-09-02T04:42:21.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000581b214259de05b1-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.059 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.060 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.060 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.061 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.061 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.062 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.063 DEBUG:boto:Data: 2022-09-02T04:42:21.063 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.064 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.065 DEBUG:boto:Params: {} 2022-09-02T04:42:21.065 DEBUG:boto:Token: None 2022-09-02T04:42:21.066 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3GS5o0R7S1cQhuSkVASB4uSiW5w= 2022-09-02T04:42:21.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3GS5o0R7S1cQhuSkVASB4uSiW5w=', 'Content-Length': '0'} 2022-09-02T04:42:21.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e6a798026f5fc9f-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.070 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.071 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.071 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.072 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.072 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.073 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.073 DEBUG:boto:Data: 2022-09-02T04:42:21.074 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.074 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.075 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.075 DEBUG:boto:Params: {} 2022-09-02T04:42:21.076 DEBUG:boto:Token: None 2022-09-02T04:42:21.076 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3GS5o0R7S1cQhuSkVASB4uSiW5w= 2022-09-02T04:42:21.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3GS5o0R7S1cQhuSkVASB4uSiW5w=', 'Content-Length': '0'} 2022-09-02T04:42:21.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000300c5621b3a4bec3-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.081 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.082 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.083 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.083 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.084 DEBUG:boto:Method: GET 2022-09-02T04:42:21.084 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.085 DEBUG:boto:Data: 2022-09-02T04:42:21.085 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.087 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.087 DEBUG:boto:Params: {} 2022-09-02T04:42:21.088 DEBUG:boto:Token: None 2022-09-02T04:42:21.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs= 2022-09-02T04:42:21.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs=', 'Content-Length': '0'} 2022-09-02T04:42:21.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000780fcd744ab5c492-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.093 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.093 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.094 DEBUG:boto:path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.094 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.095 DEBUG:boto:Method: GET 2022-09-02T04:42:21.096 DEBUG:boto:Path: /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.096 DEBUG:boto:Data: 2022-09-02T04:42:21.097 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.098 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.098 DEBUG:boto:Params: {} 2022-09-02T04:42:21.099 DEBUG:boto:Token: None 2022-09-02T04:42:21.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l 2022-09-02T04:42:21.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs= 2022-09-02T04:42:21.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6HUmKBfphhvxxPqL1ti+adV8Wjs=', 'Content-Length': '0'} 2022-09-02T04:42:21.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '4TKoLa.-emUUJiNYIKXx8QDFZGXqr3l'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000954435890d1ef940-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.146 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.147 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.148 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.149 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.150 DEBUG:boto:Method: GET 2022-09-02T04:42:21.151 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.152 DEBUG:boto:Data: 2022-09-02T04:42:21.153 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.155 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.156 DEBUG:boto:Params: {} 2022-09-02T04:42:21.157 DEBUG:boto:Token: None 2022-09-02T04:42:21.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d02b28664231fd8-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.165 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.166 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.167 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.168 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.169 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.170 DEBUG:boto:Method: GET 2022-09-02T04:42:21.171 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.172 DEBUG:boto:Data: 2022-09-02T04:42:21.173 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.175 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.175 DEBUG:boto:Params: {} 2022-09-02T04:42:21.176 DEBUG:boto:Token: None 2022-09-02T04:42:21.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072785e2e2efe80f4-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.183 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.184 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.186 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.186 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.187 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:21.187 DEBUG:boto:Data: 2022-09-02T04:42:21.188 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.189 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.189 DEBUG:boto:Params: {} 2022-09-02T04:42:21.190 DEBUG:boto:Token: None 2022-09-02T04:42:21.191 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj 2022-09-02T04:42:21.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg= 2022-09-02T04:42:21.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg=', 'Content-Length': '0'} 2022-09-02T04:42:21.196 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006752d90042ff6c39-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.197 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.197 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.198 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.199 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:21.200 DEBUG:boto:Data: 2022-09-02T04:42:21.201 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.202 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.203 DEBUG:boto:Params: {} 2022-09-02T04:42:21.204 DEBUG:boto:Token: None 2022-09-02T04:42:21.205 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj 2022-09-02T04:42:21.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg= 2022-09-02T04:42:21.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg=', 'Content-Length': '0'} 2022-09-02T04:42:21.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035673cc143ef4e82-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.213 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.214 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.214 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.215 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.216 DEBUG:boto:Method: GET 2022-09-02T04:42:21.216 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.217 DEBUG:boto:Data: 2022-09-02T04:42:21.218 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.220 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.221 DEBUG:boto:Params: {} 2022-09-02T04:42:21.222 DEBUG:boto:Token: None 2022-09-02T04:42:21.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b75b23bba3a4519c-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.270 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.271 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.271 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.272 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.272 DEBUG:boto:Method: GET 2022-09-02T04:42:21.273 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.273 DEBUG:boto:Data: 2022-09-02T04:42:21.274 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.275 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.276 DEBUG:boto:Params: {} 2022-09-02T04:42:21.277 DEBUG:boto:Token: None 2022-09-02T04:42:21.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad40f6a6d301802d-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.282 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.282 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.283 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.283 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.284 DEBUG:boto:Method: GET 2022-09-02T04:42:21.285 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.286 DEBUG:boto:Data: 2022-09-02T04:42:21.286 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.288 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.288 DEBUG:boto:Params: {} 2022-09-02T04:42:21.289 DEBUG:boto:Token: None 2022-09-02T04:42:21.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffff9c54cce6884b-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.295 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.296 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.297 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.297 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.298 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.298 DEBUG:boto:Method: GET 2022-09-02T04:42:21.299 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.299 DEBUG:boto:Data: 2022-09-02T04:42:21.300 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.301 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.302 DEBUG:boto:Params: {} 2022-09-02T04:42:21.302 DEBUG:boto:Token: None 2022-09-02T04:42:21.303 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.303 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003973807fb625633a-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.308 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.309 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.310 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.310 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.311 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.311 DEBUG:boto:Method: GET 2022-09-02T04:42:21.312 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.312 DEBUG:boto:Data: 2022-09-02T04:42:21.313 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.314 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.315 DEBUG:boto:Params: {} 2022-09-02T04:42:21.315 DEBUG:boto:Token: None 2022-09-02T04:42:21.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8= 2022-09-02T04:42:21.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8=', 'Content-Length': '0'} 2022-09-02T04:42:21.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003abab4ba04742b4d-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.320 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.321 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.321 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.322 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.322 DEBUG:boto:Method: GET 2022-09-02T04:42:21.323 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.323 DEBUG:boto:Data: 2022-09-02T04:42:21.324 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.325 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.325 DEBUG:boto:Params: {} 2022-09-02T04:42:21.326 DEBUG:boto:Token: None 2022-09-02T04:42:21.327 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8= 2022-09-02T04:42:21.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8=', 'Content-Length': '0'} 2022-09-02T04:42:21.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac4e9a8007cae6ee-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.380 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.381 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.382 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.383 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.384 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.385 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.386 DEBUG:boto:Data: 2022-09-02T04:42:21.387 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.388 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.388 DEBUG:boto:Params: {} 2022-09-02T04:42:21.389 DEBUG:boto:Token: None 2022-09-02T04:42:21.390 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m8uBF9Wdf27QI4yJ8MYp4c3P/mc= 2022-09-02T04:42:21.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m8uBF9Wdf27QI4yJ8MYp4c3P/mc=', 'Content-Length': '0'} 2022-09-02T04:42:21.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cc2453ab99699cc-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.394 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.394 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.395 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.395 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.396 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.396 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.397 DEBUG:boto:Data: 2022-09-02T04:42:21.397 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.398 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.399 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.399 DEBUG:boto:Params: {} 2022-09-02T04:42:21.400 DEBUG:boto:Token: None 2022-09-02T04:42:21.400 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m8uBF9Wdf27QI4yJ8MYp4c3P/mc= 2022-09-02T04:42:21.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m8uBF9Wdf27QI4yJ8MYp4c3P/mc=', 'Content-Length': '0'} 2022-09-02T04:42:21.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c22e955877e6ccc0-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.405 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.406 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.406 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.407 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.407 DEBUG:boto:Method: GET 2022-09-02T04:42:21.408 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.408 DEBUG:boto:Data: 2022-09-02T04:42:21.409 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.410 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.410 DEBUG:boto:Params: {} 2022-09-02T04:42:21.411 DEBUG:boto:Token: None 2022-09-02T04:42:21.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8= 2022-09-02T04:42:21.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8=', 'Content-Length': '0'} 2022-09-02T04:42:21.415 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb0d92ca040c9972-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.416 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.417 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.418 DEBUG:boto:path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.418 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.419 DEBUG:boto:Method: GET 2022-09-02T04:42:21.419 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.420 DEBUG:boto:Data: 2022-09-02T04:42:21.420 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.422 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.423 DEBUG:boto:Params: {} 2022-09-02T04:42:21.424 DEBUG:boto:Token: None 2022-09-02T04:42:21.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa 2022-09-02T04:42:21.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8= 2022-09-02T04:42:21.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QT/lvJ8bfrYm5fx6l6Xbv10Uyn8=', 'Content-Length': '0'} 2022-09-02T04:42:21.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MPSTvqfocHNpjqSSe1XHwMrhDdV5dDa'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007537a0b4d952d3cc-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.430 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.431 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.431 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.432 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.432 DEBUG:boto:Method: GET 2022-09-02T04:42:21.433 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.433 DEBUG:boto:Data: 2022-09-02T04:42:21.434 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.435 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.435 DEBUG:boto:Params: {} 2022-09-02T04:42:21.436 DEBUG:boto:Token: None 2022-09-02T04:42:21.437 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8c1a9df21eb95df-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.442 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.443 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.443 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.444 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.444 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.445 DEBUG:boto:Method: GET 2022-09-02T04:42:21.445 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.446 DEBUG:boto:Data: 2022-09-02T04:42:21.446 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.447 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.447 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.448 DEBUG:boto:Params: {} 2022-09-02T04:42:21.448 DEBUG:boto:Token: None 2022-09-02T04:42:21.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c44942303a3cb1-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.454 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.454 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.455 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.456 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.456 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:21.457 DEBUG:boto:Data: 2022-09-02T04:42:21.457 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.458 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.458 DEBUG:boto:Params: {} 2022-09-02T04:42:21.459 DEBUG:boto:Token: None 2022-09-02T04:42:21.460 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj 2022-09-02T04:42:21.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg= 2022-09-02T04:42:21.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg=', 'Content-Length': '0'} 2022-09-02T04:42:21.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8ad563bf53e5b8d-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.465 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.465 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.466 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.467 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:21.467 DEBUG:boto:Data: 2022-09-02T04:42:21.468 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.469 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.469 DEBUG:boto:Params: {} 2022-09-02T04:42:21.470 DEBUG:boto:Token: None 2022-09-02T04:42:21.471 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj 2022-09-02T04:42:21.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg= 2022-09-02T04:42:21.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg=', 'Content-Length': '0'} 2022-09-02T04:42:21.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa605b1a8e7ba11c-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.478 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.479 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.481 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.482 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.483 DEBUG:boto:Method: GET 2022-09-02T04:42:21.483 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.484 DEBUG:boto:Data: 2022-09-02T04:42:21.485 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.487 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.488 DEBUG:boto:Params: {} 2022-09-02T04:42:21.488 DEBUG:boto:Token: None 2022-09-02T04:42:21.489 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059b02f539526cb3a-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.538 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.539 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.540 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.541 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.542 DEBUG:boto:Method: GET 2022-09-02T04:42:21.543 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.544 DEBUG:boto:Data: 2022-09-02T04:42:21.545 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.546 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.547 DEBUG:boto:Params: {} 2022-09-02T04:42:21.548 DEBUG:boto:Token: None 2022-09-02T04:42:21.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018305b61bcbdeaca-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.555 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.555 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.556 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.556 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.557 DEBUG:boto:Method: GET 2022-09-02T04:42:21.557 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.558 DEBUG:boto:Data: 2022-09-02T04:42:21.558 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.559 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.560 DEBUG:boto:Params: {} 2022-09-02T04:42:21.561 DEBUG:boto:Token: None 2022-09-02T04:42:21.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f3a43a09774ba49-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.567 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.567 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.568 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.569 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.569 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.570 DEBUG:boto:Method: GET 2022-09-02T04:42:21.570 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.571 DEBUG:boto:Data: 2022-09-02T04:42:21.572 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.573 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.574 DEBUG:boto:Params: {} 2022-09-02T04:42:21.575 DEBUG:boto:Token: None 2022-09-02T04:42:21.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000940280bd909c92a3-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.580 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.581 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.582 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.583 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.583 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.584 DEBUG:boto:Method: GET 2022-09-02T04:42:21.584 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.585 DEBUG:boto:Data: 2022-09-02T04:42:21.585 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.586 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.587 DEBUG:boto:Params: {} 2022-09-02T04:42:21.587 DEBUG:boto:Token: None 2022-09-02T04:42:21.588 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ= 2022-09-02T04:42:21.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ=', 'Content-Length': '0'} 2022-09-02T04:42:21.592 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a74c4b75e6adeae7-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.593 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.593 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.594 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.594 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.595 DEBUG:boto:Method: GET 2022-09-02T04:42:21.595 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.596 DEBUG:boto:Data: 2022-09-02T04:42:21.596 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.597 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.598 DEBUG:boto:Params: {} 2022-09-02T04:42:21.598 DEBUG:boto:Token: None 2022-09-02T04:42:21.599 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ= 2022-09-02T04:42:21.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ=', 'Content-Length': '0'} 2022-09-02T04:42:21.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020fee8ba7ac718e7-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.604 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.605 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.606 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.607 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.608 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.608 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.609 DEBUG:boto:Data: 2022-09-02T04:42:21.610 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.612 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.613 DEBUG:boto:Params: {} 2022-09-02T04:42:21.614 DEBUG:boto:Token: None 2022-09-02T04:42:21.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9cWordUr+2MKqjt7nqbG3HCGalU= 2022-09-02T04:42:21.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9cWordUr+2MKqjt7nqbG3HCGalU=', 'Content-Length': '0'} 2022-09-02T04:42:21.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b660abdb2fe5c836-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.621 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.622 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.623 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.624 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.625 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.625 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.626 DEBUG:boto:Data: 2022-09-02T04:42:21.627 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.628 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.629 DEBUG:boto:Params: {} 2022-09-02T04:42:21.630 DEBUG:boto:Token: None 2022-09-02T04:42:21.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9cWordUr+2MKqjt7nqbG3HCGalU= 2022-09-02T04:42:21.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9cWordUr+2MKqjt7nqbG3HCGalU=', 'Content-Length': '0'} 2022-09-02T04:42:21.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000670f48301c5d1fd4-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.635 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.635 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.636 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.636 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.637 DEBUG:boto:Method: GET 2022-09-02T04:42:21.638 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.639 DEBUG:boto:Data: 2022-09-02T04:42:21.639 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.640 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.641 DEBUG:boto:Params: {} 2022-09-02T04:42:21.642 DEBUG:boto:Token: None 2022-09-02T04:42:21.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ= 2022-09-02T04:42:21.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ=', 'Content-Length': '0'} 2022-09-02T04:42:21.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd146cda1b9e15cc-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.647 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.648 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.649 DEBUG:boto:path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.650 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.651 DEBUG:boto:Method: GET 2022-09-02T04:42:21.652 DEBUG:boto:Path: /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.653 DEBUG:boto:Data: 2022-09-02T04:42:21.654 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.656 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.657 DEBUG:boto:Params: {} 2022-09-02T04:42:21.657 DEBUG:boto:Token: None 2022-09-02T04:42:21.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=riISQQdNpXBqmhJe5KXTuGBBXBg640R 2022-09-02T04:42:21.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ= 2022-09-02T04:42:21.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gYYSAEn3g6dCDxsYu8BTb23N/PQ=', 'Content-Length': '0'} 2022-09-02T04:42:21.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'riISQQdNpXBqmhJe5KXTuGBBXBg640R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dcc5d59b0da776b-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.706 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.707 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.708 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.709 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.710 DEBUG:boto:Method: GET 2022-09-02T04:42:21.711 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.712 DEBUG:boto:Data: 2022-09-02T04:42:21.712 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.714 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.715 DEBUG:boto:Params: {} 2022-09-02T04:42:21.716 DEBUG:boto:Token: None 2022-09-02T04:42:21.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec17cf5e6e03111f-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.723 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.724 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.725 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.725 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.726 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.727 DEBUG:boto:Method: GET 2022-09-02T04:42:21.727 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.728 DEBUG:boto:Data: 2022-09-02T04:42:21.728 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.729 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.730 DEBUG:boto:Params: {} 2022-09-02T04:42:21.730 DEBUG:boto:Token: None 2022-09-02T04:42:21.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fb7bf3151f44d0d-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.736 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.737 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.737 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.738 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.738 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:21.739 DEBUG:boto:Data: 2022-09-02T04:42:21.739 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.741 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.741 DEBUG:boto:Params: {} 2022-09-02T04:42:21.742 DEBUG:boto:Token: None 2022-09-02T04:42:21.742 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj 2022-09-02T04:42:21.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg= 2022-09-02T04:42:21.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg=', 'Content-Length': '0'} 2022-09-02T04:42:21.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b6e11dfe0f88130-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.747 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.748 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.748 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.749 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:21.749 DEBUG:boto:Data: 2022-09-02T04:42:21.750 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.751 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.752 DEBUG:boto:Params: {} 2022-09-02T04:42:21.752 DEBUG:boto:Token: None 2022-09-02T04:42:21.753 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj 2022-09-02T04:42:21.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg= 2022-09-02T04:42:21.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:266fVODaAVNXR+1CUjP5hZni8Hg=', 'Content-Length': '0'} 2022-09-02T04:42:21.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3e8e6bd4f68c312-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.759 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.760 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.760 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.761 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.762 DEBUG:boto:Method: GET 2022-09-02T04:42:21.762 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.763 DEBUG:boto:Data: 2022-09-02T04:42:21.763 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.765 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.765 DEBUG:boto:Params: {} 2022-09-02T04:42:21.766 DEBUG:boto:Token: None 2022-09-02T04:42:21.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053c433971d1e8b2a-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.814 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.815 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.816 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.817 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.818 DEBUG:boto:Method: GET 2022-09-02T04:42:21.819 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.820 DEBUG:boto:Data: 2022-09-02T04:42:21.821 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.823 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.824 DEBUG:boto:Params: {} 2022-09-02T04:42:21.825 DEBUG:boto:Token: None 2022-09-02T04:42:21.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:21.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60= 2022-09-02T04:42:21.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVZz2ckGnocg9WFMEppEgqvns60=', 'Content-Length': '0'} 2022-09-02T04:42:21.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003059547db00f59aa-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.833 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.833 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.834 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.835 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.836 DEBUG:boto:Method: GET 2022-09-02T04:42:21.837 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.838 DEBUG:boto:Data: 2022-09-02T04:42:21.839 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.841 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.842 DEBUG:boto:Params: {} 2022-09-02T04:42:21.843 DEBUG:boto:Token: None 2022-09-02T04:42:21.844 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a99837bfee264d-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.851 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.852 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.853 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.853 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.854 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.855 DEBUG:boto:Method: GET 2022-09-02T04:42:21.856 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.856 DEBUG:boto:Data: 2022-09-02T04:42:21.857 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.859 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.859 DEBUG:boto:Params: {} 2022-09-02T04:42:21.860 DEBUG:boto:Token: None 2022-09-02T04:42:21.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f720d56a3cb6ffe9-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.906 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:21.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.908 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.908 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.909 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.910 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.911 DEBUG:boto:Method: GET 2022-09-02T04:42:21.912 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.912 DEBUG:boto:Data: 2022-09-02T04:42:21.913 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.914 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.914 DEBUG:boto:Params: {} 2022-09-02T04:42:21.915 DEBUG:boto:Token: None 2022-09-02T04:42:21.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA= 2022-09-02T04:42:21.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA=', 'Content-Length': '0'} 2022-09-02T04:42:21.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b17ee2783c93e396-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.920 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.921 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.921 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.922 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.923 DEBUG:boto:Method: GET 2022-09-02T04:42:21.924 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.925 DEBUG:boto:Data: 2022-09-02T04:42:21.926 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.928 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.928 DEBUG:boto:Params: {} 2022-09-02T04:42:21.929 DEBUG:boto:Token: None 2022-09-02T04:42:21.930 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.931 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA= 2022-09-02T04:42:21.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA=', 'Content-Length': '0'} 2022-09-02T04:42:21.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ab0ca02ce7a5b4e-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.935 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.935 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.936 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.936 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.937 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.937 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.938 DEBUG:boto:Data: 2022-09-02T04:42:21.939 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.940 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.940 DEBUG:boto:Params: {} 2022-09-02T04:42:21.941 DEBUG:boto:Token: None 2022-09-02T04:42:21.941 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s1sMEIM95qAV9can94gAPb8dzO4= 2022-09-02T04:42:21.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s1sMEIM95qAV9can94gAPb8dzO4=', 'Content-Length': '0'} 2022-09-02T04:42:21.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007d03bc050c4cf29-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.946 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.947 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.947 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.948 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.948 DEBUG:boto:Method: HEAD 2022-09-02T04:42:21.949 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.949 DEBUG:boto:Data: 2022-09-02T04:42:21.950 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.951 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.951 DEBUG:boto:Params: {} 2022-09-02T04:42:21.952 DEBUG:boto:Token: None 2022-09-02T04:42:21.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s1sMEIM95qAV9can94gAPb8dzO4= 2022-09-02T04:42:21.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s1sMEIM95qAV9can94gAPb8dzO4=', 'Content-Length': '0'} 2022-09-02T04:42:21.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019b07f7a3e40e4e9-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:21.958 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.958 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.959 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.960 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.960 DEBUG:boto:Method: GET 2022-09-02T04:42:21.961 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.961 DEBUG:boto:Data: 2022-09-02T04:42:21.962 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.963 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.963 DEBUG:boto:Params: {} 2022-09-02T04:42:21.964 DEBUG:boto:Token: None 2022-09-02T04:42:21.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA= 2022-09-02T04:42:21.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA=', 'Content-Length': '0'} 2022-09-02T04:42:21.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048ce7c3f3545787c-00631189ad-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.969 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:21.969 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:21.970 DEBUG:boto:path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.971 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.972 DEBUG:boto:Method: GET 2022-09-02T04:42:21.973 DEBUG:boto:Path: /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.973 DEBUG:boto:Data: 2022-09-02T04:42:21.974 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:21.975 DEBUG:boto:Port: 8000 2022-09-02T04:42:21.976 DEBUG:boto:Params: {} 2022-09-02T04:42:21.977 DEBUG:boto:Token: None 2022-09-02T04:42:21.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/obj?versionId=C.Anncl.p5MU1JnCw8HH1LhxTImrIT2 2022-09-02T04:42:21.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA= 2022-09-02T04:42:21.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCQsb4QxS95DE6/JpV6rGQKbmUA=', 'Content-Length': '0'} 2022-09-02T04:42:21.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'C.Anncl.p5MU1JnCw8HH1LhxTImrIT2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a794cb3f951074e-00631189ad-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.983 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:21.984 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:21.985 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.986 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:21.987 DEBUG:boto:Method: GET 2022-09-02T04:42:21.987 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:21.988 DEBUG:boto:Data: 2022-09-02T04:42:21.989 DEBUG:boto:Headers: {} 2022-09-02T04:42:21.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:21.990 DEBUG:boto:Port: 8001 2022-09-02T04:42:21.991 DEBUG:boto:Params: {} 2022-09-02T04:42:21.992 DEBUG:boto:Token: None 2022-09-02T04:42:21.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:21 GMT /xljbkf-19/ 2022-09-02T04:42:21.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI= 2022-09-02T04:42:21.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4okoE/XW57TXNiPeWOjZl77uyXI=', 'Content-Length': '0'} 2022-09-02T04:42:21.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000442b834b78c4e2d8-00631189ad-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:21.998 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.000 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.001 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.002 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.003 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.003 DEBUG:boto:Method: GET 2022-09-02T04:42:22.004 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.005 DEBUG:boto:Data: 2022-09-02T04:42:22.006 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.007 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.008 DEBUG:boto:Params: {} 2022-09-02T04:42:22.009 DEBUG:boto:Token: None 2022-09-02T04:42:22.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081f4cce5d59e786-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.014 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.015 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.015 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.016 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.017 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.017 DEBUG:boto:Data: 2022-09-02T04:42:22.018 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.019 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.020 DEBUG:boto:Params: {} 2022-09-02T04:42:22.020 DEBUG:boto:Token: None 2022-09-02T04:42:22.021 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.022 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000272e34ed787180a3-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.027 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.028 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.028 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.029 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.030 DEBUG:boto:Data: 2022-09-02T04:42:22.031 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.032 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.032 DEBUG:boto:Params: {} 2022-09-02T04:42:22.033 DEBUG:boto:Token: None 2022-09-02T04:42:22.034 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.035 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bb2ebd15066262f-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.041 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.041 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.042 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.042 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.043 DEBUG:boto:Method: GET 2022-09-02T04:42:22.044 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.044 DEBUG:boto:Data: 2022-09-02T04:42:22.045 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.046 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.047 DEBUG:boto:Params: {} 2022-09-02T04:42:22.048 DEBUG:boto:Token: None 2022-09-02T04:42:22.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078b7a476de403377-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.053 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.054 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.054 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.055 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.056 DEBUG:boto:Method: GET 2022-09-02T04:42:22.057 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.057 DEBUG:boto:Data: 2022-09-02T04:42:22.058 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.059 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.059 DEBUG:boto:Params: {} 2022-09-02T04:42:22.060 DEBUG:boto:Token: None 2022-09-02T04:42:22.061 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b839bbd8cbdbd9e-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.106 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.108 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.109 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.109 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.110 DEBUG:boto:Method: GET 2022-09-02T04:42:22.110 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.111 DEBUG:boto:Data: 2022-09-02T04:42:22.112 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.113 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.114 DEBUG:boto:Params: {} 2022-09-02T04:42:22.115 DEBUG:boto:Token: None 2022-09-02T04:42:22.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c621111c3163880-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.120 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.121 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.122 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.123 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.124 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.125 DEBUG:boto:Method: GET 2022-09-02T04:42:22.126 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.127 DEBUG:boto:Data: 2022-09-02T04:42:22.128 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.130 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.131 DEBUG:boto:Params: {} 2022-09-02T04:42:22.132 DEBUG:boto:Token: None 2022-09-02T04:42:22.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004ef9f9b0bf0ba84-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.138 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.140 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.140 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.141 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.141 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.142 DEBUG:boto:Method: GET 2022-09-02T04:42:22.142 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.143 DEBUG:boto:Data: 2022-09-02T04:42:22.143 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.144 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.145 DEBUG:boto:Params: {} 2022-09-02T04:42:22.145 DEBUG:boto:Token: None 2022-09-02T04:42:22.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A= 2022-09-02T04:42:22.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A=', 'Content-Length': '0'} 2022-09-02T04:42:22.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d2fa0084eab08bb-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.190 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.191 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.192 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.194 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.195 DEBUG:boto:Method: GET 2022-09-02T04:42:22.196 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.197 DEBUG:boto:Data: 2022-09-02T04:42:22.198 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.200 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.201 DEBUG:boto:Params: {} 2022-09-02T04:42:22.202 DEBUG:boto:Token: None 2022-09-02T04:42:22.203 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A= 2022-09-02T04:42:22.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A=', 'Content-Length': '0'} 2022-09-02T04:42:22.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018b80436cf59b7e7-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.208 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.209 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.210 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.211 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.211 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.212 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.212 DEBUG:boto:Data: 2022-09-02T04:42:22.213 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.214 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.215 DEBUG:boto:Params: {} 2022-09-02T04:42:22.215 DEBUG:boto:Token: None 2022-09-02T04:42:22.216 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/c4K1r5oXwF2Uc3NON+XwqscEvE= 2022-09-02T04:42:22.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/c4K1r5oXwF2Uc3NON+XwqscEvE=', 'Content-Length': '0'} 2022-09-02T04:42:22.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b991f0c1d265a99d-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.220 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.221 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.221 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.222 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.223 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.224 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.224 DEBUG:boto:Data: 2022-09-02T04:42:22.225 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.226 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.226 DEBUG:boto:Params: {} 2022-09-02T04:42:22.227 DEBUG:boto:Token: None 2022-09-02T04:42:22.227 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/c4K1r5oXwF2Uc3NON+XwqscEvE= 2022-09-02T04:42:22.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/c4K1r5oXwF2Uc3NON+XwqscEvE=', 'Content-Length': '0'} 2022-09-02T04:42:22.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000841cf4918187799d-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.233 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.234 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.235 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.236 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.237 DEBUG:boto:Method: GET 2022-09-02T04:42:22.238 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.238 DEBUG:boto:Data: 2022-09-02T04:42:22.239 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.240 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.241 DEBUG:boto:Params: {} 2022-09-02T04:42:22.241 DEBUG:boto:Token: None 2022-09-02T04:42:22.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A= 2022-09-02T04:42:22.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A=', 'Content-Length': '0'} 2022-09-02T04:42:22.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000771783b97535c285-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.247 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.247 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.248 DEBUG:boto:path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.248 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.249 DEBUG:boto:Method: GET 2022-09-02T04:42:22.249 DEBUG:boto:Path: /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.250 DEBUG:boto:Data: 2022-09-02T04:42:22.251 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.252 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.252 DEBUG:boto:Params: {} 2022-09-02T04:42:22.253 DEBUG:boto:Token: None 2022-09-02T04:42:22.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=WdeoY5zneE2FAv88Q0QypGUzSkECcdd 2022-09-02T04:42:22.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A= 2022-09-02T04:42:22.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8fMrCzoSbiximld2AemAgFmdZ/A=', 'Content-Length': '0'} 2022-09-02T04:42:22.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'WdeoY5zneE2FAv88Q0QypGUzSkECcdd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f893674b4aa8cf-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.298 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.300 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.300 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.301 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.302 DEBUG:boto:Method: GET 2022-09-02T04:42:22.303 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.304 DEBUG:boto:Data: 2022-09-02T04:42:22.305 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.307 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.307 DEBUG:boto:Params: {} 2022-09-02T04:42:22.308 DEBUG:boto:Token: None 2022-09-02T04:42:22.309 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf894f2aa137386-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.315 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.316 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.316 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.317 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.318 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.319 DEBUG:boto:Method: GET 2022-09-02T04:42:22.320 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.321 DEBUG:boto:Data: 2022-09-02T04:42:22.322 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.323 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.323 DEBUG:boto:Params: {} 2022-09-02T04:42:22.324 DEBUG:boto:Token: None 2022-09-02T04:42:22.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca22d7e042a6881-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.330 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.331 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.331 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.332 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.332 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.333 DEBUG:boto:Data: 2022-09-02T04:42:22.333 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.335 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.335 DEBUG:boto:Params: {} 2022-09-02T04:42:22.336 DEBUG:boto:Token: None 2022-09-02T04:42:22.336 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5cd4f6a1cfefa4f-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.343 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.343 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.344 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.344 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.345 DEBUG:boto:Data: 2022-09-02T04:42:22.345 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.346 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.347 DEBUG:boto:Params: {} 2022-09-02T04:42:22.347 DEBUG:boto:Token: None 2022-09-02T04:42:22.348 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048d259d830c8d1ba-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.355 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.356 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.356 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.357 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.357 DEBUG:boto:Method: GET 2022-09-02T04:42:22.358 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.359 DEBUG:boto:Data: 2022-09-02T04:42:22.359 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.360 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.361 DEBUG:boto:Params: {} 2022-09-02T04:42:22.361 DEBUG:boto:Token: None 2022-09-02T04:42:22.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee5cf617f8d13192-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.366 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.367 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.367 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.368 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.368 DEBUG:boto:Method: GET 2022-09-02T04:42:22.369 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.369 DEBUG:boto:Data: 2022-09-02T04:42:22.370 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.370 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.371 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.371 DEBUG:boto:Params: {} 2022-09-02T04:42:22.372 DEBUG:boto:Token: None 2022-09-02T04:42:22.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae252aee5bb93442-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.377 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.378 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.378 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.379 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.380 DEBUG:boto:Method: GET 2022-09-02T04:42:22.380 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.381 DEBUG:boto:Data: 2022-09-02T04:42:22.381 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.382 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.383 DEBUG:boto:Params: {} 2022-09-02T04:42:22.384 DEBUG:boto:Token: None 2022-09-02T04:42:22.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3b0a81c9730637-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.389 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.390 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.390 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.391 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.391 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.392 DEBUG:boto:Method: GET 2022-09-02T04:42:22.392 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.393 DEBUG:boto:Data: 2022-09-02T04:42:22.393 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.394 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.394 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.395 DEBUG:boto:Params: {} 2022-09-02T04:42:22.396 DEBUG:boto:Token: None 2022-09-02T04:42:22.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006bad5010d015dd7-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.401 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.402 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.403 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.404 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.404 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.405 DEBUG:boto:Method: GET 2022-09-02T04:42:22.405 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.406 DEBUG:boto:Data: 2022-09-02T04:42:22.406 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.407 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.408 DEBUG:boto:Params: {} 2022-09-02T04:42:22.409 DEBUG:boto:Token: None 2022-09-02T04:42:22.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w= 2022-09-02T04:42:22.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w=', 'Content-Length': '0'} 2022-09-02T04:42:22.413 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000716efe2916262f72-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.414 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.415 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.415 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.416 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.416 DEBUG:boto:Method: GET 2022-09-02T04:42:22.417 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.418 DEBUG:boto:Data: 2022-09-02T04:42:22.419 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.421 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.421 DEBUG:boto:Params: {} 2022-09-02T04:42:22.422 DEBUG:boto:Token: None 2022-09-02T04:42:22.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w= 2022-09-02T04:42:22.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w=', 'Content-Length': '0'} 2022-09-02T04:42:22.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000367db99816512920-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.427 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.428 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.428 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.429 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.430 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.430 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.431 DEBUG:boto:Data: 2022-09-02T04:42:22.431 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.432 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.433 DEBUG:boto:Params: {} 2022-09-02T04:42:22.434 DEBUG:boto:Token: None 2022-09-02T04:42:22.434 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12xJ9h5L79eweGY/xmWzOAEKNqE= 2022-09-02T04:42:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12xJ9h5L79eweGY/xmWzOAEKNqE=', 'Content-Length': '0'} 2022-09-02T04:42:22.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9f02d34e692a05b-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.439 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.439 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.440 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.440 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.441 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.441 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.442 DEBUG:boto:Data: 2022-09-02T04:42:22.442 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.443 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.444 DEBUG:boto:Params: {} 2022-09-02T04:42:22.444 DEBUG:boto:Token: None 2022-09-02T04:42:22.445 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:12xJ9h5L79eweGY/xmWzOAEKNqE= 2022-09-02T04:42:22.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:12xJ9h5L79eweGY/xmWzOAEKNqE=', 'Content-Length': '0'} 2022-09-02T04:42:22.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008538cbd892bd0a46-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.450 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.451 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.452 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.452 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.453 DEBUG:boto:Method: GET 2022-09-02T04:42:22.454 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.454 DEBUG:boto:Data: 2022-09-02T04:42:22.455 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.457 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.458 DEBUG:boto:Params: {} 2022-09-02T04:42:22.458 DEBUG:boto:Token: None 2022-09-02T04:42:22.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w= 2022-09-02T04:42:22.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w=', 'Content-Length': '0'} 2022-09-02T04:42:22.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081c7c8a8bf708ade-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.464 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.464 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.465 DEBUG:boto:path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.465 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.466 DEBUG:boto:Method: GET 2022-09-02T04:42:22.467 DEBUG:boto:Path: /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.467 DEBUG:boto:Data: 2022-09-02T04:42:22.468 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.469 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.470 DEBUG:boto:Params: {} 2022-09-02T04:42:22.472 DEBUG:boto:Token: None 2022-09-02T04:42:22.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=gA.6Quiy8aYXdSNGParYlZMUxQBATsL 2022-09-02T04:42:22.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w= 2022-09-02T04:42:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pF5Uoo/0Ijy6QjcBw//JNN1Zx9w=', 'Content-Length': '0'} 2022-09-02T04:42:22.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'gA.6Quiy8aYXdSNGParYlZMUxQBATsL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000866407c288e51b1d-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.479 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.480 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.481 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.482 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.483 DEBUG:boto:Method: GET 2022-09-02T04:42:22.484 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.485 DEBUG:boto:Data: 2022-09-02T04:42:22.486 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.488 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.489 DEBUG:boto:Params: {} 2022-09-02T04:42:22.489 DEBUG:boto:Token: None 2022-09-02T04:42:22.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06191d2c9d5900b-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.538 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.540 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.541 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.542 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.543 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.543 DEBUG:boto:Method: GET 2022-09-02T04:42:22.544 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.545 DEBUG:boto:Data: 2022-09-02T04:42:22.546 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.547 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.548 DEBUG:boto:Params: {} 2022-09-02T04:42:22.549 DEBUG:boto:Token: None 2022-09-02T04:42:22.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a762cdfd71fafca9-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.555 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.556 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.556 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.557 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.557 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.558 DEBUG:boto:Data: 2022-09-02T04:42:22.558 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.559 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.560 DEBUG:boto:Params: {} 2022-09-02T04:42:22.560 DEBUG:boto:Token: None 2022-09-02T04:42:22.561 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000161a8aa5789f1ca9-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.568 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.569 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.570 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.570 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.571 DEBUG:boto:Data: 2022-09-02T04:42:22.572 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.572 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.573 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.573 DEBUG:boto:Params: {} 2022-09-02T04:42:22.574 DEBUG:boto:Token: None 2022-09-02T04:42:22.575 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbb899d85a19a389-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.580 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.581 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.582 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.583 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.584 DEBUG:boto:Method: GET 2022-09-02T04:42:22.585 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.586 DEBUG:boto:Data: 2022-09-02T04:42:22.587 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.588 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.589 DEBUG:boto:Params: {} 2022-09-02T04:42:22.590 DEBUG:boto:Token: None 2022-09-02T04:42:22.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb3fd472ca2d0f80-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.595 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.596 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.596 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.597 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.597 DEBUG:boto:Method: GET 2022-09-02T04:42:22.598 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.598 DEBUG:boto:Data: 2022-09-02T04:42:22.599 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.600 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.600 DEBUG:boto:Params: {} 2022-09-02T04:42:22.601 DEBUG:boto:Token: None 2022-09-02T04:42:22.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b078a7e7c0cf0344-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.646 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.647 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.648 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.648 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.649 DEBUG:boto:Method: GET 2022-09-02T04:42:22.650 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.651 DEBUG:boto:Data: 2022-09-02T04:42:22.652 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.653 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.654 DEBUG:boto:Params: {} 2022-09-02T04:42:22.654 DEBUG:boto:Token: None 2022-09-02T04:42:22.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003153d31c274499e-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.660 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.662 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.663 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.664 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.665 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.666 DEBUG:boto:Method: GET 2022-09-02T04:42:22.666 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.667 DEBUG:boto:Data: 2022-09-02T04:42:22.668 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.670 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.670 DEBUG:boto:Params: {} 2022-09-02T04:42:22.671 DEBUG:boto:Token: None 2022-09-02T04:42:22.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030fd2b3a364de24f-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.677 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.679 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.679 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.680 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.681 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.682 DEBUG:boto:Method: GET 2022-09-02T04:42:22.683 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.684 DEBUG:boto:Data: 2022-09-02T04:42:22.685 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.687 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.688 DEBUG:boto:Params: {} 2022-09-02T04:42:22.689 DEBUG:boto:Token: None 2022-09-02T04:42:22.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE= 2022-09-02T04:42:22.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE=', 'Content-Length': '0'} 2022-09-02T04:42:22.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051490da36119f5bc-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.742 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.743 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.743 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.745 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.746 DEBUG:boto:Method: GET 2022-09-02T04:42:22.746 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.747 DEBUG:boto:Data: 2022-09-02T04:42:22.747 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.748 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.749 DEBUG:boto:Params: {} 2022-09-02T04:42:22.750 DEBUG:boto:Token: None 2022-09-02T04:42:22.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE= 2022-09-02T04:42:22.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE=', 'Content-Length': '0'} 2022-09-02T04:42:22.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ff805d008ad7d57-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.756 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.757 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.758 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.759 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.759 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.760 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.760 DEBUG:boto:Data: 2022-09-02T04:42:22.761 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.762 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.762 DEBUG:boto:Params: {} 2022-09-02T04:42:22.763 DEBUG:boto:Token: None 2022-09-02T04:42:22.764 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YB/u+Tez47/v9KH0h5JsiyWU7ww= 2022-09-02T04:42:22.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YB/u+Tez47/v9KH0h5JsiyWU7ww=', 'Content-Length': '0'} 2022-09-02T04:42:22.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fdeec7d6975492d-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.769 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.770 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.770 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.771 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.772 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.772 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.773 DEBUG:boto:Data: 2022-09-02T04:42:22.773 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.775 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.775 DEBUG:boto:Params: {} 2022-09-02T04:42:22.777 DEBUG:boto:Token: None 2022-09-02T04:42:22.777 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YB/u+Tez47/v9KH0h5JsiyWU7ww= 2022-09-02T04:42:22.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YB/u+Tez47/v9KH0h5JsiyWU7ww=', 'Content-Length': '0'} 2022-09-02T04:42:22.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1aec695cc26ecce-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.784 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.785 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.786 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.787 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.788 DEBUG:boto:Method: GET 2022-09-02T04:42:22.788 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.789 DEBUG:boto:Data: 2022-09-02T04:42:22.789 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.790 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.791 DEBUG:boto:Params: {} 2022-09-02T04:42:22.792 DEBUG:boto:Token: None 2022-09-02T04:42:22.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE= 2022-09-02T04:42:22.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE=', 'Content-Length': '0'} 2022-09-02T04:42:22.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d66d01f543869de6-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.797 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.797 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.798 DEBUG:boto:path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.799 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.799 DEBUG:boto:Method: GET 2022-09-02T04:42:22.800 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.800 DEBUG:boto:Data: 2022-09-02T04:42:22.801 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.801 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.802 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.802 DEBUG:boto:Params: {} 2022-09-02T04:42:22.803 DEBUG:boto:Token: None 2022-09-02T04:42:22.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES 2022-09-02T04:42:22.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE= 2022-09-02T04:42:22.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+FpzHbL5d5pwJ1N181iuj/FrNSE=', 'Content-Length': '0'} 2022-09-02T04:42:22.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '7Yc35uuFZ1bDipSO-0afZ1kQQqX6kES'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7898fe8726a5c28-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.850 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.851 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.851 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.852 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.852 DEBUG:boto:Method: GET 2022-09-02T04:42:22.853 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.853 DEBUG:boto:Data: 2022-09-02T04:42:22.854 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.855 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.856 DEBUG:boto:Params: {} 2022-09-02T04:42:22.856 DEBUG:boto:Token: None 2022-09-02T04:42:22.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5c38e6d9c6d0260-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.862 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.862 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.863 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.864 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.864 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.865 DEBUG:boto:Method: GET 2022-09-02T04:42:22.865 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.866 DEBUG:boto:Data: 2022-09-02T04:42:22.866 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.867 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.868 DEBUG:boto:Params: {} 2022-09-02T04:42:22.869 DEBUG:boto:Token: None 2022-09-02T04:42:22.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000164da60bae6a7bff-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.876 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.877 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.878 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.879 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.880 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.880 DEBUG:boto:Data: 2022-09-02T04:42:22.881 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.882 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.882 DEBUG:boto:Params: {} 2022-09-02T04:42:22.883 DEBUG:boto:Token: None 2022-09-02T04:42:22.884 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cacc8f034c58615-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.889 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.890 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.890 DEBUG:boto:Method: HEAD 2022-09-02T04:42:22.891 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:22.891 DEBUG:boto:Data: 2022-09-02T04:42:22.892 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.893 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.893 DEBUG:boto:Params: {} 2022-09-02T04:42:22.894 DEBUG:boto:Token: None 2022-09-02T04:42:22.894 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj 2022-09-02T04:42:22.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY= 2022-09-02T04:42:22.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Yp0j4D45+SkmZgrHJXr3JILoLY=', 'Content-Length': '0'} 2022-09-02T04:42:22.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000297f379798394627-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:22.900 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.901 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.901 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.902 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.903 DEBUG:boto:Method: GET 2022-09-02T04:42:22.903 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.904 DEBUG:boto:Data: 2022-09-02T04:42:22.905 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.906 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.907 DEBUG:boto:Params: {} 2022-09-02T04:42:22.909 DEBUG:boto:Token: None 2022-09-02T04:42:22.909 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad82a7b8839aa3ff-00631189ae-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.913 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:22.914 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:22.915 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.915 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.916 DEBUG:boto:Method: GET 2022-09-02T04:42:22.916 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.917 DEBUG:boto:Data: 2022-09-02T04:42:22.917 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.918 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.919 DEBUG:boto:Params: {} 2022-09-02T04:42:22.919 DEBUG:boto:Token: None 2022-09-02T04:42:22.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:22.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E= 2022-09-02T04:42:22.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K//fMvvhbtSbm0w7RoVR5aLoo/E=', 'Content-Length': '0'} 2022-09-02T04:42:22.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bbbc6ab5af1f0a4-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.970 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.971 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.971 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.972 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.972 DEBUG:boto:Method: GET 2022-09-02T04:42:22.973 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.974 DEBUG:boto:Data: 2022-09-02T04:42:22.974 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:22.975 DEBUG:boto:Port: 8001 2022-09-02T04:42:22.976 DEBUG:boto:Params: {} 2022-09-02T04:42:22.976 DEBUG:boto:Token: None 2022-09-02T04:42:22.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f06a8818028b40f-00631189ae-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.982 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:22.983 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:22.983 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:22.984 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.985 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:22.985 DEBUG:boto:Method: GET 2022-09-02T04:42:22.987 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:22.987 DEBUG:boto:Data: 2022-09-02T04:42:22.988 DEBUG:boto:Headers: {} 2022-09-02T04:42:22.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:22.989 DEBUG:boto:Port: 8000 2022-09-02T04:42:22.989 DEBUG:boto:Params: {} 2022-09-02T04:42:22.991 DEBUG:boto:Token: None 2022-09-02T04:42:22.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:22 GMT /xljbkf-19/ 2022-09-02T04:42:22.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI= 2022-09-02T04:42:22.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyYyNyx1zQPSnvFWqiW91VDMYbI=', 'Content-Length': '0'} 2022-09-02T04:42:22.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f260e1a3237b6cde-00631189ae-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:22.998 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.001 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.002 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.003 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.004 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.005 DEBUG:boto:Method: GET 2022-09-02T04:42:23.006 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.007 DEBUG:boto:Data: 2022-09-02T04:42:23.008 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.009 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.010 DEBUG:boto:Params: {} 2022-09-02T04:42:23.011 DEBUG:boto:Token: None 2022-09-02T04:42:23.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4= 2022-09-02T04:42:23.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4=', 'Content-Length': '0'} 2022-09-02T04:42:23.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc47325e5b5ebfb9-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.016 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.017 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.017 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.018 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.019 DEBUG:boto:Method: GET 2022-09-02T04:42:23.019 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.020 DEBUG:boto:Data: 2022-09-02T04:42:23.020 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.021 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.021 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.022 DEBUG:boto:Params: {} 2022-09-02T04:42:23.023 DEBUG:boto:Token: None 2022-09-02T04:42:23.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4= 2022-09-02T04:42:23.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4=', 'Content-Length': '0'} 2022-09-02T04:42:23.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e19bfea18a67f8bc-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.027 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.028 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.028 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.029 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.029 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.030 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.031 DEBUG:boto:Data: 2022-09-02T04:42:23.031 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.032 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.033 DEBUG:boto:Params: {} 2022-09-02T04:42:23.033 DEBUG:boto:Token: None 2022-09-02T04:42:23.034 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.034 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pDo8TJfL5pce/4n4M4mKIpZZOwc= 2022-09-02T04:42:23.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pDo8TJfL5pce/4n4M4mKIpZZOwc=', 'Content-Length': '0'} 2022-09-02T04:42:23.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4e15e64fc75ae82-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.038 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.038 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.039 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.039 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.040 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.040 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.041 DEBUG:boto:Data: 2022-09-02T04:42:23.041 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.042 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.043 DEBUG:boto:Params: {} 2022-09-02T04:42:23.043 DEBUG:boto:Token: None 2022-09-02T04:42:23.044 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pDo8TJfL5pce/4n4M4mKIpZZOwc= 2022-09-02T04:42:23.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pDo8TJfL5pce/4n4M4mKIpZZOwc=', 'Content-Length': '0'} 2022-09-02T04:42:23.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2fc33202b12db9e-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.049 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.049 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.050 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.050 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.051 DEBUG:boto:Method: GET 2022-09-02T04:42:23.051 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.052 DEBUG:boto:Data: 2022-09-02T04:42:23.053 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.054 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.054 DEBUG:boto:Params: {} 2022-09-02T04:42:23.055 DEBUG:boto:Token: None 2022-09-02T04:42:23.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4= 2022-09-02T04:42:23.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4=', 'Content-Length': '0'} 2022-09-02T04:42:23.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2709e7975a963f7-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.102 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.103 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.104 DEBUG:boto:path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.104 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.105 DEBUG:boto:Method: GET 2022-09-02T04:42:23.106 DEBUG:boto:Path: /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.107 DEBUG:boto:Data: 2022-09-02T04:42:23.109 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.110 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.111 DEBUG:boto:Params: {} 2022-09-02T04:42:23.112 DEBUG:boto:Token: None 2022-09-02T04:42:23.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG 2022-09-02T04:42:23.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4= 2022-09-02T04:42:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PfEcbJ1I/ZELRapSMJIbsIhJOb4=', 'Content-Length': '0'} 2022-09-02T04:42:23.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'MK9VPPFy6IG1KOcSvpN3jERnLiRbAyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e5d11ba26fb5945-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.118 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.119 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.119 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.120 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.120 DEBUG:boto:Method: GET 2022-09-02T04:42:23.121 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.121 DEBUG:boto:Data: 2022-09-02T04:42:23.122 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.123 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.123 DEBUG:boto:Params: {} 2022-09-02T04:42:23.124 DEBUG:boto:Token: None 2022-09-02T04:42:23.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d50e944e0fac8f-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.129 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.131 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.132 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.133 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.134 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.135 DEBUG:boto:Method: GET 2022-09-02T04:42:23.136 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.136 DEBUG:boto:Data: 2022-09-02T04:42:23.138 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.140 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.140 DEBUG:boto:Params: {} 2022-09-02T04:42:23.141 DEBUG:boto:Token: None 2022-09-02T04:42:23.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ec6d966a6baedfe-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.147 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.148 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.149 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.150 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.151 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.151 DEBUG:boto:Data: 2022-09-02T04:42:23.152 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.154 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.155 DEBUG:boto:Params: {} 2022-09-02T04:42:23.155 DEBUG:boto:Token: None 2022-09-02T04:42:23.156 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8e1200a047b1b4e-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.162 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.163 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.163 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.164 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.164 DEBUG:boto:Data: 2022-09-02T04:42:23.165 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.166 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.166 DEBUG:boto:Params: {} 2022-09-02T04:42:23.167 DEBUG:boto:Token: None 2022-09-02T04:42:23.168 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044dba5ebcd66c5cc-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.174 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.174 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.175 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.176 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.176 DEBUG:boto:Method: GET 2022-09-02T04:42:23.177 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.177 DEBUG:boto:Data: 2022-09-02T04:42:23.178 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.179 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.179 DEBUG:boto:Params: {} 2022-09-02T04:42:23.180 DEBUG:boto:Token: None 2022-09-02T04:42:23.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2d88584deec1a06-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.185 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.186 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.187 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.187 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.188 DEBUG:boto:Method: GET 2022-09-02T04:42:23.188 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.189 DEBUG:boto:Data: 2022-09-02T04:42:23.189 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.190 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.191 DEBUG:boto:Params: {} 2022-09-02T04:42:23.192 DEBUG:boto:Token: None 2022-09-02T04:42:23.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb16337170c1c6fe-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.198 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.199 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.199 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.200 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.201 DEBUG:boto:Method: GET 2022-09-02T04:42:23.201 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.202 DEBUG:boto:Data: 2022-09-02T04:42:23.203 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.204 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.204 DEBUG:boto:Params: {} 2022-09-02T04:42:23.205 DEBUG:boto:Token: None 2022-09-02T04:42:23.206 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e8c9ea8aa0b7c0-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.211 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.212 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.212 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.213 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.214 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.214 DEBUG:boto:Method: GET 2022-09-02T04:42:23.215 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.216 DEBUG:boto:Data: 2022-09-02T04:42:23.217 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.218 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.219 DEBUG:boto:Params: {} 2022-09-02T04:42:23.221 DEBUG:boto:Token: None 2022-09-02T04:42:23.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f73d909378cb14e0-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.228 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.230 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.231 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.232 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.232 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.233 DEBUG:boto:Method: GET 2022-09-02T04:42:23.233 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.234 DEBUG:boto:Data: 2022-09-02T04:42:23.235 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.236 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.237 DEBUG:boto:Params: {} 2022-09-02T04:42:23.237 DEBUG:boto:Token: None 2022-09-02T04:42:23.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto= 2022-09-02T04:42:23.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto=', 'Content-Length': '0'} 2022-09-02T04:42:23.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f4db54bb03d6243-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.243 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.243 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.244 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.244 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.245 DEBUG:boto:Method: GET 2022-09-02T04:42:23.246 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.246 DEBUG:boto:Data: 2022-09-02T04:42:23.247 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.248 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.249 DEBUG:boto:Params: {} 2022-09-02T04:42:23.249 DEBUG:boto:Token: None 2022-09-02T04:42:23.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto= 2022-09-02T04:42:23.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto=', 'Content-Length': '0'} 2022-09-02T04:42:23.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000857148718ad359ba-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.298 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.299 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.300 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.300 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.301 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.302 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.302 DEBUG:boto:Data: 2022-09-02T04:42:23.303 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.304 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.304 DEBUG:boto:Params: {} 2022-09-02T04:42:23.305 DEBUG:boto:Token: None 2022-09-02T04:42:23.305 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MYYpOhcaKybAnSr50hmW6pgSXbo= 2022-09-02T04:42:23.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MYYpOhcaKybAnSr50hmW6pgSXbo=', 'Content-Length': '0'} 2022-09-02T04:42:23.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4e9f9df468303fd-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.310 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.311 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.312 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.312 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.313 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.314 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.314 DEBUG:boto:Data: 2022-09-02T04:42:23.315 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.316 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.317 DEBUG:boto:Params: {} 2022-09-02T04:42:23.318 DEBUG:boto:Token: None 2022-09-02T04:42:23.319 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MYYpOhcaKybAnSr50hmW6pgSXbo= 2022-09-02T04:42:23.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MYYpOhcaKybAnSr50hmW6pgSXbo=', 'Content-Length': '0'} 2022-09-02T04:42:23.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ffded600fe430c-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.326 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.327 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.327 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.328 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.328 DEBUG:boto:Method: GET 2022-09-02T04:42:23.329 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.329 DEBUG:boto:Data: 2022-09-02T04:42:23.330 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.331 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.332 DEBUG:boto:Params: {} 2022-09-02T04:42:23.332 DEBUG:boto:Token: None 2022-09-02T04:42:23.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto= 2022-09-02T04:42:23.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto=', 'Content-Length': '0'} 2022-09-02T04:42:23.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa7ef4b7dff0b844-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.338 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.339 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.339 DEBUG:boto:path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.340 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.340 DEBUG:boto:Method: GET 2022-09-02T04:42:23.341 DEBUG:boto:Path: /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.341 DEBUG:boto:Data: 2022-09-02T04:42:23.342 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.343 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.344 DEBUG:boto:Params: {} 2022-09-02T04:42:23.345 DEBUG:boto:Token: None 2022-09-02T04:42:23.346 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=zAZYkI9foiNudwZegqrWCiM-9J1OYVU 2022-09-02T04:42:23.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto= 2022-09-02T04:42:23.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pUUmDKukEOOhHfTgjJaPiiBGWto=', 'Content-Length': '0'} 2022-09-02T04:42:23.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', 'zAZYkI9foiNudwZegqrWCiM-9J1OYVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b5496a52f8ba09e-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.350 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.351 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.351 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.352 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.353 DEBUG:boto:Method: GET 2022-09-02T04:42:23.353 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.354 DEBUG:boto:Data: 2022-09-02T04:42:23.355 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.357 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.357 DEBUG:boto:Params: {} 2022-09-02T04:42:23.358 DEBUG:boto:Token: None 2022-09-02T04:42:23.358 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f8cfb21636d7e9-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.364 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.366 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.367 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.368 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.369 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.370 DEBUG:boto:Method: GET 2022-09-02T04:42:23.371 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.372 DEBUG:boto:Data: 2022-09-02T04:42:23.372 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.374 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.375 DEBUG:boto:Params: {} 2022-09-02T04:42:23.376 DEBUG:boto:Token: None 2022-09-02T04:42:23.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009271ca528f701bb-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.382 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.383 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.384 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.385 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.385 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.386 DEBUG:boto:Data: 2022-09-02T04:42:23.386 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.387 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.388 DEBUG:boto:Params: {} 2022-09-02T04:42:23.388 DEBUG:boto:Token: None 2022-09-02T04:42:23.389 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da7dc0be01fe6186-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.394 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.395 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.395 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.396 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.397 DEBUG:boto:Data: 2022-09-02T04:42:23.397 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.398 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.398 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.399 DEBUG:boto:Params: {} 2022-09-02T04:42:23.400 DEBUG:boto:Token: None 2022-09-02T04:42:23.400 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4b53d007a9eb0c9-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.406 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.407 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.408 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.408 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.409 DEBUG:boto:Method: GET 2022-09-02T04:42:23.409 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.410 DEBUG:boto:Data: 2022-09-02T04:42:23.410 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.411 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.412 DEBUG:boto:Params: {} 2022-09-02T04:42:23.413 DEBUG:boto:Token: None 2022-09-02T04:42:23.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ef748325d6d0719-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.462 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.463 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.463 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.464 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.464 DEBUG:boto:Method: GET 2022-09-02T04:42:23.465 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.466 DEBUG:boto:Data: 2022-09-02T04:42:23.466 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.467 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.468 DEBUG:boto:Params: {} 2022-09-02T04:42:23.469 DEBUG:boto:Token: None 2022-09-02T04:42:23.469 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5dd78fea6f2a0ec-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.475 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.475 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.476 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.476 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.477 DEBUG:boto:Method: GET 2022-09-02T04:42:23.477 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.478 DEBUG:boto:Data: 2022-09-02T04:42:23.478 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.479 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.480 DEBUG:boto:Params: {} 2022-09-02T04:42:23.481 DEBUG:boto:Token: None 2022-09-02T04:42:23.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007d7f36d45fd6e33-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.486 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.487 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.488 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.488 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.489 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.489 DEBUG:boto:Method: GET 2022-09-02T04:42:23.490 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.491 DEBUG:boto:Data: 2022-09-02T04:42:23.491 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.492 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.493 DEBUG:boto:Params: {} 2022-09-02T04:42:23.493 DEBUG:boto:Token: None 2022-09-02T04:42:23.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000867d50798069c51d-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.499 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.500 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.501 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.502 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.502 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.503 DEBUG:boto:Method: GET 2022-09-02T04:42:23.504 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.504 DEBUG:boto:Data: 2022-09-02T04:42:23.505 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.507 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.508 DEBUG:boto:Params: {} 2022-09-02T04:42:23.508 DEBUG:boto:Token: None 2022-09-02T04:42:23.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE= 2022-09-02T04:42:23.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE=', 'Content-Length': '0'} 2022-09-02T04:42:23.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d6ade1b708e0f56-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.514 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.515 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.515 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.516 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.517 DEBUG:boto:Method: GET 2022-09-02T04:42:23.517 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.518 DEBUG:boto:Data: 2022-09-02T04:42:23.519 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.520 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.521 DEBUG:boto:Params: {} 2022-09-02T04:42:23.522 DEBUG:boto:Token: None 2022-09-02T04:42:23.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE= 2022-09-02T04:42:23.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE=', 'Content-Length': '0'} 2022-09-02T04:42:23.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bd87126ee796123-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.527 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.528 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.528 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.529 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.530 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.530 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.531 DEBUG:boto:Data: 2022-09-02T04:42:23.531 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.532 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.533 DEBUG:boto:Params: {} 2022-09-02T04:42:23.534 DEBUG:boto:Token: None 2022-09-02T04:42:23.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eyHqf4xP14KwGUIIje5HUmh2Gow= 2022-09-02T04:42:23.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eyHqf4xP14KwGUIIje5HUmh2Gow=', 'Content-Length': '0'} 2022-09-02T04:42:23.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002631b7cefb6eb94f-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.541 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.542 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.542 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.543 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.543 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.544 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.544 DEBUG:boto:Data: 2022-09-02T04:42:23.545 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.546 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.546 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.547 DEBUG:boto:Params: {} 2022-09-02T04:42:23.547 DEBUG:boto:Token: None 2022-09-02T04:42:23.548 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eyHqf4xP14KwGUIIje5HUmh2Gow= 2022-09-02T04:42:23.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eyHqf4xP14KwGUIIje5HUmh2Gow=', 'Content-Length': '0'} 2022-09-02T04:42:23.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b31b77f0d45c7b9-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.553 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.553 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.554 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.555 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.556 DEBUG:boto:Method: GET 2022-09-02T04:42:23.556 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.557 DEBUG:boto:Data: 2022-09-02T04:42:23.558 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.560 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.561 DEBUG:boto:Params: {} 2022-09-02T04:42:23.563 DEBUG:boto:Token: None 2022-09-02T04:42:23.563 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE= 2022-09-02T04:42:23.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE=', 'Content-Length': '0'} 2022-09-02T04:42:23.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a4a6bc18ea87d3b-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.568 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.569 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.570 DEBUG:boto:path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.571 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.572 DEBUG:boto:Method: GET 2022-09-02T04:42:23.572 DEBUG:boto:Path: /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.573 DEBUG:boto:Data: 2022-09-02T04:42:23.574 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.575 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.576 DEBUG:boto:Params: {} 2022-09-02T04:42:23.577 DEBUG:boto:Token: None 2022-09-02T04:42:23.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB 2022-09-02T04:42:23.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE= 2022-09-02T04:42:23.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ijz3IbNwyafMPWY2mUTpzJ+2wkE=', 'Content-Length': '0'} 2022-09-02T04:42:23.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '5tFnNuY9Y0ZyGJIWsg-QWvubsDNtXeB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061ee70f8d2bd7e06-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.623 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.624 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.625 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.625 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.626 DEBUG:boto:Method: GET 2022-09-02T04:42:23.627 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.627 DEBUG:boto:Data: 2022-09-02T04:42:23.628 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.630 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.631 DEBUG:boto:Params: {} 2022-09-02T04:42:23.631 DEBUG:boto:Token: None 2022-09-02T04:42:23.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009077a4617bb5c869-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.638 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.640 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.641 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.641 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.642 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.642 DEBUG:boto:Method: GET 2022-09-02T04:42:23.643 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.644 DEBUG:boto:Data: 2022-09-02T04:42:23.645 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.646 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.647 DEBUG:boto:Params: {} 2022-09-02T04:42:23.647 DEBUG:boto:Token: None 2022-09-02T04:42:23.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.649 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000779cc5d24c8da948-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.654 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.654 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.655 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.656 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.656 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.657 DEBUG:boto:Data: 2022-09-02T04:42:23.658 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.659 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.660 DEBUG:boto:Params: {} 2022-09-02T04:42:23.660 DEBUG:boto:Token: None 2022-09-02T04:42:23.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee1ba7157a821316-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.667 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.667 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.668 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.668 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.669 DEBUG:boto:Data: 2022-09-02T04:42:23.670 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.671 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.672 DEBUG:boto:Params: {} 2022-09-02T04:42:23.672 DEBUG:boto:Token: None 2022-09-02T04:42:23.673 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f04ec42145259ed0-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.679 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.680 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.680 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.681 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.682 DEBUG:boto:Method: GET 2022-09-02T04:42:23.682 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.683 DEBUG:boto:Data: 2022-09-02T04:42:23.683 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.684 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.685 DEBUG:boto:Params: {} 2022-09-02T04:42:23.686 DEBUG:boto:Token: None 2022-09-02T04:42:23.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c07e1d6e2d8721bb-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.690 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.691 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.692 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.692 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.693 DEBUG:boto:Method: GET 2022-09-02T04:42:23.693 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.694 DEBUG:boto:Data: 2022-09-02T04:42:23.694 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.695 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.696 DEBUG:boto:Params: {} 2022-09-02T04:42:23.697 DEBUG:boto:Token: None 2022-09-02T04:42:23.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba45b34f02ea0e17-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.702 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.702 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.703 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.704 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.704 DEBUG:boto:Method: GET 2022-09-02T04:42:23.705 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.705 DEBUG:boto:Data: 2022-09-02T04:42:23.706 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.707 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.708 DEBUG:boto:Params: {} 2022-09-02T04:42:23.708 DEBUG:boto:Token: None 2022-09-02T04:42:23.709 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009242d0a0f815131b-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.715 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.717 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.718 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.719 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.719 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.720 DEBUG:boto:Method: GET 2022-09-02T04:42:23.721 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.721 DEBUG:boto:Data: 2022-09-02T04:42:23.722 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.723 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.724 DEBUG:boto:Params: {} 2022-09-02T04:42:23.724 DEBUG:boto:Token: None 2022-09-02T04:42:23.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015218ef347da4436-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.730 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.731 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.732 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.732 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.733 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.734 DEBUG:boto:Method: GET 2022-09-02T04:42:23.734 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.735 DEBUG:boto:Data: 2022-09-02T04:42:23.736 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.737 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.738 DEBUG:boto:Params: {} 2022-09-02T04:42:23.738 DEBUG:boto:Token: None 2022-09-02T04:42:23.739 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc= 2022-09-02T04:42:23.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc=', 'Content-Length': '0'} 2022-09-02T04:42:23.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000964f1e93ef86f880-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.743 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.744 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.744 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.745 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.746 DEBUG:boto:Method: GET 2022-09-02T04:42:23.747 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.747 DEBUG:boto:Data: 2022-09-02T04:42:23.748 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.749 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.749 DEBUG:boto:Params: {} 2022-09-02T04:42:23.750 DEBUG:boto:Token: None 2022-09-02T04:42:23.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc= 2022-09-02T04:42:23.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc=', 'Content-Length': '0'} 2022-09-02T04:42:23.754 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa2ec5fea0d932d1-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.755 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.756 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.757 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.757 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.758 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.759 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.760 DEBUG:boto:Data: 2022-09-02T04:42:23.761 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.762 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.763 DEBUG:boto:Params: {} 2022-09-02T04:42:23.764 DEBUG:boto:Token: None 2022-09-02T04:42:23.765 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EwfpE5Q0iwdsZYxmY8EL/HVKPBU= 2022-09-02T04:42:23.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EwfpE5Q0iwdsZYxmY8EL/HVKPBU=', 'Content-Length': '0'} 2022-09-02T04:42:23.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d4455cb8dc038b-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.770 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.771 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.771 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.772 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.772 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.773 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.773 DEBUG:boto:Data: 2022-09-02T04:42:23.774 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.775 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.775 DEBUG:boto:Params: {} 2022-09-02T04:42:23.776 DEBUG:boto:Token: None 2022-09-02T04:42:23.776 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EwfpE5Q0iwdsZYxmY8EL/HVKPBU= 2022-09-02T04:42:23.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EwfpE5Q0iwdsZYxmY8EL/HVKPBU=', 'Content-Length': '0'} 2022-09-02T04:42:23.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daac21adcb013716-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.781 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.781 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.782 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.782 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.783 DEBUG:boto:Method: GET 2022-09-02T04:42:23.783 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.784 DEBUG:boto:Data: 2022-09-02T04:42:23.785 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.786 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.787 DEBUG:boto:Params: {} 2022-09-02T04:42:23.787 DEBUG:boto:Token: None 2022-09-02T04:42:23.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc= 2022-09-02T04:42:23.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc=', 'Content-Length': '0'} 2022-09-02T04:42:23.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087964247155bd424-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.834 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.835 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.836 DEBUG:boto:path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.837 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.838 DEBUG:boto:Method: GET 2022-09-02T04:42:23.838 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.839 DEBUG:boto:Data: 2022-09-02T04:42:23.839 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.841 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.841 DEBUG:boto:Params: {} 2022-09-02T04:42:23.842 DEBUG:boto:Token: None 2022-09-02T04:42:23.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF 2022-09-02T04:42:23.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc= 2022-09-02T04:42:23.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M2C8Gp7SqCX3n0WvM6GYVDJyiRc=', 'Content-Length': '0'} 2022-09-02T04:42:23.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7UcPbrXXpYJiKq4c0YVIg6PwKnJRTgF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dc1f9dc7cefc810-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.847 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.847 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.848 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.849 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.849 DEBUG:boto:Method: GET 2022-09-02T04:42:23.850 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.850 DEBUG:boto:Data: 2022-09-02T04:42:23.851 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.852 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.852 DEBUG:boto:Params: {} 2022-09-02T04:42:23.853 DEBUG:boto:Token: None 2022-09-02T04:42:23.854 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1df6db8bfe0830-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.859 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.860 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.861 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.862 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.863 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.864 DEBUG:boto:Method: GET 2022-09-02T04:42:23.865 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.865 DEBUG:boto:Data: 2022-09-02T04:42:23.866 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.868 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.869 DEBUG:boto:Params: {} 2022-09-02T04:42:23.870 DEBUG:boto:Token: None 2022-09-02T04:42:23.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed36d2239f64ae37-00631189af-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.918 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:23.920 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.920 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.921 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.922 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.922 DEBUG:boto:Data: 2022-09-02T04:42:23.923 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.926 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.927 DEBUG:boto:Params: {} 2022-09-02T04:42:23.928 DEBUG:boto:Token: None 2022-09-02T04:42:23.929 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000045ba53aa80af887-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.936 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.937 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.938 DEBUG:boto:Method: HEAD 2022-09-02T04:42:23.939 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:23.939 DEBUG:boto:Data: 2022-09-02T04:42:23.940 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.942 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.943 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.943 DEBUG:boto:Params: {} 2022-09-02T04:42:23.944 DEBUG:boto:Token: None 2022-09-02T04:42:23.945 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj 2022-09-02T04:42:23.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k= 2022-09-02T04:42:23.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CitvMroj4bxC6Ol29wTdHi/t0k=', 'Content-Length': '0'} 2022-09-02T04:42:23.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1ffd66fbeb1be16-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:23.952 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.953 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.953 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.954 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.955 DEBUG:boto:Method: GET 2022-09-02T04:42:23.955 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.956 DEBUG:boto:Data: 2022-09-02T04:42:23.957 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.958 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.958 DEBUG:boto:Params: {} 2022-09-02T04:42:23.959 DEBUG:boto:Token: None 2022-09-02T04:42:23.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e945029c3e4c5680-00631189af-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.965 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:23.967 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:23.968 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.968 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.969 DEBUG:boto:Method: GET 2022-09-02T04:42:23.970 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.971 DEBUG:boto:Data: 2022-09-02T04:42:23.972 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:23.974 DEBUG:boto:Port: 8000 2022-09-02T04:42:23.975 DEBUG:boto:Params: {} 2022-09-02T04:42:23.976 DEBUG:boto:Token: None 2022-09-02T04:42:23.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:23.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54= 2022-09-02T04:42:23.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G35W2C+s1rCVmQR2hcswon8+K54=', 'Content-Length': '0'} 2022-09-02T04:42:23.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e2c4608e0a54763-00631189af-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:23.983 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:23.984 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:23.984 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.985 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:23.986 DEBUG:boto:Method: GET 2022-09-02T04:42:23.986 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:23.987 DEBUG:boto:Data: 2022-09-02T04:42:23.988 DEBUG:boto:Headers: {} 2022-09-02T04:42:23.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:23.990 DEBUG:boto:Port: 8001 2022-09-02T04:42:23.991 DEBUG:boto:Params: {} 2022-09-02T04:42:23.992 DEBUG:boto:Token: None 2022-09-02T04:42:23.993 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:23 GMT /xljbkf-19/ 2022-09-02T04:42:23.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA= 2022-09-02T04:42:23.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UjVpA9Qa34Or7ONcQZ6liXhmYmA=', 'Content-Length': '0'} 2022-09-02T04:42:23.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c14f6f3f87c031-00631189af-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.003 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.005 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.005 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.006 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.006 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.007 DEBUG:boto:Method: GET 2022-09-02T04:42:24.007 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.008 DEBUG:boto:Data: 2022-09-02T04:42:24.008 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.009 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.010 DEBUG:boto:Params: {} 2022-09-02T04:42:24.011 DEBUG:boto:Token: None 2022-09-02T04:42:24.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a02aea5e426d0e8-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.017 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.019 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.019 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.020 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.020 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.021 DEBUG:boto:Method: GET 2022-09-02T04:42:24.023 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.024 DEBUG:boto:Data: 2022-09-02T04:42:24.024 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.026 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.027 DEBUG:boto:Params: {} 2022-09-02T04:42:24.028 DEBUG:boto:Token: None 2022-09-02T04:42:24.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw= 2022-09-02T04:42:24.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw=', 'Content-Length': '0'} 2022-09-02T04:42:24.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1413ba8fe66730a-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.082 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.083 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.084 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.085 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.086 DEBUG:boto:Method: GET 2022-09-02T04:42:24.087 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.088 DEBUG:boto:Data: 2022-09-02T04:42:24.089 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.090 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.091 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.091 DEBUG:boto:Params: {} 2022-09-02T04:42:24.092 DEBUG:boto:Token: None 2022-09-02T04:42:24.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw= 2022-09-02T04:42:24.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw=', 'Content-Length': '0'} 2022-09-02T04:42:24.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e93881e654d1ffd7-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.097 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.098 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.098 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.099 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.099 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.100 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.100 DEBUG:boto:Data: 2022-09-02T04:42:24.101 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.102 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.102 DEBUG:boto:Params: {} 2022-09-02T04:42:24.103 DEBUG:boto:Token: None 2022-09-02T04:42:24.104 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gz3/YKOHwih4E/l1wmJOgD1gW/E= 2022-09-02T04:42:24.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gz3/YKOHwih4E/l1wmJOgD1gW/E=', 'Content-Length': '0'} 2022-09-02T04:42:24.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044eb80216ea57b79-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.107 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.108 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.108 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.109 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.109 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.110 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.110 DEBUG:boto:Data: 2022-09-02T04:42:24.111 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.112 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.112 DEBUG:boto:Params: {} 2022-09-02T04:42:24.113 DEBUG:boto:Token: None 2022-09-02T04:42:24.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gz3/YKOHwih4E/l1wmJOgD1gW/E= 2022-09-02T04:42:24.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gz3/YKOHwih4E/l1wmJOgD1gW/E=', 'Content-Length': '0'} 2022-09-02T04:42:24.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000435b6996bf75d4b0-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.118 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.119 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.119 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.120 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.120 DEBUG:boto:Method: GET 2022-09-02T04:42:24.121 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.122 DEBUG:boto:Data: 2022-09-02T04:42:24.122 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.123 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.124 DEBUG:boto:Params: {} 2022-09-02T04:42:24.125 DEBUG:boto:Token: None 2022-09-02T04:42:24.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw= 2022-09-02T04:42:24.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw=', 'Content-Length': '0'} 2022-09-02T04:42:24.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b36f22883cce9a33-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.131 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.131 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.132 DEBUG:boto:path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.133 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.134 DEBUG:boto:Method: GET 2022-09-02T04:42:24.134 DEBUG:boto:Path: /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.135 DEBUG:boto:Data: 2022-09-02T04:42:24.136 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.137 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.137 DEBUG:boto:Params: {} 2022-09-02T04:42:24.138 DEBUG:boto:Token: None 2022-09-02T04:42:24.139 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=UB2I40j-SHaYLw4bCdkcpcE--BvSAl4 2022-09-02T04:42:24.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw= 2022-09-02T04:42:24.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3cCZL8YKS/F3nwSfUYBn11HnzBw=', 'Content-Length': '0'} 2022-09-02T04:42:24.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'UB2I40j-SHaYLw4bCdkcpcE--BvSAl4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b1eab2d28763188-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.144 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.144 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.145 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.146 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.147 DEBUG:boto:Method: GET 2022-09-02T04:42:24.147 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.148 DEBUG:boto:Data: 2022-09-02T04:42:24.149 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.150 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.151 DEBUG:boto:Params: {} 2022-09-02T04:42:24.152 DEBUG:boto:Token: None 2022-09-02T04:42:24.153 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac8d118e10d8f88-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.158 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.159 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.159 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.160 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.160 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.161 DEBUG:boto:Method: GET 2022-09-02T04:42:24.161 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.162 DEBUG:boto:Data: 2022-09-02T04:42:24.163 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.164 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.165 DEBUG:boto:Params: {} 2022-09-02T04:42:24.166 DEBUG:boto:Token: None 2022-09-02T04:42:24.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe02649aa65c165-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.218 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.219 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.220 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.220 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.221 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.221 DEBUG:boto:Data: 2022-09-02T04:42:24.222 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.223 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.224 DEBUG:boto:Params: {} 2022-09-02T04:42:24.224 DEBUG:boto:Token: None 2022-09-02T04:42:24.225 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:24.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:24.229 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d18bb6011337c8-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.230 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.231 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.231 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.232 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.232 DEBUG:boto:Data: 2022-09-02T04:42:24.233 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.235 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.235 DEBUG:boto:Params: {} 2022-09-02T04:42:24.236 DEBUG:boto:Token: None 2022-09-02T04:42:24.237 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:24.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:24.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001aaa2c4ac9db8bae-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.244 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.244 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.245 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.245 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.246 DEBUG:boto:Method: GET 2022-09-02T04:42:24.246 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.247 DEBUG:boto:Data: 2022-09-02T04:42:24.247 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.248 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.249 DEBUG:boto:Params: {} 2022-09-02T04:42:24.250 DEBUG:boto:Token: None 2022-09-02T04:42:24.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg= 2022-09-02T04:42:24.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg=', 'Content-Length': '0'} 2022-09-02T04:42:24.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed9682523235678a-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.255 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.256 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.256 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.257 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.257 DEBUG:boto:Method: GET 2022-09-02T04:42:24.258 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.258 DEBUG:boto:Data: 2022-09-02T04:42:24.259 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.260 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.262 DEBUG:boto:Params: {} 2022-09-02T04:42:24.262 DEBUG:boto:Token: None 2022-09-02T04:42:24.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg= 2022-09-02T04:42:24.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg=', 'Content-Length': '0'} 2022-09-02T04:42:24.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066b7b35431df83e2-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.268 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.269 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.269 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.270 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.271 DEBUG:boto:Method: GET 2022-09-02T04:42:24.272 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.272 DEBUG:boto:Data: 2022-09-02T04:42:24.273 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.275 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.275 DEBUG:boto:Params: {} 2022-09-02T04:42:24.276 DEBUG:boto:Token: None 2022-09-02T04:42:24.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b21fc6adc069c3-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.322 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.323 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.324 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.324 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.325 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.325 DEBUG:boto:Method: GET 2022-09-02T04:42:24.326 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.327 DEBUG:boto:Data: 2022-09-02T04:42:24.328 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.330 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.331 DEBUG:boto:Params: {} 2022-09-02T04:42:24.332 DEBUG:boto:Token: None 2022-09-02T04:42:24.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000659798cf71060aee-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.340 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.341 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.342 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.342 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.343 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.344 DEBUG:boto:Method: GET 2022-09-02T04:42:24.344 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.345 DEBUG:boto:Data: 2022-09-02T04:42:24.345 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.347 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.347 DEBUG:boto:Params: {} 2022-09-02T04:42:24.348 DEBUG:boto:Token: None 2022-09-02T04:42:24.348 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM= 2022-09-02T04:42:24.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM=', 'Content-Length': '0'} 2022-09-02T04:42:24.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000473c6bce02a2e336-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.354 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.355 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.356 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.357 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.358 DEBUG:boto:Method: GET 2022-09-02T04:42:24.359 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.360 DEBUG:boto:Data: 2022-09-02T04:42:24.361 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.363 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.364 DEBUG:boto:Params: {} 2022-09-02T04:42:24.365 DEBUG:boto:Token: None 2022-09-02T04:42:24.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM= 2022-09-02T04:42:24.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM=', 'Content-Length': '0'} 2022-09-02T04:42:24.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063e168ed37db4a33-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.373 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.374 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.375 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.376 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.377 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.378 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.379 DEBUG:boto:Data: 2022-09-02T04:42:24.380 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.381 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.382 DEBUG:boto:Params: {} 2022-09-02T04:42:24.383 DEBUG:boto:Token: None 2022-09-02T04:42:24.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YHREe7wgdBGW/o2wOYtDvUkNTi0= 2022-09-02T04:42:24.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YHREe7wgdBGW/o2wOYtDvUkNTi0=', 'Content-Length': '0'} 2022-09-02T04:42:24.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ec6857ffbb37837-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.390 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.391 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.391 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.392 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.393 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.394 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.396 DEBUG:boto:Data: 2022-09-02T04:42:24.396 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.399 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.399 DEBUG:boto:Params: {} 2022-09-02T04:42:24.400 DEBUG:boto:Token: None 2022-09-02T04:42:24.401 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YHREe7wgdBGW/o2wOYtDvUkNTi0= 2022-09-02T04:42:24.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YHREe7wgdBGW/o2wOYtDvUkNTi0=', 'Content-Length': '0'} 2022-09-02T04:42:24.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9e6aa8945f57ab7-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.405 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.406 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.407 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.407 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.408 DEBUG:boto:Method: GET 2022-09-02T04:42:24.408 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.409 DEBUG:boto:Data: 2022-09-02T04:42:24.409 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.410 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.411 DEBUG:boto:Params: {} 2022-09-02T04:42:24.411 DEBUG:boto:Token: None 2022-09-02T04:42:24.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM= 2022-09-02T04:42:24.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM=', 'Content-Length': '0'} 2022-09-02T04:42:24.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa4d052aa3eaf9a8-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.419 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.420 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.421 DEBUG:boto:path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.422 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.423 DEBUG:boto:Method: GET 2022-09-02T04:42:24.424 DEBUG:boto:Path: /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.425 DEBUG:boto:Data: 2022-09-02T04:42:24.425 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.427 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.427 DEBUG:boto:Params: {} 2022-09-02T04:42:24.428 DEBUG:boto:Token: None 2022-09-02T04:42:24.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=CsNSL980h52zVI65V9bT5gZak-3oUO6 2022-09-02T04:42:24.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM= 2022-09-02T04:42:24.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J6eiE/xEcxUPmaHArVy8XlmeTLM=', 'Content-Length': '0'} 2022-09-02T04:42:24.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'CsNSL980h52zVI65V9bT5gZak-3oUO6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004383d77f7b63d534-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.434 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.435 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.436 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.437 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.438 DEBUG:boto:Method: GET 2022-09-02T04:42:24.438 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.439 DEBUG:boto:Data: 2022-09-02T04:42:24.439 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.441 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.441 DEBUG:boto:Params: {} 2022-09-02T04:42:24.442 DEBUG:boto:Token: None 2022-09-02T04:42:24.443 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d35ff0cfb1f933c8-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.447 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.448 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.449 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.449 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.451 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.452 DEBUG:boto:Method: GET 2022-09-02T04:42:24.453 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.454 DEBUG:boto:Data: 2022-09-02T04:42:24.454 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.455 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.456 DEBUG:boto:Params: {} 2022-09-02T04:42:24.456 DEBUG:boto:Token: None 2022-09-02T04:42:24.457 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d599211d1a3cef5-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.462 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.463 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.463 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.464 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.464 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.465 DEBUG:boto:Data: 2022-09-02T04:42:24.465 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.466 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.467 DEBUG:boto:Params: {} 2022-09-02T04:42:24.467 DEBUG:boto:Token: None 2022-09-02T04:42:24.468 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:24.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:24.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaee8dfbfb9fbb5d-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.473 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.474 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.474 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.475 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.475 DEBUG:boto:Data: 2022-09-02T04:42:24.476 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.477 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.477 DEBUG:boto:Params: {} 2022-09-02T04:42:24.478 DEBUG:boto:Token: None 2022-09-02T04:42:24.479 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:24.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:24.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006885f982c56b249-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.487 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.487 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.488 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.488 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.489 DEBUG:boto:Method: GET 2022-09-02T04:42:24.489 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.490 DEBUG:boto:Data: 2022-09-02T04:42:24.491 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.492 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.492 DEBUG:boto:Params: {} 2022-09-02T04:42:24.493 DEBUG:boto:Token: None 2022-09-02T04:42:24.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg= 2022-09-02T04:42:24.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg=', 'Content-Length': '0'} 2022-09-02T04:42:24.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d298cb3e102ee26d-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.498 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.499 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.499 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.500 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.500 DEBUG:boto:Method: GET 2022-09-02T04:42:24.501 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.501 DEBUG:boto:Data: 2022-09-02T04:42:24.502 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.502 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.503 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.503 DEBUG:boto:Params: {} 2022-09-02T04:42:24.504 DEBUG:boto:Token: None 2022-09-02T04:42:24.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg= 2022-09-02T04:42:24.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg=', 'Content-Length': '0'} 2022-09-02T04:42:24.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f62f1e44461f9bf5-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.550 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.551 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.551 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.552 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.552 DEBUG:boto:Method: GET 2022-09-02T04:42:24.553 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.554 DEBUG:boto:Data: 2022-09-02T04:42:24.554 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.555 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.556 DEBUG:boto:Params: {} 2022-09-02T04:42:24.556 DEBUG:boto:Token: None 2022-09-02T04:42:24.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5129faf3f1c72b0-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.562 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.563 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.564 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.564 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.565 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.565 DEBUG:boto:Method: GET 2022-09-02T04:42:24.566 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.567 DEBUG:boto:Data: 2022-09-02T04:42:24.567 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.568 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.568 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.569 DEBUG:boto:Params: {} 2022-09-02T04:42:24.570 DEBUG:boto:Token: None 2022-09-02T04:42:24.570 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.571 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054ebfa5d4d55953b-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.575 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.577 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.577 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.578 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.579 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.579 DEBUG:boto:Method: GET 2022-09-02T04:42:24.580 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.580 DEBUG:boto:Data: 2022-09-02T04:42:24.581 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.582 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.583 DEBUG:boto:Params: {} 2022-09-02T04:42:24.583 DEBUG:boto:Token: None 2022-09-02T04:42:24.584 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs= 2022-09-02T04:42:24.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs=', 'Content-Length': '0'} 2022-09-02T04:42:24.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfb958b303faf45c-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.588 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.589 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.589 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.590 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.591 DEBUG:boto:Method: GET 2022-09-02T04:42:24.591 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.592 DEBUG:boto:Data: 2022-09-02T04:42:24.592 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.593 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.594 DEBUG:boto:Params: {} 2022-09-02T04:42:24.595 DEBUG:boto:Token: None 2022-09-02T04:42:24.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs= 2022-09-02T04:42:24.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs=', 'Content-Length': '0'} 2022-09-02T04:42:24.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a482b0f26e6d498-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.599 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.600 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.601 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.601 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.602 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.602 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.603 DEBUG:boto:Data: 2022-09-02T04:42:24.603 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.604 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.605 DEBUG:boto:Params: {} 2022-09-02T04:42:24.605 DEBUG:boto:Token: None 2022-09-02T04:42:24.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ziLCEsHGPTZGWZLCmdiOQxgtCq8= 2022-09-02T04:42:24.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ziLCEsHGPTZGWZLCmdiOQxgtCq8=', 'Content-Length': '0'} 2022-09-02T04:42:24.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b5a275faa35efde-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.611 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.611 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.612 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.612 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.613 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.613 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.614 DEBUG:boto:Data: 2022-09-02T04:42:24.614 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.615 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.616 DEBUG:boto:Params: {} 2022-09-02T04:42:24.617 DEBUG:boto:Token: None 2022-09-02T04:42:24.617 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ziLCEsHGPTZGWZLCmdiOQxgtCq8= 2022-09-02T04:42:24.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ziLCEsHGPTZGWZLCmdiOQxgtCq8=', 'Content-Length': '0'} 2022-09-02T04:42:24.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9600c289bfffeb9-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.622 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.623 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.624 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.625 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.625 DEBUG:boto:Method: GET 2022-09-02T04:42:24.626 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.626 DEBUG:boto:Data: 2022-09-02T04:42:24.627 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.628 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.629 DEBUG:boto:Params: {} 2022-09-02T04:42:24.629 DEBUG:boto:Token: None 2022-09-02T04:42:24.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs= 2022-09-02T04:42:24.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs=', 'Content-Length': '0'} 2022-09-02T04:42:24.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002150fbc0123caaf8-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.634 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.635 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.635 DEBUG:boto:path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.636 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.637 DEBUG:boto:Method: GET 2022-09-02T04:42:24.637 DEBUG:boto:Path: /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.638 DEBUG:boto:Data: 2022-09-02T04:42:24.639 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.640 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.640 DEBUG:boto:Params: {} 2022-09-02T04:42:24.641 DEBUG:boto:Token: None 2022-09-02T04:42:24.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=LjNxkl08kx14W4bqr69fYD4nB2y0HPM 2022-09-02T04:42:24.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs= 2022-09-02T04:42:24.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyjmqlUm5JRUBFy0iKuRpJj6Fjs=', 'Content-Length': '0'} 2022-09-02T04:42:24.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'LjNxkl08kx14W4bqr69fYD4nB2y0HPM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008b01b5520e4b021-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.686 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.687 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.688 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.688 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.689 DEBUG:boto:Method: GET 2022-09-02T04:42:24.689 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.690 DEBUG:boto:Data: 2022-09-02T04:42:24.690 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.691 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.692 DEBUG:boto:Params: {} 2022-09-02T04:42:24.693 DEBUG:boto:Token: None 2022-09-02T04:42:24.693 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7390627a4ae85a-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.698 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.699 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.700 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.706 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.706 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.707 DEBUG:boto:Method: GET 2022-09-02T04:42:24.707 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.708 DEBUG:boto:Data: 2022-09-02T04:42:24.708 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.710 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.710 DEBUG:boto:Params: {} 2022-09-02T04:42:24.711 DEBUG:boto:Token: None 2022-09-02T04:42:24.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f100fa0f2a5b21d-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.716 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.717 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.718 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.718 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.719 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.719 DEBUG:boto:Data: 2022-09-02T04:42:24.720 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.721 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.721 DEBUG:boto:Params: {} 2022-09-02T04:42:24.722 DEBUG:boto:Token: None 2022-09-02T04:42:24.723 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:24.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:24.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e8ee2b5f35318ca-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.729 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.730 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.730 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.731 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.731 DEBUG:boto:Data: 2022-09-02T04:42:24.732 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.733 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.734 DEBUG:boto:Params: {} 2022-09-02T04:42:24.734 DEBUG:boto:Token: None 2022-09-02T04:42:24.735 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:24.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:24.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b223a4063b3f814b-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.741 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.742 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.742 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.743 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.743 DEBUG:boto:Method: GET 2022-09-02T04:42:24.744 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.744 DEBUG:boto:Data: 2022-09-02T04:42:24.745 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.746 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.747 DEBUG:boto:Params: {} 2022-09-02T04:42:24.747 DEBUG:boto:Token: None 2022-09-02T04:42:24.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg= 2022-09-02T04:42:24.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg=', 'Content-Length': '0'} 2022-09-02T04:42:24.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f2ff1458c52f1e0-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.798 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.799 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.800 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.800 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.801 DEBUG:boto:Method: GET 2022-09-02T04:42:24.802 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.802 DEBUG:boto:Data: 2022-09-02T04:42:24.803 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.804 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.805 DEBUG:boto:Params: {} 2022-09-02T04:42:24.806 DEBUG:boto:Token: None 2022-09-02T04:42:24.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:24.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg= 2022-09-02T04:42:24.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSa43FK8WGFa2CMWt7rihQEsbsg=', 'Content-Length': '0'} 2022-09-02T04:42:24.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae1d7833690a1ad-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.813 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.814 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.815 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.816 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.816 DEBUG:boto:Method: GET 2022-09-02T04:42:24.817 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.818 DEBUG:boto:Data: 2022-09-02T04:42:24.818 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.820 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.820 DEBUG:boto:Params: {} 2022-09-02T04:42:24.821 DEBUG:boto:Token: None 2022-09-02T04:42:24.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2504f00a3a6dbbc-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.827 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.827 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.828 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.829 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.829 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.830 DEBUG:boto:Method: GET 2022-09-02T04:42:24.830 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.831 DEBUG:boto:Data: 2022-09-02T04:42:24.831 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.832 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.833 DEBUG:boto:Params: {} 2022-09-02T04:42:24.834 DEBUG:boto:Token: None 2022-09-02T04:42:24.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1518f7c7e28e68-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.839 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.841 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.841 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.842 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.842 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.843 DEBUG:boto:Method: GET 2022-09-02T04:42:24.843 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.844 DEBUG:boto:Data: 2022-09-02T04:42:24.845 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.846 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.846 DEBUG:boto:Params: {} 2022-09-02T04:42:24.847 DEBUG:boto:Token: None 2022-09-02T04:42:24.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.848 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g= 2022-09-02T04:42:24.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g=', 'Content-Length': '0'} 2022-09-02T04:42:24.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf39420413583423-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.852 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.853 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.853 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.854 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.854 DEBUG:boto:Method: GET 2022-09-02T04:42:24.855 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.855 DEBUG:boto:Data: 2022-09-02T04:42:24.856 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.857 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.858 DEBUG:boto:Params: {} 2022-09-02T04:42:24.858 DEBUG:boto:Token: None 2022-09-02T04:42:24.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g= 2022-09-02T04:42:24.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g=', 'Content-Length': '0'} 2022-09-02T04:42:24.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005303c37cec20d4e1-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.906 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.908 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.909 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.910 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.911 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.912 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.912 DEBUG:boto:Data: 2022-09-02T04:42:24.913 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.914 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.915 DEBUG:boto:Params: {} 2022-09-02T04:42:24.916 DEBUG:boto:Token: None 2022-09-02T04:42:24.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6+OB/LJuA+xsaPHzfwuBnOdoFSU= 2022-09-02T04:42:24.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6+OB/LJuA+xsaPHzfwuBnOdoFSU=', 'Content-Length': '0'} 2022-09-02T04:42:24.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a15f198dcf937e1d-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.922 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.923 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.923 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.924 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.924 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.925 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.925 DEBUG:boto:Data: 2022-09-02T04:42:24.926 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.927 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.927 DEBUG:boto:Params: {} 2022-09-02T04:42:24.928 DEBUG:boto:Token: None 2022-09-02T04:42:24.928 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6+OB/LJuA+xsaPHzfwuBnOdoFSU= 2022-09-02T04:42:24.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6+OB/LJuA+xsaPHzfwuBnOdoFSU=', 'Content-Length': '0'} 2022-09-02T04:42:24.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001265263593cac07b-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:24.933 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.933 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.934 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.935 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.935 DEBUG:boto:Method: GET 2022-09-02T04:42:24.936 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.936 DEBUG:boto:Data: 2022-09-02T04:42:24.937 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.938 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.938 DEBUG:boto:Params: {} 2022-09-02T04:42:24.939 DEBUG:boto:Token: None 2022-09-02T04:42:24.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g= 2022-09-02T04:42:24.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g=', 'Content-Length': '0'} 2022-09-02T04:42:24.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dde387d29249b015-00631189b0-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.944 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.944 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.945 DEBUG:boto:path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.946 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.947 DEBUG:boto:Method: GET 2022-09-02T04:42:24.948 DEBUG:boto:Path: /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.949 DEBUG:boto:Data: 2022-09-02T04:42:24.950 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.952 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.953 DEBUG:boto:Params: {} 2022-09-02T04:42:24.953 DEBUG:boto:Token: None 2022-09-02T04:42:24.954 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj?versionId=u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4 2022-09-02T04:42:24.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g= 2022-09-02T04:42:24.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvHdiJCH8/HZjniEeubaFPGpm3g=', 'Content-Length': '0'} 2022-09-02T04:42:24.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'u0TBmiuoo9kWSjCI8dtEPp.2cyghZt4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e85e1629508d2355-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.959 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.959 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.960 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.960 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.961 DEBUG:boto:Method: GET 2022-09-02T04:42:24.962 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.963 DEBUG:boto:Data: 2022-09-02T04:42:24.964 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.966 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.967 DEBUG:boto:Params: {} 2022-09-02T04:42:24.968 DEBUG:boto:Token: None 2022-09-02T04:42:24.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672aac3728779ef6-00631189b0-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.973 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.974 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:24.975 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:24.975 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.976 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:24.976 DEBUG:boto:Method: GET 2022-09-02T04:42:24.977 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:24.977 DEBUG:boto:Data: 2022-09-02T04:42:24.978 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:24.979 DEBUG:boto:Port: 8000 2022-09-02T04:42:24.980 DEBUG:boto:Params: {} 2022-09-02T04:42:24.980 DEBUG:boto:Token: None 2022-09-02T04:42:24.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/ 2022-09-02T04:42:24.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY= 2022-09-02T04:42:24.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ir35dYJhU26PLA2+AKZyhNHuFBY=', 'Content-Length': '0'} 2022-09-02T04:42:24.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3820e6c111d17f-00631189b0-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:24.987 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:24.988 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:24.989 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:24.990 DEBUG:boto:Method: HEAD 2022-09-02T04:42:24.991 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:24.991 DEBUG:boto:Data: 2022-09-02T04:42:24.992 DEBUG:boto:Headers: {} 2022-09-02T04:42:24.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:24.995 DEBUG:boto:Port: 8001 2022-09-02T04:42:24.996 DEBUG:boto:Params: {} 2022-09-02T04:42:24.997 DEBUG:boto:Token: None 2022-09-02T04:42:24.998 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:24 GMT /xljbkf-19/obj 2022-09-02T04:42:24.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc= 2022-09-02T04:42:25.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vJoR3wB1dbuuK10xg3P/KQXVPOc=', 'Content-Length': '0'} 2022-09-02T04:42:25.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000197eaa6c2fd324aa-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.005 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.006 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.006 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.007 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.007 DEBUG:boto:Data: 2022-09-02T04:42:25.008 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.009 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.010 DEBUG:boto:Params: {} 2022-09-02T04:42:25.010 DEBUG:boto:Token: None 2022-09-02T04:42:25.011 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083a1a61fc4be762a-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.017 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.017 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.018 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.019 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.019 DEBUG:boto:Method: GET 2022-09-02T04:42:25.020 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.020 DEBUG:boto:Data: 2022-09-02T04:42:25.021 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.022 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.023 DEBUG:boto:Params: {} 2022-09-02T04:42:25.024 DEBUG:boto:Token: None 2022-09-02T04:42:25.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051a799125dc81133-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.070 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.071 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.071 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.072 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.073 DEBUG:boto:Method: GET 2022-09-02T04:42:25.073 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.074 DEBUG:boto:Data: 2022-09-02T04:42:25.074 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.076 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.076 DEBUG:boto:Params: {} 2022-09-02T04:42:25.077 DEBUG:boto:Token: None 2022-09-02T04:42:25.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bad5dea9f2c627f9-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.082 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.082 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.083 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.083 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.084 DEBUG:boto:Method: GET 2022-09-02T04:42:25.085 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.085 DEBUG:boto:Data: 2022-09-02T04:42:25.086 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.087 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.087 DEBUG:boto:Params: {} 2022-09-02T04:42:25.088 DEBUG:boto:Token: None 2022-09-02T04:42:25.088 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cbe46ccad3cc3fc-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.093 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.094 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.094 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.095 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.095 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.096 DEBUG:boto:Method: GET 2022-09-02T04:42:25.096 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.097 DEBUG:boto:Data: 2022-09-02T04:42:25.097 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.098 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.099 DEBUG:boto:Params: {} 2022-09-02T04:42:25.100 DEBUG:boto:Token: None 2022-09-02T04:42:25.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000364d1643a4adb501-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.105 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.107 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.107 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.108 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.108 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.109 DEBUG:boto:Method: GET 2022-09-02T04:42:25.109 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.110 DEBUG:boto:Data: 2022-09-02T04:42:25.110 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.111 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.113 DEBUG:boto:Params: {} 2022-09-02T04:42:25.114 DEBUG:boto:Token: None 2022-09-02T04:42:25.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg= 2022-09-02T04:42:25.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg=', 'Content-Length': '0'} 2022-09-02T04:42:25.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ddaef6e69cf842f-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.119 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.120 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.120 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.121 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.122 DEBUG:boto:Method: GET 2022-09-02T04:42:25.122 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.123 DEBUG:boto:Data: 2022-09-02T04:42:25.123 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.125 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.125 DEBUG:boto:Params: {} 2022-09-02T04:42:25.126 DEBUG:boto:Token: None 2022-09-02T04:42:25.127 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg= 2022-09-02T04:42:25.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg=', 'Content-Length': '0'} 2022-09-02T04:42:25.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008524eabc60cc6274-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.132 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.133 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.133 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.134 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.134 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.135 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.136 DEBUG:boto:Data: 2022-09-02T04:42:25.136 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.137 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.138 DEBUG:boto:Params: {} 2022-09-02T04:42:25.138 DEBUG:boto:Token: None 2022-09-02T04:42:25.139 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:34dv3drgqnJXKyAiQLP6nefXlfM= 2022-09-02T04:42:25.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:34dv3drgqnJXKyAiQLP6nefXlfM=', 'Content-Length': '0'} 2022-09-02T04:42:25.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a71af531195ecd4-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.144 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.145 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.146 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.146 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.147 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.148 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.149 DEBUG:boto:Data: 2022-09-02T04:42:25.149 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.150 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.151 DEBUG:boto:Params: {} 2022-09-02T04:42:25.151 DEBUG:boto:Token: None 2022-09-02T04:42:25.152 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:34dv3drgqnJXKyAiQLP6nefXlfM= 2022-09-02T04:42:25.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:34dv3drgqnJXKyAiQLP6nefXlfM=', 'Content-Length': '0'} 2022-09-02T04:42:25.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009033d903b70be4f7-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.156 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.157 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.157 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.158 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.158 DEBUG:boto:Method: GET 2022-09-02T04:42:25.159 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.159 DEBUG:boto:Data: 2022-09-02T04:42:25.160 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.161 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.161 DEBUG:boto:Params: {} 2022-09-02T04:42:25.162 DEBUG:boto:Token: None 2022-09-02T04:42:25.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg= 2022-09-02T04:42:25.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg=', 'Content-Length': '0'} 2022-09-02T04:42:25.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f717f6beb034dfcd-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.210 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.211 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.212 DEBUG:boto:path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.212 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.213 DEBUG:boto:Method: GET 2022-09-02T04:42:25.213 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.214 DEBUG:boto:Data: 2022-09-02T04:42:25.215 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.216 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.217 DEBUG:boto:Params: {} 2022-09-02T04:42:25.218 DEBUG:boto:Token: None 2022-09-02T04:42:25.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ 2022-09-02T04:42:25.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg= 2022-09-02T04:42:25.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPq8FBLDNZIL2+0H2ues1k/1TGg=', 'Content-Length': '0'} 2022-09-02T04:42:25.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7ZDBr9FWnMFnRUcqmgJGTjvYGROoFlQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f13b0860f074280-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.226 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.226 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.227 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.227 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.228 DEBUG:boto:Method: GET 2022-09-02T04:42:25.228 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.229 DEBUG:boto:Data: 2022-09-02T04:42:25.230 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.231 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.231 DEBUG:boto:Params: {} 2022-09-02T04:42:25.232 DEBUG:boto:Token: None 2022-09-02T04:42:25.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce7516be987f72d3-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.237 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.238 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.239 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.239 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.240 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.240 DEBUG:boto:Method: GET 2022-09-02T04:42:25.241 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.241 DEBUG:boto:Data: 2022-09-02T04:42:25.242 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.243 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.243 DEBUG:boto:Params: {} 2022-09-02T04:42:25.244 DEBUG:boto:Token: None 2022-09-02T04:42:25.245 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.245 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5eba8fdcd9bed47-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.250 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.251 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.252 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.252 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.253 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.253 DEBUG:boto:Data: 2022-09-02T04:42:25.254 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.255 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.255 DEBUG:boto:Params: {} 2022-09-02T04:42:25.256 DEBUG:boto:Token: None 2022-09-02T04:42:25.256 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041fe8fd6d731a1ca-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.262 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.262 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.263 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.263 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.264 DEBUG:boto:Data: 2022-09-02T04:42:25.265 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.265 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.266 DEBUG:boto:Params: {} 2022-09-02T04:42:25.267 DEBUG:boto:Token: None 2022-09-02T04:42:25.267 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7a139e2f53da458-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.274 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.275 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.276 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.277 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.278 DEBUG:boto:Method: GET 2022-09-02T04:42:25.278 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.279 DEBUG:boto:Data: 2022-09-02T04:42:25.280 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.281 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.281 DEBUG:boto:Params: {} 2022-09-02T04:42:25.282 DEBUG:boto:Token: None 2022-09-02T04:42:25.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079f24e803e3837f9-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.287 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.287 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.288 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.288 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.289 DEBUG:boto:Method: GET 2022-09-02T04:42:25.289 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.290 DEBUG:boto:Data: 2022-09-02T04:42:25.291 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.292 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.292 DEBUG:boto:Params: {} 2022-09-02T04:42:25.293 DEBUG:boto:Token: None 2022-09-02T04:42:25.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063ce27cd57f716df-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.298 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.299 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.299 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.300 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.300 DEBUG:boto:Method: GET 2022-09-02T04:42:25.301 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.301 DEBUG:boto:Data: 2022-09-02T04:42:25.302 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.303 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.304 DEBUG:boto:Params: {} 2022-09-02T04:42:25.305 DEBUG:boto:Token: None 2022-09-02T04:42:25.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083088e26c1391c93-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.310 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.311 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.311 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.312 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.312 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.313 DEBUG:boto:Method: GET 2022-09-02T04:42:25.313 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.314 DEBUG:boto:Data: 2022-09-02T04:42:25.315 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.316 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.316 DEBUG:boto:Params: {} 2022-09-02T04:42:25.317 DEBUG:boto:Token: None 2022-09-02T04:42:25.317 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092781856cb2defee-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.323 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.324 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.325 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.326 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.327 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.327 DEBUG:boto:Method: GET 2022-09-02T04:42:25.328 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.329 DEBUG:boto:Data: 2022-09-02T04:42:25.329 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.331 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.332 DEBUG:boto:Params: {} 2022-09-02T04:42:25.332 DEBUG:boto:Token: None 2022-09-02T04:42:25.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4= 2022-09-02T04:42:25.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4=', 'Content-Length': '0'} 2022-09-02T04:42:25.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000600ce4945cb3798b-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.338 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.339 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.340 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.341 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.342 DEBUG:boto:Method: GET 2022-09-02T04:42:25.343 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.344 DEBUG:boto:Data: 2022-09-02T04:42:25.344 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.346 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.347 DEBUG:boto:Params: {} 2022-09-02T04:42:25.348 DEBUG:boto:Token: None 2022-09-02T04:42:25.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4= 2022-09-02T04:42:25.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4=', 'Content-Length': '0'} 2022-09-02T04:42:25.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ca7641df09fb04e-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.357 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.358 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.358 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.359 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.359 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.360 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.360 DEBUG:boto:Data: 2022-09-02T04:42:25.361 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.362 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.362 DEBUG:boto:Params: {} 2022-09-02T04:42:25.363 DEBUG:boto:Token: None 2022-09-02T04:42:25.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h2UPcCzp6luJ/Z1OcEf/Oj0Drs0= 2022-09-02T04:42:25.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h2UPcCzp6luJ/Z1OcEf/Oj0Drs0=', 'Content-Length': '0'} 2022-09-02T04:42:25.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046d9655f39b839ba-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.368 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.368 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.369 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.369 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.370 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.370 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.371 DEBUG:boto:Data: 2022-09-02T04:42:25.371 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.372 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.373 DEBUG:boto:Params: {} 2022-09-02T04:42:25.373 DEBUG:boto:Token: None 2022-09-02T04:42:25.374 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h2UPcCzp6luJ/Z1OcEf/Oj0Drs0= 2022-09-02T04:42:25.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h2UPcCzp6luJ/Z1OcEf/Oj0Drs0=', 'Content-Length': '0'} 2022-09-02T04:42:25.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e730590a91b3cfc-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.381 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.383 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.384 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.384 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.386 DEBUG:boto:Method: GET 2022-09-02T04:42:25.387 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.388 DEBUG:boto:Data: 2022-09-02T04:42:25.389 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.390 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.391 DEBUG:boto:Params: {} 2022-09-02T04:42:25.391 DEBUG:boto:Token: None 2022-09-02T04:42:25.392 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4= 2022-09-02T04:42:25.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4=', 'Content-Length': '0'} 2022-09-02T04:42:25.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e157794166590910-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.396 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.397 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.397 DEBUG:boto:path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.398 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.398 DEBUG:boto:Method: GET 2022-09-02T04:42:25.399 DEBUG:boto:Path: /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.399 DEBUG:boto:Data: 2022-09-02T04:42:25.400 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.401 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.401 DEBUG:boto:Params: {} 2022-09-02T04:42:25.402 DEBUG:boto:Token: None 2022-09-02T04:42:25.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm 2022-09-02T04:42:25.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4= 2022-09-02T04:42:25.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2ZsmkuBI3usvGmzVIhOa3+goRH4=', 'Content-Length': '0'} 2022-09-02T04:42:25.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', '7v56mxcaqqipIwpQ7G9zjnh3lbrVkDm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c67c2bb0ea6048d-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.456 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.458 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.459 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.460 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.461 DEBUG:boto:Method: GET 2022-09-02T04:42:25.461 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.463 DEBUG:boto:Data: 2022-09-02T04:42:25.464 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.466 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.467 DEBUG:boto:Params: {} 2022-09-02T04:42:25.468 DEBUG:boto:Token: None 2022-09-02T04:42:25.469 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81069ecdaf6539f-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.479 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.480 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.481 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.482 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.482 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.483 DEBUG:boto:Method: GET 2022-09-02T04:42:25.483 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.484 DEBUG:boto:Data: 2022-09-02T04:42:25.484 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.486 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.486 DEBUG:boto:Params: {} 2022-09-02T04:42:25.487 DEBUG:boto:Token: None 2022-09-02T04:42:25.488 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a005e2c0c4bebe79-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.494 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.495 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.496 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.497 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.498 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.499 DEBUG:boto:Data: 2022-09-02T04:42:25.500 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.501 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.502 DEBUG:boto:Params: {} 2022-09-02T04:42:25.503 DEBUG:boto:Token: None 2022-09-02T04:42:25.504 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053768a55c90f5b11-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.513 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.514 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.515 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.516 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.517 DEBUG:boto:Data: 2022-09-02T04:42:25.517 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.518 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.519 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.519 DEBUG:boto:Params: {} 2022-09-02T04:42:25.520 DEBUG:boto:Token: None 2022-09-02T04:42:25.521 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.522 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035e19a8d054d8e52-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.529 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.530 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.531 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.532 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.533 DEBUG:boto:Method: GET 2022-09-02T04:42:25.533 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.534 DEBUG:boto:Data: 2022-09-02T04:42:25.535 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.536 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.537 DEBUG:boto:Params: {} 2022-09-02T04:42:25.538 DEBUG:boto:Token: None 2022-09-02T04:42:25.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9051fd27ac7ab80-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.586 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.587 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.587 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.588 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.588 DEBUG:boto:Method: GET 2022-09-02T04:42:25.589 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.590 DEBUG:boto:Data: 2022-09-02T04:42:25.590 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.591 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.592 DEBUG:boto:Params: {} 2022-09-02T04:42:25.593 DEBUG:boto:Token: None 2022-09-02T04:42:25.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009057f0dcb9aed4d6-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.598 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.598 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.600 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.601 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.601 DEBUG:boto:Method: GET 2022-09-02T04:42:25.602 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.603 DEBUG:boto:Data: 2022-09-02T04:42:25.604 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.606 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.607 DEBUG:boto:Params: {} 2022-09-02T04:42:25.608 DEBUG:boto:Token: None 2022-09-02T04:42:25.609 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208261f91dd3619e-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.616 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.617 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.617 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.618 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.618 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.619 DEBUG:boto:Method: GET 2022-09-02T04:42:25.620 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.621 DEBUG:boto:Data: 2022-09-02T04:42:25.622 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.624 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.625 DEBUG:boto:Params: {} 2022-09-02T04:42:25.626 DEBUG:boto:Token: None 2022-09-02T04:42:25.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c6ee87d5919ae9-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.634 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.636 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.637 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.637 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.638 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.638 DEBUG:boto:Method: GET 2022-09-02T04:42:25.639 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.640 DEBUG:boto:Data: 2022-09-02T04:42:25.640 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.641 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.642 DEBUG:boto:Params: {} 2022-09-02T04:42:25.642 DEBUG:boto:Token: None 2022-09-02T04:42:25.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg= 2022-09-02T04:42:25.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg=', 'Content-Length': '0'} 2022-09-02T04:42:25.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000799726de7e96dcc7-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.648 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.649 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.649 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.650 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.651 DEBUG:boto:Method: GET 2022-09-02T04:42:25.652 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.653 DEBUG:boto:Data: 2022-09-02T04:42:25.654 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.656 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.657 DEBUG:boto:Params: {} 2022-09-02T04:42:25.658 DEBUG:boto:Token: None 2022-09-02T04:42:25.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg= 2022-09-02T04:42:25.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg=', 'Content-Length': '0'} 2022-09-02T04:42:25.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edc80ba799cddaf4-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.665 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.666 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.666 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.667 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.667 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.668 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.668 DEBUG:boto:Data: 2022-09-02T04:42:25.669 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.670 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.671 DEBUG:boto:Params: {} 2022-09-02T04:42:25.671 DEBUG:boto:Token: None 2022-09-02T04:42:25.672 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2WM+PnX60wo8bSIwFJbOp8oDJw= 2022-09-02T04:42:25.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2WM+PnX60wo8bSIwFJbOp8oDJw=', 'Content-Length': '0'} 2022-09-02T04:42:25.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff7e2ecfec71d054-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.677 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.677 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.678 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.679 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.680 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.680 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.681 DEBUG:boto:Data: 2022-09-02T04:42:25.681 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.683 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.683 DEBUG:boto:Params: {} 2022-09-02T04:42:25.685 DEBUG:boto:Token: None 2022-09-02T04:42:25.685 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2WM+PnX60wo8bSIwFJbOp8oDJw= 2022-09-02T04:42:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2WM+PnX60wo8bSIwFJbOp8oDJw=', 'Content-Length': '0'} 2022-09-02T04:42:25.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027df6d45db80d2c8-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.694 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.695 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.696 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.697 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.698 DEBUG:boto:Method: GET 2022-09-02T04:42:25.699 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.700 DEBUG:boto:Data: 2022-09-02T04:42:25.701 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.703 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.704 DEBUG:boto:Params: {} 2022-09-02T04:42:25.704 DEBUG:boto:Token: None 2022-09-02T04:42:25.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg= 2022-09-02T04:42:25.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg=', 'Content-Length': '0'} 2022-09-02T04:42:25.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e0c240ba5f55269-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.754 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.756 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.756 DEBUG:boto:path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.757 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.759 DEBUG:boto:Method: GET 2022-09-02T04:42:25.760 DEBUG:boto:Path: /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.761 DEBUG:boto:Data: 2022-09-02T04:42:25.761 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.763 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.764 DEBUG:boto:Params: {} 2022-09-02T04:42:25.765 DEBUG:boto:Token: None 2022-09-02T04:42:25.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz 2022-09-02T04:42:25.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg= 2022-09-02T04:42:25.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POPc2jt1agRE3+ejdSHiHtVq2fg=', 'Content-Length': '0'} 2022-09-02T04:42:25.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'FdBlPONqZDnEX45Z7KNq2vsHOx6AhIz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005655ccabdc8a8cdf-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.771 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.771 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.772 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.772 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.773 DEBUG:boto:Method: GET 2022-09-02T04:42:25.773 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.774 DEBUG:boto:Data: 2022-09-02T04:42:25.775 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.776 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.776 DEBUG:boto:Params: {} 2022-09-02T04:42:25.777 DEBUG:boto:Token: None 2022-09-02T04:42:25.777 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e351d4b2958dd6e-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.783 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.784 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.784 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.785 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.785 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.786 DEBUG:boto:Method: GET 2022-09-02T04:42:25.787 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.787 DEBUG:boto:Data: 2022-09-02T04:42:25.788 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.789 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.790 DEBUG:boto:Params: {} 2022-09-02T04:42:25.790 DEBUG:boto:Token: None 2022-09-02T04:42:25.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d58389d007acf4-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.796 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.797 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.798 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.799 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.800 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.801 DEBUG:boto:Data: 2022-09-02T04:42:25.802 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.804 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.804 DEBUG:boto:Params: {} 2022-09-02T04:42:25.806 DEBUG:boto:Token: None 2022-09-02T04:42:25.807 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b42d99b16aedda8-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.816 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.818 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.819 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.820 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:25.821 DEBUG:boto:Data: 2022-09-02T04:42:25.821 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.823 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.824 DEBUG:boto:Params: {} 2022-09-02T04:42:25.825 DEBUG:boto:Token: None 2022-09-02T04:42:25.826 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj 2022-09-02T04:42:25.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk= 2022-09-02T04:42:25.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQSZIQg2hfExS1baiqabBeko5Uk=', 'Content-Length': '0'} 2022-09-02T04:42:25.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ceb39459a4d6d7d-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.834 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.835 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.835 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.836 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.837 DEBUG:boto:Method: GET 2022-09-02T04:42:25.838 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.839 DEBUG:boto:Data: 2022-09-02T04:42:25.840 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.842 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.843 DEBUG:boto:Params: {} 2022-09-02T04:42:25.844 DEBUG:boto:Token: None 2022-09-02T04:42:25.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d792d367f2a2c1ac-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.852 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.853 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.854 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.855 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.856 DEBUG:boto:Method: GET 2022-09-02T04:42:25.857 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.859 DEBUG:boto:Data: 2022-09-02T04:42:25.860 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.862 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.863 DEBUG:boto:Params: {} 2022-09-02T04:42:25.864 DEBUG:boto:Token: None 2022-09-02T04:42:25.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:25.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU= 2022-09-02T04:42:25.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ln+c3HmiAPK9fgqPXM1oyFWImU=', 'Content-Length': '0'} 2022-09-02T04:42:25.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b76985aa03ef920-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.910 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.911 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.912 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.912 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.913 DEBUG:boto:Method: GET 2022-09-02T04:42:25.913 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.914 DEBUG:boto:Data: 2022-09-02T04:42:25.914 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.916 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.917 DEBUG:boto:Params: {} 2022-09-02T04:42:25.918 DEBUG:boto:Token: None 2022-09-02T04:42:25.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb8f5a9532d0e32-00631189b1-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.925 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.926 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:25.927 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:25.928 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.929 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:25.930 DEBUG:boto:Method: GET 2022-09-02T04:42:25.930 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:25.931 DEBUG:boto:Data: 2022-09-02T04:42:25.931 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.932 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.933 DEBUG:boto:Params: {} 2022-09-02T04:42:25.934 DEBUG:boto:Token: None 2022-09-02T04:42:25.935 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/ 2022-09-02T04:42:25.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk= 2022-09-02T04:42:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uTkMYaqTbNvh6KjoprNOcpxRBYk=', 'Content-Length': '0'} 2022-09-02T04:42:25.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560c1ad1913a53ea-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.940 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:25.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.943 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.944 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.945 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.946 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.947 DEBUG:boto:Method: GET 2022-09-02T04:42:25.947 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.948 DEBUG:boto:Data: 2022-09-02T04:42:25.949 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.951 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.951 DEBUG:boto:Params: {} 2022-09-02T04:42:25.952 DEBUG:boto:Token: None 2022-09-02T04:42:25.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dpder9YCAs8Wvphta2WAAmZTB1M= 2022-09-02T04:42:25.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dpder9YCAs8Wvphta2WAAmZTB1M=', 'Content-Length': '0'} 2022-09-02T04:42:25.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f5c5d85948f5f06-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.957 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.958 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.959 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.960 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.961 DEBUG:boto:Method: GET 2022-09-02T04:42:25.962 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.963 DEBUG:boto:Data: 2022-09-02T04:42:25.964 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.965 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.966 DEBUG:boto:Params: {} 2022-09-02T04:42:25.966 DEBUG:boto:Token: None 2022-09-02T04:42:25.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dpder9YCAs8Wvphta2WAAmZTB1M= 2022-09-02T04:42:25.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dpder9YCAs8Wvphta2WAAmZTB1M=', 'Content-Length': '0'} 2022-09-02T04:42:25.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b1ec60a163e0299-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.971 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.972 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.972 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.973 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.973 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.974 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.974 DEBUG:boto:Data: 2022-09-02T04:42:25.975 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:25.976 DEBUG:boto:Port: 8001 2022-09-02T04:42:25.977 DEBUG:boto:Params: {} 2022-09-02T04:42:25.977 DEBUG:boto:Token: None 2022-09-02T04:42:25.978 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nUmFTpwuvkaiBafTMgpJDflSB/o= 2022-09-02T04:42:25.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nUmFTpwuvkaiBafTMgpJDflSB/o=', 'Content-Length': '0'} 2022-09-02T04:42:25.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b56e534de687c5d6-00631189b1-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.983 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.984 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.984 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.985 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.986 DEBUG:boto:Method: HEAD 2022-09-02T04:42:25.986 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.987 DEBUG:boto:Data: 2022-09-02T04:42:25.987 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.988 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:25.988 DEBUG:boto:Port: 8000 2022-09-02T04:42:25.989 DEBUG:boto:Params: {} 2022-09-02T04:42:25.990 DEBUG:boto:Token: None 2022-09-02T04:42:25.990 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:25 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nUmFTpwuvkaiBafTMgpJDflSB/o= 2022-09-02T04:42:25.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nUmFTpwuvkaiBafTMgpJDflSB/o=', 'Content-Length': '0'} 2022-09-02T04:42:25.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5d632e9abfb3d59-00631189b1-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:25.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:25.995 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:25.995 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:25.996 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.997 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.997 DEBUG:boto:Method: GET 2022-09-02T04:42:25.998 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:25.998 DEBUG:boto:Data: 2022-09-02T04:42:25.999 DEBUG:boto:Headers: {} 2022-09-02T04:42:25.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.000 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.000 DEBUG:boto:Params: {} 2022-09-02T04:42:26.001 DEBUG:boto:Token: None 2022-09-02T04:42:26.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:26.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kw1gRqKEYqBMSzQ5Fx9D3rkKVRQ= 2022-09-02T04:42:26.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kw1gRqKEYqBMSzQ5Fx9D3rkKVRQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e38650c8002d6d54-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.046 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.047 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.048 DEBUG:boto:path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:26.048 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:26.049 DEBUG:boto:Method: GET 2022-09-02T04:42:26.049 DEBUG:boto:Path: /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:26.050 DEBUG:boto:Data: 2022-09-02T04:42:26.050 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.052 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.052 DEBUG:boto:Params: {} 2022-09-02T04:42:26.053 DEBUG:boto:Token: None 2022-09-02T04:42:26.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj 2022-09-02T04:42:26.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kw1gRqKEYqBMSzQ5Fx9D3rkKVRQ= 2022-09-02T04:42:26.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kw1gRqKEYqBMSzQ5Fx9D3rkKVRQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'Doe5jdwZtcMR7HzAlJy83bxJ3p.mKxj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6bf1b2efe2aa768-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.059 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.060 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.060 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.061 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.061 DEBUG:boto:Method: GET 2022-09-02T04:42:26.062 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.063 DEBUG:boto:Data: 2022-09-02T04:42:26.063 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.064 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.065 DEBUG:boto:Params: {} 2022-09-02T04:42:26.065 DEBUG:boto:Token: None 2022-09-02T04:42:26.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056c9b9c0d05ed468-00631189b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.072 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.072 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.073 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.074 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.074 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.075 DEBUG:boto:Method: GET 2022-09-02T04:42:26.075 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.076 DEBUG:boto:Data: 2022-09-02T04:42:26.077 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.078 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.079 DEBUG:boto:Params: {} 2022-09-02T04:42:26.079 DEBUG:boto:Token: None 2022-09-02T04:42:26.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6e624e6036c1b76-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.085 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.086 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.087 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.087 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.088 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:26.088 DEBUG:boto:Data: 2022-09-02T04:42:26.089 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.090 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.090 DEBUG:boto:Params: {} 2022-09-02T04:42:26.091 DEBUG:boto:Token: None 2022-09-02T04:42:26.091 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj 2022-09-02T04:42:26.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ= 2022-09-02T04:42:26.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd41e22e6bcd02da-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.097 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.098 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.098 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.099 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:26.099 DEBUG:boto:Data: 2022-09-02T04:42:26.100 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.101 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.101 DEBUG:boto:Params: {} 2022-09-02T04:42:26.102 DEBUG:boto:Token: None 2022-09-02T04:42:26.102 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj 2022-09-02T04:42:26.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ= 2022-09-02T04:42:26.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe20aa7dacb3993a-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.109 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.110 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.110 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.111 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.111 DEBUG:boto:Method: GET 2022-09-02T04:42:26.112 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.112 DEBUG:boto:Data: 2022-09-02T04:42:26.113 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.114 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.115 DEBUG:boto:Params: {} 2022-09-02T04:42:26.115 DEBUG:boto:Token: None 2022-09-02T04:42:26.116 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A= 2022-09-02T04:42:26.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A=', 'Content-Length': '0'} 2022-09-02T04:42:26.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000738b5176b0f6ee83-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.162 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.163 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.164 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.164 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.165 DEBUG:boto:Method: GET 2022-09-02T04:42:26.165 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.166 DEBUG:boto:Data: 2022-09-02T04:42:26.167 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.168 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.168 DEBUG:boto:Params: {} 2022-09-02T04:42:26.169 DEBUG:boto:Token: None 2022-09-02T04:42:26.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A= 2022-09-02T04:42:26.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A=', 'Content-Length': '0'} 2022-09-02T04:42:26.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004c9f52a3ad5a2b3-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.174 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.175 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.175 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.176 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.176 DEBUG:boto:Method: GET 2022-09-02T04:42:26.177 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.177 DEBUG:boto:Data: 2022-09-02T04:42:26.178 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.179 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.180 DEBUG:boto:Params: {} 2022-09-02T04:42:26.180 DEBUG:boto:Token: None 2022-09-02T04:42:26.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d894885e9cbb3575-00631189b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.186 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.186 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.187 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.187 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.188 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.188 DEBUG:boto:Method: GET 2022-09-02T04:42:26.189 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.189 DEBUG:boto:Data: 2022-09-02T04:42:26.190 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.191 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.191 DEBUG:boto:Params: {} 2022-09-02T04:42:26.192 DEBUG:boto:Token: None 2022-09-02T04:42:26.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.193 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3fbed83f17babf-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.197 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.198 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.199 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.199 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.200 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.201 DEBUG:boto:Method: GET 2022-09-02T04:42:26.201 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.202 DEBUG:boto:Data: 2022-09-02T04:42:26.202 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.203 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.204 DEBUG:boto:Params: {} 2022-09-02T04:42:26.204 DEBUG:boto:Token: None 2022-09-02T04:42:26.205 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE= 2022-09-02T04:42:26.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE=', 'Content-Length': '0'} 2022-09-02T04:42:26.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016fed4040c27ae64-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.210 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.210 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.211 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.211 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.212 DEBUG:boto:Method: GET 2022-09-02T04:42:26.212 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.213 DEBUG:boto:Data: 2022-09-02T04:42:26.213 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.215 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.215 DEBUG:boto:Params: {} 2022-09-02T04:42:26.216 DEBUG:boto:Token: None 2022-09-02T04:42:26.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE= 2022-09-02T04:42:26.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE=', 'Content-Length': '0'} 2022-09-02T04:42:26.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005965a1785902b8a0-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.222 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.222 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.223 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.223 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.224 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.224 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.225 DEBUG:boto:Data: 2022-09-02T04:42:26.226 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.227 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.228 DEBUG:boto:Params: {} 2022-09-02T04:42:26.228 DEBUG:boto:Token: None 2022-09-02T04:42:26.229 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8j6TDAnhyfxViAG8MkwaI7Gm+Bc= 2022-09-02T04:42:26.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8j6TDAnhyfxViAG8MkwaI7Gm+Bc=', 'Content-Length': '0'} 2022-09-02T04:42:26.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e80a79706f3072c-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.233 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.234 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.234 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.235 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.235 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.236 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.236 DEBUG:boto:Data: 2022-09-02T04:42:26.237 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.238 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.239 DEBUG:boto:Params: {} 2022-09-02T04:42:26.239 DEBUG:boto:Token: None 2022-09-02T04:42:26.240 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8j6TDAnhyfxViAG8MkwaI7Gm+Bc= 2022-09-02T04:42:26.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8j6TDAnhyfxViAG8MkwaI7Gm+Bc=', 'Content-Length': '0'} 2022-09-02T04:42:26.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de2087275a93bc40-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.244 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.245 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.245 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.246 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.246 DEBUG:boto:Method: GET 2022-09-02T04:42:26.247 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.247 DEBUG:boto:Data: 2022-09-02T04:42:26.248 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.249 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.249 DEBUG:boto:Params: {} 2022-09-02T04:42:26.250 DEBUG:boto:Token: None 2022-09-02T04:42:26.251 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE= 2022-09-02T04:42:26.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE=', 'Content-Length': '0'} 2022-09-02T04:42:26.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c8146ee64abf1ec-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.255 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.256 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.256 DEBUG:boto:path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.257 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.257 DEBUG:boto:Method: GET 2022-09-02T04:42:26.258 DEBUG:boto:Path: /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.259 DEBUG:boto:Data: 2022-09-02T04:42:26.259 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.260 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.261 DEBUG:boto:Params: {} 2022-09-02T04:42:26.262 DEBUG:boto:Token: None 2022-09-02T04:42:26.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=crGGeHMhO34Ab7pKILSilubA2BPH4WT 2022-09-02T04:42:26.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE= 2022-09-02T04:42:26.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSeBPoxolWW+avpgSxx/dnDQ8hE=', 'Content-Length': '0'} 2022-09-02T04:42:26.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 'crGGeHMhO34Ab7pKILSilubA2BPH4WT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051c2c600cda119c0-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.272 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.272 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.273 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.273 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.274 DEBUG:boto:Method: GET 2022-09-02T04:42:26.274 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.275 DEBUG:boto:Data: 2022-09-02T04:42:26.276 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.277 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.278 DEBUG:boto:Params: {} 2022-09-02T04:42:26.278 DEBUG:boto:Token: None 2022-09-02T04:42:26.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed9deaaa7eeaf62-00631189b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.285 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.285 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.286 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.286 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.287 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.287 DEBUG:boto:Method: GET 2022-09-02T04:42:26.288 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.289 DEBUG:boto:Data: 2022-09-02T04:42:26.290 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.291 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.292 DEBUG:boto:Params: {} 2022-09-02T04:42:26.292 DEBUG:boto:Token: None 2022-09-02T04:42:26.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e14e2dbe2208819-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.304 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.305 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.306 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.306 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.307 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:26.307 DEBUG:boto:Data: 2022-09-02T04:42:26.308 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.309 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.309 DEBUG:boto:Params: {} 2022-09-02T04:42:26.310 DEBUG:boto:Token: None 2022-09-02T04:42:26.311 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj 2022-09-02T04:42:26.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ= 2022-09-02T04:42:26.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcb567f860a80cae-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.316 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.317 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.318 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.318 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:26.319 DEBUG:boto:Data: 2022-09-02T04:42:26.319 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.320 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.321 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.321 DEBUG:boto:Params: {} 2022-09-02T04:42:26.322 DEBUG:boto:Token: None 2022-09-02T04:42:26.323 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj 2022-09-02T04:42:26.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ= 2022-09-02T04:42:26.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0b46585241bce02-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.331 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.332 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.333 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.333 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.335 DEBUG:boto:Method: GET 2022-09-02T04:42:26.336 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.337 DEBUG:boto:Data: 2022-09-02T04:42:26.337 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.339 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.340 DEBUG:boto:Params: {} 2022-09-02T04:42:26.341 DEBUG:boto:Token: None 2022-09-02T04:42:26.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A= 2022-09-02T04:42:26.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A=', 'Content-Length': '0'} 2022-09-02T04:42:26.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003734d3bd7d9fe933-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.394 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.396 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.397 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.398 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.399 DEBUG:boto:Method: GET 2022-09-02T04:42:26.401 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.402 DEBUG:boto:Data: 2022-09-02T04:42:26.403 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.404 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.405 DEBUG:boto:Params: {} 2022-09-02T04:42:26.406 DEBUG:boto:Token: None 2022-09-02T04:42:26.407 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A= 2022-09-02T04:42:26.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A=', 'Content-Length': '0'} 2022-09-02T04:42:26.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000475542153f7728d5-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.412 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.413 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.413 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.414 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.415 DEBUG:boto:Method: GET 2022-09-02T04:42:26.415 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.416 DEBUG:boto:Data: 2022-09-02T04:42:26.417 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.418 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.419 DEBUG:boto:Params: {} 2022-09-02T04:42:26.420 DEBUG:boto:Token: None 2022-09-02T04:42:26.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd07ed7c52efab5-00631189b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.426 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.427 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.428 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.428 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.429 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.429 DEBUG:boto:Method: GET 2022-09-02T04:42:26.430 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.430 DEBUG:boto:Data: 2022-09-02T04:42:26.431 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.432 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.432 DEBUG:boto:Params: {} 2022-09-02T04:42:26.433 DEBUG:boto:Token: None 2022-09-02T04:42:26.434 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d106d10c68f53a02-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.438 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.440 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.440 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.441 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.441 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.442 DEBUG:boto:Method: GET 2022-09-02T04:42:26.443 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.443 DEBUG:boto:Data: 2022-09-02T04:42:26.444 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.445 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.445 DEBUG:boto:Params: {} 2022-09-02T04:42:26.446 DEBUG:boto:Token: None 2022-09-02T04:42:26.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw= 2022-09-02T04:42:26.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw=', 'Content-Length': '0'} 2022-09-02T04:42:26.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012abd2b7d27f65fd-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.451 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.451 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.452 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.452 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.453 DEBUG:boto:Method: GET 2022-09-02T04:42:26.454 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.454 DEBUG:boto:Data: 2022-09-02T04:42:26.455 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.455 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.456 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.456 DEBUG:boto:Params: {} 2022-09-02T04:42:26.457 DEBUG:boto:Token: None 2022-09-02T04:42:26.458 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw= 2022-09-02T04:42:26.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw=', 'Content-Length': '0'} 2022-09-02T04:42:26.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e6de9df2a736a61-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.462 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.462 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.463 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.463 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.464 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.464 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.465 DEBUG:boto:Data: 2022-09-02T04:42:26.465 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.466 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.467 DEBUG:boto:Params: {} 2022-09-02T04:42:26.467 DEBUG:boto:Token: None 2022-09-02T04:42:26.468 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jg+hXLAsH5E3Xokhldz+HlEaypk= 2022-09-02T04:42:26.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jg+hXLAsH5E3Xokhldz+HlEaypk=', 'Content-Length': '0'} 2022-09-02T04:42:26.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029474c21ea921577-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.472 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.473 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.473 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.474 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.474 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.475 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.475 DEBUG:boto:Data: 2022-09-02T04:42:26.476 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.477 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.477 DEBUG:boto:Params: {} 2022-09-02T04:42:26.478 DEBUG:boto:Token: None 2022-09-02T04:42:26.479 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jg+hXLAsH5E3Xokhldz+HlEaypk= 2022-09-02T04:42:26.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jg+hXLAsH5E3Xokhldz+HlEaypk=', 'Content-Length': '0'} 2022-09-02T04:42:26.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a4ac2757b200bbc-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.483 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.484 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.484 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.485 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.485 DEBUG:boto:Method: GET 2022-09-02T04:42:26.486 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.486 DEBUG:boto:Data: 2022-09-02T04:42:26.487 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.488 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.489 DEBUG:boto:Params: {} 2022-09-02T04:42:26.489 DEBUG:boto:Token: None 2022-09-02T04:42:26.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw= 2022-09-02T04:42:26.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw=', 'Content-Length': '0'} 2022-09-02T04:42:26.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8a35d9d78247933-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.494 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.495 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.495 DEBUG:boto:path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.496 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.496 DEBUG:boto:Method: GET 2022-09-02T04:42:26.497 DEBUG:boto:Path: /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.498 DEBUG:boto:Data: 2022-09-02T04:42:26.498 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.499 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.500 DEBUG:boto:Params: {} 2022-09-02T04:42:26.500 DEBUG:boto:Token: None 2022-09-02T04:42:26.501 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=t9jRBTjPWPur8r4C2pn1Lomi1F28VA0 2022-09-02T04:42:26.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw= 2022-09-02T04:42:26.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUQhwyJB/TQsAnlSEID98M9qaKw=', 'Content-Length': '0'} 2022-09-02T04:42:26.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:41 GMT'), ('x-amz-version-id', 't9jRBTjPWPur8r4C2pn1Lomi1F28VA0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060f1f5a1d6a450dd-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.505 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.506 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.506 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.507 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.507 DEBUG:boto:Method: GET 2022-09-02T04:42:26.508 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.509 DEBUG:boto:Data: 2022-09-02T04:42:26.509 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.510 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.511 DEBUG:boto:Params: {} 2022-09-02T04:42:26.511 DEBUG:boto:Token: None 2022-09-02T04:42:26.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c22f524a5a40d5-00631189b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.517 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.518 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.518 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.519 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.520 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.520 DEBUG:boto:Method: GET 2022-09-02T04:42:26.521 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.521 DEBUG:boto:Data: 2022-09-02T04:42:26.522 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.523 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.524 DEBUG:boto:Params: {} 2022-09-02T04:42:26.525 DEBUG:boto:Token: None 2022-09-02T04:42:26.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f33692973e2ad1-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.530 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.531 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.532 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.532 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.533 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:26.533 DEBUG:boto:Data: 2022-09-02T04:42:26.534 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.535 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.536 DEBUG:boto:Params: {} 2022-09-02T04:42:26.536 DEBUG:boto:Token: None 2022-09-02T04:42:26.537 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj 2022-09-02T04:42:26.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ= 2022-09-02T04:42:26.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000481c32a9c614b20e-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.544 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.544 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.545 DEBUG:boto:Method: HEAD 2022-09-02T04:42:26.546 DEBUG:boto:Path: /xljbkf-19/obj 2022-09-02T04:42:26.546 DEBUG:boto:Data: 2022-09-02T04:42:26.547 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.549 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.549 DEBUG:boto:Params: {} 2022-09-02T04:42:26.550 DEBUG:boto:Token: None 2022-09-02T04:42:26.551 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj 2022-09-02T04:42:26.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ= 2022-09-02T04:42:26.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I+qHWsku1hF4Ouq24EdAo2sKmLQ=', 'Content-Length': '0'} 2022-09-02T04:42:26.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017ade081d225c584-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:42:26.564 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.565 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.566 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.566 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.567 DEBUG:boto:Method: GET 2022-09-02T04:42:26.567 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.568 DEBUG:boto:Data: 2022-09-02T04:42:26.568 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.569 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.570 DEBUG:boto:Params: {} 2022-09-02T04:42:26.571 DEBUG:boto:Token: None 2022-09-02T04:42:26.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A= 2022-09-02T04:42:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A=', 'Content-Length': '0'} 2022-09-02T04:42:26.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bb253c6e26bbca4-00631189b2-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.622 DEBUG:boto:path=/xljbkf-19/obj 2022-09-02T04:42:26.623 DEBUG:boto:auth_path=/xljbkf-19/obj 2022-09-02T04:42:26.623 DEBUG:boto:path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.624 DEBUG:boto:auth_path=/xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.624 DEBUG:boto:Method: GET 2022-09-02T04:42:26.625 DEBUG:boto:Path: /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.625 DEBUG:boto:Data: 2022-09-02T04:42:26.626 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.627 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.628 DEBUG:boto:Params: {} 2022-09-02T04:42:26.635 DEBUG:boto:Token: None 2022-09-02T04:42:26.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/obj?versionId=36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz 2022-09-02T04:42:26.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A= 2022-09-02T04:42:26.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uFKPX1BMnlbnnlQufVAXxr8FU0A=', 'Content-Length': '0'} 2022-09-02T04:42:26.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:35:42 GMT'), ('x-amz-version-id', '36t6VhJtxX.ZyMlHbv8JhmzmPGxxpGz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eac9a2fc8cb41574-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.643 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.643 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.645 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.645 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.646 DEBUG:boto:Method: GET 2022-09-02T04:42:26.647 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.647 DEBUG:boto:Data: 2022-09-02T04:42:26.648 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:42:26.649 DEBUG:boto:Port: 8001 2022-09-02T04:42:26.650 DEBUG:boto:Params: {} 2022-09-02T04:42:26.651 DEBUG:boto:Token: None 2022-09-02T04:42:26.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cff79f21d42c42e-00631189b2-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.657 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.658 DEBUG:boto:path=/xljbkf-19/ 2022-09-02T04:42:26.658 DEBUG:boto:auth_path=/xljbkf-19/ 2022-09-02T04:42:26.659 DEBUG:boto:path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.659 DEBUG:boto:auth_path=/xljbkf-19/?prefix=obj 2022-09-02T04:42:26.660 DEBUG:boto:Method: GET 2022-09-02T04:42:26.661 DEBUG:boto:Path: /xljbkf-19/?prefix=obj 2022-09-02T04:42:26.662 DEBUG:boto:Data: 2022-09-02T04:42:26.662 DEBUG:boto:Headers: {} 2022-09-02T04:42:26.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:42:26.663 DEBUG:boto:Port: 8000 2022-09-02T04:42:26.664 DEBUG:boto:Params: {} 2022-09-02T04:42:26.665 DEBUG:boto:Token: None 2022-09-02T04:42:26.666 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:42:26 GMT /xljbkf-19/ 2022-09-02T04:42:26.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8= 2022-09-02T04:42:26.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/LSc1vOHYXSunBf1110YM8Oz4C8=', 'Content-Length': '0'} 2022-09-02T04:42:26.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec88f56a7530350b-00631189b2-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:42:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:26.672 DEBUG:boto:b'xljbkf-19obj1000falseobj2022-09-02T04:35:42.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:42:26.673 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-19 zones={test-zone3, test-zone2} 2022-09-02T04:42:26.674 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-02T04:42:26.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:26.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:26.676 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:26.794 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:42:26.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:26.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:42:26.795 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:42:26.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:26.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:26.796 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001781", 2022-09-02T04:42:26.796 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:38:19.327262Z" 2022-09-02T04:42:26.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:26.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:26.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:42:26.798 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:42:26.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:26.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:26.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:26.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:42:26.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:26.799 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:42:26.800 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:26.801 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-02T04:42:26.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:26.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:26.803 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:26.865 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:26.865 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:26.909 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:42:26.910 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:42:26.910 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:42:26.910 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:42:26.911 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:42:26.911 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 11009958948854697431 2022-09-02T04:42:26.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:26.911 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:42:26.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:26.912 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:42:26.912 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:26.913 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:26.913 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:42:26.913 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:26.914 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:26.914 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:26.914 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:42:26.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:26.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:26.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:26.915 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:42:26.916 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:26.916 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:26.916 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001781", 2022-09-02T04:42:26.917 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:26.917 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:26.918 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:26.918 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:38:19.327262Z" 2022-09-02T04:42:26.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:26.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:26.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:26.920 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:42:26.920 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:26.921 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:26.921 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:42:26.921 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:26.922 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:26.922 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:26.923 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:42:26.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:26.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:26.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:26.924 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:42:26.925 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:26.925 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:26.926 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:26.926 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:26.926 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:26.927 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:26.927 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000" 2022-09-02T04:42:26.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:26.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:26.928 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:42:26.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:26.929 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:42:26.930 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:42:26.930 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:42:26.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:26.931 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:42:26Z" 2022-09-02T04:42:26.931 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:42:26.932 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11009958948854697431 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001781", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:38:19.327262Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:26Z" } 2022-09-02T04:42:26.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:38:19.327262Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:26.934 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:26.934 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:26.935 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-02T04:42:26.935 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-02T04:42:26.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:26.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:26.937 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:26.987 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:26.987 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.055 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:42:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:42:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:42:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001781", 2022-09-02T04:42:27.058 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:38:19.327262Z" 2022-09-02T04:42:27.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:42:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:42:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:27.060 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:42:27.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.060 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:42:27.061 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:27.062 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-02T04:42:27.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.064 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.165 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:42:27.165 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:42:27.166 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:42:27.166 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:42:27.167 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:42:27.168 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1295418732353758134 2022-09-02T04:42:27.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.168 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:42:27.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:42:27.169 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.170 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:42:27.170 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.170 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.170 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.171 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:42:27.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:42:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.172 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001781", 2022-09-02T04:42:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.173 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:38:19.327262Z" 2022-09-02T04:42:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:42:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.175 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:42:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.176 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:42:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:42:27.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:27.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2022-09-02T04:42:27.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.180 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:42:27.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.181 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:42:27.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:42:27.181 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:42:27.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.182 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:42:27Z" 2022-09-02T04:42:27.182 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:42:27.183 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1295418732353758134 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001781", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:38:19.327262Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:27Z" } 2022-09-02T04:42:27.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:38:19.327262Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:27.185 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:27.185 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:27.186 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-02T04:42:27.186 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-02T04:42:27.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.188 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.281 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:42:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:42:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:42:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001781", 2022-09-02T04:42:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:38:19.327262Z" 2022-09-02T04:42:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:42:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:42:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:42:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.286 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:42:27.287 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:27.288 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-02T04:42:27.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.290 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.349 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.350 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.391 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:42:27.391 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:42:27.392 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:42:27.392 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:42:27.392 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:42:27.392 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 2207523797324220113 2022-09-02T04:42:27.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.393 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:42:27.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.395 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:42:27.395 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:27.396 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.396 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:42:27.397 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:27.397 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:27.398 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:27.398 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:42:27.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.400 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:42:27.400 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:27.400 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.401 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001781", 2022-09-02T04:42:27.401 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:27.401 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:27.401 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:27.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:38:19.327262Z" 2022-09-02T04:42:27.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.403 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:42:27.403 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:27.404 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.404 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:42:27.404 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:27.405 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:27.405 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:27.405 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:42:27.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.406 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:42:27.406 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:27.407 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.407 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:27.407 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:27.407 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:27.408 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:27.408 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000" 2022-09-02T04:42:27.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.409 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:42:27.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.409 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:42:27.409 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:42:27.410 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:42:27.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.410 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:42:27Z" 2022-09-02T04:42:27.410 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:42:27.412 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2207523797324220113 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001781", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:38:19.327262Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:27Z" } 2022-09-02T04:42:27.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001781', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:38:19.327262Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:27.413 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:27.413 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000001781', 2: '00000000000000000000:00000000000000006152', 3: ''} 2022-09-02T04:42:27.414 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-02T04:42:27.414 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-02T04:42:27.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.416 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.466 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.466 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.513 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:42:27.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.514 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:27.514 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:42:27.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.515 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002204", 2022-09-02T04:42:27.515 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:41:59.256694Z" 2022-09-02T04:42:27.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.516 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:42:27.516 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:42:27.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.517 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:27.517 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2022-09-02T04:42:27.517 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.518 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:42:27.519 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000002204', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-02T04:42:27.520 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-02T04:42:27.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.521 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.574 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.628 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:42:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:42:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:42:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:42:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:42:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T04:42:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:42:27.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.632 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:42:27.633 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.633 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.634 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:27.634 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.636 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:42:27.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.638 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:42:27.638 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.639 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002204", 2022-09-02T04:42:27.640 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.642 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:41:59.256694Z" 2022-09-02T04:42:27.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:42:27.645 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.645 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:42:27.647 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.647 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:42:27.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.651 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:42:27.651 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.652 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.652 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:27.653 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.653 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.655 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2022-09-02T04:42:27.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.658 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:42:27.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.659 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:42:27.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:42:27.661 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:42:27.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.662 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:42:27Z" 2022-09-02T04:42:27.663 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:42:27.665 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:41:59.256694Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:40.064266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:27Z" } 2022-09-02T04:42:27.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:41:59.256694Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:40.064266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:27.667 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000002204', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-02T04:42:27.668 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000002204', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-02T04:42:27.669 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-02T04:42:27.670 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-02T04:42:27.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.674 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.724 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.784 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:42:27.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.785 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:27.786 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:42:27.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.787 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002204", 2022-09-02T04:42:27.788 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:41:59.256694Z" 2022-09-02T04:42:27.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.789 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:42:27.790 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:42:27.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:27.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:27.791 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:27.792 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2022-09-02T04:42:27.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:27.793 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:42:27.795 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000002204', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-02T04:42:27.796 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-02T04:42:27.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:27.920 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:42:27.921 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:42:27.921 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:42:27.921 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:42:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:42:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1900644279603854970 2022-09-02T04:42:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:42:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:42:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:42:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:42:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002204", 2022-09-02T04:42:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:41:59.256694Z" 2022-09-02T04:42:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:42:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:42:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:42:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:42:27.933 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:27.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:27.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:27.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:27.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:27.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:27.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2022-09-02T04:42:27.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:27.935 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:42:27.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.936 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:42:27.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:42:27.936 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:42:27.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:27.937 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:42:27Z" 2022-09-02T04:42:27.937 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:42:27.938 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1900644279603854970 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002204", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:41:59.256694Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:40.064266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:27Z" } 2022-09-02T04:42:27.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002204', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:41:59.256694Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:40.064266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:27.940 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000002204', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-02T04:42:27.940 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000002204', 2: '00000000000000000000:00000000000000006434', 3: ''} 2022-09-02T04:42:27.940 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-02T04:42:27.941 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-02T04:42:27.942 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-02T04:42:27.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:27.944 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:27.993 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:27.994 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:28.047 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:42:28.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.047 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:28.048 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:42:28.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-02T04:42:28.049 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:37:02.975340Z" 2022-09-02T04:42:28.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:42:28.050 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:42:28.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:28.052 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:42:28.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.053 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:42:28.055 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005588', 3: ''} 2022-09-02T04:42:28.056 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-02T04:42:28.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.060 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:28.112 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:28.112 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:28.164 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:42:28.164 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:42:28.164 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:42:28.165 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:42:28.165 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:42:28.165 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10070206082575430316 2022-09-02T04:42:28.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.166 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:42:28.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.167 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:42:28.167 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:28.167 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.167 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:28.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:28.168 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:28.168 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:28.168 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:42:28.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:42:28.170 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:28.170 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.170 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-02T04:42:28.170 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:28.171 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:28.171 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:28.171 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:37:02.975340Z" 2022-09-02T04:42:28.172 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:42:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:42:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:28.176 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:42:28.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.177 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:42:28.177 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:28.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.177 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:28.178 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:28.178 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:28.178 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:28.178 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2022-09-02T04:42:28.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.179 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:42:28.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.180 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:42:28.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:42:28.180 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:42:28.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.181 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:42:28Z" 2022-09-02T04:42:28.181 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:42:28.183 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10070206082575430316 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:20.360817Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:37:02.975340Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:33:00.969288Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:28Z" } 2022-09-02T04:42:28.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:20.360817Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:37:02.975340Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:33:00.969288Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:28.184 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005588', 3: ''} 2022-09-02T04:42:28.185 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005588', 3: ''} 2022-09-02T04:42:28.185 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-02T04:42:28.186 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-02T04:42:28.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.188 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:28.236 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:28.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:28.323 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:42:28.323 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.324 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:28.324 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:42:28.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-02T04:42:28.325 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:37:02.975340Z" 2022-09-02T04:42:28.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.326 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:42:28.326 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:42:28.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:28.327 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:42:28.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.328 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:42:28.329 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005588', 3: ''} 2022-09-02T04:42:28.330 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-02T04:42:28.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.332 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:28.385 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:28.386 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:28.435 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:42:28.435 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:42:28.436 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:42:28.436 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:42:28.436 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:42:28.436 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 5969670707592890178 2022-09-02T04:42:28.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.437 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:42:28.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.438 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:42:28.438 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.438 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.439 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:42:28.439 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.440 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.440 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.440 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:42:28.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.441 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:42:28.441 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.442 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.442 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000001499", 2022-09-02T04:42:28.443 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.444 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.444 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.445 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:37:02.975340Z" 2022-09-02T04:42:28.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.446 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:42:28.446 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.446 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.447 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:42:28.447 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.447 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.447 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.448 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:42:28.448 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.449 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:42:28.449 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:42:28.450 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:28.450 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.450 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.451 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.451 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000" 2022-09-02T04:42:28.452 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.452 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.452 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:42:28.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.453 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:42:28.453 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:42:28.453 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:42:28.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.454 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:42:28Z" 2022-09-02T04:42:28.455 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:42:28.456 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5969670707592890178 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:20.360817Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:37:02.975340Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:33:00.969288Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:42:28Z" } 2022-09-02T04:42:28.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:20.360817Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:37:02.975340Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:33:00.969288Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2022-09-02T04:42:28.457 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005588', 3: ''} 2022-09-02T04:42:28.458 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005588', 3: ''} 2022-09-02T04:42:28.458 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-02T04:42:28.459 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-02T04:42:28.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2022-09-02T04:42:28.463 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:42:28.464 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:42:28.465 DEBUG:boto:Method: PUT 2022-09-02T04:42:28.465 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:42:28.466 DEBUG:boto:Data: 2022-09-02T04:42:28.467 DEBUG:boto:Headers: {} 2022-09-02T04:42:28.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:28.468 DEBUG:boto:Port: 8000 2022-09-02T04:42:28.468 DEBUG:boto:Params: {} 2022-09-02T04:42:28.469 DEBUG:boto:Token: None 2022-09-02T04:42:28.469 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:42:28 GMT /xljbkf-20/ 2022-09-02T04:42:28.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xKG2vIrjxRJnrpVrn3E5luk2WNg= 2022-09-02T04:42:28.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xKG2vIrjxRJnrpVrn3E5luk2WNg=', 'Content-Length': '0'} 2022-09-02T04:42:28.484 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d072173fa2e39d27-00631189b4-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:28.484 DEBUG:rgw_multi.tests:created bucket=xljbkf-20 2022-09-02T04:42:28.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.486 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:28.536 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:28.566 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:42:28.580 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:42:28.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.581 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:42:28.582 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:42:28.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.583 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:28.584 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:42:28.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.585 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093748.483984_153.1", 2022-09-02T04:42:28.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:42:28.483984Z" 2022-09-02T04:42:28.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:28.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:28.587 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:42:28.587 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:42:28.588 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:28.588 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:42:28.589 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:28.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:42:28.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:28.593 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:28.645 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:28.645 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:28.694 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:28.694+0000 7fbdc27be600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:28.697 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:28.695+0000 7fbdc27be600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:28.724 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:42:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:42:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:42:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:42:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:42:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:42:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:42:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:42:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:42:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:42:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:42:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:42:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:42:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:42:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093133.866884_134.1", 2022-09-02T04:42:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.866884Z", 2022-09-02T04:42:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:42:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:42:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:42:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:42:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:42:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:42:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:42:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:42:28Z" 2022-09-02T04:42:28.743 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:42:28.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093133.866884_134.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:32:13.866884Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:42:28Z" } 2022-09-02T04:42:28.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093133.866884_134.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:32:13.866884Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:42:28.745 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:28.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093133.866884_134.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:28.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662093748.483984_153.1 target=1_1662093133.866884_134.1 2022-09-02T04:42:33.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:33.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:33.749 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:33.807 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:33.808 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:33.838 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:33.838+0000 7f9a4f16b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:33.839 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:33.840+0000 7f9a4f16b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:33.854 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:42:33.854 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:42:33.854 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:42:33.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:42:33.855 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:42:33.855 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:42:33.856 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:33.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:33.856 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:42:33.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:33.857 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:42:33.858 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:33.858 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:33.858 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:42:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:42:33.860 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:33.861 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:42:33.861 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:33.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:33.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:33.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:33.863 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:42:33.863 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:33.864 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:33.864 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:33.864 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:33.865 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:33.865 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:33.865 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:42:33.865 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:33.866 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:33.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:33.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:33.868 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:42:33.868 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:33.869 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:33.869 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093748.483984_153.1", 2022-09-02T04:42:33.869 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:33.869 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:33.870 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:33.870 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.483984Z", 2022-09-02T04:42:33.871 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:33.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:33.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:33.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:33.872 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:42:33.872 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:33.872 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:33.873 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:42:33.873 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:33.873 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:33.874 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:33.874 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:42:33.874 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:33.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:33.875 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:33.875 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:42:33.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:33.875 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:42:33.876 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:42:33.876 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:42:33.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:33.877 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:42:33Z" 2022-09-02T04:42:33.877 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:42:33.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093748.483984_153.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:42:28.483984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:42:33Z" } 2022-09-02T04:42:33.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093748.483984_153.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:42:28.483984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:42:33.880 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:33.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:33.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:42:33.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:42:33.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:33.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:33.885 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:33.940 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:33.941 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:33.990 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:42:33.991+0000 7fceec503600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:33.991 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:42:33.992+0000 7fceec503600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:34.016 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:42:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:42:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:42:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:42:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:42:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:42:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:42:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:42:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:34.019 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:42:34.019 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:34.019 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:42:34.019 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:42:34.020 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:34.020 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:42:34.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:34.020 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:42:34.021 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:42:34.021 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:34.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:34.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:34.022 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:42:34.023 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:34.023 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:42:34.023 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:42:34.023 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:34.024 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:34.024 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:34.024 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:42:34.025 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:42:34.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:34.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:34.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:34.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:42:34.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:34.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:42:34.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093748.483984_153.1", 2022-09-02T04:42:34.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:34.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:34.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:34.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.483984Z", 2022-09-02T04:42:34.032 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:42:34.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:34.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:34.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:42:34.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:42:34.034 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:42:34.034 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:42:34.034 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:42:34.035 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:42:34.035 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:42:34.035 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:42:34.035 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:42:34.036 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:42:34.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:34.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:42:34.037 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:42:34.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:34.038 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:42:34.038 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:42:34.038 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:42:34.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:42:34.039 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:42:33Z" 2022-09-02T04:42:34.039 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:42:34.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093748.483984_153.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:42:28.483984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:42:33Z" } 2022-09-02T04:42:34.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093748.483984_153.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:42:28.483984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:42:34.042 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:34.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:34.043 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:42:34.043 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:42:34.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:34.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:42:34.045 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:42:34.096 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:34.097 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:34.131 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:34.131+0000 7f386d539600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:34.132 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:34.132+0000 7f386d539600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:42:34.148 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:42:34.149 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:42:34.149 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:42:34.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:42:34.150 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:42:34.150 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:42:34.150 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:34.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:34.151 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:42:34.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:34.151 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:42:34.151 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:34.152 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:34.152 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:42:34.152 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:34.153 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:42:34.153 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:34.153 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:42:34.153 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:34.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:34.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:34.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:34.154 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:42:34.155 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:34.155 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:34.155 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:42:34.156 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:34.156 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:34.156 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:34.157 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:42:34.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:34.157 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:34.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:34.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:34.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:42:34.158 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:34.159 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:34.159 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093748.483984_153.1", 2022-09-02T04:42:34.159 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:34.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:34.160 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:34.160 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.483984Z", 2022-09-02T04:42:34.161 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:34.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:34.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:34.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:42:34.162 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:42:34.162 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:42:34.163 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:42:34.163 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:42:34.163 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:42:34.164 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:42:34.164 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:42:34.164 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:42:34.165 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:42:34.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:34.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:42:34.166 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:42:34.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:34.166 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:42:34.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:42:34.167 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:42:34.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:42:34.167 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:42:34Z" 2022-09-02T04:42:34.167 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:42:34.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093748.483984_153.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:42:28.483984Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:42:34Z" } 2022-09-02T04:42:34.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093748.483984_153.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:42:28.483984Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:42:34.170 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:34.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093748.483984_153.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:42:34.171 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:42:34.172 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:42:34.172 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:42:34.173 DEBUG:boto:Method: HEAD 2022-09-02T04:42:34.174 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:42:34.174 DEBUG:boto:Data: 2022-09-02T04:42:34.175 DEBUG:boto:Headers: {} 2022-09-02T04:42:34.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:34.176 DEBUG:boto:Port: 8000 2022-09-02T04:42:34.176 DEBUG:boto:Params: {} 2022-09-02T04:42:34.177 DEBUG:boto:Token: None 2022-09-02T04:42:34.178 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:42:34 GMT /xljbkf-20/ 2022-09-02T04:42:34.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cQCLOfwEt9+NHq3GC2/RudxTUWQ= 2022-09-02T04:42:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:42:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cQCLOfwEt9+NHq3GC2/RudxTUWQ=', 'Content-Length': '0'} 2022-09-02T04:42:34.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4913484dfdcb862-00631189ba-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:34.184 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:42:34.185 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:42:34.186 DEBUG:boto:Method: PUT 2022-09-02T04:42:34.187 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:42:34.187 DEBUG:boto:Data: 2022-09-02T04:42:34.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:42:34.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:42:34.189 DEBUG:boto:Port: 8000 2022-09-02T04:42:34.189 DEBUG:boto:Params: {} 2022-09-02T04:42:34.190 DEBUG:boto:Token: None 2022-09-02T04:42:34.191 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:42:34 GMT /xljbkf-20/obj 2022-09-02T04:42:34.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6Q3fjrcckrh+qpCgYNHjSMvNMvg= 2022-09-02T04:42:34.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:42:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6Q3fjrcckrh+qpCgYNHjSMvNMvg='} 2022-09-02T04:42:34.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c01f594393c6937-00631189ba-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:42:34.201 DEBUG:rgw_multi.tests:created initial version id=None 2022-09-02T04:42:34.202 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-20 2022-09-02T04:42:34.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:42:34.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:42:34.204 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:42:34.260 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:42:34.261 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:42:34.306 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:34.306+0000 7ff817629600 1 waiting to reach incremental sync.. 2022-09-02T04:42:39.308 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:39.307+0000 7ff817629600 1 waiting to reach incremental sync.. 2022-09-02T04:42:44.308 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:44.307+0000 7ff817629600 1 waiting to reach incremental sync.. 2022-09-02T04:42:49.308 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:49.308+0000 7ff817629600 1 waiting to reach incremental sync.. 2022-09-02T04:42:54.309 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:54.309+0000 7ff817629600 1 waiting to reach incremental sync.. 2022-09-02T04:42:59.309 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:42:59.311+0000 7ff817629600 1 waiting to reach incremental sync.. 2022-09-02T04:43:04.312 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:04.312+0000 7ff817629600 1 bucket sync caught up with source: 2022-09-02T04:43:04.312 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:04.313 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:04.314 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:04.312+0000 7ff817629600 0 bucket checkpoint complete 2022-09-02T04:43:04.333 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-20 2022-09-02T04:43:04.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.337 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:04.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.457+0000 7fecd7186600 1 bucket sync caught up with source: 2022-09-02T04:43:04.457 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:04.457 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.457+0000 7fecd7186600 0 bucket checkpoint complete 2022-09-02T04:43:04.478 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:43:04.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.481 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:04.548 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:04.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:04.592 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:04.593+0000 7f9b07458600 1 bucket sync caught up with source: 2022-09-02T04:43:04.593 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:04.593 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:04.593+0000 7f9b07458600 0 bucket checkpoint complete 2022-09-02T04:43:04.607 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-20 2022-09-02T04:43:04.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.609 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:04.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:04.670 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:04.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.722+0000 7fbff475e600 1 bucket sync caught up with source: 2022-09-02T04:43:04.722 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2022-09-02T04:43:04.723 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.71.5, , ] 2022-09-02T04:43:04.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.722+0000 7fbff475e600 0 bucket checkpoint complete 2022-09-02T04:43:04.735 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-20 2022-09-02T04:43:04.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.737 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:04.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:04.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:04.828 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.829+0000 7eff90ed8600 1 bucket sync caught up with source: 2022-09-02T04:43:04.829 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2022-09-02T04:43:04.829 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.71.5, , ] 2022-09-02T04:43:04.829 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.829+0000 7eff90ed8600 0 bucket checkpoint complete 2022-09-02T04:43:04.843 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:43:04.844 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-20 2022-09-02T04:43:04.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.846 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:04.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:04.903 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:04.959 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.959+0000 7fae4943f600 1 bucket sync caught up with source: 2022-09-02T04:43:04.960 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.65.5, , ] 2022-09-02T04:43:04.960 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.65.5, , ] 2022-09-02T04:43:04.960 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:04.959+0000 7fae4943f600 0 bucket checkpoint complete 2022-09-02T04:43:04.976 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-20 2022-09-02T04:43:04.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:04.980 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:05.040 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:05.041 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:05.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:05.089+0000 7f8e3bd73600 1 bucket sync caught up with source: 2022-09-02T04:43:05.089 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.65.5, , ] 2022-09-02T04:43:05.090 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.65.5, , ] 2022-09-02T04:43:05.091 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:05.089+0000 7f8e3bd73600 0 bucket checkpoint complete 2022-09-02T04:43:05.112 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:43:05.113 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone2, test-zone1} 2022-09-02T04:43:05.114 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.115 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.116 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.117 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:05.118 DEBUG:boto:Data: 2022-09-02T04:43:05.119 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.120 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.120 DEBUG:boto:Params: {} 2022-09-02T04:43:05.122 DEBUG:boto:Token: None 2022-09-02T04:43:05.123 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM= 2022-09-02T04:43:05.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM=', 'Content-Length': '0'} 2022-09-02T04:43:05.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013779a34d47c07f0-00631189d9-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.136 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.137 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.137 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.138 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:05.139 DEBUG:boto:Data: 2022-09-02T04:43:05.139 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.141 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.141 DEBUG:boto:Params: {} 2022-09-02T04:43:05.143 DEBUG:boto:Token: None 2022-09-02T04:43:05.143 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM= 2022-09-02T04:43:05.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM=', 'Content-Length': '0'} 2022-09-02T04:43:05.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051aa7d2859554192-00631189d9-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:43:05.149 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.149 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.150 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.151 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.155 DEBUG:boto:Method: GET 2022-09-02T04:43:05.156 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.156 DEBUG:boto:Data: 2022-09-02T04:43:05.157 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.158 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.159 DEBUG:boto:Params: {} 2022-09-02T04:43:05.159 DEBUG:boto:Token: None 2022-09-02T04:43:05.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d2f915f8760249-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.166 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:43:05.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:43:05.170 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.171 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.171 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.172 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.172 DEBUG:boto:Method: GET 2022-09-02T04:43:05.173 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.174 DEBUG:boto:Data: 2022-09-02T04:43:05.174 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.175 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.176 DEBUG:boto:Params: {} 2022-09-02T04:43:05.177 DEBUG:boto:Token: None 2022-09-02T04:43:05.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036458077c0d545a7-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.222 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:43:05.224 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.225 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.226 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.226 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.227 DEBUG:boto:Method: GET 2022-09-02T04:43:05.227 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.228 DEBUG:boto:Data: 2022-09-02T04:43:05.229 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.230 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.230 DEBUG:boto:Params: {} 2022-09-02T04:43:05.231 DEBUG:boto:Token: None 2022-09-02T04:43:05.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a217c9a087101d59-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.237 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.238 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.239 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.239 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.240 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.240 DEBUG:boto:Method: GET 2022-09-02T04:43:05.241 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.242 DEBUG:boto:Data: 2022-09-02T04:43:05.243 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.246 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.247 DEBUG:boto:Params: {} 2022-09-02T04:43:05.248 DEBUG:boto:Token: None 2022-09-02T04:43:05.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068b5cd19cc830984-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.254 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.255 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.256 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.256 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.257 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.257 DEBUG:boto:Method: GET 2022-09-02T04:43:05.258 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.258 DEBUG:boto:Data: 2022-09-02T04:43:05.259 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.260 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.261 DEBUG:boto:Params: {} 2022-09-02T04:43:05.261 DEBUG:boto:Token: None 2022-09-02T04:43:05.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002626edacf2e0e570-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.266 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.267 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.267 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.268 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.268 DEBUG:boto:Method: GET 2022-09-02T04:43:05.269 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.269 DEBUG:boto:Data: 2022-09-02T04:43:05.270 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.271 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.271 DEBUG:boto:Params: {} 2022-09-02T04:43:05.272 DEBUG:boto:Token: None 2022-09-02T04:43:05.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035e83de86cd0994c-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.277 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.277 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.278 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.278 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.279 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.279 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.280 DEBUG:boto:Data: 2022-09-02T04:43:05.280 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.281 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.282 DEBUG:boto:Params: {} 2022-09-02T04:43:05.282 DEBUG:boto:Token: None 2022-09-02T04:43:05.283 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.284 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c= 2022-09-02T04:43:05.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c=', 'Content-Length': '0'} 2022-09-02T04:43:05.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c10aca225296ce9-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.288 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.289 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.290 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.291 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.291 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.292 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.293 DEBUG:boto:Data: 2022-09-02T04:43:05.294 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.295 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.296 DEBUG:boto:Params: {} 2022-09-02T04:43:05.297 DEBUG:boto:Token: None 2022-09-02T04:43:05.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c= 2022-09-02T04:43:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c=', 'Content-Length': '0'} 2022-09-02T04:43:05.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bfc4778f77fbcfdf-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.305 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.305 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.306 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.307 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.307 DEBUG:boto:Method: GET 2022-09-02T04:43:05.308 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.308 DEBUG:boto:Data: 2022-09-02T04:43:05.309 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.310 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.311 DEBUG:boto:Params: {} 2022-09-02T04:43:05.312 DEBUG:boto:Token: None 2022-09-02T04:43:05.313 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f9a965c1da1e2d7-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.319 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.319 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.320 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.320 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.321 DEBUG:boto:Method: GET 2022-09-02T04:43:05.321 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.322 DEBUG:boto:Data: 2022-09-02T04:43:05.323 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.324 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.324 DEBUG:boto:Params: {} 2022-09-02T04:43:05.325 DEBUG:boto:Token: None 2022-09-02T04:43:05.326 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8a7e2b8e98f92c9-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.330 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.330 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.331 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.332 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.332 DEBUG:boto:Method: GET 2022-09-02T04:43:05.333 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.334 DEBUG:boto:Data: 2022-09-02T04:43:05.334 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.335 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.336 DEBUG:boto:Params: {} 2022-09-02T04:43:05.336 DEBUG:boto:Token: None 2022-09-02T04:43:05.337 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e93615221927d3-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.342 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.343 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.344 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.344 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.345 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.345 DEBUG:boto:Method: GET 2022-09-02T04:43:05.346 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.347 DEBUG:boto:Data: 2022-09-02T04:43:05.348 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.349 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.350 DEBUG:boto:Params: {} 2022-09-02T04:43:05.351 DEBUG:boto:Token: None 2022-09-02T04:43:05.352 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030304737a266b144-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.357 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.358 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.359 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.359 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.360 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.360 DEBUG:boto:Data: 2022-09-02T04:43:05.361 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.362 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.362 DEBUG:boto:Params: {} 2022-09-02T04:43:05.363 DEBUG:boto:Token: None 2022-09-02T04:43:05.363 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k= 2022-09-02T04:43:05.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k=', 'Content-Length': '0'} 2022-09-02T04:43:05.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008712ad519a192426-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.367 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.368 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.368 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.369 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.369 DEBUG:boto:Data: 2022-09-02T04:43:05.370 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.371 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.371 DEBUG:boto:Params: {} 2022-09-02T04:43:05.372 DEBUG:boto:Token: None 2022-09-02T04:43:05.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k= 2022-09-02T04:43:05.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k=', 'Content-Length': '0'} 2022-09-02T04:43:05.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c6d3c0f9d4feb6f-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.377 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.378 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.378 DEBUG:boto:Method: GET 2022-09-02T04:43:05.379 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.379 DEBUG:boto:Data: 2022-09-02T04:43:05.380 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.381 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.381 DEBUG:boto:Params: {} 2022-09-02T04:43:05.382 DEBUG:boto:Token: None 2022-09-02T04:43:05.383 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E= 2022-09-02T04:43:05.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E=', 'Content-Length': '0'} 2022-09-02T04:43:05.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d0b1e8b95dae519-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.387 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.387 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.388 DEBUG:boto:Method: GET 2022-09-02T04:43:05.388 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.389 DEBUG:boto:Data: 2022-09-02T04:43:05.390 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.391 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.391 DEBUG:boto:Params: {} 2022-09-02T04:43:05.392 DEBUG:boto:Token: None 2022-09-02T04:43:05.392 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E= 2022-09-02T04:43:05.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E=', 'Content-Length': '0'} 2022-09-02T04:43:05.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000052929fc0e625dc9-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.397 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.398 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.398 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.399 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.399 DEBUG:boto:Method: GET 2022-09-02T04:43:05.400 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.401 DEBUG:boto:Data: 2022-09-02T04:43:05.401 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.402 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.402 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.403 DEBUG:boto:Params: {} 2022-09-02T04:43:05.403 DEBUG:boto:Token: None 2022-09-02T04:43:05.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094b6b6007bf541ba-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.450 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.452 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.453 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.454 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.455 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.455 DEBUG:boto:Method: GET 2022-09-02T04:43:05.456 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.456 DEBUG:boto:Data: 2022-09-02T04:43:05.457 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.459 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.460 DEBUG:boto:Params: {} 2022-09-02T04:43:05.461 DEBUG:boto:Token: None 2022-09-02T04:43:05.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005535000340552982-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.467 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.468 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-20 zones={test-zone2, test-zone1} 2022-09-02T04:43:05.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone3, test-zone1} 2022-09-02T04:43:05.469 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.470 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.470 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.471 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:05.471 DEBUG:boto:Data: 2022-09-02T04:43:05.472 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.473 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.474 DEBUG:boto:Params: {} 2022-09-02T04:43:05.475 DEBUG:boto:Token: None 2022-09-02T04:43:05.476 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM= 2022-09-02T04:43:05.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM=', 'Content-Length': '0'} 2022-09-02T04:43:05.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c334cfb4f391020a-00631189d9-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.483 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.484 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.485 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.486 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:05.487 DEBUG:boto:Data: 2022-09-02T04:43:05.488 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.489 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.490 DEBUG:boto:Params: {} 2022-09-02T04:43:05.491 DEBUG:boto:Token: None 2022-09-02T04:43:05.492 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM= 2022-09-02T04:43:05.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM=', 'Content-Length': '0'} 2022-09-02T04:43:05.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016a54e2922c7ebba-00631189d9-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.497 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:43:05.498 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.499 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.500 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.500 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.501 DEBUG:boto:Method: GET 2022-09-02T04:43:05.501 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.502 DEBUG:boto:Data: 2022-09-02T04:43:05.502 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.503 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.504 DEBUG:boto:Params: {} 2022-09-02T04:43:05.505 DEBUG:boto:Token: None 2022-09-02T04:43:05.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae36d7d26c74932-00631189d9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.510 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:43:05.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:43:05.511 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.512 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.512 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.513 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.513 DEBUG:boto:Method: GET 2022-09-02T04:43:05.514 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.514 DEBUG:boto:Data: 2022-09-02T04:43:05.515 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.516 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.516 DEBUG:boto:Params: {} 2022-09-02T04:43:05.517 DEBUG:boto:Token: None 2022-09-02T04:43:05.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f04ae6e7c9d33fad-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.564 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:43:05.567 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.568 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.569 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.570 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.570 DEBUG:boto:Method: GET 2022-09-02T04:43:05.571 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.571 DEBUG:boto:Data: 2022-09-02T04:43:05.572 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.573 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.575 DEBUG:boto:Params: {} 2022-09-02T04:43:05.576 DEBUG:boto:Token: None 2022-09-02T04:43:05.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0034af1f0772bd-00631189d9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.583 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.585 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.586 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.587 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.588 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.588 DEBUG:boto:Method: GET 2022-09-02T04:43:05.589 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.590 DEBUG:boto:Data: 2022-09-02T04:43:05.591 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.593 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.593 DEBUG:boto:Params: {} 2022-09-02T04:43:05.595 DEBUG:boto:Token: None 2022-09-02T04:43:05.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd8db7176fab56a0-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.602 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.604 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.604 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.605 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.606 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.607 DEBUG:boto:Method: GET 2022-09-02T04:43:05.608 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.608 DEBUG:boto:Data: 2022-09-02T04:43:05.609 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.610 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.611 DEBUG:boto:Params: {} 2022-09-02T04:43:05.612 DEBUG:boto:Token: None 2022-09-02T04:43:05.612 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed5f8437fe48a5ce-00631189d9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.617 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.618 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.619 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.620 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.621 DEBUG:boto:Method: GET 2022-09-02T04:43:05.623 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.623 DEBUG:boto:Data: 2022-09-02T04:43:05.624 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.627 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.627 DEBUG:boto:Params: {} 2022-09-02T04:43:05.628 DEBUG:boto:Token: None 2022-09-02T04:43:05.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d6c70ff1f2cdf31-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.633 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.634 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.634 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.635 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.635 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.636 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.636 DEBUG:boto:Data: 2022-09-02T04:43:05.637 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.638 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.638 DEBUG:boto:Params: {} 2022-09-02T04:43:05.639 DEBUG:boto:Token: None 2022-09-02T04:43:05.639 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c= 2022-09-02T04:43:05.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c=', 'Content-Length': '0'} 2022-09-02T04:43:05.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fea3c5a8e980752e-00631189d9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.644 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.644 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.645 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.645 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.646 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.647 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.647 DEBUG:boto:Data: 2022-09-02T04:43:05.648 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.649 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.649 DEBUG:boto:Params: {} 2022-09-02T04:43:05.650 DEBUG:boto:Token: None 2022-09-02T04:43:05.650 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c= 2022-09-02T04:43:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M7Q88WxoBb1ELR7sIP4JvUeOf0c=', 'Content-Length': '0'} 2022-09-02T04:43:05.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005469eff31c6a126c-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.655 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.656 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.657 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.657 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.658 DEBUG:boto:Method: GET 2022-09-02T04:43:05.658 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.659 DEBUG:boto:Data: 2022-09-02T04:43:05.659 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.661 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.661 DEBUG:boto:Params: {} 2022-09-02T04:43:05.662 DEBUG:boto:Token: None 2022-09-02T04:43:05.662 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e160d6b9b3f9cbc-00631189d9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.673 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.674 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.675 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.676 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.679 DEBUG:boto:Method: GET 2022-09-02T04:43:05.681 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.682 DEBUG:boto:Data: 2022-09-02T04:43:05.683 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.684 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.685 DEBUG:boto:Params: {} 2022-09-02T04:43:05.686 DEBUG:boto:Token: None 2022-09-02T04:43:05.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:05.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg= 2022-09-02T04:43:05.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m62HSXZU6ivWgjIPDiRdPOI7jlg=', 'Content-Length': '0'} 2022-09-02T04:43:05.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006934b7d7e3526098-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.691 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.691 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.692 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.692 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.693 DEBUG:boto:Method: GET 2022-09-02T04:43:05.694 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.694 DEBUG:boto:Data: 2022-09-02T04:43:05.695 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.696 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.696 DEBUG:boto:Params: {} 2022-09-02T04:43:05.697 DEBUG:boto:Token: None 2022-09-02T04:43:05.698 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c6d16712a0f841-00631189d9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.742 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.743 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.744 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.744 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.745 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.746 DEBUG:boto:Method: GET 2022-09-02T04:43:05.746 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.747 DEBUG:boto:Data: 2022-09-02T04:43:05.748 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.750 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.751 DEBUG:boto:Params: {} 2022-09-02T04:43:05.752 DEBUG:boto:Token: None 2022-09-02T04:43:05.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e7fa066c0050e4-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.760 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.762 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.763 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.764 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.765 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.766 DEBUG:boto:Data: 2022-09-02T04:43:05.767 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.769 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.770 DEBUG:boto:Params: {} 2022-09-02T04:43:05.772 DEBUG:boto:Token: None 2022-09-02T04:43:05.773 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k= 2022-09-02T04:43:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k=', 'Content-Length': '0'} 2022-09-02T04:43:05.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dfd04367c4b52a20-00631189d9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.779 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.780 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.780 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.782 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.783 DEBUG:boto:Data: 2022-09-02T04:43:05.784 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.786 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.787 DEBUG:boto:Params: {} 2022-09-02T04:43:05.788 DEBUG:boto:Token: None 2022-09-02T04:43:05.790 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k= 2022-09-02T04:43:05.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TPMAclrjGJmfT4p7eq+A29yw40k=', 'Content-Length': '0'} 2022-09-02T04:43:05.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f8c92c3cdbe83bb-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.798 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.799 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.800 DEBUG:boto:Method: GET 2022-09-02T04:43:05.801 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.802 DEBUG:boto:Data: 2022-09-02T04:43:05.803 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.804 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.805 DEBUG:boto:Params: {} 2022-09-02T04:43:05.807 DEBUG:boto:Token: None 2022-09-02T04:43:05.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E= 2022-09-02T04:43:05.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E=', 'Content-Length': '0'} 2022-09-02T04:43:05.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e445f73149bb4372-00631189d9-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.815 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.816 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.817 DEBUG:boto:Method: GET 2022-09-02T04:43:05.818 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:05.819 DEBUG:boto:Data: 2022-09-02T04:43:05.820 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.822 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.823 DEBUG:boto:Params: {} 2022-09-02T04:43:05.824 DEBUG:boto:Token: None 2022-09-02T04:43:05.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/obj 2022-09-02T04:43:05.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E= 2022-09-02T04:43:05.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oZqclSVzYAQiU1d06SI8Ds6mG+E=', 'Content-Length': '0'} 2022-09-02T04:43:05.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000bf059a7715aeed0-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.829 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.830 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.830 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.831 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.832 DEBUG:boto:Method: GET 2022-09-02T04:43:05.833 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.833 DEBUG:boto:Data: 2022-09-02T04:43:05.834 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.835 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.835 DEBUG:boto:Params: {} 2022-09-02T04:43:05.836 DEBUG:boto:Token: None 2022-09-02T04:43:05.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ebcca0e9df98d85-00631189d9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.842 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.843 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.844 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.845 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.846 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:05.847 DEBUG:boto:Method: GET 2022-09-02T04:43:05.848 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:05.849 DEBUG:boto:Data: 2022-09-02T04:43:05.850 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:05.852 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.853 DEBUG:boto:Params: {} 2022-09-02T04:43:05.854 DEBUG:boto:Token: None 2022-09-02T04:43:05.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA= 2022-09-02T04:43:05.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bWX92r1XCY9z519POW3vyYRLnDA=', 'Content-Length': '0'} 2022-09-02T04:43:05.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007785f23e17af82c3-00631189d9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.902 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-20 zones={test-zone3, test-zone1} 2022-09-02T04:43:05.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone3, test-zone2} 2022-09-02T04:43:05.904 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.905 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.906 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.906 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:05.907 DEBUG:boto:Data: 2022-09-02T04:43:05.908 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.909 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.910 DEBUG:boto:Params: {} 2022-09-02T04:43:05.910 DEBUG:boto:Token: None 2022-09-02T04:43:05.911 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM= 2022-09-02T04:43:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM=', 'Content-Length': '0'} 2022-09-02T04:43:05.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf0c4d0124f92a27-00631189d9-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.916 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.917 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.917 DEBUG:boto:Method: HEAD 2022-09-02T04:43:05.918 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:05.919 DEBUG:boto:Data: 2022-09-02T04:43:05.920 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.920 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.921 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.921 DEBUG:boto:Params: {} 2022-09-02T04:43:05.922 DEBUG:boto:Token: None 2022-09-02T04:43:05.923 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/ 2022-09-02T04:43:05.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM= 2022-09-02T04:43:05.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o7mJWxXVlXQNdDJ04MOWaBomkyM=', 'Content-Length': '0'} 2022-09-02T04:43:05.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf4e5c4585e1a1b8-00631189d9-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:43:05.930 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.931 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.932 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.932 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.933 DEBUG:boto:Method: GET 2022-09-02T04:43:05.933 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.934 DEBUG:boto:Data: 2022-09-02T04:43:05.934 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.935 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.936 DEBUG:boto:Params: {} 2022-09-02T04:43:05.937 DEBUG:boto:Token: None 2022-09-02T04:43:05.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fd7cd77c290815-00631189d9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.943 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:43:05.945 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:43:05.945 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.946 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.946 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.947 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.947 DEBUG:boto:Method: GET 2022-09-02T04:43:05.948 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.948 DEBUG:boto:Data: 2022-09-02T04:43:05.949 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.950 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.951 DEBUG:boto:Params: {} 2022-09-02T04:43:05.951 DEBUG:boto:Token: None 2022-09-02T04:43:05.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061ac8ffd0991c290-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.958 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:43:05.961 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.962 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.963 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.963 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.964 DEBUG:boto:Method: GET 2022-09-02T04:43:05.965 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.966 DEBUG:boto:Data: 2022-09-02T04:43:05.967 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:05.969 DEBUG:boto:Port: 8001 2022-09-02T04:43:05.970 DEBUG:boto:Params: {} 2022-09-02T04:43:05.971 DEBUG:boto:Token: None 2022-09-02T04:43:05.972 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008915705498b00e14-00631189d9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.979 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.980 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:05.980 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:05.981 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.981 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.982 DEBUG:boto:Method: GET 2022-09-02T04:43:05.982 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:43:05.983 DEBUG:boto:Data: 2022-09-02T04:43:05.983 DEBUG:boto:Headers: {} 2022-09-02T04:43:05.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:05.985 DEBUG:boto:Port: 8000 2022-09-02T04:43:05.986 DEBUG:boto:Params: {} 2022-09-02T04:43:05.986 DEBUG:boto:Token: None 2022-09-02T04:43:05.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:05 GMT /xljbkf-20/?versions 2022-09-02T04:43:05.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE= 2022-09-02T04:43:05.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5xYfxq5sStAjyg9Xg5/cMRV7GWE=', 'Content-Length': '0'} 2022-09-02T04:43:05.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c5249b96badfb3-00631189d9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:05.995 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:05.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:05.997 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:05.998 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:05.999 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.000 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.001 DEBUG:boto:Method: GET 2022-09-02T04:43:06.002 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.003 DEBUG:boto:Data: 2022-09-02T04:43:06.005 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.007 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.008 DEBUG:boto:Params: {} 2022-09-02T04:43:06.009 DEBUG:boto:Token: None 2022-09-02T04:43:06.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI= 2022-09-02T04:43:06.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI=', 'Content-Length': '0'} 2022-09-02T04:43:06.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b4545ff94da02b0-00631189da-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.016 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.017 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.018 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.019 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.020 DEBUG:boto:Method: GET 2022-09-02T04:43:06.021 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.022 DEBUG:boto:Data: 2022-09-02T04:43:06.022 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.023 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.025 DEBUG:boto:Params: {} 2022-09-02T04:43:06.026 DEBUG:boto:Token: None 2022-09-02T04:43:06.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI= 2022-09-02T04:43:06.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI=', 'Content-Length': '0'} 2022-09-02T04:43:06.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096579ad3498f6164-00631189da-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.033 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.034 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.035 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.035 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.036 DEBUG:boto:Method: HEAD 2022-09-02T04:43:06.036 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.037 DEBUG:boto:Data: 2022-09-02T04:43:06.037 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.039 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.039 DEBUG:boto:Params: {} 2022-09-02T04:43:06.040 DEBUG:boto:Token: None 2022-09-02T04:43:06.041 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GBKuBFdOl9WxwWcLFAGFfvc2Dfw= 2022-09-02T04:43:06.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GBKuBFdOl9WxwWcLFAGFfvc2Dfw=', 'Content-Length': '0'} 2022-09-02T04:43:06.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c8df30ce46847b2-00631189da-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.047 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.048 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.049 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.050 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.051 DEBUG:boto:Method: HEAD 2022-09-02T04:43:06.051 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.052 DEBUG:boto:Data: 2022-09-02T04:43:06.053 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.055 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.057 DEBUG:boto:Params: {} 2022-09-02T04:43:06.058 DEBUG:boto:Token: None 2022-09-02T04:43:06.059 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GBKuBFdOl9WxwWcLFAGFfvc2Dfw= 2022-09-02T04:43:06.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GBKuBFdOl9WxwWcLFAGFfvc2Dfw=', 'Content-Length': '0'} 2022-09-02T04:43:06.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e71c74805ddaec31-00631189da-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:06.066 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.066 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.067 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.067 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.068 DEBUG:boto:Method: GET 2022-09-02T04:43:06.069 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.070 DEBUG:boto:Data: 2022-09-02T04:43:06.071 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.073 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.074 DEBUG:boto:Params: {} 2022-09-02T04:43:06.075 DEBUG:boto:Token: None 2022-09-02T04:43:06.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI= 2022-09-02T04:43:06.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI=', 'Content-Length': '0'} 2022-09-02T04:43:06.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090ccc26da2b09c38-00631189da-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.082 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.083 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.084 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.085 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.086 DEBUG:boto:Method: GET 2022-09-02T04:43:06.087 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.087 DEBUG:boto:Data: 2022-09-02T04:43:06.088 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.089 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.090 DEBUG:boto:Params: {} 2022-09-02T04:43:06.091 DEBUG:boto:Token: None 2022-09-02T04:43:06.091 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:43:06.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI= 2022-09-02T04:43:06.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ck0AyDkIy0U2f9f/5u68k3/3vxI=', 'Content-Length': '0'} 2022-09-02T04:43:06.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee195a524c9019b6-00631189da-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.098 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:06.098 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:06.099 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.100 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.101 DEBUG:boto:Method: GET 2022-09-02T04:43:06.101 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:06.102 DEBUG:boto:Data: 2022-09-02T04:43:06.102 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.103 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.104 DEBUG:boto:Params: {} 2022-09-02T04:43:06.104 DEBUG:boto:Token: None 2022-09-02T04:43:06.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/ 2022-09-02T04:43:06.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0= 2022-09-02T04:43:06.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0=', 'Content-Length': '0'} 2022-09-02T04:43:06.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030d0b44e6898ff8f-00631189da-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.150 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:06.152 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:06.153 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:06.154 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.155 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.156 DEBUG:boto:Method: GET 2022-09-02T04:43:06.157 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:06.157 DEBUG:boto:Data: 2022-09-02T04:43:06.158 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.159 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.160 DEBUG:boto:Params: {} 2022-09-02T04:43:06.160 DEBUG:boto:Token: None 2022-09-02T04:43:06.161 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/ 2022-09-02T04:43:06.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0= 2022-09-02T04:43:06.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0=', 'Content-Length': '0'} 2022-09-02T04:43:06.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e103245da6881477-00631189da-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.167 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:06.167 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.168 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.168 DEBUG:boto:Method: HEAD 2022-09-02T04:43:06.169 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:06.169 DEBUG:boto:Data: 2022-09-02T04:43:06.170 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.171 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.171 DEBUG:boto:Params: {} 2022-09-02T04:43:06.172 DEBUG:boto:Token: None 2022-09-02T04:43:06.173 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj 2022-09-02T04:43:06.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hnVq1Fzz7d08mH5Dj4a1FYJ2NR4= 2022-09-02T04:43:06.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hnVq1Fzz7d08mH5Dj4a1FYJ2NR4=', 'Content-Length': '0'} 2022-09-02T04:43:06.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002256c98056336584-00631189da-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.177 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.177 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.178 DEBUG:boto:Method: HEAD 2022-09-02T04:43:06.178 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:06.179 DEBUG:boto:Data: 2022-09-02T04:43:06.180 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.182 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.183 DEBUG:boto:Params: {} 2022-09-02T04:43:06.183 DEBUG:boto:Token: None 2022-09-02T04:43:06.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj 2022-09-02T04:43:06.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hnVq1Fzz7d08mH5Dj4a1FYJ2NR4= 2022-09-02T04:43:06.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hnVq1Fzz7d08mH5Dj4a1FYJ2NR4=', 'Content-Length': '0'} 2022-09-02T04:43:06.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f38f0bda9a6e089f-00631189da-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:43:06.191 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.191 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.192 DEBUG:boto:Method: GET 2022-09-02T04:43:06.193 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:06.194 DEBUG:boto:Data: 2022-09-02T04:43:06.195 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.196 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.197 DEBUG:boto:Params: {} 2022-09-02T04:43:06.199 DEBUG:boto:Token: None 2022-09-02T04:43:06.200 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj 2022-09-02T04:43:06.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDIUWRWHC32v3mfSxiYi6jffokc= 2022-09-02T04:43:06.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDIUWRWHC32v3mfSxiYi6jffokc=', 'Content-Length': '0'} 2022-09-02T04:43:06.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f6fd750d18418e7-00631189da-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.206 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.207 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.207 DEBUG:boto:Method: GET 2022-09-02T04:43:06.208 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:06.208 DEBUG:boto:Data: 2022-09-02T04:43:06.209 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.210 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.211 DEBUG:boto:Params: {} 2022-09-02T04:43:06.212 DEBUG:boto:Token: None 2022-09-02T04:43:06.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj 2022-09-02T04:43:06.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDIUWRWHC32v3mfSxiYi6jffokc= 2022-09-02T04:43:06.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDIUWRWHC32v3mfSxiYi6jffokc=', 'Content-Length': '0'} 2022-09-02T04:43:06.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad3e3453e8c4db2c-00631189da-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.217 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:06.218 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:06.219 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.219 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.220 DEBUG:boto:Method: GET 2022-09-02T04:43:06.220 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:06.221 DEBUG:boto:Data: 2022-09-02T04:43:06.222 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:43:06.223 DEBUG:boto:Port: 8001 2022-09-02T04:43:06.224 DEBUG:boto:Params: {} 2022-09-02T04:43:06.225 DEBUG:boto:Token: None 2022-09-02T04:43:06.226 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/ 2022-09-02T04:43:06.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0= 2022-09-02T04:43:06.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0=', 'Content-Length': '0'} 2022-09-02T04:43:06.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1cb90458fae0826-00631189da-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.231 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:06.232 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:06.233 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:06.233 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.234 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:43:06.234 DEBUG:boto:Method: GET 2022-09-02T04:43:06.235 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:43:06.235 DEBUG:boto:Data: 2022-09-02T04:43:06.237 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:43:06.238 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.239 DEBUG:boto:Params: {} 2022-09-02T04:43:06.239 DEBUG:boto:Token: None 2022-09-02T04:43:06.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/ 2022-09-02T04:43:06.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0= 2022-09-02T04:43:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9JxesO52xMD9lgb8j6q5HrmMMD0=', 'Content-Length': '0'} 2022-09-02T04:43:06.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000424ea2b667ff7537-00631189da-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.245 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:43:06.246 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-20 zones={test-zone3, test-zone2} 2022-09-02T04:43:06.246 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:06.247 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:06.247 DEBUG:boto:path=/xljbkf-20/?versioning 2022-09-02T04:43:06.248 DEBUG:boto:auth_path=/xljbkf-20/?versioning 2022-09-02T04:43:06.248 DEBUG:boto:Method: PUT 2022-09-02T04:43:06.249 DEBUG:boto:Path: /xljbkf-20/?versioning 2022-09-02T04:43:06.249 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:43:06.250 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:06.251 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.251 DEBUG:boto:Params: {} 2022-09-02T04:43:06.252 DEBUG:boto:Token: None 2022-09-02T04:43:06.252 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/?versioning 2022-09-02T04:43:06.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:14MDMc0tApzlN2/mB7j/7zE5FZk= 2022-09-02T04:43:06.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:14MDMc0tApzlN2/mB7j/7zE5FZk=', 'Content-Length': '228'} 2022-09-02T04:43:06.260 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f3f5c0bcc6ae3a02-00631189da-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.262 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:43:06.316 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:06.316 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:43:06.372 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:43:06.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.373 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:43:06.373 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:43:06.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.374 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:43:06.375 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:43:06.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.376 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093786.261185_155.1", 2022-09-02T04:43:06.376 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:43:06.261185Z" 2022-09-02T04:43:06.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.377 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:43:06.377 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:43:06.377 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:43:06.377 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:43:06.380 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.381 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:43:06.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.384 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:43:06.442 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:06.443 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:06.473+0000 7f73a701a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:43:06.473 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:06.474+0000 7f73a701a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:43:06.485 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:43:06.486 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:43:06.486 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:43:06.487 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:43:06.487 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:43:06.488 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:43:06.488 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.489 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:43:06.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.491 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:43:06.491 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.492 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.493 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:43:06.493 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.494 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:43:06.494 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.495 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:43:06.495 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.497 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:43:06.497 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.498 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.498 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:43:06.498 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.498 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:43:06.499 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.499 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:43:06.499 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.500 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:43:06.501 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.501 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.501 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093786.261185_155.1", 2022-09-02T04:43:06.501 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.502 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:43:06.502 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.502 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:43:06.261185Z", 2022-09-02T04:43:06.503 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.505 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:43:06.505 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.505 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.505 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:43:06.506 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.506 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:43:06.506 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.507 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:43:06.507 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.508 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:43:06.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.508 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:43:06.509 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:43:06.509 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:43:06.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.509 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:43:06Z" 2022-09-02T04:43:06.510 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:43:06.511 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093786.261185_155.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:43:06.261185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:43:06Z" } 2022-09-02T04:43:06.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093786.261185_155.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:43:06.261185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:43:06.513 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.515 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:43:06.516 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:43:06.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.518 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:43:06.572 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:06.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:06.615 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:06.615+0000 7f17c9072600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:43:06.616 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:43:06.617+0000 7f17c9072600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:43:06.639 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:43:06.639 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:43:06.640 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:43:06.640 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:43:06.641 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:43:06.641 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:43:06.642 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:43:06.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.644 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:43:06.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:43:06.645 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:43:06.645 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:43:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:43:06.646 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:43:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:43:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:43:06.647 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:43:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:43:06.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:43:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.649 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:43:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:43:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:43:06.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:43:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:43:06.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:43:06.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:43:06.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:43:06.653 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:43:06.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:43:06.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.655 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:43:06.655 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:43:06.656 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:43:06.656 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093786.261185_155.1", 2022-09-02T04:43:06.656 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:43:06.657 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:43:06.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:43:06.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:43:06.261185Z", 2022-09-02T04:43:06.658 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:43:06.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:43:06.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:43:06.660 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:43:06.661 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:43:06.662 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:43:06.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:43:06.663 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:43:06.663 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:43:06.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:43:06.664 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:43:06.665 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:43:06.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:43:06.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:43:06.666 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:43:06.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.666 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:43:06.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:43:06.667 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:43:06.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:43:06.668 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:43:06Z" 2022-09-02T04:43:06.669 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:43:06.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093786.261185_155.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:43:06.261185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:43:06Z" } 2022-09-02T04:43:06.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093786.261185_155.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:43:06.261185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:43:06.673 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:43:06.676 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:43:06.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:43:06.678 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:43:06.734 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:06.735 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:06.760 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:06.761+0000 7f670304a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:43:06.760 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:06.762+0000 7f670304a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:43:06.781 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:43:06.781 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:43:06.781 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:43:06.782 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:43:06.782 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:43:06.782 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:43:06.783 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.783 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:43:06.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.784 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:43:06.784 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.784 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.785 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:43:06.785 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.785 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:43:06.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.786 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:43:06.786 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.787 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:43:06.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.788 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:43:06.788 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:43:06.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.789 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:43:06.789 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.790 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:43:06.791 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.791 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.791 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093786.261185_155.1", 2022-09-02T04:43:06.791 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.792 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:43:06.792 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.792 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:43:06.261185Z", 2022-09-02T04:43:06.792 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:43:06.794 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:43:06.794 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:43:06.795 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:43:06.795 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:43:06.796 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:43:06.796 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:43:06.796 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:43:06.797 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:43:06.797 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:43:06.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:43:06.799 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:43:06.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.800 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:43:06.800 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:43:06.801 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:43:06.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:43:06.802 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:43:06Z" 2022-09-02T04:43:06.803 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:43:06.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093786.261185_155.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:43:06.261185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:43:06Z" } 2022-09-02T04:43:06.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093786.261185_155.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:43:06.261185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:43:06.806 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.807 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093786.261185_155.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:43:06.808 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:43:06.809 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:43:06.809 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:43:06.810 DEBUG:boto:Method: HEAD 2022-09-02T04:43:06.811 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:43:06.812 DEBUG:boto:Data: 2022-09-02T04:43:06.813 DEBUG:boto:Headers: {} 2022-09-02T04:43:06.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:06.815 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.816 DEBUG:boto:Params: {} 2022-09-02T04:43:06.817 DEBUG:boto:Token: None 2022-09-02T04:43:06.818 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/ 2022-09-02T04:43:06.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UmeA3sIY3kQR7aqytNhjHIDQ5YQ= 2022-09-02T04:43:06.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UmeA3sIY3kQR7aqytNhjHIDQ5YQ=', 'Content-Length': '0'} 2022-09-02T04:43:06.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000542eb30a3b1f7d7b-00631189da-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.824 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:43:06.825 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:43:06.825 DEBUG:boto:Method: PUT 2022-09-02T04:43:06.826 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:43:06.826 DEBUG:boto:Data: 2022-09-02T04:43:06.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:43:06.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:43:06.828 DEBUG:boto:Port: 8000 2022-09-02T04:43:06.829 DEBUG:boto:Params: {} 2022-09-02T04:43:06.829 DEBUG:boto:Token: None 2022-09-02T04:43:06.830 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:43:06 GMT /xljbkf-20/obj 2022-09-02T04:43:06.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+C/Wd6Ejrrmkg7KDRy06dE+MDk= 2022-09-02T04:43:06.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:43:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+C/Wd6Ejrrmkg7KDRy06dE+MDk='} 2022-09-02T04:43:06.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-amz-request-id', 'tx00000410a690420c0b077-00631189da-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:43:06.849 DEBUG:rgw_multi.tests:created new version id=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:43:06.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-20 2022-09-02T04:43:06.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:06.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:43:06.852 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:43:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:43:06.912 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:43:06.968 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:06.969+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:06.969 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:06.970 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:11.970 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:11.970+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:11.970 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:11.970 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:16.972 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:16.971+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:16.973 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:16.973 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:21.977 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:21.973+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:21.977 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:21.978 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:26.974 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:26.974+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:26.974 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:26.975 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:31.976 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:31.976+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:31.976 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:31.977 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:36.977 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:36.977+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:36.977 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:36.977 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:41.978 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:41.978+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:41.979 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:41.979 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:46.982 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:46.980+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:46.982 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:46.982 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:51.981 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:51.981+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:51.982 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:51.982 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:43:56.983 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:43:56.983+0000 7f04d03a8600 1 waiting for incremental sync to catch up: 2022-09-02T04:43:56.984 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:43:56.985 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:01.985 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:44:01.984+0000 7f04d03a8600 1 bucket sync caught up with source: 2022-09-02T04:44:01.985 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:01.988 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:01.988 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:44:01.984+0000 7f04d03a8600 0 bucket checkpoint complete 2022-09-02T04:44:02.001 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-20 2022-09-02T04:44:02.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:02.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:02.003 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:44:02.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:44:02.067 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:44:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:02.113+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:02.113 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:02.114 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:07.114 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:07.115+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:07.115 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:07.116 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:12.115 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:12.116+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:12.116 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:12.116 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:17.117 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:17.117+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:17.117 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:17.118 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:22.118 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:22.119+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:22.119 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:22.120 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:27.120+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:27.120 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:27.121 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:32.122 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:32.122+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:32.122 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:32.123 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:37.123 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:37.123+0000 7fefa0ef7600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:37.123 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-09-02T04:44:37.124 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:42.126 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.125+0000 7fefa0ef7600 1 bucket sync caught up with source: 2022-09-02T04:44:42.127 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:42.128 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:42.128 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.125+0000 7fefa0ef7600 0 bucket checkpoint complete 2022-09-02T04:44:42.151 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:44:42.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.156 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:44:42.225 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:44:42.225 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:44:42.271 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:44:42.272+0000 7fdb6e5e0600 1 bucket sync caught up with source: 2022-09-02T04:44:42.272 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:42.272 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:44:42.273 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:44:42.272+0000 7fdb6e5e0600 0 bucket checkpoint complete 2022-09-02T04:44:42.284 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-20 2022-09-02T04:44:42.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.287 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:44:42.351 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:44:42.351 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:44:42.399 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.399+0000 7f3ede1a1600 1 bucket sync caught up with source: 2022-09-02T04:44:42.399 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.75.12, , ] 2022-09-02T04:44:42.399 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.75.12, , ] 2022-09-02T04:44:42.400 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.399+0000 7f3ede1a1600 0 bucket checkpoint complete 2022-09-02T04:44:42.411 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-20 2022-09-02T04:44:42.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.414 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:44:42.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:44:42.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:44:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.508+0000 7fc5541a7600 1 bucket sync caught up with source: 2022-09-02T04:44:42.508 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.75.12, , ] 2022-09-02T04:44:42.508 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.75.12, , ] 2022-09-02T04:44:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.508+0000 7fc5541a7600 0 bucket checkpoint complete 2022-09-02T04:44:42.521 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:44:42.522 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-20 2022-09-02T04:44:42.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:44:42.526 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:44:42.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:44:42.584 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:44:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:42.625+0000 7f66ce1e4600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:42.625 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.69.12, , ] 2022-09-02T04:44:42.625 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:44:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:47.627+0000 7f66ce1e4600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:47.630 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.69.12, , ] 2022-09-02T04:44:47.631 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:44:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:52.628+0000 7f66ce1e4600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:52.629 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.69.12, , ] 2022-09-02T04:44:52.629 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:44:57.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:44:57.629+0000 7f66ce1e4600 1 waiting for incremental sync to catch up: 2022-09-02T04:44:57.630 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.69.12, , ] 2022-09-02T04:44:57.631 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:45:02.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:02.631+0000 7f66ce1e4600 1 bucket sync caught up with source: 2022-09-02T04:45:02.631 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:45:02.631 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:45:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:02.631+0000 7f66ce1e4600 0 bucket checkpoint complete 2022-09-02T04:45:02.652 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-20 2022-09-02T04:45:02.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:02.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:02.654 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:02.722 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:02.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:02.770 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:02.771+0000 7fd2df819600 1 bucket sync caught up with source: 2022-09-02T04:45:02.771 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:45:02.771 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000004.71.9, , ] 2022-09-02T04:45:02.771 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:02.771+0000 7fd2df819600 0 bucket checkpoint complete 2022-09-02T04:45:02.792 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:45:02.793 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone2, test-zone1} 2022-09-02T04:45:02.794 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.794 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.795 DEBUG:boto:Method: HEAD 2022-09-02T04:45:02.796 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:02.796 DEBUG:boto:Data: 2022-09-02T04:45:02.797 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.798 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.798 DEBUG:boto:Params: {} 2022-09-02T04:45:02.799 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:45:02.800 DEBUG:boto:Token: None 2022-09-02T04:45:02.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/ 2022-09-02T04:45:02.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/0GS9ej4/yG/X1lqjxMHEHJjzGg= 2022-09-02T04:45:02.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/0GS9ej4/yG/X1lqjxMHEHJjzGg=', 'Content-Length': '0'} 2022-09-02T04:45:02.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038f250c87562b12a-0063118a4e-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.808 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.808 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.809 DEBUG:boto:Method: HEAD 2022-09-02T04:45:02.809 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:02.810 DEBUG:boto:Data: 2022-09-02T04:45:02.811 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:02.812 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.813 DEBUG:boto:Params: {} 2022-09-02T04:45:02.814 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:45:02.815 DEBUG:boto:Token: None 2022-09-02T04:45:02.816 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/ 2022-09-02T04:45:02.817 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/0GS9ej4/yG/X1lqjxMHEHJjzGg= 2022-09-02T04:45:02.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/0GS9ej4/yG/X1lqjxMHEHJjzGg=', 'Content-Length': '0'} 2022-09-02T04:45:02.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004737509d643f35ac-0063118a4e-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.823 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:45:02.824 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.824 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.825 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.826 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.826 DEBUG:boto:Method: GET 2022-09-02T04:45:02.827 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.828 DEBUG:boto:Data: 2022-09-02T04:45:02.831 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.833 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.835 DEBUG:boto:Params: {} 2022-09-02T04:45:02.836 DEBUG:boto:Token: None 2022-09-02T04:45:02.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/?versions 2022-09-02T04:45:02.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0= 2022-09-02T04:45:02.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0=', 'Content-Length': '0'} 2022-09-02T04:45:02.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8843e08a1fba61f-0063118a4e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.842 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:02.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:02.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:02.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:45:02.848 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.849 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.850 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.850 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.852 DEBUG:boto:Method: GET 2022-09-02T04:45:02.853 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.854 DEBUG:boto:Data: 2022-09-02T04:45:02.855 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:02.857 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.858 DEBUG:boto:Params: {} 2022-09-02T04:45:02.859 DEBUG:boto:Token: None 2022-09-02T04:45:02.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/?versions 2022-09-02T04:45:02.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0= 2022-09-02T04:45:02.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0=', 'Content-Length': '0'} 2022-09-02T04:45:02.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b561280d25b751b-0063118a4e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.867 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:02.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:02.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:02.870 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.871 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.871 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.872 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.873 DEBUG:boto:Method: GET 2022-09-02T04:45:02.874 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.875 DEBUG:boto:Data: 2022-09-02T04:45:02.875 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.876 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.877 DEBUG:boto:Params: {} 2022-09-02T04:45:02.877 DEBUG:boto:Token: None 2022-09-02T04:45:02.878 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/?versions 2022-09-02T04:45:02.879 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0= 2022-09-02T04:45:02.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0=', 'Content-Length': '0'} 2022-09-02T04:45:02.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6770098c3b791a4-0063118a4e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.884 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:02.886 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.887 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.887 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.888 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.888 DEBUG:boto:Method: GET 2022-09-02T04:45:02.889 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:02.890 DEBUG:boto:Data: 2022-09-02T04:45:02.890 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:02.891 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.892 DEBUG:boto:Params: {} 2022-09-02T04:45:02.893 DEBUG:boto:Token: None 2022-09-02T04:45:02.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/?versions 2022-09-02T04:45:02.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0= 2022-09-02T04:45:02.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bHSRsuwCVWyoKV+BpfIr4GgTUC0=', 'Content-Length': '0'} 2022-09-02T04:45:02.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c6c5091a4a4c8c-0063118a4e-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.902 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:02.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:02.905 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:02.906 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:02.907 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.908 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.909 DEBUG:boto:Method: GET 2022-09-02T04:45:02.911 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.912 DEBUG:boto:Data: 2022-09-02T04:45:02.912 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.913 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.914 DEBUG:boto:Params: {} 2022-09-02T04:45:02.915 DEBUG:boto:Token: None 2022-09-02T04:45:02.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ= 2022-09-02T04:45:02.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ=', 'Content-Length': '0'} 2022-09-02T04:45:02.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057cb626b7428d639-0063118a4e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.919 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:02.920 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:02.920 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.921 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.921 DEBUG:boto:Method: GET 2022-09-02T04:45:02.922 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.923 DEBUG:boto:Data: 2022-09-02T04:45:02.923 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:02.924 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.925 DEBUG:boto:Params: {} 2022-09-02T04:45:02.925 DEBUG:boto:Token: None 2022-09-02T04:45:02.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ= 2022-09-02T04:45:02.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ=', 'Content-Length': '0'} 2022-09-02T04:45:02.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc83df55996ab17b-0063118a4e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.930 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:02.931 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:02.931 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.932 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.932 DEBUG:boto:Method: HEAD 2022-09-02T04:45:02.933 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.933 DEBUG:boto:Data: 2022-09-02T04:45:02.934 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.935 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.935 DEBUG:boto:Params: {} 2022-09-02T04:45:02.936 DEBUG:boto:Token: None 2022-09-02T04:45:02.936 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fEnS3Df/J7LeiI/998HH/GkIUOM= 2022-09-02T04:45:02.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fEnS3Df/J7LeiI/998HH/GkIUOM=', 'Content-Length': '0'} 2022-09-02T04:45:02.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b31e4d6e48895e1-0063118a4e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.940 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:02.941 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:02.941 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.942 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.942 DEBUG:boto:Method: HEAD 2022-09-02T04:45:02.943 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.943 DEBUG:boto:Data: 2022-09-02T04:45:02.944 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:02.945 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.946 DEBUG:boto:Params: {} 2022-09-02T04:45:02.946 DEBUG:boto:Token: None 2022-09-02T04:45:02.947 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fEnS3Df/J7LeiI/998HH/GkIUOM= 2022-09-02T04:45:02.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fEnS3Df/J7LeiI/998HH/GkIUOM=', 'Content-Length': '0'} 2022-09-02T04:45:02.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e2bf4bfb6f466b1-0063118a4e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:02.952 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:02.952 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:02.953 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.954 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.954 DEBUG:boto:Method: GET 2022-09-02T04:45:02.955 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.956 DEBUG:boto:Data: 2022-09-02T04:45:02.957 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.958 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.959 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.960 DEBUG:boto:Params: {} 2022-09-02T04:45:02.961 DEBUG:boto:Token: None 2022-09-02T04:45:02.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ= 2022-09-02T04:45:02.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ=', 'Content-Length': '0'} 2022-09-02T04:45:02.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a9f720bc973786d-0063118a4e-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.968 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:02.969 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:02.970 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.970 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.971 DEBUG:boto:Method: GET 2022-09-02T04:45:02.972 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.972 DEBUG:boto:Data: 2022-09-02T04:45:02.973 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:02.974 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.974 DEBUG:boto:Params: {} 2022-09-02T04:45:02.975 DEBUG:boto:Token: None 2022-09-02T04:45:02.975 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:02.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ= 2022-09-02T04:45:02.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gzrRu1vI8TZOw+Oa5o35vLtQ2NQ=', 'Content-Length': '0'} 2022-09-02T04:45:02.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a7e748ede333b29-0063118a4e-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:02.980 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:02.980 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:02.981 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:02.981 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:02.982 DEBUG:boto:Method: GET 2022-09-02T04:45:02.982 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:02.983 DEBUG:boto:Data: 2022-09-02T04:45:02.984 DEBUG:boto:Headers: {} 2022-09-02T04:45:02.984 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:02.985 DEBUG:boto:Port: 8000 2022-09-02T04:45:02.985 DEBUG:boto:Params: {} 2022-09-02T04:45:02.986 DEBUG:boto:Token: None 2022-09-02T04:45:02.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:02 GMT /xljbkf-20/ 2022-09-02T04:45:02.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xxm4gQxsFgtw6Zjp8R0eO6ZCQug= 2022-09-02T04:45:02.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xxm4gQxsFgtw6Zjp8R0eO6ZCQug=', 'Content-Length': '0'} 2022-09-02T04:45:02.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be5ca12e6d443364-0063118a4e-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.038 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.039 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.040 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.040 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.041 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.041 DEBUG:boto:Method: GET 2022-09-02T04:45:03.042 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.043 DEBUG:boto:Data: 2022-09-02T04:45:03.043 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.044 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.045 DEBUG:boto:Params: {} 2022-09-02T04:45:03.045 DEBUG:boto:Token: None 2022-09-02T04:45:03.046 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9387a2e70dbe81e-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.050 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.051 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.052 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.052 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.053 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.054 DEBUG:boto:Data: 2022-09-02T04:45:03.054 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.055 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.056 DEBUG:boto:Params: {} 2022-09-02T04:45:03.056 DEBUG:boto:Token: None 2022-09-02T04:45:03.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc8802f7b8d8e453-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.062 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.062 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.063 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.063 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.064 DEBUG:boto:Data: 2022-09-02T04:45:03.064 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.065 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.066 DEBUG:boto:Params: {} 2022-09-02T04:45:03.067 DEBUG:boto:Token: None 2022-09-02T04:45:03.067 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec5fdc79618bac37-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.073 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.074 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.074 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.075 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.075 DEBUG:boto:Method: GET 2022-09-02T04:45:03.076 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.076 DEBUG:boto:Data: 2022-09-02T04:45:03.077 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.078 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.079 DEBUG:boto:Params: {} 2022-09-02T04:45:03.080 DEBUG:boto:Token: None 2022-09-02T04:45:03.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007db4581c1f39d6ec-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.085 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.085 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.086 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.087 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.087 DEBUG:boto:Method: GET 2022-09-02T04:45:03.088 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.089 DEBUG:boto:Data: 2022-09-02T04:45:03.089 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.090 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.091 DEBUG:boto:Params: {} 2022-09-02T04:45:03.092 DEBUG:boto:Token: None 2022-09-02T04:45:03.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdbd931f7b190921-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.097 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.098 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.098 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.099 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.099 DEBUG:boto:Method: GET 2022-09-02T04:45:03.100 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.100 DEBUG:boto:Data: 2022-09-02T04:45:03.101 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.102 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.103 DEBUG:boto:Params: {} 2022-09-02T04:45:03.103 DEBUG:boto:Token: None 2022-09-02T04:45:03.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aecb18c8f466c595-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.109 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.111 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.111 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.112 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.114 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.116 DEBUG:boto:Method: GET 2022-09-02T04:45:03.117 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.117 DEBUG:boto:Data: 2022-09-02T04:45:03.118 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.120 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.120 DEBUG:boto:Params: {} 2022-09-02T04:45:03.121 DEBUG:boto:Token: None 2022-09-02T04:45:03.121 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6c0754d4fc5400-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.129 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.131 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.132 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.133 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.134 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.135 DEBUG:boto:Method: GET 2022-09-02T04:45:03.136 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.137 DEBUG:boto:Data: 2022-09-02T04:45:03.138 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.140 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.141 DEBUG:boto:Params: {} 2022-09-02T04:45:03.142 DEBUG:boto:Token: None 2022-09-02T04:45:03.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000530c720bdff67740-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.150 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.150 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.151 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.151 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.152 DEBUG:boto:Method: GET 2022-09-02T04:45:03.152 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.153 DEBUG:boto:Data: 2022-09-02T04:45:03.153 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.155 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.155 DEBUG:boto:Params: {} 2022-09-02T04:45:03.156 DEBUG:boto:Token: None 2022-09-02T04:45:03.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000312532dc3dc9873-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.161 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.161 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.162 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.162 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.163 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.163 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.164 DEBUG:boto:Data: 2022-09-02T04:45:03.164 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.165 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.166 DEBUG:boto:Params: {} 2022-09-02T04:45:03.167 DEBUG:boto:Token: None 2022-09-02T04:45:03.167 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM= 2022-09-02T04:45:03.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM=', 'Content-Length': '0'} 2022-09-02T04:45:03.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056082b898ca55116-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.171 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.172 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.172 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.173 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.173 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.174 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.174 DEBUG:boto:Data: 2022-09-02T04:45:03.175 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.176 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.176 DEBUG:boto:Params: {} 2022-09-02T04:45:03.177 DEBUG:boto:Token: None 2022-09-02T04:45:03.177 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM= 2022-09-02T04:45:03.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM=', 'Content-Length': '0'} 2022-09-02T04:45:03.181 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069377d8c4f4aa2a2-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.183 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.183 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.184 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.184 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.185 DEBUG:boto:Method: GET 2022-09-02T04:45:03.185 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.186 DEBUG:boto:Data: 2022-09-02T04:45:03.187 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.188 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.188 DEBUG:boto:Params: {} 2022-09-02T04:45:03.189 DEBUG:boto:Token: None 2022-09-02T04:45:03.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000685ec854abead939-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.197 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.197 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.198 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.199 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.199 DEBUG:boto:Method: GET 2022-09-02T04:45:03.200 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.200 DEBUG:boto:Data: 2022-09-02T04:45:03.201 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.202 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.203 DEBUG:boto:Params: {} 2022-09-02T04:45:03.203 DEBUG:boto:Token: None 2022-09-02T04:45:03.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae99b2a19ed90777-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.209 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.209 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.210 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.210 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.211 DEBUG:boto:Method: GET 2022-09-02T04:45:03.211 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.212 DEBUG:boto:Data: 2022-09-02T04:45:03.212 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.214 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.214 DEBUG:boto:Params: {} 2022-09-02T04:45:03.215 DEBUG:boto:Token: None 2022-09-02T04:45:03.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c924cc37f92898e-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.266 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.267 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.268 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.269 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.269 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.271 DEBUG:boto:Method: GET 2022-09-02T04:45:03.272 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.273 DEBUG:boto:Data: 2022-09-02T04:45:03.273 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.276 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.276 DEBUG:boto:Params: {} 2022-09-02T04:45:03.277 DEBUG:boto:Token: None 2022-09-02T04:45:03.278 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bd3ad093cbdfa91-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.282 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.283 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.284 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.284 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.285 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.285 DEBUG:boto:Data: 2022-09-02T04:45:03.286 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.286 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.287 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.287 DEBUG:boto:Params: {} 2022-09-02T04:45:03.288 DEBUG:boto:Token: None 2022-09-02T04:45:03.289 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c0314802c63139f-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.293 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.294 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.294 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.295 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.295 DEBUG:boto:Data: 2022-09-02T04:45:03.296 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.297 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.297 DEBUG:boto:Params: {} 2022-09-02T04:45:03.298 DEBUG:boto:Token: None 2022-09-02T04:45:03.299 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000566dd3592ee7439f-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.304 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.304 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.305 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.306 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.306 DEBUG:boto:Method: GET 2022-09-02T04:45:03.307 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.307 DEBUG:boto:Data: 2022-09-02T04:45:03.308 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.309 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.309 DEBUG:boto:Params: {} 2022-09-02T04:45:03.310 DEBUG:boto:Token: None 2022-09-02T04:45:03.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d40b48991183424-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.316 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.316 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.317 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.317 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.318 DEBUG:boto:Method: GET 2022-09-02T04:45:03.318 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.319 DEBUG:boto:Data: 2022-09-02T04:45:03.320 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.321 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.321 DEBUG:boto:Params: {} 2022-09-02T04:45:03.322 DEBUG:boto:Token: None 2022-09-02T04:45:03.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042e357e8abb4ade9-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.328 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.328 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.329 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.329 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.330 DEBUG:boto:Method: GET 2022-09-02T04:45:03.330 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.331 DEBUG:boto:Data: 2022-09-02T04:45:03.331 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:03.332 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.333 DEBUG:boto:Params: {} 2022-09-02T04:45:03.334 DEBUG:boto:Token: None 2022-09-02T04:45:03.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a56059a19bba4401-0063118a4f-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.339 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.340 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.341 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.342 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.342 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.343 DEBUG:boto:Method: GET 2022-09-02T04:45:03.344 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.345 DEBUG:boto:Data: 2022-09-02T04:45:03.345 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.346 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.347 DEBUG:boto:Params: {} 2022-09-02T04:45:03.348 DEBUG:boto:Token: None 2022-09-02T04:45:03.348 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000595c319c020b93f-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.354 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.356 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-20 zones={test-zone2, test-zone1} 2022-09-02T04:45:03.356 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone3, test-zone1} 2022-09-02T04:45:03.357 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.358 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.359 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.360 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:03.360 DEBUG:boto:Data: 2022-09-02T04:45:03.361 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.363 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.364 DEBUG:boto:Params: {} 2022-09-02T04:45:03.365 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:45:03.366 DEBUG:boto:Token: None 2022-09-02T04:45:03.367 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HPSuOftashQOrU6Me5szEXWkG4= 2022-09-02T04:45:03.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HPSuOftashQOrU6Me5szEXWkG4=', 'Content-Length': '0'} 2022-09-02T04:45:03.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008128167c591a43dd-0063118a4f-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.375 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.376 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.377 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.378 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:03.379 DEBUG:boto:Data: 2022-09-02T04:45:03.380 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.382 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.383 DEBUG:boto:Params: {} 2022-09-02T04:45:03.384 DEBUG:boto:Token: None 2022-09-02T04:45:03.385 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HPSuOftashQOrU6Me5szEXWkG4= 2022-09-02T04:45:03.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HPSuOftashQOrU6Me5szEXWkG4=', 'Content-Length': '0'} 2022-09-02T04:45:03.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d010efa24f88d262-0063118a4f-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:45:03.393 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.394 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.396 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.397 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.398 DEBUG:boto:Method: GET 2022-09-02T04:45:03.399 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.400 DEBUG:boto:Data: 2022-09-02T04:45:03.401 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.402 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.403 DEBUG:boto:Params: {} 2022-09-02T04:45:03.404 DEBUG:boto:Token: None 2022-09-02T04:45:03.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/?versions 2022-09-02T04:45:03.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I= 2022-09-02T04:45:03.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I=', 'Content-Length': '0'} 2022-09-02T04:45:03.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca2ceb0b6d7d561-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.409 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:03.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:03.412 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:45:03.413 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.414 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.414 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.415 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.416 DEBUG:boto:Method: GET 2022-09-02T04:45:03.416 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.417 DEBUG:boto:Data: 2022-09-02T04:45:03.417 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.418 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.419 DEBUG:boto:Params: {} 2022-09-02T04:45:03.420 DEBUG:boto:Token: None 2022-09-02T04:45:03.420 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/?versions 2022-09-02T04:45:03.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I= 2022-09-02T04:45:03.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I=', 'Content-Length': '0'} 2022-09-02T04:45:03.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e23056a81f89c1-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.470 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:03.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:03.472 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.473 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.474 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.475 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.475 DEBUG:boto:Method: GET 2022-09-02T04:45:03.476 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.477 DEBUG:boto:Data: 2022-09-02T04:45:03.477 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.479 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.479 DEBUG:boto:Params: {} 2022-09-02T04:45:03.480 DEBUG:boto:Token: None 2022-09-02T04:45:03.480 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/?versions 2022-09-02T04:45:03.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I= 2022-09-02T04:45:03.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I=', 'Content-Length': '0'} 2022-09-02T04:45:03.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4cf07197499fb9-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.485 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.486 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.487 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.488 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.490 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.491 DEBUG:boto:Method: GET 2022-09-02T04:45:03.491 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:03.492 DEBUG:boto:Data: 2022-09-02T04:45:03.493 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.495 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.496 DEBUG:boto:Params: {} 2022-09-02T04:45:03.497 DEBUG:boto:Token: None 2022-09-02T04:45:03.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/?versions 2022-09-02T04:45:03.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I= 2022-09-02T04:45:03.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ew6MCpNCGTBHYKE+HmfDlhTSb6I=', 'Content-Length': '0'} 2022-09-02T04:45:03.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a4f53f612e679ed-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.503 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.505 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.506 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.506 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.507 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.508 DEBUG:boto:Method: GET 2022-09-02T04:45:03.509 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.510 DEBUG:boto:Data: 2022-09-02T04:45:03.510 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.512 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.513 DEBUG:boto:Params: {} 2022-09-02T04:45:03.514 DEBUG:boto:Token: None 2022-09-02T04:45:03.515 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d86a6125fe79f575-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.525 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.525 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.526 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.526 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.527 DEBUG:boto:Method: GET 2022-09-02T04:45:03.528 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.528 DEBUG:boto:Data: 2022-09-02T04:45:03.529 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.530 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.530 DEBUG:boto:Params: {} 2022-09-02T04:45:03.531 DEBUG:boto:Token: None 2022-09-02T04:45:03.532 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000047106469ee22a15-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.536 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.536 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.537 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.537 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.538 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.538 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.539 DEBUG:boto:Data: 2022-09-02T04:45:03.540 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.541 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.541 DEBUG:boto:Params: {} 2022-09-02T04:45:03.542 DEBUG:boto:Token: None 2022-09-02T04:45:03.542 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v5YaXw8OCa0Joim5RO6sO2gauSA= 2022-09-02T04:45:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v5YaXw8OCa0Joim5RO6sO2gauSA=', 'Content-Length': '0'} 2022-09-02T04:45:03.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adde8aada34d9e2f-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.547 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.548 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.549 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.550 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.550 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.551 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.552 DEBUG:boto:Data: 2022-09-02T04:45:03.553 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.554 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.555 DEBUG:boto:Params: {} 2022-09-02T04:45:03.556 DEBUG:boto:Token: None 2022-09-02T04:45:03.556 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v5YaXw8OCa0Joim5RO6sO2gauSA= 2022-09-02T04:45:03.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v5YaXw8OCa0Joim5RO6sO2gauSA=', 'Content-Length': '0'} 2022-09-02T04:45:03.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6174b750e5b76c9-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.563 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.564 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.564 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.565 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.566 DEBUG:boto:Method: GET 2022-09-02T04:45:03.567 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.567 DEBUG:boto:Data: 2022-09-02T04:45:03.568 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.569 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.569 DEBUG:boto:Params: {} 2022-09-02T04:45:03.571 DEBUG:boto:Token: None 2022-09-02T04:45:03.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee8d5e11cc042e08-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.578 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.579 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.579 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.580 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.580 DEBUG:boto:Method: GET 2022-09-02T04:45:03.581 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.581 DEBUG:boto:Data: 2022-09-02T04:45:03.582 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.583 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.584 DEBUG:boto:Params: {} 2022-09-02T04:45:03.584 DEBUG:boto:Token: None 2022-09-02T04:45:03.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000284065119655d824-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.589 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.590 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.590 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.591 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.591 DEBUG:boto:Method: GET 2022-09-02T04:45:03.592 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.592 DEBUG:boto:Data: 2022-09-02T04:45:03.593 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.594 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.594 DEBUG:boto:Params: {} 2022-09-02T04:45:03.595 DEBUG:boto:Token: None 2022-09-02T04:45:03.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009266a02655a06057-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.601 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.602 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.602 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.603 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.603 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.604 DEBUG:boto:Method: GET 2022-09-02T04:45:03.604 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.605 DEBUG:boto:Data: 2022-09-02T04:45:03.605 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.606 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.607 DEBUG:boto:Params: {} 2022-09-02T04:45:03.607 DEBUG:boto:Token: None 2022-09-02T04:45:03.608 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009182d1fd8147cc93-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.613 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.614 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.614 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.615 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.615 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.616 DEBUG:boto:Data: 2022-09-02T04:45:03.616 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.617 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.618 DEBUG:boto:Params: {} 2022-09-02T04:45:03.618 DEBUG:boto:Token: None 2022-09-02T04:45:03.619 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030571234877cd58e-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.624 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.624 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.625 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.625 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.626 DEBUG:boto:Data: 2022-09-02T04:45:03.627 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.628 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.628 DEBUG:boto:Params: {} 2022-09-02T04:45:03.629 DEBUG:boto:Token: None 2022-09-02T04:45:03.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061e7de33f3bc420d-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.635 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.635 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.636 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.636 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.637 DEBUG:boto:Method: GET 2022-09-02T04:45:03.637 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.638 DEBUG:boto:Data: 2022-09-02T04:45:03.639 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.640 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.640 DEBUG:boto:Params: {} 2022-09-02T04:45:03.641 DEBUG:boto:Token: None 2022-09-02T04:45:03.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d27cd492b8e013e-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.646 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.646 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.647 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.647 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.648 DEBUG:boto:Method: GET 2022-09-02T04:45:03.648 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.649 DEBUG:boto:Data: 2022-09-02T04:45:03.649 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.651 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.651 DEBUG:boto:Params: {} 2022-09-02T04:45:03.652 DEBUG:boto:Token: None 2022-09-02T04:45:03.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad696d7b1382559e-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.657 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.657 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.658 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.659 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.659 DEBUG:boto:Method: GET 2022-09-02T04:45:03.660 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.660 DEBUG:boto:Data: 2022-09-02T04:45:03.661 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.662 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.663 DEBUG:boto:Params: {} 2022-09-02T04:45:03.663 DEBUG:boto:Token: None 2022-09-02T04:45:03.664 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c43d1bbffee850b-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.714 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.715 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.716 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.716 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.717 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.717 DEBUG:boto:Method: GET 2022-09-02T04:45:03.718 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.719 DEBUG:boto:Data: 2022-09-02T04:45:03.719 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.720 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.721 DEBUG:boto:Params: {} 2022-09-02T04:45:03.722 DEBUG:boto:Token: None 2022-09-02T04:45:03.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ffa356667a797a3-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.728 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.729 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.730 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.730 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.731 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.731 DEBUG:boto:Method: GET 2022-09-02T04:45:03.732 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.732 DEBUG:boto:Data: 2022-09-02T04:45:03.733 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.734 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.735 DEBUG:boto:Params: {} 2022-09-02T04:45:03.735 DEBUG:boto:Token: None 2022-09-02T04:45:03.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000337c4eaf7e371e1-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.741 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.742 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.742 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.743 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.743 DEBUG:boto:Method: GET 2022-09-02T04:45:03.744 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.744 DEBUG:boto:Data: 2022-09-02T04:45:03.745 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.746 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.747 DEBUG:boto:Params: {} 2022-09-02T04:45:03.748 DEBUG:boto:Token: None 2022-09-02T04:45:03.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f0bd07c251a6b65-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.752 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.753 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.754 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.754 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.755 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.755 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.756 DEBUG:boto:Data: 2022-09-02T04:45:03.756 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.757 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.758 DEBUG:boto:Params: {} 2022-09-02T04:45:03.758 DEBUG:boto:Token: None 2022-09-02T04:45:03.759 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM= 2022-09-02T04:45:03.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM=', 'Content-Length': '0'} 2022-09-02T04:45:03.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1a9a4586a2632a1-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.764 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.765 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.766 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.767 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.768 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.769 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.770 DEBUG:boto:Data: 2022-09-02T04:45:03.771 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.772 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.773 DEBUG:boto:Params: {} 2022-09-02T04:45:03.774 DEBUG:boto:Token: None 2022-09-02T04:45:03.774 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM= 2022-09-02T04:45:03.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BLglpkY+pZkkKQ9xRIy6IftSfkM=', 'Content-Length': '0'} 2022-09-02T04:45:03.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b18147e8772f518-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.779 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.780 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.780 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.781 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.781 DEBUG:boto:Method: GET 2022-09-02T04:45:03.782 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.782 DEBUG:boto:Data: 2022-09-02T04:45:03.783 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.785 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.785 DEBUG:boto:Params: {} 2022-09-02T04:45:03.786 DEBUG:boto:Token: None 2022-09-02T04:45:03.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000017b3f9e7f2ec27c-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.791 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.792 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.792 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.793 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.793 DEBUG:boto:Method: GET 2022-09-02T04:45:03.794 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.795 DEBUG:boto:Data: 2022-09-02T04:45:03.795 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.796 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.797 DEBUG:boto:Params: {} 2022-09-02T04:45:03.797 DEBUG:boto:Token: None 2022-09-02T04:45:03.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:03.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM= 2022-09-02T04:45:03.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djIzsCUrLsJs9TRue5aJi3lZgcM=', 'Content-Length': '0'} 2022-09-02T04:45:03.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059471cd432507519-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.803 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.803 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.804 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.805 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.806 DEBUG:boto:Method: GET 2022-09-02T04:45:03.807 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.807 DEBUG:boto:Data: 2022-09-02T04:45:03.808 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.809 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.810 DEBUG:boto:Params: {} 2022-09-02T04:45:03.811 DEBUG:boto:Token: None 2022-09-02T04:45:03.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9605e18c6047781-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.866 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.867 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.868 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.868 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.869 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.869 DEBUG:boto:Method: GET 2022-09-02T04:45:03.870 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.870 DEBUG:boto:Data: 2022-09-02T04:45:03.871 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.872 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.872 DEBUG:boto:Params: {} 2022-09-02T04:45:03.873 DEBUG:boto:Token: None 2022-09-02T04:45:03.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3a9e3dfbef4ed4-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.879 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.881 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.882 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.883 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.884 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.885 DEBUG:boto:Data: 2022-09-02T04:45:03.885 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.887 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.888 DEBUG:boto:Params: {} 2022-09-02T04:45:03.889 DEBUG:boto:Token: None 2022-09-02T04:45:03.889 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a01eaee5fc60b044-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.895 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.895 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.896 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.897 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:03.897 DEBUG:boto:Data: 2022-09-02T04:45:03.898 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.899 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.900 DEBUG:boto:Params: {} 2022-09-02T04:45:03.901 DEBUG:boto:Token: None 2022-09-02T04:45:03.901 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj 2022-09-02T04:45:03.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60= 2022-09-02T04:45:03.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLJmKehn6uDPMhONRYvSiqtft60=', 'Content-Length': '0'} 2022-09-02T04:45:03.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001e9511ff1f4d536-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:03.908 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.908 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.909 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.910 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.911 DEBUG:boto:Method: GET 2022-09-02T04:45:03.912 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.912 DEBUG:boto:Data: 2022-09-02T04:45:03.913 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.915 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.916 DEBUG:boto:Params: {} 2022-09-02T04:45:03.917 DEBUG:boto:Token: None 2022-09-02T04:45:03.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.923 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054df6a375bf07832-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.924 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:03.925 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:03.926 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.929 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.930 DEBUG:boto:Method: GET 2022-09-02T04:45:03.940 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.941 DEBUG:boto:Data: 2022-09-02T04:45:03.942 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.943 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.944 DEBUG:boto:Params: {} 2022-09-02T04:45:03.945 DEBUG:boto:Token: None 2022-09-02T04:45:03.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:03.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I= 2022-09-02T04:45:03.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:43D1mKN2Z/YqUQX3gtygw2bcE3I=', 'Content-Length': '0'} 2022-09-02T04:45:03.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007626d139844aa88e-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.951 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.952 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.952 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.953 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.954 DEBUG:boto:Method: GET 2022-09-02T04:45:03.954 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.955 DEBUG:boto:Data: 2022-09-02T04:45:03.956 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.957 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.957 DEBUG:boto:Params: {} 2022-09-02T04:45:03.958 DEBUG:boto:Token: None 2022-09-02T04:45:03.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065f5f82a5718eaf7-0063118a4f-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.965 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.966 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.967 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.967 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.968 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:03.969 DEBUG:boto:Method: GET 2022-09-02T04:45:03.975 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:03.976 DEBUG:boto:Data: 2022-09-02T04:45:03.976 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:03.978 DEBUG:boto:Port: 8000 2022-09-02T04:45:03.978 DEBUG:boto:Params: {} 2022-09-02T04:45:03.979 DEBUG:boto:Token: None 2022-09-02T04:45:03.980 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg= 2022-09-02T04:45:03.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gqc5GffNIDgIiUZYB1Jocxa5YWg=', 'Content-Length': '0'} 2022-09-02T04:45:03.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f99476991dd9a2-0063118a4f-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.987 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:03.988 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-20 zones={test-zone3, test-zone1} 2022-09-02T04:45:03.988 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone3, test-zone2} 2022-09-02T04:45:03.989 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:03.990 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:03.990 DEBUG:boto:Method: HEAD 2022-09-02T04:45:03.991 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:03.991 DEBUG:boto:Data: 2022-09-02T04:45:03.992 DEBUG:boto:Headers: {} 2022-09-02T04:45:03.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:03.993 DEBUG:boto:Port: 8001 2022-09-02T04:45:03.993 DEBUG:boto:Params: {} 2022-09-02T04:45:03.994 DEBUG:boto:Token: None 2022-09-02T04:45:03.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:03 GMT /xljbkf-20/ 2022-09-02T04:45:03.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HPSuOftashQOrU6Me5szEXWkG4= 2022-09-02T04:45:03.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HPSuOftashQOrU6Me5szEXWkG4=', 'Content-Length': '0'} 2022-09-02T04:45:03.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d6bf2c9fe735548-0063118a4f-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:03.999 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.000 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.001 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.002 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:04.003 DEBUG:boto:Data: 2022-09-02T04:45:04.004 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.006 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.007 DEBUG:boto:Params: {} 2022-09-02T04:45:04.009 DEBUG:boto:Token: None 2022-09-02T04:45:04.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p3A4pBa7cHI4KAIZqnXZDIY8pTQ= 2022-09-02T04:45:04.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p3A4pBa7cHI4KAIZqnXZDIY8pTQ=', 'Content-Length': '0'} 2022-09-02T04:45:04.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acd8ca7e664c028d-0063118a50-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.016 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:45:04.017 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.018 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.019 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.020 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.021 DEBUG:boto:Method: GET 2022-09-02T04:45:04.022 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.023 DEBUG:boto:Data: 2022-09-02T04:45:04.024 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.025 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.026 DEBUG:boto:Params: {} 2022-09-02T04:45:04.027 DEBUG:boto:Token: None 2022-09-02T04:45:04.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/?versions 2022-09-02T04:45:04.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M= 2022-09-02T04:45:04.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M=', 'Content-Length': '0'} 2022-09-02T04:45:04.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054ba1aa7b9710a11-0063118a50-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.078 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:04.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:04.081 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:45:04.082 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.083 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.083 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.084 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.084 DEBUG:boto:Method: GET 2022-09-02T04:45:04.085 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.085 DEBUG:boto:Data: 2022-09-02T04:45:04.086 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.088 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.089 DEBUG:boto:Params: {} 2022-09-02T04:45:04.090 DEBUG:boto:Token: None 2022-09-02T04:45:04.091 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/?versions 2022-09-02T04:45:04.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M= 2022-09-02T04:45:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M=', 'Content-Length': '0'} 2022-09-02T04:45:04.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fdb19c27384ae1b-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.097 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:04.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:45:04.098 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.099 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.100 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.100 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.101 DEBUG:boto:Method: GET 2022-09-02T04:45:04.101 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.102 DEBUG:boto:Data: 2022-09-02T04:45:04.103 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.104 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.104 DEBUG:boto:Params: {} 2022-09-02T04:45:04.105 DEBUG:boto:Token: None 2022-09-02T04:45:04.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/?versions 2022-09-02T04:45:04.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M= 2022-09-02T04:45:04.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M=', 'Content-Length': '0'} 2022-09-02T04:45:04.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006828ce267d4afdb6-0063118a50-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.110 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.111 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.111 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.112 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.112 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.113 DEBUG:boto:Method: GET 2022-09-02T04:45:04.113 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:45:04.114 DEBUG:boto:Data: 2022-09-02T04:45:04.114 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.115 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.116 DEBUG:boto:Params: {} 2022-09-02T04:45:04.116 DEBUG:boto:Token: None 2022-09-02T04:45:04.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/?versions 2022-09-02T04:45:04.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M= 2022-09-02T04:45:04.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aq4KmeUPM3HO/7yiGCrvIHoMm6M=', 'Content-Length': '0'} 2022-09-02T04:45:04.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72ae3c164f4a381-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.122 DEBUG:boto:b'xljbkf-20999falseobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98true2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:04.124 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.124 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.125 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.126 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.127 DEBUG:boto:Method: GET 2022-09-02T04:45:04.128 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.128 DEBUG:boto:Data: 2022-09-02T04:45:04.129 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.130 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.130 DEBUG:boto:Params: {} 2022-09-02T04:45:04.131 DEBUG:boto:Token: None 2022-09-02T04:45:04.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083cefe8e861ad36f-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.139 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.140 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.141 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.143 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.143 DEBUG:boto:Method: GET 2022-09-02T04:45:04.144 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.145 DEBUG:boto:Data: 2022-09-02T04:45:04.145 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.147 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.148 DEBUG:boto:Params: {} 2022-09-02T04:45:04.149 DEBUG:boto:Token: None 2022-09-02T04:45:04.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000903830647103ca06-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.157 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.158 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.158 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.159 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.160 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.161 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.161 DEBUG:boto:Data: 2022-09-02T04:45:04.162 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.163 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.164 DEBUG:boto:Params: {} 2022-09-02T04:45:04.164 DEBUG:boto:Token: None 2022-09-02T04:45:04.165 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZuQffDf5kkkYoDYWrX1UqFEk+HY= 2022-09-02T04:45:04.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZuQffDf5kkkYoDYWrX1UqFEk+HY=', 'Content-Length': '0'} 2022-09-02T04:45:04.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000011f78e705f731219-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.169 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.169 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.170 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.170 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.171 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.172 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.172 DEBUG:boto:Data: 2022-09-02T04:45:04.173 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.174 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.174 DEBUG:boto:Params: {} 2022-09-02T04:45:04.176 DEBUG:boto:Token: None 2022-09-02T04:45:04.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZuQffDf5kkkYoDYWrX1UqFEk+HY= 2022-09-02T04:45:04.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZuQffDf5kkkYoDYWrX1UqFEk+HY=', 'Content-Length': '0'} 2022-09-02T04:45:04.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d55547f6ce7d9db6-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:04.181 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.182 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.183 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.184 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.184 DEBUG:boto:Method: GET 2022-09-02T04:45:04.185 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.186 DEBUG:boto:Data: 2022-09-02T04:45:04.187 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.188 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.189 DEBUG:boto:Params: {} 2022-09-02T04:45:04.189 DEBUG:boto:Token: None 2022-09-02T04:45:04.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c8a233fcff83f03-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.194 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.195 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.196 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.196 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.197 DEBUG:boto:Method: GET 2022-09-02T04:45:04.198 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.198 DEBUG:boto:Data: 2022-09-02T04:45:04.199 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.200 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.200 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.201 DEBUG:boto:Params: {} 2022-09-02T04:45:04.201 DEBUG:boto:Token: None 2022-09-02T04:45:04.202 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ea512ef94690cee-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.207 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.207 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.208 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.209 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.209 DEBUG:boto:Method: GET 2022-09-02T04:45:04.210 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.210 DEBUG:boto:Data: 2022-09-02T04:45:04.212 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.213 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.214 DEBUG:boto:Params: {} 2022-09-02T04:45:04.215 DEBUG:boto:Token: None 2022-09-02T04:45:04.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ce478f9e40cb92-0063118a50-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.220 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.221 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.222 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.223 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.223 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.224 DEBUG:boto:Method: GET 2022-09-02T04:45:04.224 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.226 DEBUG:boto:Data: 2022-09-02T04:45:04.227 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.229 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.229 DEBUG:boto:Params: {} 2022-09-02T04:45:04.230 DEBUG:boto:Token: None 2022-09-02T04:45:04.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085aad51b8e710119-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.236 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.237 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.238 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.239 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.239 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:04.240 DEBUG:boto:Data: 2022-09-02T04:45:04.240 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.243 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.244 DEBUG:boto:Params: {} 2022-09-02T04:45:04.245 DEBUG:boto:Token: None 2022-09-02T04:45:04.246 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj 2022-09-02T04:45:04.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4= 2022-09-02T04:45:04.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4=', 'Content-Length': '0'} 2022-09-02T04:45:04.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e06fc42291b011d2-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.251 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.252 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.253 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.253 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:04.254 DEBUG:boto:Data: 2022-09-02T04:45:04.255 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.257 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.259 DEBUG:boto:Params: {} 2022-09-02T04:45:04.261 DEBUG:boto:Token: None 2022-09-02T04:45:04.262 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj 2022-09-02T04:45:04.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4= 2022-09-02T04:45:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4=', 'Content-Length': '0'} 2022-09-02T04:45:04.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cdd86dd140e6eea-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:04.273 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.274 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.275 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.276 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.278 DEBUG:boto:Method: GET 2022-09-02T04:45:04.279 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.280 DEBUG:boto:Data: 2022-09-02T04:45:04.281 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.282 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.283 DEBUG:boto:Params: {} 2022-09-02T04:45:04.284 DEBUG:boto:Token: None 2022-09-02T04:45:04.284 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5177fa29f1e7789-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.305 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.306 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.307 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.308 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.309 DEBUG:boto:Method: GET 2022-09-02T04:45:04.310 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.311 DEBUG:boto:Data: 2022-09-02T04:45:04.312 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.314 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.315 DEBUG:boto:Params: {} 2022-09-02T04:45:04.316 DEBUG:boto:Token: None 2022-09-02T04:45:04.317 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003240accd1f1f9119-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.325 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.326 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.327 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.331 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.332 DEBUG:boto:Method: GET 2022-09-02T04:45:04.332 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.333 DEBUG:boto:Data: 2022-09-02T04:45:04.334 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.335 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.336 DEBUG:boto:Params: {} 2022-09-02T04:45:04.337 DEBUG:boto:Token: None 2022-09-02T04:45:04.339 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004047520c139b23ff-0063118a50-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.387 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.389 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.390 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.391 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.392 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.393 DEBUG:boto:Method: GET 2022-09-02T04:45:04.395 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.396 DEBUG:boto:Data: 2022-09-02T04:45:04.400 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.401 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.402 DEBUG:boto:Params: {} 2022-09-02T04:45:04.403 DEBUG:boto:Token: None 2022-09-02T04:45:04.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da078e4ee404fff7-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.414 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:04.417 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.419 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.420 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.421 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.422 DEBUG:boto:Method: GET 2022-09-02T04:45:04.423 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.424 DEBUG:boto:Data: 2022-09-02T04:45:04.425 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.427 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.428 DEBUG:boto:Params: {} 2022-09-02T04:45:04.429 DEBUG:boto:Token: None 2022-09-02T04:45:04.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw= 2022-09-02T04:45:04.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw=', 'Content-Length': '0'} 2022-09-02T04:45:04.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e754b67737df62d1-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.441 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.442 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.443 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.444 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.451 DEBUG:boto:Method: GET 2022-09-02T04:45:04.453 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.454 DEBUG:boto:Data: 2022-09-02T04:45:04.455 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.458 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.459 DEBUG:boto:Params: {} 2022-09-02T04:45:04.460 DEBUG:boto:Token: None 2022-09-02T04:45:04.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw= 2022-09-02T04:45:04.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw=', 'Content-Length': '0'} 2022-09-02T04:45:04.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a927f582585832d-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.465 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.474 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.475 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.476 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.477 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.478 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.484 DEBUG:boto:Data: 2022-09-02T04:45:04.485 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.487 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.488 DEBUG:boto:Params: {} 2022-09-02T04:45:04.489 DEBUG:boto:Token: None 2022-09-02T04:45:04.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v0qnuozXbdGotsvRUBPTRHe2TJw= 2022-09-02T04:45:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v0qnuozXbdGotsvRUBPTRHe2TJw=', 'Content-Length': '0'} 2022-09-02T04:45:04.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000094512570659fc7b2-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.498 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.499 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.501 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.502 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.503 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.503 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.504 DEBUG:boto:Data: 2022-09-02T04:45:04.505 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.507 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.508 DEBUG:boto:Params: {} 2022-09-02T04:45:04.509 DEBUG:boto:Token: None 2022-09-02T04:45:04.510 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v0qnuozXbdGotsvRUBPTRHe2TJw= 2022-09-02T04:45:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v0qnuozXbdGotsvRUBPTRHe2TJw=', 'Content-Length': '0'} 2022-09-02T04:45:04.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f17a526365888517-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:04.517 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.518 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.519 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.519 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.520 DEBUG:boto:Method: GET 2022-09-02T04:45:04.520 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.521 DEBUG:boto:Data: 2022-09-02T04:45:04.521 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.523 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.525 DEBUG:boto:Params: {} 2022-09-02T04:45:04.526 DEBUG:boto:Token: None 2022-09-02T04:45:04.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw= 2022-09-02T04:45:04.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw=', 'Content-Length': '0'} 2022-09-02T04:45:04.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf04ae781ca4e975-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.535 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.536 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.537 DEBUG:boto:path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.549 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.561 DEBUG:boto:Method: GET 2022-09-02T04:45:04.562 DEBUG:boto:Path: /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.562 DEBUG:boto:Data: 2022-09-02T04:45:04.563 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.563 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.564 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.564 DEBUG:boto:Params: {} 2022-09-02T04:45:04.565 DEBUG:boto:Token: None 2022-09-02T04:45:04.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=null 2022-09-02T04:45:04.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw= 2022-09-02T04:45:04.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KAVB5X9ioGMMFlvMmYfXwUyOJGw=', 'Content-Length': '0'} 2022-09-02T04:45:04.575 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:42:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041df4268c985f5de-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.577 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.579 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.580 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.581 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.582 DEBUG:boto:Method: GET 2022-09-02T04:45:04.583 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.584 DEBUG:boto:Data: 2022-09-02T04:45:04.584 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.586 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.586 DEBUG:boto:Params: {} 2022-09-02T04:45:04.587 DEBUG:boto:Token: None 2022-09-02T04:45:04.588 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf51866945936ef-0063118a50-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.603 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.605 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.607 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.609 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.610 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.610 DEBUG:boto:Method: GET 2022-09-02T04:45:04.611 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.612 DEBUG:boto:Data: 2022-09-02T04:45:04.613 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.615 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.615 DEBUG:boto:Params: {} 2022-09-02T04:45:04.617 DEBUG:boto:Token: None 2022-09-02T04:45:04.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98a0d6da6caabd6-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.623 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.624 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.625 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.625 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.626 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:04.626 DEBUG:boto:Data: 2022-09-02T04:45:04.627 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.628 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.628 DEBUG:boto:Params: {} 2022-09-02T04:45:04.629 DEBUG:boto:Token: None 2022-09-02T04:45:04.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj 2022-09-02T04:45:04.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4= 2022-09-02T04:45:04.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4=', 'Content-Length': '0'} 2022-09-02T04:45:04.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e02f5393058b626-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.635 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.635 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.636 DEBUG:boto:Method: HEAD 2022-09-02T04:45:04.636 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:04.637 DEBUG:boto:Data: 2022-09-02T04:45:04.637 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.638 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.638 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.639 DEBUG:boto:Params: {} 2022-09-02T04:45:04.640 DEBUG:boto:Token: None 2022-09-02T04:45:04.640 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj 2022-09-02T04:45:04.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4= 2022-09-02T04:45:04.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I0IAGFfd3LSkJkbJ4K0YtIAE6d4=', 'Content-Length': '0'} 2022-09-02T04:45:04.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b5760a43db791c0-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:45:04.646 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.646 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.647 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.649 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.650 DEBUG:boto:Method: GET 2022-09-02T04:45:04.651 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.652 DEBUG:boto:Data: 2022-09-02T04:45:04.653 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.655 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.657 DEBUG:boto:Params: {} 2022-09-02T04:45:04.658 DEBUG:boto:Token: None 2022-09-02T04:45:04.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057a7b5c928820941-0063118a50-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.674 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:04.675 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:04.676 DEBUG:boto:path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.677 DEBUG:boto:auth_path=/xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.678 DEBUG:boto:Method: GET 2022-09-02T04:45:04.679 DEBUG:boto:Path: /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.681 DEBUG:boto:Data: 2022-09-02T04:45:04.681 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.683 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.683 DEBUG:boto:Params: {} 2022-09-02T04:45:04.685 DEBUG:boto:Token: None 2022-09-02T04:45:04.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/obj?versionId=mjwVyZH3Y4RD68IKdN8-LP-goi9oa98 2022-09-02T04:45:04.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA= 2022-09-02T04:45:04.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CvU7nu/HPV1g2pZvX3RQHS5wFOA=', 'Content-Length': '0'} 2022-09-02T04:45:04.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:43:06 GMT'), ('x-amz-version-id', 'mjwVyZH3Y4RD68IKdN8-LP-goi9oa98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079fbb13b0e74316b-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.692 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.693 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.693 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.694 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.694 DEBUG:boto:Method: GET 2022-09-02T04:45:04.695 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.696 DEBUG:boto:Data: 2022-09-02T04:45:04.696 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:45:04.697 DEBUG:boto:Port: 8001 2022-09-02T04:45:04.698 DEBUG:boto:Params: {} 2022-09-02T04:45:04.699 DEBUG:boto:Token: None 2022-09-02T04:45:04.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddbe758f8f3a1f91-0063118a50-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.709 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.710 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.712 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.713 DEBUG:boto:path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.714 DEBUG:boto:auth_path=/xljbkf-20/?prefix=obj 2022-09-02T04:45:04.715 DEBUG:boto:Method: GET 2022-09-02T04:45:04.716 DEBUG:boto:Path: /xljbkf-20/?prefix=obj 2022-09-02T04:45:04.718 DEBUG:boto:Data: 2022-09-02T04:45:04.719 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:45:04.721 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.722 DEBUG:boto:Params: {} 2022-09-02T04:45:04.723 DEBUG:boto:Token: None 2022-09-02T04:45:04.727 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/ 2022-09-02T04:45:04.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg= 2022-09-02T04:45:04.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Hh0IcVclyQpln7Q59hsrIK1/sg=', 'Content-Length': '0'} 2022-09-02T04:45:04.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058f71f48ac72c7f4-0063118a50-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.735 DEBUG:boto:b'xljbkf-20obj1000falseobj2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:45:04.736 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-20 zones={test-zone3, test-zone2} 2022-09-02T04:45:04.738 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:04.739 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:04.739 DEBUG:boto:path=/xljbkf-20/?versioning 2022-09-02T04:45:04.740 DEBUG:boto:auth_path=/xljbkf-20/?versioning 2022-09-02T04:45:04.741 DEBUG:boto:Method: PUT 2022-09-02T04:45:04.742 DEBUG:boto:Path: /xljbkf-20/?versioning 2022-09-02T04:45:04.743 DEBUG:boto:Data: Suspended Disabled 2022-09-02T04:45:04.744 DEBUG:boto:Headers: {} 2022-09-02T04:45:04.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:04.745 DEBUG:boto:Port: 8000 2022-09-02T04:45:04.746 DEBUG:boto:Params: {} 2022-09-02T04:45:04.747 DEBUG:boto:Token: None 2022-09-02T04:45:04.748 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:45:04 GMT /xljbkf-20/?versioning 2022-09-02T04:45:04.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wXCCXa9cC8qYBqkD10CWkknLku8= 2022-09-02T04:45:04.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wXCCXa9cC8qYBqkD10CWkknLku8=', 'Content-Length': '230'} 2022-09-02T04:45:04.763 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b283220cb4570168-0063118a50-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:04.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:04.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:04.768 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:45:04.825 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:04.826 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:04.855 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:45:04.867 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:45:04.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:04.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:45:04.869 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:45:04.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:04.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:04.871 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:45:04.871 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:45:04.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:04.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:04.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093904.761841_157.1", 2022-09-02T04:45:04.873 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:45:04.761841Z" 2022-09-02T04:45:04.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:04.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:04.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:45:04.875 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:45:04.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:45:04.876 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:45:04.890 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:04.891 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:45:04.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:04.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:04.894 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:45:04.958 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:04.961 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:04.989 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:04.985+0000 7f6920ae2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:04.990 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:04.986+0000 7f6920ae2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:04.998 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:45:04.999 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:45:04.999 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:45:04.999 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:45:05.000 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:45:05.000 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:45:05.001 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:05.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:05.002 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:45:05.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:05.002 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:45:05.002 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:05.003 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:05.003 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:45:05.003 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:05.003 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:45:05.004 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:05.005 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:45:05.005 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:05.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:05.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:05.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:05.006 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:45:05.006 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:05.007 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:05.007 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:45:05.007 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:05.007 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:05.008 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:05.008 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:45:05.009 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:05.009 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:05.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:05.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:05.010 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:45:05.010 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:05.010 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:05.010 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093904.758354_156.1", 2022-09-02T04:45:05.011 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:05.011 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:05.011 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:05.011 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.758354Z", 2022-09-02T04:45:05.012 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:05.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:05.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:05.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:05.013 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:45:05.013 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:05.013 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:05.014 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:45:05.014 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:05.014 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:05.014 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:05.015 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:45:05.015 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:05.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:05.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:05.016 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:45:05.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:05.017 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:45:05.017 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:45:05.017 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:45:05.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:05.018 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:45:04Z" 2022-09-02T04:45:05.018 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:45:05.020 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093904.758354_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:45:04.758354Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:45:04Z" } 2022-09-02T04:45:05.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093904.758354_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:45:04.758354Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:45:05.021 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:05.022 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.758354_156.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:05.022 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662093904.761841_157.1 target=1_1662093904.758354_156.1 2022-09-02T04:45:10.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:10.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:10.026 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:45:10.086 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:10.087 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:10.138 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:10.140+0000 7effd460b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:10.139 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:10.141+0000 7effd460b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:10.156 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:45:10.156 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:45:10.157 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:45:10.157 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:45:10.159 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:45:10.160 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:45:10.160 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.161 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:45:10.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.161 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:45:10.162 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.163 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.163 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:45:10.163 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.164 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:45:10.164 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.165 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:45:10.165 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.166 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.167 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:45:10.167 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.167 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.168 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:45:10.168 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.168 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:10.168 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.169 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:45:10.169 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.171 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:45:10.171 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.171 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.172 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093904.761841_157.1", 2022-09-02T04:45:10.172 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.173 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:10.173 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.173 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.761841Z", 2022-09-02T04:45:10.174 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.174 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.175 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:45:10.175 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.176 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.176 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:45:10.177 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.178 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:10.178 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.178 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:45:10.178 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.179 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:45:10.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.180 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:45:10.180 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:45:10.180 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:45:10.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.181 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:45:10Z" 2022-09-02T04:45:10.182 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:45:10.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093904.761841_157.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:45:04.761841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:45:10Z" } 2022-09-02T04:45:10.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093904.761841_157.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:45:04.761841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:45:10.184 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:10.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:10.186 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:45:10.186 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:45:10.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:10.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:10.188 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:45:10.245 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:10.246 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:10.292 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:10.292+0000 7fa75eb80600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:10.292 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:10.293+0000 7fa75eb80600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:10.330 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:45:10.331 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:45:10.332 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:45:10.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:45:10.333 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:45:10.333 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:45:10.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:45:10.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:10.335 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:45:10.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:10.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:45:10.336 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:45:10.336 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:45:10.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:45:10.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:45:10.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:45:10.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:45:10.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:45:10.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:45:10.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:45:10.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:10.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:10.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:45:10.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:45:10.342 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:45:10.342 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:45:10.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:45:10.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:45:10.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:45:10.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:45:10.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:45:10.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:45:10.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:10.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:10.346 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:45:10.346 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:45:10.346 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:45:10.346 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093904.761841_157.1", 2022-09-02T04:45:10.347 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:45:10.347 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:45:10.347 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:45:10.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:45:04.761841Z", 2022-09-02T04:45:10.348 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:45:10.348 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:45:10.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:10.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:45:10.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:45:10.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:45:10.349 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:45:10.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:45:10.350 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:45:10.350 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:45:10.350 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:45:10.350 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:45:10.351 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:45:10.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:45:10.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:45:10.352 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:45:10.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:10.352 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:45:10.352 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:45:10.352 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:45:10.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:45:10.353 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:45:10Z" 2022-09-02T04:45:10.353 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:45:10.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093904.761841_157.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:45:04.761841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:45:10Z" } 2022-09-02T04:45:10.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093904.761841_157.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:45:04.761841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:45:10.356 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:10.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:10.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:45:10.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:45:10.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:10.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:45:10.359 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:45:10.411 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:10.412 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:10.449 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:10.450+0000 7f14166fe600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:10.450 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:10.451+0000 7f14166fe600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:45:10.461 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:45:10.462 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:45:10.463 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:45:10.463 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:45:10.463 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:45:10.464 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:45:10.464 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.465 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:45:10.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.466 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:45:10.466 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.467 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.467 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:45:10.467 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:45:10.468 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.468 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:45:10.469 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.469 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.470 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:45:10.470 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.471 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.471 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:45:10.471 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.471 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:10.472 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.472 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:45:10.472 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.474 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:45:10.474 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.475 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.475 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093904.761841_157.1", 2022-09-02T04:45:10.476 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.476 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:10.476 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.477 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.761841Z", 2022-09-02T04:45:10.477 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:45:10.478 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:45:10.478 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:45:10.479 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:45:10.479 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:45:10.480 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:45:10.480 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:45:10.480 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:45:10.481 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:45:10.481 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:45:10.481 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:45:10.482 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:45:10.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.483 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:45:10.483 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:45:10.483 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:45:10.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:45:10.484 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:45:10Z" 2022-09-02T04:45:10.484 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:45:10.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093904.761841_157.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:45:04.761841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:45:10Z" } 2022-09-02T04:45:10.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093904.761841_157.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:45:04.761841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:45:10.487 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:10.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093904.761841_157.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:45:10.489 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:45:10.489 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:45:10.490 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:45:10.490 DEBUG:boto:Method: HEAD 2022-09-02T04:45:10.491 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:45:10.492 DEBUG:boto:Data: 2022-09-02T04:45:10.492 DEBUG:boto:Headers: {} 2022-09-02T04:45:10.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:10.493 DEBUG:boto:Port: 8000 2022-09-02T04:45:10.494 DEBUG:boto:Params: {} 2022-09-02T04:45:10.495 DEBUG:boto:Token: None 2022-09-02T04:45:10.495 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:45:10 GMT /xljbkf-20/ 2022-09-02T04:45:10.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U1xiZNy9RjnBUHsqc1uIFiAQyLg= 2022-09-02T04:45:10.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:45:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U1xiZNy9RjnBUHsqc1uIFiAQyLg=', 'Content-Length': '0'} 2022-09-02T04:45:10.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d61db05a15d67421-0063118a56-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:45:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:10.502 DEBUG:boto:path=/xljbkf-20/obj 2022-09-02T04:45:10.503 DEBUG:boto:auth_path=/xljbkf-20/obj 2022-09-02T04:45:10.504 DEBUG:boto:Method: PUT 2022-09-02T04:45:10.505 DEBUG:boto:Path: /xljbkf-20/obj 2022-09-02T04:45:10.506 DEBUG:boto:Data: 2022-09-02T04:45:10.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:45:10.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:45:10.508 DEBUG:boto:Port: 8000 2022-09-02T04:45:10.509 DEBUG:boto:Params: {} 2022-09-02T04:45:10.509 DEBUG:boto:Token: None 2022-09-02T04:45:10.510 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:45:10 GMT /xljbkf-20/obj 2022-09-02T04:45:10.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lyWcoyIontsMqgy3daWUuE1D+Lk= 2022-09-02T04:45:10.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:45:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lyWcoyIontsMqgy3daWUuE1D+Lk='} 2022-09-02T04:45:10.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0f41c6b711eb0ce-0063118a56-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:45:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:45:10.529 DEBUG:rgw_multi.tests:created null version id=None 2022-09-02T04:45:10.529 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-20 2022-09-02T04:45:10.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:10.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:10.531 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:10.587 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:10.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:10.632 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:10.633+0000 7fc995bd3600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:10.632 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:45:10.632 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:15.633 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:15.635+0000 7fc995bd3600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:15.634 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:45:15.634 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:20.635 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:20.635+0000 7fc995bd3600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:20.636 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:45:20.636 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:25.636 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:25.636+0000 7fc995bd3600 1 bucket sync caught up with source: 2022-09-02T04:45:25.637 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:25.637 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:25.637 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:25.636+0000 7fc995bd3600 0 bucket checkpoint complete 2022-09-02T04:45:25.659 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-20 2022-09-02T04:45:25.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:25.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:25.664 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:25.731 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:25.731 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:25.779 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:25.779+0000 7f3438900600 1 bucket sync caught up with source: 2022-09-02T04:45:25.779 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:25.780 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:25.780 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:25.779+0000 7f3438900600 0 bucket checkpoint complete 2022-09-02T04:45:25.797 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:45:25.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:25.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:25.800 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:25.863 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:25.864 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:25.899 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:25.900+0000 7fc184d83600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:25.900 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-09-02T04:45:25.900 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:30.901 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:30.901+0000 7fc184d83600 1 bucket sync caught up with source: 2022-09-02T04:45:30.901 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:30.902 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-09-02T04:45:30.902 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:45:30.901+0000 7fc184d83600 0 bucket checkpoint complete 2022-09-02T04:45:30.917 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-20 2022-09-02T04:45:30.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:30.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:30.920 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:30.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:30.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:31.024 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:31.025+0000 7f86e173d600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:31.024 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000004.77.9, , ] 2022-09-02T04:45:31.025 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.79.17, , ] 2022-09-02T04:45:36.026 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:36.026+0000 7f86e173d600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:36.026 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000004.77.9, , ] 2022-09-02T04:45:36.026 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.79.17, , ] 2022-09-02T04:45:41.027 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:41.028+0000 7f86e173d600 1 bucket sync caught up with source: 2022-09-02T04:45:41.028 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000006.82.14, , ] 2022-09-02T04:45:41.028 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.79.17, , ] 2022-09-02T04:45:41.028 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:41.028+0000 7f86e173d600 0 bucket checkpoint complete 2022-09-02T04:45:41.043 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-20 2022-09-02T04:45:41.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:41.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:41.045 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:41.108 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:41.109 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:41.160 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:41.161+0000 7f6610308600 1 bucket sync caught up with source: 2022-09-02T04:45:41.161 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000006.82.14, , ] 2022-09-02T04:45:41.161 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000006.82.14, , ] 2022-09-02T04:45:41.162 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:41.161+0000 7f6610308600 0 bucket checkpoint complete 2022-09-02T04:45:41.183 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:45:41.184 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-20 2022-09-02T04:45:41.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:41.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:45:41.187 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:45:41.246 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:45:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:45:41.285 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:41.286+0000 7f667755a600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:41.286 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.73.17, , ] 2022-09-02T04:45:41.286 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:45:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:46.287+0000 7f667755a600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:46.289 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.73.17, , ] 2022-09-02T04:45:46.289 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:45:51.288 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:51.289+0000 7f667755a600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.73.17, , ] 2022-09-02T04:45:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:45:56.290 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:45:56.290+0000 7f667755a600 1 waiting for incremental sync to catch up: 2022-09-02T04:45:56.291 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.73.17, , ] 2022-09-02T04:45:56.292 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:46:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:01.292+0000 7f667755a600 1 bucket sync caught up with source: 2022-09-02T04:46:01.293 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:46:01.293 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:46:01.295 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:01.292+0000 7f667755a600 0 bucket checkpoint complete 2022-09-02T04:46:01.311 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-20 2022-09-02T04:46:01.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:01.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:01.313 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:01.382 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:01.383 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:01.428 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:01.428+0000 7fd1b8277600 1 bucket sync caught up with source: 2022-09-02T04:46:01.429 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:46:01.429 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000006.77.14, , ] 2022-09-02T04:46:01.430 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:01.428+0000 7fd1b8277600 0 bucket checkpoint complete 2022-09-02T04:46:01.447 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-20 2022-09-02T04:46:01.448 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-20 zones={test-zone2, test-zone1} 2022-09-02T04:46:01.449 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:46:01.450 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:46:01.452 DEBUG:boto:Method: HEAD 2022-09-02T04:46:01.453 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:46:01.454 DEBUG:boto:Data: 2022-09-02T04:46:01.455 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:01.458 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.459 DEBUG:boto:Params: {} 2022-09-02T04:46:01.460 DEBUG:boto:Token: None 2022-09-02T04:46:01.461 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-20/ 2022-09-02T04:46:01.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XX097KyrnKL/QuoNomrTArTgKbY= 2022-09-02T04:46:01.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XX097KyrnKL/QuoNomrTArTgKbY=', 'Content-Length': '0'} 2022-09-02T04:46:01.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000893a7dea19597bf1-0063118a89-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.469 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:46:01.470 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:46:01.470 DEBUG:boto:Method: HEAD 2022-09-02T04:46:01.471 DEBUG:boto:Path: /xljbkf-20/ 2022-09-02T04:46:01.471 DEBUG:boto:Data: 2022-09-02T04:46:01.472 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:01.473 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.473 DEBUG:boto:Params: {} 2022-09-02T04:46:01.474 DEBUG:boto:Token: None 2022-09-02T04:46:01.476 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-20/ 2022-09-02T04:46:01.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XX097KyrnKL/QuoNomrTArTgKbY= 2022-09-02T04:46:01.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XX097KyrnKL/QuoNomrTArTgKbY=', 'Content-Length': '0'} 2022-09-02T04:46:01.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086fb382d125f9a01-0063118a89-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:46:01.482 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:46:01.484 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:46:01.485 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.486 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.487 DEBUG:boto:Method: GET 2022-09-02T04:46:01.488 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.488 DEBUG:boto:Data: 2022-09-02T04:46:01.489 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:01.491 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.492 DEBUG:boto:Params: {} 2022-09-02T04:46:01.495 DEBUG:boto:Token: None 2022-09-02T04:46:01.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-20/?versions 2022-09-02T04:46:01.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA= 2022-09-02T04:46:01.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA=', 'Content-Length': '0'} 2022-09-02T04:46:01.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acfd7cceb8267dd2-0063118a89-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.503 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98false2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:01.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:01.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:01.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:46:01.508 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:46:01.508 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:46:01.509 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.510 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.511 DEBUG:boto:Method: GET 2022-09-02T04:46:01.512 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.514 DEBUG:boto:Data: 2022-09-02T04:46:01.514 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:01.515 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.516 DEBUG:boto:Params: {} 2022-09-02T04:46:01.517 DEBUG:boto:Token: None 2022-09-02T04:46:01.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-20/?versions 2022-09-02T04:46:01.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA= 2022-09-02T04:46:01.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA=', 'Content-Length': '0'} 2022-09-02T04:46:01.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b74efed8694a36f-0063118a89-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.526 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:45:10.516Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98false2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:01.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:01.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:01.528 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:46:01.529 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:46:01.529 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.530 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.530 DEBUG:boto:Method: GET 2022-09-02T04:46:01.531 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.531 DEBUG:boto:Data: 2022-09-02T04:46:01.532 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:01.533 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.534 DEBUG:boto:Params: {} 2022-09-02T04:46:01.534 DEBUG:boto:Token: None 2022-09-02T04:46:01.535 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-20/?versions 2022-09-02T04:46:01.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA= 2022-09-02T04:46:01.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA=', 'Content-Length': '0'} 2022-09-02T04:46:01.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c762fc14c99eb86e-0063118a89-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.542 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:42:34.196Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98false2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:01.543 DEBUG:boto:path=/xljbkf-20/ 2022-09-02T04:46:01.543 DEBUG:boto:auth_path=/xljbkf-20/ 2022-09-02T04:46:01.544 DEBUG:boto:path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.545 DEBUG:boto:auth_path=/xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.546 DEBUG:boto:Method: GET 2022-09-02T04:46:01.546 DEBUG:boto:Path: /xljbkf-20/?versions&max-keys=999 2022-09-02T04:46:01.547 DEBUG:boto:Data: 2022-09-02T04:46:01.547 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:01.548 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.549 DEBUG:boto:Params: {} 2022-09-02T04:46:01.549 DEBUG:boto:Token: None 2022-09-02T04:46:01.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-20/?versions 2022-09-02T04:46:01.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA= 2022-09-02T04:46:01.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7YbAhW44IJAvvv1NrKs2/4cPfYA=', 'Content-Length': '0'} 2022-09-02T04:46:01.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7fcf4f3a07a747-0063118a89-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.603 DEBUG:boto:b'xljbkf-20999falseobjnulltrue2022-09-02T04:45:10.516Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmjwVyZH3Y4RD68IKdN8-LP-goi9oa98false2022-09-02T04:43:06.835Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:01.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:01.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2022-09-02T04:46:01.620 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-21 2022-09-02T04:46:01.621 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:01.621 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:01.623 DEBUG:boto:Method: PUT 2022-09-02T04:46:01.623 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:01.624 DEBUG:boto:Data: 2022-09-02T04:46:01.625 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:01.627 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.628 DEBUG:boto:Params: {} 2022-09-02T04:46:01.629 DEBUG:boto:Token: None 2022-09-02T04:46:01.630 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-21/ 2022-09-02T04:46:01.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f1DLrrBwUlUb4WrY/RCuXcWry0U= 2022-09-02T04:46:01.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f1DLrrBwUlUb4WrY/RCuXcWry0U=', 'Content-Length': '0'} 2022-09-02T04:46:01.650 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3f4a847f6d8e6b6-0063118a89-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.651 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-22 2022-09-02T04:46:01.652 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:46:01.652 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:46:01.653 DEBUG:boto:Method: PUT 2022-09-02T04:46:01.654 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:46:01.654 DEBUG:boto:Data: 2022-09-02T04:46:01.655 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:01.656 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.657 DEBUG:boto:Params: {} 2022-09-02T04:46:01.658 DEBUG:boto:Token: None 2022-09-02T04:46:01.658 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-22/ 2022-09-02T04:46:01.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Eei32a4EVYnFyHuBlr8r9gTNe44= 2022-09-02T04:46:01.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Eei32a4EVYnFyHuBlr8r9gTNe44=', 'Content-Length': '0'} 2022-09-02T04:46:01.697 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003fb007cffce276bf-0063118a89-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.698 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-23 2022-09-02T04:46:01.699 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:46:01.699 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:46:01.700 DEBUG:boto:Method: PUT 2022-09-02T04:46:01.700 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:46:01.701 DEBUG:boto:Data: 2022-09-02T04:46:01.701 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:01.702 DEBUG:boto:Port: 8001 2022-09-02T04:46:01.703 DEBUG:boto:Params: {} 2022-09-02T04:46:01.704 DEBUG:boto:Token: None 2022-09-02T04:46:01.705 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-23/ 2022-09-02T04:46:01.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcvLPJMxB2pW3WJNV3ZmTZWdtOE= 2022-09-02T04:46:01.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcvLPJMxB2pW3WJNV3ZmTZWdtOE=', 'Content-Length': '0'} 2022-09-02T04:46:01.736 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa9bcb0c89f2d34d-0063118a89-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.737 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:01.737 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:01.738 DEBUG:boto:path=/xljbkf-21/?versioning 2022-09-02T04:46:01.738 DEBUG:boto:auth_path=/xljbkf-21/?versioning 2022-09-02T04:46:01.739 DEBUG:boto:Method: PUT 2022-09-02T04:46:01.739 DEBUG:boto:Path: /xljbkf-21/?versioning 2022-09-02T04:46:01.740 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:46:01.740 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:01.741 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.742 DEBUG:boto:Params: {} 2022-09-02T04:46:01.742 DEBUG:boto:Token: None 2022-09-02T04:46:01.743 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-21/?versioning 2022-09-02T04:46:01.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sgiSWCoJ1HjxkSuXHdjD8JyK3W0= 2022-09-02T04:46:01.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sgiSWCoJ1HjxkSuXHdjD8JyK3W0=', 'Content-Length': '228'} 2022-09-02T04:46:01.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8b6e54b5c3662ec-0063118a89-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.753 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:46:01.754 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:46:01.755 DEBUG:boto:path=/xljbkf-22/?versioning 2022-09-02T04:46:01.755 DEBUG:boto:auth_path=/xljbkf-22/?versioning 2022-09-02T04:46:01.756 DEBUG:boto:Method: PUT 2022-09-02T04:46:01.757 DEBUG:boto:Path: /xljbkf-22/?versioning 2022-09-02T04:46:01.758 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:46:01.759 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:01.761 DEBUG:boto:Port: 8000 2022-09-02T04:46:01.762 DEBUG:boto:Params: {} 2022-09-02T04:46:01.763 DEBUG:boto:Token: None 2022-09-02T04:46:01.764 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-22/?versioning 2022-09-02T04:46:01.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ro//LzDyQjCD4DO+oGjBtQojUyQ= 2022-09-02T04:46:01.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ro//LzDyQjCD4DO+oGjBtQojUyQ=', 'Content-Length': '228'} 2022-09-02T04:46:01.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7eb761f8784fd09-0063118a89-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.782 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:46:01.782 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:46:01.783 DEBUG:boto:path=/xljbkf-23/?versioning 2022-09-02T04:46:01.784 DEBUG:boto:auth_path=/xljbkf-23/?versioning 2022-09-02T04:46:01.784 DEBUG:boto:Method: PUT 2022-09-02T04:46:01.785 DEBUG:boto:Path: /xljbkf-23/?versioning 2022-09-02T04:46:01.785 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:46:01.786 DEBUG:boto:Headers: {} 2022-09-02T04:46:01.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:01.787 DEBUG:boto:Port: 8001 2022-09-02T04:46:01.787 DEBUG:boto:Params: {} 2022-09-02T04:46:01.789 DEBUG:boto:Token: None 2022-09-02T04:46:01.790 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:46:01 GMT /xljbkf-23/?versioning 2022-09-02T04:46:01.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dFprWNv1JLGNMR13dDqu8P+G+Gs= 2022-09-02T04:46:01.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dFprWNv1JLGNMR13dDqu8P+G+Gs=', 'Content-Length': '228'} 2022-09-02T04:46:01.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c52aafd1d65261c9-0063118a89-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:01.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:01.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:01.805 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:46:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:01.900 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:46:01.919 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:46:01.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:01.920 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:46:01.920 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:46:01.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:01.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:01.921 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:46:01.921 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:46:01.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:01.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:01.922 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093961.799397_175.1", 2022-09-02T04:46:01.922 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:46:01.799397Z" 2022-09-02T04:46:01.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:01.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:01.923 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:46:01.923 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:46:01.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:46:01.924 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:46:01.925 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:01.926 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:46:01.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:01.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:01.929 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:46:01.992 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:01.992 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:02.037 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:02.036+0000 7fe943f80600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:02.038 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:02.037+0000 7fe943f80600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:02.048 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:46:02.049 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:46:02.050 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:46:02.050 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:46:02.050 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:46:02.050 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:46:02.051 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:02.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:02.051 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:46:02.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:02.052 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:46:02.052 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:02.052 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:02.052 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:46:02.053 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:02.053 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:46:02.053 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:02.053 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:46:02.053 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:02.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:02.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:02.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:02.055 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:46:02.056 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:02.056 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:02.058 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:46:02.059 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:02.059 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:02.059 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:02.060 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:46:02.060 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:02.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:02.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:02.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:02.062 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:46:02.063 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:02.064 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:02.065 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093961.748449_170.1", 2022-09-02T04:46:02.065 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:02.066 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:02.066 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:02.067 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.748449Z", 2022-09-02T04:46:02.067 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:02.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:02.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:02.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:02.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:46:02.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:02.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:02.071 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:46:02.071 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:02.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:02.072 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:02.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:46:02.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:02.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:02.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:02.075 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:46:02.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:02.076 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:46:02.076 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:46:02.077 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:46:02.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:02.078 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:46:02Z" 2022-09-02T04:46:02.079 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:46:02.082 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093961.748449_170.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:46:01.748449Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:46:02Z" } 2022-09-02T04:46:02.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093961.748449_170.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:46:01.748449Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:46:02.084 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:02.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.748449_170.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:02.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662093961.799397_175.1 target=1_1662093961.748449_170.1 2022-09-02T04:46:07.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:07.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:07.089 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:46:07.141 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:07.141 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:07.172 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:07.170+0000 7f5908d53600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:07.173 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:07.171+0000 7f5908d53600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:07.187 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:46:07.189 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:46:07.189 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:46:07.190 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:46:07.190 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:46:07.191 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:46:07.191 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.192 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:46:07.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.193 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:46:07.193 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.194 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.194 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:46:07.195 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.195 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:46:07.195 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.195 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:46:07.196 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.197 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:46:07.197 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.197 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.197 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:46:07.198 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.198 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:07.198 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.198 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:46:07.199 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.201 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:46:07.201 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.201 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.201 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093961.799397_175.1", 2022-09-02T04:46:07.202 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.202 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:07.202 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.202 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.799397Z", 2022-09-02T04:46:07.203 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.203 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.204 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:46:07.204 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.204 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.205 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:46:07.205 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.205 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:07.206 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.206 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:46:07.206 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.207 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:46:07.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.208 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:46:07.208 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:46:07.208 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:46:07.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.209 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:46:07Z" 2022-09-02T04:46:07.209 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:46:07.210 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093961.799397_175.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:46:01.799397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:46:07Z" } 2022-09-02T04:46:07.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093961.799397_175.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:46:01.799397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:46:07.212 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:07.213 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:07.213 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:46:07.214 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:46:07.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:07.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:07.216 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:46:07.266 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:07.267 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:07.312+0000 7f4819bb7600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:07.313+0000 7f4819bb7600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:07.326 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:46:07.327 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:46:07.328 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:46:07.328 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:46:07.329 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:46:07.329 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:46:07.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:46:07.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:07.331 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:46:07.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:07.332 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:46:07.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:46:07.333 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:46:07.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:46:07.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:46:07.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:46:07.336 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:46:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:46:07.337 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:46:07.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:46:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:07.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:46:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:46:07.340 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:46:07.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:46:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:46:07.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:46:07.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:46:07.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:46:07.344 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:46:07.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:46:07.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:07.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:07.345 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:46:07.345 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:46:07.345 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:46:07.345 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662093961.799397_175.1", 2022-09-02T04:46:07.346 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:46:07.346 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:46:07.346 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:46:07.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.799397Z", 2022-09-02T04:46:07.347 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:46:07.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:46:07.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:07.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:46:07.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:46:07.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:46:07.350 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:46:07.350 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:46:07.351 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:46:07.351 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:46:07.352 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:46:07.353 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:46:07.353 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:46:07.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:46:07.355 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:46:07.355 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:46:07.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:07.356 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:46:07.357 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:46:07.357 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:46:07.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:46:07.358 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:46:07Z" 2022-09-02T04:46:07.358 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:46:07.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093961.799397_175.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:46:01.799397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:46:07Z" } 2022-09-02T04:46:07.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093961.799397_175.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:46:01.799397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:46:07.362 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:07.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:07.363 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:46:07.364 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:46:07.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:07.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:46:07.367 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:46:07.418 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:07.418 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:07.454+0000 7f3fae19e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:07.455+0000 7f3fae19e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:46:07.466 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:46:07.467 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:46:07.467 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:46:07.467 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:46:07.468 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:46:07.468 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:46:07.469 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.470 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:46:07.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.471 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:46:07.471 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.471 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.472 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:46:07.472 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.472 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:46:07.473 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.473 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:46:07.473 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.474 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.475 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:46:07.475 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.475 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.476 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:46:07.476 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.476 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:07.477 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.477 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:46:07.478 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.480 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:46:07.480 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.481 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.482 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662093961.799397_175.1", 2022-09-02T04:46:07.482 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.799397Z", 2022-09-02T04:46:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.484 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:46:07.484 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:46:07.485 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:46:07.485 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:46:07.485 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:46:07.486 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:46:07.486 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:46:07.487 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:46:07.487 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:46:07.487 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:46:07.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:46:07.488 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:46:07.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.488 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:46:07.489 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:46:07.489 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:46:07.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:46:07.489 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:46:07Z" 2022-09-02T04:46:07.490 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:46:07.491 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662093961.799397_175.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:46:01.799397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:46:07Z" } 2022-09-02T04:46:07.492 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662093961.799397_175.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:46:01.799397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:46:07.493 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:07.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662093961.799397_175.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:46:07.494 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:46:07.494 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:07.495 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:07.495 DEBUG:boto:Method: HEAD 2022-09-02T04:46:07.496 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:07.496 DEBUG:boto:Data: 2022-09-02T04:46:07.497 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:07.498 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.498 DEBUG:boto:Params: {} 2022-09-02T04:46:07.499 DEBUG:boto:Token: None 2022-09-02T04:46:07.499 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-21/ 2022-09-02T04:46:07.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xAf4Aepyu72v1g5DeE3Q/M7MayE= 2022-09-02T04:46:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xAf4Aepyu72v1g5DeE3Q/M7MayE=', 'Content-Length': '0'} 2022-09-02T04:46:07.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e94a3013e60fe336-0063118a8f-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.504 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:07.504 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:07.505 DEBUG:boto:Method: PUT 2022-09-02T04:46:07.505 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:07.506 DEBUG:boto:Data: 2022-09-02T04:46:07.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:46:07.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:07.507 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.508 DEBUG:boto:Params: {} 2022-09-02T04:46:07.508 DEBUG:boto:Token: None 2022-09-02T04:46:07.509 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-21/obj 2022-09-02T04:46:07.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QtGpNr3XE9cUEdJN2YOfiWjiRFs= 2022-09-02T04:46:07.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QtGpNr3XE9cUEdJN2YOfiWjiRFs='} 2022-09-02T04:46:07.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-amz-request-id', 'tx0000081d15b42a839b96c-0063118a8f-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.535 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:07.535 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:07.536 DEBUG:boto:Method: HEAD 2022-09-02T04:46:07.536 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:07.537 DEBUG:boto:Data: 2022-09-02T04:46:07.537 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:07.538 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.539 DEBUG:boto:Params: {} 2022-09-02T04:46:07.539 DEBUG:boto:Token: None 2022-09-02T04:46:07.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-21/ 2022-09-02T04:46:07.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xAf4Aepyu72v1g5DeE3Q/M7MayE= 2022-09-02T04:46:07.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xAf4Aepyu72v1g5DeE3Q/M7MayE=', 'Content-Length': '0'} 2022-09-02T04:46:07.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6a7a51415f2c2eb-0063118a8f-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.544 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:07.545 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:07.545 DEBUG:boto:Method: DELETE 2022-09-02T04:46:07.546 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:07.547 DEBUG:boto:Data: 2022-09-02T04:46:07.547 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:07.548 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.549 DEBUG:boto:Params: {} 2022-09-02T04:46:07.549 DEBUG:boto:Token: None 2022-09-02T04:46:07.550 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-21/obj 2022-09-02T04:46:07.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ee3JL7Ld93wYZwsJDndJLc6uolw= 2022-09-02T04:46:07.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ee3JL7Ld93wYZwsJDndJLc6uolw=', 'Content-Length': '0'} 2022-09-02T04:46:07.558 DEBUG:boto:Response headers: [('x-amz-version-id', '1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000006a7ad68a234c575a-0063118a8f-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.559 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:46:07.560 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:46:07.561 DEBUG:boto:Method: HEAD 2022-09-02T04:46:07.562 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:46:07.563 DEBUG:boto:Data: 2022-09-02T04:46:07.564 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:07.566 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.566 DEBUG:boto:Params: {} 2022-09-02T04:46:07.567 DEBUG:boto:Token: None 2022-09-02T04:46:07.568 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-22/ 2022-09-02T04:46:07.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PPrpgr8Ei6SBLCwJjI1rmO8ymu4= 2022-09-02T04:46:07.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PPrpgr8Ei6SBLCwJjI1rmO8ymu4=', 'Content-Length': '0'} 2022-09-02T04:46:07.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a06fd1ee98c26c32-0063118a8f-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.573 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:46:07.574 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:46:07.575 DEBUG:boto:Method: PUT 2022-09-02T04:46:07.576 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:46:07.577 DEBUG:boto:Data: 2022-09-02T04:46:07.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:46:07.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:07.580 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.581 DEBUG:boto:Params: {} 2022-09-02T04:46:07.582 DEBUG:boto:Token: None 2022-09-02T04:46:07.583 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-22/obj 2022-09-02T04:46:07.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M6eiUWbXVSN3k65pSN5XgtVwUqU= 2022-09-02T04:46:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M6eiUWbXVSN3k65pSN5XgtVwUqU='} 2022-09-02T04:46:07.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-amz-request-id', 'tx00000c5310fce9b4897f2-0063118a8f-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.596 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:46:07.597 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:46:07.598 DEBUG:boto:Method: HEAD 2022-09-02T04:46:07.598 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:46:07.599 DEBUG:boto:Data: 2022-09-02T04:46:07.600 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:07.601 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.602 DEBUG:boto:Params: {} 2022-09-02T04:46:07.603 DEBUG:boto:Token: None 2022-09-02T04:46:07.604 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-22/ 2022-09-02T04:46:07.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PPrpgr8Ei6SBLCwJjI1rmO8ymu4= 2022-09-02T04:46:07.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PPrpgr8Ei6SBLCwJjI1rmO8ymu4=', 'Content-Length': '0'} 2022-09-02T04:46:07.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad7f64f4577f0b54-0063118a8f-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.610 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:46:07.611 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:46:07.612 DEBUG:boto:Method: DELETE 2022-09-02T04:46:07.613 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:46:07.614 DEBUG:boto:Data: 2022-09-02T04:46:07.614 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:07.616 DEBUG:boto:Port: 8000 2022-09-02T04:46:07.616 DEBUG:boto:Params: {} 2022-09-02T04:46:07.617 DEBUG:boto:Token: None 2022-09-02T04:46:07.617 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-22/obj 2022-09-02T04:46:07.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x7SLh4Lgd5h/QpJOYjfDoBETop8= 2022-09-02T04:46:07.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x7SLh4Lgd5h/QpJOYjfDoBETop8=', 'Content-Length': '0'} 2022-09-02T04:46:07.626 DEBUG:boto:Response headers: [('x-amz-version-id', 'AVj9vyprFEqLOBYKqTB.RaiKLGeZ8qA'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000086344f7b0fcd3fc1-0063118a8f-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.627 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:46:07.628 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:46:07.629 DEBUG:boto:Method: HEAD 2022-09-02T04:46:07.631 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:46:07.631 DEBUG:boto:Data: 2022-09-02T04:46:07.632 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:07.634 DEBUG:boto:Port: 8001 2022-09-02T04:46:07.635 DEBUG:boto:Params: {} 2022-09-02T04:46:07.636 DEBUG:boto:Token: None 2022-09-02T04:46:07.637 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-23/ 2022-09-02T04:46:07.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22BbrQqswuHp3r4R5eH8A2rbuqg= 2022-09-02T04:46:07.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22BbrQqswuHp3r4R5eH8A2rbuqg=', 'Content-Length': '0'} 2022-09-02T04:46:07.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baf8ba84535a84c6-0063118a8f-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.643 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:46:07.644 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:46:07.645 DEBUG:boto:Method: PUT 2022-09-02T04:46:07.645 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:46:07.646 DEBUG:boto:Data: 2022-09-02T04:46:07.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:46:07.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:07.649 DEBUG:boto:Port: 8001 2022-09-02T04:46:07.650 DEBUG:boto:Params: {} 2022-09-02T04:46:07.651 DEBUG:boto:Token: None 2022-09-02T04:46:07.654 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-23/obj 2022-09-02T04:46:07.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sAHFbK+oImQ+ErJ/tmnvBfqqNfA= 2022-09-02T04:46:07.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sAHFbK+oImQ+ErJ/tmnvBfqqNfA='} 2022-09-02T04:46:07.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-amz-request-id', 'tx000008b5ffbe9e7a9ea25-0063118a8f-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.669 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:46:07.670 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:46:07.671 DEBUG:boto:Method: HEAD 2022-09-02T04:46:07.672 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:46:07.672 DEBUG:boto:Data: 2022-09-02T04:46:07.673 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:07.675 DEBUG:boto:Port: 8001 2022-09-02T04:46:07.675 DEBUG:boto:Params: {} 2022-09-02T04:46:07.676 DEBUG:boto:Token: None 2022-09-02T04:46:07.676 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-23/ 2022-09-02T04:46:07.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22BbrQqswuHp3r4R5eH8A2rbuqg= 2022-09-02T04:46:07.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22BbrQqswuHp3r4R5eH8A2rbuqg=', 'Content-Length': '0'} 2022-09-02T04:46:07.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b86f7e7eddf85815-0063118a8f-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.681 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:46:07.681 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:46:07.682 DEBUG:boto:Method: DELETE 2022-09-02T04:46:07.683 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:46:07.683 DEBUG:boto:Data: 2022-09-02T04:46:07.684 DEBUG:boto:Headers: {} 2022-09-02T04:46:07.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:07.685 DEBUG:boto:Port: 8001 2022-09-02T04:46:07.686 DEBUG:boto:Params: {} 2022-09-02T04:46:07.687 DEBUG:boto:Token: None 2022-09-02T04:46:07.687 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:46:07 GMT /xljbkf-23/obj 2022-09-02T04:46:07.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hvbi+uVfx4J1jmqZfpDimagcDzU= 2022-09-02T04:46:07.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hvbi+uVfx4J1jmqZfpDimagcDzU=', 'Content-Length': '0'} 2022-09-02T04:46:07.695 DEBUG:boto:Response headers: [('x-amz-version-id', 'VLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdX'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000004257745a6dd2faa8-0063118a8f-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:07.696 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-21 2022-09-02T04:46:07.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:07.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:07.698 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:07.754 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:07.754 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:07.785 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:07.786+0000 7ff942390600 1 waiting to reach incremental sync.. 2022-09-02T04:46:12.787 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:12.787+0000 7ff942390600 1 waiting to reach incremental sync.. 2022-09-02T04:46:17.787 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:17.788+0000 7ff942390600 1 waiting to reach incremental sync.. 2022-09-02T04:46:22.789 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:22.789+0000 7ff942390600 1 bucket sync caught up with source: 2022-09-02T04:46:22.789 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.8.11, , ] 2022-09-02T04:46:22.789 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.8.11, , ] 2022-09-02T04:46:22.790 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:22.789+0000 7ff942390600 0 bucket checkpoint complete 2022-09-02T04:46:22.808 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-21 2022-09-02T04:46:22.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:22.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:22.811 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:22.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:22.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:22.927 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:22.927+0000 7f23b2777600 1 bucket sync caught up with source: 2022-09-02T04:46:22.927 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.11, , ] 2022-09-02T04:46:22.928 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.11, , ] 2022-09-02T04:46:22.928 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:22.927+0000 7f23b2777600 0 bucket checkpoint complete 2022-09-02T04:46:22.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-21 2022-09-02T04:46:22.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:22.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:22.961 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:23.026 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:23.026 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:23.080 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:23.076+0000 7f0e8f1c0600 1 bucket sync caught up with source: 2022-09-02T04:46:23.081 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.8.11, , ] 2022-09-02T04:46:23.082 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.8.11, , ] 2022-09-02T04:46:23.082 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:23.076+0000 7f0e8f1c0600 0 bucket checkpoint complete 2022-09-02T04:46:23.091 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-21 2022-09-02T04:46:23.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:23.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:23.094 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:23.152 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:23.213 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:23.213+0000 7f44ce51c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:28.214 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:28.214+0000 7f44ce51c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:33.214 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:33.215+0000 7f44ce51c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:38.216 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:38.216+0000 7f44ce51c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:43.221 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:43.217+0000 7f44ce51c600 1 bucket sync caught up with source: 2022-09-02T04:46:43.221 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.43.5, , ] 2022-09-02T04:46:43.221 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.43.5, , ] 2022-09-02T04:46:43.222 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:43.217+0000 7f44ce51c600 0 bucket checkpoint complete 2022-09-02T04:46:43.238 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-21 2022-09-02T04:46:43.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:43.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:43.241 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:43.302 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:43.303 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:43.354 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:43.354+0000 7f239f0eb600 1 bucket sync caught up with source: 2022-09-02T04:46:43.355 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.43.5, , ] 2022-09-02T04:46:43.355 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.43.5, , ] 2022-09-02T04:46:43.355 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:43.354+0000 7f239f0eb600 0 bucket checkpoint complete 2022-09-02T04:46:43.369 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-21 2022-09-02T04:46:43.370 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-21 2022-09-02T04:46:43.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:43.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:43.371 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:43.427 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:43.428 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:43.465 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:43.465+0000 7fd2e6dc3600 1 bucket sync caught up with source: 2022-09-02T04:46:43.465 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.69.5, , ] 2022-09-02T04:46:43.466 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.69.5, , ] 2022-09-02T04:46:43.466 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:46:43.465+0000 7fd2e6dc3600 0 bucket checkpoint complete 2022-09-02T04:46:43.477 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-21 2022-09-02T04:46:43.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:43.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:43.479 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:43.544 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:43.581 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:43.581+0000 7fb4fef5b600 1 bucket sync caught up with source: 2022-09-02T04:46:43.581 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.69.5, , ] 2022-09-02T04:46:43.581 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.69.5, , ] 2022-09-02T04:46:43.581 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:43.581+0000 7fb4fef5b600 0 bucket checkpoint complete 2022-09-02T04:46:43.595 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-21 2022-09-02T04:46:43.596 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-21 zones={test-zone2, test-zone1} 2022-09-02T04:46:43.597 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.598 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.601 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.602 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:43.602 DEBUG:boto:Data: 2022-09-02T04:46:43.603 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.604 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.605 DEBUG:boto:Params: {} 2022-09-02T04:46:43.606 DEBUG:boto:Token: None 2022-09-02T04:46:43.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/ 2022-09-02T04:46:43.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I= 2022-09-02T04:46:43.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I=', 'Content-Length': '0'} 2022-09-02T04:46:43.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d3e4d43558e4cd8-0063118ab3-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.611 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.612 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.612 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.613 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:43.613 DEBUG:boto:Data: 2022-09-02T04:46:43.614 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.615 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.616 DEBUG:boto:Params: {} 2022-09-02T04:46:43.616 DEBUG:boto:Token: None 2022-09-02T04:46:43.617 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/ 2022-09-02T04:46:43.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I= 2022-09-02T04:46:43.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I=', 'Content-Length': '0'} 2022-09-02T04:46:43.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004205c9e40bfe58d-0063118ab3-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:46:43.625 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.625 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.626 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.626 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.627 DEBUG:boto:Method: GET 2022-09-02T04:46:43.627 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.628 DEBUG:boto:Data: 2022-09-02T04:46:43.629 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.630 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.631 DEBUG:boto:Params: {} 2022-09-02T04:46:43.631 DEBUG:boto:Token: None 2022-09-02T04:46:43.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/?versions 2022-09-02T04:46:43.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM= 2022-09-02T04:46:43.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM=', 'Content-Length': '0'} 2022-09-02T04:46:43.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c1491b7e49808b-0063118ab3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.639 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:43.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:43.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:46:43.644 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.645 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.646 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.647 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.647 DEBUG:boto:Method: GET 2022-09-02T04:46:43.648 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.649 DEBUG:boto:Data: 2022-09-02T04:46:43.650 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.652 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.652 DEBUG:boto:Params: {} 2022-09-02T04:46:43.654 DEBUG:boto:Token: None 2022-09-02T04:46:43.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/?versions 2022-09-02T04:46:43.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM= 2022-09-02T04:46:43.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM=', 'Content-Length': '0'} 2022-09-02T04:46:43.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c095d7445aa4c1-0063118ab3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.661 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:43.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:43.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:43.664 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.664 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.665 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.666 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.667 DEBUG:boto:Method: GET 2022-09-02T04:46:43.667 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.668 DEBUG:boto:Data: 2022-09-02T04:46:43.668 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.669 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.669 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.670 DEBUG:boto:Params: {} 2022-09-02T04:46:43.671 DEBUG:boto:Token: None 2022-09-02T04:46:43.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/?versions 2022-09-02T04:46:43.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM= 2022-09-02T04:46:43.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM=', 'Content-Length': '0'} 2022-09-02T04:46:43.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191baabc6a5e2b64-0063118ab3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.676 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:43.677 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.677 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.678 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.678 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.679 DEBUG:boto:Method: GET 2022-09-02T04:46:43.680 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.680 DEBUG:boto:Data: 2022-09-02T04:46:43.681 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.682 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.682 DEBUG:boto:Params: {} 2022-09-02T04:46:43.683 DEBUG:boto:Token: None 2022-09-02T04:46:43.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/?versions 2022-09-02T04:46:43.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM= 2022-09-02T04:46:43.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM=', 'Content-Length': '0'} 2022-09-02T04:46:43.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004742cb52c75b4d09-0063118ab3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.688 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:43.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:43.690 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.690 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.691 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.692 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:43.692 DEBUG:boto:Data: 2022-09-02T04:46:43.693 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.694 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.694 DEBUG:boto:Params: {} 2022-09-02T04:46:43.695 DEBUG:boto:Token: None 2022-09-02T04:46:43.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj 2022-09-02T04:46:43.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28= 2022-09-02T04:46:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28=', 'Content-Length': '0'} 2022-09-02T04:46:43.700 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000bcc9a600423f0c7b-0063118ab3-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.701 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.701 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.702 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.702 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:43.703 DEBUG:boto:Data: 2022-09-02T04:46:43.703 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.704 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.705 DEBUG:boto:Params: {} 2022-09-02T04:46:43.706 DEBUG:boto:Token: None 2022-09-02T04:46:43.707 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj 2022-09-02T04:46:43.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28= 2022-09-02T04:46:43.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28=', 'Content-Length': '0'} 2022-09-02T04:46:43.711 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e9fb0a9919011d5d-0063118ab3-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:43.713 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.714 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.715 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.716 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.717 DEBUG:boto:Method: GET 2022-09-02T04:46:43.718 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.719 DEBUG:boto:Data: 2022-09-02T04:46:43.721 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.723 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.723 DEBUG:boto:Params: {} 2022-09-02T04:46:43.724 DEBUG:boto:Token: None 2022-09-02T04:46:43.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc= 2022-09-02T04:46:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc=', 'Content-Length': '0'} 2022-09-02T04:46:43.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f7d0489f3784c9d-0063118ab3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.729 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.730 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.731 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.732 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.732 DEBUG:boto:Method: GET 2022-09-02T04:46:43.733 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.734 DEBUG:boto:Data: 2022-09-02T04:46:43.735 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.737 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.737 DEBUG:boto:Params: {} 2022-09-02T04:46:43.738 DEBUG:boto:Token: None 2022-09-02T04:46:43.738 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc= 2022-09-02T04:46:43.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc=', 'Content-Length': '0'} 2022-09-02T04:46:43.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000292aaf69be1a5e84-0063118ab3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.744 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.745 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.745 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.746 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.746 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.747 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.748 DEBUG:boto:Data: 2022-09-02T04:46:43.749 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.750 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.751 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.752 DEBUG:boto:Params: {} 2022-09-02T04:46:43.753 DEBUG:boto:Token: None 2022-09-02T04:46:43.754 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eagCSV9PqqO40ttFAHVG5DREMbE= 2022-09-02T04:46:43.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eagCSV9PqqO40ttFAHVG5DREMbE=', 'Content-Length': '0'} 2022-09-02T04:46:43.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001aea273b19c326f5-0063118ab3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.760 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.761 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.762 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.763 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.764 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.765 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.766 DEBUG:boto:Data: 2022-09-02T04:46:43.767 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.769 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.770 DEBUG:boto:Params: {} 2022-09-02T04:46:43.771 DEBUG:boto:Token: None 2022-09-02T04:46:43.772 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eagCSV9PqqO40ttFAHVG5DREMbE= 2022-09-02T04:46:43.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eagCSV9PqqO40ttFAHVG5DREMbE=', 'Content-Length': '0'} 2022-09-02T04:46:43.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f390c13d7ac6d84-0063118ab3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:43.783 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.784 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.785 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.786 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.791 DEBUG:boto:Method: GET 2022-09-02T04:46:43.794 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.794 DEBUG:boto:Data: 2022-09-02T04:46:43.795 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.798 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.799 DEBUG:boto:Params: {} 2022-09-02T04:46:43.800 DEBUG:boto:Token: None 2022-09-02T04:46:43.800 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc= 2022-09-02T04:46:43.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc=', 'Content-Length': '0'} 2022-09-02T04:46:43.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009095afd8e4433f93-0063118ab3-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.805 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.806 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.807 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.808 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.809 DEBUG:boto:Method: GET 2022-09-02T04:46:43.810 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.811 DEBUG:boto:Data: 2022-09-02T04:46:43.811 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.813 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.813 DEBUG:boto:Params: {} 2022-09-02T04:46:43.814 DEBUG:boto:Token: None 2022-09-02T04:46:43.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:43.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc= 2022-09-02T04:46:43.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CDF1K053X5Cr7T5fyY86oSCVHdc=', 'Content-Length': '0'} 2022-09-02T04:46:43.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c259d1f9a46cb676-0063118ab3-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.820 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.820 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.821 DEBUG:boto:path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:43.822 DEBUG:boto:auth_path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:43.822 DEBUG:boto:Method: GET 2022-09-02T04:46:43.823 DEBUG:boto:Path: /xljbkf-21/?prefix=obj 2022-09-02T04:46:43.823 DEBUG:boto:Data: 2022-09-02T04:46:43.824 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.826 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.826 DEBUG:boto:Params: {} 2022-09-02T04:46:43.827 DEBUG:boto:Token: None 2022-09-02T04:46:43.828 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/ 2022-09-02T04:46:43.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rSO5zFI797XMXl56BZWWi447Dl4= 2022-09-02T04:46:43.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rSO5zFI797XMXl56BZWWi447Dl4=', 'Content-Length': '0'} 2022-09-02T04:46:43.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000334f72ad7ca6a694-0063118ab3-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.874 DEBUG:boto:b'xljbkf-21obj1000false' 2022-09-02T04:46:43.876 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.877 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.878 DEBUG:boto:path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:43.879 DEBUG:boto:auth_path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:43.880 DEBUG:boto:Method: GET 2022-09-02T04:46:43.881 DEBUG:boto:Path: /xljbkf-21/?prefix=obj 2022-09-02T04:46:43.882 DEBUG:boto:Data: 2022-09-02T04:46:43.883 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.888 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.889 DEBUG:boto:Params: {} 2022-09-02T04:46:43.891 DEBUG:boto:Token: None 2022-09-02T04:46:43.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/ 2022-09-02T04:46:43.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rSO5zFI797XMXl56BZWWi447Dl4= 2022-09-02T04:46:43.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rSO5zFI797XMXl56BZWWi447Dl4=', 'Content-Length': '0'} 2022-09-02T04:46:43.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d0814d7e702eaa4-0063118ab3-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.897 DEBUG:boto:b'xljbkf-21obj1000false' 2022-09-02T04:46:43.899 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.899 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.900 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.901 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:43.902 DEBUG:boto:Data: 2022-09-02T04:46:43.903 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.904 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:43.905 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.906 DEBUG:boto:Params: {} 2022-09-02T04:46:43.907 DEBUG:boto:Token: None 2022-09-02T04:46:43.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj 2022-09-02T04:46:43.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28= 2022-09-02T04:46:43.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28=', 'Content-Length': '0'} 2022-09-02T04:46:43.913 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000072debc982ee60a01-0063118ab3-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.914 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:43.915 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:43.916 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.918 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:43.920 DEBUG:boto:Data: 2022-09-02T04:46:43.921 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.923 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.924 DEBUG:boto:Params: {} 2022-09-02T04:46:43.925 DEBUG:boto:Token: None 2022-09-02T04:46:43.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/obj 2022-09-02T04:46:43.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28= 2022-09-02T04:46:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCz2doe2qZCE65kwc+i0u4PBm28=', 'Content-Length': '0'} 2022-09-02T04:46:43.931 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000958fd83ad73d1702-0063118ab3-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-21 zones={test-zone2, test-zone1} 2022-09-02T04:46:43.933 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-21 zones={test-zone3, test-zone1} 2022-09-02T04:46:43.934 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.935 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.936 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.937 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:43.938 DEBUG:boto:Data: 2022-09-02T04:46:43.939 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:43.940 DEBUG:boto:Port: 8001 2022-09-02T04:46:43.941 DEBUG:boto:Params: {} 2022-09-02T04:46:43.942 DEBUG:boto:Token: None 2022-09-02T04:46:43.943 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/ 2022-09-02T04:46:43.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I= 2022-09-02T04:46:43.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I=', 'Content-Length': '0'} 2022-09-02T04:46:43.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8e1a9f54b3740d4-0063118ab3-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.950 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.951 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.952 DEBUG:boto:Method: HEAD 2022-09-02T04:46:43.953 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:43.954 DEBUG:boto:Data: 2022-09-02T04:46:43.955 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.956 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.957 DEBUG:boto:Params: {} 2022-09-02T04:46:43.957 DEBUG:boto:Token: None 2022-09-02T04:46:43.958 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/ 2022-09-02T04:46:43.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I= 2022-09-02T04:46:43.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:koLwZwzoR2kwuFPCzXqFq4p7y7I=', 'Content-Length': '0'} 2022-09-02T04:46:43.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff629c3fc879c298-0063118ab3-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.967 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:46:43.968 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.969 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.970 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.970 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.971 DEBUG:boto:Method: GET 2022-09-02T04:46:43.972 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.974 DEBUG:boto:Data: 2022-09-02T04:46:43.975 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:43.976 DEBUG:boto:Port: 8001 2022-09-02T04:46:43.977 DEBUG:boto:Params: {} 2022-09-02T04:46:43.978 DEBUG:boto:Token: None 2022-09-02T04:46:43.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/?versions 2022-09-02T04:46:43.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM= 2022-09-02T04:46:43.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM=', 'Content-Length': '0'} 2022-09-02T04:46:43.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000512094cc1f90c0c7-0063118ab3-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:43.984 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:43.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:43.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:43.986 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:46:43.987 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:43.988 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:43.989 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.989 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.990 DEBUG:boto:Method: GET 2022-09-02T04:46:43.991 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:43.992 DEBUG:boto:Data: 2022-09-02T04:46:43.993 DEBUG:boto:Headers: {} 2022-09-02T04:46:43.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:43.995 DEBUG:boto:Port: 8000 2022-09-02T04:46:43.996 DEBUG:boto:Params: {} 2022-09-02T04:46:43.997 DEBUG:boto:Token: None 2022-09-02T04:46:43.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:43 GMT /xljbkf-21/?versions 2022-09-02T04:46:43.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM= 2022-09-02T04:46:44.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xhyLGFZqwMT8FLRAEVddg8dW9JM=', 'Content-Length': '0'} 2022-09-02T04:46:44.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000144be342b33b50d6-0063118ab4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.046 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:44.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:44.048 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.049 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.049 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.050 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.050 DEBUG:boto:Method: GET 2022-09-02T04:46:44.051 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.051 DEBUG:boto:Data: 2022-09-02T04:46:44.052 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.053 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.053 DEBUG:boto:Params: {} 2022-09-02T04:46:44.054 DEBUG:boto:Token: None 2022-09-02T04:46:44.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/?versions 2022-09-02T04:46:44.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4= 2022-09-02T04:46:44.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4=', 'Content-Length': '0'} 2022-09-02T04:46:44.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9f74adaacfe2fa-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.062 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.063 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.064 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.064 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.065 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.065 DEBUG:boto:Method: GET 2022-09-02T04:46:44.066 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.066 DEBUG:boto:Data: 2022-09-02T04:46:44.067 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.068 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.068 DEBUG:boto:Params: {} 2022-09-02T04:46:44.069 DEBUG:boto:Token: None 2022-09-02T04:46:44.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/?versions 2022-09-02T04:46:44.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4= 2022-09-02T04:46:44.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4=', 'Content-Length': '0'} 2022-09-02T04:46:44.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7b4ac01ce68a39-0063118ab4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.074 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:44.076 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.076 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.077 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.077 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.078 DEBUG:boto:Data: 2022-09-02T04:46:44.078 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.079 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.080 DEBUG:boto:Params: {} 2022-09-02T04:46:44.080 DEBUG:boto:Token: None 2022-09-02T04:46:44.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.084 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000696e6d30fbcd578d-0063118ab4-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.085 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.085 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.086 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.086 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.087 DEBUG:boto:Data: 2022-09-02T04:46:44.087 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.089 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.089 DEBUG:boto:Params: {} 2022-09-02T04:46:44.090 DEBUG:boto:Token: None 2022-09-02T04:46:44.091 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.097 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a916cb5a0aa890a5-0063118ab4-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:44.098 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.099 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.099 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.100 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.101 DEBUG:boto:Method: GET 2022-09-02T04:46:44.101 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.102 DEBUG:boto:Data: 2022-09-02T04:46:44.102 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.103 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.104 DEBUG:boto:Params: {} 2022-09-02T04:46:44.104 DEBUG:boto:Token: None 2022-09-02T04:46:44.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023c378cbdd51b54d-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.110 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.111 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.111 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.112 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.112 DEBUG:boto:Method: GET 2022-09-02T04:46:44.113 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.114 DEBUG:boto:Data: 2022-09-02T04:46:44.114 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.115 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.116 DEBUG:boto:Params: {} 2022-09-02T04:46:44.116 DEBUG:boto:Token: None 2022-09-02T04:46:44.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.120 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7b6b61f9a058d3c-0063118ab4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.125 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.126 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.126 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.127 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.128 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.128 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.129 DEBUG:boto:Data: 2022-09-02T04:46:44.129 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.130 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.131 DEBUG:boto:Params: {} 2022-09-02T04:46:44.131 DEBUG:boto:Token: None 2022-09-02T04:46:44.133 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0= 2022-09-02T04:46:44.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0=', 'Content-Length': '0'} 2022-09-02T04:46:44.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045cc5ea2c181d0a9-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.137 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.137 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.138 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.139 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.139 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.140 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.140 DEBUG:boto:Data: 2022-09-02T04:46:44.141 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.142 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.144 DEBUG:boto:Params: {} 2022-09-02T04:46:44.144 DEBUG:boto:Token: None 2022-09-02T04:46:44.145 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0= 2022-09-02T04:46:44.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0=', 'Content-Length': '0'} 2022-09-02T04:46:44.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001253106ccd6adaf4-0063118ab4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:44.152 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.152 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.153 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.153 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.154 DEBUG:boto:Method: GET 2022-09-02T04:46:44.154 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.155 DEBUG:boto:Data: 2022-09-02T04:46:44.155 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.156 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.157 DEBUG:boto:Params: {} 2022-09-02T04:46:44.158 DEBUG:boto:Token: None 2022-09-02T04:46:44.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073418007d574d8c7-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.164 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.165 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.166 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.166 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.167 DEBUG:boto:Method: GET 2022-09-02T04:46:44.167 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.168 DEBUG:boto:Data: 2022-09-02T04:46:44.168 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.170 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.170 DEBUG:boto:Params: {} 2022-09-02T04:46:44.171 DEBUG:boto:Token: None 2022-09-02T04:46:44.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082f643d05fe55bff-0063118ab4-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.177 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.177 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.178 DEBUG:boto:path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.178 DEBUG:boto:auth_path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.179 DEBUG:boto:Method: GET 2022-09-02T04:46:44.179 DEBUG:boto:Path: /xljbkf-21/?prefix=obj 2022-09-02T04:46:44.180 DEBUG:boto:Data: 2022-09-02T04:46:44.181 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.183 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.183 DEBUG:boto:Params: {} 2022-09-02T04:46:44.184 DEBUG:boto:Token: None 2022-09-02T04:46:44.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/ 2022-09-02T04:46:44.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw= 2022-09-02T04:46:44.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw=', 'Content-Length': '0'} 2022-09-02T04:46:44.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ab3b4b32d6a736d-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.189 DEBUG:boto:b'xljbkf-21obj1000false' 2022-09-02T04:46:44.190 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.190 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.191 DEBUG:boto:path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.191 DEBUG:boto:auth_path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.192 DEBUG:boto:Method: GET 2022-09-02T04:46:44.193 DEBUG:boto:Path: /xljbkf-21/?prefix=obj 2022-09-02T04:46:44.194 DEBUG:boto:Data: 2022-09-02T04:46:44.194 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.195 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.196 DEBUG:boto:Params: {} 2022-09-02T04:46:44.196 DEBUG:boto:Token: None 2022-09-02T04:46:44.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/ 2022-09-02T04:46:44.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw= 2022-09-02T04:46:44.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw=', 'Content-Length': '0'} 2022-09-02T04:46:44.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0db41a711fcd5fe-0063118ab4-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.246 DEBUG:boto:b'xljbkf-21obj1000false' 2022-09-02T04:46:44.247 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.248 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.249 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.250 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.251 DEBUG:boto:Data: 2022-09-02T04:46:44.251 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.252 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.253 DEBUG:boto:Params: {} 2022-09-02T04:46:44.253 DEBUG:boto:Token: None 2022-09-02T04:46:44.254 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.263 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000079b9e63d508fcb61-0063118ab4-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.264 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.264 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.265 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.265 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.266 DEBUG:boto:Data: 2022-09-02T04:46:44.266 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:46:44.267 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.268 DEBUG:boto:Params: {} 2022-09-02T04:46:44.268 DEBUG:boto:Token: None 2022-09-02T04:46:44.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.273 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000000bb30bd480cf7693-0063118ab4-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-21 zones={test-zone3, test-zone1} 2022-09-02T04:46:44.274 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-21 zones={test-zone3, test-zone2} 2022-09-02T04:46:44.275 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.275 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.276 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.276 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:44.277 DEBUG:boto:Data: 2022-09-02T04:46:44.277 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.278 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.279 DEBUG:boto:Params: {} 2022-09-02T04:46:44.280 DEBUG:boto:Token: None 2022-09-02T04:46:44.280 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/ 2022-09-02T04:46:44.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aaaV8SaKlzn6iQd8gR0/uxMKO7o= 2022-09-02T04:46:44.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aaaV8SaKlzn6iQd8gR0/uxMKO7o=', 'Content-Length': '0'} 2022-09-02T04:46:44.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d44ed3b1b542080-0063118ab4-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.285 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.285 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.286 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.287 DEBUG:boto:Path: /xljbkf-21/ 2022-09-02T04:46:44.287 DEBUG:boto:Data: 2022-09-02T04:46:44.288 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.289 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.289 DEBUG:boto:Params: {} 2022-09-02T04:46:44.290 DEBUG:boto:Token: None 2022-09-02T04:46:44.290 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/ 2022-09-02T04:46:44.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aaaV8SaKlzn6iQd8gR0/uxMKO7o= 2022-09-02T04:46:44.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aaaV8SaKlzn6iQd8gR0/uxMKO7o=', 'Content-Length': '0'} 2022-09-02T04:46:44.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e99297b3880913c-0063118ab4-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.295 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:46:44.296 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.297 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.298 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.299 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.299 DEBUG:boto:Method: GET 2022-09-02T04:46:44.300 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.301 DEBUG:boto:Data: 2022-09-02T04:46:44.301 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.303 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.304 DEBUG:boto:Params: {} 2022-09-02T04:46:44.305 DEBUG:boto:Token: None 2022-09-02T04:46:44.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/?versions 2022-09-02T04:46:44.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4= 2022-09-02T04:46:44.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4=', 'Content-Length': '0'} 2022-09-02T04:46:44.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ee61cca0f420e82-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.314 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:44.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:44.317 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:46:44.318 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.319 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.320 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.321 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.322 DEBUG:boto:Method: GET 2022-09-02T04:46:44.323 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.324 DEBUG:boto:Data: 2022-09-02T04:46:44.325 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.327 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.328 DEBUG:boto:Params: {} 2022-09-02T04:46:44.329 DEBUG:boto:Token: None 2022-09-02T04:46:44.330 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/?versions 2022-09-02T04:46:44.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4= 2022-09-02T04:46:44.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4=', 'Content-Length': '0'} 2022-09-02T04:46:44.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e2bc2239d00f831-0063118ab4-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.336 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:44.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:46:44.338 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.339 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.339 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.340 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.341 DEBUG:boto:Method: GET 2022-09-02T04:46:44.342 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.343 DEBUG:boto:Data: 2022-09-02T04:46:44.344 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.346 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.347 DEBUG:boto:Params: {} 2022-09-02T04:46:44.348 DEBUG:boto:Token: None 2022-09-02T04:46:44.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/?versions 2022-09-02T04:46:44.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4= 2022-09-02T04:46:44.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4=', 'Content-Length': '0'} 2022-09-02T04:46:44.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f584689bba69f0-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.355 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.356 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.357 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.358 DEBUG:boto:path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.359 DEBUG:boto:auth_path=/xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.360 DEBUG:boto:Method: GET 2022-09-02T04:46:44.361 DEBUG:boto:Path: /xljbkf-21/?versions&max-keys=999 2022-09-02T04:46:44.361 DEBUG:boto:Data: 2022-09-02T04:46:44.362 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.364 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.365 DEBUG:boto:Params: {} 2022-09-02T04:46:44.366 DEBUG:boto:Token: None 2022-09-02T04:46:44.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/?versions 2022-09-02T04:46:44.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4= 2022-09-02T04:46:44.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XHI2DPXEgJ83l2Cg5d4PtlYIT4=', 'Content-Length': '0'} 2022-09-02T04:46:44.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b8d00001df8fd0-0063118ab4-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.372 DEBUG:boto:b'xljbkf-21999falseobj1vgzt9FPi5A.NlxXJwcRhkfHYe6T9I8true2022-09-02T04:46:07.555Zrgw-multisite-test-userMultisite Test UserNormalobjP8wEqgTTMlZonts0x3A9gpMWAwG5S-Gfalse2022-09-02T04:46:07.515Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:46:44.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:44.375 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.376 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.377 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.378 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.379 DEBUG:boto:Data: 2022-09-02T04:46:44.379 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.382 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.383 DEBUG:boto:Params: {} 2022-09-02T04:46:44.384 DEBUG:boto:Token: None 2022-09-02T04:46:44.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.389 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000b77dc1f1e49baf5b-0063118ab4-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.389 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.390 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.390 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.391 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.391 DEBUG:boto:Data: 2022-09-02T04:46:44.392 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.393 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.393 DEBUG:boto:Params: {} 2022-09-02T04:46:44.394 DEBUG:boto:Token: None 2022-09-02T04:46:44.395 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.398 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000007bbd7b583ae5b840-0063118ab4-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:44.399 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.400 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.401 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.401 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.402 DEBUG:boto:Method: GET 2022-09-02T04:46:44.402 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.403 DEBUG:boto:Data: 2022-09-02T04:46:44.403 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.405 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.405 DEBUG:boto:Params: {} 2022-09-02T04:46:44.406 DEBUG:boto:Token: None 2022-09-02T04:46:44.407 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026bbeed15d88a670-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.411 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.412 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.413 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.414 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.415 DEBUG:boto:Method: GET 2022-09-02T04:46:44.415 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.416 DEBUG:boto:Data: 2022-09-02T04:46:44.417 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.418 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.418 DEBUG:boto:Params: {} 2022-09-02T04:46:44.419 DEBUG:boto:Token: None 2022-09-02T04:46:44.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f91cde1fc80666e9-0063118ab4-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.425 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.426 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.427 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.428 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.429 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.430 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.431 DEBUG:boto:Data: 2022-09-02T04:46:44.431 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.433 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.434 DEBUG:boto:Params: {} 2022-09-02T04:46:44.435 DEBUG:boto:Token: None 2022-09-02T04:46:44.436 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0= 2022-09-02T04:46:44.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0=', 'Content-Length': '0'} 2022-09-02T04:46:44.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072597345d3d9e1c5-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.441 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.442 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.443 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.444 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.444 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.445 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.445 DEBUG:boto:Data: 2022-09-02T04:46:44.446 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.447 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.447 DEBUG:boto:Params: {} 2022-09-02T04:46:44.448 DEBUG:boto:Token: None 2022-09-02T04:46:44.448 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0= 2022-09-02T04:46:44.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:piTV/GGPAh/YGXWtyAwtkvESoz0=', 'Content-Length': '0'} 2022-09-02T04:46:44.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002495413473835f7f-0063118ab4-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:46:44.453 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.454 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.454 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.455 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.455 DEBUG:boto:Method: GET 2022-09-02T04:46:44.456 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.456 DEBUG:boto:Data: 2022-09-02T04:46:44.457 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.458 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.459 DEBUG:boto:Params: {} 2022-09-02T04:46:44.459 DEBUG:boto:Token: None 2022-09-02T04:46:44.460 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055c1d55dfa3825bb-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.465 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.465 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.466 DEBUG:boto:path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.466 DEBUG:boto:auth_path=/xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.467 DEBUG:boto:Method: GET 2022-09-02T04:46:44.467 DEBUG:boto:Path: /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.468 DEBUG:boto:Data: 2022-09-02T04:46:44.468 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.469 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.470 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.470 DEBUG:boto:Params: {} 2022-09-02T04:46:44.471 DEBUG:boto:Token: None 2022-09-02T04:46:44.471 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj?versionId=P8wEqgTTMlZonts0x3A9gpMWAwG5S-G 2022-09-02T04:46:44.472 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ= 2022-09-02T04:46:44.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5gywhAeTSyrIZ+oJZ3hvxihZHQ=', 'Content-Length': '0'} 2022-09-02T04:46:44.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'P8wEqgTTMlZonts0x3A9gpMWAwG5S-G'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054010973d024744b-0063118ab4-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.475 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.476 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.476 DEBUG:boto:path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.477 DEBUG:boto:auth_path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.477 DEBUG:boto:Method: GET 2022-09-02T04:46:44.478 DEBUG:boto:Path: /xljbkf-21/?prefix=obj 2022-09-02T04:46:44.479 DEBUG:boto:Data: 2022-09-02T04:46:44.479 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.480 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.481 DEBUG:boto:Params: {} 2022-09-02T04:46:44.481 DEBUG:boto:Token: None 2022-09-02T04:46:44.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/ 2022-09-02T04:46:44.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw= 2022-09-02T04:46:44.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw=', 'Content-Length': '0'} 2022-09-02T04:46:44.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff169b53ce08ba12-0063118ab4-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.487 DEBUG:boto:b'xljbkf-21obj1000false' 2022-09-02T04:46:44.487 DEBUG:boto:path=/xljbkf-21/ 2022-09-02T04:46:44.488 DEBUG:boto:auth_path=/xljbkf-21/ 2022-09-02T04:46:44.489 DEBUG:boto:path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.489 DEBUG:boto:auth_path=/xljbkf-21/?prefix=obj 2022-09-02T04:46:44.490 DEBUG:boto:Method: GET 2022-09-02T04:46:44.490 DEBUG:boto:Path: /xljbkf-21/?prefix=obj 2022-09-02T04:46:44.491 DEBUG:boto:Data: 2022-09-02T04:46:44.491 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.493 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.493 DEBUG:boto:Params: {} 2022-09-02T04:46:44.494 DEBUG:boto:Token: None 2022-09-02T04:46:44.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/ 2022-09-02T04:46:44.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw= 2022-09-02T04:46:44.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vE7GVXDUNp4Bsm1Y4J7Zf5D3YKw=', 'Content-Length': '0'} 2022-09-02T04:46:44.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079be5fdb1c16cd12-0063118ab4-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.538 DEBUG:boto:b'xljbkf-21obj1000false' 2022-09-02T04:46:44.539 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.540 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.540 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.541 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.542 DEBUG:boto:Data: 2022-09-02T04:46:44.542 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:46:44.543 DEBUG:boto:Port: 8001 2022-09-02T04:46:44.544 DEBUG:boto:Params: {} 2022-09-02T04:46:44.544 DEBUG:boto:Token: None 2022-09-02T04:46:44.545 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.549 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000b4b33e01748d7779-0063118ab4-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.549 DEBUG:boto:path=/xljbkf-21/obj 2022-09-02T04:46:44.550 DEBUG:boto:auth_path=/xljbkf-21/obj 2022-09-02T04:46:44.550 DEBUG:boto:Method: HEAD 2022-09-02T04:46:44.551 DEBUG:boto:Path: /xljbkf-21/obj 2022-09-02T04:46:44.552 DEBUG:boto:Data: 2022-09-02T04:46:44.552 DEBUG:boto:Headers: {} 2022-09-02T04:46:44.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:46:44.553 DEBUG:boto:Port: 8000 2022-09-02T04:46:44.554 DEBUG:boto:Params: {} 2022-09-02T04:46:44.555 DEBUG:boto:Token: None 2022-09-02T04:46:44.555 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:46:44 GMT /xljbkf-21/obj 2022-09-02T04:46:44.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74= 2022-09-02T04:46:44.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:46:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TTXF21GTIcJAAAKckTiGt9zb+74=', 'Content-Length': '0'} 2022-09-02T04:46:44.559 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000c67370586fdf1f21-0063118ab4-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:46:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:46:44.559 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-21 zones={test-zone3, test-zone2} 2022-09-02T04:46:44.560 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-22 2022-09-02T04:46:44.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:44.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:46:44.562 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:46:44.622 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:46:44.623 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:46:44.656 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:44.656+0000 7fcad1b7c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:49.656 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:49.657+0000 7fcad1b7c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:54.657 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:54.657+0000 7fcad1b7c600 1 waiting to reach incremental sync.. 2022-09-02T04:46:59.658 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:46:59.658+0000 7fcad1b7c600 1 waiting to reach incremental sync.. 2022-09-02T04:47:04.661 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:04.660+0000 7fcad1b7c600 1 bucket sync caught up with source: 2022-09-02T04:47:04.662 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.129.5, , ] 2022-09-02T04:47:04.662 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.129.5, , ] 2022-09-02T04:47:04.663 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:04.660+0000 7fcad1b7c600 0 bucket checkpoint complete 2022-09-02T04:47:04.676 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-22 2022-09-02T04:47:04.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:04.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:04.679 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:04.745 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:04.746 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:04.790 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:04.789+0000 7f936433a600 1 bucket sync caught up with source: 2022-09-02T04:47:04.791 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.129.5, , ] 2022-09-02T04:47:04.791 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.129.5, , ] 2022-09-02T04:47:04.792 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:04.789+0000 7f936433a600 0 bucket checkpoint complete 2022-09-02T04:47:04.801 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-22 2022-09-02T04:47:04.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:04.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:04.805 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:04.921 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:04.921+0000 7f8ee695d600 1 bucket sync caught up with source: 2022-09-02T04:47:04.922 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.129.5, , ] 2022-09-02T04:47:04.923 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.129.5, , ] 2022-09-02T04:47:04.923 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:04.921+0000 7f8ee695d600 0 bucket checkpoint complete 2022-09-02T04:47:04.943 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-22 2022-09-02T04:47:04.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:04.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:04.946 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:05.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:05.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:05.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:05.047+0000 7fe40458e600 1 bucket sync caught up with source: 2022-09-02T04:47:05.048 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.218.11, , ] 2022-09-02T04:47:05.048 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.218.11, , ] 2022-09-02T04:47:05.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:05.047+0000 7fe40458e600 0 bucket checkpoint complete 2022-09-02T04:47:05.066 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-22 2022-09-02T04:47:05.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:05.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:05.068 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:05.125 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:05.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:05.170 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:05.170+0000 7fbc4860e600 1 bucket sync caught up with source: 2022-09-02T04:47:05.170 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.218.11, , ] 2022-09-02T04:47:05.171 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.218.11, , ] 2022-09-02T04:47:05.171 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:05.170+0000 7fbc4860e600 0 bucket checkpoint complete 2022-09-02T04:47:05.189 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-22 2022-09-02T04:47:05.189 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-22 2022-09-02T04:47:05.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:05.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:05.191 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:05.249 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:05.297 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:05.298+0000 7fc6662a9600 1 bucket sync caught up with source: 2022-09-02T04:47:05.298 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.240.5, , ] 2022-09-02T04:47:05.298 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.240.5, , ] 2022-09-02T04:47:05.298 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:05.298+0000 7fc6662a9600 0 bucket checkpoint complete 2022-09-02T04:47:05.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-22 2022-09-02T04:47:05.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:05.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:05.321 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:05.392 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:05.393 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:05.437 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:05.436+0000 7f5b5a2c7600 1 bucket sync caught up with source: 2022-09-02T04:47:05.438 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.240.5, , ] 2022-09-02T04:47:05.438 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.240.5, , ] 2022-09-02T04:47:05.439 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:05.436+0000 7f5b5a2c7600 0 bucket checkpoint complete 2022-09-02T04:47:05.452 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-22 2022-09-02T04:47:05.453 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-22 zones={test-zone2, test-zone1} 2022-09-02T04:47:05.454 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.454 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.455 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.455 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:47:05.456 DEBUG:boto:Data: 2022-09-02T04:47:05.456 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.457 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.458 DEBUG:boto:Params: {} 2022-09-02T04:47:05.459 DEBUG:boto:Token: None 2022-09-02T04:47:05.459 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/ 2022-09-02T04:47:05.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0= 2022-09-02T04:47:05.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0=', 'Content-Length': '0'} 2022-09-02T04:47:05.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a0cc09e4e428ec0-0063118ac9-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.464 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.465 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.465 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.466 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:47:05.466 DEBUG:boto:Data: 2022-09-02T04:47:05.467 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.468 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.468 DEBUG:boto:Params: {} 2022-09-02T04:47:05.469 DEBUG:boto:Token: None 2022-09-02T04:47:05.470 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/ 2022-09-02T04:47:05.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0= 2022-09-02T04:47:05.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0=', 'Content-Length': '0'} 2022-09-02T04:47:05.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a42e3e011f34466-0063118ac9-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.474 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:47:05.475 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.476 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.476 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.477 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.477 DEBUG:boto:Method: GET 2022-09-02T04:47:05.478 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.478 DEBUG:boto:Data: 2022-09-02T04:47:05.479 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.480 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.480 DEBUG:boto:Params: {} 2022-09-02T04:47:05.481 DEBUG:boto:Token: None 2022-09-02T04:47:05.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf7f30e8d100e5f-0063118ac9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.493 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.497 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:47:05.498 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.499 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.500 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.500 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.501 DEBUG:boto:Method: GET 2022-09-02T04:47:05.503 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.504 DEBUG:boto:Data: 2022-09-02T04:47:05.505 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.510 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.511 DEBUG:boto:Params: {} 2022-09-02T04:47:05.512 DEBUG:boto:Token: None 2022-09-02T04:47:05.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ddb4365d4e0145a-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.523 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.525 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.526 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.526 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.527 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.527 DEBUG:boto:Method: GET 2022-09-02T04:47:05.528 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.529 DEBUG:boto:Data: 2022-09-02T04:47:05.530 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.531 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.532 DEBUG:boto:Params: {} 2022-09-02T04:47:05.537 DEBUG:boto:Token: None 2022-09-02T04:47:05.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce835fc9ec0172b5-0063118ac9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.547 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.548 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.549 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.550 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.551 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.552 DEBUG:boto:Method: GET 2022-09-02T04:47:05.553 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.554 DEBUG:boto:Data: 2022-09-02T04:47:05.555 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.557 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.558 DEBUG:boto:Params: {} 2022-09-02T04:47:05.559 DEBUG:boto:Token: None 2022-09-02T04:47:05.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6448c6503c0345-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.566 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:05.568 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.568 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.569 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.569 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:05.571 DEBUG:boto:Data: 2022-09-02T04:47:05.572 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.573 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.574 DEBUG:boto:Params: {} 2022-09-02T04:47:05.574 DEBUG:boto:Token: None 2022-09-02T04:47:05.575 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj 2022-09-02T04:47:05.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo= 2022-09-02T04:47:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo=', 'Content-Length': '0'} 2022-09-02T04:47:05.590 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008f4788c705adbd08-0063118ac9-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.592 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.598 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.599 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.600 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:05.601 DEBUG:boto:Data: 2022-09-02T04:47:05.602 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.604 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.604 DEBUG:boto:Params: {} 2022-09-02T04:47:05.605 DEBUG:boto:Token: None 2022-09-02T04:47:05.605 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj 2022-09-02T04:47:05.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo= 2022-09-02T04:47:05.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo=', 'Content-Length': '0'} 2022-09-02T04:47:05.609 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000005370244a811863-0063118ac9-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:05.611 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.611 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.612 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.612 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.613 DEBUG:boto:Method: GET 2022-09-02T04:47:05.614 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.615 DEBUG:boto:Data: 2022-09-02T04:47:05.615 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.616 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.617 DEBUG:boto:Params: {} 2022-09-02T04:47:05.620 DEBUG:boto:Token: None 2022-09-02T04:47:05.621 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI= 2022-09-02T04:47:05.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI=', 'Content-Length': '0'} 2022-09-02T04:47:05.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001baac2c16105ae76-0063118ac9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.627 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.628 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.630 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.631 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.632 DEBUG:boto:Method: GET 2022-09-02T04:47:05.633 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.634 DEBUG:boto:Data: 2022-09-02T04:47:05.636 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.638 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.639 DEBUG:boto:Params: {} 2022-09-02T04:47:05.640 DEBUG:boto:Token: None 2022-09-02T04:47:05.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI= 2022-09-02T04:47:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI=', 'Content-Length': '0'} 2022-09-02T04:47:05.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bc308f573f6e7a6-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.652 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.653 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.654 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.655 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.656 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.657 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.658 DEBUG:boto:Data: 2022-09-02T04:47:05.659 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.671 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.672 DEBUG:boto:Params: {} 2022-09-02T04:47:05.672 DEBUG:boto:Token: None 2022-09-02T04:47:05.673 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KiFsARPspdf3x5nb1pocReSNQSE= 2022-09-02T04:47:05.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KiFsARPspdf3x5nb1pocReSNQSE=', 'Content-Length': '0'} 2022-09-02T04:47:05.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001080656d7904acba-0063118ac9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.678 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.679 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.680 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.680 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.681 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.681 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.682 DEBUG:boto:Data: 2022-09-02T04:47:05.682 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.683 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.684 DEBUG:boto:Params: {} 2022-09-02T04:47:05.685 DEBUG:boto:Token: None 2022-09-02T04:47:05.685 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KiFsARPspdf3x5nb1pocReSNQSE= 2022-09-02T04:47:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KiFsARPspdf3x5nb1pocReSNQSE=', 'Content-Length': '0'} 2022-09-02T04:47:05.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f577c733bf92fef4-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:05.690 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.691 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.691 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.692 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.693 DEBUG:boto:Method: GET 2022-09-02T04:47:05.693 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.694 DEBUG:boto:Data: 2022-09-02T04:47:05.695 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.699 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.699 DEBUG:boto:Params: {} 2022-09-02T04:47:05.700 DEBUG:boto:Token: None 2022-09-02T04:47:05.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI= 2022-09-02T04:47:05.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI=', 'Content-Length': '0'} 2022-09-02T04:47:05.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b8a9cf78464b600-0063118ac9-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.709 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.710 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.710 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.711 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.711 DEBUG:boto:Method: GET 2022-09-02T04:47:05.712 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.713 DEBUG:boto:Data: 2022-09-02T04:47:05.713 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.715 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.716 DEBUG:boto:Params: {} 2022-09-02T04:47:05.717 DEBUG:boto:Token: None 2022-09-02T04:47:05.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:05.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI= 2022-09-02T04:47:05.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AdYZiaY+wBVlfJ680F5ki2QS1yI=', 'Content-Length': '0'} 2022-09-02T04:47:05.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c5cbfcbe8f4fe5f1-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.722 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.723 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.724 DEBUG:boto:path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:05.724 DEBUG:boto:auth_path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:05.725 DEBUG:boto:Method: GET 2022-09-02T04:47:05.726 DEBUG:boto:Path: /xljbkf-22/?prefix=obj 2022-09-02T04:47:05.726 DEBUG:boto:Data: 2022-09-02T04:47:05.727 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.728 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.729 DEBUG:boto:Params: {} 2022-09-02T04:47:05.729 DEBUG:boto:Token: None 2022-09-02T04:47:05.730 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/ 2022-09-02T04:47:05.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5+EgNrveinPja0WetMdtyGnJhdc= 2022-09-02T04:47:05.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5+EgNrveinPja0WetMdtyGnJhdc=', 'Content-Length': '0'} 2022-09-02T04:47:05.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d217fce7017f97f-0063118ac9-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.778 DEBUG:boto:b'xljbkf-22obj1000false' 2022-09-02T04:47:05.779 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.780 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.781 DEBUG:boto:path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:05.781 DEBUG:boto:auth_path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:05.782 DEBUG:boto:Method: GET 2022-09-02T04:47:05.782 DEBUG:boto:Path: /xljbkf-22/?prefix=obj 2022-09-02T04:47:05.783 DEBUG:boto:Data: 2022-09-02T04:47:05.784 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.785 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.785 DEBUG:boto:Params: {} 2022-09-02T04:47:05.786 DEBUG:boto:Token: None 2022-09-02T04:47:05.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/ 2022-09-02T04:47:05.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5+EgNrveinPja0WetMdtyGnJhdc= 2022-09-02T04:47:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5+EgNrveinPja0WetMdtyGnJhdc=', 'Content-Length': '0'} 2022-09-02T04:47:05.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002614b1a718e479a1-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.796 DEBUG:boto:b'xljbkf-22obj1000false' 2022-09-02T04:47:05.797 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.798 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.799 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.804 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:05.806 DEBUG:boto:Data: 2022-09-02T04:47:05.807 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:05.808 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.809 DEBUG:boto:Params: {} 2022-09-02T04:47:05.809 DEBUG:boto:Token: None 2022-09-02T04:47:05.810 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj 2022-09-02T04:47:05.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo= 2022-09-02T04:47:05.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo=', 'Content-Length': '0'} 2022-09-02T04:47:05.816 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000008f0975932fe958b-0063118ac9-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.817 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.818 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.819 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.820 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:05.821 DEBUG:boto:Data: 2022-09-02T04:47:05.822 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.824 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.825 DEBUG:boto:Params: {} 2022-09-02T04:47:05.826 DEBUG:boto:Token: None 2022-09-02T04:47:05.827 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj 2022-09-02T04:47:05.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo= 2022-09-02T04:47:05.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo=', 'Content-Length': '0'} 2022-09-02T04:47:05.834 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000003189a5f7c1bc2ee-0063118ac9-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.835 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-22 zones={test-zone2, test-zone1} 2022-09-02T04:47:05.836 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-22 zones={test-zone3, test-zone1} 2022-09-02T04:47:05.836 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.837 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.837 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.838 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:47:05.838 DEBUG:boto:Data: 2022-09-02T04:47:05.839 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:05.840 DEBUG:boto:Port: 8001 2022-09-02T04:47:05.841 DEBUG:boto:Params: {} 2022-09-02T04:47:05.841 DEBUG:boto:Token: None 2022-09-02T04:47:05.842 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/ 2022-09-02T04:47:05.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0= 2022-09-02T04:47:05.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0=', 'Content-Length': '0'} 2022-09-02T04:47:05.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c1001060d332c1-0063118ac9-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.847 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.848 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.849 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.850 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:47:05.851 DEBUG:boto:Data: 2022-09-02T04:47:05.852 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.853 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.854 DEBUG:boto:Params: {} 2022-09-02T04:47:05.855 DEBUG:boto:Token: None 2022-09-02T04:47:05.856 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/ 2022-09-02T04:47:05.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0= 2022-09-02T04:47:05.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Umxs3IzfwaN3E+FMt8s7C6PgY0=', 'Content-Length': '0'} 2022-09-02T04:47:05.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c608af57b052067-0063118ac9-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.862 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:47:05.863 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.864 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.865 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.866 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.867 DEBUG:boto:Method: GET 2022-09-02T04:47:05.868 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.869 DEBUG:boto:Data: 2022-09-02T04:47:05.870 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:05.872 DEBUG:boto:Port: 8001 2022-09-02T04:47:05.872 DEBUG:boto:Params: {} 2022-09-02T04:47:05.873 DEBUG:boto:Token: None 2022-09-02T04:47:05.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d746a733e1ff5e56-0063118ac9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.926 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.929 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:47:05.930 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.931 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.932 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.933 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.935 DEBUG:boto:Method: GET 2022-09-02T04:47:05.936 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.937 DEBUG:boto:Data: 2022-09-02T04:47:05.937 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.940 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.941 DEBUG:boto:Params: {} 2022-09-02T04:47:05.941 DEBUG:boto:Token: None 2022-09-02T04:47:05.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000813ba843e25b7e1c-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.947 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:05.950 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.952 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.953 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.954 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.956 DEBUG:boto:Method: GET 2022-09-02T04:47:05.957 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.958 DEBUG:boto:Data: 2022-09-02T04:47:05.959 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:05.961 DEBUG:boto:Port: 8001 2022-09-02T04:47:05.962 DEBUG:boto:Params: {} 2022-09-02T04:47:05.963 DEBUG:boto:Token: None 2022-09-02T04:47:05.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031e0d3d977663fd4-0063118ac9-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.968 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.970 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:05.971 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:05.971 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.972 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.972 DEBUG:boto:Method: GET 2022-09-02T04:47:05.974 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:05.974 DEBUG:boto:Data: 2022-09-02T04:47:05.975 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:05.976 DEBUG:boto:Port: 8000 2022-09-02T04:47:05.976 DEBUG:boto:Params: {} 2022-09-02T04:47:05.978 DEBUG:boto:Token: None 2022-09-02T04:47:05.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/?versions 2022-09-02T04:47:05.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw= 2022-09-02T04:47:05.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iBl6845SprweAllwMkXsTyN26yw=', 'Content-Length': '0'} 2022-09-02T04:47:05.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcb8f80db469acd1-0063118ac9-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.985 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:05.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:05.987 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.987 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.988 DEBUG:boto:Method: HEAD 2022-09-02T04:47:05.988 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:05.989 DEBUG:boto:Data: 2022-09-02T04:47:05.990 DEBUG:boto:Headers: {} 2022-09-02T04:47:05.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:05.991 DEBUG:boto:Port: 8001 2022-09-02T04:47:05.992 DEBUG:boto:Params: {} 2022-09-02T04:47:05.992 DEBUG:boto:Token: None 2022-09-02T04:47:05.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:05 GMT /xljbkf-22/obj 2022-09-02T04:47:05.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo= 2022-09-02T04:47:05.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LW+nA5V6yeDTn61q9k8mgLiYZOo=', 'Content-Length': '0'} 2022-09-02T04:47:05.997 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005b783c7df7b75ad7-0063118ac9-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:05.998 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:05.999 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:05.999 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.000 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.000 DEBUG:boto:Data: 2022-09-02T04:47:06.001 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:06.002 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.002 DEBUG:boto:Params: {} 2022-09-02T04:47:06.003 DEBUG:boto:Token: None 2022-09-02T04:47:06.003 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.007 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000009ebd20f1aa89ca52-0063118aca-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:06.010 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.010 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.011 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.011 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.012 DEBUG:boto:Method: GET 2022-09-02T04:47:06.013 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.013 DEBUG:boto:Data: 2022-09-02T04:47:06.019 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.020 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.021 DEBUG:boto:Params: {} 2022-09-02T04:47:06.022 DEBUG:boto:Token: None 2022-09-02T04:47:06.022 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002e742ee2e19676c-0063118aca-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.031 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.032 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.033 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.034 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.035 DEBUG:boto:Method: GET 2022-09-02T04:47:06.036 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.037 DEBUG:boto:Data: 2022-09-02T04:47:06.037 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:06.039 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.040 DEBUG:boto:Params: {} 2022-09-02T04:47:06.041 DEBUG:boto:Token: None 2022-09-02T04:47:06.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cc9e313c19055f7-0063118aca-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.051 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.052 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.053 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.053 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.054 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.055 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.056 DEBUG:boto:Data: 2022-09-02T04:47:06.057 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.063 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.064 DEBUG:boto:Params: {} 2022-09-02T04:47:06.066 DEBUG:boto:Token: None 2022-09-02T04:47:06.067 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U= 2022-09-02T04:47:06.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U=', 'Content-Length': '0'} 2022-09-02T04:47:06.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004af1b8075b0c5c23-0063118aca-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.076 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.077 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.082 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.083 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.085 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.097 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.098 DEBUG:boto:Data: 2022-09-02T04:47:06.098 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:06.099 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.100 DEBUG:boto:Params: {} 2022-09-02T04:47:06.101 DEBUG:boto:Token: None 2022-09-02T04:47:06.102 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U= 2022-09-02T04:47:06.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U=', 'Content-Length': '0'} 2022-09-02T04:47:06.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8a943f8deb8cb2b-0063118aca-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:06.115 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.116 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.116 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.117 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.117 DEBUG:boto:Method: GET 2022-09-02T04:47:06.118 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.118 DEBUG:boto:Data: 2022-09-02T04:47:06.119 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.120 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.120 DEBUG:boto:Params: {} 2022-09-02T04:47:06.122 DEBUG:boto:Token: None 2022-09-02T04:47:06.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee6048822cf34239-0063118aca-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.127 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.128 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.128 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.129 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.129 DEBUG:boto:Method: GET 2022-09-02T04:47:06.130 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.130 DEBUG:boto:Data: 2022-09-02T04:47:06.131 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:06.133 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.133 DEBUG:boto:Params: {} 2022-09-02T04:47:06.134 DEBUG:boto:Token: None 2022-09-02T04:47:06.134 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025cd684c09bfb40a-0063118aca-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.139 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.139 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.140 DEBUG:boto:path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.140 DEBUG:boto:auth_path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.141 DEBUG:boto:Method: GET 2022-09-02T04:47:06.141 DEBUG:boto:Path: /xljbkf-22/?prefix=obj 2022-09-02T04:47:06.143 DEBUG:boto:Data: 2022-09-02T04:47:06.143 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.144 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.145 DEBUG:boto:Params: {} 2022-09-02T04:47:06.146 DEBUG:boto:Token: None 2022-09-02T04:47:06.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/ 2022-09-02T04:47:06.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E= 2022-09-02T04:47:06.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E=', 'Content-Length': '0'} 2022-09-02T04:47:06.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef31f415b45b5633-0063118aca-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.151 DEBUG:boto:b'xljbkf-22obj1000false' 2022-09-02T04:47:06.152 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.152 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.156 DEBUG:boto:path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.158 DEBUG:boto:auth_path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.159 DEBUG:boto:Method: GET 2022-09-02T04:47:06.160 DEBUG:boto:Path: /xljbkf-22/?prefix=obj 2022-09-02T04:47:06.161 DEBUG:boto:Data: 2022-09-02T04:47:06.162 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:06.164 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.165 DEBUG:boto:Params: {} 2022-09-02T04:47:06.166 DEBUG:boto:Token: None 2022-09-02T04:47:06.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/ 2022-09-02T04:47:06.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E= 2022-09-02T04:47:06.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E=', 'Content-Length': '0'} 2022-09-02T04:47:06.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059b889135d7f450-0063118aca-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.175 DEBUG:boto:b'xljbkf-22obj1000false' 2022-09-02T04:47:06.177 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.178 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.179 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.180 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.181 DEBUG:boto:Data: 2022-09-02T04:47:06.182 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.184 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.185 DEBUG:boto:Params: {} 2022-09-02T04:47:06.185 DEBUG:boto:Token: None 2022-09-02T04:47:06.187 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.192 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008709f47e20ff7c7a-0063118aca-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.193 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.194 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.195 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.196 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.197 DEBUG:boto:Data: 2022-09-02T04:47:06.198 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:06.200 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.200 DEBUG:boto:Params: {} 2022-09-02T04:47:06.201 DEBUG:boto:Token: None 2022-09-02T04:47:06.203 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.208 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000bcfc2289a69168ee-0063118aca-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-22 zones={test-zone3, test-zone1} 2022-09-02T04:47:06.210 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-22 zones={test-zone3, test-zone2} 2022-09-02T04:47:06.211 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.212 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.213 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.214 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:47:06.215 DEBUG:boto:Data: 2022-09-02T04:47:06.216 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.219 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.220 DEBUG:boto:Params: {} 2022-09-02T04:47:06.221 DEBUG:boto:Token: None 2022-09-02T04:47:06.222 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/ 2022-09-02T04:47:06.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FfS0/eMWpLbkOEoGu4O4Z7BaLrQ= 2022-09-02T04:47:06.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FfS0/eMWpLbkOEoGu4O4Z7BaLrQ=', 'Content-Length': '0'} 2022-09-02T04:47:06.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ece88da37bbd052-0063118aca-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.229 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.230 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.231 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.232 DEBUG:boto:Path: /xljbkf-22/ 2022-09-02T04:47:06.233 DEBUG:boto:Data: 2022-09-02T04:47:06.234 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.236 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.237 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.238 DEBUG:boto:Params: {} 2022-09-02T04:47:06.239 DEBUG:boto:Token: None 2022-09-02T04:47:06.240 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/ 2022-09-02T04:47:06.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FfS0/eMWpLbkOEoGu4O4Z7BaLrQ= 2022-09-02T04:47:06.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FfS0/eMWpLbkOEoGu4O4Z7BaLrQ=', 'Content-Length': '0'} 2022-09-02T04:47:06.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000217b350039bcf7a3-0063118aca-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.247 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:47:06.248 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.249 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.250 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.251 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.251 DEBUG:boto:Method: GET 2022-09-02T04:47:06.252 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.253 DEBUG:boto:Data: 2022-09-02T04:47:06.254 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.256 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.257 DEBUG:boto:Params: {} 2022-09-02T04:47:06.259 DEBUG:boto:Token: None 2022-09-02T04:47:06.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/?versions 2022-09-02T04:47:06.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM= 2022-09-02T04:47:06.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM=', 'Content-Length': '0'} 2022-09-02T04:47:06.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2811afd6a24ee30-0063118aca-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.266 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:06.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:06.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:06.271 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:47:06.272 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.273 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.274 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.275 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.275 DEBUG:boto:Method: GET 2022-09-02T04:47:06.276 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.277 DEBUG:boto:Data: 2022-09-02T04:47:06.279 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.281 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.281 DEBUG:boto:Params: {} 2022-09-02T04:47:06.283 DEBUG:boto:Token: None 2022-09-02T04:47:06.284 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/?versions 2022-09-02T04:47:06.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM= 2022-09-02T04:47:06.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM=', 'Content-Length': '0'} 2022-09-02T04:47:06.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ecb95925463f78e-0063118aca-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.291 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:06.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:06.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:06.297 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.298 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.299 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.300 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.301 DEBUG:boto:Method: GET 2022-09-02T04:47:06.302 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.304 DEBUG:boto:Data: 2022-09-02T04:47:06.305 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.306 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.307 DEBUG:boto:Params: {} 2022-09-02T04:47:06.308 DEBUG:boto:Token: None 2022-09-02T04:47:06.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/?versions 2022-09-02T04:47:06.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM= 2022-09-02T04:47:06.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM=', 'Content-Length': '0'} 2022-09-02T04:47:06.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f18ae991e7b666b-0063118aca-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.318 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:06.320 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.321 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.322 DEBUG:boto:path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.323 DEBUG:boto:auth_path=/xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.323 DEBUG:boto:Method: GET 2022-09-02T04:47:06.324 DEBUG:boto:Path: /xljbkf-22/?versions&max-keys=999 2022-09-02T04:47:06.325 DEBUG:boto:Data: 2022-09-02T04:47:06.326 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.328 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.329 DEBUG:boto:Params: {} 2022-09-02T04:47:06.330 DEBUG:boto:Token: None 2022-09-02T04:47:06.332 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/?versions 2022-09-02T04:47:06.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM= 2022-09-02T04:47:06.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDP+Nox6PD3peNybuz0OADXQrM=', 'Content-Length': '0'} 2022-09-02T04:47:06.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd3285d3cb199ea9-0063118aca-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.341 DEBUG:boto:b'xljbkf-22999falseobjAVj9vyprFEqLOBYKqTB.RaiKLGeZ8qAtrue2022-09-02T04:46:07.623Zrgw-multisite-test-userMultisite Test UserNormalobjNdLSASkArjeqyzGutpg2Fk6qAJxkD4Cfalse2022-09-02T04:46:07.588Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:06.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:06.343 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.350 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.351 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.352 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.352 DEBUG:boto:Data: 2022-09-02T04:47:06.353 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.354 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.354 DEBUG:boto:Params: {} 2022-09-02T04:47:06.355 DEBUG:boto:Token: None 2022-09-02T04:47:06.355 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.359 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000822534ab28b70b77-0063118aca-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.362 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.363 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.363 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.364 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.365 DEBUG:boto:Data: 2022-09-02T04:47:06.366 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.368 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.368 DEBUG:boto:Params: {} 2022-09-02T04:47:06.370 DEBUG:boto:Token: None 2022-09-02T04:47:06.371 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.376 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000dd721bdde7421bce-0063118aca-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:06.379 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.379 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.380 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.380 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.381 DEBUG:boto:Method: GET 2022-09-02T04:47:06.381 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.382 DEBUG:boto:Data: 2022-09-02T04:47:06.382 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.383 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.384 DEBUG:boto:Params: {} 2022-09-02T04:47:06.385 DEBUG:boto:Token: None 2022-09-02T04:47:06.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053be72e429d93e66-0063118aca-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.391 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.392 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.392 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.393 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.393 DEBUG:boto:Method: GET 2022-09-02T04:47:06.394 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.394 DEBUG:boto:Data: 2022-09-02T04:47:06.395 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.395 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.396 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.399 DEBUG:boto:Params: {} 2022-09-02T04:47:06.400 DEBUG:boto:Token: None 2022-09-02T04:47:06.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001563ddaa494caa60-0063118aca-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.407 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.408 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.409 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.410 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.411 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.412 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.413 DEBUG:boto:Data: 2022-09-02T04:47:06.414 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.415 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.416 DEBUG:boto:Params: {} 2022-09-02T04:47:06.417 DEBUG:boto:Token: None 2022-09-02T04:47:06.418 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U= 2022-09-02T04:47:06.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U=', 'Content-Length': '0'} 2022-09-02T04:47:06.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca804c183009a723-0063118aca-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.424 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.424 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.425 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.426 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.427 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.429 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.430 DEBUG:boto:Data: 2022-09-02T04:47:06.430 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.433 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.434 DEBUG:boto:Params: {} 2022-09-02T04:47:06.435 DEBUG:boto:Token: None 2022-09-02T04:47:06.436 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U= 2022-09-02T04:47:06.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dJFVON2E3skgXfWnvYBvCraH81U=', 'Content-Length': '0'} 2022-09-02T04:47:06.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009644928806d058fd-0063118aca-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:06.442 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.443 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.443 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.444 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.445 DEBUG:boto:Method: GET 2022-09-02T04:47:06.445 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.446 DEBUG:boto:Data: 2022-09-02T04:47:06.447 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.448 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.449 DEBUG:boto:Params: {} 2022-09-02T04:47:06.450 DEBUG:boto:Token: None 2022-09-02T04:47:06.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a037f9b8f513ad8-0063118aca-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.456 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.456 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.457 DEBUG:boto:path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.457 DEBUG:boto:auth_path=/xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.458 DEBUG:boto:Method: GET 2022-09-02T04:47:06.459 DEBUG:boto:Path: /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.461 DEBUG:boto:Data: 2022-09-02T04:47:06.461 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.462 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.463 DEBUG:boto:Params: {} 2022-09-02T04:47:06.464 DEBUG:boto:Token: None 2022-09-02T04:47:06.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj?versionId=NdLSASkArjeqyzGutpg2Fk6qAJxkD4C 2022-09-02T04:47:06.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs= 2022-09-02T04:47:06.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nUKnlqtUegdBEVEs8k/DgSEPvs=', 'Content-Length': '0'} 2022-09-02T04:47:06.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'NdLSASkArjeqyzGutpg2Fk6qAJxkD4C'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054f4522714ab1202-0063118aca-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.470 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.470 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.471 DEBUG:boto:path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.471 DEBUG:boto:auth_path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.472 DEBUG:boto:Method: GET 2022-09-02T04:47:06.473 DEBUG:boto:Path: /xljbkf-22/?prefix=obj 2022-09-02T04:47:06.474 DEBUG:boto:Data: 2022-09-02T04:47:06.474 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.475 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.476 DEBUG:boto:Params: {} 2022-09-02T04:47:06.477 DEBUG:boto:Token: None 2022-09-02T04:47:06.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/ 2022-09-02T04:47:06.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E= 2022-09-02T04:47:06.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E=', 'Content-Length': '0'} 2022-09-02T04:47:06.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005320e560dce44ce7-0063118aca-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.523 DEBUG:boto:b'xljbkf-22obj1000false' 2022-09-02T04:47:06.525 DEBUG:boto:path=/xljbkf-22/ 2022-09-02T04:47:06.527 DEBUG:boto:auth_path=/xljbkf-22/ 2022-09-02T04:47:06.528 DEBUG:boto:path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.529 DEBUG:boto:auth_path=/xljbkf-22/?prefix=obj 2022-09-02T04:47:06.530 DEBUG:boto:Method: GET 2022-09-02T04:47:06.531 DEBUG:boto:Path: /xljbkf-22/?prefix=obj 2022-09-02T04:47:06.533 DEBUG:boto:Data: 2022-09-02T04:47:06.534 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.535 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.536 DEBUG:boto:Params: {} 2022-09-02T04:47:06.536 DEBUG:boto:Token: None 2022-09-02T04:47:06.537 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/ 2022-09-02T04:47:06.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E= 2022-09-02T04:47:06.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bRsbI/wdzW7oSnPAm+IeR9Ngs/E=', 'Content-Length': '0'} 2022-09-02T04:47:06.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129278dbd22fd84b-0063118aca-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.542 DEBUG:boto:b'xljbkf-22obj1000false' 2022-09-02T04:47:06.543 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.544 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.544 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.545 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.545 DEBUG:boto:Data: 2022-09-02T04:47:06.548 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:06.555 DEBUG:boto:Port: 8001 2022-09-02T04:47:06.556 DEBUG:boto:Params: {} 2022-09-02T04:47:06.557 DEBUG:boto:Token: None 2022-09-02T04:47:06.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.564 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000730bfde3e464ce28-0063118aca-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.565 DEBUG:boto:path=/xljbkf-22/obj 2022-09-02T04:47:06.566 DEBUG:boto:auth_path=/xljbkf-22/obj 2022-09-02T04:47:06.567 DEBUG:boto:Method: HEAD 2022-09-02T04:47:06.568 DEBUG:boto:Path: /xljbkf-22/obj 2022-09-02T04:47:06.569 DEBUG:boto:Data: 2022-09-02T04:47:06.570 DEBUG:boto:Headers: {} 2022-09-02T04:47:06.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:06.572 DEBUG:boto:Port: 8000 2022-09-02T04:47:06.573 DEBUG:boto:Params: {} 2022-09-02T04:47:06.574 DEBUG:boto:Token: None 2022-09-02T04:47:06.575 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:06 GMT /xljbkf-22/obj 2022-09-02T04:47:06.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0= 2022-09-02T04:47:06.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ghl7T8FvvRYEAfmnXF+o3P9nAv0=', 'Content-Length': '0'} 2022-09-02T04:47:06.583 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d405d0ebc39a9230-0063118aca-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:06.584 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-22 zones={test-zone3, test-zone2} 2022-09-02T04:47:06.585 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-23 2022-09-02T04:47:06.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.588 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:06.668 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:06.669 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:06.702 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:06.702+0000 7f9746354600 1 bucket sync caught up with source: 2022-09-02T04:47:06.703 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2022-09-02T04:47:06.704 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2022-09-02T04:47:06.704 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:06.702+0000 7f9746354600 0 bucket checkpoint complete 2022-09-02T04:47:06.727 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-23 2022-09-02T04:47:06.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.730 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:06.830 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:06.831+0000 7f7e3507e600 1 bucket sync caught up with source: 2022-09-02T04:47:06.831 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2022-09-02T04:47:06.832 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2022-09-02T04:47:06.833 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:06.831+0000 7f7e3507e600 0 bucket checkpoint complete 2022-09-02T04:47:06.844 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-23 2022-09-02T04:47:06.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.847 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:06.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:06.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:06.956 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:06.957+0000 7f1240a59600 1 bucket sync caught up with source: 2022-09-02T04:47:06.957 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2022-09-02T04:47:06.957 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2022-09-02T04:47:06.957 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:06.957+0000 7f1240a59600 0 bucket checkpoint complete 2022-09-02T04:47:06.970 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-23 2022-09-02T04:47:06.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:06.982 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:07.084 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:07.085+0000 7f7e86ef0600 1 bucket sync caught up with source: 2022-09-02T04:47:07.085 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.78.5, , ] 2022-09-02T04:47:07.086 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.78.5, , ] 2022-09-02T04:47:07.086 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:07.085+0000 7f7e86ef0600 0 bucket checkpoint complete 2022-09-02T04:47:07.104 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-23 2022-09-02T04:47:07.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:07.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:07.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:07.210+0000 7f2fe3b83600 1 bucket sync caught up with source: 2022-09-02T04:47:07.211 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.78.5, , ] 2022-09-02T04:47:07.211 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.78.5, , ] 2022-09-02T04:47:07.212 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:07.210+0000 7f2fe3b83600 0 bucket checkpoint complete 2022-09-02T04:47:07.223 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-23 2022-09-02T04:47:07.224 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-23 2022-09-02T04:47:07.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:07.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:07.226 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:07.282 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:07.283 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:07.325+0000 7ff172c1f600 1 bucket sync caught up with source: 2022-09-02T04:47:07.325 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.100.11, , ] 2022-09-02T04:47:07.325 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.100.11, , ] 2022-09-02T04:47:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:07.325+0000 7ff172c1f600 0 bucket checkpoint complete 2022-09-02T04:47:07.338 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-23 2022-09-02T04:47:07.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:07.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:07.340 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:07.396 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:07.397 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:07.455+0000 7f5e09b7b600 1 bucket sync caught up with source: 2022-09-02T04:47:07.455 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.100.11, , ] 2022-09-02T04:47:07.455 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.100.11, , ] 2022-09-02T04:47:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:07.455+0000 7f5e09b7b600 0 bucket checkpoint complete 2022-09-02T04:47:07.478 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-23 2022-09-02T04:47:07.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-23 zones={test-zone2, test-zone1} 2022-09-02T04:47:07.479 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.480 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.480 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.481 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:47:07.481 DEBUG:boto:Data: 2022-09-02T04:47:07.482 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.483 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.483 DEBUG:boto:Params: {} 2022-09-02T04:47:07.484 DEBUG:boto:Token: None 2022-09-02T04:47:07.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/ 2022-09-02T04:47:07.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY= 2022-09-02T04:47:07.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY=', 'Content-Length': '0'} 2022-09-02T04:47:07.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e01bfa7a19468d47-0063118acb-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.490 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.491 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.491 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.492 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:47:07.492 DEBUG:boto:Data: 2022-09-02T04:47:07.493 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.494 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.494 DEBUG:boto:Params: {} 2022-09-02T04:47:07.495 DEBUG:boto:Token: None 2022-09-02T04:47:07.495 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/ 2022-09-02T04:47:07.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY= 2022-09-02T04:47:07.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY=', 'Content-Length': '0'} 2022-09-02T04:47:07.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000073649de5d7e3d37-0063118acb-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:47:07.503 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.505 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.506 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.506 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.507 DEBUG:boto:Method: GET 2022-09-02T04:47:07.507 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.508 DEBUG:boto:Data: 2022-09-02T04:47:07.509 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.510 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.510 DEBUG:boto:Params: {} 2022-09-02T04:47:07.511 DEBUG:boto:Token: None 2022-09-02T04:47:07.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e9ae7f21d8b7ea-0063118acb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.516 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:47:07.519 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.519 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.520 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.520 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.521 DEBUG:boto:Method: GET 2022-09-02T04:47:07.521 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.522 DEBUG:boto:Data: 2022-09-02T04:47:07.522 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.523 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.524 DEBUG:boto:Params: {} 2022-09-02T04:47:07.524 DEBUG:boto:Token: None 2022-09-02T04:47:07.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8b854cb98b0662b-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.532 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.536 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.537 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.537 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.539 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.540 DEBUG:boto:Method: GET 2022-09-02T04:47:07.541 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.542 DEBUG:boto:Data: 2022-09-02T04:47:07.543 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.544 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.545 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.546 DEBUG:boto:Params: {} 2022-09-02T04:47:07.547 DEBUG:boto:Token: None 2022-09-02T04:47:07.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d4cd63549db1b7-0063118acb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.555 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.556 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.556 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.557 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.559 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.560 DEBUG:boto:Method: GET 2022-09-02T04:47:07.561 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.562 DEBUG:boto:Data: 2022-09-02T04:47:07.563 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.566 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.567 DEBUG:boto:Params: {} 2022-09-02T04:47:07.569 DEBUG:boto:Token: None 2022-09-02T04:47:07.570 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.571 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2d9f01738de31f0-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.575 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:07.578 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.579 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.580 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.581 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:07.582 DEBUG:boto:Data: 2022-09-02T04:47:07.583 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.584 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.586 DEBUG:boto:Params: {} 2022-09-02T04:47:07.588 DEBUG:boto:Token: None 2022-09-02T04:47:07.589 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj 2022-09-02T04:47:07.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M= 2022-09-02T04:47:07.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M=', 'Content-Length': '0'} 2022-09-02T04:47:07.593 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000008c6bd4d6c517a514-0063118acb-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.594 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.594 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.595 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.595 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:07.596 DEBUG:boto:Data: 2022-09-02T04:47:07.596 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.597 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.602 DEBUG:boto:Params: {} 2022-09-02T04:47:07.603 DEBUG:boto:Token: None 2022-09-02T04:47:07.604 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj 2022-09-02T04:47:07.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M= 2022-09-02T04:47:07.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M=', 'Content-Length': '0'} 2022-09-02T04:47:07.609 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f093643e76eabe78-0063118acb-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:07.611 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.612 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.613 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.614 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.615 DEBUG:boto:Method: GET 2022-09-02T04:47:07.615 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.616 DEBUG:boto:Data: 2022-09-02T04:47:07.616 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.618 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.618 DEBUG:boto:Params: {} 2022-09-02T04:47:07.619 DEBUG:boto:Token: None 2022-09-02T04:47:07.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:07.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088d79a74625e9b15-0063118acb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.623 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.624 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.624 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.625 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.626 DEBUG:boto:Method: GET 2022-09-02T04:47:07.626 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.627 DEBUG:boto:Data: 2022-09-02T04:47:07.627 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.628 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.629 DEBUG:boto:Params: {} 2022-09-02T04:47:07.629 DEBUG:boto:Token: None 2022-09-02T04:47:07.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:07.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ca441af9518ce60-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.634 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.635 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.635 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.636 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.636 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.637 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.637 DEBUG:boto:Data: 2022-09-02T04:47:07.638 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.638 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.639 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.640 DEBUG:boto:Params: {} 2022-09-02T04:47:07.640 DEBUG:boto:Token: None 2022-09-02T04:47:07.641 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU= 2022-09-02T04:47:07.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU=', 'Content-Length': '0'} 2022-09-02T04:47:07.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078445d2b3f71f4ae-0063118acb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.646 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.646 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.647 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.647 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.648 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.648 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.649 DEBUG:boto:Data: 2022-09-02T04:47:07.650 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.651 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.651 DEBUG:boto:Params: {} 2022-09-02T04:47:07.652 DEBUG:boto:Token: None 2022-09-02T04:47:07.653 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU= 2022-09-02T04:47:07.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU=', 'Content-Length': '0'} 2022-09-02T04:47:07.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b42fe4b917e58e7d-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:07.657 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.658 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.658 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.659 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.659 DEBUG:boto:Method: GET 2022-09-02T04:47:07.660 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.660 DEBUG:boto:Data: 2022-09-02T04:47:07.661 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.662 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.663 DEBUG:boto:Params: {} 2022-09-02T04:47:07.664 DEBUG:boto:Token: None 2022-09-02T04:47:07.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:07.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000300e33bba8949b49-0063118acb-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.671 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.671 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.672 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.673 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.673 DEBUG:boto:Method: GET 2022-09-02T04:47:07.674 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.674 DEBUG:boto:Data: 2022-09-02T04:47:07.675 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.676 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.677 DEBUG:boto:Params: {} 2022-09-02T04:47:07.677 DEBUG:boto:Token: None 2022-09-02T04:47:07.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:07.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005da63b97a28d6898-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.691 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.691 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.692 DEBUG:boto:path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:07.693 DEBUG:boto:auth_path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:07.694 DEBUG:boto:Method: GET 2022-09-02T04:47:07.695 DEBUG:boto:Path: /xljbkf-23/?prefix=obj 2022-09-02T04:47:07.696 DEBUG:boto:Data: 2022-09-02T04:47:07.696 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.697 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.698 DEBUG:boto:Params: {} 2022-09-02T04:47:07.698 DEBUG:boto:Token: None 2022-09-02T04:47:07.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/ 2022-09-02T04:47:07.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DcC504hUOpXhLxajHJJpHf0LWmc= 2022-09-02T04:47:07.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DcC504hUOpXhLxajHJJpHf0LWmc=', 'Content-Length': '0'} 2022-09-02T04:47:07.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e82ac7f1037876-0063118acb-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.747 DEBUG:boto:b'xljbkf-23obj1000false' 2022-09-02T04:47:07.749 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.749 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.750 DEBUG:boto:path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:07.750 DEBUG:boto:auth_path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:07.751 DEBUG:boto:Method: GET 2022-09-02T04:47:07.752 DEBUG:boto:Path: /xljbkf-23/?prefix=obj 2022-09-02T04:47:07.752 DEBUG:boto:Data: 2022-09-02T04:47:07.753 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.754 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.754 DEBUG:boto:Params: {} 2022-09-02T04:47:07.755 DEBUG:boto:Token: None 2022-09-02T04:47:07.756 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/ 2022-09-02T04:47:07.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DcC504hUOpXhLxajHJJpHf0LWmc= 2022-09-02T04:47:07.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DcC504hUOpXhLxajHJJpHf0LWmc=', 'Content-Length': '0'} 2022-09-02T04:47:07.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba9d550df4038ca7-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.760 DEBUG:boto:b'xljbkf-23obj1000false' 2022-09-02T04:47:07.761 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.762 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.762 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.763 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:07.764 DEBUG:boto:Data: 2022-09-02T04:47:07.765 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:07.766 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.766 DEBUG:boto:Params: {} 2022-09-02T04:47:07.767 DEBUG:boto:Token: None 2022-09-02T04:47:07.767 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj 2022-09-02T04:47:07.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M= 2022-09-02T04:47:07.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M=', 'Content-Length': '0'} 2022-09-02T04:47:07.771 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000050e8874ae073d644-0063118acb-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.771 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.772 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.772 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.773 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:07.773 DEBUG:boto:Data: 2022-09-02T04:47:07.774 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.775 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.776 DEBUG:boto:Params: {} 2022-09-02T04:47:07.776 DEBUG:boto:Token: None 2022-09-02T04:47:07.777 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj 2022-09-02T04:47:07.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M= 2022-09-02T04:47:07.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M=', 'Content-Length': '0'} 2022-09-02T04:47:07.780 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000bd52a51f04656758-0063118acb-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.781 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-23 zones={test-zone2, test-zone1} 2022-09-02T04:47:07.781 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-23 zones={test-zone3, test-zone1} 2022-09-02T04:47:07.782 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.783 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.783 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.788 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:47:07.788 DEBUG:boto:Data: 2022-09-02T04:47:07.789 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.790 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.791 DEBUG:boto:Params: {} 2022-09-02T04:47:07.793 DEBUG:boto:Token: None 2022-09-02T04:47:07.794 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/ 2022-09-02T04:47:07.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY= 2022-09-02T04:47:07.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY=', 'Content-Length': '0'} 2022-09-02T04:47:07.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bbddeb42ddb28f8-0063118acb-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.800 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.801 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.802 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.803 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:47:07.804 DEBUG:boto:Data: 2022-09-02T04:47:07.806 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.807 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.808 DEBUG:boto:Params: {} 2022-09-02T04:47:07.809 DEBUG:boto:Token: None 2022-09-02T04:47:07.809 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/ 2022-09-02T04:47:07.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY= 2022-09-02T04:47:07.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ytsqk7QJHv0qmGcUH5hjFibcUpY=', 'Content-Length': '0'} 2022-09-02T04:47:07.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac9f9e57942d4a0c-0063118acb-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.814 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:47:07.814 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.815 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.815 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.816 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.817 DEBUG:boto:Method: GET 2022-09-02T04:47:07.817 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.818 DEBUG:boto:Data: 2022-09-02T04:47:07.818 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.820 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.820 DEBUG:boto:Params: {} 2022-09-02T04:47:07.821 DEBUG:boto:Token: None 2022-09-02T04:47:07.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c9938c7ec947e6-0063118acb-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.827 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:47:07.831 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.832 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.833 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.834 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.835 DEBUG:boto:Method: GET 2022-09-02T04:47:07.836 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.837 DEBUG:boto:Data: 2022-09-02T04:47:07.838 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.840 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.841 DEBUG:boto:Params: {} 2022-09-02T04:47:07.842 DEBUG:boto:Token: None 2022-09-02T04:47:07.843 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.844 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54e18329ef84161-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.849 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:07.853 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.854 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.854 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.855 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.856 DEBUG:boto:Method: GET 2022-09-02T04:47:07.857 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.858 DEBUG:boto:Data: 2022-09-02T04:47:07.858 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.859 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.860 DEBUG:boto:Params: {} 2022-09-02T04:47:07.861 DEBUG:boto:Token: None 2022-09-02T04:47:07.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056da4c96d04cc855-0063118acb-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.868 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.869 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:07.870 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:07.870 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.871 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.872 DEBUG:boto:Method: GET 2022-09-02T04:47:07.873 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:07.874 DEBUG:boto:Data: 2022-09-02T04:47:07.874 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.876 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.877 DEBUG:boto:Params: {} 2022-09-02T04:47:07.878 DEBUG:boto:Token: None 2022-09-02T04:47:07.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/?versions 2022-09-02T04:47:07.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8= 2022-09-02T04:47:07.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bL7VYfGJupEm0rn1vP68xQ3mAz8=', 'Content-Length': '0'} 2022-09-02T04:47:07.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb0d35947444373-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.885 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:07.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:07.888 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.889 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.890 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.891 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:07.892 DEBUG:boto:Data: 2022-09-02T04:47:07.893 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.894 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.896 DEBUG:boto:Params: {} 2022-09-02T04:47:07.897 DEBUG:boto:Token: None 2022-09-02T04:47:07.898 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj 2022-09-02T04:47:07.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M= 2022-09-02T04:47:07.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M=', 'Content-Length': '0'} 2022-09-02T04:47:07.902 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000005c9d00c69cc9741e-0063118acb-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.903 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.903 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.904 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.904 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:07.905 DEBUG:boto:Data: 2022-09-02T04:47:07.905 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.907 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.908 DEBUG:boto:Params: {} 2022-09-02T04:47:07.908 DEBUG:boto:Token: None 2022-09-02T04:47:07.909 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj 2022-09-02T04:47:07.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M= 2022-09-02T04:47:07.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4gr5LtVnKkdj+tetL3Lo81i/0M=', 'Content-Length': '0'} 2022-09-02T04:47:07.912 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000e154d4ec0b7e097c-0063118acb-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:07.914 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.915 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.916 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.917 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.918 DEBUG:boto:Method: GET 2022-09-02T04:47:07.919 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.920 DEBUG:boto:Data: 2022-09-02T04:47:07.921 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.922 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.923 DEBUG:boto:Params: {} 2022-09-02T04:47:07.925 DEBUG:boto:Token: None 2022-09-02T04:47:07.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:07.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d845b2652e72f02b-0063118acb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.933 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.933 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.934 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.935 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.936 DEBUG:boto:Method: GET 2022-09-02T04:47:07.937 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.937 DEBUG:boto:Data: 2022-09-02T04:47:07.938 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.940 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.941 DEBUG:boto:Params: {} 2022-09-02T04:47:07.942 DEBUG:boto:Token: None 2022-09-02T04:47:07.943 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:07.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a36bdea5701f7f82-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.950 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.950 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.951 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.952 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.953 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.954 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.955 DEBUG:boto:Data: 2022-09-02T04:47:07.955 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.956 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.957 DEBUG:boto:Params: {} 2022-09-02T04:47:07.957 DEBUG:boto:Token: None 2022-09-02T04:47:07.958 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU= 2022-09-02T04:47:07.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU=', 'Content-Length': '0'} 2022-09-02T04:47:07.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf3564cc180699ef-0063118acb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.962 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.963 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.963 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.964 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.964 DEBUG:boto:Method: HEAD 2022-09-02T04:47:07.965 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.965 DEBUG:boto:Data: 2022-09-02T04:47:07.966 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:07.967 DEBUG:boto:Port: 8000 2022-09-02T04:47:07.968 DEBUG:boto:Params: {} 2022-09-02T04:47:07.969 DEBUG:boto:Token: None 2022-09-02T04:47:07.972 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU= 2022-09-02T04:47:07.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5nwruYktr/MlpTr5uQcyNsCPZHU=', 'Content-Length': '0'} 2022-09-02T04:47:07.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069967e6e08348a58-0063118acb-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:07.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:07.978 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:07.979 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:07.980 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.981 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.982 DEBUG:boto:Method: GET 2022-09-02T04:47:07.983 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.984 DEBUG:boto:Data: 2022-09-02T04:47:07.985 DEBUG:boto:Headers: {} 2022-09-02T04:47:07.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:07.987 DEBUG:boto:Port: 8001 2022-09-02T04:47:07.988 DEBUG:boto:Params: {} 2022-09-02T04:47:07.989 DEBUG:boto:Token: None 2022-09-02T04:47:07.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:07 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:07.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ= 2022-09-02T04:47:07.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/374vBKCVUw1pioJwEJTl3DlMEQ=', 'Content-Length': '0'} 2022-09-02T04:47:08.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf29abc4b4f49948-0063118acb-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.001 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.002 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.003 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.004 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.005 DEBUG:boto:Method: GET 2022-09-02T04:47:08.006 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.007 DEBUG:boto:Data: 2022-09-02T04:47:08.008 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:08.011 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.011 DEBUG:boto:Params: {} 2022-09-02T04:47:08.012 DEBUG:boto:Token: None 2022-09-02T04:47:08.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM= 2022-09-02T04:47:08.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM=', 'Content-Length': '0'} 2022-09-02T04:47:08.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb0eb68db9ae8e2c-0063118acc-10fc-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.017 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.017 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.018 DEBUG:boto:path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.018 DEBUG:boto:auth_path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.019 DEBUG:boto:Method: GET 2022-09-02T04:47:08.019 DEBUG:boto:Path: /xljbkf-23/?prefix=obj 2022-09-02T04:47:08.020 DEBUG:boto:Data: 2022-09-02T04:47:08.020 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.022 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.022 DEBUG:boto:Params: {} 2022-09-02T04:47:08.023 DEBUG:boto:Token: None 2022-09-02T04:47:08.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/ 2022-09-02T04:47:08.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc= 2022-09-02T04:47:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc=', 'Content-Length': '0'} 2022-09-02T04:47:08.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b43d81e493423f9e-0063118acc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.070 DEBUG:boto:b'xljbkf-23obj1000false' 2022-09-02T04:47:08.071 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.072 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.072 DEBUG:boto:path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.073 DEBUG:boto:auth_path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.073 DEBUG:boto:Method: GET 2022-09-02T04:47:08.074 DEBUG:boto:Path: /xljbkf-23/?prefix=obj 2022-09-02T04:47:08.074 DEBUG:boto:Data: 2022-09-02T04:47:08.075 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:08.076 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.076 DEBUG:boto:Params: {} 2022-09-02T04:47:08.077 DEBUG:boto:Token: None 2022-09-02T04:47:08.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/ 2022-09-02T04:47:08.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc= 2022-09-02T04:47:08.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc=', 'Content-Length': '0'} 2022-09-02T04:47:08.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96d6055b1ecf850-0063118acc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.082 DEBUG:boto:b'xljbkf-23obj1000false' 2022-09-02T04:47:08.083 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.084 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.084 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.085 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:08.085 DEBUG:boto:Data: 2022-09-02T04:47:08.086 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.087 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.087 DEBUG:boto:Params: {} 2022-09-02T04:47:08.088 DEBUG:boto:Token: None 2022-09-02T04:47:08.089 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj 2022-09-02T04:47:08.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4= 2022-09-02T04:47:08.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4=', 'Content-Length': '0'} 2022-09-02T04:47:08.094 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000004b423d08a66251bf-0063118acc-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.096 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.097 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.098 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.099 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:08.099 DEBUG:boto:Data: 2022-09-02T04:47:08.100 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:08.103 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.104 DEBUG:boto:Params: {} 2022-09-02T04:47:08.105 DEBUG:boto:Token: None 2022-09-02T04:47:08.106 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj 2022-09-02T04:47:08.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4= 2022-09-02T04:47:08.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4=', 'Content-Length': '0'} 2022-09-02T04:47:08.112 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006ccdb473b6cefbbb-0063118acc-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.113 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-23 zones={test-zone3, test-zone1} 2022-09-02T04:47:08.114 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-23 zones={test-zone3, test-zone2} 2022-09-02T04:47:08.116 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.116 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.117 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.118 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:47:08.119 DEBUG:boto:Data: 2022-09-02T04:47:08.121 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.123 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.124 DEBUG:boto:Params: {} 2022-09-02T04:47:08.125 DEBUG:boto:Token: None 2022-09-02T04:47:08.126 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/ 2022-09-02T04:47:08.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EixyTojRBofvXeP+Lq57fADB2dQ= 2022-09-02T04:47:08.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EixyTojRBofvXeP+Lq57fADB2dQ=', 'Content-Length': '0'} 2022-09-02T04:47:08.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693f7641e5dda947-0063118acc-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.147 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.147 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.148 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.149 DEBUG:boto:Path: /xljbkf-23/ 2022-09-02T04:47:08.149 DEBUG:boto:Data: 2022-09-02T04:47:08.150 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.151 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.151 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.152 DEBUG:boto:Params: {} 2022-09-02T04:47:08.152 DEBUG:boto:Token: None 2022-09-02T04:47:08.153 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/ 2022-09-02T04:47:08.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EixyTojRBofvXeP+Lq57fADB2dQ= 2022-09-02T04:47:08.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EixyTojRBofvXeP+Lq57fADB2dQ=', 'Content-Length': '0'} 2022-09-02T04:47:08.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006eb3aa176ffaeb16-0063118acc-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:47:08.160 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.161 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.161 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.162 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.163 DEBUG:boto:Method: GET 2022-09-02T04:47:08.164 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.165 DEBUG:boto:Data: 2022-09-02T04:47:08.166 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.167 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.168 DEBUG:boto:Params: {} 2022-09-02T04:47:08.169 DEBUG:boto:Token: None 2022-09-02T04:47:08.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/?versions 2022-09-02T04:47:08.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg= 2022-09-02T04:47:08.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg=', 'Content-Length': '0'} 2022-09-02T04:47:08.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e3e2f729041f0ea-0063118acc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.220 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:08.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:08.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:08.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:47:08.223 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.224 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.224 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.225 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.226 DEBUG:boto:Method: GET 2022-09-02T04:47:08.226 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.227 DEBUG:boto:Data: 2022-09-02T04:47:08.227 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.229 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.229 DEBUG:boto:Params: {} 2022-09-02T04:47:08.230 DEBUG:boto:Token: None 2022-09-02T04:47:08.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/?versions 2022-09-02T04:47:08.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg= 2022-09-02T04:47:08.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg=', 'Content-Length': '0'} 2022-09-02T04:47:08.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f777f57a6b4e74e2-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.244 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:08.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:08.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:47:08.246 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.247 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.247 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.248 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.248 DEBUG:boto:Method: GET 2022-09-02T04:47:08.249 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.249 DEBUG:boto:Data: 2022-09-02T04:47:08.250 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.251 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.252 DEBUG:boto:Params: {} 2022-09-02T04:47:08.253 DEBUG:boto:Token: None 2022-09-02T04:47:08.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/?versions 2022-09-02T04:47:08.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg= 2022-09-02T04:47:08.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg=', 'Content-Length': '0'} 2022-09-02T04:47:08.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021cb55dcd1b8957-0063118acc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.259 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:08.260 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.261 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.262 DEBUG:boto:path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.263 DEBUG:boto:auth_path=/xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.263 DEBUG:boto:Method: GET 2022-09-02T04:47:08.264 DEBUG:boto:Path: /xljbkf-23/?versions&max-keys=999 2022-09-02T04:47:08.264 DEBUG:boto:Data: 2022-09-02T04:47:08.265 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.266 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.266 DEBUG:boto:Params: {} 2022-09-02T04:47:08.267 DEBUG:boto:Token: None 2022-09-02T04:47:08.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/?versions 2022-09-02T04:47:08.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg= 2022-09-02T04:47:08.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+bTHIITBsZxy1Eu98SS6/8KVeg=', 'Content-Length': '0'} 2022-09-02T04:47:08.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee39a099d9db0102-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.272 DEBUG:boto:b'xljbkf-23999falseobjVLhtAeY4N-ag2Pw.cW5lZTaM8o.nOdXtrue2022-09-02T04:46:07.692Zrgw-multisite-test-userMultisite Test UserNormalobjMmsYVb7PTMI6q2fqHNjK1pRdW3pv4RYfalse2022-09-02T04:46:07.659Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:47:08.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:08.274 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.274 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.275 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.275 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:08.276 DEBUG:boto:Data: 2022-09-02T04:47:08.277 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.278 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.279 DEBUG:boto:Params: {} 2022-09-02T04:47:08.279 DEBUG:boto:Token: None 2022-09-02T04:47:08.280 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj 2022-09-02T04:47:08.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4= 2022-09-02T04:47:08.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4=', 'Content-Length': '0'} 2022-09-02T04:47:08.285 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000311bf84882239420-0063118acc-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.287 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.288 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.289 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.290 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:08.291 DEBUG:boto:Data: 2022-09-02T04:47:08.291 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.294 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.296 DEBUG:boto:Params: {} 2022-09-02T04:47:08.298 DEBUG:boto:Token: None 2022-09-02T04:47:08.299 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj 2022-09-02T04:47:08.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4= 2022-09-02T04:47:08.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4=', 'Content-Length': '0'} 2022-09-02T04:47:08.304 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000919d884fb44c7662-0063118acc-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:08.306 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.307 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.307 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.308 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.308 DEBUG:boto:Method: GET 2022-09-02T04:47:08.309 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.309 DEBUG:boto:Data: 2022-09-02T04:47:08.310 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.311 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.311 DEBUG:boto:Params: {} 2022-09-02T04:47:08.312 DEBUG:boto:Token: None 2022-09-02T04:47:08.313 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM= 2022-09-02T04:47:08.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM=', 'Content-Length': '0'} 2022-09-02T04:47:08.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084cac58d25b69582-0063118acc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.317 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.318 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.318 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.319 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.319 DEBUG:boto:Method: GET 2022-09-02T04:47:08.320 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.321 DEBUG:boto:Data: 2022-09-02T04:47:08.321 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.323 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.323 DEBUG:boto:Params: {} 2022-09-02T04:47:08.324 DEBUG:boto:Token: None 2022-09-02T04:47:08.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM= 2022-09-02T04:47:08.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM=', 'Content-Length': '0'} 2022-09-02T04:47:08.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e4a8889e56b7d34-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.329 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.331 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.332 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.332 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.334 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.335 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.336 DEBUG:boto:Data: 2022-09-02T04:47:08.337 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.338 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.339 DEBUG:boto:Params: {} 2022-09-02T04:47:08.340 DEBUG:boto:Token: None 2022-09-02T04:47:08.341 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2j3Yf3VwjKDV33d5PwitAaokHt0= 2022-09-02T04:47:08.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2j3Yf3VwjKDV33d5PwitAaokHt0=', 'Content-Length': '0'} 2022-09-02T04:47:08.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000796a508f7ff67cc3-0063118acc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.347 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.348 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.348 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.349 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.350 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.350 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.351 DEBUG:boto:Data: 2022-09-02T04:47:08.352 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.353 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.353 DEBUG:boto:Params: {} 2022-09-02T04:47:08.354 DEBUG:boto:Token: None 2022-09-02T04:47:08.355 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2j3Yf3VwjKDV33d5PwitAaokHt0= 2022-09-02T04:47:08.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2j3Yf3VwjKDV33d5PwitAaokHt0=', 'Content-Length': '0'} 2022-09-02T04:47:08.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf0693dbb59374ca-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:47:08.359 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.360 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.361 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.361 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.362 DEBUG:boto:Method: GET 2022-09-02T04:47:08.363 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.363 DEBUG:boto:Data: 2022-09-02T04:47:08.364 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.365 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.365 DEBUG:boto:Params: {} 2022-09-02T04:47:08.366 DEBUG:boto:Token: None 2022-09-02T04:47:08.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM= 2022-09-02T04:47:08.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM=', 'Content-Length': '0'} 2022-09-02T04:47:08.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5c7e866d7b9b104-0063118acc-1100-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.373 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.373 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.374 DEBUG:boto:path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.375 DEBUG:boto:auth_path=/xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.376 DEBUG:boto:Method: GET 2022-09-02T04:47:08.377 DEBUG:boto:Path: /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.378 DEBUG:boto:Data: 2022-09-02T04:47:08.378 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.380 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.381 DEBUG:boto:Params: {} 2022-09-02T04:47:08.382 DEBUG:boto:Token: None 2022-09-02T04:47:08.383 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj?versionId=MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY 2022-09-02T04:47:08.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM= 2022-09-02T04:47:08.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GerYY/dKHBHGyzhbVwbAO6JRcwM=', 'Content-Length': '0'} 2022-09-02T04:47:08.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:46:07 GMT'), ('x-amz-version-id', 'MmsYVb7PTMI6q2fqHNjK1pRdW3pv4RY'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa0f9f0526e64212-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.389 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.390 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.391 DEBUG:boto:path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.392 DEBUG:boto:auth_path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.393 DEBUG:boto:Method: GET 2022-09-02T04:47:08.394 DEBUG:boto:Path: /xljbkf-23/?prefix=obj 2022-09-02T04:47:08.394 DEBUG:boto:Data: 2022-09-02T04:47:08.395 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.397 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.397 DEBUG:boto:Params: {} 2022-09-02T04:47:08.399 DEBUG:boto:Token: None 2022-09-02T04:47:08.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/ 2022-09-02T04:47:08.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc= 2022-09-02T04:47:08.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc=', 'Content-Length': '0'} 2022-09-02T04:47:08.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a6f81811396077-0063118acc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.406 DEBUG:boto:b'xljbkf-23obj1000false' 2022-09-02T04:47:08.407 DEBUG:boto:path=/xljbkf-23/ 2022-09-02T04:47:08.408 DEBUG:boto:auth_path=/xljbkf-23/ 2022-09-02T04:47:08.409 DEBUG:boto:path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.410 DEBUG:boto:auth_path=/xljbkf-23/?prefix=obj 2022-09-02T04:47:08.411 DEBUG:boto:Method: GET 2022-09-02T04:47:08.412 DEBUG:boto:Path: /xljbkf-23/?prefix=obj 2022-09-02T04:47:08.413 DEBUG:boto:Data: 2022-09-02T04:47:08.414 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.414 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.415 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.416 DEBUG:boto:Params: {} 2022-09-02T04:47:08.417 DEBUG:boto:Token: None 2022-09-02T04:47:08.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/ 2022-09-02T04:47:08.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc= 2022-09-02T04:47:08.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XfFWx7DKwExFly4mmbg79+tG3pc=', 'Content-Length': '0'} 2022-09-02T04:47:08.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005852c01fe03d4a71-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.428 DEBUG:boto:b'xljbkf-23obj1000false' 2022-09-02T04:47:08.429 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.431 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.432 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.432 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:08.433 DEBUG:boto:Data: 2022-09-02T04:47:08.434 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.435 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.436 DEBUG:boto:Params: {} 2022-09-02T04:47:08.437 DEBUG:boto:Token: None 2022-09-02T04:47:08.437 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj 2022-09-02T04:47:08.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4= 2022-09-02T04:47:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4=', 'Content-Length': '0'} 2022-09-02T04:47:08.441 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000b6821b32d589be23-0063118acc-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.442 DEBUG:boto:path=/xljbkf-23/obj 2022-09-02T04:47:08.443 DEBUG:boto:auth_path=/xljbkf-23/obj 2022-09-02T04:47:08.444 DEBUG:boto:Method: HEAD 2022-09-02T04:47:08.445 DEBUG:boto:Path: /xljbkf-23/obj 2022-09-02T04:47:08.446 DEBUG:boto:Data: 2022-09-02T04:47:08.447 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.448 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.449 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.450 DEBUG:boto:Params: {} 2022-09-02T04:47:08.451 DEBUG:boto:Token: None 2022-09-02T04:47:08.452 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-23/obj 2022-09-02T04:47:08.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4= 2022-09-02T04:47:08.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dz6Z7kcafSVcT1/gHtMsTtLVwg4=', 'Content-Length': '0'} 2022-09-02T04:47:08.457 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d403c741b811c90a-0063118acc-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-23 zones={test-zone3, test-zone2} 2022-09-02T04:47:08.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2022-09-02T04:47:08.461 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-24 2022-09-02T04:47:08.461 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:47:08.462 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:47:08.463 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.464 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:47:08.465 DEBUG:boto:Data: 2022-09-02T04:47:08.466 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:08.468 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.469 DEBUG:boto:Params: {} 2022-09-02T04:47:08.469 DEBUG:boto:Token: None 2022-09-02T04:47:08.471 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-24/ 2022-09-02T04:47:08.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4ieh120dKxHYhK5uJr5FEfD/oY8= 2022-09-02T04:47:08.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4ieh120dKxHYhK5uJr5FEfD/oY8=', 'Content-Length': '0'} 2022-09-02T04:47:08.487 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b0ee0750f28f5316-0063118acc-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.488 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-25 2022-09-02T04:47:08.488 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:47:08.489 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:47:08.489 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.490 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:47:08.490 DEBUG:boto:Data: 2022-09-02T04:47:08.491 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.492 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.492 DEBUG:boto:Params: {} 2022-09-02T04:47:08.493 DEBUG:boto:Token: None 2022-09-02T04:47:08.494 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-25/ 2022-09-02T04:47:08.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KibJQOkk+xKI8/GHKrAz7V6oZYk= 2022-09-02T04:47:08.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KibJQOkk+xKI8/GHKrAz7V6oZYk=', 'Content-Length': '0'} 2022-09-02T04:47:08.528 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e130e3ca9cb63128-0063118acc-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.529 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-26 2022-09-02T04:47:08.530 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:47:08.530 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:47:08.531 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.531 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:47:08.532 DEBUG:boto:Data: 2022-09-02T04:47:08.533 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.537 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.537 DEBUG:boto:Params: {} 2022-09-02T04:47:08.538 DEBUG:boto:Token: None 2022-09-02T04:47:08.538 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-26/ 2022-09-02T04:47:08.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/NKn3hXO/CrsOav774gC6xdUr4I= 2022-09-02T04:47:08.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/NKn3hXO/CrsOav774gC6xdUr4I=', 'Content-Length': '0'} 2022-09-02T04:47:08.564 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1fea1aec05a0d5a-0063118acc-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.565 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:47:08.566 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:47:08.568 DEBUG:boto:path=/xljbkf-24/?versioning 2022-09-02T04:47:08.569 DEBUG:boto:auth_path=/xljbkf-24/?versioning 2022-09-02T04:47:08.570 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.571 DEBUG:boto:Path: /xljbkf-24/?versioning 2022-09-02T04:47:08.572 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:47:08.574 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:08.575 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.576 DEBUG:boto:Params: {} 2022-09-02T04:47:08.577 DEBUG:boto:Token: None 2022-09-02T04:47:08.578 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-24/?versioning 2022-09-02T04:47:08.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gp3su6bvLrC07v3fr2RG33AmwmQ= 2022-09-02T04:47:08.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gp3su6bvLrC07v3fr2RG33AmwmQ=', 'Content-Length': '228'} 2022-09-02T04:47:08.588 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000884bf2e64345954-0063118acc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.589 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:47:08.590 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:47:08.593 DEBUG:boto:path=/xljbkf-24/?versioning 2022-09-02T04:47:08.594 DEBUG:boto:auth_path=/xljbkf-24/?versioning 2022-09-02T04:47:08.595 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.597 DEBUG:boto:Path: /xljbkf-24/?versioning 2022-09-02T04:47:08.598 DEBUG:boto:Data: Suspended Disabled 2022-09-02T04:47:08.599 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:08.601 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.601 DEBUG:boto:Params: {} 2022-09-02T04:47:08.602 DEBUG:boto:Token: None 2022-09-02T04:47:08.603 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-24/?versioning 2022-09-02T04:47:08.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gp3su6bvLrC07v3fr2RG33AmwmQ= 2022-09-02T04:47:08.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gp3su6bvLrC07v3fr2RG33AmwmQ=', 'Content-Length': '230'} 2022-09-02T04:47:08.614 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081318838b10899e4-0063118acc-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.615 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:47:08.616 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:47:08.617 DEBUG:boto:path=/xljbkf-25/?versioning 2022-09-02T04:47:08.618 DEBUG:boto:auth_path=/xljbkf-25/?versioning 2022-09-02T04:47:08.619 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.620 DEBUG:boto:Path: /xljbkf-25/?versioning 2022-09-02T04:47:08.621 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:47:08.622 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.623 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.628 DEBUG:boto:Params: {} 2022-09-02T04:47:08.629 DEBUG:boto:Token: None 2022-09-02T04:47:08.629 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-25/?versioning 2022-09-02T04:47:08.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:izk+NkhZ/jOLLxwi4q6GnA8dMB0= 2022-09-02T04:47:08.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:izk+NkhZ/jOLLxwi4q6GnA8dMB0=', 'Content-Length': '228'} 2022-09-02T04:47:08.642 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000330afc816063f996-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.643 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:47:08.644 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:47:08.645 DEBUG:boto:path=/xljbkf-25/?versioning 2022-09-02T04:47:08.646 DEBUG:boto:auth_path=/xljbkf-25/?versioning 2022-09-02T04:47:08.647 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.648 DEBUG:boto:Path: /xljbkf-25/?versioning 2022-09-02T04:47:08.649 DEBUG:boto:Data: Suspended Disabled 2022-09-02T04:47:08.651 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:08.653 DEBUG:boto:Port: 8000 2022-09-02T04:47:08.654 DEBUG:boto:Params: {} 2022-09-02T04:47:08.655 DEBUG:boto:Token: None 2022-09-02T04:47:08.655 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-25/?versioning 2022-09-02T04:47:08.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:izk+NkhZ/jOLLxwi4q6GnA8dMB0= 2022-09-02T04:47:08.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:izk+NkhZ/jOLLxwi4q6GnA8dMB0=', 'Content-Length': '230'} 2022-09-02T04:47:08.666 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be6473cc704def18-0063118acc-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.667 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:47:08.668 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:47:08.668 DEBUG:boto:path=/xljbkf-26/?versioning 2022-09-02T04:47:08.669 DEBUG:boto:auth_path=/xljbkf-26/?versioning 2022-09-02T04:47:08.670 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.670 DEBUG:boto:Path: /xljbkf-26/?versioning 2022-09-02T04:47:08.671 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:47:08.671 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.672 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.673 DEBUG:boto:Params: {} 2022-09-02T04:47:08.674 DEBUG:boto:Token: None 2022-09-02T04:47:08.675 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-26/?versioning 2022-09-02T04:47:08.678 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IeLuqwgFTSAIGbKcV9JReQMm7nY= 2022-09-02T04:47:08.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IeLuqwgFTSAIGbKcV9JReQMm7nY=', 'Content-Length': '228'} 2022-09-02T04:47:08.689 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1d77883eaf7c9e3-0063118acc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.690 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:47:08.690 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:47:08.691 DEBUG:boto:path=/xljbkf-26/?versioning 2022-09-02T04:47:08.691 DEBUG:boto:auth_path=/xljbkf-26/?versioning 2022-09-02T04:47:08.692 DEBUG:boto:Method: PUT 2022-09-02T04:47:08.692 DEBUG:boto:Path: /xljbkf-26/?versioning 2022-09-02T04:47:08.693 DEBUG:boto:Data: Suspended Disabled 2022-09-02T04:47:08.693 DEBUG:boto:Headers: {} 2022-09-02T04:47:08.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:08.694 DEBUG:boto:Port: 8001 2022-09-02T04:47:08.695 DEBUG:boto:Params: {} 2022-09-02T04:47:08.695 DEBUG:boto:Token: None 2022-09-02T04:47:08.696 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:47:08 GMT /xljbkf-26/?versioning 2022-09-02T04:47:08.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IeLuqwgFTSAIGbKcV9JReQMm7nY= 2022-09-02T04:47:08.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IeLuqwgFTSAIGbKcV9JReQMm7nY=', 'Content-Length': '230'} 2022-09-02T04:47:08.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000131f865028d81863-0063118acc-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:08.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:08.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:08.717 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:08.772 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:08.773 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:08.806 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:47:08.833 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:47:08.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:08.834 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:08.835 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:47:08.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:08.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:08.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:47:08.836 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:47:08.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:08.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:08.837 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094028.711326_201.1", 2022-09-02T04:47:08.838 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:47:08.711326Z" 2022-09-02T04:47:08.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:08.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:08.839 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:08.839 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:47:08.839 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:08.840 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:47:08.841 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:08.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:47:08.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:08.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:08.845 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:08.899 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:08.900 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:08.931+0000 7f096816b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:08.936 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:08.932+0000 7f096816b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:08.951 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:47:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:47:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:47:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:47:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:47:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:47:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:47:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:47:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:47:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:47:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:47:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:47:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094028.508152_182.1", 2022-09-02T04:47:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.508152Z", 2022-09-02T04:47:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:47:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:47:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:47:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:47:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:47:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:47:08Z" 2022-09-02T04:47:08.983 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:47:08.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.508152_182.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.508152Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:08Z" } 2022-09-02T04:47:08.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.508152_182.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.508152Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:08.985 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:08.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.508152_182.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:08.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094028.711326_201.1 target=1_1662094028.508152_182.1 2022-09-02T04:47:13.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:13.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:13.990 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:14.052 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:14.052 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:14.088 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:14.087+0000 7f5d35b15600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:14.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:14.089+0000 7f5d35b15600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:14.102 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:47:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:47:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:47:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:47:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:47:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:47:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:47:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:47:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:14.115 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:14.115 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:14.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:14.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:14.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:14.119 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:47:14.119 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:14.120 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:14.120 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:47:14.121 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:14.121 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:14.122 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:14.123 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:47:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:14.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:14.130 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:47:14.130 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:14.131 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:14.131 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094028.512912_184.1", 2022-09-02T04:47:14.132 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:14.132 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:14.133 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:14.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.512912Z", 2022-09-02T04:47:14.134 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:14.135 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:14.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:14.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:14.136 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:47:14.137 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:14.137 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:14.138 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:14.138 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:14.138 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:14.139 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:14.139 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:14.140 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:14.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:14.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:14.140 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:47:14.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:14.141 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:47:14.141 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:47:14.141 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:47:14.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:14.142 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:47:14Z" 2022-09-02T04:47:14.142 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:47:14.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.512912_184.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.512912Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:14Z" } 2022-09-02T04:47:14.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.512912_184.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.512912Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:14.145 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:14.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.512912_184.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:14.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094028.711326_201.1 target=1_1662094028.512912_184.1 2022-09-02T04:47:19.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:19.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:19.150 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:19.204 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:19.205 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:19.245 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:19.245+0000 7f5ac1c3f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:19.246 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:19.246+0000 7f5ac1c3f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:19.260 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:47:19.261 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:47:19.261 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:47:19.261 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:47:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:47:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:19.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:47:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:47:19.264 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:19.265 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:47:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:19.266 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:47:19.267 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:19.294 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:19.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:47:19.295 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:19.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:19.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:19.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:47:19.297 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:19.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:19.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:19.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:19.300 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:47:19.301 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:19.307 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:19.307 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094028.508152_182.1", 2022-09-02T04:47:19.307 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:19.308 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:19.308 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:19.309 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.508152Z", 2022-09-02T04:47:19.309 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:19.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:19.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:19.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:19.315 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:47:19.316 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:19.316 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:19.317 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:19.317 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:19.320 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:19.321 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:19.321 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:19.322 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:19.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:19.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:19.324 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:47:19.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:19.325 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:47:19.326 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:47:19.326 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:47:19.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:19.331 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:47:19Z" 2022-09-02T04:47:19.332 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:47:19.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.508152_182.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.508152Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:19Z" } 2022-09-02T04:47:19.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.508152_182.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.508152Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:19.334 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:19.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.508152_182.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:19.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094028.711326_201.1 target=1_1662094028.508152_182.1 2022-09-02T04:47:24.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:24.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:24.347 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:24.403 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:24.403 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:24.441 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:24.442+0000 7f24816b3600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:24.443+0000 7f24816b3600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:24.458 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:47:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:47:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:47:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:47:24.459 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:47:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:24.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:47:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:47:24.461 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:24.462 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:24.462 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:24.462 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:24.464 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:47:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:24.465 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:24.466 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:47:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:47:24.467 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:24.468 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:47:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:24.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:47:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:24.470 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:24.471 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094028.508152_182.1", 2022-09-02T04:47:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:24.472 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.508152Z", 2022-09-02T04:47:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:24.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:47:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:24.474 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:24.475 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:24.476 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:47:24.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:24.478 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:47:24.478 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:47:24.479 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:47:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:47:24Z" 2022-09-02T04:47:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:47:24.482 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.508152_182.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.508152Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:24Z" } 2022-09-02T04:47:24.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.508152_182.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.508152Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:24.483 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:24.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.508152_182.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:24.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094028.711326_201.1 target=1_1662094028.508152_182.1 2022-09-02T04:47:29.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:29.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:29.488 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:29.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:29.550 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:29.587 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:29.587+0000 7f2c5067d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:29.591 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:29.588+0000 7f2c5067d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:29.606 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:47:29.607 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:47:29.607 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:47:29.608 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:47:29.608 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:47:29.609 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:29.609 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:29.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:29.610 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:47:29.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:29.611 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:47:29.613 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:29.614 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:29.614 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:29.615 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:29.615 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:47:29.616 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:29.616 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:29.617 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:29.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:29.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:29.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:29.619 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:47:29.619 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:29.620 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:29.620 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:47:29.621 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:29.622 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:29.622 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:29.623 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:47:29.623 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:29.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:29.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:29.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:29.625 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:47:29.625 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:29.625 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:29.626 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094028.711326_201.1", 2022-09-02T04:47:29.626 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:29.627 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:29.627 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:29.628 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.711326Z", 2022-09-02T04:47:29.628 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:29.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:29.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:29.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:29.631 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:47:29.631 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:29.632 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:29.632 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:29.633 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:29.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:29.634 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:29.634 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:29.635 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:29.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:29.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:29.636 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:47:29.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:29.637 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:47:29.638 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:47:29.638 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:47:29.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:29.639 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:47:29Z" 2022-09-02T04:47:29.639 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:47:29.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.711326_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.711326Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:29Z" } 2022-09-02T04:47:29.641 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.711326_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.711326Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:29.642 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:29.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:29.644 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:47:29.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:47:29.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:29.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:29.648 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:29.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:29.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:29.733+0000 7f437e9fd600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:29.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:29.734+0000 7f437e9fd600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:29.788 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:47:29.789 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:47:29.789 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:47:29.789 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:47:29.790 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:47:29.790 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:29.790 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:29.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:29.791 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:47:29.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:29.792 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:47:29.792 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:29.792 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:29.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:29.793 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:29.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:47:29.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:29.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:29.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:29.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:29.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:29.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:29.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:47:29.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:29.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:29.800 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:47:29.800 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:29.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:47:29.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:29.802 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:47:29.803 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:29.804 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:29.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:29.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:29.806 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:47:29.806 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:29.807 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:29.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094028.480028_178.1", 2022-09-02T04:47:29.811 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:29.811 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:47:29.813 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:29.814 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.480028Z", 2022-09-02T04:47:29.814 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:29.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:29.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:29.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:29.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:47:29.817 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:29.829 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:29.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:29.831 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:29.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:47:29.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:29.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:29.833 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:29.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:29.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:29.837 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:47:29.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:29.839 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:47:29.839 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:47:29.840 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:47:29.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:29.841 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:47:29Z" 2022-09-02T04:47:29.841 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:47:29.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.480028_178.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.480028Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:29Z" } 2022-09-02T04:47:29.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.480028_178.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.480028Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:29.845 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:29.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.480028_178.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:29.847 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1662094028.711326_201.1 target=1_1662094028.480028_178.1 2022-09-02T04:47:34.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:34.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:34.859 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:34.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:34.919 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:34.964 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:34.961+0000 7fa67546b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:34.965 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:34.963+0000 7fa67546b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:34.981 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:47:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:47:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:47:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:47:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:47:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:47:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:47:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:47:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:34.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:34.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:34.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:34.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:34.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:34.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:47:34.995 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:34.995 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:34.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:47:34.998 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:34.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:47:34.999 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:34.999 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:47:35.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:35.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:35.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:35.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:35.003 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:47:35.003 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:35.004 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:35.004 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094028.711326_201.1", 2022-09-02T04:47:35.004 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:35.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:47:35.006 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:35.006 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.711326Z", 2022-09-02T04:47:35.007 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:35.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:47:35.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:47:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:47:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:47:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:47:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:47:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:47:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:35.018 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:47:35.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:35.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:47:35.020 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:47:35.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:35.021 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:47:35.021 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:47:35.022 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:47:35.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:47:35.023 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:47:34Z" 2022-09-02T04:47:35.023 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:47:35.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.711326_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.711326Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:34Z" } 2022-09-02T04:47:35.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.711326_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.711326Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:35.026 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:35.027 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:35.028 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:47:35.028 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:47:35.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:35.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:47:35.032 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:47:35.089 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:35.090 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:35.135 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:35.132+0000 7fd0be264600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:35.136 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:35.134+0000 7fd0be264600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:47:35.145 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:47:35.145 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:47:35.145 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:47:35.146 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:47:35.146 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:47:35.147 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:47:35.147 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:35.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:35.148 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:47:35.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:35.149 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:47:35.149 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:35.150 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:35.150 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:47:35.151 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:35.151 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:47:35.152 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:35.152 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:47:35.153 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:35.153 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:35.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:35.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:35.155 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:47:35.156 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:35.156 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:35.157 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:47:35.157 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:35.158 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:35.158 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:35.159 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:47:35.159 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:35.160 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:35.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:35.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:35.161 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:47:35.161 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:35.162 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:35.172 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094028.711326_201.1", 2022-09-02T04:47:35.173 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:35.174 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:35.174 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:35.175 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.711326Z", 2022-09-02T04:47:35.175 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:35.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:35.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:35.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:47:35.180 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:47:35.180 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:47:35.181 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:47:35.182 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:47:35.182 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:47:35.183 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:47:35.183 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:47:35.184 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:47:35.184 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:47:35.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:35.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:47:35.189 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:47:35.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:35.191 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:47:35.192 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:47:35.193 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:47:35.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:47:35.193 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:47:35Z" 2022-09-02T04:47:35.195 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:47:35.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094028.711326_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:08.711326Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:47:35Z" } 2022-09-02T04:47:35.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094028.711326_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:08.711326Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:47:35.200 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:35.201 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094028.711326_201.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:47:35.202 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:47:35.203 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:47:35.204 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:47:35.205 DEBUG:boto:Method: HEAD 2022-09-02T04:47:35.206 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:47:35.207 DEBUG:boto:Data: 2022-09-02T04:47:35.208 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:35.209 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.210 DEBUG:boto:Params: {} 2022-09-02T04:47:35.212 DEBUG:boto:Token: None 2022-09-02T04:47:35.213 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-24/ 2022-09-02T04:47:35.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nxTIqe4qmoAzNi5aa9aCw+cSGiw= 2022-09-02T04:47:35.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nxTIqe4qmoAzNi5aa9aCw+cSGiw=', 'Content-Length': '0'} 2022-09-02T04:47:35.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea4103b9fbb7042f-0063118ae7-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.221 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:47:35.226 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:47:35.227 DEBUG:boto:Method: PUT 2022-09-02T04:47:35.228 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:47:35.231 DEBUG:boto:Data: 2022-09-02T04:47:35.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:47:35.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:35.234 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.235 DEBUG:boto:Params: {} 2022-09-02T04:47:35.236 DEBUG:boto:Token: None 2022-09-02T04:47:35.237 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-24/obj 2022-09-02T04:47:35.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9muZ6pOORtmEluFEMC96Cx6hM5c= 2022-09-02T04:47:35.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9muZ6pOORtmEluFEMC96Cx6hM5c='} 2022-09-02T04:47:35.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3a19b001f0a9371-0063118ae7-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.251 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:47:35.252 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:47:35.252 DEBUG:boto:Method: HEAD 2022-09-02T04:47:35.253 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:47:35.253 DEBUG:boto:Data: 2022-09-02T04:47:35.254 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:35.255 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.255 DEBUG:boto:Params: {} 2022-09-02T04:47:35.256 DEBUG:boto:Token: None 2022-09-02T04:47:35.256 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-24/ 2022-09-02T04:47:35.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nxTIqe4qmoAzNi5aa9aCw+cSGiw= 2022-09-02T04:47:35.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nxTIqe4qmoAzNi5aa9aCw+cSGiw=', 'Content-Length': '0'} 2022-09-02T04:47:35.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073c65da5f944c2fb-0063118ae7-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.273 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:47:35.273 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:47:35.274 DEBUG:boto:Method: DELETE 2022-09-02T04:47:35.275 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:47:35.275 DEBUG:boto:Data: 2022-09-02T04:47:35.276 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:47:35.277 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.277 DEBUG:boto:Params: {} 2022-09-02T04:47:35.278 DEBUG:boto:Token: None 2022-09-02T04:47:35.279 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-24/obj 2022-09-02T04:47:35.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q2tJxXkhnyRFgwj2OgqJeJVRIe8= 2022-09-02T04:47:35.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q2tJxXkhnyRFgwj2OgqJeJVRIe8=', 'Content-Length': '0'} 2022-09-02T04:47:35.291 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000584665da8a79a1cd-0063118ae7-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.292 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:47:35.293 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:47:35.296 DEBUG:boto:Method: HEAD 2022-09-02T04:47:35.297 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:47:35.300 DEBUG:boto:Data: 2022-09-02T04:47:35.301 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:35.303 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.304 DEBUG:boto:Params: {} 2022-09-02T04:47:35.308 DEBUG:boto:Token: None 2022-09-02T04:47:35.309 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-25/ 2022-09-02T04:47:35.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HCPPpW4ijwSftBO20YqtzWqwZkM= 2022-09-02T04:47:35.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HCPPpW4ijwSftBO20YqtzWqwZkM=', 'Content-Length': '0'} 2022-09-02T04:47:35.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031124f040dba98c9-0063118ae7-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.327 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:47:35.328 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:47:35.329 DEBUG:boto:Method: PUT 2022-09-02T04:47:35.330 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:47:35.331 DEBUG:boto:Data: 2022-09-02T04:47:35.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:47:35.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:35.337 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.339 DEBUG:boto:Params: {} 2022-09-02T04:47:35.340 DEBUG:boto:Token: None 2022-09-02T04:47:35.341 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-25/obj 2022-09-02T04:47:35.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xsu1mRMzuW/psOAzaBxhNZaIaw0= 2022-09-02T04:47:35.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xsu1mRMzuW/psOAzaBxhNZaIaw0='} 2022-09-02T04:47:35.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc6ffb0145d32169-0063118ae7-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.352 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:47:35.352 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:47:35.353 DEBUG:boto:Method: HEAD 2022-09-02T04:47:35.353 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:47:35.354 DEBUG:boto:Data: 2022-09-02T04:47:35.354 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:35.355 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.356 DEBUG:boto:Params: {} 2022-09-02T04:47:35.357 DEBUG:boto:Token: None 2022-09-02T04:47:35.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-25/ 2022-09-02T04:47:35.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HCPPpW4ijwSftBO20YqtzWqwZkM= 2022-09-02T04:47:35.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HCPPpW4ijwSftBO20YqtzWqwZkM=', 'Content-Length': '0'} 2022-09-02T04:47:35.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1ba8daad66e37ec-0063118ae7-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.363 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:47:35.364 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:47:35.365 DEBUG:boto:Method: DELETE 2022-09-02T04:47:35.365 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:47:35.366 DEBUG:boto:Data: 2022-09-02T04:47:35.366 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:47:35.369 DEBUG:boto:Port: 8000 2022-09-02T04:47:35.369 DEBUG:boto:Params: {} 2022-09-02T04:47:35.371 DEBUG:boto:Token: None 2022-09-02T04:47:35.372 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-25/obj 2022-09-02T04:47:35.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ldoG++k/9TroSZqVC2LBGd9I6JA= 2022-09-02T04:47:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ldoG++k/9TroSZqVC2LBGd9I6JA=', 'Content-Length': '0'} 2022-09-02T04:47:35.384 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000e88fc1f8c9d3fc03-0063118ae7-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.385 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:47:35.386 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:47:35.387 DEBUG:boto:Method: HEAD 2022-09-02T04:47:35.387 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:47:35.388 DEBUG:boto:Data: 2022-09-02T04:47:35.388 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:35.390 DEBUG:boto:Port: 8001 2022-09-02T04:47:35.391 DEBUG:boto:Params: {} 2022-09-02T04:47:35.392 DEBUG:boto:Token: None 2022-09-02T04:47:35.392 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-26/ 2022-09-02T04:47:35.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Mn9kbI93L5OwrleNrLVWEQbeyE= 2022-09-02T04:47:35.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Mn9kbI93L5OwrleNrLVWEQbeyE=', 'Content-Length': '0'} 2022-09-02T04:47:35.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b619ba84cde4c16c-0063118ae7-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.399 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:47:35.400 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:47:35.401 DEBUG:boto:Method: PUT 2022-09-02T04:47:35.402 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:47:35.403 DEBUG:boto:Data: 2022-09-02T04:47:35.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2022-09-02T04:47:35.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:35.406 DEBUG:boto:Port: 8001 2022-09-02T04:47:35.407 DEBUG:boto:Params: {} 2022-09-02T04:47:35.408 DEBUG:boto:Token: None 2022-09-02T04:47:35.409 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-26/obj 2022-09-02T04:47:35.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OkYDTpNG0oqGs/oi72BS2I9hZok= 2022-09-02T04:47:35.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OkYDTpNG0oqGs/oi72BS2I9hZok='} 2022-09-02T04:47:35.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d452b63fb32ed8c7-0063118ae7-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.420 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:47:35.421 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:47:35.422 DEBUG:boto:Method: HEAD 2022-09-02T04:47:35.425 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:47:35.426 DEBUG:boto:Data: 2022-09-02T04:47:35.427 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:35.429 DEBUG:boto:Port: 8001 2022-09-02T04:47:35.431 DEBUG:boto:Params: {} 2022-09-02T04:47:35.432 DEBUG:boto:Token: None 2022-09-02T04:47:35.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-26/ 2022-09-02T04:47:35.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Mn9kbI93L5OwrleNrLVWEQbeyE= 2022-09-02T04:47:35.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Mn9kbI93L5OwrleNrLVWEQbeyE=', 'Content-Length': '0'} 2022-09-02T04:47:35.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5481215481901da-0063118ae7-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.439 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:47:35.440 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:47:35.441 DEBUG:boto:Method: DELETE 2022-09-02T04:47:35.442 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:47:35.443 DEBUG:boto:Data: 2022-09-02T04:47:35.444 DEBUG:boto:Headers: {} 2022-09-02T04:47:35.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:47:35.447 DEBUG:boto:Port: 8001 2022-09-02T04:47:35.448 DEBUG:boto:Params: {} 2022-09-02T04:47:35.449 DEBUG:boto:Token: None 2022-09-02T04:47:35.450 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:47:35 GMT /xljbkf-26/obj 2022-09-02T04:47:35.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kbnK+x1bHi5qdlz+IffFGCV7BJM= 2022-09-02T04:47:35.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:47:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kbnK+x1bHi5qdlz+IffFGCV7BJM=', 'Content-Length': '0'} 2022-09-02T04:47:35.464 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000b2d8e519aff10a82-0063118ae7-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:47:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:47:35.465 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-24 2022-09-02T04:47:35.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:35.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:35.468 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:35.535 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:35.535 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:35.565 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:35.566+0000 7f53c9cd1600 1 waiting to reach incremental sync.. 2022-09-02T04:47:40.570 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:40.568+0000 7f53c9cd1600 1 bucket sync caught up with source: 2022-09-02T04:47:40.571 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.12.6, , ] 2022-09-02T04:47:40.572 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.12.6, , ] 2022-09-02T04:47:40.572 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:40.568+0000 7f53c9cd1600 0 bucket checkpoint complete 2022-09-02T04:47:40.591 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-24 2022-09-02T04:47:40.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:40.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:40.596 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:40.703 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:40.702+0000 7f1fd3d67600 1 bucket sync caught up with source: 2022-09-02T04:47:40.703 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.12.6, , ] 2022-09-02T04:47:40.704 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.12.6, , ] 2022-09-02T04:47:40.704 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:40.702+0000 7f1fd3d67600 0 bucket checkpoint complete 2022-09-02T04:47:40.716 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-24 2022-09-02T04:47:40.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:40.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:40.718 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:40.779 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:40.779 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:40.833 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:40.831+0000 7f3f95c67600 1 waiting to reach incremental sync.. 2022-09-02T04:47:45.833 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:45.832+0000 7f3f95c67600 1 waiting to reach incremental sync.. 2022-09-02T04:47:50.833 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:50.834+0000 7f3f95c67600 1 bucket sync caught up with source: 2022-09-02T04:47:50.834 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.12.6, , ] 2022-09-02T04:47:50.834 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.12.6, , ] 2022-09-02T04:47:50.834 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:47:50.834+0000 7f3f95c67600 0 bucket checkpoint complete 2022-09-02T04:47:50.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-24 2022-09-02T04:47:50.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:50.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:47:50.852 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:47:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:47:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:47:50.954 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:50.954+0000 7fd28e2fe600 1 waiting to reach incremental sync.. 2022-09-02T04:47:55.954 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:47:55.955+0000 7fd28e2fe600 1 waiting to reach incremental sync.. 2022-09-02T04:48:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:00.956+0000 7fd28e2fe600 1 bucket sync caught up with source: 2022-09-02T04:48:00.957 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.22.12, , ] 2022-09-02T04:48:00.957 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2022-09-02T04:48:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:00.956+0000 7fd28e2fe600 0 bucket checkpoint complete 2022-09-02T04:48:00.970 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-24 2022-09-02T04:48:00.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:00.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:00.979 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:01.077 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:01.077+0000 7f2d9bda4600 1 bucket sync caught up with source: 2022-09-02T04:48:01.077 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.22.12, , ] 2022-09-02T04:48:01.078 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2022-09-02T04:48:01.078 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:01.077+0000 7f2d9bda4600 0 bucket checkpoint complete 2022-09-02T04:48:01.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-24 2022-09-02T04:48:01.100 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-24 2022-09-02T04:48:01.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.102 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:01.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:01.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:01.200+0000 7f01a29d8600 1 bucket sync caught up with source: 2022-09-02T04:48:01.200 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2022-09-02T04:48:01.201 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2022-09-02T04:48:01.202 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:01.200+0000 7f01a29d8600 0 bucket checkpoint complete 2022-09-02T04:48:01.216 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-24 2022-09-02T04:48:01.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.218 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:01.278 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:01.279 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:01.331 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:01.332+0000 7f2c1d9c5600 1 bucket sync caught up with source: 2022-09-02T04:48:01.332 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2022-09-02T04:48:01.332 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2022-09-02T04:48:01.332 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:01.332+0000 7f2c1d9c5600 0 bucket checkpoint complete 2022-09-02T04:48:01.345 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-24 2022-09-02T04:48:01.346 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-24 zones={test-zone2, test-zone1} 2022-09-02T04:48:01.348 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.349 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.350 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.351 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:48:01.353 DEBUG:boto:Data: 2022-09-02T04:48:01.353 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.355 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.356 DEBUG:boto:Params: {} 2022-09-02T04:48:01.357 DEBUG:boto:Token: None 2022-09-02T04:48:01.359 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/ 2022-09-02T04:48:01.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c= 2022-09-02T04:48:01.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c=', 'Content-Length': '0'} 2022-09-02T04:48:01.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d08209b6d0b5ba7-0063118b01-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.368 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.369 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.369 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.370 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:48:01.370 DEBUG:boto:Data: 2022-09-02T04:48:01.371 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.372 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.372 DEBUG:boto:Params: {} 2022-09-02T04:48:01.373 DEBUG:boto:Token: None 2022-09-02T04:48:01.375 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/ 2022-09-02T04:48:01.376 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c= 2022-09-02T04:48:01.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c=', 'Content-Length': '0'} 2022-09-02T04:48:01.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b42b157c209121d-0063118b01-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:01.384 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.384 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.385 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.389 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.390 DEBUG:boto:Method: GET 2022-09-02T04:48:01.390 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.391 DEBUG:boto:Data: 2022-09-02T04:48:01.391 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.392 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.393 DEBUG:boto:Params: {} 2022-09-02T04:48:01.396 DEBUG:boto:Token: None 2022-09-02T04:48:01.397 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014f315d2726f5d7c-0063118b01-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.407 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:01.409 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:01.410 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.411 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.411 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.412 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.412 DEBUG:boto:Method: GET 2022-09-02T04:48:01.413 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.413 DEBUG:boto:Data: 2022-09-02T04:48:01.414 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.415 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.415 DEBUG:boto:Params: {} 2022-09-02T04:48:01.416 DEBUG:boto:Token: None 2022-09-02T04:48:01.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004029b9ea6633c40c-0063118b01-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.425 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:01.429 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.429 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.430 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.431 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.432 DEBUG:boto:Method: GET 2022-09-02T04:48:01.433 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.434 DEBUG:boto:Data: 2022-09-02T04:48:01.436 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.437 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.438 DEBUG:boto:Params: {} 2022-09-02T04:48:01.439 DEBUG:boto:Token: None 2022-09-02T04:48:01.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078474021ba6bb372-0063118b01-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.452 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.453 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.454 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.455 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.455 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.458 DEBUG:boto:Method: GET 2022-09-02T04:48:01.459 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.460 DEBUG:boto:Data: 2022-09-02T04:48:01.461 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.462 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.463 DEBUG:boto:Params: {} 2022-09-02T04:48:01.463 DEBUG:boto:Token: None 2022-09-02T04:48:01.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f17195fbea9918a3-0063118b01-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.471 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:01.474 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:48:01.474 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:48:01.475 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.476 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:48:01.477 DEBUG:boto:Data: 2022-09-02T04:48:01.478 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.481 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.482 DEBUG:boto:Params: {} 2022-09-02T04:48:01.483 DEBUG:boto:Token: None 2022-09-02T04:48:01.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/obj 2022-09-02T04:48:01.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU= 2022-09-02T04:48:01.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU=', 'Content-Length': '0'} 2022-09-02T04:48:01.492 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006d05571c3f203a29-0063118b01-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.494 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:48:01.495 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:48:01.496 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.496 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:48:01.497 DEBUG:boto:Data: 2022-09-02T04:48:01.498 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.508 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.510 DEBUG:boto:Params: {} 2022-09-02T04:48:01.511 DEBUG:boto:Token: None 2022-09-02T04:48:01.512 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/obj 2022-09-02T04:48:01.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU= 2022-09-02T04:48:01.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU=', 'Content-Length': '0'} 2022-09-02T04:48:01.519 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000004fc09c789dc03cbb-0063118b01-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-24 zones={test-zone2, test-zone1} 2022-09-02T04:48:01.520 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-24 zones={test-zone3, test-zone1} 2022-09-02T04:48:01.521 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.522 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.522 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.523 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:48:01.523 DEBUG:boto:Data: 2022-09-02T04:48:01.524 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.525 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.525 DEBUG:boto:Params: {} 2022-09-02T04:48:01.526 DEBUG:boto:Token: None 2022-09-02T04:48:01.527 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/ 2022-09-02T04:48:01.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c= 2022-09-02T04:48:01.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c=', 'Content-Length': '0'} 2022-09-02T04:48:01.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a248dea6331ba67-0063118b01-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.532 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.533 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.534 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.535 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:48:01.535 DEBUG:boto:Data: 2022-09-02T04:48:01.536 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.540 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.540 DEBUG:boto:Params: {} 2022-09-02T04:48:01.541 DEBUG:boto:Token: None 2022-09-02T04:48:01.542 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/ 2022-09-02T04:48:01.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c= 2022-09-02T04:48:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c=', 'Content-Length': '0'} 2022-09-02T04:48:01.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1e0bda78c06e046-0063118b01-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.549 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:01.550 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.551 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.552 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.553 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.554 DEBUG:boto:Method: GET 2022-09-02T04:48:01.555 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.556 DEBUG:boto:Data: 2022-09-02T04:48:01.556 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.557 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.558 DEBUG:boto:Params: {} 2022-09-02T04:48:01.559 DEBUG:boto:Token: None 2022-09-02T04:48:01.559 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2883264ad2ab274-0063118b01-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.565 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:01.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:01.567 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.568 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.569 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.569 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.570 DEBUG:boto:Method: GET 2022-09-02T04:48:01.571 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.571 DEBUG:boto:Data: 2022-09-02T04:48:01.572 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.573 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.574 DEBUG:boto:Params: {} 2022-09-02T04:48:01.575 DEBUG:boto:Token: None 2022-09-02T04:48:01.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d5bc59b92e4f57-0063118b01-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.622 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:01.624 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.624 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.625 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.626 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.626 DEBUG:boto:Method: GET 2022-09-02T04:48:01.627 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.628 DEBUG:boto:Data: 2022-09-02T04:48:01.628 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.630 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.631 DEBUG:boto:Params: {} 2022-09-02T04:48:01.631 DEBUG:boto:Token: None 2022-09-02T04:48:01.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b954707dc7fbc9b6-0063118b01-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.639 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.640 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.641 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.641 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.642 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.643 DEBUG:boto:Method: GET 2022-09-02T04:48:01.643 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.644 DEBUG:boto:Data: 2022-09-02T04:48:01.644 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.645 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.647 DEBUG:boto:Params: {} 2022-09-02T04:48:01.647 DEBUG:boto:Token: None 2022-09-02T04:48:01.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.649 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c6f262138a49969-0063118b01-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.653 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:01.655 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:48:01.655 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:48:01.656 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.656 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:48:01.657 DEBUG:boto:Data: 2022-09-02T04:48:01.657 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.658 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.659 DEBUG:boto:Params: {} 2022-09-02T04:48:01.660 DEBUG:boto:Token: None 2022-09-02T04:48:01.660 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/obj 2022-09-02T04:48:01.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU= 2022-09-02T04:48:01.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU=', 'Content-Length': '0'} 2022-09-02T04:48:01.663 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003f3f9ea05690c011-0063118b01-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.664 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:48:01.665 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:48:01.665 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.666 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:48:01.666 DEBUG:boto:Data: 2022-09-02T04:48:01.667 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:01.668 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.668 DEBUG:boto:Params: {} 2022-09-02T04:48:01.669 DEBUG:boto:Token: None 2022-09-02T04:48:01.669 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/obj 2022-09-02T04:48:01.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU= 2022-09-02T04:48:01.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU=', 'Content-Length': '0'} 2022-09-02T04:48:01.672 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003b28335b2be76aed-0063118b01-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.673 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-24 zones={test-zone3, test-zone1} 2022-09-02T04:48:01.673 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-24 zones={test-zone3, test-zone2} 2022-09-02T04:48:01.674 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.675 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.675 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.676 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:48:01.676 DEBUG:boto:Data: 2022-09-02T04:48:01.677 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.678 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.678 DEBUG:boto:Params: {} 2022-09-02T04:48:01.679 DEBUG:boto:Token: None 2022-09-02T04:48:01.680 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/ 2022-09-02T04:48:01.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c= 2022-09-02T04:48:01.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c=', 'Content-Length': '0'} 2022-09-02T04:48:01.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b63d81da275476c-0063118b01-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.684 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.684 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.685 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.686 DEBUG:boto:Path: /xljbkf-24/ 2022-09-02T04:48:01.686 DEBUG:boto:Data: 2022-09-02T04:48:01.687 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.688 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.688 DEBUG:boto:Params: {} 2022-09-02T04:48:01.689 DEBUG:boto:Token: None 2022-09-02T04:48:01.689 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/ 2022-09-02T04:48:01.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c= 2022-09-02T04:48:01.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BZwOpdRSK2ncFVdTzKzAvJf2A6c=', 'Content-Length': '0'} 2022-09-02T04:48:01.693 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004be1c635f7b2420a-0063118b01-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:01.694 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.695 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.695 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.696 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.696 DEBUG:boto:Method: GET 2022-09-02T04:48:01.697 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.697 DEBUG:boto:Data: 2022-09-02T04:48:01.698 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.699 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.700 DEBUG:boto:Params: {} 2022-09-02T04:48:01.700 DEBUG:boto:Token: None 2022-09-02T04:48:01.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ffc7ee7b3ef3296-0063118b01-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.750 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:01.753 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:01.754 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.755 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.755 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.756 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.757 DEBUG:boto:Method: GET 2022-09-02T04:48:01.758 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.759 DEBUG:boto:Data: 2022-09-02T04:48:01.760 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.761 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.762 DEBUG:boto:Params: {} 2022-09-02T04:48:01.763 DEBUG:boto:Token: None 2022-09-02T04:48:01.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b0ebaa95bc91c16-0063118b01-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.769 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:01.772 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.773 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.774 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.775 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.775 DEBUG:boto:Method: GET 2022-09-02T04:48:01.776 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.776 DEBUG:boto:Data: 2022-09-02T04:48:01.777 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.779 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.780 DEBUG:boto:Params: {} 2022-09-02T04:48:01.781 DEBUG:boto:Token: None 2022-09-02T04:48:01.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084db7ebf3950b407-0063118b01-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.791 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.793 DEBUG:boto:path=/xljbkf-24/ 2022-09-02T04:48:01.793 DEBUG:boto:auth_path=/xljbkf-24/ 2022-09-02T04:48:01.794 DEBUG:boto:path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.795 DEBUG:boto:auth_path=/xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.796 DEBUG:boto:Method: GET 2022-09-02T04:48:01.796 DEBUG:boto:Path: /xljbkf-24/?versions&max-keys=999 2022-09-02T04:48:01.797 DEBUG:boto:Data: 2022-09-02T04:48:01.798 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.800 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.801 DEBUG:boto:Params: {} 2022-09-02T04:48:01.802 DEBUG:boto:Token: None 2022-09-02T04:48:01.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/?versions 2022-09-02T04:48:01.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY= 2022-09-02T04:48:01.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJg0aoecmOA//mrliq+pNwhwgQY=', 'Content-Length': '0'} 2022-09-02T04:48:01.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c6bfc67f965add-0063118b01-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.816 DEBUG:boto:b'xljbkf-24999falseobjnulltrue2022-09-02T04:47:35.284Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:01.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:01.818 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:48:01.819 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:48:01.820 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.821 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:48:01.823 DEBUG:boto:Data: 2022-09-02T04:48:01.823 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:01.825 DEBUG:boto:Port: 8001 2022-09-02T04:48:01.826 DEBUG:boto:Params: {} 2022-09-02T04:48:01.827 DEBUG:boto:Token: None 2022-09-02T04:48:01.828 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/obj 2022-09-02T04:48:01.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU= 2022-09-02T04:48:01.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU=', 'Content-Length': '0'} 2022-09-02T04:48:01.833 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000001a41c034bc5d0ae7-0063118b01-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.834 DEBUG:boto:path=/xljbkf-24/obj 2022-09-02T04:48:01.835 DEBUG:boto:auth_path=/xljbkf-24/obj 2022-09-02T04:48:01.836 DEBUG:boto:Method: HEAD 2022-09-02T04:48:01.837 DEBUG:boto:Path: /xljbkf-24/obj 2022-09-02T04:48:01.837 DEBUG:boto:Data: 2022-09-02T04:48:01.838 DEBUG:boto:Headers: {} 2022-09-02T04:48:01.839 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:01.840 DEBUG:boto:Port: 8000 2022-09-02T04:48:01.843 DEBUG:boto:Params: {} 2022-09-02T04:48:01.844 DEBUG:boto:Token: None 2022-09-02T04:48:01.845 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:01 GMT /xljbkf-24/obj 2022-09-02T04:48:01.847 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU= 2022-09-02T04:48:01.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrcWub2kjIM9KDKibQlbjurgCJU=', 'Content-Length': '0'} 2022-09-02T04:48:01.853 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000a8def94bea08ae60-0063118b01-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:01.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-24 zones={test-zone3, test-zone2} 2022-09-02T04:48:01.854 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-25 2022-09-02T04:48:01.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.856 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:01.916 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:01.917 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:01.953 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:01.953+0000 7f11fcbab600 1 bucket sync caught up with source: 2022-09-02T04:48:01.953 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.7.12, , ] 2022-09-02T04:48:01.954 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.7.12, , ] 2022-09-02T04:48:01.954 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:01.953+0000 7f11fcbab600 0 bucket checkpoint complete 2022-09-02T04:48:01.971 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-25 2022-09-02T04:48:01.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:01.973 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:02.030 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:02.031 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:02.084 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.085+0000 7f0d7ef96600 1 bucket sync caught up with source: 2022-09-02T04:48:02.085 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.7.12, , ] 2022-09-02T04:48:02.085 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.7.12, , ] 2022-09-02T04:48:02.085 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.085+0000 7f0d7ef96600 0 bucket checkpoint complete 2022-09-02T04:48:02.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-25 2022-09-02T04:48:02.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.105 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:02.163 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:02.213 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:02.213+0000 7f5d767aa600 1 bucket sync caught up with source: 2022-09-02T04:48:02.214 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.7.12, , ] 2022-09-02T04:48:02.214 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.7.12, , ] 2022-09-02T04:48:02.215 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:02.213+0000 7f5d767aa600 0 bucket checkpoint complete 2022-09-02T04:48:02.229 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-25 2022-09-02T04:48:02.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:02.292 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:02.292 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:02.328 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.329+0000 7fa400aab600 1 bucket sync caught up with source: 2022-09-02T04:48:02.329 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.109.6, , ] 2022-09-02T04:48:02.329 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.109.6, , ] 2022-09-02T04:48:02.329 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.329+0000 7fa400aab600 0 bucket checkpoint complete 2022-09-02T04:48:02.340 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-25 2022-09-02T04:48:02.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.342 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:02.400 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:02.400 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.436+0000 7f5087063600 1 bucket sync caught up with source: 2022-09-02T04:48:02.436 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.109.6, , ] 2022-09-02T04:48:02.436 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.109.6, , ] 2022-09-02T04:48:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.436+0000 7f5087063600 0 bucket checkpoint complete 2022-09-02T04:48:02.453 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-25 2022-09-02T04:48:02.454 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-25 2022-09-02T04:48:02.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.456 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:02.576 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.573+0000 7f7a7e71d600 1 bucket sync caught up with source: 2022-09-02T04:48:02.577 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.23.12, , ] 2022-09-02T04:48:02.577 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.23.12, , ] 2022-09-02T04:48:02.578 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:02.573+0000 7f7a7e71d600 0 bucket checkpoint complete 2022-09-02T04:48:02.596 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-25 2022-09-02T04:48:02.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:02.599 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:02.656 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:02.657 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:02.702 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:02.702+0000 7f66e017a600 1 bucket sync caught up with source: 2022-09-02T04:48:02.703 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.23.12, , ] 2022-09-02T04:48:02.703 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.23.12, , ] 2022-09-02T04:48:02.704 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:02.702+0000 7f66e017a600 0 bucket checkpoint complete 2022-09-02T04:48:02.715 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-25 2022-09-02T04:48:02.716 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-25 zones={test-zone2, test-zone1} 2022-09-02T04:48:02.721 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.723 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.725 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.726 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:48:02.726 DEBUG:boto:Data: 2022-09-02T04:48:02.727 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:02.729 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.730 DEBUG:boto:Params: {} 2022-09-02T04:48:02.730 DEBUG:boto:Token: None 2022-09-02T04:48:02.731 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/ 2022-09-02T04:48:02.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo= 2022-09-02T04:48:02.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo=', 'Content-Length': '0'} 2022-09-02T04:48:02.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e00956e58a51aa4-0063118b02-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.738 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.739 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.740 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.740 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:48:02.741 DEBUG:boto:Data: 2022-09-02T04:48:02.742 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.743 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.745 DEBUG:boto:Params: {} 2022-09-02T04:48:02.746 DEBUG:boto:Token: None 2022-09-02T04:48:02.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/ 2022-09-02T04:48:02.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo= 2022-09-02T04:48:02.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo=', 'Content-Length': '0'} 2022-09-02T04:48:02.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc49cd89d18669ed-0063118b02-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.755 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:02.756 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.757 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.758 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.759 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.760 DEBUG:boto:Method: GET 2022-09-02T04:48:02.761 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.762 DEBUG:boto:Data: 2022-09-02T04:48:02.763 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:02.764 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.765 DEBUG:boto:Params: {} 2022-09-02T04:48:02.766 DEBUG:boto:Token: None 2022-09-02T04:48:02.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021cfebf2abd7f193-0063118b02-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.777 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:02.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:02.780 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.781 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.782 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.784 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.785 DEBUG:boto:Method: GET 2022-09-02T04:48:02.786 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.787 DEBUG:boto:Data: 2022-09-02T04:48:02.789 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.790 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.792 DEBUG:boto:Params: {} 2022-09-02T04:48:02.793 DEBUG:boto:Token: None 2022-09-02T04:48:02.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027069fd72bb5e479-0063118b02-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.801 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:02.803 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.804 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.804 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.805 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.806 DEBUG:boto:Method: GET 2022-09-02T04:48:02.806 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.807 DEBUG:boto:Data: 2022-09-02T04:48:02.807 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:02.808 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.809 DEBUG:boto:Params: {} 2022-09-02T04:48:02.810 DEBUG:boto:Token: None 2022-09-02T04:48:02.811 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e993bebe2029bb-0063118b02-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.816 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.817 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.818 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.819 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.820 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.821 DEBUG:boto:Method: GET 2022-09-02T04:48:02.824 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.825 DEBUG:boto:Data: 2022-09-02T04:48:02.827 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.830 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.833 DEBUG:boto:Params: {} 2022-09-02T04:48:02.834 DEBUG:boto:Token: None 2022-09-02T04:48:02.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8c04a568da32f52-0063118b02-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.848 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:02.851 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:48:02.852 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:48:02.853 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.854 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:48:02.855 DEBUG:boto:Data: 2022-09-02T04:48:02.856 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.857 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:02.858 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.859 DEBUG:boto:Params: {} 2022-09-02T04:48:02.860 DEBUG:boto:Token: None 2022-09-02T04:48:02.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/obj 2022-09-02T04:48:02.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw= 2022-09-02T04:48:02.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw=', 'Content-Length': '0'} 2022-09-02T04:48:02.865 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000eda8c61c902d9fa6-0063118b02-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.866 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:48:02.867 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:48:02.868 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.870 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:48:02.871 DEBUG:boto:Data: 2022-09-02T04:48:02.872 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.874 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.875 DEBUG:boto:Params: {} 2022-09-02T04:48:02.876 DEBUG:boto:Token: None 2022-09-02T04:48:02.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/obj 2022-09-02T04:48:02.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw= 2022-09-02T04:48:02.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw=', 'Content-Length': '0'} 2022-09-02T04:48:02.882 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d0522041be423d9a-0063118b02-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-25 zones={test-zone2, test-zone1} 2022-09-02T04:48:02.884 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-25 zones={test-zone3, test-zone1} 2022-09-02T04:48:02.885 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.886 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.887 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.887 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:48:02.888 DEBUG:boto:Data: 2022-09-02T04:48:02.889 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:02.892 DEBUG:boto:Port: 8001 2022-09-02T04:48:02.893 DEBUG:boto:Params: {} 2022-09-02T04:48:02.894 DEBUG:boto:Token: None 2022-09-02T04:48:02.894 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/ 2022-09-02T04:48:02.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo= 2022-09-02T04:48:02.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo=', 'Content-Length': '0'} 2022-09-02T04:48:02.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ddaa0a1a165c583-0063118b02-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.901 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.902 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.902 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.903 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:48:02.904 DEBUG:boto:Data: 2022-09-02T04:48:02.904 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.905 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.906 DEBUG:boto:Params: {} 2022-09-02T04:48:02.906 DEBUG:boto:Token: None 2022-09-02T04:48:02.907 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/ 2022-09-02T04:48:02.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo= 2022-09-02T04:48:02.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SFtR2cGfr4UBQb5sF2AxOkhMFNo=', 'Content-Length': '0'} 2022-09-02T04:48:02.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023a0917576d9d44e-0063118b02-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:02.913 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.914 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.914 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.915 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.915 DEBUG:boto:Method: GET 2022-09-02T04:48:02.916 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.917 DEBUG:boto:Data: 2022-09-02T04:48:02.917 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:02.920 DEBUG:boto:Port: 8001 2022-09-02T04:48:02.921 DEBUG:boto:Params: {} 2022-09-02T04:48:02.923 DEBUG:boto:Token: None 2022-09-02T04:48:02.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f37e61460e6ca29-0063118b02-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.930 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:02.932 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:02.933 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.933 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.934 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.934 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.935 DEBUG:boto:Method: GET 2022-09-02T04:48:02.935 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.936 DEBUG:boto:Data: 2022-09-02T04:48:02.936 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.937 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.938 DEBUG:boto:Params: {} 2022-09-02T04:48:02.939 DEBUG:boto:Token: None 2022-09-02T04:48:02.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc96d61ef32f34b7-0063118b02-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.945 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:02.948 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.949 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.950 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.950 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.951 DEBUG:boto:Method: GET 2022-09-02T04:48:02.952 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.952 DEBUG:boto:Data: 2022-09-02T04:48:02.953 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:02.954 DEBUG:boto:Port: 8001 2022-09-02T04:48:02.955 DEBUG:boto:Params: {} 2022-09-02T04:48:02.955 DEBUG:boto:Token: None 2022-09-02T04:48:02.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029af8552ddf6b10c-0063118b02-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.961 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.962 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:02.963 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:02.963 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.964 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.964 DEBUG:boto:Method: GET 2022-09-02T04:48:02.965 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:02.965 DEBUG:boto:Data: 2022-09-02T04:48:02.966 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.967 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.967 DEBUG:boto:Params: {} 2022-09-02T04:48:02.968 DEBUG:boto:Token: None 2022-09-02T04:48:02.968 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/?versions 2022-09-02T04:48:02.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY= 2022-09-02T04:48:02.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:peaI147J67Sqk72FO+YZ4ryjLBY=', 'Content-Length': '0'} 2022-09-02T04:48:02.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024a49ac9d2ca83a0-0063118b02-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.982 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:02.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:02.983 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:48:02.984 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:48:02.984 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.985 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:48:02.985 DEBUG:boto:Data: 2022-09-02T04:48:02.986 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:02.987 DEBUG:boto:Port: 8001 2022-09-02T04:48:02.988 DEBUG:boto:Params: {} 2022-09-02T04:48:02.988 DEBUG:boto:Token: None 2022-09-02T04:48:02.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/obj 2022-09-02T04:48:02.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw= 2022-09-02T04:48:02.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw=', 'Content-Length': '0'} 2022-09-02T04:48:02.992 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006dd887fd8e2fc0b1-0063118b02-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:02.993 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:48:02.993 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:48:02.994 DEBUG:boto:Method: HEAD 2022-09-02T04:48:02.995 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:48:02.995 DEBUG:boto:Data: 2022-09-02T04:48:02.996 DEBUG:boto:Headers: {} 2022-09-02T04:48:02.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:02.997 DEBUG:boto:Port: 8000 2022-09-02T04:48:02.997 DEBUG:boto:Params: {} 2022-09-02T04:48:02.998 DEBUG:boto:Token: None 2022-09-02T04:48:02.998 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:02 GMT /xljbkf-25/obj 2022-09-02T04:48:02.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw= 2022-09-02T04:48:03.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5KI6/QC7m45ZGrd1QRho1uLQZZw=', 'Content-Length': '0'} 2022-09-02T04:48:03.001 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000002659a0bdf32a5b8e-0063118b03-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-25 zones={test-zone3, test-zone1} 2022-09-02T04:48:03.004 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-25 zones={test-zone3, test-zone2} 2022-09-02T04:48:03.005 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:03.007 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:03.008 DEBUG:boto:Method: HEAD 2022-09-02T04:48:03.009 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:48:03.010 DEBUG:boto:Data: 2022-09-02T04:48:03.011 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:03.012 DEBUG:boto:Port: 8001 2022-09-02T04:48:03.013 DEBUG:boto:Params: {} 2022-09-02T04:48:03.014 DEBUG:boto:Token: None 2022-09-02T04:48:03.015 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/ 2022-09-02T04:48:03.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fYnAfUQcNujFjjbXJhueFsS+kuo= 2022-09-02T04:48:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fYnAfUQcNujFjjbXJhueFsS+kuo=', 'Content-Length': '0'} 2022-09-02T04:48:03.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092d4647ba0d2a059-0063118b03-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.019 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:03.020 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:03.020 DEBUG:boto:Method: HEAD 2022-09-02T04:48:03.021 DEBUG:boto:Path: /xljbkf-25/ 2022-09-02T04:48:03.021 DEBUG:boto:Data: 2022-09-02T04:48:03.022 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:03.023 DEBUG:boto:Port: 8000 2022-09-02T04:48:03.023 DEBUG:boto:Params: {} 2022-09-02T04:48:03.024 DEBUG:boto:Token: None 2022-09-02T04:48:03.024 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/ 2022-09-02T04:48:03.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fYnAfUQcNujFjjbXJhueFsS+kuo= 2022-09-02T04:48:03.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fYnAfUQcNujFjjbXJhueFsS+kuo=', 'Content-Length': '0'} 2022-09-02T04:48:03.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4445598739183dc-0063118b03-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.029 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:03.029 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:03.030 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:03.030 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.031 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.031 DEBUG:boto:Method: GET 2022-09-02T04:48:03.032 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.032 DEBUG:boto:Data: 2022-09-02T04:48:03.033 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:03.034 DEBUG:boto:Port: 8001 2022-09-02T04:48:03.034 DEBUG:boto:Params: {} 2022-09-02T04:48:03.035 DEBUG:boto:Token: None 2022-09-02T04:48:03.036 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/?versions 2022-09-02T04:48:03.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY= 2022-09-02T04:48:03.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY=', 'Content-Length': '0'} 2022-09-02T04:48:03.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bcc271701057667-0063118b03-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.085 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:03.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:03.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:03.088 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:03.088 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:03.089 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.089 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.090 DEBUG:boto:Method: GET 2022-09-02T04:48:03.090 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.091 DEBUG:boto:Data: 2022-09-02T04:48:03.091 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:03.092 DEBUG:boto:Port: 8000 2022-09-02T04:48:03.093 DEBUG:boto:Params: {} 2022-09-02T04:48:03.093 DEBUG:boto:Token: None 2022-09-02T04:48:03.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/?versions 2022-09-02T04:48:03.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY= 2022-09-02T04:48:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY=', 'Content-Length': '0'} 2022-09-02T04:48:03.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000615ef4fbbcd12f51-0063118b03-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.099 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:03.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:03.100 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:03.101 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:03.101 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.102 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.102 DEBUG:boto:Method: GET 2022-09-02T04:48:03.103 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.103 DEBUG:boto:Data: 2022-09-02T04:48:03.104 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:03.105 DEBUG:boto:Port: 8001 2022-09-02T04:48:03.105 DEBUG:boto:Params: {} 2022-09-02T04:48:03.106 DEBUG:boto:Token: None 2022-09-02T04:48:03.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/?versions 2022-09-02T04:48:03.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY= 2022-09-02T04:48:03.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY=', 'Content-Length': '0'} 2022-09-02T04:48:03.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a5169860488d1d9-0063118b03-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.112 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:03.114 DEBUG:boto:path=/xljbkf-25/ 2022-09-02T04:48:03.114 DEBUG:boto:auth_path=/xljbkf-25/ 2022-09-02T04:48:03.115 DEBUG:boto:path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.115 DEBUG:boto:auth_path=/xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.116 DEBUG:boto:Method: GET 2022-09-02T04:48:03.116 DEBUG:boto:Path: /xljbkf-25/?versions&max-keys=999 2022-09-02T04:48:03.117 DEBUG:boto:Data: 2022-09-02T04:48:03.117 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:03.118 DEBUG:boto:Port: 8000 2022-09-02T04:48:03.119 DEBUG:boto:Params: {} 2022-09-02T04:48:03.120 DEBUG:boto:Token: None 2022-09-02T04:48:03.120 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/?versions 2022-09-02T04:48:03.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY= 2022-09-02T04:48:03.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kh/jjTyCEK2A+tD8LFgj4S6kCyY=', 'Content-Length': '0'} 2022-09-02T04:48:03.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8521801ca28dc1-0063118b03-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.126 DEBUG:boto:b'xljbkf-25999falseobjnulltrue2022-09-02T04:47:35.379Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:03.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:03.128 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:48:03.129 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:48:03.130 DEBUG:boto:Method: HEAD 2022-09-02T04:48:03.131 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:48:03.131 DEBUG:boto:Data: 2022-09-02T04:48:03.132 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:03.134 DEBUG:boto:Port: 8001 2022-09-02T04:48:03.135 DEBUG:boto:Params: {} 2022-09-02T04:48:03.136 DEBUG:boto:Token: None 2022-09-02T04:48:03.136 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/obj 2022-09-02T04:48:03.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9wb+cddxjoFk3H1AYTKHsdngItk= 2022-09-02T04:48:03.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9wb+cddxjoFk3H1AYTKHsdngItk=', 'Content-Length': '0'} 2022-09-02T04:48:03.141 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000f07c519621b3202b-0063118b03-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.142 DEBUG:boto:path=/xljbkf-25/obj 2022-09-02T04:48:03.143 DEBUG:boto:auth_path=/xljbkf-25/obj 2022-09-02T04:48:03.144 DEBUG:boto:Method: HEAD 2022-09-02T04:48:03.146 DEBUG:boto:Path: /xljbkf-25/obj 2022-09-02T04:48:03.147 DEBUG:boto:Data: 2022-09-02T04:48:03.147 DEBUG:boto:Headers: {} 2022-09-02T04:48:03.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:03.148 DEBUG:boto:Port: 8000 2022-09-02T04:48:03.149 DEBUG:boto:Params: {} 2022-09-02T04:48:03.150 DEBUG:boto:Token: None 2022-09-02T04:48:03.150 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:03 GMT /xljbkf-25/obj 2022-09-02T04:48:03.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9wb+cddxjoFk3H1AYTKHsdngItk= 2022-09-02T04:48:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9wb+cddxjoFk3H1AYTKHsdngItk=', 'Content-Length': '0'} 2022-09-02T04:48:03.154 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000085cd5661c0913178-0063118b03-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:03.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-25 zones={test-zone3, test-zone2} 2022-09-02T04:48:03.156 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-26 2022-09-02T04:48:03.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.159 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.217 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.218 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.255 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:03.254+0000 7f8b9cdec600 1 bucket sync caught up with source: 2022-09-02T04:48:03.256 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2022-09-02T04:48:03.256 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2022-09-02T04:48:03.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:03.254+0000 7f8b9cdec600 0 bucket checkpoint complete 2022-09-02T04:48:03.267 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-26 2022-09-02T04:48:03.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.269 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.378 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.379+0000 7fe6f9fa4600 1 bucket sync caught up with source: 2022-09-02T04:48:03.386 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2022-09-02T04:48:03.387 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2022-09-02T04:48:03.387 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.379+0000 7fe6f9fa4600 0 bucket checkpoint complete 2022-09-02T04:48:03.392 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-26 2022-09-02T04:48:03.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.394 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.453 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.453 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.496 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:03.496+0000 7fe3d374a600 1 bucket sync caught up with source: 2022-09-02T04:48:03.496 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2022-09-02T04:48:03.497 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2022-09-02T04:48:03.497 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:03.496+0000 7fe3d374a600 0 bucket checkpoint complete 2022-09-02T04:48:03.514 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-26 2022-09-02T04:48:03.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.528 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.627 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.628+0000 7f0db0c0f600 1 bucket sync caught up with source: 2022-09-02T04:48:03.628 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.49.12, , ] 2022-09-02T04:48:03.628 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.49.12, , ] 2022-09-02T04:48:03.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.628+0000 7f0db0c0f600 0 bucket checkpoint complete 2022-09-02T04:48:03.640 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-26 2022-09-02T04:48:03.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.642 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.746 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.747+0000 7f764d723600 1 bucket sync caught up with source: 2022-09-02T04:48:03.747 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.49.12, , ] 2022-09-02T04:48:03.747 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.49.12, , ] 2022-09-02T04:48:03.747 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.747+0000 7f764d723600 0 bucket checkpoint complete 2022-09-02T04:48:03.762 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-26 2022-09-02T04:48:03.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-26 2022-09-02T04:48:03.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.764 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.857+0000 7f8f9fdfd600 1 bucket sync caught up with source: 2022-09-02T04:48:03.858 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.180.6, , ] 2022-09-02T04:48:03.859 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.180.6, , ] 2022-09-02T04:48:03.859 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:03.857+0000 7f8f9fdfd600 0 bucket checkpoint complete 2022-09-02T04:48:03.876 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-26 2022-09-02T04:48:03.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T04:48:03.878 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T04:48:03.936 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:03.937 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:03.983 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:03.983+0000 7fb7926c4600 1 bucket sync caught up with source: 2022-09-02T04:48:03.983 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000003.180.6, , ] 2022-09-02T04:48:03.983 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000003.180.6, , ] 2022-09-02T04:48:03.984 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:03.983+0000 7fb7926c4600 0 bucket checkpoint complete 2022-09-02T04:48:04.004 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-26 2022-09-02T04:48:04.005 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-26 zones={test-zone2, test-zone1} 2022-09-02T04:48:04.005 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.006 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.007 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.007 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:48:04.008 DEBUG:boto:Data: 2022-09-02T04:48:04.008 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.010 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.010 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.011 DEBUG:boto:Params: {} 2022-09-02T04:48:04.012 DEBUG:boto:Token: None 2022-09-02T04:48:04.012 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/ 2022-09-02T04:48:04.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw= 2022-09-02T04:48:04.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw=', 'Content-Length': '0'} 2022-09-02T04:48:04.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006074a0df100a48a4-0063118b04-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.017 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.017 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.018 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.019 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:48:04.019 DEBUG:boto:Data: 2022-09-02T04:48:04.020 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.021 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.021 DEBUG:boto:Params: {} 2022-09-02T04:48:04.022 DEBUG:boto:Token: None 2022-09-02T04:48:04.023 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/ 2022-09-02T04:48:04.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw= 2022-09-02T04:48:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw=', 'Content-Length': '0'} 2022-09-02T04:48:04.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b237da0698b91d5f-0063118b04-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:04.028 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.029 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.029 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.030 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.030 DEBUG:boto:Method: GET 2022-09-02T04:48:04.031 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.031 DEBUG:boto:Data: 2022-09-02T04:48:04.032 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.033 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.034 DEBUG:boto:Params: {} 2022-09-02T04:48:04.035 DEBUG:boto:Token: None 2022-09-02T04:48:04.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3f0efa18e69eb0-0063118b04-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.042 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:04.043 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:04.044 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.044 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.045 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.045 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.046 DEBUG:boto:Method: GET 2022-09-02T04:48:04.047 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.047 DEBUG:boto:Data: 2022-09-02T04:48:04.049 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.051 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.052 DEBUG:boto:Params: {} 2022-09-02T04:48:04.053 DEBUG:boto:Token: None 2022-09-02T04:48:04.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000983baf1b8bde1824-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.106 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:04.108 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.108 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.109 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.110 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.110 DEBUG:boto:Method: GET 2022-09-02T04:48:04.111 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.111 DEBUG:boto:Data: 2022-09-02T04:48:04.112 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.113 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.114 DEBUG:boto:Params: {} 2022-09-02T04:48:04.116 DEBUG:boto:Token: None 2022-09-02T04:48:04.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ee7dc36f4bb807-0063118b04-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.123 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.124 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.124 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.125 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.125 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.126 DEBUG:boto:Method: GET 2022-09-02T04:48:04.126 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.127 DEBUG:boto:Data: 2022-09-02T04:48:04.128 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.129 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.129 DEBUG:boto:Params: {} 2022-09-02T04:48:04.130 DEBUG:boto:Token: None 2022-09-02T04:48:04.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000050e7b87df66528-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.135 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:04.136 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:48:04.137 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:48:04.137 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.138 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:48:04.138 DEBUG:boto:Data: 2022-09-02T04:48:04.139 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.140 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.140 DEBUG:boto:Params: {} 2022-09-02T04:48:04.141 DEBUG:boto:Token: None 2022-09-02T04:48:04.142 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/obj 2022-09-02T04:48:04.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc= 2022-09-02T04:48:04.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc=', 'Content-Length': '0'} 2022-09-02T04:48:04.145 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000d5e698d254df503e-0063118b04-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.146 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:48:04.146 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:48:04.147 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.147 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:48:04.148 DEBUG:boto:Data: 2022-09-02T04:48:04.148 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.149 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.150 DEBUG:boto:Params: {} 2022-09-02T04:48:04.150 DEBUG:boto:Token: None 2022-09-02T04:48:04.151 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/obj 2022-09-02T04:48:04.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc= 2022-09-02T04:48:04.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc=', 'Content-Length': '0'} 2022-09-02T04:48:04.154 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000324b370706e733e8-0063118b04-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-26 zones={test-zone2, test-zone1} 2022-09-02T04:48:04.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-26 zones={test-zone3, test-zone1} 2022-09-02T04:48:04.156 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.156 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.158 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.158 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:48:04.159 DEBUG:boto:Data: 2022-09-02T04:48:04.159 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.160 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.161 DEBUG:boto:Params: {} 2022-09-02T04:48:04.162 DEBUG:boto:Token: None 2022-09-02T04:48:04.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/ 2022-09-02T04:48:04.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw= 2022-09-02T04:48:04.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw=', 'Content-Length': '0'} 2022-09-02T04:48:04.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007f442944aad4496-0063118b04-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.169 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.170 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.171 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.172 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:48:04.172 DEBUG:boto:Data: 2022-09-02T04:48:04.173 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.174 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.174 DEBUG:boto:Params: {} 2022-09-02T04:48:04.175 DEBUG:boto:Token: None 2022-09-02T04:48:04.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/ 2022-09-02T04:48:04.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw= 2022-09-02T04:48:04.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw=', 'Content-Length': '0'} 2022-09-02T04:48:04.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e29fe8b5d651df8e-0063118b04-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:04.183 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.184 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.185 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.186 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.187 DEBUG:boto:Method: GET 2022-09-02T04:48:04.188 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.189 DEBUG:boto:Data: 2022-09-02T04:48:04.190 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.192 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.192 DEBUG:boto:Params: {} 2022-09-02T04:48:04.193 DEBUG:boto:Token: None 2022-09-02T04:48:04.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f93f081dcd638478-0063118b04-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.246 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:04.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:04.250 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.251 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.252 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.253 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.254 DEBUG:boto:Method: GET 2022-09-02T04:48:04.254 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.255 DEBUG:boto:Data: 2022-09-02T04:48:04.256 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.258 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.259 DEBUG:boto:Params: {} 2022-09-02T04:48:04.260 DEBUG:boto:Token: None 2022-09-02T04:48:04.261 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c54c654d9fef4bc4-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.270 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:04.272 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.273 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.273 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.274 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.275 DEBUG:boto:Method: GET 2022-09-02T04:48:04.275 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.276 DEBUG:boto:Data: 2022-09-02T04:48:04.277 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.278 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.278 DEBUG:boto:Params: {} 2022-09-02T04:48:04.279 DEBUG:boto:Token: None 2022-09-02T04:48:04.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb49b554bb1c6b67-0063118b04-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.286 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.287 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.287 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.288 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.289 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.290 DEBUG:boto:Method: GET 2022-09-02T04:48:04.290 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.291 DEBUG:boto:Data: 2022-09-02T04:48:04.291 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.292 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.293 DEBUG:boto:Params: {} 2022-09-02T04:48:04.294 DEBUG:boto:Token: None 2022-09-02T04:48:04.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a61e1a696c3b9ed-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.347 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:04.349 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:48:04.349 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:48:04.350 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.350 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:48:04.351 DEBUG:boto:Data: 2022-09-02T04:48:04.352 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.353 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.354 DEBUG:boto:Params: {} 2022-09-02T04:48:04.355 DEBUG:boto:Token: None 2022-09-02T04:48:04.355 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/obj 2022-09-02T04:48:04.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc= 2022-09-02T04:48:04.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc=', 'Content-Length': '0'} 2022-09-02T04:48:04.358 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000062ecdef1ce8fb405-0063118b04-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.359 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:48:04.360 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:48:04.360 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.361 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:48:04.362 DEBUG:boto:Data: 2022-09-02T04:48:04.362 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.364 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.364 DEBUG:boto:Params: {} 2022-09-02T04:48:04.365 DEBUG:boto:Token: None 2022-09-02T04:48:04.366 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/obj 2022-09-02T04:48:04.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc= 2022-09-02T04:48:04.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc=', 'Content-Length': '0'} 2022-09-02T04:48:04.369 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000006e18155825e3ec9b-0063118b04-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-26 zones={test-zone3, test-zone1} 2022-09-02T04:48:04.373 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-26 zones={test-zone3, test-zone2} 2022-09-02T04:48:04.373 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.374 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.374 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.376 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:48:04.377 DEBUG:boto:Data: 2022-09-02T04:48:04.377 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.378 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.380 DEBUG:boto:Params: {} 2022-09-02T04:48:04.380 DEBUG:boto:Token: None 2022-09-02T04:48:04.381 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/ 2022-09-02T04:48:04.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw= 2022-09-02T04:48:04.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw=', 'Content-Length': '0'} 2022-09-02T04:48:04.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a73dd6db7cd526af-0063118b04-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.387 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.389 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.390 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.390 DEBUG:boto:Path: /xljbkf-26/ 2022-09-02T04:48:04.391 DEBUG:boto:Data: 2022-09-02T04:48:04.392 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.394 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.395 DEBUG:boto:Params: {} 2022-09-02T04:48:04.397 DEBUG:boto:Token: None 2022-09-02T04:48:04.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/ 2022-09-02T04:48:04.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw= 2022-09-02T04:48:04.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44IeTMuxdzd9DjOVpqELAFTpZhw=', 'Content-Length': '0'} 2022-09-02T04:48:04.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cb56149bd38737e-0063118b04-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:48:04.408 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.409 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.410 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.411 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.412 DEBUG:boto:Method: GET 2022-09-02T04:48:04.413 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.415 DEBUG:boto:Data: 2022-09-02T04:48:04.415 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.416 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.417 DEBUG:boto:Params: {} 2022-09-02T04:48:04.418 DEBUG:boto:Token: None 2022-09-02T04:48:04.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e88cf89d5ea4cd-0063118b04-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.424 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:04.425 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:48:04.426 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.426 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.427 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.428 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.428 DEBUG:boto:Method: GET 2022-09-02T04:48:04.429 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.429 DEBUG:boto:Data: 2022-09-02T04:48:04.430 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.431 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.432 DEBUG:boto:Params: {} 2022-09-02T04:48:04.432 DEBUG:boto:Token: None 2022-09-02T04:48:04.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e50a7b8c924e9b1-0063118b04-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.446 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-09-02T04:48:04.448 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.448 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.449 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.449 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.450 DEBUG:boto:Method: GET 2022-09-02T04:48:04.451 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.451 DEBUG:boto:Data: 2022-09-02T04:48:04.452 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.453 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.454 DEBUG:boto:Params: {} 2022-09-02T04:48:04.455 DEBUG:boto:Token: None 2022-09-02T04:48:04.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a1e0bdf8cdeebf-0063118b04-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.506 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.508 DEBUG:boto:path=/xljbkf-26/ 2022-09-02T04:48:04.509 DEBUG:boto:auth_path=/xljbkf-26/ 2022-09-02T04:48:04.510 DEBUG:boto:path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.511 DEBUG:boto:auth_path=/xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.512 DEBUG:boto:Method: GET 2022-09-02T04:48:04.513 DEBUG:boto:Path: /xljbkf-26/?versions&max-keys=999 2022-09-02T04:48:04.516 DEBUG:boto:Data: 2022-09-02T04:48:04.517 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.518 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.519 DEBUG:boto:Params: {} 2022-09-02T04:48:04.520 DEBUG:boto:Token: None 2022-09-02T04:48:04.521 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/?versions 2022-09-02T04:48:04.522 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU= 2022-09-02T04:48:04.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w3ND8m2IGIpNOH7dQ9Oh/RUM2QU=', 'Content-Length': '0'} 2022-09-02T04:48:04.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356aa70e04512e22-0063118b04-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.528 DEBUG:boto:b'xljbkf-26999falseobjnulltrue2022-09-02T04:47:35.456Zrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T04:48:04.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-09-02T04:48:04.532 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:48:04.532 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:48:04.534 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.535 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:48:04.536 DEBUG:boto:Data: 2022-09-02T04:48:04.536 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.540 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.540 DEBUG:boto:Params: {} 2022-09-02T04:48:04.541 DEBUG:boto:Token: None 2022-09-02T04:48:04.542 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/obj 2022-09-02T04:48:04.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc= 2022-09-02T04:48:04.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc=', 'Content-Length': '0'} 2022-09-02T04:48:04.546 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx000003fa8db4c4f084e35-0063118b04-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.547 DEBUG:boto:path=/xljbkf-26/obj 2022-09-02T04:48:04.548 DEBUG:boto:auth_path=/xljbkf-26/obj 2022-09-02T04:48:04.549 DEBUG:boto:Method: HEAD 2022-09-02T04:48:04.550 DEBUG:boto:Path: /xljbkf-26/obj 2022-09-02T04:48:04.551 DEBUG:boto:Data: 2022-09-02T04:48:04.551 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.553 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.554 DEBUG:boto:Params: {} 2022-09-02T04:48:04.555 DEBUG:boto:Token: None 2022-09-02T04:48:04.556 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-26/obj 2022-09-02T04:48:04.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc= 2022-09-02T04:48:04.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rtZzSjVz24Yd+0f1DfC8VYsDCRc=', 'Content-Length': '0'} 2022-09-02T04:48:04.566 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx0000069d6bf59fc0012d2-0063118b04-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-26 zones={test-zone3, test-zone2} 2022-09-02T04:48:04.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2022-09-02T04:48:04.570 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-27 2022-09-02T04:48:04.571 DEBUG:boto:path=/xljbkf-27/ 2022-09-02T04:48:04.572 DEBUG:boto:auth_path=/xljbkf-27/ 2022-09-02T04:48:04.572 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.573 DEBUG:boto:Path: /xljbkf-27/ 2022-09-02T04:48:04.574 DEBUG:boto:Data: 2022-09-02T04:48:04.575 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.577 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.578 DEBUG:boto:Params: {} 2022-09-02T04:48:04.580 DEBUG:boto:Token: None 2022-09-02T04:48:04.580 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-27/ 2022-09-02T04:48:04.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:26NeNyCxVD40seHDPrFhQ+GSYug= 2022-09-02T04:48:04.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:26NeNyCxVD40seHDPrFhQ+GSYug=', 'Content-Length': '0'} 2022-09-02T04:48:04.603 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1bdab79ef96c95d-0063118b04-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.604 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-28 2022-09-02T04:48:04.605 DEBUG:boto:path=/xljbkf-28/ 2022-09-02T04:48:04.606 DEBUG:boto:auth_path=/xljbkf-28/ 2022-09-02T04:48:04.607 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.607 DEBUG:boto:Path: /xljbkf-28/ 2022-09-02T04:48:04.608 DEBUG:boto:Data: 2022-09-02T04:48:04.608 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.609 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.610 DEBUG:boto:Params: {} 2022-09-02T04:48:04.611 DEBUG:boto:Token: None 2022-09-02T04:48:04.611 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-28/ 2022-09-02T04:48:04.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L1lV1gyUtlct+xFLWVmc/IQzX7k= 2022-09-02T04:48:04.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L1lV1gyUtlct+xFLWVmc/IQzX7k=', 'Content-Length': '0'} 2022-09-02T04:48:04.650 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016db9cc7fcb602e8-0063118b04-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.651 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-29 2022-09-02T04:48:04.652 DEBUG:boto:path=/xljbkf-29/ 2022-09-02T04:48:04.652 DEBUG:boto:auth_path=/xljbkf-29/ 2022-09-02T04:48:04.653 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.653 DEBUG:boto:Path: /xljbkf-29/ 2022-09-02T04:48:04.654 DEBUG:boto:Data: 2022-09-02T04:48:04.655 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.657 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.658 DEBUG:boto:Params: {} 2022-09-02T04:48:04.659 DEBUG:boto:Token: None 2022-09-02T04:48:04.661 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-29/ 2022-09-02T04:48:04.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FQyCJ0QaXMjYTjy3TmjaaTydjS0= 2022-09-02T04:48:04.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FQyCJ0QaXMjYTjy3TmjaaTydjS0=', 'Content-Length': '0'} 2022-09-02T04:48:04.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033465c7dd3abee50-0063118b04-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.703 DEBUG:boto:path=/xljbkf-27/ 2022-09-02T04:48:04.704 DEBUG:boto:auth_path=/xljbkf-27/ 2022-09-02T04:48:04.705 DEBUG:boto:path=/xljbkf-27/?versioning 2022-09-02T04:48:04.705 DEBUG:boto:auth_path=/xljbkf-27/?versioning 2022-09-02T04:48:04.706 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.707 DEBUG:boto:Path: /xljbkf-27/?versioning 2022-09-02T04:48:04.707 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:48:04.708 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.709 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.710 DEBUG:boto:Params: {} 2022-09-02T04:48:04.711 DEBUG:boto:Token: None 2022-09-02T04:48:04.712 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-27/?versioning 2022-09-02T04:48:04.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JjxX8pa+q4nReU/itYLXNa3uh74= 2022-09-02T04:48:04.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JjxX8pa+q4nReU/itYLXNa3uh74=', 'Content-Length': '228'} 2022-09-02T04:48:04.722 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000015015bf9c475db76-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.724 DEBUG:boto:path=/xljbkf-27/ 2022-09-02T04:48:04.724 DEBUG:boto:auth_path=/xljbkf-27/ 2022-09-02T04:48:04.725 DEBUG:boto:path=/xljbkf-27/?versioning 2022-09-02T04:48:04.726 DEBUG:boto:auth_path=/xljbkf-27/?versioning 2022-09-02T04:48:04.727 DEBUG:boto:Method: GET 2022-09-02T04:48:04.728 DEBUG:boto:Path: /xljbkf-27/?versioning 2022-09-02T04:48:04.729 DEBUG:boto:Data: 2022-09-02T04:48:04.729 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.732 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.734 DEBUG:boto:Params: {} 2022-09-02T04:48:04.735 DEBUG:boto:Token: None 2022-09-02T04:48:04.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-27/?versioning 2022-09-02T04:48:04.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KEjsJCEKVgHU5DwEdFw+InJ9hl4= 2022-09-02T04:48:04.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KEjsJCEKVgHU5DwEdFw+InJ9hl4=', 'Content-Length': '0'} 2022-09-02T04:48:04.742 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a26401e41df97cb-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.743 DEBUG:boto:EnabledDisabled 2022-09-02T04:48:04.746 DEBUG:boto:path=/xljbkf-28/ 2022-09-02T04:48:04.747 DEBUG:boto:auth_path=/xljbkf-28/ 2022-09-02T04:48:04.747 DEBUG:boto:path=/xljbkf-28/?versioning 2022-09-02T04:48:04.749 DEBUG:boto:auth_path=/xljbkf-28/?versioning 2022-09-02T04:48:04.751 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.752 DEBUG:boto:Path: /xljbkf-28/?versioning 2022-09-02T04:48:04.753 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:48:04.754 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.755 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.756 DEBUG:boto:Params: {} 2022-09-02T04:48:04.757 DEBUG:boto:Token: None 2022-09-02T04:48:04.759 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-28/?versioning 2022-09-02T04:48:04.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h3fMclcMyn4DO3stpSfRziPMGj4= 2022-09-02T04:48:04.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h3fMclcMyn4DO3stpSfRziPMGj4=', 'Content-Length': '228'} 2022-09-02T04:48:04.770 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010795ce2dd9d552a-0063118b04-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.771 DEBUG:boto:path=/xljbkf-28/ 2022-09-02T04:48:04.772 DEBUG:boto:auth_path=/xljbkf-28/ 2022-09-02T04:48:04.772 DEBUG:boto:path=/xljbkf-28/?versioning 2022-09-02T04:48:04.773 DEBUG:boto:auth_path=/xljbkf-28/?versioning 2022-09-02T04:48:04.774 DEBUG:boto:Method: GET 2022-09-02T04:48:04.775 DEBUG:boto:Path: /xljbkf-28/?versioning 2022-09-02T04:48:04.776 DEBUG:boto:Data: 2022-09-02T04:48:04.776 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.777 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.778 DEBUG:boto:Params: {} 2022-09-02T04:48:04.778 DEBUG:boto:Token: None 2022-09-02T04:48:04.779 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-28/?versioning 2022-09-02T04:48:04.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tkf7jsYoqp7t220hyCPTOV53hyM= 2022-09-02T04:48:04.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tkf7jsYoqp7t220hyCPTOV53hyM=', 'Content-Length': '0'} 2022-09-02T04:48:04.782 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a475001f68f795d-0063118b04-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.783 DEBUG:boto:EnabledDisabled 2022-09-02T04:48:04.783 DEBUG:boto:path=/xljbkf-29/ 2022-09-02T04:48:04.784 DEBUG:boto:auth_path=/xljbkf-29/ 2022-09-02T04:48:04.784 DEBUG:boto:path=/xljbkf-29/?versioning 2022-09-02T04:48:04.785 DEBUG:boto:auth_path=/xljbkf-29/?versioning 2022-09-02T04:48:04.785 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.786 DEBUG:boto:Path: /xljbkf-29/?versioning 2022-09-02T04:48:04.786 DEBUG:boto:Data: Enabled Disabled 2022-09-02T04:48:04.787 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.788 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.788 DEBUG:boto:Params: {} 2022-09-02T04:48:04.789 DEBUG:boto:Token: None 2022-09-02T04:48:04.789 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-29/?versioning 2022-09-02T04:48:04.790 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vhMCorj8X9o9JqTnWVs0ODt/WYw= 2022-09-02T04:48:04.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vhMCorj8X9o9JqTnWVs0ODt/WYw=', 'Content-Length': '228'} 2022-09-02T04:48:04.799 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079f4fc2b9ddd2e0f-0063118b04-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.800 DEBUG:boto:path=/xljbkf-29/ 2022-09-02T04:48:04.800 DEBUG:boto:auth_path=/xljbkf-29/ 2022-09-02T04:48:04.801 DEBUG:boto:path=/xljbkf-29/?versioning 2022-09-02T04:48:04.801 DEBUG:boto:auth_path=/xljbkf-29/?versioning 2022-09-02T04:48:04.802 DEBUG:boto:Method: GET 2022-09-02T04:48:04.802 DEBUG:boto:Path: /xljbkf-29/?versioning 2022-09-02T04:48:04.803 DEBUG:boto:Data: 2022-09-02T04:48:04.803 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.804 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.805 DEBUG:boto:Params: {} 2022-09-02T04:48:04.806 DEBUG:boto:Token: None 2022-09-02T04:48:04.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-29/?versioning 2022-09-02T04:48:04.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/myWqtDCexZQ8EkwVPr+dK7N9VA= 2022-09-02T04:48:04.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/myWqtDCexZQ8EkwVPr+dK7N9VA=', 'Content-Length': '0'} 2022-09-02T04:48:04.814 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009c40008a6ac97359-0063118b04-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.816 DEBUG:boto:EnabledDisabled 2022-09-02T04:48:04.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2022-09-02T04:48:04.818 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-30 2022-09-02T04:48:04.819 DEBUG:boto:path=/xljbkf-30/ 2022-09-02T04:48:04.820 DEBUG:boto:auth_path=/xljbkf-30/ 2022-09-02T04:48:04.821 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.822 DEBUG:boto:Path: /xljbkf-30/ 2022-09-02T04:48:04.823 DEBUG:boto:Data: 2022-09-02T04:48:04.824 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.825 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.826 DEBUG:boto:Params: {} 2022-09-02T04:48:04.827 DEBUG:boto:Token: None 2022-09-02T04:48:04.828 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-30/ 2022-09-02T04:48:04.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dBG0bsJI3d7s5h3cvVquF/umLrI= 2022-09-02T04:48:04.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dBG0bsJI3d7s5h3cvVquF/umLrI=', 'Content-Length': '0'} 2022-09-02T04:48:04.855 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f651aa917de7915a-0063118b04-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.857 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-31 2022-09-02T04:48:04.858 DEBUG:boto:path=/xljbkf-31/ 2022-09-02T04:48:04.859 DEBUG:boto:auth_path=/xljbkf-31/ 2022-09-02T04:48:04.860 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.860 DEBUG:boto:Path: /xljbkf-31/ 2022-09-02T04:48:04.861 DEBUG:boto:Data: 2022-09-02T04:48:04.862 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:04.864 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.865 DEBUG:boto:Params: {} 2022-09-02T04:48:04.866 DEBUG:boto:Token: None 2022-09-02T04:48:04.867 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-31/ 2022-09-02T04:48:04.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vtVY1UxPjRgFTqSwp5+LmixfKxM= 2022-09-02T04:48:04.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vtVY1UxPjRgFTqSwp5+LmixfKxM=', 'Content-Length': '0'} 2022-09-02T04:48:04.904 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f420b4754df4811-0063118b04-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.905 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-32 2022-09-02T04:48:04.905 DEBUG:boto:path=/xljbkf-32/ 2022-09-02T04:48:04.906 DEBUG:boto:auth_path=/xljbkf-32/ 2022-09-02T04:48:04.907 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.907 DEBUG:boto:Path: /xljbkf-32/ 2022-09-02T04:48:04.908 DEBUG:boto:Data: 2022-09-02T04:48:04.909 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:04.910 DEBUG:boto:Port: 8001 2022-09-02T04:48:04.910 DEBUG:boto:Params: {} 2022-09-02T04:48:04.911 DEBUG:boto:Token: None 2022-09-02T04:48:04.912 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-32/ 2022-09-02T04:48:04.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQguyllsNte/t9c70PaWnB78TZI= 2022-09-02T04:48:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQguyllsNte/t9c70PaWnB78TZI=', 'Content-Length': '0'} 2022-09-02T04:48:04.949 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000444f83afef606d8a-0063118b04-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.950 DEBUG:boto:path=/xljbkf-30/ 2022-09-02T04:48:04.951 DEBUG:boto:auth_path=/xljbkf-30/ 2022-09-02T04:48:04.951 DEBUG:boto:path=/xljbkf-30/?acl 2022-09-02T04:48:04.952 DEBUG:boto:auth_path=/xljbkf-30/?acl 2022-09-02T04:48:04.952 DEBUG:boto:Method: GET 2022-09-02T04:48:04.953 DEBUG:boto:Path: /xljbkf-30/?acl 2022-09-02T04:48:04.953 DEBUG:boto:Data: 2022-09-02T04:48:04.954 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.955 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.955 DEBUG:boto:Params: {} 2022-09-02T04:48:04.956 DEBUG:boto:Token: None 2022-09-02T04:48:04.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-30/?acl 2022-09-02T04:48:04.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lNplt3ABWG5rnCbPq7JTX0s+Ghg= 2022-09-02T04:48:04.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lNplt3ABWG5rnCbPq7JTX0s+Ghg=', 'Content-Length': '0'} 2022-09-02T04:48:04.960 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029e2a7cd17631684-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.961 DEBUG:boto:path=/xljbkf-30/ 2022-09-02T04:48:04.962 DEBUG:boto:auth_path=/xljbkf-30/ 2022-09-02T04:48:04.963 DEBUG:boto:path=/xljbkf-30/?acl 2022-09-02T04:48:04.963 DEBUG:boto:auth_path=/xljbkf-30/?acl 2022-09-02T04:48:04.964 DEBUG:boto:Method: PUT 2022-09-02T04:48:04.964 DEBUG:boto:Path: /xljbkf-30/?acl 2022-09-02T04:48:04.965 DEBUG:boto:Data: 2022-09-02T04:48:04.965 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:48:04.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.973 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.974 DEBUG:boto:Params: {} 2022-09-02T04:48:04.974 DEBUG:boto:Token: None 2022-09-02T04:48:04.976 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:04 GMT x-amz-acl:public-read /xljbkf-30/?acl 2022-09-02T04:48:04.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QsWIRGCopWVU4K3IkUO/2oJgN7U= 2022-09-02T04:48:04.978 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QsWIRGCopWVU4K3IkUO/2oJgN7U=', 'Content-Length': '0'} 2022-09-02T04:48:04.983 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050d1d438bc419b53-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:04.984 DEBUG:boto:path=/xljbkf-30/ 2022-09-02T04:48:04.985 DEBUG:boto:auth_path=/xljbkf-30/ 2022-09-02T04:48:04.985 DEBUG:boto:path=/xljbkf-30/?acl 2022-09-02T04:48:04.986 DEBUG:boto:auth_path=/xljbkf-30/?acl 2022-09-02T04:48:04.987 DEBUG:boto:Method: GET 2022-09-02T04:48:04.987 DEBUG:boto:Path: /xljbkf-30/?acl 2022-09-02T04:48:04.988 DEBUG:boto:Data: 2022-09-02T04:48:04.988 DEBUG:boto:Headers: {} 2022-09-02T04:48:04.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:04.989 DEBUG:boto:Port: 8000 2022-09-02T04:48:04.990 DEBUG:boto:Params: {} 2022-09-02T04:48:04.990 DEBUG:boto:Token: None 2022-09-02T04:48:04.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:04 GMT /xljbkf-30/?acl 2022-09-02T04:48:04.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lNplt3ABWG5rnCbPq7JTX0s+Ghg= 2022-09-02T04:48:04.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lNplt3ABWG5rnCbPq7JTX0s+Ghg=', 'Content-Length': '0'} 2022-09-02T04:48:04.994 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ef12055b54070c8-0063118b04-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 02 Sep 2022 04:48:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.038 DEBUG:boto:path=/xljbkf-31/ 2022-09-02T04:48:05.039 DEBUG:boto:auth_path=/xljbkf-31/ 2022-09-02T04:48:05.039 DEBUG:boto:path=/xljbkf-31/?acl 2022-09-02T04:48:05.040 DEBUG:boto:auth_path=/xljbkf-31/?acl 2022-09-02T04:48:05.040 DEBUG:boto:Method: GET 2022-09-02T04:48:05.045 DEBUG:boto:Path: /xljbkf-31/?acl 2022-09-02T04:48:05.046 DEBUG:boto:Data: 2022-09-02T04:48:05.047 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.047 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:05.048 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.048 DEBUG:boto:Params: {} 2022-09-02T04:48:05.049 DEBUG:boto:Token: None 2022-09-02T04:48:05.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-31/?acl 2022-09-02T04:48:05.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+vBBP8Eg5EqLJdxK5R8JsSeeWxc= 2022-09-02T04:48:05.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+vBBP8Eg5EqLJdxK5R8JsSeeWxc=', 'Content-Length': '0'} 2022-09-02T04:48:05.053 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004653e9c0c4e061e0-0063118b05-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.055 DEBUG:boto:path=/xljbkf-31/ 2022-09-02T04:48:05.056 DEBUG:boto:auth_path=/xljbkf-31/ 2022-09-02T04:48:05.057 DEBUG:boto:path=/xljbkf-31/?acl 2022-09-02T04:48:05.057 DEBUG:boto:auth_path=/xljbkf-31/?acl 2022-09-02T04:48:05.058 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.058 DEBUG:boto:Path: /xljbkf-31/?acl 2022-09-02T04:48:05.059 DEBUG:boto:Data: 2022-09-02T04:48:05.059 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:48:05.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:05.061 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.061 DEBUG:boto:Params: {} 2022-09-02T04:48:05.062 DEBUG:boto:Token: None 2022-09-02T04:48:05.062 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:05 GMT x-amz-acl:public-read /xljbkf-31/?acl 2022-09-02T04:48:05.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TdS2yruvnagKigIhWHoTgSTL87w= 2022-09-02T04:48:05.064 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TdS2yruvnagKigIhWHoTgSTL87w=', 'Content-Length': '0'} 2022-09-02T04:48:05.080 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000974334cd1d7490f4-0063118b05-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.081 DEBUG:boto:path=/xljbkf-31/ 2022-09-02T04:48:05.082 DEBUG:boto:auth_path=/xljbkf-31/ 2022-09-02T04:48:05.082 DEBUG:boto:path=/xljbkf-31/?acl 2022-09-02T04:48:05.083 DEBUG:boto:auth_path=/xljbkf-31/?acl 2022-09-02T04:48:05.083 DEBUG:boto:Method: GET 2022-09-02T04:48:05.084 DEBUG:boto:Path: /xljbkf-31/?acl 2022-09-02T04:48:05.084 DEBUG:boto:Data: 2022-09-02T04:48:05.085 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:05.086 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.086 DEBUG:boto:Params: {} 2022-09-02T04:48:05.087 DEBUG:boto:Token: None 2022-09-02T04:48:05.088 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-31/?acl 2022-09-02T04:48:05.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+vBBP8Eg5EqLJdxK5R8JsSeeWxc= 2022-09-02T04:48:05.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+vBBP8Eg5EqLJdxK5R8JsSeeWxc=', 'Content-Length': '0'} 2022-09-02T04:48:05.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8e836b4a721eeb6-0063118b05-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.092 DEBUG:boto:path=/xljbkf-32/ 2022-09-02T04:48:05.093 DEBUG:boto:auth_path=/xljbkf-32/ 2022-09-02T04:48:05.094 DEBUG:boto:path=/xljbkf-32/?acl 2022-09-02T04:48:05.095 DEBUG:boto:auth_path=/xljbkf-32/?acl 2022-09-02T04:48:05.095 DEBUG:boto:Method: GET 2022-09-02T04:48:05.096 DEBUG:boto:Path: /xljbkf-32/?acl 2022-09-02T04:48:05.096 DEBUG:boto:Data: 2022-09-02T04:48:05.096 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:05.098 DEBUG:boto:Port: 8001 2022-09-02T04:48:05.099 DEBUG:boto:Params: {} 2022-09-02T04:48:05.099 DEBUG:boto:Token: None 2022-09-02T04:48:05.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-32/?acl 2022-09-02T04:48:05.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yM1QLJY3OMqlhAfdxu2zsw9OX0I= 2022-09-02T04:48:05.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yM1QLJY3OMqlhAfdxu2zsw9OX0I=', 'Content-Length': '0'} 2022-09-02T04:48:05.103 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000351ffcc239975183-0063118b05-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.104 DEBUG:boto:path=/xljbkf-32/ 2022-09-02T04:48:05.104 DEBUG:boto:auth_path=/xljbkf-32/ 2022-09-02T04:48:05.105 DEBUG:boto:path=/xljbkf-32/?acl 2022-09-02T04:48:05.105 DEBUG:boto:auth_path=/xljbkf-32/?acl 2022-09-02T04:48:05.106 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.107 DEBUG:boto:Path: /xljbkf-32/?acl 2022-09-02T04:48:05.107 DEBUG:boto:Data: 2022-09-02T04:48:05.108 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2022-09-02T04:48:05.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:05.109 DEBUG:boto:Port: 8001 2022-09-02T04:48:05.110 DEBUG:boto:Params: {} 2022-09-02T04:48:05.111 DEBUG:boto:Token: None 2022-09-02T04:48:05.111 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:05 GMT x-amz-acl:public-read /xljbkf-32/?acl 2022-09-02T04:48:05.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7iCYzrrYRd0yK6C25LIrq2Wy9dU= 2022-09-02T04:48:05.112 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7iCYzrrYRd0yK6C25LIrq2Wy9dU=', 'Content-Length': '0'} 2022-09-02T04:48:05.126 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e343bdcf5bccc1b-0063118b05-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.127 DEBUG:boto:path=/xljbkf-32/ 2022-09-02T04:48:05.128 DEBUG:boto:auth_path=/xljbkf-32/ 2022-09-02T04:48:05.129 DEBUG:boto:path=/xljbkf-32/?acl 2022-09-02T04:48:05.130 DEBUG:boto:auth_path=/xljbkf-32/?acl 2022-09-02T04:48:05.131 DEBUG:boto:Method: GET 2022-09-02T04:48:05.131 DEBUG:boto:Path: /xljbkf-32/?acl 2022-09-02T04:48:05.132 DEBUG:boto:Data: 2022-09-02T04:48:05.132 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:05.133 DEBUG:boto:Port: 8001 2022-09-02T04:48:05.134 DEBUG:boto:Params: {} 2022-09-02T04:48:05.137 DEBUG:boto:Token: None 2022-09-02T04:48:05.138 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-32/?acl 2022-09-02T04:48:05.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yM1QLJY3OMqlhAfdxu2zsw9OX0I= 2022-09-02T04:48:05.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yM1QLJY3OMqlhAfdxu2zsw9OX0I=', 'Content-Length': '0'} 2022-09-02T04:48:05.142 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000281df97c6f557fef-0063118b05-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2022-09-02T04:48:05.183 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-33 2022-09-02T04:48:05.184 DEBUG:boto:path=/xljbkf-33/ 2022-09-02T04:48:05.184 DEBUG:boto:auth_path=/xljbkf-33/ 2022-09-02T04:48:05.185 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.185 DEBUG:boto:Path: /xljbkf-33/ 2022-09-02T04:48:05.186 DEBUG:boto:Data: 2022-09-02T04:48:05.186 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:05.187 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.188 DEBUG:boto:Params: {} 2022-09-02T04:48:05.189 DEBUG:boto:Token: None 2022-09-02T04:48:05.189 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-33/ 2022-09-02T04:48:05.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vA89MUIo6rDz9W6Q94Th/NoEC7A= 2022-09-02T04:48:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vA89MUIo6rDz9W6Q94Th/NoEC7A=', 'Content-Length': '0'} 2022-09-02T04:48:05.219 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000882440ece7b662b2-0063118b05-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.220 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-34 2022-09-02T04:48:05.220 DEBUG:boto:path=/xljbkf-34/ 2022-09-02T04:48:05.221 DEBUG:boto:auth_path=/xljbkf-34/ 2022-09-02T04:48:05.221 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.222 DEBUG:boto:Path: /xljbkf-34/ 2022-09-02T04:48:05.223 DEBUG:boto:Data: 2022-09-02T04:48:05.223 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:05.224 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.224 DEBUG:boto:Params: {} 2022-09-02T04:48:05.225 DEBUG:boto:Token: None 2022-09-02T04:48:05.226 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-34/ 2022-09-02T04:48:05.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ef+A1mghbjni9I+2VcgVxh0Dc34= 2022-09-02T04:48:05.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ef+A1mghbjni9I+2VcgVxh0Dc34=', 'Content-Length': '0'} 2022-09-02T04:48:05.294 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7516dd428c04627-0063118b05-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.295 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-35 2022-09-02T04:48:05.295 DEBUG:boto:path=/xljbkf-35/ 2022-09-02T04:48:05.297 DEBUG:boto:auth_path=/xljbkf-35/ 2022-09-02T04:48:05.297 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.298 DEBUG:boto:Path: /xljbkf-35/ 2022-09-02T04:48:05.299 DEBUG:boto:Data: 2022-09-02T04:48:05.299 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:05.300 DEBUG:boto:Port: 8001 2022-09-02T04:48:05.301 DEBUG:boto:Params: {} 2022-09-02T04:48:05.306 DEBUG:boto:Token: None 2022-09-02T04:48:05.308 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-35/ 2022-09-02T04:48:05.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t6iFhLYJtsgCB1OxQDIsUGulkCM= 2022-09-02T04:48:05.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t6iFhLYJtsgCB1OxQDIsUGulkCM=', 'Content-Length': '0'} 2022-09-02T04:48:05.363 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010458fb9cca80c1b-0063118b05-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.364 DEBUG:boto:path=/xljbkf-33/ 2022-09-02T04:48:05.365 DEBUG:boto:auth_path=/xljbkf-33/ 2022-09-02T04:48:05.366 DEBUG:boto:path=/xljbkf-33/?cors 2022-09-02T04:48:05.372 DEBUG:boto:auth_path=/xljbkf-33/?cors 2022-09-02T04:48:05.373 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.374 DEBUG:boto:Path: /xljbkf-33/?cors 2022-09-02T04:48:05.375 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-02T04:48:05.376 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-02T04:48:05.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:05.378 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.379 DEBUG:boto:Params: {} 2022-09-02T04:48:05.381 DEBUG:boto:Token: None 2022-09-02T04:48:05.382 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-33/?cors 2022-09-02T04:48:05.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwnRgBia3vCoCQY9/MnhtGVBe6A= 2022-09-02T04:48:05.384 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwnRgBia3vCoCQY9/MnhtGVBe6A=', 'Content-Length': '218'} 2022-09-02T04:48:05.398 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c28e64b01ab282ed-0063118b05-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.399 DEBUG:boto:path=/xljbkf-33/ 2022-09-02T04:48:05.400 DEBUG:boto:auth_path=/xljbkf-33/ 2022-09-02T04:48:05.400 DEBUG:boto:path=/xljbkf-33/?cors 2022-09-02T04:48:05.401 DEBUG:boto:auth_path=/xljbkf-33/?cors 2022-09-02T04:48:05.401 DEBUG:boto:Method: GET 2022-09-02T04:48:05.405 DEBUG:boto:Path: /xljbkf-33/?cors 2022-09-02T04:48:05.406 DEBUG:boto:Data: 2022-09-02T04:48:05.407 DEBUG:boto:Headers: {} 2022-09-02T04:48:05.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:05.408 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.409 DEBUG:boto:Params: {} 2022-09-02T04:48:05.414 DEBUG:boto:Token: None 2022-09-02T04:48:05.414 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-33/?cors 2022-09-02T04:48:05.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kASL9zz5ragCa8H6OX+Fq46tgNQ= 2022-09-02T04:48:05.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kASL9zz5ragCa8H6OX+Fq46tgNQ=', 'Content-Length': '0'} 2022-09-02T04:48:05.418 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075311e3102a78d9d-0063118b05-10fc-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 02 Sep 2022 04:48:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:05.419 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-02T04:48:05.420 DEBUG:boto:path=/xljbkf-34/ 2022-09-02T04:48:05.421 DEBUG:boto:auth_path=/xljbkf-34/ 2022-09-02T04:48:05.422 DEBUG:boto:path=/xljbkf-34/?cors 2022-09-02T04:48:05.423 DEBUG:boto:auth_path=/xljbkf-34/?cors 2022-09-02T04:48:05.423 DEBUG:boto:Method: PUT 2022-09-02T04:48:05.423 DEBUG:boto:Path: /xljbkf-34/?cors 2022-09-02T04:48:05.424 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-02T04:48:05.425 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-02T04:48:05.425 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:05.426 DEBUG:boto:Port: 8000 2022-09-02T04:48:05.426 DEBUG:boto:Params: {} 2022-09-02T04:48:05.427 DEBUG:boto:Token: None 2022-09-02T04:48:05.427 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 02 Sep 2022 04:48:05 GMT /xljbkf-34/?cors 2022-09-02T04:48:05.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EXjamPnknzPIBTUFv260aQNqFMI= 2022-09-02T04:48:05.428 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EXjamPnknzPIBTUFv260aQNqFMI=', 'Content-Length': '218'} 2022-09-02T04:48:06.443 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097311e8d56d7a407-0063118b05-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:06.445 DEBUG:boto:path=/xljbkf-34/ 2022-09-02T04:48:06.452 DEBUG:boto:auth_path=/xljbkf-34/ 2022-09-02T04:48:06.452 DEBUG:boto:path=/xljbkf-34/?cors 2022-09-02T04:48:06.454 DEBUG:boto:auth_path=/xljbkf-34/?cors 2022-09-02T04:48:06.454 DEBUG:boto:Method: GET 2022-09-02T04:48:06.455 DEBUG:boto:Path: /xljbkf-34/?cors 2022-09-02T04:48:06.455 DEBUG:boto:Data: 2022-09-02T04:48:06.456 DEBUG:boto:Headers: {} 2022-09-02T04:48:06.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:06.457 DEBUG:boto:Port: 8000 2022-09-02T04:48:06.457 DEBUG:boto:Params: {} 2022-09-02T04:48:06.458 DEBUG:boto:Token: None 2022-09-02T04:48:06.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:06 GMT /xljbkf-34/?cors 2022-09-02T04:48:06.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9UGo/PTWix9BubLvmNTqV7Ov/Dw= 2022-09-02T04:48:06.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9UGo/PTWix9BubLvmNTqV7Ov/Dw=', 'Content-Length': '0'} 2022-09-02T04:48:06.464 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ccefd0a47519dfc0-0063118b06-10e7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 02 Sep 2022 04:48:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:06.507 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-02T04:48:06.509 DEBUG:boto:path=/xljbkf-35/ 2022-09-02T04:48:06.509 DEBUG:boto:auth_path=/xljbkf-35/ 2022-09-02T04:48:06.510 DEBUG:boto:path=/xljbkf-35/?cors 2022-09-02T04:48:06.510 DEBUG:boto:auth_path=/xljbkf-35/?cors 2022-09-02T04:48:06.511 DEBUG:boto:Method: PUT 2022-09-02T04:48:06.511 DEBUG:boto:Path: /xljbkf-35/?cors 2022-09-02T04:48:06.512 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2022-09-02T04:48:06.512 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2022-09-02T04:48:06.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:06.513 DEBUG:boto:Port: 8001 2022-09-02T04:48:06.514 DEBUG:boto:Params: {} 2022-09-02T04:48:06.515 DEBUG:boto:Token: None 2022-09-02T04:48:06.515 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 02 Sep 2022 04:48:06 GMT /xljbkf-35/?cors 2022-09-02T04:48:06.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rlpjp7hFtRK7MJhc0kRw3kLH7g0= 2022-09-02T04:48:06.517 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rlpjp7hFtRK7MJhc0kRw3kLH7g0=', 'Content-Length': '218'} 2022-09-02T04:48:07.530 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099ad082df7365078-0063118b06-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:07.532 DEBUG:boto:path=/xljbkf-35/ 2022-09-02T04:48:07.533 DEBUG:boto:auth_path=/xljbkf-35/ 2022-09-02T04:48:07.534 DEBUG:boto:path=/xljbkf-35/?cors 2022-09-02T04:48:07.536 DEBUG:boto:auth_path=/xljbkf-35/?cors 2022-09-02T04:48:07.537 DEBUG:boto:Method: GET 2022-09-02T04:48:07.539 DEBUG:boto:Path: /xljbkf-35/?cors 2022-09-02T04:48:07.540 DEBUG:boto:Data: 2022-09-02T04:48:07.541 DEBUG:boto:Headers: {} 2022-09-02T04:48:07.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:07.543 DEBUG:boto:Port: 8001 2022-09-02T04:48:07.544 DEBUG:boto:Params: {} 2022-09-02T04:48:07.545 DEBUG:boto:Token: None 2022-09-02T04:48:07.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:48:07 GMT /xljbkf-35/?cors 2022-09-02T04:48:07.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BJup4zGksw678wH01n/7C8/Owr8= 2022-09-02T04:48:07.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BJup4zGksw678wH01n/7C8/Owr8=', 'Content-Length': '0'} 2022-09-02T04:48:07.565 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073ced3efd03ed268-0063118b07-1100-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 02 Sep 2022 04:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:07.567 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2022-09-02T04:48:07.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2022-09-02T04:48:07.568 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-36 2022-09-02T04:48:07.569 DEBUG:boto:path=/xljbkf-36/ 2022-09-02T04:48:07.569 DEBUG:boto:auth_path=/xljbkf-36/ 2022-09-02T04:48:07.571 DEBUG:boto:Method: PUT 2022-09-02T04:48:07.571 DEBUG:boto:Path: /xljbkf-36/ 2022-09-02T04:48:07.572 DEBUG:boto:Data: 2022-09-02T04:48:07.572 DEBUG:boto:Headers: {} 2022-09-02T04:48:07.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:07.573 DEBUG:boto:Port: 8000 2022-09-02T04:48:07.574 DEBUG:boto:Params: {} 2022-09-02T04:48:07.574 DEBUG:boto:Token: None 2022-09-02T04:48:07.575 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:07 GMT /xljbkf-36/ 2022-09-02T04:48:07.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bqj+6ydmXr3IPosASnydIBLUn8= 2022-09-02T04:48:07.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bqj+6ydmXr3IPosASnydIBLUn8=', 'Content-Length': '0'} 2022-09-02T04:48:07.605 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000075cd8642392021b-0063118b07-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:07.608 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-37 2022-09-02T04:48:07.609 DEBUG:boto:path=/xljbkf-37/ 2022-09-02T04:48:07.610 DEBUG:boto:auth_path=/xljbkf-37/ 2022-09-02T04:48:07.611 DEBUG:boto:Method: PUT 2022-09-02T04:48:07.612 DEBUG:boto:Path: /xljbkf-37/ 2022-09-02T04:48:07.612 DEBUG:boto:Data: 2022-09-02T04:48:07.613 DEBUG:boto:Headers: {} 2022-09-02T04:48:07.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:07.615 DEBUG:boto:Port: 8000 2022-09-02T04:48:07.616 DEBUG:boto:Params: {} 2022-09-02T04:48:07.616 DEBUG:boto:Token: None 2022-09-02T04:48:07.617 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:07 GMT /xljbkf-37/ 2022-09-02T04:48:07.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hwjWwTcvM/rPU0jJCVaND8y/u6w= 2022-09-02T04:48:07.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hwjWwTcvM/rPU0jJCVaND8y/u6w=', 'Content-Length': '0'} 2022-09-02T04:48:07.668 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071987ac95cea5782-0063118b07-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:07.670 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-38 2022-09-02T04:48:07.671 DEBUG:boto:path=/xljbkf-38/ 2022-09-02T04:48:07.671 DEBUG:boto:auth_path=/xljbkf-38/ 2022-09-02T04:48:07.672 DEBUG:boto:Method: PUT 2022-09-02T04:48:07.673 DEBUG:boto:Path: /xljbkf-38/ 2022-09-02T04:48:07.673 DEBUG:boto:Data: 2022-09-02T04:48:07.674 DEBUG:boto:Headers: {} 2022-09-02T04:48:07.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:07.676 DEBUG:boto:Port: 8001 2022-09-02T04:48:07.677 DEBUG:boto:Params: {} 2022-09-02T04:48:07.678 DEBUG:boto:Token: None 2022-09-02T04:48:07.678 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:07 GMT /xljbkf-38/ 2022-09-02T04:48:07.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNVjMydsgCOQz7sDcQeMaPVjcuU= 2022-09-02T04:48:07.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNVjMydsgCOQz7sDcQeMaPVjcuU=', 'Content-Length': '0'} 2022-09-02T04:48:07.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000275761ff16687c5c-0063118b07-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:07.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:07.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:07.718 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:07.828 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:48:07.841 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:48:07.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:07.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:07.842 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:48:07.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:07.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:07.842 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:48:07.843 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:48:07.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:07.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:07.844 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094087.696193_282.1", 2022-09-02T04:48:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:48:07.696193Z" 2022-09-02T04:48:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:07.846 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:48:07.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:07.846 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:48:07.847 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:07.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:48:07.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:07.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:07.849 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:07.901 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:07.901 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:07.960 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:07.959+0000 7f3bbc0e4600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:07.960 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:07.960+0000 7f3bbc0e4600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:07.986 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:07.987 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:07.987 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:07.988 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:07.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:07.991 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:07.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:07.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:07.998 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:07.998 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:07.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:07.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:08.000 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:08.000 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:08.001 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:08.001 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094087.696193_282.1", 2022-09-02T04:48:08.001 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:08.002 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:08.002 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:08.002 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.696193Z", 2022-09-02T04:48:08.002 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:08.005 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:08.005 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:08.005 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:08.005 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:08.006 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.006 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:08.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.007 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:08.007 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:08.007 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:08.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.008 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:07Z" 2022-09-02T04:48:08.014 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:08.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094087.696193_282.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:07.696193Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:07Z" } 2022-09-02T04:48:08.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094087.696193_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:07.696193Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:08.020 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:08.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:08.021 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:48:08.021 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:48:08.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:08.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:08.030 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:08.090 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:08.091 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:08.111 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:08.111+0000 7f4387a30600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:08.112 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:08.112+0000 7f4387a30600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:08.123 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:48:08.124 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:48:08.125 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:48:08.125 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:48:08.126 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:48:08.127 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:08.127 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:08.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:08.132 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:48:08.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:08.132 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:48:08.133 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:08.133 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:08.133 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:08.133 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:08.134 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:48:08.134 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:08.134 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:08.141 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:08.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:08.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:08.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:08.143 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:48:08.144 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:08.144 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:08.145 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:48:08.145 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:08.146 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:08.146 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:08.147 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:48:08.147 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:08.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:08.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:08.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:08.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:48:08.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:08.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:08.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094087.696193_282.1", 2022-09-02T04:48:08.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:08.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:08.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:08.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.696193Z", 2022-09-02T04:48:08.154 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:08.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:08.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:08.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:08.155 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:48:08.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:08.155 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:08.156 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:08.156 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:08.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:08.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:08.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:08.157 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:08.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:08.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:08.158 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:48:08.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:08.158 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:48:08.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:48:08.159 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:48:08.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:08.159 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:48:08Z" 2022-09-02T04:48:08.159 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:48:08.161 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094087.696193_282.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:07.696193Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:08Z" } 2022-09-02T04:48:08.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094087.696193_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:07.696193Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:08.162 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:08.163 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:08.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:48:08.164 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:48:08.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:08.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:08.169 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:08.231 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:08.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:08.276 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:08.275+0000 7fbce2034600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:08.276 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:08.276+0000 7fbce2034600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:08.296 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094084.680208_226.1", 2022-09-02T04:48:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.680208Z", 2022-09-02T04:48:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:08Z" 2022-09-02T04:48:08.330 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:08.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094084.680208_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:04.680208Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:08Z" } 2022-09-02T04:48:08.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094084.680208_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:04.680208Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:08.334 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:08.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094084.680208_226.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:08.335 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662094087.696193_282.1 target=1_1662094084.680208_226.1 2022-09-02T04:48:13.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:13.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:13.340 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:13.398 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:13.399 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:13.430 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:13.426+0000 7fcd4954d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:13.431 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:13.427+0000 7fcd4954d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:13.438 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:13.440 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:13.440 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094084.718596_229.1", 2022-09-02T04:48:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.718596Z", 2022-09-02T04:48:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:13Z" 2022-09-02T04:48:13.465 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:13.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094084.718596_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:04.718596Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:13Z" } 2022-09-02T04:48:13.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094084.718596_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:04.718596Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:13.468 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:13.469 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094084.718596_229.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:13.469 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662094087.696193_282.1 target=1_1662094084.718596_229.1 2022-09-02T04:48:18.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:18.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:18.474 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:18.536 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:18.537 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:18.574 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:18.574+0000 7f59ee9e3600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:18.575 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:18.575+0000 7f59ee9e3600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:18.586 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:18.587 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:18.587 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:18.588 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:18.588 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:18.589 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:18.589 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:18.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:18.590 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:18.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:18.591 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:18.592 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:18.592 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:18.593 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:18.593 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:18.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:18.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:18.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:18.596 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:18.596 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:18.596 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:18.596 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:18.597 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:18.597 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:18.598 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:18.598 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:18.598 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:18.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:18.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:18.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:18.599 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:18.600 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:18.600 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:18.600 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094084.718596_229.1", 2022-09-02T04:48:18.601 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:18.601 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:18.601 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:18.601 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.718596Z", 2022-09-02T04:48:18.602 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:18.602 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:18.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:18.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:18.603 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:18.603 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:18.604 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:18.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:18.605 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:18.605 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:18.605 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:18.605 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:18.606 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:18.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:18.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:18.607 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:18.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:18.607 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:18.608 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:18.608 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:18.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:18.609 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:18Z" 2022-09-02T04:48:18.609 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:18.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094084.718596_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:04.718596Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:18Z" } 2022-09-02T04:48:18.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094084.718596_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:04.718596Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:18.611 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:18.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094084.718596_229.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:18.612 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662094087.696193_282.1 target=1_1662094084.718596_229.1 2022-09-02T04:48:23.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:23.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:23.622 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:23.679 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:23.679 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:23.715 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:23.712+0000 7f2fdca82600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:23.716 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:23.713+0000 7f2fdca82600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:23.723 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:23.724 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:23.724 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:23.724 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:23.725 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:23.725 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:23.725 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:23.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:23.726 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:23.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:23.727 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:23.727 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:23.727 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:23.727 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:23.728 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:23.728 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:23.728 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:23.728 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:23.729 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:23.729 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:23.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:23.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:23.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:23.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:23.732 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:23.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:23.733 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:23.733 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:23.734 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:23.734 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:23.735 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:23.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:23.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:23.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:23.736 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:23.737 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:23.737 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:23.738 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094087.696193_282.1", 2022-09-02T04:48:23.738 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:23.738 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:23.739 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:23.739 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.696193Z", 2022-09-02T04:48:23.740 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:23.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:23.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:23.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:23.741 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:23.741 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:23.741 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:23.741 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:23.741 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:23.744 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:23.744 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:23.744 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:23.744 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:23.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:23.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:23.746 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:23.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:23.747 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:23.747 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:23.747 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:23.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:23.748 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:23Z" 2022-09-02T04:48:23.748 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:23.749 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094087.696193_282.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:07.696193Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:23Z" } 2022-09-02T04:48:23.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094087.696193_282.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:07.696193Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:23.751 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:23.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094087.696193_282.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:23.753 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:48:23.754 DEBUG:boto:path=/xljbkf-36/ 2022-09-02T04:48:23.754 DEBUG:boto:auth_path=/xljbkf-36/ 2022-09-02T04:48:23.755 DEBUG:boto:Method: HEAD 2022-09-02T04:48:23.756 DEBUG:boto:Path: /xljbkf-36/ 2022-09-02T04:48:23.757 DEBUG:boto:Data: 2022-09-02T04:48:23.757 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:23.762 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.763 DEBUG:boto:Params: {} 2022-09-02T04:48:23.765 DEBUG:boto:Token: None 2022-09-02T04:48:23.767 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-36/ 2022-09-02T04:48:23.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ln8S+Cmz6KnhmA5ZTTmEdPepTPA= 2022-09-02T04:48:23.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ln8S+Cmz6KnhmA5ZTTmEdPepTPA=', 'Content-Length': '0'} 2022-09-02T04:48:23.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce746c25410195f3-0063118b17-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.774 DEBUG:boto:path=/xljbkf-36/foo 2022-09-02T04:48:23.775 DEBUG:boto:auth_path=/xljbkf-36/foo 2022-09-02T04:48:23.776 DEBUG:boto:Method: PUT 2022-09-02T04:48:23.776 DEBUG:boto:Path: /xljbkf-36/foo 2022-09-02T04:48:23.777 DEBUG:boto:Data: 2022-09-02T04:48:23.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T04:48:23.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:23.779 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.779 DEBUG:boto:Params: {} 2022-09-02T04:48:23.780 DEBUG:boto:Token: None 2022-09-02T04:48:23.781 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-36/foo 2022-09-02T04:48:23.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BwqiCtfX3JkWUKvzkKzIa1ddtXU= 2022-09-02T04:48:23.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BwqiCtfX3JkWUKvzkKzIa1ddtXU='} 2022-09-02T04:48:23.797 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc1c2f0c4c3cd3d9-0063118b17-10fc-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.798 DEBUG:boto:path=/xljbkf-36/ 2022-09-02T04:48:23.799 DEBUG:boto:auth_path=/xljbkf-36/ 2022-09-02T04:48:23.800 DEBUG:boto:Method: DELETE 2022-09-02T04:48:23.801 DEBUG:boto:Path: /xljbkf-36/ 2022-09-02T04:48:23.802 DEBUG:boto:Data: 2022-09-02T04:48:23.803 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:23.805 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.806 DEBUG:boto:Params: {} 2022-09-02T04:48:23.807 DEBUG:boto:Token: None 2022-09-02T04:48:23.808 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-36/ 2022-09-02T04:48:23.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zi3S2PA8JgdL2KdQDZRbaVrx7FI= 2022-09-02T04:48:23.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zi3S2PA8JgdL2KdQDZRbaVrx7FI=', 'Content-Length': '0'} 2022-09-02T04:48:23.823 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx0000040450243e41c7a87-0063118b17-10fc-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.825 DEBUG:boto:path=/xljbkf-37/ 2022-09-02T04:48:23.825 DEBUG:boto:auth_path=/xljbkf-37/ 2022-09-02T04:48:23.826 DEBUG:boto:Method: HEAD 2022-09-02T04:48:23.827 DEBUG:boto:Path: /xljbkf-37/ 2022-09-02T04:48:23.828 DEBUG:boto:Data: 2022-09-02T04:48:23.832 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:23.834 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.835 DEBUG:boto:Params: {} 2022-09-02T04:48:23.836 DEBUG:boto:Token: None 2022-09-02T04:48:23.836 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-37/ 2022-09-02T04:48:23.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:04fTLuD1rGBo+TNJekm0kg7oBMI= 2022-09-02T04:48:23.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:04fTLuD1rGBo+TNJekm0kg7oBMI=', 'Content-Length': '0'} 2022-09-02T04:48:23.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fcbdc6962b804d9-0063118b17-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.844 DEBUG:boto:path=/xljbkf-37/foo 2022-09-02T04:48:23.845 DEBUG:boto:auth_path=/xljbkf-37/foo 2022-09-02T04:48:23.846 DEBUG:boto:Method: PUT 2022-09-02T04:48:23.847 DEBUG:boto:Path: /xljbkf-37/foo 2022-09-02T04:48:23.847 DEBUG:boto:Data: 2022-09-02T04:48:23.848 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T04:48:23.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:23.849 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.849 DEBUG:boto:Params: {} 2022-09-02T04:48:23.850 DEBUG:boto:Token: None 2022-09-02T04:48:23.851 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-37/foo 2022-09-02T04:48:23.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AbPTbXVt/gESDVciRdAAHfAWpDc= 2022-09-02T04:48:23.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AbPTbXVt/gESDVciRdAAHfAWpDc='} 2022-09-02T04:48:23.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b110c0a1c16d57a4-0063118b17-10e7-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.864 DEBUG:boto:path=/xljbkf-37/ 2022-09-02T04:48:23.865 DEBUG:boto:auth_path=/xljbkf-37/ 2022-09-02T04:48:23.866 DEBUG:boto:Method: DELETE 2022-09-02T04:48:23.867 DEBUG:boto:Path: /xljbkf-37/ 2022-09-02T04:48:23.868 DEBUG:boto:Data: 2022-09-02T04:48:23.869 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:23.871 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.872 DEBUG:boto:Params: {} 2022-09-02T04:48:23.873 DEBUG:boto:Token: None 2022-09-02T04:48:23.875 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-37/ 2022-09-02T04:48:23.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h92bFRD98ajwbcK+tyMUJQ3MSTc= 2022-09-02T04:48:23.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h92bFRD98ajwbcK+tyMUJQ3MSTc=', 'Content-Length': '0'} 2022-09-02T04:48:23.884 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx0000014141ed49997fd2b-0063118b17-10e7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.890 DEBUG:boto:path=/xljbkf-38/ 2022-09-02T04:48:23.891 DEBUG:boto:auth_path=/xljbkf-38/ 2022-09-02T04:48:23.892 DEBUG:boto:Method: HEAD 2022-09-02T04:48:23.893 DEBUG:boto:Path: /xljbkf-38/ 2022-09-02T04:48:23.893 DEBUG:boto:Data: 2022-09-02T04:48:23.894 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:23.896 DEBUG:boto:Port: 8001 2022-09-02T04:48:23.898 DEBUG:boto:Params: {} 2022-09-02T04:48:23.899 DEBUG:boto:Token: None 2022-09-02T04:48:23.900 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-38/ 2022-09-02T04:48:23.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOA08iqqAypKkOOAC1JROKAQj5Q= 2022-09-02T04:48:23.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOA08iqqAypKkOOAC1JROKAQj5Q=', 'Content-Length': '0'} 2022-09-02T04:48:23.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009df39c0351435112-0063118b17-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.908 DEBUG:boto:path=/xljbkf-38/foo 2022-09-02T04:48:23.909 DEBUG:boto:auth_path=/xljbkf-38/foo 2022-09-02T04:48:23.910 DEBUG:boto:Method: PUT 2022-09-02T04:48:23.911 DEBUG:boto:Path: /xljbkf-38/foo 2022-09-02T04:48:23.912 DEBUG:boto:Data: 2022-09-02T04:48:23.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T04:48:23.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:23.924 DEBUG:boto:Port: 8001 2022-09-02T04:48:23.925 DEBUG:boto:Params: {} 2022-09-02T04:48:23.925 DEBUG:boto:Token: None 2022-09-02T04:48:23.926 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-38/foo 2022-09-02T04:48:23.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t8mHoc0USsnVGqaXrBDJkHAgGmc= 2022-09-02T04:48:23.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t8mHoc0USsnVGqaXrBDJkHAgGmc='} 2022-09-02T04:48:23.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c80745458a8ff441-0063118b17-1100-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.942 DEBUG:boto:path=/xljbkf-38/ 2022-09-02T04:48:23.943 DEBUG:boto:auth_path=/xljbkf-38/ 2022-09-02T04:48:23.944 DEBUG:boto:Method: DELETE 2022-09-02T04:48:23.944 DEBUG:boto:Path: /xljbkf-38/ 2022-09-02T04:48:23.945 DEBUG:boto:Data: 2022-09-02T04:48:23.946 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:23.948 DEBUG:boto:Port: 8001 2022-09-02T04:48:23.949 DEBUG:boto:Params: {} 2022-09-02T04:48:23.950 DEBUG:boto:Token: None 2022-09-02T04:48:23.952 DEBUG:boto:StringToSign: DELETE Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-38/ 2022-09-02T04:48:23.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udX2aqrMokbkaiTIfIYpu89KnBU= 2022-09-02T04:48:23.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udX2aqrMokbkaiTIfIYpu89KnBU=', 'Content-Length': '0'} 2022-09-02T04:48:23.963 DEBUG:boto:Response headers: [('Content-Length', '234'), ('x-amz-request-id', 'tx000005fe2dc3a4e47015c-0063118b17-1100-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.964 DEBUG:boto:path=/xljbkf-36/ 2022-09-02T04:48:23.965 DEBUG:boto:auth_path=/xljbkf-36/ 2022-09-02T04:48:23.965 DEBUG:boto:Method: HEAD 2022-09-02T04:48:23.966 DEBUG:boto:Path: /xljbkf-36/ 2022-09-02T04:48:23.966 DEBUG:boto:Data: 2022-09-02T04:48:23.967 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:23.968 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.968 DEBUG:boto:Params: {} 2022-09-02T04:48:23.969 DEBUG:boto:Token: None 2022-09-02T04:48:23.969 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-36/ 2022-09-02T04:48:23.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ln8S+Cmz6KnhmA5ZTTmEdPepTPA= 2022-09-02T04:48:23.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ln8S+Cmz6KnhmA5ZTTmEdPepTPA=', 'Content-Length': '0'} 2022-09-02T04:48:23.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f90232069a3f9aee-0063118b17-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.975 DEBUG:boto:path=/xljbkf-37/ 2022-09-02T04:48:23.975 DEBUG:boto:auth_path=/xljbkf-37/ 2022-09-02T04:48:23.976 DEBUG:boto:Method: HEAD 2022-09-02T04:48:23.976 DEBUG:boto:Path: /xljbkf-37/ 2022-09-02T04:48:23.977 DEBUG:boto:Data: 2022-09-02T04:48:23.977 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:23.978 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.979 DEBUG:boto:Params: {} 2022-09-02T04:48:23.979 DEBUG:boto:Token: None 2022-09-02T04:48:23.980 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:23 GMT /xljbkf-37/ 2022-09-02T04:48:23.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:04fTLuD1rGBo+TNJekm0kg7oBMI= 2022-09-02T04:48:23.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:04fTLuD1rGBo+TNJekm0kg7oBMI=', 'Content-Length': '0'} 2022-09-02T04:48:23.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ba06d7734643971-0063118b17-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:23.988 DEBUG:boto:path=/xljbkf-38/ 2022-09-02T04:48:23.989 DEBUG:boto:auth_path=/xljbkf-38/ 2022-09-02T04:48:23.990 DEBUG:boto:Method: HEAD 2022-09-02T04:48:23.991 DEBUG:boto:Path: /xljbkf-38/ 2022-09-02T04:48:23.992 DEBUG:boto:Data: 2022-09-02T04:48:23.992 DEBUG:boto:Headers: {} 2022-09-02T04:48:23.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:23.996 DEBUG:boto:Port: 8000 2022-09-02T04:48:23.999 DEBUG:boto:Params: {} 2022-09-02T04:48:24.000 DEBUG:boto:Token: None 2022-09-02T04:48:24.001 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:48:24 GMT /xljbkf-38/ 2022-09-02T04:48:24.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IzEUWvpAenzKKlYy8EVUY1Ce7Hc= 2022-09-02T04:48:24.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IzEUWvpAenzKKlYy8EVUY1Ce7Hc=', 'Content-Length': '0'} 2022-09-02T04:48:24.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d22de175bba5f630-0063118b18-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:24.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2022-09-02T04:48:24.013 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-39 2022-09-02T04:48:24.014 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:48:24.015 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:48:24.016 DEBUG:boto:Method: PUT 2022-09-02T04:48:24.017 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:48:24.017 DEBUG:boto:Data: 2022-09-02T04:48:24.018 DEBUG:boto:Headers: {} 2022-09-02T04:48:24.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:48:24.020 DEBUG:boto:Port: 8000 2022-09-02T04:48:24.021 DEBUG:boto:Params: {} 2022-09-02T04:48:24.024 DEBUG:boto:Token: None 2022-09-02T04:48:24.025 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:24 GMT /xljbkf-39/ 2022-09-02T04:48:24.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nvPZj5NUNQjtN20//yTlAzajeqM= 2022-09-02T04:48:24.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nvPZj5NUNQjtN20//yTlAzajeqM=', 'Content-Length': '0'} 2022-09-02T04:48:24.050 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b361fefaffe7ff1-0063118b18-10fc-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:24.052 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-40 2022-09-02T04:48:24.053 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:48:24.053 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:48:24.057 DEBUG:boto:Method: PUT 2022-09-02T04:48:24.058 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:48:24.058 DEBUG:boto:Data: 2022-09-02T04:48:24.059 DEBUG:boto:Headers: {} 2022-09-02T04:48:24.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:48:24.060 DEBUG:boto:Port: 8000 2022-09-02T04:48:24.061 DEBUG:boto:Params: {} 2022-09-02T04:48:24.061 DEBUG:boto:Token: None 2022-09-02T04:48:24.062 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:24 GMT /xljbkf-40/ 2022-09-02T04:48:24.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BNSDsDHpXNniQD1mhysTyMGmSB4= 2022-09-02T04:48:24.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BNSDsDHpXNniQD1mhysTyMGmSB4=', 'Content-Length': '0'} 2022-09-02T04:48:24.101 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b7e109fd4f40248-0063118b18-10e7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:24.102 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-41 2022-09-02T04:48:24.103 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:48:24.103 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:48:24.104 DEBUG:boto:Method: PUT 2022-09-02T04:48:24.105 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:48:24.105 DEBUG:boto:Data: 2022-09-02T04:48:24.106 DEBUG:boto:Headers: {} 2022-09-02T04:48:24.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:48:24.107 DEBUG:boto:Port: 8001 2022-09-02T04:48:24.108 DEBUG:boto:Params: {} 2022-09-02T04:48:24.108 DEBUG:boto:Token: None 2022-09-02T04:48:24.109 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:48:24 GMT /xljbkf-41/ 2022-09-02T04:48:24.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OZ8sJeAGxQFTq8r03332UsVlkCE= 2022-09-02T04:48:24.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:48:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OZ8sJeAGxQFTq8r03332UsVlkCE=', 'Content-Length': '0'} 2022-09-02T04:48:24.154 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b787070d25c1a8e-0063118b18-1100-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:48:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:48:24.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:24.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:24.157 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:24.209 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:24.209 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=e248e3d2-c4b4-4705-8303-7ccdb6560fe1 2022-09-02T04:48:24.301 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:48:24.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:24.303 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:18:12.280197Z" 2022-09-02T04:48:24.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.306 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:48:24.306 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:48:24.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.308 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094104.139676_294.1", 2022-09-02T04:48:24.308 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:48:24.139676Z" 2022-09-02T04:48:24.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.310 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:24.310 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:20:21.771218Z" 2022-09-02T04:48:24.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:24.312 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:48:24.314 DEBUG:rgw_multi.tests:master meta markers={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:24.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:48:24.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:24.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:24.316 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:24.367 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:24.419 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:24.421+0000 7f526443d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:24.421 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:24.422+0000 7f526443d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:24.434 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:24.434 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:24.435 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:24.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:24.437 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:24.438 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:24.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:24.440 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:24.441 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:24.442 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:24.442 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:24.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:24.443 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094104.139676_294.1", 2022-09-02T04:48:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:24.444 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.139676Z", 2022-09-02T04:48:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:24.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:24.446 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:24.447 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:24.448 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:24.449 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:24.450 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:24Z" 2022-09-02T04:48:24.451 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:24.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094104.139676_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:24.139676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:24Z" } 2022-09-02T04:48:24.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094104.139676_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:24.139676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:24.453 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:24.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:24.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:48:24.455 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:48:24.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:24.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:24.457 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:24.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:24.508 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:24.555 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:24.555+0000 7f7a6d53c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:24.557 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:24.557+0000 7f7a6d53c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:24.570 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:48:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:48:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:48:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:48:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:48:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:48:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:48:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:48:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:48:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:48:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:48:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:48:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094104.042350_283.1", 2022-09-02T04:48:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.042350Z", 2022-09-02T04:48:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:48:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:48:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:48:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:48:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:48:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:48:24Z" 2022-09-02T04:48:24.600 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:48:24.601 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094104.042350_283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:24.042350Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:24Z" } 2022-09-02T04:48:24.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094104.042350_283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:24.042350Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:24.603 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:24.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.042350_283.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:24.604 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1662094104.139676_294.1 target=1_1662094104.042350_283.1 2022-09-02T04:48:29.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:29.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:29.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:29.707+0000 7f13fb1c1600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:48:29.708+0000 7f13fb1c1600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:29.729 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:48:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:48:29.730 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:48:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:48:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:48:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:29.731 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:48:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:29.732 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:48:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:29.733 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:48:29.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:29.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:29.735 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:29.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:29.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:29.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:29.737 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:48:29.737 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:29.737 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:29.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:48:29.738 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:29.738 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:29.739 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:29.740 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:48:29.741 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:29.741 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:29.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:29.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:29.742 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:48:29.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:29.743 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:29.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094104.139676_294.1", 2022-09-02T04:48:29.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:29.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:29.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:29.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.139676Z", 2022-09-02T04:48:29.747 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:29.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:29.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:48:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:48:29.749 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:48:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:48:29.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:48:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:48:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:48:29.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:29.754 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:48:29.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:29.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:48:29.756 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:48:29.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:29.757 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:48:29.758 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:48:29.758 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:48:29.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:48:29.759 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:48:29Z" 2022-09-02T04:48:29.760 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:48:29.763 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094104.139676_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:24.139676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:29Z" } 2022-09-02T04:48:29.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094104.139676_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:24.139676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:29.765 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:29.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:29.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:48:29.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:48:29.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:29.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:48:29.770 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:48:29.829 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:29.829 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:29.868 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:29.869+0000 7f0bac97c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:29.869 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:29.870+0000 7f0bac97c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:29.895 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:29.895 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:48:29.896 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:48:29.896 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:48:29.896 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:48:29.897 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:29.897 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:29.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:29.897 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:48:29.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:29.898 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:48:29.898 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:29.899 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:29.899 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:48:29.900 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:29.900 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:48:29.901 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:29.901 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:48:29.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:29.902 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:29.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:29.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:29.904 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:48:29.904 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:29.904 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:29.905 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:48:29.905 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:29.906 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:29.906 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:29.907 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:48:29.907 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:29.908 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:29.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:29.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:29.909 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:48:29.909 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:29.910 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:29.911 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094104.139676_294.1", 2022-09-02T04:48:29.912 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:29.912 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:29.912 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:29.913 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.139676Z", 2022-09-02T04:48:29.913 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:29.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:29.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:29.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:29.915 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:48:29.916 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:29.916 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:48:29.916 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:48:29.917 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:48:29.917 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:48:29.918 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:48:29.918 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:48:29.918 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:48:29.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:29.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:29.919 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:29.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:29.920 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:48:29.920 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:48:29.920 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:48:29.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:29.923 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:48:29Z" 2022-09-02T04:48:29.923 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:29.924 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094104.139676_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:24.139676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:48:29Z" } 2022-09-02T04:48:29.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094104.139676_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:24.139676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:48:29.926 DEBUG:rgw_multi.tests:log_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:29.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1662092292.280197_71.1', 1: '', 2: '1_1662094104.139676_294.1', 3: '1_1662092421.771218_140.1'} 2022-09-02T04:48:29.928 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:48:29.929 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-02T04:48:29.930 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T04:48:29.932 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T04:48:29.931+0000 7f9455941700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 112990) UID: 0 2022-09-02T04:48:29.933 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T04:48:29.931+0000 7f9459bac600 -1 shutting down 2022-09-02T04:48:31.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-02T04:48:36.033 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-02T04:48:36.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:48:36.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:48:36.036 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:48:36.093 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:36.094 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:36.116 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.116+0000 7f88f99a4600 0 NOTICE: overriding master zone: b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:48:36.116 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.116+0000 7f88f99a4600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:48:36.117 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.116+0000 7f88f99a4600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:48:36.118 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.118+0000 7f88f99a4600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:48:36.118 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.118+0000 7f88f99a4600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:48:36.125 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:36.126 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:48:36.127 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:48:36.127 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:48:36.128 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:48:36.128 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:48:36.128 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:48:36.129 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:48:36.129 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:48:36.130 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:48:36.130 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:48:36.131 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:48:36.131 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:48:36.131 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:48:36.132 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:48:36.132 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:48:36.133 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:48:36.133 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:48:36.134 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:48:36.134 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:48:36.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.135 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:48:36.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.136 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:48:36.137 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:48:36.137 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:48:36.137 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:48:36.138 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:48:36.138 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-02T04:48:36.138 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:48:36.139 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.139 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:48:36.140 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:48:36.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.140 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.141 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.141 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:48:36.141 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:48:36.141 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:36.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:48:36.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:48:36.143 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:48:36.196 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:48:36.197 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:48:36.239 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.235+0000 7f1d3cf7b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:36.240 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:48:36.236+0000 7f1d3cf7b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:48:36.247 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:48:36.247 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:48:36.248 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 1, 2022-09-02T04:48:36.248 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:48:36.249 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2022-09-02T04:48:36.249 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662092292.280197_71.1", 2022-09-02T04:48:36.249 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:48:36.250 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662094104.139676_294.1", 2022-09-02T04:48:36.250 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662092421.771218_140.1" 2022-09-02T04:48:36.251 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.251 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:48:36.252 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:48:36.252 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:48:36.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.253 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:48:36.253 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:48:36.254 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:48:36.254 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:48:36.254 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:48:36.255 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:48:36.255 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.256 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:48:36.256 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:48:36.256 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:48:36.257 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:48:36.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.258 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:48:36.258 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:48:36.259 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:48:36.259 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:48:36.259 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.260 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:48:36.260 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:48:36.261 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:48:36.261 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:48:36.262 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:48:36.262 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:48:36.263 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:48:36.263 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:48:36.263 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:48:36.264 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:48:36.264 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.270 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:48:36.271 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:48:36.271 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:48:36.272 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:48:36.272 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.273 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:48:36.273 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:48:36.274 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:48:36.274 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:48:36.276 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:48:36.276 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:48:36.279 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:48:36.279 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:48:36.279 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:48:36.280 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:48:36.280 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.282 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:48:36.282 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:48:36.283 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:48:36.283 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:48:36.283 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.284 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:48:36.284 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:48:36.285 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:48:36.285 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:48:36.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:48:36.287 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:48:36.287 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:48:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:48:36.288 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:48:36.289 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:48:36.289 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:48:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:48:36.291 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:48:36.292 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:48:36.292 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:48:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:48:36.293 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:48:36.295 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:48:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:48:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:48:36.296 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:48:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:48:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:48:36.297 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:48:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:48:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.298 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:48:36.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.301 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:48:36.301 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:48:36.302 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:48:36.302 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:48:36.302 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.303 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.303 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:48:36.304 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:48:36.304 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:48:36.304 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:48:36.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.305 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:48:36.305 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:48:36.305 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.306 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:48:36.306 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:48:36.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.307 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:48:36.307 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:48:36.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.308 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:48:36.309 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:48:36.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.310 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:48:36.310 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:48:36.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:48:36.311 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:48:36.311 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:48:36.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.312 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:48:36.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.312 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:48:36.312 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:48:36.313 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:48:36.313 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:48:36.313 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:48:36.313 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:48:36.314 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:48:36.314 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:48:36.314 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:48:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:48:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:48:36.315 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:48:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:48:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:48:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:48:36.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:48:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:48:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:48:36.317 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:48:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:48:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:48:36.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:48:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:48:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:48:36.319 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:48:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:48:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:48:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.320 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:48:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:48:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:48:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:48:36.321 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:48:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:48:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:48:36.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:48:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:48:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:48:36.323 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 3 2022-09-02T04:48:36.323 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:48:36.324 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2022-09-02T04:48:37.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-02T04:49:36.326 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-42 2022-09-02T04:49:36.327 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:49:36.328 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:49:36.329 DEBUG:boto:Method: PUT 2022-09-02T04:49:36.329 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:49:36.330 DEBUG:boto:Data: 2022-09-02T04:49:36.330 DEBUG:boto:Headers: {} 2022-09-02T04:49:36.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:49:36.331 DEBUG:boto:Port: 8000 2022-09-02T04:49:36.332 DEBUG:boto:Params: {} 2022-09-02T04:49:36.333 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:49:36.333 DEBUG:boto:Token: None 2022-09-02T04:49:36.335 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:49:36 GMT /xljbkf-42/ 2022-09-02T04:49:36.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CWVXo5HcM9REXncwFSMG3fiJh04= 2022-09-02T04:49:36.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:49:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CWVXo5HcM9REXncwFSMG3fiJh04=', 'Content-Length': '0'} 2022-09-02T04:49:36.373 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad9ee30a7e1ebfd0-0063118b60-140f-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:49:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:49:36.375 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-43 2022-09-02T04:49:36.376 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:49:36.376 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:49:36.377 DEBUG:boto:Method: PUT 2022-09-02T04:49:36.378 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:49:36.379 DEBUG:boto:Data: 2022-09-02T04:49:36.380 DEBUG:boto:Headers: {} 2022-09-02T04:49:36.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:49:36.382 DEBUG:boto:Port: 8000 2022-09-02T04:49:36.390 DEBUG:boto:Params: {} 2022-09-02T04:49:36.391 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:49:36.393 DEBUG:boto:Token: None 2022-09-02T04:49:36.394 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:49:36 GMT /xljbkf-43/ 2022-09-02T04:49:36.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HixmCGpdWfymW0uKyvMHFwMVCIc= 2022-09-02T04:49:36.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:49:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HixmCGpdWfymW0uKyvMHFwMVCIc=', 'Content-Length': '0'} 2022-09-02T04:49:36.418 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000276a57c1cfa60e28-0063118b60-1441-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:49:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:49:36.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:49:36.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:49:36.420 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:49:36.477 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:49:36.477 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:49:36.527 INFO:teuthology.orchestra.run.smithi160.stderr:No --period given, using current period=ffa7b930-977f-415c-8bb4-bf6fd394dc85 2022-09-02T04:49:36.554 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:49:36.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:49:36.555 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:49:36.555 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2022-09-02T04:49:36.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:49:36.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:49:36.557 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:49:36.558 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2022-09-02T04:49:36.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:49:36.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:49:36.559 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094176.417998_108.1", 2022-09-02T04:49:36.560 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:49:36.417998Z" 2022-09-02T04:49:36.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:49:36.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:49:36.561 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:49:36.562 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "0.000000" 2022-09-02T04:49:36.562 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:49:36.563 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:49:36.564 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094176.417998_108.1', 3: ''} 2022-09-02T04:49:36.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2022-09-02T04:49:36.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:49:36.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:49:36.568 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:49:36.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:49:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:49:36.664 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.664+0000 7f5febd30600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:49:36.665 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.666+0000 7f5febd30600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:49:36.681 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:49:36.683 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:49:36.683 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:49:36.683 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:49:36.683 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:49:36.684 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:49:36.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 3 2022-09-02T04:49:36.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.685 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:49:36.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:49:36.686 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:49:36.686 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 0, 2022-09-02T04:49:36.686 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "bucket.instance:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:49:36.686 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:49:36.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 81, 2022-09-02T04:49:36.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 4, 2022-09-02T04:49:36.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:49:36.688 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 0 2022-09-02T04:49:36.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:49:36.689 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:49:36.689 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:49:36.689 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:49:36.690 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:49:36.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:49:36.690 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:49:36.691 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:49:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 3 2022-09-02T04:49:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:49:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:49:36.693 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:49:36.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094176.417998_108.1", 2022-09-02T04:49:36.694 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:49:36.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:49:36.695 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:49:36.696 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.417998Z", 2022-09-02T04:49:36.696 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 3 2022-09-02T04:49:36.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.697 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:49:36.698 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:49:36.698 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:49:36.698 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:49:36.698 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:49:36.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:49:36.699 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:49:36.700 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:49:36.700 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 3 2022-09-02T04:49:36.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.701 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:36.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.701 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:49:36.702 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 81, 2022-09-02T04:49:36.702 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 4 2022-09-02T04:49:36.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.703 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:49:36Z" 2022-09-02T04:49:36.703 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:49:36.704 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 0, "marker": "bucket.instance:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", "next_step_marker": "", "total_entries": 81, "pos": 4, "timestamp": "0.000000", "realm_epoch": 0 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094176.417998_108.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:49:36.417998Z", "realm_epoch": 3 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 81, "complete": 4 }, "current_time": "2022-09-02T04:49:36Z" } 2022-09-02T04:49:36.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 0, 'marker': 'bucket.instance:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13', 'next_step_marker': '', 'total_entries': 81, 'pos': 4, 'timestamp': '0.000000', 'realm_epoch': 0}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094176.417998_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:49:36.417998Z', 'realm_epoch': 3}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 3}}] 2022-09-02T04:49:36.706 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094176.417998_108.1', 3: ''} 2022-09-02T04:49:36.707 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094176.417998_108.1', 3: ''} 2022-09-02T04:49:36.707 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2022-09-02T04:49:36.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:49:36.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:49:36.711 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:49:36.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:49:36.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:49:36.782 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.782+0000 7f9162506600 0 NOTICE: overriding master zone: 8b4ce987-f0fe-4038-9bfd-de8d46ec635c 2022-09-02T04:49:36.783 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.782+0000 7f9162506600 0 failed reading obj info from .rgw.root:zone_info.8b4ce987-f0fe-4038-9bfd-de8d46ec635c: (2) No such file or directory 2022-09-02T04:49:36.784 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.782+0000 7f9162506600 0 WARNING: could not read zone params for zone id=8b4ce987-f0fe-4038-9bfd-de8d46ec635c name=test-zone2 2022-09-02T04:49:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.785+0000 7f9162506600 0 failed reading obj info from .rgw.root:zone_info.fc4dbd55-e667-4fb2-9ede-75f3100f9d19: (2) No such file or directory 2022-09-02T04:49:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.785+0000 7f9162506600 0 WARNING: could not read zone params for zone id=fc4dbd55-e667-4fb2-9ede-75f3100f9d19 name=test-zone4 2022-09-02T04:49:36.790 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:49:36.790 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:49:36.791 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:49:36.791 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-09-02T04:49:36.792 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "test-zone1.rgw.control", 2022-09-02T04:49:36.793 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-09-02T04:49:36.793 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-09-02T04:49:36.794 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "test-zone1.rgw.log", 2022-09-02T04:49:36.794 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-09-02T04:49:36.794 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-09-02T04:49:36.795 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-09-02T04:49:36.795 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-09-02T04:49:36.795 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-09-02T04:49:36.795 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-09-02T04:49:36.796 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-09-02T04:49:36.796 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-09-02T04:49:36.797 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-09-02T04:49:36.797 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:49:36.797 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:49:36.798 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:49:36.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.798 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:49:36.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:49:36.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:49:36.799 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-09-02T04:49:36.800 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:49:36.800 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:49:36.800 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-09-02T04:49:36.800 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2022-09-02T04:49:36.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.801 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-09-02T04:49:36.802 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:49:36.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.802 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.803 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:49:36.803 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-09-02T04:49:36.803 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:49:36.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:49:36.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:49:36.806 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:49:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:49:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:49:36.908 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.908+0000 7f21dee3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:49:36.908 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:49:36.909+0000 7f21dee3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:49:36.922 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:49:36.923 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:49:36.923 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1, 2022-09-02T04:49:36.924 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:49:36.924 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [ 2022-09-02T04:49:36.925 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:49:36.925 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:49:36.926 INFO:teuthology.orchestra.run.smithi099.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:49:36.927 INFO:teuthology.orchestra.run.smithi099.stdout: "" 2022-09-02T04:49:36.927 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.928 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2022-09-02T04:49:36.928 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:49:36.929 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2022-09-02T04:49:36.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.930 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:49:36.930 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:49:36.931 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:49:36.931 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:49:36.932 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:49:36.932 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:49:36.933 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.934 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:49:36.934 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:49:36.935 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:49:36.935 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:49:36.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.936 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:49:36.937 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:49:36.938 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:49:36.938 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:49:36.939 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.939 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:49:36.940 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:49:36.941 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:49:36.941 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:49:36.942 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:49:36.942 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:49:36.943 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:49:36.943 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:49:36.944 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:49:36.944 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:49:36.945 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:36.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.947 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:49:36.951 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone2", 2022-09-02T04:49:36.951 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:49:36.952 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:49:36.953 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.954 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:49:36.957 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:49:36.957 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:49:36.958 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:49:36.958 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:49:36.959 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:49:36.959 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:49:36.960 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:49:36.962 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:49:36.962 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:49:36.963 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:36.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.965 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:49:36.966 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:49:36.966 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:49:36.967 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:49:36.967 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.968 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:49:36.968 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:49:36.969 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:49:36.969 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:49:36.970 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:49:36.970 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:49:36.971 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:49:36.972 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:49:36.973 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:49:36.973 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:49:36.974 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:36.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:36.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.975 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:49:36.976 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone4", 2022-09-02T04:49:36.976 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:49:36.977 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:49:36.977 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.978 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:49:36.978 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:49:36.979 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:49:36.979 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:49:36.987 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "pubsub", 2022-09-02T04:49:36.988 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2022-09-02T04:49:36.989 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [ 2022-09-02T04:49:36.989 INFO:teuthology.orchestra.run.smithi099.stdout: "test-zone1" 2022-09-02T04:49:36.990 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.990 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:49:36.991 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:49:36.991 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:49:36.992 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:36.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:36.993 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:36.993 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:49:36.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:36.995 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:49:36.997 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:49:36.998 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:49:36.998 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:49:36.999 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:36.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:37.000 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:37.000 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:49:37.001 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:49:37.001 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:49:37.002 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:49:37.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.003 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:49:37.004 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:49:37.004 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:37.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:37.005 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:49:37.005 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2022-09-02T04:49:37.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:37.006 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:49:37.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3855502858 2022-09-02T04:49:37.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:37.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:49:37.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 805948903 2022-09-02T04:49:37.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:37.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:49:37.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2020452060 2022-09-02T04:49:37.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:49:37.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:49:37.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4226356300 2022-09-02T04:49:37.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:37.014 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:49:37.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.015 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:49:37.015 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:49:37.016 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2022-09-02T04:49:37.016 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:49:37.016 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:49:37.017 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:49:37.017 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:49:37.017 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:49:37.017 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:49:37.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.018 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:49:37.018 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:49:37.019 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:49:37.019 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:49:37.019 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:49:37.019 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:49:37.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.020 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2022-09-02T04:49:37.020 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:49:37.021 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:49:37.021 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:49:37.021 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:49:37.021 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:49:37.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.022 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2022-09-02T04:49:37.022 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:49:37.023 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:49:37.023 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:49:37.023 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:49:37.023 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:49:37.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.024 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2022-09-02T04:49:37.024 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:49:37.025 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:49:37.025 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:49:37.026 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:49:37.026 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:49:37.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:49:37.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:49:37.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:49:37.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2022-09-02T04:49:37.029 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:49:37.029 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:49:37.030 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2022-09-02T04:50:37.031 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-44 2022-09-02T04:50:37.033 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:37.034 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:37.035 DEBUG:boto:Method: PUT 2022-09-02T04:50:37.035 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:37.037 DEBUG:boto:Data: 2022-09-02T04:50:37.038 DEBUG:boto:Headers: {} 2022-09-02T04:50:37.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:37.039 DEBUG:boto:Port: 8000 2022-09-02T04:50:37.040 DEBUG:boto:Params: {} 2022-09-02T04:50:37.042 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:50:37.043 DEBUG:boto:Token: None 2022-09-02T04:50:37.044 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:50:37 GMT /xljbkf-44/ 2022-09-02T04:50:37.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NMvLDdYKQEs6rZbOSiyMZzj6IUU= 2022-09-02T04:50:37.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NMvLDdYKQEs6rZbOSiyMZzj6IUU=', 'Content-Length': '0'} 2022-09-02T04:50:37.077 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd419d494877e9a8-0063118b9d-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:37.079 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-45 2022-09-02T04:50:37.080 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:37.081 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:37.081 DEBUG:boto:Method: PUT 2022-09-02T04:50:37.082 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:37.083 DEBUG:boto:Data: 2022-09-02T04:50:37.084 DEBUG:boto:Headers: {} 2022-09-02T04:50:37.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:37.089 DEBUG:boto:Port: 8000 2022-09-02T04:50:37.090 DEBUG:boto:Params: {} 2022-09-02T04:50:37.091 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:50:37.092 DEBUG:boto:Token: None 2022-09-02T04:50:37.093 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:50:37 GMT /xljbkf-45/ 2022-09-02T04:50:37.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H32BKPeFVDCvPPvGMVQu4smjMR4= 2022-09-02T04:50:37.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H32BKPeFVDCvPPvGMVQu4smjMR4=', 'Content-Length': '0'} 2022-09-02T04:50:37.135 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6cc1c45497a8548-0063118b9d-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:37.137 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-02T04:50:37.138 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-02T04:50:37.144 INFO:tasks.rgw.c1.client.1:Started 2022-09-02T04:50:37.146 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T04:50:37.179 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:50:37.180 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:50:37.181 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2022-09-02T04:50:38.181 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:38.205 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T04:50:38.206 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T04:50:38.206 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T04:50:38.208 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2022-09-02T04:50:38.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:38.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:38.211 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:38.296 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:38.297 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:38.337 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T04:50:38.351 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:38.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:38.352 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:38.352 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:50:38.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:38.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:38.354 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:38.354 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:50:38.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:38.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:38.356 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094237.119686_113.1", 2022-09-02T04:50:38.356 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:37.119686Z" 2022-09-02T04:50:38.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:38.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:38.358 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:38.358 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:50:38.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:38.359 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:38.361 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:38.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:50:38.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:38.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:38.363 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:38.416 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:38.417 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:38.457 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:38.457+0000 7f213ce38600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:38.458 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:38.458+0000 7f213ce38600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:38.474 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:50:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:50:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:50:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:50:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:50:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", 2022-09-02T04:50:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:50:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:50:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:50:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:50:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:50:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:50:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:50:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:50:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:50:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094104.139676_294.1", 2022-09-02T04:50:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.139676Z", 2022-09-02T04:50:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:50:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:38.489 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:50:38.489 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:38.489 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:38.490 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:38.490 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:50:38.490 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:38.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:50:38.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:38.492 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:50:38.492 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:50:38.492 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:50:38.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:38.493 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:50:38Z" 2022-09-02T04:50:38.493 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:50:38.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "e248e3d2-c4b4-4705-8303-7ccdb6560fe1", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094104.139676_294.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:48:24.139676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:50:38Z" } 2022-09-02T04:50:38.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094104.139676_294.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:48:24.139676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:50:38.496 WARNING:rgw_multi.tests:zone test-zone2 is syncing realm epoch=2, behind current realm epoch=4 2022-09-02T04:50:43.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:43.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:43.500 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:43.576 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:43.577 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:43.624 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:43.611+0000 7f9a92c1c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:43.625 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:43.614+0000 7f9a92c1c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:43.631 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:50:43.632 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:50:43.632 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:50:43.632 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:50:43.633 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:50:43.635 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:50:43.635 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:50:43.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:43.636 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:50:43.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:43.638 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:50:43.638 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:43.639 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:43.639 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:50:43.640 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:43.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:50:43.641 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:43.642 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:50:43.642 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:43.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:43.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:43.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:43.644 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:50:43.647 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:43.648 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:43.649 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:50:43.650 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:43.650 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:43.673 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:50:43.674 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:43.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:43.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:43.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:43.700 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:50:43.700 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:43.701 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:43.701 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094237.119686_113.1", 2022-09-02T04:50:43.702 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:43.710 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:43.711 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:43.711 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.119686Z", 2022-09-02T04:50:43.712 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:50:43.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:43.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:43.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:43.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:50:43.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:43.736 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:43.736 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:50:43.736 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:43.737 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:43.737 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:43.738 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:50:43.743 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:43.744 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:43.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:43.745 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:50:43.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:43.746 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:50:43.747 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:50:43.747 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:50:43.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:43.748 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:50:43Z" 2022-09-02T04:50:43.748 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:50:43.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094237.119686_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.119686Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:50:43Z" } 2022-09-02T04:50:43.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094237.119686_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.119686Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:50:43.753 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:43.754 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:43.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:50:43.755 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:50:43.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:43.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:43.763 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:43.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:43.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:43.875 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:50:43.874+0000 7ff0d5152600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:43.875 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:50:43.875+0000 7ff0d5152600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:43.895 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:50:43.896 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:50:43.896 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:50:43.896 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:50:43.897 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:50:43.897 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:50:43.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:50:43.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:43.898 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:50:43.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:43.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:50:43.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:50:43.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:50:43.899 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:50:43.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:50:43.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:50:43.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:50:43.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:50:43.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:50:43.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:43.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:43.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:43.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:50:43.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:50:43.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:50:43.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:43.903 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:50:43.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:50:43.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:50:43.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:50:43.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:50:43.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:43.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:43.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:50:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:50:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:50:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094237.119686_113.1", 2022-09-02T04:50:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:50:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:50:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:50:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.119686Z", 2022-09-02T04:50:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:50:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:50:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:50:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:50:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:50:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:50:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:50:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:50:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:50:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:50:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:50:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:50:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:50:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:50:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:50:43Z" 2022-09-02T04:50:43.913 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:50:43.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094237.119686_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.119686Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:50:43Z" } 2022-09-02T04:50:43.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094237.119686_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.119686Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:50:43.917 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:43.917 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:43.918 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:50:43.919 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:50:43.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:43.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:43.920 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:43.974 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:43.974 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:44.008 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:44.001+0000 7f32b08d4600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:44.009 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:44.002+0000 7f32b08d4600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:44.013 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:50:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:50:44.014 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:50:44.015 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:50:44.015 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:50:44.016 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:50:44.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:44.017 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:50:44.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:44.017 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:50:44.018 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:44.018 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:44.018 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:50:44.019 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:44.019 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:50:44.019 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:44.020 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:50:44.020 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:44.020 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:44.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:44.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:44.021 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:50:44.022 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:44.022 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:44.022 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:50:44.022 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:50:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:50:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094237.119686_113.1", 2022-09-02T04:50:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.119686Z", 2022-09-02T04:50:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:50:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:50:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:50:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:50:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:50:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:50:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:50:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:50:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:50:44Z" 2022-09-02T04:50:44.032 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:50:44.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094237.119686_113.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.119686Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:50:44Z" } 2022-09-02T04:50:44.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094237.119686_113.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.119686Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:50:44.035 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:44.035 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094237.119686_113.1', 3: ''} 2022-09-02T04:50:44.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:50:44.036 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-39 zones={test-zone2, test-zone1} 2022-09-02T04:50:44.038 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.039 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.039 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.040 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.040 DEBUG:boto:Data: 2022-09-02T04:50:44.041 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.042 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.042 DEBUG:boto:Params: {} 2022-09-02T04:50:44.043 DEBUG:boto:Token: None 2022-09-02T04:50:44.044 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005d5e21ba287b615-0063118ba4-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.054 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.055 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.058 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.058 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.063 DEBUG:boto:Data: 2022-09-02T04:50:44.064 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.065 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.066 DEBUG:boto:Params: {} 2022-09-02T04:50:44.067 DEBUG:boto:Token: None 2022-09-02T04:50:44.067 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d81b3f606969fad-0063118ba4-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.073 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.074 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.074 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.075 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.075 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.076 DEBUG:boto:Method: GET 2022-09-02T04:50:44.077 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.078 DEBUG:boto:Data: 2022-09-02T04:50:44.078 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.079 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.079 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.080 DEBUG:boto:Params: {} 2022-09-02T04:50:44.080 DEBUG:boto:Token: None 2022-09-02T04:50:44.081 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e0f1e462cf29e0-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.086 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.088 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.089 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.089 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.091 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.092 DEBUG:boto:Method: GET 2022-09-02T04:50:44.093 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.094 DEBUG:boto:Data: 2022-09-02T04:50:44.095 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.097 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.098 DEBUG:boto:Params: {} 2022-09-02T04:50:44.099 DEBUG:boto:Token: None 2022-09-02T04:50:44.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c39eb36e41b480f0-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.112 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.113 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.113 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.114 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.114 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.115 DEBUG:boto:Method: GET 2022-09-02T04:50:44.115 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.116 DEBUG:boto:Data: 2022-09-02T04:50:44.116 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.118 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.119 DEBUG:boto:Params: {} 2022-09-02T04:50:44.119 DEBUG:boto:Token: None 2022-09-02T04:50:44.120 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b29923a1fda9f1c-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.128 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.130 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.131 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.132 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.133 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.134 DEBUG:boto:Method: GET 2022-09-02T04:50:44.135 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.136 DEBUG:boto:Data: 2022-09-02T04:50:44.137 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.140 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.141 DEBUG:boto:Params: {} 2022-09-02T04:50:44.142 DEBUG:boto:Token: None 2022-09-02T04:50:44.144 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f5af6a47ca989f-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.150 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.152 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-39 zones={test-zone2, test-zone1} 2022-09-02T04:50:44.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-39 zones={test-zone3, test-zone1} 2022-09-02T04:50:44.154 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.156 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.156 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.159 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.160 DEBUG:boto:Data: 2022-09-02T04:50:44.161 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.166 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.167 DEBUG:boto:Params: {} 2022-09-02T04:50:44.168 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:50:44.177 DEBUG:boto:Token: None 2022-09-02T04:50:44.178 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000005e0509b7c19345-0063118ba4-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.188 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.189 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.191 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.191 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.192 DEBUG:boto:Data: 2022-09-02T04:50:44.192 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.195 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.196 DEBUG:boto:Params: {} 2022-09-02T04:50:44.197 DEBUG:boto:Token: None 2022-09-02T04:50:44.198 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028d8b8f23549f925-0063118ba4-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.205 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.206 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.207 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.208 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.209 DEBUG:boto:Method: GET 2022-09-02T04:50:44.210 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.211 DEBUG:boto:Data: 2022-09-02T04:50:44.211 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.213 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.214 DEBUG:boto:Params: {} 2022-09-02T04:50:44.215 DEBUG:boto:Token: None 2022-09-02T04:50:44.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000090f90fa15b02135-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.225 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.227 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.228 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.229 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.230 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.231 DEBUG:boto:Method: GET 2022-09-02T04:50:44.232 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.232 DEBUG:boto:Data: 2022-09-02T04:50:44.233 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.235 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.236 DEBUG:boto:Params: {} 2022-09-02T04:50:44.237 DEBUG:boto:Token: None 2022-09-02T04:50:44.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6281bfaaded562f-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.290 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.300 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.302 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.303 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.304 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.305 DEBUG:boto:Method: GET 2022-09-02T04:50:44.306 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.307 DEBUG:boto:Data: 2022-09-02T04:50:44.307 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.309 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.310 DEBUG:boto:Params: {} 2022-09-02T04:50:44.311 DEBUG:boto:Token: None 2022-09-02T04:50:44.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6d57d99d8ff895e-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.329 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.331 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.332 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.333 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.334 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.334 DEBUG:boto:Method: GET 2022-09-02T04:50:44.335 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.336 DEBUG:boto:Data: 2022-09-02T04:50:44.337 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.340 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.341 DEBUG:boto:Params: {} 2022-09-02T04:50:44.342 DEBUG:boto:Token: None 2022-09-02T04:50:44.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000984b336b810f4754-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.356 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.357 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-39 zones={test-zone3, test-zone1} 2022-09-02T04:50:44.358 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-39 zones={test-zone1, test-zone2} 2022-09-02T04:50:44.359 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.360 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.361 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.362 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.362 DEBUG:boto:Data: 2022-09-02T04:50:44.364 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.367 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.368 DEBUG:boto:Params: {} 2022-09-02T04:50:44.369 DEBUG:boto:Token: None 2022-09-02T04:50:44.370 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000421b11e6f60978-0063118ba4-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.377 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.378 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.378 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.379 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.380 DEBUG:boto:Data: 2022-09-02T04:50:44.381 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.386 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.386 DEBUG:boto:Params: {} 2022-09-02T04:50:44.387 DEBUG:boto:Token: None 2022-09-02T04:50:44.388 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000766f7b2e484428ab-0063118ba4-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.393 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.394 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.394 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.395 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.396 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.397 DEBUG:boto:Method: GET 2022-09-02T04:50:44.397 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.398 DEBUG:boto:Data: 2022-09-02T04:50:44.398 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.399 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.400 DEBUG:boto:Params: {} 2022-09-02T04:50:44.400 DEBUG:boto:Token: None 2022-09-02T04:50:44.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051fcdbad178261b3-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.409 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.411 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.412 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.413 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.413 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.414 DEBUG:boto:Method: GET 2022-09-02T04:50:44.415 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.415 DEBUG:boto:Data: 2022-09-02T04:50:44.416 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.417 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.417 DEBUG:boto:Params: {} 2022-09-02T04:50:44.418 DEBUG:boto:Token: None 2022-09-02T04:50:44.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f9306813d0a101-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.424 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.425 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.425 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.426 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.427 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.427 DEBUG:boto:Method: GET 2022-09-02T04:50:44.428 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.428 DEBUG:boto:Data: 2022-09-02T04:50:44.429 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.430 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.430 DEBUG:boto:Params: {} 2022-09-02T04:50:44.431 DEBUG:boto:Token: None 2022-09-02T04:50:44.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049bc2bf61ae4e956-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.437 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.438 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.438 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.439 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.439 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.440 DEBUG:boto:Method: GET 2022-09-02T04:50:44.441 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.441 DEBUG:boto:Data: 2022-09-02T04:50:44.442 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.443 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.443 DEBUG:boto:Params: {} 2022-09-02T04:50:44.444 DEBUG:boto:Token: None 2022-09-02T04:50:44.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f31a2beaaa292ec-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.455 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.455 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-39 zones={test-zone1, test-zone2} 2022-09-02T04:50:44.456 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-39 zones={test-zone3, test-zone2} 2022-09-02T04:50:44.456 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.457 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.458 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.459 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.460 DEBUG:boto:Data: 2022-09-02T04:50:44.460 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.463 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.464 DEBUG:boto:Params: {} 2022-09-02T04:50:44.464 DEBUG:boto:Token: None 2022-09-02T04:50:44.465 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000017c51988bb0e2c1-0063118ba4-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.477 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.478 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.479 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.480 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.481 DEBUG:boto:Data: 2022-09-02T04:50:44.482 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.485 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.486 DEBUG:boto:Params: {} 2022-09-02T04:50:44.487 DEBUG:boto:Token: None 2022-09-02T04:50:44.487 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9015ad8ebe4bcce-0063118ba4-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.493 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.494 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.495 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.498 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.500 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.501 DEBUG:boto:Method: GET 2022-09-02T04:50:44.502 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.503 DEBUG:boto:Data: 2022-09-02T04:50:44.504 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.513 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.514 DEBUG:boto:Params: {} 2022-09-02T04:50:44.515 DEBUG:boto:Token: None 2022-09-02T04:50:44.516 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b964847686d9b91b-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.524 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.525 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.526 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.527 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.530 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.531 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.532 DEBUG:boto:Method: GET 2022-09-02T04:50:44.533 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.534 DEBUG:boto:Data: 2022-09-02T04:50:44.535 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.537 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.539 DEBUG:boto:Params: {} 2022-09-02T04:50:44.540 DEBUG:boto:Token: None 2022-09-02T04:50:44.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c9a03ed148910bc-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.556 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.557 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.561 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.562 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.564 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.565 DEBUG:boto:Method: GET 2022-09-02T04:50:44.566 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.568 DEBUG:boto:Data: 2022-09-02T04:50:44.577 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.579 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.580 DEBUG:boto:Params: {} 2022-09-02T04:50:44.581 DEBUG:boto:Token: None 2022-09-02T04:50:44.582 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.583 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df12d6999f464bd2-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.588 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.589 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.590 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.591 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.592 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.593 DEBUG:boto:Method: GET 2022-09-02T04:50:44.594 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.595 DEBUG:boto:Data: 2022-09-02T04:50:44.596 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.598 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.599 DEBUG:boto:Params: {} 2022-09-02T04:50:44.600 DEBUG:boto:Token: None 2022-09-02T04:50:44.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef43011072f4b5f-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.610 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.612 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-39 zones={test-zone3, test-zone2} 2022-09-02T04:50:44.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-39 zones={test-zone1, test-zone3} 2022-09-02T04:50:44.614 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.615 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.616 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.617 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.618 DEBUG:boto:Data: 2022-09-02T04:50:44.619 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.622 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.623 DEBUG:boto:Params: {} 2022-09-02T04:50:44.624 DEBUG:boto:Token: None 2022-09-02T04:50:44.625 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5da058281db1d20-0063118ba4-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.635 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.636 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.641 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.642 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.643 DEBUG:boto:Data: 2022-09-02T04:50:44.644 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.646 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.647 DEBUG:boto:Params: {} 2022-09-02T04:50:44.648 DEBUG:boto:Token: None 2022-09-02T04:50:44.649 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fbc8fdd6a20b333-0063118ba4-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.655 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.656 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.657 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.658 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.658 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.659 DEBUG:boto:Method: GET 2022-09-02T04:50:44.659 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.660 DEBUG:boto:Data: 2022-09-02T04:50:44.661 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.664 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.665 DEBUG:boto:Params: {} 2022-09-02T04:50:44.666 DEBUG:boto:Token: None 2022-09-02T04:50:44.667 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1b36248c95c64f-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.714 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.716 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.717 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.717 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.719 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.720 DEBUG:boto:Method: GET 2022-09-02T04:50:44.721 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.722 DEBUG:boto:Data: 2022-09-02T04:50:44.723 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.724 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.725 DEBUG:boto:Params: {} 2022-09-02T04:50:44.726 DEBUG:boto:Token: None 2022-09-02T04:50:44.729 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1b8a6e25cc9ef0d-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.744 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.745 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.747 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.748 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.749 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.750 DEBUG:boto:Method: GET 2022-09-02T04:50:44.751 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.752 DEBUG:boto:Data: 2022-09-02T04:50:44.753 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.755 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.756 DEBUG:boto:Params: {} 2022-09-02T04:50:44.757 DEBUG:boto:Token: None 2022-09-02T04:50:44.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000280728c9bbf19319-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.765 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.766 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.767 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.767 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.768 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.769 DEBUG:boto:Method: GET 2022-09-02T04:50:44.770 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.771 DEBUG:boto:Data: 2022-09-02T04:50:44.771 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.772 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.773 DEBUG:boto:Params: {} 2022-09-02T04:50:44.773 DEBUG:boto:Token: None 2022-09-02T04:50:44.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc624c24c164c6b6-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.779 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.780 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-39 zones={test-zone1, test-zone3} 2022-09-02T04:50:44.780 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-39 zones={test-zone2, test-zone3} 2022-09-02T04:50:44.781 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.781 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.782 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.783 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.783 DEBUG:boto:Data: 2022-09-02T04:50:44.784 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.785 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.786 DEBUG:boto:Params: {} 2022-09-02T04:50:44.786 DEBUG:boto:Token: None 2022-09-02T04:50:44.787 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046100cf37745a759-0063118ba4-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.794 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.795 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.795 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.796 DEBUG:boto:Path: /xljbkf-39/ 2022-09-02T04:50:44.796 DEBUG:boto:Data: 2022-09-02T04:50:44.797 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.800 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.801 DEBUG:boto:Params: {} 2022-09-02T04:50:44.801 DEBUG:boto:Token: None 2022-09-02T04:50:44.802 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/ 2022-09-02T04:50:44.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY= 2022-09-02T04:50:44.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SrAEvg8ltlm1I7DS5pJ3RikfbsY=', 'Content-Length': '0'} 2022-09-02T04:50:44.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e10074fc4612225-0063118ba4-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.808 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.808 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.809 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.810 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.811 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.812 DEBUG:boto:Method: GET 2022-09-02T04:50:44.813 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.814 DEBUG:boto:Data: 2022-09-02T04:50:44.815 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.816 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.817 DEBUG:boto:Params: {} 2022-09-02T04:50:44.818 DEBUG:boto:Token: None 2022-09-02T04:50:44.819 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c0a60732a797bf-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.825 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.826 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.827 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.828 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.829 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.830 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.831 DEBUG:boto:Method: GET 2022-09-02T04:50:44.832 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.833 DEBUG:boto:Data: 2022-09-02T04:50:44.834 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.836 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.836 DEBUG:boto:Params: {} 2022-09-02T04:50:44.837 DEBUG:boto:Token: None 2022-09-02T04:50:44.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d1d2fbc1a15eb5-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.845 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.847 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.848 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.849 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.850 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.851 DEBUG:boto:Method: GET 2022-09-02T04:50:44.852 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.852 DEBUG:boto:Data: 2022-09-02T04:50:44.853 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.855 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.857 DEBUG:boto:Params: {} 2022-09-02T04:50:44.857 DEBUG:boto:Token: None 2022-09-02T04:50:44.858 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ba3e61c9c39f52-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.863 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.863 DEBUG:boto:path=/xljbkf-39/ 2022-09-02T04:50:44.864 DEBUG:boto:auth_path=/xljbkf-39/ 2022-09-02T04:50:44.864 DEBUG:boto:path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.865 DEBUG:boto:auth_path=/xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.865 DEBUG:boto:Method: GET 2022-09-02T04:50:44.866 DEBUG:boto:Path: /xljbkf-39/?versions&max-keys=999 2022-09-02T04:50:44.866 DEBUG:boto:Data: 2022-09-02T04:50:44.868 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.870 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.870 DEBUG:boto:Params: {} 2022-09-02T04:50:44.871 DEBUG:boto:Token: None 2022-09-02T04:50:44.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-39/?versions 2022-09-02T04:50:44.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E= 2022-09-02T04:50:44.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gwZippdtjJpAM6tgmf7ATrxF5E=', 'Content-Length': '0'} 2022-09-02T04:50:44.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0b3c3b5115af57-0063118ba4-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.880 DEBUG:boto:b'xljbkf-39999false' 2022-09-02T04:50:44.881 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-39 zones={test-zone2, test-zone3} 2022-09-02T04:50:44.881 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-40 zones={test-zone2, test-zone1} 2022-09-02T04:50:44.882 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.882 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.883 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.884 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:44.884 DEBUG:boto:Data: 2022-09-02T04:50:44.885 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.886 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.887 DEBUG:boto:Params: {} 2022-09-02T04:50:44.887 DEBUG:boto:Token: None 2022-09-02T04:50:44.888 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/ 2022-09-02T04:50:44.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NECIqd1kki2gTFDKuqRl5kkqlvs= 2022-09-02T04:50:44.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NECIqd1kki2gTFDKuqRl5kkqlvs=', 'Content-Length': '0'} 2022-09-02T04:50:44.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000580cdd4708f4ac6f-0063118ba4-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.896 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.896 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.897 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.897 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:44.898 DEBUG:boto:Data: 2022-09-02T04:50:44.899 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.900 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.900 DEBUG:boto:Params: {} 2022-09-02T04:50:44.901 DEBUG:boto:Token: None 2022-09-02T04:50:44.901 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/ 2022-09-02T04:50:44.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NECIqd1kki2gTFDKuqRl5kkqlvs= 2022-09-02T04:50:44.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NECIqd1kki2gTFDKuqRl5kkqlvs=', 'Content-Length': '0'} 2022-09-02T04:50:44.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e44e201bf911dbf-0063118ba4-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:44.912 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.912 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.913 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.914 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.914 DEBUG:boto:Method: GET 2022-09-02T04:50:44.915 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.915 DEBUG:boto:Data: 2022-09-02T04:50:44.916 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.917 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.917 DEBUG:boto:Params: {} 2022-09-02T04:50:44.919 DEBUG:boto:Token: None 2022-09-02T04:50:44.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/?versions 2022-09-02T04:50:44.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og= 2022-09-02T04:50:44.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og=', 'Content-Length': '0'} 2022-09-02T04:50:44.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd00dbc612cc259-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.925 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:44.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:44.926 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.927 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.928 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.928 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.929 DEBUG:boto:Method: GET 2022-09-02T04:50:44.929 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.930 DEBUG:boto:Data: 2022-09-02T04:50:44.931 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.932 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.933 DEBUG:boto:Params: {} 2022-09-02T04:50:44.934 DEBUG:boto:Token: None 2022-09-02T04:50:44.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/?versions 2022-09-02T04:50:44.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og= 2022-09-02T04:50:44.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og=', 'Content-Length': '0'} 2022-09-02T04:50:44.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d40b503695547d3e-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.939 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:44.940 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.941 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.943 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.944 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.945 DEBUG:boto:Method: GET 2022-09-02T04:50:44.946 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.947 DEBUG:boto:Data: 2022-09-02T04:50:44.948 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:44.950 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.951 DEBUG:boto:Params: {} 2022-09-02T04:50:44.954 DEBUG:boto:Token: None 2022-09-02T04:50:44.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/?versions 2022-09-02T04:50:44.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og= 2022-09-02T04:50:44.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og=', 'Content-Length': '0'} 2022-09-02T04:50:44.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000847130b034a47644-0063118ba4-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.963 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:44.964 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.965 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.966 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.966 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.967 DEBUG:boto:Method: GET 2022-09-02T04:50:44.967 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:44.968 DEBUG:boto:Data: 2022-09-02T04:50:44.968 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:44.970 DEBUG:boto:Port: 8000 2022-09-02T04:50:44.970 DEBUG:boto:Params: {} 2022-09-02T04:50:44.971 DEBUG:boto:Token: None 2022-09-02T04:50:44.971 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/?versions 2022-09-02T04:50:44.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og= 2022-09-02T04:50:44.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RW0pqq44SnTQiE9Lzzbb9c9u2Og=', 'Content-Length': '0'} 2022-09-02T04:50:44.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb016734e63b550a-0063118ba4-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:44.976 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:44.977 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-40 zones={test-zone2, test-zone1} 2022-09-02T04:50:44.979 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-40 zones={test-zone3, test-zone1} 2022-09-02T04:50:44.980 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:44.981 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:44.982 DEBUG:boto:Method: HEAD 2022-09-02T04:50:44.983 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:44.984 DEBUG:boto:Data: 2022-09-02T04:50:44.984 DEBUG:boto:Headers: {} 2022-09-02T04:50:44.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:44.986 DEBUG:boto:Port: 8001 2022-09-02T04:50:44.987 DEBUG:boto:Params: {} 2022-09-02T04:50:44.988 DEBUG:boto:Token: None 2022-09-02T04:50:44.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:44 GMT /xljbkf-40/ 2022-09-02T04:50:44.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NECIqd1kki2gTFDKuqRl5kkqlvs= 2022-09-02T04:50:44.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NECIqd1kki2gTFDKuqRl5kkqlvs=', 'Content-Length': '0'} 2022-09-02T04:50:45.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005972b1a88c11f68d-0063118ba4-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.002 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.003 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.003 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.005 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.005 DEBUG:boto:Data: 2022-09-02T04:50:45.006 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.007 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.008 DEBUG:boto:Params: {} 2022-09-02T04:50:45.009 DEBUG:boto:Token: None 2022-09-02T04:50:45.010 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beea6888a343239f-0063118ba5-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.023 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.024 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.025 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.026 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.027 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.029 DEBUG:boto:Method: GET 2022-09-02T04:50:45.030 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.030 DEBUG:boto:Data: 2022-09-02T04:50:45.031 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.033 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.033 DEBUG:boto:Params: {} 2022-09-02T04:50:45.034 DEBUG:boto:Token: None 2022-09-02T04:50:45.034 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.035 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be52fed77c68c134-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.086 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.089 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.090 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.091 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.093 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.094 DEBUG:boto:Method: GET 2022-09-02T04:50:45.095 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.097 DEBUG:boto:Data: 2022-09-02T04:50:45.099 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.101 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.102 DEBUG:boto:Params: {} 2022-09-02T04:50:45.103 DEBUG:boto:Token: None 2022-09-02T04:50:45.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d7366cdb0f791d1-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.114 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.116 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.117 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.118 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.119 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.120 DEBUG:boto:Method: GET 2022-09-02T04:50:45.121 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.122 DEBUG:boto:Data: 2022-09-02T04:50:45.123 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.124 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.125 DEBUG:boto:Params: {} 2022-09-02T04:50:45.126 DEBUG:boto:Token: None 2022-09-02T04:50:45.127 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001abd31428ae5a1cc-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.134 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.135 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.136 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.136 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.137 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.137 DEBUG:boto:Method: GET 2022-09-02T04:50:45.138 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.139 DEBUG:boto:Data: 2022-09-02T04:50:45.139 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.141 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.141 DEBUG:boto:Params: {} 2022-09-02T04:50:45.142 DEBUG:boto:Token: None 2022-09-02T04:50:45.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc6a3723510f3f8-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.149 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-40 zones={test-zone3, test-zone1} 2022-09-02T04:50:45.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-40 zones={test-zone1, test-zone2} 2022-09-02T04:50:45.152 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.152 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.153 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.154 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.155 DEBUG:boto:Data: 2022-09-02T04:50:45.156 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.158 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.159 DEBUG:boto:Params: {} 2022-09-02T04:50:45.160 DEBUG:boto:Token: None 2022-09-02T04:50:45.161 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037e495d020bdbc66-0063118ba5-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.168 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.169 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.170 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.171 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.172 DEBUG:boto:Data: 2022-09-02T04:50:45.173 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.174 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.175 DEBUG:boto:Params: {} 2022-09-02T04:50:45.176 DEBUG:boto:Token: None 2022-09-02T04:50:45.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aea69cf3be7cb938-0063118ba5-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.182 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.182 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.183 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.183 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.184 DEBUG:boto:Method: GET 2022-09-02T04:50:45.185 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.185 DEBUG:boto:Data: 2022-09-02T04:50:45.186 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.187 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.187 DEBUG:boto:Params: {} 2022-09-02T04:50:45.188 DEBUG:boto:Token: None 2022-09-02T04:50:45.188 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e45cdfb78a897e-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.234 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.237 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.238 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.239 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.240 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.242 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.243 DEBUG:boto:Method: GET 2022-09-02T04:50:45.244 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.246 DEBUG:boto:Data: 2022-09-02T04:50:45.246 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.247 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.248 DEBUG:boto:Params: {} 2022-09-02T04:50:45.248 DEBUG:boto:Token: None 2022-09-02T04:50:45.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb1cc065accae83-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.259 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.260 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.261 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.262 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.263 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.271 DEBUG:boto:Method: GET 2022-09-02T04:50:45.272 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.273 DEBUG:boto:Data: 2022-09-02T04:50:45.274 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.276 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.277 DEBUG:boto:Params: {} 2022-09-02T04:50:45.278 DEBUG:boto:Token: None 2022-09-02T04:50:45.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13ac0376b5b0b96-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.287 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.288 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.289 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.290 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.291 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.292 DEBUG:boto:Method: GET 2022-09-02T04:50:45.293 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.302 DEBUG:boto:Data: 2022-09-02T04:50:45.303 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.305 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.312 DEBUG:boto:Params: {} 2022-09-02T04:50:45.314 DEBUG:boto:Token: None 2022-09-02T04:50:45.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a598db0d1669ff8-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.321 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.322 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-40 zones={test-zone1, test-zone2} 2022-09-02T04:50:45.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-40 zones={test-zone3, test-zone2} 2022-09-02T04:50:45.323 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.323 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.324 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.324 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.325 DEBUG:boto:Data: 2022-09-02T04:50:45.325 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.326 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.327 DEBUG:boto:Params: {} 2022-09-02T04:50:45.328 DEBUG:boto:Token: None 2022-09-02T04:50:45.328 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ea2a1f7f3e50a59-0063118ba5-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.334 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.335 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.335 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.336 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.336 DEBUG:boto:Data: 2022-09-02T04:50:45.337 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.337 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.338 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.338 DEBUG:boto:Params: {} 2022-09-02T04:50:45.339 DEBUG:boto:Token: None 2022-09-02T04:50:45.342 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ec971c0fb731334-0063118ba5-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.351 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.355 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.356 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.358 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.359 DEBUG:boto:Method: GET 2022-09-02T04:50:45.360 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.361 DEBUG:boto:Data: 2022-09-02T04:50:45.361 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.363 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.374 DEBUG:boto:Params: {} 2022-09-02T04:50:45.375 DEBUG:boto:Token: None 2022-09-02T04:50:45.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.376 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1a0d29e24467f0f-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.396 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.397 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.398 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.399 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.400 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.401 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.401 DEBUG:boto:Method: GET 2022-09-02T04:50:45.402 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.404 DEBUG:boto:Data: 2022-09-02T04:50:45.405 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.407 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.408 DEBUG:boto:Params: {} 2022-09-02T04:50:45.409 DEBUG:boto:Token: None 2022-09-02T04:50:45.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e107af1bfc2131-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.418 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.419 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.419 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.420 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.420 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.421 DEBUG:boto:Method: GET 2022-09-02T04:50:45.422 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.422 DEBUG:boto:Data: 2022-09-02T04:50:45.423 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.424 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.424 DEBUG:boto:Params: {} 2022-09-02T04:50:45.425 DEBUG:boto:Token: None 2022-09-02T04:50:45.426 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081dd0507a3789b6a-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.433 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.434 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.435 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.436 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.437 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.447 DEBUG:boto:Method: GET 2022-09-02T04:50:45.448 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.449 DEBUG:boto:Data: 2022-09-02T04:50:45.450 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.453 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.455 DEBUG:boto:Params: {} 2022-09-02T04:50:45.456 DEBUG:boto:Token: None 2022-09-02T04:50:45.457 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e643c600bded3bf8-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.464 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-40 zones={test-zone3, test-zone2} 2022-09-02T04:50:45.467 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-40 zones={test-zone1, test-zone3} 2022-09-02T04:50:45.467 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.468 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.470 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.471 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.472 DEBUG:boto:Data: 2022-09-02T04:50:45.473 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.476 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.477 DEBUG:boto:Params: {} 2022-09-02T04:50:45.478 DEBUG:boto:Token: None 2022-09-02T04:50:45.480 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000641dbadfa73a120a-0063118ba5-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.487 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.490 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.491 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.492 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.493 DEBUG:boto:Data: 2022-09-02T04:50:45.493 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.495 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.496 DEBUG:boto:Params: {} 2022-09-02T04:50:45.498 DEBUG:boto:Token: None 2022-09-02T04:50:45.499 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e75b3d5e10ca969e-0063118ba5-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.509 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.510 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.511 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.513 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.514 DEBUG:boto:Method: GET 2022-09-02T04:50:45.516 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.518 DEBUG:boto:Data: 2022-09-02T04:50:45.518 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.520 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.521 DEBUG:boto:Params: {} 2022-09-02T04:50:45.522 DEBUG:boto:Token: None 2022-09-02T04:50:45.526 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0acffe518f6e99b-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.539 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.541 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.542 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.543 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.544 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.545 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.546 DEBUG:boto:Method: GET 2022-09-02T04:50:45.548 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.549 DEBUG:boto:Data: 2022-09-02T04:50:45.550 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.552 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.553 DEBUG:boto:Params: {} 2022-09-02T04:50:45.554 DEBUG:boto:Token: None 2022-09-02T04:50:45.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000217b6b0ad573f62c-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.565 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.567 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.568 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.569 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.571 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.572 DEBUG:boto:Method: GET 2022-09-02T04:50:45.573 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.574 DEBUG:boto:Data: 2022-09-02T04:50:45.575 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.577 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.577 DEBUG:boto:Params: {} 2022-09-02T04:50:45.579 DEBUG:boto:Token: None 2022-09-02T04:50:45.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fae9d520afab520c-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.586 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.588 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.589 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.590 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.591 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.592 DEBUG:boto:Method: GET 2022-09-02T04:50:45.593 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.594 DEBUG:boto:Data: 2022-09-02T04:50:45.596 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.597 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.598 DEBUG:boto:Params: {} 2022-09-02T04:50:45.599 DEBUG:boto:Token: None 2022-09-02T04:50:45.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dfb0ddc42aa480d-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.611 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.612 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-40 zones={test-zone1, test-zone3} 2022-09-02T04:50:45.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-40 zones={test-zone2, test-zone3} 2022-09-02T04:50:45.615 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.616 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.617 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.617 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.618 DEBUG:boto:Data: 2022-09-02T04:50:45.618 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.619 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.620 DEBUG:boto:Params: {} 2022-09-02T04:50:45.621 DEBUG:boto:Token: None 2022-09-02T04:50:45.621 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf1d3ed8b5a006f3-0063118ba5-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.626 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.627 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.627 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.628 DEBUG:boto:Path: /xljbkf-40/ 2022-09-02T04:50:45.628 DEBUG:boto:Data: 2022-09-02T04:50:45.629 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.631 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.631 DEBUG:boto:Params: {} 2022-09-02T04:50:45.632 DEBUG:boto:Token: None 2022-09-02T04:50:45.632 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/ 2022-09-02T04:50:45.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0= 2022-09-02T04:50:45.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ou/DusulXPPVQ61X0ov8byPY0=', 'Content-Length': '0'} 2022-09-02T04:50:45.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9ec307d3b195abd-0063118ba5-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.639 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.639 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.640 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.641 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.644 DEBUG:boto:Method: GET 2022-09-02T04:50:45.645 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.646 DEBUG:boto:Data: 2022-09-02T04:50:45.646 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.649 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.650 DEBUG:boto:Params: {} 2022-09-02T04:50:45.652 DEBUG:boto:Token: None 2022-09-02T04:50:45.653 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2dd89f651ca95f-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.659 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.660 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.660 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.661 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.662 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.662 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.663 DEBUG:boto:Method: GET 2022-09-02T04:50:45.663 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.664 DEBUG:boto:Data: 2022-09-02T04:50:45.665 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.666 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.667 DEBUG:boto:Params: {} 2022-09-02T04:50:45.667 DEBUG:boto:Token: None 2022-09-02T04:50:45.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118eed16b6a35fa9-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.714 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.716 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.716 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.717 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.718 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.719 DEBUG:boto:Method: GET 2022-09-02T04:50:45.721 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.723 DEBUG:boto:Data: 2022-09-02T04:50:45.727 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.728 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.729 DEBUG:boto:Params: {} 2022-09-02T04:50:45.731 DEBUG:boto:Token: None 2022-09-02T04:50:45.732 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000415a1263fb73c689-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.746 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.748 DEBUG:boto:path=/xljbkf-40/ 2022-09-02T04:50:45.748 DEBUG:boto:auth_path=/xljbkf-40/ 2022-09-02T04:50:45.749 DEBUG:boto:path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.750 DEBUG:boto:auth_path=/xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.750 DEBUG:boto:Method: GET 2022-09-02T04:50:45.751 DEBUG:boto:Path: /xljbkf-40/?versions&max-keys=999 2022-09-02T04:50:45.752 DEBUG:boto:Data: 2022-09-02T04:50:45.752 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.753 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.754 DEBUG:boto:Params: {} 2022-09-02T04:50:45.755 DEBUG:boto:Token: None 2022-09-02T04:50:45.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-40/?versions 2022-09-02T04:50:45.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI= 2022-09-02T04:50:45.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3wHWdHj5dgPB2rBEHD6A5XfJLI=', 'Content-Length': '0'} 2022-09-02T04:50:45.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000374c7b279f218bef-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.763 DEBUG:boto:b'xljbkf-40999false' 2022-09-02T04:50:45.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-40 zones={test-zone2, test-zone3} 2022-09-02T04:50:45.765 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-41 zones={test-zone2, test-zone1} 2022-09-02T04:50:45.766 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.767 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.768 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.769 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:45.770 DEBUG:boto:Data: 2022-09-02T04:50:45.771 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.773 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.773 DEBUG:boto:Params: {} 2022-09-02T04:50:45.775 DEBUG:boto:Token: None 2022-09-02T04:50:45.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/ 2022-09-02T04:50:45.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c= 2022-09-02T04:50:45.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c=', 'Content-Length': '0'} 2022-09-02T04:50:45.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000350a617a24e254f7-0063118ba5-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.783 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.784 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.785 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.786 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:45.787 DEBUG:boto:Data: 2022-09-02T04:50:45.788 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.789 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.789 DEBUG:boto:Params: {} 2022-09-02T04:50:45.791 DEBUG:boto:Token: None 2022-09-02T04:50:45.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/ 2022-09-02T04:50:45.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c= 2022-09-02T04:50:45.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c=', 'Content-Length': '0'} 2022-09-02T04:50:45.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5bef10e3e8ee14e-0063118ba5-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.800 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.801 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.801 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.802 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.802 DEBUG:boto:Method: GET 2022-09-02T04:50:45.803 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.803 DEBUG:boto:Data: 2022-09-02T04:50:45.807 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.808 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.809 DEBUG:boto:Params: {} 2022-09-02T04:50:45.809 DEBUG:boto:Token: None 2022-09-02T04:50:45.810 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4dbeb0070d68a0f-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.858 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.860 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.861 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.862 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.863 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.864 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.865 DEBUG:boto:Method: GET 2022-09-02T04:50:45.867 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.868 DEBUG:boto:Data: 2022-09-02T04:50:45.869 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.871 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.872 DEBUG:boto:Params: {} 2022-09-02T04:50:45.874 DEBUG:boto:Token: None 2022-09-02T04:50:45.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004814ee068bdb71e9-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.882 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.883 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.884 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.884 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.885 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.885 DEBUG:boto:Method: GET 2022-09-02T04:50:45.886 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.887 DEBUG:boto:Data: 2022-09-02T04:50:45.887 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:45.889 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.889 DEBUG:boto:Params: {} 2022-09-02T04:50:45.890 DEBUG:boto:Token: None 2022-09-02T04:50:45.891 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea12aee13924bc3-0063118ba5-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.895 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.896 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.897 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.897 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.898 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.899 DEBUG:boto:Method: GET 2022-09-02T04:50:45.899 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.900 DEBUG:boto:Data: 2022-09-02T04:50:45.900 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.902 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.902 DEBUG:boto:Params: {} 2022-09-02T04:50:45.903 DEBUG:boto:Token: None 2022-09-02T04:50:45.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000be47cdbdbf0f971-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.908 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.909 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-41 zones={test-zone2, test-zone1} 2022-09-02T04:50:45.909 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-41 zones={test-zone3, test-zone1} 2022-09-02T04:50:45.910 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.910 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.911 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.912 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:45.912 DEBUG:boto:Data: 2022-09-02T04:50:45.913 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.914 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.915 DEBUG:boto:Params: {} 2022-09-02T04:50:45.916 DEBUG:boto:Token: None 2022-09-02T04:50:45.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/ 2022-09-02T04:50:45.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c= 2022-09-02T04:50:45.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c=', 'Content-Length': '0'} 2022-09-02T04:50:45.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007168363c1be21de9-0063118ba5-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.922 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.922 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.923 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.924 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:45.924 DEBUG:boto:Data: 2022-09-02T04:50:45.925 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.926 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.927 DEBUG:boto:Params: {} 2022-09-02T04:50:45.927 DEBUG:boto:Token: None 2022-09-02T04:50:45.928 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/ 2022-09-02T04:50:45.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c= 2022-09-02T04:50:45.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c=', 'Content-Length': '0'} 2022-09-02T04:50:45.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b921d0a2518c418f-0063118ba5-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:45.934 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.934 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.935 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.935 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.936 DEBUG:boto:Method: GET 2022-09-02T04:50:45.936 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.937 DEBUG:boto:Data: 2022-09-02T04:50:45.937 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.939 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.939 DEBUG:boto:Params: {} 2022-09-02T04:50:45.940 DEBUG:boto:Token: None 2022-09-02T04:50:45.941 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000476f9910c1970023-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.946 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:45.948 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.948 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.949 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.949 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.950 DEBUG:boto:Method: GET 2022-09-02T04:50:45.951 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.952 DEBUG:boto:Data: 2022-09-02T04:50:45.952 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.953 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.954 DEBUG:boto:Params: {} 2022-09-02T04:50:45.957 DEBUG:boto:Token: None 2022-09-02T04:50:45.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e66661267d528718-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.964 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.964 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.965 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.966 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.966 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.967 DEBUG:boto:Method: GET 2022-09-02T04:50:45.967 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.968 DEBUG:boto:Data: 2022-09-02T04:50:45.968 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:45.970 DEBUG:boto:Port: 8001 2022-09-02T04:50:45.970 DEBUG:boto:Params: {} 2022-09-02T04:50:45.971 DEBUG:boto:Token: None 2022-09-02T04:50:45.971 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099483a15c49c7521-0063118ba5-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.976 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.977 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.977 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.978 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.978 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.979 DEBUG:boto:Method: GET 2022-09-02T04:50:45.979 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:45.980 DEBUG:boto:Data: 2022-09-02T04:50:45.980 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.982 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.982 DEBUG:boto:Params: {} 2022-09-02T04:50:45.983 DEBUG:boto:Token: None 2022-09-02T04:50:45.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/?versions 2022-09-02T04:50:45.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM= 2022-09-02T04:50:45.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:942hmUW2jddlTXdB9rdrmyZXgFM=', 'Content-Length': '0'} 2022-09-02T04:50:45.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fee83ce486734c8-0063118ba5-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:45.988 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:45.989 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-41 zones={test-zone3, test-zone1} 2022-09-02T04:50:45.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-41 zones={test-zone1, test-zone2} 2022-09-02T04:50:45.990 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:45.990 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:45.991 DEBUG:boto:Method: HEAD 2022-09-02T04:50:45.992 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:45.992 DEBUG:boto:Data: 2022-09-02T04:50:45.993 DEBUG:boto:Headers: {} 2022-09-02T04:50:45.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:45.994 DEBUG:boto:Port: 8000 2022-09-02T04:50:45.994 DEBUG:boto:Params: {} 2022-09-02T04:50:45.995 DEBUG:boto:Token: None 2022-09-02T04:50:45.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:45 GMT /xljbkf-41/ 2022-09-02T04:50:45.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c= 2022-09-02T04:50:45.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h3wupFvnl4a6742KqcOO7i1x97c=', 'Content-Length': '0'} 2022-09-02T04:50:46.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ec4f404adce8040-0063118ba5-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.000 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.001 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.002 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.002 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.003 DEBUG:boto:Data: 2022-09-02T04:50:46.003 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.005 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.005 DEBUG:boto:Params: {} 2022-09-02T04:50:46.006 DEBUG:boto:Token: None 2022-09-02T04:50:46.006 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b3195a91beb0afb-0063118ba6-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.011 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.011 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.012 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.013 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.013 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.014 DEBUG:boto:Method: GET 2022-09-02T04:50:46.014 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.015 DEBUG:boto:Data: 2022-09-02T04:50:46.015 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.017 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.017 DEBUG:boto:Params: {} 2022-09-02T04:50:46.018 DEBUG:boto:Token: None 2022-09-02T04:50:46.019 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2a7ee59738d17d-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.071 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.072 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.073 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.073 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.074 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.074 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.075 DEBUG:boto:Method: GET 2022-09-02T04:50:46.075 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.076 DEBUG:boto:Data: 2022-09-02T04:50:46.077 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.078 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.078 DEBUG:boto:Params: {} 2022-09-02T04:50:46.079 DEBUG:boto:Token: None 2022-09-02T04:50:46.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e88eebb85ed8ec-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.084 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.085 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.086 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.086 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.087 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.088 DEBUG:boto:Method: GET 2022-09-02T04:50:46.088 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.089 DEBUG:boto:Data: 2022-09-02T04:50:46.089 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.090 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.091 DEBUG:boto:Params: {} 2022-09-02T04:50:46.091 DEBUG:boto:Token: None 2022-09-02T04:50:46.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087edb4aa6e2bcfc1-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.097 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.097 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.098 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.098 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.099 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.100 DEBUG:boto:Method: GET 2022-09-02T04:50:46.100 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.101 DEBUG:boto:Data: 2022-09-02T04:50:46.101 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.102 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.103 DEBUG:boto:Params: {} 2022-09-02T04:50:46.103 DEBUG:boto:Token: None 2022-09-02T04:50:46.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e0b60a601bf4273-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.112 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.113 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-41 zones={test-zone1, test-zone2} 2022-09-02T04:50:46.113 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-41 zones={test-zone3, test-zone2} 2022-09-02T04:50:46.114 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.114 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.115 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.116 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.116 DEBUG:boto:Data: 2022-09-02T04:50:46.117 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.118 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.119 DEBUG:boto:Params: {} 2022-09-02T04:50:46.119 DEBUG:boto:Token: None 2022-09-02T04:50:46.120 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f7a3341018d6550-0063118ba6-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.131 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.132 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.132 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.133 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.133 DEBUG:boto:Data: 2022-09-02T04:50:46.134 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.135 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.135 DEBUG:boto:Params: {} 2022-09-02T04:50:46.136 DEBUG:boto:Token: None 2022-09-02T04:50:46.137 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093372806c76f8992-0063118ba6-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.145 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.146 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.146 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.148 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.149 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.150 DEBUG:boto:Method: GET 2022-09-02T04:50:46.151 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.151 DEBUG:boto:Data: 2022-09-02T04:50:46.152 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.154 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.156 DEBUG:boto:Params: {} 2022-09-02T04:50:46.157 DEBUG:boto:Token: None 2022-09-02T04:50:46.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000800b15fe2f77a44d-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.167 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.172 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.173 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.174 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.174 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.175 DEBUG:boto:Method: GET 2022-09-02T04:50:46.175 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.176 DEBUG:boto:Data: 2022-09-02T04:50:46.176 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.177 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.178 DEBUG:boto:Params: {} 2022-09-02T04:50:46.178 DEBUG:boto:Token: None 2022-09-02T04:50:46.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2468278dd290a27-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.188 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.191 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.192 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.193 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.194 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.195 DEBUG:boto:Method: GET 2022-09-02T04:50:46.197 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.197 DEBUG:boto:Data: 2022-09-02T04:50:46.198 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.199 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.199 DEBUG:boto:Params: {} 2022-09-02T04:50:46.200 DEBUG:boto:Token: None 2022-09-02T04:50:46.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f17a8ab78614d0-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.207 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.208 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.208 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.209 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.209 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.210 DEBUG:boto:Method: GET 2022-09-02T04:50:46.210 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.211 DEBUG:boto:Data: 2022-09-02T04:50:46.212 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.214 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.214 DEBUG:boto:Params: {} 2022-09-02T04:50:46.215 DEBUG:boto:Token: None 2022-09-02T04:50:46.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6fd2bc5b2a11e4-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.221 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.223 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-41 zones={test-zone3, test-zone2} 2022-09-02T04:50:46.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-41 zones={test-zone1, test-zone3} 2022-09-02T04:50:46.224 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.225 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.225 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.226 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.226 DEBUG:boto:Data: 2022-09-02T04:50:46.227 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.229 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.229 DEBUG:boto:Params: {} 2022-09-02T04:50:46.230 DEBUG:boto:Token: None 2022-09-02T04:50:46.231 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c05f1da047aac60f-0063118ba6-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.236 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.237 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.238 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.239 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.239 DEBUG:boto:Data: 2022-09-02T04:50:46.240 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.241 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.242 DEBUG:boto:Params: {} 2022-09-02T04:50:46.242 DEBUG:boto:Token: None 2022-09-02T04:50:46.243 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da3f8c311b1f4b2a-0063118ba6-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.247 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.248 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.248 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.249 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.250 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.250 DEBUG:boto:Method: GET 2022-09-02T04:50:46.251 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.252 DEBUG:boto:Data: 2022-09-02T04:50:46.252 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.254 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.254 DEBUG:boto:Params: {} 2022-09-02T04:50:46.255 DEBUG:boto:Token: None 2022-09-02T04:50:46.256 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bda14114589a80-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.264 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.265 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.265 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.266 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.267 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.267 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.268 DEBUG:boto:Method: GET 2022-09-02T04:50:46.268 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.269 DEBUG:boto:Data: 2022-09-02T04:50:46.269 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.271 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.271 DEBUG:boto:Params: {} 2022-09-02T04:50:46.272 DEBUG:boto:Token: None 2022-09-02T04:50:46.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13f4f0181be541d-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.326 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.327 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.328 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.328 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.329 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.329 DEBUG:boto:Method: GET 2022-09-02T04:50:46.330 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.331 DEBUG:boto:Data: 2022-09-02T04:50:46.331 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.332 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.333 DEBUG:boto:Params: {} 2022-09-02T04:50:46.334 DEBUG:boto:Token: None 2022-09-02T04:50:46.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aefc53523395cc61-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.341 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.342 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.343 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.344 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.345 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.346 DEBUG:boto:Method: GET 2022-09-02T04:50:46.347 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.348 DEBUG:boto:Data: 2022-09-02T04:50:46.350 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.351 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.351 DEBUG:boto:Params: {} 2022-09-02T04:50:46.352 DEBUG:boto:Token: None 2022-09-02T04:50:46.352 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002efee3610409108a-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.357 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.358 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-41 zones={test-zone1, test-zone3} 2022-09-02T04:50:46.358 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-41 zones={test-zone2, test-zone3} 2022-09-02T04:50:46.359 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.359 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.360 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.361 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.361 DEBUG:boto:Data: 2022-09-02T04:50:46.362 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.363 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.363 DEBUG:boto:Params: {} 2022-09-02T04:50:46.364 DEBUG:boto:Token: None 2022-09-02T04:50:46.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000248397c464b3d367-0063118ba6-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.370 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.371 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.372 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.373 DEBUG:boto:Path: /xljbkf-41/ 2022-09-02T04:50:46.374 DEBUG:boto:Data: 2022-09-02T04:50:46.375 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.376 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.377 DEBUG:boto:Params: {} 2022-09-02T04:50:46.379 DEBUG:boto:Token: None 2022-09-02T04:50:46.380 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/ 2022-09-02T04:50:46.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M= 2022-09-02T04:50:46.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gSfhWI0ajKlV2F/fD8fpSNMn16M=', 'Content-Length': '0'} 2022-09-02T04:50:46.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e36e893bce30d91e-0063118ba6-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.387 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.388 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.388 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.389 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.389 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.390 DEBUG:boto:Method: GET 2022-09-02T04:50:46.390 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.391 DEBUG:boto:Data: 2022-09-02T04:50:46.392 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.392 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.393 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.393 DEBUG:boto:Params: {} 2022-09-02T04:50:46.394 DEBUG:boto:Token: None 2022-09-02T04:50:46.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e6998d2b3c32e7-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.401 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.402 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.402 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.403 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.404 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.404 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.405 DEBUG:boto:Method: GET 2022-09-02T04:50:46.405 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.406 DEBUG:boto:Data: 2022-09-02T04:50:46.407 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.408 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.408 DEBUG:boto:Params: {} 2022-09-02T04:50:46.409 DEBUG:boto:Token: None 2022-09-02T04:50:46.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000607fc74c47ed2b31-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.415 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.416 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.416 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.417 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.417 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.418 DEBUG:boto:Method: GET 2022-09-02T04:50:46.418 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.419 DEBUG:boto:Data: 2022-09-02T04:50:46.419 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.420 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.420 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.421 DEBUG:boto:Params: {} 2022-09-02T04:50:46.421 DEBUG:boto:Token: None 2022-09-02T04:50:46.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000150b54cae1aabaea-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.428 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.429 DEBUG:boto:path=/xljbkf-41/ 2022-09-02T04:50:46.430 DEBUG:boto:auth_path=/xljbkf-41/ 2022-09-02T04:50:46.431 DEBUG:boto:path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.432 DEBUG:boto:auth_path=/xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.433 DEBUG:boto:Method: GET 2022-09-02T04:50:46.434 DEBUG:boto:Path: /xljbkf-41/?versions&max-keys=999 2022-09-02T04:50:46.435 DEBUG:boto:Data: 2022-09-02T04:50:46.435 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.437 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.439 DEBUG:boto:Params: {} 2022-09-02T04:50:46.440 DEBUG:boto:Token: None 2022-09-02T04:50:46.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-41/?versions 2022-09-02T04:50:46.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E= 2022-09-02T04:50:46.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gbrQ3FiTzQssJHox2JaiocFOl0E=', 'Content-Length': '0'} 2022-09-02T04:50:46.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009618d8fd1a3a95a2-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.490 DEBUG:boto:b'xljbkf-41999false' 2022-09-02T04:50:46.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-41 zones={test-zone2, test-zone3} 2022-09-02T04:50:46.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-42 zones={test-zone2, test-zone1} 2022-09-02T04:50:46.493 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.493 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.494 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.495 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.495 DEBUG:boto:Data: 2022-09-02T04:50:46.496 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.497 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.497 DEBUG:boto:Params: {} 2022-09-02T04:50:46.498 DEBUG:boto:Token: None 2022-09-02T04:50:46.499 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000535a9fdb7800e40b-0063118ba6-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.506 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.507 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.508 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.509 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.510 DEBUG:boto:Data: 2022-09-02T04:50:46.511 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.513 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.513 DEBUG:boto:Params: {} 2022-09-02T04:50:46.514 DEBUG:boto:Token: None 2022-09-02T04:50:46.515 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef30fbaeb4f6d36c-0063118ba6-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.523 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.524 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.525 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.527 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.528 DEBUG:boto:Method: GET 2022-09-02T04:50:46.529 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.530 DEBUG:boto:Data: 2022-09-02T04:50:46.531 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.532 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.532 DEBUG:boto:Params: {} 2022-09-02T04:50:46.533 DEBUG:boto:Token: None 2022-09-02T04:50:46.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d20f87aaaaed206-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.541 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.542 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.542 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.543 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.544 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.544 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.545 DEBUG:boto:Method: GET 2022-09-02T04:50:46.546 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.547 DEBUG:boto:Data: 2022-09-02T04:50:46.548 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.550 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.551 DEBUG:boto:Params: {} 2022-09-02T04:50:46.552 DEBUG:boto:Token: None 2022-09-02T04:50:46.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8173638f77e7d9-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.562 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.563 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.564 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.565 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.566 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.567 DEBUG:boto:Method: GET 2022-09-02T04:50:46.568 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.569 DEBUG:boto:Data: 2022-09-02T04:50:46.570 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.572 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.573 DEBUG:boto:Params: {} 2022-09-02T04:50:46.574 DEBUG:boto:Token: None 2022-09-02T04:50:46.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040cc4de0c6e9ad74-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.626 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.628 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.628 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.629 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.629 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.630 DEBUG:boto:Method: GET 2022-09-02T04:50:46.630 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.631 DEBUG:boto:Data: 2022-09-02T04:50:46.632 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.634 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.635 DEBUG:boto:Params: {} 2022-09-02T04:50:46.636 DEBUG:boto:Token: None 2022-09-02T04:50:46.637 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe1e8c6516e5303-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.643 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.644 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-42 zones={test-zone2, test-zone1} 2022-09-02T04:50:46.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-42 zones={test-zone3, test-zone1} 2022-09-02T04:50:46.645 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.646 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.647 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.648 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.650 DEBUG:boto:Data: 2022-09-02T04:50:46.651 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.653 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.654 DEBUG:boto:Params: {} 2022-09-02T04:50:46.655 DEBUG:boto:Token: None 2022-09-02T04:50:46.656 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9a88856687e7821-0063118ba6-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.668 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.669 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.670 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.675 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.676 DEBUG:boto:Data: 2022-09-02T04:50:46.676 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.677 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.678 DEBUG:boto:Params: {} 2022-09-02T04:50:46.679 DEBUG:boto:Token: None 2022-09-02T04:50:46.679 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.684 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b8da33bf17b4e71-0063118ba6-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.688 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.689 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.689 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.690 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.690 DEBUG:boto:Method: GET 2022-09-02T04:50:46.691 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.691 DEBUG:boto:Data: 2022-09-02T04:50:46.692 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.693 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.696 DEBUG:boto:Params: {} 2022-09-02T04:50:46.696 DEBUG:boto:Token: None 2022-09-02T04:50:46.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c9f241f957de4cb-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.703 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.704 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.705 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.706 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.707 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.708 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.709 DEBUG:boto:Method: GET 2022-09-02T04:50:46.710 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.711 DEBUG:boto:Data: 2022-09-02T04:50:46.712 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.713 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.715 DEBUG:boto:Params: {} 2022-09-02T04:50:46.716 DEBUG:boto:Token: None 2022-09-02T04:50:46.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc68f0ab69960f7-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.767 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.768 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.768 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.769 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.769 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.770 DEBUG:boto:Method: GET 2022-09-02T04:50:46.771 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.771 DEBUG:boto:Data: 2022-09-02T04:50:46.772 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.773 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.773 DEBUG:boto:Params: {} 2022-09-02T04:50:46.774 DEBUG:boto:Token: None 2022-09-02T04:50:46.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f2455f6587fc769-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.779 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.780 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.780 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.781 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.781 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.782 DEBUG:boto:Method: GET 2022-09-02T04:50:46.783 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.783 DEBUG:boto:Data: 2022-09-02T04:50:46.784 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.785 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.785 DEBUG:boto:Params: {} 2022-09-02T04:50:46.786 DEBUG:boto:Token: None 2022-09-02T04:50:46.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b09a85601f0714e-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.794 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.795 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-42 zones={test-zone3, test-zone1} 2022-09-02T04:50:46.795 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-42 zones={test-zone1, test-zone2} 2022-09-02T04:50:46.796 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.796 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.797 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.797 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.798 DEBUG:boto:Data: 2022-09-02T04:50:46.798 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.799 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.800 DEBUG:boto:Params: {} 2022-09-02T04:50:46.800 DEBUG:boto:Token: None 2022-09-02T04:50:46.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a1f69fc2053421f-0063118ba6-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.807 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.807 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.808 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.808 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.809 DEBUG:boto:Data: 2022-09-02T04:50:46.809 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.810 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.811 DEBUG:boto:Params: {} 2022-09-02T04:50:46.811 DEBUG:boto:Token: None 2022-09-02T04:50:46.812 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009748d0f9fdfebcfc-0063118ba6-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.817 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.818 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.818 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.819 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.819 DEBUG:boto:Method: GET 2022-09-02T04:50:46.820 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.820 DEBUG:boto:Data: 2022-09-02T04:50:46.821 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.822 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.823 DEBUG:boto:Params: {} 2022-09-02T04:50:46.823 DEBUG:boto:Token: None 2022-09-02T04:50:46.824 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18c96126371ee38-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.870 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.872 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.873 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.873 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.875 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.875 DEBUG:boto:Method: GET 2022-09-02T04:50:46.876 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.877 DEBUG:boto:Data: 2022-09-02T04:50:46.878 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.880 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.880 DEBUG:boto:Params: {} 2022-09-02T04:50:46.881 DEBUG:boto:Token: None 2022-09-02T04:50:46.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2868944f35ffe0-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.889 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.890 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.891 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.892 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.892 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.893 DEBUG:boto:Method: GET 2022-09-02T04:50:46.894 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.894 DEBUG:boto:Data: 2022-09-02T04:50:46.895 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:46.896 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.896 DEBUG:boto:Params: {} 2022-09-02T04:50:46.897 DEBUG:boto:Token: None 2022-09-02T04:50:46.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf2a66b8f4d52dc0-0063118ba6-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.902 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.903 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.904 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.904 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.905 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.906 DEBUG:boto:Method: GET 2022-09-02T04:50:46.906 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.907 DEBUG:boto:Data: 2022-09-02T04:50:46.908 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.909 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.910 DEBUG:boto:Params: {} 2022-09-02T04:50:46.911 DEBUG:boto:Token: None 2022-09-02T04:50:46.913 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb294f281d8efcc-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.918 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.919 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-42 zones={test-zone1, test-zone2} 2022-09-02T04:50:46.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-42 zones={test-zone3, test-zone2} 2022-09-02T04:50:46.920 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.921 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.921 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.922 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.922 DEBUG:boto:Data: 2022-09-02T04:50:46.923 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.924 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.924 DEBUG:boto:Params: {} 2022-09-02T04:50:46.925 DEBUG:boto:Token: None 2022-09-02T04:50:46.925 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006563ca545f632885-0063118ba6-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.933 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.933 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.935 DEBUG:boto:Method: HEAD 2022-09-02T04:50:46.935 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:46.936 DEBUG:boto:Data: 2022-09-02T04:50:46.937 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.938 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.939 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.939 DEBUG:boto:Params: {} 2022-09-02T04:50:46.940 DEBUG:boto:Token: None 2022-09-02T04:50:46.941 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/ 2022-09-02T04:50:46.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8= 2022-09-02T04:50:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XloUDTgJrmSbf4fhXlpvz3DfTd8=', 'Content-Length': '0'} 2022-09-02T04:50:46.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000355491a630fb07a1-0063118ba6-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:46.948 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.949 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.950 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.950 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.951 DEBUG:boto:Method: GET 2022-09-02T04:50:46.951 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.952 DEBUG:boto:Data: 2022-09-02T04:50:46.952 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.953 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.954 DEBUG:boto:Params: {} 2022-09-02T04:50:46.954 DEBUG:boto:Token: None 2022-09-02T04:50:46.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b0bdb3ee2d078c-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.961 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:46.963 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.964 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.965 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.966 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.967 DEBUG:boto:Method: GET 2022-09-02T04:50:46.968 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.969 DEBUG:boto:Data: 2022-09-02T04:50:46.970 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:46.972 DEBUG:boto:Port: 8000 2022-09-02T04:50:46.973 DEBUG:boto:Params: {} 2022-09-02T04:50:46.974 DEBUG:boto:Token: None 2022-09-02T04:50:46.975 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13853a86d19d91e-0063118ba6-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:46.979 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:46.980 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:46.981 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:46.981 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.982 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.982 DEBUG:boto:Method: GET 2022-09-02T04:50:46.983 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:46.983 DEBUG:boto:Data: 2022-09-02T04:50:46.984 DEBUG:boto:Headers: {} 2022-09-02T04:50:46.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:46.985 DEBUG:boto:Port: 8001 2022-09-02T04:50:46.985 DEBUG:boto:Params: {} 2022-09-02T04:50:46.986 DEBUG:boto:Token: None 2022-09-02T04:50:46.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:46 GMT /xljbkf-42/?versions 2022-09-02T04:50:46.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI= 2022-09-02T04:50:46.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XYEAcUGSBvjtEpU6wcPsosaClDI=', 'Content-Length': '0'} 2022-09-02T04:50:46.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0649a75d821782c-0063118ba6-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.043 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.048 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.050 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.051 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.053 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.058 DEBUG:boto:Method: GET 2022-09-02T04:50:47.059 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.060 DEBUG:boto:Data: 2022-09-02T04:50:47.061 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.064 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.065 DEBUG:boto:Params: {} 2022-09-02T04:50:47.066 DEBUG:boto:Token: None 2022-09-02T04:50:47.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0562ca191c24d3f-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.093 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-42 zones={test-zone3, test-zone2} 2022-09-02T04:50:47.095 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-42 zones={test-zone1, test-zone3} 2022-09-02T04:50:47.097 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.098 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.099 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.100 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:47.101 DEBUG:boto:Data: 2022-09-02T04:50:47.102 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.104 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.104 DEBUG:boto:Params: {} 2022-09-02T04:50:47.105 DEBUG:boto:Token: None 2022-09-02T04:50:47.105 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/ 2022-09-02T04:50:47.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo= 2022-09-02T04:50:47.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo=', 'Content-Length': '0'} 2022-09-02T04:50:47.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c91c7c9bb602b49-0063118ba7-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.110 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.111 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.112 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.112 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:47.113 DEBUG:boto:Data: 2022-09-02T04:50:47.113 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.116 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.118 DEBUG:boto:Params: {} 2022-09-02T04:50:47.119 DEBUG:boto:Token: None 2022-09-02T04:50:47.120 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/ 2022-09-02T04:50:47.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo= 2022-09-02T04:50:47.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo=', 'Content-Length': '0'} 2022-09-02T04:50:47.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000749daf6e665f986-0063118ba7-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.128 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.129 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.130 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.131 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.133 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.134 DEBUG:boto:Method: GET 2022-09-02T04:50:47.135 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.136 DEBUG:boto:Data: 2022-09-02T04:50:47.137 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.138 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.139 DEBUG:boto:Params: {} 2022-09-02T04:50:47.140 DEBUG:boto:Token: None 2022-09-02T04:50:47.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180dfe651d399284-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.151 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.153 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.154 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.154 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.155 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.156 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.156 DEBUG:boto:Method: GET 2022-09-02T04:50:47.157 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.158 DEBUG:boto:Data: 2022-09-02T04:50:47.159 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.161 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.161 DEBUG:boto:Params: {} 2022-09-02T04:50:47.162 DEBUG:boto:Token: None 2022-09-02T04:50:47.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.164 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa1018d16594c26-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.177 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.179 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.180 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.180 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.181 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.182 DEBUG:boto:Method: GET 2022-09-02T04:50:47.183 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.184 DEBUG:boto:Data: 2022-09-02T04:50:47.185 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.187 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.188 DEBUG:boto:Params: {} 2022-09-02T04:50:47.189 DEBUG:boto:Token: None 2022-09-02T04:50:47.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc33f848ff4edc4-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.197 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.199 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.201 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.201 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.202 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.203 DEBUG:boto:Method: GET 2022-09-02T04:50:47.204 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.205 DEBUG:boto:Data: 2022-09-02T04:50:47.206 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.208 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.209 DEBUG:boto:Params: {} 2022-09-02T04:50:47.210 DEBUG:boto:Token: None 2022-09-02T04:50:47.211 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cc92307df25ea2-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.218 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.219 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-42 zones={test-zone1, test-zone3} 2022-09-02T04:50:47.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-42 zones={test-zone2, test-zone3} 2022-09-02T04:50:47.221 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.222 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.223 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.224 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:47.225 DEBUG:boto:Data: 2022-09-02T04:50:47.226 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.228 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.228 DEBUG:boto:Params: {} 2022-09-02T04:50:47.229 DEBUG:boto:Token: None 2022-09-02T04:50:47.230 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/ 2022-09-02T04:50:47.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo= 2022-09-02T04:50:47.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo=', 'Content-Length': '0'} 2022-09-02T04:50:47.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f9e73c1f90a80a1-0063118ba7-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.235 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.236 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.237 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.238 DEBUG:boto:Path: /xljbkf-42/ 2022-09-02T04:50:47.238 DEBUG:boto:Data: 2022-09-02T04:50:47.239 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.241 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.242 DEBUG:boto:Params: {} 2022-09-02T04:50:47.244 DEBUG:boto:Token: None 2022-09-02T04:50:47.245 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/ 2022-09-02T04:50:47.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo= 2022-09-02T04:50:47.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UY7pPa/aq9twJ9zWYLjrr8ueuGo=', 'Content-Length': '0'} 2022-09-02T04:50:47.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b63c55ab0d4a0c8b-0063118ba7-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.255 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.256 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.256 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.257 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.257 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.258 DEBUG:boto:Method: GET 2022-09-02T04:50:47.258 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.259 DEBUG:boto:Data: 2022-09-02T04:50:47.259 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.261 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.262 DEBUG:boto:Params: {} 2022-09-02T04:50:47.262 DEBUG:boto:Token: None 2022-09-02T04:50:47.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c29113627af4262-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.268 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.269 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.270 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.270 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.271 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.271 DEBUG:boto:Method: GET 2022-09-02T04:50:47.272 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.272 DEBUG:boto:Data: 2022-09-02T04:50:47.273 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.275 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.276 DEBUG:boto:Params: {} 2022-09-02T04:50:47.276 DEBUG:boto:Token: None 2022-09-02T04:50:47.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea3052336c0bdfbd-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.287 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.289 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.294 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.294 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.295 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.295 DEBUG:boto:Method: GET 2022-09-02T04:50:47.296 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.296 DEBUG:boto:Data: 2022-09-02T04:50:47.297 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.297 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.298 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.298 DEBUG:boto:Params: {} 2022-09-02T04:50:47.299 DEBUG:boto:Token: None 2022-09-02T04:50:47.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.300 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1472f12cbad6996-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.306 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.307 DEBUG:boto:path=/xljbkf-42/ 2022-09-02T04:50:47.308 DEBUG:boto:auth_path=/xljbkf-42/ 2022-09-02T04:50:47.310 DEBUG:boto:path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.311 DEBUG:boto:auth_path=/xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.312 DEBUG:boto:Method: GET 2022-09-02T04:50:47.313 DEBUG:boto:Path: /xljbkf-42/?versions&max-keys=999 2022-09-02T04:50:47.314 DEBUG:boto:Data: 2022-09-02T04:50:47.314 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.316 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.317 DEBUG:boto:Params: {} 2022-09-02T04:50:47.318 DEBUG:boto:Token: None 2022-09-02T04:50:47.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-42/?versions 2022-09-02T04:50:47.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw= 2022-09-02T04:50:47.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/cDFLHAaKLMtPXg6WlEBZKRyyw=', 'Content-Length': '0'} 2022-09-02T04:50:47.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda7addb207263ad-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.326 DEBUG:boto:b'xljbkf-42999false' 2022-09-02T04:50:47.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-42 zones={test-zone2, test-zone3} 2022-09-02T04:50:47.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-43 zones={test-zone2, test-zone1} 2022-09-02T04:50:47.329 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.331 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.331 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.332 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.332 DEBUG:boto:Data: 2022-09-02T04:50:47.333 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.334 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.335 DEBUG:boto:Params: {} 2022-09-02T04:50:47.335 DEBUG:boto:Token: None 2022-09-02T04:50:47.336 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028ddbbdc473a8ece-0063118ba7-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.341 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.342 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.342 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.343 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.344 DEBUG:boto:Data: 2022-09-02T04:50:47.345 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.348 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.349 DEBUG:boto:Params: {} 2022-09-02T04:50:47.350 DEBUG:boto:Token: None 2022-09-02T04:50:47.351 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018f053603022d6a0-0063118ba7-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.359 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.360 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.360 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.361 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.362 DEBUG:boto:Method: GET 2022-09-02T04:50:47.363 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.364 DEBUG:boto:Data: 2022-09-02T04:50:47.365 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.368 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.369 DEBUG:boto:Params: {} 2022-09-02T04:50:47.370 DEBUG:boto:Token: None 2022-09-02T04:50:47.371 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d07ff550bf88c2c5-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.422 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.424 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.425 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.426 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.427 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.428 DEBUG:boto:Method: GET 2022-09-02T04:50:47.429 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.429 DEBUG:boto:Data: 2022-09-02T04:50:47.430 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.432 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.433 DEBUG:boto:Params: {} 2022-09-02T04:50:47.434 DEBUG:boto:Token: None 2022-09-02T04:50:47.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55dd843f2cf1d79-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.440 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.441 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.441 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.442 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.442 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.443 DEBUG:boto:Method: GET 2022-09-02T04:50:47.443 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.444 DEBUG:boto:Data: 2022-09-02T04:50:47.444 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.446 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.446 DEBUG:boto:Params: {} 2022-09-02T04:50:47.448 DEBUG:boto:Token: None 2022-09-02T04:50:47.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0bdea29a5e2dac-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.455 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.456 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.457 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.458 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.458 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.459 DEBUG:boto:Method: GET 2022-09-02T04:50:47.459 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.460 DEBUG:boto:Data: 2022-09-02T04:50:47.460 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.462 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.463 DEBUG:boto:Params: {} 2022-09-02T04:50:47.464 DEBUG:boto:Token: None 2022-09-02T04:50:47.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4b22758a5cd568-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.518 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-43 zones={test-zone2, test-zone1} 2022-09-02T04:50:47.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-43 zones={test-zone3, test-zone1} 2022-09-02T04:50:47.520 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.522 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.523 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.524 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.525 DEBUG:boto:Data: 2022-09-02T04:50:47.526 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.527 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.527 DEBUG:boto:Params: {} 2022-09-02T04:50:47.528 DEBUG:boto:Token: None 2022-09-02T04:50:47.528 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098310a761689cfc2-0063118ba7-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.533 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.533 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.534 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.534 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.535 DEBUG:boto:Data: 2022-09-02T04:50:47.535 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.536 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.537 DEBUG:boto:Params: {} 2022-09-02T04:50:47.537 DEBUG:boto:Token: None 2022-09-02T04:50:47.538 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c58d4a18e7c54fb4-0063118ba7-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.542 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.543 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.543 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.544 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.544 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.545 DEBUG:boto:Method: GET 2022-09-02T04:50:47.545 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.546 DEBUG:boto:Data: 2022-09-02T04:50:47.546 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.547 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.548 DEBUG:boto:Params: {} 2022-09-02T04:50:47.548 DEBUG:boto:Token: None 2022-09-02T04:50:47.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908ba4813d8df018-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.554 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.555 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.556 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.557 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.558 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.559 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.559 DEBUG:boto:Method: GET 2022-09-02T04:50:47.560 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.561 DEBUG:boto:Data: 2022-09-02T04:50:47.561 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.563 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.563 DEBUG:boto:Params: {} 2022-09-02T04:50:47.564 DEBUG:boto:Token: None 2022-09-02T04:50:47.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7cc268d18c01579-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.569 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.570 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.571 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.571 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.572 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.572 DEBUG:boto:Method: GET 2022-09-02T04:50:47.573 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.573 DEBUG:boto:Data: 2022-09-02T04:50:47.574 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.576 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.577 DEBUG:boto:Params: {} 2022-09-02T04:50:47.577 DEBUG:boto:Token: None 2022-09-02T04:50:47.578 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069c32ca8f2471894-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.583 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.584 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.585 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.585 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.586 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.586 DEBUG:boto:Method: GET 2022-09-02T04:50:47.587 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.587 DEBUG:boto:Data: 2022-09-02T04:50:47.588 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.589 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.590 DEBUG:boto:Params: {} 2022-09-02T04:50:47.592 DEBUG:boto:Token: None 2022-09-02T04:50:47.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db61c40a71b890b-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.600 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-43 zones={test-zone3, test-zone1} 2022-09-02T04:50:47.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-43 zones={test-zone1, test-zone2} 2022-09-02T04:50:47.604 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.604 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.605 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.605 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.606 DEBUG:boto:Data: 2022-09-02T04:50:47.607 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.609 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.610 DEBUG:boto:Params: {} 2022-09-02T04:50:47.611 DEBUG:boto:Token: None 2022-09-02T04:50:47.612 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065581c3d61ed10d3-0063118ba7-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.619 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.620 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.620 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.621 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.621 DEBUG:boto:Data: 2022-09-02T04:50:47.622 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.623 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.624 DEBUG:boto:Params: {} 2022-09-02T04:50:47.624 DEBUG:boto:Token: None 2022-09-02T04:50:47.625 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8e96dfdc6c53e39-0063118ba7-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.630 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.631 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.632 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.632 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.633 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.634 DEBUG:boto:Method: GET 2022-09-02T04:50:47.635 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.635 DEBUG:boto:Data: 2022-09-02T04:50:47.636 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.637 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.638 DEBUG:boto:Params: {} 2022-09-02T04:50:47.639 DEBUG:boto:Token: None 2022-09-02T04:50:47.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f40f4f2462da39-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.645 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.647 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.647 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.648 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.648 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.649 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.649 DEBUG:boto:Method: GET 2022-09-02T04:50:47.650 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.651 DEBUG:boto:Data: 2022-09-02T04:50:47.652 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.655 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.656 DEBUG:boto:Params: {} 2022-09-02T04:50:47.657 DEBUG:boto:Token: None 2022-09-02T04:50:47.658 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b737d96c8a85985-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.664 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.666 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.666 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.667 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.667 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.668 DEBUG:boto:Method: GET 2022-09-02T04:50:47.668 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.669 DEBUG:boto:Data: 2022-09-02T04:50:47.669 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.670 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.671 DEBUG:boto:Params: {} 2022-09-02T04:50:47.672 DEBUG:boto:Token: None 2022-09-02T04:50:47.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000490fb2c6a706926a-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.681 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.682 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.684 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.685 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.685 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.687 DEBUG:boto:Method: GET 2022-09-02T04:50:47.688 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.690 DEBUG:boto:Data: 2022-09-02T04:50:47.691 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.693 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.694 DEBUG:boto:Params: {} 2022-09-02T04:50:47.695 DEBUG:boto:Token: None 2022-09-02T04:50:47.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7df8910a9c46dc6-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.711 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-43 zones={test-zone1, test-zone2} 2022-09-02T04:50:47.714 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-43 zones={test-zone3, test-zone2} 2022-09-02T04:50:47.715 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.716 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.716 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.717 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.718 DEBUG:boto:Data: 2022-09-02T04:50:47.718 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.719 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.720 DEBUG:boto:Params: {} 2022-09-02T04:50:47.720 DEBUG:boto:Token: None 2022-09-02T04:50:47.721 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000499b83e1db72176d-0063118ba7-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.726 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.727 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.727 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.728 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.728 DEBUG:boto:Data: 2022-09-02T04:50:47.729 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.729 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.730 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.731 DEBUG:boto:Params: {} 2022-09-02T04:50:47.731 DEBUG:boto:Token: None 2022-09-02T04:50:47.732 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007622ccb474d14b18-0063118ba7-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.747 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.748 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.749 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.749 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.750 DEBUG:boto:Method: GET 2022-09-02T04:50:47.750 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.751 DEBUG:boto:Data: 2022-09-02T04:50:47.753 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.755 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.755 DEBUG:boto:Params: {} 2022-09-02T04:50:47.756 DEBUG:boto:Token: None 2022-09-02T04:50:47.756 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003451b7c670aa6bd7-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.767 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.768 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.768 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.769 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.769 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.770 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.770 DEBUG:boto:Method: GET 2022-09-02T04:50:47.771 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.771 DEBUG:boto:Data: 2022-09-02T04:50:47.772 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.773 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.773 DEBUG:boto:Params: {} 2022-09-02T04:50:47.774 DEBUG:boto:Token: None 2022-09-02T04:50:47.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c05f7b7c7a8054-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.822 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.823 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.824 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.825 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.825 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.826 DEBUG:boto:Method: GET 2022-09-02T04:50:47.827 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.827 DEBUG:boto:Data: 2022-09-02T04:50:47.828 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.829 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.830 DEBUG:boto:Params: {} 2022-09-02T04:50:47.835 DEBUG:boto:Token: None 2022-09-02T04:50:47.836 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000036a97860859a3e3-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.843 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.843 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.844 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.845 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.845 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.846 DEBUG:boto:Method: GET 2022-09-02T04:50:47.846 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.848 DEBUG:boto:Data: 2022-09-02T04:50:47.848 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.850 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.850 DEBUG:boto:Params: {} 2022-09-02T04:50:47.851 DEBUG:boto:Token: None 2022-09-02T04:50:47.854 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba9930c4d43fb489-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.862 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.863 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-43 zones={test-zone3, test-zone2} 2022-09-02T04:50:47.864 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-43 zones={test-zone1, test-zone3} 2022-09-02T04:50:47.865 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.866 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.867 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.868 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.869 DEBUG:boto:Data: 2022-09-02T04:50:47.870 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.872 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.873 DEBUG:boto:Params: {} 2022-09-02T04:50:47.874 DEBUG:boto:Token: None 2022-09-02T04:50:47.875 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016ce6be138dfecba-0063118ba7-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.882 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.883 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.884 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.885 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.886 DEBUG:boto:Data: 2022-09-02T04:50:47.888 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.890 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.890 DEBUG:boto:Params: {} 2022-09-02T04:50:47.891 DEBUG:boto:Token: None 2022-09-02T04:50:47.892 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ced3061bc53883e-0063118ba7-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.898 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.898 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.899 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.900 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.900 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.901 DEBUG:boto:Method: GET 2022-09-02T04:50:47.901 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.902 DEBUG:boto:Data: 2022-09-02T04:50:47.902 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.903 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.904 DEBUG:boto:Params: {} 2022-09-02T04:50:47.904 DEBUG:boto:Token: None 2022-09-02T04:50:47.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fda287fd7ca9dea-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.909 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.910 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.911 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.911 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.912 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.912 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.913 DEBUG:boto:Method: GET 2022-09-02T04:50:47.913 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.914 DEBUG:boto:Data: 2022-09-02T04:50:47.914 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.916 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.916 DEBUG:boto:Params: {} 2022-09-02T04:50:47.917 DEBUG:boto:Token: None 2022-09-02T04:50:47.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092180cf58997c0dc-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.922 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.923 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.923 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.924 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.924 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.925 DEBUG:boto:Method: GET 2022-09-02T04:50:47.925 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.926 DEBUG:boto:Data: 2022-09-02T04:50:47.926 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:47.927 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.928 DEBUG:boto:Params: {} 2022-09-02T04:50:47.928 DEBUG:boto:Token: None 2022-09-02T04:50:47.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002863cfa5c01c6310-0063118ba7-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.937 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.938 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.939 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.940 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.941 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.941 DEBUG:boto:Method: GET 2022-09-02T04:50:47.942 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.943 DEBUG:boto:Data: 2022-09-02T04:50:47.943 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.944 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.945 DEBUG:boto:Params: {} 2022-09-02T04:50:47.946 DEBUG:boto:Token: None 2022-09-02T04:50:47.947 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4f64a87b792518-0063118ba7-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.952 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-43 zones={test-zone1, test-zone3} 2022-09-02T04:50:47.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-43 zones={test-zone2, test-zone3} 2022-09-02T04:50:47.954 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.954 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.955 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.955 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.956 DEBUG:boto:Data: 2022-09-02T04:50:47.957 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.958 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.959 DEBUG:boto:Params: {} 2022-09-02T04:50:47.959 DEBUG:boto:Token: None 2022-09-02T04:50:47.960 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.961 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6a28bb978801f3d-0063118ba7-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.965 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.966 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.966 DEBUG:boto:Method: HEAD 2022-09-02T04:50:47.968 DEBUG:boto:Path: /xljbkf-43/ 2022-09-02T04:50:47.968 DEBUG:boto:Data: 2022-09-02T04:50:47.969 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:47.970 DEBUG:boto:Port: 8001 2022-09-02T04:50:47.970 DEBUG:boto:Params: {} 2022-09-02T04:50:47.971 DEBUG:boto:Token: None 2022-09-02T04:50:47.972 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/ 2022-09-02T04:50:47.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ= 2022-09-02T04:50:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ZVqLMY3Eg2lzD8NQJpzagTcasQ=', 'Content-Length': '0'} 2022-09-02T04:50:47.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d3ed8775cf07b29-0063118ba7-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.976 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:47.976 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.977 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.978 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.979 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.979 DEBUG:boto:Method: GET 2022-09-02T04:50:47.980 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.981 DEBUG:boto:Data: 2022-09-02T04:50:47.981 DEBUG:boto:Headers: {} 2022-09-02T04:50:47.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:47.982 DEBUG:boto:Port: 8000 2022-09-02T04:50:47.983 DEBUG:boto:Params: {} 2022-09-02T04:50:47.984 DEBUG:boto:Token: None 2022-09-02T04:50:47.984 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:47 GMT /xljbkf-43/?versions 2022-09-02T04:50:47.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY= 2022-09-02T04:50:47.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUugctqSDeeP9WYVZoY4psAq7aY=', 'Content-Length': '0'} 2022-09-02T04:50:47.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005119e936b3d5a709-0063118ba7-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:47.989 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:47.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:47.992 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:47.993 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:47.994 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.994 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.995 DEBUG:boto:Method: GET 2022-09-02T04:50:47.996 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:47.998 DEBUG:boto:Data: 2022-09-02T04:50:47.999 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.001 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.001 DEBUG:boto:Params: {} 2022-09-02T04:50:48.002 DEBUG:boto:Token: None 2022-09-02T04:50:48.003 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-43/?versions 2022-09-02T04:50:48.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i64hfUtZDMfOsuqb61sZlMW8oqU= 2022-09-02T04:50:48.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i64hfUtZDMfOsuqb61sZlMW8oqU=', 'Content-Length': '0'} 2022-09-02T04:50:48.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cad79cc4fa6ffe-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.050 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:48.052 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:48.052 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:48.053 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:48.053 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:48.054 DEBUG:boto:Method: GET 2022-09-02T04:50:48.055 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:48.055 DEBUG:boto:Data: 2022-09-02T04:50:48.056 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.057 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.059 DEBUG:boto:Params: {} 2022-09-02T04:50:48.060 DEBUG:boto:Token: None 2022-09-02T04:50:48.060 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-43/?versions 2022-09-02T04:50:48.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i64hfUtZDMfOsuqb61sZlMW8oqU= 2022-09-02T04:50:48.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i64hfUtZDMfOsuqb61sZlMW8oqU=', 'Content-Length': '0'} 2022-09-02T04:50:48.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a58b7cd63b5a35a-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.066 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:48.067 DEBUG:boto:path=/xljbkf-43/ 2022-09-02T04:50:48.067 DEBUG:boto:auth_path=/xljbkf-43/ 2022-09-02T04:50:48.068 DEBUG:boto:path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:48.068 DEBUG:boto:auth_path=/xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:48.069 DEBUG:boto:Method: GET 2022-09-02T04:50:48.070 DEBUG:boto:Path: /xljbkf-43/?versions&max-keys=999 2022-09-02T04:50:48.070 DEBUG:boto:Data: 2022-09-02T04:50:48.071 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.072 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.073 DEBUG:boto:Params: {} 2022-09-02T04:50:48.073 DEBUG:boto:Token: None 2022-09-02T04:50:48.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-43/?versions 2022-09-02T04:50:48.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i64hfUtZDMfOsuqb61sZlMW8oqU= 2022-09-02T04:50:48.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i64hfUtZDMfOsuqb61sZlMW8oqU=', 'Content-Length': '0'} 2022-09-02T04:50:48.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b12aff003706172-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.079 DEBUG:boto:b'xljbkf-43999false' 2022-09-02T04:50:48.080 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-43 zones={test-zone2, test-zone3} 2022-09-02T04:50:48.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-44 zones={test-zone2, test-zone1} 2022-09-02T04:50:48.081 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.082 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.082 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.083 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.084 DEBUG:boto:Data: 2022-09-02T04:50:48.085 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.087 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.088 DEBUG:boto:Params: {} 2022-09-02T04:50:48.089 DEBUG:boto:Token: None 2022-09-02T04:50:48.090 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7644f99f7be2d87-0063118ba8-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.096 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.097 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.097 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.098 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.098 DEBUG:boto:Data: 2022-09-02T04:50:48.099 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.100 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.100 DEBUG:boto:Params: {} 2022-09-02T04:50:48.101 DEBUG:boto:Token: None 2022-09-02T04:50:48.101 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b2754f159d87c96-0063118ba8-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.106 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.107 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.107 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.108 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.108 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.109 DEBUG:boto:Method: GET 2022-09-02T04:50:48.110 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.110 DEBUG:boto:Data: 2022-09-02T04:50:48.111 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.112 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.112 DEBUG:boto:Params: {} 2022-09-02T04:50:48.113 DEBUG:boto:Token: None 2022-09-02T04:50:48.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca3e54e45b8ea39-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.119 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.120 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.121 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.121 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.122 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.122 DEBUG:boto:Method: GET 2022-09-02T04:50:48.123 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.123 DEBUG:boto:Data: 2022-09-02T04:50:48.124 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.125 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.126 DEBUG:boto:Params: {} 2022-09-02T04:50:48.127 DEBUG:boto:Token: None 2022-09-02T04:50:48.127 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ce1acab14c02d5-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.132 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.133 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.133 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.134 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.134 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.135 DEBUG:boto:Method: GET 2022-09-02T04:50:48.135 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.136 DEBUG:boto:Data: 2022-09-02T04:50:48.137 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.138 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.138 DEBUG:boto:Params: {} 2022-09-02T04:50:48.139 DEBUG:boto:Token: None 2022-09-02T04:50:48.140 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4c6236b2569cb23-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.147 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.148 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.149 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.149 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.150 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.150 DEBUG:boto:Method: GET 2022-09-02T04:50:48.151 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.152 DEBUG:boto:Data: 2022-09-02T04:50:48.153 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.155 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.155 DEBUG:boto:Params: {} 2022-09-02T04:50:48.156 DEBUG:boto:Token: None 2022-09-02T04:50:48.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e3c4605f542a694-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.202 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-44 zones={test-zone2, test-zone1} 2022-09-02T04:50:48.204 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-44 zones={test-zone3, test-zone1} 2022-09-02T04:50:48.204 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.205 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.206 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.206 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.207 DEBUG:boto:Data: 2022-09-02T04:50:48.207 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.208 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.209 DEBUG:boto:Params: {} 2022-09-02T04:50:48.209 DEBUG:boto:Token: None 2022-09-02T04:50:48.210 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbe1a76e87eb6378-0063118ba8-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.215 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.216 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.216 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.217 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.217 DEBUG:boto:Data: 2022-09-02T04:50:48.218 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.219 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.220 DEBUG:boto:Params: {} 2022-09-02T04:50:48.220 DEBUG:boto:Token: None 2022-09-02T04:50:48.221 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.221 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67d37a833b0cd1e-0063118ba8-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.228 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.228 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.229 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.230 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.231 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.232 DEBUG:boto:Method: GET 2022-09-02T04:50:48.233 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.234 DEBUG:boto:Data: 2022-09-02T04:50:48.235 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.237 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.238 DEBUG:boto:Params: {} 2022-09-02T04:50:48.239 DEBUG:boto:Token: None 2022-09-02T04:50:48.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b085180321cbe6f-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.246 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.247 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.248 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.249 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.250 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.251 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.252 DEBUG:boto:Method: GET 2022-09-02T04:50:48.253 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.254 DEBUG:boto:Data: 2022-09-02T04:50:48.255 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.257 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.258 DEBUG:boto:Params: {} 2022-09-02T04:50:48.259 DEBUG:boto:Token: None 2022-09-02T04:50:48.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b56503421fe07e54-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.266 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.267 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.268 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.269 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.269 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.270 DEBUG:boto:Method: GET 2022-09-02T04:50:48.271 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.272 DEBUG:boto:Data: 2022-09-02T04:50:48.273 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.275 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.276 DEBUG:boto:Params: {} 2022-09-02T04:50:48.277 DEBUG:boto:Token: None 2022-09-02T04:50:48.278 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004807f736e5cfeb61-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.284 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.285 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.286 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.287 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.288 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.289 DEBUG:boto:Method: GET 2022-09-02T04:50:48.290 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.291 DEBUG:boto:Data: 2022-09-02T04:50:48.291 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.292 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.293 DEBUG:boto:Params: {} 2022-09-02T04:50:48.294 DEBUG:boto:Token: None 2022-09-02T04:50:48.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000408b0b574f0014f2-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.299 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.300 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-44 zones={test-zone3, test-zone1} 2022-09-02T04:50:48.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-44 zones={test-zone1, test-zone2} 2022-09-02T04:50:48.302 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.302 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.303 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.303 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.304 DEBUG:boto:Data: 2022-09-02T04:50:48.304 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.305 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.306 DEBUG:boto:Params: {} 2022-09-02T04:50:48.307 DEBUG:boto:Token: None 2022-09-02T04:50:48.307 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000478bf56bcd0c438e-0063118ba8-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.315 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.315 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.316 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.316 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.317 DEBUG:boto:Data: 2022-09-02T04:50:48.317 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.319 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.320 DEBUG:boto:Params: {} 2022-09-02T04:50:48.320 DEBUG:boto:Token: None 2022-09-02T04:50:48.321 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000101124716a9aef70-0063118ba8-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.333 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.335 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.335 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.336 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.336 DEBUG:boto:Method: GET 2022-09-02T04:50:48.337 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.338 DEBUG:boto:Data: 2022-09-02T04:50:48.339 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.340 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.341 DEBUG:boto:Params: {} 2022-09-02T04:50:48.342 DEBUG:boto:Token: None 2022-09-02T04:50:48.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f5b760c3dd99b76-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.394 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.396 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.397 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.398 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.398 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.399 DEBUG:boto:Method: GET 2022-09-02T04:50:48.399 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.400 DEBUG:boto:Data: 2022-09-02T04:50:48.400 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.401 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.402 DEBUG:boto:Params: {} 2022-09-02T04:50:48.403 DEBUG:boto:Token: None 2022-09-02T04:50:48.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e14efbe08bd4014-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.409 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.409 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.410 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.411 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.411 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.412 DEBUG:boto:Method: GET 2022-09-02T04:50:48.412 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.413 DEBUG:boto:Data: 2022-09-02T04:50:48.413 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.414 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.415 DEBUG:boto:Params: {} 2022-09-02T04:50:48.415 DEBUG:boto:Token: None 2022-09-02T04:50:48.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f3a28e2c9bd62e4-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.422 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.423 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.424 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.424 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.425 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.425 DEBUG:boto:Method: GET 2022-09-02T04:50:48.426 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.427 DEBUG:boto:Data: 2022-09-02T04:50:48.440 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.442 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.442 DEBUG:boto:Params: {} 2022-09-02T04:50:48.444 DEBUG:boto:Token: None 2022-09-02T04:50:48.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d51d0ba77777dc9-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.452 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.453 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-44 zones={test-zone1, test-zone2} 2022-09-02T04:50:48.454 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-44 zones={test-zone3, test-zone2} 2022-09-02T04:50:48.455 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.456 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.456 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.457 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.457 DEBUG:boto:Data: 2022-09-02T04:50:48.458 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.459 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.460 DEBUG:boto:Params: {} 2022-09-02T04:50:48.460 DEBUG:boto:Token: None 2022-09-02T04:50:48.461 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dc9792bf54053fc-0063118ba8-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.468 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.469 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.469 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.470 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.470 DEBUG:boto:Data: 2022-09-02T04:50:48.471 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.472 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.472 DEBUG:boto:Params: {} 2022-09-02T04:50:48.473 DEBUG:boto:Token: None 2022-09-02T04:50:48.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094197d23da69fbdc-0063118ba8-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.482 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.483 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.484 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.485 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.486 DEBUG:boto:Method: GET 2022-09-02T04:50:48.488 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.490 DEBUG:boto:Data: 2022-09-02T04:50:48.491 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.493 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.494 DEBUG:boto:Params: {} 2022-09-02T04:50:48.495 DEBUG:boto:Token: None 2022-09-02T04:50:48.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e97a76cf29aa1e-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.542 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.544 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.545 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.546 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.547 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.550 DEBUG:boto:Method: GET 2022-09-02T04:50:48.550 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.551 DEBUG:boto:Data: 2022-09-02T04:50:48.551 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.553 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.553 DEBUG:boto:Params: {} 2022-09-02T04:50:48.555 DEBUG:boto:Token: None 2022-09-02T04:50:48.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ffd62ba68d7978-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.568 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.570 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.571 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.576 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.577 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.578 DEBUG:boto:Method: GET 2022-09-02T04:50:48.579 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.580 DEBUG:boto:Data: 2022-09-02T04:50:48.581 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.587 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.588 DEBUG:boto:Params: {} 2022-09-02T04:50:48.590 DEBUG:boto:Token: None 2022-09-02T04:50:48.591 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670c50c216e45005-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.600 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.602 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.603 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.604 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.605 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.606 DEBUG:boto:Method: GET 2022-09-02T04:50:48.607 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.608 DEBUG:boto:Data: 2022-09-02T04:50:48.609 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.612 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.613 DEBUG:boto:Params: {} 2022-09-02T04:50:48.614 DEBUG:boto:Token: None 2022-09-02T04:50:48.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000906fb44ae98ede60-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.627 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.629 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-44 zones={test-zone3, test-zone2} 2022-09-02T04:50:48.630 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-44 zones={test-zone1, test-zone3} 2022-09-02T04:50:48.631 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.633 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.633 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.634 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.635 DEBUG:boto:Data: 2022-09-02T04:50:48.636 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.639 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.640 DEBUG:boto:Params: {} 2022-09-02T04:50:48.641 DEBUG:boto:Token: None 2022-09-02T04:50:48.642 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f32de5a4c7e6060-0063118ba8-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.650 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.651 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.652 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.653 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.659 DEBUG:boto:Data: 2022-09-02T04:50:48.660 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.662 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.663 DEBUG:boto:Params: {} 2022-09-02T04:50:48.664 DEBUG:boto:Token: None 2022-09-02T04:50:48.665 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084b12097fee38796-0063118ba8-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.673 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.674 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.675 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.676 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.677 DEBUG:boto:Method: GET 2022-09-02T04:50:48.678 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.679 DEBUG:boto:Data: 2022-09-02T04:50:48.679 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.681 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.682 DEBUG:boto:Params: {} 2022-09-02T04:50:48.683 DEBUG:boto:Token: None 2022-09-02T04:50:48.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066bb85a8279b4c2e-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.692 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.693 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.694 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.695 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.698 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.699 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.699 DEBUG:boto:Method: GET 2022-09-02T04:50:48.700 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.700 DEBUG:boto:Data: 2022-09-02T04:50:48.701 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.704 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.705 DEBUG:boto:Params: {} 2022-09-02T04:50:48.706 DEBUG:boto:Token: None 2022-09-02T04:50:48.707 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094adb398dd1f9946-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.716 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.717 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.724 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.725 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.726 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.726 DEBUG:boto:Method: GET 2022-09-02T04:50:48.727 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.727 DEBUG:boto:Data: 2022-09-02T04:50:48.728 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.730 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.730 DEBUG:boto:Params: {} 2022-09-02T04:50:48.731 DEBUG:boto:Token: None 2022-09-02T04:50:48.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009629333b7d22a744-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.741 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.742 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.743 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.744 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.745 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.746 DEBUG:boto:Method: GET 2022-09-02T04:50:48.747 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.748 DEBUG:boto:Data: 2022-09-02T04:50:48.749 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.750 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.751 DEBUG:boto:Params: {} 2022-09-02T04:50:48.752 DEBUG:boto:Token: None 2022-09-02T04:50:48.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb328bd9da30be67-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.761 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-44 zones={test-zone1, test-zone3} 2022-09-02T04:50:48.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-44 zones={test-zone2, test-zone3} 2022-09-02T04:50:48.764 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.765 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.766 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.767 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.768 DEBUG:boto:Data: 2022-09-02T04:50:48.770 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.772 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.773 DEBUG:boto:Params: {} 2022-09-02T04:50:48.774 DEBUG:boto:Token: None 2022-09-02T04:50:48.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024b288a4f78897a2-0063118ba8-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.782 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.783 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.784 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.785 DEBUG:boto:Path: /xljbkf-44/ 2022-09-02T04:50:48.786 DEBUG:boto:Data: 2022-09-02T04:50:48.787 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.792 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.793 DEBUG:boto:Params: {} 2022-09-02T04:50:48.794 DEBUG:boto:Token: None 2022-09-02T04:50:48.794 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/ 2022-09-02T04:50:48.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw= 2022-09-02T04:50:48.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:giO5r/H4DZ+GSxfYYBrw75IHnbw=', 'Content-Length': '0'} 2022-09-02T04:50:48.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025bfce9310279982-0063118ba8-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.802 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.802 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.803 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.804 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.805 DEBUG:boto:Method: GET 2022-09-02T04:50:48.805 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.806 DEBUG:boto:Data: 2022-09-02T04:50:48.807 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.810 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.811 DEBUG:boto:Params: {} 2022-09-02T04:50:48.812 DEBUG:boto:Token: None 2022-09-02T04:50:48.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001053a97b87e98298-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.820 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.821 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.822 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.823 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.824 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.825 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.826 DEBUG:boto:Method: GET 2022-09-02T04:50:48.826 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.827 DEBUG:boto:Data: 2022-09-02T04:50:48.828 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.830 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.831 DEBUG:boto:Params: {} 2022-09-02T04:50:48.832 DEBUG:boto:Token: None 2022-09-02T04:50:48.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009775628c595ec903-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.841 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.842 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.843 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.843 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.844 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.844 DEBUG:boto:Method: GET 2022-09-02T04:50:48.845 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.846 DEBUG:boto:Data: 2022-09-02T04:50:48.847 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.848 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.848 DEBUG:boto:Params: {} 2022-09-02T04:50:48.849 DEBUG:boto:Token: None 2022-09-02T04:50:48.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041005cf9f9afd5e3-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.856 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.857 DEBUG:boto:path=/xljbkf-44/ 2022-09-02T04:50:48.857 DEBUG:boto:auth_path=/xljbkf-44/ 2022-09-02T04:50:48.858 DEBUG:boto:path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.859 DEBUG:boto:auth_path=/xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.859 DEBUG:boto:Method: GET 2022-09-02T04:50:48.861 DEBUG:boto:Path: /xljbkf-44/?versions&max-keys=999 2022-09-02T04:50:48.862 DEBUG:boto:Data: 2022-09-02T04:50:48.863 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:48.864 DEBUG:boto:Port: 8001 2022-09-02T04:50:48.865 DEBUG:boto:Params: {} 2022-09-02T04:50:48.865 DEBUG:boto:Token: None 2022-09-02T04:50:48.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-44/?versions 2022-09-02T04:50:48.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc= 2022-09-02T04:50:48.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pEad3D2hFPtAoAgjuSPKgjxCXqc=', 'Content-Length': '0'} 2022-09-02T04:50:48.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dca76ad6f41459f8-0063118ba8-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.914 DEBUG:boto:b'xljbkf-44999false' 2022-09-02T04:50:48.916 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-44 zones={test-zone2, test-zone3} 2022-09-02T04:50:48.917 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-45 zones={test-zone2, test-zone1} 2022-09-02T04:50:48.917 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:48.918 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:48.919 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.919 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:48.920 DEBUG:boto:Data: 2022-09-02T04:50:48.921 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.922 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.922 DEBUG:boto:Params: {} 2022-09-02T04:50:48.923 DEBUG:boto:Token: None 2022-09-02T04:50:48.925 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-45/ 2022-09-02T04:50:48.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mu0LHbbGf/F3MvcOcR8vK2SgAwA= 2022-09-02T04:50:48.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mu0LHbbGf/F3MvcOcR8vK2SgAwA=', 'Content-Length': '0'} 2022-09-02T04:50:48.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d8839623ebf6482-0063118ba8-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.932 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:48.932 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:48.933 DEBUG:boto:Method: HEAD 2022-09-02T04:50:48.934 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:48.936 DEBUG:boto:Data: 2022-09-02T04:50:48.937 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.938 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.939 DEBUG:boto:Params: {} 2022-09-02T04:50:48.940 DEBUG:boto:Token: None 2022-09-02T04:50:48.941 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-45/ 2022-09-02T04:50:48.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mu0LHbbGf/F3MvcOcR8vK2SgAwA= 2022-09-02T04:50:48.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mu0LHbbGf/F3MvcOcR8vK2SgAwA=', 'Content-Length': '0'} 2022-09-02T04:50:48.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022ea9b8025641a1c-0063118ba8-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:48.949 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:48.950 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:48.950 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.951 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.951 DEBUG:boto:Method: GET 2022-09-02T04:50:48.952 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.953 DEBUG:boto:Data: 2022-09-02T04:50:48.954 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.956 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.956 DEBUG:boto:Params: {} 2022-09-02T04:50:48.958 DEBUG:boto:Token: None 2022-09-02T04:50:48.959 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-45/?versions 2022-09-02T04:50:48.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MiEniqgi/2IloHNOavRyy1wEHiI= 2022-09-02T04:50:48.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MiEniqgi/2IloHNOavRyy1wEHiI=', 'Content-Length': '0'} 2022-09-02T04:50:48.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff0b62c88d1369d-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.964 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:48.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:48.965 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:48.965 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:48.966 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.967 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.967 DEBUG:boto:Method: GET 2022-09-02T04:50:48.968 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.969 DEBUG:boto:Data: 2022-09-02T04:50:48.969 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:48.970 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.971 DEBUG:boto:Params: {} 2022-09-02T04:50:48.972 DEBUG:boto:Token: None 2022-09-02T04:50:48.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-45/?versions 2022-09-02T04:50:48.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MiEniqgi/2IloHNOavRyy1wEHiI= 2022-09-02T04:50:48.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MiEniqgi/2IloHNOavRyy1wEHiI=', 'Content-Length': '0'} 2022-09-02T04:50:48.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000710b72e4091b9e1a-0063118ba8-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:48.978 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:48.978 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:48.979 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:48.979 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.980 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.980 DEBUG:boto:Method: GET 2022-09-02T04:50:48.981 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:48.981 DEBUG:boto:Data: 2022-09-02T04:50:48.982 DEBUG:boto:Headers: {} 2022-09-02T04:50:48.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:48.983 DEBUG:boto:Port: 8000 2022-09-02T04:50:48.983 DEBUG:boto:Params: {} 2022-09-02T04:50:48.984 DEBUG:boto:Token: None 2022-09-02T04:50:48.985 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:48 GMT /xljbkf-45/?versions 2022-09-02T04:50:48.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MiEniqgi/2IloHNOavRyy1wEHiI= 2022-09-02T04:50:48.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MiEniqgi/2IloHNOavRyy1wEHiI=', 'Content-Length': '0'} 2022-09-02T04:50:48.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000661ad1b13d0fe123-0063118ba8-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.034 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.035 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.036 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.036 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.037 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.037 DEBUG:boto:Method: GET 2022-09-02T04:50:49.038 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.039 DEBUG:boto:Data: 2022-09-02T04:50:49.039 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.040 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.041 DEBUG:boto:Params: {} 2022-09-02T04:50:49.041 DEBUG:boto:Token: None 2022-09-02T04:50:49.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069400df633aa9f4f-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.047 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-45 zones={test-zone2, test-zone1} 2022-09-02T04:50:49.048 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-45 zones={test-zone3, test-zone1} 2022-09-02T04:50:49.048 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.049 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.050 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.050 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.051 DEBUG:boto:Data: 2022-09-02T04:50:49.051 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.052 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.053 DEBUG:boto:Params: {} 2022-09-02T04:50:49.053 DEBUG:boto:Token: None 2022-09-02T04:50:49.054 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce28cc19f83fb178-0063118ba9-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.059 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.060 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.063 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.064 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.065 DEBUG:boto:Data: 2022-09-02T04:50:49.066 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.068 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.069 DEBUG:boto:Params: {} 2022-09-02T04:50:49.070 DEBUG:boto:Token: None 2022-09-02T04:50:49.071 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.077 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062f79c80d0aaac66-0063118ba9-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:49.079 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.079 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.080 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.080 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.081 DEBUG:boto:Method: GET 2022-09-02T04:50:49.081 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.087 DEBUG:boto:Data: 2022-09-02T04:50:49.087 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.089 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.090 DEBUG:boto:Params: {} 2022-09-02T04:50:49.090 DEBUG:boto:Token: None 2022-09-02T04:50:49.091 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057cc8118872e3b3c-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.096 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.097 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:49.097 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.098 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.098 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.099 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.100 DEBUG:boto:Method: GET 2022-09-02T04:50:49.100 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.101 DEBUG:boto:Data: 2022-09-02T04:50:49.101 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.102 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.103 DEBUG:boto:Params: {} 2022-09-02T04:50:49.104 DEBUG:boto:Token: None 2022-09-02T04:50:49.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d89ea4c1cc44628-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.154 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.156 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.156 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.157 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.157 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.158 DEBUG:boto:Method: GET 2022-09-02T04:50:49.159 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.160 DEBUG:boto:Data: 2022-09-02T04:50:49.161 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.162 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.163 DEBUG:boto:Params: {} 2022-09-02T04:50:49.163 DEBUG:boto:Token: None 2022-09-02T04:50:49.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e79c55c7f319e385-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.169 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.171 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.172 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.173 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.174 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.175 DEBUG:boto:Method: GET 2022-09-02T04:50:49.176 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.177 DEBUG:boto:Data: 2022-09-02T04:50:49.178 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.180 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.181 DEBUG:boto:Params: {} 2022-09-02T04:50:49.182 DEBUG:boto:Token: None 2022-09-02T04:50:49.182 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014dcd8da0ace9410-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.189 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.189 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-45 zones={test-zone3, test-zone1} 2022-09-02T04:50:49.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-45 zones={test-zone1, test-zone2} 2022-09-02T04:50:49.190 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.191 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.191 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.192 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.192 DEBUG:boto:Data: 2022-09-02T04:50:49.193 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.194 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.195 DEBUG:boto:Params: {} 2022-09-02T04:50:49.196 DEBUG:boto:Token: None 2022-09-02T04:50:49.197 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d93735d40a847a93-0063118ba9-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.202 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.202 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.203 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.203 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.204 DEBUG:boto:Data: 2022-09-02T04:50:49.206 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.208 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.209 DEBUG:boto:Params: {} 2022-09-02T04:50:49.210 DEBUG:boto:Token: None 2022-09-02T04:50:49.210 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f620771a4258cbc2-0063118ba9-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:49.216 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.217 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.217 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.218 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.219 DEBUG:boto:Method: GET 2022-09-02T04:50:49.220 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.221 DEBUG:boto:Data: 2022-09-02T04:50:49.222 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.223 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.224 DEBUG:boto:Params: {} 2022-09-02T04:50:49.224 DEBUG:boto:Token: None 2022-09-02T04:50:49.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b9e6ff20f158fee-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.229 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:49.231 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.232 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.233 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.234 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.235 DEBUG:boto:Method: GET 2022-09-02T04:50:49.236 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.237 DEBUG:boto:Data: 2022-09-02T04:50:49.238 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.240 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.240 DEBUG:boto:Params: {} 2022-09-02T04:50:49.241 DEBUG:boto:Token: None 2022-09-02T04:50:49.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067cd943634462c53-0063118ba9-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.246 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.248 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.248 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.249 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.249 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.250 DEBUG:boto:Method: GET 2022-09-02T04:50:49.251 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.251 DEBUG:boto:Data: 2022-09-02T04:50:49.252 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.253 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.253 DEBUG:boto:Params: {} 2022-09-02T04:50:49.254 DEBUG:boto:Token: None 2022-09-02T04:50:49.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808e087fd76a02c2-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.262 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.263 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.264 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.265 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.266 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.267 DEBUG:boto:Method: GET 2022-09-02T04:50:49.268 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.269 DEBUG:boto:Data: 2022-09-02T04:50:49.270 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.271 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.271 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.272 DEBUG:boto:Params: {} 2022-09-02T04:50:49.273 DEBUG:boto:Token: None 2022-09-02T04:50:49.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4a8e86927d5e4e-0063118ba9-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.279 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.280 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-45 zones={test-zone1, test-zone2} 2022-09-02T04:50:49.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-45 zones={test-zone3, test-zone2} 2022-09-02T04:50:49.281 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.282 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.283 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.283 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.284 DEBUG:boto:Data: 2022-09-02T04:50:49.285 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.286 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.286 DEBUG:boto:Params: {} 2022-09-02T04:50:49.287 DEBUG:boto:Token: None 2022-09-02T04:50:49.288 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a33ebe824ba24c36-0063118ba9-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.292 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.293 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.294 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.295 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.295 DEBUG:boto:Data: 2022-09-02T04:50:49.296 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.297 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.298 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.299 DEBUG:boto:Params: {} 2022-09-02T04:50:49.300 DEBUG:boto:Token: None 2022-09-02T04:50:49.301 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.302 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d08369bc051b2ab6-0063118ba9-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:49.309 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.310 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.311 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.312 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.313 DEBUG:boto:Method: GET 2022-09-02T04:50:49.314 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.315 DEBUG:boto:Data: 2022-09-02T04:50:49.316 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.317 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.318 DEBUG:boto:Params: {} 2022-09-02T04:50:49.319 DEBUG:boto:Token: None 2022-09-02T04:50:49.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a6feed24bb4703-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.324 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.325 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:49.325 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.327 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.328 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.329 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.331 DEBUG:boto:Method: GET 2022-09-02T04:50:49.332 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.332 DEBUG:boto:Data: 2022-09-02T04:50:49.333 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.335 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.336 DEBUG:boto:Params: {} 2022-09-02T04:50:49.337 DEBUG:boto:Token: None 2022-09-02T04:50:49.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd55c06c3c5f750-0063118ba9-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.343 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.344 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.345 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.345 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.346 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.346 DEBUG:boto:Method: GET 2022-09-02T04:50:49.349 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.350 DEBUG:boto:Data: 2022-09-02T04:50:49.350 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.351 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.352 DEBUG:boto:Params: {} 2022-09-02T04:50:49.352 DEBUG:boto:Token: None 2022-09-02T04:50:49.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003efe59d55d7efc20-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.398 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.399 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.400 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.400 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.401 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.401 DEBUG:boto:Method: GET 2022-09-02T04:50:49.403 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.403 DEBUG:boto:Data: 2022-09-02T04:50:49.404 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.405 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.406 DEBUG:boto:Params: {} 2022-09-02T04:50:49.406 DEBUG:boto:Token: None 2022-09-02T04:50:49.407 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004abf29026d3f757d-0063118ba9-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.412 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.413 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-45 zones={test-zone3, test-zone2} 2022-09-02T04:50:49.414 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-45 zones={test-zone1, test-zone3} 2022-09-02T04:50:49.414 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.415 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.415 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.416 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.416 DEBUG:boto:Data: 2022-09-02T04:50:49.418 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.419 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.419 DEBUG:boto:Params: {} 2022-09-02T04:50:49.420 DEBUG:boto:Token: None 2022-09-02T04:50:49.421 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b700e7b25cd6def-0063118ba9-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.427 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.427 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.428 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.428 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.429 DEBUG:boto:Data: 2022-09-02T04:50:49.429 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.431 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.431 DEBUG:boto:Params: {} 2022-09-02T04:50:49.432 DEBUG:boto:Token: None 2022-09-02T04:50:49.432 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072c1852bb3177f06-0063118ba9-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:49.437 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.438 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.438 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.439 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.440 DEBUG:boto:Method: GET 2022-09-02T04:50:49.440 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.441 DEBUG:boto:Data: 2022-09-02T04:50:49.441 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.443 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.443 DEBUG:boto:Params: {} 2022-09-02T04:50:49.444 DEBUG:boto:Token: None 2022-09-02T04:50:49.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09518a3b6b88242-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.449 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:49.450 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.451 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.451 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.452 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.452 DEBUG:boto:Method: GET 2022-09-02T04:50:49.453 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.453 DEBUG:boto:Data: 2022-09-02T04:50:49.454 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.455 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.456 DEBUG:boto:Params: {} 2022-09-02T04:50:49.456 DEBUG:boto:Token: None 2022-09-02T04:50:49.457 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5db0e2ba9966b0a-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.462 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.463 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.464 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.465 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.467 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.469 DEBUG:boto:Method: GET 2022-09-02T04:50:49.470 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.470 DEBUG:boto:Data: 2022-09-02T04:50:49.471 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:49.473 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.474 DEBUG:boto:Params: {} 2022-09-02T04:50:49.475 DEBUG:boto:Token: None 2022-09-02T04:50:49.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000713875931e7df1de-0063118ba9-1425-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.481 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.482 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.483 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.483 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.484 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.484 DEBUG:boto:Method: GET 2022-09-02T04:50:49.485 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.485 DEBUG:boto:Data: 2022-09-02T04:50:49.486 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.487 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.487 DEBUG:boto:Params: {} 2022-09-02T04:50:49.488 DEBUG:boto:Token: None 2022-09-02T04:50:49.489 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.489 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000074e8d2e4ef3a270-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.494 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-45 zones={test-zone1, test-zone3} 2022-09-02T04:50:49.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-45 zones={test-zone2, test-zone3} 2022-09-02T04:50:49.497 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.498 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.498 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.499 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.500 DEBUG:boto:Data: 2022-09-02T04:50:49.500 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.502 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.503 DEBUG:boto:Params: {} 2022-09-02T04:50:49.504 DEBUG:boto:Token: None 2022-09-02T04:50:49.505 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2db36c8e06a7c58-0063118ba9-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.513 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.514 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.515 DEBUG:boto:Method: HEAD 2022-09-02T04:50:49.516 DEBUG:boto:Path: /xljbkf-45/ 2022-09-02T04:50:49.517 DEBUG:boto:Data: 2022-09-02T04:50:49.518 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.520 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.520 DEBUG:boto:Params: {} 2022-09-02T04:50:49.522 DEBUG:boto:Token: None 2022-09-02T04:50:49.523 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/ 2022-09-02T04:50:49.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM= 2022-09-02T04:50:49.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wPahZwr2jbfTtJk64QjTJUMGSXM=', 'Content-Length': '0'} 2022-09-02T04:50:49.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3cb2e3191620821-0063118ba9-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.529 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T04:50:49.530 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.531 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.531 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.532 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.541 DEBUG:boto:Method: GET 2022-09-02T04:50:49.542 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.543 DEBUG:boto:Data: 2022-09-02T04:50:49.546 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.546 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.551 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.552 DEBUG:boto:Params: {} 2022-09-02T04:50:49.553 DEBUG:boto:Token: None 2022-09-02T04:50:49.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ef6d9b7c4ceba8-0063118ba9-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.561 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.563 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T04:50:49.564 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.565 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.566 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.567 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.567 DEBUG:boto:Method: GET 2022-09-02T04:50:49.568 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.569 DEBUG:boto:Data: 2022-09-02T04:50:49.570 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.572 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.573 DEBUG:boto:Params: {} 2022-09-02T04:50:49.575 DEBUG:boto:Token: None 2022-09-02T04:50:49.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d951ed1cbe07d0c9-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.584 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.586 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.587 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.588 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.589 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.590 DEBUG:boto:Method: GET 2022-09-02T04:50:49.591 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.592 DEBUG:boto:Data: 2022-09-02T04:50:49.592 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:49.594 DEBUG:boto:Port: 8000 2022-09-02T04:50:49.595 DEBUG:boto:Params: {} 2022-09-02T04:50:49.596 DEBUG:boto:Token: None 2022-09-02T04:50:49.597 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ba0dc5b4d762ca-0063118ba9-1453-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.606 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.608 DEBUG:boto:path=/xljbkf-45/ 2022-09-02T04:50:49.609 DEBUG:boto:auth_path=/xljbkf-45/ 2022-09-02T04:50:49.610 DEBUG:boto:path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.611 DEBUG:boto:auth_path=/xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.613 DEBUG:boto:Method: GET 2022-09-02T04:50:49.614 DEBUG:boto:Path: /xljbkf-45/?versions&max-keys=999 2022-09-02T04:50:49.615 DEBUG:boto:Data: 2022-09-02T04:50:49.616 DEBUG:boto:Headers: {} 2022-09-02T04:50:49.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:49.617 DEBUG:boto:Port: 8001 2022-09-02T04:50:49.618 DEBUG:boto:Params: {} 2022-09-02T04:50:49.619 DEBUG:boto:Token: None 2022-09-02T04:50:49.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:50:49 GMT /xljbkf-45/?versions 2022-09-02T04:50:49.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM= 2022-09-02T04:50:49.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pR5gxb3MoYF1Nf3YUjVAeBpclPM=', 'Content-Length': '0'} 2022-09-02T04:50:49.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000310e74fd140bfdbf-0063118ba9-142b-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:50:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:49.628 DEBUG:boto:b'xljbkf-45999false' 2022-09-02T04:50:49.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-45 zones={test-zone2, test-zone3} 2022-09-02T04:50:49.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:49.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:49.633 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:49.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:49.731 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:49.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.731 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092292.275513_70.1", 2022-09-02T04:50:49.732 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "user", 2022-09-02T04:50:49.732 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:49.732 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.275513716Z", 2022-09-02T04:50:49.732 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.733 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.733 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.733 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.734 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.734 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:49.734 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.736 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092292.280197_71.1", 2022-09-02T04:50:49.736 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "user", 2022-09-02T04:50:49.737 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:49.737 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197578Z", 2022-09-02T04:50:49.737 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.737 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.738 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.738 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.738 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.739 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:49.739 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.739 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.740 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.750378_51.1", 2022-09-02T04:50:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.750378020Z", 2022-09-02T04:50:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.746 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.753068_53.1", 2022-09-02T04:50:49.746 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.746 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:49.747 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.753068141Z", 2022-09-02T04:50:49.747 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.747 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.747 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.754140_54.1", 2022-09-02T04:50:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10", 2022-09-02T04:50:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.754140083Z", 2022-09-02T04:50:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.753 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.753 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.753 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.754 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:49.754 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.755 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.755 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.756 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.756550_55.1", 2022-09-02T04:50:49.756 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.756 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10", 2022-09-02T04:50:49.757 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.756550326Z", 2022-09-02T04:50:49.757 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.757 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.758 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.758 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.758 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.758 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:49.759 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.761 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.780936_56.1", 2022-09-02T04:50:49.761 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.761 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:49.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.780936022Z", 2022-09-02T04:50:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.783480_57.1", 2022-09-02T04:50:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.766 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:49.766 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.783480461Z", 2022-09-02T04:50:49.766 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.767 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.767 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.767 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.768 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.769 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:49.769 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.770 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.770 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.770 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.771 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.784364_58.1", 2022-09-02T04:50:49.772 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.772 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11", 2022-09-02T04:50:49.772 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.784364362Z", 2022-09-02T04:50:49.772 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.773 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.774 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.774 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.775 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.775 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:49.776 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.776 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.778 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.780 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.786791_59.1", 2022-09-02T04:50:49.780 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.780 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11", 2022-09-02T04:50:49.781 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.786791946Z", 2022-09-02T04:50:49.781 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.781 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.781 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.782 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.782 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.783 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:49.783 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.783 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.784 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.786 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.818066_60.1", 2022-09-02T04:50:49.786 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.786 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:49.787 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.818066873Z", 2022-09-02T04:50:49.787 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.787 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.787 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.788 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.789 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.790 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:49.790 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.791 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.791 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.792 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.820503_61.1", 2022-09-02T04:50:49.793 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.793 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:49.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.820503127Z", 2022-09-02T04:50:49.794 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.794 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.795 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.795 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.796 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.797 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:49.797 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.798 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.798 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.808 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.821417_62.1", 2022-09-02T04:50:49.809 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.809 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12", 2022-09-02T04:50:49.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.821417990Z", 2022-09-02T04:50:49.810 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.811 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.811 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.812 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.813 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.813 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:49.814 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.817 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.818 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.821 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.823736_63.1", 2022-09-02T04:50:49.822 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.823 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12", 2022-09-02T04:50:49.827 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.823736708Z", 2022-09-02T04:50:49.827 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.828 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.828 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.829 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.829 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.830 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:49.831 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.832 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.832 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.835 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.494157_68.1", 2022-09-02T04:50:49.835 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.836 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:49.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.494157445Z", 2022-09-02T04:50:49.837 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.837 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.838 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.839 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.840 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.840 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:49.841 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.842 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.842 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.843 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.846 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.500451_69.1", 2022-09-02T04:50:49.846 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.847 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:49.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.500451205Z", 2022-09-02T04:50:49.848 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.848 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.848 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.849 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.850 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.851 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:49.851 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.852 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.852 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.854 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.502516_70.1", 2022-09-02T04:50:49.854 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.855 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13", 2022-09-02T04:50:49.856 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.502516287Z", 2022-09-02T04:50:49.856 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.857 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.857 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.858 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.859 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.859 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:49.859 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.860 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.860 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.863 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.508242_71.1", 2022-09-02T04:50:49.863 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.864 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13", 2022-09-02T04:50:49.864 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.508242760Z", 2022-09-02T04:50:49.865 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.865 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.865 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.866 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.867 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.867 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:49.868 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.869 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.869 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.563183_72.1", 2022-09-02T04:50:49.872 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:49.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.563183823Z", 2022-09-02T04:50:49.874 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.875 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.877 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.877 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:49.878 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.879 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.879 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.882 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.571784_73.1", 2022-09-02T04:50:49.882 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.883 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:49.883 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.571784929Z", 2022-09-02T04:50:49.884 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.884 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.884 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.885 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.886 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.887 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:49.887 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.889 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.891 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.572807_74.1", 2022-09-02T04:50:49.892 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.892 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14", 2022-09-02T04:50:49.893 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.572807912Z", 2022-09-02T04:50:49.893 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.894 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.894 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.895 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.896 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.897 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:49.897 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.900 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.900 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.901 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.578754_75.1", 2022-09-02T04:50:49.902 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.902 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14", 2022-09-02T04:50:49.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.578754707Z", 2022-09-02T04:50:49.903 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.903 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.903 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.903 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.904 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.904 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:49.905 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.905 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.906 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.907 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.659538_76.1", 2022-09-02T04:50:49.907 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.908 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:49.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.659538083Z", 2022-09-02T04:50:49.908 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.909 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.909 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.909 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.910 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:49.910 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.911 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.911 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.913 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.662121_77.1", 2022-09-02T04:50:49.913 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.913 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:49.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.662121126Z", 2022-09-02T04:50:49.914 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.914 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.914 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.915 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.915 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.915 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:49.916 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.916 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.918 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.663114_78.1", 2022-09-02T04:50:49.918 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.918 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15", 2022-09-02T04:50:49.919 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.663114920Z", 2022-09-02T04:50:49.919 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.919 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.919 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.920 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.920 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.921 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:49.921 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.921 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.922 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.923 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.665862_79.1", 2022-09-02T04:50:49.923 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.923 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15", 2022-09-02T04:50:49.924 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.665862159Z", 2022-09-02T04:50:49.924 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.924 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.924 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.925 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.925 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.926 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:49.926 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.927 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.927 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.927 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.927 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.928 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.289122_89.1", 2022-09-02T04:50:49.929 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.929 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:49.929 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.289122249Z", 2022-09-02T04:50:49.929 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.930 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.930 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.930 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.931 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.931 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:49.931 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.933 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.293860_90.1", 2022-09-02T04:50:49.933 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.934 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:49.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.293860164Z", 2022-09-02T04:50:49.935 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.935 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.935 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.936 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.936 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.937 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:49.937 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.938 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.939 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.295284_91.1", 2022-09-02T04:50:49.939 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.940 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16", 2022-09-02T04:50:49.940 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.295284709Z", 2022-09-02T04:50:49.940 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.940 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.940 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.941 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.941 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.942 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:49.942 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.943 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.944 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.298399_92.1", 2022-09-02T04:50:49.944 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.944 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16", 2022-09-02T04:50:49.945 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.298399993Z", 2022-09-02T04:50:49.945 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.945 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.946 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.946 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.946 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:49.947 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.947 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.948 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.949 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.327712_93.1", 2022-09-02T04:50:49.949 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.949 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:49.950 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.327712318Z", 2022-09-02T04:50:49.950 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.950 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.950 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.951 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.951 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.951 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:49.952 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.952 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.952 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.954 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.332779_94.1", 2022-09-02T04:50:49.954 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.955 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:49.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.332779056Z", 2022-09-02T04:50:49.956 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.956 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.956 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.957 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.958 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.958 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:49.959 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.959 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.960 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.960 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.962 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.334588_95.1", 2022-09-02T04:50:49.963 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.963 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17", 2022-09-02T04:50:49.963 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.334588743Z", 2022-09-02T04:50:49.964 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.964 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.965 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.965 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.966 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.967 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:49.967 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.968 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.969 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.971 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.337431_96.1", 2022-09-02T04:50:49.971 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.972 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17", 2022-09-02T04:50:49.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.337431441Z", 2022-09-02T04:50:49.972 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.972 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.973 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.973 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.974 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.974 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:49.974 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.975 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.975 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.976 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.365816_97.1", 2022-09-02T04:50:49.977 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.977 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:49.978 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.365816803Z", 2022-09-02T04:50:49.978 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.979 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.979 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.980 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.981 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.981 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:49.982 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.983 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.983 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.984 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.986 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.371582_98.1", 2022-09-02T04:50:49.986 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:49.986 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:49.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.371582029Z", 2022-09-02T04:50:49.987 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.987 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.987 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.988 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.988 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.988 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:49.989 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.989 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.991 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.373810_99.1", 2022-09-02T04:50:49.991 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.991 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18", 2022-09-02T04:50:49.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.373810553Z", 2022-09-02T04:50:49.992 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.992 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.992 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.993 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.993 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.993 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:49.994 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.994 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.994 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:49.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:49.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:49.996 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.380421_100.1", 2022-09-02T04:50:49.996 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:49.996 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18", 2022-09-02T04:50:49.996 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.380421451Z", 2022-09-02T04:50:49.997 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:49.997 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:49.997 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:49.997 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:49.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.998 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:49.998 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:49.998 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:49.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:49.999 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:49.999 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:49.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.001 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.406502_101.1", 2022-09-02T04:50:50.001 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.001 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:50.001 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.406502378Z", 2022-09-02T04:50:50.002 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.002 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.002 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:50.002 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.003 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.003 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:50.003 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.004 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.005 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.006 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.409376_102.1", 2022-09-02T04:50:50.006 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.007 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:50.007 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.409376514Z", 2022-09-02T04:50:50.007 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.007 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.008 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:50.008 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.008 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.009 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:50.009 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.011 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.424417_103.1", 2022-09-02T04:50:50.011 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.012 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:50.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.424417536Z", 2022-09-02T04:50:50.012 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.012 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.013 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:50.013 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.013 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.014 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:50.014 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.014 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.015 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.016 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.428769_104.1", 2022-09-02T04:50:50.016 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.017 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:50.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.428769794Z", 2022-09-02T04:50:50.017 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.017 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.018 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:50.018 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.018 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.019 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:50.019 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.019 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.020 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.020 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.020 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.021 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.448268_105.1", 2022-09-02T04:50:50.021 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.022 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:50.022 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.448268867Z", 2022-09-02T04:50:50.022 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.022 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.023 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:50.023 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.023 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.024 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:50.024 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.024 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.025 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.026 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.462134_106.1", 2022-09-02T04:50:50.026 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.026 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:50.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.462134251Z", 2022-09-02T04:50:50.027 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.031 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.839577_129.1", 2022-09-02T04:50:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.839577631Z", 2022-09-02T04:50:50.032 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.033 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.033 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.033 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.034 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.034 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:50.034 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.843007_130.1", 2022-09-02T04:50:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:50.037 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.843007979Z", 2022-09-02T04:50:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.038 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.039 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.039 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.040 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.042 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.844233_131.1", 2022-09-02T04:50:50.043 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.043 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19", 2022-09-02T04:50:50.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.844233977Z", 2022-09-02T04:50:50.043 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.044 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.044 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.044 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.045 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.045 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:50.045 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.046 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.046 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.047 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.847167_132.1", 2022-09-02T04:50:50.048 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.048 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19", 2022-09-02T04:50:50.048 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.847167513Z", 2022-09-02T04:50:50.049 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.049 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.049 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.049 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.050 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.050 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:50.050 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.053 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.863844_133.1", 2022-09-02T04:50:50.053 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.053 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:50.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.863844546Z", 2022-09-02T04:50:50.054 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.054 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.054 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:50.054 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.055 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.055 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:50.055 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.056 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.056 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.058 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.866884_134.1", 2022-09-02T04:50:50.058 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.058 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:50.059 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.866884082Z", 2022-09-02T04:50:50.059 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.059 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.059 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:50.060 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.060 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.060 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:50.061 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.061 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.061 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.063 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.478355_147.1", 2022-09-02T04:50:50.063 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.063 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:50.064 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.478355264Z", 2022-09-02T04:50:50.064 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.064 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.064 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.065 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.065 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.065 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.066 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.066 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.066 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.068 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.480846_151.1", 2022-09-02T04:50:50.068 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.069 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:50.069 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.480846805Z", 2022-09-02T04:50:50.069 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.069 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.070 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.070 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.071 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.071 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.071 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.072 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.072 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.072 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.072 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.073 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.481682_152.1", 2022-09-02T04:50:50.073 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.074 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20", 2022-09-02T04:50:50.074 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.481682426Z", 2022-09-02T04:50:50.074 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.074 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.075 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.075 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.076 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.076 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:50.076 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.077 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.077 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.078 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.483984_153.1", 2022-09-02T04:50:50.078 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.079 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20", 2022-09-02T04:50:50.079 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.483984075Z", 2022-09-02T04:50:50.079 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.079 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.080 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.080 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.080 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.081 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:50.081 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.081 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.082 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.083 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093786.258112_154.1", 2022-09-02T04:50:50.084 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.084 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:50.084 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:43:06.258112912Z", 2022-09-02T04:50:50.084 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.085 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.085 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.085 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.086 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.086 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.087 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.087 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.087 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.089 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093786.261185_155.1", 2022-09-02T04:50:50.089 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.089 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:50.089 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:43:06.261185635Z", 2022-09-02T04:50:50.090 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.090 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.090 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.091 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.091 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.091 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.092 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.092 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.092 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.094 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093904.758354_156.1", 2022-09-02T04:50:50.094 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.094 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:50.095 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:45:04.758354256Z", 2022-09-02T04:50:50.095 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.095 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.096 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.096 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.096 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.097 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.097 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.097 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.098 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.099 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093904.761841_157.1", 2022-09-02T04:50:50.099 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.099 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:50.100 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:45:04.761841925Z", 2022-09-02T04:50:50.100 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.101 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.101 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.101 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.102 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.102 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:50.102 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.103 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.103 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.105 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.643237_158.1", 2022-09-02T04:50:50.105 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.106 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:50.106 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.643237395Z", 2022-09-02T04:50:50.106 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.106 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.107 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.107 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.107 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.108 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:50.108 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.108 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.109 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.646348_159.1", 2022-09-02T04:50:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:50.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.646348674Z", 2022-09-02T04:50:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.112 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:50.113 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.647493_160.1", 2022-09-02T04:50:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21", 2022-09-02T04:50:50.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.647493391Z", 2022-09-02T04:50:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.117 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:50.118 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.119 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.120 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.650453_161.1", 2022-09-02T04:50:50.121 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.121 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21", 2022-09-02T04:50:50.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.650453266Z", 2022-09-02T04:50:50.122 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.122 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.122 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.123 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.123 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.123 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:50.124 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.125 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.126 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.671385_162.1", 2022-09-02T04:50:50.127 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.128 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:50.128 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.671385092Z", 2022-09-02T04:50:50.129 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.129 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.130 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.130 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.131 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.131 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:50.132 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.133 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.133 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.134 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.134 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.136 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.677725_163.1", 2022-09-02T04:50:50.136 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.137 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:50.137 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.677725006Z", 2022-09-02T04:50:50.138 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.138 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.139 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.139 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.140 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.140 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:50.141 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.142 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.142 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.144 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.679743_164.1", 2022-09-02T04:50:50.145 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.145 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22", 2022-09-02T04:50:50.146 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.679743170Z", 2022-09-02T04:50:50.146 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.147 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.147 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.148 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.148 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.149 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:50.149 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.150 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.151 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.153 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.685252_165.1", 2022-09-02T04:50:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22", 2022-09-02T04:50:50.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.685252272Z", 2022-09-02T04:50:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.155 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.156 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.159 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.720122_166.1", 2022-09-02T04:50:50.159 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.720122481Z", 2022-09-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.160 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.161 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.162 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.722787_167.1", 2022-09-02T04:50:50.164 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.165 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:50.165 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.722787347Z", 2022-09-02T04:50:50.165 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.166 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.166 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.166 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.167 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.167 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:50.167 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.168 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.168 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.168 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.170 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.723810_168.1", 2022-09-02T04:50:50.170 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.170 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23", 2022-09-02T04:50:50.170 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.723810483Z", 2022-09-02T04:50:50.171 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.171 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.171 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.172 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.172 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.172 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:50.173 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.173 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.173 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.175 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.726983_169.1", 2022-09-02T04:50:50.175 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.175 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23", 2022-09-02T04:50:50.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.726983239Z", 2022-09-02T04:50:50.176 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.176 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.176 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.177 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.177 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.177 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:50.178 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.180 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.748449_170.1", 2022-09-02T04:50:50.180 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.180 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:50.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.748449921Z", 2022-09-02T04:50:50.181 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.181 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.181 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:50.182 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.182 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:50.183 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.185 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.752174_171.1", 2022-09-02T04:50:50.185 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.185 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:50.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.752174857Z", 2022-09-02T04:50:50.186 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.186 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.186 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:50.186 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.187 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.187 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:50.187 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.189 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.772370_172.1", 2022-09-02T04:50:50.190 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.190 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:50.190 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.772370930Z", 2022-09-02T04:50:50.191 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.191 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.191 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:50.192 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.192 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.193 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:50.193 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.193 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.194 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.195 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.775210_173.1", 2022-09-02T04:50:50.195 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.196 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:50.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.775210687Z", 2022-09-02T04:50:50.196 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.196 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.196 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:50.197 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.197 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.198 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:50.198 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.198 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.200 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.796060_174.1", 2022-09-02T04:50:50.200 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.200 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:50.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.796060843Z", 2022-09-02T04:50:50.201 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.201 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.201 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:50.202 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.202 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.203 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:50.203 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.203 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.204 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.205 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.799397_175.1", 2022-09-02T04:50:50.205 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.206 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:50.206 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.799397586Z", 2022-09-02T04:50:50.206 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.206 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.207 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:50.207 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.207 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.208 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:50.208 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.209 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.209 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.210 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.480028_178.1", 2022-09-02T04:50:50.211 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.211 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:50.211 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.480028626Z", 2022-09-02T04:50:50.212 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.212 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.212 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.212 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.213 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.213 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.213 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.214 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.214 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.216 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.483732_179.1", 2022-09-02T04:50:50.216 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.217 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:50.217 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.483732364Z", 2022-09-02T04:50:50.218 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.218 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.219 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.219 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.220 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.220 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.221 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.222 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.222 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.225 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.484607_180.1", 2022-09-02T04:50:50.225 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.226 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24", 2022-09-02T04:50:50.226 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.484607016Z", 2022-09-02T04:50:50.226 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.227 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.228 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.229 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.230 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.232 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.487046_181.1", 2022-09-02T04:50:50.232 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.233 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24", 2022-09-02T04:50:50.233 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.487046711Z", 2022-09-02T04:50:50.233 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.233 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.234 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.234 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.234 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.235 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:50.235 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.236 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.236 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.237 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.508152_182.1", 2022-09-02T04:50:50.237 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.238 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:50.238 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.508152912Z", 2022-09-02T04:50:50.238 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.511557_183.1", 2022-09-02T04:50:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.511557664Z", 2022-09-02T04:50:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.512912_184.1", 2022-09-02T04:50:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25", 2022-09-02T04:50:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.512912424Z", 2022-09-02T04:50:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.250 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.250 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:50.250 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.251 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.251 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.252 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.253 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.515118_185.1", 2022-09-02T04:50:50.253 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.253 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25", 2022-09-02T04:50:50.253 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.515118385Z", 2022-09-02T04:50:50.254 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.254 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.254 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.254 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.255 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.255 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:50.255 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.256 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.256 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.257 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.257 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.259 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.550248_186.1", 2022-09-02T04:50:50.259 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.260 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:50.260 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.550248103Z", 2022-09-02T04:50:50.261 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.261 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.262 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.262 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.263 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.264 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.264 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.265 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.266 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.268 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.552820_187.1", 2022-09-02T04:50:50.269 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.269 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:50.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.552820973Z", 2022-09-02T04:50:50.271 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.271 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.272 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.272 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.273 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.273 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.274 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.275 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.276 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.277 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.279 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.553829_188.1", 2022-09-02T04:50:50.279 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.280 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26", 2022-09-02T04:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.553829203Z", 2022-09-02T04:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.281 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.282 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.283 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.284 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:50.284 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.285 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.285 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.287 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.556273_189.1", 2022-09-02T04:50:50.287 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.287 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26", 2022-09-02T04:50:50.287 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.556273838Z", 2022-09-02T04:50:50.288 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.288 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.288 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.289 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.289 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:50.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.290 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.292 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.583634_190.1", 2022-09-02T04:50:50.292 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.292 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:50.293 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.583634061Z", 2022-09-02T04:50:50.293 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.293 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.294 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.294 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.294 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.295 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.295 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.295 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.297 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.587708_191.1", 2022-09-02T04:50:50.297 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.298 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:50.298 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.587708089Z", 2022-09-02T04:50:50.298 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.299 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.299 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.299 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.300 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.301 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.301 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.302 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.303 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.305 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.607965_192.1", 2022-09-02T04:50:50.306 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.306 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:50.306 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.607965005Z", 2022-09-02T04:50:50.307 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.307 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.307 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.307 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.308 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.308 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.308 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.309 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.309 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.311 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.614764_193.1", 2022-09-02T04:50:50.311 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.311 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:50.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.614764199Z", 2022-09-02T04:50:50.312 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.312 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.312 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.313 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.313 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.313 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:50.314 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.314 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.316 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.635864_194.1", 2022-09-02T04:50:50.316 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.316 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:50.317 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.635864908Z", 2022-09-02T04:50:50.317 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.317 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.317 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.318 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.318 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.318 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.319 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.319 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.319 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.321 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.640272_195.1", 2022-09-02T04:50:50.321 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.321 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:50.322 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.640272673Z", 2022-09-02T04:50:50.322 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.322 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.323 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.323 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.323 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.324 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.324 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.324 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.325 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.325 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.326 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.661758_196.1", 2022-09-02T04:50:50.326 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.327 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:50.327 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.661758657Z", 2022-09-02T04:50:50.327 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.328 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.328 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.329 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.330 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.330 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.331 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.335 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.664046_197.1", 2022-09-02T04:50:50.335 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.335 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:50.336 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.664046043Z", 2022-09-02T04:50:50.336 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.336 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.337 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.337 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.337 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.338 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:50.338 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.342 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.683056_198.1", 2022-09-02T04:50:50.343 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.343 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:50.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.683056320Z", 2022-09-02T04:50:50.344 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.344 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.345 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.345 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.347 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.347 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.347 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.348 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.349 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.352 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.685741_199.1", 2022-09-02T04:50:50.352 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.353 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:50.353 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.685741330Z", 2022-09-02T04:50:50.354 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.354 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.355 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.355 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.356 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.357 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.358 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.359 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.359 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.362 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.701672_200.1", 2022-09-02T04:50:50.362 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.363 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:50.363 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.701672940Z", 2022-09-02T04:50:50.364 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.364 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.365 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.365 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.366 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.367 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.367 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.369 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.369 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.372 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.711326_201.1", 2022-09-02T04:50:50.373 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.373 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:50.374 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.711326169Z", 2022-09-02T04:50:50.374 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.375 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.375 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.376 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.377 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.377 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:50.378 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:50.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.379 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.380 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.382 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.596190_217.1", 2022-09-02T04:50:50.382 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.383 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:50.383 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.596190558Z", 2022-09-02T04:50:50.384 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.384 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.385 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.385 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.386 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:50.387 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.388 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.599340_218.1", 2022-09-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.391 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.599340364Z", 2022-09-02T04:50:50.392 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.600264_219.1", 2022-09-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27", 2022-09-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.600264022Z", 2022-09-02T04:50:50.401 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.402 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.403 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.404 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:50.405 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.406 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.602614_220.1", 2022-09-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.409 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27", 2022-09-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.602614533Z", 2022-09-02T04:50:50.410 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.412 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.415 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.624417_221.1", 2022-09-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.416 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.624417644Z", 2022-09-02T04:50:50.417 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.418 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:50.420 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.628627_222.1", 2022-09-02T04:50:50.424 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.628627812Z", 2022-09-02T04:50:50.425 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.629582_223.1", 2022-09-02T04:50:50.429 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28", 2022-09-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.629582051Z", 2022-09-02T04:50:50.430 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.634049_224.1", 2022-09-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28", 2022-09-02T04:50:50.435 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.634049176Z", 2022-09-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.436 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.437 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:50.438 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.439 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.674265_225.1", 2022-09-02T04:50:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.674265806Z", 2022-09-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.456 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.465 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.470 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.472 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.680208_226.1", 2022-09-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.475 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.680208008Z", 2022-09-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.476 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.477 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.478 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.479 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.482 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.682665_227.1", 2022-09-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.483 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29", 2022-09-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.682665330Z", 2022-09-02T04:50:50.485 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.486 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.487 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.488 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.489 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:50.490 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.491 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.492 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.494 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.686711_228.1", 2022-09-02T04:50:50.495 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29", 2022-09-02T04:50:50.496 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.686711368Z", 2022-09-02T04:50:50.497 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.498 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.499 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:50.500 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.502 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.504 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.718596_229.1", 2022-09-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.505 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:50.506 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.718596260Z", 2022-09-02T04:50:50.507 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.508 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.510 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:50.511 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.513 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.514 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.722825_230.1", 2022-09-02T04:50:50.520 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:50.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.722825082Z", 2022-09-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.523 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.525 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.527 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.528 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.529 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.765767_231.1", 2022-09-02T04:50:50.530 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:50.531 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.765767335Z", 2022-09-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.532 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:50.533 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:50.534 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.535 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.768248_232.1", 2022-09-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.537 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.768248326Z", 2022-09-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.538 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:50.539 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.541 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:50.542 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.543 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.795442_233.1", 2022-09-02T04:50:50.546 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:50.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.795442979Z", 2022-09-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.548 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.550 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.554 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.797953_234.1", 2022-09-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.555 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.797953373Z", 2022-09-02T04:50:50.556 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.557 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.559 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.560 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.563 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.841868_235.1", 2022-09-02T04:50:50.564 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.564 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:50.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.841868706Z", 2022-09-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.565 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.566 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.566 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.567 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.567 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:50.568 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.568 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.569 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.571 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.846202_236.1", 2022-09-02T04:50:50.571 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.572 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:50.572 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.846202210Z", 2022-09-02T04:50:50.573 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.573 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.574 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.574 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.575 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.575 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:50.576 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.577 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.577 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.579 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.847338_237.1", 2022-09-02T04:50:50.580 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.580 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30", 2022-09-02T04:50:50.581 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.847338068Z", 2022-09-02T04:50:50.581 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.581 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.582 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.582 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.583 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.583 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:50.584 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.585 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.585 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.588 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.851413_238.1", 2022-09-02T04:50:50.588 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.588 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30", 2022-09-02T04:50:50.589 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.851413431Z", 2022-09-02T04:50:50.589 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.590 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.590 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.591 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.592 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.592 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:50.592 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.593 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.594 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.595 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.596 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.882196_239.1", 2022-09-02T04:50:50.596 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.597 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:50.597 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.882196828Z", 2022-09-02T04:50:50.598 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.598 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.598 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.599 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.600 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.600 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:50.601 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.601 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.886381_240.1", 2022-09-02T04:50:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.886381079Z", 2022-09-02T04:50:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.887265_241.1", 2022-09-02T04:50:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31", 2022-09-02T04:50:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.887265472Z", 2022-09-02T04:50:50.615 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.615 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.615 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.616 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.617 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.617 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:50.617 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.618 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.619 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.621 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.891145_242.1", 2022-09-02T04:50:50.621 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.622 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31", 2022-09-02T04:50:50.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.891145853Z", 2022-09-02T04:50:50.622 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.623 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.623 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.624 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.625 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.625 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:50.626 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.626 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.627 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.629 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.923935_243.1", 2022-09-02T04:50:50.629 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.630 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:50.630 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.923935375Z", 2022-09-02T04:50:50.631 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.631 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.631 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.632 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.633 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.634 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:50.634 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.635 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.635 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.638 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.927811_244.1", 2022-09-02T04:50:50.638 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.639 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:50.639 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.927811078Z", 2022-09-02T04:50:50.639 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.640 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.640 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.641 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.642 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.643 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:50.643 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.645 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.647 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.928867_245.1", 2022-09-02T04:50:50.648 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.648 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32", 2022-09-02T04:50:50.649 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.928867563Z", 2022-09-02T04:50:50.649 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.650 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.650 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.650 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.651 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.651 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:50.652 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.933635_246.1", 2022-09-02T04:50:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32", 2022-09-02T04:50:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.933635936Z", 2022-09-02T04:50:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.660 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.663 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.665 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.981889_247.1", 2022-09-02T04:50:50.666 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.666 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.981889010Z", 2022-09-02T04:50:50.667 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.669 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:50.670 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.671 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.673 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.674 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.984537_248.1", 2022-09-02T04:50:50.675 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.675 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:50.676 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.984537015Z", 2022-09-02T04:50:50.676 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.677 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.677 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:50.677 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.678 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.679 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:50.679 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.680 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.680 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.683 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.069441_249.1", 2022-09-02T04:50:50.683 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.683 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:50.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.069441995Z", 2022-09-02T04:50:50.684 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.685 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.685 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:50.685 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.078532_250.1", 2022-09-02T04:50:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.078532139Z", 2022-09-02T04:50:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.695 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:50.695 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.696 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.697 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.698 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.117134_251.1", 2022-09-02T04:50:50.699 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.699 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:50.700 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.117134511Z", 2022-09-02T04:50:50.700 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.701 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.701 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:50.701 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.702 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.703 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:50.703 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.704 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.705 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.706 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.707 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.121445_252.1", 2022-09-02T04:50:50.708 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.708 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:50.709 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.121445603Z", 2022-09-02T04:50:50.709 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.710 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.710 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:50.711 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.712 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.712 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:50.713 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.714 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.714 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.714 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.716 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.200772_253.1", 2022-09-02T04:50:50.716 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.716 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:50.716 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.200772049Z", 2022-09-02T04:50:50.716 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.717 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.717 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.718 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.718 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.719 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:50.719 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.204385_254.1", 2022-09-02T04:50:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.204385610Z", 2022-09-02T04:50:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.205324_255.1", 2022-09-02T04:50:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33", 2022-09-02T04:50:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.205324745Z", 2022-09-02T04:50:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.743 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.211398_256.1", 2022-09-02T04:50:50.744 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.744 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33", 2022-09-02T04:50:50.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.211398303Z", 2022-09-02T04:50:50.745 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.745 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.746 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.746 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.747 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.748 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:50.748 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.750 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.752 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.247878_257.1", 2022-09-02T04:50:50.753 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.753 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:50.754 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.247878027Z", 2022-09-02T04:50:50.754 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.755 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.755 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.756 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.757 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.757 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:50.757 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.762 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.251068_258.1", 2022-09-02T04:50:50.762 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.762 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:50.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.251068384Z", 2022-09-02T04:50:50.763 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.764 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.764 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.764 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.765 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.766 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:50.766 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.769 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.252037_259.1", 2022-09-02T04:50:50.769 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.770 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34", 2022-09-02T04:50:50.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.252037843Z", 2022-09-02T04:50:50.771 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.771 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.771 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.772 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.772 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.773 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:50.773 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.774 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.775 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.777 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.260507_260.1", 2022-09-02T04:50:50.778 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.778 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34", 2022-09-02T04:50:50.779 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.260507523Z", 2022-09-02T04:50:50.779 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.780 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.780 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.781 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.782 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.782 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:50.783 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.784 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.784 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.785 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.786 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.324260_261.1", 2022-09-02T04:50:50.787 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.787 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:50.787 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.324260578Z", 2022-09-02T04:50:50.788 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.788 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.789 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.789 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.790 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.791 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:50.791 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.792 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.793 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.795 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.328657_262.1", 2022-09-02T04:50:50.795 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.796 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:50.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.328657368Z", 2022-09-02T04:50:50.796 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.796 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.797 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.797 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.798 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.798 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:50.799 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.800 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.800 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.802 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.330962_263.1", 2022-09-02T04:50:50.803 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.803 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35", 2022-09-02T04:50:50.803 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.330962269Z", 2022-09-02T04:50:50.804 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.804 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.805 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.805 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.806 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.807 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:50.807 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.808 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.809 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.811 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.335420_264.1", 2022-09-02T04:50:50.811 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.812 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35", 2022-09-02T04:50:50.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.335420823Z", 2022-09-02T04:50:50.813 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.813 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.814 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.814 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.815 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.815 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:50.815 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.816 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.816 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.818 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.389275_265.1", 2022-09-02T04:50:50.818 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.818 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:50.819 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.389275554Z", 2022-09-02T04:50:50.819 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.819 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.820 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:50.820 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.821 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.822 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:50.822 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.823 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.824 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.827 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.393872_266.1", 2022-09-02T04:50:50.827 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.828 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:50.828 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.393872877Z", 2022-09-02T04:50:50.829 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.829 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.829 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:50.829 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.830 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.830 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:50.831 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.833 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094086.434234_267.1", 2022-09-02T04:50:50.833 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.833 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:50.833 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:06.434234679Z", 2022-09-02T04:50:50.834 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.834 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.835 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:50.835 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.836 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.836 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:50.836 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.837 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.838 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.839 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.840 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094086.438494_268.1", 2022-09-02T04:50:50.841 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.841 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:50.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:06.438494895Z", 2022-09-02T04:50:50.842 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.842 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.843 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:50.843 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.844 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.845 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:50.845 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.846 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.847 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.849 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.522239_269.1", 2022-09-02T04:50:50.850 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.850 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:50.850 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.522239231Z", 2022-09-02T04:50:50.850 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.526590_270.1", 2022-09-02T04:50:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.526590558Z", 2022-09-02T04:50:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.588437_271.1", 2022-09-02T04:50:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.588437443Z", 2022-09-02T04:50:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.594122_272.1", 2022-09-02T04:50:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.594122094Z", 2022-09-02T04:50:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.596187_273.1", 2022-09-02T04:50:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36", 2022-09-02T04:50:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.596187284Z", 2022-09-02T04:50:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.601998_274.1", 2022-09-02T04:50:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36", 2022-09-02T04:50:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.601998619Z", 2022-09-02T04:50:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.638123_275.1", 2022-09-02T04:50:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.638123665Z", 2022-09-02T04:50:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.890 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.640624_276.1", 2022-09-02T04:50:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.640624183Z", 2022-09-02T04:50:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.641616_277.1", 2022-09-02T04:50:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37", 2022-09-02T04:50:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.641616287Z", 2022-09-02T04:50:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.643956_278.1", 2022-09-02T04:50:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37", 2022-09-02T04:50:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.643956748Z", 2022-09-02T04:50:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.689255_279.1", 2022-09-02T04:50:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.689255290Z", 2022-09-02T04:50:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.691696_280.1", 2022-09-02T04:50:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.691696253Z", 2022-09-02T04:50:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.692552_281.1", 2022-09-02T04:50:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38", 2022-09-02T04:50:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.692552094Z", 2022-09-02T04:50:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.696193_282.1", 2022-09-02T04:50:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38", 2022-09-02T04:50:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.696193220Z", 2022-09-02T04:50:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.042350_283.1", 2022-09-02T04:50:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.042350878Z", 2022-09-02T04:50:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.045432_284.1", 2022-09-02T04:50:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.045432525Z", 2022-09-02T04:50:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.047334_285.1", 2022-09-02T04:50:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39", 2022-09-02T04:50:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.047334461Z", 2022-09-02T04:50:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.050679_286.1", 2022-09-02T04:50:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39", 2022-09-02T04:50:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.050679259Z", 2022-09-02T04:50:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.075859_287.1", 2022-09-02T04:50:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.075859654Z", 2022-09-02T04:50:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.963 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.963 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.964 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.965 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:50.965 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.966 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.966 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.967 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.967 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.968 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.078656_288.1", 2022-09-02T04:50:50.969 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.970 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:50.970 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.078656456Z", 2022-09-02T04:50:50.971 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.971 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.972 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.972 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.973 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.973 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:50.973 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.974 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.974 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.976 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.080841_289.1", 2022-09-02T04:50:50.976 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.976 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40", 2022-09-02T04:50:50.976 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.080841716Z", 2022-09-02T04:50:50.977 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.977 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.977 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.977 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.979 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.979 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.980 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.980 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.981 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.086877_290.1", 2022-09-02T04:50:50.981 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.982 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40", 2022-09-02T04:50:50.982 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.086877398Z", 2022-09-02T04:50:50.983 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.983 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.983 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.983 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.984 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.984 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:50.984 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.985 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.985 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.987 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.125974_291.1", 2022-09-02T04:50:50.988 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.988 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:50.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.125974705Z", 2022-09-02T04:50:50.988 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.989 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.989 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.989 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.990 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.990 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:50.990 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.991 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.991 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:50.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.992 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.132649_292.1", 2022-09-02T04:50:50.992 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:50.993 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:50.993 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.132649142Z", 2022-09-02T04:50:50.993 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.993 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.994 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:50.994 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:50.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.994 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:50.995 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:50.995 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:50.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.995 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:50.996 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:50.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:50.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:50.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:50.997 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.133657_293.1", 2022-09-02T04:50:50.997 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:50.998 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41", 2022-09-02T04:50:50.999 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.133657858Z", 2022-09-02T04:50:50.999 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:50.999 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:50.999 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.000 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.000 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.000 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:51.001 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.001 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.001 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.003 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.139676_294.1", 2022-09-02T04:50:51.003 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.003 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41", 2022-09-02T04:50:51.004 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.139676296Z", 2022-09-02T04:50:51.004 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.004 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.005 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.005 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.005 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.006 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:51.006 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.010 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.573151_93.1", 2022-09-02T04:50:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:51.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.573151032Z", 2022-09-02T04:50:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.012 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.013 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.014 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.015 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:51.015 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.016 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.016 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.017 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.579025_94.1", 2022-09-02T04:50:51.018 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.018 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:51.018 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.579025648Z", 2022-09-02T04:50:51.019 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.019 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.019 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.020 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.021 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.021 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:51.022 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.023 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.024 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.024 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.026 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.581233_95.1", 2022-09-02T04:50:51.026 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.027 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1", 2022-09-02T04:50:51.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.581233957Z", 2022-09-02T04:50:51.028 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.028 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.029 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.029 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.030 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.585812_96.1", 2022-09-02T04:50:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1", 2022-09-02T04:50:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.585812830Z", 2022-09-02T04:50:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.617016_97.1", 2022-09-02T04:50:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.617016945Z", 2022-09-02T04:50:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.048 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.048 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.049 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.050 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.050 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:51.051 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.054 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.620981_98.1", 2022-09-02T04:50:51.055 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.055 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:51.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.620981265Z", 2022-09-02T04:50:51.056 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.057 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.057 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.057 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.058 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.058 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:51.059 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.060 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.060 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.063 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.622018_99.1", 2022-09-02T04:50:51.064 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.064 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2", 2022-09-02T04:50:51.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.622018558Z", 2022-09-02T04:50:51.065 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.066 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.067 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.067 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.068 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.069 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:51.069 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.070 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.071 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.072 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.073 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.624757_100.1", 2022-09-02T04:50:51.074 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.074 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2", 2022-09-02T04:50:51.075 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.624757525Z", 2022-09-02T04:50:51.075 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.076 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.076 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.077 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.078 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.078 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:51.079 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.083 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.630272_101.1", 2022-09-02T04:50:51.083 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.084 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:51.084 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.630272240Z", 2022-09-02T04:50:51.085 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.085 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.086 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.086 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.088 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.088 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:51.089 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.090 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.090 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.091 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.091 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.093 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.633085_102.1", 2022-09-02T04:50:51.093 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.094 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:51.094 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.633085325Z", 2022-09-02T04:50:51.095 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.095 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.096 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.096 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.097 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.098 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:51.098 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.106 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.106 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.108 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.634091_103.1", 2022-09-02T04:50:51.109 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.109 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3", 2022-09-02T04:50:51.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.634091584Z", 2022-09-02T04:50:51.111 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.111 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.112 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.112 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.113 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.113 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:51.113 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.116 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.636577_104.1", 2022-09-02T04:50:51.117 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.117 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3", 2022-09-02T04:50:51.118 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.636577040Z", 2022-09-02T04:50:51.119 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.119 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.120 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.120 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.121 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.121 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:51.121 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.122 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.122 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.123 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.570754_105.1", 2022-09-02T04:50:51.124 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.124 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:51.124 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.570754424Z", 2022-09-02T04:50:51.124 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.125 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.125 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.125 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.126 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.126 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:51.127 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.128 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.128 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.131 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.573776_106.1", 2022-09-02T04:50:51.131 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.131 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:51.132 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.573776177Z", 2022-09-02T04:50:51.132 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.133 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.133 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.134 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.134 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.135 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:51.135 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.136 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.137 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.139 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.574795_107.1", 2022-09-02T04:50:51.139 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.140 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4", 2022-09-02T04:50:51.140 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.574795537Z", 2022-09-02T04:50:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.143 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.143 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:51.144 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.144 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.145 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.147 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.577651_108.1", 2022-09-02T04:50:51.147 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.148 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4", 2022-09-02T04:50:51.148 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.577651116Z", 2022-09-02T04:50:51.149 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.149 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.149 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.150 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.151 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.151 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:51.152 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.153 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.153 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.155 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.609918_109.1", 2022-09-02T04:50:51.155 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.155 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:51.156 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.609918035Z", 2022-09-02T04:50:51.156 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.156 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.156 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.157 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.157 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.157 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:51.158 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.160 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.613337_110.1", 2022-09-02T04:50:51.160 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.160 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:51.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.613337726Z", 2022-09-02T04:50:51.161 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.161 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.161 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.162 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.162 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.162 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:51.163 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.165 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.614272_111.1", 2022-09-02T04:50:51.165 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.165 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5", 2022-09-02T04:50:51.165 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.614272060Z", 2022-09-02T04:50:51.166 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.166 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.166 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.167 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.167 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.167 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:51.168 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.168 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.168 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.170 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.618746_112.1", 2022-09-02T04:50:51.170 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.170 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5", 2022-09-02T04:50:51.170 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.618746992Z", 2022-09-02T04:50:51.171 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.171 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.171 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.171 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.172 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.172 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:51.172 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.173 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.173 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.173 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.174 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.654802_113.1", 2022-09-02T04:50:51.175 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.175 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:51.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.654802951Z", 2022-09-02T04:50:51.175 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.657918_114.1", 2022-09-02T04:50:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:51.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.657918674Z", 2022-09-02T04:50:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.181 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.182 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:51.182 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.184 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.660134_115.1", 2022-09-02T04:50:51.185 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.185 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6", 2022-09-02T04:50:51.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.660134944Z", 2022-09-02T04:50:51.185 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.186 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.186 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.186 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.187 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.187 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:51.187 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.189 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.665018_116.1", 2022-09-02T04:50:51.189 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.190 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6", 2022-09-02T04:50:51.190 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.665018151Z", 2022-09-02T04:50:51.190 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.190 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.191 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.191 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.191 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.192 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:51.192 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.193 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.194 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.769933_117.1", 2022-09-02T04:50:51.194 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.194 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:51.195 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.769933252Z", 2022-09-02T04:50:51.195 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.196 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.196 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.196 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.197 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.198 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:51.198 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.203 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.772485_118.1", 2022-09-02T04:50:51.203 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.203 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:51.204 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.772485812Z", 2022-09-02T04:50:51.204 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.205 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.205 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.205 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.206 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.207 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:51.207 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.208 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.208 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.210 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.773480_119.1", 2022-09-02T04:50:51.210 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.211 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:51.211 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.773480228Z", 2022-09-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.212 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.213 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.214 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.214 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:51.214 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.215 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.216 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.218 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.776581_120.1", 2022-09-02T04:50:51.218 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.218 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:51.219 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.776581648Z", 2022-09-02T04:50:51.219 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.220 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.220 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.220 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.221 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.221 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:51.222 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.223 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.223 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.227 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.801099_121.1", 2022-09-02T04:50:51.227 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.227 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:51.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.801099540Z", 2022-09-02T04:50:51.228 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.229 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.229 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.229 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.230 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.230 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:51.231 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.232 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.232 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.232 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.234 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.804401_122.1", 2022-09-02T04:50:51.235 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.235 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:51.235 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.804401413Z", 2022-09-02T04:50:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.237 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.237 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.238 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:51.238 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.238 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.239 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.240 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.805277_123.1", 2022-09-02T04:50:51.240 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.240 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:51.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.805277428Z", 2022-09-02T04:50:51.241 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.241 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.241 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.242 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.242 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.242 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:51.243 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.243 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.243 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.245 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.807715_124.1", 2022-09-02T04:50:51.245 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.245 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:51.246 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.807715596Z", 2022-09-02T04:50:51.246 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.246 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.246 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.247 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.247 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.247 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:51.248 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.248 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.248 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.842266_125.1", 2022-09-02T04:50:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.842266349Z", 2022-09-02T04:50:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.853430_126.1", 2022-09-02T04:50:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.853430958Z", 2022-09-02T04:50:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.855671_127.1", 2022-09-02T04:50:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.855671199Z", 2022-09-02T04:50:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.862622_128.1", 2022-09-02T04:50:51.264 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.265 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:51.265 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.862622338Z", 2022-09-02T04:50:51.265 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.266 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.266 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:51.266 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:51.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.267 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.267 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:51.267 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.268 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.268 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.269 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.580527_129.1", 2022-09-02T04:50:51.269 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.270 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:51.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.580527505Z", 2022-09-02T04:50:51.270 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.270 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.271 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:51.271 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.271 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.272 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:51.272 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.272 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.273 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.274 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.583272_130.1", 2022-09-02T04:50:51.274 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.275 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:51.275 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.583272249Z", 2022-09-02T04:50:51.275 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.275 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.275 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:51.276 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.276 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.276 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:51.277 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.277 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.277 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.279 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.595303_131.1", 2022-09-02T04:50:51.279 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.279 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:51.280 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.595303694Z", 2022-09-02T04:50:51.280 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.280 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.280 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:51.281 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.281 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.281 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:51.282 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.282 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.282 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "remove" 2022-09-02T04:50:51.283 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.283 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.284 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.599169_132.1", 2022-09-02T04:50:51.284 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.284 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:51.284 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.599169915Z", 2022-09-02T04:50:51.285 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.285 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.285 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:51.286 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.288 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.288 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:51.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.290 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.294 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.638539_133.1", 2022-09-02T04:50:51.294 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.295 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:51.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.638539023Z", 2022-09-02T04:50:51.296 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.296 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.298 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:51.298 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.299 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.300 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:51.300 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.302 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.302 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.305 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.645764_134.1", 2022-09-02T04:50:51.305 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.305 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:51.306 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.645764849Z", 2022-09-02T04:50:51.306 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.306 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.306 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:51.307 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.307 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.307 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:51.308 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.308 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.308 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.310 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.659515_135.1", 2022-09-02T04:50:51.310 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.310 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:51.310 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.659515593Z", 2022-09-02T04:50:51.311 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.311 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.311 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:51.311 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.312 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.312 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:51.313 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.313 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.314 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "remove" 2022-09-02T04:50:51.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.662895_136.1", 2022-09-02T04:50:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.315 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.662895104Z", 2022-09-02T04:50:51.316 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.317 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.318 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.319 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.320 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.321 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.322 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.751579_137.1", 2022-09-02T04:50:51.322 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.322 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:51.323 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.751579938Z", 2022-09-02T04:50:51.323 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.323 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.323 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:51.324 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.324 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.324 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:51.325 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:51.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.328 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.754025_138.1", 2022-09-02T04:50:51.328 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.329 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:51.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.754025410Z", 2022-09-02T04:50:51.329 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.329 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.330 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:51.330 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.330 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.331 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:51.332 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.336 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.769139_139.1", 2022-09-02T04:50:51.336 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.337 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:51.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.769139327Z", 2022-09-02T04:50:51.338 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.338 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.339 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:51.339 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.340 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.341 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:51.341 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.342 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.343 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "remove" 2022-09-02T04:50:51.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:51.348 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.771218_140.1", 2022-09-02T04:50:51.349 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:51.349 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:51.350 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218052Z", 2022-09-02T04:50:51.350 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:51.351 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:51.351 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:51.352 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:51.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.353 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:51.353 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:51.354 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:51.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:51.355 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:51.355 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:51.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:51.357 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:51.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:51.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:51.366 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:51.424 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:51.425 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:51.472 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:50:51.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.474 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092292.275513_70.1", 2022-09-02T04:50:51.475 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2022-09-02T04:50:51.475 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:51.476 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.275513716Z", 2022-09-02T04:50:51.476 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.477 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.477 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.478 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.480 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.481 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:51.481 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.482 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.482 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.484 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092292.280197_71.1", 2022-09-02T04:50:51.485 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2022-09-02T04:50:51.485 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:51.485 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197578Z", 2022-09-02T04:50:51.486 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.486 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.486 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.486 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.487 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.487 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:51.488 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.488 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.488 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.490 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.750378_51.1", 2022-09-02T04:50:51.491 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.491 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:51.491 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.750378020Z", 2022-09-02T04:50:51.491 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.492 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.492 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.492 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.493 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.493 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:51.494 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.494 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.494 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.496 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.753068_53.1", 2022-09-02T04:50:51.496 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.496 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:51.496 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.753068141Z", 2022-09-02T04:50:51.497 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.497 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.497 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.497 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.498 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.498 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:51.498 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.499 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.499 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.500 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.501 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.754140_54.1", 2022-09-02T04:50:51.501 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.502 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10", 2022-09-02T04:50:51.502 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.754140083Z", 2022-09-02T04:50:51.502 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.503 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.503 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.503 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.504 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.504 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:51.504 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.505 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.505 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.506 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.506 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.756550_55.1", 2022-09-02T04:50:51.507 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.507 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10", 2022-09-02T04:50:51.507 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.756550326Z", 2022-09-02T04:50:51.507 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.508 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.508 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.508 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.509 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.509 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:51.509 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.510 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.510 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.511 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.512 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.780936_56.1", 2022-09-02T04:50:51.512 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.512 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:51.512 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.780936022Z", 2022-09-02T04:50:51.513 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.513 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.513 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.513 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.514 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.514 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:51.515 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.516 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.516 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.517 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.783480_57.1", 2022-09-02T04:50:51.517 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.518 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:51.518 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.783480461Z", 2022-09-02T04:50:51.518 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.518 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.519 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.519 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.519 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.520 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:51.520 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.520 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.522 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.784364_58.1", 2022-09-02T04:50:51.522 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11", 2022-09-02T04:50:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.784364362Z", 2022-09-02T04:50:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:51.525 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.527 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.786791_59.1", 2022-09-02T04:50:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11", 2022-09-02T04:50:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.786791946Z", 2022-09-02T04:50:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.530 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.530 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:51.531 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.531 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.531 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.818066_60.1", 2022-09-02T04:50:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:51.534 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.818066873Z", 2022-09-02T04:50:51.534 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.534 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.534 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.820503_61.1", 2022-09-02T04:50:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.820503127Z", 2022-09-02T04:50:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:51.541 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.541 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.541 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.542 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.542 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.543 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.821417_62.1", 2022-09-02T04:50:51.543 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.543 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12", 2022-09-02T04:50:51.544 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.821417990Z", 2022-09-02T04:50:51.544 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.544 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.544 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.544 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.545 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.545 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:51.545 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.546 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.546 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.547 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.823736_63.1", 2022-09-02T04:50:51.548 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.548 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12", 2022-09-02T04:50:51.549 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.823736708Z", 2022-09-02T04:50:51.549 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.549 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.549 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.550 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.550 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.550 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:51.551 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.552 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.552 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.553 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.494157_68.1", 2022-09-02T04:50:51.554 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.554 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:51.554 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.494157445Z", 2022-09-02T04:50:51.554 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.555 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.555 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.555 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.556 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.556 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:51.556 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.557 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.557 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.558 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.558 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.500451_69.1", 2022-09-02T04:50:51.559 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.559 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:51.559 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.500451205Z", 2022-09-02T04:50:51.559 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.560 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.560 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.561 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.561 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:51.561 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.562 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.562 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.562 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.562 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.563 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.502516_70.1", 2022-09-02T04:50:51.563 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.564 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13", 2022-09-02T04:50:51.564 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.502516287Z", 2022-09-02T04:50:51.564 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.565 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.566 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.567 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.508242_71.1", 2022-09-02T04:50:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13", 2022-09-02T04:50:51.569 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.508242760Z", 2022-09-02T04:50:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.570 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.571 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.571 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:51.572 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.572 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.572 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.573 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.573 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.563183_72.1", 2022-09-02T04:50:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.574 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.563183823Z", 2022-09-02T04:50:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.575 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.576 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.571784_73.1", 2022-09-02T04:50:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.571784929Z", 2022-09-02T04:50:51.580 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.581 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.581 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.581 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.582 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.582 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:51.582 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.583 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.583 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.584 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.572807_74.1", 2022-09-02T04:50:51.585 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.585 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14", 2022-09-02T04:50:51.585 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.572807912Z", 2022-09-02T04:50:51.585 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.586 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.586 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.586 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.587 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.587 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:51.587 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.588 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.588 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.588 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.589 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.578754_75.1", 2022-09-02T04:50:51.590 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.590 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14", 2022-09-02T04:50:51.590 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.578754707Z", 2022-09-02T04:50:51.591 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.591 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.591 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.591 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.592 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.592 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:51.593 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.593 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.594 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.595 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.659538_76.1", 2022-09-02T04:50:51.595 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.596 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:51.596 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.659538083Z", 2022-09-02T04:50:51.596 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.596 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.597 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.597 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.597 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.598 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:51.598 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.598 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.599 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.600 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.662121_77.1", 2022-09-02T04:50:51.600 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.601 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:51.601 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.662121126Z", 2022-09-02T04:50:51.601 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.601 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.602 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.602 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.603 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:51.603 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.604 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.604 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.604 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.607 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.663114_78.1", 2022-09-02T04:50:51.607 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.608 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15", 2022-09-02T04:50:51.608 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.663114920Z", 2022-09-02T04:50:51.609 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.609 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.609 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.611 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.612 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:51.613 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.614 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.614 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.617 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.665862_79.1", 2022-09-02T04:50:51.617 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.617 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15", 2022-09-02T04:50:51.618 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.665862159Z", 2022-09-02T04:50:51.618 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.619 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.620 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.620 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.621 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.621 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:51.622 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.623 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.623 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.625 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.289122_89.1", 2022-09-02T04:50:51.625 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.626 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:51.626 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.289122249Z", 2022-09-02T04:50:51.627 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.627 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.628 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.628 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.629 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.630 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:51.630 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.631 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.632 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.632 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.634 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.293860_90.1", 2022-09-02T04:50:51.635 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.635 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:51.636 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.293860164Z", 2022-09-02T04:50:51.636 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.637 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.637 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.638 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.638 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.639 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:51.639 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.640 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.641 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.643 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.295284_91.1", 2022-09-02T04:50:51.644 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.644 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16", 2022-09-02T04:50:51.645 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.295284709Z", 2022-09-02T04:50:51.645 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.646 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.647 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.648 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.648 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:51.649 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.649 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.650 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.650 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.654 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.298399_92.1", 2022-09-02T04:50:51.655 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.657 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16", 2022-09-02T04:50:51.657 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.298399993Z", 2022-09-02T04:50:51.658 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.659 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.659 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.659 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.660 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.660 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:51.660 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.661 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.661 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.662 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.327712_93.1", 2022-09-02T04:50:51.663 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.663 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:51.663 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.327712318Z", 2022-09-02T04:50:51.663 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.664 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.664 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.664 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.665 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.665 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:51.665 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.666 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.666 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.668 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.332779_94.1", 2022-09-02T04:50:51.668 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.668 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:51.668 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.332779056Z", 2022-09-02T04:50:51.669 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.669 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.669 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.669 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.670 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:51.670 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.671 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.672 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.334588_95.1", 2022-09-02T04:50:51.673 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.673 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17", 2022-09-02T04:50:51.673 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.334588743Z", 2022-09-02T04:50:51.673 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.674 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.674 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.674 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.675 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.675 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:51.675 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.676 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.676 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.677 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.337431_96.1", 2022-09-02T04:50:51.678 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.678 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17", 2022-09-02T04:50:51.678 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.337431441Z", 2022-09-02T04:50:51.678 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.679 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.679 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.679 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.680 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.680 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:51.680 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.681 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.681 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.682 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.365816_97.1", 2022-09-02T04:50:51.683 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.683 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:51.683 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.365816803Z", 2022-09-02T04:50:51.683 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.684 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.684 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.684 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.685 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.685 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:51.685 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.686 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.686 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.687 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.371582_98.1", 2022-09-02T04:50:51.687 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.688 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:51.688 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.371582029Z", 2022-09-02T04:50:51.688 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.688 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.689 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.689 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.690 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:51.690 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.690 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.691 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.692 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.373810_99.1", 2022-09-02T04:50:51.692 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.692 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18", 2022-09-02T04:50:51.693 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.373810553Z", 2022-09-02T04:50:51.693 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.693 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.693 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.694 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.694 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.694 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:51.695 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.695 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.696 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.697 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.380421_100.1", 2022-09-02T04:50:51.697 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.705 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18", 2022-09-02T04:50:51.711 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.380421451Z", 2022-09-02T04:50:51.712 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.712 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.712 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.713 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.714 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.714 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:51.715 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.716 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.716 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.719 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.406502_101.1", 2022-09-02T04:50:51.719 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.719 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:51.720 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.406502378Z", 2022-09-02T04:50:51.720 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.721 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.722 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:51.722 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.723 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.724 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:51.724 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.726 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.726 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.729 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.409376_102.1", 2022-09-02T04:50:51.729 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.730 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:51.731 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.409376514Z", 2022-09-02T04:50:51.731 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.732 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.733 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:51.733 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.734 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.735 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:51.735 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.736 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.736 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.739 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.424417_103.1", 2022-09-02T04:50:51.740 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.740 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:51.741 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.424417536Z", 2022-09-02T04:50:51.741 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.742 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.742 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:51.742 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.743 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.743 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:51.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.744 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.745 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.747 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.428769_104.1", 2022-09-02T04:50:51.747 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.748 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:51.748 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.428769794Z", 2022-09-02T04:50:51.749 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.749 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.750 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:51.750 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.751 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.752 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:51.752 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.753 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.753 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.753 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.755 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.448268_105.1", 2022-09-02T04:50:51.756 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.756 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:51.756 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.448268867Z", 2022-09-02T04:50:51.756 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.757 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:51.757 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.758 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.758 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:51.759 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.759 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.760 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.761 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.462134_106.1", 2022-09-02T04:50:51.762 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.762 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:51.763 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.462134251Z", 2022-09-02T04:50:51.763 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.764 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.764 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:51.765 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.766 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.766 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:51.766 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.767 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.768 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.770 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.839577_129.1", 2022-09-02T04:50:51.771 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.771 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:51.771 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.839577631Z", 2022-09-02T04:50:51.771 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.772 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.773 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.773 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.773 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:51.774 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.775 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.775 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.777 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.843007_130.1", 2022-09-02T04:50:51.778 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.779 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:51.779 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.843007979Z", 2022-09-02T04:50:51.780 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.780 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.781 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.781 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.782 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:51.783 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.784 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.785 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.789 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.844233_131.1", 2022-09-02T04:50:51.789 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.789 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19", 2022-09-02T04:50:51.789 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.844233977Z", 2022-09-02T04:50:51.790 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.790 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.790 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.791 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.791 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.791 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:51.792 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.793 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.794 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.795 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.847167_132.1", 2022-09-02T04:50:51.795 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.796 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19", 2022-09-02T04:50:51.796 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.847167513Z", 2022-09-02T04:50:51.797 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.797 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.798 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.799 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.799 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.800 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:51.800 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.801 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.801 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.802 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.863844_133.1", 2022-09-02T04:50:51.802 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.803 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:51.803 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.863844546Z", 2022-09-02T04:50:51.803 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.803 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.804 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:51.804 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.805 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.805 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:51.806 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.807 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.807 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.807 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.807 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.808 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.866884_134.1", 2022-09-02T04:50:51.808 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.866884082Z", 2022-09-02T04:50:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.809 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:51.810 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.811 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.811 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:51.811 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.812 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.812 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.813 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.478355_147.1", 2022-09-02T04:50:51.814 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.814 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:51.814 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.478355264Z", 2022-09-02T04:50:51.814 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.817 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.480846_151.1", 2022-09-02T04:50:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.480846805Z", 2022-09-02T04:50:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.481682_152.1", 2022-09-02T04:50:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20", 2022-09-02T04:50:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.481682426Z", 2022-09-02T04:50:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.483984_153.1", 2022-09-02T04:50:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20", 2022-09-02T04:50:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.483984075Z", 2022-09-02T04:50:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093786.258112_154.1", 2022-09-02T04:50:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:43:06.258112912Z", 2022-09-02T04:50:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093786.261185_155.1", 2022-09-02T04:50:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:43:06.261185635Z", 2022-09-02T04:50:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093904.758354_156.1", 2022-09-02T04:50:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.758354256Z", 2022-09-02T04:50:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.858 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093904.761841_157.1", 2022-09-02T04:50:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.861 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.761841925Z", 2022-09-02T04:50:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.862 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.863 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.864 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.643237_158.1", 2022-09-02T04:50:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.866 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.643237395Z", 2022-09-02T04:50:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.867 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:51.868 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.869 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.870 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.646348_159.1", 2022-09-02T04:50:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.646348674Z", 2022-09-02T04:50:51.871 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.873 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.875 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.647493_160.1", 2022-09-02T04:50:51.878 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21", 2022-09-02T04:50:51.879 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.647493391Z", 2022-09-02T04:50:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.880 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.881 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.882 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:51.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.885 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.886 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.887 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.650453_161.1", 2022-09-02T04:50:51.888 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.888 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21", 2022-09-02T04:50:51.888 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.650453266Z", 2022-09-02T04:50:51.889 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.889 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.889 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.889 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.890 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.890 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:51.890 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.891 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.892 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.671385_162.1", 2022-09-02T04:50:51.893 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:51.894 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.671385092Z", 2022-09-02T04:50:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.895 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.896 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.897 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.677725_163.1", 2022-09-02T04:50:51.899 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.677725006Z", 2022-09-02T04:50:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.679743_164.1", 2022-09-02T04:50:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22", 2022-09-02T04:50:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.679743170Z", 2022-09-02T04:50:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.685252_165.1", 2022-09-02T04:50:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22", 2022-09-02T04:50:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.685252272Z", 2022-09-02T04:50:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.720122_166.1", 2022-09-02T04:50:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.720122481Z", 2022-09-02T04:50:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.918 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.722787_167.1", 2022-09-02T04:50:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:51.920 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.722787347Z", 2022-09-02T04:50:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.921 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:51.922 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.924 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.723810_168.1", 2022-09-02T04:50:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23", 2022-09-02T04:50:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.723810483Z", 2022-09-02T04:50:51.925 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.929 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.726983_169.1", 2022-09-02T04:50:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23", 2022-09-02T04:50:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.726983239Z", 2022-09-02T04:50:51.930 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.748449_170.1", 2022-09-02T04:50:51.934 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.935 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.748449921Z", 2022-09-02T04:50:51.936 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:51.938 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.942 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.752174_171.1", 2022-09-02T04:50:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:51.943 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.752174857Z", 2022-09-02T04:50:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.944 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:51.945 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.946 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.948 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.950 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.772370_172.1", 2022-09-02T04:50:51.951 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.951 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.772370930Z", 2022-09-02T04:50:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.952 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.953 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.955 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.956 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.775210_173.1", 2022-09-02T04:50:51.957 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.775210687Z", 2022-09-02T04:50:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.958 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:51.959 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:51.960 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.961 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.796060_174.1", 2022-09-02T04:50:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.796060843Z", 2022-09-02T04:50:51.965 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.799397_175.1", 2022-09-02T04:50:51.969 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.799397586Z", 2022-09-02T04:50:51.970 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:51.972 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.973 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.975 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.480028_178.1", 2022-09-02T04:50:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.977 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.480028626Z", 2022-09-02T04:50:51.978 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.979 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.980 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.483732_179.1", 2022-09-02T04:50:51.984 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:51.985 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.483732364Z", 2022-09-02T04:50:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.986 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.987 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.988 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.988 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:51.989 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.989 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.990 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:51.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:51.992 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.484607_180.1", 2022-09-02T04:50:51.992 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:51.993 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24", 2022-09-02T04:50:51.993 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.484607016Z", 2022-09-02T04:50:51.994 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:51.994 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:51.995 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:51.995 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:51.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.996 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:51.996 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:51.997 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:51.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:51.998 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:51.998 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:51.999 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:51.999 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.000 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.487046_181.1", 2022-09-02T04:50:52.001 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.001 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24", 2022-09-02T04:50:52.002 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.487046711Z", 2022-09-02T04:50:52.002 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.002 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.003 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.004 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.004 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:52.005 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.006 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.006 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.009 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.508152_182.1", 2022-09-02T04:50:52.009 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.010 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:52.010 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.508152912Z", 2022-09-02T04:50:52.010 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.011 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.012 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.012 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.013 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.013 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.015 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.015 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.018 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.511557_183.1", 2022-09-02T04:50:52.019 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.019 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:52.019 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.511557664Z", 2022-09-02T04:50:52.020 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.020 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.021 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.021 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.022 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.023 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.023 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.023 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.024 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.025 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.512912_184.1", 2022-09-02T04:50:52.025 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.025 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25", 2022-09-02T04:50:52.026 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.512912424Z", 2022-09-02T04:50:52.026 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.026 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.026 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.027 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.027 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.027 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:52.028 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.028 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.028 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.030 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.515118_185.1", 2022-09-02T04:50:52.030 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.030 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25", 2022-09-02T04:50:52.030 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.515118385Z", 2022-09-02T04:50:52.031 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.031 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.031 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.031 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.032 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.032 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:52.032 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.033 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.033 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.034 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.550248_186.1", 2022-09-02T04:50:52.035 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.035 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:52.035 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.550248103Z", 2022-09-02T04:50:52.036 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.036 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.036 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.037 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.037 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.037 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.037 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.041 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.552820_187.1", 2022-09-02T04:50:52.042 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.042 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:52.042 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.552820973Z", 2022-09-02T04:50:52.043 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.043 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.047 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.047 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.048 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.049 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.049 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.050 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.051 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.553829_188.1", 2022-09-02T04:50:52.052 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.052 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26", 2022-09-02T04:50:52.053 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.553829203Z", 2022-09-02T04:50:52.053 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.053 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.054 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.055 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.057 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.057 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:52.057 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.058 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.059 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.061 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.556273_189.1", 2022-09-02T04:50:52.062 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.062 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26", 2022-09-02T04:50:52.062 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.556273838Z", 2022-09-02T04:50:52.063 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.063 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.065 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.067 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.067 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:52.068 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.070 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.071 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.071 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.583634_190.1", 2022-09-02T04:50:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:52.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.583634061Z", 2022-09-02T04:50:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.074 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.075 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.075 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.075 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.076 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.587708_191.1", 2022-09-02T04:50:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.078 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.587708089Z", 2022-09-02T04:50:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.079 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.080 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.081 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.081 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.082 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.083 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.607965_192.1", 2022-09-02T04:50:52.083 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.083 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:52.084 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.607965005Z", 2022-09-02T04:50:52.084 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.084 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.085 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.085 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.085 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.086 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.086 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:52.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.088 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.614764_193.1", 2022-09-02T04:50:52.088 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.089 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:52.089 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.614764199Z", 2022-09-02T04:50:52.089 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.089 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.090 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.090 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.091 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.091 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:52.091 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:52.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.092 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.092 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.093 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.635864_194.1", 2022-09-02T04:50:52.093 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.094 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:52.094 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.635864908Z", 2022-09-02T04:50:52.094 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.095 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.095 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.095 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.096 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.096 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.096 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.097 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.097 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.098 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.640272_195.1", 2022-09-02T04:50:52.099 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.099 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:52.099 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.640272673Z", 2022-09-02T04:50:52.099 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.100 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.100 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.100 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.101 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.101 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.101 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.102 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.102 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.103 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.103 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.104 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.661758_196.1", 2022-09-02T04:50:52.104 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.104 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:52.104 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.661758657Z", 2022-09-02T04:50:52.105 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.105 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.105 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.105 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.106 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.106 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.107 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:52.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.107 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.107 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.108 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.108 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.109 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.664046_197.1", 2022-09-02T04:50:52.109 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.109 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:52.109 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.664046043Z", 2022-09-02T04:50:52.110 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.110 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.110 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.111 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.112 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.112 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:52.112 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:52.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.113 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.113 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.114 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.683056_198.1", 2022-09-02T04:50:52.115 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.115 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:52.115 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.683056320Z", 2022-09-02T04:50:52.115 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.116 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.116 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.116 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.117 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.117 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.117 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.118 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.118 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.118 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.119 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.685741_199.1", 2022-09-02T04:50:52.120 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.120 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:52.120 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.685741330Z", 2022-09-02T04:50:52.120 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.121 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.121 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.121 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.122 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.122 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.122 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.123 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.123 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.123 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.124 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.701672_200.1", 2022-09-02T04:50:52.125 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.125 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:52.125 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.701672940Z", 2022-09-02T04:50:52.125 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.126 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.126 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.126 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.127 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.127 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:52.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.128 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.128 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.129 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.130 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.711326_201.1", 2022-09-02T04:50:52.130 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.130 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:52.131 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.711326169Z", 2022-09-02T04:50:52.131 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.131 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.131 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.132 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.132 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.132 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:52.133 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:52.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.133 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.134 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.135 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.596190_217.1", 2022-09-02T04:50:52.135 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.135 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:52.136 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.596190558Z", 2022-09-02T04:50:52.136 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.136 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.137 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.137 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.137 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.138 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:52.138 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.139 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.140 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.599340_218.1", 2022-09-02T04:50:52.140 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.141 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:52.141 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.599340364Z", 2022-09-02T04:50:52.141 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.141 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.142 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.142 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.143 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.143 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:52.143 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.144 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.144 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.144 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.144 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.145 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.600264_219.1", 2022-09-02T04:50:52.145 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.146 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27", 2022-09-02T04:50:52.146 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.600264022Z", 2022-09-02T04:50:52.146 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.146 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.147 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.147 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.147 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.148 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:52.148 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.149 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.150 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.602614_220.1", 2022-09-02T04:50:52.150 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.151 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27", 2022-09-02T04:50:52.151 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.602614533Z", 2022-09-02T04:50:52.151 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.152 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.152 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.152 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.153 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.153 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:52.153 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.154 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.154 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.156 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.624417_221.1", 2022-09-02T04:50:52.156 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.156 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:52.156 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.624417644Z", 2022-09-02T04:50:52.157 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.157 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.157 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.158 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.158 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.159 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:52.159 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.159 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.160 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.160 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.160 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.161 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.628627_222.1", 2022-09-02T04:50:52.161 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.162 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:52.162 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.628627812Z", 2022-09-02T04:50:52.162 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.162 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.163 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.163 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.163 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.164 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:52.164 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.165 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.165 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.165 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.166 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.629582_223.1", 2022-09-02T04:50:52.166 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.167 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28", 2022-09-02T04:50:52.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.629582051Z", 2022-09-02T04:50:52.167 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.167 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.168 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.168 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.169 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.169 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:52.169 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.170 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.170 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.170 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.170 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.171 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.634049_224.1", 2022-09-02T04:50:52.171 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.172 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28", 2022-09-02T04:50:52.172 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.634049176Z", 2022-09-02T04:50:52.172 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.173 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.173 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.173 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.174 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.174 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:52.174 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.175 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.175 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.177 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.674265_225.1", 2022-09-02T04:50:52.177 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.177 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:52.178 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.674265806Z", 2022-09-02T04:50:52.178 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.178 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.178 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.179 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.179 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.179 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:52.180 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.180 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.180 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.182 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.680208_226.1", 2022-09-02T04:50:52.182 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.182 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:52.183 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.680208008Z", 2022-09-02T04:50:52.183 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.183 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.184 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.184 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.185 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:52.185 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.186 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.186 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.187 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.682665_227.1", 2022-09-02T04:50:52.187 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.188 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29", 2022-09-02T04:50:52.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.682665330Z", 2022-09-02T04:50:52.188 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.188 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.189 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.189 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.190 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.190 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:52.190 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.191 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.191 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.192 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.686711_228.1", 2022-09-02T04:50:52.192 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.193 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29", 2022-09-02T04:50:52.193 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.686711368Z", 2022-09-02T04:50:52.193 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.194 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.194 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.194 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.195 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.195 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:52.195 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.196 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.196 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.197 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.718596_229.1", 2022-09-02T04:50:52.198 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.198 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:52.198 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.718596260Z", 2022-09-02T04:50:52.198 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.199 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.199 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:52.199 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.200 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.200 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.203 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.722825_230.1", 2022-09-02T04:50:52.203 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.203 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:52.203 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.722825082Z", 2022-09-02T04:50:52.204 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.204 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.204 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:52.204 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.205 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.205 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:52.206 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.206 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.206 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.208 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.765767_231.1", 2022-09-02T04:50:52.208 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.208 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:52.208 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.765767335Z", 2022-09-02T04:50:52.209 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.209 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.209 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:52.209 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.210 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.210 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:52.211 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.211 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.211 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.213 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.768248_232.1", 2022-09-02T04:50:52.213 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.213 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:52.214 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.768248326Z", 2022-09-02T04:50:52.214 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.214 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:52.215 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.215 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.216 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:52.216 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.216 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.217 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.218 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.795442_233.1", 2022-09-02T04:50:52.218 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.219 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:52.219 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.795442979Z", 2022-09-02T04:50:52.219 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.219 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:52.220 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.220 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.221 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:52.221 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.797953_234.1", 2022-09-02T04:50:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.797953373Z", 2022-09-02T04:50:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.227 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.227 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.228 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.228 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.841868_235.1", 2022-09-02T04:50:52.228 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.229 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:52.229 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.841868706Z", 2022-09-02T04:50:52.229 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.230 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.230 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.230 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.231 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.231 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:52.231 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.232 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.232 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.232 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.232 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.233 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.846202_236.1", 2022-09-02T04:50:52.233 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.234 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:52.234 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.846202210Z", 2022-09-02T04:50:52.234 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.235 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.235 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.235 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.236 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.236 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.237 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:52.237 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.238 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.239 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.239 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.240 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.241 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.847338_237.1", 2022-09-02T04:50:52.241 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.242 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30", 2022-09-02T04:50:52.242 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.847338068Z", 2022-09-02T04:50:52.243 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.243 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.244 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.244 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.245 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.246 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:52.246 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.247 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.248 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.248 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.250 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.851413_238.1", 2022-09-02T04:50:52.251 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.251 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30", 2022-09-02T04:50:52.252 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.851413431Z", 2022-09-02T04:50:52.252 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.253 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.253 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.254 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.255 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.256 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:52.256 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.257 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.257 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.260 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.882196_239.1", 2022-09-02T04:50:52.260 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.261 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:52.261 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.882196828Z", 2022-09-02T04:50:52.262 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.262 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.263 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.263 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.264 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.265 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:52.265 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.267 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.267 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.268 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.269 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.886381_240.1", 2022-09-02T04:50:52.270 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.270 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:52.271 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.886381079Z", 2022-09-02T04:50:52.272 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.272 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.272 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.273 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.274 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.274 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:52.275 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.276 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.276 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.279 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.887265_241.1", 2022-09-02T04:50:52.279 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.280 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31", 2022-09-02T04:50:52.280 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.887265472Z", 2022-09-02T04:50:52.281 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.281 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.282 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.282 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.283 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.284 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:52.284 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.285 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.286 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.286 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.288 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.891145_242.1", 2022-09-02T04:50:52.288 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.289 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31", 2022-09-02T04:50:52.289 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.891145853Z", 2022-09-02T04:50:52.290 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.290 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.291 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.292 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.293 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:52.293 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.294 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.295 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.297 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.923935_243.1", 2022-09-02T04:50:52.298 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.298 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:52.299 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.923935375Z", 2022-09-02T04:50:52.299 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.300 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.300 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.301 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.302 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.303 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:52.303 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.304 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.305 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.305 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.307 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.927811_244.1", 2022-09-02T04:50:52.308 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.308 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:52.309 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.927811078Z", 2022-09-02T04:50:52.309 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.310 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.310 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.310 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.311 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.311 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:52.312 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.312 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.313 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.314 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.928867_245.1", 2022-09-02T04:50:52.315 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.315 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32", 2022-09-02T04:50:52.316 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.928867563Z", 2022-09-02T04:50:52.316 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.317 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.317 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.318 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.319 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.319 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:52.320 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.321 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.321 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.324 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.933635_246.1", 2022-09-02T04:50:52.324 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.325 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32", 2022-09-02T04:50:52.325 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.933635936Z", 2022-09-02T04:50:52.326 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.326 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.326 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.326 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.327 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.327 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:52.328 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.328 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.328 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.330 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.981889_247.1", 2022-09-02T04:50:52.331 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.331 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:52.332 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.981889010Z", 2022-09-02T04:50:52.332 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.333 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.333 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:52.333 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.334 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.334 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:52.334 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.335 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.335 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.336 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.338 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.984537_248.1", 2022-09-02T04:50:52.339 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.339 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:52.340 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.984537015Z", 2022-09-02T04:50:52.340 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.341 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.341 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:52.341 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.342 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.343 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:52.343 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.344 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.345 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.346 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.347 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.069441_249.1", 2022-09-02T04:50:52.348 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.348 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:52.348 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.069441995Z", 2022-09-02T04:50:52.349 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:52.351 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.352 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.353 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.078532_250.1", 2022-09-02T04:50:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:52.354 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.078532139Z", 2022-09-02T04:50:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.355 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.356 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:52.357 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.358 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.361 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.117134_251.1", 2022-09-02T04:50:52.362 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.362 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:52.362 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.117134511Z", 2022-09-02T04:50:52.363 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.363 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.363 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:52.363 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.364 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.365 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:52.365 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.365 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.366 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.367 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.121445_252.1", 2022-09-02T04:50:52.367 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.368 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:52.368 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.121445603Z", 2022-09-02T04:50:52.368 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.369 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.369 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:52.369 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.370 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.371 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:52.371 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.373 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.373 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.375 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.200772_253.1", 2022-09-02T04:50:52.375 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.376 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:52.376 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.200772049Z", 2022-09-02T04:50:52.376 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.377 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.377 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.377 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.378 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.379 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:52.379 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.379 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.380 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.383 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.204385_254.1", 2022-09-02T04:50:52.383 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.384 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:52.384 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.204385610Z", 2022-09-02T04:50:52.385 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.385 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.386 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.386 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.388 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.390 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.205324_255.1", 2022-09-02T04:50:52.391 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.391 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33", 2022-09-02T04:50:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.205324745Z", 2022-09-02T04:50:52.392 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.393 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.394 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.395 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:52.396 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.397 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.397 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.399 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.211398_256.1", 2022-09-02T04:50:52.399 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.399 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33", 2022-09-02T04:50:52.399 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.211398303Z", 2022-09-02T04:50:52.400 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.400 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.401 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.401 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.402 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.403 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:52.403 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.404 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.405 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.411 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.247878_257.1", 2022-09-02T04:50:52.411 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.412 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:52.412 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.247878027Z", 2022-09-02T04:50:52.413 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.413 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.414 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.414 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.415 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.416 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:52.416 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.417 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.417 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.419 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.251068_258.1", 2022-09-02T04:50:52.419 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.420 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:52.420 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.251068384Z", 2022-09-02T04:50:52.420 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.421 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.422 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.422 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.422 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.423 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:52.423 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.423 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.424 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.424 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.424 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.425 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.252037_259.1", 2022-09-02T04:50:52.425 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.426 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34", 2022-09-02T04:50:52.426 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.252037843Z", 2022-09-02T04:50:52.427 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.427 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.428 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.428 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.429 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.430 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:52.431 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.432 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.432 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.433 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.433 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.436 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.260507_260.1", 2022-09-02T04:50:52.436 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.436 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34", 2022-09-02T04:50:52.437 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.260507523Z", 2022-09-02T04:50:52.437 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.438 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.438 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.439 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.439 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.440 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:52.440 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.441 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.441 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.442 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.442 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.443 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.324260_261.1", 2022-09-02T04:50:52.443 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.444 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:52.444 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.324260578Z", 2022-09-02T04:50:52.445 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.445 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.446 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.446 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.447 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.448 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:52.448 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.450 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.451 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.452 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.328657_262.1", 2022-09-02T04:50:52.453 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.453 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:52.454 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.328657368Z", 2022-09-02T04:50:52.454 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.454 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.455 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.455 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.455 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.456 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:52.456 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.457 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.457 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.458 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.459 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.460 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.330962_263.1", 2022-09-02T04:50:52.460 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.460 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35", 2022-09-02T04:50:52.461 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.330962269Z", 2022-09-02T04:50:52.461 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.461 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.462 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.462 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.463 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.464 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:52.464 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.465 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.465 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.466 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.466 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.468 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.335420_264.1", 2022-09-02T04:50:52.469 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.469 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35", 2022-09-02T04:50:52.470 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.335420823Z", 2022-09-02T04:50:52.470 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.470 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.470 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.471 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.471 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.471 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:52.472 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.474 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.389275_265.1", 2022-09-02T04:50:52.474 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.474 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:52.475 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.389275554Z", 2022-09-02T04:50:52.475 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.475 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.475 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:52.476 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.476 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.476 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:52.477 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.477 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.478 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.479 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.393872_266.1", 2022-09-02T04:50:52.479 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.479 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:52.480 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.393872877Z", 2022-09-02T04:50:52.480 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.480 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.480 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:52.481 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.481 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.481 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:52.482 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.482 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.483 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.484 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094086.434234_267.1", 2022-09-02T04:50:52.484 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.484 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:52.485 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:06.434234679Z", 2022-09-02T04:50:52.485 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.485 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.485 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:52.486 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.486 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.486 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:52.487 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.487 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.487 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.489 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094086.438494_268.1", 2022-09-02T04:50:52.489 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.489 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:52.489 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:06.438494895Z", 2022-09-02T04:50:52.490 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.490 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:52.490 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.491 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.491 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:52.491 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.492 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.492 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.492 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.493 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.493 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.522239_269.1", 2022-09-02T04:50:52.494 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.494 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:52.494 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.522239231Z", 2022-09-02T04:50:52.495 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.495 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.495 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:52.495 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.496 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.496 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:52.496 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.497 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.497 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.498 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.499 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.526590_270.1", 2022-09-02T04:50:52.500 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.500 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:52.500 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.526590558Z", 2022-09-02T04:50:52.500 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.501 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.501 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:52.501 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.502 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.502 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:52.502 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.503 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.503 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.504 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.588437_271.1", 2022-09-02T04:50:52.505 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.505 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:52.505 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.588437443Z", 2022-09-02T04:50:52.505 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.506 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.506 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.506 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.507 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.507 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:52.507 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.508 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.508 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.510 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.594122_272.1", 2022-09-02T04:50:52.510 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.510 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:52.510 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.594122094Z", 2022-09-02T04:50:52.511 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.511 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.511 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.511 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.512 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.512 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:52.512 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.513 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.513 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.515 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.596187_273.1", 2022-09-02T04:50:52.515 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.515 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36", 2022-09-02T04:50:52.515 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.596187284Z", 2022-09-02T04:50:52.516 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.516 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.516 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.517 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.517 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:52.517 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.518 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.518 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.520 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.601998_274.1", 2022-09-02T04:50:52.520 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.520 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36", 2022-09-02T04:50:52.520 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.601998619Z", 2022-09-02T04:50:52.521 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.521 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.521 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.521 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.522 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.522 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:52.523 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.523 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.523 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.524 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.524 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.638123_275.1", 2022-09-02T04:50:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.638123665Z", 2022-09-02T04:50:52.526 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.526 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.526 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.526 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.527 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.527 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:52.527 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.528 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.528 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.530 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.640624_276.1", 2022-09-02T04:50:52.530 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.530 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:52.531 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.640624183Z", 2022-09-02T04:50:52.531 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.531 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.531 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.532 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.532 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.532 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:52.533 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.533 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.533 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.535 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.641616_277.1", 2022-09-02T04:50:52.535 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.535 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37", 2022-09-02T04:50:52.536 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.641616287Z", 2022-09-02T04:50:52.536 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.536 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.536 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.541 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.643956_278.1", 2022-09-02T04:50:52.542 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.542 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37", 2022-09-02T04:50:52.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.643956748Z", 2022-09-02T04:50:52.543 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.544 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.544 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.545 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.546 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.546 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:52.547 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.548 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.548 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.549 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.549 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.551 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.689255_279.1", 2022-09-02T04:50:52.551 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.552 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:52.552 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.689255290Z", 2022-09-02T04:50:52.553 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.553 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.554 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.554 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.555 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.556 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:52.556 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.557 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.558 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.558 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.560 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.691696_280.1", 2022-09-02T04:50:52.561 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.561 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:52.561 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.691696253Z", 2022-09-02T04:50:52.562 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.562 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.562 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.562 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.563 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:52.564 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.565 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.566 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.568 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.692552_281.1", 2022-09-02T04:50:52.569 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.569 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38", 2022-09-02T04:50:52.570 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.692552094Z", 2022-09-02T04:50:52.570 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.571 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.571 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.572 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.572 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:52.573 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.575 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.575 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.576 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.696193_282.1", 2022-09-02T04:50:52.577 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.577 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38", 2022-09-02T04:50:52.577 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.696193220Z", 2022-09-02T04:50:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.578 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:52.579 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.580 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.580 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.581 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.042350_283.1", 2022-09-02T04:50:52.582 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.582 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:52.582 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.042350878Z", 2022-09-02T04:50:52.582 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.583 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.583 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.583 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.584 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.584 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:52.584 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.585 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.585 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.586 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.045432_284.1", 2022-09-02T04:50:52.586 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.587 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:52.587 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.045432525Z", 2022-09-02T04:50:52.587 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.587 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.588 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.588 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.589 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.589 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:52.589 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.590 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.590 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.591 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.047334_285.1", 2022-09-02T04:50:52.591 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.592 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39", 2022-09-02T04:50:52.592 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.047334461Z", 2022-09-02T04:50:52.592 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.592 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.593 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.593 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.593 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.594 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:52.594 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.594 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.595 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.596 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.050679_286.1", 2022-09-02T04:50:52.596 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.597 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39", 2022-09-02T04:50:52.597 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.050679259Z", 2022-09-02T04:50:52.597 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.597 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.598 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.598 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.599 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.599 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:52.599 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.600 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.600 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.600 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.600 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.601 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.075859_287.1", 2022-09-02T04:50:52.601 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.602 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:52.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.075859654Z", 2022-09-02T04:50:52.602 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.603 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.603 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.604 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:52.604 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.604 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.605 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.606 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.078656_288.1", 2022-09-02T04:50:52.606 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.606 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:52.607 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.078656456Z", 2022-09-02T04:50:52.607 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.607 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.608 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.608 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:52.609 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.609 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.609 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.611 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.080841_289.1", 2022-09-02T04:50:52.612 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.612 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40", 2022-09-02T04:50:52.612 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.080841716Z", 2022-09-02T04:50:52.612 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.613 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.613 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.613 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.615 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.615 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.616 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.086877_290.1", 2022-09-02T04:50:52.616 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.617 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40", 2022-09-02T04:50:52.617 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.086877398Z", 2022-09-02T04:50:52.617 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.618 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.618 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.618 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.619 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.619 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:52.619 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.620 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.620 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.621 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.125974_291.1", 2022-09-02T04:50:52.621 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.628 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:52.629 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.125974705Z", 2022-09-02T04:50:52.629 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.630 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.630 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.631 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.632 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:52.633 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.634 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.634 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.636 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.132649_292.1", 2022-09-02T04:50:52.637 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.637 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:52.638 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.132649142Z", 2022-09-02T04:50:52.638 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.638 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.639 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.639 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.640 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.641 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:52.642 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.645 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.133657_293.1", 2022-09-02T04:50:52.646 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.646 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41", 2022-09-02T04:50:52.646 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.133657858Z", 2022-09-02T04:50:52.647 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.647 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.648 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.648 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.650 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.650 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:52.651 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.652 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.652 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.654 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.139676_294.1", 2022-09-02T04:50:52.655 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.655 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41", 2022-09-02T04:50:52.656 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.139676296Z", 2022-09-02T04:50:52.657 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.657 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.658 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.658 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.659 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.659 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:52.660 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.661 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.661 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.664 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.573151_93.1", 2022-09-02T04:50:52.665 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.665 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:52.666 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.573151032Z", 2022-09-02T04:50:52.666 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.667 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.667 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.667 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.668 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.669 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:52.669 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.671 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.674 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.579025_94.1", 2022-09-02T04:50:52.674 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.675 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:52.675 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.579025648Z", 2022-09-02T04:50:52.676 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.676 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.677 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.677 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.678 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.678 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:52.679 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.680 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.680 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.581233_95.1", 2022-09-02T04:50:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1", 2022-09-02T04:50:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.581233957Z", 2022-09-02T04:50:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.585812_96.1", 2022-09-02T04:50:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1", 2022-09-02T04:50:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.585812830Z", 2022-09-02T04:50:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.617016_97.1", 2022-09-02T04:50:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.617016945Z", 2022-09-02T04:50:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.620981_98.1", 2022-09-02T04:50:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.620981265Z", 2022-09-02T04:50:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.622018_99.1", 2022-09-02T04:50:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2", 2022-09-02T04:50:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.622018558Z", 2022-09-02T04:50:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.624757_100.1", 2022-09-02T04:50:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2", 2022-09-02T04:50:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.624757525Z", 2022-09-02T04:50:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.630272_101.1", 2022-09-02T04:50:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.630272240Z", 2022-09-02T04:50:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.740 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.633085_102.1", 2022-09-02T04:50:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.633085325Z", 2022-09-02T04:50:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.634091_103.1", 2022-09-02T04:50:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3", 2022-09-02T04:50:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.634091584Z", 2022-09-02T04:50:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.762 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.763 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.764 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:52.764 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.765 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.766 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.766 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.768 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.636577_104.1", 2022-09-02T04:50:52.769 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.769 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3", 2022-09-02T04:50:52.769 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.636577040Z", 2022-09-02T04:50:52.770 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.770 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.771 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.771 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.771 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:52.772 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.772 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.773 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.774 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.570754_105.1", 2022-09-02T04:50:52.774 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.775 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:52.775 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.570754424Z", 2022-09-02T04:50:52.775 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.775 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.776 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.776 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.777 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.777 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:52.777 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.778 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.778 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.780 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.573776_106.1", 2022-09-02T04:50:52.780 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.780 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:52.781 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.573776177Z", 2022-09-02T04:50:52.781 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.781 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.782 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.782 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.782 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:52.783 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.783 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.784 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.785 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.574795_107.1", 2022-09-02T04:50:52.785 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.785 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4", 2022-09-02T04:50:52.786 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.574795537Z", 2022-09-02T04:50:52.786 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.786 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.786 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.787 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.787 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.788 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:52.788 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.789 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.789 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.791 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.577651_108.1", 2022-09-02T04:50:52.791 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.791 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4", 2022-09-02T04:50:52.791 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.577651116Z", 2022-09-02T04:50:52.792 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.792 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.792 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.793 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.793 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.795 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:52.796 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.796 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.797 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.797 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.799 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.609918_109.1", 2022-09-02T04:50:52.800 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.800 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:52.801 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.609918035Z", 2022-09-02T04:50:52.801 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.803 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.804 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.804 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.805 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.805 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:52.805 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.806 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.806 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.807 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.807 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.808 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.613337_110.1", 2022-09-02T04:50:52.808 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.808 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:52.808 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.613337726Z", 2022-09-02T04:50:52.809 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.809 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.809 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.809 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.810 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:52.811 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.811 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.812 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.812 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.813 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.614272_111.1", 2022-09-02T04:50:52.813 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.821 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5", 2022-09-02T04:50:52.821 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.614272060Z", 2022-09-02T04:50:52.822 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.822 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.822 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.823 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.823 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.824 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:52.824 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.825 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.825 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.825 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.825 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.827 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.618746_112.1", 2022-09-02T04:50:52.828 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.828 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5", 2022-09-02T04:50:52.828 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.618746992Z", 2022-09-02T04:50:52.828 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.829 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.829 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.830 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.830 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:52.830 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.831 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.831 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.832 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.832 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.654802_113.1", 2022-09-02T04:50:52.833 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.833 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:52.834 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.654802951Z", 2022-09-02T04:50:52.834 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.835 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.835 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.835 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.836 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.836 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:52.836 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.838 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.839 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.839 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.839 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.840 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.657918_114.1", 2022-09-02T04:50:52.841 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.841 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:52.841 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.657918674Z", 2022-09-02T04:50:52.842 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.842 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.842 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.843 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.843 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.843 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:52.844 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.844 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.844 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.846 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.660134_115.1", 2022-09-02T04:50:52.846 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.846 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6", 2022-09-02T04:50:52.847 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.660134944Z", 2022-09-02T04:50:52.847 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.848 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.848 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.848 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.849 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.849 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:52.849 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.850 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.850 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.851 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.852 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.665018_116.1", 2022-09-02T04:50:52.852 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.852 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6", 2022-09-02T04:50:52.852 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.665018151Z", 2022-09-02T04:50:52.853 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.853 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.853 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.853 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.854 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:52.855 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.855 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.857 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.769933_117.1", 2022-09-02T04:50:52.858 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.858 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:52.859 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.769933252Z", 2022-09-02T04:50:52.859 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.859 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.860 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.860 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.860 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.861 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:52.861 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.862 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.865 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.772485_118.1", 2022-09-02T04:50:52.865 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.865 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:52.867 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.772485812Z", 2022-09-02T04:50:52.868 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.868 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.869 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.869 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.870 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.870 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:52.870 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.871 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.871 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.872 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.872 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.773480_119.1", 2022-09-02T04:50:52.873 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.873 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:52.873 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.773480228Z", 2022-09-02T04:50:52.873 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.874 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.874 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.874 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.875 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.875 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:52.875 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.876 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.876 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.876 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.878 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.776581_120.1", 2022-09-02T04:50:52.878 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.878 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:52.884 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.776581648Z", 2022-09-02T04:50:52.885 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.885 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.885 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.886 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.887 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.887 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:52.888 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.888 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.888 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.890 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.801099_121.1", 2022-09-02T04:50:52.891 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.891 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:52.891 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.801099540Z", 2022-09-02T04:50:52.892 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.892 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.892 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.892 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.893 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.893 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:52.894 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.894 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.894 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.896 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.804401_122.1", 2022-09-02T04:50:52.896 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.896 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:52.897 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.804401413Z", 2022-09-02T04:50:52.897 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.897 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.898 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.898 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.898 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.899 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:52.899 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.900 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.900 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.902 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.805277_123.1", 2022-09-02T04:50:52.902 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.902 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:52.903 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.805277428Z", 2022-09-02T04:50:52.903 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.903 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.903 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.904 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.904 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.904 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:52.905 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.905 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.906 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.906 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.906 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.907 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.807715_124.1", 2022-09-02T04:50:52.907 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.908 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:52.908 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.807715596Z", 2022-09-02T04:50:52.908 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.908 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.909 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.909 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.910 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.911 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:52.911 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.914 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.914 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.915 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.915 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.842266_125.1", 2022-09-02T04:50:52.916 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.916 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:52.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.842266349Z", 2022-09-02T04:50:52.917 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.917 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.917 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.917 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.918 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.918 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:52.919 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.919 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.920 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.920 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.920 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.922 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.853430_126.1", 2022-09-02T04:50:52.923 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.923 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:52.924 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.853430958Z", 2022-09-02T04:50:52.924 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.925 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.925 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.926 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.927 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.927 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:52.928 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.929 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.933 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.855671_127.1", 2022-09-02T04:50:52.933 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.934 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:52.934 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.855671199Z", 2022-09-02T04:50:52.935 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.935 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.936 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.936 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.937 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.938 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:52.939 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.940 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.940 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.941 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.941 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.862622_128.1", 2022-09-02T04:50:52.943 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.943 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:52.943 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.862622338Z", 2022-09-02T04:50:52.943 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.944 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.944 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:52.945 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:52.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.946 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.946 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:52.947 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.948 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.948 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.951 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.580527_129.1", 2022-09-02T04:50:52.951 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.952 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:52.952 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.580527505Z", 2022-09-02T04:50:52.953 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.954 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.954 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:52.955 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.956 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.956 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:52.957 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.958 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.959 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.961 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.583272_130.1", 2022-09-02T04:50:52.962 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.962 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:52.963 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.583272249Z", 2022-09-02T04:50:52.963 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.964 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.965 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:52.965 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.966 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:52.967 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.967 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.970 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.595303_131.1", 2022-09-02T04:50:52.971 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.971 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:52.972 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.595303694Z", 2022-09-02T04:50:52.972 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.976 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:52.976 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.977 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.977 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2022-09-02T04:50:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.599169_132.1", 2022-09-02T04:50:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.599169915Z", 2022-09-02T04:50:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.638539_133.1", 2022-09-02T04:50:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.638539023Z", 2022-09-02T04:50:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.645764_134.1", 2022-09-02T04:50:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:52.996 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.645764849Z", 2022-09-02T04:50:52.996 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:52.996 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:52.997 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:52.998 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:52.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:52.999 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:52.999 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:52.999 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.000 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.001 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:53.001 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.002 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:53.003 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.659515_135.1", 2022-09-02T04:50:53.003 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:53.004 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:53.004 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.659515593Z", 2022-09-02T04:50:53.004 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:53.005 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:53.005 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:53.005 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:53.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.006 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:53.006 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:53.006 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.007 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.007 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2022-09-02T04:50:53.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.008 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:53.008 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.662895_136.1", 2022-09-02T04:50:53.009 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:53.009 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:53.009 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.662895104Z", 2022-09-02T04:50:53.009 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:53.010 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:53.010 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:53.010 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:53.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.011 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:53.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:53.011 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.012 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.012 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:53.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:53.013 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.751579_137.1", 2022-09-02T04:50:53.014 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.014 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:53.014 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.751579938Z", 2022-09-02T04:50:53.014 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:53.015 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:53.015 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:53.015 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:53.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.024 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:53.024 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:53.024 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.025 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.025 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:53.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:53.027 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.754025_138.1", 2022-09-02T04:50:53.027 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.027 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:53.028 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.754025410Z", 2022-09-02T04:50:53.028 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:53.028 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:53.029 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:53.029 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:53.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.030 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:53.030 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:53.030 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.031 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.031 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:53.031 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:53.032 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.769139_139.1", 2022-09-02T04:50:53.033 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:53.033 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:53.033 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.769139327Z", 2022-09-02T04:50:53.033 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:53.034 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:53.034 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:53.034 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:53.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.035 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:53.035 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:53.035 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.036 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.036 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2022-09-02T04:50:53.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:53.039 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.771218_140.1", 2022-09-02T04:50:53.039 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:53.040 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:53.040 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218052Z", 2022-09-02T04:50:53.040 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:53.040 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:53.041 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:53.041 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:53.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.042 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:53.042 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:53.042 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:53.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:53.043 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:53.043 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:53.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:53.044 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:50:53.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:53.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:53.048 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:53.105 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:53.105 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:53.153 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092292.275513_70.1", 2022-09-02T04:50:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "user", 2022-09-02T04:50:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.275513716Z", 2022-09-02T04:50:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092292.280197_71.1", 2022-09-02T04:50:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "user", 2022-09-02T04:50:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197578Z", 2022-09-02T04:50:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.750378_51.1", 2022-09-02T04:50:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.750378020Z", 2022-09-02T04:50:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.753068_53.1", 2022-09-02T04:50:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.753068141Z", 2022-09-02T04:50:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.754140_54.1", 2022-09-02T04:50:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10", 2022-09-02T04:50:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.754140083Z", 2022-09-02T04:50:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.756550_55.1", 2022-09-02T04:50:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-10", 2022-09-02T04:50:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.756550326Z", 2022-09-02T04:50:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.780936_56.1", 2022-09-02T04:50:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.780936022Z", 2022-09-02T04:50:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.783480_57.1", 2022-09-02T04:50:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.783480461Z", 2022-09-02T04:50:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.784364_58.1", 2022-09-02T04:50:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11", 2022-09-02T04:50:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.784364362Z", 2022-09-02T04:50:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.786791_59.1", 2022-09-02T04:50:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-11", 2022-09-02T04:50:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.786791946Z", 2022-09-02T04:50:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.818066_60.1", 2022-09-02T04:50:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.818066873Z", 2022-09-02T04:50:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.820503_61.1", 2022-09-02T04:50:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.820503127Z", 2022-09-02T04:50:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.821417_62.1", 2022-09-02T04:50:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12", 2022-09-02T04:50:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.821417990Z", 2022-09-02T04:50:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092427.823736_63.1", 2022-09-02T04:50:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-12", 2022-09-02T04:50:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:27.823736708Z", 2022-09-02T04:50:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.494157_68.1", 2022-09-02T04:50:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.494157445Z", 2022-09-02T04:50:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.500451_69.1", 2022-09-02T04:50:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.500451205Z", 2022-09-02T04:50:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.502516_70.1", 2022-09-02T04:50:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13", 2022-09-02T04:50:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.502516287Z", 2022-09-02T04:50:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.508242_71.1", 2022-09-02T04:50:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-13", 2022-09-02T04:50:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.508242760Z", 2022-09-02T04:50:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.563183_72.1", 2022-09-02T04:50:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.563183823Z", 2022-09-02T04:50:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.571784_73.1", 2022-09-02T04:50:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.571784929Z", 2022-09-02T04:50:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.572807_74.1", 2022-09-02T04:50:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14", 2022-09-02T04:50:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.572807912Z", 2022-09-02T04:50:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.578754_75.1", 2022-09-02T04:50:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-14", 2022-09-02T04:50:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.578754707Z", 2022-09-02T04:50:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.659538_76.1", 2022-09-02T04:50:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.659538083Z", 2022-09-02T04:50:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.284 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.662121_77.1", 2022-09-02T04:50:53.284 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.285 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:53.285 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.662121126Z", 2022-09-02T04:50:53.286 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.286 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.287 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.288 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.289 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:53.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.290 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.294 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.663114_78.1", 2022-09-02T04:50:53.294 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.295 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15", 2022-09-02T04:50:53.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.663114920Z", 2022-09-02T04:50:53.296 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.296 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.296 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.297 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.297 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.298 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:53.298 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.298 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.300 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092762.665862_79.1", 2022-09-02T04:50:53.301 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.301 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-15", 2022-09-02T04:50:53.301 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:26:02.665862159Z", 2022-09-02T04:50:53.302 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.302 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.302 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.302 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.303 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.303 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:53.304 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.304 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.304 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.306 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.289122_89.1", 2022-09-02T04:50:53.306 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.306 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:53.307 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.289122249Z", 2022-09-02T04:50:53.307 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.307 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.307 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.308 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.308 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.308 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:53.309 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.309 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.309 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.311 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.293860_90.1", 2022-09-02T04:50:53.311 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.311 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:53.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.293860164Z", 2022-09-02T04:50:53.312 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.312 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.313 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.313 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.313 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.318 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.295284_91.1", 2022-09-02T04:50:53.319 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.319 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16", 2022-09-02T04:50:53.320 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.295284709Z", 2022-09-02T04:50:53.320 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.321 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.321 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.298399_92.1", 2022-09-02T04:50:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16", 2022-09-02T04:50:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.298399993Z", 2022-09-02T04:50:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.333 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:53.333 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.334 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.335 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.337 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.327712_93.1", 2022-09-02T04:50:53.337 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.337 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:53.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.327712318Z", 2022-09-02T04:50:53.338 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.338 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.338 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.338 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.339 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.339 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:53.339 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.342 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.332779_94.1", 2022-09-02T04:50:53.342 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.342 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:53.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.332779056Z", 2022-09-02T04:50:53.343 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.343 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.343 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.343 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.344 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.344 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:53.344 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.345 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.345 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.347 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.334588_95.1", 2022-09-02T04:50:53.347 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.347 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17", 2022-09-02T04:50:53.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.334588743Z", 2022-09-02T04:50:53.348 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.348 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.348 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.348 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.349 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.349 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:53.349 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.350 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.350 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.352 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.337431_96.1", 2022-09-02T04:50:53.352 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.352 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17", 2022-09-02T04:50:53.352 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.337431441Z", 2022-09-02T04:50:53.353 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.353 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.353 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.353 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.354 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.354 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:53.354 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.355 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.355 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.357 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.365816_97.1", 2022-09-02T04:50:53.357 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.357 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:53.357 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.365816803Z", 2022-09-02T04:50:53.358 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.358 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.358 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.359 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.359 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.359 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:53.360 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.360 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.360 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.362 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.371582_98.1", 2022-09-02T04:50:53.362 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.362 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:53.362 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.371582029Z", 2022-09-02T04:50:53.363 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.363 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.363 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.363 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.364 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.364 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:53.364 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.365 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.365 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.365 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.367 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.373810_99.1", 2022-09-02T04:50:53.367 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.367 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18", 2022-09-02T04:50:53.367 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.373810553Z", 2022-09-02T04:50:53.368 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.368 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.368 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.368 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.369 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.369 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:53.369 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.370 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.370 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.371 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.371 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.380421_100.1", 2022-09-02T04:50:53.372 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.372 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18", 2022-09-02T04:50:53.372 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.380421451Z", 2022-09-02T04:50:53.372 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.373 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.373 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.373 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.374 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.374 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:53.374 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.375 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.375 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.376 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.406502_101.1", 2022-09-02T04:50:53.377 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.377 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:53.377 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.406502378Z", 2022-09-02T04:50:53.377 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.378 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.378 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:53.378 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.379 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.379 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:53.379 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.380 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.380 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.381 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.409376_102.1", 2022-09-02T04:50:53.382 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.382 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:53.382 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.409376514Z", 2022-09-02T04:50:53.383 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.383 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.383 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:53.383 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.384 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.384 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:53.384 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.386 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.424417_103.1", 2022-09-02T04:50:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:53.387 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.424417536Z", 2022-09-02T04:50:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:53.388 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:53.389 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.428769_104.1", 2022-09-02T04:50:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.428769794Z", 2022-09-02T04:50:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.448268_105.1", 2022-09-02T04:50:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.448268867Z", 2022-09-02T04:50:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092864.462134_106.1", 2022-09-02T04:50:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:27:44.462134251Z", 2022-09-02T04:50:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.839577_129.1", 2022-09-02T04:50:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.839577631Z", 2022-09-02T04:50:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.843007_130.1", 2022-09-02T04:50:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.843007979Z", 2022-09-02T04:50:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.844233_131.1", 2022-09-02T04:50:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19", 2022-09-02T04:50:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.844233977Z", 2022-09-02T04:50:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.847167_132.1", 2022-09-02T04:50:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19", 2022-09-02T04:50:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.847167513Z", 2022-09-02T04:50:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.863844_133.1", 2022-09-02T04:50:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.863844546Z", 2022-09-02T04:50:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093133.866884_134.1", 2022-09-02T04:50:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:32:13.866884082Z", 2022-09-02T04:50:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.478355_147.1", 2022-09-02T04:50:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.478355264Z", 2022-09-02T04:50:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.480846_151.1", 2022-09-02T04:50:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.480846805Z", 2022-09-02T04:50:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.481682_152.1", 2022-09-02T04:50:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20", 2022-09-02T04:50:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.481682426Z", 2022-09-02T04:50:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093748.483984_153.1", 2022-09-02T04:50:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20", 2022-09-02T04:50:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:42:28.483984075Z", 2022-09-02T04:50:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093786.258112_154.1", 2022-09-02T04:50:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:43:06.258112912Z", 2022-09-02T04:50:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093786.261185_155.1", 2022-09-02T04:50:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:43:06.261185635Z", 2022-09-02T04:50:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093904.758354_156.1", 2022-09-02T04:50:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:45:04.758354256Z", 2022-09-02T04:50:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093904.761841_157.1", 2022-09-02T04:50:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:45:04.761841925Z", 2022-09-02T04:50:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.643237_158.1", 2022-09-02T04:50:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.643237395Z", 2022-09-02T04:50:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.646348_159.1", 2022-09-02T04:50:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.646348674Z", 2022-09-02T04:50:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.647493_160.1", 2022-09-02T04:50:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21", 2022-09-02T04:50:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.647493391Z", 2022-09-02T04:50:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.650453_161.1", 2022-09-02T04:50:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21", 2022-09-02T04:50:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.650453266Z", 2022-09-02T04:50:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.671385_162.1", 2022-09-02T04:50:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.671385092Z", 2022-09-02T04:50:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.516 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.516 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.677725_163.1", 2022-09-02T04:50:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.677725006Z", 2022-09-02T04:50:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.679743_164.1", 2022-09-02T04:50:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22", 2022-09-02T04:50:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.679743170Z", 2022-09-02T04:50:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.685252_165.1", 2022-09-02T04:50:53.532 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.532 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22", 2022-09-02T04:50:53.532 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.685252272Z", 2022-09-02T04:50:53.533 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.533 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.533 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.534 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.534 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.536 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.720122_166.1", 2022-09-02T04:50:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.720122481Z", 2022-09-02T04:50:53.538 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.541 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.722787_167.1", 2022-09-02T04:50:53.543 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.544 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:53.544 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.722787347Z", 2022-09-02T04:50:53.544 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.545 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.545 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.546 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.547 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.547 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:53.548 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.549 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.550 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.723810_168.1", 2022-09-02T04:50:53.551 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.551 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23", 2022-09-02T04:50:53.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.723810483Z", 2022-09-02T04:50:53.551 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.553 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.554 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.555 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.556 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.556 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:53.556 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.557 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.557 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.559 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.726983_169.1", 2022-09-02T04:50:53.559 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.560 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23", 2022-09-02T04:50:53.560 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.726983239Z", 2022-09-02T04:50:53.560 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.561 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.561 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.561 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.562 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.562 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:53.563 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.563 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.564 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.566 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.748449_170.1", 2022-09-02T04:50:53.567 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.567 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:53.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.748449921Z", 2022-09-02T04:50:53.568 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.569 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.569 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:53.570 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.572 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.572 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:53.573 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.577 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.578 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.580 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.582 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.752174_171.1", 2022-09-02T04:50:53.582 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.583 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:53.583 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.752174857Z", 2022-09-02T04:50:53.584 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.584 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.585 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:53.585 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.587 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.587 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:53.588 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.589 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.589 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.592 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.772370_172.1", 2022-09-02T04:50:53.592 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.593 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:53.593 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.772370930Z", 2022-09-02T04:50:53.594 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.595 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.595 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:53.596 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.597 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.597 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:53.598 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.599 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.599 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.602 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.775210_173.1", 2022-09-02T04:50:53.602 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.603 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:53.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.775210687Z", 2022-09-02T04:50:53.604 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.604 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.605 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:53.605 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.607 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.607 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:53.607 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.608 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.608 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.609 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.796060_174.1", 2022-09-02T04:50:53.610 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.610 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:53.610 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.796060843Z", 2022-09-02T04:50:53.611 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.612 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.612 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:53.612 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.613 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.613 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:53.613 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.614 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.614 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.616 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662093961.799397_175.1", 2022-09-02T04:50:53.616 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.616 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:53.616 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:46:01.799397586Z", 2022-09-02T04:50:53.617 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.617 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.617 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:53.617 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.618 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.618 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:53.619 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.619 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.619 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.621 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.480028_178.1", 2022-09-02T04:50:53.621 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.621 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:53.621 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.480028626Z", 2022-09-02T04:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.483732_179.1", 2022-09-02T04:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:53.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.483732364Z", 2022-09-02T04:50:53.627 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.627 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.627 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.628 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.628 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.628 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.629 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.629 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.629 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.631 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.484607_180.1", 2022-09-02T04:50:53.632 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.632 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24", 2022-09-02T04:50:53.632 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.484607016Z", 2022-09-02T04:50:53.633 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.633 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.633 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.634 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.635 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.635 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:53.635 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.636 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.637 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.639 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.487046_181.1", 2022-09-02T04:50:53.639 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.639 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24", 2022-09-02T04:50:53.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.487046711Z", 2022-09-02T04:50:53.640 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.641 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.641 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.642 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.643 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.643 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:53.644 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.508152_182.1", 2022-09-02T04:50:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.508152912Z", 2022-09-02T04:50:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.511557_183.1", 2022-09-02T04:50:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.511557664Z", 2022-09-02T04:50:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.654 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.654 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.655 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.655 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.657 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.512912_184.1", 2022-09-02T04:50:53.657 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.657 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25", 2022-09-02T04:50:53.657 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.512912424Z", 2022-09-02T04:50:53.658 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.658 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.663 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.663 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.664 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.664 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:53.665 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.665 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.666 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.515118_185.1", 2022-09-02T04:50:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25", 2022-09-02T04:50:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.515118385Z", 2022-09-02T04:50:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.550248_186.1", 2022-09-02T04:50:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:53.676 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.550248103Z", 2022-09-02T04:50:53.676 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.676 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.676 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.677 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.677 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.677 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.678 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.678 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.679 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.680 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.681 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.552820_187.1", 2022-09-02T04:50:53.681 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.682 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:53.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.552820973Z", 2022-09-02T04:50:53.683 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.683 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.684 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.684 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.685 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.685 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.686 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.687 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.687 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.689 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.553829_188.1", 2022-09-02T04:50:53.690 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.690 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26", 2022-09-02T04:50:53.690 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.553829203Z", 2022-09-02T04:50:53.690 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.691 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.691 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.692 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.693 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.693 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:53.693 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.694 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.695 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.697 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.556273_189.1", 2022-09-02T04:50:53.698 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.698 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26", 2022-09-02T04:50:53.699 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.556273838Z", 2022-09-02T04:50:53.699 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.699 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.700 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.700 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.701 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.701 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:53.702 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.703 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.703 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.704 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.583634_190.1", 2022-09-02T04:50:53.704 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.705 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:53.705 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.583634061Z", 2022-09-02T04:50:53.705 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.706 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.706 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.706 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.707 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.708 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.708 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.709 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.709 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.709 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.710 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.711 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.587708_191.1", 2022-09-02T04:50:53.711 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.711 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:53.711 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.587708089Z", 2022-09-02T04:50:53.712 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.712 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.712 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.713 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.713 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.713 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.714 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.714 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.715 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.716 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.607965_192.1", 2022-09-02T04:50:53.716 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.716 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:53.716 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.607965005Z", 2022-09-02T04:50:53.717 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.717 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.717 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.717 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.718 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.718 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.719 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.719 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.719 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.721 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.614764_193.1", 2022-09-02T04:50:53.721 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.721 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:53.721 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.614764199Z", 2022-09-02T04:50:53.722 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.722 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.722 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.723 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.723 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.724 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:53.724 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.725 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.726 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.728 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.635864_194.1", 2022-09-02T04:50:53.728 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.729 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:53.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.635864908Z", 2022-09-02T04:50:53.730 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.730 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.730 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.731 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.732 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.732 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.732 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.733 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.733 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.640272_195.1", 2022-09-02T04:50:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.640272673Z", 2022-09-02T04:50:53.736 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.736 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.737 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.737 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.738 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.738 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.739 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.740 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.740 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.741 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.741 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.661758_196.1", 2022-09-02T04:50:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.661758657Z", 2022-09-02T04:50:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.748 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.752 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.664046_197.1", 2022-09-02T04:50:53.753 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.753 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:53.754 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.664046043Z", 2022-09-02T04:50:53.754 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.755 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.755 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.756 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.756 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.757 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:53.757 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.683056_198.1", 2022-09-02T04:50:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.683056320Z", 2022-09-02T04:50:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.685741_199.1", 2022-09-02T04:50:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.685741330Z", 2022-09-02T04:50:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.767 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.769 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.701672_200.1", 2022-09-02T04:50:53.769 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.769 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:53.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.701672940Z", 2022-09-02T04:50:53.770 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.770 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.770 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.770 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.771 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.771 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.771 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.772 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.772 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.773 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094028.711326_201.1", 2022-09-02T04:50:53.774 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.774 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:53.774 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:08.711326169Z", 2022-09-02T04:50:53.774 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.775 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.775 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.775 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.776 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.776 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:53.776 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 3 2022-09-02T04:50:53.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.777 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.777 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.596190_217.1", 2022-09-02T04:50:53.778 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.596190558Z", 2022-09-02T04:50:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.779 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.780 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.781 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.599340_218.1", 2022-09-02T04:50:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.783 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:53.784 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.599340364Z", 2022-09-02T04:50:53.784 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.784 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.784 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.785 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.785 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.785 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:53.786 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.786 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.786 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.787 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.600264_219.1", 2022-09-02T04:50:53.788 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.788 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27", 2022-09-02T04:50:53.788 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.600264022Z", 2022-09-02T04:50:53.788 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.789 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.789 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.789 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.790 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.790 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:53.790 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.791 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.791 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.792 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.602614_220.1", 2022-09-02T04:50:53.793 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.793 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27", 2022-09-02T04:50:53.793 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.602614533Z", 2022-09-02T04:50:53.793 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.794 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.794 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.794 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.795 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.795 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:53.795 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.796 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.796 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.797 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.624417_221.1", 2022-09-02T04:50:53.797 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.798 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:53.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.624417644Z", 2022-09-02T04:50:53.798 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.798 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.799 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.799 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.799 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.800 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:53.800 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.800 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.800 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.802 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.628627_222.1", 2022-09-02T04:50:53.802 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.802 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:53.803 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.628627812Z", 2022-09-02T04:50:53.803 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.803 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.803 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.805 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.805 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.805 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.629582_223.1", 2022-09-02T04:50:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28", 2022-09-02T04:50:53.807 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.629582051Z", 2022-09-02T04:50:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.634049_224.1", 2022-09-02T04:50:53.811 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.812 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28", 2022-09-02T04:50:53.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.634049176Z", 2022-09-02T04:50:53.812 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.812 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.813 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.813 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.814 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.814 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:53.814 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.815 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.815 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.816 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.674265_225.1", 2022-09-02T04:50:53.816 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.817 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:53.817 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.674265806Z", 2022-09-02T04:50:53.817 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.817 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.818 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.818 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.818 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.819 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:53.819 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.819 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.820 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.821 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.680208_226.1", 2022-09-02T04:50:53.821 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.821 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:53.822 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.680208008Z", 2022-09-02T04:50:53.822 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.822 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.822 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.823 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.823 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.823 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:53.824 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.824 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.824 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.826 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.682665_227.1", 2022-09-02T04:50:53.826 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.826 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29", 2022-09-02T04:50:53.826 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.682665330Z", 2022-09-02T04:50:53.827 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.827 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.827 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.827 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.828 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.828 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:53.828 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.830 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.686711_228.1", 2022-09-02T04:50:53.831 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.831 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29", 2022-09-02T04:50:53.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.686711368Z", 2022-09-02T04:50:53.831 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.832 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.832 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.832 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.832 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.833 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:53.833 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.834 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.834 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.836 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.718596_229.1", 2022-09-02T04:50:53.836 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.836 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:53.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.718596260Z", 2022-09-02T04:50:53.837 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.837 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.837 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:53.837 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.838 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.838 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:53.838 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.839 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.839 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.839 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.840 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.722825_230.1", 2022-09-02T04:50:53.840 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.841 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:53.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.722825082Z", 2022-09-02T04:50:53.841 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.841 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.842 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:53.842 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.842 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.843 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:53.843 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.843 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.844 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.846 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.765767_231.1", 2022-09-02T04:50:53.846 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:53.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.765767335Z", 2022-09-02T04:50:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.848 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.853 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.768248_232.1", 2022-09-02T04:50:53.854 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.854 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:53.854 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.768248326Z", 2022-09-02T04:50:53.854 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.855 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.855 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:53.856 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.857 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.857 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:53.857 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.858 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.859 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.859 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.859 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.860 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.795442_233.1", 2022-09-02T04:50:53.860 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.860 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:53.861 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.795442979Z", 2022-09-02T04:50:53.861 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.861 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.862 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:53.862 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.862 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.863 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:53.863 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.864 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.864 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.865 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.866 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.797953_234.1", 2022-09-02T04:50:53.866 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.867 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:53.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.797953373Z", 2022-09-02T04:50:53.868 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.868 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.868 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:53.869 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.870 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.872 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.874 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.841868_235.1", 2022-09-02T04:50:53.874 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.875 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:53.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.841868706Z", 2022-09-02T04:50:53.876 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.876 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.877 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.877 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.878 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.879 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:53.879 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.880 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.881 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.883 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.846202_236.1", 2022-09-02T04:50:53.883 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.884 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:53.884 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.846202210Z", 2022-09-02T04:50:53.884 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.885 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.885 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.885 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.887 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.887 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:53.888 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.890 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.847338_237.1", 2022-09-02T04:50:53.891 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.891 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30", 2022-09-02T04:50:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.847338068Z", 2022-09-02T04:50:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.851413_238.1", 2022-09-02T04:50:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30", 2022-09-02T04:50:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.851413431Z", 2022-09-02T04:50:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.901 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.882196_239.1", 2022-09-02T04:50:53.901 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.901 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:53.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.882196828Z", 2022-09-02T04:50:53.902 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.902 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.902 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.902 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.903 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.903 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:53.903 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.904 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.904 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.905 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.886381_240.1", 2022-09-02T04:50:53.906 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.906 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:53.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.886381079Z", 2022-09-02T04:50:53.906 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.907 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.907 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.907 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.908 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.908 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:53.908 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.909 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.909 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.910 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.887265_241.1", 2022-09-02T04:50:53.911 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.911 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31", 2022-09-02T04:50:53.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.887265472Z", 2022-09-02T04:50:53.911 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.912 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.912 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.912 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.913 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.913 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:53.913 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.914 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.914 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.915 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.891145_242.1", 2022-09-02T04:50:53.915 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.916 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31", 2022-09-02T04:50:53.916 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.891145853Z", 2022-09-02T04:50:53.916 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.916 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.917 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.917 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.917 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.918 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:53.918 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.918 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.919 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.919 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.919 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.920 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.923935_243.1", 2022-09-02T04:50:53.920 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.920 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:53.921 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.923935375Z", 2022-09-02T04:50:53.921 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.921 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.921 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.921 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.922 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.922 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:53.923 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.923 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.923 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.925 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.927811_244.1", 2022-09-02T04:50:53.925 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.925 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:53.925 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.927811078Z", 2022-09-02T04:50:53.926 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.926 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.926 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.926 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.927 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.927 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:53.927 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.928 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.928 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.929 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.928867_245.1", 2022-09-02T04:50:53.929 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.930 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32", 2022-09-02T04:50:53.930 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.928867563Z", 2022-09-02T04:50:53.930 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.930 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.931 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.931 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.931 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.933635_246.1", 2022-09-02T04:50:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32", 2022-09-02T04:50:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.933635936Z", 2022-09-02T04:50:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.981889_247.1", 2022-09-02T04:50:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.981889010Z", 2022-09-02T04:50:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094084.984537_248.1", 2022-09-02T04:50:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:04.984537015Z", 2022-09-02T04:50:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:53.946 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.946 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.946 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.948 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.069441_249.1", 2022-09-02T04:50:53.948 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.948 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:53.948 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.069441995Z", 2022-09-02T04:50:53.949 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.949 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.949 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:53.949 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.950 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.950 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:53.950 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.078532_250.1", 2022-09-02T04:50:53.952 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.078532139Z", 2022-09-02T04:50:53.953 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.955 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.956 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.957 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.958 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.117134_251.1", 2022-09-02T04:50:53.958 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.958 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:53.959 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.117134511Z", 2022-09-02T04:50:53.959 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.959 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.959 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:53.960 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.960 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.960 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:53.961 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.961 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.961 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.121445_252.1", 2022-09-02T04:50:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:53.963 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.121445603Z", 2022-09-02T04:50:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:53.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.966 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.966 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.967 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.200772_253.1", 2022-09-02T04:50:53.968 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.968 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:53.968 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.200772049Z", 2022-09-02T04:50:53.968 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.969 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.969 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.969 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.970 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.970 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:53.970 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.971 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.971 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.971 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.972 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.204385_254.1", 2022-09-02T04:50:53.973 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.973 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:53.973 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.204385610Z", 2022-09-02T04:50:53.973 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.974 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.974 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.974 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.975 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.975 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:53.975 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.976 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.976 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.978 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.205324_255.1", 2022-09-02T04:50:53.978 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.978 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33", 2022-09-02T04:50:53.978 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.205324745Z", 2022-09-02T04:50:53.979 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.979 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.979 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.979 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.980 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.980 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:53.980 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.981 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.981 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.981 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.982 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.982 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.211398_256.1", 2022-09-02T04:50:53.983 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.983 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33", 2022-09-02T04:50:53.983 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.211398303Z", 2022-09-02T04:50:53.983 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.984 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.984 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.984 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.985 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.985 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:53.985 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.985 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.986 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.987 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.247878_257.1", 2022-09-02T04:50:53.987 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.987 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:53.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.247878027Z", 2022-09-02T04:50:53.988 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.988 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.988 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.989 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.989 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.989 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:53.990 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:53.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.992 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.251068_258.1", 2022-09-02T04:50:53.992 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:53.992 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:53.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.251068384Z", 2022-09-02T04:50:53.993 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:53.993 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:53.993 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:53.993 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:53.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.994 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:53.994 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:53.995 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:53.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.996 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:53.996 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:53.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.997 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:53.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:53.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:53.998 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.252037_259.1", 2022-09-02T04:50:53.998 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:53.999 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34", 2022-09-02T04:50:53.999 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.252037843Z", 2022-09-02T04:50:53.999 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.000 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.000 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.000 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.001 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.002 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:54.002 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.003 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.003 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.005 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.260507_260.1", 2022-09-02T04:50:54.005 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.005 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34", 2022-09-02T04:50:54.005 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.260507523Z", 2022-09-02T04:50:54.006 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.006 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.006 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.006 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.007 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.007 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:54.007 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.009 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.324260_261.1", 2022-09-02T04:50:54.010 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.010 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:54.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.324260578Z", 2022-09-02T04:50:54.010 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.011 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.011 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.011 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.012 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.012 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:54.012 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.013 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.013 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.014 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.328657_262.1", 2022-09-02T04:50:54.014 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.015 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:54.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.328657368Z", 2022-09-02T04:50:54.015 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.015 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.016 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.016 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.016 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.016 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:54.017 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.017 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.017 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.330962_263.1", 2022-09-02T04:50:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35", 2022-09-02T04:50:54.020 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.330962269Z", 2022-09-02T04:50:54.020 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.020 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.020 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.020 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.021 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.021 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:54.022 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.022 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.022 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.023 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.335420_264.1", 2022-09-02T04:50:54.024 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.024 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35", 2022-09-02T04:50:54.024 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.335420823Z", 2022-09-02T04:50:54.024 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.025 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.025 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.025 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.026 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.026 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:54.026 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.027 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.027 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.028 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.389275_265.1", 2022-09-02T04:50:54.028 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.029 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:54.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.389275554Z", 2022-09-02T04:50:54.029 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.029 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.030 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:54.030 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.030 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.031 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:54.031 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.031 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.032 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.033 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094085.393872_266.1", 2022-09-02T04:50:54.033 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.033 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:54.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:05.393872877Z", 2022-09-02T04:50:54.034 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.034 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.034 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:54.035 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.035 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.035 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:54.035 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.036 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.036 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.038 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094086.434234_267.1", 2022-09-02T04:50:54.038 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.038 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:54.039 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:06.434234679Z", 2022-09-02T04:50:54.039 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.039 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.039 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:54.039 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.040 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.040 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:54.040 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.042 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094086.438494_268.1", 2022-09-02T04:50:54.043 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.043 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:54.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:06.438494895Z", 2022-09-02T04:50:54.043 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.044 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.044 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:54.044 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.045 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.045 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:54.045 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.046 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.046 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.047 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.522239_269.1", 2022-09-02T04:50:54.047 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.048 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:54.048 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.522239231Z", 2022-09-02T04:50:54.048 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.526590_270.1", 2022-09-02T04:50:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.526590558Z", 2022-09-02T04:50:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.588437_271.1", 2022-09-02T04:50:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.588437443Z", 2022-09-02T04:50:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.059 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.059 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.059 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.060 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:54.060 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.060 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.594122_272.1", 2022-09-02T04:50:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:54.063 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.594122094Z", 2022-09-02T04:50:54.063 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.063 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.063 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.063 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.064 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.064 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:54.065 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.067 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.596187_273.1", 2022-09-02T04:50:54.067 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.067 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36", 2022-09-02T04:50:54.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.596187284Z", 2022-09-02T04:50:54.068 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.068 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.068 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.068 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.069 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.069 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:54.069 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.070 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.070 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.070 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.071 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.601998_274.1", 2022-09-02T04:50:54.072 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.072 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-36", 2022-09-02T04:50:54.072 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.601998619Z", 2022-09-02T04:50:54.072 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.073 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.073 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.073 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.074 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.074 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:54.074 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.075 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.075 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.076 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.638123_275.1", 2022-09-02T04:50:54.076 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.076 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:54.077 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.638123665Z", 2022-09-02T04:50:54.077 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.077 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.077 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.078 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.078 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.079 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:54.079 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.079 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.081 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.640624_276.1", 2022-09-02T04:50:54.081 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.081 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:54.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.640624183Z", 2022-09-02T04:50:54.082 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.082 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.082 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.083 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.083 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.083 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:54.084 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.084 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.084 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.085 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.641616_277.1", 2022-09-02T04:50:54.086 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.086 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37", 2022-09-02T04:50:54.086 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.641616287Z", 2022-09-02T04:50:54.086 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.087 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.087 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.087 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.088 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.088 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:54.088 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.089 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.089 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.089 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.090 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.643956_278.1", 2022-09-02T04:50:54.091 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.091 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-37", 2022-09-02T04:50:54.091 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.643956748Z", 2022-09-02T04:50:54.091 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.092 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.092 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.092 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.093 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.093 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:54.093 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.094 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.094 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.094 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.094 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.095 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.689255_279.1", 2022-09-02T04:50:54.095 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.095 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:54.096 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.689255290Z", 2022-09-02T04:50:54.096 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.096 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.097 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.097 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.099 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.099 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.100 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.691696_280.1", 2022-09-02T04:50:54.100 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.101 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:54.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.691696253Z", 2022-09-02T04:50:54.101 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.101 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.102 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.102 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.102 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.103 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:54.103 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.103 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.104 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.105 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.692552_281.1", 2022-09-02T04:50:54.105 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.105 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38", 2022-09-02T04:50:54.106 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.692552094Z", 2022-09-02T04:50:54.106 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.106 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.106 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.107 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.107 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.107 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:54.108 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.108 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.108 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.110 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094087.696193_282.1", 2022-09-02T04:50:54.110 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.110 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-38", 2022-09-02T04:50:54.110 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:07.696193220Z", 2022-09-02T04:50:54.111 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.111 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.111 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.111 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.112 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.112 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:54.112 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.113 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.113 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.115 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.042350_283.1", 2022-09-02T04:50:54.115 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.115 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:54.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.042350878Z", 2022-09-02T04:50:54.116 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.116 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.116 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.116 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.117 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.117 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:54.117 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.118 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.118 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.119 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.045432_284.1", 2022-09-02T04:50:54.119 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.120 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:54.120 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.045432525Z", 2022-09-02T04:50:54.120 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.120 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.121 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.121 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.121 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.122 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:54.122 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.122 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.123 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.124 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.047334_285.1", 2022-09-02T04:50:54.124 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.124 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39", 2022-09-02T04:50:54.124 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.047334461Z", 2022-09-02T04:50:54.125 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.125 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.125 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.125 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.126 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.126 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:54.126 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.127 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.127 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.128 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.128 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.050679_286.1", 2022-09-02T04:50:54.129 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.129 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-39", 2022-09-02T04:50:54.129 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.050679259Z", 2022-09-02T04:50:54.129 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.129 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.130 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.130 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.130 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.131 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:54.131 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.131 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.132 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.132 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.132 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.133 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.075859_287.1", 2022-09-02T04:50:54.133 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.133 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:54.134 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.075859654Z", 2022-09-02T04:50:54.134 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.134 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.134 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.135 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.135 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.135 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:54.136 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.136 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.136 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.138 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.078656_288.1", 2022-09-02T04:50:54.138 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.138 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:54.138 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.078656456Z", 2022-09-02T04:50:54.139 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.139 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.139 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.139 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.140 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.140 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:54.140 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.141 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.141 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.080841_289.1", 2022-09-02T04:50:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40", 2022-09-02T04:50:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.080841716Z", 2022-09-02T04:50:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.086877_290.1", 2022-09-02T04:50:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-40", 2022-09-02T04:50:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.086877398Z", 2022-09-02T04:50:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.125974_291.1", 2022-09-02T04:50:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.125974705Z", 2022-09-02T04:50:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.154 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.154 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.154 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:54.155 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.155 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.155 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.157 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.132649_292.1", 2022-09-02T04:50:54.157 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.157 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:54.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.132649142Z", 2022-09-02T04:50:54.158 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.158 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.158 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.158 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.159 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.159 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:54.159 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.160 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.160 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.161 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.133657_293.1", 2022-09-02T04:50:54.161 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.162 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41", 2022-09-02T04:50:54.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.133657858Z", 2022-09-02T04:50:54.162 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.162 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.163 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.163 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.163 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.164 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:54.164 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.164 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.165 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.166 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094104.139676_294.1", 2022-09-02T04:50:54.166 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.166 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-41", 2022-09-02T04:50:54.167 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:48:24.139676296Z", 2022-09-02T04:50:54.167 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.167 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.167 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.167 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.168 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.168 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:54.168 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.170 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.170 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.573151_93.1", 2022-09-02T04:50:54.171 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.171 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:54.171 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.573151032Z", 2022-09-02T04:50:54.171 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.172 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.172 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.172 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.173 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.173 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:54.173 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.174 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.174 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.175 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.579025_94.1", 2022-09-02T04:50:54.175 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.176 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:54.176 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.579025648Z", 2022-09-02T04:50:54.176 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.176 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.177 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.177 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.177 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.177 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.581233_95.1", 2022-09-02T04:50:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1", 2022-09-02T04:50:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.581233957Z", 2022-09-02T04:50:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.585812_96.1", 2022-09-02T04:50:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-1", 2022-09-02T04:50:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.585812830Z", 2022-09-02T04:50:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.617016_97.1", 2022-09-02T04:50:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.190 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:54.190 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.617016945Z", 2022-09-02T04:50:54.190 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.190 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.191 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.191 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.191 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.192 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:54.192 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.194 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.620981_98.1", 2022-09-02T04:50:54.194 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.194 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:54.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.620981265Z", 2022-09-02T04:50:54.195 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.195 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.195 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.195 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.196 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.196 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:54.196 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.197 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.197 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.197 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.198 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.622018_99.1", 2022-09-02T04:50:54.199 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.199 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2", 2022-09-02T04:50:54.199 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.622018558Z", 2022-09-02T04:50:54.199 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.200 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.200 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.200 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.200 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.201 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:54.201 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.201 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.202 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.203 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092409.624757_100.1", 2022-09-02T04:50:54.203 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.203 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-2", 2022-09-02T04:50:54.204 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:09.624757525Z", 2022-09-02T04:50:54.204 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.204 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.204 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.205 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.205 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.205 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:54.206 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.206 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.206 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.208 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.630272_101.1", 2022-09-02T04:50:54.208 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.208 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:54.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.630272240Z", 2022-09-02T04:50:54.209 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.209 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.209 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.209 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.210 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.210 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:54.210 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.211 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.212 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.633085_102.1", 2022-09-02T04:50:54.213 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.213 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:54.213 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.633085325Z", 2022-09-02T04:50:54.213 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.215 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.634091_103.1", 2022-09-02T04:50:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.217 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3", 2022-09-02T04:50:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.634091584Z", 2022-09-02T04:50:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.218 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.219 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.220 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.221 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092412.636577_104.1", 2022-09-02T04:50:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-3", 2022-09-02T04:50:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:12.636577040Z", 2022-09-02T04:50:54.222 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.224 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.224 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:54.224 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.225 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.225 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.225 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.226 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.570754_105.1", 2022-09-02T04:50:54.226 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.227 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:54.227 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.570754424Z", 2022-09-02T04:50:54.227 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.227 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.228 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.228 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.228 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.229 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:54.229 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.229 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.229 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.231 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.573776_106.1", 2022-09-02T04:50:54.231 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.231 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:54.231 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.573776177Z", 2022-09-02T04:50:54.232 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.232 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.232 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.232 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.574795_107.1", 2022-09-02T04:50:54.236 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.236 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4", 2022-09-02T04:50:54.236 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.574795537Z", 2022-09-02T04:50:54.236 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.236 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.237 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.237 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.237 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.238 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:54.238 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.238 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.239 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.240 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.577651_108.1", 2022-09-02T04:50:54.240 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.240 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-4", 2022-09-02T04:50:54.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.577651116Z", 2022-09-02T04:50:54.241 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.241 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.241 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.242 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.242 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.242 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:54.243 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.243 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.243 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.243 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.244 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.609918_109.1", 2022-09-02T04:50:54.245 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.245 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:54.245 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.609918035Z", 2022-09-02T04:50:54.245 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.246 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.246 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.246 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.247 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.247 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:54.247 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.248 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.248 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.249 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.613337_110.1", 2022-09-02T04:50:54.249 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.250 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:54.250 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.613337726Z", 2022-09-02T04:50:54.250 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.250 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.251 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.251 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.251 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.252 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:54.252 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.252 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.253 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.254 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.614272_111.1", 2022-09-02T04:50:54.254 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.254 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5", 2022-09-02T04:50:54.255 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.614272060Z", 2022-09-02T04:50:54.255 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.255 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.255 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.256 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.256 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.256 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:54.257 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.257 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.257 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.257 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.258 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.259 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.618746_112.1", 2022-09-02T04:50:54.259 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.259 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-5", 2022-09-02T04:50:54.259 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.618746992Z", 2022-09-02T04:50:54.259 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.260 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.260 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.260 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.261 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:54.261 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.262 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.262 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.263 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.654802_113.1", 2022-09-02T04:50:54.263 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.264 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:54.264 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.654802951Z", 2022-09-02T04:50:54.264 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.264 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.265 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.265 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.265 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.266 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:54.266 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.266 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.267 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.268 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.657918_114.1", 2022-09-02T04:50:54.268 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.268 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:54.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.657918674Z", 2022-09-02T04:50:54.269 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.269 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.269 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.269 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.270 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.270 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:54.270 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.271 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.271 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.271 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.272 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.660134_115.1", 2022-09-02T04:50:54.273 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.273 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6", 2022-09-02T04:50:54.273 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.660134944Z", 2022-09-02T04:50:54.273 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.274 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.274 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.274 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.275 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.275 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:54.275 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.276 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.276 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.278 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092418.665018_116.1", 2022-09-02T04:50:54.278 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.278 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-6", 2022-09-02T04:50:54.278 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:18.665018151Z", 2022-09-02T04:50:54.279 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.279 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.279 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.279 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.280 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.280 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:54.280 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.281 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.281 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.282 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.769933_117.1", 2022-09-02T04:50:54.283 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.283 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:54.283 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.769933252Z", 2022-09-02T04:50:54.283 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.284 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.284 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.284 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.285 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.285 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:54.285 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.286 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.286 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.287 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.772485_118.1", 2022-09-02T04:50:54.288 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.288 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:54.288 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.772485812Z", 2022-09-02T04:50:54.289 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.289 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.289 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.290 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.290 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:54.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.292 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.773480_119.1", 2022-09-02T04:50:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.773480228Z", 2022-09-02T04:50:54.293 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.776581_120.1", 2022-09-02T04:50:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.297 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.776581648Z", 2022-09-02T04:50:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.299 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.299 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.299 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:54.300 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.300 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.300 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.301 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.301 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.302 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.801099_121.1", 2022-09-02T04:50:54.302 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.303 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:54.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.801099540Z", 2022-09-02T04:50:54.304 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.304 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.305 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.305 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.306 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.306 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:54.307 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.308 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.308 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.308 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.311 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.804401_122.1", 2022-09-02T04:50:54.311 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.311 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:54.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.804401413Z", 2022-09-02T04:50:54.312 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.313 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.313 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.315 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.316 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.805277_123.1", 2022-09-02T04:50:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.318 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.805277428Z", 2022-09-02T04:50:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.319 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.320 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:54.321 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.321 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.321 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.322 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.322 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.323 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.807715_124.1", 2022-09-02T04:50:54.323 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.323 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:54.323 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.807715596Z", 2022-09-02T04:50:54.324 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.324 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.324 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.324 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.325 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.325 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:54.325 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.326 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.326 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.327 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.842266_125.1", 2022-09-02T04:50:54.328 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.328 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:54.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.842266349Z", 2022-09-02T04:50:54.328 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.329 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.329 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.329 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.330 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.330 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:54.330 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.331 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.331 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.332 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.853430_126.1", 2022-09-02T04:50:54.332 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.332 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:54.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.853430958Z", 2022-09-02T04:50:54.333 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.333 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.333 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.334 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.334 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.334 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:54.335 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.335 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.335 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.337 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.855671_127.1", 2022-09-02T04:50:54.337 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.337 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:54.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.855671199Z", 2022-09-02T04:50:54.338 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.338 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.338 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.338 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.339 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.339 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:54.339 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.341 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092420.862622_128.1", 2022-09-02T04:50:54.342 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.342 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:54.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:20.862622338Z", 2022-09-02T04:50:54.342 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.343 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.343 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:54.343 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:54.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.343 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.344 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:54.344 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.344 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.345 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.346 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.580527_129.1", 2022-09-02T04:50:54.346 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.346 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:54.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.580527505Z", 2022-09-02T04:50:54.347 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.347 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.347 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:54.348 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.348 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.348 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:54.349 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.349 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.349 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.351 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.583272_130.1", 2022-09-02T04:50:54.351 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.351 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:54.351 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.583272249Z", 2022-09-02T04:50:54.352 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.352 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.352 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:54.352 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.353 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.353 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:54.353 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.354 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.354 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.355 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.595303_131.1", 2022-09-02T04:50:54.355 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.356 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:54.356 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.595303694Z", 2022-09-02T04:50:54.356 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.356 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.357 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:54.357 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.357 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.357 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:54.358 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.358 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.359 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "remove" 2022-09-02T04:50:54.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.360 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.599169_132.1", 2022-09-02T04:50:54.362 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.362 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-7", 2022-09-02T04:50:54.362 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.599169915Z", 2022-09-02T04:50:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.638539_133.1", 2022-09-02T04:50:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.638539023Z", 2022-09-02T04:50:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:54.371 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.372 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.372 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:54.373 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.376 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.645764_134.1", 2022-09-02T04:50:54.377 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.377 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:54.377 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.645764849Z", 2022-09-02T04:50:54.377 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.378 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.378 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:54.379 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.379 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.379 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:54.380 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.380 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.381 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.382 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.659515_135.1", 2022-09-02T04:50:54.382 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.383 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:54.383 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.659515593Z", 2022-09-02T04:50:54.384 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.384 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.385 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:54.385 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.386 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.387 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:54.387 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.388 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.389 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "remove" 2022-09-02T04:50:54.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.391 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.662895_136.1", 2022-09-02T04:50:54.392 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.392 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-8", 2022-09-02T04:50:54.393 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.662895104Z", 2022-09-02T04:50:54.393 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.394 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.394 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:54.395 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.396 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.396 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:54.397 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.398 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.398 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.400 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.751579_137.1", 2022-09-02T04:50:54.401 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.402 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:54.403 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.751579938Z", 2022-09-02T04:50:54.403 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.403 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.404 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:54.404 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.405 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.406 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:54.407 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.408 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.408 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:54.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.411 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.754025_138.1", 2022-09-02T04:50:54.411 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.412 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:54.412 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.754025410Z", 2022-09-02T04:50:54.413 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.413 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.414 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:54.414 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.415 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.416 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:54.416 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.418 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.418 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.419 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.769139_139.1", 2022-09-02T04:50:54.420 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.420 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:54.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.769139327Z", 2022-09-02T04:50:54.420 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.421 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.421 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:54.421 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.422 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.422 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:54.422 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.423 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.423 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "remove" 2022-09-02T04:50:54.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:54.426 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662092421.771218_140.1", 2022-09-02T04:50:54.427 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:54.427 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-9", 2022-09-02T04:50:54.428 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218052Z", 2022-09-02T04:50:54.428 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:54.428 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:54.429 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:54.429 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:54.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.430 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:54.430 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:54.431 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 2 2022-09-02T04:50:54.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:54.432 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:54.432 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:54.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:54.434 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:54.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:54.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:54.440 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:54.498 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:54.499 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:54.547 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:50:54.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.547 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092292.275513_70.1", 2022-09-02T04:50:54.548 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2022-09-02T04:50:54.548 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:54.548 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.275513716Z", 2022-09-02T04:50:54.548 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.549 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.549 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.549 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.550 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.550 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:54.551 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.552 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.552 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.554 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092292.280197_71.1", 2022-09-02T04:50:54.555 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "user", 2022-09-02T04:50:54.555 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "rgw-multisite-test-user", 2022-09-02T04:50:54.555 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197578Z", 2022-09-02T04:50:54.555 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.556 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.556 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.556 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.557 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UNTd1t0ErtL3kXUFcqANiPV", 2022-09-02T04:50:54.557 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.558 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.559 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.560 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.561 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.750378_51.1", 2022-09-02T04:50:54.562 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.562 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:54.563 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.750378020Z", 2022-09-02T04:50:54.563 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.564 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.564 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.565 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.566 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:54.567 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.568 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.568 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.569 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.569 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.753068_53.1", 2022-09-02T04:50:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T04:50:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.753068141Z", 2022-09-02T04:50:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kxMPjxnUcDnnfCObGDttZuZ", 2022-09-02T04:50:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.754140_54.1", 2022-09-02T04:50:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10", 2022-09-02T04:50:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.754140083Z", 2022-09-02T04:50:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.756550_55.1", 2022-09-02T04:50:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-10", 2022-09-02T04:50:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.756550326Z", 2022-09-02T04:50:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_RUfsg0UYKDvo7xY4VPn2JVp", 2022-09-02T04:50:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.780936_56.1", 2022-09-02T04:50:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.780936022Z", 2022-09-02T04:50:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.783480_57.1", 2022-09-02T04:50:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T04:50:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.783480461Z", 2022-09-02T04:50:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p5dFNoTbntj8BJFpwgwyX_g", 2022-09-02T04:50:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.784364_58.1", 2022-09-02T04:50:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11", 2022-09-02T04:50:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.784364362Z", 2022-09-02T04:50:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.786791_59.1", 2022-09-02T04:50:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-11", 2022-09-02T04:50:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.786791946Z", 2022-09-02T04:50:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gPpHiTR12ZfVi8eOMCjwmvH", 2022-09-02T04:50:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.818066_60.1", 2022-09-02T04:50:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.818066873Z", 2022-09-02T04:50:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.820503_61.1", 2022-09-02T04:50:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T04:50:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.820503127Z", 2022-09-02T04:50:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_rnx7BuluXh1oIZe5cRjOo0E", 2022-09-02T04:50:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.821417_62.1", 2022-09-02T04:50:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12", 2022-09-02T04:50:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.821417990Z", 2022-09-02T04:50:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092427.823736_63.1", 2022-09-02T04:50:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-12", 2022-09-02T04:50:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:27.823736708Z", 2022-09-02T04:50:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Hc6ItsP2CxtBMktTWrPRYi4", 2022-09-02T04:50:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.494157_68.1", 2022-09-02T04:50:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.494157445Z", 2022-09-02T04:50:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.500451_69.1", 2022-09-02T04:50:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T04:50:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.500451205Z", 2022-09-02T04:50:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SjZatDP8nh_SKYajJjbQNqZ", 2022-09-02T04:50:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.502516_70.1", 2022-09-02T04:50:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13", 2022-09-02T04:50:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.502516287Z", 2022-09-02T04:50:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.508242_71.1", 2022-09-02T04:50:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-13", 2022-09-02T04:50:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.508242760Z", 2022-09-02T04:50:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.681 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.681 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.682 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.683 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ZevFls0QozEqzknkLpalckf", 2022-09-02T04:50:54.684 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.684 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.684 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.686 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.563183_72.1", 2022-09-02T04:50:54.686 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.686 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:54.687 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.563183823Z", 2022-09-02T04:50:54.687 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.687 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.687 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.688 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.688 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.688 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:54.689 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.689 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.689 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.691 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.571784_73.1", 2022-09-02T04:50:54.691 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.691 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T04:50:54.692 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.571784929Z", 2022-09-02T04:50:54.692 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.692 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.692 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.693 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.693 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.693 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Y6Geur0-7HcaUcF591KzPiW", 2022-09-02T04:50:54.694 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.695 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.695 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.698 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.572807_74.1", 2022-09-02T04:50:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14", 2022-09-02T04:50:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.572807912Z", 2022-09-02T04:50:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.704 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.705 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:54.705 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.706 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.707 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.709 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.578754_75.1", 2022-09-02T04:50:54.710 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.710 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-14", 2022-09-02T04:50:54.711 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.578754707Z", 2022-09-02T04:50:54.712 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.712 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.713 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.713 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.714 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.715 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_L5aiYOUysVlVLn3-eKitBXe", 2022-09-02T04:50:54.715 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.716 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.718 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.720 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.659538_76.1", 2022-09-02T04:50:54.721 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.721 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:54.722 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.659538083Z", 2022-09-02T04:50:54.723 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.723 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.724 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.724 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.725 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.725 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:54.726 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.727 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.727 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.730 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.662121_77.1", 2022-09-02T04:50:54.730 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.731 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T04:50:54.731 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.662121126Z", 2022-09-02T04:50:54.732 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.732 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.733 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.733 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.734 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.735 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SuhXevy-Acn1xRgkAXGHOkC", 2022-09-02T04:50:54.735 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.736 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.737 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.739 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.663114_78.1", 2022-09-02T04:50:54.741 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.741 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15", 2022-09-02T04:50:54.742 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.663114920Z", 2022-09-02T04:50:54.742 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.743 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.743 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.744 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.745 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.745 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:54.746 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.747 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.747 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.748 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.751 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092762.665862_79.1", 2022-09-02T04:50:54.751 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.752 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-15", 2022-09-02T04:50:54.754 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:26:02.665862159Z", 2022-09-02T04:50:54.754 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.755 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.755 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.756 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.757 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_DtiL3DU1uIxRLY1ybQ42d2h", 2022-09-02T04:50:54.758 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.759 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.759 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.760 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.763 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.289122_89.1", 2022-09-02T04:50:54.763 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.764 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:54.764 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.289122249Z", 2022-09-02T04:50:54.766 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.766 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.767 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.767 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.768 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.769 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:54.769 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.770 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.771 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.774 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.293860_90.1", 2022-09-02T04:50:54.774 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.775 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:54.775 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.293860164Z", 2022-09-02T04:50:54.776 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.777 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.777 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.778 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.779 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.779 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:54.780 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.781 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.783 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.786 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.295284_91.1", 2022-09-02T04:50:54.786 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.787 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16", 2022-09-02T04:50:54.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.295284709Z", 2022-09-02T04:50:54.788 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.788 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.788 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.789 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.790 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.790 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:54.791 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.791 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.795 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.298399_92.1", 2022-09-02T04:50:54.795 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.796 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16", 2022-09-02T04:50:54.796 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.298399993Z", 2022-09-02T04:50:54.796 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.797 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.797 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.798 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.798 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.799 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8V0QfoW0tAYik_1h8jh80Ni", 2022-09-02T04:50:54.799 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.800 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.801 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.803 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.327712_93.1", 2022-09-02T04:50:54.803 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.803 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:54.804 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.327712318Z", 2022-09-02T04:50:54.804 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.805 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.805 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.805 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.812 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.813 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:54.813 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.813 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.814 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.815 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.332779_94.1", 2022-09-02T04:50:54.816 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.816 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:54.816 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.332779056Z", 2022-09-02T04:50:54.816 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.817 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.817 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.817 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.818 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.818 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:54.818 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.819 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.819 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.820 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.820 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.334588_95.1", 2022-09-02T04:50:54.821 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.821 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17", 2022-09-02T04:50:54.821 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.334588743Z", 2022-09-02T04:50:54.822 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.823 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.823 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.824 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.825 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.825 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:54.826 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.827 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.828 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.828 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.829 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.830 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.337431_96.1", 2022-09-02T04:50:54.831 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.831 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17", 2022-09-02T04:50:54.831 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.337431441Z", 2022-09-02T04:50:54.832 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.832 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.833 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.833 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.834 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.835 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_UgXIeDrfZT1F3UhJsmcJvmc", 2022-09-02T04:50:54.835 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.836 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.837 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.837 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.839 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.365816_97.1", 2022-09-02T04:50:54.839 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.840 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:54.840 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.365816803Z", 2022-09-02T04:50:54.841 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.841 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.842 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.842 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.842 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.843 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:54.843 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.843 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.844 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.845 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.371582_98.1", 2022-09-02T04:50:54.846 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.846 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:54.847 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.371582029Z", 2022-09-02T04:50:54.847 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.848 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.848 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.849 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.849 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.850 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:54.850 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.851 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.852 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.852 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.854 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.373810_99.1", 2022-09-02T04:50:54.854 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.855 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18", 2022-09-02T04:50:54.855 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.373810553Z", 2022-09-02T04:50:54.856 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.856 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.857 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.857 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.380421_100.1", 2022-09-02T04:50:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18", 2022-09-02T04:50:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.380421451Z", 2022-09-02T04:50:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XtvSShOElWH1GaJ3DprXxiW", 2022-09-02T04:50:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.871 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.872 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.872 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.873 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.874 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.406502_101.1", 2022-09-02T04:50:54.875 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.875 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:54.875 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.406502378Z", 2022-09-02T04:50:54.876 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.876 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.876 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:54.876 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.877 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.877 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:54.877 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.878 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.878 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.878 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.879 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.880 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.409376_102.1", 2022-09-02T04:50:54.880 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.880 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T04:50:54.881 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.409376514Z", 2022-09-02T04:50:54.881 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.882 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.882 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:54.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.884 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.884 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nRkW_BvwgBqkU7IdUIwQJy8", 2022-09-02T04:50:54.884 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.885 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.886 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.889 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.424417_103.1", 2022-09-02T04:50:54.889 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.890 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:54.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.424417536Z", 2022-09-02T04:50:54.891 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.891 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:54.891 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.892 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.893 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:54.893 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.894 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.894 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.896 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.428769_104.1", 2022-09-02T04:50:54.896 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.897 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T04:50:54.897 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.428769794Z", 2022-09-02T04:50:54.897 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.897 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.898 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:54.898 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.899 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.900 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_vFAjJT-9HgOGd_m0muWNwGm", 2022-09-02T04:50:54.900 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.901 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.902 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.902 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.903 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.904 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.448268_105.1", 2022-09-02T04:50:54.904 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.905 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:54.905 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.448268867Z", 2022-09-02T04:50:54.906 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.906 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.907 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:54.908 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.909 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.909 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:54.910 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.911 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.911 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.914 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092864.462134_106.1", 2022-09-02T04:50:54.915 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.915 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T04:50:54.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:27:44.462134251Z", 2022-09-02T04:50:54.916 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.916 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.917 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:54.917 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.918 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.919 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ts0kfgeAtAhtokMNE_QimI9", 2022-09-02T04:50:54.919 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.920 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.921 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.923 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.839577_129.1", 2022-09-02T04:50:54.923 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.924 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:54.924 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.839577631Z", 2022-09-02T04:50:54.925 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.925 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.926 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.927 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.928 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.928 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:54.929 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.930 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.930 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.843007_130.1", 2022-09-02T04:50:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.933 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:54.933 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.843007979Z", 2022-09-02T04:50:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.938 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.844233_131.1", 2022-09-02T04:50:54.939 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.939 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19", 2022-09-02T04:50:54.939 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.844233977Z", 2022-09-02T04:50:54.939 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.940 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.940 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.940 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.941 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.941 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:54.941 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.942 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.942 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.943 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.847167_132.1", 2022-09-02T04:50:54.944 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.944 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19", 2022-09-02T04:50:54.944 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.847167513Z", 2022-09-02T04:50:54.944 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.945 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.945 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.945 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.946 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.946 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_kPWK_Wv0ieYeE_BaUj3gzWr", 2022-09-02T04:50:54.946 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.947 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.948 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.863844_133.1", 2022-09-02T04:50:54.949 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.949 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:54.949 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.863844546Z", 2022-09-02T04:50:54.949 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.950 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.950 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:54.950 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.951 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.951 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:54.951 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.952 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.952 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.953 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093133.866884_134.1", 2022-09-02T04:50:54.953 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.954 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T04:50:54.954 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:32:13.866884082Z", 2022-09-02T04:50:54.954 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.954 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.955 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:54.955 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.955 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.956 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KwjW3PecupwHdr9iKxKR_MH", 2022-09-02T04:50:54.956 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.956 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.957 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.958 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.478355_147.1", 2022-09-02T04:50:54.958 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.958 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:54.959 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.478355264Z", 2022-09-02T04:50:54.959 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.959 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.959 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.960 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.960 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.960 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:54.961 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.961 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.961 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.963 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.480846_151.1", 2022-09-02T04:50:54.963 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.963 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:54.963 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.480846805Z", 2022-09-02T04:50:54.964 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.964 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.964 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.965 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.965 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:54.966 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.967 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.967 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.968 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.481682_152.1", 2022-09-02T04:50:54.968 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.969 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20", 2022-09-02T04:50:54.969 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.481682426Z", 2022-09-02T04:50:54.969 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.969 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.970 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.970 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.970 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.971 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:54.971 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.971 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.972 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.973 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093748.483984_153.1", 2022-09-02T04:50:54.973 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:54.973 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20", 2022-09-02T04:50:54.974 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:42:28.483984075Z", 2022-09-02T04:50:54.974 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.974 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.974 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:54.975 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:54.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.975 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.975 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4bhbDRXmCTeTC-tOtjs0lRB", 2022-09-02T04:50:54.976 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.976 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.976 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:54.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.983 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093786.258112_154.1", 2022-09-02T04:50:54.984 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.986 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:54.986 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:43:06.258112912Z", 2022-09-02T04:50:54.987 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:54.988 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:54.988 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:54.988 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:54.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.989 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:54.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:54.990 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:54.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.991 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:54.993 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:54.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:54.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093786.261185_155.1", 2022-09-02T04:50:54.996 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:54.997 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:54.997 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:43:06.261185635Z", 2022-09-02T04:50:54.999 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.000 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.000 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:55.001 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.002 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.002 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:55.003 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.004 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.005 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.008 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093904.758354_156.1", 2022-09-02T04:50:55.008 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.010 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:55.010 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.758354256Z", 2022-09-02T04:50:55.010 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.011 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:55.012 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.013 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.013 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:55.014 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.015 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.015 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.017 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093904.761841_157.1", 2022-09-02T04:50:55.018 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.019 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T04:50:55.020 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:45:04.761841925Z", 2022-09-02T04:50:55.021 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.021 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.022 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:55.022 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.023 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.023 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_zc-Kata49XH9uePchM9uCXb", 2022-09-02T04:50:55.023 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.024 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.024 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.027 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.643237_158.1", 2022-09-02T04:50:55.027 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.028 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:55.028 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.643237395Z", 2022-09-02T04:50:55.028 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.029 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.030 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.030 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.031 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.031 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:55.032 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.033 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.034 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.034 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.035 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.036 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.646348_159.1", 2022-09-02T04:50:55.036 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.037 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:55.037 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.646348674Z", 2022-09-02T04:50:55.038 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.038 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.039 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.039 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.040 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.040 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:55.041 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.042 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.042 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.044 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.647493_160.1", 2022-09-02T04:50:55.045 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.045 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21", 2022-09-02T04:50:55.045 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.647493391Z", 2022-09-02T04:50:55.046 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.046 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.047 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.047 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.048 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.048 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:55.049 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.050 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.050 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.052 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.650453_161.1", 2022-09-02T04:50:55.053 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.053 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21", 2022-09-02T04:50:55.054 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.650453266Z", 2022-09-02T04:50:55.054 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.054 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.055 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.055 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.056 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.057 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ku5uw7mOhuoajP4whti092T", 2022-09-02T04:50:55.057 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.058 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.059 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.059 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.061 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.671385_162.1", 2022-09-02T04:50:55.061 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.062 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:55.062 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.671385092Z", 2022-09-02T04:50:55.063 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.063 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.064 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.065 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.065 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:55.066 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.067 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.067 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.069 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.677725_163.1", 2022-09-02T04:50:55.070 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.071 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:55.071 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.677725006Z", 2022-09-02T04:50:55.072 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.072 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.073 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.073 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.074 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.074 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:55.075 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.076 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.076 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.079 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.679743_164.1", 2022-09-02T04:50:55.080 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.080 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22", 2022-09-02T04:50:55.081 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.679743170Z", 2022-09-02T04:50:55.081 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.082 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.083 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.084 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.084 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:55.085 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.085 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.086 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.090 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.685252_165.1", 2022-09-02T04:50:55.090 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.090 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22", 2022-09-02T04:50:55.091 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.685252272Z", 2022-09-02T04:50:55.091 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.091 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.091 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.092 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.093 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.093 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_PzzfcnyFiAyj0NmLkGI4rbM", 2022-09-02T04:50:55.093 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.094 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.094 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.094 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.095 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.095 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.720122_166.1", 2022-09-02T04:50:55.096 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.096 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:55.096 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.720122481Z", 2022-09-02T04:50:55.096 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.097 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.097 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.097 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.098 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.098 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:55.098 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.099 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.099 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.099 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.099 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.100 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.722787_167.1", 2022-09-02T04:50:55.100 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.101 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:55.101 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.722787347Z", 2022-09-02T04:50:55.101 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.102 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.102 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.102 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.103 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.103 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:55.103 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.104 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.104 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.106 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.723810_168.1", 2022-09-02T04:50:55.107 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.107 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23", 2022-09-02T04:50:55.108 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.723810483Z", 2022-09-02T04:50:55.108 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.109 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.109 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.115 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.117 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.117 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:55.118 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.119 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.119 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.120 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.120 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.121 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.726983_169.1", 2022-09-02T04:50:55.122 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.123 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23", 2022-09-02T04:50:55.123 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.726983239Z", 2022-09-02T04:50:55.124 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.124 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.124 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.125 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.126 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.126 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_gm9lzq8cKMljU1MUpMEPo31", 2022-09-02T04:50:55.127 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.128 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.128 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.129 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.129 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.131 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.748449_170.1", 2022-09-02T04:50:55.131 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.132 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:55.132 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.748449921Z", 2022-09-02T04:50:55.133 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.134 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.141 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:55.141 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.142 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.143 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:55.143 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.144 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.144 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.147 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.752174_171.1", 2022-09-02T04:50:55.147 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.148 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T04:50:55.148 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.752174857Z", 2022-09-02T04:50:55.149 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.149 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.150 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:55.150 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.151 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.152 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_CnDmS806S2weqEtCqUVKIvG", 2022-09-02T04:50:55.152 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.153 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.153 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.772370_172.1", 2022-09-02T04:50:55.156 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.157 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:55.157 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.772370930Z", 2022-09-02T04:50:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.158 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:55.159 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.160 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.161 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:55.161 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.162 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.163 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.164 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.166 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.775210_173.1", 2022-09-02T04:50:55.166 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T04:50:55.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.775210687Z", 2022-09-02T04:50:55.168 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.168 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.169 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:55.169 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.170 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.170 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_c_LYTFboJf9t8VARF_i4c_o", 2022-09-02T04:50:55.171 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.172 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.173 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.174 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.174 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.796060_174.1", 2022-09-02T04:50:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.796060843Z", 2022-09-02T04:50:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:55.178 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662093961.799397_175.1", 2022-09-02T04:50:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T04:50:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:46:01.799397586Z", 2022-09-02T04:50:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.183 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_qUI3pCb7qhCqKzoKhi-CHW1", 2022-09-02T04:50:55.183 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.185 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.185 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.480028_178.1", 2022-09-02T04:50:55.188 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.188 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:55.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.480028626Z", 2022-09-02T04:50:55.188 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.195 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.196 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.198 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.198 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.199 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.199 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.200 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.200 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.200 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.202 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.483732_179.1", 2022-09-02T04:50:55.203 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.203 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:55.204 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.483732364Z", 2022-09-02T04:50:55.204 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.204 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.205 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.205 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.206 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.207 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.207 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.211 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.484607_180.1", 2022-09-02T04:50:55.211 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.212 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24", 2022-09-02T04:50:55.212 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.484607016Z", 2022-09-02T04:50:55.213 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.213 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.213 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.214 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:55.216 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.219 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.487046_181.1", 2022-09-02T04:50:55.220 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24", 2022-09-02T04:50:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.487046711Z", 2022-09-02T04:50:55.222 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.222 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.223 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.223 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.224 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.224 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_csnpoUYuMe5U8LjOEugXvjz", 2022-09-02T04:50:55.225 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.226 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.227 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.228 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.228 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.228 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.229 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.508152_182.1", 2022-09-02T04:50:55.229 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.229 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:55.230 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.508152912Z", 2022-09-02T04:50:55.230 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.230 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.511557_183.1", 2022-09-02T04:50:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.511557664Z", 2022-09-02T04:50:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.512912_184.1", 2022-09-02T04:50:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25", 2022-09-02T04:50:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.512912424Z", 2022-09-02T04:50:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.515118_185.1", 2022-09-02T04:50:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25", 2022-09-02T04:50:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.515118385Z", 2022-09-02T04:50:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_WOxitqMSZWCNB48JPD6zBvX", 2022-09-02T04:50:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.550248_186.1", 2022-09-02T04:50:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.550248103Z", 2022-09-02T04:50:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.552820_187.1", 2022-09-02T04:50:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.552820973Z", 2022-09-02T04:50:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.553829_188.1", 2022-09-02T04:50:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26", 2022-09-02T04:50:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.553829203Z", 2022-09-02T04:50:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.556273_189.1", 2022-09-02T04:50:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26", 2022-09-02T04:50:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.556273838Z", 2022-09-02T04:50:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4dGM8Pdte0vDvsqdhlIEVGQ", 2022-09-02T04:50:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.583634_190.1", 2022-09-02T04:50:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.583634061Z", 2022-09-02T04:50:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.587708_191.1", 2022-09-02T04:50:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.587708089Z", 2022-09-02T04:50:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.607965_192.1", 2022-09-02T04:50:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.607965005Z", 2022-09-02T04:50:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.614764_193.1", 2022-09-02T04:50:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T04:50:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.614764199Z", 2022-09-02T04:50:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KiZyjt2drN9P6jmaR5WWgtf", 2022-09-02T04:50:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.635864_194.1", 2022-09-02T04:50:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.635864908Z", 2022-09-02T04:50:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.336 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.336 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.336 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.337 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.337 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.337 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.338 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.640272_195.1", 2022-09-02T04:50:55.339 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.339 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:55.339 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.640272673Z", 2022-09-02T04:50:55.339 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.340 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.340 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.340 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.341 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.341 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.342 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.342 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.343 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.344 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.346 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.661758_196.1", 2022-09-02T04:50:55.347 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.347 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:55.348 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.661758657Z", 2022-09-02T04:50:55.348 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.349 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.349 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.350 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.351 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.351 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.352 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.353 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.353 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.354 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.355 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.356 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.664046_197.1", 2022-09-02T04:50:55.357 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.357 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T04:50:55.358 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.664046043Z", 2022-09-02T04:50:55.358 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.359 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.359 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.360 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.361 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.362 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTGzt35ODe4takOx0Ijo-Hm", 2022-09-02T04:50:55.362 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.363 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.364 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.365 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.365 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.366 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.683056_198.1", 2022-09-02T04:50:55.367 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.367 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:55.368 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.683056320Z", 2022-09-02T04:50:55.368 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.368 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.368 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.369 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.369 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.370 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.370 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.371 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.371 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.372 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.372 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.374 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.685741_199.1", 2022-09-02T04:50:55.374 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.375 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:55.376 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.685741330Z", 2022-09-02T04:50:55.377 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.377 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.378 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.378 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.379 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.380 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.380 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.381 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.382 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.384 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.701672_200.1", 2022-09-02T04:50:55.385 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.385 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:55.386 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.701672940Z", 2022-09-02T04:50:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.389 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.391 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.391 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.391 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.393 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094028.711326_201.1", 2022-09-02T04:50:55.393 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.394 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T04:50:55.394 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:08.711326169Z", 2022-09-02T04:50:55.395 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.395 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.396 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.396 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.397 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.397 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VFA0IfxAEfj9ZnIgf3caZLc", 2022-09-02T04:50:55.397 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 3 2022-09-02T04:50:55.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.398 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.399 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.399 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.399 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.401 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.596190_217.1", 2022-09-02T04:50:55.401 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.402 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:55.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.596190558Z", 2022-09-02T04:50:55.403 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.403 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.403 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.404 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.404 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.404 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:55.405 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.405 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.405 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.407 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.599340_218.1", 2022-09-02T04:50:55.407 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.407 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:55.407 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.599340364Z", 2022-09-02T04:50:55.408 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.408 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.408 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.409 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.410 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.410 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:55.410 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.411 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.411 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.411 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.412 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.412 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.600264_219.1", 2022-09-02T04:50:55.413 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.413 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27", 2022-09-02T04:50:55.413 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.600264022Z", 2022-09-02T04:50:55.413 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.414 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.414 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.414 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.415 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.415 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:55.415 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.416 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.416 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.416 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.417 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.417 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.602614_220.1", 2022-09-02T04:50:55.418 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.418 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27", 2022-09-02T04:50:55.418 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.602614533Z", 2022-09-02T04:50:55.418 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.419 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.419 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.423 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.424 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.424 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_VMKZ1hIZ4BrNvfWoN9A7j6g", 2022-09-02T04:50:55.425 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.425 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.426 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.427 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.624417_221.1", 2022-09-02T04:50:55.427 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.428 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:55.428 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.624417644Z", 2022-09-02T04:50:55.428 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.428 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.429 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.429 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.430 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.430 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:55.430 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.431 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.431 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.431 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.431 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.432 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.628627_222.1", 2022-09-02T04:50:55.432 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.433 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:55.433 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.628627812Z", 2022-09-02T04:50:55.433 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.434 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.434 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.434 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.435 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.435 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:55.435 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.436 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.436 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.437 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.629582_223.1", 2022-09-02T04:50:55.437 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.438 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28", 2022-09-02T04:50:55.438 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.629582051Z", 2022-09-02T04:50:55.439 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.439 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.440 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.440 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.441 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.441 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:55.441 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.634049_224.1", 2022-09-02T04:50:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28", 2022-09-02T04:50:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.634049176Z", 2022-09-02T04:50:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.445 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.445 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.445 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.446 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.446 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-rQG-2qCZRLxb48hq4fo6dM", 2022-09-02T04:50:55.446 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.447 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.447 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.447 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.448 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.448 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.674265_225.1", 2022-09-02T04:50:55.449 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.449 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:55.449 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.674265806Z", 2022-09-02T04:50:55.450 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.450 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.451 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.451 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.452 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.452 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:55.452 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.453 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.453 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.453 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.454 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.454 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.680208_226.1", 2022-09-02T04:50:55.455 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.455 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:55.455 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.680208008Z", 2022-09-02T04:50:55.466 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.467 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.467 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.467 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.468 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.468 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:55.468 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.469 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.469 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.469 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.470 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.471 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.682665_227.1", 2022-09-02T04:50:55.471 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.471 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29", 2022-09-02T04:50:55.472 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.682665330Z", 2022-09-02T04:50:55.472 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.472 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.473 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.473 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.474 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.474 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:55.474 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.475 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.475 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.475 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.475 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.476 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.686711_228.1", 2022-09-02T04:50:55.477 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.477 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29", 2022-09-02T04:50:55.477 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.686711368Z", 2022-09-02T04:50:55.477 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.478 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.478 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.478 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.479 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.479 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tfh5W32ktk4YCz_zFIg3V0P", 2022-09-02T04:50:55.479 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.480 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.480 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.481 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.718596_229.1", 2022-09-02T04:50:55.482 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.482 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:55.482 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.718596260Z", 2022-09-02T04:50:55.482 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.483 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.483 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:55.483 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.484 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.484 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:55.484 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.485 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.485 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.485 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.485 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.486 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.722825_230.1", 2022-09-02T04:50:55.487 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.487 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-27:b210c8c8-c433-420b-87c1-57ebfc433073.4350.27", 2022-09-02T04:50:55.487 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.722825082Z", 2022-09-02T04:50:55.487 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.488 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.488 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:55.488 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_d9CLwmHAAIERJkcIB7dQoJz", 2022-09-02T04:50:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.491 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.765767_231.1", 2022-09-02T04:50:55.491 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.492 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:55.492 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.765767335Z", 2022-09-02T04:50:55.492 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.493 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.493 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:55.493 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.494 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.494 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:55.495 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.495 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.496 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.498 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.500 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.768248_232.1", 2022-09-02T04:50:55.500 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-28:b210c8c8-c433-420b-87c1-57ebfc433073.4350.28", 2022-09-02T04:50:55.501 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.768248326Z", 2022-09-02T04:50:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.502 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.503 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:55.503 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.505 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.505 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_npWwVSYmL1EnHk-QLn6rqtn", 2022-09-02T04:50:55.506 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.507 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.507 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.509 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.795442_233.1", 2022-09-02T04:50:55.509 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.509 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:55.509 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.795442979Z", 2022-09-02T04:50:55.510 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.510 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.510 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:55.510 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.511 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.511 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:55.511 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.512 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.512 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.512 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.515 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.797953_234.1", 2022-09-02T04:50:55.515 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.515 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-29:b210c8c8-c433-420b-87c1-57ebfc433073.4350.29", 2022-09-02T04:50:55.516 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.797953373Z", 2022-09-02T04:50:55.516 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.517 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.517 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:55.518 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.519 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.519 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_tcNb8zKSaOfoO4Kh7yJojkI", 2022-09-02T04:50:55.519 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.520 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.521 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.523 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.841868_235.1", 2022-09-02T04:50:55.523 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.524 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:55.524 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.841868706Z", 2022-09-02T04:50:55.525 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.525 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.526 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.526 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.527 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.528 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:55.528 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.529 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.529 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.532 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.846202_236.1", 2022-09-02T04:50:55.532 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.533 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:55.533 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.846202210Z", 2022-09-02T04:50:55.534 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.534 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.535 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.535 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.536 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.538 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.538 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.847338_237.1", 2022-09-02T04:50:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.541 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30", 2022-09-02T04:50:55.542 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.847338068Z", 2022-09-02T04:50:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.543 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.545 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.545 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.549 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.851413_238.1", 2022-09-02T04:50:55.549 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.550 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30", 2022-09-02T04:50:55.550 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.851413431Z", 2022-09-02T04:50:55.551 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.551 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.552 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.552 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.553 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.553 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GSQVAMuk2iypNdwk6bpcLH6", 2022-09-02T04:50:55.554 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.555 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.555 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.557 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.882196_239.1", 2022-09-02T04:50:55.558 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.558 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:55.559 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.882196828Z", 2022-09-02T04:50:55.559 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.559 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.560 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.561 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.561 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:55.562 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.563 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.563 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.564 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.564 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.567 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.886381_240.1", 2022-09-02T04:50:55.568 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.569 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:55.571 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.886381079Z", 2022-09-02T04:50:55.572 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.572 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.573 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.574 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.574 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:55.575 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.576 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.576 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.579 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.887265_241.1", 2022-09-02T04:50:55.579 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.579 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31", 2022-09-02T04:50:55.580 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.887265472Z", 2022-09-02T04:50:55.582 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.583 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.583 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.584 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.585 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.586 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:55.586 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.587 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.588 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.588 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.591 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.891145_242.1", 2022-09-02T04:50:55.591 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.593 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31", 2022-09-02T04:50:55.593 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.891145853Z", 2022-09-02T04:50:55.593 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.593 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.594 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.594 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.595 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.595 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_NJppyOB8n_mRGNndsFlalJz", 2022-09-02T04:50:55.595 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.596 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.597 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.599 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.923935_243.1", 2022-09-02T04:50:55.599 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.599 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:55.600 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.923935375Z", 2022-09-02T04:50:55.600 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.600 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.601 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.601 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.602 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:55.603 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.604 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.604 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.607 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.927811_244.1", 2022-09-02T04:50:55.607 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.608 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:55.608 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.927811078Z", 2022-09-02T04:50:55.609 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.609 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.610 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.610 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.611 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:55.612 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.613 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.613 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.616 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.928867_245.1", 2022-09-02T04:50:55.616 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.617 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32", 2022-09-02T04:50:55.617 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.928867563Z", 2022-09-02T04:50:55.618 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.618 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.619 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.619 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.620 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.621 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:55.621 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.622 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.623 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.625 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.933635_246.1", 2022-09-02T04:50:55.625 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.625 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32", 2022-09-02T04:50:55.626 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.933635936Z", 2022-09-02T04:50:55.626 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.626 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.627 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.627 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.627 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.628 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Z32uECDB8fUY5EDIBuDcLg4", 2022-09-02T04:50:55.628 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.628 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.629 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.630 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.981889_247.1", 2022-09-02T04:50:55.630 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.630 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:55.631 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.981889010Z", 2022-09-02T04:50:55.631 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.631 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.631 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:55.632 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.632 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.633 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:55.633 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.633 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.635 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094084.984537_248.1", 2022-09-02T04:50:55.635 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.635 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-30:b210c8c8-c433-420b-87c1-57ebfc433073.4350.30", 2022-09-02T04:50:55.635 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:04.984537015Z", 2022-09-02T04:50:55.636 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.636 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.636 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:55.636 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.637 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.637 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LXhRIhQjiTa0YEdb9WqN9ED", 2022-09-02T04:50:55.637 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.638 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.638 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.639 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.069441_249.1", 2022-09-02T04:50:55.640 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.640 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:55.640 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.069441995Z", 2022-09-02T04:50:55.640 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.641 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.641 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:55.641 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.641 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.642 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:55.642 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.644 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.078532_250.1", 2022-09-02T04:50:55.644 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.645 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-31:b210c8c8-c433-420b-87c1-57ebfc433073.4350.31", 2022-09-02T04:50:55.645 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.078532139Z", 2022-09-02T04:50:55.645 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.645 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:55.646 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.646 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.647 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_xDX72nvdgR8oJlewr7XJua3", 2022-09-02T04:50:55.647 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.648 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.648 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.648 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.648 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.649 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.117134_251.1", 2022-09-02T04:50:55.649 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.650 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:55.650 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.117134511Z", 2022-09-02T04:50:55.650 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.650 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.651 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:55.651 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.651 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.652 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:55.652 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.653 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.653 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.653 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.654 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.121445_252.1", 2022-09-02T04:50:55.654 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.655 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-32:b210c8c8-c433-420b-87c1-57ebfc433073.4350.32", 2022-09-02T04:50:55.655 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.121445603Z", 2022-09-02T04:50:55.655 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.655 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.656 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:55.656 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.656 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.657 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_OXfkUI5redRrhD9ZjE1WrwN", 2022-09-02T04:50:55.657 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.657 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.658 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.659 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.200772_253.1", 2022-09-02T04:50:55.659 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.659 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:55.660 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.200772049Z", 2022-09-02T04:50:55.660 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.660 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.660 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.661 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.661 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.661 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:55.662 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.662 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.662 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.663 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.204385_254.1", 2022-09-02T04:50:55.664 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.664 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:55.664 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.204385610Z", 2022-09-02T04:50:55.664 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.665 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.665 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.665 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.666 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.666 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:55.666 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.667 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.667 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.668 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.205324_255.1", 2022-09-02T04:50:55.668 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.669 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33", 2022-09-02T04:50:55.669 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.205324745Z", 2022-09-02T04:50:55.669 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.669 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.670 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.670 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.671 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:55.671 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.671 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.672 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.673 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.211398_256.1", 2022-09-02T04:50:55.673 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.673 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33", 2022-09-02T04:50:55.674 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.211398303Z", 2022-09-02T04:50:55.674 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.674 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.675 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.675 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.675 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.675 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_65hdQBWsT0BjnLkD8TSj7KU", 2022-09-02T04:50:55.676 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.676 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.676 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.677 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.678 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.247878_257.1", 2022-09-02T04:50:55.678 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.678 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:55.678 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.247878027Z", 2022-09-02T04:50:55.679 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.679 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.679 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.679 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.680 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.680 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:55.680 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.681 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.681 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.683 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.251068_258.1", 2022-09-02T04:50:55.683 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.684 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:55.684 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.251068384Z", 2022-09-02T04:50:55.684 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.684 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.685 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.685 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.685 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.686 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:55.686 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.687 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.687 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.687 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.688 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.252037_259.1", 2022-09-02T04:50:55.689 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.689 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34", 2022-09-02T04:50:55.689 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.252037843Z", 2022-09-02T04:50:55.690 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.690 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.690 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.690 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.691 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.691 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:55.692 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.692 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.692 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.694 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.260507_260.1", 2022-09-02T04:50:55.694 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.694 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34", 2022-09-02T04:50:55.694 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.260507523Z", 2022-09-02T04:50:55.695 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.695 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.695 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.695 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.696 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.696 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o3oR7GTEp3cPqpOUQX5-Vlj", 2022-09-02T04:50:55.696 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.697 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.697 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.698 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.324260_261.1", 2022-09-02T04:50:55.699 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.699 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:55.699 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.324260578Z", 2022-09-02T04:50:55.699 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.699 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.700 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.700 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.701 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.701 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:55.701 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.702 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.702 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.703 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.328657_262.1", 2022-09-02T04:50:55.703 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.704 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:55.704 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.328657368Z", 2022-09-02T04:50:55.704 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.704 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.705 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.705 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.705 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.706 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:55.706 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.706 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.707 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.708 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.330962_263.1", 2022-09-02T04:50:55.708 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.709 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35", 2022-09-02T04:50:55.709 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.330962269Z", 2022-09-02T04:50:55.709 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.709 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.710 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.710 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.711 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.711 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:55.711 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.712 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.712 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.714 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.335420_264.1", 2022-09-02T04:50:55.714 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.714 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35", 2022-09-02T04:50:55.715 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.335420823Z", 2022-09-02T04:50:55.715 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.715 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.715 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.716 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.716 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.716 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_MV8SekHew1H0R7EwsQ07enh", 2022-09-02T04:50:55.716 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.717 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.717 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.719 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.389275_265.1", 2022-09-02T04:50:55.719 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.719 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:55.719 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.389275554Z", 2022-09-02T04:50:55.720 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.720 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:55.720 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.721 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.721 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:55.721 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.722 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.722 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.723 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.723 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094085.393872_266.1", 2022-09-02T04:50:55.724 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.724 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-33:b210c8c8-c433-420b-87c1-57ebfc433073.4350.33", 2022-09-02T04:50:55.724 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:05.393872877Z", 2022-09-02T04:50:55.724 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.725 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.725 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:55.725 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.726 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.726 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JupxKvRwS6hje7cDtgNncrE", 2022-09-02T04:50:55.726 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.727 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.727 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.728 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094086.434234_267.1", 2022-09-02T04:50:55.728 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.729 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:55.729 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:06.434234679Z", 2022-09-02T04:50:55.729 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.729 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.730 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:55.730 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.730 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.731 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:55.731 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.731 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.732 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.733 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094086.438494_268.1", 2022-09-02T04:50:55.733 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.733 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-34:b210c8c8-c433-420b-87c1-57ebfc433073.4350.34", 2022-09-02T04:50:55.734 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:06.438494895Z", 2022-09-02T04:50:55.734 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.734 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.735 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:55.735 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.735 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.736 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_o_vp37Wih20al_l6D6WLQ_Z", 2022-09-02T04:50:55.736 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.736 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.737 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.738 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.522239_269.1", 2022-09-02T04:50:55.738 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.738 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:55.739 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.522239231Z", 2022-09-02T04:50:55.739 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.739 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.739 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:55.740 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.740 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:55.741 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.741 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.741 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.743 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.526590_270.1", 2022-09-02T04:50:55.743 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.743 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-35:b210c8c8-c433-420b-87c1-57ebfc433073.4350.35", 2022-09-02T04:50:55.743 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.526590558Z", 2022-09-02T04:50:55.744 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.744 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.744 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:55.744 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.745 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.745 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Icq_UOUmER4UuS-lZfnSN00", 2022-09-02T04:50:55.745 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:55.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.746 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.746 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.747 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.588437_271.1", 2022-09-02T04:50:55.748 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.748 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:55.748 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.588437443Z", 2022-09-02T04:50:55.748 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.749 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.749 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.749 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.750 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.750 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:55.750 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.751 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.751 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.752 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.594122_272.1", 2022-09-02T04:50:55.752 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.753 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T04:50:55.753 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.594122094Z", 2022-09-02T04:50:55.753 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.753 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.753 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.754 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.754 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.755 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uhh0uO6_u6-CvYk40e70XxB", 2022-09-02T04:50:55.755 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.755 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.756 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.757 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.596187_273.1", 2022-09-02T04:50:55.757 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.757 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36", 2022-09-02T04:50:55.758 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.596187284Z", 2022-09-02T04:50:55.758 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.758 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.758 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.759 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.759 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.759 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:55.760 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.760 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.760 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.762 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.601998_274.1", 2022-09-02T04:50:55.762 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.762 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-36", 2022-09-02T04:50:55.762 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.601998619Z", 2022-09-02T04:50:55.763 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.763 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.763 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.763 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.764 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.764 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nYphXSr0ZcPhi0RMd53Z0d2", 2022-09-02T04:50:55.764 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.765 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.765 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.768 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.638123_275.1", 2022-09-02T04:50:55.769 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.770 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:55.771 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.638123665Z", 2022-09-02T04:50:55.771 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.772 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.773 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.774 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.774 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:55.774 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.775 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.776 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.781 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.640624_276.1", 2022-09-02T04:50:55.781 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.782 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T04:50:55.782 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.640624183Z", 2022-09-02T04:50:55.782 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.783 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.783 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.784 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.784 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KnGL3U60vDuvygqsTV2tfRK", 2022-09-02T04:50:55.784 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.785 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.785 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.787 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.641616_277.1", 2022-09-02T04:50:55.787 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.787 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37", 2022-09-02T04:50:55.788 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.641616287Z", 2022-09-02T04:50:55.788 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.788 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.788 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.789 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.791 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.791 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:55.791 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.794 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.643956_278.1", 2022-09-02T04:50:55.794 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.794 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-37", 2022-09-02T04:50:55.794 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.643956748Z", 2022-09-02T04:50:55.795 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.795 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.795 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.795 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.796 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.796 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_SXpRk-of0PBJWAFKQQVMwbW", 2022-09-02T04:50:55.797 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.797 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.797 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.799 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.689255_279.1", 2022-09-02T04:50:55.799 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.799 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:55.800 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.689255290Z", 2022-09-02T04:50:55.800 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.800 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.800 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.801 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.802 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.803 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:55.803 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.804 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.804 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.806 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.691696_280.1", 2022-09-02T04:50:55.807 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.807 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T04:50:55.809 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.691696253Z", 2022-09-02T04:50:55.809 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.809 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.810 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.811 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.811 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_z-jZ6-peOw0lw74pXDPqBAk", 2022-09-02T04:50:55.812 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.813 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.813 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.815 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.692552_281.1", 2022-09-02T04:50:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38", 2022-09-02T04:50:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.692552094Z", 2022-09-02T04:50:55.816 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.817 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.817 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.817 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.818 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.818 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:55.818 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.819 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.819 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.820 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.820 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.821 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094087.696193_282.1", 2022-09-02T04:50:55.821 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.821 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-38", 2022-09-02T04:50:55.821 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:07.696193220Z", 2022-09-02T04:50:55.822 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.822 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.822 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.822 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.823 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.823 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_0R_iVDtkaH-yN5AiZC1a-qd", 2022-09-02T04:50:55.823 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.824 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.824 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.825 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.826 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.042350_283.1", 2022-09-02T04:50:55.826 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.826 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:55.826 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.042350878Z", 2022-09-02T04:50:55.827 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.827 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.827 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.827 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.828 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.828 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:55.829 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.830 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.830 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.830 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.831 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.045432_284.1", 2022-09-02T04:50:55.831 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.832 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39:b210c8c8-c433-420b-87c1-57ebfc433073.4350.39", 2022-09-02T04:50:55.832 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.045432525Z", 2022-09-02T04:50:55.832 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.832 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.833 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.833 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.833 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.834 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9KblAUJ8OzFiqEtM28-6H_7", 2022-09-02T04:50:55.834 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.834 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.835 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.836 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.047334_285.1", 2022-09-02T04:50:55.836 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.836 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39", 2022-09-02T04:50:55.837 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.047334461Z", 2022-09-02T04:50:55.837 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.837 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.837 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.838 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.838 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.838 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:55.839 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.839 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.839 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.841 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.050679_286.1", 2022-09-02T04:50:55.841 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.842 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-39", 2022-09-02T04:50:55.842 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.050679259Z", 2022-09-02T04:50:55.842 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.842 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.843 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.843 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.843 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.844 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_U8d3YshAkaicZT7RuRvTN-q", 2022-09-02T04:50:55.844 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.844 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.845 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.846 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.075859_287.1", 2022-09-02T04:50:55.846 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.846 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:55.847 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.075859654Z", 2022-09-02T04:50:55.847 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.847 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.847 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.848 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.848 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.848 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:55.849 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.849 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.849 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.078656_288.1", 2022-09-02T04:50:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40:b210c8c8-c433-420b-87c1-57ebfc433073.4350.40", 2022-09-02T04:50:55.851 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.078656456Z", 2022-09-02T04:50:55.852 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.852 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.852 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.852 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.853 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.853 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wXJx5dyWRnHfaAfAzGH4xwA", 2022-09-02T04:50:55.853 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.854 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.854 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.854 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.855 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.080841_289.1", 2022-09-02T04:50:55.856 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.856 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40", 2022-09-02T04:50:55.856 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.080841716Z", 2022-09-02T04:50:55.856 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.857 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.857 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.857 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.858 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.858 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:55.858 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.859 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.859 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.860 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.086877_290.1", 2022-09-02T04:50:55.860 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.861 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-40", 2022-09-02T04:50:55.861 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.086877398Z", 2022-09-02T04:50:55.861 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.861 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.862 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.862 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.862 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.863 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9dMm5j-dFWKyjWUjjEpRyAp", 2022-09-02T04:50:55.863 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.863 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.865 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.125974_291.1", 2022-09-02T04:50:55.865 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.865 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:55.865 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.125974705Z", 2022-09-02T04:50:55.866 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.866 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.866 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.866 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.867 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.867 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:55.867 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.868 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.868 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.869 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.869 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.870 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.132649_292.1", 2022-09-02T04:50:55.870 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.870 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41:b210c8c8-c433-420b-87c1-57ebfc433073.4350.41", 2022-09-02T04:50:55.870 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.132649142Z", 2022-09-02T04:50:55.871 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.871 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.871 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.871 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.872 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.872 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Yvaxtbfx3bbD7jmzp7dUHyk", 2022-09-02T04:50:55.872 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.873 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.873 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.873 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.874 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.133657_293.1", 2022-09-02T04:50:55.875 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.875 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41", 2022-09-02T04:50:55.876 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.133657858Z", 2022-09-02T04:50:55.876 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.876 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.876 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.877 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.877 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.877 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:55.878 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.878 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.878 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.879 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.879 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.880 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094104.139676_294.1", 2022-09-02T04:50:55.880 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.880 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-41", 2022-09-02T04:50:55.880 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:48:24.139676296Z", 2022-09-02T04:50:55.881 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.881 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.881 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.882 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.882 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.882 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_v392DwYwdDbK4b_qJrKPnYQ", 2022-09-02T04:50:55.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.883 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.885 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.573151_93.1", 2022-09-02T04:50:55.885 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.885 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:55.886 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.573151032Z", 2022-09-02T04:50:55.886 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.886 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.886 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.887 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.887 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.887 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:55.888 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.888 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.888 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.890 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.579025_94.1", 2022-09-02T04:50:55.890 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.890 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1:b210c8c8-c433-420b-87c1-57ebfc433073.4350.1", 2022-09-02T04:50:55.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.579025648Z", 2022-09-02T04:50:55.892 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.892 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.893 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.893 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.898 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.898 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3b5wBLS00P_OjW2OQl7j2uF", 2022-09-02T04:50:55.898 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.899 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.899 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.900 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.900 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.901 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.581233_95.1", 2022-09-02T04:50:55.901 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.901 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1", 2022-09-02T04:50:55.902 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.581233957Z", 2022-09-02T04:50:55.902 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.902 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.903 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.903 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.903 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.904 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:55.904 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.904 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.904 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.906 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.585812_96.1", 2022-09-02T04:50:55.906 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.906 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-1", 2022-09-02T04:50:55.906 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.585812830Z", 2022-09-02T04:50:55.907 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.907 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.907 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.907 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.908 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.908 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_1A8L11MVlTzfxGg0iygZWyo", 2022-09-02T04:50:55.908 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.909 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.909 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.910 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.910 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.617016_97.1", 2022-09-02T04:50:55.911 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.911 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:55.911 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.617016945Z", 2022-09-02T04:50:55.912 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.912 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.912 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.912 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.913 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.913 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:55.914 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.914 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.914 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.915 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.915 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.916 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.620981_98.1", 2022-09-02T04:50:55.916 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.916 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2:b210c8c8-c433-420b-87c1-57ebfc433073.4350.2", 2022-09-02T04:50:55.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.620981265Z", 2022-09-02T04:50:55.917 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.917 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.917 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.917 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.918 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.918 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bMtSVG-w7sHma9sDySacmdF", 2022-09-02T04:50:55.918 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.919 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.920 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.922 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.924 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.622018_99.1", 2022-09-02T04:50:55.924 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.925 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2", 2022-09-02T04:50:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.622018558Z", 2022-09-02T04:50:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.926 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.927 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:55.928 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.929 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.931 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092409.624757_100.1", 2022-09-02T04:50:55.932 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.932 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-2", 2022-09-02T04:50:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:09.624757525Z", 2022-09-02T04:50:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.933 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.934 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.934 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.935 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.936 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4AXM-zyagwAYGKB93ttrGOn", 2022-09-02T04:50:55.936 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.937 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.938 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.630272_101.1", 2022-09-02T04:50:55.940 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.630272240Z", 2022-09-02T04:50:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.941 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.942 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.942 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.942 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.943 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:55.943 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.943 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.944 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.944 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.944 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.945 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.633085_102.1", 2022-09-02T04:50:55.946 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.946 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3:b210c8c8-c433-420b-87c1-57ebfc433073.4350.3", 2022-09-02T04:50:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.633085325Z", 2022-09-02T04:50:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.947 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.948 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.950 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8zH2SospBMR9KYq-awUKcwj", 2022-09-02T04:50:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.952 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.954 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.634091_103.1", 2022-09-02T04:50:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3", 2022-09-02T04:50:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.634091584Z", 2022-09-02T04:50:55.955 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.956 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:55.957 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.958 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.959 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092412.636577_104.1", 2022-09-02T04:50:55.960 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.960 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-3", 2022-09-02T04:50:55.960 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:12.636577040Z", 2022-09-02T04:50:55.960 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.961 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.961 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.961 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.962 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.962 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_syLkxLZpVEXX3ZjV0L8jZ_n", 2022-09-02T04:50:55.962 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.963 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.963 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.964 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.570754_105.1", 2022-09-02T04:50:55.965 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.965 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:55.965 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.570754424Z", 2022-09-02T04:50:55.965 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.966 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.966 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.966 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.967 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.967 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:55.967 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.968 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.969 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.573776_106.1", 2022-09-02T04:50:55.970 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:55.970 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4:b210c8c8-c433-420b-87c1-57ebfc433073.4350.4", 2022-09-02T04:50:55.970 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.573776177Z", 2022-09-02T04:50:55.970 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.971 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.971 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.971 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.972 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.972 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hpmBpbzaQj4zi0tso7qOIS7", 2022-09-02T04:50:55.972 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.973 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.973 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.973 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.973 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.974 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.574795_107.1", 2022-09-02T04:50:55.974 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.975 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4", 2022-09-02T04:50:55.975 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.574795537Z", 2022-09-02T04:50:55.975 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.976 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.977 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.978 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.979 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.984 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:55.984 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.985 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.985 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:55.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.988 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.577651_108.1", 2022-09-02T04:50:55.989 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:55.989 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-4", 2022-09-02T04:50:55.990 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.577651116Z", 2022-09-02T04:50:55.990 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:55.991 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:55.991 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:55.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:55.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.994 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:55.994 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_XqYEDYkbfEWzFo3xEarPMpM", 2022-09-02T04:50:55.995 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:55.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.997 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:55.997 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:55.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:55.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:55.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:55.999 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.609918_109.1", 2022-09-02T04:50:56.000 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.000 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:56.001 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.609918035Z", 2022-09-02T04:50:56.001 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.002 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.002 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.003 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.005 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.005 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:56.006 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.007 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.008 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.009 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.009 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.011 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.613337_110.1", 2022-09-02T04:50:56.011 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.012 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5:b210c8c8-c433-420b-87c1-57ebfc433073.4350.5", 2022-09-02T04:50:56.012 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.613337726Z", 2022-09-02T04:50:56.013 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.013 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.013 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.014 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.015 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.016 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_lAJ_YpqnGSrl-KTE8QIqP2e", 2022-09-02T04:50:56.016 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.017 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.018 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.020 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.614272_111.1", 2022-09-02T04:50:56.020 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.021 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5", 2022-09-02T04:50:56.021 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.614272060Z", 2022-09-02T04:50:56.022 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.023 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.023 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.024 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.025 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.025 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:56.026 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.027 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.028 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.028 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.030 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.618746_112.1", 2022-09-02T04:50:56.031 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.031 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-5", 2022-09-02T04:50:56.032 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.618746992Z", 2022-09-02T04:50:56.032 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.033 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.034 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.034 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.034 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.035 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_-hTZCJzTedJnsxq2bixh54a", 2022-09-02T04:50:56.035 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.035 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.036 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.036 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.036 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.037 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.654802_113.1", 2022-09-02T04:50:56.037 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.038 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:56.038 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.654802951Z", 2022-09-02T04:50:56.038 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.038 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.039 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.039 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.039 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.040 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:56.040 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.040 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.041 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.041 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.041 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.042 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.657918_114.1", 2022-09-02T04:50:56.042 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.043 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6:b210c8c8-c433-420b-87c1-57ebfc433073.4350.6", 2022-09-02T04:50:56.043 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.657918674Z", 2022-09-02T04:50:56.043 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.043 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.044 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.044 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.044 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.045 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_061zzkoLd5_nGDKCw5D2-iE", 2022-09-02T04:50:56.045 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.046 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.046 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.046 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.048 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.660134_115.1", 2022-09-02T04:50:56.048 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.048 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6", 2022-09-02T04:50:56.049 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.660134944Z", 2022-09-02T04:50:56.049 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.049 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.050 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.050 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.051 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.052 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:56.052 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.053 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.054 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.055 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.056 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092418.665018_116.1", 2022-09-02T04:50:56.057 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.057 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-6", 2022-09-02T04:50:56.058 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:18.665018151Z", 2022-09-02T04:50:56.058 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.059 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.059 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.060 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.061 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.062 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_X39T5Ps6MLi2Isnjy0knQBQ", 2022-09-02T04:50:56.062 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.063 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.064 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.066 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.769933_117.1", 2022-09-02T04:50:56.066 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.066 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:56.067 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.769933252Z", 2022-09-02T04:50:56.067 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.067 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.068 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.068 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.069 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.070 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:56.070 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.071 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.072 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.074 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.772485_118.1", 2022-09-02T04:50:56.074 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.075 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:56.075 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.772485812Z", 2022-09-02T04:50:56.076 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.076 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.076 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.077 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.078 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.078 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:56.079 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.079 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.080 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.082 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.773480_119.1", 2022-09-02T04:50:56.082 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.083 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:56.083 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.773480228Z", 2022-09-02T04:50:56.084 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.093 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.093 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.094 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.095 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.095 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:56.096 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.097 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.097 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.099 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.776581_120.1", 2022-09-02T04:50:56.100 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.100 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:56.100 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.776581648Z", 2022-09-02T04:50:56.101 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.101 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.102 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.102 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.103 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.103 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:56.104 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.105 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.105 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.107 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.801099_121.1", 2022-09-02T04:50:56.108 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.108 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:56.108 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.801099540Z", 2022-09-02T04:50:56.109 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.109 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.110 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.111 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.111 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.112 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:56.112 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.113 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.113 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.116 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.804401_122.1", 2022-09-02T04:50:56.116 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.117 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:56.117 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.804401413Z", 2022-09-02T04:50:56.118 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.118 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.119 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.119 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.120 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.121 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:56.121 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.122 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.123 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.123 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.125 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.805277_123.1", 2022-09-02T04:50:56.125 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.126 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:56.126 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.805277428Z", 2022-09-02T04:50:56.127 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.127 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.128 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.128 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.129 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.130 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:56.130 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.131 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.132 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.132 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.134 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.807715_124.1", 2022-09-02T04:50:56.135 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.135 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:56.136 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.807715596Z", 2022-09-02T04:50:56.137 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.137 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.138 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.138 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.139 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.140 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:56.140 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.141 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.142 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.143 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.144 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.842266_125.1", 2022-09-02T04:50:56.145 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.145 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:56.146 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.842266349Z", 2022-09-02T04:50:56.146 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.147 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.147 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.148 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.149 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.149 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:56.150 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.151 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.151 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.154 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.853430_126.1", 2022-09-02T04:50:56.154 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.155 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:56.155 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.853430958Z", 2022-09-02T04:50:56.155 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.156 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.156 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.157 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.158 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.158 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:56.159 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.160 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.160 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.163 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.855671_127.1", 2022-09-02T04:50:56.163 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.163 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:56.164 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.855671199Z", 2022-09-02T04:50:56.164 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.165 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.166 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.166 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.167 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.167 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:56.168 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.169 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.169 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.170 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.170 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.172 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092420.862622_128.1", 2022-09-02T04:50:56.172 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.173 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:56.173 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:20.862622338Z", 2022-09-02T04:50:56.174 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.174 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.175 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.176 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.176 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:56.177 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.178 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.179 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.181 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.580527_129.1", 2022-09-02T04:50:56.181 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.182 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:56.182 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.580527505Z", 2022-09-02T04:50:56.183 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.183 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:56.184 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.185 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.186 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:56.186 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.187 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.187 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.190 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.583272_130.1", 2022-09-02T04:50:56.190 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.191 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7:b210c8c8-c433-420b-87c1-57ebfc433073.4350.7", 2022-09-02T04:50:56.191 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.583272249Z", 2022-09-02T04:50:56.191 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.192 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.192 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:56.193 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.193 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.194 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_JObJBeIa1mqhHethXlAYtc7", 2022-09-02T04:50:56.195 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.196 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.196 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.206 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.595303_131.1", 2022-09-02T04:50:56.207 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.208 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:56.208 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.595303694Z", 2022-09-02T04:50:56.209 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.209 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.210 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:56.210 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.211 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.212 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:56.212 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.213 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.214 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2022-09-02T04:50:56.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.215 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.216 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.599169_132.1", 2022-09-02T04:50:56.217 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.217 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-7", 2022-09-02T04:50:56.218 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.599169915Z", 2022-09-02T04:50:56.219 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.219 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:56.220 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.221 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.221 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8WkKslYZOhbEGgtRHIaJ7I9", 2022-09-02T04:50:56.222 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.223 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.223 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.224 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.224 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.226 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.638539_133.1", 2022-09-02T04:50:56.226 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.227 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:56.228 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.638539023Z", 2022-09-02T04:50:56.228 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.229 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.229 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:56.230 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.231 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.231 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:56.232 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.232 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.233 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.233 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.234 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.235 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.645764_134.1", 2022-09-02T04:50:56.235 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.236 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8:b210c8c8-c433-420b-87c1-57ebfc433073.4350.8", 2022-09-02T04:50:56.236 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.645764849Z", 2022-09-02T04:50:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.238 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:56.239 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.240 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.240 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uGelrVFrz9ICcAthoWdwZJ7", 2022-09-02T04:50:56.241 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.243 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.243 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.244 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.244 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.246 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.659515_135.1", 2022-09-02T04:50:56.246 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.659515593Z", 2022-09-02T04:50:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.248 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.248 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:56.248 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.249 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.250 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:56.250 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.251 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.252 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2022-09-02T04:50:56.252 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.254 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.662895_136.1", 2022-09-02T04:50:56.254 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.255 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-8", 2022-09-02T04:50:56.255 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.662895104Z", 2022-09-02T04:50:56.255 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.256 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.256 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:56.257 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.257 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.257 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_fYxsHAScr18IoeFPZAOJ-Qb", 2022-09-02T04:50:56.258 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.259 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.260 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.262 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.751579_137.1", 2022-09-02T04:50:56.263 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.263 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:56.263 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.751579938Z", 2022-09-02T04:50:56.264 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.264 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.265 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:56.265 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.267 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.267 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:56.268 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.270 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.270 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.271 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.272 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.754025_138.1", 2022-09-02T04:50:56.272 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.272 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9:b210c8c8-c433-420b-87c1-57ebfc433073.4350.9", 2022-09-02T04:50:56.272 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.754025410Z", 2022-09-02T04:50:56.273 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.273 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.273 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:56.273 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.274 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.274 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_KJNyS8OyDkrF7B5ARMfvynA", 2022-09-02T04:50:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.275 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.769139_139.1", 2022-09-02T04:50:56.277 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.769139327Z", 2022-09-02T04:50:56.278 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:56.280 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.281 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "remove" 2022-09-02T04:50:56.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.284 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662092421.771218_140.1", 2022-09-02T04:50:56.284 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.284 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-9", 2022-09-02T04:50:56.285 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218052Z", 2022-09-02T04:50:56.286 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.286 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.286 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:56.286 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.287 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.287 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ebU_u_CGYyeS_yMVbpQ8-w-", 2022-09-02T04:50:56.287 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 2 2022-09-02T04:50:56.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.288 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.288 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.289 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:50:56.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.292 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:56.384 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.352302_101.1", 2022-09-02T04:50:56.385 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.385 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.385 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.352302088Z", 2022-09-02T04:50:56.385 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.387 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.388 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.389 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.392 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.355560_102.1", 2022-09-02T04:50:56.392 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.355560597Z", 2022-09-02T04:50:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.356896_103.1", 2022-09-02T04:50:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.356896760Z", 2022-09-02T04:50:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.402 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.403 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.403 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.403 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.404 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.405 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.407 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.359950_104.1", 2022-09-02T04:50:56.407 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.408 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.408 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.359950824Z", 2022-09-02T04:50:56.409 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.409 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.410 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.410 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.411 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.412 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.412 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.415 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.410774_105.1", 2022-09-02T04:50:56.416 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.416 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.410774824Z", 2022-09-02T04:50:56.417 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.417 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.418 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.418 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.419 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.420 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.420 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.424 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.413915_106.1", 2022-09-02T04:50:56.424 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.425 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.425 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.413915554Z", 2022-09-02T04:50:56.426 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.426 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.427 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.427 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.428 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.429 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.429 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.432 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.415076_107.1", 2022-09-02T04:50:56.433 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.433 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.433 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.415076727Z", 2022-09-02T04:50:56.433 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.434 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.434 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.434 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.435 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.435 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.436 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.436 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.436 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.438 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.417998_108.1", 2022-09-02T04:50:56.438 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.438 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.417998554Z", 2022-09-02T04:50:56.439 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.439 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.439 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.439 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.443 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:56.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.445 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:56.504 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:56.505 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:56.538 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:50:56.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.539 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.352302_101.1", 2022-09-02T04:50:56.540 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.540 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.541 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.352302088Z", 2022-09-02T04:50:56.541 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.542 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.542 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.543 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.544 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.544 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.544 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.545 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.546 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.548 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.355560_102.1", 2022-09-02T04:50:56.549 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.549 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.550 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.355560597Z", 2022-09-02T04:50:56.550 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.551 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.551 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.551 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.552 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.553 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.553 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.554 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.555 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.557 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.356896_103.1", 2022-09-02T04:50:56.558 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.559 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.559 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.356896760Z", 2022-09-02T04:50:56.560 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.560 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.560 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.561 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.561 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.561 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.563 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.563 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.564 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.359950_104.1", 2022-09-02T04:50:56.564 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.565 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.565 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.359950824Z", 2022-09-02T04:50:56.565 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.565 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.566 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.566 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.569 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.569 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.569 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.570 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.571 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.574 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.410774_105.1", 2022-09-02T04:50:56.574 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.575 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.575 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.410774824Z", 2022-09-02T04:50:56.576 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.576 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.577 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.577 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.578 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.579 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.579 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.580 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.580 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.581 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.413915_106.1", 2022-09-02T04:50:56.582 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.582 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.582 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.413915554Z", 2022-09-02T04:50:56.582 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.583 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.583 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.583 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.584 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.584 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.584 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.585 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.585 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.586 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.415076_107.1", 2022-09-02T04:50:56.586 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.587 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.587 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.415076727Z", 2022-09-02T04:50:56.588 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.588 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.589 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.589 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.589 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.590 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.590 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.590 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.591 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.592 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.417998_108.1", 2022-09-02T04:50:56.592 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.592 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.593 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.417998554Z", 2022-09-02T04:50:56.593 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.593 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.593 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.594 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.594 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.594 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.595 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.595 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.597 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.598 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:50:56.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.600 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:56.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:56.684 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:56.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.686 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.352302_101.1", 2022-09-02T04:50:56.686 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.686 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.686 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.352302088Z", 2022-09-02T04:50:56.687 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.687 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.687 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.688 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.689 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.689 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.689 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.707 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.707 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.708 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.709 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.712 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.355560_102.1", 2022-09-02T04:50:56.712 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.713 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.714 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.355560597Z", 2022-09-02T04:50:56.714 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.715 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.715 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.716 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.717 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.717 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.718 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.719 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.722 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.723 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.356896_103.1", 2022-09-02T04:50:56.723 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.723 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.724 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.356896760Z", 2022-09-02T04:50:56.724 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.724 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.724 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.725 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.725 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.725 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.726 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.726 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.727 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.728 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.359950_104.1", 2022-09-02T04:50:56.728 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.729 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.359950824Z", 2022-09-02T04:50:56.729 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.729 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.731 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.731 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.731 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.732 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.732 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.733 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.733 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.734 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.410774_105.1", 2022-09-02T04:50:56.734 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.735 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.410774824Z", 2022-09-02T04:50:56.735 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.736 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.736 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.736 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.737 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.737 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.737 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.738 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.738 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.739 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.740 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.413915_106.1", 2022-09-02T04:50:56.740 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.740 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.741 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.413915554Z", 2022-09-02T04:50:56.741 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.741 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.742 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.742 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.743 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.743 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.743 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.744 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.744 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.745 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.415076_107.1", 2022-09-02T04:50:56.746 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.746 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.415076727Z", 2022-09-02T04:50:56.746 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.747 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.747 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.747 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.748 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.748 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.748 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:56.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:56.750 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094176.417998_108.1", 2022-09-02T04:50:56.751 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:56.751 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:49:36.417998554Z", 2022-09-02T04:50:56.751 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:56.752 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:56.752 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:56.752 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:56.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.753 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:56.753 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.753 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:56.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:56.754 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:56.755 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:56.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:56.757 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:56.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.758 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:56.810 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:56.811 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:56.848 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:50:56.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.850 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.352302_101.1", 2022-09-02T04:50:56.850 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.850 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.851 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.352302088Z", 2022-09-02T04:50:56.851 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.851 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.851 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.852 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.852 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.853 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.853 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.854 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.854 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.856 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.355560_102.1", 2022-09-02T04:50:56.856 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.857 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.1", 2022-09-02T04:50:56.857 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.355560597Z", 2022-09-02T04:50:56.857 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.858 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.858 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.858 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.859 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.860 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_No69KkbzciI5DoQ_gJQiqrW", 2022-09-02T04:50:56.860 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.861 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.862 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.866 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.356896_103.1", 2022-09-02T04:50:56.866 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.867 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.867 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.356896760Z", 2022-09-02T04:50:56.868 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.868 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.869 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.869 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.870 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.871 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.871 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.873 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.873 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.875 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.875 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.877 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.359950_104.1", 2022-09-02T04:50:56.877 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.877 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-42", 2022-09-02T04:50:56.878 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.359950824Z", 2022-09-02T04:50:56.878 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.879 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.880 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.880 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_uulo7MoXad3rnLzd8zzpjwe", 2022-09-02T04:50:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.410774_105.1", 2022-09-02T04:50:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.410774824Z", 2022-09-02T04:50:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.895 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.896 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.896 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.898 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.413915_106.1", 2022-09-02T04:50:56.899 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:56.899 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43:8b4ce987-f0fe-4038-9bfd-de8d46ec635c.5187.2", 2022-09-02T04:50:56.900 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.413915554Z", 2022-09-02T04:50:56.900 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.901 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.901 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.903 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.904 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.905 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_LTxS3L1ZOWfNiFivnHt8RLy", 2022-09-02T04:50:56.905 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.906 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.907 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.909 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.415076_107.1", 2022-09-02T04:50:56.909 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.910 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.911 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.415076727Z", 2022-09-02T04:50:56.912 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.912 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.913 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.913 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.915 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.915 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.916 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.916 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.917 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:56.918 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.918 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:56.920 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094176.417998_108.1", 2022-09-02T04:50:56.920 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:56.921 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-43", 2022-09-02T04:50:56.921 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:49:36.417998554Z", 2022-09-02T04:50:56.922 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:56.923 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:56.923 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:56.924 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:56.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.926 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:56.926 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_FQUblOeW-XeGyprnzRbauU6", 2022-09-02T04:50:56.927 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:56.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:56.928 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:56.930 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:56.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:56.933 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:50:56.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:56.935 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:56.987 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:56.988 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.028 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.069331_105.1", 2022-09-02T04:50:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.069331355Z", 2022-09-02T04:50:57.030 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.030 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.030 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.030 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.031 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.031 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.032 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.032 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.033 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.033 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.033 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.035 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.072446_107.1", 2022-09-02T04:50:57.036 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.036 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.037 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.072446444Z", 2022-09-02T04:50:57.037 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.037 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.037 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.038 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.039 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.039 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.040 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.041 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.044 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.073593_108.1", 2022-09-02T04:50:57.044 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.045 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.045 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.073593325Z", 2022-09-02T04:50:57.046 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.046 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.047 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.047 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.048 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.048 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.049 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.052 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.076462_109.1", 2022-09-02T04:50:57.053 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.053 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.076462666Z", 2022-09-02T04:50:57.054 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.054 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.055 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.055 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.056 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.056 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.057 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.058 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.058 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.059 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.059 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.061 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.112726_110.1", 2022-09-02T04:50:57.061 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.062 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.062 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.112726044Z", 2022-09-02T04:50:57.063 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.063 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.064 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.064 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.065 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.066 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.066 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.067 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.068 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.070 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.115765_111.1", 2022-09-02T04:50:57.070 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.071 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.071 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.115765220Z", 2022-09-02T04:50:57.072 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.072 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.072 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.073 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.075 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.075 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.076 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.077 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.077 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.080 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.116881_112.1", 2022-09-02T04:50:57.081 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.081 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.116881653Z", 2022-09-02T04:50:57.082 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.083 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.083 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.083 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.084 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.084 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.084 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.085 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.085 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.086 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.119686_113.1", 2022-09-02T04:50:57.087 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.087 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.087 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.119686224Z", 2022-09-02T04:50:57.087 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.088 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.088 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.088 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.089 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.089 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.089 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.090 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.091 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.091 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.092 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:57.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:57.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:57.094 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:57.154 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:57.154 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.187 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:50:57.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.188 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.069331_105.1", 2022-09-02T04:50:57.188 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.189 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.189 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.069331355Z", 2022-09-02T04:50:57.189 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.189 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.190 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.190 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.191 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.191 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.191 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.192 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.192 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.193 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.193 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.072446_107.1", 2022-09-02T04:50:57.194 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.194 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.194 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.072446444Z", 2022-09-02T04:50:57.194 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.195 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.195 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.195 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.196 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.196 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.196 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.197 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.197 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.199 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.073593_108.1", 2022-09-02T04:50:57.199 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.199 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.200 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.073593325Z", 2022-09-02T04:50:57.200 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.200 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.201 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.201 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.201 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.202 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.202 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.203 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.203 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.203 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.204 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.204 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.076462_109.1", 2022-09-02T04:50:57.205 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.205 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.205 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.076462666Z", 2022-09-02T04:50:57.206 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.206 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.206 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.206 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.207 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.207 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.208 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.208 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.210 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.112726_110.1", 2022-09-02T04:50:57.210 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.210 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.211 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.112726044Z", 2022-09-02T04:50:57.211 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.212 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.212 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.213 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.214 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.215 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.216 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.217 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.219 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.115765_111.1", 2022-09-02T04:50:57.219 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.219 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.219 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.115765220Z", 2022-09-02T04:50:57.220 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.220 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.221 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.221 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.221 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.222 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.222 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.223 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.224 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.224 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.116881_112.1", 2022-09-02T04:50:57.224 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.225 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.225 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.116881653Z", 2022-09-02T04:50:57.225 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.226 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.226 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.226 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.227 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.227 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.228 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.228 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.229 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.230 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.119686_113.1", 2022-09-02T04:50:57.231 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.231 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.232 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.119686224Z", 2022-09-02T04:50:57.232 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.232 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.233 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.233 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.233 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.234 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.234 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.234 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.236 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.236 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.237 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.237 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.237 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:50:57.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:57.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:57.239 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:57.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:57.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.323 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:57.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.324 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.069331_105.1", 2022-09-02T04:50:57.325 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.325 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.069331355Z", 2022-09-02T04:50:57.326 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.327 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.327 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.327 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.328 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.329 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.329 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.331 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.331 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.333 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.072446_107.1", 2022-09-02T04:50:57.333 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.338 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.338 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.072446444Z", 2022-09-02T04:50:57.338 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.339 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.339 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.339 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.340 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.340 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.340 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.341 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.341 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.342 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.073593_108.1", 2022-09-02T04:50:57.343 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.343 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.073593325Z", 2022-09-02T04:50:57.343 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.344 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.344 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.344 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.345 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.345 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.345 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.346 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.346 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.346 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.347 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.076462_109.1", 2022-09-02T04:50:57.348 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.348 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.348 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.076462666Z", 2022-09-02T04:50:57.348 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.349 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.349 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.349 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.350 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.350 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.350 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.351 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.351 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.352 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.112726_110.1", 2022-09-02T04:50:57.353 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.353 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.353 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.112726044Z", 2022-09-02T04:50:57.354 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.354 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.354 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.354 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.355 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.355 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.356 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.356 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.357 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.358 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.115765_111.1", 2022-09-02T04:50:57.358 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.359 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.359 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.115765220Z", 2022-09-02T04:50:57.359 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.360 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.360 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.360 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.361 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.361 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.361 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.362 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.362 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.363 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.363 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.364 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.116881_112.1", 2022-09-02T04:50:57.364 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.364 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.364 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.116881653Z", 2022-09-02T04:50:57.365 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.365 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.365 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.365 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.366 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.366 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.366 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.367 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.367 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "write" 2022-09-02T04:50:57.367 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:57.368 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "1_1662094237.119686_113.1", 2022-09-02T04:50:57.369 INFO:teuthology.orchestra.run.smithi099.stdout: "section": "bucket", 2022-09-02T04:50:57.369 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.369 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.119686224Z", 2022-09-02T04:50:57.369 INFO:teuthology.orchestra.run.smithi099.stdout: "data": { 2022-09-02T04:50:57.370 INFO:teuthology.orchestra.run.smithi099.stdout: "read_version": { 2022-09-02T04:50:57.370 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "", 2022-09-02T04:50:57.370 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 0 2022-09-02T04:50:57.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.371 INFO:teuthology.orchestra.run.smithi099.stdout: "write_version": { 2022-09-02T04:50:57.371 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.371 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": 1 2022-09-02T04:50:57.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:57.372 INFO:teuthology.orchestra.run.smithi099.stdout: "status": { 2022-09-02T04:50:57.373 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "complete" 2022-09-02T04:50:57.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:57.378 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:57.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:57.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:57.380 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:57.432 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:57.432 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.475 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:50:57.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.476 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.069331_105.1", 2022-09-02T04:50:57.476 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.476 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.476 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.069331355Z", 2022-09-02T04:50:57.477 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.477 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.477 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.477 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.478 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.478 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.478 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.479 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.479 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.484 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.072446_107.1", 2022-09-02T04:50:57.484 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.484 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44:b210c8c8-c433-420b-87c1-57ebfc433073.5159.1", 2022-09-02T04:50:57.485 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.072446444Z", 2022-09-02T04:50:57.485 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.485 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.485 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.486 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.486 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.487 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ggEu6Tw_6yULP76VPgs1gqQ", 2022-09-02T04:50:57.487 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.488 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.488 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.490 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.073593_108.1", 2022-09-02T04:50:57.490 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.491 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.491 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.073593325Z", 2022-09-02T04:50:57.492 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.492 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.492 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.493 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.493 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.494 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.494 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.495 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.495 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.497 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.076462_109.1", 2022-09-02T04:50:57.497 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.497 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-44", 2022-09-02T04:50:57.498 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.076462666Z", 2022-09-02T04:50:57.498 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.498 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.499 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.499 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.500 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.500 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_09Nz5anPOBJ1ppIqAk7D0zB", 2022-09-02T04:50:57.500 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.501 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.502 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.504 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.112726_110.1", 2022-09-02T04:50:57.504 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.504 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.505 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.112726044Z", 2022-09-02T04:50:57.505 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.505 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.505 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.506 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.506 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.506 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.507 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.507 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.507 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.509 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.115765_111.1", 2022-09-02T04:50:57.509 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket.instance", 2022-09-02T04:50:57.509 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45:b210c8c8-c433-420b-87c1-57ebfc433073.5159.2", 2022-09-02T04:50:57.509 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.115765220Z", 2022-09-02T04:50:57.510 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.510 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.510 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.510 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.511 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.511 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_nTl3Jx27ewXC_h8fTqZvlQi", 2022-09-02T04:50:57.512 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.513 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.514 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.515 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.116881_112.1", 2022-09-02T04:50:57.515 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.515 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.516 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.116881653Z", 2022-09-02T04:50:57.516 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.517 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.517 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.517 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.518 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.518 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.518 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.519 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.519 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "write" 2022-09-02T04:50:57.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.520 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:57.521 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "1_1662094237.119686_113.1", 2022-09-02T04:50:57.521 INFO:teuthology.orchestra.run.smithi160.stdout: "section": "bucket", 2022-09-02T04:50:57.521 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "xljbkf-45", 2022-09-02T04:50:57.521 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.119686224Z", 2022-09-02T04:50:57.522 INFO:teuthology.orchestra.run.smithi160.stdout: "data": { 2022-09-02T04:50:57.522 INFO:teuthology.orchestra.run.smithi160.stdout: "read_version": { 2022-09-02T04:50:57.522 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "", 2022-09-02T04:50:57.523 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 0 2022-09-02T04:50:57.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.523 INFO:teuthology.orchestra.run.smithi160.stdout: "write_version": { 2022-09-02T04:50:57.523 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_psLg3-MultFdWl8Gx_0P4Jh", 2022-09-02T04:50:57.524 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": 1 2022-09-02T04:50:57.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:57.525 INFO:teuthology.orchestra.run.smithi160.stdout: "status": { 2022-09-02T04:50:57.526 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "complete" 2022-09-02T04:50:57.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:57.527 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:50:57.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:50:57.575 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.689 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:50:57.738 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:57.739 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.890 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:50:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:57.936 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:57.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:50:57.992 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:50:58.038 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.039 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.178 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.229 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.230 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.287 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:50:58.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.290 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.337 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.338 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.391 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:50:58.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.396 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.445 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.445 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.521 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:50:58.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e248e3d2-c4b4-4705-8303-7ccdb6560fe1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.531 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e248e3d2-c4b4-4705-8303-7ccdb6560fe1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.582 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.582 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.623 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:50:58.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.716 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:50:58.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.721 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.770 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.770 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.829 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:50:58.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.832 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.880 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.880 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:58.927 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:50:58.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ffa7b930-977f-415c-8bb4-bf6fd394dc85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:58.930 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ffa7b930-977f-415c-8bb4-bf6fd394dc85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:58.978 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:58.979 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.035 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:50:59.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.039 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:59.088 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:59.088 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.147 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:50:59.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.150 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:59.198 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:59.199 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.258 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:50:59.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.262 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:59.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:59.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.376 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:50:59.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b798a9cd-aa0e-480c-bb07-05f02abf3f41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.380 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b798a9cd-aa0e-480c-bb07-05f02abf3f41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:59.428 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:59.429 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.470 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:50:59.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... ok 2022-09-02T04:50:59.476 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-46 2022-09-02T04:50:59.476 DEBUG:boto:path=/xljbkf-46/ 2022-09-02T04:50:59.477 DEBUG:boto:auth_path=/xljbkf-46/ 2022-09-02T04:50:59.478 DEBUG:boto:Method: PUT 2022-09-02T04:50:59.479 DEBUG:boto:Path: /xljbkf-46/ 2022-09-02T04:50:59.479 DEBUG:boto:Data: 2022-09-02T04:50:59.480 DEBUG:boto:Headers: {} 2022-09-02T04:50:59.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:59.483 DEBUG:boto:Port: 8000 2022-09-02T04:50:59.484 DEBUG:boto:Params: {} 2022-09-02T04:50:59.486 DEBUG:boto:Token: None 2022-09-02T04:50:59.488 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-46/ 2022-09-02T04:50:59.489 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OrOj8DVKY03jxHymyw0E65Htaxk= 2022-09-02T04:50:59.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OrOj8DVKY03jxHymyw0E65Htaxk=', 'Content-Length': '0'} 2022-09-02T04:50:59.505 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d81f3f9ff69ab26c-0063118bb3-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.506 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-47 2022-09-02T04:50:59.507 DEBUG:boto:path=/xljbkf-47/ 2022-09-02T04:50:59.508 DEBUG:boto:auth_path=/xljbkf-47/ 2022-09-02T04:50:59.509 DEBUG:boto:Method: PUT 2022-09-02T04:50:59.509 DEBUG:boto:Path: /xljbkf-47/ 2022-09-02T04:50:59.510 DEBUG:boto:Data: 2022-09-02T04:50:59.510 DEBUG:boto:Headers: {} 2022-09-02T04:50:59.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:59.512 DEBUG:boto:Port: 8000 2022-09-02T04:50:59.512 DEBUG:boto:Params: {} 2022-09-02T04:50:59.513 DEBUG:boto:Token: None 2022-09-02T04:50:59.513 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-47/ 2022-09-02T04:50:59.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q8U2RD7hWP0ydBHHjO0NXCgDqgs= 2022-09-02T04:50:59.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q8U2RD7hWP0ydBHHjO0NXCgDqgs=', 'Content-Length': '0'} 2022-09-02T04:50:59.547 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008499cf59099499d-0063118bb3-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.548 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-48 2022-09-02T04:50:59.549 DEBUG:boto:path=/xljbkf-48/ 2022-09-02T04:50:59.549 DEBUG:boto:auth_path=/xljbkf-48/ 2022-09-02T04:50:59.550 DEBUG:boto:Method: PUT 2022-09-02T04:50:59.550 DEBUG:boto:Path: /xljbkf-48/ 2022-09-02T04:50:59.554 DEBUG:boto:Data: 2022-09-02T04:50:59.555 DEBUG:boto:Headers: {} 2022-09-02T04:50:59.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:59.557 DEBUG:boto:Port: 8001 2022-09-02T04:50:59.558 DEBUG:boto:Params: {} 2022-09-02T04:50:59.559 DEBUG:boto:Token: None 2022-09-02T04:50:59.560 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-48/ 2022-09-02T04:50:59.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eC4BjiWmz57nd405PHLxYA/g82Q= 2022-09-02T04:50:59.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eC4BjiWmz57nd405PHLxYA/g82Q=', 'Content-Length': '0'} 2022-09-02T04:50:59.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d9ebe890b6432c0-0063118bb3-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.589 DEBUG:boto:path=/xljbkf-46/ 2022-09-02T04:50:59.591 DEBUG:boto:auth_path=/xljbkf-46/ 2022-09-02T04:50:59.591 DEBUG:boto:Method: HEAD 2022-09-02T04:50:59.592 DEBUG:boto:Path: /xljbkf-46/ 2022-09-02T04:50:59.592 DEBUG:boto:Data: 2022-09-02T04:50:59.593 DEBUG:boto:Headers: {} 2022-09-02T04:50:59.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:59.594 DEBUG:boto:Port: 8000 2022-09-02T04:50:59.595 DEBUG:boto:Params: {} 2022-09-02T04:50:59.596 DEBUG:boto:Token: None 2022-09-02T04:50:59.597 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-46/ 2022-09-02T04:50:59.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cWCOOQW0cj2PfAsUWE3PlUBlmm0= 2022-09-02T04:50:59.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cWCOOQW0cj2PfAsUWE3PlUBlmm0=', 'Content-Length': '0'} 2022-09-02T04:50:59.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c74c49aaa5c2e7b3-0063118bb3-1425-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.603 DEBUG:boto:path=/xljbkf-46/key 2022-09-02T04:50:59.604 DEBUG:boto:auth_path=/xljbkf-46/key 2022-09-02T04:50:59.605 DEBUG:boto:Method: PUT 2022-09-02T04:50:59.606 DEBUG:boto:Path: /xljbkf-46/key 2022-09-02T04:50:59.607 DEBUG:boto:Data: 2022-09-02T04:50:59.608 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-02T04:50:59.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:50:59.609 DEBUG:boto:Port: 8000 2022-09-02T04:50:59.610 DEBUG:boto:Params: {} 2022-09-02T04:50:59.611 DEBUG:boto:Token: None 2022-09-02T04:50:59.612 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-46/key 2022-09-02T04:50:59.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:654jawBBq5LOzkRq/8DDSf3z6x4= 2022-09-02T04:50:59.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:654jawBBq5LOzkRq/8DDSf3z6x4='} 2022-09-02T04:50:59.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bcb3ab107c8eeea-0063118bb3-1425-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.621 DEBUG:boto:path=/xljbkf-47/ 2022-09-02T04:50:59.621 DEBUG:boto:auth_path=/xljbkf-47/ 2022-09-02T04:50:59.622 DEBUG:boto:Method: HEAD 2022-09-02T04:50:59.622 DEBUG:boto:Path: /xljbkf-47/ 2022-09-02T04:50:59.623 DEBUG:boto:Data: 2022-09-02T04:50:59.623 DEBUG:boto:Headers: {} 2022-09-02T04:50:59.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:59.625 DEBUG:boto:Port: 8000 2022-09-02T04:50:59.625 DEBUG:boto:Params: {} 2022-09-02T04:50:59.626 DEBUG:boto:Token: None 2022-09-02T04:50:59.627 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-47/ 2022-09-02T04:50:59.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YcwdkBDjUSCHYvsDXItGYEpyMI4= 2022-09-02T04:50:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YcwdkBDjUSCHYvsDXItGYEpyMI4=', 'Content-Length': '0'} 2022-09-02T04:50:59.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000296e348cdec0058d-0063118bb3-1453-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.634 DEBUG:boto:path=/xljbkf-47/key 2022-09-02T04:50:59.635 DEBUG:boto:auth_path=/xljbkf-47/key 2022-09-02T04:50:59.636 DEBUG:boto:Method: PUT 2022-09-02T04:50:59.637 DEBUG:boto:Path: /xljbkf-47/key 2022-09-02T04:50:59.638 DEBUG:boto:Data: 2022-09-02T04:50:59.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-02T04:50:59.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:50:59.641 DEBUG:boto:Port: 8000 2022-09-02T04:50:59.642 DEBUG:boto:Params: {} 2022-09-02T04:50:59.643 DEBUG:boto:Token: None 2022-09-02T04:50:59.644 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-47/key 2022-09-02T04:50:59.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1DNNsRQ4ivPrsRv4sV7S82bPAQc= 2022-09-02T04:50:59.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1DNNsRQ4ivPrsRv4sV7S82bPAQc='} 2022-09-02T04:50:59.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b2a8683bd20dc15-0063118bb3-1453-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.696 DEBUG:boto:path=/xljbkf-48/ 2022-09-02T04:50:59.697 DEBUG:boto:auth_path=/xljbkf-48/ 2022-09-02T04:50:59.699 DEBUG:boto:Method: HEAD 2022-09-02T04:50:59.700 DEBUG:boto:Path: /xljbkf-48/ 2022-09-02T04:50:59.701 DEBUG:boto:Data: 2022-09-02T04:50:59.702 DEBUG:boto:Headers: {} 2022-09-02T04:50:59.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:59.703 DEBUG:boto:Port: 8001 2022-09-02T04:50:59.704 DEBUG:boto:Params: {} 2022-09-02T04:50:59.705 DEBUG:boto:Token: None 2022-09-02T04:50:59.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-48/ 2022-09-02T04:50:59.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TvDY6kJSxfPxY0YqSjAsyotvdb8= 2022-09-02T04:50:59.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TvDY6kJSxfPxY0YqSjAsyotvdb8=', 'Content-Length': '0'} 2022-09-02T04:50:59.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f3500e924cb2840-0063118bb3-142b-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.712 DEBUG:boto:path=/xljbkf-48/key 2022-09-02T04:50:59.713 DEBUG:boto:auth_path=/xljbkf-48/key 2022-09-02T04:50:59.714 DEBUG:boto:Method: PUT 2022-09-02T04:50:59.714 DEBUG:boto:Path: /xljbkf-48/key 2022-09-02T04:50:59.715 DEBUG:boto:Data: 2022-09-02T04:50:59.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2022-09-02T04:50:59.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:50:59.716 DEBUG:boto:Port: 8001 2022-09-02T04:50:59.717 DEBUG:boto:Params: {} 2022-09-02T04:50:59.718 DEBUG:boto:Token: None 2022-09-02T04:50:59.718 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 02 Sep 2022 04:50:59 GMT /xljbkf-48/key 2022-09-02T04:50:59.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0h3blFBuB2w5/7qvv9TlkObFhIs= 2022-09-02T04:50:59.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:50:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0h3blFBuB2w5/7qvv9TlkObFhIs='} 2022-09-02T04:50:59.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bafedc66f87737c5-0063118bb3-142b-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:50:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:50:59.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.733 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:59.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:59.784 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.814 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T04:50:59.825 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:50:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:50:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:50:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:50:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:59.579973Z" 2022-09-02T04:50:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:50:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:50:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:50:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:50:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:50:59.830 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:50:59.831 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:50:59.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:50:59.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:50:59.834 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:50:59.883 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:50:59.884 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:50:59.918 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:59.917+0000 7f8bd0786600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:59.919 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:50:59.918+0000 7f8bd0786600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:50:59.930 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:50:59.932 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:50:59.932 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:50:59.933 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:50:59.933 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:50:59.933 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:50:59.934 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:50:59.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:59.935 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:50:59.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:59.935 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:50:59.936 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:59.936 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:59.937 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:50:59.937 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:59.937 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:50:59.938 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:59.938 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:50:59.939 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:59.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:59.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:59.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:59.941 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:50:59.943 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:59.943 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:59.944 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:50:59.944 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:59.945 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:59.945 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:59.946 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:50:59.946 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:59.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:59.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:59.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:59.948 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:50:59.949 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:59.950 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:59.950 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.499859_119.1", 2022-09-02T04:50:59.951 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:59.951 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:59.953 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:59.953 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.499859Z", 2022-09-02T04:50:59.953 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:50:59.954 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:59.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:59.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:50:59.955 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:50:59.956 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:50:59.956 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:50:59.957 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:50:59.957 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:50:59.958 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:50:59.958 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:50:59.958 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:50:59.959 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:50:59.959 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:59.960 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:50:59.960 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:50:59.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:59.961 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:50:59.962 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:50:59.963 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:50:59.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:50:59.964 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:50:59Z" 2022-09-02T04:50:59.964 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:50:59.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.499859_119.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.499859Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:50:59Z" } 2022-09-02T04:50:59.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.499859_119.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.499859Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:50:59.967 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:50:59.967 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.499859_119.1', 3: ''} 2022-09-02T04:50:59.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094259.579973_131.1 target=1_1662094259.499859_119.1 2022-09-02T04:51:04.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:04.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:04.972 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:05.029 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:05.030 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:05.073 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:05.072+0000 7fe73ff60600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:51:05.073 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:05.073+0000 7fe73ff60600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:51:05.086 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:05.086 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:05.087 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:05.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:05.088 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:05.088 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:51:05.089 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:51:05.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.090 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:05.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.094 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:05.095 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.095 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.096 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:51:05.096 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.097 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:51:05.097 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.098 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:51:05.098 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:51:05.099 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.101 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:05.101 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.102 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.102 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:51:05.102 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.103 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.103 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.104 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:51:05.104 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:51:05.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.106 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:05.107 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.107 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.107 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:51:05.108 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.108 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.109 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.109 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:51:05.110 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:51:05.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.111 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:05.112 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.112 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.112 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:51:05.113 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.113 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.114 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.115 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:51:05.115 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:51:05.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.120 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:05.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.121 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:05.121 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:51:05.122 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:51:05.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.123 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:05Z" 2022-09-02T04:51:05.123 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:05.126 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:51:05Z" } 2022-09-02T04:51:05.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:51:05.128 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:51:05.129 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:51:05.130 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:51:05.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:51:05.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.133 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:05.195 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:05.196 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:51:05.217+0000 7f4816b50600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:51:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:51:05.218+0000 7f4816b50600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:51:05.229 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:05.230 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:05.230 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:05.230 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:05.231 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:05.231 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:51:05.232 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:51:05.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.232 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:05.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.233 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:05.233 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:05.233 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:51:05.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:51:05.234 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:05.235 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:51:05.235 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:05.235 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:51:05.236 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:51:05.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:05.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.237 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:05.237 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:05.237 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:51:05.237 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:51:05.238 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:05.238 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:05.239 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:05.239 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:51:05.239 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:51:05.240 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:05.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.241 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:05.241 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:05.241 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:51:05.241 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:51:05.242 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:05.242 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:05.243 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:05.243 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:51:05.243 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:51:05.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:05.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.244 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:05.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:05.245 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:51:05.245 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:51:05.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:05.246 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:05.247 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:05.247 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:51:05.247 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:51:05.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:05.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:05.248 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:05.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.249 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:05.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:51:05.249 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:51:05.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.250 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:05Z" 2022-09-02T04:51:05.251 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:05.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:51:05Z" } 2022-09-02T04:51:05.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:51:05.253 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:51:05.254 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:51:05.255 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:51:05.255 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:51:05.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.257 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:05.309 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:05.309 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:05.339 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:05.338+0000 7f3be96ae600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:51:05.340 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:05.339+0000 7f3be96ae600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:51:05.350 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:05.350 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:05.351 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:05.351 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:05.352 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:05.353 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:51:05.353 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:51:05.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.354 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:05.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:05.358 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.358 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.359 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:51:05.359 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.363 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:51:05.363 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.364 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:51:05.364 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:51:05.364 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.365 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:05.367 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.368 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.369 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:51:05.369 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.370 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.370 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.371 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:51:05.371 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:51:05.371 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.372 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:05.372 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.372 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.373 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:51:05.374 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.375 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.375 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:51:05.375 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:51:05.376 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.377 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:05.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.378 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:51:05.379 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:51:05.380 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.380 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.381 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.382 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:51:05.383 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:51:05.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.384 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.384 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:05.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.385 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:05.386 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:51:05.386 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:51:05.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.387 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:05Z" 2022-09-02T04:51:05.388 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:05.390 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:51:05Z" } 2022-09-02T04:51:05.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:51:05.393 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:51:05.394 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:51:05.395 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:51:05.396 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-02T04:51:05.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.397 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:05.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:05.449 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:05.493 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:51:05.493 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.493 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:05.494 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:05.494 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.495 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.495 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004601", 2022-09-02T04:51:05.495 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:59.621615Z" 2022-09-02T04:51:05.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.496 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.496 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:05.496 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:05.496 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:05.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:05.497 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:05.497 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:05.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:05.498 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:51:05.499 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:05.500 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-02T04:51:05.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:05.502 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:05.558 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:05.559 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:05.610 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:05.611 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:05.612 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:05.613 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 11009958948854697431 2022-09-02T04:51:05.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.614 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:05.615 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.615 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:05.616 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.617 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.617 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.617 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:05.618 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.620 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:05.620 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.620 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:05.621 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004460", 2022-09-02T04:51:05.621 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.621 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.621 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.622 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:47:40.929660Z" 2022-09-02T04:51:05.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.623 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:05.624 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.624 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:05.634 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:05.635 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.635 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.636 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.637 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:05.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:05.638 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:05.639 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:05.639 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:05.640 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000654", 2022-09-02T04:51:05.640 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:05.641 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:05.641 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:05.642 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:08.825120Z" 2022-09-02T04:51:05.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:05.645 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:05.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.646 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:05.646 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:51:05.646 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:51:05.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:05.655 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:05Z" 2022-09-02T04:51:05.655 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:05.656 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11009958948854697431 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:40.929660Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000654", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:08.825120Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:05Z" } 2022-09-02T04:51:05.657 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:40.929660Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000654', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:08.825120Z'}}] 2022-09-02T04:51:05.658 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:05.659 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000654'} 2022-09-02T04:51:05.660 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000004601 target=00000000000000000000:00000000000000004460, shard=3 master=00000000000000000000:00000000000000000796 target=00000000000000000000:00000000000000000654 2022-09-02T04:51:10.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:10.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:10.663 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:10.723 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:10.724 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:10.782 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:10.783 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:10.783 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:10.785 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 11009958948854697431 2022-09-02T04:51:10.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:10.785 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:10.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:10.786 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:10.786 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:10.786 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:10.787 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:10.787 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:10.787 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:10.788 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:10.788 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:10.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:10.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:10.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:10.798 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:10.798 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:10.799 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:10.799 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004601", 2022-09-02T04:51:10.800 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:10.800 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:10.800 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:10.801 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.621615Z" 2022-09-02T04:51:10.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:10.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:10.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:10.803 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:10.803 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:10.804 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:10.806 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:10.807 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:10.807 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:10.808 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:10.808 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:10.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:10.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:10.811 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:10.811 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:10.811 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:10.812 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:10.812 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:10.813 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:10.813 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:10.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:10.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:10.815 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:10.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:10.817 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:10.817 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:51:10.818 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:51:10.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:10.819 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:10Z" 2022-09-02T04:51:10.819 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:10.820 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11009958948854697431 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004601", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.621615Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:10Z" } 2022-09-02T04:51:10.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.621615Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T04:51:10.822 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:10.823 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:10.824 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-02T04:51:10.825 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-02T04:51:10.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:10.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:10.829 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:10.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:10.940 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:51:10.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:10.942 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:10.942 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:10.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:10.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:10.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004741", 2022-09-02T04:51:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:51:08.863130Z" 2022-09-02T04:51:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:10.952 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:51:10.954 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004741', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:10.959 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-02T04:51:10.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:10.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:10.962 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:11.059 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:11.060 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:11.060 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:11.061 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:11.061 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:11.063 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1295418732353758134 2022-09-02T04:51:11.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:11.064 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:11.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:11.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:11.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:11.066 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:11.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:11.067 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:11.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:11.068 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:11.068 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:11.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:11.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:11.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:11.070 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:11.071 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:11.071 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:11.072 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004460", 2022-09-02T04:51:11.072 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:11.073 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:11.073 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:11.074 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:40.929660Z" 2022-09-02T04:51:11.074 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:11.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:11.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:11.076 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:11.078 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:11.078 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:11.079 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:11.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:11.080 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:11.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:11.081 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:11.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:11.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:11.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:11.083 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:11.084 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:11.084 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:11.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:11.086 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:11.087 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:11.087 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:11.088 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:11.089 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:11.089 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:11.090 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:11.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:11.091 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:11.091 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:11.092 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:11.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:11.093 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:11Z" 2022-09-02T04:51:11.093 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:11.095 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1295418732353758134 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:40.929660Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:11Z" } 2022-09-02T04:51:11.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:40.929660Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T04:51:11.098 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004741', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:11.100 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:11.101 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000004741 target=00000000000000000000:00000000000000004460 2022-09-02T04:51:16.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:16.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:16.104 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:16.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:16.165 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:16.222 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:16.223 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:16.223 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:16.224 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:16.224 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:16.224 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1295418732353758134 2022-09-02T04:51:16.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:16.225 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:16.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:16.229 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:16.230 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:16.230 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:16.231 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:16.231 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:16.232 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:16.232 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:16.233 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:16.234 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:16.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:16.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:16.236 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:16.237 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:16.237 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:16.239 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004460", 2022-09-02T04:51:16.240 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:16.240 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:16.240 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:16.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:47:40.929660Z" 2022-09-02T04:51:16.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:16.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:16.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:16.244 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:16.244 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:16.245 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:16.245 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:16.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:16.246 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:16.247 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:16.247 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:16.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:16.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:16.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:16.250 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:16.251 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:16.251 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:16.252 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:16.252 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:16.252 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:16.253 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:16.253 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:16.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:16.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:16.254 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:16.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:16.256 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:16.256 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:16.257 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:16.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:16.257 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:16Z" 2022-09-02T04:51:16.258 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:16.261 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1295418732353758134 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:47:40.929660Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:16Z" } 2022-09-02T04:51:16.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:47:40.929660Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T04:51:16.265 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004741', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:16.266 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:16.267 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000004741 target=00000000000000000000:00000000000000004460 2022-09-02T04:51:21.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.279 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:21.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:21.339 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:21.404 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:21.405 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:21.405 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:21.405 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:21.405 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:21.406 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1295418732353758134 2022-09-02T04:51:21.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.406 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:21.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.407 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:21.407 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.407 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.408 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:21.408 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.408 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.408 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.409 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:21.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.410 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:21.410 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.410 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.411 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004881", 2022-09-02T04:51:21.411 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.411 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.411 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.412 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:51:17.171208Z" 2022-09-02T04:51:21.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.413 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:21.413 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.413 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:21.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.415 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:21.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:21.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.416 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:21.417 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.417 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:21.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.418 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:21.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.419 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:21.419 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:21.419 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:21.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.420 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:21Z" 2022-09-02T04:51:21.421 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:21.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1295418732353758134 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004881", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:17.171208Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:21Z" } 2022-09-02T04:51:21.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:17.171208Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T04:51:21.423 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004741', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.424 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.424 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-02T04:51:21.425 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-02T04:51:21.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.426 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:21.522 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:51:21.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:21.524 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:21.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004881", 2022-09-02T04:51:21.525 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:51:17.171208Z" 2022-09-02T04:51:21.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.525 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:21.526 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:21.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.527 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:21.527 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:21.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.529 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:51:21.530 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.531 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-02T04:51:21.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.532 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:21.593 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:21.594 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:21.640 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:21.641 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 2207523797324220113 2022-09-02T04:51:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.642 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:21.643 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:21.644 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.646 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004881", 2022-09-02T04:51:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:21.647 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:51:17.171208Z" 2022-09-02T04:51:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.649 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:21.650 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:21.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:21.652 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:21.653 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:21.654 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:21.655 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:51:21.656 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:51:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.657 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:21Z" 2022-09-02T04:51:21.657 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:21.659 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2207523797324220113 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004881", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:17.171208Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006152", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:39.466172Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:21Z" } 2022-09-02T04:51:21.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:17.171208Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006152', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:39.466172Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T04:51:21.660 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.660 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006152', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.661 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-02T04:51:21.662 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-02T04:51:21.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.664 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:21.715 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:21.715 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:21.773 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:51:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:21.774 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005446", 2022-09-02T04:51:21.775 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:51:08.814390Z" 2022-09-02T04:51:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:21.776 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:21.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:21.778 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:21.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:21.779 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:51:21.780 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.781 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-02T04:51:21.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:21.783 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:21.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:21.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:21.876 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:21.876 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:21.876 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:21.877 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:21.877 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:21.877 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T04:51:21.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.878 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:21.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.878 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:21.878 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.879 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.879 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:21.879 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.880 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.880 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.880 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:21.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.882 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:21.882 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.883 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005306", 2022-09-02T04:51:21.885 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.886 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.886 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.886 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:59.657887Z" 2022-09-02T04:51:21.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.887 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:21.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:21.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.889 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:21.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:21.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:21.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:21.891 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:21.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:21.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:21.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:21.892 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:21.892 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:21.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:21.893 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:21.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.893 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:21.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:21.894 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:21.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:21.894 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:21Z" 2022-09-02T04:51:21.895 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:21.896 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.657887Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:40.064266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:38.798708Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:21Z" } 2022-09-02T04:51:21.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.657887Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:40.064266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:38.798708Z'}}] 2022-09-02T04:51:21.897 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005306', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:21.901 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000005446 target=00000000000000000000:00000000000000005306 2022-09-02T04:51:26.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:26.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:26.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:26.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:26.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:27.045 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:27.045 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:27.045 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:27.046 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:27.046 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:27.046 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T04:51:27.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:27.047 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:27.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:27.047 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:27.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:27.048 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:27.048 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:27.048 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:27.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:27.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:27.049 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:27.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:27.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:27.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:27.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:27.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:27.051 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:27.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005306", 2022-09-02T04:51:27.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:27.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:59.657887Z" 2022-09-02T04:51:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:27.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:27.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:27.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:27.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:27.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:27.058 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:27.058 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:27.058 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:27.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:27.060 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:27.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:27.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:27.061 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:27.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:27.062 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:27.062 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:27.062 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:27.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:27.066 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:27Z" 2022-09-02T04:51:27.066 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:27.072 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005306", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.657887Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:40.064266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:38.798708Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:27Z" } 2022-09-02T04:51:27.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005306', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.657887Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:40.064266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:38.798708Z'}}] 2022-09-02T04:51:27.073 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:27.074 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005306', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:27.074 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000005446 target=00000000000000000000:00000000000000005306 2022-09-02T04:51:32.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.078 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:32.137 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:32.138 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:32.186 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:32.187 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:32.187 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:32.187 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:32.187 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T04:51:32.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.188 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:32.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.191 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:32.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.200 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.201 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:32.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.202 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.202 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:32.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.204 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:32.205 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.205 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005446", 2022-09-02T04:51:32.206 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.207 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:51:08.814390Z" 2022-09-02T04:51:32.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.210 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.212 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:32.213 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.213 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.214 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:32.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.216 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:32.217 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.217 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.218 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:32.219 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.219 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.220 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.220 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:32.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:32.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.224 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:32.224 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:32.224 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:32.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.225 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:32Z" 2022-09-02T04:51:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:32.230 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005446", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:08.814390Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:40.064266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:38.798708Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:32Z" } 2022-09-02T04:51:32.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005446', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:08.814390Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:40.064266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:38.798708Z'}}] 2022-09-02T04:51:32.232 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.233 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.234 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-02T04:51:32.236 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-02T04:51:32.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.242 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:32.297 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:32.298 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:32.362 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:51:32.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:32.362 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005446", 2022-09-02T04:51:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:51:08.814390Z" 2022-09-02T04:51:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:32.367 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:51:32.370 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.371 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-02T04:51:32.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.376 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:32.431 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:32.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:32.496 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:32.496 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:32.496 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:32.497 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:32.497 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1900644279603854970 2022-09-02T04:51:32.497 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.498 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:32.502 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.502 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:32.503 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.503 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.503 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:32.503 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.504 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.504 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.504 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:32.504 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.505 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.505 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:32.505 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.506 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.506 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005446", 2022-09-02T04:51:32.507 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.507 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.508 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:51:08.814390Z" 2022-09-02T04:51:32.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.509 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:32.509 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.509 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.509 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:32.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.510 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:32.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.512 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:32.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.512 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.513 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:32.513 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.513 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.514 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:32.514 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.514 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.514 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:32Z" 2022-09-02T04:51:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:32.523 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1900644279603854970 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005446", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:08.814390Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:35:40.064266Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:38.798708Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:32Z" } 2022-09-02T04:51:32.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005446', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:08.814390Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:35:40.064266Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:38.798708Z'}}] 2022-09-02T04:51:32.524 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.525 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005446', 2: '00000000000000000000:00000000000000006434', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.525 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-02T04:51:32.525 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-02T04:51:32.529 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-02T04:51:32.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.531 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:32.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:32.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:32.651 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:51:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:51:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005020", 2022-09-02T04:51:32.656 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:51:21.279465Z" 2022-09-02T04:51:32.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:51:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:51:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:32.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:37.538287Z" 2022-09-02T04:51:32.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:51:32.659 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.660 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-02T04:51:32.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:32.661 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:32.713 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:32.713 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:32.784 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:32.784 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:32.784 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:32.785 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:32.785 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:32.785 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10070206082575430316 2022-09-02T04:51:32.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.786 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:32.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.787 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:32.787 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.788 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:32.789 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.789 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:51:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004460", 2022-09-02T04:51:32.806 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.806 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.807 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.807 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:59.729126Z" 2022-09-02T04:51:32.808 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:51:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:51:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:32.816 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.538287Z" 2022-09-02T04:51:32.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:32Z" 2022-09-02T04:51:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:32.819 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10070206082575430316 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:20.360817Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.729126Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:33:00.969288Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.538287Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:32Z" } 2022-09-02T04:51:32.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:20.360817Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.729126Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:33:00.969288Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.538287Z'}}] 2022-09-02T04:51:32.820 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004460', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:32.821 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=1 master=00000000000000000000:00000000000000005020 target=00000000000000000000:00000000000000004460 2022-09-02T04:51:37.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:37.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:37.826 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:37.887 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:51:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:51:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10070206082575430316 2022-09-02T04:51:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:51:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:51:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:51:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:51:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005020", 2022-09-02T04:51:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:51:21.279465Z" 2022-09-02T04:51:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:51:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:51:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:51:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:51:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:51:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T04:51:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:51:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:37.538287Z" 2022-09-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:51:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:51:37Z" 2022-09-02T04:51:37.979 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:51:37.980 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10070206082575430316 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:20.360817Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:21.279465Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:33:00.969288Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.538287Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:37Z" } 2022-09-02T04:51:37.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:20.360817Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:21.279465Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:33:00.969288Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.538287Z'}}] 2022-09-02T04:51:37.982 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:37.982 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:37.983 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-02T04:51:37.984 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-02T04:51:37.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:37.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:37.988 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:38.121 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:51:38.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:38.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:38.122 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:51:38.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:38.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:38.122 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005020", 2022-09-02T04:51:38.123 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:51:21.279465Z" 2022-09-02T04:51:38.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:38.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:38.123 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:51:38.124 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:51:38.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:38.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:38.124 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:38.126 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:37.538287Z" 2022-09-02T04:51:38.126 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:38.127 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:51:38.128 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:38.128 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-02T04:51:38.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:38.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:38.130 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:38.193 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:38.194 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:38.256 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 5969670707592890178 2022-09-02T04:51:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:51:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004740", 2022-09-02T04:51:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:51:17.530894Z" 2022-09-02T04:51:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:51:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:51:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.538287Z" 2022-09-02T04:51:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:51:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:51:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:38Z" 2022-09-02T04:51:38.297 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:38.299 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5969670707592890178 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:20.360817Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004740", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:17.530894Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:33:00.969288Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.538287Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:38Z" } 2022-09-02T04:51:38.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:20.360817Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004740', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:17.530894Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:33:00.969288Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.538287Z'}}] 2022-09-02T04:51:38.300 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:38.300 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000004740', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:38.301 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone3: shard=1 master=00000000000000000000:00000000000000005020 target=00000000000000000000:00000000000000004740 2022-09-02T04:51:43.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.305 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:43.365 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:43.365 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:43.425 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:43.426 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:51:43.427 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:51:43.427 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:51:43.428 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:51:43.428 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 5969670707592890178 2022-09-02T04:51:43.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:51:43.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:51:43.431 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:43.432 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:43.433 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:43.434 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:43.435 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:30:20.360817Z" 2022-09-02T04:51:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:43.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:43.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:51:43.438 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:43.439 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005020", 2022-09-02T04:51:43.440 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:43.441 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:43.442 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:51:21.279465Z" 2022-09-02T04:51:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:43.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:51:43.444 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:43.445 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005588", 2022-09-02T04:51:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:43.446 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:33:00.969288Z" 2022-09-02T04:51:43.447 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:51:43.448 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T04:51:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:43.449 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:51:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:51:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:51:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:37.538287Z" 2022-09-02T04:51:43.450 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:43.451 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:51:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T04:51:43.452 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T04:51:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:43.453 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:51:43Z" 2022-09-02T04:51:43.453 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:43.455 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5969670707592890178 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:20.360817Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005020", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:21.279465Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:33:00.969288Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:37.538287Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T04:51:43Z" } 2022-09-02T04:51:43.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:20.360817Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005020', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:21.279465Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:33:00.969288Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:37.538287Z'}}] 2022-09-02T04:51:43.457 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:43.459 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005020', 2: '00000000000000000000:00000000000000005588', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T04:51:43.460 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-02T04:51:43.461 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-02T04:51:43.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.463 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:43.517 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:43.518 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:43.571 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:51:43.571 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:43.572 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T04:51:43.572 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:31:59.361841Z" 2022-09-02T04:51:43.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:43.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:43.573 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004881", 2022-09-02T04:51:43.573 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:51:17.171208Z" 2022-09-02T04:51:43.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:43.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:43.575 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006152", 2022-09-02T04:51:43.575 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:35:39.466172Z" 2022-09-02T04:51:43.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:51:43.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:51:43.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:43.577 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:50:48.826574Z" 2022-09-02T04:51:43.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:51:43.578 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:51:43.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:43.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:43.582 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:51:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:43.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.704 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:43.756 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:43.756 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:43.807 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:51:43.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.810 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:43.861 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:43.862 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:43.906 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:51:43.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:43.908 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:43.960 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:43.961 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:44.029 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:51:44.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:44.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:44.032 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:44.120 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:44.121 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:44.175 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2022-09-02T04:51:44.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:44.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:44.178 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:44.233 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:44.234 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:44.307 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:51:44.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:44.308 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T04:51:44.308 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:30:11.070047Z" 2022-09-02T04:51:44.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:44.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:44.311 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000005586", 2022-09-02T04:51:44.311 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:51:38.749572Z" 2022-09-02T04:51:44.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:44.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:44.313 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006434", 2022-09-02T04:51:44.314 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:35:40.064266Z" 2022-09-02T04:51:44.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:44.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:44.317 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T04:51:44.317 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T04:50:38.798708Z" 2022-09-02T04:51:44.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:44.318 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:51:44.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:44.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:44.324 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:51:44.373 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:44.374 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:45.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:45.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:45.446 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 0 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:45.545 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:45.546 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:45.620 INFO:teuthology.orchestra.run.smithi160.stdout:[] 2022-09-02T04:51:45.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:45.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:51:45.624 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --shard-id 1 --max-entries 1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:51:45.673 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:45.674 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:45.731 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T04:51:45.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:45.732 INFO:teuthology.orchestra.run.smithi160.stdout: "entity_type": "bucket", 2022-09-02T04:51:45.732 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T04:51:45.732 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:51:38.749205Z", 2022-09-02T04:51:45.733 INFO:teuthology.orchestra.run.smithi160.stdout: "gen": 0 2022-09-02T04:51:45.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:45.733 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T04:51:45.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2022-09-02T04:51:45.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:45.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:45.743 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:51:45.794 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:45.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:45.814 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:45.814+0000 7fab0943f600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:51:45.815 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:45.814+0000 7fab0943f600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:51:45.817 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:45.818+0000 7fab0943f600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:51:45.818 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:51:45.818+0000 7fab0943f600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:51:45.821 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:45.822 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:51:45.822 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:51:45.823 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:51:45.824 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:51:45.825 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:51:45.826 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:51:45.826 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:51:45.826 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:51:45.827 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:51:45.828 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:51:45.828 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:51:45.829 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:51:45.829 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:51:45.830 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:51:45.830 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:51:45.831 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:51:45.831 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:51:45.832 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:51:45.832 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:51:45.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:45.833 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:51:45.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:45.835 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:51:45.835 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:51:45.836 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:51:45.836 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:51:45.837 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:51:45.837 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-02T04:51:45.838 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:51:45.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:45.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:45.839 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:51:45.840 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:51:45.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:45.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:45.841 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:45.841 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:51:45.841 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:51:45.841 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:45.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:45.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:51:45.846 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:51:45.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:51:45.896 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:51:45.938 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:51:46.977 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:51:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:51:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2, 2022-09-02T04:51:46.978 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:51:46.979 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2022-09-02T04:51:46.979 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:51:46.979 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:51:46.979 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:51:46.980 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2022-09-02T04:51:46.980 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:46.980 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:51:46.981 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:51:46.981 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:51:46.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:46.982 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:51:46.982 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:51:46.983 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:51:46.983 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:51:46.984 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:51:46.984 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:51:46.984 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:46.985 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:51:46.985 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:51:46.985 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:51:46.986 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:51:46.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:46.987 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:51:46.987 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:51:46.987 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:51:46.988 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:51:46.988 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:46.989 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:51:46.989 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:51:46.990 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:51:46.990 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:51:46.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:51:46.991 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:51:46.991 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:51:46.992 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:51:46.992 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:51:46.993 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:51:46.993 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:46.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:46.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:46.994 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:51:46.995 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:51:46.995 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:51:46.995 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:51:46.996 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:46.996 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:51:46.996 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:51:46.996 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:51:46.997 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:51:46.997 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:51:46.997 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:51:46.997 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:51:46.998 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:51:46.998 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:51:46.998 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:51:46.999 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:46.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:46.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:46.999 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:51:47.000 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:51:47.000 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:51:47.000 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:51:47.001 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:47.001 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:51:47.001 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:51:47.001 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:51:47.001 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:51:47.002 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:51:47.002 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:51:47.002 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:51:47.003 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:51:47.003 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:51:47.003 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:51:47.003 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:47.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:47.004 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:51:47.005 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:51:47.005 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:51:47.005 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:51:47.006 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:47.006 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:51:47.006 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:51:47.006 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:51:47.007 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:51:47.007 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:51:47.008 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:51:47.008 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:51:47.009 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:51:47.009 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:47.009 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:51:47.010 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:51:47.010 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:51:47.010 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:47.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:47.011 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:47.011 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:51:47.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:47.012 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:51:47.014 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:51:47.014 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:51:47.014 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:51:47.015 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:47.015 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:47.016 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:47.016 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:51:47.016 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:51:47.016 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:51:47.017 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:51:47.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.017 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:51:47.018 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:51:47.018 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:47.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:47.019 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:51:47.019 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:51:47.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:47.020 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:51:47.020 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:51:47.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:47.022 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:51:47.022 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:51:47.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:47.023 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:51:47.023 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:51:47.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:51:47.024 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:51:47.024 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:51:47.024 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:47.025 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:51:47.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.026 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:51:47.026 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:51:47.027 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:51:47.027 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:51:47.027 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:51:47.028 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:51:47.028 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:51:47.028 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:51:47.029 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:51:47.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.029 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:51:47.029 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:51:47.030 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:51:47.030 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:51:47.030 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:51:47.031 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:51:47.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.031 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:51:47.032 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:51:47.032 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:51:47.032 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:51:47.032 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:51:47.033 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:51:47.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.033 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:51:47.033 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:51:47.034 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:51:47.034 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:51:47.035 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:51:47.035 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:51:47.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.036 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:51:47.036 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:51:47.037 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:51:47.037 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:51:47.037 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:51:47.038 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:51:47.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:51:47.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:51:47.038 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:51:47.039 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:51:47.039 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:51:47.039 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:51:47.040 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2022-09-02T04:52:47.042 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-49 2022-09-02T04:52:47.044 DEBUG:boto:path=/xljbkf-49/ 2022-09-02T04:52:47.045 DEBUG:boto:auth_path=/xljbkf-49/ 2022-09-02T04:52:47.046 DEBUG:boto:Method: PUT 2022-09-02T04:52:47.047 DEBUG:boto:Path: /xljbkf-49/ 2022-09-02T04:52:47.048 DEBUG:boto:Data: 2022-09-02T04:52:47.050 DEBUG:boto:Headers: {} 2022-09-02T04:52:47.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:52:47.051 DEBUG:boto:Port: 8000 2022-09-02T04:52:47.052 DEBUG:boto:Params: {} 2022-09-02T04:52:47.055 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:52:47.056 DEBUG:boto:Token: None 2022-09-02T04:52:47.057 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:52:47 GMT /xljbkf-49/ 2022-09-02T04:52:47.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z/FIpGkqBPWcyTomwB07Ou9ZCvA= 2022-09-02T04:52:47.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:52:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z/FIpGkqBPWcyTomwB07Ou9ZCvA=', 'Content-Length': '0'} 2022-09-02T04:52:47.084 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096bbf91f42d387c3-0063118c1f-1519-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:52:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:52:47.085 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:52:47.086 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:52:47.086 DEBUG:boto:Method: PUT 2022-09-02T04:52:47.087 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:52:47.088 DEBUG:boto:Data: 2022-09-02T04:52:47.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T04:52:47.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:52:47.091 DEBUG:boto:Port: 8000 2022-09-02T04:52:47.092 DEBUG:boto:Params: {} 2022-09-02T04:52:47.093 DEBUG:boto:Token: None 2022-09-02T04:52:47.094 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 02 Sep 2022 04:52:47 GMT /xljbkf-49/testredirect 2022-09-02T04:52:47.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SaYpikMcagLRebT9YLQX0hRzDU0= 2022-09-02T04:52:47.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:52:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SaYpikMcagLRebT9YLQX0hRzDU0='} 2022-09-02T04:52:47.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053ee560ab8eddcde-0063118c1f-1519-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:52:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:52:47.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:47.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:47.114 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:52:47.176 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:52:47.177 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:52:47.220 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T04:52:47.241 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:52:47.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:52:47.242 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:52:47.242 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:52:47.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:52:47.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:52:47.245 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:52:47.245 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:52:47.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:52:47.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:52:47.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:52:47.246 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:52:47.083839Z" 2022-09-02T04:52:47.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:52:47.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:52:47.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:52:47.247 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:52:47.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:52:47.248 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:52:47.249 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:52:47.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:52:47.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:47.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:47.252 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:52:47.316 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:52:47.317 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:52:47.352 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:52:47.350+0000 7f266936b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:52:47.352 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:52:47.351+0000 7f266936b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:52:47.372 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:52:47.372 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:52:47.373 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:52:47.373 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:52:47.374 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:52:47.374 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:52:47.376 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:52:47.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:47.377 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:52:47.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:52:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:47.379 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:52:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:52:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:47.381 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:52:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:47.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:47.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:52:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:47.385 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:52:47.387 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:47.388 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:47.389 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:52:47.390 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:47.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:47.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:47.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:47.392 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:52:47.392 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:47.393 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:47.393 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:52:47.394 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:47.394 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:47.395 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:47.395 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:52:47.396 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:52:47.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:47.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:47.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:47.399 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:52:47.399 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:47.400 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:47.400 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:52:47.401 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:47.401 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:47.402 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:47.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:52:47.403 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:47.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:47.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:47.405 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:52:47.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:47.406 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:52:47.406 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:52:47.407 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:52:47.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:47.409 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:52:47Z" 2022-09-02T04:52:47.409 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:52:47.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:52:47Z" } 2022-09-02T04:52:47.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:52:47.412 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:52:47.413 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:52:47.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:52:52.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:52.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:52.418 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:52:52.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:52:52.477 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:52:52.508 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:52:52.508+0000 7f2ce18b5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:52:52.509 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:52:52.510+0000 7f2ce18b5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:52:52.523 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:52:52.524 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:52:52.524 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:52:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:52:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:52:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:52:52.526 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:52:52.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:52.527 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:52:52.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:52.528 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:52:52.528 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:52.529 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:52.529 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:52:52.530 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:52.530 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:52:52.531 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:52.531 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:52:52.532 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:52.532 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:52.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:52.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:52.534 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:52:52.534 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:52.534 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:52.535 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:52:52.535 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:52.536 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:52.536 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:52:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:52.537 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:52.538 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:52:52.539 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:52.539 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:52.539 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:52:52.539 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:52.540 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:52.540 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:52.540 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:52:52.540 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:52:52.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:52.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:52.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:52.541 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:52:52.542 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:52.542 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:52.542 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:52:52.542 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:52.543 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:52.543 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:52.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:52:52.543 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:52.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:52.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:52.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:52:52.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:52.545 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:52:52.545 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:52:52.545 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:52:52.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:52.546 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:52:52Z" 2022-09-02T04:52:52.546 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:52:52.547 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:52:52Z" } 2022-09-02T04:52:52.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:52:52.549 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:52:52.549 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:52:52.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:52:57.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:57.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:52:57.553 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:52:57.613 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:52:57.614 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:52:57.662 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:52:57.663+0000 7fd6ffb85600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:52:57.663 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:52:57.664+0000 7fd6ffb85600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:52:57.680 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:52:57.681 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:52:57.682 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:52:57.683 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:52:57.683 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:52:57.684 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:52:57.684 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:52:57.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:57.685 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:52:57.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:57.686 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:52:57.687 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:57.687 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:57.688 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:52:57.689 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:57.689 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:52:57.689 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:57.690 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:52:57.690 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:57.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:57.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:57.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:57.691 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:52:57.692 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:57.692 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:57.692 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:52:57.692 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:57.693 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:57.693 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:57.693 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:52:57.693 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:57.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:57.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:57.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:57.694 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:52:57.695 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:57.695 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:57.695 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:52:57.695 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:57.696 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:57.696 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:57.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:52:57.697 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:52:57.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:57.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:57.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:52:57.699 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:52:57.699 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:52:57.699 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:52:57.700 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:52:57.700 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:52:57.700 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:52:57.701 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:52:57.701 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:52:57.702 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:52:57.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:57.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:52:57.703 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:52:57.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:57.704 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:52:57.705 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:52:57.705 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:52:57.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:52:57.706 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:52:57Z" 2022-09-02T04:52:57.706 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:52:57.708 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:52:57Z" } 2022-09-02T04:52:57.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:52:57.710 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:52:57.711 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:52:57.711 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:02.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:02.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:02.714 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:02.773 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:02.773 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:02.808 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:02.809+0000 7f0ac1fb0600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:02.809 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:02.810+0000 7f0ac1fb0600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:02.826 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:02.827 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:02.827 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:02.828 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:02.828 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:02.829 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:02.829 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:02.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:02.830 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:02.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:02.831 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:02.832 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:02.832 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:02.833 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:02.833 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:02.833 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:02.834 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:02.834 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:02.835 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:02.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:02.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:02.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:02.837 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:02.837 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:02.838 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:02.838 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:02.839 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:02.839 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:02.839 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:02.840 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:02.840 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:02.841 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:02.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:02.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:02.842 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:02.843 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:02.844 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:02.845 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:02.845 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:02.846 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:02.846 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:02.847 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:02.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:02.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:02.849 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:02.851 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:02.852 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:02.853 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:02.853 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:02.854 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:02.854 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:02.855 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:02.855 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:02.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:02.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:02.857 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:02.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:02.858 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:02.859 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:02.860 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:02.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:02.860 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:02Z" 2022-09-02T04:53:02.861 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:02.863 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:02Z" } 2022-09-02T04:53:02.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:02.865 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:02.866 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:02.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:07.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:07.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:07.872 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:07.930 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:07.931 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:07.960 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:07.961+0000 7f4c8e960600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:07.963 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:07.962+0000 7f4c8e960600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:07.974 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:07.975 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:07.976 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:07.976 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:07.977 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:07.977 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:07.980 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:07.980 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:07.981 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:07.981 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:07.982 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:07.982 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:07.983 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:07.983 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:07.984 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:07.984 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:07.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:07.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:07.986 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:07.986 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:07.986 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:07.987 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:07.987 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:07.987 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:07.988 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:07.988 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:07.988 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:07.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:07.989 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:07.990 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:07.990 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:07.990 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:07.991 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:07.991 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:07.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:07.993 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:07.994 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:07.995 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:07.995 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:07.995 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:07.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:07.996 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:07.997 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:07Z" 2022-09-02T04:53:07.998 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:07.999 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:07Z" } 2022-09-02T04:53:08.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:08.000 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:08.001 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:08.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:13.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:13.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:13.007 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:13.067 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:13.067 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:13.120 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:13.120+0000 7fc1c5e0a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:13.121 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:13.121+0000 7fc1c5e0a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:13.144 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:13.144 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:13.144 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:13.145 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:13.145 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:13.145 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:13.146 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:13.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:13.146 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:13.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:13.147 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:13.147 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:13.147 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:13.147 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:13.148 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:13.148 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:13.148 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:13.148 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:13.149 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:13.149 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:13.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:13.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:13.150 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:13.150 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:13.150 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:13.150 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:13.151 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:13.151 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:13.151 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:13.151 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:13.152 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:13.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:13.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:13.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:13.153 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:13.153 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:13.153 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:13.153 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:13.154 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:13.154 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:13.154 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:13.154 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:13.155 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:13.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:13.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:13.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:13.156 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:13.156 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:13.156 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:13.156 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:13.157 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:13.157 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:13.157 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:13.157 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:13.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:13.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:13.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:13.160 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:13Z" 2022-09-02T04:53:13.160 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:13.161 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:13Z" } 2022-09-02T04:53:13.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:13.162 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:13.163 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:13.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:18.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:18.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:18.167 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:18.225 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:18.226 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:18.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:18.258+0000 7f5aeed3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:18.258 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:18.259+0000 7f5aeed3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:18.273 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:18.273 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:18.273 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:18.274 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:18.274 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:18.274 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:18.274 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:18.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:18.275 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:18.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:18.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:18.276 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:18.276 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:18.277 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:18.277 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:18.277 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:18.278 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:18.278 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:18.278 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:18.279 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:18.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:18.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:18.280 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:18.280 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:18.280 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:18.280 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:18.281 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:18.281 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:18.281 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:18.281 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:18.282 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:18.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:18.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:18.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:18.283 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:18.283 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:18.283 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:18.283 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:18.284 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:18.284 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:18.284 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:18.284 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:18.285 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:18.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:18.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:18.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:18.285 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:18.286 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:18.286 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:18.286 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:18.286 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:18.287 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:18.287 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:18.287 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:18.287 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:18.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:18.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:18.288 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:18.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:18.289 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:18.289 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:18.289 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:18.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:18.290 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:18Z" 2022-09-02T04:53:18.290 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:18.292 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:18Z" } 2022-09-02T04:53:18.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:18.294 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:18.295 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:18.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:23.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:23.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:23.299 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:23.357 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:23.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:23.409 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:23.407+0000 7f6260e5b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:23.409 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:23.409+0000 7f6260e5b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:23.427 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:23.428 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:23.428 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:23.428 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:23.429 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:23.429 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:23.429 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:23.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:23.430 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:23.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:23.431 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:23.431 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:23.431 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:23.431 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:23.432 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:23.432 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:23.432 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:23.432 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:23.433 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:23.433 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:23.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:23.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:23.434 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:23.434 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:23.434 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:23.435 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:23.435 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:23.435 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:23.435 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:23.436 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:23.436 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:23.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:23.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:23.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:23.437 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:23.437 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:23.438 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:23.438 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:23.438 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:23.438 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:23.439 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:23.439 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:23.439 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:23.439 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:23.440 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:23.440 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:23.441 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:23.441 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:23.441 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:23.442 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:23.442 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:23.442 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:23.442 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:23.443 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:23.443 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:23.443 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:23.444 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:23.444 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:23.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:23.445 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:23.445 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:23.445 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:23.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:23.446 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:23Z" 2022-09-02T04:53:23.446 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:23.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:23Z" } 2022-09-02T04:53:23.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:23.449 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:23.450 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:23.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:28.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:28.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:28.456 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:28.516 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:28.517 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:28.568 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:28.569+0000 7f2f57b7f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:28.569 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:28.570+0000 7f2f57b7f600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:28.592 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:28Z" 2022-09-02T04:53:28.609 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:28.610 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:28Z" } 2022-09-02T04:53:28.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:28.611 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:28.612 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:28.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:33.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:33.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:33.615 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:33.674 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:33.674 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:33.708 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:33.708+0000 7f1a296e0600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:33.708 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:33.709+0000 7f1a296e0600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:33.722 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:33.723 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:33.723 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:33.723 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:33.724 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:33.724 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:33.724 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:33.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:33.725 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:33.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:33.725 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:33.726 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:33.726 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:33.726 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:33.726 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:33.727 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:33.727 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:33.727 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:33.727 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:33.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:33.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:33.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:33.728 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:33.729 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:33.729 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:33.729 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:33.729 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:33.730 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:33.730 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:33.730 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:33.730 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:33.731 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:33.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:33.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:33.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:33.732 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:33.732 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:33.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:33.732 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:33.732 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:33.733 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:33.733 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:33.733 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:33.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:33.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:33.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:33.734 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:33.734 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:33.735 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:33.735 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:33.735 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:33.735 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:33.736 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:33.736 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:33.736 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:33.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:33.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:33.737 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:33.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:33.737 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:33.738 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:33.738 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:33.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:33.738 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:33Z" 2022-09-02T04:53:33.739 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:33.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:33Z" } 2022-09-02T04:53:33.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:33.742 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:33.742 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:33.743 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:38.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:38.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:38.747 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:38.806 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:38.806 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:38.855 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:38.857+0000 7f7b87c8e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:38.856 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:38.858+0000 7f7b87c8e600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:38.878 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:38.878 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:38.879 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:38.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:38.881 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:38.881 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:38.881 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:38.882 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:38.882 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:38.882 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:38.882 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:38.883 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:38.883 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:38.883 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:38.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:38.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:38.884 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:38.884 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:38.884 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:38.885 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:38.885 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:38.885 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:38.885 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:38.886 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:38.886 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:38.886 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:38.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:38.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:38.887 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:38.887 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:38.887 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:38.888 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:38.888 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:38.888 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:38.888 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:38.889 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:38.889 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:38.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:38.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:38.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:38.890 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:38.890 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:38.890 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:38.890 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:38.891 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:38.891 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:38.891 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:38.891 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:38.892 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:38.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:38.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:38.892 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:38.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:38.893 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:38.893 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:38.893 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:38.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:38.894 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:38Z" 2022-09-02T04:53:38.894 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:38.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:38Z" } 2022-09-02T04:53:38.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:38.897 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:38.897 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:38.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:43.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:43.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:43.902 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:43.959 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:43.960 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:43.990 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:43.992+0000 7f55c3b24600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:43.991 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:43.993+0000 7f55c3b24600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:44.005 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:44.006 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:44.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:44.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:44.009 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:44.010 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:44.011 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:44.012 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:44.013 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:44.018 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:44.018 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:44.018 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:44.019 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:44.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:44.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:44.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:44.021 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:44.022 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:44.022 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094259.579973_131.1", 2022-09-02T04:53:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:44.023 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:59.579973Z", 2022-09-02T04:53:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:44.032 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:44.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:44.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:44.033 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:44.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:44.034 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:44.035 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:44.035 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:44.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:44.036 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:43Z" 2022-09-02T04:53:44.037 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:44.039 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094259.579973_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:59.579973Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:43Z" } 2022-09-02T04:53:44.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094259.579973_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:59.579973Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:44.041 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:44.042 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094259.579973_131.1', 3: ''} 2022-09-02T04:53:44.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094367.083839_156.1 target=1_1662094259.579973_131.1 2022-09-02T04:53:49.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:49.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:49.047 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:49.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:49.104 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:49.136 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.137+0000 7efc1295b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:49.137 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.138+0000 7efc1295b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:49.154 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:49.155 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:49.155 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:49.156 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:49.156 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:49.157 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:49.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:49.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.158 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:49.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.159 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:49.159 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.160 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.160 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:49.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.161 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:49.161 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.162 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:49.162 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:49.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.164 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:49.164 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.164 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.165 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:49.165 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:49.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.167 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:49.167 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:49.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.169 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:49.169 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.169 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.170 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:53:49.170 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.171 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:49.171 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.171 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:53:49.172 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:49.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.173 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:49.174 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.174 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.175 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:49.175 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.175 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:49.176 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.176 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:49.176 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:49.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.177 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.178 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:49.178 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:49.178 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:49.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.182 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:49Z" 2022-09-02T04:53:49.182 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:49.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:49Z" } 2022-09-02T04:53:49.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:49.185 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:49.186 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:49.187 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:53:49.187 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:53:49.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:49.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:49.189 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:49.246 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:49.246 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:49.277 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:53:49.278+0000 7ff8a670a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:49.278 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:53:49.279+0000 7ff8a670a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:49.298 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:53:49.298 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:53:49.299 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:53:49.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:53:49.300 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:53:49.300 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:49.301 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:53:49.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:53:49.301 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:53:49.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:53:49.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:53:49.302 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:53:49.302 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:53:49.303 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:49.303 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:53:49.303 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:53:49.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:53:49.304 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:49.304 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:53:49.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:53:49.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:53:49.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:53:49.305 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:53:49.306 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:53:49.306 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:53:49.306 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:53:49.307 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:53:49.307 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:53:49.308 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:53:49.308 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:53:49.309 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:53:49.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:53:49.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:53:49.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:53:49.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:53:49.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:53:49.312 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:53:49.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:53:49.313 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:53:49.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:53:49.313 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:53:49.313 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:53:49.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:53:49.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:53:49.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:53:49.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:53:49.315 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:53:49.315 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:53:49.315 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:53:49.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:49.316 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:53:49.316 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:53:49.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:53:49.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:49.317 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:53:49.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:53:49.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:53:49.318 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:53:49.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:53:49.318 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:53:49.318 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:53:49.319 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:53:49.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:53:49.319 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:53:49Z" 2022-09-02T04:53:49.319 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:53:49.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:49Z" } 2022-09-02T04:53:49.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:49.322 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:49.323 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:49.323 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:53:49.324 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:53:49.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:49.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:53:49.325 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:53:49.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:49.378 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:49.410 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.412+0000 7f6aca715600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:49.411 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.413+0000 7f6aca715600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:53:49.425 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:49.425 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:53:49.425 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:53:49.426 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:53:49.426 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:53:49.426 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:49.427 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:49.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.427 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:53:49.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.428 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:53:49.428 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.428 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.429 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:53:49.429 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.429 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:53:49.430 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.430 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:53:49.430 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:49.430 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.431 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:53:49.431 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.432 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.432 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:53:49.432 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.432 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:49.433 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.433 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:53:49.433 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:49.433 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.434 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:53:49.435 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.435 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.435 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:53:49.435 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.435 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:49.436 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.436 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:53:49.436 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:49.436 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.437 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:53:49.437 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.438 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:53:49.438 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:53:49.439 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:53:49.439 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:53:49.439 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:53:49.439 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:53:49.440 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:53:49.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.440 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.441 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:53:49.441 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:53:49.441 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:53:49.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.442 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:53:49Z" 2022-09-02T04:53:49.442 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:49.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:53:49Z" } 2022-09-02T04:53:49.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:53:49.445 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:49.445 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:53:49.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:53:49.446 DEBUG:boto:path=/xljbkf-49/ 2022-09-02T04:53:49.447 DEBUG:boto:auth_path=/xljbkf-49/ 2022-09-02T04:53:49.447 DEBUG:boto:Method: HEAD 2022-09-02T04:53:49.448 DEBUG:boto:Path: /xljbkf-49/ 2022-09-02T04:53:49.449 DEBUG:boto:Data: 2022-09-02T04:53:49.449 DEBUG:boto:Headers: {} 2022-09-02T04:53:49.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:53:49.451 DEBUG:boto:Port: 8000 2022-09-02T04:53:49.452 DEBUG:boto:Params: {} 2022-09-02T04:53:49.454 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:53:49.455 DEBUG:boto:Token: None 2022-09-02T04:53:49.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:53:49 GMT /xljbkf-49/ 2022-09-02T04:53:49.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UnLmk4k6KkCk/KJFw2KslwN3Wlw= 2022-09-02T04:53:49.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:53:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UnLmk4k6KkCk/KJFw2KslwN3Wlw=', 'Content-Length': '0'} 2022-09-02T04:53:49.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078a911abe4ebe3ba-0063118c5d-1521-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:53:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:53:49.464 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:53:49.466 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:53:49.466 DEBUG:boto:Method: HEAD 2022-09-02T04:53:49.467 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:53:49.467 DEBUG:boto:Data: 2022-09-02T04:53:49.468 DEBUG:boto:Headers: {} 2022-09-02T04:53:49.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:53:49.469 DEBUG:boto:Port: 8000 2022-09-02T04:53:49.469 DEBUG:boto:Params: {} 2022-09-02T04:53:49.470 DEBUG:boto:Token: None 2022-09-02T04:53:49.470 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:53:49 GMT /xljbkf-49/testredirect 2022-09-02T04:53:49.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2AjCDspGxV4nfBoho4+zaFPqMRU= 2022-09-02T04:53:49.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:53:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2AjCDspGxV4nfBoho4+zaFPqMRU=', 'Content-Length': '0'} 2022-09-02T04:53:49.474 DEBUG:boto:Response headers: [('Content-Length', '229'), ('x-amz-request-id', 'tx00000da6a8463a48ce16f-0063118c5d-1521-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:53:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:53:49.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=b210c8c8-c433-420b-87c1-57ebfc433073', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:53:49.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=b210c8c8-c433-420b-87c1-57ebfc433073', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:53:49.477 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=b210c8c8-c433-420b-87c1-57ebfc433073 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:53:49.525 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:49.526 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:49.550 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.551+0000 7f80fdba3600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:53:49.550 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.551+0000 7f80fdba3600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:53:49.551 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.552+0000 7f80fdba3600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:53:49.551 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:53:49.552+0000 7f80fdba3600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:53:49.560 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:53:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:53:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:53:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:53:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:53:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:53:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:53:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:53:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:53:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:53:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:53:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:53:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:53:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:53:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:53:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:53:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:53:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:53:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:53:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:53:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:53:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:53:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:53:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:53:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:53:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-02T04:53:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:53:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:53:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:53:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:53:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:53:49.576 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:49.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:53:49.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:53:49.580 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:53:49.629 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:53:49.630 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:53:49.687 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:53:49.723 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:53:49.724 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:49.724 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3, 2022-09-02T04:53:49.725 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:53:49.725 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2022-09-02T04:53:49.726 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:53:49.726 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:53:49.726 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:53:49.727 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2022-09-02T04:53:49.727 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.727 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:53:49.727 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:53:49.728 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:53:49.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.728 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:53:49.728 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:53:49.729 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:53:49.729 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:53:49.729 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:53:49.729 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:53:49.730 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.730 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:53:49.730 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:53:49.730 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:53:49.731 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:53:49.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.731 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:53:49.731 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:53:49.732 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:53:49.732 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:53:49.732 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.732 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:53:49.733 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:53:49.733 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:53:49.733 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:53:49.733 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:53:49.734 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:53:49.734 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:53:49.734 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:53:49.735 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:53:49.735 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:53:49.735 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.737 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:53:49.737 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:53:49.738 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:53:49.738 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:53:49.739 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.739 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:53:49.739 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:53:49.740 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:53:49.740 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:53:49.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:53:49.741 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:53:49.741 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:53:49.742 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:53:49.742 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:53:49.742 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:53:49.743 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.744 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:53:49.744 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:53:49.744 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:53:49.744 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:53:49.745 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.745 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:53:49.745 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:53:49.745 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:53:49.746 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:53:49.746 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:53:49.746 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:53:49.746 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:53:49.747 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:53:49.747 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:53:49.747 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:53:49.747 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.748 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:53:49.749 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:53:49.749 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:53:49.749 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:53:49.750 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.750 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:53:49.751 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:53:49.751 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:53:49.752 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:53:49.752 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:53:49.753 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:53:49.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:53:49.754 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:53:49.754 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.754 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:53:49.755 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:53:49.755 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:53:49.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.756 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.757 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:53:49.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.759 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:53:49.760 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:53:49.760 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:53:49.760 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:53:49.760 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.761 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.761 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:53:49.761 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:53:49.762 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:53:49.762 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:53:49.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.763 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:53:49.763 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:53:49.763 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.763 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.764 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:53:49.764 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:53:49.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.764 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:53:49.765 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:53:49.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.765 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:53:49.766 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:53:49.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.766 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:53:49.767 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:53:49.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:53:49.768 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:53:49.768 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:53:49.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:53:49.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.769 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:53:49.770 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:53:49.770 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:53:49.771 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:53:49.771 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:53:49.771 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:53:49.771 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:53:49.771 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:53:49.772 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:53:49.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.772 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:53:49.772 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:53:49.773 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:53:49.773 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:53:49.773 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:53:49.774 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:53:49.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.774 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:53:49.775 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:53:49.775 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:53:49.775 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:53:49.776 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:53:49.776 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:53:49.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.776 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:53:49.777 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:53:49.777 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:53:49.777 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:53:49.778 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:53:49.778 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:53:49.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.778 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:53:49.779 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:53:49.779 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:53:49.779 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:53:49.779 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:53:49.780 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:53:49.780 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:53:49.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:53:49.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:53:49.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:53:49.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:53:49.781 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:53:49.782 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "b210c8c8-c433-420b-87c1-57ebfc433073" 2022-09-02T04:54:49.783 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.785 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.786 DEBUG:boto:Method: HEAD 2022-09-02T04:54:49.786 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.787 DEBUG:boto:Data: 2022-09-02T04:54:49.787 DEBUG:boto:Headers: {} 2022-09-02T04:54:49.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:54:49.788 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.789 DEBUG:boto:Params: {} 2022-09-02T04:54:49.790 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.790 DEBUG:boto:Token: None 2022-09-02T04:54:49.791 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4vHQknfVs1FjJjLPJFV9pGJK5g= 2022-09-02T04:54:49.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4vHQknfVs1FjJjLPJFV9pGJK5g=', 'Content-Length': '0'} 2022-09-02T04:54:49.798 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/xljbkf-49/testredirect?'), ('x-amz-request-id', 'tx000001d73ba44bbc586ce-0063118c99-1589-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.799 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/xljbkf-49/testredirect 2022-09-02T04:54:49.799 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2022-09-02T04:54:49.800 DEBUG:boto:Token: None 2022-09-02T04:54:49.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4vHQknfVs1FjJjLPJFV9pGJK5g= 2022-09-02T04:54:49.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4vHQknfVs1FjJjLPJFV9pGJK5g=', 'Content-Length': '0'} 2022-09-02T04:54:49.807 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000001a39b9cd9ec47c19-0063118c99-152d-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.808 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.809 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.810 DEBUG:boto:Method: GET 2022-09-02T04:54:49.810 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.811 DEBUG:boto:Data: 2022-09-02T04:54:49.811 DEBUG:boto:Headers: {} 2022-09-02T04:54:49.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:54:49.812 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.813 DEBUG:boto:Params: {} 2022-09-02T04:54:49.813 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.814 DEBUG:boto:Token: None 2022-09-02T04:54:49.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.819 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/xljbkf-49/testredirect?'), ('x-amz-request-id', 'tx00000baaf6f60bcb7a1b8-0063118c99-1589-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.819 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/xljbkf-49/testredirect 2022-09-02T04:54:49.820 DEBUG:boto:Token: None 2022-09-02T04:54:49.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.824 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:52:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000096838038cca2a168-0063118c99-152d-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.825 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.826 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.827 DEBUG:boto:Method: PUT 2022-09-02T04:54:49.828 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.828 DEBUG:boto:Data: 2022-09-02T04:54:49.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T04:54:49.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:54:49.830 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.831 DEBUG:boto:Params: {} 2022-09-02T04:54:49.832 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.833 DEBUG:boto:Token: None 2022-09-02T04:54:49.833 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ut7TEWpjkMkoyxzcN50yxOicXSs= 2022-09-02T04:54:49.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ut7TEWpjkMkoyxzcN50yxOicXSs='} 2022-09-02T04:54:49.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6ffe19d366d46de-0063118c99-152d-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.850 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.851 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.852 DEBUG:boto:Method: GET 2022-09-02T04:54:49.852 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.853 DEBUG:boto:Data: 2022-09-02T04:54:49.853 DEBUG:boto:Headers: {} 2022-09-02T04:54:49.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:54:49.854 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.855 DEBUG:boto:Params: {} 2022-09-02T04:54:49.855 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.856 DEBUG:boto:Token: None 2022-09-02T04:54:49.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.860 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/xljbkf-49/testredirect?'), ('x-amz-request-id', 'tx00000f472027a803d33c8-0063118c99-1589-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.861 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/xljbkf-49/testredirect 2022-09-02T04:54:49.862 DEBUG:boto:Token: None 2022-09-02T04:54:49.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.866 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000267096bcb9aeb2b6-0063118c99-152d-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.867 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.867 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.868 DEBUG:boto:Method: PUT 2022-09-02T04:54:49.868 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.869 DEBUG:boto:Data: 2022-09-02T04:54:49.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T04:54:49.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:54:49.870 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.871 DEBUG:boto:Params: {} 2022-09-02T04:54:49.871 DEBUG:boto:Token: None 2022-09-02T04:54:49.872 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mEkQ3Tz/hB37ql6kKGf/k9jnUls= 2022-09-02T04:54:49.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mEkQ3Tz/hB37ql6kKGf/k9jnUls='} 2022-09-02T04:54:49.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ada248103be4622-0063118c99-152d-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.881 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.882 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.882 DEBUG:boto:Method: GET 2022-09-02T04:54:49.883 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.883 DEBUG:boto:Data: 2022-09-02T04:54:49.884 DEBUG:boto:Headers: {} 2022-09-02T04:54:49.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:54:49.885 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.885 DEBUG:boto:Params: {} 2022-09-02T04:54:49.886 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.886 DEBUG:boto:Token: None 2022-09-02T04:54:49.887 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.891 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/xljbkf-49/testredirect?'), ('x-amz-request-id', 'tx00000fc2f33fb0d424c5d-0063118c99-1589-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.891 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/xljbkf-49/testredirect 2022-09-02T04:54:49.892 DEBUG:boto:Token: None 2022-09-02T04:54:49.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.896 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx000000bb7161d933dcab1-0063118c99-152d-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.897 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.898 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.898 DEBUG:boto:Method: PUT 2022-09-02T04:54:49.899 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.899 DEBUG:boto:Data: 2022-09-02T04:54:49.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T04:54:49.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:54:49.901 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.901 DEBUG:boto:Params: {} 2022-09-02T04:54:49.902 DEBUG:boto:Token: None 2022-09-02T04:54:49.903 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvj5yuctqHPal4kVhiqbbnGmXTM= 2022-09-02T04:54:49.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvj5yuctqHPal4kVhiqbbnGmXTM='} 2022-09-02T04:54:49.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3712872866bb1f8-0063118c99-152d-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.911 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.912 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.912 DEBUG:boto:Method: GET 2022-09-02T04:54:49.913 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.914 DEBUG:boto:Data: 2022-09-02T04:54:49.915 DEBUG:boto:Headers: {} 2022-09-02T04:54:49.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:54:49.916 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.917 DEBUG:boto:Params: {} 2022-09-02T04:54:49.917 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.918 DEBUG:boto:Token: None 2022-09-02T04:54:49.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.923 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/xljbkf-49/testredirect?'), ('x-amz-request-id', 'tx00000cde1d1f291b92b44-0063118c99-1589-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.924 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/xljbkf-49/testredirect 2022-09-02T04:54:49.925 DEBUG:boto:Token: None 2022-09-02T04:54:49.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.929 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx0000016d59b77684d701d-0063118c99-152d-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.930 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.931 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.931 DEBUG:boto:Method: PUT 2022-09-02T04:54:49.932 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.932 DEBUG:boto:Data: 2022-09-02T04:54:49.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T04:54:49.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:54:49.934 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.935 DEBUG:boto:Params: {} 2022-09-02T04:54:49.935 DEBUG:boto:Token: None 2022-09-02T04:54:49.936 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ceinlknAFKACTOUxJg8bBXP+jBQ= 2022-09-02T04:54:49.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ceinlknAFKACTOUxJg8bBXP+jBQ='} 2022-09-02T04:54:49.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac588e042247d720-0063118c99-152d-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.983 DEBUG:boto:path=/xljbkf-49/testredirect 2022-09-02T04:54:49.984 DEBUG:boto:auth_path=/xljbkf-49/testredirect 2022-09-02T04:54:49.984 DEBUG:boto:Method: GET 2022-09-02T04:54:49.985 DEBUG:boto:Path: /xljbkf-49/testredirect 2022-09-02T04:54:49.985 DEBUG:boto:Data: 2022-09-02T04:54:49.986 DEBUG:boto:Headers: {} 2022-09-02T04:54:49.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:54:49.987 DEBUG:boto:Port: 8000 2022-09-02T04:54:49.988 DEBUG:boto:Params: {} 2022-09-02T04:54:49.989 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:54:49.989 DEBUG:boto:Token: None 2022-09-02T04:54:49.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:49.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:49.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:49.995 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/xljbkf-49/testredirect?'), ('x-amz-request-id', 'tx00000e7c4a3ad1e9c96e0-0063118c99-1589-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:49.996 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/xljbkf-49/testredirect 2022-09-02T04:54:49.997 DEBUG:boto:Token: None 2022-09-02T04:54:49.999 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:54:49 GMT /xljbkf-49/testredirect 2022-09-02T04:54:50.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio= 2022-09-02T04:54:50.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:54:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqkN5oWqiXdNfhP2I1jYXC5IFio=', 'Content-Length': '0'} 2022-09-02T04:54:50.003 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 04:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000005926b2fe12c91976-0063118c9a-152d-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 04:54:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:54:50.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:54:50.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:54:50.005 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:54:50.062 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:54:50.063 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:54:50.086 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:54:50.087+0000 7f29b3101600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:54:50.087 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:54:50.087+0000 7f29b3101600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:54:50.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:54:50.091+0000 7f29b3101600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:54:50.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:54:50.091+0000 7f29b3101600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:54:50.097 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:54:50.097 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:54:50.098 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:54:50.098 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:54:50.098 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:54:50.098 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:54:50.099 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:54:50.099 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:54:50.099 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:54:50.100 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:54:50.100 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:54:50.100 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:54:50.100 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:54:50.101 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:54:50.101 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:54:50.101 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:54:50.102 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:54:50.102 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:54:50.102 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:54:50.103 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:54:50.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.104 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:54:50.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.105 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:54:50.105 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:54:50.106 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:54:50.106 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:54:50.107 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:54:50.107 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-02T04:54:50.108 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:54:50.108 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.109 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:54:50.110 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:54:50.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.111 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.111 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.112 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:54:50.112 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:54:50.113 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:54:50.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:54:50.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:54:50.117 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:54:50.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:54:50.168 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:54:50.221 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:54:50.255 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:54:50.255 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:54:50.256 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4, 2022-09-02T04:54:50.256 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:54:50.257 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2022-09-02T04:54:50.257 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:54:50.258 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:54:50.258 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:54:50.258 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2022-09-02T04:54:50.258 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.259 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:54:50.259 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:54:50.259 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:54:50.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.260 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:54:50.260 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:54:50.260 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:54:50.261 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:54:50.261 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:54:50.261 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:54:50.261 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.262 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:54:50.262 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:54:50.262 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:54:50.262 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:54:50.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.263 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:54:50.263 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:54:50.263 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:54:50.264 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:54:50.264 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.264 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:54:50.264 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:54:50.265 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:54:50.265 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:54:50.265 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:54:50.265 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:54:50.266 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:54:50.266 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:54:50.266 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:54:50.266 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:54:50.267 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.267 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:54:50.268 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:54:50.268 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:54:50.268 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:54:50.268 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.269 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:54:50.269 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:54:50.269 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:54:50.269 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:54:50.270 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:54:50.270 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:54:50.270 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:54:50.270 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:54:50.271 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:54:50.271 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:54:50.271 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.272 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:54:50.272 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:54:50.272 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:54:50.273 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:54:50.273 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.273 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:54:50.274 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:54:50.274 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:54:50.274 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:54:50.274 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:54:50.274 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:54:50.275 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:54:50.275 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:54:50.275 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:54:50.275 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:54:50.276 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.276 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:54:50.277 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:54:50.277 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:54:50.277 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:54:50.277 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.278 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:54:50.278 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:54:50.278 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:54:50.278 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:54:50.279 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:54:50.279 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:54:50.279 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:54:50.279 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:54:50.280 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.280 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:54:50.280 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:54:50.280 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:54:50.281 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.281 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.281 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:54:50.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.282 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:54:50.283 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:54:50.283 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:54:50.283 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:54:50.284 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.284 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.284 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:54:50.285 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:54:50.285 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:54:50.285 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:54:50.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.286 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:54:50.286 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:54:50.286 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.286 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.287 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:54:50.287 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:54:50.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.287 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:54:50.287 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:54:50.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.288 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:54:50.288 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:54:50.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.289 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:54:50.289 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:54:50.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:54:50.290 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:54:50.290 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:54:50.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.291 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:54:50.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.291 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:54:50.292 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:54:50.292 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:54:50.292 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:54:50.292 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:54:50.293 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:54:50.293 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:54:50.293 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:54:50.293 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:54:50.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.294 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:54:50.294 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:54:50.294 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:54:50.295 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:54:50.295 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:54:50.295 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:54:50.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.296 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:54:50.296 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:54:50.296 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:54:50.296 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:54:50.297 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:54:50.297 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:54:50.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.297 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:54:50.298 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:54:50.298 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:54:50.298 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:54:50.298 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:54:50.299 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:54:50.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.299 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:54:50.299 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:54:50.300 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:54:50.300 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:54:50.300 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:54:50.300 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:54:50.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:54:50.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:54:50.301 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:54:50.301 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:54:50.302 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:54:50.302 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:54:50.303 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2022-09-02T04:55:50.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:55:50.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:55:50.308 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:55:50.367 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:55:50.367 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:55:50.390 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:55:50.392+0000 7f04c02fc600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:55:50.391 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:55:50.392+0000 7f04c02fc600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:55:50.393 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:55:50.395+0000 7f04c02fc600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:55:50.394 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:55:50.395+0000 7f04c02fc600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:55:50.403 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:55:50.403 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:55:50.404 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:55:50.404 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-09-02T04:55:50.404 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "test-zone2.rgw.control", 2022-09-02T04:55:50.405 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-09-02T04:55:50.405 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-09-02T04:55:50.407 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "test-zone2.rgw.log", 2022-09-02T04:55:50.407 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-09-02T04:55:50.408 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-09-02T04:55:50.408 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-09-02T04:55:50.409 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-09-02T04:55:50.409 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-09-02T04:55:50.410 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-09-02T04:55:50.410 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-09-02T04:55:50.411 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-09-02T04:55:50.411 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-09-02T04:55:50.411 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:55:50.411 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:55:50.412 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:55:50.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.413 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:55:50.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.413 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:55:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:55:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-09-02T04:55:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:55:50.414 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:55:50.415 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-09-02T04:55:50.415 INFO:teuthology.orchestra.run.smithi160.stdout: "compression_type": "random" 2022-09-02T04:55:50.415 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.416 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-09-02T04:55:50.416 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:55:50.416 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.416 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.417 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.417 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:55:50.418 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-09-02T04:55:50.418 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:55:50.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:55:50.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:55:50.421 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:55:50.472 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:55:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:55:50.518 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:55:50.550 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:55:50.551 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:55:50.551 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5, 2022-09-02T04:55:50.551 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:55:50.552 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2022-09-02T04:55:50.552 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:55:50.552 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:55:50.552 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:55:50.553 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2022-09-02T04:55:50.553 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.553 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:55:50.553 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:55:50.554 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:55:50.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.554 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:55:50.554 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:55:50.555 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:55:50.555 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:55:50.555 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:55:50.555 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:55:50.556 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.556 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:55:50.556 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:55:50.556 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:55:50.557 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:55:50.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.557 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:55:50.557 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:55:50.558 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:55:50.558 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:55:50.558 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.558 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:55:50.559 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:55:50.559 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:55:50.559 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:55:50.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:55:50.560 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:55:50.560 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:55:50.560 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:55:50.561 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:55:50.561 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:55:50.562 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.562 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:55:50.563 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:55:50.563 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:55:50.563 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:55:50.563 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.564 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:55:50.564 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:55:50.564 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:55:50.565 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:55:50.565 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:55:50.565 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:55:50.565 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:55:50.565 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:55:50.566 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:55:50.566 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:55:50.566 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.567 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:55:50.567 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:55:50.568 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:55:50.568 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:55:50.568 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.568 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:55:50.569 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:55:50.569 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:55:50.569 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:55:50.569 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:55:50.570 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:55:50.570 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:55:50.570 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:55:50.571 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:55:50.571 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:55:50.571 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.572 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:55:50.573 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:55:50.573 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:55:50.573 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:55:50.573 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.574 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:55:50.574 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:55:50.574 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:55:50.574 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:55:50.575 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:55:50.575 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:55:50.575 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:55:50.575 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:55:50.576 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.576 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:55:50.576 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:55:50.576 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:55:50.577 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.577 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.577 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:55:50.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.578 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:55:50.580 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:55:50.580 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:55:50.580 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:55:50.581 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.581 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.581 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.581 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:55:50.582 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:55:50.582 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:55:50.582 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:55:50.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.583 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:55:50.584 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:55:50.584 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.584 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:55:50.585 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:55:50.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.585 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:55:50.585 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:55:50.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.586 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:55:50.587 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:55:50.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.587 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:55:50.588 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:55:50.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:55:50.589 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:55:50.589 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:55:50.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.590 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:55:50.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.591 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:55:50.591 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:55:50.591 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:55:50.592 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:55:50.592 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:55:50.593 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:55:50.593 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:55:50.594 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:55:50.594 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:55:50.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.594 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:55:50.595 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:55:50.595 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:55:50.595 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:55:50.595 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:55:50.596 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:55:50.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.596 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:55:50.596 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:55:50.597 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:55:50.597 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:55:50.597 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:55:50.597 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:55:50.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.598 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:55:50.598 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:55:50.599 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:55:50.599 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:55:50.599 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:55:50.599 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:55:50.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.600 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:55:50.600 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:55:50.601 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:55:50.601 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:55:50.601 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:55:50.601 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:55:50.602 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:55:50.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:55:50.602 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:55:50.602 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:55:50.603 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:55:50.603 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:55:50.603 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2022-09-02T04:56:50.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2022-09-02T04:56:50.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:50.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:50.610 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:56:50.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:50.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:50.708 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T04:56:50.727 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:56:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:56:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:56:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:56:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:56:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:56:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:52:47.083839Z" 2022-09-02T04:56:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:56:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:56:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:50.736 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:56:50.739 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:50.740 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:56:50.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:50.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:50.744 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:56:50.802 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:50.803 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:50.861 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:50.862+0000 7f052a007600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:56:50.862 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:50.863+0000 7f052a007600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:56:50.884 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:56:50.885 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:56:50.885 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:56:50.886 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:56:50.886 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:56:50.887 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:50.887 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:56:50.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:50.888 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:56:50.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:50.888 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:56:50.889 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:50.889 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:50.889 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:56:50.889 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:50.890 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:56:50.890 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:50.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:56:50.891 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:56:50.891 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:50.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:50.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:50.892 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:56:50.892 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:50.892 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:50.892 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:56:50.892 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:50.893 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:56:50.893 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:50.893 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:56:50.894 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:56:50.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:50.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:50.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:50.895 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:56:50.895 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:50.896 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:50.896 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:56:50.896 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:50.897 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:56:50.897 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:50.897 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:56:50.897 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:56:50.898 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:50.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:50.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:50.898 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:56:50.899 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:50.899 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:50.899 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:56:50.900 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:50.900 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:56:50.900 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:50.900 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:56:50.901 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:56:50.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:50.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:50.901 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:50.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:50.902 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:56:50.902 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:56:50.903 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:56:50.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:50.903 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:56:50Z" 2022-09-02T04:56:50.903 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:56:50.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:56:50Z" } 2022-09-02T04:56:50.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:56:50.906 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:50.907 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:50.907 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T04:56:50.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T04:56:50.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:50.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:50.909 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:56:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:50.996 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:50.997+0000 7fc31e7c2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:56:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:50.998+0000 7fc31e7c2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:56:51.020 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:51.021 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T04:56:51.021 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T04:56:51.022 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T04:56:51.022 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T04:56:51.023 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:51.023 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:56:51.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:51.024 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T04:56:51.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:51.025 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T04:56:51.025 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:56:51.026 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:56:51.026 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:56:51.027 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:56:51.027 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T04:56:51.027 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:56:51.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:56:51.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:56:51.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:51.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:51.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:51.030 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T04:56:51.030 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:56:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:56:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:56:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:56:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:56:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:56:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T04:56:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:56:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T04:56:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:56:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:56:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:56:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:56:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:56:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:56:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:56:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:56:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T04:56:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:56:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T04:56:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:56:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T04:56:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T04:56:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T04:56:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:56:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T04:56:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T04:56:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T04:56:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T04:56:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T04:56:51Z" 2022-09-02T04:56:51.047 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:51.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:56:51Z" } 2022-09-02T04:56:51.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:56:51.049 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:51.050 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:51.050 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T04:56:51.051 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T04:56:51.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:51.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:56:51.052 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:56:51.103 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:51.103 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:51.131 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.133+0000 7fd229914600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:56:51.132 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.134+0000 7fd229914600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:56:51.144 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:56:51.144 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:56:51.145 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:56:51.145 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:56:51.145 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:56:51.146 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:51.146 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:56:51.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.146 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:56:51.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:51.147 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:56:51.147 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:51.147 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:51.148 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:56:51.148 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:51.148 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:56:51.149 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:51.150 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:56:51.150 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:56:51.150 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:51.151 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:56:51.151 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:51.152 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:51.152 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:56:51.152 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:51.152 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:56:51.153 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:51.153 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:56:51.153 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:56:51.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:51.154 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:56:51.155 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:51.156 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:51.156 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:56:51.156 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:51.157 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:56:51.157 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:51.157 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:56:51.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:56:51.158 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:51.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:56:51.159 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:51.159 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:56:51.160 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:56:51.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:56:51.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:56:51.161 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:56:51.161 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:56:51.161 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:56:51.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.162 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:51.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.162 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:56:51.163 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:56:51.163 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:56:51.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.163 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:56:51Z" 2022-09-02T04:56:51.164 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:56:51.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:56:51Z" } 2022-09-02T04:56:51.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:56:51.166 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:51.167 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:56:51.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T04:56:51.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:51.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '6M6WHK7PDUE2FR0Y', '--secret', 'bI7gPpPYJPjU68G5buW0tUqRaRbabhZM', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:51.169 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 6M6WHK7PDUE2FR0Y --secret bI7gPpPYJPjU68G5buW0tUqRaRbabhZM --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:51.221 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:51.222 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:51.246 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.248+0000 7f26e2142600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:56:51.247 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.248+0000 7f26e2142600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:56:51.247 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.248+0000 7f26e2142600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:56:51.248 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.248+0000 7f26e2142600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:56:51.250 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:56:51.250 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "25e63244-a799-4a80-909f-adf397b2c9b2", 2022-09-02T04:56:51.251 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "remove", 2022-09-02T04:56:51.251 INFO:teuthology.orchestra.run.smithi160.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-02T04:56:51.251 INFO:teuthology.orchestra.run.smithi160.stdout: "control_pool": "remove.rgw.control", 2022-09-02T04:56:51.251 INFO:teuthology.orchestra.run.smithi160.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-02T04:56:51.252 INFO:teuthology.orchestra.run.smithi160.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-02T04:56:51.252 INFO:teuthology.orchestra.run.smithi160.stdout: "log_pool": "remove.rgw.log", 2022-09-02T04:56:51.252 INFO:teuthology.orchestra.run.smithi160.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-02T04:56:51.253 INFO:teuthology.orchestra.run.smithi160.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-02T04:56:51.253 INFO:teuthology.orchestra.run.smithi160.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-02T04:56:51.253 INFO:teuthology.orchestra.run.smithi160.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-02T04:56:51.253 INFO:teuthology.orchestra.run.smithi160.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-02T04:56:51.254 INFO:teuthology.orchestra.run.smithi160.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-02T04:56:51.254 INFO:teuthology.orchestra.run.smithi160.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-02T04:56:51.254 INFO:teuthology.orchestra.run.smithi160.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-02T04:56:51.254 INFO:teuthology.orchestra.run.smithi160.stdout: "otp_pool": "remove.rgw.otp", 2022-09-02T04:56:51.255 INFO:teuthology.orchestra.run.smithi160.stdout: "system_key": { 2022-09-02T04:56:51.255 INFO:teuthology.orchestra.run.smithi160.stdout: "access_key": "6M6WHK7PDUE2FR0Y", 2022-09-02T04:56:51.255 INFO:teuthology.orchestra.run.smithi160.stdout: "secret_key": "bI7gPpPYJPjU68G5buW0tUqRaRbabhZM" 2022-09-02T04:56:51.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.256 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_pools": [ 2022-09-02T04:56:51.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:51.256 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "default-placement", 2022-09-02T04:56:51.257 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:56:51.257 INFO:teuthology.orchestra.run.smithi160.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-02T04:56:51.257 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": { 2022-09-02T04:56:51.257 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD": { 2022-09-02T04:56:51.257 INFO:teuthology.orchestra.run.smithi160.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-02T04:56:51.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:51.259 INFO:teuthology.orchestra.run.smithi160.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-02T04:56:51.259 INFO:teuthology.orchestra.run.smithi160.stdout: "index_type": 0 2022-09-02T04:56:51.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:51.260 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:51.260 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:51.260 INFO:teuthology.orchestra.run.smithi160.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-02T04:56:51.260 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:56:51.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:51.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:51.264 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:51.312 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:51.313 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:51.321 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:51.323+0000 7fecd4fd7600 0 period (b798a9cd-aa0e-480c-bb07-05f02abf3f41 does not have zone 25e63244-a799-4a80-909f-adf397b2c9b2 configured 2022-09-02T04:56:58.133 INFO:teuthology.orchestra.run.smithi160.stderr:Sending period to new master zone b210c8c8-c433-420b-87c1-57ebfc433073 2022-09-02T04:56:58.166 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:56:58.167 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:58.167 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 6, 2022-09-02T04:56:58.168 INFO:teuthology.orchestra.run.smithi160.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:56:58.168 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": [ 2022-09-02T04:56:58.168 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:56:58.169 INFO:teuthology.orchestra.run.smithi160.stdout: "", 2022-09-02T04:56:58.169 INFO:teuthology.orchestra.run.smithi160.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:56:58.170 INFO:teuthology.orchestra.run.smithi160.stdout: "" 2022-09-02T04:56:58.170 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.170 INFO:teuthology.orchestra.run.smithi160.stdout: "period_map": { 2022-09-02T04:56:58.171 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:58.171 INFO:teuthology.orchestra.run.smithi160.stdout: "zonegroups": [ 2022-09-02T04:56:58.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.172 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "test-zonegroup", 2022-09-02T04:56:58.172 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zonegroup", 2022-09-02T04:56:58.173 INFO:teuthology.orchestra.run.smithi160.stdout: "api_name": "", 2022-09-02T04:56:58.173 INFO:teuthology.orchestra.run.smithi160.stdout: "is_master": true, 2022-09-02T04:56:58.174 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:56:58.174 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.175 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.175 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames": [], 2022-09-02T04:56:58.176 INFO:teuthology.orchestra.run.smithi160.stdout: "hostnames_s3website": [], 2022-09-02T04:56:58.176 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.177 INFO:teuthology.orchestra.run.smithi160.stdout: "zones": [ 2022-09-02T04:56:58.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.177 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "25e63244-a799-4a80-909f-adf397b2c9b2", 2022-09-02T04:56:58.178 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "remove", 2022-09-02T04:56:58.178 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [], 2022-09-02T04:56:58.178 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:56:58.178 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:56:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:56:58.179 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:56:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:56:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:56:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:56:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:56:58.180 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:56:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone3", 2022-09-02T04:56:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:56:58.182 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:56:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:56:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:56:58.183 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:56:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:56:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:56:58.184 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:56:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:56:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:56:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:56:58.185 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:58.186 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone2", 2022-09-02T04:56:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:56:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.187 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.188 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:56:58.188 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:56:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:56:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:56:58.189 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:56:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:56:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:56:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:56:58.190 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:56:58.191 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.192 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.192 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone1", 2022-09-02T04:56:58.192 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:56:58.192 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.193 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.193 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:56:58.193 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:56:58.193 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.194 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:56:58.194 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "", 2022-09-02T04:56:58.194 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": true, 2022-09-02T04:56:58.195 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [], 2022-09-02T04:56:58.195 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:56:58.196 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:56:58.196 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:56:58.197 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.198 INFO:teuthology.orchestra.run.smithi160.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:58.198 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "test-zone4", 2022-09-02T04:56:58.199 INFO:teuthology.orchestra.run.smithi160.stdout: "endpoints": [ 2022-09-02T04:56:58.199 INFO:teuthology.orchestra.run.smithi160.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:56:58.200 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.200 INFO:teuthology.orchestra.run.smithi160.stdout: "log_meta": false, 2022-09-02T04:56:58.202 INFO:teuthology.orchestra.run.smithi160.stdout: "log_data": true, 2022-09-02T04:56:58.203 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.203 INFO:teuthology.orchestra.run.smithi160.stdout: "read_only": false, 2022-09-02T04:56:58.204 INFO:teuthology.orchestra.run.smithi160.stdout: "tier_type": "pubsub", 2022-09-02T04:56:58.205 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from_all": false, 2022-09-02T04:56:58.205 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_from": [ 2022-09-02T04:56:58.206 INFO:teuthology.orchestra.run.smithi160.stdout: "test-zone1" 2022-09-02T04:56:58.206 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.206 INFO:teuthology.orchestra.run.smithi160.stdout: "redirect_zone": "", 2022-09-02T04:56:58.207 INFO:teuthology.orchestra.run.smithi160.stdout: "supported_features": [ 2022-09-02T04:56:58.207 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:56:58.208 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.208 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: "placement_targets": [ 2022-09-02T04:56:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "default-placement", 2022-09-02T04:56:58.211 INFO:teuthology.orchestra.run.smithi160.stdout: "tags": [], 2022-09-02T04:56:58.211 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_classes": [ 2022-09-02T04:56:58.212 INFO:teuthology.orchestra.run.smithi160.stdout: "STANDARD" 2022-09-02T04:56:58.213 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:58.213 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.214 INFO:teuthology.orchestra.run.smithi160.stdout: "default_placement": "default-placement", 2022-09-02T04:56:58.214 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:58.215 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_policy": { 2022-09-02T04:56:58.215 INFO:teuthology.orchestra.run.smithi160.stdout: "groups": [] 2022-09-02T04:56:58.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.216 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled_features": [ 2022-09-02T04:56:58.217 INFO:teuthology.orchestra.run.smithi160.stdout: "resharding" 2022-09-02T04:56:58.217 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:58.218 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2022-09-02T04:56:58.219 INFO:teuthology.orchestra.run.smithi160.stdout: "short_zone_ids": [ 2022-09-02T04:56:58.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.220 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "25e63244-a799-4a80-909f-adf397b2c9b2", 2022-09-02T04:56:58.220 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 804564347 2022-09-02T04:56:58.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.221 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.221 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:58.222 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 3855502858 2022-09-02T04:56:58.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.223 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:58.224 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 805948903 2022-09-02T04:56:58.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.225 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.226 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 2020452060 2022-09-02T04:56:58.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:56:58.227 INFO:teuthology.orchestra.run.smithi160.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:58.228 INFO:teuthology.orchestra.run.smithi160.stdout: "val": 4226356300 2022-09-02T04:56:58.228 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:58.229 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:56:58.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.230 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:56:58.231 INFO:teuthology.orchestra.run.smithi160.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.231 INFO:teuthology.orchestra.run.smithi160.stdout: "period_config": { 2022-09-02T04:56:58.232 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_quota": { 2022-09-02T04:56:58.233 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:56:58.233 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:56:58.234 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:56:58.234 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:56:58.235 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:56:58.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.236 INFO:teuthology.orchestra.run.smithi160.stdout: "user_quota": { 2022-09-02T04:56:58.237 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false, 2022-09-02T04:56:58.237 INFO:teuthology.orchestra.run.smithi160.stdout: "check_on_raw": false, 2022-09-02T04:56:58.238 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size": -1, 2022-09-02T04:56:58.238 INFO:teuthology.orchestra.run.smithi160.stdout: "max_size_kb": 0, 2022-09-02T04:56:58.239 INFO:teuthology.orchestra.run.smithi160.stdout: "max_objects": -1 2022-09-02T04:56:58.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.240 INFO:teuthology.orchestra.run.smithi160.stdout: "user_ratelimit": { 2022-09-02T04:56:58.240 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:56:58.241 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:56:58.241 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:56:58.242 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:56:58.242 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:56:58.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.243 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket_ratelimit": { 2022-09-02T04:56:58.244 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:56:58.245 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:56:58.245 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:56:58.246 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:56:58.246 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:56:58.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.247 INFO:teuthology.orchestra.run.smithi160.stdout: "anonymous_ratelimit": { 2022-09-02T04:56:58.248 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_ops": 0, 2022-09-02T04:56:58.249 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_ops": 0, 2022-09-02T04:56:58.249 INFO:teuthology.orchestra.run.smithi160.stdout: "max_read_bytes": 0, 2022-09-02T04:56:58.250 INFO:teuthology.orchestra.run.smithi160.stdout: "max_write_bytes": 0, 2022-09-02T04:56:58.250 INFO:teuthology.orchestra.run.smithi160.stdout: "enabled": false 2022-09-02T04:56:58.250 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:56:58.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:56:58.251 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:58.251 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_name": "test-realm", 2022-09-02T04:56:58.251 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:56:58.252 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:56:58.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.254 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:58.308 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:58.321 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:58.322+0000 7f595aaaa600 0 failed reading obj info from .rgw.root:zone_info.8b4ce987-f0fe-4038-9bfd-de8d46ec635c: (2) No such file or directory 2022-09-02T04:56:58.322 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:58.322+0000 7f595aaaa600 0 WARNING: could not read zone params for zone id=8b4ce987-f0fe-4038-9bfd-de8d46ec635c name=test-zone2 2022-09-02T04:56:58.322 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:58.323+0000 7f595aaaa600 0 failed reading obj info from .rgw.root:zone_info.fc4dbd55-e667-4fb2-9ede-75f3100f9d19: (2) No such file or directory 2022-09-02T04:56:58.323 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:58.323+0000 7f595aaaa600 0 WARNING: could not read zone params for zone id=fc4dbd55-e667-4fb2-9ede-75f3100f9d19 name=test-zone4 2022-09-02T04:56:58.325 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:58.326 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:56:58.326 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:56:58.326 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:56:58.327 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:56:58.327 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.327 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.328 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.328 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:56:58.328 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:56:58.328 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.329 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:56:58.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.329 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:58.329 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:56:58.330 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.330 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:56:58.330 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.330 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.331 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.331 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.331 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.331 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:58.332 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:58.332 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:58.332 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.332 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.332 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.333 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.333 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:58.334 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone2", 2022-09-02T04:56:58.334 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.334 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.334 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.335 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.335 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.335 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.335 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.336 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:58.336 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:58.336 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:58.336 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.337 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.337 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.337 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.338 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.338 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:56:58.338 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.339 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.339 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.339 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.339 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.340 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.340 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.340 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:58.340 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:58.341 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:58.341 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.341 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.341 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.342 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.342 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:58.343 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone4", 2022-09-02T04:56:58.343 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.343 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:56:58.344 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.344 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.344 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.344 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.345 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.345 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "pubsub", 2022-09-02T04:56:58.345 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2022-09-02T04:56:58.345 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [ 2022-09-02T04:56:58.346 INFO:teuthology.orchestra.run.smithi099.stdout: "test-zone1" 2022-09-02T04:56:58.346 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.346 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.346 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.347 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.347 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.347 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.348 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:56:58.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.348 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:56:58.348 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:56:58.349 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:56:58.349 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:56:58.349 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.349 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.349 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.350 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:56:58.350 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:58.350 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:56:58.350 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:56:58.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.351 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:56:58.351 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.352 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.352 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:58.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.355 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:58.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:58.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:58.428 INFO:teuthology.orchestra.run.smithi099.stderr:zone name remove not found in zonegroup test-zonegroup 2022-09-02T04:56:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-09-02T04:56:58.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.434 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:58.497 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:58.497 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:58.541+0000 7f011f671600 0 failed reading obj info from .rgw.root:zone_info.839886eb-91c7-4e3c-80d4-fd4e89474f26: (2) No such file or directory 2022-09-02T04:56:58.540 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:58.541+0000 7f011f671600 0 WARNING: could not read zone params for zone id=839886eb-91c7-4e3c-80d4-fd4e89474f26 name=test-zone3 2022-09-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:58.542+0000 7f011f671600 0 failed reading obj info from .rgw.root:zone_info.b210c8c8-c433-420b-87c1-57ebfc433073: (2) No such file or directory 2022-09-02T04:56:58.541 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:58.542+0000 7f011f671600 0 WARNING: could not read zone params for zone id=b210c8c8-c433-420b-87c1-57ebfc433073 name=test-zone1 2022-09-02T04:56:58.544 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:56:58.546+0000 7f011f671600 0 zone id 25e63244-a799-4a80-909f-adf397b2c9b2 is not a part of zonegroup default 2022-09-02T04:56:58.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.562 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:58.668 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:58.669+0000 7f90c19fd600 0 WARNING: can't generate connection for zone 25e63244-a799-4a80-909f-adf397b2c9b2 id remove: no endpoints defined 2022-09-02T04:56:58.796 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7, 2022-09-02T04:56:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:56:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [ 2022-09-02T04:56:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:56:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:56:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:56:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "" 2022-09-02T04:56:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2022-09-02T04:56:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2022-09-02T04:56:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:56:58.804 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:56:58.804 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:56:58.805 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:56:58.805 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.806 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.806 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.806 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:56:58.807 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:56:58.807 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.807 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:56:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:56:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:56:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.810 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.810 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.810 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.811 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:58.811 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:58.811 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:58.812 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.812 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.812 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.813 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:58.813 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone2", 2022-09-02T04:56:58.814 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.814 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.814 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.815 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.815 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.815 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.815 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.816 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:58.816 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:58.816 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:58.816 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.817 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.817 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.818 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.818 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:56:58.818 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.819 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:58.819 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.819 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.819 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.820 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.820 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.820 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:58.821 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:58.821 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:58.821 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.821 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.822 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.822 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.823 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:58.823 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone4", 2022-09-02T04:56:58.823 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:58.823 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:56:58.824 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.824 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:58.824 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:58.824 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:58.825 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:58.825 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "pubsub", 2022-09-02T04:56:58.825 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2022-09-02T04:56:58.825 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [ 2022-09-02T04:56:58.826 INFO:teuthology.orchestra.run.smithi099.stdout: "test-zone1" 2022-09-02T04:56:58.826 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.826 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:58.826 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:58.827 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.827 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.827 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.828 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:56:58.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.828 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:56:58.830 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:56:58.830 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:56:58.830 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:56:58.831 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.831 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.831 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:56:58.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:58.832 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:56:58.832 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:56:58.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.833 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:56:58.833 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.833 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.834 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:58.834 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2022-09-02T04:56:58.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.834 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:58.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3855502858 2022-09-02T04:56:58.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:58.836 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 805948903 2022-09-02T04:56:58.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.837 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.838 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2020452060 2022-09-02T04:56:58.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:58.839 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:58.839 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4226356300 2022-09-02T04:56:58.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.840 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.840 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:56:58.841 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:58.841 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2022-09-02T04:56:58.841 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:56:58.841 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:56:58.842 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:56:58.842 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:56:58.842 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:56:58.842 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:56:58.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.843 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:56:58.843 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:56:58.843 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:56:58.844 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:56:58.844 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:56:58.844 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:56:58.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2022-09-02T04:56:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:58.846 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:58.846 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:58.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.847 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2022-09-02T04:56:58.847 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:58.847 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:58.848 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:58.848 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:58.849 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:58.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.849 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2022-09-02T04:56:58.850 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:58.850 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:58.851 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:58.851 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:58.852 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:58.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:58.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.853 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:58.853 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2022-09-02T04:56:58.853 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:56:58.853 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:58.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2022-09-02T04:56:58.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.856 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:58.906 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:58.907 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:58.940 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:58.940 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "95854e69-1ac9-443c-b381-8dbcc5a88e82", 2022-09-02T04:56:58.940 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove_zg", 2022-09-02T04:56:58.940 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "remove_zg", 2022-09-02T04:56:58.941 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2022-09-02T04:56:58.941 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2022-09-02T04:56:58.941 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:56:58.942 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:56:58.942 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "", 2022-09-02T04:56:58.942 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [], 2022-09-02T04:56:58.942 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [], 2022-09-02T04:56:58.943 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2022-09-02T04:56:58.943 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:58.943 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:56:58.943 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:56:58.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:58.944 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:56:58.944 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:58.944 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:58.945 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:58.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:58.948 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:59.044 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:59.044 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "c1d5412b-f307-4809-b256-63b543bb5dc5", 2022-09-02T04:56:59.045 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2022-09-02T04:56:59.045 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "remove.rgw.meta:root", 2022-09-02T04:56:59.048 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "remove.rgw.control", 2022-09-02T04:56:59.048 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "remove.rgw.log:gc", 2022-09-02T04:56:59.048 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "remove.rgw.log:lc", 2022-09-02T04:56:59.049 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "remove.rgw.log", 2022-09-02T04:56:59.049 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-09-02T04:56:59.049 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-09-02T04:56:59.050 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-09-02T04:56:59.050 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-09-02T04:56:59.050 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-09-02T04:56:59.051 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-09-02T04:56:59.051 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-09-02T04:56:59.052 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-09-02T04:56:59.052 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "remove.rgw.otp", 2022-09-02T04:56:59.052 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2022-09-02T04:56:59.053 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2022-09-02T04:56:59.053 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2022-09-02T04:56:59.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.053 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2022-09-02T04:56:59.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2022-09-02T04:56:59.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T04:56:59.055 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "remove.rgw.buckets.index", 2022-09-02T04:56:59.055 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2022-09-02T04:56:59.055 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2022-09-02T04:56:59.055 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "remove.rgw.buckets.data" 2022-09-02T04:56:59.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.057 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-09-02T04:56:59.057 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0 2022-09-02T04:56:59.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "remove.rgw.log:notif" 2022-09-02T04:56:59.060 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:59.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.063 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:59.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:59.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:59.229 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:59.229 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:59.230 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8, 2022-09-02T04:56:59.230 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:56:59.231 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [ 2022-09-02T04:56:59.231 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:56:59.231 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:56:59.231 INFO:teuthology.orchestra.run.smithi099.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:56:59.232 INFO:teuthology.orchestra.run.smithi099.stdout: "" 2022-09-02T04:56:59.232 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.233 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2022-09-02T04:56:59.233 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:59.233 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2022-09-02T04:56:59.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.234 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "95854e69-1ac9-443c-b381-8dbcc5a88e82", 2022-09-02T04:56:59.235 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove_zg", 2022-09-02T04:56:59.235 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "remove_zg", 2022-09-02T04:56:59.235 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2022-09-02T04:56:59.235 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2022-09-02T04:56:59.236 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:56:59.236 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:56:59.236 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "c1d5412b-f307-4809-b256-63b543bb5dc5", 2022-09-02T04:56:59.236 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:56:59.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.237 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "c1d5412b-f307-4809-b256-63b543bb5dc5", 2022-09-02T04:56:59.237 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2022-09-02T04:56:59.238 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2022-09-02T04:56:59.238 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.238 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": false, 2022-09-02T04:56:59.239 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.239 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.239 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.239 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.240 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.240 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.240 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.240 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.241 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.241 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.241 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:56:59.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.242 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:56:59.242 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:56:59.243 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:56:59.243 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:56:59.243 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.244 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.245 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:56:59.245 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:59.245 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:56:59.245 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:56:59.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.246 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:56:59.246 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.247 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.247 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:56:59.248 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:56:59.248 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:56:59.248 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:56:59.248 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.249 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:59.249 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.249 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:56:59.249 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:56:59.250 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.250 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:56:59.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.250 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:59.251 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:56:59.251 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.251 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:56:59.252 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.252 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.253 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.253 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.253 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.254 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.254 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.254 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.255 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.255 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.255 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.256 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.256 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:59.257 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone2", 2022-09-02T04:56:59.257 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.257 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:56:59.257 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.258 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.258 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.259 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.259 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.260 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.260 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.261 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.261 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.262 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.263 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.263 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.265 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.265 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:56:59.266 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.266 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:59.267 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.267 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.267 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.268 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.268 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.268 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.268 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.269 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.269 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.269 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.269 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.270 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.271 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:59.272 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone4", 2022-09-02T04:56:59.272 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.273 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:56:59.273 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.274 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.274 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.275 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.275 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.275 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "pubsub", 2022-09-02T04:56:59.276 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2022-09-02T04:56:59.276 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [ 2022-09-02T04:56:59.276 INFO:teuthology.orchestra.run.smithi099.stdout: "test-zone1" 2022-09-02T04:56:59.276 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.277 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.277 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.277 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.278 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.278 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.279 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:56:59.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.279 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:56:59.280 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:56:59.280 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:56:59.280 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:56:59.280 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.281 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.282 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:56:59.282 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:59.283 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:56:59.283 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:56:59.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.284 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:56:59.285 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.285 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.286 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.286 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2022-09-02T04:56:59.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.287 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:59.287 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3855502858 2022-09-02T04:56:59.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.288 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:59.288 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 805948903 2022-09-02T04:56:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2020452060 2022-09-02T04:56:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "c1d5412b-f307-4809-b256-63b543bb5dc5", 2022-09-02T04:56:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 756792064 2022-09-02T04:56:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4226356300 2022-09-02T04:56:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:56:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2022-09-02T04:56:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:56:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:56:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:56:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:56:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:56:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:56:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:56:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:56:59.298 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:56:59.298 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:56:59.298 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:56:59.299 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:56:59.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.300 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2022-09-02T04:56:59.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:59.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:59.301 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:59.301 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:59.301 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:59.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.302 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2022-09-02T04:56:59.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:59.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:59.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:59.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:59.303 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:59.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.304 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2022-09-02T04:56:59.304 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:59.305 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:59.305 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:59.305 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:59.306 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:59.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.307 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:59.307 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2022-09-02T04:56:59.307 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:56:59.307 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:59.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.309 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:59.360 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:59.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:59.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:59.395+0000 7f272d7d5600 0 zone id c1d5412b-f307-4809-b256-63b543bb5dc5 is not a part of zonegroup test-zonegroup 2022-09-02T04:56:59.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:59.396+0000 7f272d7d5600 0 zone id c1d5412b-f307-4809-b256-63b543bb5dc5 is not a part of zonegroup default 2022-09-02T04:56:59.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.409 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:59.463 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:59.463 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:59.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T04:56:59.494+0000 7f9548fc6600 0 ERROR: zonegroup remove_zg has a non existent master zone 2022-09-02T04:56:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-09-02T04:56:59.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.499 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:59.557 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:59.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T04:56:59.605 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T04:56:59.664 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:56:59.665 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:56:59.787 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T04:56:59.788 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:59.788 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9, 2022-09-02T04:56:59.789 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "ffa7b930-977f-415c-8bb4-bf6fd394dc85", 2022-09-02T04:56:59.789 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [ 2022-09-02T04:56:59.789 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:56:59.790 INFO:teuthology.orchestra.run.smithi099.stdout: "", 2022-09-02T04:56:59.790 INFO:teuthology.orchestra.run.smithi099.stdout: "1_1662094176.417998_108.1", 2022-09-02T04:56:59.790 INFO:teuthology.orchestra.run.smithi099.stdout: "" 2022-09-02T04:56:59.790 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.791 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2022-09-02T04:56:59.791 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:56:59.791 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2022-09-02T04:56:59.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.792 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "test-zonegroup", 2022-09-02T04:56:59.792 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zonegroup", 2022-09-02T04:56:59.792 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2022-09-02T04:56:59.792 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2022-09-02T04:56:59.793 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.793 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:59.793 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.793 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2022-09-02T04:56:59.794 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2022-09-02T04:56:59.794 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.794 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2022-09-02T04:56:59.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.795 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:59.795 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone3", 2022-09-02T04:56:59.795 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.795 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2022-09-02T04:56:59.796 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.796 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.796 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.797 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.797 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.797 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.797 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.798 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.798 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.798 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.798 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.799 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.799 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:59.800 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone2", 2022-09-02T04:56:59.800 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.800 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8000" 2022-09-02T04:56:59.800 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.801 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.801 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.801 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.801 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.802 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.802 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.802 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.802 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.803 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.803 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.803 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.804 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.804 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone1", 2022-09-02T04:56:59.804 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.804 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2022-09-02T04:56:59.805 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.805 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.805 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.805 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.806 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.806 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2022-09-02T04:56:59.806 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2022-09-02T04:56:59.806 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2022-09-02T04:56:59.807 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.807 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.807 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.807 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.808 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:59.808 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-zone4", 2022-09-02T04:56:59.809 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2022-09-02T04:56:59.809 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi160.front.sepia.ceph.com:8001" 2022-09-02T04:56:59.809 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.809 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2022-09-02T04:56:59.810 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2022-09-02T04:56:59.810 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2022-09-02T04:56:59.810 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2022-09-02T04:56:59.810 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "pubsub", 2022-09-02T04:56:59.811 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2022-09-02T04:56:59.811 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [ 2022-09-02T04:56:59.811 INFO:teuthology.orchestra.run.smithi099.stdout: "test-zone1" 2022-09-02T04:56:59.811 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.812 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2022-09-02T04:56:59.812 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2022-09-02T04:56:59.812 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.813 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.813 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2022-09-02T04:56:59.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.814 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2022-09-02T04:56:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2022-09-02T04:56:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2022-09-02T04:56:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2022-09-02T04:56:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2022-09-02T04:56:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2022-09-02T04:56:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2022-09-02T04:56:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2022-09-02T04:56:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2022-09-02T04:56:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2022-09-02T04:56:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2022-09-02T04:56:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T04:56:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3855502858 2022-09-02T04:56:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T04:56:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 805948903 2022-09-02T04:56:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 2020452060 2022-09-02T04:56:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:56:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "fc4dbd55-e667-4fb2-9ede-75f3100f9d19", 2022-09-02T04:56:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4226356300 2022-09-02T04:56:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T04:56:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "test-zonegroup", 2022-09-02T04:56:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T04:56:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2022-09-02T04:56:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2022-09-02T04:56:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:56:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:56:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:56:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:56:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:56:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2022-09-02T04:56:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2022-09-02T04:56:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2022-09-02T04:56:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2022-09-02T04:56:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2022-09-02T04:56:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2022-09-02T04:56:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2022-09-02T04:56:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2022-09-02T04:56:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2022-09-02T04:56:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2022-09-02T04:56:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2022-09-02T04:56:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2022-09-02T04:56:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2022-09-02T04:56:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2022-09-02T04:56:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:56:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:56:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "d53aaf3c-e9e8-4189-80a7-dfda98d01df2", 2022-09-02T04:56:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2022-09-02T04:56:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T04:56:59.861 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T04:56:59.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2022-09-02T04:56:59.864 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-50 2022-09-02T04:56:59.864 DEBUG:boto:path=/xljbkf-50/ 2022-09-02T04:56:59.865 DEBUG:boto:auth_path=/xljbkf-50/ 2022-09-02T04:56:59.866 DEBUG:boto:Method: PUT 2022-09-02T04:56:59.866 DEBUG:boto:Path: /xljbkf-50/ 2022-09-02T04:56:59.867 DEBUG:boto:Data: 2022-09-02T04:56:59.867 DEBUG:boto:Headers: {} 2022-09-02T04:56:59.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:56:59.868 DEBUG:boto:Port: 8000 2022-09-02T04:56:59.869 DEBUG:boto:Params: {} 2022-09-02T04:56:59.870 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:56:59.870 DEBUG:boto:Token: None 2022-09-02T04:56:59.871 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:56:59 GMT /xljbkf-50/ 2022-09-02T04:56:59.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vFU37UJ2CZv/SJjpZgn5w3fvaYc= 2022-09-02T04:56:59.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:56:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vFU37UJ2CZv/SJjpZgn5w3fvaYc=', 'Content-Length': '0'} 2022-09-02T04:57:00.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009985f2876bd5daaf-0063118d1b-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.003 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-51 2022-09-02T04:57:00.003 DEBUG:boto:path=/xljbkf-51/ 2022-09-02T04:57:00.004 DEBUG:boto:auth_path=/xljbkf-51/ 2022-09-02T04:57:00.004 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.005 DEBUG:boto:Path: /xljbkf-51/ 2022-09-02T04:57:00.005 DEBUG:boto:Data: 2022-09-02T04:57:00.006 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:57:00.007 DEBUG:boto:Port: 8000 2022-09-02T04:57:00.007 DEBUG:boto:Params: {} 2022-09-02T04:57:00.008 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T04:57:00.008 DEBUG:boto:Token: None 2022-09-02T04:57:00.009 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-51/ 2022-09-02T04:57:00.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VsBYn8mFYz66pXY2G5khQRIWUDI= 2022-09-02T04:57:00.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VsBYn8mFYz66pXY2G5khQRIWUDI=', 'Content-Length': '0'} 2022-09-02T04:57:00.375 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f4959e9243689add-0063118d1c-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.376 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-52 2022-09-02T04:57:00.377 DEBUG:boto:path=/xljbkf-52/ 2022-09-02T04:57:00.377 DEBUG:boto:auth_path=/xljbkf-52/ 2022-09-02T04:57:00.378 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.379 DEBUG:boto:Path: /xljbkf-52/ 2022-09-02T04:57:00.379 DEBUG:boto:Data: 2022-09-02T04:57:00.380 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:57:00.381 DEBUG:boto:Port: 8001 2022-09-02T04:57:00.381 DEBUG:boto:Params: {} 2022-09-02T04:57:00.382 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T04:57:00.383 DEBUG:boto:Token: None 2022-09-02T04:57:00.384 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-52/ 2022-09-02T04:57:00.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rhAoakpVtT2DVDHF0KIwSQ91Ess= 2022-09-02T04:57:00.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rhAoakpVtT2DVDHF0KIwSQ91Ess=', 'Content-Length': '0'} 2022-09-02T04:57:00.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb87736fe53d721a-0063118d1c-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.729 DEBUG:boto:path=/xljbkf-50/ 2022-09-02T04:57:00.730 DEBUG:boto:auth_path=/xljbkf-50/ 2022-09-02T04:57:00.731 DEBUG:boto:path=/xljbkf-50/?website 2022-09-02T04:57:00.732 DEBUG:boto:auth_path=/xljbkf-50/?website 2022-09-02T04:57:00.733 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.734 DEBUG:boto:Path: /xljbkf-50/?website 2022-09-02T04:57:00.735 DEBUG:boto:Data: index.htmlerror.html 2022-09-02T04:57:00.735 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:57:00.737 DEBUG:boto:Port: 8000 2022-09-02T04:57:00.738 DEBUG:boto:Params: {} 2022-09-02T04:57:00.738 DEBUG:boto:Token: None 2022-09-02T04:57:00.739 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-50/?website 2022-09-02T04:57:00.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gEzyqGpv4MB4EcXxnP25yo0OETE= 2022-09-02T04:57:00.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gEzyqGpv4MB4EcXxnP25yo0OETE=', 'Content-Length': '241'} 2022-09-02T04:57:00.742 DEBUG:boto:Response headers: [('Content-Length', '202'), ('x-amz-request-id', 'tx000000c2b6449b8dce7e0-0063118d1c-1595-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2022-09-02T04:57:00.745 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-53 2022-09-02T04:57:00.745 DEBUG:boto:path=/xljbkf-53/ 2022-09-02T04:57:00.746 DEBUG:boto:auth_path=/xljbkf-53/ 2022-09-02T04:57:00.747 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.748 DEBUG:boto:Path: /xljbkf-53/ 2022-09-02T04:57:00.748 DEBUG:boto:Data: 2022-09-02T04:57:00.749 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:57:00.751 DEBUG:boto:Port: 8000 2022-09-02T04:57:00.752 DEBUG:boto:Params: {} 2022-09-02T04:57:00.753 DEBUG:boto:Token: None 2022-09-02T04:57:00.754 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-53/ 2022-09-02T04:57:00.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F1HA3Jwu3APBQiPGIHZlpDh6xVo= 2022-09-02T04:57:00.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F1HA3Jwu3APBQiPGIHZlpDh6xVo=', 'Content-Length': '0'} 2022-09-02T04:57:00.809 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003053897ab78021c6-0063118d1c-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.810 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-54 2022-09-02T04:57:00.811 DEBUG:boto:path=/xljbkf-54/ 2022-09-02T04:57:00.812 DEBUG:boto:auth_path=/xljbkf-54/ 2022-09-02T04:57:00.813 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.814 DEBUG:boto:Path: /xljbkf-54/ 2022-09-02T04:57:00.815 DEBUG:boto:Data: 2022-09-02T04:57:00.816 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:57:00.818 DEBUG:boto:Port: 8000 2022-09-02T04:57:00.819 DEBUG:boto:Params: {} 2022-09-02T04:57:00.820 DEBUG:boto:Token: None 2022-09-02T04:57:00.821 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-54/ 2022-09-02T04:57:00.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HZeAteX6G08d+Ugv5mM9WHQskFk= 2022-09-02T04:57:00.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HZeAteX6G08d+Ugv5mM9WHQskFk=', 'Content-Length': '0'} 2022-09-02T04:57:00.887 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010a162cbe43545ad-0063118d1c-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.888 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-55 2022-09-02T04:57:00.889 DEBUG:boto:path=/xljbkf-55/ 2022-09-02T04:57:00.889 DEBUG:boto:auth_path=/xljbkf-55/ 2022-09-02T04:57:00.890 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.890 DEBUG:boto:Path: /xljbkf-55/ 2022-09-02T04:57:00.891 DEBUG:boto:Data: 2022-09-02T04:57:00.891 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:57:00.892 DEBUG:boto:Port: 8001 2022-09-02T04:57:00.893 DEBUG:boto:Params: {} 2022-09-02T04:57:00.893 DEBUG:boto:Token: None 2022-09-02T04:57:00.894 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-55/ 2022-09-02T04:57:00.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x/HspQB6jXkWFSETQETGItUckJc= 2022-09-02T04:57:00.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x/HspQB6jXkWFSETQETGItUckJc=', 'Content-Length': '0'} 2022-09-02T04:57:00.963 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bbc213f69653f82e-0063118d1c-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.963 DEBUG:boto:path=/xljbkf-53/ 2022-09-02T04:57:00.964 DEBUG:boto:auth_path=/xljbkf-53/ 2022-09-02T04:57:00.965 DEBUG:boto:path=/xljbkf-53/?policy 2022-09-02T04:57:00.965 DEBUG:boto:auth_path=/xljbkf-53/?policy 2022-09-02T04:57:00.966 DEBUG:boto:Method: PUT 2022-09-02T04:57:00.966 DEBUG:boto:Path: /xljbkf-53/?policy 2022-09-02T04:57:00.967 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-02T04:57:00.967 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:57:00.968 DEBUG:boto:Port: 8000 2022-09-02T04:57:00.968 DEBUG:boto:Params: {} 2022-09-02T04:57:00.969 DEBUG:boto:Token: None 2022-09-02T04:57:00.970 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-53/?policy 2022-09-02T04:57:00.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qQgxgwECw+v+aCVYcYZEGSsOfVw= 2022-09-02T04:57:00.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qQgxgwECw+v+aCVYcYZEGSsOfVw=', 'Content-Length': '97'} 2022-09-02T04:57:00.976 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000c89719d74debe89-0063118d1c-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 04:57:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:00.988 DEBUG:boto:path=/xljbkf-53/ 2022-09-02T04:57:00.988 DEBUG:boto:auth_path=/xljbkf-53/ 2022-09-02T04:57:00.989 DEBUG:boto:path=/xljbkf-53/?policy 2022-09-02T04:57:00.990 DEBUG:boto:auth_path=/xljbkf-53/?policy 2022-09-02T04:57:00.991 DEBUG:boto:Method: GET 2022-09-02T04:57:00.992 DEBUG:boto:Path: /xljbkf-53/?policy 2022-09-02T04:57:00.993 DEBUG:boto:Data: 2022-09-02T04:57:00.994 DEBUG:boto:Headers: {} 2022-09-02T04:57:00.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:57:00.996 DEBUG:boto:Port: 8000 2022-09-02T04:57:00.996 DEBUG:boto:Params: {} 2022-09-02T04:57:00.997 DEBUG:boto:Token: None 2022-09-02T04:57:00.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:57:00 GMT /xljbkf-53/?policy 2022-09-02T04:57:00.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hlPVgobybGxYg0LhsDRr0QmhRm0= 2022-09-02T04:57:01.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hlPVgobybGxYg0LhsDRr0QmhRm0=', 'Content-Length': '0'} 2022-09-02T04:57:01.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095250bbb91b8bb39-0063118d1d-1595-test-zone1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 02 Sep 2022 04:57:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:01.003 DEBUG:boto:path=/xljbkf-54/ 2022-09-02T04:57:01.004 DEBUG:boto:auth_path=/xljbkf-54/ 2022-09-02T04:57:01.004 DEBUG:boto:path=/xljbkf-54/?policy 2022-09-02T04:57:01.005 DEBUG:boto:auth_path=/xljbkf-54/?policy 2022-09-02T04:57:01.006 DEBUG:boto:Method: PUT 2022-09-02T04:57:01.007 DEBUG:boto:Path: /xljbkf-54/?policy 2022-09-02T04:57:01.008 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-02T04:57:01.008 DEBUG:boto:Headers: {} 2022-09-02T04:57:01.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:57:01.009 DEBUG:boto:Port: 8000 2022-09-02T04:57:01.010 DEBUG:boto:Params: {} 2022-09-02T04:57:01.010 DEBUG:boto:Token: None 2022-09-02T04:57:01.011 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:01 GMT /xljbkf-54/?policy 2022-09-02T04:57:01.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pszbnAsxI3R8/dOfRHZ296xU6MQ= 2022-09-02T04:57:01.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pszbnAsxI3R8/dOfRHZ296xU6MQ=', 'Content-Length': '97'} 2022-09-02T04:57:02.023 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000040eece55f89eae69-0063118d1d-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 04:57:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:02.024 DEBUG:boto:path=/xljbkf-54/ 2022-09-02T04:57:02.024 DEBUG:boto:auth_path=/xljbkf-54/ 2022-09-02T04:57:02.025 DEBUG:boto:path=/xljbkf-54/?policy 2022-09-02T04:57:02.025 DEBUG:boto:auth_path=/xljbkf-54/?policy 2022-09-02T04:57:02.026 DEBUG:boto:Method: GET 2022-09-02T04:57:02.026 DEBUG:boto:Path: /xljbkf-54/?policy 2022-09-02T04:57:02.027 DEBUG:boto:Data: 2022-09-02T04:57:02.027 DEBUG:boto:Headers: {} 2022-09-02T04:57:02.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:57:02.028 DEBUG:boto:Port: 8000 2022-09-02T04:57:02.029 DEBUG:boto:Params: {} 2022-09-02T04:57:02.029 DEBUG:boto:Token: None 2022-09-02T04:57:02.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:57:02 GMT /xljbkf-54/?policy 2022-09-02T04:57:02.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qO4I1vImQ8ahd6y1TqIvVsgFvcg= 2022-09-02T04:57:02.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qO4I1vImQ8ahd6y1TqIvVsgFvcg=', 'Content-Length': '0'} 2022-09-02T04:57:02.033 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042e651eb32803a25-0063118d1e-15e3-test-zone2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 02 Sep 2022 04:57:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:02.033 DEBUG:boto:path=/xljbkf-55/ 2022-09-02T04:57:02.034 DEBUG:boto:auth_path=/xljbkf-55/ 2022-09-02T04:57:02.034 DEBUG:boto:path=/xljbkf-55/?policy 2022-09-02T04:57:02.035 DEBUG:boto:auth_path=/xljbkf-55/?policy 2022-09-02T04:57:02.035 DEBUG:boto:Method: PUT 2022-09-02T04:57:02.036 DEBUG:boto:Path: /xljbkf-55/?policy 2022-09-02T04:57:02.036 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2022-09-02T04:57:02.037 DEBUG:boto:Headers: {} 2022-09-02T04:57:02.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:57:02.039 DEBUG:boto:Port: 8001 2022-09-02T04:57:02.040 DEBUG:boto:Params: {} 2022-09-02T04:57:02.041 DEBUG:boto:Token: None 2022-09-02T04:57:02.042 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:02 GMT /xljbkf-55/?policy 2022-09-02T04:57:02.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:StRepO1egYjrA6zalGb9P5s/dGM= 2022-09-02T04:57:02.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:StRepO1egYjrA6zalGb9P5s/dGM=', 'Content-Length': '97'} 2022-09-02T04:57:03.057 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006977a3e2b573fb65-0063118d1e-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 04:57:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:03.058 DEBUG:boto:path=/xljbkf-55/ 2022-09-02T04:57:03.059 DEBUG:boto:auth_path=/xljbkf-55/ 2022-09-02T04:57:03.060 DEBUG:boto:path=/xljbkf-55/?policy 2022-09-02T04:57:03.061 DEBUG:boto:auth_path=/xljbkf-55/?policy 2022-09-02T04:57:03.061 DEBUG:boto:Method: GET 2022-09-02T04:57:03.062 DEBUG:boto:Path: /xljbkf-55/?policy 2022-09-02T04:57:03.063 DEBUG:boto:Data: 2022-09-02T04:57:03.064 DEBUG:boto:Headers: {} 2022-09-02T04:57:03.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:57:03.066 DEBUG:boto:Port: 8001 2022-09-02T04:57:03.067 DEBUG:boto:Params: {} 2022-09-02T04:57:03.068 DEBUG:boto:Token: None 2022-09-02T04:57:03.069 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 04:57:03 GMT /xljbkf-55/?policy 2022-09-02T04:57:03.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:80cxsuK4QtS8yQtnjn+eXaVYAMU= 2022-09-02T04:57:03.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:80cxsuK4QtS8yQtnjn+eXaVYAMU=', 'Content-Length': '0'} 2022-09-02T04:57:03.075 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008a86ce3b86826955-0063118d1f-1599-test-zone3'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 02 Sep 2022 04:57:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:03.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2022-09-02T04:57:03.078 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-56 2022-09-02T04:57:03.078 DEBUG:boto:path=/xljbkf-56/ 2022-09-02T04:57:03.079 DEBUG:boto:auth_path=/xljbkf-56/ 2022-09-02T04:57:03.079 DEBUG:boto:Method: PUT 2022-09-02T04:57:03.080 DEBUG:boto:Path: /xljbkf-56/ 2022-09-02T04:57:03.080 DEBUG:boto:Data: 2022-09-02T04:57:03.081 DEBUG:boto:Headers: {} 2022-09-02T04:57:03.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T04:57:03.082 DEBUG:boto:Port: 8000 2022-09-02T04:57:03.082 DEBUG:boto:Params: {} 2022-09-02T04:57:03.083 DEBUG:boto:Token: None 2022-09-02T04:57:03.084 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:03 GMT /xljbkf-56/ 2022-09-02T04:57:03.084 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8Gw+bv11lEw66rvItJnEXO0jXxU= 2022-09-02T04:57:03.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8Gw+bv11lEw66rvItJnEXO0jXxU=', 'Content-Length': '0'} 2022-09-02T04:57:03.118 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da591d1737a8dfc0-0063118d1f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:03.119 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-57 2022-09-02T04:57:03.120 DEBUG:boto:path=/xljbkf-57/ 2022-09-02T04:57:03.121 DEBUG:boto:auth_path=/xljbkf-57/ 2022-09-02T04:57:03.122 DEBUG:boto:Method: PUT 2022-09-02T04:57:03.122 DEBUG:boto:Path: /xljbkf-57/ 2022-09-02T04:57:03.123 DEBUG:boto:Data: 2022-09-02T04:57:03.124 DEBUG:boto:Headers: {} 2022-09-02T04:57:03.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T04:57:03.125 DEBUG:boto:Port: 8000 2022-09-02T04:57:03.126 DEBUG:boto:Params: {} 2022-09-02T04:57:03.127 DEBUG:boto:Token: None 2022-09-02T04:57:03.127 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:03 GMT /xljbkf-57/ 2022-09-02T04:57:03.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:57CtJ3FX2qLQJrce3Xgo+f9eqTY= 2022-09-02T04:57:03.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:57CtJ3FX2qLQJrce3Xgo+f9eqTY=', 'Content-Length': '0'} 2022-09-02T04:57:03.177 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b17970b7f1cb806b-0063118d1f-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:03.178 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-58 2022-09-02T04:57:03.179 DEBUG:boto:path=/xljbkf-58/ 2022-09-02T04:57:03.179 DEBUG:boto:auth_path=/xljbkf-58/ 2022-09-02T04:57:03.180 DEBUG:boto:Method: PUT 2022-09-02T04:57:03.181 DEBUG:boto:Path: /xljbkf-58/ 2022-09-02T04:57:03.181 DEBUG:boto:Data: 2022-09-02T04:57:03.182 DEBUG:boto:Headers: {} 2022-09-02T04:57:03.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T04:57:03.183 DEBUG:boto:Port: 8001 2022-09-02T04:57:03.184 DEBUG:boto:Params: {} 2022-09-02T04:57:03.184 DEBUG:boto:Token: None 2022-09-02T04:57:03.185 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 04:57:03 GMT /xljbkf-58/ 2022-09-02T04:57:03.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T7KqSY37F6fA3lnx56+phXeL1OI= 2022-09-02T04:57:03.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 04:57:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T7KqSY37F6fA3lnx56+phXeL1OI=', 'Content-Length': '0'} 2022-09-02T04:57:03.231 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000259a68d0e248fd1-0063118d1f-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 04:57:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T04:57:03.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:03.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:03.233 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:03.285 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:03.285 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:03.327 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T04:57:03.344 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T04:57:03.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:57:03.345 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:57:03.346 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:57:03.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:57:03.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:57:03.347 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:57:03.347 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:57:03.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:57:03.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:57:03.348 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094623.215257_258.1", 2022-09-02T04:57:03.348 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T04:57:03.215257Z" 2022-09-02T04:57:03.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T04:57:03.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T04:57:03.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T04:57:03.349 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T04:57:03.349 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T04:57:03.350 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T04:57:03.352 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:03.352 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T04:57:03.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:03.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:03.354 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:03.412 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:03.412 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:03.469 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:03.470+0000 7fbc3a7c6600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:03.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:03.471+0000 7fbc3a7c6600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:03.493 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:03.494 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:03.494 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:03.494 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:03.494 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:03.495 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:03.495 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:03.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:03.495 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:03.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:03.496 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:03.496 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:03.496 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:03.497 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:03.497 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:03.497 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:03.498 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:03.498 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:03.498 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:03.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:03.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:03.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:03.500 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:03.500 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:03.500 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:03.501 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:03.501 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:03.501 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:03.501 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:03.502 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:03.502 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:03.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:03.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:03.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:03.503 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:03.503 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:03.504 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:03.504 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:03.505 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:03.505 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:03.505 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:03.506 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:03.506 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:03.506 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:03.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:03.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:03.507 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:03.507 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:03.508 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:03.508 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:03.508 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:03.508 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:03.509 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:03.509 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:03.509 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:03.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:03.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:03.510 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:03.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:03.511 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:03.512 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:03.512 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:03.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:03.513 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:03Z" 2022-09-02T04:57:03.513 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:03.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:03Z" } 2022-09-02T04:57:03.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:03.517 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:03.517 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:03.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:08.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:08.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:08.520 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:08.577 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:08.577 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:08.610 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:08.611+0000 7f8e606f2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:08.611 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:08.612+0000 7f8e606f2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:08.625 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:08.625 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:08.625 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:08.626 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:08.626 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:08.626 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:08.627 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:08.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:08.627 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:08.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:08.628 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:08.628 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:08.628 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:08.628 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:08.629 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:08.629 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:08.629 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:08.630 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:08.630 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:08.630 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:08.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:08.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:08.631 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:08.631 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:08.632 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:08.632 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:08.632 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:08.632 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:08.633 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:08.633 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:08.633 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:08.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:08.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:08.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:08.634 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:08.634 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:08.635 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:08.635 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:08.635 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:08.635 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:08.636 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:08.636 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:08.636 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:08.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:08.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:08.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:08.637 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:08.637 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:08.638 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:08.638 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:08.638 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:08.638 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:08.639 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:08.639 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:08.639 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:08.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:08.640 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:08.640 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:08.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:08.640 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:08.641 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:08.641 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:08.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:08.641 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:08Z" 2022-09-02T04:57:08.642 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:08.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:08Z" } 2022-09-02T04:57:08.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:08.644 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:08.645 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:08.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:13.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:13.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:13.648 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:13.707 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:13.707 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:13.763 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:13.764+0000 7fc2efacf600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:13.764 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:13.765+0000 7fc2efacf600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:13.787 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:13.788 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:13.788 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:13.788 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:13.788 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:13.789 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:13.789 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:13.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:13.790 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:13.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:13.790 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:13.791 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:13.791 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:13.792 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:13.792 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:13.793 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:13.793 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:13.794 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:13.794 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:13.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:13.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:13.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:13.796 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:13.797 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:13.798 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:13.798 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:13.799 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:13.799 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:13.800 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:13.800 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:13.800 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:13.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:13.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:13.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:13.802 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:13.806 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:13.806 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:13.807 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:13.807 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:13.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:13.810 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:13.810 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:13.810 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:13.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:13.811 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:13Z" 2022-09-02T04:57:13.811 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:13.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:13Z" } 2022-09-02T04:57:13.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:13.814 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:13.814 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:13.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:18.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:18.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:18.817 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:18.876 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:18.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:18.911 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:18.912+0000 7f9a39935600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:18.912 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:18.913+0000 7f9a39935600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:18.935 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:18.936 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:18.937 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:18.937 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:18.937 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:18.938 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:18.938 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:18.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:18.938 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:18.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:18.939 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:18.939 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:18.939 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:18.940 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:18.940 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:18.940 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:18.940 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:18.941 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:18.942 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:18.943 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:18.944 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:18.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:18.946 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:18.947 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:18.947 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:18.947 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:18.948 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:18.948 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:18.948 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:18.948 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:18.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:18.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:18.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:18.949 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:18.950 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:18.950 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:18.950 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:18.951 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:18.952 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:18.952 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:18.953 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:18.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:18.954 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:18.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:18.956 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:18Z" 2022-09-02T04:57:18.956 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:18.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:18Z" } 2022-09-02T04:57:18.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:18.959 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:18.959 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:18.960 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:23.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:23.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:23.963 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:24.022 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:24.022 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:24.060 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:24.060+0000 7f63b7c85600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:24.060 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:24.061+0000 7f63b7c85600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:24.079 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:24.079 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:24.079 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:24.080 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:24.080 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:24.080 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:24.080 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:24.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:24.081 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:24.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:24.081 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:24.082 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:24.082 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:24.082 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:24.082 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:24.083 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:24.083 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:24.083 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:24.083 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:24.084 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:24.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:24.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:24.084 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:24.085 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:24.085 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:24.085 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:24.085 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:24.086 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:24.086 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:24.086 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:24.087 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:24.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:24.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:24.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:24.087 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:24.088 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:24.088 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:24.088 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:24.088 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:24.089 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:24.089 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:24.089 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:24.089 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:24.090 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:24.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:24.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:24.091 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:24.091 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:24.091 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:24.091 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:24.092 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:24.092 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:24.092 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:24.092 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:24.093 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:24.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:24.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:24.093 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:24.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:24.094 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:24.094 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:24.094 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:24.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:24.095 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:24Z" 2022-09-02T04:57:24.095 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:24.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:24Z" } 2022-09-02T04:57:24.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:24.097 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:24.098 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:24.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:29.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:29.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:29.101 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:29.159 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:29.159 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:29.206 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:29.206+0000 7fd412ccc600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:29.207 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:29.208+0000 7fd412ccc600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:29.230 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:29.230 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:29.231 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:29.231 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:29.231 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:29.232 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:29.232 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:29.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:29.233 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:29.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:29.233 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:29.233 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:29.234 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:29.234 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:29.234 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:29.234 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:29.235 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:29.235 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:29.235 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:29.235 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:29.236 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:29.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:29.236 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:29.236 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:29.237 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:29.237 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:29.237 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:29.237 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:29.237 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:29.238 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:29.238 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:29.238 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:29.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:29.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:29.239 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:29.239 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:29.239 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:29.240 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:29.240 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:29.240 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:29.240 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:29.241 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:29.241 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:29.241 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:29.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:29.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:29.242 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:29.242 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:29.243 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:29.243 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:29.243 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:29.244 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:29.244 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:29.244 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:29.245 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:29.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:29.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:29.245 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:29.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:29.246 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:29.246 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:29.246 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:29.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:29.247 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:29Z" 2022-09-02T04:57:29.247 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:29.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:29Z" } 2022-09-02T04:57:29.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:29.250 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:29.250 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:29.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:34.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:34.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:34.253 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:34.310 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:34.311 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:34.346 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:34.346+0000 7feb9c793600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:34.347 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:34.348+0000 7feb9c793600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:34.366 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:34.368 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:34.369 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:34.370 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:34.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:34.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:34.374 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:34.375 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:34.376 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:34.377 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:34.378 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:34.379 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:34.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:34.381 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:34.382 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:34.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:34.384 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:34.385 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:34Z" 2022-09-02T04:57:34.386 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:34.386 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:34Z" } 2022-09-02T04:57:34.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:34.387 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:34.388 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:34.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:39.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:39.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:39.392 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:39.449 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:39.450 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:39.480 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:39.481+0000 7efe4b4ac600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:39.482 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:39.482+0000 7efe4b4ac600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:39.496 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:39Z" 2022-09-02T04:57:39.515 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:39.516 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:39Z" } 2022-09-02T04:57:39.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:39.518 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:39.518 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:39.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:44.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:44.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:44.521 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:44.578 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:44.579 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:44.612 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:44.613+0000 7f27072b5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:44.613 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:44.614+0000 7f27072b5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:44.633 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:44.634 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:44.634 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:44.634 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:44.635 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:44.635 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:44.635 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:44.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:44.636 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:44.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:44.636 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:44.636 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:44.637 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:44.637 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:44.637 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:44.638 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:44.638 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:44.638 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:44.638 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:44.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:44.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:44.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:44.640 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:44.640 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:44.640 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:44.641 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:44.641 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:44.641 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:44.641 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:44.642 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:44.642 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:44.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:44.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:44.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:44.644 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:44.644 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:44.645 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:44.645 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:44.645 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:44.645 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:44.646 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:44.646 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:44.646 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:44.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:44.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:44.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:44.647 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:44.647 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:44.648 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:44.648 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:44.648 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:44.648 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:44.649 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:44.649 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:44.649 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:44.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:44.650 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:44.650 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:44.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:44.650 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:44.651 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:44.651 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:44.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:44.652 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:44Z" 2022-09-02T04:57:44.652 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:44.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:44Z" } 2022-09-02T04:57:44.654 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:44.654 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:44.655 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:44.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:49.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:49.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:49.658 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:49.716 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:49.716 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:49.768 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:49.769+0000 7f69163f5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:49.769 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:49.770+0000 7f69163f5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:49.786 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:49.787 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:49.787 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:49.787 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:49.787 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:49.788 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:49.788 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:49.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:49.788 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:49.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:49.789 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:49.789 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:49.789 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:49.790 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:49.790 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:49.790 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:49.790 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:49.791 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:49.791 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:49.791 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:49.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:49.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:49.792 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:49.792 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:49.792 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:49.793 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:49.793 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:49.793 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:49.793 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:49.794 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:49.794 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:49.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:49.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:49.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:49.795 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:49.795 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:49.795 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:49.796 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:49.796 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:49.796 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:49.796 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:49.797 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:49.797 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:49.797 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:49.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:49.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:49.798 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:49.798 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:49.799 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:49.799 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:49.799 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:49.799 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:49.800 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:49.800 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:49.800 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:49.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:49.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:49.801 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:49.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:49.801 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:49.801 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:49.802 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:49.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:49.802 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:49Z" 2022-09-02T04:57:49.802 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:49.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:49Z" } 2022-09-02T04:57:49.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:49.805 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:49.805 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:49.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:54.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:54.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:54.809 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:57:54.866 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:57:54.867 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:57:54.900 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:54.901+0000 7f4bcc437600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:54.900 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:57:54.902+0000 7f4bcc437600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:57:54.914 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:57:54.915 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:57:54.915 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:57:54.915 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:57:54.916 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:57:54.916 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:57:54.917 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:54.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:54.918 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:57:54.918 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:54.919 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:57:54.919 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:54.920 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:54.920 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:57:54.921 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:54.921 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:57:54.922 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:54.922 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:57:54.923 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:54.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:54.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:54.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:54.925 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:57:54.925 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:54.926 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:54.926 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:57:54.927 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:54.927 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:54.927 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:54.928 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:57:54.928 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:54.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:54.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:54.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:54.929 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:57:54.929 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:54.930 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:54.930 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:57:54.930 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:54.931 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:54.931 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:54.931 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:57:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:57:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:54.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:57:54.933 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:57:54.933 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:57:54.933 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:57:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:57:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:57:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:57:54.934 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:57:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:57:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:57:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:54.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:57:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:57:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:57:54.936 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:57:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:57:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:57:54.937 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:57:54Z" 2022-09-02T04:57:54.938 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:57:54.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:57:54Z" } 2022-09-02T04:57:54.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:57:54.941 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:57:54.941 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:57:54.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:57:59.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:59.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:57:59.946 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:00.004 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:00.005 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:00.062 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:00.064+0000 7f940a34b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:00.063 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:00.065+0000 7f940a34b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:00.086 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:00.086 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:00.087 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:00.087 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:00.087 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:00.087 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:00.088 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:00.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:00.088 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:00.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:00.089 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:00.090 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:00.090 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:00.090 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:00.091 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:00.091 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:00.091 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:00.091 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:00.092 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:00.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:00.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:00.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:00.093 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:00.093 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:00.093 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:00.093 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:00.094 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:00.094 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:00.094 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:00.094 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:00.095 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:00.095 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:00.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:00.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:00.096 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:00.096 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:00.096 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:00.096 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:00.097 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:00.097 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:00.097 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:00.098 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:00.098 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:00.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:00.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:00.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:00.099 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:00.099 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:00.100 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:00.100 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:00.100 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:00.100 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:00.101 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:00.101 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:00.101 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:00.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:00.102 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:00.102 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:00.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:00.103 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:00.103 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:00.103 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:00.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:00.104 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:00Z" 2022-09-02T04:58:00.104 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:00.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:00Z" } 2022-09-02T04:58:00.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:00.106 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:00.107 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:00.107 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:05.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:05.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:05.110 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:05.166 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:05.166 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:05.211 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:05.213+0000 7fab41420600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:05.212 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:05.214+0000 7fab41420600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:05.225 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:05.225 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:05.226 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:05.226 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:05.226 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:05.226 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:05.227 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:05.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:05.227 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:05.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:05.228 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:05.228 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:05.228 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:05.229 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:05.229 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:05.229 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:05.229 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:05.230 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:05.230 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:05.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:05.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:05.231 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:05.231 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:05.231 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:05.231 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:05.232 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:05.232 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:05.232 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:05.233 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:05.233 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:05.234 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:05.235 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:05.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:05.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:05.235 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:05.236 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:05.236 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:05.236 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:05.236 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:05.237 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:05.237 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:05.238 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:05.238 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:05.238 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:05.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:05.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:05.240 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:05.240 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:05.241 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:05.241 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:05.242 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:05.242 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:05.243 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:05.243 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:05.243 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:05.243 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:05.244 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:05.244 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:05.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:05.245 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:05.245 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:05Z" 2022-09-02T04:58:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:05.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:05Z" } 2022-09-02T04:58:05.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:05.249 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:05.249 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:05.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:10.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:10.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:10.253 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:10.312 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:10.312 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:10.343 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:10.344+0000 7f62c6290600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:10.344 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:10.345+0000 7f62c6290600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:10.358 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:10.358 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:10.359 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:10.359 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:10.359 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:10.359 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:10.360 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:10.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:10.360 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:10.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:10.361 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:10.361 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:10.361 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:10.361 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:10.362 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:10.362 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:10.362 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:10.363 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:10.363 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:10.363 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:10.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:10.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:10.364 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:10.364 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:10.364 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:10.365 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:10.365 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:10.365 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:10.365 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:10.366 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:10.366 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:10.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:10.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:10.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:10.367 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:10.367 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:10.367 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:10.368 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:10.368 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:10.368 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:10.368 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:10.369 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:10.369 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:10.369 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:10.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:10.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:10.370 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:10.371 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:10.371 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:10.371 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:10.371 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:10.372 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:10.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:10.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:10.372 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:10.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:10.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:10.373 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:10.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:10.374 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:10.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:10.374 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:10.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:10.375 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:10Z" 2022-09-02T04:58:10.375 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:10.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:10Z" } 2022-09-02T04:58:10.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:10.377 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:10.378 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:10.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:15.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:15.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:15.381 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:15.438 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:15.439 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:15.480 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:15.480+0000 7f8fd7d71600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:15.481 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:15.481+0000 7f8fd7d71600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:15.504 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:15.505 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:15.506 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:15.507 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:15.508 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:15.509 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:15.510 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:15.511 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:15.512 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:15.513 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:15.514 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:15.515 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:15.516 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:15.517 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:15.518 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:15.519 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:15.520 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:15.521 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:15.522 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:15.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:15.524 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:15.525 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:15.526 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:15.526 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:15.527 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:15.527 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:15.528 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:15.528 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:15.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:15.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:15.529 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:15.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:15.530 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:15.530 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:15.530 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:15.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:15.531 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:15Z" 2022-09-02T04:58:15.531 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:15.532 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:15Z" } 2022-09-02T04:58:15.533 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:15.534 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:15.534 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:15.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:20.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:20.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:20.538 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:20.595 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:20.596 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:20.636 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:20.637+0000 7f68e7cef600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:20.637 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:20.638+0000 7f68e7cef600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:20.654 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:20.655 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:20.655 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:20.655 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:20.655 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:20.656 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:20.656 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:20.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:20.656 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:20.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:20.657 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:20.657 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:20.658 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:20.658 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:20.658 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:20.658 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:20.659 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:20.659 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:20.659 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:20.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:20.660 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:20.661 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:20.661 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:20.661 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:20.661 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:20.662 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:20.662 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:20.662 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:20.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:20.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:20.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:20.663 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:20.663 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:20.664 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:20.664 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:20.664 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:20.664 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:20.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:20.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:20.666 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:20.666 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:20.666 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:20.667 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:20.667 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:20.667 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:20.667 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:20.668 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:20.668 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:20.668 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:20.668 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:20.669 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:20.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:20.669 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:20.669 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:20.670 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:20.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:20.670 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:20Z" 2022-09-02T04:58:20.670 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:20.672 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:20Z" } 2022-09-02T04:58:20.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:20.673 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:20.673 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:20.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:25.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:25.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:25.677 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:25.735 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:25.735 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:25.767 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:25.768+0000 7f05d53d5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:25.768 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:25.769+0000 7f05d53d5600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:25.782 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:25.782 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:25.783 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:25.783 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:25.783 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:25.783 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:25.784 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:25.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:25.784 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:25.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:25.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:25.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:25.785 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:25.786 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:25.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:25.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:25.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:25.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:25.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:25.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:25.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:25.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:25.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:25.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:25.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:25.789 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:25.789 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:25.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:25.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:25.790 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:25.790 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:25.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:25.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:25.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:25.791 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:25.791 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:25.791 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:25.792 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:25.792 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:25.792 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:25.792 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:25.793 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:25.793 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:25.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:25.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:25.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:25.794 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:25.794 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:25.794 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:25.795 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:25.795 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:25.795 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:25.795 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:25.796 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:25.796 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:25.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:25.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:25.797 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:25.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:25.797 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:25.797 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:25.798 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:25.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:25.798 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:25Z" 2022-09-02T04:58:25.798 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:25.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:25Z" } 2022-09-02T04:58:25.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:25.801 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:25.801 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:25.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:30.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:30.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:30.804 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:30.863 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:30.863 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:30.894 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:30.894+0000 7f74fdeca600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:30.895 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:30.895+0000 7f74fdeca600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:30.911 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:30.911 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:30.912 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:30.912 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:30.912 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:30.913 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:30.913 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:30.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:30.914 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:30.915 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:30.916 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:30.917 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:30.918 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:30.919 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:30.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:30.921 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:30.921 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:30.921 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:30.921 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:30.922 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:30.922 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:30.922 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:30.922 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:30.923 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:30.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:30.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:30.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:30.924 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:30.924 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:30.924 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:30.924 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:30.925 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:30.925 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:30.925 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:30.925 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:30.926 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:30.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:30.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:30.926 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:30.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:30.927 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:30.927 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:30.927 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:30.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:30.928 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:30Z" 2022-09-02T04:58:30.928 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:30.929 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:30Z" } 2022-09-02T04:58:30.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:30.931 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:30.931 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:30.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:35.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:35.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:35.935 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:35.990 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:35.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:36.047 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:36.048+0000 7f77d80da600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:36.048 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:36.049+0000 7f77d80da600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:36.070 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:36.071 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:36.072 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:36.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:36.074 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:36.075 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:36.076 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:36.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:36.079 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:36.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:36.081 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:36.082 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:36.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:36.084 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:36.085 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:36.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:36.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:36.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:36.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:36.087 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:36.088 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:36.088 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:36.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:36.088 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:36Z" 2022-09-02T04:58:36.089 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:36.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:36Z" } 2022-09-02T04:58:36.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:36.091 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:36.092 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:36.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:41.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:41.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:41.095 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:41.152 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:41.153 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:41.193 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:41.194+0000 7f981f27d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:41.193 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:41.195+0000 7f981f27d600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:41.212 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:41.213 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:41.213 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:41.213 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:41.214 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:41.214 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:41.214 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:41.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:41.215 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:41.215 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:41.216 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:41.216 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:41.216 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:41.216 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:41.216 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:41.217 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:41.217 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:41.218 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:41.218 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:41.219 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:41.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:41.220 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:41.220 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:41.221 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:41.221 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:41.222 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:41.222 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:41.223 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:41.223 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:41.223 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:41.224 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:41.224 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:41.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:41.224 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:41.225 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:41.225 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:41.225 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:41.225 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:41.226 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:41.226 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:41.226 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:41.227 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:41.227 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:41.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:41.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:41.228 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:41.228 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:41.228 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:41.228 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:41.229 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:41.229 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:41.229 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:41.229 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:41.230 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:41.230 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:41.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:41.231 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:41.231 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:41.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:41.232 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:41.232 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:41.232 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:41.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:41.233 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:41Z" 2022-09-02T04:58:41.233 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:41.235 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:41Z" } 2022-09-02T04:58:41.235 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:41.236 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:41.237 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:41.238 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:46.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:46.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:46.241 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:46.298 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:46.298 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:46.353 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:46.353+0000 7fd8174cd600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:46.353 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:46.354+0000 7fd8174cd600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:46.376 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:46Z" 2022-09-02T04:58:46.405 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:46.407 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:46Z" } 2022-09-02T04:58:46.407 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:46.408 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:46.409 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:46.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:51.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:51.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:51.412 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:51.470 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:51.470 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:51.505 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:51.505+0000 7fb31c519600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:51.506 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:51.507+0000 7fb31c519600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:51.519 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:51.520 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:51.520 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:51.521 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:51.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:51.522 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:51.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:51.524 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:51.525 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:51.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:51.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:51.526 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:51.527 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:51.527 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:51.528 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:51.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:51.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:51.530 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:51.531 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:51.531 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:51.532 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:51.532 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:51.533 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:51.534 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:51.536 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:51.537 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:51.538 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:51.539 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:51.540 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:51Z" 2022-09-02T04:58:51.540 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:51.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:51Z" } 2022-09-02T04:58:51.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:51.544 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:51.545 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:51.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:58:56.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:56.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:58:56.549 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:58:56.607 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:58:56.607 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:58:56.663 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:56.664+0000 7ff7ae40a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:56.664 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:58:56.665+0000 7ff7ae40a600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:58:56.683 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:58:56.683 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:58:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:58:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:58:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:58:56.684 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:58:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:58:56.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:58:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:56.686 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:58:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:58:56.687 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:58:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:56.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:58:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:56.689 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:58:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:56.690 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:58:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:56.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:58:56.692 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094367.083839_156.1", 2022-09-02T04:58:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:56.693 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:52:47.083839Z", 2022-09-02T04:58:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:58:56.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:58:56.695 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:58:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:58:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:58:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:58:56.696 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:58:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:58:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:58:56.697 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:58:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:58:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:58:56.698 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:58:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:56.699 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:58:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:58:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:58:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:58:56.700 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:58:56Z" 2022-09-02T04:58:56.701 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:58:56.702 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094367.083839_156.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:52:47.083839Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:58:56Z" } 2022-09-02T04:58:56.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094367.083839_156.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:52:47.083839Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:58:56.703 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:58:56.704 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094367.083839_156.1', 3: ''} 2022-09-02T04:58:56.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094367.083839_156.1 2022-09-02T04:59:01.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:01.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:01.707 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:01.759 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:01.759 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:01.789 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:01.791+0000 7f6fcebe2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:01.790 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:01.792+0000 7f6fcebe2600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:01.803 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:01.803 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:01.804 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:01.804 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:01.804 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:01.804 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:01.805 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:01.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:01.805 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:01.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:01.806 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:01.806 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:01.806 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:01.807 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:01.807 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:01.807 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:01.807 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:01.808 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:01.808 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:01.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:01.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:01.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:01.809 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:01.809 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:01.809 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:01.809 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:01.810 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:01.810 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:01.810 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:01.811 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:01.811 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:01.811 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:01.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:01.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:01.812 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:01.812 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:01.812 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:01.813 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.663864_225.1", 2022-09-02T04:59:01.813 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:01.813 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:01.813 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:01.814 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.663864Z", 2022-09-02T04:59:01.814 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:01.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:01.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:01.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:01.815 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:01.815 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:01.816 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:01.816 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:01.816 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:01.816 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:01.817 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:01.817 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:01.817 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:01.817 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:01.818 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:01.818 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:01.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:01.819 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:01.819 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:01.819 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:01.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:01.820 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:01Z" 2022-09-02T04:59:01.820 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:01.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.663864_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.663864Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:01Z" } 2022-09-02T04:59:01.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.663864_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.663864Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:01.822 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:01.823 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.663864_225.1', 3: ''} 2022-09-02T04:59:01.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.663864_225.1 2022-09-02T04:59:06.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:06.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:06.826 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:06.876 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:06.877 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:06.910 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:06.912+0000 7f8f3225c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:06.911 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:06.913+0000 7f8f3225c600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:06.924 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:06.925 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:06.925 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:06.925 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:06.926 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:06.926 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:06.926 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:06.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:06.927 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:06.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:06.927 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:06.928 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:06.928 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:06.928 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:06.928 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:06.928 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:06.929 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:06.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:06.929 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:06.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:06.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:06.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:06.930 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:06.931 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:06.931 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:06.931 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:06.931 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:06.932 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:06.932 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:06.932 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:06.932 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:06.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:06.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:06.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:06.933 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:06.934 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:06.934 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:06.934 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.678035_227.1", 2022-09-02T04:59:06.934 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:06.935 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:06.935 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:06.935 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.678035Z", 2022-09-02T04:59:06.936 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:06.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:06.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:06.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:06.937 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:06.937 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:06.937 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:06.937 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:06.938 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:06.938 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:06.938 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:06.939 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:06.939 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:06.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:06.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:06.939 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:06.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:06.940 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:06.940 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:06.940 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:06.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:06.941 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:06Z" 2022-09-02T04:59:06.941 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:06.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.678035_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.678035Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:06Z" } 2022-09-02T04:59:06.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.678035_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.678035Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:06.944 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:06.945 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.678035_227.1', 3: ''} 2022-09-02T04:59:06.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.678035_227.1 2022-09-02T04:59:11.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:11.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:11.948 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:12.000 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:12.001 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:12.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:12.040+0000 7fb05dbd8600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:12.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:12.041+0000 7fb05dbd8600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:12.056 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:12.057 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:12.057 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:12.057 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:12.058 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:12.058 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:12.058 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:12.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:12.059 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:12.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:12.059 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:12.059 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:12.060 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:12.060 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:12.060 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:12.060 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:12.061 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:12.061 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:12.061 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:12.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:12.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:12.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:12.062 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:12.062 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:12.063 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:12.063 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:12.063 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:12.063 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:12.064 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:12.064 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:12.064 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:12.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:12.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:12.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:12.067 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:12.067 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:12.068 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:12.068 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.996186_212.1", 2022-09-02T04:59:12.069 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:12.069 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:12.070 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:12.071 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.996186Z", 2022-09-02T04:59:12.071 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:12.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:12.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:12.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:12.073 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:12.074 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:12.074 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:12.075 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:12.075 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:12.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:12.076 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:12.076 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:12.076 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:12.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:12.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:12.077 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:12.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:12.078 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:12.078 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:12.078 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:12.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:12.079 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:12Z" 2022-09-02T04:59:12.079 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:12.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.996186_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.996186Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:12Z" } 2022-09-02T04:59:12.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.996186_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.996186Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:12.082 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:12.082 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.996186_212.1', 3: ''} 2022-09-02T04:59:12.083 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.996186_212.1 2022-09-02T04:59:17.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:17.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:17.086 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:17.137 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:17.138 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:17.192 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:17.193+0000 7f055e909600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:17.192 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:17.193+0000 7f055e909600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:17.210 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:17.211 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:17.211 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:17.211 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:17.211 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:17.212 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:17.212 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:17.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:17.213 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:17.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:17.213 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:17.214 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:17.214 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:17.215 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:17.215 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:17.216 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:17.217 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:17.218 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:17.218 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:17.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:17.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:17.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:17.219 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:17.219 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:17.219 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:17.220 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:17.220 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:17.220 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:17.221 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:17.221 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:17.221 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:17.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:17.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:17.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:17.222 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:17.222 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:17.223 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:17.223 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.977875_209.1", 2022-09-02T04:59:17.223 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:17.223 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:17.224 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:17.224 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.977875Z", 2022-09-02T04:59:17.225 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:17.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:17.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:17.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:17.226 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:17.226 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:17.227 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:17.227 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:17.227 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:17.228 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:17.228 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:17.228 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:17.228 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:17.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:17.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:17.229 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:17.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:17.230 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:17.230 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:17.230 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:17.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:17.231 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:17Z" 2022-09-02T04:59:17.231 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:17.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.977875_209.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.977875Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:17Z" } 2022-09-02T04:59:17.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.977875_209.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.977875Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:17.234 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:17.234 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.977875_209.1', 3: ''} 2022-09-02T04:59:17.235 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.977875_209.1 2022-09-02T04:59:22.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:22.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:22.238 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:22.289 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:22.289 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:22.332 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:22.333+0000 7ff5ac003600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:22.333 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:22.334+0000 7ff5ac003600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:22.345 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:22.346 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:22.346 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:22.346 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:22.347 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:22.347 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:22.347 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:22.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:22.348 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:22.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:22.348 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:22.348 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:22.349 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:22.349 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:22.349 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:22.349 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:22.350 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:22.350 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:22.350 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:22.350 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:22.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:22.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:22.351 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:22.351 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:22.352 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:22.352 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:22.353 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:22.353 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:22.353 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:22.353 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:22.354 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:22.354 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:22.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:22.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:22.355 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:22.355 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:22.355 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:22.355 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.985540_210.1", 2022-09-02T04:59:22.356 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:22.356 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:22.356 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:22.356 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.985540Z", 2022-09-02T04:59:22.357 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:22.357 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:22.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:22.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:22.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:22.358 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:22.358 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:22.358 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:22.359 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:22.359 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:22.359 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:22.359 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:22.360 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:22.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:22.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:22.360 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:22.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:22.361 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:22.361 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:22.361 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:22.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:22.362 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:22Z" 2022-09-02T04:59:22.362 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:22.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.985540_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.985540Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:22Z" } 2022-09-02T04:59:22.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.985540_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.985540Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:22.365 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:22.365 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.985540_210.1', 3: ''} 2022-09-02T04:59:22.366 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.985540_210.1 2022-09-02T04:59:27.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:27.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:27.368 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:27.426 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:27.426 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:27.459 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:27.460+0000 7f6c1e215600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:27.460 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:27.461+0000 7f6c1e215600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:27.482 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:27.482 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:27.483 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:27.483 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:27.483 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:27.483 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:27.484 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:27.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:27.484 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:27.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:27.485 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:27.485 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:27.485 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:27.485 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:27.486 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:27.486 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:27.486 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:27.486 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:27.487 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:27.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:27.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:27.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:27.488 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:27.488 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:27.488 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:27.489 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:27.489 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:27.489 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:27.489 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:27.490 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:27.490 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:27.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:27.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:27.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:27.491 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:27.491 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:27.491 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:27.492 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.769306_229.1", 2022-09-02T04:59:27.492 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:27.492 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:27.492 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:27.493 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.769306Z", 2022-09-02T04:59:27.493 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:27.493 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:27.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:27.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:27.494 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:27.494 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:27.494 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:27.495 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:27.495 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:27.495 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:27.495 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:27.496 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:27.496 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:27.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:27.496 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:27.498 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:27.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:27.498 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:27Z" 2022-09-02T04:59:27.498 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:27.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.769306_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.769306Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:27Z" } 2022-09-02T04:59:27.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.769306_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.769306Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:27.501 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:27.502 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.769306_229.1', 3: ''} 2022-09-02T04:59:27.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.769306_229.1 2022-09-02T04:59:32.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:32.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:32.505 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:32.556 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:32.556 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:32.594 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:32.595+0000 7fa091e30600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:32.595 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:32.596+0000 7fa091e30600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:32.609 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:32.609 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:32.610 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:32.610 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:32.610 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:32.611 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:32.611 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:32.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:32.612 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:32.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:32.612 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:32.613 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:32.613 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:32.613 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:32.614 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:32.614 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:32.614 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:32.614 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:32.615 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:32.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:32.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:32.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:32.617 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:32.617 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:32.618 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:32.618 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:32.618 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:32.618 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:32.619 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:32.619 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:32.619 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:32.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:32.620 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.769306_229.1", 2022-09-02T04:59:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:32.621 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.769306Z", 2022-09-02T04:59:32.622 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:32.623 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:32.624 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:32.625 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:32.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:32.627 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:32.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:32.629 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:32Z" 2022-09-02T04:59:32.629 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:32.631 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.769306_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.769306Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:32Z" } 2022-09-02T04:59:32.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.769306_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.769306Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:32.633 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:32.634 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.769306_229.1', 3: ''} 2022-09-02T04:59:32.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.769306_229.1 2022-09-02T04:59:37.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:37.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:37.640 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:37.691 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:37.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:37.745 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:37.746+0000 7f459cd70600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:37.746 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:37.747+0000 7f459cd70600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:37.764 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.985540_210.1", 2022-09-02T04:59:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.985540Z", 2022-09-02T04:59:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:37Z" 2022-09-02T04:59:37.786 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:37.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.985540_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.985540Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:37Z" } 2022-09-02T04:59:37.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.985540_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.985540Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:37.789 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:37.789 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.985540_210.1', 3: ''} 2022-09-02T04:59:37.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.985540_210.1 2022-09-02T04:59:42.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:42.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:42.793 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:42.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:42.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:42.897 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:42.898+0000 7f807c4b0600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:42.898 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:42.899+0000 7f807c4b0600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:42.921 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:42.922 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:42.922 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:42.922 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:42.923 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:42.923 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:42.923 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:42.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:42.924 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:42.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:42.924 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:42.924 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:42.925 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:42.925 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:42.925 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:42.926 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:42.926 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:42.926 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:42.926 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:42.927 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:42.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:42.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:42.927 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:42.928 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:42.928 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:42.928 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:42.929 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:42.929 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:42.929 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:42.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:42.930 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:42.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:42.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:42.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:42.931 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:42.931 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:42.931 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:42.931 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.349825_222.1", 2022-09-02T04:59:42.932 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:42.932 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:42.932 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:42.933 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.349825Z", 2022-09-02T04:59:42.933 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:42.933 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:42.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:42.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:42.934 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:42.934 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:42.935 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:42.935 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:42.935 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:42.935 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:42.936 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:42.936 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:42.936 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:42.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:42.937 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:42.937 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:42.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:42.937 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:42.938 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:42.938 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:42.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:42.938 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:42Z" 2022-09-02T04:59:42.939 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:42.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.349825_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.349825Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:42Z" } 2022-09-02T04:59:42.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.349825_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.349825Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:42.941 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:42.942 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.349825_222.1', 3: ''} 2022-09-02T04:59:42.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.349825_222.1 2022-09-02T04:59:47.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:47.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:47.946 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:48.005 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:48.006 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:48.038 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:48.039+0000 7ffaee1a8600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:48.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:48.040+0000 7ffaee1a8600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:48.053 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.663864_225.1", 2022-09-02T04:59:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.663864Z", 2022-09-02T04:59:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:48Z" 2022-09-02T04:59:48.085 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:48.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.663864_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.663864Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:48Z" } 2022-09-02T04:59:48.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.663864_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.663864Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:48.088 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:48.088 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.663864_225.1', 3: ''} 2022-09-02T04:59:48.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.663864_225.1 2022-09-02T04:59:53.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:53.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:53.092 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:53.150 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:53.150 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:53.260 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:53.261+0000 7f9ad012b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:53.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:53.263+0000 7f9ad012b600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:53.281 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:53.282 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:53.282 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:53.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:53.285 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:53.286 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:53.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:53.288 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:53.289 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:53.290 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:53.291 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.673080_226.1", 2022-09-02T04:59:53.292 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.673080Z", 2022-09-02T04:59:53.293 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:53.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:53.295 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:53.295 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:53.295 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:53.295 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:53.296 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:53.296 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:53.296 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:53.296 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:53.297 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:53.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:53.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:53.297 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:53.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:53.298 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:53.298 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:53.299 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:53.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:53.299 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:53Z" 2022-09-02T04:59:53.300 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:53.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.673080_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.673080Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:53Z" } 2022-09-02T04:59:53.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.673080_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.673080Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:53.302 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:53.303 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.673080_226.1', 3: ''} 2022-09-02T04:59:53.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.673080_226.1 2022-09-02T04:59:58.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:58.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T04:59:58.306 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T04:59:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T04:59:58.358 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T04:59:58.386 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:58.387+0000 7f7e7a392600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:58.387 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T04:59:58.388+0000 7f7e7a392600 20 RGW-SYNC:meta: read sync status 2022-09-02T04:59:58.400 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T04:59:58.400 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T04:59:58.400 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T04:59:58.401 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T04:59:58.401 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T04:59:58.401 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T04:59:58.401 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:58.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:58.402 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T04:59:58.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:58.402 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T04:59:58.403 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:58.403 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:58.403 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T04:59:58.404 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:58.404 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T04:59:58.404 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:58.404 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T04:59:58.405 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:58.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:58.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:58.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:58.406 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T04:59:58.406 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:58.406 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:58.406 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T04:59:58.407 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:58.407 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:58.407 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:58.408 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T04:59:58.408 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:58.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:58.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:58.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:58.409 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T04:59:58.410 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:58.410 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:58.410 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.345339_221.1", 2022-09-02T04:59:58.411 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:58.411 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:58.411 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:58.411 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.345339Z", 2022-09-02T04:59:58.412 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T04:59:58.412 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:58.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:58.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T04:59:58.413 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T04:59:58.413 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T04:59:58.413 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T04:59:58.413 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T04:59:58.414 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T04:59:58.414 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T04:59:58.414 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T04:59:58.414 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T04:59:58.415 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T04:59:58.415 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:58.415 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T04:59:58.415 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T04:59:58.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:58.416 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T04:59:58.416 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T04:59:58.416 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T04:59:58.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T04:59:58.417 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T04:59:58Z" 2022-09-02T04:59:58.417 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T04:59:58.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.345339_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.345339Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T04:59:58Z" } 2022-09-02T04:59:58.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.345339_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.345339Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T04:59:58.420 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T04:59:58.420 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.345339_221.1', 3: ''} 2022-09-02T04:59:58.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.345339_221.1 2022-09-02T05:00:03.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:03.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:03.424 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:03.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:03.477 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:03.520 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:03.521+0000 7fab4878c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:03.521 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:03.522+0000 7fab4878c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:03.534 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:03.535 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:03.535 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:03.535 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:03.536 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:03.536 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:03.536 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:03.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:03.537 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:03.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:03.537 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:03.538 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:03.538 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:03.538 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:03.538 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:03.539 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:03.539 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:03.539 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:03.540 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:03.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:03.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:03.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:03.541 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:03.541 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:03.541 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:03.541 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:03.542 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:03.542 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:03.542 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:03.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:03.543 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:03.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:03.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:03.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:03.544 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:03.544 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:03.544 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:03.545 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.345339_221.1", 2022-09-02T05:00:03.545 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:03.545 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:03.545 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:03.546 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.345339Z", 2022-09-02T05:00:03.546 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:03.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:03.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:03.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:03.547 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:03.547 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:03.547 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:03.548 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:03.548 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:03.548 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:03.549 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:03.549 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:03.549 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:03.549 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:03.550 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:03.550 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:03.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:03.550 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:03.551 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:03.551 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:03.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:03.551 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:03Z" 2022-09-02T05:00:03.552 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:03.553 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.345339_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.345339Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:03Z" } 2022-09-02T05:00:03.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.345339_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.345339Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:03.554 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:03.555 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.345339_221.1', 3: ''} 2022-09-02T05:00:03.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.345339_221.1 2022-09-02T05:00:08.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:08.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:08.558 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:08.609 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:08.610 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:08.636 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:08.638+0000 7f6f2d13c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:08.637 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:08.639+0000 7f6f2d13c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:08.650 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:08.650 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:08.651 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:08.651 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:08.652 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:08.652 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:08.652 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:08.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:08.653 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:08.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:08.653 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:08.654 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:08.654 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:08.654 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:08.654 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:08.655 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:08.655 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:08.655 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:08.655 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:08.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:08.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:08.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:08.657 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:08.657 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:08.657 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:08.657 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:08.658 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:08.658 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:08.658 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:08.658 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:08.659 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:08.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:08.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:08.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:08.660 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:08.660 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:08.660 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:08.660 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.663864_225.1", 2022-09-02T05:00:08.661 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:08.661 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:08.661 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:08.662 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.663864Z", 2022-09-02T05:00:08.662 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:08.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:08.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:08.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:08.663 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:08.663 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:08.664 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:08.664 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:08.664 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:08.665 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:08.665 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:08.665 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:08.665 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:08.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:08.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:08.666 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:08.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:08.667 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:08.667 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:08.667 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:08.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:08.668 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:08Z" 2022-09-02T05:00:08.668 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:08.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.663864_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.663864Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:08Z" } 2022-09-02T05:00:08.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.663864_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.663864Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:08.671 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:08.671 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.663864_225.1', 3: ''} 2022-09-02T05:00:08.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.663864_225.1 2022-09-02T05:00:13.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:13.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:13.675 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:13.728 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:13.729 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:13.772 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:13.773+0000 7f3d96cbf600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:13.773 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:13.775+0000 7f3d96cbf600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:13.791 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:13.791 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:13.791 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:13.792 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:13.792 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:13.792 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:13.792 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:13.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:13.794 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:13.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:13.794 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:13.795 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:13.795 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:13.795 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:13.796 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:13.796 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:13.796 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:13.796 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:13.797 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:13.797 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:13.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:13.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:13.798 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:13.798 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:13.798 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:13.799 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:13.800 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:13.800 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:13.801 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:13.801 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:13.801 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:13.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:13.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:13.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:13.802 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.769306_229.1", 2022-09-02T05:00:13.803 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.769306Z", 2022-09-02T05:00:13.804 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:13.805 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:13.806 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:13.806 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:13.806 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:13.806 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:13.807 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:13.807 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:13.807 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:13.807 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:13.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:13.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:13.810 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:13Z" 2022-09-02T05:00:13.810 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:13.812 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.769306_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.769306Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:13Z" } 2022-09-02T05:00:13.812 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.769306_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.769306Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:13.813 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:13.813 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.769306_229.1', 3: ''} 2022-09-02T05:00:13.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.769306_229.1 2022-09-02T05:00:18.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:18.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:18.816 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:18.867 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:18.868 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:18.964 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:18.965+0000 7f4e89ee7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:18.966 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:18.966+0000 7f4e89ee7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:18.993 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:18.994 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:18.994 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:18.995 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:18.995 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:18.995 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:18.996 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:18.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:18.996 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:18.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:18.997 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:18.997 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:18.998 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:18.998 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:18.998 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:18.999 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:18.999 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:18.999 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:18.999 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:19.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:19.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:19.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:19.001 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:19.001 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:19.001 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:19.001 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:19.002 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:19.002 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:19.002 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:19.003 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:19.003 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:19.003 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:19.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:19.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:19.004 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:19.004 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:19.005 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:19.005 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.975057_241.1", 2022-09-02T05:00:19.005 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:19.005 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:19.006 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:19.006 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.975057Z", 2022-09-02T05:00:19.006 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:19.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:19.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:19.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:19.007 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:19.008 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:19.008 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:19.008 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:19.008 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:19.009 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:19.009 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:19.009 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:19.010 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:19.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:19.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:19.010 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:19.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:19.011 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:19.011 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:19.011 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:19.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:19.012 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:18Z" 2022-09-02T05:00:19.012 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:19.013 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.975057_241.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.975057Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:18Z" } 2022-09-02T05:00:19.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.975057_241.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.975057Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:19.015 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:19.015 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.975057_241.1', 3: ''} 2022-09-02T05:00:19.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.975057_241.1 2022-09-02T05:00:24.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:24.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:24.019 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:24.072 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:24.072 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:24.115 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:24.116+0000 7f93419d9600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:24.116 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:24.117+0000 7f93419d9600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:24.145 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:24.146 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:24.146 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:24.146 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:24.147 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:24.147 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:24.147 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:24.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:24.148 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:24.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:24.148 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:24.149 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:24.149 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:24.149 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:24.151 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:24.151 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:24.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:24.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:24.152 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:24.152 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:24.152 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:24.153 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:24.153 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:24.153 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:24.153 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:24.154 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:24.154 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:24.154 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:24.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:24.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:24.155 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:24.155 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:24.155 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:24.156 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.769306_229.1", 2022-09-02T05:00:24.156 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:24.156 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:24.157 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:24.157 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.769306Z", 2022-09-02T05:00:24.157 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:24.157 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:24.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:24.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:24.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:24.158 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:24.159 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:24.159 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:24.159 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:24Z" 2022-09-02T05:00:24.163 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:24.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.769306_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.769306Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:24Z" } 2022-09-02T05:00:24.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.769306_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.769306Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:24.166 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:24.168 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.769306_229.1', 3: ''} 2022-09-02T05:00:24.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.769306_229.1 2022-09-02T05:00:29.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:29.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:29.173 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:29.246 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:29.246 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:29.341 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:29.342+0000 7fedf0bd7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:29.342 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:29.343+0000 7fedf0bd7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:29.358 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:29.359 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:29.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:29.361 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:29.362 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:29.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:29.364 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:29.365 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:29.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:29.367 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.349825_222.1", 2022-09-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:29.368 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.349825Z", 2022-09-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:29.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:29.370 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:29.371 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:29.372 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:29.373 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:29.374 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:29Z" 2022-09-02T05:00:29.375 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:29.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.349825_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.349825Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:29Z" } 2022-09-02T05:00:29.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.349825_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.349825Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:29.377 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:29.378 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.349825_222.1', 3: ''} 2022-09-02T05:00:29.378 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.349825_222.1 2022-09-02T05:00:34.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:34.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:34.381 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:34.435 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:34.435 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:34.474 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:34.474+0000 7efdacc78600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:34.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:34.476+0000 7efdacc78600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:34.493 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.340302_219.1", 2022-09-02T05:00:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.340302Z", 2022-09-02T05:00:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:34Z" 2022-09-02T05:00:34.512 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:34.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.340302_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.340302Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:34Z" } 2022-09-02T05:00:34.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.340302_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.340302Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:34.515 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:34.516 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.340302_219.1', 3: ''} 2022-09-02T05:00:34.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.340302_219.1 2022-09-02T05:00:39.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:39.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:39.519 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:39.575 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:39.576 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:39.617 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:39.618+0000 7f90b9a4f600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:39.623 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:39.624+0000 7f90b9a4f600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:39.721 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:39.722 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:39.722 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:39.722 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:39.723 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:39.723 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:39.723 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:39.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:39.724 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:39.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:39.725 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:39.725 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:39.725 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:39.725 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:39.726 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:39.726 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:39.726 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:39.727 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:39.727 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:39.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:39.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:39.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:39.728 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:39.728 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:39.728 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:39.729 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:39.729 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:39.729 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:39.729 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:39.730 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:39.730 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:39.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:39.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:39.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:39.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:39.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:39.731 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:39.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.340302_219.1", 2022-09-02T05:00:39.732 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:39.732 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:39.732 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:39.733 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.340302Z", 2022-09-02T05:00:39.733 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:39.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:39.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:39.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:39.734 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:39.734 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:39.734 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:39.735 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:39.735 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:39.735 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:39.736 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:39.736 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:39.736 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:39.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:39.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:39.737 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:39.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:39.737 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:39.738 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:39.738 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:39.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:39.738 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:39Z" 2022-09-02T05:00:39.739 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:39.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.340302_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.340302Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:39Z" } 2022-09-02T05:00:39.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.340302_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.340302Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:39.741 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:39.742 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.340302_219.1', 3: ''} 2022-09-02T05:00:39.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.340302_219.1 2022-09-02T05:00:44.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:44.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:44.745 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:44.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:44.795 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:44.825 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:44.825+0000 7f1220ec9600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:44.827 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:44.827+0000 7f1220ec9600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:44.839 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:44.840 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:44.840 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:44.841 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:44.841 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:44.842 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:44.842 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:44.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:44.844 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:44.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:44.845 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:44.845 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:44.846 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:44.847 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:44.848 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:44.848 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:44.849 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:44.849 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:44.850 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:44.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:44.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:44.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:44.851 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:44.851 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:44.852 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:44.852 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:44.853 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:44.853 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:44.853 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:44.853 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:44.853 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:44.854 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:44.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:44.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:44.855 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:44.855 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:44.855 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:44.855 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.349825_222.1", 2022-09-02T05:00:44.855 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:44.856 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:44.856 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:44.856 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.349825Z", 2022-09-02T05:00:44.856 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:44.857 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:44.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:44.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:44.858 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:44.858 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:44.858 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:44.859 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:44.859 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:44.859 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:44.859 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:44.860 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:44.860 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:44.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:44.861 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:44.861 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:44.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:44.862 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:44.862 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:44.862 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:44.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:44.863 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:44Z" 2022-09-02T05:00:44.863 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:44.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.349825_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.349825Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:44Z" } 2022-09-02T05:00:44.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.349825_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.349825Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:44.866 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:44.867 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.349825_222.1', 3: ''} 2022-09-02T05:00:44.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.349825_222.1 2022-09-02T05:00:49.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:49.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:49.871 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:49.925 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:49.925 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:49.966 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:49.967+0000 7f5e25f3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:49.967 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:49.968+0000 7f5e25f3e600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:49.980 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:49.981 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:49.981 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:49.981 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:49.982 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:49.982 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:49.982 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:49.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:49.983 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:49.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:49.984 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:49.984 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:49.984 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:49.985 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:49.985 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:49.986 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:49.986 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:49.986 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:49.986 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:49.987 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:49.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:49.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:49.988 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:49.988 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:49.988 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:49.989 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:49.989 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:49.989 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:49.989 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:49.990 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:49.990 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:49.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:49.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:49.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:49.991 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:49.991 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:49.991 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:49.992 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.663864_225.1", 2022-09-02T05:00:49.992 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:49.992 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:49.992 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:49.993 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.663864Z", 2022-09-02T05:00:49.993 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:49.993 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:49.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:49.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:49.994 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:49.994 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:49.995 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:49.995 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:49.995 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:49.995 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:49.996 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:49.996 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:49.996 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:49.996 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:49.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:49.997 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:49.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:49.997 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:49.998 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:49.998 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:49.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:49.998 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:49Z" 2022-09-02T05:00:49.999 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:50.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.663864_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.663864Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:49Z" } 2022-09-02T05:00:50.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.663864_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.663864Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:50.001 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:50.002 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.663864_225.1', 3: ''} 2022-09-02T05:00:50.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.663864_225.1 2022-09-02T05:00:55.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:55.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:00:55.006 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:00:55.068 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:00:55.068 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:00:55.119 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:55.120+0000 7fab95d7d600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:55.121 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:00:55.122+0000 7fab95d7d600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:00:55.171 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:00:55.172 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:00:55.172 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:00:55.172 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:00:55.173 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:00:55.173 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:00:55.173 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:55.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:55.174 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:00:55.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:55.174 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:00:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:55.175 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:00:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:00:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:00:55.176 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:55.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:55.178 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:00:55.178 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:55.178 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:55.178 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:00:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:55.179 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:00:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:55.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:00:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:55.181 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.996186_212.1", 2022-09-02T05:00:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:55.182 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.996186Z", 2022-09-02T05:00:55.183 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:00:55.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:55.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:55.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:00:55.184 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:00:55.184 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:00:55.184 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:00:55.184 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:00:55.185 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:00:55.185 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:00:55.185 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:00:55.185 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:00:55.186 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:00:55.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:55.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:00:55.186 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:00:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:00:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:00:55.187 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:00:55.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:00:55.188 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:00:55Z" 2022-09-02T05:00:55.188 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:00:55.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.996186_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.996186Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:00:55Z" } 2022-09-02T05:00:55.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.996186_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.996186Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:00:55.191 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:00:55.191 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.996186_212.1', 3: ''} 2022-09-02T05:00:55.192 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.996186_212.1 2022-09-02T05:01:00.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:00.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:00.194 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:00.246 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:00.246 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:00.301 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:00.302+0000 7fe662e53600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:00.302 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:00.303+0000 7fe662e53600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:00.325 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:00.325 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:00.326 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:00.326 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:00.326 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:00.326 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:00.327 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:00.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:00.327 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:00.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:00.328 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:00.328 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:00.328 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:00.329 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:00.329 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:00.329 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:00.330 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:00.330 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:00.330 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:00.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:00.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:00.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:00.331 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:00.331 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:00.332 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:00.332 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:00.332 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:00.333 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:00.333 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:00.333 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:00.333 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:00.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:00.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:00.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:00.334 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:00.335 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:00.335 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:00.335 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.663864_225.1", 2022-09-02T05:01:00.335 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:00.336 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:00.336 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:00.336 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.663864Z", 2022-09-02T05:01:00.336 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:00.337 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:00.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:00.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:00.337 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:00.338 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:00.338 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:00.338 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:00.339 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:00.339 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:00.339 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:00.339 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:00.340 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:00.340 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:00.340 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:00.340 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:00.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:00.341 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:00.341 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:00.341 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:00.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:00.342 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:00Z" 2022-09-02T05:01:00.342 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:00.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.663864_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.663864Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:00Z" } 2022-09-02T05:01:00.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.663864_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.663864Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:00.345 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:00.345 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.663864_225.1', 3: ''} 2022-09-02T05:01:00.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.663864_225.1 2022-09-02T05:01:05.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:05.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:05.349 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:05.404 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:05.405 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:05.438 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:05.439+0000 7f6567caf600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:05.438 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:05.440+0000 7f6567caf600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:05.453 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:05.453 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:05.453 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:05.454 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:05.454 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:05.454 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:05.455 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:05.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:05.455 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:05.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:05.456 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:05.456 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:05.456 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:05.456 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:05.457 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:05.457 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:05.457 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:05.457 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:05.458 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:05.458 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:05.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:05.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:05.459 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:05.459 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:05.459 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:05.460 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:05.460 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:05.460 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:05.461 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:05.461 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:05.461 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:05.461 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:05.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:05.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:05.462 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:05.462 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:05.463 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:05.463 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.996186_212.1", 2022-09-02T05:01:05.463 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:05.464 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:05.464 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:05.464 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.996186Z", 2022-09-02T05:01:05.464 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:05.465 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:05.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:05.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:05.465 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:05.466 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:05.466 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:05.466 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:05.467 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:05.467 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:05.467 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:05.467 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:05.468 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:05.468 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:05.468 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:05.469 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:05.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:05.469 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:05.469 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:05.470 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:05.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:05.470 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:05Z" 2022-09-02T05:01:05.471 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:05.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.996186_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.996186Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:05Z" } 2022-09-02T05:01:05.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.996186_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.996186Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:05.473 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:05.474 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.996186_212.1', 3: ''} 2022-09-02T05:01:05.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.996186_212.1 2022-09-02T05:01:10.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:10.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:10.477 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:10.527 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:10.527 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:10.566+0000 7f0975ab6600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:10.567+0000 7f0975ab6600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:10.585 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:10.586 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:10.586 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:10.587 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:10.587 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:10.587 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:10.588 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:10.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:10.589 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:10.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:10.590 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:10.590 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:10.591 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:10.591 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:10.592 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:10.592 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:10.593 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:10.594 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:10.594 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:10.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:10.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:10.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:10.596 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:10.596 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:10.597 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:10.597 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:10.598 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:10.598 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:10.598 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:10.598 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:10.599 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:10.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:10.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:10.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:10.600 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:10.600 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:10.600 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:10.600 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.975057_241.1", 2022-09-02T05:01:10.601 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:10.601 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:10.601 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:10.602 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.975057Z", 2022-09-02T05:01:10.602 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:10.602 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:10.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:10.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:10.603 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:10.603 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:10.603 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:10.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:10.604 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:10.604 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:10.604 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:10.605 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:10.605 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:10.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:10.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:10.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:10.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:10.606 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:10.607 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:10.607 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:10.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:10.607 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:10Z" 2022-09-02T05:01:10.608 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:10.609 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.975057_241.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.975057Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:10Z" } 2022-09-02T05:01:10.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.975057_241.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.975057Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:10.610 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:10.611 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.975057_241.1', 3: ''} 2022-09-02T05:01:10.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.975057_241.1 2022-09-02T05:01:15.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:15.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:15.614 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:15.671 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:15.671 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:15.710 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:15.712+0000 7ff0f6533600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:15.713+0000 7ff0f6533600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:15.724 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:15.725 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:15.725 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:15.725 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:15.726 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:15.726 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:15.726 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:15.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.727 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:15.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.727 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:15.727 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.728 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.728 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:15.728 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.729 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:15.729 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.729 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:15.729 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:15.731 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:15.732 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.734 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094623.215257_258.1", 2022-09-02T05:01:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.735 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:03.215257Z", 2022-09-02T05:01:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.737 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:15.738 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.740 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:15.741 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.742 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:15Z" 2022-09-02T05:01:15.743 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:15.745 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094623.215257_258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:03.215257Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:15Z" } 2022-09-02T05:01:15.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094623.215257_258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:03.215257Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:15.747 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:15.747 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:15.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:01:15.749 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:01:15.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:15.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:15.751 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:15.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:15.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:15.848 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:01:15.850+0000 7fd6e642a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:01:15.851+0000 7fd6e642a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:15.862 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:15.862 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:15.863 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:15.863 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:15.863 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:15.863 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:15.864 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:01:15.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:15.864 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:15.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:01:15.865 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:01:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:15.866 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:01:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:15.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:15.868 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:01:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:01:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:15.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:15.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:01:15.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:01:15.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:15.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:15.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:15.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:15.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:15.871 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:01:15.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094623.215257_258.1", 2022-09-02T05:01:15.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:15.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:15.872 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:15.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:57:03.215257Z", 2022-09-02T05:01:15.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:01:15.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:15.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:15.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:15.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:15.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:15.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:01:15.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:15.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:15.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:15.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:15.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:15.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:01:15.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:15.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:15.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:15.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:15.877 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:15.877 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:01:15.878 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:01:15.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:15.878 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:15Z" 2022-09-02T05:01:15.878 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:15.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094623.215257_258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:03.215257Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:15Z" } 2022-09-02T05:01:15.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094623.215257_258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:03.215257Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:15.881 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:15.881 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:15.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:01:15.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:01:15.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:15.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:15.884 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:15.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:15.934 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:15.961 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:15.963+0000 7f2596f90600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:15.962 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:15.965+0000 7f2596f90600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:15.979 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:15.979 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:15.980 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:15.980 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:15.980 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:15.981 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:15.981 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:15.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.982 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:15.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.982 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:15.982 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.983 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.983 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:15.983 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.983 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:15.984 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.984 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:15.984 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:15.984 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.985 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:15.985 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.986 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.986 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:15.986 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.986 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:15.987 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.987 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:15.987 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:15.987 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.988 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:15.989 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.989 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.989 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.343887_220.1", 2022-09-02T05:01:15.989 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.989 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:15.990 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.990 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.343887Z", 2022-09-02T05:01:15.990 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:15.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:15.991 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:15.992 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:15.992 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:15.992 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:15.992 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:15.993 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:15.993 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:15.993 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:15.994 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:15.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:15.994 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:15.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.995 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:15.995 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:15.995 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:15.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:15.996 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:15Z" 2022-09-02T05:01:15.996 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:15.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.343887_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.343887Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:15Z" } 2022-09-02T05:01:15.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.343887_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.343887Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:15.999 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:15.999 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.343887_220.1', 3: ''} 2022-09-02T05:01:16.000 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.343887_220.1 2022-09-02T05:01:21.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:21.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:21.002 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:21.055 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:21.055 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:21.096 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:21.098+0000 7fb24f025600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:21.097 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:21.099+0000 7fb24f025600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:21.118 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:21.118 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:21.119 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:21.119 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:21.119 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:21.120 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:21.120 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:21.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:21.120 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:21.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:21.121 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:21.121 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:21.122 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:21.122 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:21.122 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:21.122 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:21.123 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:21.123 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:21.123 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:21.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:21.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:21.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:21.125 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:21.125 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:21.125 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:21.125 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:21.126 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:21.126 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:21.126 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:21.127 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:21.127 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:21.127 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:21.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:21.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:21.128 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:21.128 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:21.128 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:21.129 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094619.985540_210.1", 2022-09-02T05:01:21.129 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:21.129 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:21.129 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:21.130 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:56:59.985540Z", 2022-09-02T05:01:21.130 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:21.131 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:21.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:21.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:21.131 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:21.132 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:21.132 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:21.132 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:21.132 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:21.133 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:21.133 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:21.133 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:21.134 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:21.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:21.134 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:21.135 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:21.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:21.135 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:21.135 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:21.136 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:21.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:21.136 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:21Z" 2022-09-02T05:01:21.136 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:21.138 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094619.985540_210.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:56:59.985540Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:21Z" } 2022-09-02T05:01:21.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094619.985540_210.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:56:59.985540Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:21.139 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:21.140 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094619.985540_210.1', 3: ''} 2022-09-02T05:01:21.140 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094619.985540_210.1 2022-09-02T05:01:26.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:26.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:26.143 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:26.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:26.202 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:26.249 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:26.250+0000 7f5e6dd75600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:26.250 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:26.252+0000 7f5e6dd75600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:26.269 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:26.269 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:26.270 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:26.271 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:26.272 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:26.273 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:26.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:26.275 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:26.276 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:26.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:26.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:26.278 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:26.279 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:26.279 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:26.279 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094620.769306_229.1", 2022-09-02T05:01:26.279 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:00.769306Z", 2022-09-02T05:01:26.280 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:26.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:26.282 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:26.282 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:26.283 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:26.284 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:26.285 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:26.286 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:26.287 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:26Z" 2022-09-02T05:01:26.287 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:26.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094620.769306_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:00.769306Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:26Z" } 2022-09-02T05:01:26.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094620.769306_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:00.769306Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:26.290 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:26.290 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094620.769306_229.1', 3: ''} 2022-09-02T05:01:26.291 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662094623.215257_258.1 target=1_1662094620.769306_229.1 2022-09-02T05:01:31.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:31.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:31.293 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:31.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:31.352 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:31.381 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:31.382+0000 7fdc5588e600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:31.382 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:31.383+0000 7fdc5588e600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:31.395 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:31.395 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:31.395 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:31.395 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:31.396 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:31.396 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:31.396 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:31.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:31.397 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:31.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:31.397 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:31.398 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:31.398 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:31.398 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:31.399 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:31.399 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:31.399 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:31.399 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:31.400 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:31.400 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:31.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:31.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:31.401 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:31.401 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:31.401 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:31.401 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:31.402 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:31.402 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:31.402 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:31.403 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:31.403 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:31.403 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:31.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:31.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:31.404 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:31.404 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:31.404 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:31.405 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094623.215257_258.1", 2022-09-02T05:01:31.405 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:31.405 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:31.405 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:31.406 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:57:03.215257Z", 2022-09-02T05:01:31.406 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:31.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:31.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:31.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:31.407 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:31.407 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:31.408 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:31.408 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:31.408 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:31.409 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:31.409 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:31.409 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:31.410 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:31.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:31.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:31.410 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:31.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:31.411 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:31.412 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:31.412 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:31.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:31.413 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:31Z" 2022-09-02T05:01:31.414 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:31.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094623.215257_258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:57:03.215257Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:31Z" } 2022-09-02T05:01:31.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094623.215257_258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:57:03.215257Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:31.418 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:31.419 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094623.215257_258.1', 3: ''} 2022-09-02T05:01:31.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:01:31.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xljbkf-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:01:31.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xljbkf-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:01:31.422 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xljbkf-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:01:31.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:31.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:31.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xljbkf-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:01:31.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xljbkf-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:01:31.580 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xljbkf-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:01:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:31.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xljbkf-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:01:31.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xljbkf-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:01:31.720 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xljbkf-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:01:31.771 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:31.772 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:31.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:31.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:31.858 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:31.905 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:31.905 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:31.971 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:31.972 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:31.972 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000078", 2022-09-02T05:01:31.972 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.history", 2022-09-02T05:01:31.972 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:31.973 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:31.973 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.23", 2022-09-02T05:01:31.973 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:31.973 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000070", 2022-09-02T05:01:31.974 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:31.974 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:31.974 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000104", 2022-09-02T05:01:31.974 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:31.975 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000026", 2022-09-02T05:01:31.975 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:31.975 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:31.976 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:31.976 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:31.976 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:31.976 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:31.977 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.8", 2022-09-02T05:01:31.977 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:31.977 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000028", 2022-09-02T05:01:31.977 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:31.978 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:31.978 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:31.979 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:31.979 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:31.979 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:31.979 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:31.980 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:31.980 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:31.980 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:31.980 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:31.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000040", 2022-09-02T05:01:31.981 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:31.981 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:31.981 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:31.982 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:31.982 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.6", 2022-09-02T05:01:31.982 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:31.983 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:31.983 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:31.983 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:31.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000015", 2022-09-02T05:01:31.984 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:31.984 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:31.984 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:31.984 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000069", 2022-09-02T05:01:31.985 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000095", 2022-09-02T05:01:31.985 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000003", 2022-09-02T05:01:31.985 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:31.985 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000047", 2022-09-02T05:01:31.986 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:31.986 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:31.986 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:31.986 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:31.987 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:31.987 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:31.987 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:31.988 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:31.988 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:31.988 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:31.989 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:31.989 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:31.989 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:31.989 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:31.990 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:31.990 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1.5fUk9mgrkjz.0", 2022-09-02T05:01:31.990 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000035", 2022-09-02T05:01:31.990 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:31.991 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:31.991 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000037", 2022-09-02T05:01:31.991 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:31.991 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:31.992 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:31.992 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:31.992 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:31.992 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:31.993 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:31.993 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:31.993 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:31.993 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:31.994 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000024", 2022-09-02T05:01:31.994 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:31.994 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:31.995 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000000", 2022-09-02T05:01:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000031", 2022-09-02T05:01:31.996 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:31.997 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.0.retry", 2022-09-02T05:01:31.997 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:31.997 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:31.997 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:31.998 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3", 2022-09-02T05:01:31.998 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:31.998 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:31.999 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:31.999 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:31.999 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:31.999 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:32.000 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:32.000 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:32.000 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:32.000 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:32.001 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.001 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.001 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.001 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:32.002 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:32.002 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:32.002 INFO:teuthology.orchestra.run.smithi099.stdout: "data_loggenerations_metadata", 2022-09-02T05:01:32.002 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:32.003 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:32.003 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:32.003 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.003 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:32.004 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:32.004 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.1.retry", 2022-09-02T05:01:32.004 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:32.004 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000076", 2022-09-02T05:01:32.005 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.005 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000113", 2022-09-02T05:01:32.005 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000005", 2022-09-02T05:01:32.005 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.006 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000011", 2022-09-02T05:01:32.006 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000058", 2022-09-02T05:01:32.007 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000052", 2022-09-02T05:01:32.007 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:32.007 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.007 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:32.008 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.21", 2022-09-02T05:01:32.008 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.008 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:32.008 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:32.009 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.009 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:32.009 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:32.009 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.010 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:32.010 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:32.010 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:32.010 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:32.011 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:32.011 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.011 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:32.011 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:32.012 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000088", 2022-09-02T05:01:32.012 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:32.012 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000080", 2022-09-02T05:01:32.012 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.013 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000090", 2022-09-02T05:01:32.013 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:32.013 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.013 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:32.014 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000110", 2022-09-02T05:01:32.014 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:32.014 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:32.015 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:32.015 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.015 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.015 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.016 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:32.016 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:32.016 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:32.017 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000096", 2022-09-02T05:01:32.017 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:32.017 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:32.018 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:32.018 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.3", 2022-09-02T05:01:32.018 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:32.018 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000087", 2022-09-02T05:01:32.019 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:32.019 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000008", 2022-09-02T05:01:32.019 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:32.019 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:32.020 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10", 2022-09-02T05:01:32.020 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000006", 2022-09-02T05:01:32.020 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000029", 2022-09-02T05:01:32.020 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:32.021 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000089", 2022-09-02T05:01:32.021 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.021 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:32.022 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000022", 2022-09-02T05:01:32.022 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.022 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.022 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:32.023 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:32.023 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:32.023 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.023 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog.trim", 2022-09-02T05:01:32.024 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:32.024 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.024 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000016", 2022-09-02T05:01:32.025 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:32.025 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:32.025 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:32.025 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:32.026 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:32.026 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:32.026 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000041", 2022-09-02T05:01:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:32.027 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7", 2022-09-02T05:01:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000018", 2022-09-02T05:01:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000092", 2022-09-02T05:01:32.028 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.4", 2022-09-02T05:01:32.029 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:32.030 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.2.retry", 2022-09-02T05:01:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:32.031 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000014", 2022-09-02T05:01:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.032 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:32.033 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.033 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:32.033 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000112", 2022-09-02T05:01:32.033 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:32.034 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000007", 2022-09-02T05:01:32.034 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000021", 2022-09-02T05:01:32.034 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.16", 2022-09-02T05:01:32.035 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:32.035 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000064", 2022-09-02T05:01:32.036 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:32.036 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:32.036 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000071", 2022-09-02T05:01:32.036 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:32.037 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:32.037 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:32.037 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000074", 2022-09-02T05:01:32.037 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.038 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.038 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:32.038 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:32.038 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.039 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.039 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:32.039 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.039 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:32.040 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000081", 2022-09-02T05:01:32.041 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:32.041 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000009", 2022-09-02T05:01:32.041 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000121", 2022-09-02T05:01:32.041 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1", 2022-09-02T05:01:32.042 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:32.042 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:32.042 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:32.043 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.043 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.17", 2022-09-02T05:01:32.043 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.043 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:32.044 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.044 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000125", 2022-09-02T05:01:32.044 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:32.044 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.045 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000082", 2022-09-02T05:01:32.045 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.046 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.046 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:32.046 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000105", 2022-09-02T05:01:32.046 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000059", 2022-09-02T05:01:32.047 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:32.047 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:32.047 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:32.047 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:32.048 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.048 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.048 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000077", 2022-09-02T05:01:32.048 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:32.049 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000032", 2022-09-02T05:01:32.049 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:32.049 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:32.049 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:32.050 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.1", 2022-09-02T05:01:32.050 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:32.050 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:32.051 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0.VffrRyl4br9.0", 2022-09-02T05:01:32.051 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T05:01:32.051 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:32.051 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:32.052 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.052 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:32.052 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.052 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T05:01:32.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:32.053 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.2", 2022-09-02T05:01:32.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.053 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000053", 2022-09-02T05:01:32.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:32.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:32.054 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000091", 2022-09-02T05:01:32.055 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:32.055 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:32.055 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:32.055 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000065", 2022-09-02T05:01:32.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:32.056 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:32.056 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000083", 2022-09-02T05:01:32.056 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000010", 2022-09-02T05:01:32.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:32.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:32.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:32.057 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:32.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.058 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:32.059 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:32.059 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:32.059 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:32.059 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.7", 2022-09-02T05:01:32.060 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.3", 2022-09-02T05:01:32.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:32.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:32.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:32.061 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:32.061 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:32.061 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.0", 2022-09-02T05:01:32.061 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:32.062 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:32.062 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.062 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:32.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:32.063 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.2", 2022-09-02T05:01:32.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:32.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7", 2022-09-02T05:01:32.064 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000045", 2022-09-02T05:01:32.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:32.064 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:32.064 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000002", 2022-09-02T05:01:32.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:32.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.065 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.12", 2022-09-02T05:01:32.065 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:32.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.066 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000116", 2022-09-02T05:01:32.066 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:32.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:32.067 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:32.068 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000034", 2022-09-02T05:01:32.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:32.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:32.068 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:32.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:32.069 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000101", 2022-09-02T05:01:32.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:32.069 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:32.070 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.13", 2022-09-02T05:01:32.070 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.1", 2022-09-02T05:01:32.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.070 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:32.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:32.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:32.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:32.071 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:32.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:32.072 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000079", 2022-09-02T05:01:32.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.072 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000049", 2022-09-02T05:01:32.073 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000117", 2022-09-02T05:01:32.073 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000044", 2022-09-02T05:01:32.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:32.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.074 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:32.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:32.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:32.075 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.14", 2022-09-02T05:01:32.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.076 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000066", 2022-09-02T05:01:32.076 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.0.retry", 2022-09-02T05:01:32.076 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:32.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:32.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:32.077 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:32.077 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000068", 2022-09-02T05:01:32.078 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000085", 2022-09-02T05:01:32.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.078 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:32.079 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.1", 2022-09-02T05:01:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.080 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000073", 2022-09-02T05:01:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000038", 2022-09-02T05:01:32.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.083 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:32.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:32.084 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.10", 2022-09-02T05:01:32.084 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:32.085 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000118", 2022-09-02T05:01:32.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:32.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:32.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:32.086 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000036", 2022-09-02T05:01:32.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:32.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:32.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:32.087 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.088 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000103", 2022-09-02T05:01:32.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:32.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:32.088 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:32.089 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.3", 2022-09-02T05:01:32.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:32.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:32.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:32.090 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000119", 2022-09-02T05:01:32.091 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.091 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000098", 2022-09-02T05:01:32.091 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:32.091 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:32.092 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:32.092 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:32.092 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:32.093 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.0", 2022-09-02T05:01:32.093 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.093 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.093 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:32.094 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:32.094 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:32.094 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:32.095 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000027", 2022-09-02T05:01:32.095 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.1.retry", 2022-09-02T05:01:32.095 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.095 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:32.096 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status", 2022-09-02T05:01:32.096 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000019", 2022-09-02T05:01:32.096 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:32.096 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:32.097 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:32.097 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000039", 2022-09-02T05:01:32.097 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:32.097 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:32.098 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:32.098 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.098 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:32.098 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:32.099 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:32.099 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.099 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:32.100 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000100", 2022-09-02T05:01:32.100 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.100 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:32.100 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:32.101 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.101 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.101 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:32.102 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:32.102 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.102 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:32.102 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:32.103 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:32.103 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.103 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:32.103 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000093", 2022-09-02T05:01:32.104 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:32.104 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:32.104 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:32.104 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:32.105 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000004", 2022-09-02T05:01:32.105 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000063", 2022-09-02T05:01:32.105 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.106 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000122", 2022-09-02T05:01:32.106 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000057", 2022-09-02T05:01:32.106 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000054", 2022-09-02T05:01:32.107 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000114", 2022-09-02T05:01:32.107 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:32.107 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:32.107 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:32.108 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.2", 2022-09-02T05:01:32.108 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.108 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0", 2022-09-02T05:01:32.108 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.109 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:32.109 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:32.109 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.109 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.110 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.110 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:32.110 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000012", 2022-09-02T05:01:32.110 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:32.111 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:32.111 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.112 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.112 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000084", 2022-09-02T05:01:32.112 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000043", 2022-09-02T05:01:32.112 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:32.113 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:32.113 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:32.113 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:32.113 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.114 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.11", 2022-09-02T05:01:32.114 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:32.114 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:32.114 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.115 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:32.115 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:32.115 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:32.115 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:32.116 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.116 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000111", 2022-09-02T05:01:32.116 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000048", 2022-09-02T05:01:32.116 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:32.117 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:32.117 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:32.117 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000020", 2022-09-02T05:01:32.117 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.118 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.118 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000099", 2022-09-02T05:01:32.118 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000056", 2022-09-02T05:01:32.118 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.119 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:32.119 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.119 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:32.119 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.19", 2022-09-02T05:01:32.120 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:32.120 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:32.120 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:32.121 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:32.121 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000072", 2022-09-02T05:01:32.121 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:32.121 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:32.122 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:32.122 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:32.122 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000062", 2022-09-02T05:01:32.123 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000115", 2022-09-02T05:01:32.123 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.123 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:32.123 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:32.124 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:32.124 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.1", 2022-09-02T05:01:32.124 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:32.124 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000033", 2022-09-02T05:01:32.125 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:32.125 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:32.125 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.125 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:32.126 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:32.126 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.126 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.126 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:32.127 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:32.127 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000025", 2022-09-02T05:01:32.127 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:32.127 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000123", 2022-09-02T05:01:32.128 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.128 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:32.128 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:32.128 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.18", 2022-09-02T05:01:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.3", 2022-09-02T05:01:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000108", 2022-09-02T05:01:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000094", 2022-09-02T05:01:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000060", 2022-09-02T05:01:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000109", 2022-09-02T05:01:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.15", 2022-09-02T05:01:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000013", 2022-09-02T05:01:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000051", 2022-09-02T05:01:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000106", 2022-09-02T05:01:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000017", 2022-09-02T05:01:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.2", 2022-09-02T05:01:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000023", 2022-09-02T05:01:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000107", 2022-09-02T05:01:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000046", 2022-09-02T05:01:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.22", 2022-09-02T05:01:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2", 2022-09-02T05:01:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000067", 2022-09-02T05:01:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.b798a9cd-aa0e-480c-bb07-05f02abf3f41.2", 2022-09-02T05:01:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.9", 2022-09-02T05:01:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000086", 2022-09-02T05:01:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.5", 2022-09-02T05:01:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:32.156 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:32.156 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:32.156 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:32.156 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:32.157 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:32.157 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:32.157 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:32.157 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.20", 2022-09-02T05:01:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.0", 2022-09-02T05:01:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000050", 2022-09-02T05:01:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000102", 2022-09-02T05:01:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000124", 2022-09-02T05:01:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3.wcVxYLWEpQZ.0", 2022-09-02T05:01:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000001", 2022-09-02T05:01:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000126", 2022-09-02T05:01:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000055", 2022-09-02T05:01:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000061", 2022-09-02T05:01:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000097", 2022-09-02T05:01:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.2.retry", 2022-09-02T05:01:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000042", 2022-09-02T05:01:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000075", 2022-09-02T05:01:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000120", 2022-09-02T05:01:32.175 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000030", 2022-09-02T05:01:32.175 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.175 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2.WGZfK1CA7KC.0", 2022-09-02T05:01:32.175 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:32.176 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10" 2022-09-02T05:01:32.176 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:32.176 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T05:01:32.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:32.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:32.179 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:32.234 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:32.235 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:32.314 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T05:01:32.315 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:32.315 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000078", 2022-09-02T05:01:32.315 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.history", 2022-09-02T05:01:32.315 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.316 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.316 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:32.316 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:32.317 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000070", 2022-09-02T05:01:32.317 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.317 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.317 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:32.318 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000104", 2022-09-02T05:01:32.318 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:32.318 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000026", 2022-09-02T05:01:32.319 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:32.319 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:32.319 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.319 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:32.320 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:32.320 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.8", 2022-09-02T05:01:32.320 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:32.320 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000028", 2022-09-02T05:01:32.321 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:32.321 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.321 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.321 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:32.322 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:32.322 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:32.322 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.322 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:32.323 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:32.323 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.323 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.323 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:32.324 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:32.324 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000040", 2022-09-02T05:01:32.324 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:32.324 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.325 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.6", 2022-09-02T05:01:32.325 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.325 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:32.325 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:32.326 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000015", 2022-09-02T05:01:32.326 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:32.326 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:32.327 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000069", 2022-09-02T05:01:32.327 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000095", 2022-09-02T05:01:32.327 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000003", 2022-09-02T05:01:32.327 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000047", 2022-09-02T05:01:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:32.330 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:32.330 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:32.330 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T05:01:32.330 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:32.331 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000035", 2022-09-02T05:01:32.331 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:32.331 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000037", 2022-09-02T05:01:32.331 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.332 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.332 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.332 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:32.332 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.333 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:32.333 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000024", 2022-09-02T05:01:32.333 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:32.333 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:32.334 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:32.334 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000000", 2022-09-02T05:01:32.334 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.335 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000031", 2022-09-02T05:01:32.335 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:32.335 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.335 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.336 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:32.336 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:32.336 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.3", 2022-09-02T05:01:32.336 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:32.337 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:32.337 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:32.337 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:32.338 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.338 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:32.338 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:32.338 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:32.339 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.339 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.339 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.339 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:32.340 INFO:teuthology.orchestra.run.smithi160.stdout: "data_loggenerations_metadata", 2022-09-02T05:01:32.340 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:32.340 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.340 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.341 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:32.341 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:32.341 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.1.retry", 2022-09-02T05:01:32.341 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000076", 2022-09-02T05:01:32.342 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.342 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000113", 2022-09-02T05:01:32.342 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:32.343 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000005", 2022-09-02T05:01:32.343 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000011", 2022-09-02T05:01:32.343 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000058", 2022-09-02T05:01:32.343 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000052", 2022-09-02T05:01:32.344 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.344 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:32.344 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.1", 2022-09-02T05:01:32.344 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.345 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:32.345 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:32.345 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:32.345 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:32.346 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:32.346 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:32.346 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.346 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.347 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000088", 2022-09-02T05:01:32.347 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:32.347 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000080", 2022-09-02T05:01:32.347 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.348 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000090", 2022-09-02T05:01:32.348 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:32.348 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.348 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:32.349 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000110", 2022-09-02T05:01:32.349 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.349 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:32.349 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.350 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.350 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.350 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.351 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.351 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:32.351 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:32.352 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000096", 2022-09-02T05:01:32.352 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:32.352 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:32.352 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:32.353 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:32.353 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:32.353 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000087", 2022-09-02T05:01:32.353 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.354 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000008", 2022-09-02T05:01:32.354 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:32.354 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.354 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.355 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.355 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.355 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:32.356 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10", 2022-09-02T05:01:32.356 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.356 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.356 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000006", 2022-09-02T05:01:32.357 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000029", 2022-09-02T05:01:32.357 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.357 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:32.357 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:32.358 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:32.358 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:32.358 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000089", 2022-09-02T05:01:32.358 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.359 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.359 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:32.359 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000022", 2022-09-02T05:01:32.359 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.360 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.360 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.360 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:32.360 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.361 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:32.361 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.361 INFO:teuthology.orchestra.run.smithi160.stdout: "bilog.trim", 2022-09-02T05:01:32.361 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.362 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000016", 2022-09-02T05:01:32.362 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:32.362 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:32.363 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000041", 2022-09-02T05:01:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.364 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7", 2022-09-02T05:01:32.365 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000018", 2022-09-02T05:01:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000092", 2022-09-02T05:01:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:32.366 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.367 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:32.367 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.4", 2022-09-02T05:01:32.367 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.367 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:32.368 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:32.368 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:32.368 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.368 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.369 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.369 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.369 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.369 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000014", 2022-09-02T05:01:32.370 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.370 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:32.370 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:32.370 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:32.371 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.371 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000112", 2022-09-02T05:01:32.371 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:32.372 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.372 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000007", 2022-09-02T05:01:32.372 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000021", 2022-09-02T05:01:32.372 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.373 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.373 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.16", 2022-09-02T05:01:32.373 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.3.retry", 2022-09-02T05:01:32.374 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.374 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000064", 2022-09-02T05:01:32.374 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:32.374 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:32.375 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:32.375 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:32.375 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000071", 2022-09-02T05:01:32.375 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:32.376 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.2", 2022-09-02T05:01:32.376 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.376 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:32.376 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000074", 2022-09-02T05:01:32.377 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.377 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.377 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:32.377 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:32.378 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:32.378 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.384 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.384 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000081", 2022-09-02T05:01:32.385 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:32.385 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000009", 2022-09-02T05:01:32.385 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:32.386 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000121", 2022-09-02T05:01:32.386 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10", 2022-09-02T05:01:32.386 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.1", 2022-09-02T05:01:32.386 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:32.387 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.387 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:32.387 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.388 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.388 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.388 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.388 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.17", 2022-09-02T05:01:32.389 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.389 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000125", 2022-09-02T05:01:32.389 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.390 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000082", 2022-09-02T05:01:32.390 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.390 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:32.390 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.391 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:32.391 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000105", 2022-09-02T05:01:32.391 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000059", 2022-09-02T05:01:32.392 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:32.392 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:32.392 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:32.392 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:32.393 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:32.393 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:32.393 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.393 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.394 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000077", 2022-09-02T05:01:32.394 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:32.394 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000032", 2022-09-02T05:01:32.395 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:32.395 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:32.395 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:32.395 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26", 2022-09-02T05:01:32.396 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:32.396 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:32.396 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.396 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:32.397 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.397 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.3./90tWz13lrT.0", 2022-09-02T05:01:32.397 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:32.398 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:32.398 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.398 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000053", 2022-09-02T05:01:32.398 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:32.399 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000091", 2022-09-02T05:01:32.399 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:32.399 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:32.399 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:32.400 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000065", 2022-09-02T05:01:32.400 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:32.400 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000083", 2022-09-02T05:01:32.400 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000010", 2022-09-02T05:01:32.401 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:32.401 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:32.401 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:32.401 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.402 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:32.402 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.402 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:32.402 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:32.403 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:32.403 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.7", 2022-09-02T05:01:32.403 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:32.403 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.3", 2022-09-02T05:01:32.404 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:32.404 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:32.404 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:32.404 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:32.405 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:32.405 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:32.405 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:32.405 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:32.406 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:32.406 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.406 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:32.407 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:32.407 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:32.407 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:32.407 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.2", 2022-09-02T05:01:32.408 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.3.retry", 2022-09-02T05:01:32.408 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:32.408 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:32.408 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:32.409 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000045", 2022-09-02T05:01:32.409 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:32.409 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000002", 2022-09-02T05:01:32.409 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:32.410 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:32.410 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.410 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.12", 2022-09-02T05:01:32.411 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:32.411 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:32.411 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.411 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:32.412 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000116", 2022-09-02T05:01:32.412 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:32.412 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:32.412 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:32.413 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:32.413 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:32.413 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:32.413 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000034", 2022-09-02T05:01:32.414 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:32.414 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:32.414 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:32.415 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000101", 2022-09-02T05:01:32.415 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:32.415 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:32.415 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:32.416 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.13", 2022-09-02T05:01:32.416 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.1", 2022-09-02T05:01:32.416 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.416 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:32.417 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:32.417 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:32.417 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:32.417 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:32.418 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:32.418 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000079", 2022-09-02T05:01:32.418 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000049", 2022-09-02T05:01:32.418 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000117", 2022-09-02T05:01:32.419 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000044", 2022-09-02T05:01:32.419 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:32.419 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:32.419 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:32.420 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:32.420 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.14", 2022-09-02T05:01:32.420 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.421 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000066", 2022-09-02T05:01:32.421 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.0.retry", 2022-09-02T05:01:32.421 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:32.421 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:32.422 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:32.422 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:32.422 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000068", 2022-09-02T05:01:32.422 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000085", 2022-09-02T05:01:32.423 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:32.423 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.423 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:32.423 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.1", 2022-09-02T05:01:32.424 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000073", 2022-09-02T05:01:32.424 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.424 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:32.425 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000038", 2022-09-02T05:01:32.425 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.425 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:32.426 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.426 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:32.426 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:32.426 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.10", 2022-09-02T05:01:32.427 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:32.427 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:32.427 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000118", 2022-09-02T05:01:32.427 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:32.428 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.428 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:32.428 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:32.429 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:32.429 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:32.429 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:32.429 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000036", 2022-09-02T05:01:32.430 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:32.430 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:32.430 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:32.431 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.431 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:32.431 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000103", 2022-09-02T05:01:32.431 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:32.432 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:32.432 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:32.432 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:32.432 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.3", 2022-09-02T05:01:32.433 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.433 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:32.433 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000119", 2022-09-02T05:01:32.433 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.434 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000098", 2022-09-02T05:01:32.434 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:32.434 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:32.435 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:32.435 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:32.435 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:32.435 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:32.436 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.0", 2022-09-02T05:01:32.436 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.436 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.2.XXKeFfGipcz.0", 2022-09-02T05:01:32.436 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:32.437 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:32.437 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:32.437 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:32.437 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000027", 2022-09-02T05:01:32.438 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.438 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status", 2022-09-02T05:01:32.438 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000019", 2022-09-02T05:01:32.438 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:32.439 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:32.439 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:32.439 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:32.439 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000039", 2022-09-02T05:01:32.440 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:32.440 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:32.440 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:32.440 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:32.441 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:32.441 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:32.441 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:32.441 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:32.442 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:32.442 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:32.442 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:32.442 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.443 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:32.443 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000100", 2022-09-02T05:01:32.443 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:32.443 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.444 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:32.444 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:32.444 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:32.445 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:32.445 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.445 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:32.445 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:32.446 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:32.446 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:32.446 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:32.446 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.447 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000093", 2022-09-02T05:01:32.447 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:32.447 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:32.448 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:32.448 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:32.448 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:32.449 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:32.449 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000004", 2022-09-02T05:01:32.449 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000063", 2022-09-02T05:01:32.449 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000122", 2022-09-02T05:01:32.450 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:32.450 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000057", 2022-09-02T05:01:32.450 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:32.451 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:32.451 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000054", 2022-09-02T05:01:32.451 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000114", 2022-09-02T05:01:32.451 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:32.452 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:32.452 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:32.452 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:32.452 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.2", 2022-09-02T05:01:32.453 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.453 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0", 2022-09-02T05:01:32.453 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:32.453 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:32.454 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:32.454 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:32.454 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:32.454 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:32.455 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:32.455 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.455 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:32.455 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000012", 2022-09-02T05:01:32.456 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:32.456 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000084", 2022-09-02T05:01:32.456 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000043", 2022-09-02T05:01:32.457 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:32.457 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.1.W2sfBssElkR.0", 2022-09-02T05:01:32.457 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:32.457 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:32.458 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:32.458 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:32.458 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:32.458 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:32.459 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.11", 2022-09-02T05:01:32.459 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:32.459 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:32.459 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:32.460 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.460 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:32.460 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:32.460 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.461 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000111", 2022-09-02T05:01:32.461 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:32.461 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000048", 2022-09-02T05:01:32.461 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:32.462 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:32.462 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:32.462 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:32.463 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000020", 2022-09-02T05:01:32.463 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.463 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:32.463 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:32.464 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000099", 2022-09-02T05:01:32.464 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000056", 2022-09-02T05:01:32.464 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:32.464 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:32.465 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.465 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:32.465 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:32.465 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.19", 2022-09-02T05:01:32.466 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:32.466 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:32.466 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000072", 2022-09-02T05:01:32.467 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:32.467 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:32.467 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:32.467 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:32.468 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000062", 2022-09-02T05:01:32.468 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000115", 2022-09-02T05:01:32.468 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:32.468 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:32.469 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.1", 2022-09-02T05:01:32.469 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:32.469 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000033", 2022-09-02T05:01:32.469 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:32.470 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:32.470 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:32.470 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:32.470 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:32.471 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.471 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:32.471 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:32.471 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:32.472 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:32.472 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:32.472 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:32.472 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:32.473 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000025", 2022-09-02T05:01:32.473 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000123", 2022-09-02T05:01:32.473 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.473 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:32.474 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:32.474 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.18", 2022-09-02T05:01:32.474 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.3", 2022-09-02T05:01:32.475 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000108", 2022-09-02T05:01:32.475 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:32.475 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000094", 2022-09-02T05:01:32.475 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:32.476 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.476 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:32.476 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:32.476 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:32.477 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.477 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000060", 2022-09-02T05:01:32.477 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:32.477 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.478 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000109", 2022-09-02T05:01:32.478 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.15", 2022-09-02T05:01:32.478 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:32.478 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:32.479 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:32.479 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000013", 2022-09-02T05:01:32.479 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.2.retry", 2022-09-02T05:01:32.479 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:32.480 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:32.480 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:32.480 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.480 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:32.481 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:32.481 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:32.481 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:32.481 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:32.482 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:32.482 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:32.482 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:32.482 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:32.483 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.0.retry", 2022-09-02T05:01:32.483 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:32.483 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:32.483 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.484 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:32.484 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:32.484 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000051", 2022-09-02T05:01:32.484 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.485 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000106", 2022-09-02T05:01:32.485 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:32.485 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000017", 2022-09-02T05:01:32.485 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:32.486 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.2", 2022-09-02T05:01:32.486 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:32.486 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:32.487 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:32.487 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:32.487 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.487 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.3", 2022-09-02T05:01:32.488 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000023", 2022-09-02T05:01:32.488 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:32.488 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:32.488 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:32.489 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:32.489 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:32.489 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:32.489 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000107", 2022-09-02T05:01:32.490 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:32.490 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:32.490 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000046", 2022-09-02T05:01:32.490 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:32.491 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:32.491 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:32.491 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:32.492 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:32.492 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.492 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.1.retry", 2022-09-02T05:01:32.492 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:32.493 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0.lVkqDXNy/8c.0", 2022-09-02T05:01:32.493 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:32.493 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:32.494 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:32.494 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:32.494 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:32.494 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.2", 2022-09-02T05:01:32.495 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:32.495 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000067", 2022-09-02T05:01:32.495 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.495 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:32.496 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:32.496 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:32.496 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.496 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.b798a9cd-aa0e-480c-bb07-05f02abf3f41.2", 2022-09-02T05:01:32.497 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:32.497 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:32.497 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.9", 2022-09-02T05:01:32.497 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000086", 2022-09-02T05:01:32.498 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.5", 2022-09-02T05:01:32.498 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.498 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:32.498 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:32.499 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:32.499 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:32.499 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:32.499 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:32.500 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:32.500 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:32.500 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:32.500 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:32.501 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:32.501 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:32.501 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:32.502 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.502 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:32.502 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:32.502 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.0", 2022-09-02T05:01:32.503 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.503 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:32.503 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:32.503 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.504 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:32.504 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:32.504 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:32.504 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:32.505 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.505 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:32.505 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000050", 2022-09-02T05:01:32.505 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:32.506 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000102", 2022-09-02T05:01:32.506 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:32.506 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:32.507 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:32.507 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:32.507 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:32.507 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:32.508 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:32.508 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:32.508 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:32.508 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:32.509 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000124", 2022-09-02T05:01:32.509 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.509 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000001", 2022-09-02T05:01:32.509 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:32.510 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:32.510 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:32.510 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:32.510 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:32.511 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000126", 2022-09-02T05:01:32.511 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:32.511 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:32.512 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:32.512 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.512 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000055", 2022-09-02T05:01:32.512 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:32.513 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:32.513 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000061", 2022-09-02T05:01:32.513 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:32.513 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:32.514 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000097", 2022-09-02T05:01:32.514 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:32.514 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:32.514 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.515 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.0", 2022-09-02T05:01:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.839886eb-91c7-4e3c-80d4-fd4e89474f26.2.retry", 2022-09-02T05:01:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:32.516 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000042", 2022-09-02T05:01:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:32.517 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:32.518 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:32.518 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000075", 2022-09-02T05:01:32.518 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.518 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:32.519 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:32.519 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.839886eb-91c7-4e3c-80d4-fd4e89474f26:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:32.519 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000120", 2022-09-02T05:01:32.519 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000030", 2022-09-02T05:01:32.520 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:32.520 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:32.520 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7" 2022-09-02T05:01:32.520 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T05:01:32.521 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T05:01:32.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:32.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:32.523 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:32.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:32.585 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:32.628 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000078", 2022-09-02T05:01:32.628 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.history", 2022-09-02T05:01:32.628 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.629 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.629 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:32.629 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000070", 2022-09-02T05:01:32.629 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:32.630 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.630 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.630 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000104", 2022-09-02T05:01:32.631 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000026", 2022-09-02T05:01:32.631 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:32.631 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:32.631 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:32.632 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.632 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:32.632 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.8", 2022-09-02T05:01:32.632 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000028", 2022-09-02T05:01:32.633 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:32.633 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.633 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:32.633 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:32.634 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.634 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:32.634 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.635 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.635 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.635 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:32.636 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:32.636 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:32.636 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:32.637 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:32.637 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000040", 2022-09-02T05:01:32.637 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:32.637 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:32.638 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:32.638 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.639 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.6", 2022-09-02T05:01:32.639 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:32.639 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.639 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:32.640 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000015", 2022-09-02T05:01:32.640 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:32.640 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:32.640 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000069", 2022-09-02T05:01:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000095", 2022-09-02T05:01:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000003", 2022-09-02T05:01:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000047", 2022-09-02T05:01:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T05:01:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000035", 2022-09-02T05:01:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000037", 2022-09-02T05:01:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.3.retry", 2022-09-02T05:01:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000024", 2022-09-02T05:01:32.650 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000000", 2022-09-02T05:01:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:32.651 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000031", 2022-09-02T05:01:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.0.retry", 2022-09-02T05:01:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.652 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3", 2022-09-02T05:01:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:32.653 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:32.654 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:32.655 INFO:teuthology.orchestra.run.smithi099.stdout: "data_loggenerations_metadata", 2022-09-02T05:01:32.656 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:32.656 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:32.657 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.658 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.658 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:32.658 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:32.659 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000076", 2022-09-02T05:01:32.659 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.659 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.659 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000113", 2022-09-02T05:01:32.660 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:32.660 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000005", 2022-09-02T05:01:32.660 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.660 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000011", 2022-09-02T05:01:32.661 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000058", 2022-09-02T05:01:32.661 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000052", 2022-09-02T05:01:32.661 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:32.661 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.662 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:32.662 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:32.662 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:32.663 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.663 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:32.663 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.663 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.e248e3d2-c4b4-4705-8303-7ccdb6560fe1.3", 2022-09-02T05:01:32.664 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:32.664 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.1", 2022-09-02T05:01:32.664 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.664 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:32.665 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:32.665 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:32.665 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.666 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.666 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:32.666 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.666 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000088", 2022-09-02T05:01:32.667 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0.VjswNmqXu5Q.0", 2022-09-02T05:01:32.667 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000080", 2022-09-02T05:01:32.667 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000090", 2022-09-02T05:01:32.668 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.668 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.668 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000110", 2022-09-02T05:01:32.669 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:32.669 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.669 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:32.670 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.670 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.670 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.670 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.671 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.671 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.671 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.672 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.672 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:32.672 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000096", 2022-09-02T05:01:32.672 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:32.673 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:32.673 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:32.673 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.3", 2022-09-02T05:01:32.673 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000087", 2022-09-02T05:01:32.674 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.674 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:32.674 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000008", 2022-09-02T05:01:32.674 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.675 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.675 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.675 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.676 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.676 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.676 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000006", 2022-09-02T05:01:32.676 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000029", 2022-09-02T05:01:32.677 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.677 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:32.677 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:32.677 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:32.678 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000089", 2022-09-02T05:01:32.678 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.678 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000022", 2022-09-02T05:01:32.679 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.679 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.679 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.679 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.680 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:32.680 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:32.680 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:32.681 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.681 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.681 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog.trim", 2022-09-02T05:01:32.681 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:32.682 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000016", 2022-09-02T05:01:32.682 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:32.682 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:32.682 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000041", 2022-09-02T05:01:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000018", 2022-09-02T05:01:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000092", 2022-09-02T05:01:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.4", 2022-09-02T05:01:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1.s2yqZ/e3W+T.0", 2022-09-02T05:01:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.2.retry", 2022-09-02T05:01:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000014", 2022-09-02T05:01:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000112", 2022-09-02T05:01:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000007", 2022-09-02T05:01:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000021", 2022-09-02T05:01:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000064", 2022-09-02T05:01:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000071", 2022-09-02T05:01:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.2", 2022-09-02T05:01:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000074", 2022-09-02T05:01:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000081", 2022-09-02T05:01:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000009", 2022-09-02T05:01:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000121", 2022-09-02T05:01:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10", 2022-09-02T05:01:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.1", 2022-09-02T05:01:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:32.703 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.703 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000125", 2022-09-02T05:01:32.703 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:32.704 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000082", 2022-09-02T05:01:32.704 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:32.704 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:32.704 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:32.705 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000105", 2022-09-02T05:01:32.705 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000059", 2022-09-02T05:01:32.705 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2.+T15fYBQKpl.0", 2022-09-02T05:01:32.705 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:32.706 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:32.706 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000077", 2022-09-02T05:01:32.706 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000032", 2022-09-02T05:01:32.706 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:32.707 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:32.707 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:32.707 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:32.707 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:32.708 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.1", 2022-09-02T05:01:32.708 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:32.708 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:32.709 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:32.709 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:32.709 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:32.709 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c", 2022-09-02T05:01:32.710 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:32.710 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.2", 2022-09-02T05:01:32.711 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000053", 2022-09-02T05:01:32.711 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:32.711 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000091", 2022-09-02T05:01:32.711 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000065", 2022-09-02T05:01:32.712 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:32.712 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000083", 2022-09-02T05:01:32.712 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000010", 2022-09-02T05:01:32.712 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:32.713 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:32.713 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:32.713 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:32.714 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:32.714 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:32.714 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:32.714 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.7", 2022-09-02T05:01:32.715 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:32.715 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.3", 2022-09-02T05:01:32.715 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:32.715 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:32.716 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:32.716 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:32.716 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:32.716 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.0", 2022-09-02T05:01:32.717 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:32.717 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:32.717 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:32.718 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:32.718 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:32.718 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:32.719 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:32.719 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:32.721 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:32.721 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:32.721 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:32.722 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.2", 2022-09-02T05:01:32.722 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:32.722 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7", 2022-09-02T05:01:32.722 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:32.723 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:32.723 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:32.723 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000045", 2022-09-02T05:01:32.724 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:32.724 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000002", 2022-09-02T05:01:32.724 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:32.724 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:32.725 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:32.725 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000116", 2022-09-02T05:01:32.725 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:32.725 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.726 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:32.726 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.726 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:32.727 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.3.m0Xycl5Mtc0.0", 2022-09-02T05:01:32.727 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:32.727 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:32.727 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:32.728 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:32.728 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000034", 2022-09-02T05:01:32.728 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:32.728 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:32.729 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:32.729 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000101", 2022-09-02T05:01:32.729 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:32.729 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:32.730 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:32.730 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.1", 2022-09-02T05:01:32.730 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:32.730 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:32.731 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:32.731 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:32.732 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000079", 2022-09-02T05:01:32.732 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.732 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000049", 2022-09-02T05:01:32.732 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000117", 2022-09-02T05:01:32.733 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000044", 2022-09-02T05:01:32.733 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.733 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.733 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.734 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000066", 2022-09-02T05:01:32.734 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:32.734 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:32.734 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000068", 2022-09-02T05:01:32.735 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000085", 2022-09-02T05:01:32.735 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:32.735 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:32.736 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.736 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:32.736 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.737 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.737 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.737 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.737 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:32.738 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000073", 2022-09-02T05:01:32.738 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.738 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:32.739 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:32.739 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000038", 2022-09-02T05:01:32.739 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.739 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:32.740 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000118", 2022-09-02T05:01:32.740 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:32.740 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:32.741 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:32.741 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:32.741 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:32.742 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000036", 2022-09-02T05:01:32.742 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:32.742 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:32.743 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.743 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:32.743 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.743 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000103", 2022-09-02T05:01:32.744 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:32.744 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:32.744 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:32.745 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.3", 2022-09-02T05:01:32.745 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:32.745 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.745 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:32.746 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:32.746 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:32.746 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000119", 2022-09-02T05:01:32.746 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000098", 2022-09-02T05:01:32.747 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:32.747 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:32.747 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:32.748 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:32.748 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:32.748 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:32.748 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:32.749 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status.shard.0", 2022-09-02T05:01:32.749 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:32.749 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.749 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:32.750 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000027", 2022-09-02T05:01:32.750 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.8b4ce987-f0fe-4038-9bfd-de8d46ec635c.1.retry", 2022-09-02T05:01:32.750 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:32.751 INFO:teuthology.orchestra.run.smithi099.stdout: "mdlog.sync-status", 2022-09-02T05:01:32.751 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000019", 2022-09-02T05:01:32.751 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:32.751 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:32.752 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:32.752 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:32.753 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:32.753 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000039", 2022-09-02T05:01:32.753 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:32.754 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:32.754 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:32.754 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.754 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:32.755 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:32.755 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:32.756 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:32.756 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000100", 2022-09-02T05:01:32.757 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:32.757 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:32.761 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:32.761 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:32.762 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.762 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:32.763 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:32.763 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000093", 2022-09-02T05:01:32.764 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:32.764 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:32.765 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:32.765 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:32.766 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:32.766 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:32.767 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000004", 2022-09-02T05:01:32.767 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000063", 2022-09-02T05:01:32.768 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.768 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000122", 2022-09-02T05:01:32.768 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:32.768 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000057", 2022-09-02T05:01:32.769 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:32.769 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:32.769 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000054", 2022-09-02T05:01:32.770 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000114", 2022-09-02T05:01:32.770 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:32.770 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:32.770 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.2", 2022-09-02T05:01:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.0", 2022-09-02T05:01:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.e248e3d2-c4b4-4705-8303-7ccdb6560fe1.2", 2022-09-02T05:01:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000012", 2022-09-02T05:01:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000084", 2022-09-02T05:01:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000043", 2022-09-02T05:01:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:32.780 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:32.780 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:32.780 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:32.781 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:32.781 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:32.781 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000111", 2022-09-02T05:01:32.781 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:32.782 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000048", 2022-09-02T05:01:32.782 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:32.782 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:32.782 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:32.783 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000020", 2022-09-02T05:01:32.783 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:32.783 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.784 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:32.784 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.e248e3d2-c4b4-4705-8303-7ccdb6560fe1.0", 2022-09-02T05:01:32.784 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000099", 2022-09-02T05:01:32.785 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000056", 2022-09-02T05:01:32.785 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:32.785 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:32.786 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.786 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:32.787 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:32.787 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000072", 2022-09-02T05:01:32.788 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:32.788 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:32.789 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000062", 2022-09-02T05:01:32.789 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000115", 2022-09-02T05:01:32.790 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.790 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:32.791 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.1", 2022-09-02T05:01:32.791 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000033", 2022-09-02T05:01:32.791 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:32.792 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:32.792 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:32.792 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:32.793 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:32.793 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.793 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:32.793 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:32.794 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.794 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:32.795 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:32.795 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:32.796 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:32.796 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:32.796 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000025", 2022-09-02T05:01:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:32.797 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000123", 2022-09-02T05:01:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000108", 2022-09-02T05:01:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000094", 2022-09-02T05:01:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:32.801 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000060", 2022-09-02T05:01:32.802 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000109", 2022-09-02T05:01:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000013", 2022-09-02T05:01:32.803 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.2.retry", 2022-09-02T05:01:32.804 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:32.804 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:32.804 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:32.805 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:32.805 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:32.806 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:32.806 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.806 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.806 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:32.807 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.807 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:32.807 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:32.807 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.0.retry", 2022-09-02T05:01:32.808 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:32.808 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:32.808 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:32.809 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000051", 2022-09-02T05:01:32.809 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:32.809 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:32.809 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000106", 2022-09-02T05:01:32.810 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:32.810 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000017", 2022-09-02T05:01:32.810 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:32.811 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:32.812 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.3", 2022-09-02T05:01:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000023", 2022-09-02T05:01:32.813 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:32.814 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.815 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:32.816 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000107", 2022-09-02T05:01:32.816 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:32.816 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000046", 2022-09-02T05:01:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.817 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.818 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.1.retry", 2022-09-02T05:01:32.819 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:32.819 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:32.819 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:32.819 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:32.820 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:32.820 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.820 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.820 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:32.821 INFO:teuthology.orchestra.run.smithi099.stdout: "data_log.2", 2022-09-02T05:01:32.821 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:32.821 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.ffa7b930-977f-415c-8bb4-bf6fd394dc85.2", 2022-09-02T05:01:32.821 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000067", 2022-09-02T05:01:32.822 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.822 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:32.822 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:32.823 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:32.823 INFO:teuthology.orchestra.run.smithi099.stdout: "meta.log.b798a9cd-aa0e-480c-bb07-05f02abf3f41.2", 2022-09-02T05:01:32.823 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:32.824 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:32.824 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000086", 2022-09-02T05:01:32.824 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.824 INFO:teuthology.orchestra.run.smithi099.stdout: "sync.error-log.5", 2022-09-02T05:01:32.825 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:32.825 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.825 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:32.825 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:32.826 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:32.826 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:32.832 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000050", 2022-09-02T05:01:32.832 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:32.832 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000102", 2022-09-02T05:01:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000124", 2022-09-02T05:01:32.837 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:32.837 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.837 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000001", 2022-09-02T05:01:32.837 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:32.838 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:32.838 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000126", 2022-09-02T05:01:32.838 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000055", 2022-09-02T05:01:32.840 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:32.840 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000061", 2022-09-02T05:01:32.840 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:32.840 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:32.841 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:32.841 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:32.841 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:32.842 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:32.842 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000097", 2022-09-02T05:01:32.842 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.843 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:32.843 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:32.843 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.843 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:32.844 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:32.844 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:32.844 INFO:teuthology.orchestra.run.smithi099.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.0", 2022-09-02T05:01:32.844 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:32.845 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000042", 2022-09-02T05:01:32.845 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:32.845 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:32.845 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:32.846 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:32.846 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:32.846 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:32.846 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:32.847 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000075", 2022-09-02T05:01:32.847 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:32.848 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:32.848 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:32.848 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000120", 2022-09-02T05:01:32.848 INFO:teuthology.orchestra.run.smithi099.stdout: "obj_delete_at_hint.0000000030", 2022-09-02T05:01:32.849 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:32.849 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.849 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:32.849 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7", 2022-09-02T05:01:32.850 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:32.850 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket.sync-status.8b4ce987-f0fe-4038-9bfd-de8d46ec635c:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10" 2022-09-02T05:01:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:32.851 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-09-02T05:01:32.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:32.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:32.852 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:32.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:32.906 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:32.958 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T05:01:32.958 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000078", 2022-09-02T05:01:32.959 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.history", 2022-09-02T05:01:32.959 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:8", 2022-09-02T05:01:32.959 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:9", 2022-09-02T05:01:32.960 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000070", 2022-09-02T05:01:32.960 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:0", 2022-09-02T05:01:32.960 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:3", 2022-09-02T05:01:32.960 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000104", 2022-09-02T05:01:32.961 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000026", 2022-09-02T05:01:32.961 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:4", 2022-09-02T05:01:32.961 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:0", 2022-09-02T05:01:32.961 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.8", 2022-09-02T05:01:32.962 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000028", 2022-09-02T05:01:32.962 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:9", 2022-09-02T05:01:32.962 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:9", 2022-09-02T05:01:32.962 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:0", 2022-09-02T05:01:32.963 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:7", 2022-09-02T05:01:32.963 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:7", 2022-09-02T05:01:32.963 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:10", 2022-09-02T05:01:32.963 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:9", 2022-09-02T05:01:32.964 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:9", 2022-09-02T05:01:32.964 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000040", 2022-09-02T05:01:32.964 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:10", 2022-09-02T05:01:32.964 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.6", 2022-09-02T05:01:32.965 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:7", 2022-09-02T05:01:32.965 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000015", 2022-09-02T05:01:32.965 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:8", 2022-09-02T05:01:32.965 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000069", 2022-09-02T05:01:32.966 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000095", 2022-09-02T05:01:32.966 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000003", 2022-09-02T05:01:32.966 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000047", 2022-09-02T05:01:32.966 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:2", 2022-09-02T05:01:32.967 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:0", 2022-09-02T05:01:32.967 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:2", 2022-09-02T05:01:32.967 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073", 2022-09-02T05:01:32.967 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000035", 2022-09-02T05:01:32.968 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000037", 2022-09-02T05:01:32.968 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:10", 2022-09-02T05:01:32.968 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:1", 2022-09-02T05:01:32.968 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:3", 2022-09-02T05:01:32.969 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:8", 2022-09-02T05:01:32.969 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:1", 2022-09-02T05:01:32.969 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000024", 2022-09-02T05:01:32.969 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000000", 2022-09-02T05:01:32.970 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:6", 2022-09-02T05:01:32.970 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000031", 2022-09-02T05:01:32.970 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:5", 2022-09-02T05:01:32.970 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:2", 2022-09-02T05:01:32.971 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:2", 2022-09-02T05:01:32.971 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:7", 2022-09-02T05:01:32.971 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:9", 2022-09-02T05:01:32.971 INFO:teuthology.orchestra.run.smithi160.stdout: "data_loggenerations_metadata", 2022-09-02T05:01:32.972 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:1", 2022-09-02T05:01:32.972 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:8", 2022-09-02T05:01:32.972 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15", 2022-09-02T05:01:32.972 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000076", 2022-09-02T05:01:32.973 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:4", 2022-09-02T05:01:32.973 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000113", 2022-09-02T05:01:32.973 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:8", 2022-09-02T05:01:32.973 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000005", 2022-09-02T05:01:32.974 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000011", 2022-09-02T05:01:32.974 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000058", 2022-09-02T05:01:32.974 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000052", 2022-09-02T05:01:32.975 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.e248e3d2-c4b4-4705-8303-7ccdb6560fe1.3", 2022-09-02T05:01:32.975 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.1", 2022-09-02T05:01:32.975 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:8", 2022-09-02T05:01:32.975 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:1", 2022-09-02T05:01:32.976 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:1", 2022-09-02T05:01:32.976 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:0", 2022-09-02T05:01:32.976 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18", 2022-09-02T05:01:32.976 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000088", 2022-09-02T05:01:32.976 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000080", 2022-09-02T05:01:32.977 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000090", 2022-09-02T05:01:32.977 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:6", 2022-09-02T05:01:32.977 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000110", 2022-09-02T05:01:32.977 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:6", 2022-09-02T05:01:32.978 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:8", 2022-09-02T05:01:32.978 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:0", 2022-09-02T05:01:32.978 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:2", 2022-09-02T05:01:32.978 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:6", 2022-09-02T05:01:32.979 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:10", 2022-09-02T05:01:32.979 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000096", 2022-09-02T05:01:32.979 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:3", 2022-09-02T05:01:32.979 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:1", 2022-09-02T05:01:32.980 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000087", 2022-09-02T05:01:32.980 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:10", 2022-09-02T05:01:32.980 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000008", 2022-09-02T05:01:32.980 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:1", 2022-09-02T05:01:32.981 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:8", 2022-09-02T05:01:32.981 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:0", 2022-09-02T05:01:32.981 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:5", 2022-09-02T05:01:32.981 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11", 2022-09-02T05:01:32.982 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:2", 2022-09-02T05:01:32.982 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000006", 2022-09-02T05:01:32.982 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000029", 2022-09-02T05:01:32.983 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:1", 2022-09-02T05:01:32.983 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:10", 2022-09-02T05:01:32.983 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:0", 2022-09-02T05:01:32.983 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:6", 2022-09-02T05:01:32.984 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000089", 2022-09-02T05:01:32.984 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22", 2022-09-02T05:01:32.984 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000022", 2022-09-02T05:01:32.984 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:2", 2022-09-02T05:01:32.984 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:4", 2022-09-02T05:01:32.985 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:8", 2022-09-02T05:01:32.985 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:7", 2022-09-02T05:01:32.985 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:0", 2022-09-02T05:01:32.985 INFO:teuthology.orchestra.run.smithi160.stdout: "bilog.trim", 2022-09-02T05:01:32.986 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000016", 2022-09-02T05:01:32.986 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:10", 2022-09-02T05:01:32.986 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000041", 2022-09-02T05:01:32.987 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:6", 2022-09-02T05:01:32.987 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:3", 2022-09-02T05:01:32.987 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5", 2022-09-02T05:01:32.987 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:3", 2022-09-02T05:01:32.988 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000018", 2022-09-02T05:01:32.988 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000092", 2022-09-02T05:01:32.988 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:3", 2022-09-02T05:01:32.988 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:10", 2022-09-02T05:01:32.988 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:2", 2022-09-02T05:01:32.989 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.4", 2022-09-02T05:01:32.989 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:10", 2022-09-02T05:01:32.990 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:3", 2022-09-02T05:01:32.990 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:8", 2022-09-02T05:01:32.990 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17", 2022-09-02T05:01:32.991 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000014", 2022-09-02T05:01:32.991 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:8", 2022-09-02T05:01:32.991 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:0", 2022-09-02T05:01:32.991 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:7", 2022-09-02T05:01:32.992 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000112", 2022-09-02T05:01:32.992 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:4", 2022-09-02T05:01:32.992 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16", 2022-09-02T05:01:32.992 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000007", 2022-09-02T05:01:32.993 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000021", 2022-09-02T05:01:32.993 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:7", 2022-09-02T05:01:32.993 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:4", 2022-09-02T05:01:32.994 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:5", 2022-09-02T05:01:32.994 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000064", 2022-09-02T05:01:32.994 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:7", 2022-09-02T05:01:32.994 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:1", 2022-09-02T05:01:32.995 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000071", 2022-09-02T05:01:32.995 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.2", 2022-09-02T05:01:32.995 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:1", 2022-09-02T05:01:32.995 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000074", 2022-09-02T05:01:32.996 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:3", 2022-09-02T05:01:32.996 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38", 2022-09-02T05:01:32.996 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:9", 2022-09-02T05:01:32.997 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:6", 2022-09-02T05:01:32.997 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000081", 2022-09-02T05:01:32.997 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000009", 2022-09-02T05:01:32.997 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:6", 2022-09-02T05:01:32.997 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000121", 2022-09-02T05:01:32.998 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:10", 2022-09-02T05:01:32.998 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20", 2022-09-02T05:01:32.998 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:10", 2022-09-02T05:01:32.999 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:7", 2022-09-02T05:01:32.999 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:0", 2022-09-02T05:01:32.999 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000125", 2022-09-02T05:01:32.999 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000082", 2022-09-02T05:01:33.000 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23", 2022-09-02T05:01:33.000 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:4", 2022-09-02T05:01:33.000 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000105", 2022-09-02T05:01:33.000 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000059", 2022-09-02T05:01:33.001 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:5", 2022-09-02T05:01:33.001 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:2", 2022-09-02T05:01:33.001 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000077", 2022-09-02T05:01:33.001 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000032", 2022-09-02T05:01:33.002 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:7", 2022-09-02T05:01:33.002 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1", 2022-09-02T05:01:33.002 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:2", 2022-09-02T05:01:33.002 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12", 2022-09-02T05:01:33.003 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:9", 2022-09-02T05:01:33.003 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000053", 2022-09-02T05:01:33.003 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000091", 2022-09-02T05:01:33.003 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000065", 2022-09-02T05:01:33.004 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000083", 2022-09-02T05:01:33.004 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000010", 2022-09-02T05:01:33.004 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:7", 2022-09-02T05:01:33.004 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:10", 2022-09-02T05:01:33.005 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36", 2022-09-02T05:01:33.005 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.7", 2022-09-02T05:01:33.005 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:7", 2022-09-02T05:01:33.005 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.3", 2022-09-02T05:01:33.006 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:1", 2022-09-02T05:01:33.006 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:3", 2022-09-02T05:01:33.006 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:3", 2022-09-02T05:01:33.006 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:3", 2022-09-02T05:01:33.007 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:5", 2022-09-02T05:01:33.007 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:8", 2022-09-02T05:01:33.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:0", 2022-09-02T05:01:33.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:9", 2022-09-02T05:01:33.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:10", 2022-09-02T05:01:33.008 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:3", 2022-09-02T05:01:33.009 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.2", 2022-09-02T05:01:33.009 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.3.retry", 2022-09-02T05:01:33.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:3", 2022-09-02T05:01:33.009 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:6", 2022-09-02T05:01:33.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:10", 2022-09-02T05:01:33.010 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000045", 2022-09-02T05:01:33.010 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000002", 2022-09-02T05:01:33.010 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:5", 2022-09-02T05:01:33.011 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:4", 2022-09-02T05:01:33.011 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:2", 2022-09-02T05:01:33.011 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000116", 2022-09-02T05:01:33.011 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:0", 2022-09-02T05:01:33.012 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:10", 2022-09-02T05:01:33.012 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:6", 2022-09-02T05:01:33.012 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:2", 2022-09-02T05:01:33.012 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:4", 2022-09-02T05:01:33.013 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000034", 2022-09-02T05:01:33.013 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:2", 2022-09-02T05:01:33.013 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000101", 2022-09-02T05:01:33.013 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:0", 2022-09-02T05:01:33.014 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26", 2022-09-02T05:01:33.014 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.1", 2022-09-02T05:01:33.014 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:3", 2022-09-02T05:01:33.014 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:7", 2022-09-02T05:01:33.015 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000079", 2022-09-02T05:01:33.015 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000049", 2022-09-02T05:01:33.015 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000117", 2022-09-02T05:01:33.015 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000044", 2022-09-02T05:01:33.016 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000066", 2022-09-02T05:01:33.016 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19", 2022-09-02T05:01:33.016 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000068", 2022-09-02T05:01:33.016 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000085", 2022-09-02T05:01:33.017 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:5", 2022-09-02T05:01:33.017 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000073", 2022-09-02T05:01:33.017 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:6", 2022-09-02T05:01:33.017 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000038", 2022-09-02T05:01:33.018 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.10", 2022-09-02T05:01:33.018 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:4", 2022-09-02T05:01:33.018 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000118", 2022-09-02T05:01:33.019 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:5", 2022-09-02T05:01:33.019 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:2", 2022-09-02T05:01:33.019 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:9", 2022-09-02T05:01:33.019 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:8", 2022-09-02T05:01:33.020 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000036", 2022-09-02T05:01:33.020 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:6", 2022-09-02T05:01:33.020 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:4", 2022-09-02T05:01:33.020 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000103", 2022-09-02T05:01:33.020 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:6", 2022-09-02T05:01:33.021 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:7", 2022-09-02T05:01:33.021 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.3", 2022-09-02T05:01:33.021 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:5", 2022-09-02T05:01:33.021 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000119", 2022-09-02T05:01:33.022 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000098", 2022-09-02T05:01:33.022 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:6", 2022-09-02T05:01:33.022 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:5", 2022-09-02T05:01:33.023 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:0", 2022-09-02T05:01:33.023 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:3", 2022-09-02T05:01:33.023 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status.shard.0", 2022-09-02T05:01:33.023 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:0", 2022-09-02T05:01:33.024 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000027", 2022-09-02T05:01:33.024 INFO:teuthology.orchestra.run.smithi160.stdout: "mdlog.sync-status", 2022-09-02T05:01:33.024 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000019", 2022-09-02T05:01:33.024 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10", 2022-09-02T05:01:33.025 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:8", 2022-09-02T05:01:33.025 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:10", 2022-09-02T05:01:33.025 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000039", 2022-09-02T05:01:33.025 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:1", 2022-09-02T05:01:33.026 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:9", 2022-09-02T05:01:33.026 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:4", 2022-09-02T05:01:33.026 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:8", 2022-09-02T05:01:33.027 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:9", 2022-09-02T05:01:33.027 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:3", 2022-09-02T05:01:33.027 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000100", 2022-09-02T05:01:33.028 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:10", 2022-09-02T05:01:33.028 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:4", 2022-09-02T05:01:33.028 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:2", 2022-09-02T05:01:33.028 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:4", 2022-09-02T05:01:33.029 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:1", 2022-09-02T05:01:33.029 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:2", 2022-09-02T05:01:33.029 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000093", 2022-09-02T05:01:33.029 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:2", 2022-09-02T05:01:33.030 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:7", 2022-09-02T05:01:33.030 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:5", 2022-09-02T05:01:33.030 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:0", 2022-09-02T05:01:33.030 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000004", 2022-09-02T05:01:33.031 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000063", 2022-09-02T05:01:33.031 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000122", 2022-09-02T05:01:33.031 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:9", 2022-09-02T05:01:33.031 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000057", 2022-09-02T05:01:33.032 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:0", 2022-09-02T05:01:33.032 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:6", 2022-09-02T05:01:33.032 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000054", 2022-09-02T05:01:33.032 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000114", 2022-09-02T05:01:33.033 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:5", 2022-09-02T05:01:33.033 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:8", 2022-09-02T05:01:33.033 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:3", 2022-09-02T05:01:33.034 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.2", 2022-09-02T05:01:33.034 INFO:teuthology.orchestra.run.smithi160.stdout: "data_log.0", 2022-09-02T05:01:33.034 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:10", 2022-09-02T05:01:33.034 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:0", 2022-09-02T05:01:33.035 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:5", 2022-09-02T05:01:33.035 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:1", 2022-09-02T05:01:33.035 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:1", 2022-09-02T05:01:33.036 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.e248e3d2-c4b4-4705-8303-7ccdb6560fe1.2", 2022-09-02T05:01:33.036 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:9", 2022-09-02T05:01:33.036 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000012", 2022-09-02T05:01:33.036 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000084", 2022-09-02T05:01:33.037 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000043", 2022-09-02T05:01:33.037 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:9", 2022-09-02T05:01:33.038 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:2", 2022-09-02T05:01:33.038 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:7", 2022-09-02T05:01:33.038 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:5", 2022-09-02T05:01:33.038 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:4", 2022-09-02T05:01:33.039 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:1", 2022-09-02T05:01:33.039 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:5", 2022-09-02T05:01:33.039 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:4", 2022-09-02T05:01:33.040 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000111", 2022-09-02T05:01:33.040 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:7", 2022-09-02T05:01:33.040 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000048", 2022-09-02T05:01:33.040 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:8", 2022-09-02T05:01:33.041 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:0", 2022-09-02T05:01:33.041 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000020", 2022-09-02T05:01:33.041 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:3", 2022-09-02T05:01:33.042 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:1", 2022-09-02T05:01:33.042 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.e248e3d2-c4b4-4705-8303-7ccdb6560fe1.0", 2022-09-02T05:01:33.042 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000099", 2022-09-02T05:01:33.042 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000056", 2022-09-02T05:01:33.043 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:10", 2022-09-02T05:01:33.043 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:5", 2022-09-02T05:01:33.043 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000072", 2022-09-02T05:01:33.043 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21", 2022-09-02T05:01:33.044 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000062", 2022-09-02T05:01:33.044 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000115", 2022-09-02T05:01:33.044 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.1", 2022-09-02T05:01:33.044 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000033", 2022-09-02T05:01:33.045 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:1", 2022-09-02T05:01:33.045 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:9", 2022-09-02T05:01:33.045 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25", 2022-09-02T05:01:33.045 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:1", 2022-09-02T05:01:33.046 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:3", 2022-09-02T05:01:33.046 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37", 2022-09-02T05:01:33.046 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:9", 2022-09-02T05:01:33.046 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:9", 2022-09-02T05:01:33.047 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:6", 2022-09-02T05:01:33.047 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:4", 2022-09-02T05:01:33.047 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000025", 2022-09-02T05:01:33.047 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000123", 2022-09-02T05:01:33.048 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:3", 2022-09-02T05:01:33.048 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:7", 2022-09-02T05:01:33.048 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000108", 2022-09-02T05:01:33.048 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:6", 2022-09-02T05:01:33.049 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000094", 2022-09-02T05:01:33.049 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:4", 2022-09-02T05:01:33.049 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:8", 2022-09-02T05:01:33.049 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:1", 2022-09-02T05:01:33.050 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000060", 2022-09-02T05:01:33.050 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000109", 2022-09-02T05:01:33.050 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:0", 2022-09-02T05:01:33.050 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000013", 2022-09-02T05:01:33.051 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.2.retry", 2022-09-02T05:01:33.051 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:5", 2022-09-02T05:01:33.051 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:2", 2022-09-02T05:01:33.052 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:4", 2022-09-02T05:01:33.052 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:3", 2022-09-02T05:01:33.052 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24", 2022-09-02T05:01:33.052 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3", 2022-09-02T05:01:33.053 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-15:b210c8c8-c433-420b-87c1-57ebfc433073.4350.15:2", 2022-09-02T05:01:33.053 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:3", 2022-09-02T05:01:33.053 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.0.retry", 2022-09-02T05:01:33.053 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:0", 2022-09-02T05:01:33.054 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:5", 2022-09-02T05:01:33.054 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000051", 2022-09-02T05:01:33.054 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000106", 2022-09-02T05:01:33.055 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:7", 2022-09-02T05:01:33.055 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000017", 2022-09-02T05:01:33.055 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:8", 2022-09-02T05:01:33.055 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-18:b210c8c8-c433-420b-87c1-57ebfc433073.4350.18:6", 2022-09-02T05:01:33.056 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:4", 2022-09-02T05:01:33.056 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13", 2022-09-02T05:01:33.056 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.3", 2022-09-02T05:01:33.056 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000023", 2022-09-02T05:01:33.057 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:8", 2022-09-02T05:01:33.057 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:3", 2022-09-02T05:01:33.057 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:1", 2022-09-02T05:01:33.057 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:10", 2022-09-02T05:01:33.058 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:10", 2022-09-02T05:01:33.058 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000107", 2022-09-02T05:01:33.058 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:6", 2022-09-02T05:01:33.058 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000046", 2022-09-02T05:01:33.059 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:7", 2022-09-02T05:01:33.059 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.1.retry", 2022-09-02T05:01:33.059 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:2", 2022-09-02T05:01:33.060 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:10", 2022-09-02T05:01:33.060 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-11:b210c8c8-c433-420b-87c1-57ebfc433073.4350.11:5", 2022-09-02T05:01:33.060 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.ffa7b930-977f-415c-8bb4-bf6fd394dc85.2", 2022-09-02T05:01:33.061 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000067", 2022-09-02T05:01:33.061 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-26:b210c8c8-c433-420b-87c1-57ebfc433073.4350.26:5", 2022-09-02T05:01:33.061 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:5", 2022-09-02T05:01:33.061 INFO:teuthology.orchestra.run.smithi160.stdout: "meta.log.b798a9cd-aa0e-480c-bb07-05f02abf3f41.2", 2022-09-02T05:01:33.062 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:9", 2022-09-02T05:01:33.062 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-24:b210c8c8-c433-420b-87c1-57ebfc433073.4350.24:4", 2022-09-02T05:01:33.062 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.9", 2022-09-02T05:01:33.063 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000086", 2022-09-02T05:01:33.063 INFO:teuthology.orchestra.run.smithi160.stdout: "sync.error-log.5", 2022-09-02T05:01:33.063 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:0", 2022-09-02T05:01:33.063 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-22:b210c8c8-c433-420b-87c1-57ebfc433073.4350.22:5", 2022-09-02T05:01:33.064 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-23:b210c8c8-c433-420b-87c1-57ebfc433073.4350.23:8", 2022-09-02T05:01:33.064 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-38:b210c8c8-c433-420b-87c1-57ebfc433073.4350.38:6", 2022-09-02T05:01:33.064 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:1", 2022-09-02T05:01:33.065 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:10", 2022-09-02T05:01:33.065 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:8", 2022-09-02T05:01:33.065 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:8", 2022-09-02T05:01:33.065 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-37:b210c8c8-c433-420b-87c1-57ebfc433073.4350.37:10", 2022-09-02T05:01:33.066 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-19:b210c8c8-c433-420b-87c1-57ebfc433073.4350.19:0", 2022-09-02T05:01:33.066 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000050", 2022-09-02T05:01:33.066 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:5", 2022-09-02T05:01:33.066 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000102", 2022-09-02T05:01:33.067 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:9", 2022-09-02T05:01:33.067 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-13:b210c8c8-c433-420b-87c1-57ebfc433073.4350.13:2", 2022-09-02T05:01:33.067 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:9", 2022-09-02T05:01:33.068 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:4", 2022-09-02T05:01:33.068 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:4", 2022-09-02T05:01:33.068 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-46:b210c8c8-c433-420b-87c1-57ebfc433073.5159.3:4", 2022-09-02T05:01:33.068 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:4", 2022-09-02T05:01:33.069 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000124", 2022-09-02T05:01:33.069 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000001", 2022-09-02T05:01:33.069 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-20:b210c8c8-c433-420b-87c1-57ebfc433073.4350.20:7", 2022-09-02T05:01:33.070 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-36:b210c8c8-c433-420b-87c1-57ebfc433073.4350.36:9", 2022-09-02T05:01:33.070 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000126", 2022-09-02T05:01:33.070 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:7", 2022-09-02T05:01:33.070 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14:6", 2022-09-02T05:01:33.071 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-10:b210c8c8-c433-420b-87c1-57ebfc433073.4350.10:6", 2022-09-02T05:01:33.071 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000055", 2022-09-02T05:01:33.071 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:2", 2022-09-02T05:01:33.071 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000061", 2022-09-02T05:01:33.072 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-16:b210c8c8-c433-420b-87c1-57ebfc433073.4350.16:6", 2022-09-02T05:01:33.072 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:3", 2022-09-02T05:01:33.072 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000097", 2022-09-02T05:01:33.072 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:1", 2022-09-02T05:01:33.073 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-49:b210c8c8-c433-420b-87c1-57ebfc433073.5403.1:8", 2022-09-02T05:01:33.073 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-48:b210c8c8-c433-420b-87c1-57ebfc433073.5159.5:2", 2022-09-02T05:01:33.073 INFO:teuthology.orchestra.run.smithi160.stdout: "datalog.sync-status.shard.b210c8c8-c433-420b-87c1-57ebfc433073.0", 2022-09-02T05:01:33.073 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-21:b210c8c8-c433-420b-87c1-57ebfc433073.4350.21:5", 2022-09-02T05:01:33.074 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000042", 2022-09-02T05:01:33.074 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4:1", 2022-09-02T05:01:33.074 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:9", 2022-09-02T05:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-47:b210c8c8-c433-420b-87c1-57ebfc433073.5159.4", 2022-09-02T05:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.full-sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-14:b210c8c8-c433-420b-87c1-57ebfc433073.4350.14", 2022-09-02T05:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000075", 2022-09-02T05:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:9", 2022-09-02T05:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000120", 2022-09-02T05:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "obj_delete_at_hint.0000000030", 2022-09-02T05:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-12:b210c8c8-c433-420b-87c1-57ebfc433073.4350.12:6", 2022-09-02T05:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-17:b210c8c8-c433-420b-87c1-57ebfc433073.4350.17:7", 2022-09-02T05:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "bucket.sync-status.b210c8c8-c433-420b-87c1-57ebfc433073:xljbkf-25:b210c8c8-c433-420b-87c1-57ebfc433073.4350.25:7" 2022-09-02T05:01:33.077 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T05:01:33.077 INFO:teuthology.orchestra.run.smithi160.stdout: 2022-09-02T05:01:33.078 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2022-09-02T05:01:33.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:33.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:33.080 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:33.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:33.176 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-02T05:01:33.176 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.836581Z" 2022-09-02T05:01:33.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:33.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:33.177 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006141", 2022-09-02T05:01:33.177 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.837696Z" 2022-09-02T05:01:33.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:33.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:33.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007833", 2022-09-02T05:01:33.178 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.838740Z" 2022-09-02T05:01:33.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:33.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:33.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:33.179 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.839767Z" 2022-09-02T05:01:33.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:33.180 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:33.181 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:33.182 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-02T05:01:33.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:33.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:33.184 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:33.236 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:33.236 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:33.295 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:33.296 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:33.296 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:33.296 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:33.297 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:33.297 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 11009958948854697431 2022-09-02T05:01:33.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:33.298 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:33.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:33.298 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:33.299 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T05:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:33.300 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T05:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:33.301 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004881", 2022-09-02T05:01:33.302 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:51:17.171208Z" 2022-09-02T05:01:33.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:33.304 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006573", 2022-09-02T05:01:33.305 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:33.306 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:54:50.298578Z" 2022-09-02T05:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:33.307 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:33.308 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T05:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:33.309 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T05:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:33.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:33.311 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:33.311 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T05:01:33.311 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T05:01:33.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:33.312 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:33Z" 2022-09-02T05:01:33.312 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:33.313 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11009958948854697431 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004881", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:17.171208Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006573", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:54:50.298578Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:33Z" } 2022-09-02T05:01:33.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:17.171208Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006573', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:54:50.298578Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T05:01:33.314 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:33.315 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006573', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T05:01:33.315 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000002909, shard=1 master=00000000000000000000:00000000000000006141 target=00000000000000000000:00000000000000004881, shard=2 master=00000000000000000000:00000000000000007833 target=00000000000000000000:00000000000000006573, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-02T05:01:38.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:38.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:38.318 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:38.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:38.377 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:38.433 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:38.433 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:38.434 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:38.434 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:38.434 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:38.434 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 11009958948854697431 2022-09-02T05:01:38.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:38.435 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:38.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:38.435 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:38.436 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:38.436 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:38.436 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002909", 2022-09-02T05:01:38.436 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:38.437 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:38.437 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:38.437 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:31:59.361841Z" 2022-09-02T05:01:38.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:38.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:38.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:38.438 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:38.438 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:38.439 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:38.439 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000004881", 2022-09-02T05:01:38.439 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:38.440 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:38.440 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:38.440 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:51:17.171208Z" 2022-09-02T05:01:38.440 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:38.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:38.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:38.441 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:38.441 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:38.442 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:38.442 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006573", 2022-09-02T05:01:38.442 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:38.442 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:38.443 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:38.443 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:54:50.298578Z" 2022-09-02T05:01:38.443 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:38.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:38.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:38.444 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:38.444 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:38.444 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:38.445 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T05:01:38.445 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:38.445 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:38.445 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:38.446 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:50:48.826574Z" 2022-09-02T05:01:38.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:38.446 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:38.446 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:38.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:38.447 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:38.447 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T05:01:38.447 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T05:01:38.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:38.448 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:38Z" 2022-09-02T05:01:38.448 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:38.449 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11009958948854697431 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:31:59.361841Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004881", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:17.171208Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006573", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:54:50.298578Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:48.826574Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:38Z" } 2022-09-02T05:01:38.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:31:59.361841Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004881', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:17.171208Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006573', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:54:50.298578Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:48.826574Z'}}] 2022-09-02T05:01:38.450 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:38.450 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002909', 1: '00000000000000000000:00000000000000004881', 2: '00000000000000000000:00000000000000006573', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T05:01:38.451 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003750 target=00000000000000000000:00000000000000002909, shard=1 master=00000000000000000000:00000000000000006141 target=00000000000000000000:00000000000000004881, shard=2 master=00000000000000000000:00000000000000007833 target=00000000000000000000:00000000000000006573, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-02T05:01:43.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.453 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:43.513 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:43.513 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:43.564 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:43.564 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:43.565 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 11009958948854697431 2022-09-02T05:01:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:43.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.567 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-02T05:01:43.568 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:43.568 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:43.568 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:43.568 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.836581Z" 2022-09-02T05:01:43.569 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:43.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:43.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:43.570 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:43.570 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:43.570 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006141", 2022-09-02T05:01:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:43.571 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.837696Z" 2022-09-02T05:01:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:43.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:43.573 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007833", 2022-09-02T05:01:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:43.574 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.838740Z" 2022-09-02T05:01:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:43.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:43.576 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:43.576 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:43.576 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.577 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:43.577 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:43.577 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:43.578 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:43.578 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.839767Z" 2022-09-02T05:01:43.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:43.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:43.579 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:43.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:43.579 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T05:01:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T05:01:43.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:43.581 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:43Z" 2022-09-02T05:01:43.581 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:43.582 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11009958948854697431 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.836581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006141", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.837696Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007833", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.838740Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.839767Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:43Z" } 2022-09-02T05:01:43.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.836581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.837696Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007833', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.838740Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.839767Z'}}] 2022-09-02T05:01:43.584 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:43.584 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:43.585 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-09-02T05:01:43.585 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2022-09-02T05:01:43.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.587 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:43.642 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:43.643 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:43.692 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-02T05:01:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.836581Z" 2022-09-02T05:01:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006141", 2022-09-02T05:01:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.837696Z" 2022-09-02T05:01:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007833", 2022-09-02T05:01:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.838740Z" 2022-09-02T05:01:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.839767Z" 2022-09-02T05:01:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.697 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:43.698 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:43.698 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-02T05:01:43.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.700 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:43.753 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:43.753 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:43.802 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:43.803 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:43.803 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:43.803 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:43.803 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:43.804 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1295418732353758134 2022-09-02T05:01:43.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.804 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:43.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.805 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:43.805 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:43.805 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.806 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-02T05:01:43.806 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:43.806 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:43.806 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:43.807 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:31.836581Z" 2022-09-02T05:01:43.807 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.808 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:43.808 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:43.808 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006141", 2022-09-02T05:01:43.809 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:43.809 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:43.809 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:43.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:31.837696Z" 2022-09-02T05:01:43.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.811 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:43.811 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:43.811 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.811 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007833", 2022-09-02T05:01:43.812 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:43.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:43.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:43.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:31.838740Z" 2022-09-02T05:01:43.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.813 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:43.814 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:43.814 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:43.814 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:43.814 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:43.815 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:43.815 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:43.815 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:31.839767Z" 2022-09-02T05:01:43.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.816 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:43.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.816 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:43.817 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T05:01:43.817 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T05:01:43.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.817 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:43Z" 2022-09-02T05:01:43.818 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:43.819 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1295418732353758134 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.836581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006141", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.837696Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007833", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.838740Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.839767Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:43Z" } 2022-09-02T05:01:43.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.836581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.837696Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007833', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.838740Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.839767Z'}}] 2022-09-02T05:01:43.820 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:43.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:43.821 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-09-02T05:01:43.822 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone4 2022-09-02T05:01:43.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.823 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:43.871 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:43.872 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:43.921 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:43.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.922 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-02T05:01:43.922 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.836581Z" 2022-09-02T05:01:43.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.923 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006141", 2022-09-02T05:01:43.923 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.837696Z" 2022-09-02T05:01:43.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.924 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007833", 2022-09-02T05:01:43.924 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.838740Z" 2022-09-02T05:01:43.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:43.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:43.925 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:43.925 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:31.839767Z" 2022-09-02T05:01:43.925 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:43.926 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:43.927 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:43.927 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-02T05:01:43.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:43.929 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:43.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:43.986 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:44.027 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 2207523797324220113 2022-09-02T05:01:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.029 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:44.030 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003750", 2022-09-02T05:01:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:44.031 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:44.032 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:44.032 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.836581Z" 2022-09-02T05:01:44.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:44.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.034 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:44.034 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:44.034 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.035 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006141", 2022-09-02T05:01:44.035 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:44.036 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:44.036 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:44.036 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.837696Z" 2022-09-02T05:01:44.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:44.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.037 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:44.038 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:44.038 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.038 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007833", 2022-09-02T05:01:44.038 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:44.039 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:44.039 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:44.039 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.838740Z" 2022-09-02T05:01:44.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:44.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.040 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:44.040 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:44.041 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.041 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:44.041 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:44.041 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:44.041 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:44.042 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:31.839767Z" 2022-09-02T05:01:44.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:44.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:44.042 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:44.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.043 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:44.043 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T05:01:44.043 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T05:01:44.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.044 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:44Z" 2022-09-02T05:01:44.044 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:44.045 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 2207523797324220113 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.836581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006141", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.837696Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007833", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.838740Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:31.839767Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:44Z" } 2022-09-02T05:01:44.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.836581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.837696Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007833', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.838740Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:31.839767Z'}}] 2022-09-02T05:01:44.047 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:44.047 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003750', 1: '00000000000000000000:00000000000000006141', 2: '00000000000000000000:00000000000000007833', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:44.047 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-09-02T05:01:44.048 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2022-09-02T05:01:44.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:44.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:44.050 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:44.099 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:44.099 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:44.148 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T05:01:44.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.149 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:44.149 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.089751Z" 2022-09-02T05:01:44.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.150 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006846", 2022-09-02T05:01:44.150 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.091331Z" 2022-09-02T05:01:44.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.151 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007975", 2022-09-02T05:01:44.151 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.092680Z" 2022-09-02T05:01:44.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:44.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:44.152 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:44.152 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.094015Z" 2022-09-02T05:01:44.153 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:44.153 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T05:01:44.154 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:44.155 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-02T05:01:44.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:44.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:44.157 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:44.209 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:44.209 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:44.269 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:44.270 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:44.270 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:44.271 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:44.271 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:44.271 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T05:01:44.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:44.272 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:44.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:44.272 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:44.272 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:44.273 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.273 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T05:01:44.273 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:44.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:44.274 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:44.274 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T05:01:44.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:44.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:44.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:44.275 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005586", 2022-09-02T05:01:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:51:38.749572Z" 2022-09-02T05:01:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:44.279 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:44.279 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:44.279 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.280 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006715", 2022-09-02T05:01:44.280 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:44.280 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:44.280 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:44.281 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:55:12.493357Z" 2022-09-02T05:01:44.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:44.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:44.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:44.282 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:44.282 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T05:01:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:38.798708Z" 2022-09-02T05:01:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T05:01:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T05:01:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:44Z" 2022-09-02T05:01:44.287 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:44.288 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005586", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:38.749572Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006715", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:55:12.493357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:38.798708Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:44Z" } 2022-09-02T05:01:44.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005586', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:38.749572Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006715', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:55:12.493357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:38.798708Z'}}] 2022-09-02T05:01:44.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:44.290 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000006715', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T05:01:44.290 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000003327 target=00000000000000000000:00000000000000002486, shard=1 master=00000000000000000000:00000000000000006846 target=00000000000000000000:00000000000000005586, shard=2 master=00000000000000000000:00000000000000007975 target=00000000000000000000:00000000000000006715, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-02T05:01:49.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:49.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:49.294 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:49.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:49.354 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:49.412 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:49.412 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:49.412 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:49.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:49.413 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:49.413 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T05:01:49.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:49.414 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:49.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:49.414 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:49.415 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:49.415 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:49.415 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2022-09-02T05:01:49.415 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:49.416 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:49.416 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:49.416 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:30:11.070047Z" 2022-09-02T05:01:49.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:49.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:49.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:49.417 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:49.418 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:49.418 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:49.418 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005586", 2022-09-02T05:01:49.418 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:49.419 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:49.419 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:49.419 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:51:38.749572Z" 2022-09-02T05:01:49.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:49.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:49.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:49.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006715", 2022-09-02T05:01:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:55:12.493357Z" 2022-09-02T05:01:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000796", 2022-09-02T05:01:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:50:38.798708Z" 2022-09-02T05:01:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T05:01:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T05:01:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:49Z" 2022-09-02T05:01:49.428 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:49.429 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:30:11.070047Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005586", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:51:38.749572Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006715", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:55:12.493357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000796", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:50:38.798708Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:49Z" } 2022-09-02T05:01:49.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:30:11.070047Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005586', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:51:38.749572Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006715', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:55:12.493357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000796', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:50:38.798708Z'}}] 2022-09-02T05:01:49.430 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:49.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002486', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000006715', 3: '00000000000000000000:00000000000000000796'} 2022-09-02T05:01:49.431 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=00000000000000000000:00000000000000003327 target=00000000000000000000:00000000000000002486, shard=1 master=00000000000000000000:00000000000000006846 target=00000000000000000000:00000000000000005586, shard=2 master=00000000000000000000:00000000000000007975 target=00000000000000000000:00000000000000006715, shard=3 master=00000000000000000000:00000000000000002060 target=00000000000000000000:00000000000000000796 2022-09-02T05:01:54.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.434 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:54.494 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:54.494 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:54.543 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:54.543 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:54.544 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:54.544 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:54.544 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:54.544 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 3759133696547852314 2022-09-02T05:01:54.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.545 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:54.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.545 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:54.546 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.546 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.546 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:54.547 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.547 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.547 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.089751Z" 2022-09-02T05:01:54.548 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.548 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:54.549 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.549 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006846", 2022-09-02T05:01:54.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.550 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.091331Z" 2022-09-02T05:01:54.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:54.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007975", 2022-09-02T05:01:54.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.092680Z" 2022-09-02T05:01:54.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:54.560 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.561 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:54.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.562 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.094015Z" 2022-09-02T05:01:54.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.564 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:54.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.565 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:54.565 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T05:01:54.565 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T05:01:54.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.566 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:54Z" 2022-09-02T05:01:54.566 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:54.567 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3759133696547852314 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003327", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.089751Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006846", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.091331Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007975", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.092680Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.094015Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:54Z" } 2022-09-02T05:01:54.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.089751Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006846', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.091331Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007975', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.092680Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.094015Z'}}] 2022-09-02T05:01:54.569 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:54.569 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:54.570 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-09-02T05:01:54.570 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2022-09-02T05:01:54.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.572 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:54.628 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:54.629 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:54.700 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T05:01:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:54.701 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.089751Z" 2022-09-02T05:01:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:54.702 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006846", 2022-09-02T05:01:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.091331Z" 2022-09-02T05:01:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:54.703 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007975", 2022-09-02T05:01:54.704 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.092680Z" 2022-09-02T05:01:54.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:54.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:54.704 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:54.705 INFO:teuthology.orchestra.run.smithi160.stdout: "last_update": "2022-09-02T05:01:36.094015Z" 2022-09-02T05:01:54.705 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:54.705 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T05:01:54.706 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:54.707 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-02T05:01:54.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:54.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:54.763 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:54.828 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:54.828 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:54.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:54.830 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1900644279603854970 2022-09-02T05:01:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.831 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.833 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:54.834 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.835 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.089751Z" 2022-09-02T05:01:54.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:54.838 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.839 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006846", 2022-09-02T05:01:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.091331Z" 2022-09-02T05:01:54.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:54.843 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.844 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.844 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007975", 2022-09-02T05:01:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.845 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.846 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.092680Z" 2022-09-02T05:01:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.848 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:54.849 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:54.850 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:54.851 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:36.094015Z" 2022-09-02T05:01:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:54.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:54.854 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T05:01:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T05:01:54.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.856 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:54Z" 2022-09-02T05:01:54.856 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:54.858 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1900644279603854970 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003327", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.089751Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006846", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.091331Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007975", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.092680Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:36.094015Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:54Z" } 2022-09-02T05:01:54.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.089751Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006846', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.091331Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007975', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.092680Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:36.094015Z'}}] 2022-09-02T05:01:54.860 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:54.861 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006846', 2: '00000000000000000000:00000000000000007975', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:54.862 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-09-02T05:01:54.863 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone4 2022-09-02T05:01:54.864 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2022-09-02T05:01:54.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:54.867 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:54.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:54.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:54.994 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:54.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:54.995 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.792990Z" 2022-09-02T05:01:54.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.995 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006280", 2022-09-02T05:01:54.996 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.796167Z" 2022-09-02T05:01:54.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007129", 2022-09-02T05:01:54.997 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.802058Z" 2022-09-02T05:01:54.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:54.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:54.997 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:54.998 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.807756Z" 2022-09-02T05:01:54.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:54.998 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:54.999 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006280', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:55.000 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-02T05:01:55.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:55.053 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:55.053 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:55.101 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:01:55.101 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:01:55.102 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:01:55.102 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:01:55.102 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:01:55.103 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 10070206082575430316 2022-09-02T05:01:55.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.103 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:01:55.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.104 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:01:55.104 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:55.104 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.104 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:55.105 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:55.105 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:55.105 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:55.105 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:38.792990Z" 2022-09-02T05:01:55.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:55.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:01:55.107 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:55.107 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.107 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006280", 2022-09-02T05:01:55.107 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:55.108 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:55.108 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:55.108 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:38.796167Z" 2022-09-02T05:01:55.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:55.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.109 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:01:55.109 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:55.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.110 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007129", 2022-09-02T05:01:55.110 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:55.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:55.111 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:55.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:38.802058Z" 2022-09-02T05:01:55.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:55.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:01:55.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:01:55.113 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:55.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:01:55.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:01:55.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:01:55.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:38.807756Z" 2022-09-02T05:01:55.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:55.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:55.115 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:01:55.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.115 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:01:55.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2022-09-02T05:01:55.116 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2022-09-02T05:01:55.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.116 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:01:55Z" 2022-09-02T05:01:55.116 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:01:55.118 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10070206082575430316 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003327", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.792990Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006280", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.796167Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007129", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.802058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.807756Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:55Z" } 2022-09-02T05:01:55.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.792990Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006280', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.796167Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.802058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.807756Z'}}] 2022-09-02T05:01:55.119 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006280', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:55.120 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006280', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:55.120 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-09-02T05:01:55.120 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2022-09-02T05:01:55.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.122 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:55.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:55.172 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:55.244 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:55.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:55.246 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.792990Z" 2022-09-02T05:01:55.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.248 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006280", 2022-09-02T05:01:55.248 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.796167Z" 2022-09-02T05:01:55.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007129", 2022-09-02T05:01:55.249 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.802058Z" 2022-09-02T05:01:55.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:55.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:55.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:55.250 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:38.807756Z" 2022-09-02T05:01:55.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:55.250 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:55.252 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006280', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:55.252 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-02T05:01:55.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.254 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:55.309 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:55.309 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:55.371 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:55.372 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:55.372 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:55.373 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:55.373 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:55.373 INFO:teuthology.orchestra.run.smithi160.stdout: "instance_id": 5969670707592890178 2022-09-02T05:01:55.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:55.374 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:55.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:55.374 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:55.375 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:55.375 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.375 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000003327", 2022-09-02T05:01:55.375 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:55.376 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:55.376 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:55.376 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:38.792990Z" 2022-09-02T05:01:55.376 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:55.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:55.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:55.377 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:55.377 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:55.378 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.378 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000006280", 2022-09-02T05:01:55.378 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:55.379 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:55.379 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:55.379 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:38.796167Z" 2022-09-02T05:01:55.379 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:55.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:55.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:55.380 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:55.380 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:55.381 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.381 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000007129", 2022-09-02T05:01:55.381 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:55.382 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:55.382 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:55.383 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:38.802058Z" 2022-09-02T05:01:55.383 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:55.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:55.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:55.384 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:55.385 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:55.385 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "incremental-sync", 2022-09-02T05:01:55.385 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "00000000000000000000:00000000000000002060", 2022-09-02T05:01:55.385 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:55.386 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:55.386 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:55.386 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:38.807756Z" 2022-09-02T05:01:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:55.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 0, 2022-09-02T05:01:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 0 2022-09-02T05:01:55.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:55.389 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:55Z" 2022-09-02T05:01:55.389 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:55.390 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5969670707592890178 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003327", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.792990Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006280", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.796167Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007129", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.802058Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:38.807756Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2022-09-02T05:01:55Z" } 2022-09-02T05:01:55.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003327', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.792990Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006280', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.796167Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.802058Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:38.807756Z'}}] 2022-09-02T05:01:55.391 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006280', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:55.392 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003327', 1: '00000000000000000000:00000000000000006280', 2: '00000000000000000000:00000000000000007129', 3: '00000000000000000000:00000000000000002060'} 2022-09-02T05:01:55.392 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-09-02T05:01:55.393 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone4 2022-09-02T05:01:55.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2022-09-02T05:01:55.397 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-59 2022-09-02T05:01:55.398 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:01:55.398 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:01:55.399 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.400 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:01:55.400 DEBUG:boto:Data: 2022-09-02T05:01:55.401 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.402 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.402 DEBUG:boto:Params: {} 2022-09-02T05:01:55.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:01:55.404 DEBUG:boto:Token: None 2022-09-02T05:01:55.405 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/ 2022-09-02T05:01:55.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nXUG8P9ipD4vmr5DuCY1UngzkIk= 2022-09-02T05:01:55.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nXUG8P9ipD4vmr5DuCY1UngzkIk=', 'Content-Length': '0'} 2022-09-02T05:01:55.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000289e6845cd26618c-0063118e43-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.439 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-60 2022-09-02T05:01:55.439 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:01:55.440 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:01:55.440 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.441 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:01:55.441 DEBUG:boto:Data: 2022-09-02T05:01:55.442 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.443 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.443 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.443 DEBUG:boto:Params: {} 2022-09-02T05:01:55.444 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:01:55.445 DEBUG:boto:Token: None 2022-09-02T05:01:55.445 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/ 2022-09-02T05:01:55.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:stt+JoTp8QNr7g/FNK4eep/byRs= 2022-09-02T05:01:55.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:stt+JoTp8QNr7g/FNK4eep/byRs=', 'Content-Length': '0'} 2022-09-02T05:01:55.482 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d711f80c5d332ac-0063118e43-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.483 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-61 2022-09-02T05:01:55.484 DEBUG:boto:path=/xljbkf-61/ 2022-09-02T05:01:55.484 DEBUG:boto:auth_path=/xljbkf-61/ 2022-09-02T05:01:55.485 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.485 DEBUG:boto:Path: /xljbkf-61/ 2022-09-02T05:01:55.486 DEBUG:boto:Data: 2022-09-02T05:01:55.486 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.487 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.488 DEBUG:boto:Params: {} 2022-09-02T05:01:55.489 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:01:55.489 DEBUG:boto:Token: None 2022-09-02T05:01:55.490 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/ 2022-09-02T05:01:55.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BoL81b4hT1So2wO5o6uWnnOTVtM= 2022-09-02T05:01:55.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BoL81b4hT1So2wO5o6uWnnOTVtM=', 'Content-Length': '0'} 2022-09-02T05:01:55.535 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000be22739a97210f5-0063118e43-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.535 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:01:55.536 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:01:55.537 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.537 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:01:55.538 DEBUG:boto:Data: 2022-09-02T05:01:55.538 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.539 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.540 DEBUG:boto:Params: {} 2022-09-02T05:01:55.540 DEBUG:boto:Token: None 2022-09-02T05:01:55.541 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/ 2022-09-02T05:01:55.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o= 2022-09-02T05:01:55.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o=', 'Content-Length': '0'} 2022-09-02T05:01:55.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918b6099e4bb1b59-0063118e43-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.546 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:01:55.546 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:01:55.547 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.547 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:01:55.548 DEBUG:boto:Data: 2022-09-02T05:01:55.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.550 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.550 DEBUG:boto:Params: {} 2022-09-02T05:01:55.551 DEBUG:boto:Token: None 2022-09-02T05:01:55.551 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/obj1 2022-09-02T05:01:55.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvNX4WhL2lRu7FBHiK5Z0d4wKA4= 2022-09-02T05:01:55.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvNX4WhL2lRu7FBHiK5Z0d4wKA4='} 2022-09-02T05:01:55.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c04420b2f07020cc-0063118e43-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.560 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:01:55.560 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:01:55.561 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.561 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:01:55.562 DEBUG:boto:Data: 2022-09-02T05:01:55.562 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.563 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.564 DEBUG:boto:Params: {} 2022-09-02T05:01:55.564 DEBUG:boto:Token: None 2022-09-02T05:01:55.565 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/ 2022-09-02T05:01:55.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o= 2022-09-02T05:01:55.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o=', 'Content-Length': '0'} 2022-09-02T05:01:55.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae82ead1f00416e5-0063118e43-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.569 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:01:55.570 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:01:55.570 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.571 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:01:55.571 DEBUG:boto:Data: 2022-09-02T05:01:55.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.573 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.573 DEBUG:boto:Params: {} 2022-09-02T05:01:55.574 DEBUG:boto:Token: None 2022-09-02T05:01:55.574 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/obj2 2022-09-02T05:01:55.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vc2BYEXlvmTc3B0TmPam8NL6da4= 2022-09-02T05:01:55.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vc2BYEXlvmTc3B0TmPam8NL6da4='} 2022-09-02T05:01:55.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004480f9d7cd22cba7-0063118e43-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.588 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:01:55.588 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:01:55.589 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.590 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:01:55.590 DEBUG:boto:Data: 2022-09-02T05:01:55.591 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.592 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.592 DEBUG:boto:Params: {} 2022-09-02T05:01:55.593 DEBUG:boto:Token: None 2022-09-02T05:01:55.593 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/ 2022-09-02T05:01:55.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o= 2022-09-02T05:01:55.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o=', 'Content-Length': '0'} 2022-09-02T05:01:55.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000143bbec1ebe2c07b-0063118e43-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.597 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:01:55.598 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:01:55.598 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.599 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:01:55.599 DEBUG:boto:Data: 2022-09-02T05:01:55.600 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.601 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.601 DEBUG:boto:Params: {} 2022-09-02T05:01:55.602 DEBUG:boto:Token: None 2022-09-02T05:01:55.603 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/obj3 2022-09-02T05:01:55.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MC2a+L/nWeCbnoMOQCqNf2P+xBU= 2022-09-02T05:01:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MC2a+L/nWeCbnoMOQCqNf2P+xBU='} 2022-09-02T05:01:55.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000885060d488cb124a-0063118e43-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.614 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:01:55.614 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:01:55.615 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.615 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:01:55.616 DEBUG:boto:Data: 2022-09-02T05:01:55.616 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.617 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.618 DEBUG:boto:Params: {} 2022-09-02T05:01:55.618 DEBUG:boto:Token: None 2022-09-02T05:01:55.619 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/ 2022-09-02T05:01:55.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o= 2022-09-02T05:01:55.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YcKsW5VG6SvrNZRYay+g2ior3/o=', 'Content-Length': '0'} 2022-09-02T05:01:55.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003020f2b545b9bda1-0063118e43-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.623 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:01:55.623 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:01:55.624 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.624 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:01:55.625 DEBUG:boto:Data: 2022-09-02T05:01:55.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:01:55.626 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.627 DEBUG:boto:Params: {} 2022-09-02T05:01:55.627 DEBUG:boto:Token: None 2022-09-02T05:01:55.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-59/obj4 2022-09-02T05:01:55.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nAZ7BZEZt/TjQS1DaBMzhVz0WQ= 2022-09-02T05:01:55.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nAZ7BZEZt/TjQS1DaBMzhVz0WQ='} 2022-09-02T05:01:55.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000caf5b49d66762eb3-0063118e43-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.675 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:01:55.676 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:01:55.676 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.677 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:01:55.677 DEBUG:boto:Data: 2022-09-02T05:01:55.678 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.679 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.679 DEBUG:boto:Params: {} 2022-09-02T05:01:55.680 DEBUG:boto:Token: None 2022-09-02T05:01:55.681 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/ 2022-09-02T05:01:55.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY= 2022-09-02T05:01:55.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY=', 'Content-Length': '0'} 2022-09-02T05:01:55.684 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f669ef017006b79-0063118e43-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.685 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:01:55.686 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:01:55.686 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.687 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:01:55.687 DEBUG:boto:Data: 2022-09-02T05:01:55.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.688 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.689 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.689 DEBUG:boto:Params: {} 2022-09-02T05:01:55.690 DEBUG:boto:Token: None 2022-09-02T05:01:55.691 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/obj1 2022-09-02T05:01:55.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lfWQTVgPOltvreiNZMZxwHGgD1s= 2022-09-02T05:01:55.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lfWQTVgPOltvreiNZMZxwHGgD1s='} 2022-09-02T05:01:55.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b130d1b1337fed1-0063118e43-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.700 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:01:55.701 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:01:55.701 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.702 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:01:55.702 DEBUG:boto:Data: 2022-09-02T05:01:55.703 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.704 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.704 DEBUG:boto:Params: {} 2022-09-02T05:01:55.705 DEBUG:boto:Token: None 2022-09-02T05:01:55.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/ 2022-09-02T05:01:55.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY= 2022-09-02T05:01:55.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY=', 'Content-Length': '0'} 2022-09-02T05:01:55.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009aeabfbbe7fb0aca-0063118e43-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.710 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:01:55.711 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:01:55.712 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.713 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:01:55.714 DEBUG:boto:Data: 2022-09-02T05:01:55.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.717 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.718 DEBUG:boto:Params: {} 2022-09-02T05:01:55.719 DEBUG:boto:Token: None 2022-09-02T05:01:55.720 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/obj2 2022-09-02T05:01:55.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KgZDW3At8jKQ15aGM5JsCBJdPY0= 2022-09-02T05:01:55.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KgZDW3At8jKQ15aGM5JsCBJdPY0='} 2022-09-02T05:01:55.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006de6ebccef6bc93b-0063118e43-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.728 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:01:55.728 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:01:55.729 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.729 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:01:55.730 DEBUG:boto:Data: 2022-09-02T05:01:55.730 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.731 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.732 DEBUG:boto:Params: {} 2022-09-02T05:01:55.732 DEBUG:boto:Token: None 2022-09-02T05:01:55.733 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/ 2022-09-02T05:01:55.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY= 2022-09-02T05:01:55.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY=', 'Content-Length': '0'} 2022-09-02T05:01:55.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007028434f8616dec4-0063118e43-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.737 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:01:55.738 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:01:55.738 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.739 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:01:55.739 DEBUG:boto:Data: 2022-09-02T05:01:55.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.741 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.741 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.742 DEBUG:boto:Params: {} 2022-09-02T05:01:55.742 DEBUG:boto:Token: None 2022-09-02T05:01:55.743 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/obj3 2022-09-02T05:01:55.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+aCaRS3bY8FmhK+v73BJcrygDys= 2022-09-02T05:01:55.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+aCaRS3bY8FmhK+v73BJcrygDys='} 2022-09-02T05:01:55.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038061d31d2b5c337-0063118e43-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.751 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:01:55.751 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:01:55.752 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.753 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:01:55.753 DEBUG:boto:Data: 2022-09-02T05:01:55.754 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.755 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.755 DEBUG:boto:Params: {} 2022-09-02T05:01:55.756 DEBUG:boto:Token: None 2022-09-02T05:01:55.756 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/ 2022-09-02T05:01:55.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY= 2022-09-02T05:01:55.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5h/G8u9k0attE/uQiEXM4UeQkRY=', 'Content-Length': '0'} 2022-09-02T05:01:55.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067388e7ac7439107-0063118e43-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.761 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:01:55.761 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:01:55.762 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.762 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:01:55.763 DEBUG:boto:Data: 2022-09-02T05:01:55.763 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:01:55.764 DEBUG:boto:Port: 8000 2022-09-02T05:01:55.765 DEBUG:boto:Params: {} 2022-09-02T05:01:55.765 DEBUG:boto:Token: None 2022-09-02T05:01:55.766 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-60/obj4 2022-09-02T05:01:55.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffm0u2wK+xTsioWiX9C9i4ypUAo= 2022-09-02T05:01:55.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffm0u2wK+xTsioWiX9C9i4ypUAo='} 2022-09-02T05:01:55.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030be70a70ab2ba91-0063118e43-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.815 DEBUG:boto:path=/xljbkf-61/ 2022-09-02T05:01:55.816 DEBUG:boto:auth_path=/xljbkf-61/ 2022-09-02T05:01:55.816 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.817 DEBUG:boto:Path: /xljbkf-61/ 2022-09-02T05:01:55.818 DEBUG:boto:Data: 2022-09-02T05:01:55.819 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.820 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.820 DEBUG:boto:Params: {} 2022-09-02T05:01:55.821 DEBUG:boto:Token: None 2022-09-02T05:01:55.821 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/ 2022-09-02T05:01:55.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ= 2022-09-02T05:01:55.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ=', 'Content-Length': '0'} 2022-09-02T05:01:55.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000723d9cb60d022c08-0063118e43-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.826 DEBUG:boto:path=/xljbkf-61/obj1 2022-09-02T05:01:55.826 DEBUG:boto:auth_path=/xljbkf-61/obj1 2022-09-02T05:01:55.827 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.827 DEBUG:boto:Path: /xljbkf-61/obj1 2022-09-02T05:01:55.828 DEBUG:boto:Data: 2022-09-02T05:01:55.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.830 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.831 DEBUG:boto:Params: {} 2022-09-02T05:01:55.832 DEBUG:boto:Token: None 2022-09-02T05:01:55.833 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/obj1 2022-09-02T05:01:55.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KQTdg+UmZv4uWymFKPzBKSluKgQ= 2022-09-02T05:01:55.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KQTdg+UmZv4uWymFKPzBKSluKgQ='} 2022-09-02T05:01:55.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fcf0fe2c6fc74f4-0063118e43-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.844 DEBUG:boto:path=/xljbkf-61/ 2022-09-02T05:01:55.845 DEBUG:boto:auth_path=/xljbkf-61/ 2022-09-02T05:01:55.846 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.847 DEBUG:boto:Path: /xljbkf-61/ 2022-09-02T05:01:55.848 DEBUG:boto:Data: 2022-09-02T05:01:55.849 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.851 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.851 DEBUG:boto:Params: {} 2022-09-02T05:01:55.852 DEBUG:boto:Token: None 2022-09-02T05:01:55.853 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/ 2022-09-02T05:01:55.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ= 2022-09-02T05:01:55.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ=', 'Content-Length': '0'} 2022-09-02T05:01:55.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004011e7a52eb7ad02-0063118e43-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.859 DEBUG:boto:path=/xljbkf-61/obj2 2022-09-02T05:01:55.860 DEBUG:boto:auth_path=/xljbkf-61/obj2 2022-09-02T05:01:55.861 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.862 DEBUG:boto:Path: /xljbkf-61/obj2 2022-09-02T05:01:55.863 DEBUG:boto:Data: 2022-09-02T05:01:55.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.864 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.865 DEBUG:boto:Params: {} 2022-09-02T05:01:55.866 DEBUG:boto:Token: None 2022-09-02T05:01:55.866 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/obj2 2022-09-02T05:01:55.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eFfyko7/4hAX6ZXWo2csKc+ggv4= 2022-09-02T05:01:55.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eFfyko7/4hAX6ZXWo2csKc+ggv4='} 2022-09-02T05:01:55.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5664c0ac87f6a08-0063118e43-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.879 DEBUG:boto:path=/xljbkf-61/ 2022-09-02T05:01:55.880 DEBUG:boto:auth_path=/xljbkf-61/ 2022-09-02T05:01:55.881 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.881 DEBUG:boto:Path: /xljbkf-61/ 2022-09-02T05:01:55.882 DEBUG:boto:Data: 2022-09-02T05:01:55.882 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.883 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.884 DEBUG:boto:Params: {} 2022-09-02T05:01:55.884 DEBUG:boto:Token: None 2022-09-02T05:01:55.885 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/ 2022-09-02T05:01:55.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ= 2022-09-02T05:01:55.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ=', 'Content-Length': '0'} 2022-09-02T05:01:55.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b47ed76514e5b886-0063118e43-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.889 DEBUG:boto:path=/xljbkf-61/obj3 2022-09-02T05:01:55.890 DEBUG:boto:auth_path=/xljbkf-61/obj3 2022-09-02T05:01:55.890 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.891 DEBUG:boto:Path: /xljbkf-61/obj3 2022-09-02T05:01:55.891 DEBUG:boto:Data: 2022-09-02T05:01:55.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.893 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.893 DEBUG:boto:Params: {} 2022-09-02T05:01:55.894 DEBUG:boto:Token: None 2022-09-02T05:01:55.894 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/obj3 2022-09-02T05:01:55.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mw9XWvz+rwwd2flNB9T3PKtTrTk= 2022-09-02T05:01:55.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mw9XWvz+rwwd2flNB9T3PKtTrTk='} 2022-09-02T05:01:55.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000365703893bf71d48-0063118e43-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.907 DEBUG:boto:path=/xljbkf-61/ 2022-09-02T05:01:55.907 DEBUG:boto:auth_path=/xljbkf-61/ 2022-09-02T05:01:55.908 DEBUG:boto:Method: HEAD 2022-09-02T05:01:55.909 DEBUG:boto:Path: /xljbkf-61/ 2022-09-02T05:01:55.909 DEBUG:boto:Data: 2022-09-02T05:01:55.910 DEBUG:boto:Headers: {} 2022-09-02T05:01:55.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.911 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.912 DEBUG:boto:Params: {} 2022-09-02T05:01:55.912 DEBUG:boto:Token: None 2022-09-02T05:01:55.913 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/ 2022-09-02T05:01:55.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ= 2022-09-02T05:01:55.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JHkX8W0ZaKm/xrLaLfDOO1hLJgQ=', 'Content-Length': '0'} 2022-09-02T05:01:55.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000673660722913e910-0063118e43-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.918 DEBUG:boto:path=/xljbkf-61/obj4 2022-09-02T05:01:55.919 DEBUG:boto:auth_path=/xljbkf-61/obj4 2022-09-02T05:01:55.919 DEBUG:boto:Method: PUT 2022-09-02T05:01:55.920 DEBUG:boto:Path: /xljbkf-61/obj4 2022-09-02T05:01:55.921 DEBUG:boto:Data: 2022-09-02T05:01:55.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:01:55.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:01:55.923 DEBUG:boto:Port: 8001 2022-09-02T05:01:55.923 DEBUG:boto:Params: {} 2022-09-02T05:01:55.924 DEBUG:boto:Token: None 2022-09-02T05:01:55.925 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:01:55 GMT /xljbkf-61/obj4 2022-09-02T05:01:55.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MUj7jxWZe8YyMLj3T/HBEJRbPJw= 2022-09-02T05:01:55.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:01:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MUj7jxWZe8YyMLj3T/HBEJRbPJw='} 2022-09-02T05:01:55.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cec344ffb0dd781-0063118e43-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:01:55.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:55.976 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:56.027 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:56.027 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:56.082 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:01:56.105 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:01:56.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:56.106 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:01:56.106 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:01:56.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:56.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:56.108 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:01:56.108 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:01:56.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:56.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:56.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094915.519565_303.1", 2022-09-02T05:01:56.109 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:01:55.519565Z" 2022-09-02T05:01:56.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:01:56.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:01:56.110 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:01:56.111 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:01:56.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:01:56.111 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:01:56.113 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:01:56.113 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:01:56.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:56.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:01:56.116 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:01:56.166 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:01:56.167 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:01:56.210 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:56.211+0000 7f0f23f10600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:56.211 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:01:56.212+0000 7f0f23f10600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:01:56.235 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:01:56.235 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:01:56.236 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:01:56.236 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:01:56.236 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:01:56.236 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:01:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:01:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:56.238 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:01:56.238 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:56.238 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:56.239 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:01:56.239 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:56.239 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:01:56.239 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:56.240 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:01:56.240 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:56.240 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:56.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:56.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:56.241 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:01:56.241 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:56.241 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:56.242 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:01:56.242 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:56.242 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:56.243 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:56.243 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:01:56.243 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:56.244 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:56.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:56.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:56.244 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:01:56.245 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:56.245 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:56.245 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094915.506562_300.1", 2022-09-02T05:01:56.245 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:56.246 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:56.246 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:56.246 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:55.506562Z", 2022-09-02T05:01:56.246 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:01:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:01:56.247 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:01:56.248 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:01:56.248 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:01:56.249 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:01:56.249 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:01:56.249 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:01:56.250 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:01:56.250 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:01:56.250 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:01:56.251 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:56.251 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:01:56.251 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:01:56.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:56.252 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:01:56.252 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:01:56.252 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:01:56.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:01:56.253 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:01:56Z" 2022-09-02T05:01:56.253 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:01:56.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094915.506562_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:55.506562Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:01:56Z" } 2022-09-02T05:01:56.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094915.506562_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:55.506562Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:01:56.263 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:01:56.264 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094915.506562_300.1', 3: ''} 2022-09-02T05:01:56.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662094915.519565_303.1 target=1_1662094915.506562_300.1 2022-09-02T05:02:01.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:02:01.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:02:01.268 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:02:01.325 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:01.326 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:01.382 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:01.382+0000 7f63fde2c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:02:01.382 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:01.384+0000 7f63fde2c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:02:01.411 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:02:01.411 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:02:01.412 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:02:01.412 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:02:01.412 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:02:01.413 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:02:01.413 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:02:01.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.413 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:02:01.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.414 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:02:01.414 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.414 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.415 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:02:01.415 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.415 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:02:01.415 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.416 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:02:01.416 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:02:01.416 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.417 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:02:01.417 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.417 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.418 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:02:01.418 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.418 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:02:01.418 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.419 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:02:01.419 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:02:01.419 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.420 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:02:01.420 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.420 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.421 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094915.519565_303.1", 2022-09-02T05:02:01.421 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.421 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:02:01.421 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.422 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:55.519565Z", 2022-09-02T05:02:01.422 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:02:01.422 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.423 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:02:01.423 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.423 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.424 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:02:01.424 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.424 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:02:01.424 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.425 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:02:01.425 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:02:01.425 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.425 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.426 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:02:01.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.426 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:02:01.426 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:02:01.427 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:02:01.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.427 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:02:01Z" 2022-09-02T05:02:01.427 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:02:01.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094915.519565_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:55.519565Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:02:01Z" } 2022-09-02T05:02:01.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094915.519565_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:55.519565Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:02:01.430 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:02:01.431 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:02:01.432 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:02:01.432 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:02:01.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:02:01.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:02:01.435 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:02:01.497 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:01.498 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:01.529 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:01.530+0000 7f01c423d600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:02:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:01.532+0000 7f01c423d600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:02:01.544 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:02:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:02:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:02:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:02:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:02:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:02:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:02:01.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:02:01.547 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:02:01.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:02:01.547 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:02:01.548 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:02:01.548 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:02:01.548 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:02:01.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:02:01.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:02:01.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:02:01.549 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:02:01.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:02:01.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:02:01.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:02:01.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:02:01.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:02:01.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:02:01.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:02:01.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:02:01.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:02:01.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:02:01.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:02:01.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:02:01.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:02:01.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:02:01.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:02:01.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:02:01.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:02:01.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:02:01.555 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:02:01.555 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662094915.519565_303.1", 2022-09-02T05:02:01.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:02:01.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:02:01.556 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:02:01.556 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:01:55.519565Z", 2022-09-02T05:02:01.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:02:01.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:02:01.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:02:01.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:02:01.557 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:02:01.558 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:02:01.558 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:02:01.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:02:01.558 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:02:01.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:02:01.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:02:01.559 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:02:01.559 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:02:01.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:02:01.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:02:01.560 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:02:01.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:02:01.561 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:02:01.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:02:01.561 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:02:01.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:02:01.562 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:02:01Z" 2022-09-02T05:02:01.562 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:02:01.563 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094915.519565_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:55.519565Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:02:01Z" } 2022-09-02T05:02:01.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094915.519565_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:55.519565Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:02:01.565 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:02:01.565 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:02:01.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:02:01.566 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:02:01.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:02:01.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:02:01.568 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:02:01.618 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:01.619 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:01.658 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:01.659+0000 7f42c5d16600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:02:01.659 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:01.660+0000 7f42c5d16600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:02:01.671 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:02:01.671 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:02:01.672 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:02:01.672 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:02:01.672 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:02:01.672 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:02:01.673 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:02:01.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.673 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:02:01.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.674 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:02:01.674 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.674 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.675 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:02:01.675 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.675 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:02:01.676 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.676 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:02:01.676 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:02:01.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.677 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:02:01.677 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.678 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.678 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:02:01.678 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.679 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:02:01.679 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.679 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:02:01.679 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:02:01.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.681 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:02:01.681 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.681 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.681 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662094915.519565_303.1", 2022-09-02T05:02:01.682 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.682 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:02:01.682 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.682 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:01:55.519565Z", 2022-09-02T05:02:01.683 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:02:01.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:02:01.684 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:02:01.684 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:02:01.684 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:02:01.685 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:02:01.685 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:02:01.685 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:02:01.685 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:02:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:02:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:02:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:02:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:02:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.687 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:02:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:02:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:02:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:02:01.688 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:02:01Z" 2022-09-02T05:02:01.689 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:02:01.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662094915.519565_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:01:55.519565Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:02:01Z" } 2022-09-02T05:02:01.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662094915.519565_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:01:55.519565Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:02:01.691 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:02:01.692 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662094915.519565_303.1', 3: ''} 2022-09-02T05:02:01.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:02:01.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-59 2022-09-02T05:02:01.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:01.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:01.695 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:01.764 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:01.764 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:01.811 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:01.812+0000 7f6d987e9600 1 bucket sync caught up with source: 2022-09-02T05:02:01.812 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.46.5, , 00000000001.40.5, , 00000000001.30.5, , , 00000000001.34.5, , , ] 2022-09-02T05:02:01.812 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.46.5, , 00000000001.40.5, , 00000000001.30.5, , , 00000000001.34.5, , , ] 2022-09-02T05:02:01.812 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:01.812+0000 7f6d987e9600 0 bucket checkpoint complete 2022-09-02T05:02:01.832 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-59 2022-09-02T05:02:01.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:01.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:01.834 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:01.894 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:01.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:01.950+0000 7fb9a9f62600 1 bucket sync caught up with source: 2022-09-02T05:02:01.949 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.46.5, , 00000000001.40.5, , 00000000001.30.5, , , 00000000001.34.5, , , ] 2022-09-02T05:02:01.950 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.46.5, , 00000000001.40.5, , 00000000001.30.5, , , 00000000001.34.5, , , ] 2022-09-02T05:02:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:01.950+0000 7fb9a9f62600 0 bucket checkpoint complete 2022-09-02T05:02:01.961 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-59 2022-09-02T05:02:01.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:01.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:01.963 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:02.019 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:02.020 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:02.062 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:02.063+0000 7fa7b547c600 1 bucket sync caught up with source: 2022-09-02T05:02:02.062 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.46.5, , 00000000001.40.5, , 00000000001.30.5, , , 00000000001.34.5, , , ] 2022-09-02T05:02:02.063 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.46.5, , 00000000001.40.5, , 00000000001.30.5, , , 00000000001.34.5, , , ] 2022-09-02T05:02:02.063 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:02.063+0000 7fa7b547c600 0 bucket checkpoint complete 2022-09-02T05:02:02.081 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-59 2022-09-02T05:02:02.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:02.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:02.082 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:02.177+0000 7f236d8fd600 1 waiting to reach incremental sync.. 2022-09-02T05:02:07.176 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:07.177+0000 7f236d8fd600 1 waiting to reach incremental sync.. 2022-09-02T05:02:12.178 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:12.179+0000 7f236d8fd600 1 bucket sync caught up with source: 2022-09-02T05:02:12.178 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1038.5, , 00000000001.687.5, , 00000000001.648.5, , , 00000000001.866.5, , , ] 2022-09-02T05:02:12.179 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.1038.5, , 00000000001.687.5, , 00000000001.648.5, , , 00000000001.866.5, , , ] 2022-09-02T05:02:12.179 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:12.179+0000 7f236d8fd600 0 bucket checkpoint complete 2022-09-02T05:02:12.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-59 2022-09-02T05:02:12.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:12.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:12.201 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:12.302 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:12.303+0000 7f70f7ac8600 1 bucket sync caught up with source: 2022-09-02T05:02:12.302 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1038.5, , 00000000001.687.5, , 00000000001.648.5, , , 00000000001.866.5, , , ] 2022-09-02T05:02:12.302 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.1038.5, , 00000000001.687.5, , 00000000001.648.5, , , 00000000001.866.5, , , ] 2022-09-02T05:02:12.303 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:12.303+0000 7f70f7ac8600 0 bucket checkpoint complete 2022-09-02T05:02:12.314 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-59 2022-09-02T05:02:12.315 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-59 2022-09-02T05:02:12.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:12.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:12.316 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:12.409 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:12.411+0000 7fb4ec552600 1 bucket sync caught up with source: 2022-09-02T05:02:12.410 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.146.5, , 00000000001.240.5, , 00000000001.184.5, , , 00000000001.298.5, , , ] 2022-09-02T05:02:12.410 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.146.5, , 00000000001.240.5, , 00000000001.184.5, , , 00000000001.298.5, , , ] 2022-09-02T05:02:12.410 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:12.411+0000 7fb4ec552600 0 bucket checkpoint complete 2022-09-02T05:02:12.425 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-59 2022-09-02T05:02:12.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:12.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:12.427 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:12.492 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:12.492 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:12.531 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:12.532+0000 7f36f8172600 1 bucket sync caught up with source: 2022-09-02T05:02:12.532 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.146.5, , 00000000001.240.5, , 00000000001.184.5, , , 00000000001.298.5, , , ] 2022-09-02T05:02:12.532 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.146.5, , 00000000001.240.5, , 00000000001.184.5, , , 00000000001.298.5, , , ] 2022-09-02T05:02:12.532 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:12.532+0000 7f36f8172600 0 bucket checkpoint complete 2022-09-02T05:02:12.549 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-59 2022-09-02T05:02:12.549 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-59 zones={test-zone2, test-zone1} 2022-09-02T05:02:12.550 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.551 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.551 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.552 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:02:12.552 DEBUG:boto:Data: 2022-09-02T05:02:12.553 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.553 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.554 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.554 DEBUG:boto:Params: {} 2022-09-02T05:02:12.555 DEBUG:boto:Token: None 2022-09-02T05:02:12.556 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T3Pk0xGGJLSmLHr2CfwZ3hAUFyY= 2022-09-02T05:02:12.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T3Pk0xGGJLSmLHr2CfwZ3hAUFyY=', 'Content-Length': '0'} 2022-09-02T05:02:12.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000087f6c1a2926d7e1-0063118e54-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.560 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.561 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.561 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.562 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:02:12.563 DEBUG:boto:Data: 2022-09-02T05:02:12.563 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.564 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.565 DEBUG:boto:Params: {} 2022-09-02T05:02:12.565 DEBUG:boto:Token: None 2022-09-02T05:02:12.566 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T3Pk0xGGJLSmLHr2CfwZ3hAUFyY= 2022-09-02T05:02:12.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T3Pk0xGGJLSmLHr2CfwZ3hAUFyY=', 'Content-Length': '0'} 2022-09-02T05:02:12.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9bc57fdf17b24b7-0063118e54-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.570 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:02:12.571 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.571 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.572 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.572 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.573 DEBUG:boto:Method: GET 2022-09-02T05:02:12.573 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.574 DEBUG:boto:Data: 2022-09-02T05:02:12.574 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.575 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.575 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.576 DEBUG:boto:Params: {} 2022-09-02T05:02:12.576 DEBUG:boto:Token: None 2022-09-02T05:02:12.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/?versions 2022-09-02T05:02:12.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ= 2022-09-02T05:02:12.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ=', 'Content-Length': '0'} 2022-09-02T05:02:12.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c41c4da12c07e5c-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.581 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:02:12.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:02:12.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:02:12.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:02:12.585 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:02:12.585 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.586 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.587 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.587 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.588 DEBUG:boto:Method: GET 2022-09-02T05:02:12.588 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.589 DEBUG:boto:Data: 2022-09-02T05:02:12.589 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.590 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.591 DEBUG:boto:Params: {} 2022-09-02T05:02:12.591 DEBUG:boto:Token: None 2022-09-02T05:02:12.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/?versions 2022-09-02T05:02:12.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ= 2022-09-02T05:02:12.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ=', 'Content-Length': '0'} 2022-09-02T05:02:12.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072afe8842ad68009-0063118e54-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.596 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:02:12.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:02:12.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:02:12.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:02:12.600 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.600 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.601 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.601 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.602 DEBUG:boto:Method: GET 2022-09-02T05:02:12.602 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.603 DEBUG:boto:Data: 2022-09-02T05:02:12.603 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.604 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.605 DEBUG:boto:Params: {} 2022-09-02T05:02:12.606 DEBUG:boto:Token: None 2022-09-02T05:02:12.606 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/?versions 2022-09-02T05:02:12.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ= 2022-09-02T05:02:12.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ=', 'Content-Length': '0'} 2022-09-02T05:02:12.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036fbf81dd4cdce5e-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.611 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.611 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.612 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.612 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.613 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.613 DEBUG:boto:Method: GET 2022-09-02T05:02:12.614 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:12.615 DEBUG:boto:Data: 2022-09-02T05:02:12.615 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.616 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.617 DEBUG:boto:Params: {} 2022-09-02T05:02:12.617 DEBUG:boto:Token: None 2022-09-02T05:02:12.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/?versions 2022-09-02T05:02:12.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ= 2022-09-02T05:02:12.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kWTGUWZ8bFnjQ7jGop6wN/18QeQ=', 'Content-Length': '0'} 2022-09-02T05:02:12.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aba3a25c5965017c-0063118e54-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.622 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:12.624 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.624 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.625 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.625 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.626 DEBUG:boto:Method: GET 2022-09-02T05:02:12.626 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.627 DEBUG:boto:Data: 2022-09-02T05:02:12.628 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.629 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.629 DEBUG:boto:Params: {} 2022-09-02T05:02:12.630 DEBUG:boto:Token: None 2022-09-02T05:02:12.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I= 2022-09-02T05:02:12.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I=', 'Content-Length': '0'} 2022-09-02T05:02:12.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ef7781795ff6d8a-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.634 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.635 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.636 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.636 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.637 DEBUG:boto:Method: GET 2022-09-02T05:02:12.637 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.638 DEBUG:boto:Data: 2022-09-02T05:02:12.638 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.640 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.640 DEBUG:boto:Params: {} 2022-09-02T05:02:12.641 DEBUG:boto:Token: None 2022-09-02T05:02:12.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I= 2022-09-02T05:02:12.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I=', 'Content-Length': '0'} 2022-09-02T05:02:12.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040eaaf2e5b7c767a-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.645 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.646 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.646 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.647 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.647 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.648 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.648 DEBUG:boto:Data: 2022-09-02T05:02:12.649 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.650 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.651 DEBUG:boto:Params: {} 2022-09-02T05:02:12.651 DEBUG:boto:Token: None 2022-09-02T05:02:12.652 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XwgMJ7VkPzyklls9uapd7LLOvKE= 2022-09-02T05:02:12.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XwgMJ7VkPzyklls9uapd7LLOvKE=', 'Content-Length': '0'} 2022-09-02T05:02:12.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032090159924be6fd-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.655 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.656 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.657 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.657 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.658 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.658 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.659 DEBUG:boto:Data: 2022-09-02T05:02:12.659 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.660 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.661 DEBUG:boto:Params: {} 2022-09-02T05:02:12.661 DEBUG:boto:Token: None 2022-09-02T05:02:12.662 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XwgMJ7VkPzyklls9uapd7LLOvKE= 2022-09-02T05:02:12.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XwgMJ7VkPzyklls9uapd7LLOvKE=', 'Content-Length': '0'} 2022-09-02T05:02:12.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004bdf21591c9413a-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:12.666 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.667 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.667 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.668 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.668 DEBUG:boto:Method: GET 2022-09-02T05:02:12.669 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.670 DEBUG:boto:Data: 2022-09-02T05:02:12.670 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.671 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.672 DEBUG:boto:Params: {} 2022-09-02T05:02:12.672 DEBUG:boto:Token: None 2022-09-02T05:02:12.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I= 2022-09-02T05:02:12.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I=', 'Content-Length': '0'} 2022-09-02T05:02:12.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009343bf943ca0c106-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.677 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.678 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.678 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.679 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.679 DEBUG:boto:Method: GET 2022-09-02T05:02:12.680 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.681 DEBUG:boto:Data: 2022-09-02T05:02:12.681 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.682 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.683 DEBUG:boto:Params: {} 2022-09-02T05:02:12.683 DEBUG:boto:Token: None 2022-09-02T05:02:12.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:12.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I= 2022-09-02T05:02:12.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DObuK2hCWzYPyPpTZiLnmOCTU1I=', 'Content-Length': '0'} 2022-09-02T05:02:12.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c66a73f5d438e4a-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.688 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.688 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.689 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.689 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.690 DEBUG:boto:Method: GET 2022-09-02T05:02:12.691 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.691 DEBUG:boto:Data: 2022-09-02T05:02:12.692 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.693 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.693 DEBUG:boto:Params: {} 2022-09-02T05:02:12.694 DEBUG:boto:Token: None 2022-09-02T05:02:12.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cf058491aa8ea25-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.705 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.706 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.706 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.707 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.707 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.708 DEBUG:boto:Method: GET 2022-09-02T05:02:12.708 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.709 DEBUG:boto:Data: 2022-09-02T05:02:12.709 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.710 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.711 DEBUG:boto:Params: {} 2022-09-02T05:02:12.711 DEBUG:boto:Token: None 2022-09-02T05:02:12.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ed9e4f94b293419-0063118e54-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.716 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.717 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.718 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.718 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.719 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:12.719 DEBUG:boto:Data: 2022-09-02T05:02:12.720 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.721 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.721 DEBUG:boto:Params: {} 2022-09-02T05:02:12.722 DEBUG:boto:Token: None 2022-09-02T05:02:12.723 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1 2022-09-02T05:02:12.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZdNLDNXEB0xeeRGgvXMvILI9nVQ= 2022-09-02T05:02:12.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZdNLDNXEB0xeeRGgvXMvILI9nVQ=', 'Content-Length': '0'} 2022-09-02T05:02:12.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abff466758d40e08-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.726 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.727 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.727 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.728 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:12.728 DEBUG:boto:Data: 2022-09-02T05:02:12.729 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.730 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.730 DEBUG:boto:Params: {} 2022-09-02T05:02:12.731 DEBUG:boto:Token: None 2022-09-02T05:02:12.732 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1 2022-09-02T05:02:12.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZdNLDNXEB0xeeRGgvXMvILI9nVQ= 2022-09-02T05:02:12.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZdNLDNXEB0xeeRGgvXMvILI9nVQ=', 'Content-Length': '0'} 2022-09-02T05:02:12.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000771ee70b3f2b6e41-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:12.736 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.736 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.737 DEBUG:boto:Method: GET 2022-09-02T05:02:12.737 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:12.738 DEBUG:boto:Data: 2022-09-02T05:02:12.738 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.740 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.740 DEBUG:boto:Params: {} 2022-09-02T05:02:12.741 DEBUG:boto:Token: None 2022-09-02T05:02:12.741 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1 2022-09-02T05:02:12.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ceCDrFDW0cd/Rrbj2ndoV0x/IZg= 2022-09-02T05:02:12.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ceCDrFDW0cd/Rrbj2ndoV0x/IZg=', 'Content-Length': '0'} 2022-09-02T05:02:12.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072beb231c050d665-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.745 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:12.746 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:12.746 DEBUG:boto:Method: GET 2022-09-02T05:02:12.747 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:12.747 DEBUG:boto:Data: 2022-09-02T05:02:12.748 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.749 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.750 DEBUG:boto:Params: {} 2022-09-02T05:02:12.750 DEBUG:boto:Token: None 2022-09-02T05:02:12.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj1 2022-09-02T05:02:12.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ceCDrFDW0cd/Rrbj2ndoV0x/IZg= 2022-09-02T05:02:12.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ceCDrFDW0cd/Rrbj2ndoV0x/IZg=', 'Content-Length': '0'} 2022-09-02T05:02:12.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d555b757879c311b-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.755 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.755 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.756 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.757 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.757 DEBUG:boto:Method: GET 2022-09-02T05:02:12.758 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.758 DEBUG:boto:Data: 2022-09-02T05:02:12.759 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.760 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.760 DEBUG:boto:Params: {} 2022-09-02T05:02:12.761 DEBUG:boto:Token: None 2022-09-02T05:02:12.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbb1727dcee50442-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.766 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.767 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.767 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.768 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.768 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.769 DEBUG:boto:Method: GET 2022-09-02T05:02:12.769 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:12.770 DEBUG:boto:Data: 2022-09-02T05:02:12.770 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.772 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.772 DEBUG:boto:Params: {} 2022-09-02T05:02:12.773 DEBUG:boto:Token: None 2022-09-02T05:02:12.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4bf2496aefcf14d-0063118e54-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.778 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:12.779 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.780 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.780 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.781 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.781 DEBUG:boto:Method: GET 2022-09-02T05:02:12.782 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.782 DEBUG:boto:Data: 2022-09-02T05:02:12.783 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.784 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.785 DEBUG:boto:Params: {} 2022-09-02T05:02:12.785 DEBUG:boto:Token: None 2022-09-02T05:02:12.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q= 2022-09-02T05:02:12.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q=', 'Content-Length': '0'} 2022-09-02T05:02:12.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042ed1ca3ae333d01-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.790 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.791 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.791 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.792 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.792 DEBUG:boto:Method: GET 2022-09-02T05:02:12.793 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.793 DEBUG:boto:Data: 2022-09-02T05:02:12.794 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.795 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.795 DEBUG:boto:Params: {} 2022-09-02T05:02:12.796 DEBUG:boto:Token: None 2022-09-02T05:02:12.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q= 2022-09-02T05:02:12.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q=', 'Content-Length': '0'} 2022-09-02T05:02:12.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c876ee6cce70528-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.800 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.801 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.802 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.802 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.803 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.803 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.804 DEBUG:boto:Data: 2022-09-02T05:02:12.804 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.805 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.805 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.806 DEBUG:boto:Params: {} 2022-09-02T05:02:12.806 DEBUG:boto:Token: None 2022-09-02T05:02:12.807 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dru/YZfh+uAD2QtVOMcKgxRiinY= 2022-09-02T05:02:12.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dru/YZfh+uAD2QtVOMcKgxRiinY=', 'Content-Length': '0'} 2022-09-02T05:02:12.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d11a38985c8dcc8e-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.810 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.811 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.811 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.812 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.812 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.813 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.813 DEBUG:boto:Data: 2022-09-02T05:02:12.814 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.815 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.816 DEBUG:boto:Params: {} 2022-09-02T05:02:12.816 DEBUG:boto:Token: None 2022-09-02T05:02:12.817 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.817 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dru/YZfh+uAD2QtVOMcKgxRiinY= 2022-09-02T05:02:12.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dru/YZfh+uAD2QtVOMcKgxRiinY=', 'Content-Length': '0'} 2022-09-02T05:02:12.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6426b7acc361cc2-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:12.821 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.822 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.822 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.823 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.823 DEBUG:boto:Method: GET 2022-09-02T05:02:12.824 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.825 DEBUG:boto:Data: 2022-09-02T05:02:12.825 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.826 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.826 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.827 DEBUG:boto:Params: {} 2022-09-02T05:02:12.827 DEBUG:boto:Token: None 2022-09-02T05:02:12.828 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q= 2022-09-02T05:02:12.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q=', 'Content-Length': '0'} 2022-09-02T05:02:12.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c83e8f13e54ff5ab-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.832 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.832 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.833 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.833 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.834 DEBUG:boto:Method: GET 2022-09-02T05:02:12.834 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.835 DEBUG:boto:Data: 2022-09-02T05:02:12.835 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.836 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.837 DEBUG:boto:Params: {} 2022-09-02T05:02:12.838 DEBUG:boto:Token: None 2022-09-02T05:02:12.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:12.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q= 2022-09-02T05:02:12.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G0VnhpaiR2xtLZZPlmvP+Jk4H6Q=', 'Content-Length': '0'} 2022-09-02T05:02:12.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000984a9691309e6250-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.842 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.843 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.843 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.844 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.844 DEBUG:boto:Method: GET 2022-09-02T05:02:12.845 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.845 DEBUG:boto:Data: 2022-09-02T05:02:12.846 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.846 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.847 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.847 DEBUG:boto:Params: {} 2022-09-02T05:02:12.848 DEBUG:boto:Token: None 2022-09-02T05:02:12.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7a92867e6af754-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.853 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.854 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.854 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.855 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.855 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.856 DEBUG:boto:Method: GET 2022-09-02T05:02:12.857 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.857 DEBUG:boto:Data: 2022-09-02T05:02:12.858 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.859 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.859 DEBUG:boto:Params: {} 2022-09-02T05:02:12.860 DEBUG:boto:Token: None 2022-09-02T05:02:12.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b4a85f206455c8-0063118e54-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.865 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.866 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.866 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.867 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.867 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:12.868 DEBUG:boto:Data: 2022-09-02T05:02:12.868 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.869 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.870 DEBUG:boto:Params: {} 2022-09-02T05:02:12.870 DEBUG:boto:Token: None 2022-09-02T05:02:12.871 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2 2022-09-02T05:02:12.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ugS30TV/F0mChMHfAmkyRCBfvzc= 2022-09-02T05:02:12.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ugS30TV/F0mChMHfAmkyRCBfvzc=', 'Content-Length': '0'} 2022-09-02T05:02:12.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bd2f990a0b1c584-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.874 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.875 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.875 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.876 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:12.877 DEBUG:boto:Data: 2022-09-02T05:02:12.877 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.878 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.879 DEBUG:boto:Params: {} 2022-09-02T05:02:12.879 DEBUG:boto:Token: None 2022-09-02T05:02:12.880 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2 2022-09-02T05:02:12.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ugS30TV/F0mChMHfAmkyRCBfvzc= 2022-09-02T05:02:12.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ugS30TV/F0mChMHfAmkyRCBfvzc=', 'Content-Length': '0'} 2022-09-02T05:02:12.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ffbafbcc9b04f27-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:12.884 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.885 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.885 DEBUG:boto:Method: GET 2022-09-02T05:02:12.886 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:12.886 DEBUG:boto:Data: 2022-09-02T05:02:12.887 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.888 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.888 DEBUG:boto:Params: {} 2022-09-02T05:02:12.889 DEBUG:boto:Token: None 2022-09-02T05:02:12.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2 2022-09-02T05:02:12.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaQf/BAIR7BWxiwqJDsehBEI68I= 2022-09-02T05:02:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaQf/BAIR7BWxiwqJDsehBEI68I=', 'Content-Length': '0'} 2022-09-02T05:02:12.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c7d73a0ee703501-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.893 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:12.894 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:12.895 DEBUG:boto:Method: GET 2022-09-02T05:02:12.895 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:12.896 DEBUG:boto:Data: 2022-09-02T05:02:12.896 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.897 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.898 DEBUG:boto:Params: {} 2022-09-02T05:02:12.898 DEBUG:boto:Token: None 2022-09-02T05:02:12.899 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj2 2022-09-02T05:02:12.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaQf/BAIR7BWxiwqJDsehBEI68I= 2022-09-02T05:02:12.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaQf/BAIR7BWxiwqJDsehBEI68I=', 'Content-Length': '0'} 2022-09-02T05:02:12.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bed1cc608445396-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.946 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.947 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.947 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.948 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.948 DEBUG:boto:Method: GET 2022-09-02T05:02:12.949 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.949 DEBUG:boto:Data: 2022-09-02T05:02:12.950 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.951 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.952 DEBUG:boto:Params: {} 2022-09-02T05:02:12.952 DEBUG:boto:Token: None 2022-09-02T05:02:12.953 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000920b4816fd28fcfe-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.958 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.958 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:12.959 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:12.960 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.960 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.961 DEBUG:boto:Method: GET 2022-09-02T05:02:12.961 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:12.962 DEBUG:boto:Data: 2022-09-02T05:02:12.962 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.963 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.964 DEBUG:boto:Params: {} 2022-09-02T05:02:12.964 DEBUG:boto:Token: None 2022-09-02T05:02:12.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/ 2022-09-02T05:02:12.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck= 2022-09-02T05:02:12.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ffwa3bdDRyslOZtai2tL8DMbqck=', 'Content-Length': '0'} 2022-09-02T05:02:12.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b34ce6f4ae6ca547-0063118e54-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.969 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:12.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:12.971 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:12.971 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:12.972 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.972 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.973 DEBUG:boto:Method: GET 2022-09-02T05:02:12.973 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.974 DEBUG:boto:Data: 2022-09-02T05:02:12.974 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.975 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.976 DEBUG:boto:Params: {} 2022-09-02T05:02:12.977 DEBUG:boto:Token: None 2022-09-02T05:02:12.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EoJv0xDDR7i+27hSfWYh1osjedU= 2022-09-02T05:02:12.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EoJv0xDDR7i+27hSfWYh1osjedU=', 'Content-Length': '0'} 2022-09-02T05:02:12.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039920182dafb3737-0063118e54-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.981 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:12.982 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:12.983 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.983 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.984 DEBUG:boto:Method: GET 2022-09-02T05:02:12.984 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.985 DEBUG:boto:Data: 2022-09-02T05:02:12.985 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:12.986 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.987 DEBUG:boto:Params: {} 2022-09-02T05:02:12.987 DEBUG:boto:Token: None 2022-09-02T05:02:12.988 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EoJv0xDDR7i+27hSfWYh1osjedU= 2022-09-02T05:02:12.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EoJv0xDDR7i+27hSfWYh1osjedU=', 'Content-Length': '0'} 2022-09-02T05:02:12.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c07f56c2ce5e274-0063118e54-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:12.992 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:12.993 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:12.993 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.994 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.995 DEBUG:boto:Method: HEAD 2022-09-02T05:02:12.995 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:12.996 DEBUG:boto:Data: 2022-09-02T05:02:12.996 DEBUG:boto:Headers: {} 2022-09-02T05:02:12.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:12.997 DEBUG:boto:Port: 8000 2022-09-02T05:02:12.998 DEBUG:boto:Params: {} 2022-09-02T05:02:12.999 DEBUG:boto:Token: None 2022-09-02T05:02:12.999 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:12 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HVQsCxRPlUgFeBsw79T5NBWktbY= 2022-09-02T05:02:13.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HVQsCxRPlUgFeBsw79T5NBWktbY=', 'Content-Length': '0'} 2022-09-02T05:02:13.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c311ce5effc39d0a-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.003 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.003 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.004 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.004 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.005 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.005 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.006 DEBUG:boto:Data: 2022-09-02T05:02:13.006 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.007 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.008 DEBUG:boto:Params: {} 2022-09-02T05:02:13.008 DEBUG:boto:Token: None 2022-09-02T05:02:13.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:48becb+yWbaOeuJhtgMZTSsIsjA= 2022-09-02T05:02:13.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:48becb+yWbaOeuJhtgMZTSsIsjA=', 'Content-Length': '0'} 2022-09-02T05:02:13.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2679a5d15f400a9-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:13.013 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.014 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.014 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.015 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.015 DEBUG:boto:Method: GET 2022-09-02T05:02:13.016 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.016 DEBUG:boto:Data: 2022-09-02T05:02:13.017 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.018 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.019 DEBUG:boto:Params: {} 2022-09-02T05:02:13.019 DEBUG:boto:Token: None 2022-09-02T05:02:13.020 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4= 2022-09-02T05:02:13.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4=', 'Content-Length': '0'} 2022-09-02T05:02:13.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000930705cd98023841-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.066 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.067 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.067 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.068 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.069 DEBUG:boto:Method: GET 2022-09-02T05:02:13.069 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.070 DEBUG:boto:Data: 2022-09-02T05:02:13.070 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.071 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.072 DEBUG:boto:Params: {} 2022-09-02T05:02:13.072 DEBUG:boto:Token: None 2022-09-02T05:02:13.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4= 2022-09-02T05:02:13.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4=', 'Content-Length': '0'} 2022-09-02T05:02:13.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b04ab7033168a92-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.078 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.078 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.079 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.079 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.080 DEBUG:boto:Method: GET 2022-09-02T05:02:13.080 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.081 DEBUG:boto:Data: 2022-09-02T05:02:13.081 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.082 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.083 DEBUG:boto:Params: {} 2022-09-02T05:02:13.084 DEBUG:boto:Token: None 2022-09-02T05:02:13.084 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ba5347f67da790-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.089 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.090 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.090 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.091 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.092 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.092 DEBUG:boto:Method: GET 2022-09-02T05:02:13.093 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.093 DEBUG:boto:Data: 2022-09-02T05:02:13.094 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.095 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.096 DEBUG:boto:Params: {} 2022-09-02T05:02:13.096 DEBUG:boto:Token: None 2022-09-02T05:02:13.097 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edd02eb6461868ad-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.102 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.103 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.104 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.105 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.106 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:13.107 DEBUG:boto:Data: 2022-09-02T05:02:13.108 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.109 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.110 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.110 DEBUG:boto:Params: {} 2022-09-02T05:02:13.111 DEBUG:boto:Token: None 2022-09-02T05:02:13.111 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3 2022-09-02T05:02:13.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LdbtpIULW7LK3Y8mi7P+Uh/ttnE= 2022-09-02T05:02:13.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LdbtpIULW7LK3Y8mi7P+Uh/ttnE=', 'Content-Length': '0'} 2022-09-02T05:02:13.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7c4eeb9051909e0-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.115 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.116 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.117 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.117 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:13.118 DEBUG:boto:Data: 2022-09-02T05:02:13.118 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.119 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.120 DEBUG:boto:Params: {} 2022-09-02T05:02:13.120 DEBUG:boto:Token: None 2022-09-02T05:02:13.121 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3 2022-09-02T05:02:13.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LdbtpIULW7LK3Y8mi7P+Uh/ttnE= 2022-09-02T05:02:13.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LdbtpIULW7LK3Y8mi7P+Uh/ttnE=', 'Content-Length': '0'} 2022-09-02T05:02:13.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ca4e767ca158b81-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:13.125 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.126 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.127 DEBUG:boto:Method: GET 2022-09-02T05:02:13.127 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:13.128 DEBUG:boto:Data: 2022-09-02T05:02:13.129 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.130 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.130 DEBUG:boto:Params: {} 2022-09-02T05:02:13.131 DEBUG:boto:Token: None 2022-09-02T05:02:13.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3 2022-09-02T05:02:13.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:glbZxugP3xIVQH7TSXw77LW15Kc= 2022-09-02T05:02:13.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:glbZxugP3xIVQH7TSXw77LW15Kc=', 'Content-Length': '0'} 2022-09-02T05:02:13.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba5a4d6e93bcb9c7-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.136 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.136 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.137 DEBUG:boto:Method: GET 2022-09-02T05:02:13.138 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:13.138 DEBUG:boto:Data: 2022-09-02T05:02:13.139 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.140 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.140 DEBUG:boto:Params: {} 2022-09-02T05:02:13.141 DEBUG:boto:Token: None 2022-09-02T05:02:13.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3 2022-09-02T05:02:13.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:glbZxugP3xIVQH7TSXw77LW15Kc= 2022-09-02T05:02:13.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:glbZxugP3xIVQH7TSXw77LW15Kc=', 'Content-Length': '0'} 2022-09-02T05:02:13.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b74f8c06b1a8d83e-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.146 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.146 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.147 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.147 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.148 DEBUG:boto:Method: GET 2022-09-02T05:02:13.148 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.149 DEBUG:boto:Data: 2022-09-02T05:02:13.149 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.151 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.151 DEBUG:boto:Params: {} 2022-09-02T05:02:13.152 DEBUG:boto:Token: None 2022-09-02T05:02:13.153 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b29df69af6dd5e35-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.159 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.160 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.160 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.161 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.161 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.162 DEBUG:boto:Method: GET 2022-09-02T05:02:13.162 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.163 DEBUG:boto:Data: 2022-09-02T05:02:13.163 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.165 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.165 DEBUG:boto:Params: {} 2022-09-02T05:02:13.166 DEBUG:boto:Token: None 2022-09-02T05:02:13.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb8401fda1310f7d-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.171 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:13.172 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.173 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.173 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.174 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.175 DEBUG:boto:Method: GET 2022-09-02T05:02:13.175 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.176 DEBUG:boto:Data: 2022-09-02T05:02:13.176 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.177 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.178 DEBUG:boto:Params: {} 2022-09-02T05:02:13.178 DEBUG:boto:Token: None 2022-09-02T05:02:13.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c= 2022-09-02T05:02:13.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c=', 'Content-Length': '0'} 2022-09-02T05:02:13.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dfe0a1a6a9b3e0b-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.183 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.184 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.184 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.185 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.185 DEBUG:boto:Method: GET 2022-09-02T05:02:13.186 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.186 DEBUG:boto:Data: 2022-09-02T05:02:13.187 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.188 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.189 DEBUG:boto:Params: {} 2022-09-02T05:02:13.190 DEBUG:boto:Token: None 2022-09-02T05:02:13.191 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c= 2022-09-02T05:02:13.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c=', 'Content-Length': '0'} 2022-09-02T05:02:13.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000391599a68ddf5b7a-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.195 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.195 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.196 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.196 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.197 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.197 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.198 DEBUG:boto:Data: 2022-09-02T05:02:13.198 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.200 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.200 DEBUG:boto:Params: {} 2022-09-02T05:02:13.201 DEBUG:boto:Token: None 2022-09-02T05:02:13.201 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7lzJdlBNJ7nQlVbG+keeyvkgkg= 2022-09-02T05:02:13.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7lzJdlBNJ7nQlVbG+keeyvkgkg=', 'Content-Length': '0'} 2022-09-02T05:02:13.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f034b72ae8fbbbf4-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.205 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.206 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.207 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.208 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.208 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.209 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.209 DEBUG:boto:Data: 2022-09-02T05:02:13.210 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.211 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.211 DEBUG:boto:Params: {} 2022-09-02T05:02:13.212 DEBUG:boto:Token: None 2022-09-02T05:02:13.213 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7lzJdlBNJ7nQlVbG+keeyvkgkg= 2022-09-02T05:02:13.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7lzJdlBNJ7nQlVbG+keeyvkgkg=', 'Content-Length': '0'} 2022-09-02T05:02:13.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfa6f6211639c015-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:13.219 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.219 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.220 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.221 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.222 DEBUG:boto:Method: GET 2022-09-02T05:02:13.222 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.223 DEBUG:boto:Data: 2022-09-02T05:02:13.223 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.225 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.225 DEBUG:boto:Params: {} 2022-09-02T05:02:13.226 DEBUG:boto:Token: None 2022-09-02T05:02:13.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c= 2022-09-02T05:02:13.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c=', 'Content-Length': '0'} 2022-09-02T05:02:13.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b81fb3af7233483-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.274 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.275 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.275 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.276 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.276 DEBUG:boto:Method: GET 2022-09-02T05:02:13.277 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.277 DEBUG:boto:Data: 2022-09-02T05:02:13.278 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.279 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.280 DEBUG:boto:Params: {} 2022-09-02T05:02:13.280 DEBUG:boto:Token: None 2022-09-02T05:02:13.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:13.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c= 2022-09-02T05:02:13.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YGOYTS1vDqpkSOWFghcEp9Xv35c=', 'Content-Length': '0'} 2022-09-02T05:02:13.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2d9b0fcd88208e6-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.286 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.286 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.287 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.287 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.288 DEBUG:boto:Method: GET 2022-09-02T05:02:13.288 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.289 DEBUG:boto:Data: 2022-09-02T05:02:13.289 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.291 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.291 DEBUG:boto:Params: {} 2022-09-02T05:02:13.292 DEBUG:boto:Token: None 2022-09-02T05:02:13.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b797af36b0849468-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.297 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.298 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.299 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.299 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.300 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.300 DEBUG:boto:Method: GET 2022-09-02T05:02:13.301 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.301 DEBUG:boto:Data: 2022-09-02T05:02:13.302 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.303 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.304 DEBUG:boto:Params: {} 2022-09-02T05:02:13.304 DEBUG:boto:Token: None 2022-09-02T05:02:13.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061c7493f3e1815bf-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.310 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.310 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.311 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.311 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.312 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:13.312 DEBUG:boto:Data: 2022-09-02T05:02:13.313 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.314 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.315 DEBUG:boto:Params: {} 2022-09-02T05:02:13.315 DEBUG:boto:Token: None 2022-09-02T05:02:13.316 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4 2022-09-02T05:02:13.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hDVd5DYctChHtXL4n+SXZckS6sk= 2022-09-02T05:02:13.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hDVd5DYctChHtXL4n+SXZckS6sk=', 'Content-Length': '0'} 2022-09-02T05:02:13.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b0da8784f28b9bb-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.319 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.320 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.320 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.321 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:13.321 DEBUG:boto:Data: 2022-09-02T05:02:13.322 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.323 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.323 DEBUG:boto:Params: {} 2022-09-02T05:02:13.324 DEBUG:boto:Token: None 2022-09-02T05:02:13.324 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4 2022-09-02T05:02:13.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hDVd5DYctChHtXL4n+SXZckS6sk= 2022-09-02T05:02:13.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hDVd5DYctChHtXL4n+SXZckS6sk=', 'Content-Length': '0'} 2022-09-02T05:02:13.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8a31d0c570695b1-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:13.329 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.329 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.330 DEBUG:boto:Method: GET 2022-09-02T05:02:13.330 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:13.331 DEBUG:boto:Data: 2022-09-02T05:02:13.332 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.333 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.333 DEBUG:boto:Params: {} 2022-09-02T05:02:13.334 DEBUG:boto:Token: None 2022-09-02T05:02:13.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4 2022-09-02T05:02:13.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h47oQhvGQ9sbn2jkcCKurZ0DIS8= 2022-09-02T05:02:13.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h47oQhvGQ9sbn2jkcCKurZ0DIS8=', 'Content-Length': '0'} 2022-09-02T05:02:13.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017661c9584251d5d-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.338 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:13.339 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:13.340 DEBUG:boto:Method: GET 2022-09-02T05:02:13.340 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:13.341 DEBUG:boto:Data: 2022-09-02T05:02:13.341 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.342 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.343 DEBUG:boto:Params: {} 2022-09-02T05:02:13.343 DEBUG:boto:Token: None 2022-09-02T05:02:13.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj4 2022-09-02T05:02:13.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h47oQhvGQ9sbn2jkcCKurZ0DIS8= 2022-09-02T05:02:13.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h47oQhvGQ9sbn2jkcCKurZ0DIS8=', 'Content-Length': '0'} 2022-09-02T05:02:13.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097b8b914621d6498-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.348 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.349 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.350 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.351 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.352 DEBUG:boto:Method: GET 2022-09-02T05:02:13.353 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.354 DEBUG:boto:Data: 2022-09-02T05:02:13.355 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:13.357 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.358 DEBUG:boto:Params: {} 2022-09-02T05:02:13.359 DEBUG:boto:Token: None 2022-09-02T05:02:13.359 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808dbb33afc5c6ca-0063118e55-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.364 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.364 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.365 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.366 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.366 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.367 DEBUG:boto:Method: GET 2022-09-02T05:02:13.367 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:13.368 DEBUG:boto:Data: 2022-09-02T05:02:13.368 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.369 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.370 DEBUG:boto:Params: {} 2022-09-02T05:02:13.371 DEBUG:boto:Token: None 2022-09-02T05:02:13.371 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd737272d7299e8-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.376 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.377 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-59 zones={test-zone2, test-zone1} 2022-09-02T05:02:13.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-59 zones={test-zone3, test-zone1} 2022-09-02T05:02:13.378 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.379 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.379 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.380 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:02:13.380 DEBUG:boto:Data: 2022-09-02T05:02:13.381 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.382 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.382 DEBUG:boto:Params: {} 2022-09-02T05:02:13.383 DEBUG:boto:Token: None 2022-09-02T05:02:13.383 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rh98zy7UAL2spTJXEmEf8M+BiHw= 2022-09-02T05:02:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rh98zy7UAL2spTJXEmEf8M+BiHw=', 'Content-Length': '0'} 2022-09-02T05:02:13.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835e3e5fb32b3865-0063118e55-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.388 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.388 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.389 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.390 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:02:13.391 DEBUG:boto:Data: 2022-09-02T05:02:13.391 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.393 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.393 DEBUG:boto:Params: {} 2022-09-02T05:02:13.394 DEBUG:boto:Token: None 2022-09-02T05:02:13.394 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rh98zy7UAL2spTJXEmEf8M+BiHw= 2022-09-02T05:02:13.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rh98zy7UAL2spTJXEmEf8M+BiHw=', 'Content-Length': '0'} 2022-09-02T05:02:13.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005600fcb45d9ac87d-0063118e55-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:02:13.399 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.400 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.400 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.401 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.401 DEBUG:boto:Method: GET 2022-09-02T05:02:13.402 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.402 DEBUG:boto:Data: 2022-09-02T05:02:13.403 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.404 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.404 DEBUG:boto:Params: {} 2022-09-02T05:02:13.405 DEBUG:boto:Token: None 2022-09-02T05:02:13.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/?versions 2022-09-02T05:02:13.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y= 2022-09-02T05:02:13.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y=', 'Content-Length': '0'} 2022-09-02T05:02:13.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f78deacb2e4b001-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.410 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:02:13.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:02:13.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:02:13.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:02:13.413 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:02:13.413 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.414 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.414 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.415 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.415 DEBUG:boto:Method: GET 2022-09-02T05:02:13.416 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.416 DEBUG:boto:Data: 2022-09-02T05:02:13.417 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.418 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.419 DEBUG:boto:Params: {} 2022-09-02T05:02:13.419 DEBUG:boto:Token: None 2022-09-02T05:02:13.420 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/?versions 2022-09-02T05:02:13.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y= 2022-09-02T05:02:13.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y=', 'Content-Length': '0'} 2022-09-02T05:02:13.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040efa020e4363418-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.424 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:02:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:02:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:02:13.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:02:13.427 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.428 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.428 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.429 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.429 DEBUG:boto:Method: GET 2022-09-02T05:02:13.430 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.430 DEBUG:boto:Data: 2022-09-02T05:02:13.431 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.433 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.434 DEBUG:boto:Params: {} 2022-09-02T05:02:13.436 DEBUG:boto:Token: None 2022-09-02T05:02:13.437 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/?versions 2022-09-02T05:02:13.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y= 2022-09-02T05:02:13.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y=', 'Content-Length': '0'} 2022-09-02T05:02:13.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb9ec65efb948ec-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.442 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.443 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.444 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.444 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.445 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.445 DEBUG:boto:Method: GET 2022-09-02T05:02:13.446 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:13.447 DEBUG:boto:Data: 2022-09-02T05:02:13.447 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.448 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.449 DEBUG:boto:Params: {} 2022-09-02T05:02:13.449 DEBUG:boto:Token: None 2022-09-02T05:02:13.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/?versions 2022-09-02T05:02:13.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y= 2022-09-02T05:02:13.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhJrD1EcSq11ajWF0v+HGWOft3Y=', 'Content-Length': '0'} 2022-09-02T05:02:13.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f66d4d952a68de2-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.454 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:13.456 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.456 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.457 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.457 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.458 DEBUG:boto:Method: GET 2022-09-02T05:02:13.459 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.459 DEBUG:boto:Data: 2022-09-02T05:02:13.460 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.461 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.461 DEBUG:boto:Params: {} 2022-09-02T05:02:13.462 DEBUG:boto:Token: None 2022-09-02T05:02:13.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw= 2022-09-02T05:02:13.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw=', 'Content-Length': '0'} 2022-09-02T05:02:13.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ff8ece3df2dead2-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.467 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.468 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.468 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.469 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.469 DEBUG:boto:Method: GET 2022-09-02T05:02:13.470 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.470 DEBUG:boto:Data: 2022-09-02T05:02:13.471 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.472 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.473 DEBUG:boto:Params: {} 2022-09-02T05:02:13.474 DEBUG:boto:Token: None 2022-09-02T05:02:13.475 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw= 2022-09-02T05:02:13.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw=', 'Content-Length': '0'} 2022-09-02T05:02:13.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002c2d8c3013eb5d9-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.481 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.482 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.482 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.483 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.483 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.484 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.484 DEBUG:boto:Data: 2022-09-02T05:02:13.485 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.486 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.486 DEBUG:boto:Params: {} 2022-09-02T05:02:13.487 DEBUG:boto:Token: None 2022-09-02T05:02:13.488 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cNk5W9tMRMWOory2bCCAsd2cZQM= 2022-09-02T05:02:13.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cNk5W9tMRMWOory2bCCAsd2cZQM=', 'Content-Length': '0'} 2022-09-02T05:02:13.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc2e3ccc2e7d58e4-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.491 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.492 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.493 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.493 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.494 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.494 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.495 DEBUG:boto:Data: 2022-09-02T05:02:13.495 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.496 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.497 DEBUG:boto:Params: {} 2022-09-02T05:02:13.497 DEBUG:boto:Token: None 2022-09-02T05:02:13.498 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cNk5W9tMRMWOory2bCCAsd2cZQM= 2022-09-02T05:02:13.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cNk5W9tMRMWOory2bCCAsd2cZQM=', 'Content-Length': '0'} 2022-09-02T05:02:13.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cea1f3001a2ca270-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:13.504 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.505 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.506 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.507 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.507 DEBUG:boto:Method: GET 2022-09-02T05:02:13.508 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.509 DEBUG:boto:Data: 2022-09-02T05:02:13.510 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.512 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.512 DEBUG:boto:Params: {} 2022-09-02T05:02:13.513 DEBUG:boto:Token: None 2022-09-02T05:02:13.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw= 2022-09-02T05:02:13.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw=', 'Content-Length': '0'} 2022-09-02T05:02:13.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf8e1fbdb9568aa9-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.558 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.559 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.560 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.561 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.561 DEBUG:boto:Method: GET 2022-09-02T05:02:13.562 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.562 DEBUG:boto:Data: 2022-09-02T05:02:13.563 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.564 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.565 DEBUG:boto:Params: {} 2022-09-02T05:02:13.565 DEBUG:boto:Token: None 2022-09-02T05:02:13.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:13.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw= 2022-09-02T05:02:13.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8g7nhV73+tey1wUCdy085zWt4nw=', 'Content-Length': '0'} 2022-09-02T05:02:13.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e9deb0388eb1757-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.570 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.571 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.572 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.572 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.573 DEBUG:boto:Method: GET 2022-09-02T05:02:13.573 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.574 DEBUG:boto:Data: 2022-09-02T05:02:13.574 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.575 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.576 DEBUG:boto:Params: {} 2022-09-02T05:02:13.577 DEBUG:boto:Token: None 2022-09-02T05:02:13.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a81aac513de5fc08-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.582 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.583 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.583 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.584 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.584 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.585 DEBUG:boto:Method: GET 2022-09-02T05:02:13.585 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.586 DEBUG:boto:Data: 2022-09-02T05:02:13.587 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.588 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.588 DEBUG:boto:Params: {} 2022-09-02T05:02:13.589 DEBUG:boto:Token: None 2022-09-02T05:02:13.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b65396677a2cb59-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.594 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.595 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.596 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.597 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.598 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:13.599 DEBUG:boto:Data: 2022-09-02T05:02:13.600 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.601 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.602 DEBUG:boto:Params: {} 2022-09-02T05:02:13.603 DEBUG:boto:Token: None 2022-09-02T05:02:13.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1 2022-09-02T05:02:13.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t8HSdf9nWc2dS4J6aJHgOumBifY= 2022-09-02T05:02:13.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t8HSdf9nWc2dS4J6aJHgOumBifY=', 'Content-Length': '0'} 2022-09-02T05:02:13.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3bebc222f6b2119-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.607 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.608 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.608 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.609 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:13.609 DEBUG:boto:Data: 2022-09-02T05:02:13.610 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.611 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.611 DEBUG:boto:Params: {} 2022-09-02T05:02:13.612 DEBUG:boto:Token: None 2022-09-02T05:02:13.612 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1 2022-09-02T05:02:13.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t8HSdf9nWc2dS4J6aJHgOumBifY= 2022-09-02T05:02:13.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t8HSdf9nWc2dS4J6aJHgOumBifY=', 'Content-Length': '0'} 2022-09-02T05:02:13.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b66de186762dccb1-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:13.617 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.618 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.618 DEBUG:boto:Method: GET 2022-09-02T05:02:13.619 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:13.620 DEBUG:boto:Data: 2022-09-02T05:02:13.620 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.621 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.622 DEBUG:boto:Params: {} 2022-09-02T05:02:13.623 DEBUG:boto:Token: None 2022-09-02T05:02:13.623 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1 2022-09-02T05:02:13.624 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O/+ucvDqOYLk0vSopM4OnjpJ0Ko= 2022-09-02T05:02:13.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O/+ucvDqOYLk0vSopM4OnjpJ0Ko=', 'Content-Length': '0'} 2022-09-02T05:02:13.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b49d425fae6f1634-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.627 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:13.628 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:13.628 DEBUG:boto:Method: GET 2022-09-02T05:02:13.629 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:13.629 DEBUG:boto:Data: 2022-09-02T05:02:13.630 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.631 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.632 DEBUG:boto:Params: {} 2022-09-02T05:02:13.632 DEBUG:boto:Token: None 2022-09-02T05:02:13.633 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj1 2022-09-02T05:02:13.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O/+ucvDqOYLk0vSopM4OnjpJ0Ko= 2022-09-02T05:02:13.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O/+ucvDqOYLk0vSopM4OnjpJ0Ko=', 'Content-Length': '0'} 2022-09-02T05:02:13.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001437f97f7cdab3be-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.637 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.638 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.638 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.639 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.639 DEBUG:boto:Method: GET 2022-09-02T05:02:13.640 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.640 DEBUG:boto:Data: 2022-09-02T05:02:13.641 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.642 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.643 DEBUG:boto:Params: {} 2022-09-02T05:02:13.643 DEBUG:boto:Token: None 2022-09-02T05:02:13.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6e53698c957495-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.690 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.691 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.692 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.692 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.693 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.693 DEBUG:boto:Method: GET 2022-09-02T05:02:13.694 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:13.694 DEBUG:boto:Data: 2022-09-02T05:02:13.695 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.696 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.697 DEBUG:boto:Params: {} 2022-09-02T05:02:13.697 DEBUG:boto:Token: None 2022-09-02T05:02:13.698 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a01c403ebd1117b-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.702 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:13.704 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.705 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.705 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.706 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.706 DEBUG:boto:Method: GET 2022-09-02T05:02:13.707 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.707 DEBUG:boto:Data: 2022-09-02T05:02:13.708 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.709 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.709 DEBUG:boto:Params: {} 2022-09-02T05:02:13.710 DEBUG:boto:Token: None 2022-09-02T05:02:13.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c= 2022-09-02T05:02:13.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c=', 'Content-Length': '0'} 2022-09-02T05:02:13.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000927168b4dd5f4513-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.715 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.715 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.716 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.717 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.717 DEBUG:boto:Method: GET 2022-09-02T05:02:13.718 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.718 DEBUG:boto:Data: 2022-09-02T05:02:13.719 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.720 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.721 DEBUG:boto:Params: {} 2022-09-02T05:02:13.721 DEBUG:boto:Token: None 2022-09-02T05:02:13.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c= 2022-09-02T05:02:13.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c=', 'Content-Length': '0'} 2022-09-02T05:02:13.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000335897fbad625929-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.726 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.727 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.727 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.728 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.728 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.729 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.729 DEBUG:boto:Data: 2022-09-02T05:02:13.730 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.731 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.731 DEBUG:boto:Params: {} 2022-09-02T05:02:13.732 DEBUG:boto:Token: None 2022-09-02T05:02:13.733 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3OmIj8O7VJ4uamCDM5pz1uyKqs0= 2022-09-02T05:02:13.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3OmIj8O7VJ4uamCDM5pz1uyKqs0=', 'Content-Length': '0'} 2022-09-02T05:02:13.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000763a462dfec394cf-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.736 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.737 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.737 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.738 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.738 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.739 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.739 DEBUG:boto:Data: 2022-09-02T05:02:13.740 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.741 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.741 DEBUG:boto:Params: {} 2022-09-02T05:02:13.742 DEBUG:boto:Token: None 2022-09-02T05:02:13.743 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3OmIj8O7VJ4uamCDM5pz1uyKqs0= 2022-09-02T05:02:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3OmIj8O7VJ4uamCDM5pz1uyKqs0=', 'Content-Length': '0'} 2022-09-02T05:02:13.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000091d6491acf5d18e-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:13.747 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.747 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.748 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.749 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.749 DEBUG:boto:Method: GET 2022-09-02T05:02:13.750 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.750 DEBUG:boto:Data: 2022-09-02T05:02:13.751 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.752 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.753 DEBUG:boto:Params: {} 2022-09-02T05:02:13.753 DEBUG:boto:Token: None 2022-09-02T05:02:13.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c= 2022-09-02T05:02:13.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c=', 'Content-Length': '0'} 2022-09-02T05:02:13.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057c1764d5d773b83-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.758 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.758 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.759 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.759 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.760 DEBUG:boto:Method: GET 2022-09-02T05:02:13.761 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.761 DEBUG:boto:Data: 2022-09-02T05:02:13.762 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.763 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.763 DEBUG:boto:Params: {} 2022-09-02T05:02:13.764 DEBUG:boto:Token: None 2022-09-02T05:02:13.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:13.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c= 2022-09-02T05:02:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CQblQA4TL7YUeDZl42pzhzL1s5c=', 'Content-Length': '0'} 2022-09-02T05:02:13.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bd28964c72ac462-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.769 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.769 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.770 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.771 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.771 DEBUG:boto:Method: GET 2022-09-02T05:02:13.772 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.772 DEBUG:boto:Data: 2022-09-02T05:02:13.773 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.774 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.774 DEBUG:boto:Params: {} 2022-09-02T05:02:13.775 DEBUG:boto:Token: None 2022-09-02T05:02:13.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a84acda6ece1319-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.780 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.781 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.782 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.782 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.783 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.783 DEBUG:boto:Method: GET 2022-09-02T05:02:13.784 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.785 DEBUG:boto:Data: 2022-09-02T05:02:13.785 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.786 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.787 DEBUG:boto:Params: {} 2022-09-02T05:02:13.787 DEBUG:boto:Token: None 2022-09-02T05:02:13.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043d3a0d266793ede-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.792 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.793 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.794 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.794 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.795 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:13.795 DEBUG:boto:Data: 2022-09-02T05:02:13.796 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.797 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.797 DEBUG:boto:Params: {} 2022-09-02T05:02:13.798 DEBUG:boto:Token: None 2022-09-02T05:02:13.799 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2 2022-09-02T05:02:13.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44tiAZt4/m91QmdYt0tbxQ2/ujw= 2022-09-02T05:02:13.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44tiAZt4/m91QmdYt0tbxQ2/ujw=', 'Content-Length': '0'} 2022-09-02T05:02:13.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e1c05e76b1f7b8a-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.803 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.803 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.804 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.804 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:13.805 DEBUG:boto:Data: 2022-09-02T05:02:13.805 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.806 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.807 DEBUG:boto:Params: {} 2022-09-02T05:02:13.807 DEBUG:boto:Token: None 2022-09-02T05:02:13.808 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2 2022-09-02T05:02:13.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:44tiAZt4/m91QmdYt0tbxQ2/ujw= 2022-09-02T05:02:13.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:44tiAZt4/m91QmdYt0tbxQ2/ujw=', 'Content-Length': '0'} 2022-09-02T05:02:13.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081268c838f0e3887-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:13.812 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.813 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.813 DEBUG:boto:Method: GET 2022-09-02T05:02:13.814 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:13.815 DEBUG:boto:Data: 2022-09-02T05:02:13.816 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.818 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.819 DEBUG:boto:Params: {} 2022-09-02T05:02:13.820 DEBUG:boto:Token: None 2022-09-02T05:02:13.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2 2022-09-02T05:02:13.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x+jSU/oG6DeO3TeJFvedOllbI2s= 2022-09-02T05:02:13.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x+jSU/oG6DeO3TeJFvedOllbI2s=', 'Content-Length': '0'} 2022-09-02T05:02:13.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e119dbfacd0e7ef0-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.825 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:13.826 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:13.826 DEBUG:boto:Method: GET 2022-09-02T05:02:13.827 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:13.827 DEBUG:boto:Data: 2022-09-02T05:02:13.828 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.829 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.830 DEBUG:boto:Params: {} 2022-09-02T05:02:13.830 DEBUG:boto:Token: None 2022-09-02T05:02:13.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj2 2022-09-02T05:02:13.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x+jSU/oG6DeO3TeJFvedOllbI2s= 2022-09-02T05:02:13.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x+jSU/oG6DeO3TeJFvedOllbI2s=', 'Content-Length': '0'} 2022-09-02T05:02:13.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012959c576e404c81-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.835 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.835 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.836 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.836 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.837 DEBUG:boto:Method: GET 2022-09-02T05:02:13.838 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.838 DEBUG:boto:Data: 2022-09-02T05:02:13.839 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.840 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.840 DEBUG:boto:Params: {} 2022-09-02T05:02:13.841 DEBUG:boto:Token: None 2022-09-02T05:02:13.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4fbdaba71ef928-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.846 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.847 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.847 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.848 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.848 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.849 DEBUG:boto:Method: GET 2022-09-02T05:02:13.849 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:13.850 DEBUG:boto:Data: 2022-09-02T05:02:13.851 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.852 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.852 DEBUG:boto:Params: {} 2022-09-02T05:02:13.853 DEBUG:boto:Token: None 2022-09-02T05:02:13.853 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004acc13848cfbc51e-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.858 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:13.859 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.859 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.860 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.861 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.862 DEBUG:boto:Method: GET 2022-09-02T05:02:13.863 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.864 DEBUG:boto:Data: 2022-09-02T05:02:13.865 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.867 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.868 DEBUG:boto:Params: {} 2022-09-02T05:02:13.869 DEBUG:boto:Token: None 2022-09-02T05:02:13.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4= 2022-09-02T05:02:13.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4=', 'Content-Length': '0'} 2022-09-02T05:02:13.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007759f82589feae01-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.873 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.874 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.874 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.875 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.875 DEBUG:boto:Method: GET 2022-09-02T05:02:13.876 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.877 DEBUG:boto:Data: 2022-09-02T05:02:13.877 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.878 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.879 DEBUG:boto:Params: {} 2022-09-02T05:02:13.879 DEBUG:boto:Token: None 2022-09-02T05:02:13.880 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4= 2022-09-02T05:02:13.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4=', 'Content-Length': '0'} 2022-09-02T05:02:13.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6ad16d860bd333e-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.884 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.885 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.885 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.886 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.886 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.887 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.887 DEBUG:boto:Data: 2022-09-02T05:02:13.888 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.889 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.889 DEBUG:boto:Params: {} 2022-09-02T05:02:13.890 DEBUG:boto:Token: None 2022-09-02T05:02:13.891 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:48becb+yWbaOeuJhtgMZTSsIsjA= 2022-09-02T05:02:13.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:48becb+yWbaOeuJhtgMZTSsIsjA=', 'Content-Length': '0'} 2022-09-02T05:02:13.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017405389a3581d64-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.894 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.895 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.895 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.896 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.896 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.897 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.897 DEBUG:boto:Data: 2022-09-02T05:02:13.898 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.899 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.899 DEBUG:boto:Params: {} 2022-09-02T05:02:13.900 DEBUG:boto:Token: None 2022-09-02T05:02:13.901 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:48becb+yWbaOeuJhtgMZTSsIsjA= 2022-09-02T05:02:13.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:48becb+yWbaOeuJhtgMZTSsIsjA=', 'Content-Length': '0'} 2022-09-02T05:02:13.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e24bb9efaa0d07fc-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:13.905 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.906 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.906 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.907 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.907 DEBUG:boto:Method: GET 2022-09-02T05:02:13.908 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.908 DEBUG:boto:Data: 2022-09-02T05:02:13.909 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.911 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.912 DEBUG:boto:Params: {} 2022-09-02T05:02:13.913 DEBUG:boto:Token: None 2022-09-02T05:02:13.914 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4= 2022-09-02T05:02:13.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4=', 'Content-Length': '0'} 2022-09-02T05:02:13.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000862199eff77ea480-0063118e55-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.920 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.921 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.921 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.922 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.922 DEBUG:boto:Method: GET 2022-09-02T05:02:13.923 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.923 DEBUG:boto:Data: 2022-09-02T05:02:13.924 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.925 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.926 DEBUG:boto:Params: {} 2022-09-02T05:02:13.926 DEBUG:boto:Token: None 2022-09-02T05:02:13.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:13.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4= 2022-09-02T05:02:13.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ni+xaEzapM/7tWhjaoux5hmULH4=', 'Content-Length': '0'} 2022-09-02T05:02:13.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c6d710198ac143-0063118e55-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.931 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.932 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.932 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.933 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.933 DEBUG:boto:Method: GET 2022-09-02T05:02:13.934 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.934 DEBUG:boto:Data: 2022-09-02T05:02:13.935 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.936 DEBUG:boto:Port: 8001 2022-09-02T05:02:13.937 DEBUG:boto:Params: {} 2022-09-02T05:02:13.937 DEBUG:boto:Token: None 2022-09-02T05:02:13.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d0707e85d192f6-0063118e55-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.942 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.943 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:13.943 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:13.944 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.945 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.945 DEBUG:boto:Method: GET 2022-09-02T05:02:13.946 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:13.946 DEBUG:boto:Data: 2022-09-02T05:02:13.947 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:13.948 DEBUG:boto:Port: 8000 2022-09-02T05:02:13.948 DEBUG:boto:Params: {} 2022-09-02T05:02:13.949 DEBUG:boto:Token: None 2022-09-02T05:02:13.949 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:13 GMT /xljbkf-59/ 2022-09-02T05:02:13.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE= 2022-09-02T05:02:13.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+quYEaDAr57+afFDk5b4Lh41CE=', 'Content-Length': '0'} 2022-09-02T05:02:13.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a70b0a32ef8d1a9-0063118e55-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:13.994 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:13.995 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:13.996 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:13.996 DEBUG:boto:Method: HEAD 2022-09-02T05:02:13.997 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:13.997 DEBUG:boto:Data: 2022-09-02T05:02:13.998 DEBUG:boto:Headers: {} 2022-09-02T05:02:13.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:13.999 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.000 DEBUG:boto:Params: {} 2022-09-02T05:02:14.000 DEBUG:boto:Token: None 2022-09-02T05:02:14.001 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ= 2022-09-02T05:02:14.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ=', 'Content-Length': '0'} 2022-09-02T05:02:14.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d91a7905cd987cea-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.005 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.005 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.006 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.006 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.007 DEBUG:boto:Data: 2022-09-02T05:02:14.007 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.009 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.009 DEBUG:boto:Params: {} 2022-09-02T05:02:14.010 DEBUG:boto:Token: None 2022-09-02T05:02:14.010 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ= 2022-09-02T05:02:14.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ=', 'Content-Length': '0'} 2022-09-02T05:02:14.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000209e87410d0a008b-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:14.015 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.015 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.016 DEBUG:boto:Method: GET 2022-09-02T05:02:14.016 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.017 DEBUG:boto:Data: 2022-09-02T05:02:14.017 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.018 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.019 DEBUG:boto:Params: {} 2022-09-02T05:02:14.020 DEBUG:boto:Token: None 2022-09-02T05:02:14.020 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI= 2022-09-02T05:02:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI=', 'Content-Length': '0'} 2022-09-02T05:02:14.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfcb358aed266d19-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.024 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.025 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.025 DEBUG:boto:Method: GET 2022-09-02T05:02:14.026 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.026 DEBUG:boto:Data: 2022-09-02T05:02:14.027 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.028 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.028 DEBUG:boto:Params: {} 2022-09-02T05:02:14.029 DEBUG:boto:Token: None 2022-09-02T05:02:14.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI= 2022-09-02T05:02:14.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI=', 'Content-Length': '0'} 2022-09-02T05:02:14.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b831ebedca2bc6ec-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.034 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.034 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.035 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.035 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.036 DEBUG:boto:Method: GET 2022-09-02T05:02:14.037 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.037 DEBUG:boto:Data: 2022-09-02T05:02:14.038 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.039 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.039 DEBUG:boto:Params: {} 2022-09-02T05:02:14.040 DEBUG:boto:Token: None 2022-09-02T05:02:14.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c9a3a75507fb783-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.045 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.046 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.046 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.047 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.048 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.048 DEBUG:boto:Method: GET 2022-09-02T05:02:14.049 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.049 DEBUG:boto:Data: 2022-09-02T05:02:14.050 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.051 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.051 DEBUG:boto:Params: {} 2022-09-02T05:02:14.052 DEBUG:boto:Token: None 2022-09-02T05:02:14.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc7cec57a928e70-0063118e56-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.057 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:14.058 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.059 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.059 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.060 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.061 DEBUG:boto:Method: GET 2022-09-02T05:02:14.061 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.062 DEBUG:boto:Data: 2022-09-02T05:02:14.062 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.063 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.064 DEBUG:boto:Params: {} 2022-09-02T05:02:14.064 DEBUG:boto:Token: None 2022-09-02T05:02:14.065 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA= 2022-09-02T05:02:14.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA=', 'Content-Length': '0'} 2022-09-02T05:02:14.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000474cc458e4ae2cf-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.069 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.070 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.070 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.071 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.071 DEBUG:boto:Method: GET 2022-09-02T05:02:14.072 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.072 DEBUG:boto:Data: 2022-09-02T05:02:14.073 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.074 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.075 DEBUG:boto:Params: {} 2022-09-02T05:02:14.075 DEBUG:boto:Token: None 2022-09-02T05:02:14.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA= 2022-09-02T05:02:14.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA=', 'Content-Length': '0'} 2022-09-02T05:02:14.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707497cd2b6383d9-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.080 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.081 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.081 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.082 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.082 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.083 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.083 DEBUG:boto:Data: 2022-09-02T05:02:14.084 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.085 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.085 DEBUG:boto:Params: {} 2022-09-02T05:02:14.086 DEBUG:boto:Token: None 2022-09-02T05:02:14.087 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0KPauRn03Z05nHhDPvpkcQJRKw= 2022-09-02T05:02:14.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0KPauRn03Z05nHhDPvpkcQJRKw=', 'Content-Length': '0'} 2022-09-02T05:02:14.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000932d36e0ae2920e6-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.090 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.091 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.091 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.092 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.092 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.093 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.093 DEBUG:boto:Data: 2022-09-02T05:02:14.094 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.095 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.096 DEBUG:boto:Params: {} 2022-09-02T05:02:14.096 DEBUG:boto:Token: None 2022-09-02T05:02:14.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0KPauRn03Z05nHhDPvpkcQJRKw= 2022-09-02T05:02:14.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0KPauRn03Z05nHhDPvpkcQJRKw=', 'Content-Length': '0'} 2022-09-02T05:02:14.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dd00094ee6cb9e3-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:14.101 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.102 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.102 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.103 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.103 DEBUG:boto:Method: GET 2022-09-02T05:02:14.104 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.104 DEBUG:boto:Data: 2022-09-02T05:02:14.105 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.106 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.107 DEBUG:boto:Params: {} 2022-09-02T05:02:14.107 DEBUG:boto:Token: None 2022-09-02T05:02:14.108 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA= 2022-09-02T05:02:14.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA=', 'Content-Length': '0'} 2022-09-02T05:02:14.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001baccbf8ec53bf03-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.112 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.112 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.113 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.114 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.114 DEBUG:boto:Method: GET 2022-09-02T05:02:14.115 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.115 DEBUG:boto:Data: 2022-09-02T05:02:14.116 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.117 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.118 DEBUG:boto:Params: {} 2022-09-02T05:02:14.118 DEBUG:boto:Token: None 2022-09-02T05:02:14.119 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA= 2022-09-02T05:02:14.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA=', 'Content-Length': '0'} 2022-09-02T05:02:14.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4d6406408e5234c-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.123 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.123 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.124 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.124 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.125 DEBUG:boto:Method: GET 2022-09-02T05:02:14.126 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.126 DEBUG:boto:Data: 2022-09-02T05:02:14.127 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.128 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.128 DEBUG:boto:Params: {} 2022-09-02T05:02:14.129 DEBUG:boto:Token: None 2022-09-02T05:02:14.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0317a8da0078d80-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.134 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.135 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.136 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.136 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.137 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.137 DEBUG:boto:Method: GET 2022-09-02T05:02:14.138 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.139 DEBUG:boto:Data: 2022-09-02T05:02:14.139 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.140 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.141 DEBUG:boto:Params: {} 2022-09-02T05:02:14.141 DEBUG:boto:Token: None 2022-09-02T05:02:14.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c11bfe47260b73-0063118e56-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.147 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.147 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.148 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.148 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.149 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:14.150 DEBUG:boto:Data: 2022-09-02T05:02:14.150 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.151 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.152 DEBUG:boto:Params: {} 2022-09-02T05:02:14.152 DEBUG:boto:Token: None 2022-09-02T05:02:14.153 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4 2022-09-02T05:02:14.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EAHJlCTqdf0Skw2yMOG9c3lytOo= 2022-09-02T05:02:14.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EAHJlCTqdf0Skw2yMOG9c3lytOo=', 'Content-Length': '0'} 2022-09-02T05:02:14.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fedb6b657365d364-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.157 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.157 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.158 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.158 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:14.159 DEBUG:boto:Data: 2022-09-02T05:02:14.159 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.160 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.161 DEBUG:boto:Params: {} 2022-09-02T05:02:14.162 DEBUG:boto:Token: None 2022-09-02T05:02:14.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4 2022-09-02T05:02:14.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EAHJlCTqdf0Skw2yMOG9c3lytOo= 2022-09-02T05:02:14.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EAHJlCTqdf0Skw2yMOG9c3lytOo=', 'Content-Length': '0'} 2022-09-02T05:02:14.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000500d6ee703a39c7b-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:14.167 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.167 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.168 DEBUG:boto:Method: GET 2022-09-02T05:02:14.168 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:14.169 DEBUG:boto:Data: 2022-09-02T05:02:14.169 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.170 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.171 DEBUG:boto:Params: {} 2022-09-02T05:02:14.172 DEBUG:boto:Token: None 2022-09-02T05:02:14.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4 2022-09-02T05:02:14.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bN3EJPBXfpFeFhjcSpmhMSq6K30= 2022-09-02T05:02:14.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bN3EJPBXfpFeFhjcSpmhMSq6K30=', 'Content-Length': '0'} 2022-09-02T05:02:14.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046b5d3317ba2afef-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.176 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.177 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.177 DEBUG:boto:Method: GET 2022-09-02T05:02:14.178 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:14.178 DEBUG:boto:Data: 2022-09-02T05:02:14.179 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.180 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.181 DEBUG:boto:Params: {} 2022-09-02T05:02:14.181 DEBUG:boto:Token: None 2022-09-02T05:02:14.182 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4 2022-09-02T05:02:14.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bN3EJPBXfpFeFhjcSpmhMSq6K30= 2022-09-02T05:02:14.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bN3EJPBXfpFeFhjcSpmhMSq6K30=', 'Content-Length': '0'} 2022-09-02T05:02:14.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cc02586e2bfafa2-0063118e56-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.226 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.227 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.227 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.228 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.229 DEBUG:boto:Method: GET 2022-09-02T05:02:14.229 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.230 DEBUG:boto:Data: 2022-09-02T05:02:14.230 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.231 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.232 DEBUG:boto:Params: {} 2022-09-02T05:02:14.233 DEBUG:boto:Token: None 2022-09-02T05:02:14.233 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad9b066a8320ff7-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.238 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.239 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.239 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.240 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.240 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.241 DEBUG:boto:Method: GET 2022-09-02T05:02:14.242 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:14.242 DEBUG:boto:Data: 2022-09-02T05:02:14.243 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:02:14.244 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.244 DEBUG:boto:Params: {} 2022-09-02T05:02:14.245 DEBUG:boto:Token: None 2022-09-02T05:02:14.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a2fcb35c706479-0063118e56-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.250 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.251 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-59 zones={test-zone3, test-zone1} 2022-09-02T05:02:14.252 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-59 zones={test-zone3, test-zone2} 2022-09-02T05:02:14.252 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.253 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.253 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.254 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:02:14.254 DEBUG:boto:Data: 2022-09-02T05:02:14.255 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.256 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.256 DEBUG:boto:Params: {} 2022-09-02T05:02:14.257 DEBUG:boto:Token: None 2022-09-02T05:02:14.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lX2MesaCT8aIkLQe5SGM5A/fn1U= 2022-09-02T05:02:14.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lX2MesaCT8aIkLQe5SGM5A/fn1U=', 'Content-Length': '0'} 2022-09-02T05:02:14.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c5457b0b9430600-0063118e56-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.262 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.263 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.263 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.264 DEBUG:boto:Path: /xljbkf-59/ 2022-09-02T05:02:14.264 DEBUG:boto:Data: 2022-09-02T05:02:14.265 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.266 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.266 DEBUG:boto:Params: {} 2022-09-02T05:02:14.267 DEBUG:boto:Token: None 2022-09-02T05:02:14.267 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lX2MesaCT8aIkLQe5SGM5A/fn1U= 2022-09-02T05:02:14.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lX2MesaCT8aIkLQe5SGM5A/fn1U=', 'Content-Length': '0'} 2022-09-02T05:02:14.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000088b16a7a4cc8d54-0063118e56-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.272 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:02:14.273 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.273 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.274 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.274 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.275 DEBUG:boto:Method: GET 2022-09-02T05:02:14.275 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.276 DEBUG:boto:Data: 2022-09-02T05:02:14.276 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.277 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.278 DEBUG:boto:Params: {} 2022-09-02T05:02:14.278 DEBUG:boto:Token: None 2022-09-02T05:02:14.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/?versions 2022-09-02T05:02:14.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY= 2022-09-02T05:02:14.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY=', 'Content-Length': '0'} 2022-09-02T05:02:14.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002434d4ed803bbd96-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.283 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:02:14.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:02:14.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:02:14.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:02:14.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:02:14.287 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.288 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.289 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.290 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.291 DEBUG:boto:Method: GET 2022-09-02T05:02:14.291 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.292 DEBUG:boto:Data: 2022-09-02T05:02:14.293 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.295 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.296 DEBUG:boto:Params: {} 2022-09-02T05:02:14.297 DEBUG:boto:Token: None 2022-09-02T05:02:14.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/?versions 2022-09-02T05:02:14.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY= 2022-09-02T05:02:14.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY=', 'Content-Length': '0'} 2022-09-02T05:02:14.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094dd325638b534be-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.302 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:02:14.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:02:14.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:02:14.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:02:14.305 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.306 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.306 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.307 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.308 DEBUG:boto:Method: GET 2022-09-02T05:02:14.308 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.309 DEBUG:boto:Data: 2022-09-02T05:02:14.309 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.310 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.311 DEBUG:boto:Params: {} 2022-09-02T05:02:14.311 DEBUG:boto:Token: None 2022-09-02T05:02:14.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/?versions 2022-09-02T05:02:14.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY= 2022-09-02T05:02:14.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY=', 'Content-Length': '0'} 2022-09-02T05:02:14.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001686f3ca91c370ca-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.316 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.317 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.318 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.318 DEBUG:boto:path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.319 DEBUG:boto:auth_path=/xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.319 DEBUG:boto:Method: GET 2022-09-02T05:02:14.320 DEBUG:boto:Path: /xljbkf-59/?versions&max-keys=999 2022-09-02T05:02:14.320 DEBUG:boto:Data: 2022-09-02T05:02:14.321 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.322 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.322 DEBUG:boto:Params: {} 2022-09-02T05:02:14.323 DEBUG:boto:Token: None 2022-09-02T05:02:14.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/?versions 2022-09-02T05:02:14.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY= 2022-09-02T05:02:14.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6sj7T/cxcUHREULXhmCqQS6FLkY=', 'Content-Length': '0'} 2022-09-02T05:02:14.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcd14e2b2f84b089-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.328 DEBUG:boto:b'xljbkf-59999falseobj1nulltrue2022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:14.330 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.330 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.331 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.331 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.332 DEBUG:boto:Method: GET 2022-09-02T05:02:14.332 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.333 DEBUG:boto:Data: 2022-09-02T05:02:14.333 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.334 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.335 DEBUG:boto:Params: {} 2022-09-02T05:02:14.335 DEBUG:boto:Token: None 2022-09-02T05:02:14.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo= 2022-09-02T05:02:14.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo=', 'Content-Length': '0'} 2022-09-02T05:02:14.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091548466c2315b63-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.382 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.383 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.383 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.384 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.384 DEBUG:boto:Method: GET 2022-09-02T05:02:14.385 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.385 DEBUG:boto:Data: 2022-09-02T05:02:14.386 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.387 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.388 DEBUG:boto:Params: {} 2022-09-02T05:02:14.388 DEBUG:boto:Token: None 2022-09-02T05:02:14.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo= 2022-09-02T05:02:14.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo=', 'Content-Length': '0'} 2022-09-02T05:02:14.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000148e1a37287e0fa1-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.393 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.394 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.394 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.395 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.395 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.396 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.396 DEBUG:boto:Data: 2022-09-02T05:02:14.397 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.398 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.398 DEBUG:boto:Params: {} 2022-09-02T05:02:14.399 DEBUG:boto:Token: None 2022-09-02T05:02:14.400 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.400 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZroQuJ/gA4JbUaxxqYJ6Iatdcog= 2022-09-02T05:02:14.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZroQuJ/gA4JbUaxxqYJ6Iatdcog=', 'Content-Length': '0'} 2022-09-02T05:02:14.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d473a45a168afa6c-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.403 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.404 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.404 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.405 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.405 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.406 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.406 DEBUG:boto:Data: 2022-09-02T05:02:14.407 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.408 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.408 DEBUG:boto:Params: {} 2022-09-02T05:02:14.409 DEBUG:boto:Token: None 2022-09-02T05:02:14.410 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZroQuJ/gA4JbUaxxqYJ6Iatdcog= 2022-09-02T05:02:14.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZroQuJ/gA4JbUaxxqYJ6Iatdcog=', 'Content-Length': '0'} 2022-09-02T05:02:14.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4d6a6459d256ddd-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:14.414 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.415 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.415 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.416 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.416 DEBUG:boto:Method: GET 2022-09-02T05:02:14.417 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.417 DEBUG:boto:Data: 2022-09-02T05:02:14.418 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.419 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.420 DEBUG:boto:Params: {} 2022-09-02T05:02:14.420 DEBUG:boto:Token: None 2022-09-02T05:02:14.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo= 2022-09-02T05:02:14.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo=', 'Content-Length': '0'} 2022-09-02T05:02:14.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038f17ef509c6cf86-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.425 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.425 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.426 DEBUG:boto:path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.427 DEBUG:boto:auth_path=/xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.427 DEBUG:boto:Method: GET 2022-09-02T05:02:14.428 DEBUG:boto:Path: /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.428 DEBUG:boto:Data: 2022-09-02T05:02:14.429 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.430 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.430 DEBUG:boto:Params: {} 2022-09-02T05:02:14.431 DEBUG:boto:Token: None 2022-09-02T05:02:14.431 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1?versionId=null 2022-09-02T05:02:14.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo= 2022-09-02T05:02:14.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5uvxnX9v7r/MtevO26uqDnHaxCo=', 'Content-Length': '0'} 2022-09-02T05:02:14.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3f3daea4d8713a7-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.436 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.436 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.437 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.437 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.438 DEBUG:boto:Method: GET 2022-09-02T05:02:14.438 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.439 DEBUG:boto:Data: 2022-09-02T05:02:14.439 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.440 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.441 DEBUG:boto:Params: {} 2022-09-02T05:02:14.441 DEBUG:boto:Token: None 2022-09-02T05:02:14.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d3d2c35a14f7dc9-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.446 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.447 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.448 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.448 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.449 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.449 DEBUG:boto:Method: GET 2022-09-02T05:02:14.450 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.450 DEBUG:boto:Data: 2022-09-02T05:02:14.451 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.452 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.452 DEBUG:boto:Params: {} 2022-09-02T05:02:14.453 DEBUG:boto:Token: None 2022-09-02T05:02:14.454 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1591786bacd5311-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.458 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.459 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.460 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.460 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.461 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:14.461 DEBUG:boto:Data: 2022-09-02T05:02:14.462 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.463 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.463 DEBUG:boto:Params: {} 2022-09-02T05:02:14.464 DEBUG:boto:Token: None 2022-09-02T05:02:14.464 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1 2022-09-02T05:02:14.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XT++NYfTzpHfQYvRsasf6nuFmhY= 2022-09-02T05:02:14.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XT++NYfTzpHfQYvRsasf6nuFmhY=', 'Content-Length': '0'} 2022-09-02T05:02:14.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b9c77526c53c18-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.468 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.468 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.469 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.469 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:14.470 DEBUG:boto:Data: 2022-09-02T05:02:14.470 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.471 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.472 DEBUG:boto:Params: {} 2022-09-02T05:02:14.472 DEBUG:boto:Token: None 2022-09-02T05:02:14.473 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1 2022-09-02T05:02:14.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XT++NYfTzpHfQYvRsasf6nuFmhY= 2022-09-02T05:02:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XT++NYfTzpHfQYvRsasf6nuFmhY=', 'Content-Length': '0'} 2022-09-02T05:02:14.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c243c8fcd78a725-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:02:14.478 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.478 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.479 DEBUG:boto:Method: GET 2022-09-02T05:02:14.479 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:14.480 DEBUG:boto:Data: 2022-09-02T05:02:14.481 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.482 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.482 DEBUG:boto:Params: {} 2022-09-02T05:02:14.483 DEBUG:boto:Token: None 2022-09-02T05:02:14.483 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1 2022-09-02T05:02:14.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xcLr0Zra9KkWlHXThhcUku3DGFg= 2022-09-02T05:02:14.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xcLr0Zra9KkWlHXThhcUku3DGFg=', 'Content-Length': '0'} 2022-09-02T05:02:14.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e09adad02ac0b0e-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.487 DEBUG:boto:path=/xljbkf-59/obj1 2022-09-02T05:02:14.488 DEBUG:boto:auth_path=/xljbkf-59/obj1 2022-09-02T05:02:14.488 DEBUG:boto:Method: GET 2022-09-02T05:02:14.489 DEBUG:boto:Path: /xljbkf-59/obj1 2022-09-02T05:02:14.489 DEBUG:boto:Data: 2022-09-02T05:02:14.490 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.491 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.491 DEBUG:boto:Params: {} 2022-09-02T05:02:14.492 DEBUG:boto:Token: None 2022-09-02T05:02:14.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj1 2022-09-02T05:02:14.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xcLr0Zra9KkWlHXThhcUku3DGFg= 2022-09-02T05:02:14.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xcLr0Zra9KkWlHXThhcUku3DGFg=', 'Content-Length': '0'} 2022-09-02T05:02:14.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d27943c064bc7f86-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.497 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.497 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.498 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.498 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.499 DEBUG:boto:Method: GET 2022-09-02T05:02:14.499 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.500 DEBUG:boto:Data: 2022-09-02T05:02:14.500 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.501 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.502 DEBUG:boto:Params: {} 2022-09-02T05:02:14.503 DEBUG:boto:Token: None 2022-09-02T05:02:14.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000302cdc6561c05a8d-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.550 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.551 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.552 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.552 DEBUG:boto:path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.553 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.553 DEBUG:boto:Method: GET 2022-09-02T05:02:14.554 DEBUG:boto:Path: /xljbkf-59/?prefix=obj1 2022-09-02T05:02:14.554 DEBUG:boto:Data: 2022-09-02T05:02:14.555 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.556 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.557 DEBUG:boto:Params: {} 2022-09-02T05:02:14.557 DEBUG:boto:Token: None 2022-09-02T05:02:14.558 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023ed53fd54956647-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.563 DEBUG:boto:b'xljbkf-59obj11000falseobj12022-09-02T05:01:55.558Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:14.564 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.564 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.565 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.565 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.566 DEBUG:boto:Method: GET 2022-09-02T05:02:14.567 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.567 DEBUG:boto:Data: 2022-09-02T05:02:14.568 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.569 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.569 DEBUG:boto:Params: {} 2022-09-02T05:02:14.570 DEBUG:boto:Token: None 2022-09-02T05:02:14.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.571 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y= 2022-09-02T05:02:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y=', 'Content-Length': '0'} 2022-09-02T05:02:14.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b24ca9eac6ec36-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.575 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.575 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.576 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.576 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.577 DEBUG:boto:Method: GET 2022-09-02T05:02:14.577 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.578 DEBUG:boto:Data: 2022-09-02T05:02:14.578 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.580 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.580 DEBUG:boto:Params: {} 2022-09-02T05:02:14.581 DEBUG:boto:Token: None 2022-09-02T05:02:14.581 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y= 2022-09-02T05:02:14.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y=', 'Content-Length': '0'} 2022-09-02T05:02:14.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010a9ac0524129b2f-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.585 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.586 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.586 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.587 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.587 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.588 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.588 DEBUG:boto:Data: 2022-09-02T05:02:14.589 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.590 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.591 DEBUG:boto:Params: {} 2022-09-02T05:02:14.591 DEBUG:boto:Token: None 2022-09-02T05:02:14.592 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xGT2XoH8lHvlwHwC21S+2QsJHCI= 2022-09-02T05:02:14.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xGT2XoH8lHvlwHwC21S+2QsJHCI=', 'Content-Length': '0'} 2022-09-02T05:02:14.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfe51acaf4f315d3-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.596 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.596 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.597 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.597 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.598 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.598 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.599 DEBUG:boto:Data: 2022-09-02T05:02:14.599 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.600 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.601 DEBUG:boto:Params: {} 2022-09-02T05:02:14.602 DEBUG:boto:Token: None 2022-09-02T05:02:14.602 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xGT2XoH8lHvlwHwC21S+2QsJHCI= 2022-09-02T05:02:14.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xGT2XoH8lHvlwHwC21S+2QsJHCI=', 'Content-Length': '0'} 2022-09-02T05:02:14.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e55a7d901a6cb904-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:14.607 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.608 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.608 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.609 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.609 DEBUG:boto:Method: GET 2022-09-02T05:02:14.610 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.610 DEBUG:boto:Data: 2022-09-02T05:02:14.611 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.612 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.612 DEBUG:boto:Params: {} 2022-09-02T05:02:14.613 DEBUG:boto:Token: None 2022-09-02T05:02:14.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y= 2022-09-02T05:02:14.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y=', 'Content-Length': '0'} 2022-09-02T05:02:14.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c78e132162f558c-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.618 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.618 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.619 DEBUG:boto:path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.619 DEBUG:boto:auth_path=/xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.620 DEBUG:boto:Method: GET 2022-09-02T05:02:14.620 DEBUG:boto:Path: /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.621 DEBUG:boto:Data: 2022-09-02T05:02:14.621 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.622 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.623 DEBUG:boto:Params: {} 2022-09-02T05:02:14.624 DEBUG:boto:Token: None 2022-09-02T05:02:14.624 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2?versionId=null 2022-09-02T05:02:14.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y= 2022-09-02T05:02:14.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:36d7GtMghzZxnqdNOjqsfCi0v8Y=', 'Content-Length': '0'} 2022-09-02T05:02:14.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000467db35db320006d-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.628 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.629 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.629 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.630 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.630 DEBUG:boto:Method: GET 2022-09-02T05:02:14.631 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.631 DEBUG:boto:Data: 2022-09-02T05:02:14.632 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.633 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.634 DEBUG:boto:Params: {} 2022-09-02T05:02:14.634 DEBUG:boto:Token: None 2022-09-02T05:02:14.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c08df49a4d3c7cc-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.639 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.640 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.640 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.641 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.641 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.642 DEBUG:boto:Method: GET 2022-09-02T05:02:14.642 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.643 DEBUG:boto:Data: 2022-09-02T05:02:14.643 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.644 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.644 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.645 DEBUG:boto:Params: {} 2022-09-02T05:02:14.646 DEBUG:boto:Token: None 2022-09-02T05:02:14.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee3a9f2adca50d4-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.651 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.652 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.652 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.653 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.653 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:14.654 DEBUG:boto:Data: 2022-09-02T05:02:14.654 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.655 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.656 DEBUG:boto:Params: {} 2022-09-02T05:02:14.657 DEBUG:boto:Token: None 2022-09-02T05:02:14.657 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2 2022-09-02T05:02:14.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UdQ9juyj2wIO1M4Adh1VFdODdRQ= 2022-09-02T05:02:14.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UdQ9juyj2wIO1M4Adh1VFdODdRQ=', 'Content-Length': '0'} 2022-09-02T05:02:14.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019ea512f65472276-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.661 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.662 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.662 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.663 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:14.663 DEBUG:boto:Data: 2022-09-02T05:02:14.664 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.665 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.665 DEBUG:boto:Params: {} 2022-09-02T05:02:14.666 DEBUG:boto:Token: None 2022-09-02T05:02:14.666 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2 2022-09-02T05:02:14.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UdQ9juyj2wIO1M4Adh1VFdODdRQ= 2022-09-02T05:02:14.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UdQ9juyj2wIO1M4Adh1VFdODdRQ=', 'Content-Length': '0'} 2022-09-02T05:02:14.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d424ed723831a4b1-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:02:14.671 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.671 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.672 DEBUG:boto:Method: GET 2022-09-02T05:02:14.673 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:14.673 DEBUG:boto:Data: 2022-09-02T05:02:14.674 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.675 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.675 DEBUG:boto:Params: {} 2022-09-02T05:02:14.676 DEBUG:boto:Token: None 2022-09-02T05:02:14.676 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2 2022-09-02T05:02:14.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IQpn5/f7BzsNfZQ2vz0i9X1NNeM= 2022-09-02T05:02:14.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IQpn5/f7BzsNfZQ2vz0i9X1NNeM=', 'Content-Length': '0'} 2022-09-02T05:02:14.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2fb9e95fee55e23-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.722 DEBUG:boto:path=/xljbkf-59/obj2 2022-09-02T05:02:14.723 DEBUG:boto:auth_path=/xljbkf-59/obj2 2022-09-02T05:02:14.723 DEBUG:boto:Method: GET 2022-09-02T05:02:14.724 DEBUG:boto:Path: /xljbkf-59/obj2 2022-09-02T05:02:14.725 DEBUG:boto:Data: 2022-09-02T05:02:14.725 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.727 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.727 DEBUG:boto:Params: {} 2022-09-02T05:02:14.728 DEBUG:boto:Token: None 2022-09-02T05:02:14.728 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj2 2022-09-02T05:02:14.729 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IQpn5/f7BzsNfZQ2vz0i9X1NNeM= 2022-09-02T05:02:14.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IQpn5/f7BzsNfZQ2vz0i9X1NNeM=', 'Content-Length': '0'} 2022-09-02T05:02:14.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be6d29dfe7fe8b44-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.733 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.733 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.734 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.734 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.735 DEBUG:boto:Method: GET 2022-09-02T05:02:14.735 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.736 DEBUG:boto:Data: 2022-09-02T05:02:14.737 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.738 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.739 DEBUG:boto:Params: {} 2022-09-02T05:02:14.740 DEBUG:boto:Token: None 2022-09-02T05:02:14.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f1048678fe7542-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.744 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.745 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.746 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.746 DEBUG:boto:path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.747 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.748 DEBUG:boto:Method: GET 2022-09-02T05:02:14.748 DEBUG:boto:Path: /xljbkf-59/?prefix=obj2 2022-09-02T05:02:14.749 DEBUG:boto:Data: 2022-09-02T05:02:14.749 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.750 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.750 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.751 DEBUG:boto:Params: {} 2022-09-02T05:02:14.751 DEBUG:boto:Token: None 2022-09-02T05:02:14.752 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094bba048014390ea-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.757 DEBUG:boto:b'xljbkf-59obj21000falseobj22022-09-02T05:01:55.580Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:14.758 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.759 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.759 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.760 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.760 DEBUG:boto:Method: GET 2022-09-02T05:02:14.761 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.761 DEBUG:boto:Data: 2022-09-02T05:02:14.762 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.763 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.763 DEBUG:boto:Params: {} 2022-09-02T05:02:14.764 DEBUG:boto:Token: None 2022-09-02T05:02:14.764 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc= 2022-09-02T05:02:14.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc=', 'Content-Length': '0'} 2022-09-02T05:02:14.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039dd14f95ba47b45-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.768 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.769 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.769 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.770 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.770 DEBUG:boto:Method: GET 2022-09-02T05:02:14.771 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.771 DEBUG:boto:Data: 2022-09-02T05:02:14.772 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.773 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.774 DEBUG:boto:Params: {} 2022-09-02T05:02:14.774 DEBUG:boto:Token: None 2022-09-02T05:02:14.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc= 2022-09-02T05:02:14.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc=', 'Content-Length': '0'} 2022-09-02T05:02:14.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bbd05183e3f60af-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.779 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.780 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.780 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.781 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.781 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.782 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.782 DEBUG:boto:Data: 2022-09-02T05:02:14.783 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.784 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.784 DEBUG:boto:Params: {} 2022-09-02T05:02:14.785 DEBUG:boto:Token: None 2022-09-02T05:02:14.785 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EP0QVs68xkBIs01BPemzBp8xxTY= 2022-09-02T05:02:14.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EP0QVs68xkBIs01BPemzBp8xxTY=', 'Content-Length': '0'} 2022-09-02T05:02:14.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095da94252b006898-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.789 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.789 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.790 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.790 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.791 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.791 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.792 DEBUG:boto:Data: 2022-09-02T05:02:14.792 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.793 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.794 DEBUG:boto:Params: {} 2022-09-02T05:02:14.794 DEBUG:boto:Token: None 2022-09-02T05:02:14.795 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EP0QVs68xkBIs01BPemzBp8xxTY= 2022-09-02T05:02:14.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EP0QVs68xkBIs01BPemzBp8xxTY=', 'Content-Length': '0'} 2022-09-02T05:02:14.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9c2997bc6cd9d93-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:14.799 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.800 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.800 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.801 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.802 DEBUG:boto:Method: GET 2022-09-02T05:02:14.802 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.803 DEBUG:boto:Data: 2022-09-02T05:02:14.803 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.804 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.805 DEBUG:boto:Params: {} 2022-09-02T05:02:14.805 DEBUG:boto:Token: None 2022-09-02T05:02:14.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc= 2022-09-02T05:02:14.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc=', 'Content-Length': '0'} 2022-09-02T05:02:14.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8d596d259ee6d31-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.810 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.810 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.811 DEBUG:boto:path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.811 DEBUG:boto:auth_path=/xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.812 DEBUG:boto:Method: GET 2022-09-02T05:02:14.812 DEBUG:boto:Path: /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.813 DEBUG:boto:Data: 2022-09-02T05:02:14.814 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.815 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.815 DEBUG:boto:Params: {} 2022-09-02T05:02:14.816 DEBUG:boto:Token: None 2022-09-02T05:02:14.817 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3?versionId=null 2022-09-02T05:02:14.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc= 2022-09-02T05:02:14.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFYRaZiDVPbU0kcad/TAEdj+uqc=', 'Content-Length': '0'} 2022-09-02T05:02:14.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb3328f3cac28655-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.823 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.824 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.825 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.826 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.827 DEBUG:boto:Method: GET 2022-09-02T05:02:14.828 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.829 DEBUG:boto:Data: 2022-09-02T05:02:14.830 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.832 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.833 DEBUG:boto:Params: {} 2022-09-02T05:02:14.834 DEBUG:boto:Token: None 2022-09-02T05:02:14.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f04deef9d7d4cae4-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.841 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.843 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.844 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.845 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.846 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.847 DEBUG:boto:Method: GET 2022-09-02T05:02:14.847 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.848 DEBUG:boto:Data: 2022-09-02T05:02:14.849 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.850 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.850 DEBUG:boto:Params: {} 2022-09-02T05:02:14.851 DEBUG:boto:Token: None 2022-09-02T05:02:14.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5cf264f8ea2b19-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.898 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.899 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.900 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.900 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.901 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.901 DEBUG:boto:Data: 2022-09-02T05:02:14.902 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.903 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.903 DEBUG:boto:Params: {} 2022-09-02T05:02:14.904 DEBUG:boto:Token: None 2022-09-02T05:02:14.904 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ= 2022-09-02T05:02:14.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ=', 'Content-Length': '0'} 2022-09-02T05:02:14.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6669056758568e9-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.908 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.909 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.909 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.910 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.910 DEBUG:boto:Data: 2022-09-02T05:02:14.911 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.912 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.913 DEBUG:boto:Params: {} 2022-09-02T05:02:14.913 DEBUG:boto:Token: None 2022-09-02T05:02:14.914 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ= 2022-09-02T05:02:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4AWboU976lqBz6a1JlpJfwx9HQ=', 'Content-Length': '0'} 2022-09-02T05:02:14.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f5f24304107b689-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:02:14.918 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.919 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.919 DEBUG:boto:Method: GET 2022-09-02T05:02:14.920 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.920 DEBUG:boto:Data: 2022-09-02T05:02:14.921 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.922 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.922 DEBUG:boto:Params: {} 2022-09-02T05:02:14.923 DEBUG:boto:Token: None 2022-09-02T05:02:14.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI= 2022-09-02T05:02:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI=', 'Content-Length': '0'} 2022-09-02T05:02:14.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbaa161dc87f92f3-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.928 DEBUG:boto:path=/xljbkf-59/obj3 2022-09-02T05:02:14.928 DEBUG:boto:auth_path=/xljbkf-59/obj3 2022-09-02T05:02:14.929 DEBUG:boto:Method: GET 2022-09-02T05:02:14.929 DEBUG:boto:Path: /xljbkf-59/obj3 2022-09-02T05:02:14.930 DEBUG:boto:Data: 2022-09-02T05:02:14.930 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.931 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.932 DEBUG:boto:Params: {} 2022-09-02T05:02:14.933 DEBUG:boto:Token: None 2022-09-02T05:02:14.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj3 2022-09-02T05:02:14.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI= 2022-09-02T05:02:14.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PxIwVhrpZubeLdtFux3WOO225nI=', 'Content-Length': '0'} 2022-09-02T05:02:14.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a837064ac822aac4-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.937 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.938 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.938 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.939 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.940 DEBUG:boto:Method: GET 2022-09-02T05:02:14.940 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.941 DEBUG:boto:Data: 2022-09-02T05:02:14.941 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.942 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.943 DEBUG:boto:Params: {} 2022-09-02T05:02:14.943 DEBUG:boto:Token: None 2022-09-02T05:02:14.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034465a499c8a6ea9-0063118e56-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.948 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.949 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:14.950 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:14.950 DEBUG:boto:path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.951 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.951 DEBUG:boto:Method: GET 2022-09-02T05:02:14.952 DEBUG:boto:Path: /xljbkf-59/?prefix=obj3 2022-09-02T05:02:14.952 DEBUG:boto:Data: 2022-09-02T05:02:14.953 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.954 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.954 DEBUG:boto:Params: {} 2022-09-02T05:02:14.955 DEBUG:boto:Token: None 2022-09-02T05:02:14.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/ 2022-09-02T05:02:14.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI= 2022-09-02T05:02:14.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5IQ+Y/mkhgPhYHFaPf7leiEYDMI=', 'Content-Length': '0'} 2022-09-02T05:02:14.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd32ff70097799c1-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.960 DEBUG:boto:b'xljbkf-59obj31000falseobj32022-09-02T05:01:55.608Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:14.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:14.961 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.962 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.963 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.963 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.964 DEBUG:boto:Method: GET 2022-09-02T05:02:14.964 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.965 DEBUG:boto:Data: 2022-09-02T05:02:14.965 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.966 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.967 DEBUG:boto:Params: {} 2022-09-02T05:02:14.967 DEBUG:boto:Token: None 2022-09-02T05:02:14.968 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA= 2022-09-02T05:02:14.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA=', 'Content-Length': '0'} 2022-09-02T05:02:14.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cb1937f89cf3006-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.972 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.973 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.973 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.974 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.974 DEBUG:boto:Method: GET 2022-09-02T05:02:14.975 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.975 DEBUG:boto:Data: 2022-09-02T05:02:14.976 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.977 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.977 DEBUG:boto:Params: {} 2022-09-02T05:02:14.978 DEBUG:boto:Token: None 2022-09-02T05:02:14.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA= 2022-09-02T05:02:14.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAgZMIQgGtC47uuRuSvJzcLpfA=', 'Content-Length': '0'} 2022-09-02T05:02:14.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000459234ee6d5aa666-0063118e56-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.983 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.984 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.984 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.985 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.985 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.986 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.987 DEBUG:boto:Data: 2022-09-02T05:02:14.987 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:14.988 DEBUG:boto:Port: 8001 2022-09-02T05:02:14.989 DEBUG:boto:Params: {} 2022-09-02T05:02:14.989 DEBUG:boto:Token: None 2022-09-02T05:02:14.990 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:14 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0KPauRn03Z05nHhDPvpkcQJRKw= 2022-09-02T05:02:14.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0KPauRn03Z05nHhDPvpkcQJRKw=', 'Content-Length': '0'} 2022-09-02T05:02:14.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fdd9548d97246e8-0063118e56-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:14.994 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:14.994 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:14.995 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.996 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.996 DEBUG:boto:Method: HEAD 2022-09-02T05:02:14.997 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:14.997 DEBUG:boto:Data: 2022-09-02T05:02:14.998 DEBUG:boto:Headers: {} 2022-09-02T05:02:14.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:14.999 DEBUG:boto:Port: 8000 2022-09-02T05:02:14.999 DEBUG:boto:Params: {} 2022-09-02T05:02:15.000 DEBUG:boto:Token: None 2022-09-02T05:02:15.000 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aUDGlsRb5yovupVH19ECdjmHkuw= 2022-09-02T05:02:15.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aUDGlsRb5yovupVH19ECdjmHkuw=', 'Content-Length': '0'} 2022-09-02T05:02:15.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e11053dc69a58a95-0063118e57-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:15.005 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:15.006 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:15.007 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.007 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.008 DEBUG:boto:Method: GET 2022-09-02T05:02:15.008 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.009 DEBUG:boto:Data: 2022-09-02T05:02:15.009 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:15.010 DEBUG:boto:Port: 8001 2022-09-02T05:02:15.011 DEBUG:boto:Params: {} 2022-09-02T05:02:15.011 DEBUG:boto:Token: None 2022-09-02T05:02:15.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WwiiLmhkPnLdaL0EkNYylaWhTxk= 2022-09-02T05:02:15.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WwiiLmhkPnLdaL0EkNYylaWhTxk=', 'Content-Length': '0'} 2022-09-02T05:02:15.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b212e7cd2617ac1a-0063118e57-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.016 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:15.017 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:15.017 DEBUG:boto:path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.018 DEBUG:boto:auth_path=/xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.018 DEBUG:boto:Method: GET 2022-09-02T05:02:15.019 DEBUG:boto:Path: /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.020 DEBUG:boto:Data: 2022-09-02T05:02:15.020 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.021 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:15.021 DEBUG:boto:Port: 8000 2022-09-02T05:02:15.022 DEBUG:boto:Params: {} 2022-09-02T05:02:15.023 DEBUG:boto:Token: None 2022-09-02T05:02:15.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4?versionId=null 2022-09-02T05:02:15.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WwiiLmhkPnLdaL0EkNYylaWhTxk= 2022-09-02T05:02:15.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WwiiLmhkPnLdaL0EkNYylaWhTxk=', 'Content-Length': '0'} 2022-09-02T05:02:15.027 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae94e3f1a7b13892-0063118e57-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.070 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:15.071 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:15.072 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.072 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.073 DEBUG:boto:Method: GET 2022-09-02T05:02:15.074 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.074 DEBUG:boto:Data: 2022-09-02T05:02:15.075 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:15.077 DEBUG:boto:Port: 8001 2022-09-02T05:02:15.077 DEBUG:boto:Params: {} 2022-09-02T05:02:15.078 DEBUG:boto:Token: None 2022-09-02T05:02:15.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/ 2022-09-02T05:02:15.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk= 2022-09-02T05:02:15.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk=', 'Content-Length': '0'} 2022-09-02T05:02:15.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c2a8aee696492fb-0063118e57-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.083 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:15.084 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:15.085 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:15.085 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.086 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.087 DEBUG:boto:Method: GET 2022-09-02T05:02:15.087 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.088 DEBUG:boto:Data: 2022-09-02T05:02:15.089 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:15.090 DEBUG:boto:Port: 8000 2022-09-02T05:02:15.090 DEBUG:boto:Params: {} 2022-09-02T05:02:15.091 DEBUG:boto:Token: None 2022-09-02T05:02:15.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/ 2022-09-02T05:02:15.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk= 2022-09-02T05:02:15.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk=', 'Content-Length': '0'} 2022-09-02T05:02:15.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb38cf7750993e56-0063118e57-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.097 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:15.098 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:15.099 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:15.100 DEBUG:boto:Method: HEAD 2022-09-02T05:02:15.100 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:15.101 DEBUG:boto:Data: 2022-09-02T05:02:15.101 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:15.102 DEBUG:boto:Port: 8001 2022-09-02T05:02:15.103 DEBUG:boto:Params: {} 2022-09-02T05:02:15.103 DEBUG:boto:Token: None 2022-09-02T05:02:15.104 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4 2022-09-02T05:02:15.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SYT41dEkA69lD4zk4oQCjx16cQ0= 2022-09-02T05:02:15.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SYT41dEkA69lD4zk4oQCjx16cQ0=', 'Content-Length': '0'} 2022-09-02T05:02:15.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eca40601568045cd-0063118e57-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.109 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:15.110 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:15.111 DEBUG:boto:Method: HEAD 2022-09-02T05:02:15.111 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:15.112 DEBUG:boto:Data: 2022-09-02T05:02:15.113 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:15.114 DEBUG:boto:Port: 8000 2022-09-02T05:02:15.114 DEBUG:boto:Params: {} 2022-09-02T05:02:15.115 DEBUG:boto:Token: None 2022-09-02T05:02:15.115 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4 2022-09-02T05:02:15.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SYT41dEkA69lD4zk4oQCjx16cQ0= 2022-09-02T05:02:15.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SYT41dEkA69lD4zk4oQCjx16cQ0=', 'Content-Length': '0'} 2022-09-02T05:02:15.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c9888f7522e9d06-0063118e57-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:02:15.120 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:15.121 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:15.121 DEBUG:boto:Method: GET 2022-09-02T05:02:15.122 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:15.122 DEBUG:boto:Data: 2022-09-02T05:02:15.123 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:15.124 DEBUG:boto:Port: 8001 2022-09-02T05:02:15.125 DEBUG:boto:Params: {} 2022-09-02T05:02:15.125 DEBUG:boto:Token: None 2022-09-02T05:02:15.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4 2022-09-02T05:02:15.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qt4aj8IJdBiIOg+HYaFQEsnCMnA= 2022-09-02T05:02:15.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qt4aj8IJdBiIOg+HYaFQEsnCMnA=', 'Content-Length': '0'} 2022-09-02T05:02:15.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041b32e3d2e75de9e-0063118e57-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.170 DEBUG:boto:path=/xljbkf-59/obj4 2022-09-02T05:02:15.171 DEBUG:boto:auth_path=/xljbkf-59/obj4 2022-09-02T05:02:15.171 DEBUG:boto:Method: GET 2022-09-02T05:02:15.172 DEBUG:boto:Path: /xljbkf-59/obj4 2022-09-02T05:02:15.172 DEBUG:boto:Data: 2022-09-02T05:02:15.173 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:15.174 DEBUG:boto:Port: 8000 2022-09-02T05:02:15.175 DEBUG:boto:Params: {} 2022-09-02T05:02:15.175 DEBUG:boto:Token: None 2022-09-02T05:02:15.176 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/obj4 2022-09-02T05:02:15.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qt4aj8IJdBiIOg+HYaFQEsnCMnA= 2022-09-02T05:02:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qt4aj8IJdBiIOg+HYaFQEsnCMnA=', 'Content-Length': '0'} 2022-09-02T05:02:15.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fa212e75487b894-0063118e57-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.180 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:15.181 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:15.182 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.182 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.183 DEBUG:boto:Method: GET 2022-09-02T05:02:15.183 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.184 DEBUG:boto:Data: 2022-09-02T05:02:15.184 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:02:15.185 DEBUG:boto:Port: 8001 2022-09-02T05:02:15.186 DEBUG:boto:Params: {} 2022-09-02T05:02:15.187 DEBUG:boto:Token: None 2022-09-02T05:02:15.187 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/ 2022-09-02T05:02:15.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk= 2022-09-02T05:02:15.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk=', 'Content-Length': '0'} 2022-09-02T05:02:15.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d2f0998c1a6ba1-0063118e57-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.191 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:15.192 DEBUG:boto:path=/xljbkf-59/ 2022-09-02T05:02:15.193 DEBUG:boto:auth_path=/xljbkf-59/ 2022-09-02T05:02:15.193 DEBUG:boto:path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.194 DEBUG:boto:auth_path=/xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.194 DEBUG:boto:Method: GET 2022-09-02T05:02:15.195 DEBUG:boto:Path: /xljbkf-59/?prefix=obj4 2022-09-02T05:02:15.196 DEBUG:boto:Data: 2022-09-02T05:02:15.196 DEBUG:boto:Headers: {} 2022-09-02T05:02:15.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:02:15.197 DEBUG:boto:Port: 8000 2022-09-02T05:02:15.198 DEBUG:boto:Params: {} 2022-09-02T05:02:15.199 DEBUG:boto:Token: None 2022-09-02T05:02:15.199 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:02:15 GMT /xljbkf-59/ 2022-09-02T05:02:15.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk= 2022-09-02T05:02:15.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:02:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk/1rPmCsemEctFv9R1HbvOOHRk=', 'Content-Length': '0'} 2022-09-02T05:02:15.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df9288cd1c6029b8-0063118e57-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:02:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:02:15.204 DEBUG:boto:b'xljbkf-59obj41000falseobj42022-09-02T05:01:55.634Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:02:15.205 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-59 zones={test-zone3, test-zone2} 2022-09-02T05:02:15.205 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-60 2022-09-02T05:02:15.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:15.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:15.207 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:15.272 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:15.272 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:15.324 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:15.325+0000 7ff957cf9600 1 waiting to reach incremental sync.. 2022-09-02T05:02:20.325 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:20.327+0000 7ff957cf9600 1 waiting to reach incremental sync.. 2022-09-02T05:02:25.327 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:25.329+0000 7ff957cf9600 1 bucket sync caught up with source: 2022-09-02T05:02:25.328 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:25.328 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:25.329 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:25.329+0000 7ff957cf9600 0 bucket checkpoint complete 2022-09-02T05:02:25.347 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-60 2022-09-02T05:02:25.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:25.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:25.349 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:25.414 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:25.461 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:25.462+0000 7ff1c874b600 1 bucket sync caught up with source: 2022-09-02T05:02:25.461 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:25.461 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:25.462 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:02:25.462+0000 7ff1c874b600 0 bucket checkpoint complete 2022-09-02T05:02:25.475 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-60 2022-09-02T05:02:25.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:25.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:02:25.477 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:02:25.540 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:02:25.540 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:02:25.579 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:25.581+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:25.579 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:25.579 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:30.580 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:30.581+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:30.581 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:30.581 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:35.582 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:35.583+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:35.582 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:35.583 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:40.583 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:40.584+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:40.584 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:40.585 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:45.585 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:45.586+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:45.586 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:45.586 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:50.586 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:50.587+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:50.587 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:50.587 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:02:55.587 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:02:55.588+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:02:55.588 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:02:55.588 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:00.589 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:00.590+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:03:00.589 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , 00000000001.30.5, , , ] 2022-09-02T05:03:00.589 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:05.590 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:05.591+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:03:05.591 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.73.5, , , , , 00000000001.30.5, , , ] 2022-09-02T05:03:05.591 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:10.592 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:10.592+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:03:10.592 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.73.5, , , , , 00000000001.30.5, , , ] 2022-09-02T05:03:10.592 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:15.593 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:15.594+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:03:15.594 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.73.5, , , , , 00000000001.30.5, , , ] 2022-09-02T05:03:15.594 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:20.595 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:20.596+0000 7f8e074ca600 1 waiting for incremental sync to catch up: 2022-09-02T05:03:20.595 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000001.73.5, , , , , 00000000001.30.5, , , ] 2022-09-02T05:03:20.595 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:25.596 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:25.598+0000 7f8e074ca600 1 bucket sync caught up with source: 2022-09-02T05:03:25.596 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:25.597 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.36.5, , 00000000001.73.5, , 00000000001.38.5, , , 00000000001.30.5, , , ] 2022-09-02T05:03:25.597 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:25.598+0000 7f8e074ca600 0 bucket checkpoint complete 2022-09-02T05:03:25.614 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-60 2022-09-02T05:03:25.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.618 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:03:25.682 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:03:25.683 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:03:25.735 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:25.736+0000 7f79dd7ed600 1 bucket sync caught up with source: 2022-09-02T05:03:25.735 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.862.5, , 00000000001.719.5, , 00000000001.598.5, , , 00000000001.458.5, , , ] 2022-09-02T05:03:25.736 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.862.5, , 00000000001.719.5, , 00000000001.598.5, , , 00000000001.458.5, , , ] 2022-09-02T05:03:25.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:25.736+0000 7f79dd7ed600 0 bucket checkpoint complete 2022-09-02T05:03:25.751 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-60 2022-09-02T05:03:25.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.753 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:03:25.809 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:03:25.809 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:03:25.844 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:25.845+0000 7f7998263600 1 bucket sync caught up with source: 2022-09-02T05:03:25.844 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.862.5, , 00000000001.719.5, , 00000000001.598.5, , , 00000000001.458.5, , , ] 2022-09-02T05:03:25.845 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.862.5, , 00000000001.719.5, , 00000000001.598.5, , , 00000000001.458.5, , , ] 2022-09-02T05:03:25.845 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:25.845+0000 7f7998263600 0 bucket checkpoint complete 2022-09-02T05:03:25.857 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-60 2022-09-02T05:03:25.857 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-60 2022-09-02T05:03:25.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.859 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:03:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:03:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:03:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:25.966+0000 7fc5886a3600 1 bucket sync caught up with source: 2022-09-02T05:03:25.965 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.300.5, , 00000000001.235.5, , 00000000001.120.5, , , 00000000001.80.5, , , ] 2022-09-02T05:03:25.965 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.300.5, , 00000000001.235.5, , 00000000001.120.5, , , 00000000001.80.5, , , ] 2022-09-02T05:03:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:25.966+0000 7fc5886a3600 0 bucket checkpoint complete 2022-09-02T05:03:25.980 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-60 2022-09-02T05:03:25.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:25.982 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:03:26.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:03:26.047 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:03:26.084 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:26.086+0000 7fcbd1b89600 1 bucket sync caught up with source: 2022-09-02T05:03:26.085 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.300.5, , 00000000001.235.5, , 00000000001.120.5, , , 00000000001.80.5, , , ] 2022-09-02T05:03:26.085 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.300.5, , 00000000001.235.5, , 00000000001.120.5, , , 00000000001.80.5, , , ] 2022-09-02T05:03:26.085 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:26.086+0000 7fcbd1b89600 0 bucket checkpoint complete 2022-09-02T05:03:26.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-60 2022-09-02T05:03:26.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-60 zones={test-zone2, test-zone1} 2022-09-02T05:03:26.101 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.101 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.102 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.103 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:03:26.103 DEBUG:boto:Data: 2022-09-02T05:03:26.104 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.105 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.105 DEBUG:boto:Params: {} 2022-09-02T05:03:26.106 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:03:26.107 DEBUG:boto:Token: None 2022-09-02T05:03:26.108 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc= 2022-09-02T05:03:26.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc=', 'Content-Length': '0'} 2022-09-02T05:03:26.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf76dcd6cb614071-0063118e9e-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.114 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.115 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.115 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.116 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:03:26.116 DEBUG:boto:Data: 2022-09-02T05:03:26.117 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.118 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.119 DEBUG:boto:Params: {} 2022-09-02T05:03:26.119 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:03:26.120 DEBUG:boto:Token: None 2022-09-02T05:03:26.121 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc= 2022-09-02T05:03:26.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc=', 'Content-Length': '0'} 2022-09-02T05:03:26.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000242fc096655b15c5-0063118e9e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:03:26.126 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.127 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.127 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.128 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.128 DEBUG:boto:Method: GET 2022-09-02T05:03:26.129 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.129 DEBUG:boto:Data: 2022-09-02T05:03:26.130 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.131 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.131 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.132 DEBUG:boto:Params: {} 2022-09-02T05:03:26.132 DEBUG:boto:Token: None 2022-09-02T05:03:26.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/?versions 2022-09-02T05:03:26.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08= 2022-09-02T05:03:26.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08=', 'Content-Length': '0'} 2022-09-02T05:03:26.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e97351837e1439c-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.137 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:03:26.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:03:26.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:03:26.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:03:26.140 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:03:26.141 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.141 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.142 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.142 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.143 DEBUG:boto:Method: GET 2022-09-02T05:03:26.143 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.144 DEBUG:boto:Data: 2022-09-02T05:03:26.144 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.145 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.146 DEBUG:boto:Params: {} 2022-09-02T05:03:26.147 DEBUG:boto:Token: None 2022-09-02T05:03:26.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/?versions 2022-09-02T05:03:26.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08= 2022-09-02T05:03:26.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08=', 'Content-Length': '0'} 2022-09-02T05:03:26.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b029f5a7e4d8bc73-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.151 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:03:26.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:03:26.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:03:26.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:03:26.154 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.155 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.155 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.156 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.156 DEBUG:boto:Method: GET 2022-09-02T05:03:26.157 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.157 DEBUG:boto:Data: 2022-09-02T05:03:26.158 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.159 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.159 DEBUG:boto:Params: {} 2022-09-02T05:03:26.160 DEBUG:boto:Token: None 2022-09-02T05:03:26.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/?versions 2022-09-02T05:03:26.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08= 2022-09-02T05:03:26.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08=', 'Content-Length': '0'} 2022-09-02T05:03:26.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066dfe2564d8b209f-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.165 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.166 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.166 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.167 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.167 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.168 DEBUG:boto:Method: GET 2022-09-02T05:03:26.168 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.169 DEBUG:boto:Data: 2022-09-02T05:03:26.169 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.170 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.171 DEBUG:boto:Params: {} 2022-09-02T05:03:26.171 DEBUG:boto:Token: None 2022-09-02T05:03:26.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/?versions 2022-09-02T05:03:26.172 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08= 2022-09-02T05:03:26.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PBVJMev0W7cnyF8szVjaNZHcg08=', 'Content-Length': '0'} 2022-09-02T05:03:26.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f988c3e91dee466-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.176 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:26.177 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.178 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.178 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.179 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.179 DEBUG:boto:Method: GET 2022-09-02T05:03:26.180 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.181 DEBUG:boto:Data: 2022-09-02T05:03:26.181 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.182 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.183 DEBUG:boto:Params: {} 2022-09-02T05:03:26.183 DEBUG:boto:Token: None 2022-09-02T05:03:26.184 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E= 2022-09-02T05:03:26.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E=', 'Content-Length': '0'} 2022-09-02T05:03:26.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf14b2dcf3862fc8-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.188 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.188 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.189 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.189 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.190 DEBUG:boto:Method: GET 2022-09-02T05:03:26.190 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.191 DEBUG:boto:Data: 2022-09-02T05:03:26.192 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.193 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.193 DEBUG:boto:Params: {} 2022-09-02T05:03:26.194 DEBUG:boto:Token: None 2022-09-02T05:03:26.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E= 2022-09-02T05:03:26.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E=', 'Content-Length': '0'} 2022-09-02T05:03:26.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b435411f57c32e0-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.199 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.199 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.200 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.200 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.201 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.201 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.202 DEBUG:boto:Data: 2022-09-02T05:03:26.202 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.203 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.204 DEBUG:boto:Params: {} 2022-09-02T05:03:26.204 DEBUG:boto:Token: None 2022-09-02T05:03:26.205 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ELgGvAmEklYuJbt3X1H08JvKqm0= 2022-09-02T05:03:26.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ELgGvAmEklYuJbt3X1H08JvKqm0=', 'Content-Length': '0'} 2022-09-02T05:03:26.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091d295e46995a2ab-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.208 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.209 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.209 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.210 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.210 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.211 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.212 DEBUG:boto:Data: 2022-09-02T05:03:26.212 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.213 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.214 DEBUG:boto:Params: {} 2022-09-02T05:03:26.214 DEBUG:boto:Token: None 2022-09-02T05:03:26.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ELgGvAmEklYuJbt3X1H08JvKqm0= 2022-09-02T05:03:26.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ELgGvAmEklYuJbt3X1H08JvKqm0=', 'Content-Length': '0'} 2022-09-02T05:03:26.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e60cf2611fd57e9-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:26.219 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.219 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.220 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.220 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.221 DEBUG:boto:Method: GET 2022-09-02T05:03:26.222 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.222 DEBUG:boto:Data: 2022-09-02T05:03:26.223 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.224 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.224 DEBUG:boto:Params: {} 2022-09-02T05:03:26.225 DEBUG:boto:Token: None 2022-09-02T05:03:26.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E= 2022-09-02T05:03:26.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E=', 'Content-Length': '0'} 2022-09-02T05:03:26.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dec912e94af200f-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.229 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.230 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.230 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.231 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.231 DEBUG:boto:Method: GET 2022-09-02T05:03:26.232 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.232 DEBUG:boto:Data: 2022-09-02T05:03:26.233 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.234 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.235 DEBUG:boto:Params: {} 2022-09-02T05:03:26.235 DEBUG:boto:Token: None 2022-09-02T05:03:26.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:26.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E= 2022-09-02T05:03:26.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSZPQ3OWn8uzTXynFGnm/JHCd3E=', 'Content-Length': '0'} 2022-09-02T05:03:26.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc72075052b823d9-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.240 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.240 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.241 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.241 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.242 DEBUG:boto:Method: GET 2022-09-02T05:03:26.242 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.243 DEBUG:boto:Data: 2022-09-02T05:03:26.243 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.244 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.245 DEBUG:boto:Params: {} 2022-09-02T05:03:26.246 DEBUG:boto:Token: None 2022-09-02T05:03:26.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000004199b9a4a3b1d-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.250 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.251 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.252 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.252 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.253 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.253 DEBUG:boto:Method: GET 2022-09-02T05:03:26.254 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.254 DEBUG:boto:Data: 2022-09-02T05:03:26.255 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.256 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.256 DEBUG:boto:Params: {} 2022-09-02T05:03:26.257 DEBUG:boto:Token: None 2022-09-02T05:03:26.257 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb5276d558a8678c-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.262 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.262 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.263 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.263 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.264 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:26.264 DEBUG:boto:Data: 2022-09-02T05:03:26.265 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.266 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.266 DEBUG:boto:Params: {} 2022-09-02T05:03:26.267 DEBUG:boto:Token: None 2022-09-02T05:03:26.268 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1 2022-09-02T05:03:26.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9iFxH4kZvKoN3Y/KZ8acqT8DIFQ= 2022-09-02T05:03:26.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9iFxH4kZvKoN3Y/KZ8acqT8DIFQ=', 'Content-Length': '0'} 2022-09-02T05:03:26.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000322cfa78052c8573-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.271 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.272 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.272 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.273 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:26.273 DEBUG:boto:Data: 2022-09-02T05:03:26.274 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.275 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.275 DEBUG:boto:Params: {} 2022-09-02T05:03:26.276 DEBUG:boto:Token: None 2022-09-02T05:03:26.277 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1 2022-09-02T05:03:26.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9iFxH4kZvKoN3Y/KZ8acqT8DIFQ= 2022-09-02T05:03:26.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9iFxH4kZvKoN3Y/KZ8acqT8DIFQ=', 'Content-Length': '0'} 2022-09-02T05:03:26.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085120a9332a29b87-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:26.281 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.281 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.282 DEBUG:boto:Method: GET 2022-09-02T05:03:26.282 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:26.283 DEBUG:boto:Data: 2022-09-02T05:03:26.283 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.284 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.285 DEBUG:boto:Params: {} 2022-09-02T05:03:26.285 DEBUG:boto:Token: None 2022-09-02T05:03:26.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1 2022-09-02T05:03:26.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSKnXv/WUd7AsTgREPgd86Af6RI= 2022-09-02T05:03:26.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSKnXv/WUd7AsTgREPgd86Af6RI=', 'Content-Length': '0'} 2022-09-02T05:03:26.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b712cab3894c500-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.290 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:26.291 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:26.291 DEBUG:boto:Method: GET 2022-09-02T05:03:26.292 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:26.292 DEBUG:boto:Data: 2022-09-02T05:03:26.293 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.294 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.294 DEBUG:boto:Params: {} 2022-09-02T05:03:26.295 DEBUG:boto:Token: None 2022-09-02T05:03:26.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj1 2022-09-02T05:03:26.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSKnXv/WUd7AsTgREPgd86Af6RI= 2022-09-02T05:03:26.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSKnXv/WUd7AsTgREPgd86Af6RI=', 'Content-Length': '0'} 2022-09-02T05:03:26.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005adbfc39d5a9fa8f-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.299 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.300 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.300 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.301 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.302 DEBUG:boto:Method: GET 2022-09-02T05:03:26.302 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.303 DEBUG:boto:Data: 2022-09-02T05:03:26.303 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.304 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.305 DEBUG:boto:Params: {} 2022-09-02T05:03:26.306 DEBUG:boto:Token: None 2022-09-02T05:03:26.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d069449a1c9cf59-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.311 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.312 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.312 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.313 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.313 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.314 DEBUG:boto:Method: GET 2022-09-02T05:03:26.315 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:26.315 DEBUG:boto:Data: 2022-09-02T05:03:26.316 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.317 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.317 DEBUG:boto:Params: {} 2022-09-02T05:03:26.318 DEBUG:boto:Token: None 2022-09-02T05:03:26.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f32760bd7276b3-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.323 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:26.324 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.325 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.325 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.326 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.326 DEBUG:boto:Method: GET 2022-09-02T05:03:26.327 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.328 DEBUG:boto:Data: 2022-09-02T05:03:26.328 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.329 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.330 DEBUG:boto:Params: {} 2022-09-02T05:03:26.330 DEBUG:boto:Token: None 2022-09-02T05:03:26.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc= 2022-09-02T05:03:26.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc=', 'Content-Length': '0'} 2022-09-02T05:03:26.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3b94bfe05765b15-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.335 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.335 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.336 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.336 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.337 DEBUG:boto:Method: GET 2022-09-02T05:03:26.337 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.338 DEBUG:boto:Data: 2022-09-02T05:03:26.339 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.340 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.340 DEBUG:boto:Params: {} 2022-09-02T05:03:26.341 DEBUG:boto:Token: None 2022-09-02T05:03:26.341 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc= 2022-09-02T05:03:26.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc=', 'Content-Length': '0'} 2022-09-02T05:03:26.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbd371457e192308-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.386 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.387 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.387 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.388 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.388 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.389 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.389 DEBUG:boto:Data: 2022-09-02T05:03:26.390 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.391 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.392 DEBUG:boto:Params: {} 2022-09-02T05:03:26.392 DEBUG:boto:Token: None 2022-09-02T05:03:26.393 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xr8lEZ5U6F1G+LZm1KO3dUpPMXM= 2022-09-02T05:03:26.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xr8lEZ5U6F1G+LZm1KO3dUpPMXM=', 'Content-Length': '0'} 2022-09-02T05:03:26.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a8a0f4751bb64a2-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.397 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.398 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.399 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.399 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.400 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.400 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.401 DEBUG:boto:Data: 2022-09-02T05:03:26.401 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.402 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.403 DEBUG:boto:Params: {} 2022-09-02T05:03:26.404 DEBUG:boto:Token: None 2022-09-02T05:03:26.404 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xr8lEZ5U6F1G+LZm1KO3dUpPMXM= 2022-09-02T05:03:26.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xr8lEZ5U6F1G+LZm1KO3dUpPMXM=', 'Content-Length': '0'} 2022-09-02T05:03:26.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc05d0fd53a3f66c-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:26.409 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.409 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.410 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.411 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.411 DEBUG:boto:Method: GET 2022-09-02T05:03:26.412 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.412 DEBUG:boto:Data: 2022-09-02T05:03:26.413 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.414 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.414 DEBUG:boto:Params: {} 2022-09-02T05:03:26.415 DEBUG:boto:Token: None 2022-09-02T05:03:26.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc= 2022-09-02T05:03:26.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc=', 'Content-Length': '0'} 2022-09-02T05:03:26.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c90df89744427f5d-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.420 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.420 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.421 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.421 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.422 DEBUG:boto:Method: GET 2022-09-02T05:03:26.422 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.423 DEBUG:boto:Data: 2022-09-02T05:03:26.424 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.425 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.426 DEBUG:boto:Params: {} 2022-09-02T05:03:26.426 DEBUG:boto:Token: None 2022-09-02T05:03:26.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:26.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc= 2022-09-02T05:03:26.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DVftwPEsDXtttJfudH+4rJmkghc=', 'Content-Length': '0'} 2022-09-02T05:03:26.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f245bef738e5a4b0-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.431 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.432 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.432 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.433 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.433 DEBUG:boto:Method: GET 2022-09-02T05:03:26.434 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.434 DEBUG:boto:Data: 2022-09-02T05:03:26.435 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.436 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.437 DEBUG:boto:Params: {} 2022-09-02T05:03:26.437 DEBUG:boto:Token: None 2022-09-02T05:03:26.438 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f05369b2a736adbe-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.442 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.443 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.443 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.444 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.444 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.445 DEBUG:boto:Method: GET 2022-09-02T05:03:26.446 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.446 DEBUG:boto:Data: 2022-09-02T05:03:26.447 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.448 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.449 DEBUG:boto:Params: {} 2022-09-02T05:03:26.449 DEBUG:boto:Token: None 2022-09-02T05:03:26.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008821b05096fa8d2-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.454 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.455 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.456 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.456 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.457 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:26.457 DEBUG:boto:Data: 2022-09-02T05:03:26.458 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.458 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.459 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.459 DEBUG:boto:Params: {} 2022-09-02T05:03:26.460 DEBUG:boto:Token: None 2022-09-02T05:03:26.461 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2 2022-09-02T05:03:26.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FC85ovWqtcA83cr0nD6ClU1wqwc= 2022-09-02T05:03:26.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FC85ovWqtcA83cr0nD6ClU1wqwc=', 'Content-Length': '0'} 2022-09-02T05:03:26.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000603376ef24be5901-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.464 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.465 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.465 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.466 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:26.472 DEBUG:boto:Data: 2022-09-02T05:03:26.473 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.474 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.475 DEBUG:boto:Params: {} 2022-09-02T05:03:26.475 DEBUG:boto:Token: None 2022-09-02T05:03:26.476 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2 2022-09-02T05:03:26.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FC85ovWqtcA83cr0nD6ClU1wqwc= 2022-09-02T05:03:26.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FC85ovWqtcA83cr0nD6ClU1wqwc=', 'Content-Length': '0'} 2022-09-02T05:03:26.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b20898b4dca38720-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:26.480 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.481 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.481 DEBUG:boto:Method: GET 2022-09-02T05:03:26.482 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:26.482 DEBUG:boto:Data: 2022-09-02T05:03:26.483 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.484 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.485 DEBUG:boto:Params: {} 2022-09-02T05:03:26.485 DEBUG:boto:Token: None 2022-09-02T05:03:26.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2 2022-09-02T05:03:26.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O9FNv4PUJMdBImFXCBkAHe6KPvo= 2022-09-02T05:03:26.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O9FNv4PUJMdBImFXCBkAHe6KPvo=', 'Content-Length': '0'} 2022-09-02T05:03:26.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b35b989bb794b64-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.490 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:26.490 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:26.491 DEBUG:boto:Method: GET 2022-09-02T05:03:26.492 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:26.493 DEBUG:boto:Data: 2022-09-02T05:03:26.494 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.496 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.497 DEBUG:boto:Params: {} 2022-09-02T05:03:26.498 DEBUG:boto:Token: None 2022-09-02T05:03:26.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj2 2022-09-02T05:03:26.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O9FNv4PUJMdBImFXCBkAHe6KPvo= 2022-09-02T05:03:26.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O9FNv4PUJMdBImFXCBkAHe6KPvo=', 'Content-Length': '0'} 2022-09-02T05:03:26.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087a85a35dcd5caa7-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.546 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.547 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.548 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.549 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.550 DEBUG:boto:Method: GET 2022-09-02T05:03:26.551 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.552 DEBUG:boto:Data: 2022-09-02T05:03:26.553 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.554 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.555 DEBUG:boto:Params: {} 2022-09-02T05:03:26.556 DEBUG:boto:Token: None 2022-09-02T05:03:26.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec97d45415c9e12-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.561 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.562 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.563 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.563 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.564 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.564 DEBUG:boto:Method: GET 2022-09-02T05:03:26.565 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:26.566 DEBUG:boto:Data: 2022-09-02T05:03:26.566 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.567 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.568 DEBUG:boto:Params: {} 2022-09-02T05:03:26.568 DEBUG:boto:Token: None 2022-09-02T05:03:26.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003379ff4d4ee92d00-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.573 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:26.575 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.575 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.576 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.576 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.577 DEBUG:boto:Method: GET 2022-09-02T05:03:26.577 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.578 DEBUG:boto:Data: 2022-09-02T05:03:26.579 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.580 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.581 DEBUG:boto:Params: {} 2022-09-02T05:03:26.582 DEBUG:boto:Token: None 2022-09-02T05:03:26.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20= 2022-09-02T05:03:26.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20=', 'Content-Length': '0'} 2022-09-02T05:03:26.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b23246953ccedab-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.587 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.588 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.588 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.589 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.589 DEBUG:boto:Method: GET 2022-09-02T05:03:26.590 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.591 DEBUG:boto:Data: 2022-09-02T05:03:26.591 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.593 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.594 DEBUG:boto:Params: {} 2022-09-02T05:03:26.595 DEBUG:boto:Token: None 2022-09-02T05:03:26.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20= 2022-09-02T05:03:26.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20=', 'Content-Length': '0'} 2022-09-02T05:03:26.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000189e3add95dcb76d-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.601 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.602 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.602 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.603 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.603 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.604 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.604 DEBUG:boto:Data: 2022-09-02T05:03:26.605 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.606 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.606 DEBUG:boto:Params: {} 2022-09-02T05:03:26.607 DEBUG:boto:Token: None 2022-09-02T05:03:26.608 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AjedqgA7F3QhzUSBb2bliBoZNTE= 2022-09-02T05:03:26.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AjedqgA7F3QhzUSBb2bliBoZNTE=', 'Content-Length': '0'} 2022-09-02T05:03:26.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d29f60d2f0a15e86-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.611 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.612 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.612 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.613 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.613 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.614 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.614 DEBUG:boto:Data: 2022-09-02T05:03:26.615 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.616 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.616 DEBUG:boto:Params: {} 2022-09-02T05:03:26.617 DEBUG:boto:Token: None 2022-09-02T05:03:26.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AjedqgA7F3QhzUSBb2bliBoZNTE= 2022-09-02T05:03:26.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AjedqgA7F3QhzUSBb2bliBoZNTE=', 'Content-Length': '0'} 2022-09-02T05:03:26.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3e9fc291dae942c-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:26.622 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.622 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.623 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.624 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.624 DEBUG:boto:Method: GET 2022-09-02T05:03:26.625 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.625 DEBUG:boto:Data: 2022-09-02T05:03:26.626 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.627 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.628 DEBUG:boto:Params: {} 2022-09-02T05:03:26.628 DEBUG:boto:Token: None 2022-09-02T05:03:26.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20= 2022-09-02T05:03:26.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20=', 'Content-Length': '0'} 2022-09-02T05:03:26.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc38a8c366bfa144-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.674 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.675 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.675 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.676 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.676 DEBUG:boto:Method: GET 2022-09-02T05:03:26.677 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.678 DEBUG:boto:Data: 2022-09-02T05:03:26.678 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.679 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.680 DEBUG:boto:Params: {} 2022-09-02T05:03:26.681 DEBUG:boto:Token: None 2022-09-02T05:03:26.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:26.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20= 2022-09-02T05:03:26.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aYqg7P5G06WCYJcXpjhsczImW20=', 'Content-Length': '0'} 2022-09-02T05:03:26.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7cfdfedc1b0f2a7-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.686 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.687 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.687 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.688 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.688 DEBUG:boto:Method: GET 2022-09-02T05:03:26.689 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.690 DEBUG:boto:Data: 2022-09-02T05:03:26.691 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.692 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.693 DEBUG:boto:Params: {} 2022-09-02T05:03:26.694 DEBUG:boto:Token: None 2022-09-02T05:03:26.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8d26ff39c2144f-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.700 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.701 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.702 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.702 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.703 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.703 DEBUG:boto:Method: GET 2022-09-02T05:03:26.704 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.704 DEBUG:boto:Data: 2022-09-02T05:03:26.705 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.706 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.707 DEBUG:boto:Params: {} 2022-09-02T05:03:26.707 DEBUG:boto:Token: None 2022-09-02T05:03:26.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e341f69ff248a3d8-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.712 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.713 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.714 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.714 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.715 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:26.715 DEBUG:boto:Data: 2022-09-02T05:03:26.716 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.717 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.718 DEBUG:boto:Params: {} 2022-09-02T05:03:26.718 DEBUG:boto:Token: None 2022-09-02T05:03:26.719 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3 2022-09-02T05:03:26.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VlPqmZRrmJKwAlnMhQCZRpyfSEU= 2022-09-02T05:03:26.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VlPqmZRrmJKwAlnMhQCZRpyfSEU=', 'Content-Length': '0'} 2022-09-02T05:03:26.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000550e9cb110693c01-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.723 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.723 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.724 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.725 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:26.725 DEBUG:boto:Data: 2022-09-02T05:03:26.726 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.727 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.728 DEBUG:boto:Params: {} 2022-09-02T05:03:26.729 DEBUG:boto:Token: None 2022-09-02T05:03:26.730 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3 2022-09-02T05:03:26.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VlPqmZRrmJKwAlnMhQCZRpyfSEU= 2022-09-02T05:03:26.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VlPqmZRrmJKwAlnMhQCZRpyfSEU=', 'Content-Length': '0'} 2022-09-02T05:03:26.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000937230078d981b73-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:26.734 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.735 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.735 DEBUG:boto:Method: GET 2022-09-02T05:03:26.736 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:26.736 DEBUG:boto:Data: 2022-09-02T05:03:26.737 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.738 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.739 DEBUG:boto:Params: {} 2022-09-02T05:03:26.739 DEBUG:boto:Token: None 2022-09-02T05:03:26.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3 2022-09-02T05:03:26.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HqF612kC8Psfof7YfcCskI2zhoU= 2022-09-02T05:03:26.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HqF612kC8Psfof7YfcCskI2zhoU=', 'Content-Length': '0'} 2022-09-02T05:03:26.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adda714b892121d7-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.744 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:26.744 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:26.745 DEBUG:boto:Method: GET 2022-09-02T05:03:26.745 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:26.746 DEBUG:boto:Data: 2022-09-02T05:03:26.747 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.748 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.749 DEBUG:boto:Params: {} 2022-09-02T05:03:26.749 DEBUG:boto:Token: None 2022-09-02T05:03:26.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj3 2022-09-02T05:03:26.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HqF612kC8Psfof7YfcCskI2zhoU= 2022-09-02T05:03:26.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HqF612kC8Psfof7YfcCskI2zhoU=', 'Content-Length': '0'} 2022-09-02T05:03:26.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f76f7ef9944f44e-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.754 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.754 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.755 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.755 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.756 DEBUG:boto:Method: GET 2022-09-02T05:03:26.757 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.757 DEBUG:boto:Data: 2022-09-02T05:03:26.758 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.758 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.759 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.759 DEBUG:boto:Params: {} 2022-09-02T05:03:26.760 DEBUG:boto:Token: None 2022-09-02T05:03:26.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa82a47723ef2b7c-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.765 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.766 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.766 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.767 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.768 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.768 DEBUG:boto:Method: GET 2022-09-02T05:03:26.769 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:26.769 DEBUG:boto:Data: 2022-09-02T05:03:26.770 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.771 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.771 DEBUG:boto:Params: {} 2022-09-02T05:03:26.772 DEBUG:boto:Token: None 2022-09-02T05:03:26.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b69933af2dd246a-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.777 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:26.779 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.779 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.780 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.780 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.781 DEBUG:boto:Method: GET 2022-09-02T05:03:26.781 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.782 DEBUG:boto:Data: 2022-09-02T05:03:26.783 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.784 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.785 DEBUG:boto:Params: {} 2022-09-02T05:03:26.786 DEBUG:boto:Token: None 2022-09-02T05:03:26.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc= 2022-09-02T05:03:26.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc=', 'Content-Length': '0'} 2022-09-02T05:03:26.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd19067e779271d0-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.792 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.792 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.793 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.794 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.795 DEBUG:boto:Method: GET 2022-09-02T05:03:26.795 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.796 DEBUG:boto:Data: 2022-09-02T05:03:26.796 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.797 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.798 DEBUG:boto:Params: {} 2022-09-02T05:03:26.799 DEBUG:boto:Token: None 2022-09-02T05:03:26.799 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc= 2022-09-02T05:03:26.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc=', 'Content-Length': '0'} 2022-09-02T05:03:26.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000729b8e053aa59be4-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.804 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.805 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.805 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.806 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.806 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.807 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.807 DEBUG:boto:Data: 2022-09-02T05:03:26.808 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.809 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.809 DEBUG:boto:Params: {} 2022-09-02T05:03:26.810 DEBUG:boto:Token: None 2022-09-02T05:03:26.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e0DI/x9AyAaonwFpXaEWZ+n9MfU= 2022-09-02T05:03:26.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e0DI/x9AyAaonwFpXaEWZ+n9MfU=', 'Content-Length': '0'} 2022-09-02T05:03:26.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000647be54bf227db48-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.814 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.815 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.815 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.816 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.816 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.817 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.817 DEBUG:boto:Data: 2022-09-02T05:03:26.818 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.819 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.820 DEBUG:boto:Params: {} 2022-09-02T05:03:26.820 DEBUG:boto:Token: None 2022-09-02T05:03:26.821 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e0DI/x9AyAaonwFpXaEWZ+n9MfU= 2022-09-02T05:03:26.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e0DI/x9AyAaonwFpXaEWZ+n9MfU=', 'Content-Length': '0'} 2022-09-02T05:03:26.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cf886024c6ac60f-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:26.825 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.825 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.826 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.826 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.827 DEBUG:boto:Method: GET 2022-09-02T05:03:26.827 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.828 DEBUG:boto:Data: 2022-09-02T05:03:26.828 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.830 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.830 DEBUG:boto:Params: {} 2022-09-02T05:03:26.831 DEBUG:boto:Token: None 2022-09-02T05:03:26.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc= 2022-09-02T05:03:26.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc=', 'Content-Length': '0'} 2022-09-02T05:03:26.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000503481180d3df1cd-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.835 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.836 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.836 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.837 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.837 DEBUG:boto:Method: GET 2022-09-02T05:03:26.838 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.838 DEBUG:boto:Data: 2022-09-02T05:03:26.839 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.840 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.841 DEBUG:boto:Params: {} 2022-09-02T05:03:26.841 DEBUG:boto:Token: None 2022-09-02T05:03:26.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:26.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc= 2022-09-02T05:03:26.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v1O8yU8nWB1zjJJfkJ5S5/9ckvc=', 'Content-Length': '0'} 2022-09-02T05:03:26.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041ab4eb924ceb66e-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.846 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.846 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.847 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.847 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.848 DEBUG:boto:Method: GET 2022-09-02T05:03:26.848 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.849 DEBUG:boto:Data: 2022-09-02T05:03:26.849 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.850 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.851 DEBUG:boto:Params: {} 2022-09-02T05:03:26.851 DEBUG:boto:Token: None 2022-09-02T05:03:26.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf19fcf3a9f71dce-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.898 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.899 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.900 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.900 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.901 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.901 DEBUG:boto:Method: GET 2022-09-02T05:03:26.902 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.902 DEBUG:boto:Data: 2022-09-02T05:03:26.903 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.904 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.904 DEBUG:boto:Params: {} 2022-09-02T05:03:26.905 DEBUG:boto:Token: None 2022-09-02T05:03:26.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d513a42ea4c7645-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.910 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.910 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.911 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.911 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.912 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:26.912 DEBUG:boto:Data: 2022-09-02T05:03:26.913 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.914 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.914 DEBUG:boto:Params: {} 2022-09-02T05:03:26.915 DEBUG:boto:Token: None 2022-09-02T05:03:26.915 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4 2022-09-02T05:03:26.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/Aijazy7u9AnyE6mQXJF+/gH/E= 2022-09-02T05:03:26.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/Aijazy7u9AnyE6mQXJF+/gH/E=', 'Content-Length': '0'} 2022-09-02T05:03:26.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087606a3e783f07f8-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.919 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.920 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.920 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.921 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:26.921 DEBUG:boto:Data: 2022-09-02T05:03:26.922 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.922 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.923 DEBUG:boto:Params: {} 2022-09-02T05:03:26.923 DEBUG:boto:Token: None 2022-09-02T05:03:26.924 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4 2022-09-02T05:03:26.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e/Aijazy7u9AnyE6mQXJF+/gH/E= 2022-09-02T05:03:26.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e/Aijazy7u9AnyE6mQXJF+/gH/E=', 'Content-Length': '0'} 2022-09-02T05:03:26.927 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064b4051a3aa7770d-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:26.928 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.929 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.929 DEBUG:boto:Method: GET 2022-09-02T05:03:26.930 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:26.930 DEBUG:boto:Data: 2022-09-02T05:03:26.931 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.932 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.933 DEBUG:boto:Params: {} 2022-09-02T05:03:26.933 DEBUG:boto:Token: None 2022-09-02T05:03:26.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4 2022-09-02T05:03:26.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O92UZfoMiba6TWKKSK9pzqB260g= 2022-09-02T05:03:26.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O92UZfoMiba6TWKKSK9pzqB260g=', 'Content-Length': '0'} 2022-09-02T05:03:26.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd3cd8b6d6a3834-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.938 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:26.939 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:26.940 DEBUG:boto:Method: GET 2022-09-02T05:03:26.940 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:26.941 DEBUG:boto:Data: 2022-09-02T05:03:26.941 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.943 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.943 DEBUG:boto:Params: {} 2022-09-02T05:03:26.944 DEBUG:boto:Token: None 2022-09-02T05:03:26.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/obj4 2022-09-02T05:03:26.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O92UZfoMiba6TWKKSK9pzqB260g= 2022-09-02T05:03:26.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O92UZfoMiba6TWKKSK9pzqB260g=', 'Content-Length': '0'} 2022-09-02T05:03:26.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000e095c1ad8d90fb-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.949 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.949 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.950 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.951 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.951 DEBUG:boto:Method: GET 2022-09-02T05:03:26.952 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.952 DEBUG:boto:Data: 2022-09-02T05:03:26.953 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:26.954 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.954 DEBUG:boto:Params: {} 2022-09-02T05:03:26.955 DEBUG:boto:Token: None 2022-09-02T05:03:26.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9f864f31d9eeb4-0063118e9e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.960 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.961 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.962 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.962 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.963 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.963 DEBUG:boto:Method: GET 2022-09-02T05:03:26.964 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:26.965 DEBUG:boto:Data: 2022-09-02T05:03:26.965 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.966 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.967 DEBUG:boto:Params: {} 2022-09-02T05:03:26.967 DEBUG:boto:Token: None 2022-09-02T05:03:26.968 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U= 2022-09-02T05:03:26.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FDwS9wQ0i6rk5MWZyLASqRcXS6U=', 'Content-Length': '0'} 2022-09-02T05:03:26.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2ed54c40afe878f-0063118e9e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.972 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:26.973 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-60 zones={test-zone2, test-zone1} 2022-09-02T05:03:26.974 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-60 zones={test-zone3, test-zone1} 2022-09-02T05:03:26.974 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.975 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.976 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.976 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:03:26.977 DEBUG:boto:Data: 2022-09-02T05:03:26.977 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:26.978 DEBUG:boto:Port: 8001 2022-09-02T05:03:26.979 DEBUG:boto:Params: {} 2022-09-02T05:03:26.980 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:03:26.980 DEBUG:boto:Token: None 2022-09-02T05:03:26.981 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc= 2022-09-02T05:03:26.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc=', 'Content-Length': '0'} 2022-09-02T05:03:26.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002527cfe31ce3ea06-0063118e9e-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.986 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.987 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.988 DEBUG:boto:Method: HEAD 2022-09-02T05:03:26.988 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:03:26.989 DEBUG:boto:Data: 2022-09-02T05:03:26.989 DEBUG:boto:Headers: {} 2022-09-02T05:03:26.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:26.990 DEBUG:boto:Port: 8000 2022-09-02T05:03:26.991 DEBUG:boto:Params: {} 2022-09-02T05:03:26.992 DEBUG:boto:Token: None 2022-09-02T05:03:26.992 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:26 GMT /xljbkf-60/ 2022-09-02T05:03:26.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc= 2022-09-02T05:03:26.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wL8EvzDFBX1I3o1eHx6WvWrbQxc=', 'Content-Length': '0'} 2022-09-02T05:03:26.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e850578c4408edf-0063118e9e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:03:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:26.996 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:03:26.997 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:26.997 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:26.998 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.999 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:26.999 DEBUG:boto:Method: GET 2022-09-02T05:03:27.000 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.000 DEBUG:boto:Data: 2022-09-02T05:03:27.001 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.002 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.002 DEBUG:boto:Params: {} 2022-09-02T05:03:27.003 DEBUG:boto:Token: None 2022-09-02T05:03:27.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/?versions 2022-09-02T05:03:27.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA= 2022-09-02T05:03:27.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA=', 'Content-Length': '0'} 2022-09-02T05:03:27.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ab35b1b7f8b42c-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.007 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:03:27.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:03:27.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:03:27.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:03:27.010 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:03:27.011 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.011 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.012 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.012 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.013 DEBUG:boto:Method: GET 2022-09-02T05:03:27.013 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.014 DEBUG:boto:Data: 2022-09-02T05:03:27.014 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.015 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.016 DEBUG:boto:Params: {} 2022-09-02T05:03:27.017 DEBUG:boto:Token: None 2022-09-02T05:03:27.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/?versions 2022-09-02T05:03:27.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA= 2022-09-02T05:03:27.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA=', 'Content-Length': '0'} 2022-09-02T05:03:27.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b346912d6eb304bb-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.021 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:03:27.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:03:27.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:03:27.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:03:27.024 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.025 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.025 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.026 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.026 DEBUG:boto:Method: GET 2022-09-02T05:03:27.027 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.027 DEBUG:boto:Data: 2022-09-02T05:03:27.028 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.029 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.030 DEBUG:boto:Params: {} 2022-09-02T05:03:27.030 DEBUG:boto:Token: None 2022-09-02T05:03:27.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/?versions 2022-09-02T05:03:27.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA= 2022-09-02T05:03:27.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA=', 'Content-Length': '0'} 2022-09-02T05:03:27.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d137fefca1ae5194-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.035 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.036 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.037 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.037 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.038 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.039 DEBUG:boto:Method: GET 2022-09-02T05:03:27.039 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.040 DEBUG:boto:Data: 2022-09-02T05:03:27.040 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.041 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.042 DEBUG:boto:Params: {} 2022-09-02T05:03:27.042 DEBUG:boto:Token: None 2022-09-02T05:03:27.043 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/?versions 2022-09-02T05:03:27.044 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA= 2022-09-02T05:03:27.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA=', 'Content-Length': '0'} 2022-09-02T05:03:27.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b93e535c721e30-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.048 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:27.049 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.050 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.050 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.051 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.051 DEBUG:boto:Method: GET 2022-09-02T05:03:27.052 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.052 DEBUG:boto:Data: 2022-09-02T05:03:27.053 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.054 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.055 DEBUG:boto:Params: {} 2022-09-02T05:03:27.055 DEBUG:boto:Token: None 2022-09-02T05:03:27.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ= 2022-09-02T05:03:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003280b02b7359e62c-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.060 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.061 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.061 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.062 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.062 DEBUG:boto:Method: GET 2022-09-02T05:03:27.063 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.063 DEBUG:boto:Data: 2022-09-02T05:03:27.064 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.065 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.066 DEBUG:boto:Params: {} 2022-09-02T05:03:27.066 DEBUG:boto:Token: None 2022-09-02T05:03:27.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ= 2022-09-02T05:03:27.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086b83857b6176585-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.071 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.072 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.072 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.073 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.073 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.074 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.074 DEBUG:boto:Data: 2022-09-02T05:03:27.075 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.076 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.076 DEBUG:boto:Params: {} 2022-09-02T05:03:27.077 DEBUG:boto:Token: None 2022-09-02T05:03:27.077 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4eU8bS9oJhEbnXYVMmZgBywuGCs= 2022-09-02T05:03:27.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4eU8bS9oJhEbnXYVMmZgBywuGCs=', 'Content-Length': '0'} 2022-09-02T05:03:27.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcbd78468353a6aa-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.081 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.082 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.082 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.083 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.083 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.084 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.084 DEBUG:boto:Data: 2022-09-02T05:03:27.085 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.086 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.086 DEBUG:boto:Params: {} 2022-09-02T05:03:27.087 DEBUG:boto:Token: None 2022-09-02T05:03:27.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4eU8bS9oJhEbnXYVMmZgBywuGCs= 2022-09-02T05:03:27.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4eU8bS9oJhEbnXYVMmZgBywuGCs=', 'Content-Length': '0'} 2022-09-02T05:03:27.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5a065695445b89c-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:27.092 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.093 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.093 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.094 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.094 DEBUG:boto:Method: GET 2022-09-02T05:03:27.095 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.095 DEBUG:boto:Data: 2022-09-02T05:03:27.096 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.097 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.098 DEBUG:boto:Params: {} 2022-09-02T05:03:27.098 DEBUG:boto:Token: None 2022-09-02T05:03:27.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ= 2022-09-02T05:03:27.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4508bde36dec47f-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.103 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.103 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.104 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.104 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.105 DEBUG:boto:Method: GET 2022-09-02T05:03:27.106 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.106 DEBUG:boto:Data: 2022-09-02T05:03:27.107 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.108 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.108 DEBUG:boto:Params: {} 2022-09-02T05:03:27.109 DEBUG:boto:Token: None 2022-09-02T05:03:27.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:27.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ= 2022-09-02T05:03:27.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hGFeaxqu/AIW85vOalCSTPgLLTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dae4cd995b5c6be3-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.154 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.155 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.155 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.156 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.157 DEBUG:boto:Method: GET 2022-09-02T05:03:27.157 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.158 DEBUG:boto:Data: 2022-09-02T05:03:27.158 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.159 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.160 DEBUG:boto:Params: {} 2022-09-02T05:03:27.161 DEBUG:boto:Token: None 2022-09-02T05:03:27.161 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066cee7af985f78ed-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.166 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.167 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.167 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.168 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.168 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.169 DEBUG:boto:Method: GET 2022-09-02T05:03:27.169 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.170 DEBUG:boto:Data: 2022-09-02T05:03:27.170 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.172 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.172 DEBUG:boto:Params: {} 2022-09-02T05:03:27.173 DEBUG:boto:Token: None 2022-09-02T05:03:27.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5bdf6db9e3e6525-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.178 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.179 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.179 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.180 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.180 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:27.181 DEBUG:boto:Data: 2022-09-02T05:03:27.181 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.182 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.183 DEBUG:boto:Params: {} 2022-09-02T05:03:27.183 DEBUG:boto:Token: None 2022-09-02T05:03:27.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1 2022-09-02T05:03:27.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0X9VxxMliSIrrHX7UIZTLakG33Y= 2022-09-02T05:03:27.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0X9VxxMliSIrrHX7UIZTLakG33Y=', 'Content-Length': '0'} 2022-09-02T05:03:27.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040584c20487ba9d8-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.188 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.188 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.189 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.190 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:27.190 DEBUG:boto:Data: 2022-09-02T05:03:27.191 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.192 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.192 DEBUG:boto:Params: {} 2022-09-02T05:03:27.193 DEBUG:boto:Token: None 2022-09-02T05:03:27.193 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1 2022-09-02T05:03:27.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0X9VxxMliSIrrHX7UIZTLakG33Y= 2022-09-02T05:03:27.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0X9VxxMliSIrrHX7UIZTLakG33Y=', 'Content-Length': '0'} 2022-09-02T05:03:27.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000487f84b8b5c817ca-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:27.198 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.199 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.199 DEBUG:boto:Method: GET 2022-09-02T05:03:27.200 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:27.200 DEBUG:boto:Data: 2022-09-02T05:03:27.201 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.202 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.203 DEBUG:boto:Params: {} 2022-09-02T05:03:27.203 DEBUG:boto:Token: None 2022-09-02T05:03:27.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1 2022-09-02T05:03:27.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NMOYvF+dInR59IWvvrjOXNCRNME= 2022-09-02T05:03:27.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NMOYvF+dInR59IWvvrjOXNCRNME=', 'Content-Length': '0'} 2022-09-02T05:03:27.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000443a21cd781b7c2-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.208 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:27.208 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:27.209 DEBUG:boto:Method: GET 2022-09-02T05:03:27.210 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:27.210 DEBUG:boto:Data: 2022-09-02T05:03:27.211 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.212 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.212 DEBUG:boto:Params: {} 2022-09-02T05:03:27.213 DEBUG:boto:Token: None 2022-09-02T05:03:27.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj1 2022-09-02T05:03:27.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NMOYvF+dInR59IWvvrjOXNCRNME= 2022-09-02T05:03:27.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NMOYvF+dInR59IWvvrjOXNCRNME=', 'Content-Length': '0'} 2022-09-02T05:03:27.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030c5d857d6e646c3-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.218 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.218 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.219 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.219 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.220 DEBUG:boto:Method: GET 2022-09-02T05:03:27.220 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.221 DEBUG:boto:Data: 2022-09-02T05:03:27.221 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.223 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.223 DEBUG:boto:Params: {} 2022-09-02T05:03:27.224 DEBUG:boto:Token: None 2022-09-02T05:03:27.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf25734c0b07562-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.229 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.230 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.231 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.231 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.232 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.232 DEBUG:boto:Method: GET 2022-09-02T05:03:27.233 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:27.233 DEBUG:boto:Data: 2022-09-02T05:03:27.234 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.235 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.236 DEBUG:boto:Params: {} 2022-09-02T05:03:27.237 DEBUG:boto:Token: None 2022-09-02T05:03:27.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e239a0eae21fef7-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.242 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:27.244 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.244 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.245 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.245 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.246 DEBUG:boto:Method: GET 2022-09-02T05:03:27.246 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.247 DEBUG:boto:Data: 2022-09-02T05:03:27.247 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.249 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.249 DEBUG:boto:Params: {} 2022-09-02T05:03:27.250 DEBUG:boto:Token: None 2022-09-02T05:03:27.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk= 2022-09-02T05:03:27.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk=', 'Content-Length': '0'} 2022-09-02T05:03:27.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099f75f50974a283d-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.255 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.255 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.256 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.257 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.257 DEBUG:boto:Method: GET 2022-09-02T05:03:27.258 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.258 DEBUG:boto:Data: 2022-09-02T05:03:27.259 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.260 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.260 DEBUG:boto:Params: {} 2022-09-02T05:03:27.261 DEBUG:boto:Token: None 2022-09-02T05:03:27.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk= 2022-09-02T05:03:27.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk=', 'Content-Length': '0'} 2022-09-02T05:03:27.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc2d8a3ff2ec28d5-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.266 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.267 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.268 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.268 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.269 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.269 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.270 DEBUG:boto:Data: 2022-09-02T05:03:27.270 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.272 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.272 DEBUG:boto:Params: {} 2022-09-02T05:03:27.273 DEBUG:boto:Token: None 2022-09-02T05:03:27.273 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8JvwMkru2TARHi2mf/VWKPHFbMo= 2022-09-02T05:03:27.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8JvwMkru2TARHi2mf/VWKPHFbMo=', 'Content-Length': '0'} 2022-09-02T05:03:27.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d56119e15e534b0b-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.277 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.278 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.278 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.279 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.279 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.280 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.281 DEBUG:boto:Data: 2022-09-02T05:03:27.281 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.282 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.283 DEBUG:boto:Params: {} 2022-09-02T05:03:27.283 DEBUG:boto:Token: None 2022-09-02T05:03:27.284 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8JvwMkru2TARHi2mf/VWKPHFbMo= 2022-09-02T05:03:27.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8JvwMkru2TARHi2mf/VWKPHFbMo=', 'Content-Length': '0'} 2022-09-02T05:03:27.287 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004efc2ddd605ef224-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:27.289 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.289 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.290 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.291 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.299 DEBUG:boto:Method: GET 2022-09-02T05:03:27.300 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.300 DEBUG:boto:Data: 2022-09-02T05:03:27.301 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.302 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.303 DEBUG:boto:Params: {} 2022-09-02T05:03:27.304 DEBUG:boto:Token: None 2022-09-02T05:03:27.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk= 2022-09-02T05:03:27.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk=', 'Content-Length': '0'} 2022-09-02T05:03:27.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b89da2b607fadbf-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.350 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.351 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.351 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.352 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.352 DEBUG:boto:Method: GET 2022-09-02T05:03:27.353 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.354 DEBUG:boto:Data: 2022-09-02T05:03:27.354 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.355 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.356 DEBUG:boto:Params: {} 2022-09-02T05:03:27.357 DEBUG:boto:Token: None 2022-09-02T05:03:27.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:27.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk= 2022-09-02T05:03:27.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4c1jzB4ycH31fTSxzJMY/UZEaqk=', 'Content-Length': '0'} 2022-09-02T05:03:27.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008410da6acca6fbe-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.361 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.362 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.363 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.363 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.364 DEBUG:boto:Method: GET 2022-09-02T05:03:27.364 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.365 DEBUG:boto:Data: 2022-09-02T05:03:27.365 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.367 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.367 DEBUG:boto:Params: {} 2022-09-02T05:03:27.368 DEBUG:boto:Token: None 2022-09-02T05:03:27.368 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd845f88852a4d22-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.373 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.374 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.374 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.375 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.376 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.376 DEBUG:boto:Method: GET 2022-09-02T05:03:27.377 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.377 DEBUG:boto:Data: 2022-09-02T05:03:27.378 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.379 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.379 DEBUG:boto:Params: {} 2022-09-02T05:03:27.380 DEBUG:boto:Token: None 2022-09-02T05:03:27.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27cbf5a6dfe6a86-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.385 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.386 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.386 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.387 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.388 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:27.388 DEBUG:boto:Data: 2022-09-02T05:03:27.389 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.390 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.390 DEBUG:boto:Params: {} 2022-09-02T05:03:27.391 DEBUG:boto:Token: None 2022-09-02T05:03:27.391 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2 2022-09-02T05:03:27.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0vDHbJnHyPQRJAENyF7BAWaQKbU= 2022-09-02T05:03:27.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0vDHbJnHyPQRJAENyF7BAWaQKbU=', 'Content-Length': '0'} 2022-09-02T05:03:27.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93911021e36b1f0-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.395 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.396 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.396 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.397 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:27.397 DEBUG:boto:Data: 2022-09-02T05:03:27.398 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.399 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.399 DEBUG:boto:Params: {} 2022-09-02T05:03:27.400 DEBUG:boto:Token: None 2022-09-02T05:03:27.401 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2 2022-09-02T05:03:27.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0vDHbJnHyPQRJAENyF7BAWaQKbU= 2022-09-02T05:03:27.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0vDHbJnHyPQRJAENyF7BAWaQKbU=', 'Content-Length': '0'} 2022-09-02T05:03:27.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000463ddaf0f4ebeb4f-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:27.405 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.406 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.407 DEBUG:boto:Method: GET 2022-09-02T05:03:27.407 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:27.408 DEBUG:boto:Data: 2022-09-02T05:03:27.408 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.409 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.410 DEBUG:boto:Params: {} 2022-09-02T05:03:27.411 DEBUG:boto:Token: None 2022-09-02T05:03:27.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2 2022-09-02T05:03:27.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XaJ2lJiEPIk9mEDCdFR7Lmd3mjU= 2022-09-02T05:03:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XaJ2lJiEPIk9mEDCdFR7Lmd3mjU=', 'Content-Length': '0'} 2022-09-02T05:03:27.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001122614a8bb99242-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.415 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:27.416 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:27.416 DEBUG:boto:Method: GET 2022-09-02T05:03:27.417 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:27.417 DEBUG:boto:Data: 2022-09-02T05:03:27.418 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.419 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.420 DEBUG:boto:Params: {} 2022-09-02T05:03:27.420 DEBUG:boto:Token: None 2022-09-02T05:03:27.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj2 2022-09-02T05:03:27.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XaJ2lJiEPIk9mEDCdFR7Lmd3mjU= 2022-09-02T05:03:27.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XaJ2lJiEPIk9mEDCdFR7Lmd3mjU=', 'Content-Length': '0'} 2022-09-02T05:03:27.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037b92db35deaccd7-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.425 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.425 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.426 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.426 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.427 DEBUG:boto:Method: GET 2022-09-02T05:03:27.427 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.428 DEBUG:boto:Data: 2022-09-02T05:03:27.428 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.430 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.430 DEBUG:boto:Params: {} 2022-09-02T05:03:27.431 DEBUG:boto:Token: None 2022-09-02T05:03:27.431 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ecff979f4c9452c-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.478 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.479 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.480 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.480 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.481 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.482 DEBUG:boto:Method: GET 2022-09-02T05:03:27.482 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:27.483 DEBUG:boto:Data: 2022-09-02T05:03:27.483 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.484 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.485 DEBUG:boto:Params: {} 2022-09-02T05:03:27.485 DEBUG:boto:Token: None 2022-09-02T05:03:27.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78eecf092934eac-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.491 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:27.492 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.493 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.493 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.494 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.494 DEBUG:boto:Method: GET 2022-09-02T05:03:27.495 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.495 DEBUG:boto:Data: 2022-09-02T05:03:27.496 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.497 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.497 DEBUG:boto:Params: {} 2022-09-02T05:03:27.498 DEBUG:boto:Token: None 2022-09-02T05:03:27.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q= 2022-09-02T05:03:27.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q=', 'Content-Length': '0'} 2022-09-02T05:03:27.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003495d746e07a48c2-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.503 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.504 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.505 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.506 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.507 DEBUG:boto:Method: GET 2022-09-02T05:03:27.508 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.509 DEBUG:boto:Data: 2022-09-02T05:03:27.510 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.512 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.513 DEBUG:boto:Params: {} 2022-09-02T05:03:27.514 DEBUG:boto:Token: None 2022-09-02T05:03:27.515 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q= 2022-09-02T05:03:27.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q=', 'Content-Length': '0'} 2022-09-02T05:03:27.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008903c414886f838-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.521 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.522 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.523 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.524 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.525 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.526 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.527 DEBUG:boto:Data: 2022-09-02T05:03:27.528 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.530 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.531 DEBUG:boto:Params: {} 2022-09-02T05:03:27.532 DEBUG:boto:Token: None 2022-09-02T05:03:27.533 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kSeFcNeB/OYu+gNKL1zR/eV6Toc= 2022-09-02T05:03:27.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kSeFcNeB/OYu+gNKL1zR/eV6Toc=', 'Content-Length': '0'} 2022-09-02T05:03:27.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022fcdb9c817937ec-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.539 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.540 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.541 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.541 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.542 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.543 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.544 DEBUG:boto:Data: 2022-09-02T05:03:27.545 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.547 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.548 DEBUG:boto:Params: {} 2022-09-02T05:03:27.549 DEBUG:boto:Token: None 2022-09-02T05:03:27.550 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kSeFcNeB/OYu+gNKL1zR/eV6Toc= 2022-09-02T05:03:27.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kSeFcNeB/OYu+gNKL1zR/eV6Toc=', 'Content-Length': '0'} 2022-09-02T05:03:27.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd3f2990868acb2b-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:27.557 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.558 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.559 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.560 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.560 DEBUG:boto:Method: GET 2022-09-02T05:03:27.561 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.562 DEBUG:boto:Data: 2022-09-02T05:03:27.562 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.563 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.564 DEBUG:boto:Params: {} 2022-09-02T05:03:27.565 DEBUG:boto:Token: None 2022-09-02T05:03:27.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q= 2022-09-02T05:03:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q=', 'Content-Length': '0'} 2022-09-02T05:03:27.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5cc56477d3a3e6c-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.569 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.570 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.570 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.571 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.571 DEBUG:boto:Method: GET 2022-09-02T05:03:27.572 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.572 DEBUG:boto:Data: 2022-09-02T05:03:27.573 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.574 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.575 DEBUG:boto:Params: {} 2022-09-02T05:03:27.575 DEBUG:boto:Token: None 2022-09-02T05:03:27.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:27.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q= 2022-09-02T05:03:27.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iLRRMm7DoWP2b4O6GVzegXzws+Q=', 'Content-Length': '0'} 2022-09-02T05:03:27.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7b12283fde0f3e3-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.622 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.623 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.623 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.624 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.625 DEBUG:boto:Method: GET 2022-09-02T05:03:27.626 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.626 DEBUG:boto:Data: 2022-09-02T05:03:27.627 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.628 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.628 DEBUG:boto:Params: {} 2022-09-02T05:03:27.629 DEBUG:boto:Token: None 2022-09-02T05:03:27.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009332029d23cd394-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.635 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.635 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.636 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.637 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.637 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.638 DEBUG:boto:Method: GET 2022-09-02T05:03:27.638 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.639 DEBUG:boto:Data: 2022-09-02T05:03:27.640 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.641 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.641 DEBUG:boto:Params: {} 2022-09-02T05:03:27.642 DEBUG:boto:Token: None 2022-09-02T05:03:27.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b99fae665e830be-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.648 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.649 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.650 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.651 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.652 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:27.652 DEBUG:boto:Data: 2022-09-02T05:03:27.653 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.654 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.655 DEBUG:boto:Params: {} 2022-09-02T05:03:27.655 DEBUG:boto:Token: None 2022-09-02T05:03:27.656 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3 2022-09-02T05:03:27.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9gM3KnYIebaRX12Sy+83powLZuQ= 2022-09-02T05:03:27.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9gM3KnYIebaRX12Sy+83powLZuQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c11383f0cc4364d-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.660 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.660 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.661 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.661 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:27.662 DEBUG:boto:Data: 2022-09-02T05:03:27.663 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.664 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.664 DEBUG:boto:Params: {} 2022-09-02T05:03:27.665 DEBUG:boto:Token: None 2022-09-02T05:03:27.666 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3 2022-09-02T05:03:27.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9gM3KnYIebaRX12Sy+83powLZuQ= 2022-09-02T05:03:27.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9gM3KnYIebaRX12Sy+83powLZuQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043c60659ec7eee7b-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:27.670 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.671 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.671 DEBUG:boto:Method: GET 2022-09-02T05:03:27.672 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:27.672 DEBUG:boto:Data: 2022-09-02T05:03:27.673 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.674 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.674 DEBUG:boto:Params: {} 2022-09-02T05:03:27.675 DEBUG:boto:Token: None 2022-09-02T05:03:27.676 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3 2022-09-02T05:03:27.676 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qRfYg7IUtseDXtnUiPO9RIg8Zy0= 2022-09-02T05:03:27.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qRfYg7IUtseDXtnUiPO9RIg8Zy0=', 'Content-Length': '0'} 2022-09-02T05:03:27.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054980288bd3e33f8-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.680 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:27.680 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:27.681 DEBUG:boto:Method: GET 2022-09-02T05:03:27.682 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:27.682 DEBUG:boto:Data: 2022-09-02T05:03:27.683 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.684 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.684 DEBUG:boto:Params: {} 2022-09-02T05:03:27.685 DEBUG:boto:Token: None 2022-09-02T05:03:27.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj3 2022-09-02T05:03:27.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qRfYg7IUtseDXtnUiPO9RIg8Zy0= 2022-09-02T05:03:27.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qRfYg7IUtseDXtnUiPO9RIg8Zy0=', 'Content-Length': '0'} 2022-09-02T05:03:27.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aa8aab6a41d6b4e-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.690 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.690 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.691 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.691 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.692 DEBUG:boto:Method: GET 2022-09-02T05:03:27.693 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.693 DEBUG:boto:Data: 2022-09-02T05:03:27.694 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.695 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.695 DEBUG:boto:Params: {} 2022-09-02T05:03:27.696 DEBUG:boto:Token: None 2022-09-02T05:03:27.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e3af67ea955367-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.701 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.702 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.703 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.703 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.704 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.705 DEBUG:boto:Method: GET 2022-09-02T05:03:27.705 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:27.706 DEBUG:boto:Data: 2022-09-02T05:03:27.706 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.708 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.708 DEBUG:boto:Params: {} 2022-09-02T05:03:27.709 DEBUG:boto:Token: None 2022-09-02T05:03:27.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33d443e89a570ca-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.714 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:27.715 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.715 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.716 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.716 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.717 DEBUG:boto:Method: GET 2022-09-02T05:03:27.718 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.718 DEBUG:boto:Data: 2022-09-02T05:03:27.719 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.720 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.720 DEBUG:boto:Params: {} 2022-09-02T05:03:27.721 DEBUG:boto:Token: None 2022-09-02T05:03:27.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug= 2022-09-02T05:03:27.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug=', 'Content-Length': '0'} 2022-09-02T05:03:27.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f2921374de3c4a0-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.725 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.726 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.727 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.727 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.728 DEBUG:boto:Method: GET 2022-09-02T05:03:27.728 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.729 DEBUG:boto:Data: 2022-09-02T05:03:27.730 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.731 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.731 DEBUG:boto:Params: {} 2022-09-02T05:03:27.732 DEBUG:boto:Token: None 2022-09-02T05:03:27.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug= 2022-09-02T05:03:27.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug=', 'Content-Length': '0'} 2022-09-02T05:03:27.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3f55cb7136137ad-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.738 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.738 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.739 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.739 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.740 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.741 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.742 DEBUG:boto:Data: 2022-09-02T05:03:27.743 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.745 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.746 DEBUG:boto:Params: {} 2022-09-02T05:03:27.747 DEBUG:boto:Token: None 2022-09-02T05:03:27.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+aBJ61wuc5bMquv2LbXx1zYCGUM= 2022-09-02T05:03:27.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+aBJ61wuc5bMquv2LbXx1zYCGUM=', 'Content-Length': '0'} 2022-09-02T05:03:27.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b539545eb8cb862-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.752 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.753 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.753 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.754 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.755 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.755 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.756 DEBUG:boto:Data: 2022-09-02T05:03:27.757 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.758 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.758 DEBUG:boto:Params: {} 2022-09-02T05:03:27.759 DEBUG:boto:Token: None 2022-09-02T05:03:27.760 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+aBJ61wuc5bMquv2LbXx1zYCGUM= 2022-09-02T05:03:27.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+aBJ61wuc5bMquv2LbXx1zYCGUM=', 'Content-Length': '0'} 2022-09-02T05:03:27.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eb2a75feb4b34f2-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:27.764 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.765 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.765 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.766 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.766 DEBUG:boto:Method: GET 2022-09-02T05:03:27.767 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.768 DEBUG:boto:Data: 2022-09-02T05:03:27.768 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.770 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.770 DEBUG:boto:Params: {} 2022-09-02T05:03:27.771 DEBUG:boto:Token: None 2022-09-02T05:03:27.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug= 2022-09-02T05:03:27.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug=', 'Content-Length': '0'} 2022-09-02T05:03:27.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2b2db3f0de75149-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.776 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.777 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.777 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.778 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.778 DEBUG:boto:Method: GET 2022-09-02T05:03:27.779 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.779 DEBUG:boto:Data: 2022-09-02T05:03:27.780 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.781 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.782 DEBUG:boto:Params: {} 2022-09-02T05:03:27.782 DEBUG:boto:Token: None 2022-09-02T05:03:27.783 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:27.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug= 2022-09-02T05:03:27.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YojRkfcid008FHRUq6QLyi0K+ug=', 'Content-Length': '0'} 2022-09-02T05:03:27.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d129a98a47d04892-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.830 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.831 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.832 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.832 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.833 DEBUG:boto:Method: GET 2022-09-02T05:03:27.834 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.834 DEBUG:boto:Data: 2022-09-02T05:03:27.835 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.836 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.837 DEBUG:boto:Params: {} 2022-09-02T05:03:27.837 DEBUG:boto:Token: None 2022-09-02T05:03:27.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a7bcaca0091aea-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.842 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.843 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.844 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.844 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.845 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.845 DEBUG:boto:Method: GET 2022-09-02T05:03:27.846 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.846 DEBUG:boto:Data: 2022-09-02T05:03:27.847 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.848 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.848 DEBUG:boto:Params: {} 2022-09-02T05:03:27.849 DEBUG:boto:Token: None 2022-09-02T05:03:27.850 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3ca30064f07a8ed-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.855 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.856 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.857 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.858 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.858 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:27.859 DEBUG:boto:Data: 2022-09-02T05:03:27.859 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.860 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.861 DEBUG:boto:Params: {} 2022-09-02T05:03:27.861 DEBUG:boto:Token: None 2022-09-02T05:03:27.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4 2022-09-02T05:03:27.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eIi/RiNRsRt32g2ZRuxmPu7B57k= 2022-09-02T05:03:27.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eIi/RiNRsRt32g2ZRuxmPu7B57k=', 'Content-Length': '0'} 2022-09-02T05:03:27.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efbd503e2cccb3f7-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.866 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.866 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.867 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.867 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:27.868 DEBUG:boto:Data: 2022-09-02T05:03:27.869 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.870 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.870 DEBUG:boto:Params: {} 2022-09-02T05:03:27.871 DEBUG:boto:Token: None 2022-09-02T05:03:27.872 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4 2022-09-02T05:03:27.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eIi/RiNRsRt32g2ZRuxmPu7B57k= 2022-09-02T05:03:27.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eIi/RiNRsRt32g2ZRuxmPu7B57k=', 'Content-Length': '0'} 2022-09-02T05:03:27.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b8b80afd92da0c7-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:27.876 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.877 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.877 DEBUG:boto:Method: GET 2022-09-02T05:03:27.878 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:27.878 DEBUG:boto:Data: 2022-09-02T05:03:27.879 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.880 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.880 DEBUG:boto:Params: {} 2022-09-02T05:03:27.881 DEBUG:boto:Token: None 2022-09-02T05:03:27.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4 2022-09-02T05:03:27.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yNRLpyxziYiTxVw9McRTN4kMYyo= 2022-09-02T05:03:27.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yNRLpyxziYiTxVw9McRTN4kMYyo=', 'Content-Length': '0'} 2022-09-02T05:03:27.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ba852e821165552-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.886 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:27.886 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:27.887 DEBUG:boto:Method: GET 2022-09-02T05:03:27.888 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:27.888 DEBUG:boto:Data: 2022-09-02T05:03:27.889 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.890 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.891 DEBUG:boto:Params: {} 2022-09-02T05:03:27.891 DEBUG:boto:Token: None 2022-09-02T05:03:27.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/obj4 2022-09-02T05:03:27.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yNRLpyxziYiTxVw9McRTN4kMYyo= 2022-09-02T05:03:27.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yNRLpyxziYiTxVw9McRTN4kMYyo=', 'Content-Length': '0'} 2022-09-02T05:03:27.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e072899f7da439e-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.938 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.939 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.939 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.940 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.940 DEBUG:boto:Method: GET 2022-09-02T05:03:27.941 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.941 DEBUG:boto:Data: 2022-09-02T05:03:27.942 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.943 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.944 DEBUG:boto:Params: {} 2022-09-02T05:03:27.944 DEBUG:boto:Token: None 2022-09-02T05:03:27.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620739b1089decf5-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.950 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.951 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.952 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.952 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.953 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.953 DEBUG:boto:Method: GET 2022-09-02T05:03:27.954 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:27.954 DEBUG:boto:Data: 2022-09-02T05:03:27.955 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:03:27.956 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.956 DEBUG:boto:Params: {} 2022-09-02T05:03:27.957 DEBUG:boto:Token: None 2022-09-02T05:03:27.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ= 2022-09-02T05:03:27.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:os5zIfqyEAgXhuWYFbggPuGghTQ=', 'Content-Length': '0'} 2022-09-02T05:03:27.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a217ad28fa1ee2-0063118e9f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.962 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.964 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-60 zones={test-zone3, test-zone1} 2022-09-02T05:03:27.964 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-60 zones={test-zone3, test-zone2} 2022-09-02T05:03:27.965 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.965 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.966 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.966 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:03:27.967 DEBUG:boto:Data: 2022-09-02T05:03:27.968 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.969 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.969 DEBUG:boto:Params: {} 2022-09-02T05:03:27.970 DEBUG:boto:Token: None 2022-09-02T05:03:27.970 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+9WApEIrm5At0SPg7fMYzbkoBI= 2022-09-02T05:03:27.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+9WApEIrm5At0SPg7fMYzbkoBI=', 'Content-Length': '0'} 2022-09-02T05:03:27.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003791a4e3977ad08e-0063118e9f-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.974 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.975 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.975 DEBUG:boto:Method: HEAD 2022-09-02T05:03:27.976 DEBUG:boto:Path: /xljbkf-60/ 2022-09-02T05:03:27.977 DEBUG:boto:Data: 2022-09-02T05:03:27.977 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.978 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:27.978 DEBUG:boto:Port: 8000 2022-09-02T05:03:27.979 DEBUG:boto:Params: {} 2022-09-02T05:03:27.979 DEBUG:boto:Token: None 2022-09-02T05:03:27.980 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/ 2022-09-02T05:03:27.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+9WApEIrm5At0SPg7fMYzbkoBI= 2022-09-02T05:03:27.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+9WApEIrm5At0SPg7fMYzbkoBI=', 'Content-Length': '0'} 2022-09-02T05:03:27.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b028ea8b5bdbf0b5-0063118e9f-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.984 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:03:27.985 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:27.985 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:27.986 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.987 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.987 DEBUG:boto:Method: GET 2022-09-02T05:03:27.988 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:27.988 DEBUG:boto:Data: 2022-09-02T05:03:27.989 DEBUG:boto:Headers: {} 2022-09-02T05:03:27.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:27.990 DEBUG:boto:Port: 8001 2022-09-02T05:03:27.990 DEBUG:boto:Params: {} 2022-09-02T05:03:27.991 DEBUG:boto:Token: None 2022-09-02T05:03:27.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:27 GMT /xljbkf-60/?versions 2022-09-02T05:03:27.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA= 2022-09-02T05:03:27.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cv1HGtfmwxLdgssl6z2PO4/bRmA=', 'Content-Length': '0'} 2022-09-02T05:03:27.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044138b302de1184c-0063118e9f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:27.996 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:27.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:03:27.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:03:27.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:03:27.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:03:27.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:03:28.000 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.000 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.001 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.001 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.002 DEBUG:boto:Method: GET 2022-09-02T05:03:28.003 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.003 DEBUG:boto:Data: 2022-09-02T05:03:28.004 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.005 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.006 DEBUG:boto:Params: {} 2022-09-02T05:03:28.006 DEBUG:boto:Token: None 2022-09-02T05:03:28.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/?versions 2022-09-02T05:03:28.008 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+E7lQU8VjafZeZOlW+rrWbULQks= 2022-09-02T05:03:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+E7lQU8VjafZeZOlW+rrWbULQks=', 'Content-Length': '0'} 2022-09-02T05:03:28.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004582059120701cce-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.012 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:03:28.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:03:28.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:03:28.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:03:28.017 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.017 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.018 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.018 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.019 DEBUG:boto:Method: GET 2022-09-02T05:03:28.019 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.020 DEBUG:boto:Data: 2022-09-02T05:03:28.021 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.022 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.023 DEBUG:boto:Params: {} 2022-09-02T05:03:28.024 DEBUG:boto:Token: None 2022-09-02T05:03:28.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/?versions 2022-09-02T05:03:28.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+E7lQU8VjafZeZOlW+rrWbULQks= 2022-09-02T05:03:28.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+E7lQU8VjafZeZOlW+rrWbULQks=', 'Content-Length': '0'} 2022-09-02T05:03:28.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec6758d93871d45-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.030 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.031 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.032 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.033 DEBUG:boto:path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.034 DEBUG:boto:auth_path=/xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.035 DEBUG:boto:Method: GET 2022-09-02T05:03:28.036 DEBUG:boto:Path: /xljbkf-60/?versions&max-keys=999 2022-09-02T05:03:28.036 DEBUG:boto:Data: 2022-09-02T05:03:28.037 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.039 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.040 DEBUG:boto:Params: {} 2022-09-02T05:03:28.041 DEBUG:boto:Token: None 2022-09-02T05:03:28.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/?versions 2022-09-02T05:03:28.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+E7lQU8VjafZeZOlW+rrWbULQks= 2022-09-02T05:03:28.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+E7lQU8VjafZeZOlW+rrWbULQks=', 'Content-Length': '0'} 2022-09-02T05:03:28.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e79b32cf60bc7e-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.048 DEBUG:boto:b'xljbkf-60999falseobj1nulltrue2022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:28.051 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.052 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.053 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.054 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.055 DEBUG:boto:Method: GET 2022-09-02T05:03:28.056 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.057 DEBUG:boto:Data: 2022-09-02T05:03:28.058 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.059 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.060 DEBUG:boto:Params: {} 2022-09-02T05:03:28.061 DEBUG:boto:Token: None 2022-09-02T05:03:28.062 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto= 2022-09-02T05:03:28.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto=', 'Content-Length': '0'} 2022-09-02T05:03:28.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8032d7ad133e07e-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.110 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.112 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.113 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.114 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.115 DEBUG:boto:Method: GET 2022-09-02T05:03:28.116 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.117 DEBUG:boto:Data: 2022-09-02T05:03:28.117 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.119 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.120 DEBUG:boto:Params: {} 2022-09-02T05:03:28.122 DEBUG:boto:Token: None 2022-09-02T05:03:28.123 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto= 2022-09-02T05:03:28.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto=', 'Content-Length': '0'} 2022-09-02T05:03:28.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aca61e2b6977b95b-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.128 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.129 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.129 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.130 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.130 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.131 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.131 DEBUG:boto:Data: 2022-09-02T05:03:28.132 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.133 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.133 DEBUG:boto:Params: {} 2022-09-02T05:03:28.134 DEBUG:boto:Token: None 2022-09-02T05:03:28.135 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:diJQ9e2AF/Xq477ERidbCJ7K/MA= 2022-09-02T05:03:28.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:diJQ9e2AF/Xq477ERidbCJ7K/MA=', 'Content-Length': '0'} 2022-09-02T05:03:28.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b907315eaa5aad71-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.139 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.139 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.140 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.140 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.141 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.141 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.142 DEBUG:boto:Data: 2022-09-02T05:03:28.142 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.143 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.144 DEBUG:boto:Params: {} 2022-09-02T05:03:28.144 DEBUG:boto:Token: None 2022-09-02T05:03:28.145 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:diJQ9e2AF/Xq477ERidbCJ7K/MA= 2022-09-02T05:03:28.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:diJQ9e2AF/Xq477ERidbCJ7K/MA=', 'Content-Length': '0'} 2022-09-02T05:03:28.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e54edb19a30e758-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:28.150 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.151 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.151 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.152 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.152 DEBUG:boto:Method: GET 2022-09-02T05:03:28.153 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.153 DEBUG:boto:Data: 2022-09-02T05:03:28.154 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.155 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.155 DEBUG:boto:Params: {} 2022-09-02T05:03:28.156 DEBUG:boto:Token: None 2022-09-02T05:03:28.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto= 2022-09-02T05:03:28.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto=', 'Content-Length': '0'} 2022-09-02T05:03:28.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0b71123c2b322dc-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.161 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.161 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.162 DEBUG:boto:path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.163 DEBUG:boto:auth_path=/xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.163 DEBUG:boto:Method: GET 2022-09-02T05:03:28.164 DEBUG:boto:Path: /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.164 DEBUG:boto:Data: 2022-09-02T05:03:28.165 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.166 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.166 DEBUG:boto:Params: {} 2022-09-02T05:03:28.167 DEBUG:boto:Token: None 2022-09-02T05:03:28.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1?versionId=null 2022-09-02T05:03:28.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto= 2022-09-02T05:03:28.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yIR4/mt9KheQkQ8h7qCtILj2Dto=', 'Content-Length': '0'} 2022-09-02T05:03:28.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb8459d3ee3d4768-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.172 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.172 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.173 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.173 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.174 DEBUG:boto:Method: GET 2022-09-02T05:03:28.175 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.175 DEBUG:boto:Data: 2022-09-02T05:03:28.176 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.177 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.177 DEBUG:boto:Params: {} 2022-09-02T05:03:28.178 DEBUG:boto:Token: None 2022-09-02T05:03:28.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ad0fb44910a101-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.183 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.184 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.185 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.185 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.186 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.186 DEBUG:boto:Method: GET 2022-09-02T05:03:28.187 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.187 DEBUG:boto:Data: 2022-09-02T05:03:28.188 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.189 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.190 DEBUG:boto:Params: {} 2022-09-02T05:03:28.190 DEBUG:boto:Token: None 2022-09-02T05:03:28.191 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083bd041e4dc0dd57-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.195 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.196 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.197 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.197 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.198 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:28.198 DEBUG:boto:Data: 2022-09-02T05:03:28.199 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.200 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.200 DEBUG:boto:Params: {} 2022-09-02T05:03:28.201 DEBUG:boto:Token: None 2022-09-02T05:03:28.202 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1 2022-09-02T05:03:28.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NALJuUdb4amZG7fV72BWg7P70OA= 2022-09-02T05:03:28.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NALJuUdb4amZG7fV72BWg7P70OA=', 'Content-Length': '0'} 2022-09-02T05:03:28.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3de2746ec541955-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.206 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.207 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.207 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.208 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:28.208 DEBUG:boto:Data: 2022-09-02T05:03:28.209 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.210 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.210 DEBUG:boto:Params: {} 2022-09-02T05:03:28.211 DEBUG:boto:Token: None 2022-09-02T05:03:28.212 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1 2022-09-02T05:03:28.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NALJuUdb4amZG7fV72BWg7P70OA= 2022-09-02T05:03:28.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NALJuUdb4amZG7fV72BWg7P70OA=', 'Content-Length': '0'} 2022-09-02T05:03:28.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1f0912469d31a4b-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:03:28.216 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.217 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.217 DEBUG:boto:Method: GET 2022-09-02T05:03:28.218 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:28.219 DEBUG:boto:Data: 2022-09-02T05:03:28.219 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.220 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.221 DEBUG:boto:Params: {} 2022-09-02T05:03:28.221 DEBUG:boto:Token: None 2022-09-02T05:03:28.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1 2022-09-02T05:03:28.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XEyNcgP95bonqmuuZ0P/q/dalKI= 2022-09-02T05:03:28.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XEyNcgP95bonqmuuZ0P/q/dalKI=', 'Content-Length': '0'} 2022-09-02T05:03:28.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000828cfb0be13a489-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.226 DEBUG:boto:path=/xljbkf-60/obj1 2022-09-02T05:03:28.227 DEBUG:boto:auth_path=/xljbkf-60/obj1 2022-09-02T05:03:28.227 DEBUG:boto:Method: GET 2022-09-02T05:03:28.228 DEBUG:boto:Path: /xljbkf-60/obj1 2022-09-02T05:03:28.228 DEBUG:boto:Data: 2022-09-02T05:03:28.229 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.230 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.230 DEBUG:boto:Params: {} 2022-09-02T05:03:28.231 DEBUG:boto:Token: None 2022-09-02T05:03:28.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj1 2022-09-02T05:03:28.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XEyNcgP95bonqmuuZ0P/q/dalKI= 2022-09-02T05:03:28.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XEyNcgP95bonqmuuZ0P/q/dalKI=', 'Content-Length': '0'} 2022-09-02T05:03:28.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc4ba87445156f92-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.278 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.279 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.279 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.280 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.280 DEBUG:boto:Method: GET 2022-09-02T05:03:28.281 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.281 DEBUG:boto:Data: 2022-09-02T05:03:28.282 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.283 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.284 DEBUG:boto:Params: {} 2022-09-02T05:03:28.284 DEBUG:boto:Token: None 2022-09-02T05:03:28.285 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81529682a518dc8-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.289 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.290 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.291 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.291 DEBUG:boto:path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.292 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.292 DEBUG:boto:Method: GET 2022-09-02T05:03:28.293 DEBUG:boto:Path: /xljbkf-60/?prefix=obj1 2022-09-02T05:03:28.293 DEBUG:boto:Data: 2022-09-02T05:03:28.294 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.295 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.296 DEBUG:boto:Params: {} 2022-09-02T05:03:28.296 DEBUG:boto:Token: None 2022-09-02T05:03:28.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.297 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb2a80fcba525c3-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.301 DEBUG:boto:b'xljbkf-60obj11000falseobj12022-09-02T05:01:55.698Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:28.303 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.303 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.304 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.304 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.305 DEBUG:boto:Method: GET 2022-09-02T05:03:28.305 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.306 DEBUG:boto:Data: 2022-09-02T05:03:28.307 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.308 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.309 DEBUG:boto:Params: {} 2022-09-02T05:03:28.309 DEBUG:boto:Token: None 2022-09-02T05:03:28.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8= 2022-09-02T05:03:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8=', 'Content-Length': '0'} 2022-09-02T05:03:28.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025b29660c893835b-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.314 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.314 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.315 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.315 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.316 DEBUG:boto:Method: GET 2022-09-02T05:03:28.317 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.317 DEBUG:boto:Data: 2022-09-02T05:03:28.318 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.318 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.319 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.319 DEBUG:boto:Params: {} 2022-09-02T05:03:28.320 DEBUG:boto:Token: None 2022-09-02T05:03:28.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8= 2022-09-02T05:03:28.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8=', 'Content-Length': '0'} 2022-09-02T05:03:28.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7d12d182c8e62ea-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.325 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.326 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.326 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.327 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.327 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.328 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.328 DEBUG:boto:Data: 2022-09-02T05:03:28.329 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.330 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.330 DEBUG:boto:Params: {} 2022-09-02T05:03:28.331 DEBUG:boto:Token: None 2022-09-02T05:03:28.331 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DydVPh0qvc4w1NvD75a/jy8FwWk= 2022-09-02T05:03:28.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DydVPh0qvc4w1NvD75a/jy8FwWk=', 'Content-Length': '0'} 2022-09-02T05:03:28.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c2cf435e5f6faa1-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.335 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.336 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.336 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.337 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.337 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.338 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.338 DEBUG:boto:Data: 2022-09-02T05:03:28.339 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.340 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.340 DEBUG:boto:Params: {} 2022-09-02T05:03:28.341 DEBUG:boto:Token: None 2022-09-02T05:03:28.341 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DydVPh0qvc4w1NvD75a/jy8FwWk= 2022-09-02T05:03:28.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DydVPh0qvc4w1NvD75a/jy8FwWk=', 'Content-Length': '0'} 2022-09-02T05:03:28.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c486cf1c4af68c2a-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:28.346 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.347 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.347 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.348 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.348 DEBUG:boto:Method: GET 2022-09-02T05:03:28.349 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.349 DEBUG:boto:Data: 2022-09-02T05:03:28.350 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.351 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.352 DEBUG:boto:Params: {} 2022-09-02T05:03:28.353 DEBUG:boto:Token: None 2022-09-02T05:03:28.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8= 2022-09-02T05:03:28.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8=', 'Content-Length': '0'} 2022-09-02T05:03:28.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eae9c494072e51e0-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.398 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.399 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.399 DEBUG:boto:path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.400 DEBUG:boto:auth_path=/xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.400 DEBUG:boto:Method: GET 2022-09-02T05:03:28.401 DEBUG:boto:Path: /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.402 DEBUG:boto:Data: 2022-09-02T05:03:28.402 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.403 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.404 DEBUG:boto:Params: {} 2022-09-02T05:03:28.404 DEBUG:boto:Token: None 2022-09-02T05:03:28.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2?versionId=null 2022-09-02T05:03:28.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8= 2022-09-02T05:03:28.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:njOPhCo5dIwec1NdgJK5GfUVaP8=', 'Content-Length': '0'} 2022-09-02T05:03:28.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c7990342fef4e01-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.410 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.410 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.411 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.411 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.412 DEBUG:boto:Method: GET 2022-09-02T05:03:28.412 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.413 DEBUG:boto:Data: 2022-09-02T05:03:28.413 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.414 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.415 DEBUG:boto:Params: {} 2022-09-02T05:03:28.416 DEBUG:boto:Token: None 2022-09-02T05:03:28.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5890b927a8d0d6d-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.421 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.422 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.422 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.423 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.423 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.424 DEBUG:boto:Method: GET 2022-09-02T05:03:28.425 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.425 DEBUG:boto:Data: 2022-09-02T05:03:28.426 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.427 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.427 DEBUG:boto:Params: {} 2022-09-02T05:03:28.428 DEBUG:boto:Token: None 2022-09-02T05:03:28.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d39231a477acf08e-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.433 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.434 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.435 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.435 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.436 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:28.436 DEBUG:boto:Data: 2022-09-02T05:03:28.437 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.438 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.438 DEBUG:boto:Params: {} 2022-09-02T05:03:28.439 DEBUG:boto:Token: None 2022-09-02T05:03:28.439 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2 2022-09-02T05:03:28.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OVpsgSxG8iaPGDTJVVIphHpWMjA= 2022-09-02T05:03:28.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OVpsgSxG8iaPGDTJVVIphHpWMjA=', 'Content-Length': '0'} 2022-09-02T05:03:28.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002c120179f3f91c4-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.443 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.444 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.444 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.445 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:28.445 DEBUG:boto:Data: 2022-09-02T05:03:28.446 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.447 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.447 DEBUG:boto:Params: {} 2022-09-02T05:03:28.448 DEBUG:boto:Token: None 2022-09-02T05:03:28.449 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2 2022-09-02T05:03:28.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OVpsgSxG8iaPGDTJVVIphHpWMjA= 2022-09-02T05:03:28.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OVpsgSxG8iaPGDTJVVIphHpWMjA=', 'Content-Length': '0'} 2022-09-02T05:03:28.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020ae7fcf3dd883a7-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:03:28.453 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.454 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.454 DEBUG:boto:Method: GET 2022-09-02T05:03:28.455 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:28.455 DEBUG:boto:Data: 2022-09-02T05:03:28.456 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.457 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.458 DEBUG:boto:Params: {} 2022-09-02T05:03:28.458 DEBUG:boto:Token: None 2022-09-02T05:03:28.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2 2022-09-02T05:03:28.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4eUuEPhNP8iGpZnsQtewmUtlBQo= 2022-09-02T05:03:28.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4eUuEPhNP8iGpZnsQtewmUtlBQo=', 'Content-Length': '0'} 2022-09-02T05:03:28.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a98a22a431aaaa7f-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.462 DEBUG:boto:path=/xljbkf-60/obj2 2022-09-02T05:03:28.463 DEBUG:boto:auth_path=/xljbkf-60/obj2 2022-09-02T05:03:28.464 DEBUG:boto:Method: GET 2022-09-02T05:03:28.464 DEBUG:boto:Path: /xljbkf-60/obj2 2022-09-02T05:03:28.465 DEBUG:boto:Data: 2022-09-02T05:03:28.465 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.466 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.466 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.467 DEBUG:boto:Params: {} 2022-09-02T05:03:28.467 DEBUG:boto:Token: None 2022-09-02T05:03:28.468 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj2 2022-09-02T05:03:28.468 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4eUuEPhNP8iGpZnsQtewmUtlBQo= 2022-09-02T05:03:28.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4eUuEPhNP8iGpZnsQtewmUtlBQo=', 'Content-Length': '0'} 2022-09-02T05:03:28.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5584c407a8ec987-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.472 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.473 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.473 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.474 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.474 DEBUG:boto:Method: GET 2022-09-02T05:03:28.475 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.475 DEBUG:boto:Data: 2022-09-02T05:03:28.476 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.477 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.477 DEBUG:boto:Params: {} 2022-09-02T05:03:28.478 DEBUG:boto:Token: None 2022-09-02T05:03:28.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031026922e68002af-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.483 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.484 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.484 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.485 DEBUG:boto:path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.485 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.486 DEBUG:boto:Method: GET 2022-09-02T05:03:28.487 DEBUG:boto:Path: /xljbkf-60/?prefix=obj2 2022-09-02T05:03:28.488 DEBUG:boto:Data: 2022-09-02T05:03:28.489 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.490 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.490 DEBUG:boto:Params: {} 2022-09-02T05:03:28.491 DEBUG:boto:Token: None 2022-09-02T05:03:28.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d1ad13bc2d98bc-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.496 DEBUG:boto:b'xljbkf-60obj21000falseobj22022-09-02T05:01:55.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:28.497 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.498 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.498 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.499 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.500 DEBUG:boto:Method: GET 2022-09-02T05:03:28.500 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.501 DEBUG:boto:Data: 2022-09-02T05:03:28.501 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.502 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.503 DEBUG:boto:Params: {} 2022-09-02T05:03:28.504 DEBUG:boto:Token: None 2022-09-02T05:03:28.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8= 2022-09-02T05:03:28.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8=', 'Content-Length': '0'} 2022-09-02T05:03:28.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b4918b9eac90376-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.550 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.551 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.551 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.552 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.552 DEBUG:boto:Method: GET 2022-09-02T05:03:28.553 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.554 DEBUG:boto:Data: 2022-09-02T05:03:28.554 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.555 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.556 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.556 DEBUG:boto:Params: {} 2022-09-02T05:03:28.557 DEBUG:boto:Token: None 2022-09-02T05:03:28.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8= 2022-09-02T05:03:28.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8=', 'Content-Length': '0'} 2022-09-02T05:03:28.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093f6e8179e25809b-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.562 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.563 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.563 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.564 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.564 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.565 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.565 DEBUG:boto:Data: 2022-09-02T05:03:28.566 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.567 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.567 DEBUG:boto:Params: {} 2022-09-02T05:03:28.568 DEBUG:boto:Token: None 2022-09-02T05:03:28.569 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NPuWNX6YpPx3D869/r26tnGzAKY= 2022-09-02T05:03:28.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NPuWNX6YpPx3D869/r26tnGzAKY=', 'Content-Length': '0'} 2022-09-02T05:03:28.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000839e39330303f44a-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.573 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.574 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.574 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.575 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.576 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.576 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.577 DEBUG:boto:Data: 2022-09-02T05:03:28.577 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.578 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.579 DEBUG:boto:Params: {} 2022-09-02T05:03:28.580 DEBUG:boto:Token: None 2022-09-02T05:03:28.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NPuWNX6YpPx3D869/r26tnGzAKY= 2022-09-02T05:03:28.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NPuWNX6YpPx3D869/r26tnGzAKY=', 'Content-Length': '0'} 2022-09-02T05:03:28.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000929ac88e9561469c-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:28.587 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.588 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.589 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.590 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.591 DEBUG:boto:Method: GET 2022-09-02T05:03:28.591 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.592 DEBUG:boto:Data: 2022-09-02T05:03:28.593 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.594 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.595 DEBUG:boto:Params: {} 2022-09-02T05:03:28.595 DEBUG:boto:Token: None 2022-09-02T05:03:28.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8= 2022-09-02T05:03:28.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8=', 'Content-Length': '0'} 2022-09-02T05:03:28.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ff4e21b2c7de14a-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.600 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.601 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.601 DEBUG:boto:path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.602 DEBUG:boto:auth_path=/xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.602 DEBUG:boto:Method: GET 2022-09-02T05:03:28.603 DEBUG:boto:Path: /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.603 DEBUG:boto:Data: 2022-09-02T05:03:28.604 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.605 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.606 DEBUG:boto:Params: {} 2022-09-02T05:03:28.607 DEBUG:boto:Token: None 2022-09-02T05:03:28.607 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3?versionId=null 2022-09-02T05:03:28.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8= 2022-09-02T05:03:28.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8FqJd61cVoqTxSX/fyv6SU0Kq8=', 'Content-Length': '0'} 2022-09-02T05:03:28.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a3d65b7a0ce02c2-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.612 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.612 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.613 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.613 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.614 DEBUG:boto:Method: GET 2022-09-02T05:03:28.614 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.615 DEBUG:boto:Data: 2022-09-02T05:03:28.615 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.617 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.617 DEBUG:boto:Params: {} 2022-09-02T05:03:28.618 DEBUG:boto:Token: None 2022-09-02T05:03:28.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b75f4851c38fa6fc-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.623 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.624 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.624 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.625 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.626 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.626 DEBUG:boto:Method: GET 2022-09-02T05:03:28.627 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.627 DEBUG:boto:Data: 2022-09-02T05:03:28.628 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.629 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.629 DEBUG:boto:Params: {} 2022-09-02T05:03:28.630 DEBUG:boto:Token: None 2022-09-02T05:03:28.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080e7829bbac4268-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.636 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.637 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.638 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.639 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.640 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:28.640 DEBUG:boto:Data: 2022-09-02T05:03:28.641 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.642 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.643 DEBUG:boto:Params: {} 2022-09-02T05:03:28.643 DEBUG:boto:Token: None 2022-09-02T05:03:28.644 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3 2022-09-02T05:03:28.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DZyZaZanLNDlcWRJoyWTmTyVI2k= 2022-09-02T05:03:28.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DZyZaZanLNDlcWRJoyWTmTyVI2k=', 'Content-Length': '0'} 2022-09-02T05:03:28.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7194323faac311e-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.648 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.649 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.649 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.650 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:28.650 DEBUG:boto:Data: 2022-09-02T05:03:28.651 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.652 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.653 DEBUG:boto:Params: {} 2022-09-02T05:03:28.653 DEBUG:boto:Token: None 2022-09-02T05:03:28.654 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3 2022-09-02T05:03:28.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DZyZaZanLNDlcWRJoyWTmTyVI2k= 2022-09-02T05:03:28.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DZyZaZanLNDlcWRJoyWTmTyVI2k=', 'Content-Length': '0'} 2022-09-02T05:03:28.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093359e4e7d8450bb-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:03:28.659 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.659 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.660 DEBUG:boto:Method: GET 2022-09-02T05:03:28.660 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:28.661 DEBUG:boto:Data: 2022-09-02T05:03:28.662 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.663 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.664 DEBUG:boto:Params: {} 2022-09-02T05:03:28.664 DEBUG:boto:Token: None 2022-09-02T05:03:28.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3 2022-09-02T05:03:28.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5BkYtjp7HIDMswITZwZtIkVWYgs= 2022-09-02T05:03:28.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5BkYtjp7HIDMswITZwZtIkVWYgs=', 'Content-Length': '0'} 2022-09-02T05:03:28.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cd1e3f13c50d785-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.669 DEBUG:boto:path=/xljbkf-60/obj3 2022-09-02T05:03:28.670 DEBUG:boto:auth_path=/xljbkf-60/obj3 2022-09-02T05:03:28.670 DEBUG:boto:Method: GET 2022-09-02T05:03:28.671 DEBUG:boto:Path: /xljbkf-60/obj3 2022-09-02T05:03:28.671 DEBUG:boto:Data: 2022-09-02T05:03:28.672 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.673 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.674 DEBUG:boto:Params: {} 2022-09-02T05:03:28.674 DEBUG:boto:Token: None 2022-09-02T05:03:28.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj3 2022-09-02T05:03:28.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5BkYtjp7HIDMswITZwZtIkVWYgs= 2022-09-02T05:03:28.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5BkYtjp7HIDMswITZwZtIkVWYgs=', 'Content-Length': '0'} 2022-09-02T05:03:28.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d9b81bd8257c8e2-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.722 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.723 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.724 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.724 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.725 DEBUG:boto:Method: GET 2022-09-02T05:03:28.725 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.726 DEBUG:boto:Data: 2022-09-02T05:03:28.726 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.728 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.728 DEBUG:boto:Params: {} 2022-09-02T05:03:28.729 DEBUG:boto:Token: None 2022-09-02T05:03:28.730 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000611694ff1ec73a8a-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.734 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.735 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.736 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.736 DEBUG:boto:path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.737 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.738 DEBUG:boto:Method: GET 2022-09-02T05:03:28.738 DEBUG:boto:Path: /xljbkf-60/?prefix=obj3 2022-09-02T05:03:28.739 DEBUG:boto:Data: 2022-09-02T05:03:28.739 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.740 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.741 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.741 DEBUG:boto:Params: {} 2022-09-02T05:03:28.742 DEBUG:boto:Token: None 2022-09-02T05:03:28.742 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e248a1d293ecea9-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.747 DEBUG:boto:b'xljbkf-60obj31000falseobj32022-09-02T05:01:55.749Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:28.749 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.749 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.750 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.750 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.751 DEBUG:boto:Method: GET 2022-09-02T05:03:28.751 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.752 DEBUG:boto:Data: 2022-09-02T05:03:28.752 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.754 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.754 DEBUG:boto:Params: {} 2022-09-02T05:03:28.755 DEBUG:boto:Token: None 2022-09-02T05:03:28.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM= 2022-09-02T05:03:28.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM=', 'Content-Length': '0'} 2022-09-02T05:03:28.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ef7d4f2659c5912-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.760 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.760 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.761 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.761 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.762 DEBUG:boto:Method: GET 2022-09-02T05:03:28.763 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.763 DEBUG:boto:Data: 2022-09-02T05:03:28.764 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.765 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.765 DEBUG:boto:Params: {} 2022-09-02T05:03:28.766 DEBUG:boto:Token: None 2022-09-02T05:03:28.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM= 2022-09-02T05:03:28.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM=', 'Content-Length': '0'} 2022-09-02T05:03:28.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9fd0a925f916586-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.771 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.772 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.773 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.774 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.775 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.776 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.777 DEBUG:boto:Data: 2022-09-02T05:03:28.778 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.779 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.780 DEBUG:boto:Params: {} 2022-09-02T05:03:28.781 DEBUG:boto:Token: None 2022-09-02T05:03:28.782 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7SfENuWqOYamZfSjqpKgu8B1gTY= 2022-09-02T05:03:28.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7SfENuWqOYamZfSjqpKgu8B1gTY=', 'Content-Length': '0'} 2022-09-02T05:03:28.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d433be97d1804a17-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.786 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.787 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.787 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.788 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.788 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.789 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.789 DEBUG:boto:Data: 2022-09-02T05:03:28.790 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.791 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.792 DEBUG:boto:Params: {} 2022-09-02T05:03:28.792 DEBUG:boto:Token: None 2022-09-02T05:03:28.793 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7SfENuWqOYamZfSjqpKgu8B1gTY= 2022-09-02T05:03:28.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7SfENuWqOYamZfSjqpKgu8B1gTY=', 'Content-Length': '0'} 2022-09-02T05:03:28.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003da2ec4fb00601a5-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:28.798 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.798 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.799 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.799 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.800 DEBUG:boto:Method: GET 2022-09-02T05:03:28.801 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.802 DEBUG:boto:Data: 2022-09-02T05:03:28.802 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.803 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.804 DEBUG:boto:Params: {} 2022-09-02T05:03:28.805 DEBUG:boto:Token: None 2022-09-02T05:03:28.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM= 2022-09-02T05:03:28.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM=', 'Content-Length': '0'} 2022-09-02T05:03:28.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003265d5b8add13d4-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.850 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.851 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.851 DEBUG:boto:path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.852 DEBUG:boto:auth_path=/xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.853 DEBUG:boto:Method: GET 2022-09-02T05:03:28.853 DEBUG:boto:Path: /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.854 DEBUG:boto:Data: 2022-09-02T05:03:28.854 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.856 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.856 DEBUG:boto:Params: {} 2022-09-02T05:03:28.857 DEBUG:boto:Token: None 2022-09-02T05:03:28.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4?versionId=null 2022-09-02T05:03:28.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM= 2022-09-02T05:03:28.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WauSpr4UEe9evxmXtP0cUtr+cDM=', 'Content-Length': '0'} 2022-09-02T05:03:28.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057567af67fed0eef-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.862 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.863 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.863 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.864 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.864 DEBUG:boto:Method: GET 2022-09-02T05:03:28.865 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.866 DEBUG:boto:Data: 2022-09-02T05:03:28.866 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.867 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.868 DEBUG:boto:Params: {} 2022-09-02T05:03:28.869 DEBUG:boto:Token: None 2022-09-02T05:03:28.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abbcd4c045767cd8-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.874 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.875 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.876 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.876 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.877 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.877 DEBUG:boto:Method: GET 2022-09-02T05:03:28.878 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.879 DEBUG:boto:Data: 2022-09-02T05:03:28.879 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.880 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.881 DEBUG:boto:Params: {} 2022-09-02T05:03:28.881 DEBUG:boto:Token: None 2022-09-02T05:03:28.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e8ba0724778602c-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.887 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.888 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.889 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.889 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.890 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:28.890 DEBUG:boto:Data: 2022-09-02T05:03:28.891 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.893 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.894 DEBUG:boto:Params: {} 2022-09-02T05:03:28.895 DEBUG:boto:Token: None 2022-09-02T05:03:28.896 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4 2022-09-02T05:03:28.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:puU1T2c9OpjODAZKthzHsi+vuKM= 2022-09-02T05:03:28.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:puU1T2c9OpjODAZKthzHsi+vuKM=', 'Content-Length': '0'} 2022-09-02T05:03:28.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d538d3d3b6fb932-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.902 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.903 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.903 DEBUG:boto:Method: HEAD 2022-09-02T05:03:28.904 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:28.904 DEBUG:boto:Data: 2022-09-02T05:03:28.905 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.906 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.906 DEBUG:boto:Params: {} 2022-09-02T05:03:28.907 DEBUG:boto:Token: None 2022-09-02T05:03:28.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4 2022-09-02T05:03:28.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:puU1T2c9OpjODAZKthzHsi+vuKM= 2022-09-02T05:03:28.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:puU1T2c9OpjODAZKthzHsi+vuKM=', 'Content-Length': '0'} 2022-09-02T05:03:28.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a22f8bd736748ec-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:03:28.912 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.913 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.913 DEBUG:boto:Method: GET 2022-09-02T05:03:28.914 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:28.914 DEBUG:boto:Data: 2022-09-02T05:03:28.915 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.916 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.917 DEBUG:boto:Params: {} 2022-09-02T05:03:28.917 DEBUG:boto:Token: None 2022-09-02T05:03:28.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4 2022-09-02T05:03:28.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vVySEglnmbF+mdECBg7kkesKRdI= 2022-09-02T05:03:28.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vVySEglnmbF+mdECBg7kkesKRdI=', 'Content-Length': '0'} 2022-09-02T05:03:28.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000636e7bf94761ef35-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.922 DEBUG:boto:path=/xljbkf-60/obj4 2022-09-02T05:03:28.923 DEBUG:boto:auth_path=/xljbkf-60/obj4 2022-09-02T05:03:28.923 DEBUG:boto:Method: GET 2022-09-02T05:03:28.924 DEBUG:boto:Path: /xljbkf-60/obj4 2022-09-02T05:03:28.924 DEBUG:boto:Data: 2022-09-02T05:03:28.925 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.926 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.926 DEBUG:boto:Params: {} 2022-09-02T05:03:28.927 DEBUG:boto:Token: None 2022-09-02T05:03:28.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/obj4 2022-09-02T05:03:28.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vVySEglnmbF+mdECBg7kkesKRdI= 2022-09-02T05:03:28.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vVySEglnmbF+mdECBg7kkesKRdI=', 'Content-Length': '0'} 2022-09-02T05:03:28.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:01:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7f268a9f7a4ad22-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.974 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.975 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.976 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.976 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.977 DEBUG:boto:Method: GET 2022-09-02T05:03:28.977 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.978 DEBUG:boto:Data: 2022-09-02T05:03:28.978 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:03:28.979 DEBUG:boto:Port: 8001 2022-09-02T05:03:28.980 DEBUG:boto:Params: {} 2022-09-02T05:03:28.980 DEBUG:boto:Token: None 2022-09-02T05:03:28.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f5db48884ad800-0063118ea0-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.986 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.987 DEBUG:boto:path=/xljbkf-60/ 2022-09-02T05:03:28.987 DEBUG:boto:auth_path=/xljbkf-60/ 2022-09-02T05:03:28.988 DEBUG:boto:path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.988 DEBUG:boto:auth_path=/xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.989 DEBUG:boto:Method: GET 2022-09-02T05:03:28.989 DEBUG:boto:Path: /xljbkf-60/?prefix=obj4 2022-09-02T05:03:28.990 DEBUG:boto:Data: 2022-09-02T05:03:28.991 DEBUG:boto:Headers: {} 2022-09-02T05:03:28.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:03:28.992 DEBUG:boto:Port: 8000 2022-09-02T05:03:28.992 DEBUG:boto:Params: {} 2022-09-02T05:03:28.993 DEBUG:boto:Token: None 2022-09-02T05:03:28.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:03:28 GMT /xljbkf-60/ 2022-09-02T05:03:28.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c= 2022-09-02T05:03:28.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:03:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4ZiJET/ZjKRIt2nhnJjKi8Xp7c=', 'Content-Length': '0'} 2022-09-02T05:03:28.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b343eacf92cf226-0063118ea0-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:03:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:03:28.998 DEBUG:boto:b'xljbkf-60obj41000falseobj42022-09-02T05:01:55.771Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:03:28.999 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-60 zones={test-zone3, test-zone2} 2022-09-02T05:03:29.000 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-61 2022-09-02T05:03:29.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:29.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:29.001 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:03:29.065 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:03:29.065 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:03:29.124 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:29.126+0000 7f0c41ab7600 1 bucket sync caught up with source: 2022-09-02T05:03:29.124 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:03:29.125 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:03:29.125 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:03:29.126+0000 7f0c41ab7600 0 bucket checkpoint complete 2022-09-02T05:03:29.148 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-61 2022-09-02T05:03:29.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:29.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:03:29.150 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:03:29.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:03:29.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:03:29.261 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:29.262+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:03:34.261 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:34.263+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:03:39.262 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:39.263+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:03:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:44.264+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:03:49.263 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:49.265+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:03:54.264 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:54.266+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:03:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:03:59.266+0000 7f56cf718600 1 waiting to reach incremental sync.. 2022-09-02T05:04:04.266 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:04:04.267+0000 7f56cf718600 1 bucket sync caught up with source: 2022-09-02T05:04:04.267 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:04.267 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:04.267 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:04:04.267+0000 7f56cf718600 0 bucket checkpoint complete 2022-09-02T05:04:04.284 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-61 2022-09-02T05:04:04.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:04:04.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:04:04.288 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:04:04.353 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:04:04.353 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:04:04.399 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:04.400+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:04.399 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:04.399 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:09.401 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:09.401+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:09.402 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:09.402 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:14.402 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:14.403+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:14.402 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:14.402 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:19.404 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:19.404+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:19.404 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:19.404 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:24.404 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:24.405+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:24.405 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:24.405 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:29.406 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:29.407+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:29.407 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:29.407 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:34.408 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:34.410+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:34.408 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:34.408 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:39.409 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:39.411+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:39.409 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:39.410 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:44.410 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:44.411+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:44.411 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:44.412 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:49.412 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:49.413+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:49.413 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:49.413 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:54.413 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:54.414+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:54.414 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:54.414 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:04:59.415 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:04:59.416+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:04:59.415 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:04:59.416 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:04.416 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:04.417+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:04.417 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:04.417 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:09.417 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:09.418+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:09.418 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:09.418 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:14.419 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:14.420+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:14.419 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:14.420 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:19.420 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:19.421+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:19.421 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:19.421 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:24.422 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:24.422+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:24.422 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:24.422 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:29.423 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:29.424+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:29.423 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:29.424 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:34.425 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:34.425+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:34.425 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:34.425 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:39.426 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:39.428+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:39.427 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:39.427 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:44.428 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:44.429+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:44.429 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:44.429 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:49.429 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:49.430+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:49.429 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:49.430 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:54.430 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:54.431+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:54.431 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:54.431 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:05:59.432 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:05:59.433+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:05:59.432 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:05:59.433 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:04.433 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:04.434+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:04.434 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:04.434 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:09.435 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:09.436+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:09.435 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:09.435 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:14.436 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:14.437+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:14.437 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:14.437 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:19.437 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:19.438+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:19.438 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:19.438 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:24.439 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:24.440+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:24.439 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:24.440 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:29.440 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:29.441+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:29.441 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:29.441 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:34.442 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:34.443+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:34.443 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:34.443 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:39.443 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:39.444+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:39.444 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:39.444 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:44.444 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:44.446+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:44.445 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:44.445 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:49.446 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:49.448+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:49.447 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:49.447 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:54.447 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:54.448+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:54.448 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:54.448 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:06:59.449 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:06:59.450+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:06:59.450 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:06:59.450 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:04.450 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:04.451+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:04.451 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:04.451 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:09.451 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:09.452+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:09.452 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:09.452 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:14.453 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:14.454+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:14.453 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:14.453 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:19.454 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:19.455+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:19.455 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:19.455 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:24.455 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:24.457+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:24.456 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:24.456 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:29.457 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:29.458+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:29.458 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:29.458 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:34.459+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:34.459 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:34.459 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:39.461 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:39.461+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:39.462 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:39.462 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:44.461 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:44.462+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:44.462 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:44.463 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:49.463 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:49.465+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:49.464 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:49.464 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:54.464 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:54.466+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:54.464 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:54.465 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:07:59.466 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:07:59.468+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:07:59.466 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:07:59.466 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:04.467 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:04.468+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:04.467 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:04.467 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:09.468 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:09.469+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:09.468 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:09.469 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:14.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:14.471+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:14.470 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:14.470 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:19.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:19.472+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:19.471 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:19.472 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:24.473 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:24.474+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:24.474 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:24.474 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:29.475 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:29.476+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:29.476 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:29.476 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:34.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:34.477+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:34.476 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:34.477 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:39.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:39.479+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:39.478 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:39.478 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:44.479 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:44.480+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:44.479 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:44.480 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:49.480 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:49.482+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:49.481 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:49.481 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:54.482 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:54.484+0000 7ff8f580d600 1 waiting for incremental sync to catch up: 2022-09-02T05:08:54.483 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , , , , ] 2022-09-02T05:08:54.483 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.29.5, , 00000000001.61.5, , 00000000001.48.5, , , 00000000001.48.5, , , ] 2022-09-02T05:08:59.483 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:59.485+0000 7ff8f580d600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T05:08:59.484 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:59.485+0000 7ff8f580d600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:08:59.484 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:08:59.485+0000 7ff8f580d600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:08:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T05:08:59.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2022-09-02T05:08:59.517 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-62 2022-09-02T05:08:59.519 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:08:59.520 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:08:59.521 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.521 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:08:59.522 DEBUG:boto:Data: 2022-09-02T05:08:59.522 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.523 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.524 DEBUG:boto:Params: {} 2022-09-02T05:08:59.525 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:08:59.526 DEBUG:boto:Token: None 2022-09-02T05:08:59.527 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/ 2022-09-02T05:08:59.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:82OTMUNQneKb1ZLNzPzG4yEh8Kg= 2022-09-02T05:08:59.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:82OTMUNQneKb1ZLNzPzG4yEh8Kg=', 'Content-Length': '0'} 2022-09-02T05:08:59.561 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c72961e6bf129ce7-0063118feb-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.562 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-63 2022-09-02T05:08:59.562 DEBUG:boto:path=/xljbkf-63/ 2022-09-02T05:08:59.563 DEBUG:boto:auth_path=/xljbkf-63/ 2022-09-02T05:08:59.563 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.564 DEBUG:boto:Path: /xljbkf-63/ 2022-09-02T05:08:59.564 DEBUG:boto:Data: 2022-09-02T05:08:59.565 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.566 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.567 DEBUG:boto:Params: {} 2022-09-02T05:08:59.567 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:08:59.568 DEBUG:boto:Token: None 2022-09-02T05:08:59.569 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/ 2022-09-02T05:08:59.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tlAwDEtfQXY4m0NPWCvrvOhAK2k= 2022-09-02T05:08:59.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tlAwDEtfQXY4m0NPWCvrvOhAK2k=', 'Content-Length': '0'} 2022-09-02T05:08:59.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000712823443fa7fb9d-0063118feb-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.620 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-64 2022-09-02T05:08:59.621 DEBUG:boto:path=/xljbkf-64/ 2022-09-02T05:08:59.621 DEBUG:boto:auth_path=/xljbkf-64/ 2022-09-02T05:08:59.622 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.622 DEBUG:boto:Path: /xljbkf-64/ 2022-09-02T05:08:59.623 DEBUG:boto:Data: 2022-09-02T05:08:59.623 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:08:59.624 DEBUG:boto:Port: 8001 2022-09-02T05:08:59.625 DEBUG:boto:Params: {} 2022-09-02T05:08:59.626 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:08:59.626 DEBUG:boto:Token: None 2022-09-02T05:08:59.627 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-64/ 2022-09-02T05:08:59.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0sARuqazM2RbWjVbF4wlNc777F4= 2022-09-02T05:08:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0sARuqazM2RbWjVbF4wlNc777F4=', 'Content-Length': '0'} 2022-09-02T05:08:59.665 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000490633dec8572b50-0063118feb-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.666 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:08:59.667 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:08:59.667 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.668 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:08:59.668 DEBUG:boto:Data: 2022-09-02T05:08:59.669 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.670 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.670 DEBUG:boto:Params: {} 2022-09-02T05:08:59.671 DEBUG:boto:Token: None 2022-09-02T05:08:59.671 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/ 2022-09-02T05:08:59.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA= 2022-09-02T05:08:59.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA=', 'Content-Length': '0'} 2022-09-02T05:08:59.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b9f6140ead6ad5c-0063118feb-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.677 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:08:59.678 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:08:59.679 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.680 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:08:59.681 DEBUG:boto:Data: 2022-09-02T05:08:59.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.683 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.684 DEBUG:boto:Params: {} 2022-09-02T05:08:59.685 DEBUG:boto:Token: None 2022-09-02T05:08:59.686 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/obj1 2022-09-02T05:08:59.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tgHO8LttXzLJ7RQ6ZececCmmSZk= 2022-09-02T05:08:59.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tgHO8LttXzLJ7RQ6ZececCmmSZk='} 2022-09-02T05:08:59.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067e77ce97dd74ad1-0063118feb-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.696 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:08:59.696 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:08:59.697 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.697 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:08:59.698 DEBUG:boto:Data: 2022-09-02T05:08:59.698 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.700 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.700 DEBUG:boto:Params: {} 2022-09-02T05:08:59.701 DEBUG:boto:Token: None 2022-09-02T05:08:59.702 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/ 2022-09-02T05:08:59.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA= 2022-09-02T05:08:59.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA=', 'Content-Length': '0'} 2022-09-02T05:08:59.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c468722db8a1a39a-0063118feb-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.706 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:08:59.707 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:08:59.708 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.708 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:08:59.709 DEBUG:boto:Data: 2022-09-02T05:08:59.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.712 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.713 DEBUG:boto:Params: {} 2022-09-02T05:08:59.714 DEBUG:boto:Token: None 2022-09-02T05:08:59.715 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/obj2 2022-09-02T05:08:59.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cbOwkheIyQMcczvgcpSuKJdxEOk= 2022-09-02T05:08:59.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cbOwkheIyQMcczvgcpSuKJdxEOk='} 2022-09-02T05:08:59.723 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000345f492865fb9839-0063118feb-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.724 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:08:59.724 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:08:59.725 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.725 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:08:59.726 DEBUG:boto:Data: 2022-09-02T05:08:59.727 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.728 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.728 DEBUG:boto:Params: {} 2022-09-02T05:08:59.729 DEBUG:boto:Token: None 2022-09-02T05:08:59.729 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/ 2022-09-02T05:08:59.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA= 2022-09-02T05:08:59.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA=', 'Content-Length': '0'} 2022-09-02T05:08:59.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ef1a1c5f7a26cce-0063118feb-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.734 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:08:59.734 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:08:59.735 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.735 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:08:59.736 DEBUG:boto:Data: 2022-09-02T05:08:59.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.737 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.738 DEBUG:boto:Params: {} 2022-09-02T05:08:59.738 DEBUG:boto:Token: None 2022-09-02T05:08:59.739 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/obj3 2022-09-02T05:08:59.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oTO83s+DoofBgklatHJMyk5D85c= 2022-09-02T05:08:59.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oTO83s+DoofBgklatHJMyk5D85c='} 2022-09-02T05:08:59.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098e746660be2f3dd-0063118feb-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.788 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:08:59.789 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:08:59.790 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.791 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:08:59.792 DEBUG:boto:Data: 2022-09-02T05:08:59.792 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.794 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.794 DEBUG:boto:Params: {} 2022-09-02T05:08:59.795 DEBUG:boto:Token: None 2022-09-02T05:08:59.796 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/ 2022-09-02T05:08:59.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA= 2022-09-02T05:08:59.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:irSMyfjlM9jFUAqjtY92B7D0fNA=', 'Content-Length': '0'} 2022-09-02T05:08:59.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d30e7ee3ed22c2e2-0063118feb-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.800 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:08:59.801 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:08:59.802 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.802 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:08:59.803 DEBUG:boto:Data: 2022-09-02T05:08:59.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:08:59.804 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.805 DEBUG:boto:Params: {} 2022-09-02T05:08:59.806 DEBUG:boto:Token: None 2022-09-02T05:08:59.806 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-62/obj4 2022-09-02T05:08:59.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UR1FNteRedwGy0FqQWP7swEZhx8= 2022-09-02T05:08:59.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UR1FNteRedwGy0FqQWP7swEZhx8='} 2022-09-02T05:08:59.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df88b1dd908b79f3-0063118feb-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.819 DEBUG:boto:path=/xljbkf-63/ 2022-09-02T05:08:59.819 DEBUG:boto:auth_path=/xljbkf-63/ 2022-09-02T05:08:59.820 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.820 DEBUG:boto:Path: /xljbkf-63/ 2022-09-02T05:08:59.821 DEBUG:boto:Data: 2022-09-02T05:08:59.821 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.822 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.823 DEBUG:boto:Params: {} 2022-09-02T05:08:59.824 DEBUG:boto:Token: None 2022-09-02T05:08:59.824 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/ 2022-09-02T05:08:59.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I= 2022-09-02T05:08:59.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I=', 'Content-Length': '0'} 2022-09-02T05:08:59.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cb4705aa1a97796-0063118feb-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.828 DEBUG:boto:path=/xljbkf-63/obj1 2022-09-02T05:08:59.829 DEBUG:boto:auth_path=/xljbkf-63/obj1 2022-09-02T05:08:59.829 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.830 DEBUG:boto:Path: /xljbkf-63/obj1 2022-09-02T05:08:59.831 DEBUG:boto:Data: 2022-09-02T05:08:59.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.832 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.833 DEBUG:boto:Params: {} 2022-09-02T05:08:59.833 DEBUG:boto:Token: None 2022-09-02T05:08:59.834 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/obj1 2022-09-02T05:08:59.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cGx+DcgM+RnDhiDvbyLystUPsXo= 2022-09-02T05:08:59.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cGx+DcgM+RnDhiDvbyLystUPsXo='} 2022-09-02T05:08:59.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f712c369056f06eb-0063118feb-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.851 DEBUG:boto:path=/xljbkf-63/ 2022-09-02T05:08:59.852 DEBUG:boto:auth_path=/xljbkf-63/ 2022-09-02T05:08:59.853 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.853 DEBUG:boto:Path: /xljbkf-63/ 2022-09-02T05:08:59.853 DEBUG:boto:Data: 2022-09-02T05:08:59.854 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.855 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.855 DEBUG:boto:Params: {} 2022-09-02T05:08:59.856 DEBUG:boto:Token: None 2022-09-02T05:08:59.857 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/ 2022-09-02T05:08:59.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I= 2022-09-02T05:08:59.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I=', 'Content-Length': '0'} 2022-09-02T05:08:59.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000848cd2e360a1721e-0063118feb-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.863 DEBUG:boto:path=/xljbkf-63/obj2 2022-09-02T05:08:59.864 DEBUG:boto:auth_path=/xljbkf-63/obj2 2022-09-02T05:08:59.865 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.865 DEBUG:boto:Path: /xljbkf-63/obj2 2022-09-02T05:08:59.866 DEBUG:boto:Data: 2022-09-02T05:08:59.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.868 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.868 DEBUG:boto:Params: {} 2022-09-02T05:08:59.869 DEBUG:boto:Token: None 2022-09-02T05:08:59.869 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/obj2 2022-09-02T05:08:59.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V+My1jkYWH6BFNkykZ3Xv6XE0c0= 2022-09-02T05:08:59.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V+My1jkYWH6BFNkykZ3Xv6XE0c0='} 2022-09-02T05:08:59.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000648055a938efed2e-0063118feb-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.881 DEBUG:boto:path=/xljbkf-63/ 2022-09-02T05:08:59.882 DEBUG:boto:auth_path=/xljbkf-63/ 2022-09-02T05:08:59.882 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.883 DEBUG:boto:Path: /xljbkf-63/ 2022-09-02T05:08:59.883 DEBUG:boto:Data: 2022-09-02T05:08:59.884 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.885 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.886 DEBUG:boto:Params: {} 2022-09-02T05:08:59.887 DEBUG:boto:Token: None 2022-09-02T05:08:59.888 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/ 2022-09-02T05:08:59.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I= 2022-09-02T05:08:59.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I=', 'Content-Length': '0'} 2022-09-02T05:08:59.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075e08ddefb83e09d-0063118feb-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.893 DEBUG:boto:path=/xljbkf-63/obj3 2022-09-02T05:08:59.893 DEBUG:boto:auth_path=/xljbkf-63/obj3 2022-09-02T05:08:59.894 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.894 DEBUG:boto:Path: /xljbkf-63/obj3 2022-09-02T05:08:59.895 DEBUG:boto:Data: 2022-09-02T05:08:59.895 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.897 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.897 DEBUG:boto:Params: {} 2022-09-02T05:08:59.898 DEBUG:boto:Token: None 2022-09-02T05:08:59.899 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/obj3 2022-09-02T05:08:59.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kOoIkWp925Z30sxrAy9rddQ9sPk= 2022-09-02T05:08:59.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kOoIkWp925Z30sxrAy9rddQ9sPk='} 2022-09-02T05:08:59.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f336ca78d7763d96-0063118feb-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.947 DEBUG:boto:path=/xljbkf-63/ 2022-09-02T05:08:59.948 DEBUG:boto:auth_path=/xljbkf-63/ 2022-09-02T05:08:59.948 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.949 DEBUG:boto:Path: /xljbkf-63/ 2022-09-02T05:08:59.950 DEBUG:boto:Data: 2022-09-02T05:08:59.951 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.952 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.952 DEBUG:boto:Params: {} 2022-09-02T05:08:59.953 DEBUG:boto:Token: None 2022-09-02T05:08:59.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/ 2022-09-02T05:08:59.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I= 2022-09-02T05:08:59.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M4reIF9gVQVvxlJHcAq7YOkwy7I=', 'Content-Length': '0'} 2022-09-02T05:08:59.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015975ef0b3571c76-0063118feb-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.958 DEBUG:boto:path=/xljbkf-63/obj4 2022-09-02T05:08:59.958 DEBUG:boto:auth_path=/xljbkf-63/obj4 2022-09-02T05:08:59.959 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.960 DEBUG:boto:Path: /xljbkf-63/obj4 2022-09-02T05:08:59.960 DEBUG:boto:Data: 2022-09-02T05:08:59.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:08:59.971 DEBUG:boto:Port: 8000 2022-09-02T05:08:59.971 DEBUG:boto:Params: {} 2022-09-02T05:08:59.972 DEBUG:boto:Token: None 2022-09-02T05:08:59.973 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-63/obj4 2022-09-02T05:08:59.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/s+5QyyiV/0xXG64eYtkuJSKMg= 2022-09-02T05:08:59.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/s+5QyyiV/0xXG64eYtkuJSKMg='} 2022-09-02T05:08:59.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbe53f58bfb1fab6-0063118feb-15e3-test-zone2'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.982 DEBUG:boto:path=/xljbkf-64/ 2022-09-02T05:08:59.983 DEBUG:boto:auth_path=/xljbkf-64/ 2022-09-02T05:08:59.984 DEBUG:boto:Method: HEAD 2022-09-02T05:08:59.984 DEBUG:boto:Path: /xljbkf-64/ 2022-09-02T05:08:59.985 DEBUG:boto:Data: 2022-09-02T05:08:59.986 DEBUG:boto:Headers: {} 2022-09-02T05:08:59.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:08:59.987 DEBUG:boto:Port: 8001 2022-09-02T05:08:59.988 DEBUG:boto:Params: {} 2022-09-02T05:08:59.989 DEBUG:boto:Token: None 2022-09-02T05:08:59.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:08:59 GMT /xljbkf-64/ 2022-09-02T05:08:59.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U8R7bTlk1ZW0K1Js+OdGH4uviNY= 2022-09-02T05:08:59.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:08:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U8R7bTlk1ZW0K1Js+OdGH4uviNY=', 'Content-Length': '0'} 2022-09-02T05:08:59.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fcc252d280a7083-0063118feb-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:08:59.994 DEBUG:boto:path=/xljbkf-64/obj1 2022-09-02T05:08:59.995 DEBUG:boto:auth_path=/xljbkf-64/obj1 2022-09-02T05:08:59.995 DEBUG:boto:Method: PUT 2022-09-02T05:08:59.996 DEBUG:boto:Path: /xljbkf-64/obj1 2022-09-02T05:08:59.997 DEBUG:boto:Data: 2022-09-02T05:08:59.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:08:59.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:08:59.998 DEBUG:boto:Port: 8001 2022-09-02T05:08:59.999 DEBUG:boto:Params: {} 2022-09-02T05:08:59.999 DEBUG:boto:Token: None 2022-09-02T05:09:00.000 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/obj1 2022-09-02T05:09:00.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lh6XksuNBMM0lObIBiRfe1un5ro= 2022-09-02T05:09:00.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lh6XksuNBMM0lObIBiRfe1un5ro='} 2022-09-02T05:09:00.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019ebf2b7bb68ad6d-0063118fec-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.011 DEBUG:boto:path=/xljbkf-64/ 2022-09-02T05:09:00.011 DEBUG:boto:auth_path=/xljbkf-64/ 2022-09-02T05:09:00.012 DEBUG:boto:Method: HEAD 2022-09-02T05:09:00.012 DEBUG:boto:Path: /xljbkf-64/ 2022-09-02T05:09:00.013 DEBUG:boto:Data: 2022-09-02T05:09:00.013 DEBUG:boto:Headers: {} 2022-09-02T05:09:00.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:09:00.015 DEBUG:boto:Port: 8001 2022-09-02T05:09:00.015 DEBUG:boto:Params: {} 2022-09-02T05:09:00.016 DEBUG:boto:Token: None 2022-09-02T05:09:00.016 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/ 2022-09-02T05:09:00.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TE53oNKZFvR6/5/V0ahtOrVqb6M= 2022-09-02T05:09:00.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TE53oNKZFvR6/5/V0ahtOrVqb6M=', 'Content-Length': '0'} 2022-09-02T05:09:00.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000683f65a6935948cd-0063118fec-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.021 DEBUG:boto:path=/xljbkf-64/obj2 2022-09-02T05:09:00.021 DEBUG:boto:auth_path=/xljbkf-64/obj2 2022-09-02T05:09:00.022 DEBUG:boto:Method: PUT 2022-09-02T05:09:00.022 DEBUG:boto:Path: /xljbkf-64/obj2 2022-09-02T05:09:00.023 DEBUG:boto:Data: 2022-09-02T05:09:00.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:09:00.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:09:00.024 DEBUG:boto:Port: 8001 2022-09-02T05:09:00.025 DEBUG:boto:Params: {} 2022-09-02T05:09:00.025 DEBUG:boto:Token: None 2022-09-02T05:09:00.026 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/obj2 2022-09-02T05:09:00.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sMvD4/V88+xqRTsAEREovSELxvc= 2022-09-02T05:09:00.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sMvD4/V88+xqRTsAEREovSELxvc='} 2022-09-02T05:09:00.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4ff65be62e75ff2-0063118fec-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.039 DEBUG:boto:path=/xljbkf-64/ 2022-09-02T05:09:00.040 DEBUG:boto:auth_path=/xljbkf-64/ 2022-09-02T05:09:00.041 DEBUG:boto:Method: HEAD 2022-09-02T05:09:00.042 DEBUG:boto:Path: /xljbkf-64/ 2022-09-02T05:09:00.043 DEBUG:boto:Data: 2022-09-02T05:09:00.044 DEBUG:boto:Headers: {} 2022-09-02T05:09:00.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:09:00.045 DEBUG:boto:Port: 8001 2022-09-02T05:09:00.046 DEBUG:boto:Params: {} 2022-09-02T05:09:00.047 DEBUG:boto:Token: None 2022-09-02T05:09:00.047 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/ 2022-09-02T05:09:00.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TE53oNKZFvR6/5/V0ahtOrVqb6M= 2022-09-02T05:09:00.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TE53oNKZFvR6/5/V0ahtOrVqb6M=', 'Content-Length': '0'} 2022-09-02T05:09:00.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a15b05983851f8d-0063118fec-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.052 DEBUG:boto:path=/xljbkf-64/obj3 2022-09-02T05:09:00.053 DEBUG:boto:auth_path=/xljbkf-64/obj3 2022-09-02T05:09:00.053 DEBUG:boto:Method: PUT 2022-09-02T05:09:00.054 DEBUG:boto:Path: /xljbkf-64/obj3 2022-09-02T05:09:00.054 DEBUG:boto:Data: 2022-09-02T05:09:00.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:09:00.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:09:00.056 DEBUG:boto:Port: 8001 2022-09-02T05:09:00.057 DEBUG:boto:Params: {} 2022-09-02T05:09:00.058 DEBUG:boto:Token: None 2022-09-02T05:09:00.058 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/obj3 2022-09-02T05:09:00.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uy/WjjFrlnpAtyOh38TyqpehV3g= 2022-09-02T05:09:00.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uy/WjjFrlnpAtyOh38TyqpehV3g='} 2022-09-02T05:09:00.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6b3918f1367ff1c-0063118fec-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.107 DEBUG:boto:path=/xljbkf-64/ 2022-09-02T05:09:00.108 DEBUG:boto:auth_path=/xljbkf-64/ 2022-09-02T05:09:00.109 DEBUG:boto:Method: HEAD 2022-09-02T05:09:00.109 DEBUG:boto:Path: /xljbkf-64/ 2022-09-02T05:09:00.110 DEBUG:boto:Data: 2022-09-02T05:09:00.111 DEBUG:boto:Headers: {} 2022-09-02T05:09:00.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:09:00.112 DEBUG:boto:Port: 8001 2022-09-02T05:09:00.112 DEBUG:boto:Params: {} 2022-09-02T05:09:00.113 DEBUG:boto:Token: None 2022-09-02T05:09:00.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/ 2022-09-02T05:09:00.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TE53oNKZFvR6/5/V0ahtOrVqb6M= 2022-09-02T05:09:00.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TE53oNKZFvR6/5/V0ahtOrVqb6M=', 'Content-Length': '0'} 2022-09-02T05:09:00.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085947e2d197fd491-0063118fec-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.119 DEBUG:boto:path=/xljbkf-64/obj4 2022-09-02T05:09:00.119 DEBUG:boto:auth_path=/xljbkf-64/obj4 2022-09-02T05:09:00.122 DEBUG:boto:Method: PUT 2022-09-02T05:09:00.122 DEBUG:boto:Path: /xljbkf-64/obj4 2022-09-02T05:09:00.123 DEBUG:boto:Data: 2022-09-02T05:09:00.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2022-09-02T05:09:00.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:09:00.124 DEBUG:boto:Port: 8001 2022-09-02T05:09:00.125 DEBUG:boto:Params: {} 2022-09-02T05:09:00.126 DEBUG:boto:Token: None 2022-09-02T05:09:00.126 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 02 Sep 2022 05:09:00 GMT /xljbkf-64/obj4 2022-09-02T05:09:00.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:681OY1u8j9AMqrmweioJk35yFdU= 2022-09-02T05:09:00.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:09:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:681OY1u8j9AMqrmweioJk35yFdU='} 2022-09-02T05:09:00.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000babc23244060a2c8-0063118fec-1599-test-zone3'), ('Date', 'Fri, 02 Sep 2022 05:09:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:09:00.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:00.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:00.138 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:09:00.188 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:00.188 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:09:00.242 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:09:00.245 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:09:00.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:00.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:00.245 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095339.653578_332.1", 2022-09-02T05:09:00.246 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:08:59.653578Z" 2022-09-02T05:09:00.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:09:00.247 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:09:00.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:09:00.248 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:09:00.249 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:00.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:09:00.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:00.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:00.252 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:09:00.303 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:00.304 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:00.336 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:00.339+0000 7f40fdde8600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:00.337 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:00.340+0000 7f40fdde8600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:00.355 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:09:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:09:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:09:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:09:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:09:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:09:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:09:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:00.358 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:09:00.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:00.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:09:00.359 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:00.359 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:00.359 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:09:00.360 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:00.360 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:09:00.360 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:00.361 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:09:00.361 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:00.361 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:00.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:00.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:00.362 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:09:00.363 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:00.363 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:00.363 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:09:00.364 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:00.364 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:00.365 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:00.365 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:09:00.366 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:00.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:00.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:00.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:00.367 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:09:00.367 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:00.367 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:00.367 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095339.646432_329.1", 2022-09-02T05:09:00.368 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:00.368 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:00.368 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:00.368 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:08:59.646432Z", 2022-09-02T05:09:00.369 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:09:00.369 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:00.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:00.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:00.370 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:09:00.370 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:00.370 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:00.371 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:09:00.371 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:00.371 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:00.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:00.372 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:09:00.372 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:00.372 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:00.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:00.373 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:09:00.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:00.374 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:09:00.374 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:09:00.375 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:09:00.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:00.375 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:09:00Z" 2022-09-02T05:09:00.376 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:09:00.378 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095339.646432_329.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:08:59.646432Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:09:00Z" } 2022-09-02T05:09:00.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095339.646432_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:08:59.646432Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:09:00.380 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:00.381 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095339.646432_329.1', 3: ''} 2022-09-02T05:09:00.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662095339.653578_332.1 target=1_1662095339.646432_329.1 2022-09-02T05:09:05.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:05.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:05.385 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:09:05.443 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:05.443 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:05.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:05.479+0000 7f02b9c6a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:05.478 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:05.480+0000 7f02b9c6a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:05.492 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:09:05.493 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:09:05.493 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:09:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:09:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:09:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:09:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:09:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:09:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:09:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:09:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.497 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:09:05.497 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.497 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:09:05.498 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:05.498 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.500 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:09:05.500 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:09:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:09:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:09:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095339.653578_332.1", 2022-09-02T05:09:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:08:59.653578Z", 2022-09-02T05:09:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:09:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:09:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:09:05.508 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.508 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:05.508 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.508 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:09:05.509 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:05.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.509 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:09:05.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.510 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:09:05.510 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:09:05.511 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:09:05.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.512 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:09:05Z" 2022-09-02T05:09:05.512 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:09:05.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095339.653578_332.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:08:59.653578Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:09:05Z" } 2022-09-02T05:09:05.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095339.653578_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:08:59.653578Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:09:05.516 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:05.517 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:05.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:09:05.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:09:05.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:05.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:05.522 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:09:05.580 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:05.580 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:05.613 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:09:05.614+0000 7f3bac567600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:05.614 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:09:05.616+0000 7f3bac567600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:05.627 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:09:05.628 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:09:05.628 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:09:05.629 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:09:05.629 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:09:05.629 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:09:05.629 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:09:05.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:09:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:09:05.631 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:09:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:09:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:09:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:09:05.632 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:09:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:09:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:09:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:09:05.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:09:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:09:05.634 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:09:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:09:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:09:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:09:05.635 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:09:05.636 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:09:05.636 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:09:05.636 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:09:05.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:09:05.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:05.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:05.637 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:09:05.638 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:09:05.638 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:09:05.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095339.653578_332.1", 2022-09-02T05:09:05.638 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:09:05.639 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:09:05.639 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:09:05.639 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:08:59.653578Z", 2022-09-02T05:09:05.639 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:09:05.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:09:05.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:05.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:09:05.640 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:09:05.641 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:09:05.641 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:09:05.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:09:05.641 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:09:05.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:09:05.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:09:05.643 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:09:05.643 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:09:05.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:09:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:09:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:09:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:09:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:09:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:09:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:09:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:09:05Z" 2022-09-02T05:09:05.646 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:09:05.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095339.653578_332.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:08:59.653578Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:09:05Z" } 2022-09-02T05:09:05.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095339.653578_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:08:59.653578Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:09:05.649 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:05.649 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:05.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:09:05.650 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:09:05.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:05.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:09:05.652 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:09:05.704 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:05.704 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:05.751 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:05.753+0000 7fee3e09a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:05.752 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:05.754+0000 7fee3e09a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:09:05.772 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:09:05.773 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:09:05.773 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:09:05.773 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:09:05.774 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:09:05.774 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:09:05.774 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:09:05.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.775 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:09:05.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.775 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:09:05.776 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.776 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.776 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:09:05.776 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.777 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:09:05.777 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.777 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:09:05.777 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:05.778 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.779 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:09:05.779 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.779 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.779 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:09:05.780 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.780 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:05.780 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.780 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:09:05.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:05.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.782 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:09:05.782 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.782 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.782 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095339.653578_332.1", 2022-09-02T05:09:05.783 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:05.783 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.783 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:08:59.653578Z", 2022-09-02T05:09:05.784 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:09:05.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:09:05.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:09:05.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:09:05.785 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:09:05.785 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:09:05.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:09:05.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:09:05.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:09:05.786 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:09:05.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:09:05.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:09:05.787 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:09:05.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.788 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:09:05.788 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:09:05.788 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:09:05.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:09:05.789 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:09:05Z" 2022-09-02T05:09:05.789 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:09:05.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095339.653578_332.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:08:59.653578Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:09:05Z" } 2022-09-02T05:09:05.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095339.653578_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:08:59.653578Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:09:05.792 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:05.792 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095339.653578_332.1', 3: ''} 2022-09-02T05:09:05.793 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:09:05.793 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-62 2022-09-02T05:09:05.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:09:05.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:09:05.795 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:09:05.853 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:05.853 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:05.912 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:05.914+0000 7ff9422c4600 1 bucket sync caught up with source: 2022-09-02T05:09:05.913 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:05.913 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:05.913 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:05.914+0000 7ff9422c4600 0 bucket checkpoint complete 2022-09-02T05:09:05.931 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-62 2022-09-02T05:09:05.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:09:05.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:09:05.933 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:09:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:06.042 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:09:06.044+0000 7fd770301600 1 bucket sync caught up with source: 2022-09-02T05:09:06.043 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:06.043 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:09:06.044+0000 7fd770301600 0 bucket checkpoint complete 2022-09-02T05:09:06.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-62 2022-09-02T05:09:06.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:09:06.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:09:06.069 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:09:06.126 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:09:06.127 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:09:06.176 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:06.177+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:06.176 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:06.176 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:11.177 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:11.178+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:11.178 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:11.178 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:16.178 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:16.179+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:16.179 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:16.179 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:21.179 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:21.181+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:21.180 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:21.180 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:26.181 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:26.182+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:26.182 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:26.182 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:31.182 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:31.183+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:31.183 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:31.183 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:36.183 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:36.184+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:36.184 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:36.184 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:41.185 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:41.186+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:41.185 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:41.185 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:46.186 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:46.187+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:46.187 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:46.187 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:51.188 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:51.189+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:51.188 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:51.188 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:56.189 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:09:56.190+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:09:56.190 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:09:56.190 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:10:01.191 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:01.193+0000 7f9115e8e600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:01.191 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , , , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:10:01.192 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:10:06.192 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:06.194+0000 7f9115e8e600 1 bucket sync caught up with source: 2022-09-02T05:10:06.193 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:10:06.193 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.45.5, , 00000000001.56.5, , 00000000001.58.5, , , 00000000001.40.5, , , ] 2022-09-02T05:10:06.193 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:06.194+0000 7f9115e8e600 0 bucket checkpoint complete 2022-09-02T05:10:06.209 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-62 2022-09-02T05:10:06.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.212 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:06.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:06.277 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:06.319 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:06.321+0000 7f54879df600 1 bucket sync caught up with source: 2022-09-02T05:10:06.320 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.435.5, , 00000000001.570.5, , 00000000001.1071.5, , , 00000000001.691.5, , , ] 2022-09-02T05:10:06.320 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.435.5, , 00000000001.570.5, , 00000000001.1071.5, , , 00000000001.691.5, , , ] 2022-09-02T05:10:06.320 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:06.321+0000 7f54879df600 0 bucket checkpoint complete 2022-09-02T05:10:06.344 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-62 2022-09-02T05:10:06.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.346 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:06.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:06.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:06.468 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:06.469+0000 7f19995bd600 1 bucket sync caught up with source: 2022-09-02T05:10:06.468 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.435.5, , 00000000001.570.5, , 00000000001.1071.5, , , 00000000001.691.5, , , ] 2022-09-02T05:10:06.469 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.435.5, , 00000000001.570.5, , 00000000001.1071.5, , , 00000000001.691.5, , , ] 2022-09-02T05:10:06.469 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:06.470+0000 7f19995bd600 0 bucket checkpoint complete 2022-09-02T05:10:06.488 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-62 2022-09-02T05:10:06.488 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-62 2022-09-02T05:10:06.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.496 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:06.551 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:06.552 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:06.595 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:06.596+0000 7f364caa1600 1 bucket sync caught up with source: 2022-09-02T05:10:06.595 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.240.5, , 00000000001.189.5, , 00000000001.362.5, , , 00000000001.82.5, , , ] 2022-09-02T05:10:06.595 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.240.5, , 00000000001.189.5, , 00000000001.362.5, , , 00000000001.82.5, , , ] 2022-09-02T05:10:06.596 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:06.596+0000 7f364caa1600 0 bucket checkpoint complete 2022-09-02T05:10:06.608 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-62 2022-09-02T05:10:06.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:06.610 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:06.675 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:06.676 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:06.717 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:06.720+0000 7f472df3d600 1 bucket sync caught up with source: 2022-09-02T05:10:06.718 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.240.5, , 00000000001.189.5, , 00000000001.362.5, , , 00000000001.82.5, , , ] 2022-09-02T05:10:06.718 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.240.5, , 00000000001.189.5, , 00000000001.362.5, , , 00000000001.82.5, , , ] 2022-09-02T05:10:06.718 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:06.720+0000 7f472df3d600 0 bucket checkpoint complete 2022-09-02T05:10:06.739 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-62 2022-09-02T05:10:06.740 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-62 zones={test-zone2, test-zone1} 2022-09-02T05:10:06.741 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.741 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.742 DEBUG:boto:Method: HEAD 2022-09-02T05:10:06.743 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:10:06.743 DEBUG:boto:Data: 2022-09-02T05:10:06.744 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.745 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.745 DEBUG:boto:Params: {} 2022-09-02T05:10:06.746 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:10:06.747 DEBUG:boto:Token: None 2022-09-02T05:10:06.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/ 2022-09-02T05:10:06.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qd++NbCMdzPDN3BirqMJSqTShag= 2022-09-02T05:10:06.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qd++NbCMdzPDN3BirqMJSqTShag=', 'Content-Length': '0'} 2022-09-02T05:10:06.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b05ce58484759a3-006311902e-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.754 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.754 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.755 DEBUG:boto:Method: HEAD 2022-09-02T05:10:06.755 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:10:06.756 DEBUG:boto:Data: 2022-09-02T05:10:06.756 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.757 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.758 DEBUG:boto:Params: {} 2022-09-02T05:10:06.759 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:10:06.759 DEBUG:boto:Token: None 2022-09-02T05:10:06.760 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/ 2022-09-02T05:10:06.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qd++NbCMdzPDN3BirqMJSqTShag= 2022-09-02T05:10:06.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qd++NbCMdzPDN3BirqMJSqTShag=', 'Content-Length': '0'} 2022-09-02T05:10:06.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0ddb01751345406-006311902e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.765 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:10:06.766 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.766 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.767 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.767 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.768 DEBUG:boto:Method: GET 2022-09-02T05:10:06.768 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.769 DEBUG:boto:Data: 2022-09-02T05:10:06.769 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.770 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.771 DEBUG:boto:Params: {} 2022-09-02T05:10:06.772 DEBUG:boto:Token: None 2022-09-02T05:10:06.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/?versions 2022-09-02T05:10:06.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI= 2022-09-02T05:10:06.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI=', 'Content-Length': '0'} 2022-09-02T05:10:06.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c434899099980f03-006311902e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.777 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:06.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:10:06.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:10:06.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:10:06.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:10:06.780 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:10:06.781 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.781 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.782 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.782 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.783 DEBUG:boto:Method: GET 2022-09-02T05:10:06.783 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.784 DEBUG:boto:Data: 2022-09-02T05:10:06.784 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.785 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.786 DEBUG:boto:Params: {} 2022-09-02T05:10:06.787 DEBUG:boto:Token: None 2022-09-02T05:10:06.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/?versions 2022-09-02T05:10:06.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI= 2022-09-02T05:10:06.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI=', 'Content-Length': '0'} 2022-09-02T05:10:06.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003697dfdeff768ec1-006311902e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.792 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:06.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:10:06.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:10:06.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:10:06.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:10:06.795 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.796 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.796 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.797 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.797 DEBUG:boto:Method: GET 2022-09-02T05:10:06.798 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.798 DEBUG:boto:Data: 2022-09-02T05:10:06.799 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.800 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.800 DEBUG:boto:Params: {} 2022-09-02T05:10:06.801 DEBUG:boto:Token: None 2022-09-02T05:10:06.801 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/?versions 2022-09-02T05:10:06.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI= 2022-09-02T05:10:06.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI=', 'Content-Length': '0'} 2022-09-02T05:10:06.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f20a730153a786-006311902e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.806 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:06.807 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.807 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.808 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.808 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.809 DEBUG:boto:Method: GET 2022-09-02T05:10:06.809 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:06.810 DEBUG:boto:Data: 2022-09-02T05:10:06.811 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.812 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.812 DEBUG:boto:Params: {} 2022-09-02T05:10:06.813 DEBUG:boto:Token: None 2022-09-02T05:10:06.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/?versions 2022-09-02T05:10:06.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI= 2022-09-02T05:10:06.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SxCuHic98riJavKpWE4ccKpI7LI=', 'Content-Length': '0'} 2022-09-02T05:10:06.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db8d17efaa0c2d11-006311902e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.818 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:06.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:06.820 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.820 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.821 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.821 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.822 DEBUG:boto:Method: GET 2022-09-02T05:10:06.823 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.823 DEBUG:boto:Data: 2022-09-02T05:10:06.824 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.824 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.825 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.825 DEBUG:boto:Params: {} 2022-09-02T05:10:06.826 DEBUG:boto:Token: None 2022-09-02T05:10:06.827 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY= 2022-09-02T05:10:06.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY=', 'Content-Length': '0'} 2022-09-02T05:10:06.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016352d642199a355-006311902e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.831 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.832 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.832 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.833 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.833 DEBUG:boto:Method: GET 2022-09-02T05:10:06.834 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.834 DEBUG:boto:Data: 2022-09-02T05:10:06.835 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.836 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.836 DEBUG:boto:Params: {} 2022-09-02T05:10:06.837 DEBUG:boto:Token: None 2022-09-02T05:10:06.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY= 2022-09-02T05:10:06.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY=', 'Content-Length': '0'} 2022-09-02T05:10:06.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd6d9e05198ff1b5-006311902e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.843 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.844 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.845 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.845 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.846 DEBUG:boto:Method: HEAD 2022-09-02T05:10:06.847 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.848 DEBUG:boto:Data: 2022-09-02T05:10:06.848 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.849 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.850 DEBUG:boto:Params: {} 2022-09-02T05:10:06.851 DEBUG:boto:Token: None 2022-09-02T05:10:06.851 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GIO+QMLt2W/RXvBJHKPOAGkfcjU= 2022-09-02T05:10:06.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GIO+QMLt2W/RXvBJHKPOAGkfcjU=', 'Content-Length': '0'} 2022-09-02T05:10:06.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000661c74bd13f8269d-006311902e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.855 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.856 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.856 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.857 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.858 DEBUG:boto:Method: HEAD 2022-09-02T05:10:06.858 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.859 DEBUG:boto:Data: 2022-09-02T05:10:06.859 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.860 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.861 DEBUG:boto:Params: {} 2022-09-02T05:10:06.862 DEBUG:boto:Token: None 2022-09-02T05:10:06.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GIO+QMLt2W/RXvBJHKPOAGkfcjU= 2022-09-02T05:10:06.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GIO+QMLt2W/RXvBJHKPOAGkfcjU=', 'Content-Length': '0'} 2022-09-02T05:10:06.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5e5921ace2378b2-006311902e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:06.867 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.868 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.868 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.869 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.870 DEBUG:boto:Method: GET 2022-09-02T05:10:06.870 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.871 DEBUG:boto:Data: 2022-09-02T05:10:06.871 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.872 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.873 DEBUG:boto:Params: {} 2022-09-02T05:10:06.874 DEBUG:boto:Token: None 2022-09-02T05:10:06.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY= 2022-09-02T05:10:06.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY=', 'Content-Length': '0'} 2022-09-02T05:10:06.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e4e4ca653dbbc5d-006311902e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.879 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.879 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.880 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.881 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.881 DEBUG:boto:Method: GET 2022-09-02T05:10:06.882 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.883 DEBUG:boto:Data: 2022-09-02T05:10:06.883 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.884 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.885 DEBUG:boto:Params: {} 2022-09-02T05:10:06.886 DEBUG:boto:Token: None 2022-09-02T05:10:06.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:06.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY= 2022-09-02T05:10:06.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M8Zoogxc/4nT7Ezul8j/aB90nzY=', 'Content-Length': '0'} 2022-09-02T05:10:06.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032b1e98831eed09b-006311902e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.890 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.891 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.892 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:06.892 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:06.893 DEBUG:boto:Method: GET 2022-09-02T05:10:06.893 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:06.894 DEBUG:boto:Data: 2022-09-02T05:10:06.895 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.896 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.896 DEBUG:boto:Params: {} 2022-09-02T05:10:06.897 DEBUG:boto:Token: None 2022-09-02T05:10:06.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/ 2022-09-02T05:10:06.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g+Hg0H94czb4/BbyfrqPxDdUx7I= 2022-09-02T05:10:06.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g+Hg0H94czb4/BbyfrqPxDdUx7I=', 'Content-Length': '0'} 2022-09-02T05:10:06.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba58f2e56204732c-006311902e-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.902 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:06.903 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:06.904 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:06.905 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:06.905 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:06.906 DEBUG:boto:Method: GET 2022-09-02T05:10:06.906 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:06.907 DEBUG:boto:Data: 2022-09-02T05:10:06.907 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.908 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.909 DEBUG:boto:Params: {} 2022-09-02T05:10:06.910 DEBUG:boto:Token: None 2022-09-02T05:10:06.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/ 2022-09-02T05:10:06.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g+Hg0H94czb4/BbyfrqPxDdUx7I= 2022-09-02T05:10:06.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g+Hg0H94czb4/BbyfrqPxDdUx7I=', 'Content-Length': '0'} 2022-09-02T05:10:06.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d580b0939191f09f-006311902e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.915 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:06.916 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.917 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.917 DEBUG:boto:Method: HEAD 2022-09-02T05:10:06.918 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:06.918 DEBUG:boto:Data: 2022-09-02T05:10:06.919 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.920 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.920 DEBUG:boto:Params: {} 2022-09-02T05:10:06.921 DEBUG:boto:Token: None 2022-09-02T05:10:06.921 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1 2022-09-02T05:10:06.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AstgnzBBa0vD1ET6SBQUdk/b+sg= 2022-09-02T05:10:06.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AstgnzBBa0vD1ET6SBQUdk/b+sg=', 'Content-Length': '0'} 2022-09-02T05:10:06.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079183f4ce96877b6-006311902e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.925 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.926 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.927 DEBUG:boto:Method: HEAD 2022-09-02T05:10:06.927 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:06.928 DEBUG:boto:Data: 2022-09-02T05:10:06.928 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.929 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.930 DEBUG:boto:Params: {} 2022-09-02T05:10:06.930 DEBUG:boto:Token: None 2022-09-02T05:10:06.931 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1 2022-09-02T05:10:06.931 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AstgnzBBa0vD1ET6SBQUdk/b+sg= 2022-09-02T05:10:06.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AstgnzBBa0vD1ET6SBQUdk/b+sg=', 'Content-Length': '0'} 2022-09-02T05:10:06.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1e69abc0dda45b9-006311902e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:06.935 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.936 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.936 DEBUG:boto:Method: GET 2022-09-02T05:10:06.937 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:06.938 DEBUG:boto:Data: 2022-09-02T05:10:06.938 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:06.939 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.940 DEBUG:boto:Params: {} 2022-09-02T05:10:06.940 DEBUG:boto:Token: None 2022-09-02T05:10:06.941 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1 2022-09-02T05:10:06.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+Mz+hlcGC7dnMU1K00q1WlGDKI0= 2022-09-02T05:10:06.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+Mz+hlcGC7dnMU1K00q1WlGDKI0=', 'Content-Length': '0'} 2022-09-02T05:10:06.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1649b1d227866b6-006311902e-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.986 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:06.987 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:06.988 DEBUG:boto:Method: GET 2022-09-02T05:10:06.989 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:06.990 DEBUG:boto:Data: 2022-09-02T05:10:06.991 DEBUG:boto:Headers: {} 2022-09-02T05:10:06.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:06.993 DEBUG:boto:Port: 8000 2022-09-02T05:10:06.993 DEBUG:boto:Params: {} 2022-09-02T05:10:06.994 DEBUG:boto:Token: None 2022-09-02T05:10:06.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:06 GMT /xljbkf-62/obj1 2022-09-02T05:10:06.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+Mz+hlcGC7dnMU1K00q1WlGDKI0= 2022-09-02T05:10:06.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+Mz+hlcGC7dnMU1K00q1WlGDKI0=', 'Content-Length': '0'} 2022-09-02T05:10:06.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008442368a1d0eb518-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:06.999 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.000 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.001 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:07.001 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:07.002 DEBUG:boto:Method: GET 2022-09-02T05:10:07.002 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:07.003 DEBUG:boto:Data: 2022-09-02T05:10:07.003 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.004 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.005 DEBUG:boto:Params: {} 2022-09-02T05:10:07.006 DEBUG:boto:Token: None 2022-09-02T05:10:07.006 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac3d39001de037dc-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.011 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.012 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.012 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.013 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:07.013 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:07.014 DEBUG:boto:Method: GET 2022-09-02T05:10:07.014 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:07.015 DEBUG:boto:Data: 2022-09-02T05:10:07.015 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.016 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.017 DEBUG:boto:Params: {} 2022-09-02T05:10:07.018 DEBUG:boto:Token: None 2022-09-02T05:10:07.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000958af1115389926-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.070 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:07.072 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.072 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.073 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.074 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.075 DEBUG:boto:Method: GET 2022-09-02T05:10:07.076 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.076 DEBUG:boto:Data: 2022-09-02T05:10:07.077 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.078 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.078 DEBUG:boto:Params: {} 2022-09-02T05:10:07.079 DEBUG:boto:Token: None 2022-09-02T05:10:07.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg= 2022-09-02T05:10:07.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg=', 'Content-Length': '0'} 2022-09-02T05:10:07.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005557fc73319f7206-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.084 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.085 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.085 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.086 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.086 DEBUG:boto:Method: GET 2022-09-02T05:10:07.087 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.088 DEBUG:boto:Data: 2022-09-02T05:10:07.088 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.089 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.090 DEBUG:boto:Params: {} 2022-09-02T05:10:07.090 DEBUG:boto:Token: None 2022-09-02T05:10:07.091 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg= 2022-09-02T05:10:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg=', 'Content-Length': '0'} 2022-09-02T05:10:07.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000139ce4d8747545d9-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.095 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.096 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.096 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.097 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.097 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.098 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.099 DEBUG:boto:Data: 2022-09-02T05:10:07.099 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.100 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.101 DEBUG:boto:Params: {} 2022-09-02T05:10:07.101 DEBUG:boto:Token: None 2022-09-02T05:10:07.102 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rvebWqq55uzjsvAOBT258EoRAg= 2022-09-02T05:10:07.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rvebWqq55uzjsvAOBT258EoRAg=', 'Content-Length': '0'} 2022-09-02T05:10:07.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096a0b22f45a43647-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.106 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.106 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.107 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.107 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.108 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.108 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.109 DEBUG:boto:Data: 2022-09-02T05:10:07.109 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.110 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.111 DEBUG:boto:Params: {} 2022-09-02T05:10:07.111 DEBUG:boto:Token: None 2022-09-02T05:10:07.112 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rvebWqq55uzjsvAOBT258EoRAg= 2022-09-02T05:10:07.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rvebWqq55uzjsvAOBT258EoRAg=', 'Content-Length': '0'} 2022-09-02T05:10:07.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000993118a867d1c2e2-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:07.117 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.117 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.118 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.118 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.119 DEBUG:boto:Method: GET 2022-09-02T05:10:07.119 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.120 DEBUG:boto:Data: 2022-09-02T05:10:07.121 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.122 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.122 DEBUG:boto:Params: {} 2022-09-02T05:10:07.123 DEBUG:boto:Token: None 2022-09-02T05:10:07.124 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg= 2022-09-02T05:10:07.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg=', 'Content-Length': '0'} 2022-09-02T05:10:07.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000812bf7dad68cc4b2-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.127 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.128 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.128 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.129 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.130 DEBUG:boto:Method: GET 2022-09-02T05:10:07.130 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.131 DEBUG:boto:Data: 2022-09-02T05:10:07.131 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.132 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.133 DEBUG:boto:Params: {} 2022-09-02T05:10:07.133 DEBUG:boto:Token: None 2022-09-02T05:10:07.134 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:07.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg= 2022-09-02T05:10:07.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3e8DGqPOaJzJ42qRvo3SbCy1tOg=', 'Content-Length': '0'} 2022-09-02T05:10:07.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a04f0857c81fe16a-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.138 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.139 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.139 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.140 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.141 DEBUG:boto:Method: GET 2022-09-02T05:10:07.141 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.142 DEBUG:boto:Data: 2022-09-02T05:10:07.142 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.143 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.144 DEBUG:boto:Params: {} 2022-09-02T05:10:07.144 DEBUG:boto:Token: None 2022-09-02T05:10:07.145 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff23d2951521378f-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.150 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.150 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.151 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.152 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.152 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.153 DEBUG:boto:Method: GET 2022-09-02T05:10:07.153 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.154 DEBUG:boto:Data: 2022-09-02T05:10:07.154 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.155 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.156 DEBUG:boto:Params: {} 2022-09-02T05:10:07.156 DEBUG:boto:Token: None 2022-09-02T05:10:07.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae4ba5df17ca877-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.162 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.162 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.163 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.164 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.164 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:07.165 DEBUG:boto:Data: 2022-09-02T05:10:07.165 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.166 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.167 DEBUG:boto:Params: {} 2022-09-02T05:10:07.167 DEBUG:boto:Token: None 2022-09-02T05:10:07.168 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2 2022-09-02T05:10:07.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xbzP9OHbzi6oqrBvhvK2TJMvnqY= 2022-09-02T05:10:07.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xbzP9OHbzi6oqrBvhvK2TJMvnqY=', 'Content-Length': '0'} 2022-09-02T05:10:07.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000681a371381c49929-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.172 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.172 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.173 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.173 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:07.174 DEBUG:boto:Data: 2022-09-02T05:10:07.174 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.175 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.176 DEBUG:boto:Params: {} 2022-09-02T05:10:07.177 DEBUG:boto:Token: None 2022-09-02T05:10:07.177 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2 2022-09-02T05:10:07.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xbzP9OHbzi6oqrBvhvK2TJMvnqY= 2022-09-02T05:10:07.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xbzP9OHbzi6oqrBvhvK2TJMvnqY=', 'Content-Length': '0'} 2022-09-02T05:10:07.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db75ccbb37d4924c-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:07.182 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.182 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.183 DEBUG:boto:Method: GET 2022-09-02T05:10:07.183 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:07.184 DEBUG:boto:Data: 2022-09-02T05:10:07.184 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.185 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.186 DEBUG:boto:Params: {} 2022-09-02T05:10:07.187 DEBUG:boto:Token: None 2022-09-02T05:10:07.187 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2 2022-09-02T05:10:07.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:euweV9oxhP54fz3NIyPQzE8Lfrg= 2022-09-02T05:10:07.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:euweV9oxhP54fz3NIyPQzE8Lfrg=', 'Content-Length': '0'} 2022-09-02T05:10:07.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e22c1e44f441876d-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.191 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:07.192 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:07.193 DEBUG:boto:Method: GET 2022-09-02T05:10:07.193 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:07.194 DEBUG:boto:Data: 2022-09-02T05:10:07.194 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.195 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.196 DEBUG:boto:Params: {} 2022-09-02T05:10:07.196 DEBUG:boto:Token: None 2022-09-02T05:10:07.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj2 2022-09-02T05:10:07.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:euweV9oxhP54fz3NIyPQzE8Lfrg= 2022-09-02T05:10:07.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:euweV9oxhP54fz3NIyPQzE8Lfrg=', 'Content-Length': '0'} 2022-09-02T05:10:07.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037f71f20a25d9f3e-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.201 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.202 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.202 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.203 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.203 DEBUG:boto:Method: GET 2022-09-02T05:10:07.204 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.204 DEBUG:boto:Data: 2022-09-02T05:10:07.205 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.206 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.207 DEBUG:boto:Params: {} 2022-09-02T05:10:07.207 DEBUG:boto:Token: None 2022-09-02T05:10:07.208 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000609189d1e399b65a-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.254 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.255 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.256 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.257 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.257 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.259 DEBUG:boto:Method: GET 2022-09-02T05:10:07.259 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:07.260 DEBUG:boto:Data: 2022-09-02T05:10:07.261 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.264 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.265 DEBUG:boto:Params: {} 2022-09-02T05:10:07.266 DEBUG:boto:Token: None 2022-09-02T05:10:07.267 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074eb81a64f6d6dcd-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.272 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:07.274 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.274 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.275 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.276 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.276 DEBUG:boto:Method: GET 2022-09-02T05:10:07.277 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.277 DEBUG:boto:Data: 2022-09-02T05:10:07.278 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.278 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.279 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.280 DEBUG:boto:Params: {} 2022-09-02T05:10:07.280 DEBUG:boto:Token: None 2022-09-02T05:10:07.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo= 2022-09-02T05:10:07.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo=', 'Content-Length': '0'} 2022-09-02T05:10:07.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc5c235ed8bf48fb-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.285 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.286 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.287 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.287 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.288 DEBUG:boto:Method: GET 2022-09-02T05:10:07.289 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.290 DEBUG:boto:Data: 2022-09-02T05:10:07.290 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.291 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.292 DEBUG:boto:Params: {} 2022-09-02T05:10:07.293 DEBUG:boto:Token: None 2022-09-02T05:10:07.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo= 2022-09-02T05:10:07.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo=', 'Content-Length': '0'} 2022-09-02T05:10:07.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa517a160d0a5f41-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.338 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.339 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.340 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.341 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.342 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.343 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.344 DEBUG:boto:Data: 2022-09-02T05:10:07.345 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.347 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.348 DEBUG:boto:Params: {} 2022-09-02T05:10:07.349 DEBUG:boto:Token: None 2022-09-02T05:10:07.350 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OtEpSbYEMNTzG7j2f4robS5YkSQ= 2022-09-02T05:10:07.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OtEpSbYEMNTzG7j2f4robS5YkSQ=', 'Content-Length': '0'} 2022-09-02T05:10:07.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000362cde3b9eb16a90-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.356 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.357 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.358 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.359 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.360 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.361 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.362 DEBUG:boto:Data: 2022-09-02T05:10:07.363 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.365 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.366 DEBUG:boto:Params: {} 2022-09-02T05:10:07.367 DEBUG:boto:Token: None 2022-09-02T05:10:07.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OtEpSbYEMNTzG7j2f4robS5YkSQ= 2022-09-02T05:10:07.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OtEpSbYEMNTzG7j2f4robS5YkSQ=', 'Content-Length': '0'} 2022-09-02T05:10:07.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f2fe5198614e928-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:07.376 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.377 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.378 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.379 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.380 DEBUG:boto:Method: GET 2022-09-02T05:10:07.381 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.383 DEBUG:boto:Data: 2022-09-02T05:10:07.384 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.386 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.387 DEBUG:boto:Params: {} 2022-09-02T05:10:07.388 DEBUG:boto:Token: None 2022-09-02T05:10:07.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo= 2022-09-02T05:10:07.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo=', 'Content-Length': '0'} 2022-09-02T05:10:07.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000550e53586acbbeec-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.394 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.395 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.395 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.396 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.397 DEBUG:boto:Method: GET 2022-09-02T05:10:07.397 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.398 DEBUG:boto:Data: 2022-09-02T05:10:07.399 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.400 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.400 DEBUG:boto:Params: {} 2022-09-02T05:10:07.401 DEBUG:boto:Token: None 2022-09-02T05:10:07.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:07.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo= 2022-09-02T05:10:07.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nL6NYAgW2+sVWXNEtAzzB5DV1jo=', 'Content-Length': '0'} 2022-09-02T05:10:07.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046253101ced53d7a-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.406 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.407 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.408 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.408 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.409 DEBUG:boto:Method: GET 2022-09-02T05:10:07.409 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.410 DEBUG:boto:Data: 2022-09-02T05:10:07.410 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.411 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.412 DEBUG:boto:Params: {} 2022-09-02T05:10:07.412 DEBUG:boto:Token: None 2022-09-02T05:10:07.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a266370e3fbac7d0-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.418 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.419 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.419 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.420 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.420 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.421 DEBUG:boto:Method: GET 2022-09-02T05:10:07.422 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.422 DEBUG:boto:Data: 2022-09-02T05:10:07.423 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.424 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.424 DEBUG:boto:Params: {} 2022-09-02T05:10:07.425 DEBUG:boto:Token: None 2022-09-02T05:10:07.426 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6fe171c2b49b0c1-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.430 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.431 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.432 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.432 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.433 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:07.433 DEBUG:boto:Data: 2022-09-02T05:10:07.434 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.435 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.435 DEBUG:boto:Params: {} 2022-09-02T05:10:07.436 DEBUG:boto:Token: None 2022-09-02T05:10:07.436 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3 2022-09-02T05:10:07.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7doj1klE6oALz+KhQengctVLVyI= 2022-09-02T05:10:07.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7doj1klE6oALz+KhQengctVLVyI=', 'Content-Length': '0'} 2022-09-02T05:10:07.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000070d930a7c2b296b-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.440 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.441 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.441 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.442 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:07.442 DEBUG:boto:Data: 2022-09-02T05:10:07.443 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.444 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.444 DEBUG:boto:Params: {} 2022-09-02T05:10:07.445 DEBUG:boto:Token: None 2022-09-02T05:10:07.445 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3 2022-09-02T05:10:07.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7doj1klE6oALz+KhQengctVLVyI= 2022-09-02T05:10:07.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7doj1klE6oALz+KhQengctVLVyI=', 'Content-Length': '0'} 2022-09-02T05:10:07.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002961c60cfe9c9480-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:07.450 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.451 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.451 DEBUG:boto:Method: GET 2022-09-02T05:10:07.452 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:07.452 DEBUG:boto:Data: 2022-09-02T05:10:07.453 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.453 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.454 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.454 DEBUG:boto:Params: {} 2022-09-02T05:10:07.455 DEBUG:boto:Token: None 2022-09-02T05:10:07.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3 2022-09-02T05:10:07.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LRaHh68GpIcpd0nDrmSUS9NFeQk= 2022-09-02T05:10:07.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LRaHh68GpIcpd0nDrmSUS9NFeQk=', 'Content-Length': '0'} 2022-09-02T05:10:07.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3969dae5c3d59e0-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.459 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:07.460 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:07.461 DEBUG:boto:Method: GET 2022-09-02T05:10:07.461 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:07.462 DEBUG:boto:Data: 2022-09-02T05:10:07.462 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.463 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.464 DEBUG:boto:Params: {} 2022-09-02T05:10:07.464 DEBUG:boto:Token: None 2022-09-02T05:10:07.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj3 2022-09-02T05:10:07.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LRaHh68GpIcpd0nDrmSUS9NFeQk= 2022-09-02T05:10:07.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LRaHh68GpIcpd0nDrmSUS9NFeQk=', 'Content-Length': '0'} 2022-09-02T05:10:07.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000793f54cbf4b4ab94-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.469 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.469 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.470 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.471 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.471 DEBUG:boto:Method: GET 2022-09-02T05:10:07.472 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.472 DEBUG:boto:Data: 2022-09-02T05:10:07.473 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.474 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.474 DEBUG:boto:Params: {} 2022-09-02T05:10:07.475 DEBUG:boto:Token: None 2022-09-02T05:10:07.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a121438299dd30-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.522 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.524 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.525 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.526 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.526 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.527 DEBUG:boto:Method: GET 2022-09-02T05:10:07.528 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:07.529 DEBUG:boto:Data: 2022-09-02T05:10:07.530 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.531 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.531 DEBUG:boto:Params: {} 2022-09-02T05:10:07.532 DEBUG:boto:Token: None 2022-09-02T05:10:07.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000709af292d6ed133d-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.539 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:07.541 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.542 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.542 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.543 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.543 DEBUG:boto:Method: GET 2022-09-02T05:10:07.544 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.544 DEBUG:boto:Data: 2022-09-02T05:10:07.545 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.546 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.547 DEBUG:boto:Params: {} 2022-09-02T05:10:07.547 DEBUG:boto:Token: None 2022-09-02T05:10:07.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc= 2022-09-02T05:10:07.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc=', 'Content-Length': '0'} 2022-09-02T05:10:07.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7889fcd5746d14a-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.553 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.554 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.555 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.556 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.557 DEBUG:boto:Method: GET 2022-09-02T05:10:07.558 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.559 DEBUG:boto:Data: 2022-09-02T05:10:07.560 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.562 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.563 DEBUG:boto:Params: {} 2022-09-02T05:10:07.564 DEBUG:boto:Token: None 2022-09-02T05:10:07.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc= 2022-09-02T05:10:07.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc=', 'Content-Length': '0'} 2022-09-02T05:10:07.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071ea7fde6d398ee8-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.572 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.573 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.574 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.575 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.576 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.577 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.578 DEBUG:boto:Data: 2022-09-02T05:10:07.579 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.580 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.581 DEBUG:boto:Params: {} 2022-09-02T05:10:07.582 DEBUG:boto:Token: None 2022-09-02T05:10:07.583 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANl0h2PpUFII6H7U1R42BNqGWHk= 2022-09-02T05:10:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANl0h2PpUFII6H7U1R42BNqGWHk=', 'Content-Length': '0'} 2022-09-02T05:10:07.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a1b9f4e2e83a1cf-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.587 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.588 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.588 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.589 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.589 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.590 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.590 DEBUG:boto:Data: 2022-09-02T05:10:07.591 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.592 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.592 DEBUG:boto:Params: {} 2022-09-02T05:10:07.593 DEBUG:boto:Token: None 2022-09-02T05:10:07.594 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANl0h2PpUFII6H7U1R42BNqGWHk= 2022-09-02T05:10:07.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANl0h2PpUFII6H7U1R42BNqGWHk=', 'Content-Length': '0'} 2022-09-02T05:10:07.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fbe125cb057aa90-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:07.598 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.599 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.599 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.600 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.601 DEBUG:boto:Method: GET 2022-09-02T05:10:07.602 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.603 DEBUG:boto:Data: 2022-09-02T05:10:07.604 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.606 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.607 DEBUG:boto:Params: {} 2022-09-02T05:10:07.607 DEBUG:boto:Token: None 2022-09-02T05:10:07.608 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc= 2022-09-02T05:10:07.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc=', 'Content-Length': '0'} 2022-09-02T05:10:07.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79e70b5277d4518-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.612 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.613 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.614 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.614 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.615 DEBUG:boto:Method: GET 2022-09-02T05:10:07.616 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.616 DEBUG:boto:Data: 2022-09-02T05:10:07.617 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.618 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.618 DEBUG:boto:Params: {} 2022-09-02T05:10:07.619 DEBUG:boto:Token: None 2022-09-02T05:10:07.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:07.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc= 2022-09-02T05:10:07.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:05M1/ZkaPrWVZlmqEh+C42Yh+qc=', 'Content-Length': '0'} 2022-09-02T05:10:07.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028bdd4a7eab21da6-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.666 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.667 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.668 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.668 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.669 DEBUG:boto:Method: GET 2022-09-02T05:10:07.669 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.670 DEBUG:boto:Data: 2022-09-02T05:10:07.670 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.671 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.671 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.672 DEBUG:boto:Params: {} 2022-09-02T05:10:07.673 DEBUG:boto:Token: None 2022-09-02T05:10:07.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13e669aca8e0edd-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.679 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.679 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.680 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.681 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.681 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.682 DEBUG:boto:Method: GET 2022-09-02T05:10:07.682 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.683 DEBUG:boto:Data: 2022-09-02T05:10:07.683 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.684 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.685 DEBUG:boto:Params: {} 2022-09-02T05:10:07.685 DEBUG:boto:Token: None 2022-09-02T05:10:07.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003343f28e16c37eed-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.691 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.692 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.692 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.693 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.693 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:07.694 DEBUG:boto:Data: 2022-09-02T05:10:07.695 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.696 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.697 DEBUG:boto:Params: {} 2022-09-02T05:10:07.697 DEBUG:boto:Token: None 2022-09-02T05:10:07.698 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4 2022-09-02T05:10:07.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAXCq8FwOoKeh7IK8tlttjuEVIw= 2022-09-02T05:10:07.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAXCq8FwOoKeh7IK8tlttjuEVIw=', 'Content-Length': '0'} 2022-09-02T05:10:07.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da4744c590ce61f9-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.703 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.704 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.704 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.705 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:07.706 DEBUG:boto:Data: 2022-09-02T05:10:07.706 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.707 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.708 DEBUG:boto:Params: {} 2022-09-02T05:10:07.709 DEBUG:boto:Token: None 2022-09-02T05:10:07.709 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4 2022-09-02T05:10:07.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAXCq8FwOoKeh7IK8tlttjuEVIw= 2022-09-02T05:10:07.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAXCq8FwOoKeh7IK8tlttjuEVIw=', 'Content-Length': '0'} 2022-09-02T05:10:07.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d1a1e5527139940-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:07.715 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.715 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.716 DEBUG:boto:Method: GET 2022-09-02T05:10:07.717 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:07.717 DEBUG:boto:Data: 2022-09-02T05:10:07.718 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.718 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.719 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.719 DEBUG:boto:Params: {} 2022-09-02T05:10:07.720 DEBUG:boto:Token: None 2022-09-02T05:10:07.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4 2022-09-02T05:10:07.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:caZJHcbHZp+mhsgAVVI24VCxXu0= 2022-09-02T05:10:07.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:caZJHcbHZp+mhsgAVVI24VCxXu0=', 'Content-Length': '0'} 2022-09-02T05:10:07.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002316fa87e6d868f4-006311902f-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.766 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:07.767 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:07.767 DEBUG:boto:Method: GET 2022-09-02T05:10:07.768 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:07.768 DEBUG:boto:Data: 2022-09-02T05:10:07.769 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.770 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.771 DEBUG:boto:Params: {} 2022-09-02T05:10:07.772 DEBUG:boto:Token: None 2022-09-02T05:10:07.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj4 2022-09-02T05:10:07.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:caZJHcbHZp+mhsgAVVI24VCxXu0= 2022-09-02T05:10:07.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:caZJHcbHZp+mhsgAVVI24VCxXu0=', 'Content-Length': '0'} 2022-09-02T05:10:07.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ce8117c3c0939cb-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.777 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.778 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.779 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.779 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.780 DEBUG:boto:Method: GET 2022-09-02T05:10:07.780 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.781 DEBUG:boto:Data: 2022-09-02T05:10:07.781 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.782 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:07.783 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.783 DEBUG:boto:Params: {} 2022-09-02T05:10:07.784 DEBUG:boto:Token: None 2022-09-02T05:10:07.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002312a564ae082c1-006311902f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.792 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.794 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.795 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.796 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.797 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.798 DEBUG:boto:Method: GET 2022-09-02T05:10:07.799 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:07.800 DEBUG:boto:Data: 2022-09-02T05:10:07.801 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.803 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.804 DEBUG:boto:Params: {} 2022-09-02T05:10:07.805 DEBUG:boto:Token: None 2022-09-02T05:10:07.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs= 2022-09-02T05:10:07.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K+yGZarDMVOpNDjfXXkOyAm5mDs=', 'Content-Length': '0'} 2022-09-02T05:10:07.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd647ad7a9f3504-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.812 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-62 zones={test-zone2, test-zone1} 2022-09-02T05:10:07.813 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-62 zones={test-zone3, test-zone1} 2022-09-02T05:10:07.814 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.814 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.815 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.815 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:10:07.816 DEBUG:boto:Data: 2022-09-02T05:10:07.816 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:07.817 DEBUG:boto:Port: 8001 2022-09-02T05:10:07.818 DEBUG:boto:Params: {} 2022-09-02T05:10:07.819 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:10:07.819 DEBUG:boto:Token: None 2022-09-02T05:10:07.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mjX0d6sxh8MSY/qz/kFJAUyyFGk= 2022-09-02T05:10:07.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mjX0d6sxh8MSY/qz/kFJAUyyFGk=', 'Content-Length': '0'} 2022-09-02T05:10:07.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007861e30d637d4e20-006311902f-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.825 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.826 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.826 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.827 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:10:07.827 DEBUG:boto:Data: 2022-09-02T05:10:07.828 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.829 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.829 DEBUG:boto:Params: {} 2022-09-02T05:10:07.830 DEBUG:boto:Token: None 2022-09-02T05:10:07.831 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/ 2022-09-02T05:10:07.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mjX0d6sxh8MSY/qz/kFJAUyyFGk= 2022-09-02T05:10:07.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mjX0d6sxh8MSY/qz/kFJAUyyFGk=', 'Content-Length': '0'} 2022-09-02T05:10:07.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a905e7526116cfe9-006311902f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:10:07.836 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.836 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.837 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.837 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.838 DEBUG:boto:Method: GET 2022-09-02T05:10:07.838 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.839 DEBUG:boto:Data: 2022-09-02T05:10:07.839 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:07.840 DEBUG:boto:Port: 8001 2022-09-02T05:10:07.841 DEBUG:boto:Params: {} 2022-09-02T05:10:07.841 DEBUG:boto:Token: None 2022-09-02T05:10:07.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/?versions 2022-09-02T05:10:07.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q= 2022-09-02T05:10:07.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q=', 'Content-Length': '0'} 2022-09-02T05:10:07.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014d60292092dcb39-006311902f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.847 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:10:07.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:10:07.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:10:07.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:10:07.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:10:07.851 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.851 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.852 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.852 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.853 DEBUG:boto:Method: GET 2022-09-02T05:10:07.853 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.854 DEBUG:boto:Data: 2022-09-02T05:10:07.854 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.855 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.856 DEBUG:boto:Params: {} 2022-09-02T05:10:07.856 DEBUG:boto:Token: None 2022-09-02T05:10:07.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/?versions 2022-09-02T05:10:07.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q= 2022-09-02T05:10:07.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q=', 'Content-Length': '0'} 2022-09-02T05:10:07.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005861f28d2b6fe6a8-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.862 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:10:07.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:10:07.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:10:07.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:10:07.865 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.866 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.866 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.867 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.867 DEBUG:boto:Method: GET 2022-09-02T05:10:07.868 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.868 DEBUG:boto:Data: 2022-09-02T05:10:07.869 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:07.870 DEBUG:boto:Port: 8001 2022-09-02T05:10:07.871 DEBUG:boto:Params: {} 2022-09-02T05:10:07.871 DEBUG:boto:Token: None 2022-09-02T05:10:07.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/?versions 2022-09-02T05:10:07.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q= 2022-09-02T05:10:07.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q=', 'Content-Length': '0'} 2022-09-02T05:10:07.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbdbb2b3d9f725a0-006311902f-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.877 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.878 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:07.879 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:07.879 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.880 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.880 DEBUG:boto:Method: GET 2022-09-02T05:10:07.881 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:07.881 DEBUG:boto:Data: 2022-09-02T05:10:07.882 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.883 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.884 DEBUG:boto:Params: {} 2022-09-02T05:10:07.885 DEBUG:boto:Token: None 2022-09-02T05:10:07.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/?versions 2022-09-02T05:10:07.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q= 2022-09-02T05:10:07.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gWEJH8yKkdLmuBJ+3mKzi1xFK8Q=', 'Content-Length': '0'} 2022-09-02T05:10:07.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eeb51dd1aebccf5-006311902f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.893 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:07.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:07.895 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:07.896 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:07.896 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.897 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.898 DEBUG:boto:Method: GET 2022-09-02T05:10:07.899 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.900 DEBUG:boto:Data: 2022-09-02T05:10:07.901 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:07.903 DEBUG:boto:Port: 8001 2022-09-02T05:10:07.904 DEBUG:boto:Params: {} 2022-09-02T05:10:07.905 DEBUG:boto:Token: None 2022-09-02T05:10:07.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OGbLhH3pxCZ8KKHTUnaOxr0Sscc= 2022-09-02T05:10:07.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OGbLhH3pxCZ8KKHTUnaOxr0Sscc=', 'Content-Length': '0'} 2022-09-02T05:10:07.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035dc8e23ee7e3cf3-006311902f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.911 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:07.911 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:07.912 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.913 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.913 DEBUG:boto:Method: GET 2022-09-02T05:10:07.914 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.914 DEBUG:boto:Data: 2022-09-02T05:10:07.915 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.916 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.917 DEBUG:boto:Params: {} 2022-09-02T05:10:07.917 DEBUG:boto:Token: None 2022-09-02T05:10:07.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OGbLhH3pxCZ8KKHTUnaOxr0Sscc= 2022-09-02T05:10:07.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OGbLhH3pxCZ8KKHTUnaOxr0Sscc=', 'Content-Length': '0'} 2022-09-02T05:10:07.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc7a8f4352c0563e-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.962 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:07.963 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:07.963 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.964 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.964 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.965 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.965 DEBUG:boto:Data: 2022-09-02T05:10:07.966 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:07.967 DEBUG:boto:Port: 8001 2022-09-02T05:10:07.968 DEBUG:boto:Params: {} 2022-09-02T05:10:07.968 DEBUG:boto:Token: None 2022-09-02T05:10:07.969 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qMmI5yqVC4d/glTakv9/dBku5PM= 2022-09-02T05:10:07.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qMmI5yqVC4d/glTakv9/dBku5PM=', 'Content-Length': '0'} 2022-09-02T05:10:07.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bdd6409183087f3-006311902f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.973 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:07.974 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:07.974 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.975 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.975 DEBUG:boto:Method: HEAD 2022-09-02T05:10:07.976 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.976 DEBUG:boto:Data: 2022-09-02T05:10:07.977 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:07.978 DEBUG:boto:Port: 8000 2022-09-02T05:10:07.978 DEBUG:boto:Params: {} 2022-09-02T05:10:07.979 DEBUG:boto:Token: None 2022-09-02T05:10:07.979 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qMmI5yqVC4d/glTakv9/dBku5PM= 2022-09-02T05:10:07.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qMmI5yqVC4d/glTakv9/dBku5PM=', 'Content-Length': '0'} 2022-09-02T05:10:07.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d15e8177e1a72f5-006311902f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:07.984 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:07.985 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:07.985 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.986 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.986 DEBUG:boto:Method: GET 2022-09-02T05:10:07.987 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.988 DEBUG:boto:Data: 2022-09-02T05:10:07.988 DEBUG:boto:Headers: {} 2022-09-02T05:10:07.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:07.989 DEBUG:boto:Port: 8001 2022-09-02T05:10:07.990 DEBUG:boto:Params: {} 2022-09-02T05:10:07.991 DEBUG:boto:Token: None 2022-09-02T05:10:07.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:07 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OGbLhH3pxCZ8KKHTUnaOxr0Sscc= 2022-09-02T05:10:07.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OGbLhH3pxCZ8KKHTUnaOxr0Sscc=', 'Content-Length': '0'} 2022-09-02T05:10:07.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8162b424e8ef6dc-006311902f-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:07.996 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:07.996 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:07.997 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.997 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.998 DEBUG:boto:Method: GET 2022-09-02T05:10:07.998 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:07.999 DEBUG:boto:Data: 2022-09-02T05:10:07.999 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.001 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.001 DEBUG:boto:Params: {} 2022-09-02T05:10:08.002 DEBUG:boto:Token: None 2022-09-02T05:10:08.002 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+8ztcyOhTDqnxlJPcwpO4p6Igw= 2022-09-02T05:10:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+8ztcyOhTDqnxlJPcwpO4p6Igw=', 'Content-Length': '0'} 2022-09-02T05:10:08.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dae02f85fd22bfbb-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.007 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.008 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.008 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.009 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.009 DEBUG:boto:Method: GET 2022-09-02T05:10:08.010 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.011 DEBUG:boto:Data: 2022-09-02T05:10:08.011 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.012 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.013 DEBUG:boto:Params: {} 2022-09-02T05:10:08.013 DEBUG:boto:Token: None 2022-09-02T05:10:08.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53fb5235e1dfbce-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.019 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.020 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.020 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.021 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.021 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.022 DEBUG:boto:Method: GET 2022-09-02T05:10:08.022 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.023 DEBUG:boto:Data: 2022-09-02T05:10:08.024 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.025 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.025 DEBUG:boto:Params: {} 2022-09-02T05:10:08.026 DEBUG:boto:Token: None 2022-09-02T05:10:08.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008236f9edc5cb1842-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.078 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.079 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:08.080 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:08.080 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.081 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:08.081 DEBUG:boto:Data: 2022-09-02T05:10:08.082 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.083 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.083 DEBUG:boto:Params: {} 2022-09-02T05:10:08.084 DEBUG:boto:Token: None 2022-09-02T05:10:08.084 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1 2022-09-02T05:10:08.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9lMgWh9ZkHiM896hsMfPffGKmDE= 2022-09-02T05:10:08.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9lMgWh9ZkHiM896hsMfPffGKmDE=', 'Content-Length': '0'} 2022-09-02T05:10:08.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db59fdfb74fe8eaf-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.089 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:08.090 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:08.090 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.091 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:08.091 DEBUG:boto:Data: 2022-09-02T05:10:08.092 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.093 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.093 DEBUG:boto:Params: {} 2022-09-02T05:10:08.094 DEBUG:boto:Token: None 2022-09-02T05:10:08.094 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1 2022-09-02T05:10:08.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9lMgWh9ZkHiM896hsMfPffGKmDE= 2022-09-02T05:10:08.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9lMgWh9ZkHiM896hsMfPffGKmDE=', 'Content-Length': '0'} 2022-09-02T05:10:08.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043a112a5791b604e-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:08.099 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:08.100 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:08.100 DEBUG:boto:Method: GET 2022-09-02T05:10:08.101 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:08.102 DEBUG:boto:Data: 2022-09-02T05:10:08.102 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.103 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.104 DEBUG:boto:Params: {} 2022-09-02T05:10:08.104 DEBUG:boto:Token: None 2022-09-02T05:10:08.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1 2022-09-02T05:10:08.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PGM2PCWwMHlQ6/FLklDqL2Y3HcY= 2022-09-02T05:10:08.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PGM2PCWwMHlQ6/FLklDqL2Y3HcY=', 'Content-Length': '0'} 2022-09-02T05:10:08.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000541173383b0294a9-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.110 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:08.110 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:08.111 DEBUG:boto:Method: GET 2022-09-02T05:10:08.111 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:08.112 DEBUG:boto:Data: 2022-09-02T05:10:08.112 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.114 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.114 DEBUG:boto:Params: {} 2022-09-02T05:10:08.115 DEBUG:boto:Token: None 2022-09-02T05:10:08.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1 2022-09-02T05:10:08.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PGM2PCWwMHlQ6/FLklDqL2Y3HcY= 2022-09-02T05:10:08.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PGM2PCWwMHlQ6/FLklDqL2Y3HcY=', 'Content-Length': '0'} 2022-09-02T05:10:08.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000405644e63e32e512-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.120 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.120 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.121 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.122 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.122 DEBUG:boto:Method: GET 2022-09-02T05:10:08.123 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.123 DEBUG:boto:Data: 2022-09-02T05:10:08.124 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.126 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.127 DEBUG:boto:Params: {} 2022-09-02T05:10:08.128 DEBUG:boto:Token: None 2022-09-02T05:10:08.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3de28128cd87ab-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.135 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.136 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.137 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.137 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.138 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.138 DEBUG:boto:Method: GET 2022-09-02T05:10:08.139 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:08.140 DEBUG:boto:Data: 2022-09-02T05:10:08.140 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.141 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.142 DEBUG:boto:Params: {} 2022-09-02T05:10:08.143 DEBUG:boto:Token: None 2022-09-02T05:10:08.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee2798fb1aef6ad-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.148 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:08.149 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.150 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.150 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.151 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.152 DEBUG:boto:Method: GET 2022-09-02T05:10:08.153 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.153 DEBUG:boto:Data: 2022-09-02T05:10:08.154 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.155 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.156 DEBUG:boto:Params: {} 2022-09-02T05:10:08.156 DEBUG:boto:Token: None 2022-09-02T05:10:08.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA= 2022-09-02T05:10:08.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA=', 'Content-Length': '0'} 2022-09-02T05:10:08.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b75b63fdc498aab0-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.202 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.203 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.204 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.204 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.205 DEBUG:boto:Method: GET 2022-09-02T05:10:08.205 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.206 DEBUG:boto:Data: 2022-09-02T05:10:08.207 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.209 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.210 DEBUG:boto:Params: {} 2022-09-02T05:10:08.211 DEBUG:boto:Token: None 2022-09-02T05:10:08.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA= 2022-09-02T05:10:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA=', 'Content-Length': '0'} 2022-09-02T05:10:08.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d82abef25c14b7d1-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.217 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.217 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.218 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.219 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.219 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.220 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.220 DEBUG:boto:Data: 2022-09-02T05:10:08.221 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.222 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.222 DEBUG:boto:Params: {} 2022-09-02T05:10:08.223 DEBUG:boto:Token: None 2022-09-02T05:10:08.224 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wOpmfimgf4ijFB8i+xFvMW3DrdI= 2022-09-02T05:10:08.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wOpmfimgf4ijFB8i+xFvMW3DrdI=', 'Content-Length': '0'} 2022-09-02T05:10:08.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000076825d0ef752aa9-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.228 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.228 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.229 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.230 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.230 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.231 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.231 DEBUG:boto:Data: 2022-09-02T05:10:08.232 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.233 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.233 DEBUG:boto:Params: {} 2022-09-02T05:10:08.234 DEBUG:boto:Token: None 2022-09-02T05:10:08.234 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.235 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wOpmfimgf4ijFB8i+xFvMW3DrdI= 2022-09-02T05:10:08.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wOpmfimgf4ijFB8i+xFvMW3DrdI=', 'Content-Length': '0'} 2022-09-02T05:10:08.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000713d35eb0e4a9353-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:08.239 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.240 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.240 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.241 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.241 DEBUG:boto:Method: GET 2022-09-02T05:10:08.242 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.243 DEBUG:boto:Data: 2022-09-02T05:10:08.243 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.244 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.245 DEBUG:boto:Params: {} 2022-09-02T05:10:08.245 DEBUG:boto:Token: None 2022-09-02T05:10:08.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA= 2022-09-02T05:10:08.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA=', 'Content-Length': '0'} 2022-09-02T05:10:08.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdc78b3b17f0d39e-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.250 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.251 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.251 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.252 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.253 DEBUG:boto:Method: GET 2022-09-02T05:10:08.253 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.254 DEBUG:boto:Data: 2022-09-02T05:10:08.254 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.255 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.256 DEBUG:boto:Params: {} 2022-09-02T05:10:08.257 DEBUG:boto:Token: None 2022-09-02T05:10:08.257 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:08.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA= 2022-09-02T05:10:08.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:45KDqLc/3ijs2NiumW16DN3X1iA=', 'Content-Length': '0'} 2022-09-02T05:10:08.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053e6ea466aa32889-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.262 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.263 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.264 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.265 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.266 DEBUG:boto:Method: GET 2022-09-02T05:10:08.267 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.268 DEBUG:boto:Data: 2022-09-02T05:10:08.269 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.271 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.272 DEBUG:boto:Params: {} 2022-09-02T05:10:08.273 DEBUG:boto:Token: None 2022-09-02T05:10:08.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc3c373d39c9250-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.280 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.282 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.283 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.284 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.285 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.286 DEBUG:boto:Method: GET 2022-09-02T05:10:08.287 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.287 DEBUG:boto:Data: 2022-09-02T05:10:08.288 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.290 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.291 DEBUG:boto:Params: {} 2022-09-02T05:10:08.292 DEBUG:boto:Token: None 2022-09-02T05:10:08.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f98a6b8bf75dc3-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.299 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.299 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.300 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.301 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.301 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:08.302 DEBUG:boto:Data: 2022-09-02T05:10:08.302 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.303 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.304 DEBUG:boto:Params: {} 2022-09-02T05:10:08.305 DEBUG:boto:Token: None 2022-09-02T05:10:08.305 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2 2022-09-02T05:10:08.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bl0/CtFC65YbBlttHGxLgv8PDJs= 2022-09-02T05:10:08.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bl0/CtFC65YbBlttHGxLgv8PDJs=', 'Content-Length': '0'} 2022-09-02T05:10:08.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ab08944ed46498f-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.309 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.310 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.310 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.311 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:08.311 DEBUG:boto:Data: 2022-09-02T05:10:08.312 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.313 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.313 DEBUG:boto:Params: {} 2022-09-02T05:10:08.314 DEBUG:boto:Token: None 2022-09-02T05:10:08.314 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2 2022-09-02T05:10:08.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bl0/CtFC65YbBlttHGxLgv8PDJs= 2022-09-02T05:10:08.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bl0/CtFC65YbBlttHGxLgv8PDJs=', 'Content-Length': '0'} 2022-09-02T05:10:08.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa5b62df02056d64-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:08.319 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.320 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.320 DEBUG:boto:Method: GET 2022-09-02T05:10:08.321 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:08.321 DEBUG:boto:Data: 2022-09-02T05:10:08.322 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.323 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.323 DEBUG:boto:Params: {} 2022-09-02T05:10:08.324 DEBUG:boto:Token: None 2022-09-02T05:10:08.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2 2022-09-02T05:10:08.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EoRWg5TrdLwu+eS9FUEc16bCcqE= 2022-09-02T05:10:08.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EoRWg5TrdLwu+eS9FUEc16bCcqE=', 'Content-Length': '0'} 2022-09-02T05:10:08.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a865a2f2afd1f8f-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.329 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:08.329 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:08.330 DEBUG:boto:Method: GET 2022-09-02T05:10:08.330 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:08.331 DEBUG:boto:Data: 2022-09-02T05:10:08.331 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.332 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.333 DEBUG:boto:Params: {} 2022-09-02T05:10:08.333 DEBUG:boto:Token: None 2022-09-02T05:10:08.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj2 2022-09-02T05:10:08.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EoRWg5TrdLwu+eS9FUEc16bCcqE= 2022-09-02T05:10:08.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EoRWg5TrdLwu+eS9FUEc16bCcqE=', 'Content-Length': '0'} 2022-09-02T05:10:08.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005eeb716a6b19b03e-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.379 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.379 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.380 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.380 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.381 DEBUG:boto:Method: GET 2022-09-02T05:10:08.381 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.382 DEBUG:boto:Data: 2022-09-02T05:10:08.382 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.384 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.384 DEBUG:boto:Params: {} 2022-09-02T05:10:08.385 DEBUG:boto:Token: None 2022-09-02T05:10:08.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57d8520316ae43f-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.390 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.391 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.392 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.392 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.393 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.393 DEBUG:boto:Method: GET 2022-09-02T05:10:08.394 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:08.394 DEBUG:boto:Data: 2022-09-02T05:10:08.395 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.396 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.396 DEBUG:boto:Params: {} 2022-09-02T05:10:08.397 DEBUG:boto:Token: None 2022-09-02T05:10:08.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003044a25892eaebe3-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.403 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:08.405 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.405 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.406 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.406 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.407 DEBUG:boto:Method: GET 2022-09-02T05:10:08.407 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.408 DEBUG:boto:Data: 2022-09-02T05:10:08.408 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.409 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.410 DEBUG:boto:Params: {} 2022-09-02T05:10:08.411 DEBUG:boto:Token: None 2022-09-02T05:10:08.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM= 2022-09-02T05:10:08.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM=', 'Content-Length': '0'} 2022-09-02T05:10:08.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004de89d6b0dcc9870-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.416 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.416 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.417 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.417 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.418 DEBUG:boto:Method: GET 2022-09-02T05:10:08.418 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.419 DEBUG:boto:Data: 2022-09-02T05:10:08.419 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.420 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.421 DEBUG:boto:Params: {} 2022-09-02T05:10:08.422 DEBUG:boto:Token: None 2022-09-02T05:10:08.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM= 2022-09-02T05:10:08.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM=', 'Content-Length': '0'} 2022-09-02T05:10:08.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d90b37bb7adccd3-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.427 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.428 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.428 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.429 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.429 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.430 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.430 DEBUG:boto:Data: 2022-09-02T05:10:08.431 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.432 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.432 DEBUG:boto:Params: {} 2022-09-02T05:10:08.433 DEBUG:boto:Token: None 2022-09-02T05:10:08.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5qP5Zt1VLXPFtYCbIAvENsDAxJE= 2022-09-02T05:10:08.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5qP5Zt1VLXPFtYCbIAvENsDAxJE=', 'Content-Length': '0'} 2022-09-02T05:10:08.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1c8ddd48dacdf7-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.437 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.438 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.439 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.439 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.440 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.440 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.441 DEBUG:boto:Data: 2022-09-02T05:10:08.441 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.442 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.443 DEBUG:boto:Params: {} 2022-09-02T05:10:08.443 DEBUG:boto:Token: None 2022-09-02T05:10:08.444 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5qP5Zt1VLXPFtYCbIAvENsDAxJE= 2022-09-02T05:10:08.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5qP5Zt1VLXPFtYCbIAvENsDAxJE=', 'Content-Length': '0'} 2022-09-02T05:10:08.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1d61f2659d92cc4-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:08.449 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.450 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.450 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.451 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.451 DEBUG:boto:Method: GET 2022-09-02T05:10:08.452 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.453 DEBUG:boto:Data: 2022-09-02T05:10:08.453 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.455 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.455 DEBUG:boto:Params: {} 2022-09-02T05:10:08.456 DEBUG:boto:Token: None 2022-09-02T05:10:08.457 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM= 2022-09-02T05:10:08.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM=', 'Content-Length': '0'} 2022-09-02T05:10:08.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000147e3dd68622fa2c-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.502 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.503 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.503 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.504 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.505 DEBUG:boto:Method: GET 2022-09-02T05:10:08.505 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.506 DEBUG:boto:Data: 2022-09-02T05:10:08.507 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.516 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.517 DEBUG:boto:Params: {} 2022-09-02T05:10:08.517 DEBUG:boto:Token: None 2022-09-02T05:10:08.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:08.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM= 2022-09-02T05:10:08.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kVU/kgVMEHveYowRColwAv8P8RM=', 'Content-Length': '0'} 2022-09-02T05:10:08.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac568c3a1ac8f4ba-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.523 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.524 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.524 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.525 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.525 DEBUG:boto:Method: GET 2022-09-02T05:10:08.526 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.527 DEBUG:boto:Data: 2022-09-02T05:10:08.528 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.529 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.530 DEBUG:boto:Params: {} 2022-09-02T05:10:08.530 DEBUG:boto:Token: None 2022-09-02T05:10:08.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000131ece564a7e3dc4-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.536 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.537 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.538 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.539 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.540 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.541 DEBUG:boto:Method: GET 2022-09-02T05:10:08.542 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.543 DEBUG:boto:Data: 2022-09-02T05:10:08.544 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.546 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.547 DEBUG:boto:Params: {} 2022-09-02T05:10:08.548 DEBUG:boto:Token: None 2022-09-02T05:10:08.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005474e401ba784bce-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.555 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.555 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.556 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.556 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.557 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:08.557 DEBUG:boto:Data: 2022-09-02T05:10:08.558 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.559 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.559 DEBUG:boto:Params: {} 2022-09-02T05:10:08.560 DEBUG:boto:Token: None 2022-09-02T05:10:08.561 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3 2022-09-02T05:10:08.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O6gIV1UGyo9PlPon9u0sQnNzG6c= 2022-09-02T05:10:08.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O6gIV1UGyo9PlPon9u0sQnNzG6c=', 'Content-Length': '0'} 2022-09-02T05:10:08.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5df54e1bb01ab9b-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.565 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.565 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.566 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.566 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:08.567 DEBUG:boto:Data: 2022-09-02T05:10:08.567 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.568 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.569 DEBUG:boto:Params: {} 2022-09-02T05:10:08.569 DEBUG:boto:Token: None 2022-09-02T05:10:08.570 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3 2022-09-02T05:10:08.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O6gIV1UGyo9PlPon9u0sQnNzG6c= 2022-09-02T05:10:08.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O6gIV1UGyo9PlPon9u0sQnNzG6c=', 'Content-Length': '0'} 2022-09-02T05:10:08.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1e5cc4063f47dc0-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:08.575 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.575 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.576 DEBUG:boto:Method: GET 2022-09-02T05:10:08.576 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:08.577 DEBUG:boto:Data: 2022-09-02T05:10:08.577 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.578 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.579 DEBUG:boto:Params: {} 2022-09-02T05:10:08.579 DEBUG:boto:Token: None 2022-09-02T05:10:08.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3 2022-09-02T05:10:08.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l8EUOAyJAeUMx4mNT6JlSpjDYqg= 2022-09-02T05:10:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l8EUOAyJAeUMx4mNT6JlSpjDYqg=', 'Content-Length': '0'} 2022-09-02T05:10:08.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be87b83b06848453-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.584 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:08.585 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:08.585 DEBUG:boto:Method: GET 2022-09-02T05:10:08.586 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:08.586 DEBUG:boto:Data: 2022-09-02T05:10:08.587 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.588 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.589 DEBUG:boto:Params: {} 2022-09-02T05:10:08.589 DEBUG:boto:Token: None 2022-09-02T05:10:08.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj3 2022-09-02T05:10:08.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l8EUOAyJAeUMx4mNT6JlSpjDYqg= 2022-09-02T05:10:08.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l8EUOAyJAeUMx4mNT6JlSpjDYqg=', 'Content-Length': '0'} 2022-09-02T05:10:08.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff1289fbb7fbae9e-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.594 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.595 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.595 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.596 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.596 DEBUG:boto:Method: GET 2022-09-02T05:10:08.597 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.597 DEBUG:boto:Data: 2022-09-02T05:10:08.598 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.599 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.599 DEBUG:boto:Params: {} 2022-09-02T05:10:08.600 DEBUG:boto:Token: None 2022-09-02T05:10:08.600 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cdb07bf0c47ab84-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.605 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.606 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.606 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.607 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.607 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.608 DEBUG:boto:Method: GET 2022-09-02T05:10:08.608 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:08.609 DEBUG:boto:Data: 2022-09-02T05:10:08.609 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.610 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.611 DEBUG:boto:Params: {} 2022-09-02T05:10:08.612 DEBUG:boto:Token: None 2022-09-02T05:10:08.612 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff4fcfeba47bb63d-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.616 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:08.618 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.618 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.619 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.619 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.620 DEBUG:boto:Method: GET 2022-09-02T05:10:08.620 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.621 DEBUG:boto:Data: 2022-09-02T05:10:08.622 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.623 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.623 DEBUG:boto:Params: {} 2022-09-02T05:10:08.624 DEBUG:boto:Token: None 2022-09-02T05:10:08.624 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk= 2022-09-02T05:10:08.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk=', 'Content-Length': '0'} 2022-09-02T05:10:08.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3c5234dd810eaba-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.629 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.629 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.630 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.630 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.631 DEBUG:boto:Method: GET 2022-09-02T05:10:08.631 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.632 DEBUG:boto:Data: 2022-09-02T05:10:08.633 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.634 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.634 DEBUG:boto:Params: {} 2022-09-02T05:10:08.635 DEBUG:boto:Token: None 2022-09-02T05:10:08.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk= 2022-09-02T05:10:08.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk=', 'Content-Length': '0'} 2022-09-02T05:10:08.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d40d654726b60585-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.640 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.640 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.641 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.641 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.642 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.642 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.643 DEBUG:boto:Data: 2022-09-02T05:10:08.643 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.644 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.645 DEBUG:boto:Params: {} 2022-09-02T05:10:08.646 DEBUG:boto:Token: None 2022-09-02T05:10:08.646 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dkxBU3FMwLHrGvO1a0SWTT54xyg= 2022-09-02T05:10:08.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dkxBU3FMwLHrGvO1a0SWTT54xyg=', 'Content-Length': '0'} 2022-09-02T05:10:08.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056e67fbf92014b96-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.650 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.651 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.651 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.652 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.652 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.653 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.653 DEBUG:boto:Data: 2022-09-02T05:10:08.654 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.655 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.655 DEBUG:boto:Params: {} 2022-09-02T05:10:08.656 DEBUG:boto:Token: None 2022-09-02T05:10:08.657 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dkxBU3FMwLHrGvO1a0SWTT54xyg= 2022-09-02T05:10:08.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dkxBU3FMwLHrGvO1a0SWTT54xyg=', 'Content-Length': '0'} 2022-09-02T05:10:08.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001869673c7e4c27ad-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:08.661 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.662 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.662 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.663 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.663 DEBUG:boto:Method: GET 2022-09-02T05:10:08.664 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.664 DEBUG:boto:Data: 2022-09-02T05:10:08.665 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.666 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.667 DEBUG:boto:Params: {} 2022-09-02T05:10:08.667 DEBUG:boto:Token: None 2022-09-02T05:10:08.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk= 2022-09-02T05:10:08.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk=', 'Content-Length': '0'} 2022-09-02T05:10:08.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6db06b1cfc082bd-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.672 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.673 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.673 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.674 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.674 DEBUG:boto:Method: GET 2022-09-02T05:10:08.675 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.675 DEBUG:boto:Data: 2022-09-02T05:10:08.676 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.677 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.677 DEBUG:boto:Params: {} 2022-09-02T05:10:08.678 DEBUG:boto:Token: None 2022-09-02T05:10:08.679 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:08.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk= 2022-09-02T05:10:08.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3dVH8wHtM5oEjZ2gpMdPEYsDGk=', 'Content-Length': '0'} 2022-09-02T05:10:08.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9d84f2389096a4a-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.683 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.684 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.684 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.685 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.685 DEBUG:boto:Method: GET 2022-09-02T05:10:08.686 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.686 DEBUG:boto:Data: 2022-09-02T05:10:08.687 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.688 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.688 DEBUG:boto:Params: {} 2022-09-02T05:10:08.689 DEBUG:boto:Token: None 2022-09-02T05:10:08.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cadf741afd25946-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.694 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.695 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.696 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.696 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.697 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.697 DEBUG:boto:Method: GET 2022-09-02T05:10:08.698 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.698 DEBUG:boto:Data: 2022-09-02T05:10:08.699 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.700 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.701 DEBUG:boto:Params: {} 2022-09-02T05:10:08.701 DEBUG:boto:Token: None 2022-09-02T05:10:08.702 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2302d4fa9209209-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.750 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.751 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.752 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.752 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.753 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:08.753 DEBUG:boto:Data: 2022-09-02T05:10:08.754 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.755 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.756 DEBUG:boto:Params: {} 2022-09-02T05:10:08.756 DEBUG:boto:Token: None 2022-09-02T05:10:08.757 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4 2022-09-02T05:10:08.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8AwRlV4ZopNPoVj3j0cBs+sd2hE= 2022-09-02T05:10:08.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8AwRlV4ZopNPoVj3j0cBs+sd2hE=', 'Content-Length': '0'} 2022-09-02T05:10:08.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dd1e367060ad79a-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.761 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.762 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.763 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.763 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:08.764 DEBUG:boto:Data: 2022-09-02T05:10:08.765 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.766 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.766 DEBUG:boto:Params: {} 2022-09-02T05:10:08.767 DEBUG:boto:Token: None 2022-09-02T05:10:08.768 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4 2022-09-02T05:10:08.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8AwRlV4ZopNPoVj3j0cBs+sd2hE= 2022-09-02T05:10:08.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8AwRlV4ZopNPoVj3j0cBs+sd2hE=', 'Content-Length': '0'} 2022-09-02T05:10:08.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8f38ac7f66e36e5-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:08.773 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.773 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.774 DEBUG:boto:Method: GET 2022-09-02T05:10:08.775 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:08.775 DEBUG:boto:Data: 2022-09-02T05:10:08.776 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.777 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.777 DEBUG:boto:Params: {} 2022-09-02T05:10:08.778 DEBUG:boto:Token: None 2022-09-02T05:10:08.779 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4 2022-09-02T05:10:08.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mn/2gGoacDMbwKVYsFaLyNKw2JA= 2022-09-02T05:10:08.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mn/2gGoacDMbwKVYsFaLyNKw2JA=', 'Content-Length': '0'} 2022-09-02T05:10:08.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d86ff7c7dc2df03-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.783 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:08.784 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:08.784 DEBUG:boto:Method: GET 2022-09-02T05:10:08.785 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:08.785 DEBUG:boto:Data: 2022-09-02T05:10:08.786 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.787 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.788 DEBUG:boto:Params: {} 2022-09-02T05:10:08.788 DEBUG:boto:Token: None 2022-09-02T05:10:08.789 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj4 2022-09-02T05:10:08.790 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mn/2gGoacDMbwKVYsFaLyNKw2JA= 2022-09-02T05:10:08.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mn/2gGoacDMbwKVYsFaLyNKw2JA=', 'Content-Length': '0'} 2022-09-02T05:10:08.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f03d7d3868dde464-0063119030-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.794 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.794 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.795 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.795 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.796 DEBUG:boto:Method: GET 2022-09-02T05:10:08.797 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.797 DEBUG:boto:Data: 2022-09-02T05:10:08.798 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.800 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.801 DEBUG:boto:Params: {} 2022-09-02T05:10:08.801 DEBUG:boto:Token: None 2022-09-02T05:10:08.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a40c24c92258328f-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.850 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.851 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.852 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.852 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.853 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.854 DEBUG:boto:Method: GET 2022-09-02T05:10:08.855 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:08.855 DEBUG:boto:Data: 2022-09-02T05:10:08.856 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:10:08.857 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.857 DEBUG:boto:Params: {} 2022-09-02T05:10:08.858 DEBUG:boto:Token: None 2022-09-02T05:10:08.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM= 2022-09-02T05:10:08.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QGsL7Crj+tabqQyPpQ5SroKjyEM=', 'Content-Length': '0'} 2022-09-02T05:10:08.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc97211479d0e75-0063119030-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.863 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-62 zones={test-zone3, test-zone1} 2022-09-02T05:10:08.865 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-62 zones={test-zone3, test-zone2} 2022-09-02T05:10:08.866 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.866 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.867 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.867 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:10:08.868 DEBUG:boto:Data: 2022-09-02T05:10:08.868 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.869 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.870 DEBUG:boto:Params: {} 2022-09-02T05:10:08.871 DEBUG:boto:Token: None 2022-09-02T05:10:08.871 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G8smoeXd73Jw2bQMDG/tjfc8I/s= 2022-09-02T05:10:08.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G8smoeXd73Jw2bQMDG/tjfc8I/s=', 'Content-Length': '0'} 2022-09-02T05:10:08.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000767b95437d1c71c9-0063119030-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.876 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.877 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.877 DEBUG:boto:Method: HEAD 2022-09-02T05:10:08.878 DEBUG:boto:Path: /xljbkf-62/ 2022-09-02T05:10:08.879 DEBUG:boto:Data: 2022-09-02T05:10:08.879 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:08.880 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.881 DEBUG:boto:Params: {} 2022-09-02T05:10:08.881 DEBUG:boto:Token: None 2022-09-02T05:10:08.882 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/ 2022-09-02T05:10:08.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G8smoeXd73Jw2bQMDG/tjfc8I/s= 2022-09-02T05:10:08.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G8smoeXd73Jw2bQMDG/tjfc8I/s=', 'Content-Length': '0'} 2022-09-02T05:10:08.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c7ca8886648ec9a-0063119030-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:10:08.887 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.888 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.889 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.889 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.890 DEBUG:boto:Method: GET 2022-09-02T05:10:08.890 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.891 DEBUG:boto:Data: 2022-09-02T05:10:08.891 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.892 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.893 DEBUG:boto:Params: {} 2022-09-02T05:10:08.894 DEBUG:boto:Token: None 2022-09-02T05:10:08.894 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/?versions 2022-09-02T05:10:08.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw= 2022-09-02T05:10:08.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw=', 'Content-Length': '0'} 2022-09-02T05:10:08.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000301fccd82f047318-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.899 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:10:08.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:10:08.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:10:08.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:10:08.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:10:08.903 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.904 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.904 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.905 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.905 DEBUG:boto:Method: GET 2022-09-02T05:10:08.906 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.906 DEBUG:boto:Data: 2022-09-02T05:10:08.907 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:08.908 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.908 DEBUG:boto:Params: {} 2022-09-02T05:10:08.909 DEBUG:boto:Token: None 2022-09-02T05:10:08.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/?versions 2022-09-02T05:10:08.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw= 2022-09-02T05:10:08.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw=', 'Content-Length': '0'} 2022-09-02T05:10:08.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bbb6dd12d60b3a0-0063119030-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.915 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2022-09-02T05:10:08.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2022-09-02T05:10:08.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2022-09-02T05:10:08.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2022-09-02T05:10:08.918 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.919 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.919 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.920 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.920 DEBUG:boto:Method: GET 2022-09-02T05:10:08.921 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.921 DEBUG:boto:Data: 2022-09-02T05:10:08.922 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.923 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.924 DEBUG:boto:Params: {} 2022-09-02T05:10:08.924 DEBUG:boto:Token: None 2022-09-02T05:10:08.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/?versions 2022-09-02T05:10:08.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw= 2022-09-02T05:10:08.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw=', 'Content-Length': '0'} 2022-09-02T05:10:08.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003601129a28d52ada-0063119030-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.930 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.931 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:08.931 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:08.932 DEBUG:boto:path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.932 DEBUG:boto:auth_path=/xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.933 DEBUG:boto:Method: GET 2022-09-02T05:10:08.934 DEBUG:boto:Path: /xljbkf-62/?versions&max-keys=999 2022-09-02T05:10:08.934 DEBUG:boto:Data: 2022-09-02T05:10:08.935 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:08.936 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.937 DEBUG:boto:Params: {} 2022-09-02T05:10:08.938 DEBUG:boto:Token: None 2022-09-02T05:10:08.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/?versions 2022-09-02T05:10:08.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw= 2022-09-02T05:10:08.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lp2vvyHdJh8s0Keiu3TRE15XdHw=', 'Content-Length': '0'} 2022-09-02T05:10:08.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fe5f6a105c94f0-0063119030-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.944 DEBUG:boto:b'xljbkf-62999falseobj1nulltrue2022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:08.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:08.945 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:08.946 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:08.947 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.947 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.948 DEBUG:boto:Method: GET 2022-09-02T05:10:08.948 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.949 DEBUG:boto:Data: 2022-09-02T05:10:08.949 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:08.951 DEBUG:boto:Port: 8001 2022-09-02T05:10:08.951 DEBUG:boto:Params: {} 2022-09-02T05:10:08.952 DEBUG:boto:Token: None 2022-09-02T05:10:08.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+8ztcyOhTDqnxlJPcwpO4p6Igw= 2022-09-02T05:10:08.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+8ztcyOhTDqnxlJPcwpO4p6Igw=', 'Content-Length': '0'} 2022-09-02T05:10:08.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216e187e005c3402-0063119030-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:08.957 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:08.958 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:08.959 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.960 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.961 DEBUG:boto:Method: GET 2022-09-02T05:10:08.962 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.963 DEBUG:boto:Data: 2022-09-02T05:10:08.964 DEBUG:boto:Headers: {} 2022-09-02T05:10:08.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:08.966 DEBUG:boto:Port: 8000 2022-09-02T05:10:08.967 DEBUG:boto:Params: {} 2022-09-02T05:10:08.968 DEBUG:boto:Token: None 2022-09-02T05:10:08.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:08 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:08.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+8ztcyOhTDqnxlJPcwpO4p6Igw= 2022-09-02T05:10:08.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+8ztcyOhTDqnxlJPcwpO4p6Igw=', 'Content-Length': '0'} 2022-09-02T05:10:08.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000939edb0e760620a7-0063119030-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.018 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.019 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.019 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.020 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.020 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.021 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.021 DEBUG:boto:Data: 2022-09-02T05:10:09.022 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.023 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.024 DEBUG:boto:Params: {} 2022-09-02T05:10:09.024 DEBUG:boto:Token: None 2022-09-02T05:10:09.025 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ykmB/xt/D2cFG3ZUS6p2teWmvjc= 2022-09-02T05:10:09.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ykmB/xt/D2cFG3ZUS6p2teWmvjc=', 'Content-Length': '0'} 2022-09-02T05:10:09.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c39e33db00476772-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.031 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.032 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.033 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.033 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.034 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.034 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.035 DEBUG:boto:Data: 2022-09-02T05:10:09.036 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.037 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.037 DEBUG:boto:Params: {} 2022-09-02T05:10:09.038 DEBUG:boto:Token: None 2022-09-02T05:10:09.039 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ykmB/xt/D2cFG3ZUS6p2teWmvjc= 2022-09-02T05:10:09.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ykmB/xt/D2cFG3ZUS6p2teWmvjc=', 'Content-Length': '0'} 2022-09-02T05:10:09.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006396fc480cbf7335-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:09.045 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.046 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.047 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.047 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.048 DEBUG:boto:Method: GET 2022-09-02T05:10:09.048 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.049 DEBUG:boto:Data: 2022-09-02T05:10:09.050 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.051 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.052 DEBUG:boto:Params: {} 2022-09-02T05:10:09.053 DEBUG:boto:Token: None 2022-09-02T05:10:09.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5Dcab085W01T/rqkKLOYvVpbAuM= 2022-09-02T05:10:09.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5Dcab085W01T/rqkKLOYvVpbAuM=', 'Content-Length': '0'} 2022-09-02T05:10:09.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7fcb44bff51cbe1-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.061 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.061 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.062 DEBUG:boto:path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.063 DEBUG:boto:auth_path=/xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.064 DEBUG:boto:Method: GET 2022-09-02T05:10:09.064 DEBUG:boto:Path: /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.065 DEBUG:boto:Data: 2022-09-02T05:10:09.066 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.067 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.068 DEBUG:boto:Params: {} 2022-09-02T05:10:09.069 DEBUG:boto:Token: None 2022-09-02T05:10:09.069 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1?versionId=null 2022-09-02T05:10:09.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5Dcab085W01T/rqkKLOYvVpbAuM= 2022-09-02T05:10:09.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5Dcab085W01T/rqkKLOYvVpbAuM=', 'Content-Length': '0'} 2022-09-02T05:10:09.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fbf7067a3de2577-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.074 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.075 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.075 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.076 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.076 DEBUG:boto:Method: GET 2022-09-02T05:10:09.077 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.078 DEBUG:boto:Data: 2022-09-02T05:10:09.078 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.079 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.081 DEBUG:boto:Params: {} 2022-09-02T05:10:09.082 DEBUG:boto:Token: None 2022-09-02T05:10:09.083 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.084 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ece54b0dcfb3a12d-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.090 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.091 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.092 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.093 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.093 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.094 DEBUG:boto:Method: GET 2022-09-02T05:10:09.094 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.095 DEBUG:boto:Data: 2022-09-02T05:10:09.096 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.097 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.097 DEBUG:boto:Params: {} 2022-09-02T05:10:09.098 DEBUG:boto:Token: None 2022-09-02T05:10:09.098 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000897a512f54e8dd24-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.103 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.104 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.104 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.105 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.106 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:09.106 DEBUG:boto:Data: 2022-09-02T05:10:09.107 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.109 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.109 DEBUG:boto:Params: {} 2022-09-02T05:10:09.110 DEBUG:boto:Token: None 2022-09-02T05:10:09.112 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1 2022-09-02T05:10:09.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g5H0E1XHD8QnDGMkD3LGlqjPHEc= 2022-09-02T05:10:09.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g5H0E1XHD8QnDGMkD3LGlqjPHEc=', 'Content-Length': '0'} 2022-09-02T05:10:09.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b52c3b22010f4b7-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.118 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.118 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.119 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.119 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:09.120 DEBUG:boto:Data: 2022-09-02T05:10:09.120 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.121 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.122 DEBUG:boto:Params: {} 2022-09-02T05:10:09.122 DEBUG:boto:Token: None 2022-09-02T05:10:09.123 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1 2022-09-02T05:10:09.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g5H0E1XHD8QnDGMkD3LGlqjPHEc= 2022-09-02T05:10:09.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g5H0E1XHD8QnDGMkD3LGlqjPHEc=', 'Content-Length': '0'} 2022-09-02T05:10:09.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a0978bfff88dc82-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2022-09-02T05:10:09.128 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.128 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.129 DEBUG:boto:Method: GET 2022-09-02T05:10:09.129 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:09.130 DEBUG:boto:Data: 2022-09-02T05:10:09.131 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.132 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.132 DEBUG:boto:Params: {} 2022-09-02T05:10:09.133 DEBUG:boto:Token: None 2022-09-02T05:10:09.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1 2022-09-02T05:10:09.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pMvDZ0c4Uq7byUs5/fKkThwCCxw= 2022-09-02T05:10:09.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pMvDZ0c4Uq7byUs5/fKkThwCCxw=', 'Content-Length': '0'} 2022-09-02T05:10:09.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b634a31c9ab183d2-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.138 DEBUG:boto:path=/xljbkf-62/obj1 2022-09-02T05:10:09.139 DEBUG:boto:auth_path=/xljbkf-62/obj1 2022-09-02T05:10:09.140 DEBUG:boto:Method: GET 2022-09-02T05:10:09.141 DEBUG:boto:Path: /xljbkf-62/obj1 2022-09-02T05:10:09.142 DEBUG:boto:Data: 2022-09-02T05:10:09.143 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.145 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.146 DEBUG:boto:Params: {} 2022-09-02T05:10:09.147 DEBUG:boto:Token: None 2022-09-02T05:10:09.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj1 2022-09-02T05:10:09.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pMvDZ0c4Uq7byUs5/fKkThwCCxw= 2022-09-02T05:10:09.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pMvDZ0c4Uq7byUs5/fKkThwCCxw=', 'Content-Length': '0'} 2022-09-02T05:10:09.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e6b4a2a31f790a3-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.194 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.195 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.196 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.196 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.197 DEBUG:boto:Method: GET 2022-09-02T05:10:09.198 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.198 DEBUG:boto:Data: 2022-09-02T05:10:09.199 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.200 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.200 DEBUG:boto:Params: {} 2022-09-02T05:10:09.201 DEBUG:boto:Token: None 2022-09-02T05:10:09.202 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004072eafd6f79302-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.208 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.210 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.210 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.211 DEBUG:boto:path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.212 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.213 DEBUG:boto:Method: GET 2022-09-02T05:10:09.213 DEBUG:boto:Path: /xljbkf-62/?prefix=obj1 2022-09-02T05:10:09.214 DEBUG:boto:Data: 2022-09-02T05:10:09.214 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.215 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.216 DEBUG:boto:Params: {} 2022-09-02T05:10:09.217 DEBUG:boto:Token: None 2022-09-02T05:10:09.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070245fb5877f463c-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.222 DEBUG:boto:b'xljbkf-62obj11000falseobj12022-09-02T05:08:59.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:09.223 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.224 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.225 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.225 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.226 DEBUG:boto:Method: GET 2022-09-02T05:10:09.226 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.227 DEBUG:boto:Data: 2022-09-02T05:10:09.228 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.229 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.229 DEBUG:boto:Params: {} 2022-09-02T05:10:09.230 DEBUG:boto:Token: None 2022-09-02T05:10:09.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4= 2022-09-02T05:10:09.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4=', 'Content-Length': '0'} 2022-09-02T05:10:09.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ae7a20caee2908a-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.282 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.283 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.283 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.284 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.284 DEBUG:boto:Method: GET 2022-09-02T05:10:09.285 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.286 DEBUG:boto:Data: 2022-09-02T05:10:09.286 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.288 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.289 DEBUG:boto:Params: {} 2022-09-02T05:10:09.289 DEBUG:boto:Token: None 2022-09-02T05:10:09.290 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4= 2022-09-02T05:10:09.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4=', 'Content-Length': '0'} 2022-09-02T05:10:09.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff394f031a037971-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.295 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.296 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.297 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.298 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.298 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.299 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.300 DEBUG:boto:Data: 2022-09-02T05:10:09.301 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.302 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.302 DEBUG:boto:Params: {} 2022-09-02T05:10:09.303 DEBUG:boto:Token: None 2022-09-02T05:10:09.304 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/Rn2g/GGWH6+6vbxU+r1BdGd7Y= 2022-09-02T05:10:09.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/Rn2g/GGWH6+6vbxU+r1BdGd7Y=', 'Content-Length': '0'} 2022-09-02T05:10:09.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc6da73bb4492be0-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.308 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.309 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.309 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.310 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.311 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.311 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.312 DEBUG:boto:Data: 2022-09-02T05:10:09.313 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.314 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.315 DEBUG:boto:Params: {} 2022-09-02T05:10:09.316 DEBUG:boto:Token: None 2022-09-02T05:10:09.317 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d/Rn2g/GGWH6+6vbxU+r1BdGd7Y= 2022-09-02T05:10:09.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d/Rn2g/GGWH6+6vbxU+r1BdGd7Y=', 'Content-Length': '0'} 2022-09-02T05:10:09.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d1eb65a42317597-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:09.322 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.323 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.324 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.324 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.325 DEBUG:boto:Method: GET 2022-09-02T05:10:09.326 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.326 DEBUG:boto:Data: 2022-09-02T05:10:09.327 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.328 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.328 DEBUG:boto:Params: {} 2022-09-02T05:10:09.329 DEBUG:boto:Token: None 2022-09-02T05:10:09.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4= 2022-09-02T05:10:09.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4=', 'Content-Length': '0'} 2022-09-02T05:10:09.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9945c61102b2295-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.334 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.335 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.336 DEBUG:boto:path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.336 DEBUG:boto:auth_path=/xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.337 DEBUG:boto:Method: GET 2022-09-02T05:10:09.337 DEBUG:boto:Path: /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.338 DEBUG:boto:Data: 2022-09-02T05:10:09.339 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.341 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.342 DEBUG:boto:Params: {} 2022-09-02T05:10:09.343 DEBUG:boto:Token: None 2022-09-02T05:10:09.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2?versionId=null 2022-09-02T05:10:09.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4= 2022-09-02T05:10:09.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WH2eGXKmdHyuScTdo2UGFdyahj4=', 'Content-Length': '0'} 2022-09-02T05:10:09.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e37a158d3a6e806-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.348 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.349 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.349 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.350 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.351 DEBUG:boto:Method: GET 2022-09-02T05:10:09.351 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.352 DEBUG:boto:Data: 2022-09-02T05:10:09.352 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.353 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.354 DEBUG:boto:Params: {} 2022-09-02T05:10:09.354 DEBUG:boto:Token: None 2022-09-02T05:10:09.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8e0b4baed737ed-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.359 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.360 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.361 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.361 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.362 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.362 DEBUG:boto:Method: GET 2022-09-02T05:10:09.363 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.363 DEBUG:boto:Data: 2022-09-02T05:10:09.364 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.365 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.366 DEBUG:boto:Params: {} 2022-09-02T05:10:09.366 DEBUG:boto:Token: None 2022-09-02T05:10:09.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000258af73876ae09a3-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.372 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.373 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.373 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.374 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.374 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:09.375 DEBUG:boto:Data: 2022-09-02T05:10:09.375 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.376 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.377 DEBUG:boto:Params: {} 2022-09-02T05:10:09.377 DEBUG:boto:Token: None 2022-09-02T05:10:09.378 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2 2022-09-02T05:10:09.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bsuHvUrkjKzt0baPoaGTpfYPyM4= 2022-09-02T05:10:09.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bsuHvUrkjKzt0baPoaGTpfYPyM4=', 'Content-Length': '0'} 2022-09-02T05:10:09.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001edd854147396abe-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.382 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.382 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.383 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.383 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:09.384 DEBUG:boto:Data: 2022-09-02T05:10:09.384 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.385 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.386 DEBUG:boto:Params: {} 2022-09-02T05:10:09.387 DEBUG:boto:Token: None 2022-09-02T05:10:09.387 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2 2022-09-02T05:10:09.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bsuHvUrkjKzt0baPoaGTpfYPyM4= 2022-09-02T05:10:09.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bsuHvUrkjKzt0baPoaGTpfYPyM4=', 'Content-Length': '0'} 2022-09-02T05:10:09.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb3411c442277d83-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2022-09-02T05:10:09.392 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.393 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.393 DEBUG:boto:Method: GET 2022-09-02T05:10:09.394 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:09.394 DEBUG:boto:Data: 2022-09-02T05:10:09.395 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.396 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.396 DEBUG:boto:Params: {} 2022-09-02T05:10:09.397 DEBUG:boto:Token: None 2022-09-02T05:10:09.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2 2022-09-02T05:10:09.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zYwxVR23ZjlhoxHZ0fritzIT+cM= 2022-09-02T05:10:09.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zYwxVR23ZjlhoxHZ0fritzIT+cM=', 'Content-Length': '0'} 2022-09-02T05:10:09.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a492d0f8b37582f-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.402 DEBUG:boto:path=/xljbkf-62/obj2 2022-09-02T05:10:09.402 DEBUG:boto:auth_path=/xljbkf-62/obj2 2022-09-02T05:10:09.403 DEBUG:boto:Method: GET 2022-09-02T05:10:09.404 DEBUG:boto:Path: /xljbkf-62/obj2 2022-09-02T05:10:09.404 DEBUG:boto:Data: 2022-09-02T05:10:09.405 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.406 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.406 DEBUG:boto:Params: {} 2022-09-02T05:10:09.407 DEBUG:boto:Token: None 2022-09-02T05:10:09.407 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj2 2022-09-02T05:10:09.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zYwxVR23ZjlhoxHZ0fritzIT+cM= 2022-09-02T05:10:09.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zYwxVR23ZjlhoxHZ0fritzIT+cM=', 'Content-Length': '0'} 2022-09-02T05:10:09.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002728e9524139132a-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.458 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.459 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.460 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.461 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.462 DEBUG:boto:Method: GET 2022-09-02T05:10:09.462 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.463 DEBUG:boto:Data: 2022-09-02T05:10:09.464 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.466 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.467 DEBUG:boto:Params: {} 2022-09-02T05:10:09.469 DEBUG:boto:Token: None 2022-09-02T05:10:09.470 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd979211dd99aea-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.476 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.477 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.478 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.478 DEBUG:boto:path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.479 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.479 DEBUG:boto:Method: GET 2022-09-02T05:10:09.480 DEBUG:boto:Path: /xljbkf-62/?prefix=obj2 2022-09-02T05:10:09.481 DEBUG:boto:Data: 2022-09-02T05:10:09.481 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.483 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.483 DEBUG:boto:Params: {} 2022-09-02T05:10:09.484 DEBUG:boto:Token: None 2022-09-02T05:10:09.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083dfa2f38f0a07b2-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.490 DEBUG:boto:b'xljbkf-62obj21000falseobj22022-09-02T05:08:59.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:09.491 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.492 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.492 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.493 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.494 DEBUG:boto:Method: GET 2022-09-02T05:10:09.495 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.496 DEBUG:boto:Data: 2022-09-02T05:10:09.497 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.499 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.500 DEBUG:boto:Params: {} 2022-09-02T05:10:09.501 DEBUG:boto:Token: None 2022-09-02T05:10:09.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY= 2022-09-02T05:10:09.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY=', 'Content-Length': '0'} 2022-09-02T05:10:09.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b12b6bf26ee9857-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.546 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.547 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.548 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.548 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.549 DEBUG:boto:Method: GET 2022-09-02T05:10:09.550 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.550 DEBUG:boto:Data: 2022-09-02T05:10:09.551 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.552 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.553 DEBUG:boto:Params: {} 2022-09-02T05:10:09.555 DEBUG:boto:Token: None 2022-09-02T05:10:09.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY= 2022-09-02T05:10:09.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY=', 'Content-Length': '0'} 2022-09-02T05:10:09.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033992360c75f9543-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.563 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.564 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.564 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.565 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.565 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.566 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.567 DEBUG:boto:Data: 2022-09-02T05:10:09.567 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.568 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.569 DEBUG:boto:Params: {} 2022-09-02T05:10:09.570 DEBUG:boto:Token: None 2022-09-02T05:10:09.571 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.571 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lD997BVlk8DJlHJY1QaUx0B73PM= 2022-09-02T05:10:09.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lD997BVlk8DJlHJY1QaUx0B73PM=', 'Content-Length': '0'} 2022-09-02T05:10:09.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f75cdf1607184d2-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.575 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.575 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.576 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.576 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.577 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.578 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.579 DEBUG:boto:Data: 2022-09-02T05:10:09.579 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.581 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.581 DEBUG:boto:Params: {} 2022-09-02T05:10:09.582 DEBUG:boto:Token: None 2022-09-02T05:10:09.583 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lD997BVlk8DJlHJY1QaUx0B73PM= 2022-09-02T05:10:09.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lD997BVlk8DJlHJY1QaUx0B73PM=', 'Content-Length': '0'} 2022-09-02T05:10:09.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8c285901296aa0a-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:09.589 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.590 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.591 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.592 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.593 DEBUG:boto:Method: GET 2022-09-02T05:10:09.594 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.595 DEBUG:boto:Data: 2022-09-02T05:10:09.596 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.598 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.599 DEBUG:boto:Params: {} 2022-09-02T05:10:09.600 DEBUG:boto:Token: None 2022-09-02T05:10:09.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY= 2022-09-02T05:10:09.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY=', 'Content-Length': '0'} 2022-09-02T05:10:09.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000638e456a94a73ea5-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.608 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.609 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.610 DEBUG:boto:path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.610 DEBUG:boto:auth_path=/xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.611 DEBUG:boto:Method: GET 2022-09-02T05:10:09.611 DEBUG:boto:Path: /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.612 DEBUG:boto:Data: 2022-09-02T05:10:09.613 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.615 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.615 DEBUG:boto:Params: {} 2022-09-02T05:10:09.616 DEBUG:boto:Token: None 2022-09-02T05:10:09.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3?versionId=null 2022-09-02T05:10:09.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY= 2022-09-02T05:10:09.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ohVf/uNOGbM7ms4kSheorH3HCY=', 'Content-Length': '0'} 2022-09-02T05:10:09.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000223cdaa7897b9155-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.621 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.622 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.623 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.624 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.624 DEBUG:boto:Method: GET 2022-09-02T05:10:09.625 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.625 DEBUG:boto:Data: 2022-09-02T05:10:09.626 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.627 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.628 DEBUG:boto:Params: {} 2022-09-02T05:10:09.628 DEBUG:boto:Token: None 2022-09-02T05:10:09.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5394b5d94245ec9-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.635 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.636 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.637 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.637 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.638 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.638 DEBUG:boto:Method: GET 2022-09-02T05:10:09.639 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.639 DEBUG:boto:Data: 2022-09-02T05:10:09.640 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.641 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.641 DEBUG:boto:Params: {} 2022-09-02T05:10:09.642 DEBUG:boto:Token: None 2022-09-02T05:10:09.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b90eecda21adca9-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.648 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.649 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.649 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.650 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.650 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:09.651 DEBUG:boto:Data: 2022-09-02T05:10:09.651 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.652 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.653 DEBUG:boto:Params: {} 2022-09-02T05:10:09.653 DEBUG:boto:Token: None 2022-09-02T05:10:09.654 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3 2022-09-02T05:10:09.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nzHrzhCR65ZZQcfMH8hxEL6Q+uk= 2022-09-02T05:10:09.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nzHrzhCR65ZZQcfMH8hxEL6Q+uk=', 'Content-Length': '0'} 2022-09-02T05:10:09.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7d7f144c4e528c6-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.658 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.659 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.659 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.660 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:09.660 DEBUG:boto:Data: 2022-09-02T05:10:09.661 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.662 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.662 DEBUG:boto:Params: {} 2022-09-02T05:10:09.663 DEBUG:boto:Token: None 2022-09-02T05:10:09.664 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3 2022-09-02T05:10:09.664 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nzHrzhCR65ZZQcfMH8hxEL6Q+uk= 2022-09-02T05:10:09.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nzHrzhCR65ZZQcfMH8hxEL6Q+uk=', 'Content-Length': '0'} 2022-09-02T05:10:09.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000428021364a09866a-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2022-09-02T05:10:09.668 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.669 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.669 DEBUG:boto:Method: GET 2022-09-02T05:10:09.670 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:09.671 DEBUG:boto:Data: 2022-09-02T05:10:09.671 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.672 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.673 DEBUG:boto:Params: {} 2022-09-02T05:10:09.673 DEBUG:boto:Token: None 2022-09-02T05:10:09.674 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3 2022-09-02T05:10:09.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y0FsdaDjSRNIwF7vC0OLUCrx7RI= 2022-09-02T05:10:09.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y0FsdaDjSRNIwF7vC0OLUCrx7RI=', 'Content-Length': '0'} 2022-09-02T05:10:09.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2f23e098fd451f-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.722 DEBUG:boto:path=/xljbkf-62/obj3 2022-09-02T05:10:09.723 DEBUG:boto:auth_path=/xljbkf-62/obj3 2022-09-02T05:10:09.723 DEBUG:boto:Method: GET 2022-09-02T05:10:09.724 DEBUG:boto:Path: /xljbkf-62/obj3 2022-09-02T05:10:09.725 DEBUG:boto:Data: 2022-09-02T05:10:09.725 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.727 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.728 DEBUG:boto:Params: {} 2022-09-02T05:10:09.728 DEBUG:boto:Token: None 2022-09-02T05:10:09.729 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj3 2022-09-02T05:10:09.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y0FsdaDjSRNIwF7vC0OLUCrx7RI= 2022-09-02T05:10:09.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y0FsdaDjSRNIwF7vC0OLUCrx7RI=', 'Content-Length': '0'} 2022-09-02T05:10:09.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a880369651014362-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.734 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.735 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.736 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.736 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.737 DEBUG:boto:Method: GET 2022-09-02T05:10:09.738 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.739 DEBUG:boto:Data: 2022-09-02T05:10:09.740 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.742 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.742 DEBUG:boto:Params: {} 2022-09-02T05:10:09.743 DEBUG:boto:Token: None 2022-09-02T05:10:09.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017da6cebaae265e-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.750 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.751 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.752 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.752 DEBUG:boto:path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.753 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.754 DEBUG:boto:Method: GET 2022-09-02T05:10:09.754 DEBUG:boto:Path: /xljbkf-62/?prefix=obj3 2022-09-02T05:10:09.755 DEBUG:boto:Data: 2022-09-02T05:10:09.755 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.758 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.758 DEBUG:boto:Params: {} 2022-09-02T05:10:09.759 DEBUG:boto:Token: None 2022-09-02T05:10:09.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260eb29405a2051d-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.765 DEBUG:boto:b'xljbkf-62obj31000falseobj32022-09-02T05:08:59.745Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:09.767 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.768 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.768 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.769 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.770 DEBUG:boto:Method: GET 2022-09-02T05:10:09.770 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.771 DEBUG:boto:Data: 2022-09-02T05:10:09.771 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.772 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.773 DEBUG:boto:Params: {} 2022-09-02T05:10:09.773 DEBUG:boto:Token: None 2022-09-02T05:10:09.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc= 2022-09-02T05:10:09.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc=', 'Content-Length': '0'} 2022-09-02T05:10:09.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcb4f51aa5be798f-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.785 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.785 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.786 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.786 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.787 DEBUG:boto:Method: GET 2022-09-02T05:10:09.787 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.788 DEBUG:boto:Data: 2022-09-02T05:10:09.788 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.789 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.789 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.790 DEBUG:boto:Params: {} 2022-09-02T05:10:09.791 DEBUG:boto:Token: None 2022-09-02T05:10:09.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc= 2022-09-02T05:10:09.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc=', 'Content-Length': '0'} 2022-09-02T05:10:09.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5fa9f06f9d5e896-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.798 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.798 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.799 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.799 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.800 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.800 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.801 DEBUG:boto:Data: 2022-09-02T05:10:09.801 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.803 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.803 DEBUG:boto:Params: {} 2022-09-02T05:10:09.804 DEBUG:boto:Token: None 2022-09-02T05:10:09.804 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C8nLj2+CyT3J4Zy6vxdq8/3d6ns= 2022-09-02T05:10:09.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C8nLj2+CyT3J4Zy6vxdq8/3d6ns=', 'Content-Length': '0'} 2022-09-02T05:10:09.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3861fd34711d718-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.809 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.810 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.811 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.811 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.812 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.812 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.813 DEBUG:boto:Data: 2022-09-02T05:10:09.813 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.814 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.815 DEBUG:boto:Params: {} 2022-09-02T05:10:09.816 DEBUG:boto:Token: None 2022-09-02T05:10:09.816 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.817 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C8nLj2+CyT3J4Zy6vxdq8/3d6ns= 2022-09-02T05:10:09.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C8nLj2+CyT3J4Zy6vxdq8/3d6ns=', 'Content-Length': '0'} 2022-09-02T05:10:09.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073d8dae774b57c7d-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:09.821 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.822 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.822 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.823 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.823 DEBUG:boto:Method: GET 2022-09-02T05:10:09.824 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.825 DEBUG:boto:Data: 2022-09-02T05:10:09.825 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.827 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.827 DEBUG:boto:Params: {} 2022-09-02T05:10:09.828 DEBUG:boto:Token: None 2022-09-02T05:10:09.829 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc= 2022-09-02T05:10:09.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc=', 'Content-Length': '0'} 2022-09-02T05:10:09.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9ef7476c7d35b4f-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.874 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.875 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.876 DEBUG:boto:path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.876 DEBUG:boto:auth_path=/xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.877 DEBUG:boto:Method: GET 2022-09-02T05:10:09.877 DEBUG:boto:Path: /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.878 DEBUG:boto:Data: 2022-09-02T05:10:09.878 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.879 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.880 DEBUG:boto:Params: {} 2022-09-02T05:10:09.880 DEBUG:boto:Token: None 2022-09-02T05:10:09.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4?versionId=null 2022-09-02T05:10:09.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc= 2022-09-02T05:10:09.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZQA3n+ISLm3P72t0o6pLKZ7dELc=', 'Content-Length': '0'} 2022-09-02T05:10:09.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010ea2408b41f16f4-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.887 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.888 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.889 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.889 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.890 DEBUG:boto:Method: GET 2022-09-02T05:10:09.892 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.893 DEBUG:boto:Data: 2022-09-02T05:10:09.894 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.896 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.897 DEBUG:boto:Params: {} 2022-09-02T05:10:09.898 DEBUG:boto:Token: None 2022-09-02T05:10:09.899 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d95cd42d57b373-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.906 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.907 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.908 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.909 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.910 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.911 DEBUG:boto:Method: GET 2022-09-02T05:10:09.912 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.913 DEBUG:boto:Data: 2022-09-02T05:10:09.914 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.916 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.917 DEBUG:boto:Params: {} 2022-09-02T05:10:09.918 DEBUG:boto:Token: None 2022-09-02T05:10:09.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73d7b164dc48f17-0063119031-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.926 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.927 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.928 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.929 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.930 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:09.931 DEBUG:boto:Data: 2022-09-02T05:10:09.932 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.934 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.935 DEBUG:boto:Params: {} 2022-09-02T05:10:09.936 DEBUG:boto:Token: None 2022-09-02T05:10:09.937 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4 2022-09-02T05:10:09.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gPDwORkWake2lNbUbI6JoPSsXc= 2022-09-02T05:10:09.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gPDwORkWake2lNbUbI6JoPSsXc=', 'Content-Length': '0'} 2022-09-02T05:10:09.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e46d8fc89842e5e6-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.943 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.944 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.944 DEBUG:boto:Method: HEAD 2022-09-02T05:10:09.945 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:09.945 DEBUG:boto:Data: 2022-09-02T05:10:09.946 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.947 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.947 DEBUG:boto:Params: {} 2022-09-02T05:10:09.948 DEBUG:boto:Token: None 2022-09-02T05:10:09.949 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4 2022-09-02T05:10:09.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gPDwORkWake2lNbUbI6JoPSsXc= 2022-09-02T05:10:09.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gPDwORkWake2lNbUbI6JoPSsXc=', 'Content-Length': '0'} 2022-09-02T05:10:09.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da36fc0136dd0d41-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2022-09-02T05:10:09.953 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.954 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.954 DEBUG:boto:Method: GET 2022-09-02T05:10:09.955 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:09.955 DEBUG:boto:Data: 2022-09-02T05:10:09.956 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.958 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.959 DEBUG:boto:Params: {} 2022-09-02T05:10:09.959 DEBUG:boto:Token: None 2022-09-02T05:10:09.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4 2022-09-02T05:10:09.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8NPaRxoeAB4bEavtBJRLw6wPSL4= 2022-09-02T05:10:09.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8NPaRxoeAB4bEavtBJRLw6wPSL4=', 'Content-Length': '0'} 2022-09-02T05:10:09.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c96f8f6850c86ea-0063119031-1599-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.965 DEBUG:boto:path=/xljbkf-62/obj4 2022-09-02T05:10:09.966 DEBUG:boto:auth_path=/xljbkf-62/obj4 2022-09-02T05:10:09.967 DEBUG:boto:Method: GET 2022-09-02T05:10:09.967 DEBUG:boto:Path: /xljbkf-62/obj4 2022-09-02T05:10:09.968 DEBUG:boto:Data: 2022-09-02T05:10:09.969 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:09.970 DEBUG:boto:Port: 8000 2022-09-02T05:10:09.971 DEBUG:boto:Params: {} 2022-09-02T05:10:09.972 DEBUG:boto:Token: None 2022-09-02T05:10:09.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/obj4 2022-09-02T05:10:09.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8NPaRxoeAB4bEavtBJRLw6wPSL4= 2022-09-02T05:10:09.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8NPaRxoeAB4bEavtBJRLw6wPSL4=', 'Content-Length': '0'} 2022-09-02T05:10:09.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:08:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d06dcff3f12b5ed0-0063119031-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.980 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:09.981 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:09.981 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.982 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.983 DEBUG:boto:Method: GET 2022-09-02T05:10:09.984 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:09.985 DEBUG:boto:Data: 2022-09-02T05:10:09.986 DEBUG:boto:Headers: {} 2022-09-02T05:10:09.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:10:09.988 DEBUG:boto:Port: 8001 2022-09-02T05:10:09.989 DEBUG:boto:Params: {} 2022-09-02T05:10:09.990 DEBUG:boto:Token: None 2022-09-02T05:10:09.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:09 GMT /xljbkf-62/ 2022-09-02T05:10:09.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs= 2022-09-02T05:10:09.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRskM4IoZDuoGPhZQ+jyHp3+FQs=', 'Content-Length': '0'} 2022-09-02T05:10:09.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2265dc4a11672d1-0063119031-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:09.998 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:09.999 DEBUG:boto:path=/xljbkf-62/ 2022-09-02T05:10:10.000 DEBUG:boto:auth_path=/xljbkf-62/ 2022-09-02T05:10:10.001 DEBUG:boto:path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:10.002 DEBUG:boto:auth_path=/xljbkf-62/?prefix=obj4 2022-09-02T05:10:10.003 DEBUG:boto:Method: GET 2022-09-02T05:10:10.004 DEBUG:boto:Path: /xljbkf-62/?prefix=obj4 2022-09-02T05:10:10.004 DEBUG:boto:Data: 2022-09-02T05:10:10.005 DEBUG:boto:Headers: {} 2022-09-02T05:10:10.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:10:10.006 DEBUG:boto:Port: 8000 2022-09-02T05:10:10.007 DEBUG:boto:Params: {} 2022-09-02T05:10:10.007 DEBUG:boto:Token: None 2022-09-02T05:10:10.008 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:10:10 GMT /xljbkf-62/ 2022-09-02T05:10:10.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EQnnq13lxjlNZN3iYl8CT0CgV4s= 2022-09-02T05:10:10.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:10:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EQnnq13lxjlNZN3iYl8CT0CgV4s=', 'Content-Length': '0'} 2022-09-02T05:10:10.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c473a511f6e94733-0063119032-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:10:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:10:10.058 DEBUG:boto:b'xljbkf-62obj41000falseobj42022-09-02T05:08:59.812Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:10:10.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-62 zones={test-zone3, test-zone2} 2022-09-02T05:10:10.060 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-63 2022-09-02T05:10:10.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:10.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:10.062 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:10.127 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:10.128 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:10.186 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:10.188+0000 7f4bf5621600 1 bucket sync caught up with source: 2022-09-02T05:10:10.186 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:10.187 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:10.187 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:10.188+0000 7f4bf5621600 0 bucket checkpoint complete 2022-09-02T05:10:10.209 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-63 2022-09-02T05:10:10.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:10.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:10.212 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:10.277 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:10.277 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:10.315 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:10.317+0000 7f736b710600 1 bucket sync caught up with source: 2022-09-02T05:10:10.316 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:10.316 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:10.316 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:10:10.317+0000 7f736b710600 0 bucket checkpoint complete 2022-09-02T05:10:10.338 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-63 2022-09-02T05:10:10.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:10.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:10:10.340 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:10:10.396 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:10:10.397 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:10:10.441 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:10.444+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:10.442 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:10.442 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:15.443 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:15.444+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:15.443 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:15.444 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:20.444 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:20.446+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:20.445 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:20.445 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:25.446 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:25.447+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:25.447 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:25.447 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:30.447 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:30.448+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:30.448 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:30.448 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:35.448 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:35.450+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:35.449 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:35.449 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:40.450 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:40.451+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:40.450 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:40.451 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:45.451 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:45.453+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:45.452 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:45.452 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:50.453 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:50.454+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:50.453 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:50.453 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:55.454 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:10:55.456+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:10:55.455 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:10:55.455 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:00.456 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:00.457+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:00.456 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:00.456 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:05.457 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:05.458+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:05.458 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:05.458 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:10.458 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:10.461+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:10.459 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:10.459 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:15.460 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:15.462+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:15.461 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:15.462 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:20.461 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:20.463+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:20.462 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:20.462 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:25.463 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:25.464+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:25.463 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:25.464 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:30.464 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:30.466+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:30.465 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:30.465 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:35.465 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:35.467+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:35.466 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:35.466 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:40.467 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:40.468+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:40.468 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:40.468 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:45.468 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:45.470+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:45.469 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:45.469 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:50.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:50.471+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:50.470 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:50.471 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:55.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:11:55.473+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:11:55.472 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:11:55.472 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:00.473 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:00.474+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:00.473 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:00.474 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:05.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:05.476+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:05.477 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:05.477 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:10.476 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:10.477+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:10.476 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:10.476 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:15.477 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:15.479+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:15.478 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:15.478 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:20.479 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:20.481+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:20.479 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:20.480 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:25.480 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:25.481+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:25.480 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:25.480 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:30.481 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:30.482+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:30.482 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:30.482 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:35.483 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:35.484+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:35.483 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:35.483 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:40.484 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:40.485+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:40.485 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:40.485 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:45.485 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:45.487+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:45.486 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:45.486 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:50.487 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:50.488+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:50.488 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:50.488 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:55.489 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:12:55.490+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:12:55.489 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:12:55.489 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:00.490 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:00.491+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:00.491 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:00.491 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:05.491 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:05.493+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:05.492 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:05.492 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:10.493 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:10.494+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:10.493 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:10.494 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:15.494 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:15.496+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:15.495 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:15.495 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:20.496 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:20.498+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:20.496 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:20.497 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:25.497 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:25.499+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:25.498 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:25.498 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:30.498 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:30.500+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:30.499 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:30.499 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:35.500 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:35.502+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:35.501 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:35.501 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:40.502 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:40.503+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:40.502 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:40.503 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:45.503 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:45.504+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:45.503 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:45.504 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:50.504 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:50.506+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:50.505 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:50.505 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:55.506 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:13:55.507+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:13:55.506 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:13:55.506 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:00.507 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:00.508+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:00.508 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:00.508 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:05.508 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:05.510+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:05.509 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:05.509 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:10.510 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:10.511+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:10.510 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:10.510 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:15.511 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:15.513+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:15.512 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:15.512 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:20.513 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:20.514+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:20.513 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:20.513 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:25.514 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:25.516+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:25.514 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:25.515 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:30.515 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:30.518+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:30.516 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:30.516 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:35.517 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:35.518+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:35.517 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:35.518 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:40.518 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:40.519+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:40.519 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:40.519 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:45.519 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:45.521+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:45.520 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:45.520 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:50.521 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:50.522+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:50.521 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:50.522 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:55.522 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:14:55.524+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:14:55.523 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:14:55.523 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:15:00.524 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:00.525+0000 7f50d2d70600 1 waiting for incremental sync to catch up: 2022-09-02T05:15:00.524 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, , , , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:15:00.524 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, , 00000000001.86.5, , 00000000001.80.5, , , 00000000001.78.5, , , ] 2022-09-02T05:15:05.525 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:05.527+0000 7f50d2d70600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T05:15:05.526 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:05.527+0000 7f50d2d70600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:15:05.526 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:05.527+0000 7f50d2d70600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:15:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T05:15:05.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2022-09-02T05:15:05.554 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-65 2022-09-02T05:15:05.555 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:05.556 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:05.557 DEBUG:boto:Method: PUT 2022-09-02T05:15:05.558 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:05.559 DEBUG:boto:Data: 2022-09-02T05:15:05.559 DEBUG:boto:Headers: {} 2022-09-02T05:15:05.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:05.560 DEBUG:boto:Port: 8000 2022-09-02T05:15:05.561 DEBUG:boto:Params: {} 2022-09-02T05:15:05.562 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:15:05.563 DEBUG:boto:Token: None 2022-09-02T05:15:05.564 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:15:05 GMT /xljbkf-65/ 2022-09-02T05:15:05.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H6p9D26TjB94BG1nejqUJeaOu6E= 2022-09-02T05:15:05.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H6p9D26TjB94BG1nejqUJeaOu6E=', 'Content-Length': '0'} 2022-09-02T05:15:05.594 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ad4f27599a189d1-0063119159-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:05.595 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xljbkf-66 2022-09-02T05:15:05.596 DEBUG:boto:path=/xljbkf-66/ 2022-09-02T05:15:05.596 DEBUG:boto:auth_path=/xljbkf-66/ 2022-09-02T05:15:05.597 DEBUG:boto:Method: PUT 2022-09-02T05:15:05.597 DEBUG:boto:Path: /xljbkf-66/ 2022-09-02T05:15:05.598 DEBUG:boto:Data: 2022-09-02T05:15:05.598 DEBUG:boto:Headers: {} 2022-09-02T05:15:05.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:05.599 DEBUG:boto:Port: 8000 2022-09-02T05:15:05.600 DEBUG:boto:Params: {} 2022-09-02T05:15:05.601 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:15:05.601 DEBUG:boto:Token: None 2022-09-02T05:15:05.602 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:15:05 GMT /xljbkf-66/ 2022-09-02T05:15:05.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PWv8xLxHYopKGEC8DOmrqWBjtMk= 2022-09-02T05:15:05.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PWv8xLxHYopKGEC8DOmrqWBjtMk=', 'Content-Length': '0'} 2022-09-02T05:15:05.654 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005189037b2e644cc8-0063119159-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:05.655 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xljbkf-67 2022-09-02T05:15:05.655 DEBUG:boto:path=/xljbkf-67/ 2022-09-02T05:15:05.656 DEBUG:boto:auth_path=/xljbkf-67/ 2022-09-02T05:15:05.656 DEBUG:boto:Method: PUT 2022-09-02T05:15:05.657 DEBUG:boto:Path: /xljbkf-67/ 2022-09-02T05:15:05.657 DEBUG:boto:Data: 2022-09-02T05:15:05.658 DEBUG:boto:Headers: {} 2022-09-02T05:15:05.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:05.659 DEBUG:boto:Port: 8001 2022-09-02T05:15:05.659 DEBUG:boto:Params: {} 2022-09-02T05:15:05.660 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:15:05.661 DEBUG:boto:Token: None 2022-09-02T05:15:05.661 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:15:05 GMT /xljbkf-67/ 2022-09-02T05:15:05.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5BIgCPeGp8bNgF3/82sFJK0Fv+Q= 2022-09-02T05:15:05.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5BIgCPeGp8bNgF3/82sFJK0Fv+Q=', 'Content-Length': '0'} 2022-09-02T05:15:05.703 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d58fd72956b1da0b-0063119159-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:05.704 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:05.705 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:05.705 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploads 2022-09-02T05:15:05.706 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploads 2022-09-02T05:15:05.706 DEBUG:boto:Method: POST 2022-09-02T05:15:05.707 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploads 2022-09-02T05:15:05.707 DEBUG:boto:Data: 2022-09-02T05:15:05.707 DEBUG:boto:Headers: {} 2022-09-02T05:15:05.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:05.708 DEBUG:boto:Port: 8000 2022-09-02T05:15:05.709 DEBUG:boto:Params: {} 2022-09-02T05:15:05.709 DEBUG:boto:Token: None 2022-09-02T05:15:05.710 DEBUG:boto:StringToSign: POST Fri, 02 Sep 2022 05:15:05 GMT /xljbkf-65/MULTIPART?uploads 2022-09-02T05:15:05.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/GCvhIGgcd7zyAcNhGnMLVggEOk= 2022-09-02T05:15:05.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/GCvhIGgcd7zyAcNhGnMLVggEOk=', 'Content-Length': '0'} 2022-09-02T05:15:07.060 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003296e190550adc75-0063119159-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.062 DEBUG:boto:b'xljbkf-65MULTIPART2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB' 2022-09-02T05:15:07.104 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.106 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.106 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=1 2022-09-02T05:15:07.107 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=1 2022-09-02T05:15:07.108 DEBUG:boto:Method: PUT 2022-09-02T05:15:07.108 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=1 2022-09-02T05:15:07.109 DEBUG:boto:Data: 2022-09-02T05:15:07.109 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-02T05:15:07.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:07.110 DEBUG:boto:Port: 8000 2022-09-02T05:15:07.111 DEBUG:boto:Params: {} 2022-09-02T05:15:07.111 DEBUG:boto:Token: None 2022-09-02T05:15:07.112 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Fri, 02 Sep 2022 05:15:07 GMT /xljbkf-65/MULTIPART?partNumber=1&uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:euAJJ8dzqIo7gIL+LdSs052QMX8= 2022-09-02T05:15:07.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:15:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:euAJJ8dzqIo7gIL+LdSs052QMX8='} 2022-09-02T05:15:07.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0b99d8e914b3fbf-006311915b-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.192 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.193 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.193 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=2 2022-09-02T05:15:07.194 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=2 2022-09-02T05:15:07.194 DEBUG:boto:Method: PUT 2022-09-02T05:15:07.195 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=2 2022-09-02T05:15:07.195 DEBUG:boto:Data: 2022-09-02T05:15:07.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-02T05:15:07.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:07.197 DEBUG:boto:Port: 8000 2022-09-02T05:15:07.197 DEBUG:boto:Params: {} 2022-09-02T05:15:07.198 DEBUG:boto:Token: None 2022-09-02T05:15:07.198 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Fri, 02 Sep 2022 05:15:07 GMT /xljbkf-65/MULTIPART?partNumber=2&uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:auH2Wr3PTlP03WuJWC1SqC0X0TE= 2022-09-02T05:15:07.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:15:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:auH2Wr3PTlP03WuJWC1SqC0X0TE='} 2022-09-02T05:15:07.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005402c3f5b314e3e1-006311915b-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.255 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.256 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.256 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=3 2022-09-02T05:15:07.257 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=3 2022-09-02T05:15:07.257 DEBUG:boto:Method: PUT 2022-09-02T05:15:07.258 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=3 2022-09-02T05:15:07.259 DEBUG:boto:Data: 2022-09-02T05:15:07.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-02T05:15:07.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:07.260 DEBUG:boto:Port: 8000 2022-09-02T05:15:07.261 DEBUG:boto:Params: {} 2022-09-02T05:15:07.261 DEBUG:boto:Token: None 2022-09-02T05:15:07.262 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Fri, 02 Sep 2022 05:15:07 GMT /xljbkf-65/MULTIPART?partNumber=3&uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.262 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BsPF/dGLsGobfyoEx8xkYL2kKzA= 2022-09-02T05:15:07.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:15:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BsPF/dGLsGobfyoEx8xkYL2kKzA='} 2022-09-02T05:15:07.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000881b994cf76f0e69-006311915b-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.331 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.332 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.332 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=4 2022-09-02T05:15:07.333 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=4 2022-09-02T05:15:07.333 DEBUG:boto:Method: PUT 2022-09-02T05:15:07.334 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB&partNumber=4 2022-09-02T05:15:07.334 DEBUG:boto:Data: 2022-09-02T05:15:07.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2022-09-02T05:15:07.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:07.336 DEBUG:boto:Port: 8000 2022-09-02T05:15:07.336 DEBUG:boto:Params: {} 2022-09-02T05:15:07.337 DEBUG:boto:Token: None 2022-09-02T05:15:07.338 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Fri, 02 Sep 2022 05:15:07 GMT /xljbkf-65/MULTIPART?partNumber=4&uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zPR8s5Jaj7aL1j6V6E3uZVcoSQY= 2022-09-02T05:15:07.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:15:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zPR8s5Jaj7aL1j6V6E3uZVcoSQY='} 2022-09-02T05:15:07.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c81cd6e09a4fa46-006311915b-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.397 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.397 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.398 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.398 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.399 DEBUG:boto:Method: GET 2022-09-02T05:15:07.399 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.400 DEBUG:boto:Data: 2022-09-02T05:15:07.400 DEBUG:boto:Headers: {} 2022-09-02T05:15:07.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:07.401 DEBUG:boto:Port: 8000 2022-09-02T05:15:07.402 DEBUG:boto:Params: {} 2022-09-02T05:15:07.402 DEBUG:boto:Token: None 2022-09-02T05:15:07.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:07 GMT /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z+1WFkItubNuPumtnP7NzcwvgJ8= 2022-09-02T05:15:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z+1WFkItubNuPumtnP7NzcwvgJ8=', 'Content-Length': '0'} 2022-09-02T05:15:07.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359b0359d186c04c-006311915b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.450 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.451 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:07.452 DEBUG:boto:path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.453 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.453 DEBUG:boto:Method: POST 2022-09-02T05:15:07.454 DEBUG:boto:Path: /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.454 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2022-09-02T05:15:07.455 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2022-09-02T05:15:07.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:07.456 DEBUG:boto:Port: 8000 2022-09-02T05:15:07.456 DEBUG:boto:Params: {} 2022-09-02T05:15:07.457 DEBUG:boto:Token: None 2022-09-02T05:15:07.458 DEBUG:boto:StringToSign: POST text/xml Fri, 02 Sep 2022 05:15:07 GMT /xljbkf-65/MULTIPART?uploadId=2~uKb_kyGu6Kx3X8rR1_Z68GoOfSMWttB 2022-09-02T05:15:07.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iWm8cUrwO/a4do9Ut1mHqs1w8r0= 2022-09-02T05:15:07.459 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iWm8cUrwO/a4do9Ut1mHqs1w8r0=', 'Content-Length': '460'} 2022-09-02T05:15:07.469 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e61354f66626e79-006311915b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:07.469 DEBUG:boto:http://smithi099.front.sepia.ceph.com:8000/xljbkf-65/MULTIPARTxljbkf-65MULTIPART 2022-09-02T05:15:07.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.472 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:07.552 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:15:07.565 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:15:07.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.566 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:07.566 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:15:07.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.567 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:07.567 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:15:07.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.568 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095705.687548_353.1", 2022-09-02T05:15:07.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:15:05.687548Z" 2022-09-02T05:15:07.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:07.569 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:15:07.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:07.570 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:15:07.571 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:15:07.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.573 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:07.632 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:07.633 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:07.683 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:07.684+0000 7ffa99842600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:07.684 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:07.686+0000 7ffa99842600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:07.707 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:15:07.708 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:15:07.708 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:15:07.708 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:15:07.709 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:15:07.709 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:07.709 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:07.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.710 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:15:07.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.710 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:15:07.710 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.711 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.711 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:07.711 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.711 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:15:07.712 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.712 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:07.712 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:07.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.713 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:15:07.713 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.714 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.714 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:15:07.714 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.715 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:07.715 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.715 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:15:07.715 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:07.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.716 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:15:07.716 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.717 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.717 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095705.687548_353.1", 2022-09-02T05:15:07.718 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.718 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:07.718 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.719 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:15:05.687548Z", 2022-09-02T05:15:07.719 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:07.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.720 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:15:07.720 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.720 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.721 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:07.721 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.721 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:07.721 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.721 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:07.722 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:07.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.722 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:15:07.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.723 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:15:07.723 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:15:07.723 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:15:07.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.724 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:15:07Z" 2022-09-02T05:15:07.724 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:15:07.725 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095705.687548_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:05.687548Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:07Z" } 2022-09-02T05:15:07.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095705.687548_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:05.687548Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:07.727 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.727 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.728 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:15:07.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:15:07.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.730 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:07.780 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:07.781 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:07.813 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:07.815+0000 7f29329f8600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:07.814 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:07.816+0000 7f29329f8600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:07.835 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:15:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:15:07.835 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:15:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:15:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:15:07.836 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:07.837 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:15:07.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.838 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:15:07.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.839 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:15:07.840 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:07.840 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:07.840 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:07.841 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:07.841 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:15:07.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:07.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:07.843 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:15:07.843 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:07.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.844 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:15:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:07.845 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:07.846 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:07.846 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:15:07.846 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:07.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:15:07.847 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:15:07.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:07.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.848 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:15:07.848 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:07.849 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:07.849 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095705.687548_353.1", 2022-09-02T05:15:07.849 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:07.849 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:15:07.850 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:07.850 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:15:05.687548Z", 2022-09-02T05:15:07.850 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:15:07.850 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:07.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:07.851 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:15:07.851 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:07.852 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:07.852 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:07.852 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:07.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:15:07.853 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:07.853 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:07.853 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:15:07.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:07.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:07.854 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:15:07.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.855 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:15:07.855 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:15:07.855 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:15:07.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:07.856 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:15:07Z" 2022-09-02T05:15:07.856 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:15:07.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095705.687548_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:05.687548Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:07Z" } 2022-09-02T05:15:07.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095705.687548_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:05.687548Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:07.858 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.859 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:15:07.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:15:07.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:07.863 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:07.919 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:07.920 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:07.952 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:07.954+0000 7f844d40c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:07.953 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:07.955+0000 7f844d40c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:07.964 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:15:07.965 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:15:07.965 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:15:07.965 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:15:07.966 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:15:07.966 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:07.966 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:07.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.967 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:15:07.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.968 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:15:07.968 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.968 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.968 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:07.969 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.969 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:15:07.969 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.970 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:07.970 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:07.970 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.971 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:15:07.971 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.972 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.972 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:15:07.972 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.972 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:07.973 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.973 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:15:07.973 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:07.973 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.974 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:15:07.975 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.975 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.975 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095705.687548_353.1", 2022-09-02T05:15:07.975 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.976 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:07.976 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.976 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:15:05.687548Z", 2022-09-02T05:15:07.976 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:07.977 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:07.977 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:15:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:07.980 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:07.980 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.980 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:07.980 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:15:07.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.981 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:15:07.981 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:15:07.982 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:15:07.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:07.982 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:15:07Z" 2022-09-02T05:15:07.982 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:15:07.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095705.687548_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:05.687548Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:07Z" } 2022-09-02T05:15:07.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095705.687548_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:05.687548Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:07.985 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.985 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095705.687548_353.1', 3: ''} 2022-09-02T05:15:07.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:15:07.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-65 2022-09-02T05:15:07.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:07.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:07.988 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:08.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:08.046 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:08.085+0000 7f33306b6600 1 waiting to reach incremental sync.. 2022-09-02T05:15:13.084 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:13.086+0000 7f33306b6600 1 waiting to reach incremental sync.. 2022-09-02T05:15:18.084 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:18.086+0000 7f33306b6600 1 waiting to reach incremental sync.. 2022-09-02T05:15:23.086 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:23.088+0000 7f33306b6600 1 bucket sync caught up with source: 2022-09-02T05:15:23.087 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000011.47.5, , , , , ] 2022-09-02T05:15:23.087 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000011.47.5, , , , , ] 2022-09-02T05:15:23.087 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:23.088+0000 7f33306b6600 0 bucket checkpoint complete 2022-09-02T05:15:23.105 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-65 2022-09-02T05:15:23.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:23.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:23.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:23.236 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:23.238+0000 7fc3194df600 1 bucket sync caught up with source: 2022-09-02T05:15:23.236 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000011.47.5, , , , , ] 2022-09-02T05:15:23.236 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000011.47.5, , , , , ] 2022-09-02T05:15:23.237 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:23.238+0000 7fc3194df600 0 bucket checkpoint complete 2022-09-02T05:15:23.260 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-65 2022-09-02T05:15:23.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:23.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:23.262 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:23.326 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:23.327 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:23.363 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:23.364+0000 7f5bab75d600 1 bucket sync caught up with source: 2022-09-02T05:15:23.363 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000011.47.5, , , , , ] 2022-09-02T05:15:23.363 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000011.47.5, , , , , ] 2022-09-02T05:15:23.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:23.364+0000 7f5bab75d600 0 bucket checkpoint complete 2022-09-02T05:15:23.376 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-65 2022-09-02T05:15:23.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:23.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:23.378 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:23.435 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:23.436 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:23.474 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:23.476+0000 7f9ff8f6d600 1 waiting to reach incremental sync.. 2022-09-02T05:15:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:28.476+0000 7f9ff8f6d600 1 waiting to reach incremental sync.. 2022-09-02T05:15:33.476 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:33.478+0000 7f9ff8f6d600 1 bucket sync caught up with source: 2022-09-02T05:15:33.477 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.1219.5, , , , , ] 2022-09-02T05:15:33.477 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.1219.5, , , , , ] 2022-09-02T05:15:33.478 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:33.478+0000 7f9ff8f6d600 0 bucket checkpoint complete 2022-09-02T05:15:33.501 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-65 2022-09-02T05:15:33.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:33.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:33.503 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:33.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:33.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:33.624 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:33.626+0000 7f0d48438600 1 waiting to reach incremental sync.. 2022-09-02T05:15:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:38.626+0000 7f0d48438600 1 waiting to reach incremental sync.. 2022-09-02T05:15:43.626 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:43.628+0000 7f0d48438600 1 bucket sync caught up with source: 2022-09-02T05:15:43.627 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.1219.5, , , , , ] 2022-09-02T05:15:43.627 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.1219.5, , , , , ] 2022-09-02T05:15:43.627 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:43.628+0000 7f0d48438600 0 bucket checkpoint complete 2022-09-02T05:15:43.643 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-65 2022-09-02T05:15:43.644 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-65 2022-09-02T05:15:43.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:43.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:43.646 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:43.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:43.708 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:43.745 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:43.747+0000 7fcb3c0f2600 1 bucket sync caught up with source: 2022-09-02T05:15:43.745 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.307.5, , , , , ] 2022-09-02T05:15:43.745 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.307.5, , , , , ] 2022-09-02T05:15:43.746 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:43.747+0000 7fcb3c0f2600 0 bucket checkpoint complete 2022-09-02T05:15:43.761 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-65 2022-09-02T05:15:43.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:43.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:43.764 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:43.830 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:43.830 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:43.873+0000 7f8fba034600 1 bucket sync caught up with source: 2022-09-02T05:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.307.5, , , , , ] 2022-09-02T05:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.307.5, , , , , ] 2022-09-02T05:15:43.872 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:43.873+0000 7f8fba034600 0 bucket checkpoint complete 2022-09-02T05:15:43.887 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-65 2022-09-02T05:15:43.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-65 zones={test-zone2, test-zone1} 2022-09-02T05:15:43.889 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:43.890 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:43.891 DEBUG:boto:Method: HEAD 2022-09-02T05:15:43.892 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:43.893 DEBUG:boto:Data: 2022-09-02T05:15:43.894 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.895 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:43.896 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.897 DEBUG:boto:Params: {} 2022-09-02T05:15:43.898 DEBUG:boto:Token: None 2022-09-02T05:15:43.899 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/ 2022-09-02T05:15:43.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kHOWBmQYUnzL60T+jK4Yf+w7458= 2022-09-02T05:15:43.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kHOWBmQYUnzL60T+jK4Yf+w7458=', 'Content-Length': '0'} 2022-09-02T05:15:43.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008aa9fb9190e08655-006311917f-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:43.906 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:43.907 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:43.907 DEBUG:boto:Method: HEAD 2022-09-02T05:15:43.908 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:43.909 DEBUG:boto:Data: 2022-09-02T05:15:43.910 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:43.912 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.912 DEBUG:boto:Params: {} 2022-09-02T05:15:43.913 DEBUG:boto:Token: None 2022-09-02T05:15:43.915 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/ 2022-09-02T05:15:43.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kHOWBmQYUnzL60T+jK4Yf+w7458= 2022-09-02T05:15:43.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kHOWBmQYUnzL60T+jK4Yf+w7458=', 'Content-Length': '0'} 2022-09-02T05:15:43.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd5cd3c03d338692-006311917f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:43.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:15:43.920 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:43.921 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:43.921 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.922 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.922 DEBUG:boto:Method: GET 2022-09-02T05:15:43.923 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.923 DEBUG:boto:Data: 2022-09-02T05:15:43.924 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:43.925 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.925 DEBUG:boto:Params: {} 2022-09-02T05:15:43.926 DEBUG:boto:Token: None 2022-09-02T05:15:43.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/?versions 2022-09-02T05:15:43.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0= 2022-09-02T05:15:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0=', 'Content-Length': '0'} 2022-09-02T05:15:43.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be16489559984be2-006311917f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:43.932 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:43.933 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-02T05:15:43.933 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:15:43.934 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:43.934 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:43.935 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.935 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.936 DEBUG:boto:Method: GET 2022-09-02T05:15:43.936 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.937 DEBUG:boto:Data: 2022-09-02T05:15:43.937 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:43.938 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.939 DEBUG:boto:Params: {} 2022-09-02T05:15:43.939 DEBUG:boto:Token: None 2022-09-02T05:15:43.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/?versions 2022-09-02T05:15:43.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0= 2022-09-02T05:15:43.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0=', 'Content-Length': '0'} 2022-09-02T05:15:43.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000746f566721676e72-006311917f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:43.944 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:43.945 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-02T05:15:43.946 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:43.947 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:43.947 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.948 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.949 DEBUG:boto:Method: GET 2022-09-02T05:15:43.949 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.950 DEBUG:boto:Data: 2022-09-02T05:15:43.950 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:43.951 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.952 DEBUG:boto:Params: {} 2022-09-02T05:15:43.952 DEBUG:boto:Token: None 2022-09-02T05:15:43.953 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/?versions 2022-09-02T05:15:43.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0= 2022-09-02T05:15:43.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0=', 'Content-Length': '0'} 2022-09-02T05:15:43.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb70013aff051e7-006311917f-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:43.957 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:43.958 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:43.959 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:43.959 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.960 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.961 DEBUG:boto:Method: GET 2022-09-02T05:15:43.961 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:43.962 DEBUG:boto:Data: 2022-09-02T05:15:43.962 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:43.964 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.964 DEBUG:boto:Params: {} 2022-09-02T05:15:43.965 DEBUG:boto:Token: None 2022-09-02T05:15:43.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/?versions 2022-09-02T05:15:43.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0= 2022-09-02T05:15:43.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sXgStfdmIVNniHEmsEodlBzBKQ0=', 'Content-Length': '0'} 2022-09-02T05:15:43.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db8528ee987393bf-006311917f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:43.971 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:43.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:43.973 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:43.974 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:43.975 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:43.976 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:43.977 DEBUG:boto:Method: GET 2022-09-02T05:15:43.977 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:43.978 DEBUG:boto:Data: 2022-09-02T05:15:43.979 DEBUG:boto:Headers: {} 2022-09-02T05:15:43.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:43.980 DEBUG:boto:Port: 8000 2022-09-02T05:15:43.980 DEBUG:boto:Params: {} 2022-09-02T05:15:43.982 DEBUG:boto:Token: None 2022-09-02T05:15:43.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:43 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:43.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NkpVhN2MuQ2RDD/el352yHu268I= 2022-09-02T05:15:43.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NkpVhN2MuQ2RDD/el352yHu268I=', 'Content-Length': '0'} 2022-09-02T05:15:43.987 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000213244475178b6de-006311917f-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.089 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.090 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.090 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.091 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.091 DEBUG:boto:Method: GET 2022-09-02T05:15:44.092 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.093 DEBUG:boto:Data: 2022-09-02T05:15:44.093 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.094 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.095 DEBUG:boto:Params: {} 2022-09-02T05:15:44.095 DEBUG:boto:Token: None 2022-09-02T05:15:44.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o= 2022-09-02T05:15:44.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o=', 'Content-Length': '0'} 2022-09-02T05:15:44.101 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001c4eb0e873467b64-0063119180-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.208 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.209 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.209 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.210 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.210 DEBUG:boto:Method: HEAD 2022-09-02T05:15:44.211 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.211 DEBUG:boto:Data: 2022-09-02T05:15:44.212 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:44.213 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.213 DEBUG:boto:Params: {} 2022-09-02T05:15:44.214 DEBUG:boto:Token: None 2022-09-02T05:15:44.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wU70lVg8lyZLDGku7fWcKZklecs= 2022-09-02T05:15:44.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wU70lVg8lyZLDGku7fWcKZklecs=', 'Content-Length': '0'} 2022-09-02T05:15:44.218 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000061cfe62828e0ac22-0063119180-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.219 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.220 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.220 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.221 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.221 DEBUG:boto:Method: HEAD 2022-09-02T05:15:44.222 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.222 DEBUG:boto:Data: 2022-09-02T05:15:44.223 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.224 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.224 DEBUG:boto:Params: {} 2022-09-02T05:15:44.225 DEBUG:boto:Token: None 2022-09-02T05:15:44.226 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wU70lVg8lyZLDGku7fWcKZklecs= 2022-09-02T05:15:44.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wU70lVg8lyZLDGku7fWcKZklecs=', 'Content-Length': '0'} 2022-09-02T05:15:44.231 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000005d8c5c74dde3ff5f-0063119180-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:44.233 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.234 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.235 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.236 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.237 DEBUG:boto:Method: GET 2022-09-02T05:15:44.237 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.238 DEBUG:boto:Data: 2022-09-02T05:15:44.239 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:44.240 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.240 DEBUG:boto:Params: {} 2022-09-02T05:15:44.241 DEBUG:boto:Token: None 2022-09-02T05:15:44.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o= 2022-09-02T05:15:44.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o=', 'Content-Length': '0'} 2022-09-02T05:15:44.246 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000009acebe3c8b1e5d5-0063119180-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.340 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.341 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.341 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.342 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.342 DEBUG:boto:Method: GET 2022-09-02T05:15:44.343 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.344 DEBUG:boto:Data: 2022-09-02T05:15:44.344 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.345 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.346 DEBUG:boto:Params: {} 2022-09-02T05:15:44.346 DEBUG:boto:Token: None 2022-09-02T05:15:44.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o= 2022-09-02T05:15:44.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o=', 'Content-Length': '0'} 2022-09-02T05:15:44.352 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e278ab5cb69598ac-0063119180-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.483 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.484 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.484 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.485 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.485 DEBUG:boto:Method: GET 2022-09-02T05:15:44.486 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.487 DEBUG:boto:Data: 2022-09-02T05:15:44.487 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:44.488 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.489 DEBUG:boto:Params: {} 2022-09-02T05:15:44.489 DEBUG:boto:Token: None 2022-09-02T05:15:44.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/ 2022-09-02T05:15:44.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A= 2022-09-02T05:15:44.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A=', 'Content-Length': '0'} 2022-09-02T05:15:44.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba6b576650c38389-0063119180-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.495 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.496 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.497 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.497 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.498 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.499 DEBUG:boto:Method: GET 2022-09-02T05:15:44.500 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.500 DEBUG:boto:Data: 2022-09-02T05:15:44.501 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.502 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.502 DEBUG:boto:Params: {} 2022-09-02T05:15:44.503 DEBUG:boto:Token: None 2022-09-02T05:15:44.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/ 2022-09-02T05:15:44.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A= 2022-09-02T05:15:44.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A=', 'Content-Length': '0'} 2022-09-02T05:15:44.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001befeb50d8c4d6b8-0063119180-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.508 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.509 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.510 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.510 DEBUG:boto:Method: HEAD 2022-09-02T05:15:44.511 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:44.511 DEBUG:boto:Data: 2022-09-02T05:15:44.512 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:44.513 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.513 DEBUG:boto:Params: {} 2022-09-02T05:15:44.514 DEBUG:boto:Token: None 2022-09-02T05:15:44.515 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:44.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XA0HdfWeUlJ2XIaDdr8ZVf6ViC8= 2022-09-02T05:15:44.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XA0HdfWeUlJ2XIaDdr8ZVf6ViC8=', 'Content-Length': '0'} 2022-09-02T05:15:44.518 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002ff644f6627ac73f-0063119180-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.520 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.521 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.522 DEBUG:boto:Method: HEAD 2022-09-02T05:15:44.523 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:44.524 DEBUG:boto:Data: 2022-09-02T05:15:44.525 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.526 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.527 DEBUG:boto:Params: {} 2022-09-02T05:15:44.528 DEBUG:boto:Token: None 2022-09-02T05:15:44.529 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:44.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XA0HdfWeUlJ2XIaDdr8ZVf6ViC8= 2022-09-02T05:15:44.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XA0HdfWeUlJ2XIaDdr8ZVf6ViC8=', 'Content-Length': '0'} 2022-09-02T05:15:44.534 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000014679421e6be8e68-0063119180-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:44.535 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.535 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.536 DEBUG:boto:Method: GET 2022-09-02T05:15:44.537 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:44.537 DEBUG:boto:Data: 2022-09-02T05:15:44.538 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:44.539 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.540 DEBUG:boto:Params: {} 2022-09-02T05:15:44.540 DEBUG:boto:Token: None 2022-09-02T05:15:44.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:44.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oYcTbE41sga2+IHMS6E1TTzqDWg= 2022-09-02T05:15:44.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oYcTbE41sga2+IHMS6E1TTzqDWg=', 'Content-Length': '0'} 2022-09-02T05:15:44.545 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008142aeea7ccadf6a-0063119180-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.637 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.638 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.639 DEBUG:boto:Method: GET 2022-09-02T05:15:44.639 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:44.640 DEBUG:boto:Data: 2022-09-02T05:15:44.640 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.641 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.642 DEBUG:boto:Params: {} 2022-09-02T05:15:44.643 DEBUG:boto:Token: None 2022-09-02T05:15:44.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:44.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oYcTbE41sga2+IHMS6E1TTzqDWg= 2022-09-02T05:15:44.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oYcTbE41sga2+IHMS6E1TTzqDWg=', 'Content-Length': '0'} 2022-09-02T05:15:44.648 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000005070ff7613649337-0063119180-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.746 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.747 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.747 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.748 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.748 DEBUG:boto:Method: GET 2022-09-02T05:15:44.749 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.750 DEBUG:boto:Data: 2022-09-02T05:15:44.750 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:44.751 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.752 DEBUG:boto:Params: {} 2022-09-02T05:15:44.752 DEBUG:boto:Token: None 2022-09-02T05:15:44.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/ 2022-09-02T05:15:44.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A= 2022-09-02T05:15:44.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A=', 'Content-Length': '0'} 2022-09-02T05:15:44.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaab6aa60d22823c-0063119180-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.800 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.802 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.803 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.804 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.805 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.805 DEBUG:boto:Method: GET 2022-09-02T05:15:44.806 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:44.807 DEBUG:boto:Data: 2022-09-02T05:15:44.807 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.809 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.810 DEBUG:boto:Params: {} 2022-09-02T05:15:44.810 DEBUG:boto:Token: None 2022-09-02T05:15:44.811 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/ 2022-09-02T05:15:44.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A= 2022-09-02T05:15:44.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C/FhiKUQSAtXXP299lviGthxO1A=', 'Content-Length': '0'} 2022-09-02T05:15:44.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006467967407f088c3-0063119180-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.816 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-65 zones={test-zone2, test-zone1} 2022-09-02T05:15:44.818 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-65 zones={test-zone3, test-zone1} 2022-09-02T05:15:44.818 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.819 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.819 DEBUG:boto:Method: HEAD 2022-09-02T05:15:44.820 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:44.820 DEBUG:boto:Data: 2022-09-02T05:15:44.821 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:44.822 DEBUG:boto:Port: 8001 2022-09-02T05:15:44.823 DEBUG:boto:Params: {} 2022-09-02T05:15:44.823 DEBUG:boto:Token: None 2022-09-02T05:15:44.824 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/ 2022-09-02T05:15:44.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lez9uTAnejL4s+N6SUSIvL4ooAQ= 2022-09-02T05:15:44.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lez9uTAnejL4s+N6SUSIvL4ooAQ=', 'Content-Length': '0'} 2022-09-02T05:15:44.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049e79ba9e19c4f8b-0063119180-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.829 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.830 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.830 DEBUG:boto:Method: HEAD 2022-09-02T05:15:44.831 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:44.831 DEBUG:boto:Data: 2022-09-02T05:15:44.832 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.833 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.834 DEBUG:boto:Params: {} 2022-09-02T05:15:44.834 DEBUG:boto:Token: None 2022-09-02T05:15:44.835 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/ 2022-09-02T05:15:44.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lez9uTAnejL4s+N6SUSIvL4ooAQ= 2022-09-02T05:15:44.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lez9uTAnejL4s+N6SUSIvL4ooAQ=', 'Content-Length': '0'} 2022-09-02T05:15:44.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2edf3bafc83e9c7-0063119180-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.840 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:15:44.840 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.841 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.841 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.842 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.842 DEBUG:boto:Method: GET 2022-09-02T05:15:44.843 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.844 DEBUG:boto:Data: 2022-09-02T05:15:44.844 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:44.845 DEBUG:boto:Port: 8001 2022-09-02T05:15:44.846 DEBUG:boto:Params: {} 2022-09-02T05:15:44.846 DEBUG:boto:Token: None 2022-09-02T05:15:44.847 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/?versions 2022-09-02T05:15:44.848 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs= 2022-09-02T05:15:44.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs=', 'Content-Length': '0'} 2022-09-02T05:15:44.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e2604720b6f040-0063119180-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.852 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.852 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-02T05:15:44.853 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:15:44.854 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.854 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.855 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.855 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.856 DEBUG:boto:Method: GET 2022-09-02T05:15:44.857 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.857 DEBUG:boto:Data: 2022-09-02T05:15:44.858 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.859 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.859 DEBUG:boto:Params: {} 2022-09-02T05:15:44.860 DEBUG:boto:Token: None 2022-09-02T05:15:44.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/?versions 2022-09-02T05:15:44.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs= 2022-09-02T05:15:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs=', 'Content-Length': '0'} 2022-09-02T05:15:44.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfb2a46073cbbab-0063119180-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.906 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.907 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-02T05:15:44.908 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.908 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.909 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.909 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.910 DEBUG:boto:Method: GET 2022-09-02T05:15:44.910 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.911 DEBUG:boto:Data: 2022-09-02T05:15:44.911 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:44.913 DEBUG:boto:Port: 8001 2022-09-02T05:15:44.913 DEBUG:boto:Params: {} 2022-09-02T05:15:44.914 DEBUG:boto:Token: None 2022-09-02T05:15:44.914 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/?versions 2022-09-02T05:15:44.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs= 2022-09-02T05:15:44.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs=', 'Content-Length': '0'} 2022-09-02T05:15:44.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb76e5b8a821b6c-0063119180-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.919 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.920 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:44.920 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:44.921 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.921 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.922 DEBUG:boto:Method: GET 2022-09-02T05:15:44.923 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:44.923 DEBUG:boto:Data: 2022-09-02T05:15:44.923 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:44.924 DEBUG:boto:Port: 8000 2022-09-02T05:15:44.925 DEBUG:boto:Params: {} 2022-09-02T05:15:44.925 DEBUG:boto:Token: None 2022-09-02T05:15:44.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/?versions 2022-09-02T05:15:44.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs= 2022-09-02T05:15:44.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e1l9ZvGQnf55rQgTuo3Ce8Gpqqs=', 'Content-Length': '0'} 2022-09-02T05:15:44.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca22a9132a8cf851-0063119180-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:44.931 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:44.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:44.932 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.933 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:44.933 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.934 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.934 DEBUG:boto:Method: GET 2022-09-02T05:15:44.935 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.935 DEBUG:boto:Data: 2022-09-02T05:15:44.936 DEBUG:boto:Headers: {} 2022-09-02T05:15:44.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:44.937 DEBUG:boto:Port: 8001 2022-09-02T05:15:44.937 DEBUG:boto:Params: {} 2022-09-02T05:15:44.938 DEBUG:boto:Token: None 2022-09-02T05:15:44.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:44 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:44.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o= 2022-09-02T05:15:44.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6wc5yzajlI8rjh4v5OySf3VaE+o=', 'Content-Length': '0'} 2022-09-02T05:15:44.945 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000076e2b05cddc85412-0063119180-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.017 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.018 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.018 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.019 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.019 DEBUG:boto:Method: GET 2022-09-02T05:15:45.020 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.020 DEBUG:boto:Data: 2022-09-02T05:15:45.021 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.022 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.023 DEBUG:boto:Params: {} 2022-09-02T05:15:45.023 DEBUG:boto:Token: None 2022-09-02T05:15:45.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE= 2022-09-02T05:15:45.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE=', 'Content-Length': '0'} 2022-09-02T05:15:45.029 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e7237dae79965f2a-0063119181-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.134 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.136 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.137 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.138 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.139 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.140 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.141 DEBUG:boto:Data: 2022-09-02T05:15:45.142 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.144 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.145 DEBUG:boto:Params: {} 2022-09-02T05:15:45.146 DEBUG:boto:Token: None 2022-09-02T05:15:45.147 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg= 2022-09-02T05:15:45.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg=', 'Content-Length': '0'} 2022-09-02T05:15:45.153 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000087c98a8362262ea-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.154 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.155 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.156 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.156 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.157 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.157 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.158 DEBUG:boto:Data: 2022-09-02T05:15:45.158 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.160 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.160 DEBUG:boto:Params: {} 2022-09-02T05:15:45.161 DEBUG:boto:Token: None 2022-09-02T05:15:45.161 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg= 2022-09-02T05:15:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg=', 'Content-Length': '0'} 2022-09-02T05:15:45.165 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000a66026c5cba4fd6f-0063119181-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:45.166 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.167 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.167 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.168 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.168 DEBUG:boto:Method: GET 2022-09-02T05:15:45.169 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.169 DEBUG:boto:Data: 2022-09-02T05:15:45.170 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.171 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.172 DEBUG:boto:Params: {} 2022-09-02T05:15:45.172 DEBUG:boto:Token: None 2022-09-02T05:15:45.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE= 2022-09-02T05:15:45.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE=', 'Content-Length': '0'} 2022-09-02T05:15:45.179 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b750e7c1e3fe0d4d-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.253 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.254 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.254 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.255 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.255 DEBUG:boto:Method: GET 2022-09-02T05:15:45.256 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.256 DEBUG:boto:Data: 2022-09-02T05:15:45.257 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.258 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.258 DEBUG:boto:Params: {} 2022-09-02T05:15:45.259 DEBUG:boto:Token: None 2022-09-02T05:15:45.259 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE= 2022-09-02T05:15:45.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE=', 'Content-Length': '0'} 2022-09-02T05:15:45.264 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000086091a94e73e0173-0063119181-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.366 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.367 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.367 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.368 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.368 DEBUG:boto:Method: GET 2022-09-02T05:15:45.369 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.369 DEBUG:boto:Data: 2022-09-02T05:15:45.370 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.371 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.372 DEBUG:boto:Params: {} 2022-09-02T05:15:45.372 DEBUG:boto:Token: None 2022-09-02T05:15:45.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/ 2022-09-02T05:15:45.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs= 2022-09-02T05:15:45.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs=', 'Content-Length': '0'} 2022-09-02T05:15:45.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000226542b9c26adcb2-0063119181-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.378 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.378 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.379 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.380 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.380 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.381 DEBUG:boto:Method: GET 2022-09-02T05:15:45.381 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.382 DEBUG:boto:Data: 2022-09-02T05:15:45.382 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.383 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.384 DEBUG:boto:Params: {} 2022-09-02T05:15:45.384 DEBUG:boto:Token: None 2022-09-02T05:15:45.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/ 2022-09-02T05:15:45.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs= 2022-09-02T05:15:45.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs=', 'Content-Length': '0'} 2022-09-02T05:15:45.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84a28b5b847b0d5-0063119181-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.390 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.391 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.391 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.392 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.392 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:45.393 DEBUG:boto:Data: 2022-09-02T05:15:45.393 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.394 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.395 DEBUG:boto:Params: {} 2022-09-02T05:15:45.395 DEBUG:boto:Token: None 2022-09-02T05:15:45.396 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:45.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7/31+8+tqu13A5KhqDmrMbCtG7w= 2022-09-02T05:15:45.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7/31+8+tqu13A5KhqDmrMbCtG7w=', 'Content-Length': '0'} 2022-09-02T05:15:45.399 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e8eafb7f12a270c2-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.400 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.401 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.401 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.402 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:45.402 DEBUG:boto:Data: 2022-09-02T05:15:45.403 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.404 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.404 DEBUG:boto:Params: {} 2022-09-02T05:15:45.405 DEBUG:boto:Token: None 2022-09-02T05:15:45.405 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:45.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7/31+8+tqu13A5KhqDmrMbCtG7w= 2022-09-02T05:15:45.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7/31+8+tqu13A5KhqDmrMbCtG7w=', 'Content-Length': '0'} 2022-09-02T05:15:45.409 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000aa369ced26195f4a-0063119181-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:45.410 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.411 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.411 DEBUG:boto:Method: GET 2022-09-02T05:15:45.412 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:45.413 DEBUG:boto:Data: 2022-09-02T05:15:45.413 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.415 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.415 DEBUG:boto:Params: {} 2022-09-02T05:15:45.416 DEBUG:boto:Token: None 2022-09-02T05:15:45.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:45.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Td71jP+UPQU2CAkEXlcBOa0s2Pc= 2022-09-02T05:15:45.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Td71jP+UPQU2CAkEXlcBOa0s2Pc=', 'Content-Length': '0'} 2022-09-02T05:15:45.422 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000051048c993f448a92-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.502 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.503 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.504 DEBUG:boto:Method: GET 2022-09-02T05:15:45.504 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:45.505 DEBUG:boto:Data: 2022-09-02T05:15:45.505 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.506 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.507 DEBUG:boto:Params: {} 2022-09-02T05:15:45.508 DEBUG:boto:Token: None 2022-09-02T05:15:45.508 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:45.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Td71jP+UPQU2CAkEXlcBOa0s2Pc= 2022-09-02T05:15:45.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Td71jP+UPQU2CAkEXlcBOa0s2Pc=', 'Content-Length': '0'} 2022-09-02T05:15:45.513 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000024045ecd25c6f67f-0063119181-1595-test-zone1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.619 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.620 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.620 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.621 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.622 DEBUG:boto:Method: GET 2022-09-02T05:15:45.622 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.623 DEBUG:boto:Data: 2022-09-02T05:15:45.623 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.625 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.625 DEBUG:boto:Params: {} 2022-09-02T05:15:45.626 DEBUG:boto:Token: None 2022-09-02T05:15:45.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/ 2022-09-02T05:15:45.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs= 2022-09-02T05:15:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs=', 'Content-Length': '0'} 2022-09-02T05:15:45.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fabe63b4e81244a7-0063119181-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.631 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.632 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.633 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.633 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.634 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.635 DEBUG:boto:Method: GET 2022-09-02T05:15:45.635 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:45.636 DEBUG:boto:Data: 2022-09-02T05:15:45.636 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:45.637 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.638 DEBUG:boto:Params: {} 2022-09-02T05:15:45.639 DEBUG:boto:Token: None 2022-09-02T05:15:45.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/ 2022-09-02T05:15:45.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs= 2022-09-02T05:15:45.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:joFwEwH7u+F/Wr+L+UwW7YUBVAs=', 'Content-Length': '0'} 2022-09-02T05:15:45.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c11470c145a5df7-0063119181-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.644 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.644 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-65 zones={test-zone3, test-zone1} 2022-09-02T05:15:45.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-65 zones={test-zone3, test-zone2} 2022-09-02T05:15:45.646 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.646 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.647 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.647 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:45.648 DEBUG:boto:Data: 2022-09-02T05:15:45.648 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.649 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.650 DEBUG:boto:Params: {} 2022-09-02T05:15:45.650 DEBUG:boto:Token: None 2022-09-02T05:15:45.651 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/ 2022-09-02T05:15:45.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y16DhsC+zjauLn4rtzP48i+emVY= 2022-09-02T05:15:45.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y16DhsC+zjauLn4rtzP48i+emVY=', 'Content-Length': '0'} 2022-09-02T05:15:45.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4c9d0d76837195d-0063119181-1599-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.655 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.656 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.656 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.657 DEBUG:boto:Path: /xljbkf-65/ 2022-09-02T05:15:45.658 DEBUG:boto:Data: 2022-09-02T05:15:45.658 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:45.660 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.660 DEBUG:boto:Params: {} 2022-09-02T05:15:45.661 DEBUG:boto:Token: None 2022-09-02T05:15:45.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/ 2022-09-02T05:15:45.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y16DhsC+zjauLn4rtzP48i+emVY= 2022-09-02T05:15:45.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y16DhsC+zjauLn4rtzP48i+emVY=', 'Content-Length': '0'} 2022-09-02T05:15:45.665 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ebdcee5c3bf4abb-0063119181-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.666 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:15:45.667 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.667 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.668 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.668 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.669 DEBUG:boto:Method: GET 2022-09-02T05:15:45.669 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.670 DEBUG:boto:Data: 2022-09-02T05:15:45.670 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.672 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.672 DEBUG:boto:Params: {} 2022-09-02T05:15:45.673 DEBUG:boto:Token: None 2022-09-02T05:15:45.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/?versions 2022-09-02T05:15:45.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA= 2022-09-02T05:15:45.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA=', 'Content-Length': '0'} 2022-09-02T05:15:45.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4750e61503c639-0063119181-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.678 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.679 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-02T05:15:45.680 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:15:45.680 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.681 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.682 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.682 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.683 DEBUG:boto:Method: GET 2022-09-02T05:15:45.683 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.684 DEBUG:boto:Data: 2022-09-02T05:15:45.684 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:45.685 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.686 DEBUG:boto:Params: {} 2022-09-02T05:15:45.687 DEBUG:boto:Token: None 2022-09-02T05:15:45.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/?versions 2022-09-02T05:15:45.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA= 2022-09-02T05:15:45.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA=', 'Content-Length': '0'} 2022-09-02T05:15:45.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a5366e909c6c011-0063119181-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.692 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.693 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2022-09-02T05:15:45.694 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.694 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.695 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.696 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.696 DEBUG:boto:Method: GET 2022-09-02T05:15:45.697 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.697 DEBUG:boto:Data: 2022-09-02T05:15:45.698 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.699 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.699 DEBUG:boto:Params: {} 2022-09-02T05:15:45.700 DEBUG:boto:Token: None 2022-09-02T05:15:45.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/?versions 2022-09-02T05:15:45.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA= 2022-09-02T05:15:45.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA=', 'Content-Length': '0'} 2022-09-02T05:15:45.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046aea3f9cc6561b7-0063119181-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.705 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.706 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:45.707 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:45.707 DEBUG:boto:path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.708 DEBUG:boto:auth_path=/xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.708 DEBUG:boto:Method: GET 2022-09-02T05:15:45.709 DEBUG:boto:Path: /xljbkf-65/?versions&max-keys=999 2022-09-02T05:15:45.709 DEBUG:boto:Data: 2022-09-02T05:15:45.710 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.710 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:45.711 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.711 DEBUG:boto:Params: {} 2022-09-02T05:15:45.712 DEBUG:boto:Token: None 2022-09-02T05:15:45.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/?versions 2022-09-02T05:15:45.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA= 2022-09-02T05:15:45.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s28REK+N3os29j6gsnMI8iAZTOA=', 'Content-Length': '0'} 2022-09-02T05:15:45.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ce9c7b15cc7f71-0063119181-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.717 DEBUG:boto:b'xljbkf-65999falseMULTIPARTnulltrue2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:45.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:45.719 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.720 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.720 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.721 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.721 DEBUG:boto:Method: GET 2022-09-02T05:15:45.722 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.722 DEBUG:boto:Data: 2022-09-02T05:15:45.723 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.724 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.725 DEBUG:boto:Params: {} 2022-09-02T05:15:45.725 DEBUG:boto:Token: None 2022-09-02T05:15:45.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE= 2022-09-02T05:15:45.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE=', 'Content-Length': '0'} 2022-09-02T05:15:45.732 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009f9fd5f7eeb430f6-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.809 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.810 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.810 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.811 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.811 DEBUG:boto:Method: GET 2022-09-02T05:15:45.812 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.812 DEBUG:boto:Data: 2022-09-02T05:15:45.813 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:45.814 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.815 DEBUG:boto:Params: {} 2022-09-02T05:15:45.815 DEBUG:boto:Token: None 2022-09-02T05:15:45.816 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.817 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE= 2022-09-02T05:15:45.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE=', 'Content-Length': '0'} 2022-09-02T05:15:45.821 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000908542d4f7bab05a-0063119181-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.928 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.929 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.929 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.930 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.930 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.931 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.932 DEBUG:boto:Data: 2022-09-02T05:15:45.932 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.933 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.934 DEBUG:boto:Params: {} 2022-09-02T05:15:45.934 DEBUG:boto:Token: None 2022-09-02T05:15:45.935 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg= 2022-09-02T05:15:45.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg=', 'Content-Length': '0'} 2022-09-02T05:15:45.939 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d642bcb795dfdfce-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.939 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.940 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.941 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.941 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.942 DEBUG:boto:Method: HEAD 2022-09-02T05:15:45.942 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.943 DEBUG:boto:Data: 2022-09-02T05:15:45.943 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:45.944 DEBUG:boto:Port: 8000 2022-09-02T05:15:45.945 DEBUG:boto:Params: {} 2022-09-02T05:15:45.946 DEBUG:boto:Token: None 2022-09-02T05:15:45.946 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg= 2022-09-02T05:15:45.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hLAcaEokz2nD+i9o483BnIQHWYg=', 'Content-Length': '0'} 2022-09-02T05:15:45.950 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000064c9327496fbe2cc-0063119181-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:45.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:45.951 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.952 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:45.952 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.953 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.953 DEBUG:boto:Method: GET 2022-09-02T05:15:45.954 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.955 DEBUG:boto:Data: 2022-09-02T05:15:45.955 DEBUG:boto:Headers: {} 2022-09-02T05:15:45.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:45.956 DEBUG:boto:Port: 8001 2022-09-02T05:15:45.957 DEBUG:boto:Params: {} 2022-09-02T05:15:45.958 DEBUG:boto:Token: None 2022-09-02T05:15:45.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:45 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:45.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE= 2022-09-02T05:15:45.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OzzOa5la3jq35+3tzy1LT3j/+UE=', 'Content-Length': '0'} 2022-09-02T05:15:45.964 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e65ea84a077e41d3-0063119181-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.040 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.041 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.042 DEBUG:boto:path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:46.042 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:46.043 DEBUG:boto:Method: GET 2022-09-02T05:15:46.044 DEBUG:boto:Path: /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:46.044 DEBUG:boto:Data: 2022-09-02T05:15:46.045 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:46.046 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.046 DEBUG:boto:Params: {} 2022-09-02T05:15:46.047 DEBUG:boto:Token: None 2022-09-02T05:15:46.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/MULTIPART?versionId=null 2022-09-02T05:15:46.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1fjyrRl5+F+jvAnKCQNmJhtq1ZA= 2022-09-02T05:15:46.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1fjyrRl5+F+jvAnKCQNmJhtq1ZA=', 'Content-Length': '0'} 2022-09-02T05:15:46.052 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000088078d08af5c7507-0063119182-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.158 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:46.159 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:46.160 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.160 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.161 DEBUG:boto:Method: GET 2022-09-02T05:15:46.161 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.162 DEBUG:boto:Data: 2022-09-02T05:15:46.163 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:46.164 DEBUG:boto:Port: 8001 2022-09-02T05:15:46.164 DEBUG:boto:Params: {} 2022-09-02T05:15:46.165 DEBUG:boto:Token: None 2022-09-02T05:15:46.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/ 2022-09-02T05:15:46.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc= 2022-09-02T05:15:46.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc=', 'Content-Length': '0'} 2022-09-02T05:15:46.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006911c79e784b8355-0063119182-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.214 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:46.216 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:46.217 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:46.218 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.219 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.220 DEBUG:boto:Method: GET 2022-09-02T05:15:46.221 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.222 DEBUG:boto:Data: 2022-09-02T05:15:46.223 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:46.224 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.225 DEBUG:boto:Params: {} 2022-09-02T05:15:46.226 DEBUG:boto:Token: None 2022-09-02T05:15:46.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/ 2022-09-02T05:15:46.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc= 2022-09-02T05:15:46.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc=', 'Content-Length': '0'} 2022-09-02T05:15:46.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ed3b0161c4d7fcf-0063119182-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.233 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:46.234 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.235 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.235 DEBUG:boto:Method: HEAD 2022-09-02T05:15:46.236 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:46.237 DEBUG:boto:Data: 2022-09-02T05:15:46.237 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:46.238 DEBUG:boto:Port: 8001 2022-09-02T05:15:46.239 DEBUG:boto:Params: {} 2022-09-02T05:15:46.239 DEBUG:boto:Token: None 2022-09-02T05:15:46.240 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:46.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FxK6+EjVchUEsCFLgyTsmKX0Sms= 2022-09-02T05:15:46.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FxK6+EjVchUEsCFLgyTsmKX0Sms=', 'Content-Length': '0'} 2022-09-02T05:15:46.243 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000dd1d489de6a34af7-0063119182-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.244 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.245 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.245 DEBUG:boto:Method: HEAD 2022-09-02T05:15:46.246 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:46.247 DEBUG:boto:Data: 2022-09-02T05:15:46.247 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:46.248 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.249 DEBUG:boto:Params: {} 2022-09-02T05:15:46.250 DEBUG:boto:Token: None 2022-09-02T05:15:46.250 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:46.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FxK6+EjVchUEsCFLgyTsmKX0Sms= 2022-09-02T05:15:46.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FxK6+EjVchUEsCFLgyTsmKX0Sms=', 'Content-Length': '0'} 2022-09-02T05:15:46.255 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003784b8057688be55-0063119182-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2022-09-02T05:15:46.256 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.257 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.257 DEBUG:boto:Method: GET 2022-09-02T05:15:46.258 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:46.258 DEBUG:boto:Data: 2022-09-02T05:15:46.259 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:46.260 DEBUG:boto:Port: 8001 2022-09-02T05:15:46.261 DEBUG:boto:Params: {} 2022-09-02T05:15:46.261 DEBUG:boto:Token: None 2022-09-02T05:15:46.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:46.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xMVbSjD378Hc/57cgoc6NN8qylA= 2022-09-02T05:15:46.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xMVbSjD378Hc/57cgoc6NN8qylA=', 'Content-Length': '0'} 2022-09-02T05:15:46.268 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004378c05a4b56f3d1-0063119182-1599-test-zone3'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.348 DEBUG:boto:path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.349 DEBUG:boto:auth_path=/xljbkf-65/MULTIPART 2022-09-02T05:15:46.350 DEBUG:boto:Method: GET 2022-09-02T05:15:46.350 DEBUG:boto:Path: /xljbkf-65/MULTIPART 2022-09-02T05:15:46.351 DEBUG:boto:Data: 2022-09-02T05:15:46.351 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:46.352 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.353 DEBUG:boto:Params: {} 2022-09-02T05:15:46.354 DEBUG:boto:Token: None 2022-09-02T05:15:46.354 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/MULTIPART 2022-09-02T05:15:46.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xMVbSjD378Hc/57cgoc6NN8qylA= 2022-09-02T05:15:46.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xMVbSjD378Hc/57cgoc6NN8qylA=', 'Content-Length': '0'} 2022-09-02T05:15:46.359 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000006641d264bdb1efae-0063119182-15e3-test-zone2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.484 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:46.485 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:46.485 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.486 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.487 DEBUG:boto:Method: GET 2022-09-02T05:15:46.487 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.488 DEBUG:boto:Data: 2022-09-02T05:15:46.488 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:15:46.489 DEBUG:boto:Port: 8001 2022-09-02T05:15:46.490 DEBUG:boto:Params: {} 2022-09-02T05:15:46.491 DEBUG:boto:Token: None 2022-09-02T05:15:46.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/ 2022-09-02T05:15:46.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc= 2022-09-02T05:15:46.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc=', 'Content-Length': '0'} 2022-09-02T05:15:46.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb4dcf1af57c82b6-0063119182-1599-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.496 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:46.497 DEBUG:boto:path=/xljbkf-65/ 2022-09-02T05:15:46.498 DEBUG:boto:auth_path=/xljbkf-65/ 2022-09-02T05:15:46.498 DEBUG:boto:path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.499 DEBUG:boto:auth_path=/xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.499 DEBUG:boto:Method: GET 2022-09-02T05:15:46.500 DEBUG:boto:Path: /xljbkf-65/?prefix=MULTIPART 2022-09-02T05:15:46.500 DEBUG:boto:Data: 2022-09-02T05:15:46.501 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.502 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:15:46.502 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.503 DEBUG:boto:Params: {} 2022-09-02T05:15:46.503 DEBUG:boto:Token: None 2022-09-02T05:15:46.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-65/ 2022-09-02T05:15:46.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc= 2022-09-02T05:15:46.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V8T7qmvC1P0D199Q4WmD/ne4+Bc=', 'Content-Length': '0'} 2022-09-02T05:15:46.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a107c7d38a5d41-0063119182-15e3-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.509 DEBUG:boto:b'xljbkf-65MULTIPART1000falseMULTIPART2022-09-02T05:15:07.466Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:15:46.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-65 zones={test-zone3, test-zone2} 2022-09-02T05:15:46.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2022-09-02T05:15:46.512 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-68 2022-09-02T05:15:46.512 DEBUG:boto:path=/xljbkf-68/ 2022-09-02T05:15:46.513 DEBUG:boto:auth_path=/xljbkf-68/ 2022-09-02T05:15:46.513 DEBUG:boto:Method: PUT 2022-09-02T05:15:46.514 DEBUG:boto:Path: /xljbkf-68/ 2022-09-02T05:15:46.514 DEBUG:boto:Data: 2022-09-02T05:15:46.515 DEBUG:boto:Headers: {} 2022-09-02T05:15:46.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:46.516 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.516 DEBUG:boto:Params: {} 2022-09-02T05:15:46.517 DEBUG:boto:Token: None 2022-09-02T05:15:46.518 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:15:46 GMT /xljbkf-68/ 2022-09-02T05:15:46.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QORtmOGxzdaSXvKraniLdjw4278= 2022-09-02T05:15:46.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QORtmOGxzdaSXvKraniLdjw4278=', 'Content-Length': '0'} 2022-09-02T05:15:46.546 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d6168a056b4ccde-0063119182-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.547 DEBUG:boto:path=/xljbkf-68/testobj-sse-c 2022-09-02T05:15:46.547 DEBUG:boto:auth_path=/xljbkf-68/testobj-sse-c 2022-09-02T05:15:46.548 DEBUG:boto:Method: PUT 2022-09-02T05:15:46.549 DEBUG:boto:Path: /xljbkf-68/testobj-sse-c 2022-09-02T05:15:46.549 DEBUG:boto:Data: 2022-09-02T05:15:46.550 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T05:15:46.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:46.551 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.551 DEBUG:boto:Params: {} 2022-09-02T05:15:46.552 DEBUG:boto:Token: None 2022-09-02T05:15:46.553 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 02 Sep 2022 05:15:46 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /xljbkf-68/testobj-sse-c 2022-09-02T05:15:46.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h4aFHAYLNBjBVi1EJJydXtWXQ8Y= 2022-09-02T05:15:46.554 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h4aFHAYLNBjBVi1EJJydXtWXQ8Y='} 2022-09-02T05:15:46.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000766ea22758f6d7a4-0063119182-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.569 DEBUG:boto:path=/xljbkf-68/testobj-sse-kms 2022-09-02T05:15:46.569 DEBUG:boto:auth_path=/xljbkf-68/testobj-sse-kms 2022-09-02T05:15:46.570 DEBUG:boto:Method: PUT 2022-09-02T05:15:46.570 DEBUG:boto:Path: /xljbkf-68/testobj-sse-kms 2022-09-02T05:15:46.571 DEBUG:boto:Data: 2022-09-02T05:15:46.571 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2022-09-02T05:15:46.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:15:46.572 DEBUG:boto:Port: 8000 2022-09-02T05:15:46.573 DEBUG:boto:Params: {} 2022-09-02T05:15:46.574 DEBUG:boto:Token: None 2022-09-02T05:15:46.574 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 02 Sep 2022 05:15:46 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /xljbkf-68/testobj-sse-kms 2022-09-02T05:15:46.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:poc1/CMGrSATYSOTIizTvRBHAEI= 2022-09-02T05:15:46.576 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:15:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:poc1/CMGrSATYSOTIizTvRBHAEI='} 2022-09-02T05:15:46.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6eGxqYmtmLTY4L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx0000048b8c7b4939c68dd-0063119182-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:15:46.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.588 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:46.700 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:15:46.718 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:15:46.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.718 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:46.719 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:15:46.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.720 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:46.720 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:15:46.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.721 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095746.544420_360.1", 2022-09-02T05:15:46.721 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:15:46.544420Z" 2022-09-02T05:15:46.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.722 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:46.722 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:15:46.722 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:46.723 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:15:46.724 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:46.725 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:15:46.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.726 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:46.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:46.783 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:46.821 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:46.823+0000 7fb8accf6600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:46.822 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:46.824+0000 7fb8accf6600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:46.836 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:15:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:15:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:15:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:15:46.837 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:15:46.838 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:46.838 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:46.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:15:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:15:46.839 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:46.840 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:46.840 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:46.840 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:46.840 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:15:46.841 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:46.841 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:46.841 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:46.841 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:46.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:46.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:46.842 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:15:46.842 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:46.843 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:46.843 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:15:46.843 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:46.843 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:46.844 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:46.844 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:15:46.844 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:46.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:46.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:46.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:46.846 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:15:46.847 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:46.847 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:46.848 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095746.544420_360.1", 2022-09-02T05:15:46.848 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:46.849 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:46.849 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:46.849 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:15:46.544420Z", 2022-09-02T05:15:46.850 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:46.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:46.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:46.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:46.852 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:15:46.852 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:46.852 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:46.853 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:46.853 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:46.853 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:46.854 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:46.854 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:46.854 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:46.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:46.855 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:46.855 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:15:46.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:46.856 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:15:46.856 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:15:46.856 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:15:46.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:46.856 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:15:46Z" 2022-09-02T05:15:46.857 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:15:46.858 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095746.544420_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:46.544420Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:46Z" } 2022-09-02T05:15:46.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095746.544420_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:46.544420Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:46.859 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:46.860 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:46.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:15:46.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:15:46.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.862 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:46.914 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:46.914 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:46.943 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:46.945+0000 7f5d5cc29600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:46.944 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:15:46.946+0000 7f5d5cc29600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:46.956 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:15:46.956 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:15:46.957 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:15:46.957 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:15:46.957 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:15:46.957 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:46.958 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:15:46.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.958 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:15:46.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.959 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:15:46.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:46.959 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:46.959 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:46.960 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:46.960 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:15:46.960 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:46.960 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:46.961 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:15:46.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:46.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.962 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:15:46.962 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:46.962 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:46.962 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:15:46.963 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:46.963 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:15:46.963 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:46.963 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:15:46.964 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:15:46.964 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:46.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.964 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:15:46.965 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:46.965 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:46.965 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095746.544420_360.1", 2022-09-02T05:15:46.965 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:46.966 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:15:46.966 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:46.966 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:15:46.544420Z", 2022-09-02T05:15:46.966 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:15:46.967 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:46.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:15:46.967 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:15:46.968 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:15:46.968 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:15:46.968 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:46.968 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:15:46.969 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:15:46.969 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:15:46.969 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:46.969 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:15:46.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:46.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:15:46.970 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:15:46.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.971 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:15:46.971 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:15:46.971 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:15:46.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:15:46.972 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:15:46Z" 2022-09-02T05:15:46.972 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:15:46.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095746.544420_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:46.544420Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:46Z" } 2022-09-02T05:15:46.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095746.544420_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:46.544420Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:46.975 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:46.975 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:46.976 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:15:46.976 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:15:46.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:46.978 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:47.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:47.029 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:47.055 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:47.057+0000 7fd199895600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:47.056 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:47.058+0000 7fd199895600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:47.070 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:15:47.071 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:15:47.071 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:15:47.072 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:15:47.072 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:15:47.072 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:47.072 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:47.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:47.073 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:15:47.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:47.074 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:15:47.074 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:47.074 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:47.075 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:47.075 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:47.075 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:15:47.075 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:47.076 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:47.076 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:47.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:47.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:47.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:47.077 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:15:47.077 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:47.077 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:47.078 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:15:47.078 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:47.078 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:47.078 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:47.079 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:15:47.079 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:47.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:47.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:47.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:47.080 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:15:47.080 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:47.080 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:47.081 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095746.533396_357.1", 2022-09-02T05:15:47.081 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:47.082 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:47.082 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:47.082 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:15:46.533396Z", 2022-09-02T05:15:47.082 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:47.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:47.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:47.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:47.083 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:15:47.084 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:47.084 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:47.084 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:47.084 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:47.085 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:47.085 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:47.085 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:47.085 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:47.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:47.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:47.086 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:15:47.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:47.087 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:15:47.087 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:15:47.087 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:15:47.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:47.088 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:15:47Z" 2022-09-02T05:15:47.088 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:15:47.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095746.533396_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:46.533396Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:47Z" } 2022-09-02T05:15:47.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095746.533396_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:46.533396Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:47.090 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:47.091 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095746.533396_357.1', 3: ''} 2022-09-02T05:15:47.091 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662095746.544420_360.1 target=1_1662095746.533396_357.1 2022-09-02T05:15:52.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:52.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:15:52.095 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:15:52.152 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:52.152 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:52.183 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:52.185+0000 7f8eeb45a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:52.184 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:52.186+0000 7f8eeb45a600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:15:52.200 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:15:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:15:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:15:52.201 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:15:52.202 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:15:52.202 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:15:52.203 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:52.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:52.204 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:15:52.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:52.205 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:15:52.205 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:52.206 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:52.207 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:15:52.207 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:52.208 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:15:52.208 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:52.209 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:15:52.209 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:52.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:52.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:52.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:52.211 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:15:52.212 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:52.213 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:52.213 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:15:52.214 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:52.214 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:52.215 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:52.215 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:15:52.216 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:52.216 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:52.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:52.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:52.218 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:15:52.218 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:52.219 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:52.219 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095746.544420_360.1", 2022-09-02T05:15:52.220 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:52.220 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:52.221 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:52.221 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:15:46.544420Z", 2022-09-02T05:15:52.221 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:15:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:52.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:15:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:15:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:15:52.223 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:15:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:15:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:15:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:15:52.224 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:15:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:15:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:15:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:52.225 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:15:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:15:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:52.226 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:15:52.227 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:15:52.227 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:15:52.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:15:52.228 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:15:52Z" 2022-09-02T05:15:52.228 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:15:52.230 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095746.544420_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:15:46.544420Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:15:52Z" } 2022-09-02T05:15:52.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095746.544420_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:15:46.544420Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:15:52.231 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:52.232 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095746.544420_360.1', 3: ''} 2022-09-02T05:15:52.232 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:15:52.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:52.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:15:52.234 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:15:52.292 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:15:52.292 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:15:52.352 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:52.354+0000 7f680cdfe600 1 waiting to reach incremental sync.. 2022-09-02T05:15:57.353 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:15:57.355+0000 7f680cdfe600 1 waiting to reach incremental sync.. 2022-09-02T05:16:02.355 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:02.357+0000 7f680cdfe600 1 bucket sync caught up with source: 2022-09-02T05:16:02.356 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.53.5, , , , 00000000001.52.5, , , , , ] 2022-09-02T05:16:02.356 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.53.5, , , , 00000000001.52.5, , , , , ] 2022-09-02T05:16:02.357 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:02.357+0000 7f680cdfe600 0 bucket checkpoint complete 2022-09-02T05:16:02.373 DEBUG:boto:path=/xljbkf-68/ 2022-09-02T05:16:02.374 DEBUG:boto:auth_path=/xljbkf-68/ 2022-09-02T05:16:02.375 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.376 DEBUG:boto:Path: /xljbkf-68/ 2022-09-02T05:16:02.376 DEBUG:boto:Data: 2022-09-02T05:16:02.377 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.377 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:16:02.378 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.378 DEBUG:boto:Params: {} 2022-09-02T05:16:02.379 DEBUG:boto:Token: None 2022-09-02T05:16:02.380 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-68/ 2022-09-02T05:16:02.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ty7S3ErdsH2OEOy/ZsGWfVP5pZQ= 2022-09-02T05:16:02.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ty7S3ErdsH2OEOy/ZsGWfVP5pZQ=', 'Content-Length': '0'} 2022-09-02T05:16:02.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a286ff640fbdd3c-0063119192-15e3-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.385 DEBUG:boto:path=/xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.386 DEBUG:boto:auth_path=/xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.387 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.387 DEBUG:boto:Path: /xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.388 DEBUG:boto:Data: 2022-09-02T05:16:02.389 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2022-09-02T05:16:02.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:16:02.390 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.391 DEBUG:boto:Params: {} 2022-09-02T05:16:02.391 DEBUG:boto:Token: None 2022-09-02T05:16:02.392 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JVGGV0I1iiuMHoVyA+p/MAH5tWI= 2022-09-02T05:16:02.394 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JVGGV0I1iiuMHoVyA+p/MAH5tWI=', 'Content-Length': '0'} 2022-09-02T05:16:02.396 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000bc09c5fbdb25fce9-0063119192-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.397 DEBUG:boto:path=/xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.397 DEBUG:boto:auth_path=/xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.398 DEBUG:boto:Method: GET 2022-09-02T05:16:02.399 DEBUG:boto:Path: /xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.399 DEBUG:boto:Data: 2022-09-02T05:16:02.400 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2022-09-02T05:16:02.400 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:16:02.401 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.402 DEBUG:boto:Params: {} 2022-09-02T05:16:02.403 DEBUG:boto:Token: None 2022-09-02T05:16:02.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:16:02 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /xljbkf-68/testobj-sse-c 2022-09-02T05:16:02.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nAts0B4oucj1stqoqv+qLhXJDz4= 2022-09-02T05:16:02.404 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nAts0B4oucj1stqoqv+qLhXJDz4=', 'Content-Length': '0'} 2022-09-02T05:16:02.407 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000030eda9d8429d9791-0063119192-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.408 DEBUG:boto:path=/xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.409 DEBUG:boto:auth_path=/xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.409 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.410 DEBUG:boto:Path: /xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.411 DEBUG:boto:Data: 2022-09-02T05:16:02.411 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.412 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:16:02.412 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.413 DEBUG:boto:Params: {} 2022-09-02T05:16:02.413 DEBUG:boto:Token: None 2022-09-02T05:16:02.414 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1ztKfx5F7hArzYpo29qBbC/XBng= 2022-09-02T05:16:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1ztKfx5F7hArzYpo29qBbC/XBng=', 'Content-Length': '0'} 2022-09-02T05:16:02.417 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000004bb50e9788cf128b-0063119192-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.429 DEBUG:boto:path=/xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.430 DEBUG:boto:auth_path=/xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.431 DEBUG:boto:Method: GET 2022-09-02T05:16:02.431 DEBUG:boto:Path: /xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.432 DEBUG:boto:Data: 2022-09-02T05:16:02.433 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:16:02.435 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.436 DEBUG:boto:Params: {} 2022-09-02T05:16:02.438 DEBUG:boto:Token: None 2022-09-02T05:16:02.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-68/testobj-sse-kms 2022-09-02T05:16:02.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SOjXmFfrT3WITAl/0RBl9aRoLWw= 2022-09-02T05:16:02.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SOjXmFfrT3WITAl/0RBl9aRoLWw=', 'Content-Length': '0'} 2022-09-02T05:16:02.444 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:15:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000a8eeded191145787-0063119192-15e3-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2022-09-02T05:16:02.447 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-69 2022-09-02T05:16:02.448 DEBUG:boto:path=/xljbkf-69/ 2022-09-02T05:16:02.449 DEBUG:boto:auth_path=/xljbkf-69/ 2022-09-02T05:16:02.449 DEBUG:boto:Method: PUT 2022-09-02T05:16:02.450 DEBUG:boto:Path: /xljbkf-69/ 2022-09-02T05:16:02.451 DEBUG:boto:Data: 2022-09-02T05:16:02.452 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.454 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.455 DEBUG:boto:Params: {} 2022-09-02T05:16:02.456 DEBUG:boto:Token: None 2022-09-02T05:16:02.456 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/ 2022-09-02T05:16:02.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H+LT0BKegSp/BRorQTJGSZxUKr0= 2022-09-02T05:16:02.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H+LT0BKegSp/BRorQTJGSZxUKr0=', 'Content-Length': '0'} 2022-09-02T05:16:02.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8bb4c661e3f01bf-0063119192-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.480 DEBUG:boto:path=/xljbkf-69/ 2022-09-02T05:16:02.480 DEBUG:boto:auth_path=/xljbkf-69/ 2022-09-02T05:16:02.481 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.481 DEBUG:boto:Path: /xljbkf-69/ 2022-09-02T05:16:02.482 DEBUG:boto:Data: 2022-09-02T05:16:02.482 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.483 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.484 DEBUG:boto:Params: {} 2022-09-02T05:16:02.485 DEBUG:boto:Token: None 2022-09-02T05:16:02.485 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/ 2022-09-02T05:16:02.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI= 2022-09-02T05:16:02.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI=', 'Content-Length': '0'} 2022-09-02T05:16:02.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000245d490abf582521-0063119192-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.490 DEBUG:boto:path=/xljbkf-69/a 2022-09-02T05:16:02.491 DEBUG:boto:auth_path=/xljbkf-69/a 2022-09-02T05:16:02.491 DEBUG:boto:Method: PUT 2022-09-02T05:16:02.492 DEBUG:boto:Path: /xljbkf-69/a 2022-09-02T05:16:02.492 DEBUG:boto:Data: 2022-09-02T05:16:02.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:16:02.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.494 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.494 DEBUG:boto:Params: {} 2022-09-02T05:16:02.495 DEBUG:boto:Token: None 2022-09-02T05:16:02.496 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/a 2022-09-02T05:16:02.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5AUwaqZM+X24e3CHciNa36j/Wh0= 2022-09-02T05:16:02.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5AUwaqZM+X24e3CHciNa36j/Wh0='} 2022-09-02T05:16:02.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eecfaca74d7e878-0063119192-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.543 DEBUG:boto:path=/xljbkf-69/ 2022-09-02T05:16:02.543 DEBUG:boto:auth_path=/xljbkf-69/ 2022-09-02T05:16:02.544 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.544 DEBUG:boto:Path: /xljbkf-69/ 2022-09-02T05:16:02.545 DEBUG:boto:Data: 2022-09-02T05:16:02.545 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.547 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.547 DEBUG:boto:Params: {} 2022-09-02T05:16:02.548 DEBUG:boto:Token: None 2022-09-02T05:16:02.548 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/ 2022-09-02T05:16:02.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI= 2022-09-02T05:16:02.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI=', 'Content-Length': '0'} 2022-09-02T05:16:02.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010655fe09d5deadc-0063119192-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.555 DEBUG:boto:path=/xljbkf-69/b 2022-09-02T05:16:02.555 DEBUG:boto:auth_path=/xljbkf-69/b 2022-09-02T05:16:02.556 DEBUG:boto:Method: PUT 2022-09-02T05:16:02.557 DEBUG:boto:Path: /xljbkf-69/b 2022-09-02T05:16:02.557 DEBUG:boto:Data: 2022-09-02T05:16:02.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:16:02.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.559 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.559 DEBUG:boto:Params: {} 2022-09-02T05:16:02.560 DEBUG:boto:Token: None 2022-09-02T05:16:02.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/b 2022-09-02T05:16:02.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vc0hgnkDvLBgGdU6tV2ajNn7dOk= 2022-09-02T05:16:02.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vc0hgnkDvLBgGdU6tV2ajNn7dOk='} 2022-09-02T05:16:02.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbc151dad2c81bcb-0063119192-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.569 DEBUG:boto:path=/xljbkf-69/ 2022-09-02T05:16:02.570 DEBUG:boto:auth_path=/xljbkf-69/ 2022-09-02T05:16:02.570 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.571 DEBUG:boto:Path: /xljbkf-69/ 2022-09-02T05:16:02.571 DEBUG:boto:Data: 2022-09-02T05:16:02.572 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.573 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.574 DEBUG:boto:Params: {} 2022-09-02T05:16:02.574 DEBUG:boto:Token: None 2022-09-02T05:16:02.575 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/ 2022-09-02T05:16:02.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI= 2022-09-02T05:16:02.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI=', 'Content-Length': '0'} 2022-09-02T05:16:02.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad599121f12b18d4-0063119192-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.579 DEBUG:boto:path=/xljbkf-69/c 2022-09-02T05:16:02.580 DEBUG:boto:auth_path=/xljbkf-69/c 2022-09-02T05:16:02.580 DEBUG:boto:Method: PUT 2022-09-02T05:16:02.581 DEBUG:boto:Path: /xljbkf-69/c 2022-09-02T05:16:02.581 DEBUG:boto:Data: 2022-09-02T05:16:02.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:16:02.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.583 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.583 DEBUG:boto:Params: {} 2022-09-02T05:16:02.584 DEBUG:boto:Token: None 2022-09-02T05:16:02.585 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/c 2022-09-02T05:16:02.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L7CFcPm6ramxHFgUdOq5f6nYALU= 2022-09-02T05:16:02.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L7CFcPm6ramxHFgUdOq5f6nYALU='} 2022-09-02T05:16:02.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2c442381aa090be-0063119192-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.631 DEBUG:boto:path=/xljbkf-69/ 2022-09-02T05:16:02.632 DEBUG:boto:auth_path=/xljbkf-69/ 2022-09-02T05:16:02.632 DEBUG:boto:Method: HEAD 2022-09-02T05:16:02.633 DEBUG:boto:Path: /xljbkf-69/ 2022-09-02T05:16:02.633 DEBUG:boto:Data: 2022-09-02T05:16:02.634 DEBUG:boto:Headers: {} 2022-09-02T05:16:02.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.635 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.635 DEBUG:boto:Params: {} 2022-09-02T05:16:02.636 DEBUG:boto:Token: None 2022-09-02T05:16:02.636 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/ 2022-09-02T05:16:02.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI= 2022-09-02T05:16:02.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ymjd+YmxSNESOZum4CLXW5dmQGI=', 'Content-Length': '0'} 2022-09-02T05:16:02.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028305f545ec436cc-0063119192-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.641 DEBUG:boto:path=/xljbkf-69/d 2022-09-02T05:16:02.641 DEBUG:boto:auth_path=/xljbkf-69/d 2022-09-02T05:16:02.642 DEBUG:boto:Method: PUT 2022-09-02T05:16:02.642 DEBUG:boto:Path: /xljbkf-69/d 2022-09-02T05:16:02.643 DEBUG:boto:Data: 2022-09-02T05:16:02.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:16:02.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:16:02.645 DEBUG:boto:Port: 8000 2022-09-02T05:16:02.645 DEBUG:boto:Params: {} 2022-09-02T05:16:02.646 DEBUG:boto:Token: None 2022-09-02T05:16:02.646 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:16:02 GMT /xljbkf-69/d 2022-09-02T05:16:02.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FovkHwemOit4oGz9MATbojeFqto= 2022-09-02T05:16:02.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:16:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FovkHwemOit4oGz9MATbojeFqto='} 2022-09-02T05:16:02.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007bb8d57b58aeb5f-0063119192-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:16:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:16:02.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:02.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:02.656 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:16:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:16:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:16:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:16:02.791 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:16:02.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:02.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:16:02.792 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:16:02.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:02.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:02.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:16:02.793 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:16:02.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:02.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:02.794 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095762.480132_364.1", 2022-09-02T05:16:02.794 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:16:02.480132Z" 2022-09-02T05:16:02.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:02.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:02.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:16:02.795 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:16:02.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:16:02.795 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:16:02.797 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:02.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:16:02.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:02.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:02.799 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:16:02.854 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:16:02.855 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:16:02.908 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:02.910+0000 7fd689f3b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:16:02.909 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:02.911+0000 7fd689f3b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:16:02.923 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:16:02.923 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:16:02.924 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:16:02.924 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:16:02.924 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:16:02.924 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:16:02.925 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:16:02.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:02.925 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:16:02.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:02.926 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:16:02.926 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:02.926 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:16:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:16:02.927 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:16:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:16:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:02.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:02.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:02.929 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:16:02.929 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:02.930 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:02.930 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:16:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:16:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:02.931 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:16:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:16:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:02.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:02.933 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:16:02.933 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:02.933 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:02.934 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095762.480132_364.1", 2022-09-02T05:16:02.934 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:02.934 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:16:02.934 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:16:02.480132Z", 2022-09-02T05:16:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:16:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:02.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:16:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:02.936 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:02.937 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:16:02.937 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:02.937 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:16:02.937 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:02.938 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:16:02.938 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:16:02.938 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:02.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:02.939 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:16:02.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:02.939 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:16:02.940 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:16:02.940 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:16:02.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:02.941 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:16:02Z" 2022-09-02T05:16:02.941 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:16:02.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095762.480132_364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:16:02.480132Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:16:02Z" } 2022-09-02T05:16:02.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095762.480132_364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:16:02.480132Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:16:02.944 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:02.944 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:02.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:16:02.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:16:02.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:02.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:02.947 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:16:02.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:16:02.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:16:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:03.034+0000 7fd44ce1b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:16:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:03.035+0000 7fd44ce1b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:16:03.051 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:16:03.051 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:16:03.052 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:16:03.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:16:03.053 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:16:03.053 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:16:03.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:16:03.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:03.054 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:16:03.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:03.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:16:03.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:16:03.055 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:16:03.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:16:03.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:16:03.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:16:03.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:16:03.057 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:16:03.057 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:16:03.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:16:03.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:03.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:03.058 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:16:03.058 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:16:03.058 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:16:03.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:16:03.059 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:16:03.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:16:03.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:16:03.060 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:16:03.060 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:16:03.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:16:03.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:03.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:03.061 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:16:03.061 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:16:03.061 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:16:03.062 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662095762.480132_364.1", 2022-09-02T05:16:03.062 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:16:03.062 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:16:03.062 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:16:03.063 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:16:02.480132Z", 2022-09-02T05:16:03.063 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:16:03.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:16:03.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:03.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:16:03.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:16:03.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:16:03.065 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:16:03.065 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:16:03.065 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:16:03.065 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:16:03.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:16:03.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:16:03.066 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:16:03.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:16:03.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:16:03.067 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:16:03.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:03.068 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:16:03.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:16:03.068 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:16:03.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:16:03.069 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:16:03Z" 2022-09-02T05:16:03.069 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:16:03.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095762.480132_364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:16:02.480132Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:16:03Z" } 2022-09-02T05:16:03.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095762.480132_364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:16:02.480132Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:16:03.072 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:03.072 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:03.073 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:16:03.073 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:16:03.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:03.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:16:03.075 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:16:03.125 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:16:03.126 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:16:03.152 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:03.154+0000 7fa8f4e38600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:16:03.153 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:03.155+0000 7fa8f4e38600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:16:03.165 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:16:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:16:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:16:03.166 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:16:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:16:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:16:03.167 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:16:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:16:03.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:16:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:03.169 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:16:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:16:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:03.170 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:16:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:16:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:03.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:16:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:03.172 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:16:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:16:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:03.173 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:16:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:16:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:03.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:16:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:03.175 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662095762.480132_364.1", 2022-09-02T05:16:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:16:03.176 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:16:02.480132Z", 2022-09-02T05:16:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:16:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:03.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:16:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:16:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:16:03.178 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:16:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:16:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:16:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:16:03.179 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:16:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:16:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:16:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:03.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:16:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:16:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:16:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:16:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:16:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:16:03.182 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:16:03Z" 2022-09-02T05:16:03.182 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:16:03.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662095762.480132_364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:16:02.480132Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:16:03Z" } 2022-09-02T05:16:03.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662095762.480132_364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:16:02.480132Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:16:03.185 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:03.186 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662095762.480132_364.1', 3: ''} 2022-09-02T05:16:03.186 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:16:03.187 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-69 2022-09-02T05:16:03.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:16:03.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:16:03.188 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:16:03.244 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:16:03.245 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:16:03.282 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:03.284+0000 7f7b5a624600 1 waiting to reach incremental sync.. 2022-09-02T05:16:08.283 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:08.285+0000 7f7b5a624600 1 waiting to reach incremental sync.. 2022-09-02T05:16:13.284 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:13.285+0000 7f7b5a624600 1 waiting to reach incremental sync.. 2022-09-02T05:16:18.284 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:18.286+0000 7f7b5a624600 1 waiting to reach incremental sync.. 2022-09-02T05:16:23.286 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:23.288+0000 7f7b5a624600 1 bucket sync caught up with source: 2022-09-02T05:16:23.287 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:16:23.287 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:16:23.287 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:16:23.288+0000 7f7b5a624600 0 bucket checkpoint complete 2022-09-02T05:16:23.304 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-69 2022-09-02T05:16:23.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:16:23.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:16:23.307 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:16:23.371 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:16:23.371 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:16:23.426 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:23.428+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:28.427 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:28.429+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:33.427 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:33.429+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:38.428 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:38.430+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:43.429 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:43.431+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:48.429 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:48.432+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:53.430 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:53.432+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:16:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:16:58.433+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:03.433+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:08.432 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:08.434+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:13.433 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:13.435+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:18.433 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:18.435+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:23.436+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:28.435 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:28.437+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:33.435 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:33.437+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:38.436 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:38.438+0000 7f6bb651f600 1 waiting to reach incremental sync.. 2022-09-02T05:17:43.438 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:43.440+0000 7f6bb651f600 1 bucket sync caught up with source: 2022-09-02T05:17:43.438 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:43.439 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:43.439 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:17:43.440+0000 7f6bb651f600 0 bucket checkpoint complete 2022-09-02T05:17:43.455 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-69 2022-09-02T05:17:43.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:17:43.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:17:43.458 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:17:43.521 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:17:43.522 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:17:43.560 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:17:43.563+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:17:43.561 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:43.561 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:48.562 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:17:48.565+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:17:48.562 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:48.563 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:53.563 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:17:53.565+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:17:53.564 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:53.564 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:17:58.566+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:17:58.566 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:17:58.566 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:03.566 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:03.568+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:03.566 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:03.567 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:08.567 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:08.569+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:08.568 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:08.568 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:13.569 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:13.570+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:13.569 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:13.570 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:18.572+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:18.571 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:18.571 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:23.573 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:23.574+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:23.573 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:23.573 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:28.573 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:28.575+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:28.574 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:28.574 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:33.575 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:33.577+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:33.576 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:33.576 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:38.576 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:38.578+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:38.577 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:38.577 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:43.578 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:43.579+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:43.578 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:43.578 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:48.579 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:48.582+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:48.580 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:48.580 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:53.581 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:53.583+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:53.581 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:53.581 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:58.582 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:18:58.584+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:18:58.583 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:18:58.583 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:03.584 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:03.585+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:03.584 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:03.585 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:08.585 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:08.587+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:08.586 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:08.586 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:13.587 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:13.589+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:13.588 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:13.588 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:18.589 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:18.590+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:18.589 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:18.590 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:23.590 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:23.592+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:23.591 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:23.592 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:28.592 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:28.593+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:28.592 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:28.593 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:33.593 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:33.594+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:33.594 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:33.594 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:38.594 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:38.596+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:38.595 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:38.595 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:43.595 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:43.597+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:43.596 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:43.596 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:48.596 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:48.598+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:48.597 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:48.598 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:53.598 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:53.601+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:53.599 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:53.600 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:58.599 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:19:58.602+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:19:58.600 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:19:58.600 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:03.601 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:03.604+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:03.602 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:03.602 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:08.602 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:08.604+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:08.603 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:08.603 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:13.604 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:13.606+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:13.605 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:13.605 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:18.605 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:18.607+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:18.606 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:18.606 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:23.607 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:23.609+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:23.607 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:23.608 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:28.608 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:28.610+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:28.609 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:28.609 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:33.610 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:33.612+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:33.611 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:33.611 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:38.612 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:38.613+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:38.612 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:38.613 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:43.614 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:43.615+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:43.614 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:43.615 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:48.614 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:48.616+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:48.615 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:48.615 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:53.616 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:53.617+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:53.616 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:53.616 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:58.617 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:20:58.620+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:20:58.618 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:20:58.618 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:03.619 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:03.621+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:03.619 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:03.620 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:08.620 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:08.623+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:08.621 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:08.621 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:13.621 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:13.623+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:13.622 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:13.622 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:18.624+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:18.624 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:18.624 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:23.624 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:23.626+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:23.625 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:23.625 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:28.625 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:28.627+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:28.626 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:28.626 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:33.627 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:33.629+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:33.627 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:33.628 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:38.628 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:38.630+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:38.629 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:38.629 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:43.630 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:43.631+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:43.631 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:43.631 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:48.631 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:48.633+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:48.631 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:48.632 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:53.632 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:53.634+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:53.633 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:53.633 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:58.634 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:21:58.636+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:21:58.634 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:21:58.635 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:03.636 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:03.637+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:03.636 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:03.637 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:08.637 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:08.640+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:08.637 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:08.638 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:13.639 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:13.641+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:13.639 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:13.640 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:18.640 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:18.642+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:18.641 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:18.641 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:23.641 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:23.643+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:23.642 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:23.642 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:28.642 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:28.644+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:28.643 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:28.643 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:33.644 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:33.646+0000 7f4dadcd5600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:33.645 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.36.5, , , , , , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:33.645 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.36.5, , , , 00000000001.35.5, , , 00000000001.56.5, , 00000000001.56.5] 2022-09-02T05:22:38.646 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:38.647+0000 7f4dadcd5600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T05:22:38.646 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:38.647+0000 7f4dadcd5600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:22:38.647 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:38.647+0000 7f4dadcd5600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:22:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T05:22:38.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2022-09-02T05:22:38.688 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xljbkf-70 2022-09-02T05:22:38.689 DEBUG:boto:path=/xljbkf-70/ 2022-09-02T05:22:38.690 DEBUG:boto:auth_path=/xljbkf-70/ 2022-09-02T05:22:38.691 DEBUG:boto:Method: PUT 2022-09-02T05:22:38.692 DEBUG:boto:Path: /xljbkf-70/ 2022-09-02T05:22:38.693 DEBUG:boto:Data: 2022-09-02T05:22:38.694 DEBUG:boto:Headers: {} 2022-09-02T05:22:38.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.695 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.696 DEBUG:boto:Params: {} 2022-09-02T05:22:38.697 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:22:38.698 DEBUG:boto:Token: None 2022-09-02T05:22:38.699 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/ 2022-09-02T05:22:38.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hPZHFGsIcD8Cl81jZDuj5sJPyiI= 2022-09-02T05:22:38.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hPZHFGsIcD8Cl81jZDuj5sJPyiI=', 'Content-Length': '0'} 2022-09-02T05:22:38.724 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000035cad018d51dacb3-006311931e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.725 DEBUG:boto:path=/xljbkf-70/ 2022-09-02T05:22:38.726 DEBUG:boto:auth_path=/xljbkf-70/ 2022-09-02T05:22:38.726 DEBUG:boto:Method: HEAD 2022-09-02T05:22:38.727 DEBUG:boto:Path: /xljbkf-70/ 2022-09-02T05:22:38.728 DEBUG:boto:Data: 2022-09-02T05:22:38.728 DEBUG:boto:Headers: {} 2022-09-02T05:22:38.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.730 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.731 DEBUG:boto:Params: {} 2022-09-02T05:22:38.732 DEBUG:boto:Token: None 2022-09-02T05:22:38.733 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/ 2022-09-02T05:22:38.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk= 2022-09-02T05:22:38.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk=', 'Content-Length': '0'} 2022-09-02T05:22:38.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000734f85268620ef26-006311931e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.740 DEBUG:boto:path=/xljbkf-70/a 2022-09-02T05:22:38.741 DEBUG:boto:auth_path=/xljbkf-70/a 2022-09-02T05:22:38.741 DEBUG:boto:Method: PUT 2022-09-02T05:22:38.742 DEBUG:boto:Path: /xljbkf-70/a 2022-09-02T05:22:38.742 DEBUG:boto:Data: 2022-09-02T05:22:38.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:22:38.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.744 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.744 DEBUG:boto:Params: {} 2022-09-02T05:22:38.745 DEBUG:boto:Token: None 2022-09-02T05:22:38.746 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/a 2022-09-02T05:22:38.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9c2T3lBLDs6KtNrsYGq17Q4H5Lg= 2022-09-02T05:22:38.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9c2T3lBLDs6KtNrsYGq17Q4H5Lg='} 2022-09-02T05:22:38.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f0b9de00b26fce5-006311931e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.755 DEBUG:boto:path=/xljbkf-70/ 2022-09-02T05:22:38.756 DEBUG:boto:auth_path=/xljbkf-70/ 2022-09-02T05:22:38.756 DEBUG:boto:Method: HEAD 2022-09-02T05:22:38.757 DEBUG:boto:Path: /xljbkf-70/ 2022-09-02T05:22:38.757 DEBUG:boto:Data: 2022-09-02T05:22:38.758 DEBUG:boto:Headers: {} 2022-09-02T05:22:38.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.759 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.760 DEBUG:boto:Params: {} 2022-09-02T05:22:38.761 DEBUG:boto:Token: None 2022-09-02T05:22:38.762 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/ 2022-09-02T05:22:38.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk= 2022-09-02T05:22:38.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk=', 'Content-Length': '0'} 2022-09-02T05:22:38.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066ebfebc8393c9cd-006311931e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.769 DEBUG:boto:path=/xljbkf-70/b 2022-09-02T05:22:38.770 DEBUG:boto:auth_path=/xljbkf-70/b 2022-09-02T05:22:38.770 DEBUG:boto:Method: PUT 2022-09-02T05:22:38.771 DEBUG:boto:Path: /xljbkf-70/b 2022-09-02T05:22:38.772 DEBUG:boto:Data: 2022-09-02T05:22:38.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:22:38.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.773 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.773 DEBUG:boto:Params: {} 2022-09-02T05:22:38.774 DEBUG:boto:Token: None 2022-09-02T05:22:38.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/b 2022-09-02T05:22:38.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Tsm6ckrwQQUMGWhTRMTcO8cCL5M= 2022-09-02T05:22:38.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Tsm6ckrwQQUMGWhTRMTcO8cCL5M='} 2022-09-02T05:22:38.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1bcd758c2cc325a-006311931e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.820 DEBUG:boto:path=/xljbkf-70/ 2022-09-02T05:22:38.821 DEBUG:boto:auth_path=/xljbkf-70/ 2022-09-02T05:22:38.822 DEBUG:boto:Method: HEAD 2022-09-02T05:22:38.822 DEBUG:boto:Path: /xljbkf-70/ 2022-09-02T05:22:38.823 DEBUG:boto:Data: 2022-09-02T05:22:38.823 DEBUG:boto:Headers: {} 2022-09-02T05:22:38.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.824 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.825 DEBUG:boto:Params: {} 2022-09-02T05:22:38.826 DEBUG:boto:Token: None 2022-09-02T05:22:38.826 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/ 2022-09-02T05:22:38.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk= 2022-09-02T05:22:38.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk=', 'Content-Length': '0'} 2022-09-02T05:22:38.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1998f120a9ce846-006311931e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.831 DEBUG:boto:path=/xljbkf-70/c 2022-09-02T05:22:38.831 DEBUG:boto:auth_path=/xljbkf-70/c 2022-09-02T05:22:38.832 DEBUG:boto:Method: PUT 2022-09-02T05:22:38.833 DEBUG:boto:Path: /xljbkf-70/c 2022-09-02T05:22:38.833 DEBUG:boto:Data: 2022-09-02T05:22:38.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:22:38.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.835 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.835 DEBUG:boto:Params: {} 2022-09-02T05:22:38.836 DEBUG:boto:Token: None 2022-09-02T05:22:38.837 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/c 2022-09-02T05:22:38.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5BZr0pJdVDIdiKXzdzqQLRmgfWk= 2022-09-02T05:22:38.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5BZr0pJdVDIdiKXzdzqQLRmgfWk='} 2022-09-02T05:22:38.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046ef6e0c068742d3-006311931e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.848 DEBUG:boto:path=/xljbkf-70/ 2022-09-02T05:22:38.849 DEBUG:boto:auth_path=/xljbkf-70/ 2022-09-02T05:22:38.849 DEBUG:boto:Method: HEAD 2022-09-02T05:22:38.850 DEBUG:boto:Path: /xljbkf-70/ 2022-09-02T05:22:38.851 DEBUG:boto:Data: 2022-09-02T05:22:38.852 DEBUG:boto:Headers: {} 2022-09-02T05:22:38.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.853 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.853 DEBUG:boto:Params: {} 2022-09-02T05:22:38.854 DEBUG:boto:Token: None 2022-09-02T05:22:38.854 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/ 2022-09-02T05:22:38.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk= 2022-09-02T05:22:38.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Me3nOzOubiwJIKFURrpKa1ZFQqk=', 'Content-Length': '0'} 2022-09-02T05:22:38.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000400a069f244a8f3e-006311931e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.859 DEBUG:boto:path=/xljbkf-70/d 2022-09-02T05:22:38.860 DEBUG:boto:auth_path=/xljbkf-70/d 2022-09-02T05:22:38.860 DEBUG:boto:Method: PUT 2022-09-02T05:22:38.861 DEBUG:boto:Path: /xljbkf-70/d 2022-09-02T05:22:38.861 DEBUG:boto:Data: 2022-09-02T05:22:38.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:22:38.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:22:38.863 DEBUG:boto:Port: 8000 2022-09-02T05:22:38.863 DEBUG:boto:Params: {} 2022-09-02T05:22:38.864 DEBUG:boto:Token: None 2022-09-02T05:22:38.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:22:38 GMT /xljbkf-70/d 2022-09-02T05:22:38.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+A8m2yGUqQIvK4FZ02Z3+xrPKk= 2022-09-02T05:22:38.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:22:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+A8m2yGUqQIvK4FZ02Z3+xrPKk='} 2022-09-02T05:22:38.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035790b1765aaf570-006311931e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:22:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:22:38.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:38.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:38.916 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:22:38.968 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:38.969 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:39.016 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:22:39.038 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:22:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:22:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:22:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:22:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:22:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096158.724435_371.1", 2022-09-02T05:22:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:22:38.724435Z" 2022-09-02T05:22:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:22:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:22:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:22:39.046 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:22:39.048 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:22:39.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:39.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:39.054 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:22:39.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:39.113 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:39.168 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:39.170+0000 7f0227cef600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:22:39.169 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:39.171+0000 7f0227cef600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:22:39.194 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:22:39.194 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:22:39.195 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:22:39.195 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:22:39.195 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:22:39.195 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:22:39.196 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:22:39.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.197 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:22:39.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.197 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:22:39.198 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.198 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.198 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:22:39.198 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.199 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:22:39.200 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.201 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:22:39.201 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:22:39.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.203 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:22:39.203 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.205 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:22:39.205 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.205 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:22:39.206 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.206 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:22:39.207 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:22:39.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.209 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:22:39.209 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.210 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.210 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096158.724435_371.1", 2022-09-02T05:22:39.211 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.212 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:22:39.212 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.212 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:22:38.724435Z", 2022-09-02T05:22:39.213 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:22:39.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.215 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:22:39.215 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.216 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.216 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:22:39.216 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.216 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:22:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:22:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:22:39.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:22:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.218 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:22:39.219 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:22:39.219 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:22:39.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.220 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:22:39Z" 2022-09-02T05:22:39.220 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:22:39.220 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096158.724435_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:22:38.724435Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:22:39Z" } 2022-09-02T05:22:39.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096158.724435_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:22:38.724435Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:22:39.222 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.222 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:22:39.224 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:22:39.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:39.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:39.225 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:22:39.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:39.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:39.304 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:22:39.306+0000 7fef5b185600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:22:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:22:39.307+0000 7fef5b185600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:22:39.317 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:22:39.318 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:22:39.318 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:22:39.318 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:22:39.319 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:22:39.319 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:22:39.319 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:22:39.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.320 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:22:39.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.320 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:22:39.320 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:22:39.321 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:22:39.321 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:22:39.322 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:22:39.322 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:22:39.323 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:22:39.323 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:22:39.324 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:22:39.324 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:22:39.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:22:39.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:22:39.326 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:22:39.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:22:39.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:22:39.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:22:39.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:22:39.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:22:39.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:22:39.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:22:39.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:22:39.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:22:39.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:22:39.332 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096158.724435_371.1", 2022-09-02T05:22:39.332 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:22:39.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:22:39.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:22:39.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:22:38.724435Z", 2022-09-02T05:22:39.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:22:39.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:22:39.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:22:39.336 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:22:39.337 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:22:39.337 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:22:39.337 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:22:39.338 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:22:39.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:22:39.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:22:39.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:22:39.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:22:39.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:22:39.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:22:39.342 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:22:39.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.343 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:22:39.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:22:39.343 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:22:39.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:22:39.344 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:22:39Z" 2022-09-02T05:22:39.345 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:22:39.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096158.724435_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:22:38.724435Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:22:39Z" } 2022-09-02T05:22:39.348 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096158.724435_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:22:38.724435Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:22:39.349 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.349 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.350 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:22:39.350 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:22:39.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:39.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:22:39.352 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:22:39.402 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:39.403 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:39.449 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:39.451+0000 7fb9b1c02600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:22:39.450 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:39.452+0000 7fb9b1c02600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:22:39.462 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:22:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:22:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:22:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:22:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:22:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:22:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:22:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:22:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:22:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:22:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:22:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:22:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:22:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:22:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:22:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:22:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:22:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:22:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:22:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096158.724435_371.1", 2022-09-02T05:22:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:22:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:22:38.724435Z", 2022-09-02T05:22:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:22:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:22:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:22:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:22:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:22:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:22:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:22:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:22:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:22:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:22:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:22:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:22:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:22:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:22:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:22:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:22:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:22:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:22:39Z" 2022-09-02T05:22:39.486 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:22:39.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096158.724435_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:22:38.724435Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:22:39Z" } 2022-09-02T05:22:39.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096158.724435_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:22:38.724435Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:22:39.489 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.490 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096158.724435_371.1', 3: ''} 2022-09-02T05:22:39.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:22:39.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-70 2022-09-02T05:22:39.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:22:39.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:22:39.493 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:22:39.549 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:39.550 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:39.586 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:39.588+0000 7f80100aa600 1 waiting to reach incremental sync.. 2022-09-02T05:22:44.588 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:44.589+0000 7f80100aa600 1 bucket sync caught up with source: 2022-09-02T05:22:44.589 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:44.589 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:44.590 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:44.590+0000 7f80100aa600 0 bucket checkpoint complete 2022-09-02T05:22:44.609 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-70 2022-09-02T05:22:44.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:22:44.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:22:44.611 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:22:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:44.720 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:22:44.722+0000 7f8bf9a49600 1 bucket sync caught up with source: 2022-09-02T05:22:44.721 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:44.721 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:44.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:22:44.722+0000 7f8bf9a49600 0 bucket checkpoint complete 2022-09-02T05:22:44.734 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-70 2022-09-02T05:22:44.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:22:44.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:22:44.736 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:22:44.799 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:22:44.799 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:22:44.849 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:44.851+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:44.850 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:44.850 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:49.851 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:49.853+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:49.852 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:49.853 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:54.853 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:54.854+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:54.853 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:54.853 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:59.858 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:22:59.856+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:22:59.859 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:22:59.859 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:04.856 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:04.857+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:04.857 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:04.857 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:09.859+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:09.857 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:09.858 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:14.858 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:14.861+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:14.859 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:14.859 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:19.859 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:19.862+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:19.860 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:19.860 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:24.861 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:24.863+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:24.861 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:24.861 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:29.863 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:29.864+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:29.863 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:29.863 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:34.864 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:34.865+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:34.864 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:34.865 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:39.865 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:39.867+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:39.866 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:39.866 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:44.867 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:44.869+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:44.867 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:44.868 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:49.868 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:49.870+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:49.869 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:49.869 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:54.870 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:54.872+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:54.871 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:54.871 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:59.873 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:23:59.873+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:23:59.873 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:23:59.873 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:04.873 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:04.875+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:04.873 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:04.873 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:09.874 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:09.876+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:09.875 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:09.875 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:14.875 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:14.877+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:14.876 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:14.876 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:19.878 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:19.880+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:19.878 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:19.879 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:24.879 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:24.881+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:24.879 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:24.880 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:29.880 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:29.882+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:29.880 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:29.881 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:34.881 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:34.883+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:34.881 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:34.882 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:39.883 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:39.884+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:39.883 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:39.884 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:44.884 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:44.886+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:44.885 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:44.885 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:49.885 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:49.887+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:49.886 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:49.886 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:54.887 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:54.888+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:54.888 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:54.888 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:59.888 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:24:59.890+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:24:59.889 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:24:59.889 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:04.889 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:04.891+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:04.890 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:04.890 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:09.891 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:09.893+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:09.891 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:09.891 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:14.892 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:14.894+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:14.893 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:14.893 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:19.893 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:19.895+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:19.894 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:19.894 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:24.895 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:24.898+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:24.895 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:24.896 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:29.896 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:29.899+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:29.896 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:29.897 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:34.898 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:34.900+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:34.898 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:34.899 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:39.900 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:39.901+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:39.900 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:39.901 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:44.901 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:44.902+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:44.901 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:44.902 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:49.902 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:49.904+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:49.903 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:49.903 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:54.904 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:54.905+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:54.904 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:54.904 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:59.905 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:25:59.907+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:25:59.905 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:25:59.906 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:04.906 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:04.908+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:04.907 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:04.907 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:09.908 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:09.909+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:09.908 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:09.908 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:14.910 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:14.911+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:14.910 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:14.911 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:19.910 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:19.912+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:19.911 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:19.911 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:24.912 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:24.914+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:24.913 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:24.913 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:29.913 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:29.916+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:29.914 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:29.914 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:34.915 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:34.918+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:34.915 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:34.915 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:39.918 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:39.918+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:39.919 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:39.919 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:44.917 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:44.919+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:44.918 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:44.918 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:49.919 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:49.921+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:49.919 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:49.920 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:54.920 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:54.922+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:54.920 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:54.921 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:59.921 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:26:59.923+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:26:59.922 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:26:59.922 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:04.923 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:04.925+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:04.924 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:04.924 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:09.924 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:09.926+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:09.925 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:09.926 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:14.925 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:14.927+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:14.926 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:14.926 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:19.927 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:19.929+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:19.928 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:19.928 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:24.929 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:24.930+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:24.929 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:24.930 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:29.930 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:29.932+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:29.931 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:29.932 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:34.932 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:34.934+0000 7f62a24d6600 1 waiting for incremental sync to catch up: 2022-09-02T05:27:34.932 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:34.933 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.91.5, , , 00000000001.43.5, , 00000000001.50.5] 2022-09-02T05:27:39.933 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:39.936+0000 7f62a24d6600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T05:27:39.934 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:39.936+0000 7f62a24d6600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:27:39.934 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:39.936+0000 7f62a24d6600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:27:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T05:27:39.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2022-09-02T05:27:39.956 DEBUG:boto:path=/xljbkf-71/ 2022-09-02T05:27:39.956 DEBUG:boto:auth_path=/xljbkf-71/ 2022-09-02T05:27:39.958 DEBUG:boto:Method: PUT 2022-09-02T05:27:39.959 DEBUG:boto:Path: /xljbkf-71/ 2022-09-02T05:27:39.960 DEBUG:boto:Data: 2022-09-02T05:27:39.960 DEBUG:boto:Headers: {} 2022-09-02T05:27:39.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:39.961 DEBUG:boto:Port: 8000 2022-09-02T05:27:39.962 DEBUG:boto:Params: {} 2022-09-02T05:27:39.963 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:27:39.964 DEBUG:boto:Token: None 2022-09-02T05:27:39.965 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:27:39 GMT /xljbkf-71/ 2022-09-02T05:27:39.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k7ZnuUwOijQZsZQvbVOy30R6JTY= 2022-09-02T05:27:39.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:27:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k7ZnuUwOijQZsZQvbVOy30R6JTY=', 'Content-Length': '0'} 2022-09-02T05:27:39.995 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000040342705bf0c608a-006311944b-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:27:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:39.996 DEBUG:rgw_multi.tests:created bucket=xljbkf-71 2022-09-02T05:27:39.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:39.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:39.999 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:27:40.058 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:27:40.059 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:27:40.088 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:27:40.105 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:27:40.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:40.106 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:27:40.107 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:27:40.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:40.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:40.108 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:27:40.108 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:27:40.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:40.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:40.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096459.995239_380.1", 2022-09-02T05:27:40.109 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:27:39.995239Z" 2022-09-02T05:27:40.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:40.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:40.110 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:27:40.110 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:27:40.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:27:40.111 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:27:40.113 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:40.113 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:27:40.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:40.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:40.116 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:27:40.173 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:27:40.173 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:27:40.212 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:40.214+0000 7f126c3c9600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:40.213 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:40.216+0000 7f126c3c9600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:40.247 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:27:40.248 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:27:40.248 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:27:40.249 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:27:40.249 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:27:40.249 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:27:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:27:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:27:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:27:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:27:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:27:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:40.254 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:27:40.254 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:40.255 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:40.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:40.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:40.257 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:27:40.257 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:40.258 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:40.258 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:27:40.259 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:40.259 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:40.260 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:40.260 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:27:40.261 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:40.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:40.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:40.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:40.263 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:27:40.263 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:40.264 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:40.264 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096459.985154_377.1", 2022-09-02T05:27:40.265 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:40.265 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:40.266 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:40.266 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:27:39.985154Z", 2022-09-02T05:27:40.267 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:27:40.267 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:40.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:40.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:40.269 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:27:40.269 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:40.270 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:40.270 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:27:40.271 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:40.271 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:40.272 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:40.272 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:27:40.273 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:40.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:40.274 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:40.274 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:27:40.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:40.275 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:27:40.276 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:27:40.276 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:27:40.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:40.277 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:27:40Z" 2022-09-02T05:27:40.278 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:27:40.280 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096459.985154_377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:27:39.985154Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:27:40Z" } 2022-09-02T05:27:40.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096459.985154_377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:27:39.985154Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:27:40.282 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:40.283 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096459.985154_377.1', 3: ''} 2022-09-02T05:27:40.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662096459.995239_380.1 target=1_1662096459.985154_377.1 2022-09-02T05:27:45.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:45.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:45.287 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:27:45.345 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:27:45.346 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:27:45.383 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:45.385+0000 7f8bb83f0600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:45.384 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:45.386+0000 7f8bb83f0600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:45.399 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:27:45.399 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:27:45.400 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:27:45.400 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:27:45.400 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:27:45.400 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:27:45.401 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:27:45.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.401 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:27:45.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.402 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:27:45.402 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.402 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.402 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:27:45.403 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.403 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:27:45.403 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.403 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:27:45.404 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:45.404 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.405 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:27:45.405 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.405 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.405 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.407 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:27:45.407 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:45.407 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.408 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:27:45.408 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.408 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.409 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096459.995239_380.1", 2022-09-02T05:27:45.409 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.409 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:45.409 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.410 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:27:39.995239Z", 2022-09-02T05:27:45.410 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:27:45.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.411 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:27:45.411 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.411 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.412 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:27:45.412 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.412 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:45.412 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.413 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:27:45.413 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:45.413 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.413 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.414 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:27:45.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.414 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:27:45.414 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:27:45.415 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:27:45.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.415 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:27:45Z" 2022-09-02T05:27:45.415 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:27:45.417 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096459.995239_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:27:39.995239Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:27:45Z" } 2022-09-02T05:27:45.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096459.995239_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:27:39.995239Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:27:45.418 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:45.418 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:45.419 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:27:45.419 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:27:45.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:45.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:45.421 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:27:45.478 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:27:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:27:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:27:45.529+0000 7f084f96b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:45.528 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:27:45.530+0000 7f084f96b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:45.548 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:27:45.549 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:27:45.549 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:27:45.549 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:27:45.550 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:27:45.550 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:27:45.550 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:27:45.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:45.551 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:27:45.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:45.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:27:45.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:27:45.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:27:45.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:27:45.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:27:45.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:27:45.554 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:27:45.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:27:45.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:27:45.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:27:45.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:45.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:45.555 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:27:45.556 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:27:45.556 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:27:45.556 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:27:45.557 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:27:45.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:27:45.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:27:45.558 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:27:45.558 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:27:45.559 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:27:45.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:45.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:45.560 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:27:45.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:27:45.561 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:27:45.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096459.995239_380.1", 2022-09-02T05:27:45.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:27:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:27:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:27:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:27:39.995239Z", 2022-09-02T05:27:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:27:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:27:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:27:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:27:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:27:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:27:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:27:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:27:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:27:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:27:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:27:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:27:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:27:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:27:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:27:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:27:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:27:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:27:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:27:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:27:45Z" 2022-09-02T05:27:45.571 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:27:45.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096459.995239_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:27:39.995239Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:27:45Z" } 2022-09-02T05:27:45.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096459.995239_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:27:39.995239Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:27:45.574 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:45.575 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:45.576 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:27:45.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:27:45.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:45.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:27:45.579 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:27:45.632 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:27:45.633 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:27:45.659 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:45.662+0000 7fbd82250600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:45.660 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:45.663+0000 7fbd82250600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:27:45.672 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:27:45.673 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:27:45.673 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:27:45.673 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:27:45.674 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:27:45.674 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:27:45.675 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:27:45.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.675 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:27:45.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.676 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:27:45.677 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.677 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.677 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:27:45.678 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.678 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:27:45.679 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.679 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:27:45.680 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:45.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.682 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:27:45.682 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.683 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.683 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:27:45.684 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.684 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:45.685 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.685 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:27:45.686 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:45.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.687 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:27:45.688 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.688 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.689 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096459.995239_380.1", 2022-09-02T05:27:45.689 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.689 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:45.689 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.690 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:27:39.995239Z", 2022-09-02T05:27:45.690 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:27:45.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:27:45.691 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:27:45.691 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:27:45.691 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:27:45.692 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:27:45.692 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:27:45.692 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:27:45.692 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:27:45.693 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:27:45.693 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:27:45.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:27:45.694 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:27:45.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.694 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:27:45.694 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:27:45.695 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:27:45.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:27:45.695 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:27:45Z" 2022-09-02T05:27:45.695 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:27:45.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096459.995239_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:27:39.995239Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:27:45Z" } 2022-09-02T05:27:45.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096459.995239_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:27:39.995239Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:27:45.698 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:45.698 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096459.995239_380.1', 3: ''} 2022-09-02T05:27:45.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:27:45.699 DEBUG:boto:path=/xljbkf-71/ 2022-09-02T05:27:45.700 DEBUG:boto:auth_path=/xljbkf-71/ 2022-09-02T05:27:45.701 DEBUG:boto:Method: HEAD 2022-09-02T05:27:45.702 DEBUG:boto:Path: /xljbkf-71/ 2022-09-02T05:27:45.703 DEBUG:boto:Data: 2022-09-02T05:27:45.704 DEBUG:boto:Headers: {} 2022-09-02T05:27:45.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.705 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.706 DEBUG:boto:Params: {} 2022-09-02T05:27:45.708 DEBUG:boto:Token: None 2022-09-02T05:27:45.709 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/ 2022-09-02T05:27:45.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk= 2022-09-02T05:27:45.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk=', 'Content-Length': '0'} 2022-09-02T05:27:45.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2498847ca0152f1-0063119451-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.714 DEBUG:boto:path=/xljbkf-71/a 2022-09-02T05:27:45.715 DEBUG:boto:auth_path=/xljbkf-71/a 2022-09-02T05:27:45.716 DEBUG:boto:Method: PUT 2022-09-02T05:27:45.716 DEBUG:boto:Path: /xljbkf-71/a 2022-09-02T05:27:45.717 DEBUG:boto:Data: 2022-09-02T05:27:45.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:27:45.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.718 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.719 DEBUG:boto:Params: {} 2022-09-02T05:27:45.719 DEBUG:boto:Token: None 2022-09-02T05:27:45.720 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/a 2022-09-02T05:27:45.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p2BA1FXt833XJbCoBDAWz0D/gGQ= 2022-09-02T05:27:45.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p2BA1FXt833XJbCoBDAWz0D/gGQ='} 2022-09-02T05:27:45.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6769477ec4421a1-0063119451-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.729 DEBUG:boto:path=/xljbkf-71/ 2022-09-02T05:27:45.730 DEBUG:boto:auth_path=/xljbkf-71/ 2022-09-02T05:27:45.730 DEBUG:boto:Method: HEAD 2022-09-02T05:27:45.731 DEBUG:boto:Path: /xljbkf-71/ 2022-09-02T05:27:45.731 DEBUG:boto:Data: 2022-09-02T05:27:45.732 DEBUG:boto:Headers: {} 2022-09-02T05:27:45.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.733 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.733 DEBUG:boto:Params: {} 2022-09-02T05:27:45.734 DEBUG:boto:Token: None 2022-09-02T05:27:45.734 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/ 2022-09-02T05:27:45.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk= 2022-09-02T05:27:45.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk=', 'Content-Length': '0'} 2022-09-02T05:27:45.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000841e4a8d30ea06d1-0063119451-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.739 DEBUG:boto:path=/xljbkf-71/b 2022-09-02T05:27:45.739 DEBUG:boto:auth_path=/xljbkf-71/b 2022-09-02T05:27:45.740 DEBUG:boto:Method: PUT 2022-09-02T05:27:45.740 DEBUG:boto:Path: /xljbkf-71/b 2022-09-02T05:27:45.741 DEBUG:boto:Data: 2022-09-02T05:27:45.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:27:45.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.742 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.743 DEBUG:boto:Params: {} 2022-09-02T05:27:45.744 DEBUG:boto:Token: None 2022-09-02T05:27:45.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/b 2022-09-02T05:27:45.745 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K8/mDHb03HQ4Wh+xLjM5mlFjvPk= 2022-09-02T05:27:45.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K8/mDHb03HQ4Wh+xLjM5mlFjvPk='} 2022-09-02T05:27:45.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ac8ec8b4f218903-0063119451-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.753 DEBUG:boto:path=/xljbkf-71/ 2022-09-02T05:27:45.754 DEBUG:boto:auth_path=/xljbkf-71/ 2022-09-02T05:27:45.755 DEBUG:boto:Method: HEAD 2022-09-02T05:27:45.755 DEBUG:boto:Path: /xljbkf-71/ 2022-09-02T05:27:45.756 DEBUG:boto:Data: 2022-09-02T05:27:45.756 DEBUG:boto:Headers: {} 2022-09-02T05:27:45.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.757 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.758 DEBUG:boto:Params: {} 2022-09-02T05:27:45.758 DEBUG:boto:Token: None 2022-09-02T05:27:45.759 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/ 2022-09-02T05:27:45.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk= 2022-09-02T05:27:45.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk=', 'Content-Length': '0'} 2022-09-02T05:27:45.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082bb5203251f8f81-0063119451-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.764 DEBUG:boto:path=/xljbkf-71/c 2022-09-02T05:27:45.765 DEBUG:boto:auth_path=/xljbkf-71/c 2022-09-02T05:27:45.765 DEBUG:boto:Method: PUT 2022-09-02T05:27:45.766 DEBUG:boto:Path: /xljbkf-71/c 2022-09-02T05:27:45.766 DEBUG:boto:Data: 2022-09-02T05:27:45.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:27:45.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.768 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.768 DEBUG:boto:Params: {} 2022-09-02T05:27:45.769 DEBUG:boto:Token: None 2022-09-02T05:27:45.769 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/c 2022-09-02T05:27:45.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dBDwpr0EmilVoUZjHDsqfxWEvNA= 2022-09-02T05:27:45.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dBDwpr0EmilVoUZjHDsqfxWEvNA='} 2022-09-02T05:27:45.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aff7d1caae175bb-0063119451-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.816 DEBUG:boto:path=/xljbkf-71/ 2022-09-02T05:27:45.816 DEBUG:boto:auth_path=/xljbkf-71/ 2022-09-02T05:27:45.817 DEBUG:boto:Method: HEAD 2022-09-02T05:27:45.818 DEBUG:boto:Path: /xljbkf-71/ 2022-09-02T05:27:45.818 DEBUG:boto:Data: 2022-09-02T05:27:45.819 DEBUG:boto:Headers: {} 2022-09-02T05:27:45.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.820 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.820 DEBUG:boto:Params: {} 2022-09-02T05:27:45.821 DEBUG:boto:Token: None 2022-09-02T05:27:45.822 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/ 2022-09-02T05:27:45.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk= 2022-09-02T05:27:45.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g6rXwPrgUZa2UhK1ZUSsjOE1dSk=', 'Content-Length': '0'} 2022-09-02T05:27:45.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000204f619af5e4ea98-0063119451-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.826 DEBUG:boto:path=/xljbkf-71/d 2022-09-02T05:27:45.826 DEBUG:boto:auth_path=/xljbkf-71/d 2022-09-02T05:27:45.827 DEBUG:boto:Method: PUT 2022-09-02T05:27:45.827 DEBUG:boto:Path: /xljbkf-71/d 2022-09-02T05:27:45.828 DEBUG:boto:Data: 2022-09-02T05:27:45.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:27:45.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:27:45.830 DEBUG:boto:Port: 8000 2022-09-02T05:27:45.830 DEBUG:boto:Params: {} 2022-09-02T05:27:45.831 DEBUG:boto:Token: None 2022-09-02T05:27:45.831 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:27:45 GMT /xljbkf-71/d 2022-09-02T05:27:45.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p0CEHhhjoVgd9PlR0URBdlDt5Ms= 2022-09-02T05:27:45.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:27:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p0CEHhhjoVgd9PlR0URBdlDt5Ms='} 2022-09-02T05:27:45.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006eaf763b483a597a-0063119451-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:27:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:27:45.840 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-71 2022-09-02T05:27:45.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:27:45.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:27:45.841 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:27:45.899 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:27:45.899 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:27:45.953 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:45.955+0000 7f929bfb9600 1 waiting to reach incremental sync.. 2022-09-02T05:27:50.953 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:50.955+0000 7f929bfb9600 1 waiting to reach incremental sync.. 2022-09-02T05:27:55.954 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:27:55.956+0000 7f929bfb9600 1 waiting to reach incremental sync.. 2022-09-02T05:28:00.954 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:00.956+0000 7f929bfb9600 1 waiting to reach incremental sync.. 2022-09-02T05:28:05.957 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:05.959+0000 7f929bfb9600 1 bucket sync caught up with source: 2022-09-02T05:28:05.957 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:05.958 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:05.958 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:05.959+0000 7f929bfb9600 0 bucket checkpoint complete 2022-09-02T05:28:05.976 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-71 2022-09-02T05:28:05.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:28:05.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:28:05.979 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:28:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:28:06.044 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:28:06.083 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:28:06.085+0000 7f62b4c3a600 1 bucket sync caught up with source: 2022-09-02T05:28:06.084 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:06.084 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:06.085 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:28:06.085+0000 7f62b4c3a600 0 bucket checkpoint complete 2022-09-02T05:28:06.097 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-71 2022-09-02T05:28:06.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:28:06.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:28:06.100 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:28:06.164 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:28:06.164 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:28:06.202 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:06.204+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:06.203 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:06.203 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:11.204 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:11.206+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:11.205 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:11.205 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:16.205 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:16.207+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:16.206 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:16.207 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:21.207 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:21.209+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:21.207 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:21.208 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:26.210+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:26.209 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:26.209 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:31.210 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:31.212+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:31.210 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:31.210 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:36.211 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:36.214+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:36.212 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:36.212 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:41.212 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:41.215+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:41.213 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:41.213 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:46.214 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:46.217+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:46.214 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:46.215 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:51.215 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:51.217+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:51.216 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:51.216 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:56.217 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:28:56.219+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:28:56.217 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:28:56.217 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:01.218 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:01.220+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:01.219 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:01.219 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:06.220 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:06.222+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:06.220 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:06.221 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:11.221 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:11.223+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:11.221 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:11.222 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:16.222 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:16.225+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:16.223 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:16.223 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:21.224 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:21.226+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:21.225 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:21.225 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:26.226 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:26.228+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:26.226 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:26.226 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:31.227 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:31.229+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:31.228 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:31.229 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:36.229 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:36.230+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:36.229 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:36.229 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:41.230 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:41.233+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:41.230 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:41.231 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:46.232 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:46.234+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:46.232 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:46.233 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:51.233 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:51.236+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:51.233 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:51.233 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:56.234 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:29:56.236+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:29:56.234 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:29:56.235 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:01.236 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:01.237+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:01.236 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:01.237 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:06.237 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:06.239+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:06.237 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:06.238 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:11.238 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:11.240+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:11.239 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:11.239 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:16.239 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:16.242+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:16.240 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:16.240 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:21.241 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:21.243+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:21.241 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:21.242 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:26.242 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:26.244+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:26.243 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:26.243 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:31.244 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:31.246+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:31.244 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:31.245 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:36.245 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:36.247+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:36.246 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:36.246 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:41.247 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:41.249+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:41.247 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:41.247 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:46.248 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:46.251+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:46.249 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:46.250 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:51.249 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:51.252+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:51.250 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:51.250 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:56.251 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:30:56.254+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:30:56.251 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:30:56.252 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:01.252 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:01.254+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:01.252 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:01.253 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:06.253 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:06.255+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:06.254 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:06.254 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:11.254 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:11.256+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:11.255 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:11.255 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:16.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:16.258+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:16.257 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:16.258 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:21.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:21.259+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:21.258 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:21.258 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:26.258 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:26.260+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:26.259 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:26.259 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:31.260 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:31.262+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:31.260 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:31.261 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:36.261 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:36.263+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:36.261 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:36.262 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:41.263 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:41.264+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:41.263 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:41.263 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:46.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:46.266+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:46.264 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:46.264 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:51.265 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:51.268+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:51.266 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:51.267 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:56.266 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:31:56.270+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:31:56.267 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:31:56.267 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:01.268 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:01.271+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:01.268 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:01.269 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:06.269 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:06.271+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:06.270 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:06.270 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:11.271 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:11.273+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:11.271 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:11.272 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:16.272 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:16.274+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:16.273 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:16.274 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:21.273 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:21.276+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:21.274 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:21.274 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:26.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:26.277+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:26.275 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:26.276 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:31.277 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:31.279+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:31.278 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:31.278 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:36.278 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:36.280+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:36.279 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:36.279 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:41.280 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:41.282+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:41.280 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:41.280 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:46.281 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:46.283+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:46.282 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:46.283 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:51.282 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:51.285+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:51.283 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:51.283 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:56.284 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:32:56.287+0000 7fa19c5d0600 1 waiting for incremental sync to catch up: 2022-09-02T05:32:56.284 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.56.5, , , , , , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:32:56.285 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.56.5, , , , 00000000001.58.5, , , 00000000001.46.5, , 00000000001.52.5] 2022-09-02T05:33:01.285 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:01.288+0000 7fa19c5d0600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T05:33:01.286 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:01.288+0000 7fa19c5d0600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:33:01.286 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:01.288+0000 7fa19c5d0600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:33:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T05:33:01.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2022-09-02T05:33:01.311 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:01.312 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:01.313 DEBUG:boto:Method: PUT 2022-09-02T05:33:01.314 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:01.314 DEBUG:boto:Data: 2022-09-02T05:33:01.315 DEBUG:boto:Headers: {} 2022-09-02T05:33:01.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:01.316 DEBUG:boto:Port: 8000 2022-09-02T05:33:01.316 DEBUG:boto:Params: {} 2022-09-02T05:33:01.318 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:33:01.319 DEBUG:boto:Token: None 2022-09-02T05:33:01.321 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:33:01 GMT /xljbkf-72/ 2022-09-02T05:33:01.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WIys/v9F/nFAb7MuaXu87vSSbIo= 2022-09-02T05:33:01.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WIys/v9F/nFAb7MuaXu87vSSbIo=', 'Content-Length': '0'} 2022-09-02T05:33:01.352 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006c7df3e8d03a73e1-006311958d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:01.353 DEBUG:rgw_multi.tests:created bucket=xljbkf-72 2022-09-02T05:33:01.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:01.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:01.356 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:33:01.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:01.412 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:01.443 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:33:01.458 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:33:01.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:01.459 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:33:01.459 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:33:01.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:01.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:01.460 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:33:01.461 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:33:01.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:01.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:01.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096781.353094_384.1", 2022-09-02T05:33:01.462 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:33:01.353094Z" 2022-09-02T05:33:01.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:01.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:01.464 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:33:01.464 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:33:01.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:33:01.465 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:33:01.467 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:01.468 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:33:01.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:01.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:01.472 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:33:01.527 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:01.528 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:01.562 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:01.565+0000 7fb5f8bf0600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:01.563 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:01.566+0000 7fb5f8bf0600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:01.578 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:33:01.579 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:33:01.579 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:33:01.579 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:33:01.579 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:33:01.580 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:33:01.580 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:33:01.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:01.581 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:33:01.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:01.582 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:33:01.582 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:01.582 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:01.582 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:33:01.583 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:01.583 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:33:01.583 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:01.583 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:33:01.584 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:01.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:01.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:01.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:01.585 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:33:01.585 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:01.585 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:01.586 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:33:01.586 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:01.586 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:01.588 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:01.588 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:33:01.589 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:01.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:01.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:01.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:01.590 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:33:01.590 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:01.590 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:01.591 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096781.345662_381.1", 2022-09-02T05:33:01.591 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:01.591 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:01.592 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:01.592 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:33:01.345662Z", 2022-09-02T05:33:01.592 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:33:01.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:01.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:01.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:01.593 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:33:01.593 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:01.594 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:01.594 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:33:01.594 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:01.595 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:01.595 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:01.595 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:33:01.595 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:01.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:01.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:01.596 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:33:01.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:01.597 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:33:01.597 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:33:01.598 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:33:01.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:01.598 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:33:01Z" 2022-09-02T05:33:01.598 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:33:01.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096781.345662_381.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:01.345662Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:33:01Z" } 2022-09-02T05:33:01.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096781.345662_381.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:01.345662Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:33:01.601 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:01.601 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096781.345662_381.1', 3: ''} 2022-09-02T05:33:01.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662096781.353094_384.1 target=1_1662096781.345662_381.1 2022-09-02T05:33:06.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:06.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:06.605 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:33:06.663 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:06.664 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:06.710 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:06.712+0000 7f9536f3b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:06.711 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:06.713+0000 7f9536f3b600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:06.725 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:33:06.726 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:33:06.727 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:33:06.727 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:33:06.728 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:33:06.728 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:33:06.729 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:33:06.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:06.730 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:33:06.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:06.731 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:33:06.731 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:06.732 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:06.732 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:33:06.732 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:06.733 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:33:06.733 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:06.734 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:33:06.734 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:06.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:06.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:06.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:06.735 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:33:06.736 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:06.736 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:06.736 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:33:06.737 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:06.737 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:06.737 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:06.738 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:33:06.738 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:06.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:06.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:06.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:06.739 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:33:06.739 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:06.740 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:06.740 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096781.353094_384.1", 2022-09-02T05:33:06.740 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:06.741 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:06.741 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:06.741 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:33:01.353094Z", 2022-09-02T05:33:06.742 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:33:06.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:06.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:06.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:06.744 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:33:06.744 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:06.745 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:06.745 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:33:06.745 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:06.746 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:06.746 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:06.747 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:33:06.747 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:06.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:06.748 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:06.748 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:33:06.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:06.749 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:33:06.749 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:33:06.749 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:33:06.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:06.750 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:33:06Z" 2022-09-02T05:33:06.750 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:33:06.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096781.353094_384.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:01.353094Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:33:06Z" } 2022-09-02T05:33:06.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096781.353094_384.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:01.353094Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:33:06.754 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:06.755 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:06.756 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:33:06.756 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:33:06.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:06.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:06.759 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:33:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:06.849 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:33:06.851+0000 7f36803ef600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:06.850 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:33:06.853+0000 7f36803ef600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:06.864 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:33:06.865 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:33:06.865 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:33:06.865 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:33:06.865 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:33:06.866 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:33:06.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:33:06.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:06.867 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:33:06.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:06.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:33:06.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:33:06.869 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:33:06.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:33:06.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:33:06.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:33:06.870 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:33:06.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:33:06.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:33:06.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:33:06.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:06.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:06.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:33:06.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:33:06.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:33:06.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:33:06.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:33:06.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:33:06.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:33:06.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:33:06.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:33:06.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:33:06.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:06.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:06.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:33:06.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:33:06.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:33:06.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096781.353094_384.1", 2022-09-02T05:33:06.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:33:06.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:33:06.876 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:33:06.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:33:01.353094Z", 2022-09-02T05:33:06.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:33:06.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:33:06.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:06.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:33:06.877 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:33:06.877 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:33:06.878 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:33:06.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:33:06.878 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:33:06.878 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:33:06.879 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:33:06.879 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:33:06.879 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:33:06.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:33:06.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:33:06.880 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:33:06.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:06.880 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:33:06.881 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:33:06.881 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:33:06.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:33:06.881 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:33:06Z" 2022-09-02T05:33:06.882 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:33:06.883 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096781.353094_384.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:01.353094Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:33:06Z" } 2022-09-02T05:33:06.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096781.353094_384.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:01.353094Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:33:06.884 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:06.885 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:06.885 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:33:06.886 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:33:06.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:06.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:33:06.887 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:33:06.939 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:06.940 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:06.982 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:06.985+0000 7f68dedc7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:06.983 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:06.986+0000 7f68dedc7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:33:06.999 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:33:06.999 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:33:07.000 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:33:07.000 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:33:07.000 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:33:07.001 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:33:07.001 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:33:07.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:07.001 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:33:07.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:07.002 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:33:07.002 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:07.003 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:07.003 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:33:07.003 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:07.003 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:33:07.004 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:07.004 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:33:07.004 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:07.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:07.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:07.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:07.005 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:33:07.005 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:07.006 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:07.006 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:33:07.006 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:07.006 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:07.007 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:07.007 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:33:07.007 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:07.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:07.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:07.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:07.008 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:33:07.008 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:07.009 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:07.009 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096781.353094_384.1", 2022-09-02T05:33:07.009 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:07.009 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:07.010 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:07.010 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:33:01.353094Z", 2022-09-02T05:33:07.010 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:33:07.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:07.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:07.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:33:07.011 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:33:07.011 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:33:07.012 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:33:07.012 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:33:07.012 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:33:07.012 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:33:07.013 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:33:07.013 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:33:07.013 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:33:07.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:07.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:33:07.014 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:33:07.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:07.014 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:33:07.015 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:33:07.015 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:33:07.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:33:07.015 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:33:06Z" 2022-09-02T05:33:07.016 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:33:07.017 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096781.353094_384.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:01.353094Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:33:06Z" } 2022-09-02T05:33:07.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096781.353094_384.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:01.353094Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:33:07.018 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:07.019 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096781.353094_384.1', 3: ''} 2022-09-02T05:33:07.019 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:33:07.020 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-02T05:33:07.020 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T05:33:07.021 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T05:33:07.023+0000 7f39d6ced700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 111604) UID: 0 2022-09-02T05:33:07.022 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T05:33:07.023+0000 7f39daf58600 -1 shutting down 2022-09-02T05:33:10.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-02T05:33:11.025 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2022-09-02T05:33:11.027+0000 7fb73c7d2700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-02T05:33:11.073 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2022-09-02T05:33:11.075+0000 7fb73c7d2700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-02T05:33:11.143 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2022-09-02T05:33:11.145+0000 7fb73c7d2700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-02T05:33:13.123 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-02T05:33:13.124 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-09-02T05:33:13.124 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T05:33:13.125 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T05:33:13.128+0000 7f4a33593700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 127514) UID: 0 2022-09-02T05:33:13.126 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T05:33:13.128+0000 7f4a377fe600 -1 shutting down 2022-09-02T05:33:16.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-09-02T05:33:16.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-02T05:33:19.226 INFO:tasks.rgw.c1.client.1:Stopped 2022-09-02T05:33:19.228 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.229 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.230 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.230 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.231 DEBUG:boto:Data: 2022-09-02T05:33:19.231 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.233 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.234 DEBUG:boto:Params: {} 2022-09-02T05:33:19.240 DEBUG:boto:Token: None 2022-09-02T05:33:19.240 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc5df7f5290f6def-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.246 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:33:19.247 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:33:19.248 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.248 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:33:19.249 DEBUG:boto:Data: 2022-09-02T05:33:19.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.250 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.251 DEBUG:boto:Params: {} 2022-09-02T05:33:19.251 DEBUG:boto:Token: None 2022-09-02T05:33:19.252 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/a 2022-09-02T05:33:19.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:evsHC51fqhJM2IwekEuDCnzcVd0= 2022-09-02T05:33:19.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:evsHC51fqhJM2IwekEuDCnzcVd0='} 2022-09-02T05:33:19.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dbf249ce8fe5323-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.264 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.264 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.265 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.266 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.266 DEBUG:boto:Data: 2022-09-02T05:33:19.267 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.268 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.268 DEBUG:boto:Params: {} 2022-09-02T05:33:19.269 DEBUG:boto:Token: None 2022-09-02T05:33:19.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002105d05285491484-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.275 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:33:19.276 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:33:19.277 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.278 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:33:19.279 DEBUG:boto:Data: 2022-09-02T05:33:19.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.282 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.282 DEBUG:boto:Params: {} 2022-09-02T05:33:19.284 DEBUG:boto:Token: None 2022-09-02T05:33:19.285 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/b 2022-09-02T05:33:19.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TMsiffqPXYmetL65kF1zCOqrNQU= 2022-09-02T05:33:19.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TMsiffqPXYmetL65kF1zCOqrNQU='} 2022-09-02T05:33:19.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dca88ba60831c2cf-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.297 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.298 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.298 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.299 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.299 DEBUG:boto:Data: 2022-09-02T05:33:19.300 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.301 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.302 DEBUG:boto:Params: {} 2022-09-02T05:33:19.303 DEBUG:boto:Token: None 2022-09-02T05:33:19.304 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fcabed2d04173d7-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.308 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:33:19.309 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:33:19.309 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.310 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:33:19.311 DEBUG:boto:Data: 2022-09-02T05:33:19.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.312 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.313 DEBUG:boto:Params: {} 2022-09-02T05:33:19.313 DEBUG:boto:Token: None 2022-09-02T05:33:19.314 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/c 2022-09-02T05:33:19.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ErqSGrgQgGjTrGsXnKa4Kdahxig= 2022-09-02T05:33:19.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ErqSGrgQgGjTrGsXnKa4Kdahxig='} 2022-09-02T05:33:19.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081b7f2e9207bc2f3-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.366 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.367 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.368 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.368 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.369 DEBUG:boto:Data: 2022-09-02T05:33:19.369 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.370 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.371 DEBUG:boto:Params: {} 2022-09-02T05:33:19.372 DEBUG:boto:Token: None 2022-09-02T05:33:19.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e53d6aa3b1563b09-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.378 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:33:19.379 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:33:19.379 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.380 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:33:19.380 DEBUG:boto:Data: 2022-09-02T05:33:19.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.382 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.382 DEBUG:boto:Params: {} 2022-09-02T05:33:19.383 DEBUG:boto:Token: None 2022-09-02T05:33:19.383 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/d 2022-09-02T05:33:19.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lzWHDLnOL7RhxEEGzJxnuqGxwMk= 2022-09-02T05:33:19.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lzWHDLnOL7RhxEEGzJxnuqGxwMk='} 2022-09-02T05:33:19.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f169588cb0c747df-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'xljbkf-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:33:19.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'xljbkf-72', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:33:19.396 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket xljbkf-72 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:33:19.453 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:19.454 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:19.513 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:33:19.514 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-72 2022-09-02T05:33:19.517 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2022-09-02T05:33:19.576 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.576 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.577 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.577 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.578 DEBUG:boto:Data: 2022-09-02T05:33:19.578 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.579 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.580 DEBUG:boto:Params: {} 2022-09-02T05:33:19.580 DEBUG:boto:Token: None 2022-09-02T05:33:19.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000325c43bec557092a-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.586 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:33:19.586 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:33:19.587 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.587 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:33:19.588 DEBUG:boto:Data: 2022-09-02T05:33:19.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.589 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.590 DEBUG:boto:Params: {} 2022-09-02T05:33:19.591 DEBUG:boto:Token: None 2022-09-02T05:33:19.591 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/e 2022-09-02T05:33:19.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jVcerP3rKzO6pWm+uhwfQ0nkyvI= 2022-09-02T05:33:19.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jVcerP3rKzO6pWm+uhwfQ0nkyvI='} 2022-09-02T05:33:19.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065716c1d5d1dc7ad-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.605 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.605 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.606 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.607 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.607 DEBUG:boto:Data: 2022-09-02T05:33:19.608 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.609 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.609 DEBUG:boto:Params: {} 2022-09-02T05:33:19.610 DEBUG:boto:Token: None 2022-09-02T05:33:19.610 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e68e5cbdfe9fc78c-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.614 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:33:19.615 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:33:19.616 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.616 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:33:19.617 DEBUG:boto:Data: 2022-09-02T05:33:19.617 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.618 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.619 DEBUG:boto:Params: {} 2022-09-02T05:33:19.619 DEBUG:boto:Token: None 2022-09-02T05:33:19.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/f 2022-09-02T05:33:19.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fbnaoOkG+07/oJgNYSWZdRaYRZg= 2022-09-02T05:33:19.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fbnaoOkG+07/oJgNYSWZdRaYRZg='} 2022-09-02T05:33:19.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f37b63c105ba64ac-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.633 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.634 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.634 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.635 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.635 DEBUG:boto:Data: 2022-09-02T05:33:19.636 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.637 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.637 DEBUG:boto:Params: {} 2022-09-02T05:33:19.638 DEBUG:boto:Token: None 2022-09-02T05:33:19.639 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea3f55231ee34787-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.642 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:33:19.643 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:33:19.644 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.644 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:33:19.645 DEBUG:boto:Data: 2022-09-02T05:33:19.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.648 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.649 DEBUG:boto:Params: {} 2022-09-02T05:33:19.651 DEBUG:boto:Token: None 2022-09-02T05:33:19.651 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/g 2022-09-02T05:33:19.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7yy5sOinwrzYm18qX2v+ejjZcTg= 2022-09-02T05:33:19.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7yy5sOinwrzYm18qX2v+ejjZcTg='} 2022-09-02T05:33:19.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001debf1a7e2dd32e7-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.699 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:33:19.700 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:33:19.700 DEBUG:boto:Method: HEAD 2022-09-02T05:33:19.701 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:33:19.701 DEBUG:boto:Data: 2022-09-02T05:33:19.702 DEBUG:boto:Headers: {} 2022-09-02T05:33:19.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.703 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.703 DEBUG:boto:Params: {} 2022-09-02T05:33:19.704 DEBUG:boto:Token: None 2022-09-02T05:33:19.704 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/ 2022-09-02T05:33:19.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A= 2022-09-02T05:33:19.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vOuzjVwJXL1AjhbXQR4F+ibY56A=', 'Content-Length': '0'} 2022-09-02T05:33:19.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c18c47a0bb3fa34-006311959f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.708 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:33:19.709 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:33:19.709 DEBUG:boto:Method: PUT 2022-09-02T05:33:19.710 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:33:19.711 DEBUG:boto:Data: 2022-09-02T05:33:19.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:33:19.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:33:19.712 DEBUG:boto:Port: 8000 2022-09-02T05:33:19.713 DEBUG:boto:Params: {} 2022-09-02T05:33:19.714 DEBUG:boto:Token: None 2022-09-02T05:33:19.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:33:19 GMT /xljbkf-72/h 2022-09-02T05:33:19.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B90ZhFTr2ghJ5AKcUisX4Ad4J0g= 2022-09-02T05:33:19.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:33:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B90ZhFTr2ghJ5AKcUisX4Ad4J0g='} 2022-09-02T05:33:19.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029ecb019bc88a22b-006311959f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:33:19.721 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-02T05:33:19.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-02T05:33:19.726 INFO:tasks.rgw.c2.client.0:Started 2022-09-02T05:33:19.727 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T05:33:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T05:33:19.754 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T05:33:19.755 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T05:33:19.755 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T05:33:20.756 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T05:33:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T05:33:20.786 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T05:33:20.787 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T05:33:20.787 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T05:33:21.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-09-02T05:33:22.788 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T05:33:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T05:33:22.822 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T05:33:22.822 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T05:33:22.823 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T05:33:26.824 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T05:33:26.851 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T05:33:26.852 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T05:33:26.852 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T05:33:26.853 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2022-09-02T05:33:26.854 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-09-02T05:33:26.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-09-02T05:33:26.858 INFO:tasks.rgw.c1.client.1:Started 2022-09-02T05:33:26.859 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2022-09-02T05:33:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T05:33:26.886 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T05:33:26.887 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T05:33:26.887 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2022-09-02T05:33:27.888 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2022-09-02T05:33:27.915 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T05:33:27.915 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2022-09-02T05:33:27.915 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T05:33:27.917 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2022-09-02T05:33:27.917 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-72 2022-09-02T05:33:27.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:33:27.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:33:27.919 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:33:27.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:33:27.985 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:33:28.024 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:28.026+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:33:33.025 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:33.027+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:33:38.025 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:38.027+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:33:43.026 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:43.028+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:33:48.026 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:48.028+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:33:53.027 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:53.029+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:33:58.027 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:33:58.030+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:03.028 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:03.031+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:08.028 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:08.032+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:13.029 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:13.031+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:18.030 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:18.032+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:23.030 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:23.032+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:28.031 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:28.033+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:33.032 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:33.034+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:38.032 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:38.034+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:43.033 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:43.035+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:48.034 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:48.035+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:53.033 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:53.036+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:34:58.034 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:34:58.036+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:03.035 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:03.037+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:08.035 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:08.038+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:13.036 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:13.039+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:18.037 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:18.039+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:23.037 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:23.039+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:28.038 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:28.040+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:33.038 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:33.040+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:38.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:38.041+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:43.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:43.042+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:48.041 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:48.043+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:53.041 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:53.043+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:35:58.042 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:35:58.044+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:36:03.042 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:03.045+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:36:08.043 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:08.045+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:36:13.044 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:13.047+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:36:18.045 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:18.047+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:36:23.046 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:23.047+0000 7f54c0fde600 1 waiting to reach incremental sync.. 2022-09-02T05:36:28.047 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:28.048+0000 7f54c0fde600 1 bucket sync caught up with source: 2022-09-02T05:36:28.047 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.68.5, 00000000002.39.5, 00000000002.36.5] 2022-09-02T05:36:28.048 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.68.5, 00000000002.39.5, 00000000002.36.5] 2022-09-02T05:36:28.048 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:28.048+0000 7f54c0fde600 0 bucket checkpoint complete 2022-09-02T05:36:28.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-72 2022-09-02T05:36:28.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:28.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:28.070 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:36:28.138 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:36:28.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:36:28.184 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:28.186+0000 7f869bbc9600 1 waiting to reach incremental sync.. 2022-09-02T05:36:33.185 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:33.186+0000 7f869bbc9600 1 waiting to reach incremental sync.. 2022-09-02T05:36:38.185 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:38.187+0000 7f869bbc9600 1 waiting to reach incremental sync.. 2022-09-02T05:36:43.186 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:43.188+0000 7f869bbc9600 1 waiting to reach incremental sync.. 2022-09-02T05:36:48.187 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:48.188+0000 7f869bbc9600 1 waiting to reach incremental sync.. 2022-09-02T05:36:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.190+0000 7f869bbc9600 1 bucket sync caught up with source: 2022-09-02T05:36:53.189 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.68.5, 00000000002.39.5, 00000000002.36.5] 2022-09-02T05:36:53.189 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.68.5, 00000000002.39.5, 00000000002.36.5] 2022-09-02T05:36:53.189 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.190+0000 7f869bbc9600 0 bucket checkpoint complete 2022-09-02T05:36:53.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-72 2022-09-02T05:36:53.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.214 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:36:53.277 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:36:53.278 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:36:53.315 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:53.317+0000 7f1c805ed600 1 bucket sync caught up with source: 2022-09-02T05:36:53.316 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000003.68.5, 00000000002.39.5, 00000000002.36.5] 2022-09-02T05:36:53.316 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000003.68.5, 00000000002.39.5, 00000000002.36.5] 2022-09-02T05:36:53.316 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:36:53.317+0000 7f1c805ed600 0 bucket checkpoint complete 2022-09-02T05:36:53.329 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-72 2022-09-02T05:36:53.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.331 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:36:53.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:36:53.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:36:53.442 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.444+0000 7f152eb3b600 1 bucket sync caught up with source: 2022-09-02T05:36:53.443 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.532.5, , , 00000000001.1081.5, 00000000001.1054.5, , , 00000000002.708.5, , 00000000002.359.5] 2022-09-02T05:36:53.444 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.532.5, , , 00000000001.1081.5, 00000000001.1054.5, , , 00000000002.708.5, , 00000000002.359.5] 2022-09-02T05:36:53.444 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.444+0000 7f152eb3b600 0 bucket checkpoint complete 2022-09-02T05:36:53.461 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-72 2022-09-02T05:36:53.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.465 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:36:53.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:36:53.522 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:36:53.568 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.570+0000 7f9acede8600 1 bucket sync caught up with source: 2022-09-02T05:36:53.569 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.532.5, , , 00000000001.1081.5, 00000000001.1054.5, , , 00000000002.708.5, , 00000000002.359.5] 2022-09-02T05:36:53.569 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.532.5, , , 00000000001.1081.5, 00000000001.1054.5, , , 00000000002.708.5, , 00000000002.359.5] 2022-09-02T05:36:53.569 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.570+0000 7f9acede8600 0 bucket checkpoint complete 2022-09-02T05:36:53.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-72 2022-09-02T05:36:53.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-72 2022-09-02T05:36:53.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:36:53.590 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:36:53.648 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:36:53.649 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:36:53.691 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:53.693+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:36:58.692 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:36:58.693+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:03.693 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:03.694+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:08.693 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:08.695+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:13.694 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:13.696+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:18.695 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:18.696+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:23.696 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:23.697+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:28.696 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:28.698+0000 7f339cf74600 1 waiting to reach incremental sync.. 2022-09-02T05:37:33.698 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:33.700+0000 7f339cf74600 1 bucket sync caught up with source: 2022-09-02T05:37:33.699 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.187.5, , , 00000000002.378.5, 00000000001.315.5, , , 00000000002.218.5, , 00000000002.101.5] 2022-09-02T05:37:33.699 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.187.5, , , 00000000002.378.5, 00000000001.315.5, , , 00000000002.218.5, , 00000000002.101.5] 2022-09-02T05:37:33.699 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:33.700+0000 7f339cf74600 0 bucket checkpoint complete 2022-09-02T05:37:33.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-72 2022-09-02T05:37:33.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:37:33.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-72', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:37:33.724 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-72 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:37:33.790 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:33.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:33.851 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:33.852+0000 7f06a5a95600 1 bucket sync caught up with source: 2022-09-02T05:37:33.851 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.187.5, , , 00000000002.378.5, 00000000001.315.5, , , 00000000002.218.5, , 00000000002.101.5] 2022-09-02T05:37:33.852 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.187.5, , , 00000000002.378.5, 00000000001.315.5, , , 00000000002.218.5, , 00000000002.101.5] 2022-09-02T05:37:33.852 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:33.852+0000 7f06a5a95600 0 bucket checkpoint complete 2022-09-02T05:37:33.874 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-72 2022-09-02T05:37:33.875 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-72 zones={test-zone2, test-zone1} 2022-09-02T05:37:33.876 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:33.878 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:33.879 DEBUG:boto:Method: HEAD 2022-09-02T05:37:33.881 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:37:33.881 DEBUG:boto:Data: 2022-09-02T05:37:33.882 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:33.884 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.885 DEBUG:boto:Params: {} 2022-09-02T05:37:33.887 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:37:33.888 DEBUG:boto:Token: None 2022-09-02T05:37:33.889 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/ 2022-09-02T05:37:33.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GArv6WH/SkLx7m7KrmihHJOoUFQ= 2022-09-02T05:37:33.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GArv6WH/SkLx7m7KrmihHJOoUFQ=', 'Content-Length': '0'} 2022-09-02T05:37:33.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2c5801f9daae259-006311969d-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:33.901 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:33.902 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:33.903 DEBUG:boto:Method: HEAD 2022-09-02T05:37:33.904 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:37:33.906 DEBUG:boto:Data: 2022-09-02T05:37:33.907 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:33.909 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.910 DEBUG:boto:Params: {} 2022-09-02T05:37:33.911 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:37:33.911 DEBUG:boto:Token: None 2022-09-02T05:37:33.912 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/ 2022-09-02T05:37:33.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GArv6WH/SkLx7m7KrmihHJOoUFQ= 2022-09-02T05:37:33.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GArv6WH/SkLx7m7KrmihHJOoUFQ=', 'Content-Length': '0'} 2022-09-02T05:37:33.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085e05e1e8307252e-006311969d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:33.917 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:37:33.918 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:33.919 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:33.920 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.920 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.921 DEBUG:boto:Method: GET 2022-09-02T05:37:33.922 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.923 DEBUG:boto:Data: 2022-09-02T05:37:33.924 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:33.926 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.927 DEBUG:boto:Params: {} 2022-09-02T05:37:33.928 DEBUG:boto:Token: None 2022-09-02T05:37:33.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/?versions 2022-09-02T05:37:33.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY= 2022-09-02T05:37:33.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY=', 'Content-Length': '0'} 2022-09-02T05:37:33.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002782f4e3ef2c21bb-006311969d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:33.934 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:33.936 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-02T05:37:33.937 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-02T05:37:33.938 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-02T05:37:33.938 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-02T05:37:33.939 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-02T05:37:33.939 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-02T05:37:33.940 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-02T05:37:33.940 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-02T05:37:33.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:37:33.942 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:33.942 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:33.943 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.943 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.944 DEBUG:boto:Method: GET 2022-09-02T05:37:33.944 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.945 DEBUG:boto:Data: 2022-09-02T05:37:33.946 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:33.947 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.947 DEBUG:boto:Params: {} 2022-09-02T05:37:33.948 DEBUG:boto:Token: None 2022-09-02T05:37:33.949 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/?versions 2022-09-02T05:37:33.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY= 2022-09-02T05:37:33.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY=', 'Content-Length': '0'} 2022-09-02T05:37:33.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dc2fa8ba093d9b6-006311969d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:33.953 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:33.955 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-02T05:37:33.955 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-02T05:37:33.956 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-02T05:37:33.956 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-02T05:37:33.957 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-02T05:37:33.957 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-02T05:37:33.958 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-02T05:37:33.958 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-02T05:37:33.959 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:33.960 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:33.960 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.961 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.961 DEBUG:boto:Method: GET 2022-09-02T05:37:33.962 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.963 DEBUG:boto:Data: 2022-09-02T05:37:33.963 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:33.964 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.965 DEBUG:boto:Params: {} 2022-09-02T05:37:33.965 DEBUG:boto:Token: None 2022-09-02T05:37:33.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/?versions 2022-09-02T05:37:33.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY= 2022-09-02T05:37:33.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY=', 'Content-Length': '0'} 2022-09-02T05:37:33.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcb78573f659be4b-006311969d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:33.970 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:33.971 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:33.972 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:33.973 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.973 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.974 DEBUG:boto:Method: GET 2022-09-02T05:37:33.975 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:33.975 DEBUG:boto:Data: 2022-09-02T05:37:33.976 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:33.977 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.977 DEBUG:boto:Params: {} 2022-09-02T05:37:33.978 DEBUG:boto:Token: None 2022-09-02T05:37:33.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/?versions 2022-09-02T05:37:33.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY= 2022-09-02T05:37:33.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K4obxydV17AKctfCEga+CVNk0PY=', 'Content-Length': '0'} 2022-09-02T05:37:33.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a340a23747400c-006311969d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:33.984 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:33.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:33.987 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:33.988 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:33.989 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:33.990 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:33.991 DEBUG:boto:Method: GET 2022-09-02T05:37:33.992 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:33.993 DEBUG:boto:Data: 2022-09-02T05:37:33.993 DEBUG:boto:Headers: {} 2022-09-02T05:37:33.994 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:33.994 DEBUG:boto:Port: 8000 2022-09-02T05:37:33.995 DEBUG:boto:Params: {} 2022-09-02T05:37:33.995 DEBUG:boto:Token: None 2022-09-02T05:37:33.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:33 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:33.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUg0aKqgAIWfTq4JunxSzJ5LvoU= 2022-09-02T05:37:33.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUg0aKqgAIWfTq4JunxSzJ5LvoU=', 'Content-Length': '0'} 2022-09-02T05:37:33.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877eaf92fe795004-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.000 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.001 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.001 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.002 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.002 DEBUG:boto:Method: GET 2022-09-02T05:37:34.003 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.003 DEBUG:boto:Data: 2022-09-02T05:37:34.004 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.005 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.005 DEBUG:boto:Params: {} 2022-09-02T05:37:34.006 DEBUG:boto:Token: None 2022-09-02T05:37:34.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCCyNRyHWGe5VSX/bV8EnJpfYzo= 2022-09-02T05:37:34.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCCyNRyHWGe5VSX/bV8EnJpfYzo=', 'Content-Length': '0'} 2022-09-02T05:37:34.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81aad601e71abe7-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.011 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.012 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.012 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.013 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.013 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.014 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.015 DEBUG:boto:Data: 2022-09-02T05:37:34.016 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.017 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.018 DEBUG:boto:Params: {} 2022-09-02T05:37:34.019 DEBUG:boto:Token: None 2022-09-02T05:37:34.020 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pqOzpRPDQX2wVoAhny4HDvdSPyk= 2022-09-02T05:37:34.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pqOzpRPDQX2wVoAhny4HDvdSPyk=', 'Content-Length': '0'} 2022-09-02T05:37:34.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c1fed72d7db4e7-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.026 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.027 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.028 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.029 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.029 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.030 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.031 DEBUG:boto:Data: 2022-09-02T05:37:34.032 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.033 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.033 DEBUG:boto:Params: {} 2022-09-02T05:37:34.034 DEBUG:boto:Token: None 2022-09-02T05:37:34.035 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pqOzpRPDQX2wVoAhny4HDvdSPyk= 2022-09-02T05:37:34.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pqOzpRPDQX2wVoAhny4HDvdSPyk=', 'Content-Length': '0'} 2022-09-02T05:37:34.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066635c1770da568f-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:34.041 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.042 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.042 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.043 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.043 DEBUG:boto:Method: GET 2022-09-02T05:37:34.044 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.044 DEBUG:boto:Data: 2022-09-02T05:37:34.045 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.046 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.047 DEBUG:boto:Params: {} 2022-09-02T05:37:34.048 DEBUG:boto:Token: None 2022-09-02T05:37:34.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCCyNRyHWGe5VSX/bV8EnJpfYzo= 2022-09-02T05:37:34.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCCyNRyHWGe5VSX/bV8EnJpfYzo=', 'Content-Length': '0'} 2022-09-02T05:37:34.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39c0b4744e39e97-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.054 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.054 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.055 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.055 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:34.056 DEBUG:boto:Method: GET 2022-09-02T05:37:34.057 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.057 DEBUG:boto:Data: 2022-09-02T05:37:34.058 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.059 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.059 DEBUG:boto:Params: {} 2022-09-02T05:37:34.060 DEBUG:boto:Token: None 2022-09-02T05:37:34.061 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:34.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCCyNRyHWGe5VSX/bV8EnJpfYzo= 2022-09-02T05:37:34.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCCyNRyHWGe5VSX/bV8EnJpfYzo=', 'Content-Length': '0'} 2022-09-02T05:37:34.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a55d90c4ae2e23f-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.106 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.108 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.109 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.110 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.111 DEBUG:boto:Method: GET 2022-09-02T05:37:34.112 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:34.113 DEBUG:boto:Data: 2022-09-02T05:37:34.114 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.115 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.116 DEBUG:boto:Params: {} 2022-09-02T05:37:34.117 DEBUG:boto:Token: None 2022-09-02T05:37:34.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b09ce285568f0d-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.122 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.123 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.124 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.124 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.125 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.125 DEBUG:boto:Method: GET 2022-09-02T05:37:34.126 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:34.127 DEBUG:boto:Data: 2022-09-02T05:37:34.127 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.129 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.130 DEBUG:boto:Params: {} 2022-09-02T05:37:34.131 DEBUG:boto:Token: None 2022-09-02T05:37:34.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c39fab5b40da727-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.138 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.139 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.140 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.140 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.141 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:34.141 DEBUG:boto:Data: 2022-09-02T05:37:34.142 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.143 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.144 DEBUG:boto:Params: {} 2022-09-02T05:37:34.144 DEBUG:boto:Token: None 2022-09-02T05:37:34.145 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a 2022-09-02T05:37:34.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0lHL/Oa4YOvIKMI9K9IKVyeE9nk= 2022-09-02T05:37:34.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0lHL/Oa4YOvIKMI9K9IKVyeE9nk=', 'Content-Length': '0'} 2022-09-02T05:37:34.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb587dc0f018a9a-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.149 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.150 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.150 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.151 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:34.152 DEBUG:boto:Data: 2022-09-02T05:37:34.152 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.153 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.154 DEBUG:boto:Params: {} 2022-09-02T05:37:34.154 DEBUG:boto:Token: None 2022-09-02T05:37:34.155 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a 2022-09-02T05:37:34.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0lHL/Oa4YOvIKMI9K9IKVyeE9nk= 2022-09-02T05:37:34.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0lHL/Oa4YOvIKMI9K9IKVyeE9nk=', 'Content-Length': '0'} 2022-09-02T05:37:34.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2bf73e823de8f8-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:34.160 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.161 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.161 DEBUG:boto:Method: GET 2022-09-02T05:37:34.162 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:34.163 DEBUG:boto:Data: 2022-09-02T05:37:34.163 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.164 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.165 DEBUG:boto:Params: {} 2022-09-02T05:37:34.166 DEBUG:boto:Token: None 2022-09-02T05:37:34.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a 2022-09-02T05:37:34.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UYNcB/zOh1oyUkxWvKVpxSRuqfc= 2022-09-02T05:37:34.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UYNcB/zOh1oyUkxWvKVpxSRuqfc=', 'Content-Length': '0'} 2022-09-02T05:37:34.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9954e8b0c99a90-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.173 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:34.173 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:34.174 DEBUG:boto:Method: GET 2022-09-02T05:37:34.174 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:34.175 DEBUG:boto:Data: 2022-09-02T05:37:34.175 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.176 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.177 DEBUG:boto:Params: {} 2022-09-02T05:37:34.177 DEBUG:boto:Token: None 2022-09-02T05:37:34.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/a 2022-09-02T05:37:34.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UYNcB/zOh1oyUkxWvKVpxSRuqfc= 2022-09-02T05:37:34.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UYNcB/zOh1oyUkxWvKVpxSRuqfc=', 'Content-Length': '0'} 2022-09-02T05:37:34.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c31dd682ab6d847-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.182 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.183 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.184 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.184 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.185 DEBUG:boto:Method: GET 2022-09-02T05:37:34.185 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:34.186 DEBUG:boto:Data: 2022-09-02T05:37:34.186 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.187 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.188 DEBUG:boto:Params: {} 2022-09-02T05:37:34.188 DEBUG:boto:Token: None 2022-09-02T05:37:34.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003366662741e075f0-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.194 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.194 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.195 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.196 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.196 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:34.197 DEBUG:boto:Method: GET 2022-09-02T05:37:34.197 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:34.198 DEBUG:boto:Data: 2022-09-02T05:37:34.198 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.199 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.200 DEBUG:boto:Params: {} 2022-09-02T05:37:34.200 DEBUG:boto:Token: None 2022-09-02T05:37:34.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc7b0d6ff376077d-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.205 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:34.207 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.207 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.208 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.208 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.209 DEBUG:boto:Method: GET 2022-09-02T05:37:34.210 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.210 DEBUG:boto:Data: 2022-09-02T05:37:34.211 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.212 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.212 DEBUG:boto:Params: {} 2022-09-02T05:37:34.213 DEBUG:boto:Token: None 2022-09-02T05:37:34.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM= 2022-09-02T05:37:34.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM=', 'Content-Length': '0'} 2022-09-02T05:37:34.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000393761bde10738e8-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.218 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.218 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.219 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.220 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.220 DEBUG:boto:Method: GET 2022-09-02T05:37:34.221 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.221 DEBUG:boto:Data: 2022-09-02T05:37:34.222 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.223 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.224 DEBUG:boto:Params: {} 2022-09-02T05:37:34.224 DEBUG:boto:Token: None 2022-09-02T05:37:34.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM= 2022-09-02T05:37:34.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM=', 'Content-Length': '0'} 2022-09-02T05:37:34.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170b2f67627e03b4-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.229 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.230 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.230 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.231 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.231 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.232 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.232 DEBUG:boto:Data: 2022-09-02T05:37:34.233 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.233 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.234 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.234 DEBUG:boto:Params: {} 2022-09-02T05:37:34.235 DEBUG:boto:Token: None 2022-09-02T05:37:34.236 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:efRnJ7ew/FLtORxpYXaoZcQ/8ag= 2022-09-02T05:37:34.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:efRnJ7ew/FLtORxpYXaoZcQ/8ag=', 'Content-Length': '0'} 2022-09-02T05:37:34.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233042025dbac1f7-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.240 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.240 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.241 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.241 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.242 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.242 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.243 DEBUG:boto:Data: 2022-09-02T05:37:34.243 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.244 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.245 DEBUG:boto:Params: {} 2022-09-02T05:37:34.245 DEBUG:boto:Token: None 2022-09-02T05:37:34.246 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:efRnJ7ew/FLtORxpYXaoZcQ/8ag= 2022-09-02T05:37:34.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:efRnJ7ew/FLtORxpYXaoZcQ/8ag=', 'Content-Length': '0'} 2022-09-02T05:37:34.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dca8031bc240b41-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:34.251 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.252 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.252 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.253 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.253 DEBUG:boto:Method: GET 2022-09-02T05:37:34.254 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.254 DEBUG:boto:Data: 2022-09-02T05:37:34.255 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.255 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.256 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.257 DEBUG:boto:Params: {} 2022-09-02T05:37:34.257 DEBUG:boto:Token: None 2022-09-02T05:37:34.258 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM= 2022-09-02T05:37:34.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM=', 'Content-Length': '0'} 2022-09-02T05:37:34.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000473c0b34511828b1-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.262 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.262 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.263 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.264 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:34.274 DEBUG:boto:Method: GET 2022-09-02T05:37:34.274 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.275 DEBUG:boto:Data: 2022-09-02T05:37:34.275 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.277 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.278 DEBUG:boto:Params: {} 2022-09-02T05:37:34.279 DEBUG:boto:Token: None 2022-09-02T05:37:34.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:34.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM= 2022-09-02T05:37:34.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QVIa2+/RVTGDavmQ3ZMGmlYI3EM=', 'Content-Length': '0'} 2022-09-02T05:37:34.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5105c9e5622fa4b-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.285 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.286 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.287 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.288 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.289 DEBUG:boto:Method: GET 2022-09-02T05:37:34.289 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:34.290 DEBUG:boto:Data: 2022-09-02T05:37:34.291 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.293 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.294 DEBUG:boto:Params: {} 2022-09-02T05:37:34.295 DEBUG:boto:Token: None 2022-09-02T05:37:34.296 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.297 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002644f917db67b1d7-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.302 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.303 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.304 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.304 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.305 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.305 DEBUG:boto:Method: GET 2022-09-02T05:37:34.306 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:34.307 DEBUG:boto:Data: 2022-09-02T05:37:34.307 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.308 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.309 DEBUG:boto:Params: {} 2022-09-02T05:37:34.309 DEBUG:boto:Token: None 2022-09-02T05:37:34.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000816b900f73d1a1a0-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.314 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.315 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.316 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.316 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.317 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:34.318 DEBUG:boto:Data: 2022-09-02T05:37:34.318 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.319 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.320 DEBUG:boto:Params: {} 2022-09-02T05:37:34.320 DEBUG:boto:Token: None 2022-09-02T05:37:34.321 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b 2022-09-02T05:37:34.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwojcIKto6JdcdUpRznmzMlJNzo= 2022-09-02T05:37:34.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwojcIKto6JdcdUpRznmzMlJNzo=', 'Content-Length': '0'} 2022-09-02T05:37:34.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e5566baf849754-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.325 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.326 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.326 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.327 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:34.327 DEBUG:boto:Data: 2022-09-02T05:37:34.328 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.329 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.330 DEBUG:boto:Params: {} 2022-09-02T05:37:34.330 DEBUG:boto:Token: None 2022-09-02T05:37:34.331 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b 2022-09-02T05:37:34.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwojcIKto6JdcdUpRznmzMlJNzo= 2022-09-02T05:37:34.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwojcIKto6JdcdUpRznmzMlJNzo=', 'Content-Length': '0'} 2022-09-02T05:37:34.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfe664ac5e2510b-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:34.336 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.336 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.337 DEBUG:boto:Method: GET 2022-09-02T05:37:34.337 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:34.338 DEBUG:boto:Data: 2022-09-02T05:37:34.338 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.340 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.340 DEBUG:boto:Params: {} 2022-09-02T05:37:34.341 DEBUG:boto:Token: None 2022-09-02T05:37:34.341 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b 2022-09-02T05:37:34.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uvd+8HOVZkvAsLVMzkONcMNqlNI= 2022-09-02T05:37:34.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uvd+8HOVZkvAsLVMzkONcMNqlNI=', 'Content-Length': '0'} 2022-09-02T05:37:34.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003476be0fed95d9bc-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.345 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:34.346 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:34.346 DEBUG:boto:Method: GET 2022-09-02T05:37:34.347 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:34.348 DEBUG:boto:Data: 2022-09-02T05:37:34.348 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.349 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.350 DEBUG:boto:Params: {} 2022-09-02T05:37:34.350 DEBUG:boto:Token: None 2022-09-02T05:37:34.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/b 2022-09-02T05:37:34.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uvd+8HOVZkvAsLVMzkONcMNqlNI= 2022-09-02T05:37:34.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uvd+8HOVZkvAsLVMzkONcMNqlNI=', 'Content-Length': '0'} 2022-09-02T05:37:34.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064bebbd5b9423bcb-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.402 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.403 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.403 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.404 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.404 DEBUG:boto:Method: GET 2022-09-02T05:37:34.405 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:34.406 DEBUG:boto:Data: 2022-09-02T05:37:34.406 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.407 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.408 DEBUG:boto:Params: {} 2022-09-02T05:37:34.408 DEBUG:boto:Token: None 2022-09-02T05:37:34.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000622d1e4e778e9b23-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.414 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.415 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.415 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.416 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.417 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:34.417 DEBUG:boto:Method: GET 2022-09-02T05:37:34.418 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:34.418 DEBUG:boto:Data: 2022-09-02T05:37:34.419 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.420 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.421 DEBUG:boto:Params: {} 2022-09-02T05:37:34.422 DEBUG:boto:Token: None 2022-09-02T05:37:34.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a993af2717545167-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.429 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:34.431 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.432 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.433 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.434 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.435 DEBUG:boto:Method: GET 2022-09-02T05:37:34.436 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.437 DEBUG:boto:Data: 2022-09-02T05:37:34.438 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.439 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.440 DEBUG:boto:Params: {} 2022-09-02T05:37:34.441 DEBUG:boto:Token: None 2022-09-02T05:37:34.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw= 2022-09-02T05:37:34.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw=', 'Content-Length': '0'} 2022-09-02T05:37:34.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09ad9d5aa3dea24-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.449 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.449 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.450 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.451 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.452 DEBUG:boto:Method: GET 2022-09-02T05:37:34.452 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.453 DEBUG:boto:Data: 2022-09-02T05:37:34.453 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.454 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.455 DEBUG:boto:Params: {} 2022-09-02T05:37:34.456 DEBUG:boto:Token: None 2022-09-02T05:37:34.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw= 2022-09-02T05:37:34.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw=', 'Content-Length': '0'} 2022-09-02T05:37:34.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a96486e769e3ee-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.461 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.461 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.468 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.469 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.469 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.470 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.471 DEBUG:boto:Data: 2022-09-02T05:37:34.472 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.474 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.475 DEBUG:boto:Params: {} 2022-09-02T05:37:34.476 DEBUG:boto:Token: None 2022-09-02T05:37:34.477 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JZTuuW8tMK1JsZ9Ysh1yBYutI1g= 2022-09-02T05:37:34.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JZTuuW8tMK1JsZ9Ysh1yBYutI1g=', 'Content-Length': '0'} 2022-09-02T05:37:34.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4fc0c07cb065925-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.483 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.484 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.485 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.486 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.487 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.488 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.489 DEBUG:boto:Data: 2022-09-02T05:37:34.490 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.492 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.493 DEBUG:boto:Params: {} 2022-09-02T05:37:34.494 DEBUG:boto:Token: None 2022-09-02T05:37:34.495 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JZTuuW8tMK1JsZ9Ysh1yBYutI1g= 2022-09-02T05:37:34.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JZTuuW8tMK1JsZ9Ysh1yBYutI1g=', 'Content-Length': '0'} 2022-09-02T05:37:34.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752b9855ad8fcb4a-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:34.503 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.504 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.505 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.506 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.507 DEBUG:boto:Method: GET 2022-09-02T05:37:34.508 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.509 DEBUG:boto:Data: 2022-09-02T05:37:34.510 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.511 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.512 DEBUG:boto:Params: {} 2022-09-02T05:37:34.513 DEBUG:boto:Token: None 2022-09-02T05:37:34.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw= 2022-09-02T05:37:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw=', 'Content-Length': '0'} 2022-09-02T05:37:34.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb25fbae68f2dd9-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.517 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.518 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.518 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.519 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:34.519 DEBUG:boto:Method: GET 2022-09-02T05:37:34.520 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.520 DEBUG:boto:Data: 2022-09-02T05:37:34.521 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.522 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.522 DEBUG:boto:Params: {} 2022-09-02T05:37:34.523 DEBUG:boto:Token: None 2022-09-02T05:37:34.524 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:34.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw= 2022-09-02T05:37:34.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u/fWI4hxlsf73ZmrB0CgMIZmsTw=', 'Content-Length': '0'} 2022-09-02T05:37:34.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db64c225fa91031-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.528 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.528 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.529 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.529 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.530 DEBUG:boto:Method: GET 2022-09-02T05:37:34.530 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:34.531 DEBUG:boto:Data: 2022-09-02T05:37:34.531 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.532 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.533 DEBUG:boto:Params: {} 2022-09-02T05:37:34.533 DEBUG:boto:Token: None 2022-09-02T05:37:34.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005196197ee470b5fe-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.538 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.539 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.540 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.540 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.541 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.541 DEBUG:boto:Method: GET 2022-09-02T05:37:34.542 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:34.542 DEBUG:boto:Data: 2022-09-02T05:37:34.543 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.544 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.544 DEBUG:boto:Params: {} 2022-09-02T05:37:34.545 DEBUG:boto:Token: None 2022-09-02T05:37:34.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d21ccb0b7862ae2-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.550 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.551 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.552 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.552 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.553 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:34.553 DEBUG:boto:Data: 2022-09-02T05:37:34.554 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.555 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.555 DEBUG:boto:Params: {} 2022-09-02T05:37:34.556 DEBUG:boto:Token: None 2022-09-02T05:37:34.557 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c 2022-09-02T05:37:34.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iP96lEoeY48SPKBOO8+Fn6FDAMM= 2022-09-02T05:37:34.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iP96lEoeY48SPKBOO8+Fn6FDAMM=', 'Content-Length': '0'} 2022-09-02T05:37:34.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db14cb577813ff00-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.560 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.561 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.561 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.562 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:34.562 DEBUG:boto:Data: 2022-09-02T05:37:34.563 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.564 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.564 DEBUG:boto:Params: {} 2022-09-02T05:37:34.565 DEBUG:boto:Token: None 2022-09-02T05:37:34.566 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c 2022-09-02T05:37:34.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iP96lEoeY48SPKBOO8+Fn6FDAMM= 2022-09-02T05:37:34.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iP96lEoeY48SPKBOO8+Fn6FDAMM=', 'Content-Length': '0'} 2022-09-02T05:37:34.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7181df5e450e7d1-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:34.570 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.571 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.571 DEBUG:boto:Method: GET 2022-09-02T05:37:34.572 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:34.572 DEBUG:boto:Data: 2022-09-02T05:37:34.573 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.574 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.574 DEBUG:boto:Params: {} 2022-09-02T05:37:34.575 DEBUG:boto:Token: None 2022-09-02T05:37:34.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c 2022-09-02T05:37:34.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gFlb1WESD6isTpwaegKG0adrPS4= 2022-09-02T05:37:34.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gFlb1WESD6isTpwaegKG0adrPS4=', 'Content-Length': '0'} 2022-09-02T05:37:34.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f0c99f6cc7044ac-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.579 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:34.580 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:34.580 DEBUG:boto:Method: GET 2022-09-02T05:37:34.581 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:34.581 DEBUG:boto:Data: 2022-09-02T05:37:34.582 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.583 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.584 DEBUG:boto:Params: {} 2022-09-02T05:37:34.584 DEBUG:boto:Token: None 2022-09-02T05:37:34.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/c 2022-09-02T05:37:34.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gFlb1WESD6isTpwaegKG0adrPS4= 2022-09-02T05:37:34.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gFlb1WESD6isTpwaegKG0adrPS4=', 'Content-Length': '0'} 2022-09-02T05:37:34.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b51ce452a332fe4f-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.589 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.589 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.590 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.590 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.591 DEBUG:boto:Method: GET 2022-09-02T05:37:34.592 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:34.592 DEBUG:boto:Data: 2022-09-02T05:37:34.593 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.594 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.594 DEBUG:boto:Params: {} 2022-09-02T05:37:34.595 DEBUG:boto:Token: None 2022-09-02T05:37:34.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003170f29316ca1ab-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.642 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.644 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.644 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.645 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.645 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:34.646 DEBUG:boto:Method: GET 2022-09-02T05:37:34.647 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:34.647 DEBUG:boto:Data: 2022-09-02T05:37:34.648 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.649 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.649 DEBUG:boto:Params: {} 2022-09-02T05:37:34.650 DEBUG:boto:Token: None 2022-09-02T05:37:34.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9626c66d3ef8aa7-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.655 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:34.657 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.658 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.659 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.660 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.661 DEBUG:boto:Method: GET 2022-09-02T05:37:34.662 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.663 DEBUG:boto:Data: 2022-09-02T05:37:34.664 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.666 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.667 DEBUG:boto:Params: {} 2022-09-02T05:37:34.668 DEBUG:boto:Token: None 2022-09-02T05:37:34.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8= 2022-09-02T05:37:34.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8=', 'Content-Length': '0'} 2022-09-02T05:37:34.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e9e56195fd65e1d-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.676 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.676 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.677 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.678 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.679 DEBUG:boto:Method: GET 2022-09-02T05:37:34.680 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.681 DEBUG:boto:Data: 2022-09-02T05:37:34.682 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.684 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.685 DEBUG:boto:Params: {} 2022-09-02T05:37:34.686 DEBUG:boto:Token: None 2022-09-02T05:37:34.688 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8= 2022-09-02T05:37:34.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8=', 'Content-Length': '0'} 2022-09-02T05:37:34.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070fae7558ce95b58-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.694 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.695 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.696 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.697 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.698 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.698 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.699 DEBUG:boto:Data: 2022-09-02T05:37:34.700 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.701 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.701 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.702 DEBUG:boto:Params: {} 2022-09-02T05:37:34.703 DEBUG:boto:Token: None 2022-09-02T05:37:34.704 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wnfDzzHJYtmOGSvjynnAsg0f2Ak= 2022-09-02T05:37:34.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wnfDzzHJYtmOGSvjynnAsg0f2Ak=', 'Content-Length': '0'} 2022-09-02T05:37:34.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b100829aecfdd48d-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.708 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.709 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.709 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.710 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.710 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.711 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.711 DEBUG:boto:Data: 2022-09-02T05:37:34.712 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.713 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.713 DEBUG:boto:Params: {} 2022-09-02T05:37:34.714 DEBUG:boto:Token: None 2022-09-02T05:37:34.714 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wnfDzzHJYtmOGSvjynnAsg0f2Ak= 2022-09-02T05:37:34.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wnfDzzHJYtmOGSvjynnAsg0f2Ak=', 'Content-Length': '0'} 2022-09-02T05:37:34.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e89b19161aaf4316-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:34.719 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.719 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.720 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.720 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.721 DEBUG:boto:Method: GET 2022-09-02T05:37:34.721 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.722 DEBUG:boto:Data: 2022-09-02T05:37:34.722 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.724 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.724 DEBUG:boto:Params: {} 2022-09-02T05:37:34.725 DEBUG:boto:Token: None 2022-09-02T05:37:34.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8= 2022-09-02T05:37:34.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8=', 'Content-Length': '0'} 2022-09-02T05:37:34.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e60bb1447b3ff6c-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.770 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.771 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.771 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.772 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:34.772 DEBUG:boto:Method: GET 2022-09-02T05:37:34.773 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.773 DEBUG:boto:Data: 2022-09-02T05:37:34.774 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.775 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.776 DEBUG:boto:Params: {} 2022-09-02T05:37:34.776 DEBUG:boto:Token: None 2022-09-02T05:37:34.777 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:34.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8= 2022-09-02T05:37:34.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y1EVlcDRhsyiEEiMRn06xjZNEu8=', 'Content-Length': '0'} 2022-09-02T05:37:34.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5b5a49af9c0725-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.781 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.782 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.782 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.783 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.783 DEBUG:boto:Method: GET 2022-09-02T05:37:34.784 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:34.784 DEBUG:boto:Data: 2022-09-02T05:37:34.785 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.785 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.786 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.786 DEBUG:boto:Params: {} 2022-09-02T05:37:34.787 DEBUG:boto:Token: None 2022-09-02T05:37:34.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0b99f69f23dc30b-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.792 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.793 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.794 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.794 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.795 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.795 DEBUG:boto:Method: GET 2022-09-02T05:37:34.796 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:34.796 DEBUG:boto:Data: 2022-09-02T05:37:34.797 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.798 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.798 DEBUG:boto:Params: {} 2022-09-02T05:37:34.799 DEBUG:boto:Token: None 2022-09-02T05:37:34.800 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2ba112840bc296-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.804 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.805 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.806 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.806 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.807 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:34.807 DEBUG:boto:Data: 2022-09-02T05:37:34.808 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.809 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.810 DEBUG:boto:Params: {} 2022-09-02T05:37:34.810 DEBUG:boto:Token: None 2022-09-02T05:37:34.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d 2022-09-02T05:37:34.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TDe2YptraGphe+Bd6sJv7cCtlHY= 2022-09-02T05:37:34.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TDe2YptraGphe+Bd6sJv7cCtlHY=', 'Content-Length': '0'} 2022-09-02T05:37:34.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd1f9c02f7fc04c2-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.815 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.815 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.816 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.816 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:34.817 DEBUG:boto:Data: 2022-09-02T05:37:34.817 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.818 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.819 DEBUG:boto:Params: {} 2022-09-02T05:37:34.819 DEBUG:boto:Token: None 2022-09-02T05:37:34.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d 2022-09-02T05:37:34.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TDe2YptraGphe+Bd6sJv7cCtlHY= 2022-09-02T05:37:34.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TDe2YptraGphe+Bd6sJv7cCtlHY=', 'Content-Length': '0'} 2022-09-02T05:37:34.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7fdef1400c217e1-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:34.825 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.826 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.826 DEBUG:boto:Method: GET 2022-09-02T05:37:34.827 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:34.827 DEBUG:boto:Data: 2022-09-02T05:37:34.828 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.829 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.829 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.830 DEBUG:boto:Params: {} 2022-09-02T05:37:34.830 DEBUG:boto:Token: None 2022-09-02T05:37:34.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d 2022-09-02T05:37:34.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zDK7Jm0DiWMlL+90O1/inEJLz9Y= 2022-09-02T05:37:34.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zDK7Jm0DiWMlL+90O1/inEJLz9Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9fdc3e53ed7ad4-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.878 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:34.879 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:34.880 DEBUG:boto:Method: GET 2022-09-02T05:37:34.881 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:34.881 DEBUG:boto:Data: 2022-09-02T05:37:34.882 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.883 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.884 DEBUG:boto:Params: {} 2022-09-02T05:37:34.884 DEBUG:boto:Token: None 2022-09-02T05:37:34.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/d 2022-09-02T05:37:34.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zDK7Jm0DiWMlL+90O1/inEJLz9Y= 2022-09-02T05:37:34.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zDK7Jm0DiWMlL+90O1/inEJLz9Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f987893f7c6cfcf1-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.889 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.890 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.890 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.891 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.891 DEBUG:boto:Method: GET 2022-09-02T05:37:34.892 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:34.892 DEBUG:boto:Data: 2022-09-02T05:37:34.893 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.894 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.894 DEBUG:boto:Params: {} 2022-09-02T05:37:34.895 DEBUG:boto:Token: None 2022-09-02T05:37:34.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e2fb8faa5744a2b-006311969e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.900 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.901 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:34.901 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:34.902 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.902 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:34.903 DEBUG:boto:Method: GET 2022-09-02T05:37:34.903 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:34.904 DEBUG:boto:Data: 2022-09-02T05:37:34.904 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.905 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.906 DEBUG:boto:Params: {} 2022-09-02T05:37:34.907 DEBUG:boto:Token: None 2022-09-02T05:37:34.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/ 2022-09-02T05:37:34.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y= 2022-09-02T05:37:34.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JlUKViGr9TY0iZ1+uhFrEMhN/7Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acd4841bb50b00ea-006311969e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.912 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:34.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:34.913 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:34.914 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:34.914 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.915 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.915 DEBUG:boto:Method: GET 2022-09-02T05:37:34.916 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.917 DEBUG:boto:Data: 2022-09-02T05:37:34.917 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.918 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.919 DEBUG:boto:Params: {} 2022-09-02T05:37:34.920 DEBUG:boto:Token: None 2022-09-02T05:37:34.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:viyD5XHv2OnhsjQYXHMrwRGbl6Y= 2022-09-02T05:37:34.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:viyD5XHv2OnhsjQYXHMrwRGbl6Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee88b332673a6ae8-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.925 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:34.925 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:34.926 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.926 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.927 DEBUG:boto:Method: GET 2022-09-02T05:37:34.928 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.928 DEBUG:boto:Data: 2022-09-02T05:37:34.929 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.930 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.930 DEBUG:boto:Params: {} 2022-09-02T05:37:34.931 DEBUG:boto:Token: None 2022-09-02T05:37:34.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:viyD5XHv2OnhsjQYXHMrwRGbl6Y= 2022-09-02T05:37:34.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:viyD5XHv2OnhsjQYXHMrwRGbl6Y=', 'Content-Length': '0'} 2022-09-02T05:37:34.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a7e138b8c735bd4-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.978 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:34.979 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:34.979 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.980 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.981 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.981 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.982 DEBUG:boto:Data: 2022-09-02T05:37:34.982 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:34.983 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.984 DEBUG:boto:Params: {} 2022-09-02T05:37:34.984 DEBUG:boto:Token: None 2022-09-02T05:37:34.985 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ymdcf54qVjPXsBAmq67xOwvuaqI= 2022-09-02T05:37:34.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ymdcf54qVjPXsBAmq67xOwvuaqI=', 'Content-Length': '0'} 2022-09-02T05:37:34.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe9d2ab0067280f-006311969e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.989 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:34.989 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:34.990 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.990 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:34.991 DEBUG:boto:Method: HEAD 2022-09-02T05:37:34.991 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.992 DEBUG:boto:Data: 2022-09-02T05:37:34.992 DEBUG:boto:Headers: {} 2022-09-02T05:37:34.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:34.993 DEBUG:boto:Port: 8000 2022-09-02T05:37:34.994 DEBUG:boto:Params: {} 2022-09-02T05:37:34.994 DEBUG:boto:Token: None 2022-09-02T05:37:34.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:34 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:34.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ymdcf54qVjPXsBAmq67xOwvuaqI= 2022-09-02T05:37:34.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ymdcf54qVjPXsBAmq67xOwvuaqI=', 'Content-Length': '0'} 2022-09-02T05:37:34.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037eaea7c5fe63a24-006311969e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:34.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:34.999 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:35.000 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:35.001 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:35.001 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:35.002 DEBUG:boto:Method: GET 2022-09-02T05:37:35.002 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:35.003 DEBUG:boto:Data: 2022-09-02T05:37:35.003 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.004 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.005 DEBUG:boto:Params: {} 2022-09-02T05:37:35.006 DEBUG:boto:Token: None 2022-09-02T05:37:35.006 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:35.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7vTZ1JrGVSpPD3xK1YpwzlXbTO0= 2022-09-02T05:37:35.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7vTZ1JrGVSpPD3xK1YpwzlXbTO0=', 'Content-Length': '0'} 2022-09-02T05:37:35.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff75445dd9e5b557-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.010 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:35.011 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:35.012 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:35.012 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:35.013 DEBUG:boto:Method: GET 2022-09-02T05:37:35.013 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:35.014 DEBUG:boto:Data: 2022-09-02T05:37:35.014 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.015 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.016 DEBUG:boto:Params: {} 2022-09-02T05:37:35.016 DEBUG:boto:Token: None 2022-09-02T05:37:35.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:35.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7vTZ1JrGVSpPD3xK1YpwzlXbTO0= 2022-09-02T05:37:35.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7vTZ1JrGVSpPD3xK1YpwzlXbTO0=', 'Content-Length': '0'} 2022-09-02T05:37:35.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c39ad914c7f162a-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.021 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.022 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.022 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.023 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.023 DEBUG:boto:Method: GET 2022-09-02T05:37:35.024 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:35.025 DEBUG:boto:Data: 2022-09-02T05:37:35.025 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.026 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.027 DEBUG:boto:Params: {} 2022-09-02T05:37:35.027 DEBUG:boto:Token: None 2022-09-02T05:37:35.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac0b1a3d69aa27f-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.032 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.033 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.034 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.034 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.035 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.035 DEBUG:boto:Method: GET 2022-09-02T05:37:35.036 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:35.036 DEBUG:boto:Data: 2022-09-02T05:37:35.037 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.038 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.039 DEBUG:boto:Params: {} 2022-09-02T05:37:35.039 DEBUG:boto:Token: None 2022-09-02T05:37:35.040 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd8904d9271a2719-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.045 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.045 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:35.046 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:35.047 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.047 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:35.048 DEBUG:boto:Data: 2022-09-02T05:37:35.048 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.049 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.050 DEBUG:boto:Params: {} 2022-09-02T05:37:35.050 DEBUG:boto:Token: None 2022-09-02T05:37:35.051 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/e 2022-09-02T05:37:35.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EBMdPHsR9EmgukzA6MMdWcquwI8= 2022-09-02T05:37:35.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EBMdPHsR9EmgukzA6MMdWcquwI8=', 'Content-Length': '0'} 2022-09-02T05:37:35.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e3fd14726c02f3-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.055 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:35.056 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:35.057 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.057 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:35.058 DEBUG:boto:Data: 2022-09-02T05:37:35.058 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.059 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.060 DEBUG:boto:Params: {} 2022-09-02T05:37:35.060 DEBUG:boto:Token: None 2022-09-02T05:37:35.061 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/e 2022-09-02T05:37:35.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EBMdPHsR9EmgukzA6MMdWcquwI8= 2022-09-02T05:37:35.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EBMdPHsR9EmgukzA6MMdWcquwI8=', 'Content-Length': '0'} 2022-09-02T05:37:35.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004518cc27a9a48763-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:35.066 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:35.066 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:35.067 DEBUG:boto:Method: GET 2022-09-02T05:37:35.068 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:35.068 DEBUG:boto:Data: 2022-09-02T05:37:35.069 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.069 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.070 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.070 DEBUG:boto:Params: {} 2022-09-02T05:37:35.071 DEBUG:boto:Token: None 2022-09-02T05:37:35.072 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/e 2022-09-02T05:37:35.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gPlhjw8M1gYwaCLui69ugnojmlM= 2022-09-02T05:37:35.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gPlhjw8M1gYwaCLui69ugnojmlM=', 'Content-Length': '0'} 2022-09-02T05:37:35.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000072386cdbe4068df-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.122 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:35.123 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:35.123 DEBUG:boto:Method: GET 2022-09-02T05:37:35.124 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:35.125 DEBUG:boto:Data: 2022-09-02T05:37:35.125 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.126 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.127 DEBUG:boto:Params: {} 2022-09-02T05:37:35.128 DEBUG:boto:Token: None 2022-09-02T05:37:35.128 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/e 2022-09-02T05:37:35.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gPlhjw8M1gYwaCLui69ugnojmlM= 2022-09-02T05:37:35.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gPlhjw8M1gYwaCLui69ugnojmlM=', 'Content-Length': '0'} 2022-09-02T05:37:35.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d089bbe942fc3464-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.133 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.134 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.135 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.136 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.137 DEBUG:boto:Method: GET 2022-09-02T05:37:35.138 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:35.139 DEBUG:boto:Data: 2022-09-02T05:37:35.140 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.142 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.143 DEBUG:boto:Params: {} 2022-09-02T05:37:35.144 DEBUG:boto:Token: None 2022-09-02T05:37:35.145 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bee135556b66fc32-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.151 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.153 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.154 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.155 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.155 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:35.156 DEBUG:boto:Method: GET 2022-09-02T05:37:35.157 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:35.158 DEBUG:boto:Data: 2022-09-02T05:37:35.159 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.160 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.161 DEBUG:boto:Params: {} 2022-09-02T05:37:35.162 DEBUG:boto:Token: None 2022-09-02T05:37:35.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.164 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089ce314658d3aac6-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.168 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:35.170 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.171 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.171 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.172 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.172 DEBUG:boto:Method: GET 2022-09-02T05:37:35.173 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.173 DEBUG:boto:Data: 2022-09-02T05:37:35.174 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.174 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.175 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.176 DEBUG:boto:Params: {} 2022-09-02T05:37:35.176 DEBUG:boto:Token: None 2022-09-02T05:37:35.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg= 2022-09-02T05:37:35.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg=', 'Content-Length': '0'} 2022-09-02T05:37:35.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8cffef5bb9b1467-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.182 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.183 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.184 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.185 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.185 DEBUG:boto:Method: GET 2022-09-02T05:37:35.186 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.187 DEBUG:boto:Data: 2022-09-02T05:37:35.187 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.188 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.189 DEBUG:boto:Params: {} 2022-09-02T05:37:35.189 DEBUG:boto:Token: None 2022-09-02T05:37:35.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg= 2022-09-02T05:37:35.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg=', 'Content-Length': '0'} 2022-09-02T05:37:35.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa050f073c29a89b-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.194 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.195 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.196 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.196 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.197 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.197 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.198 DEBUG:boto:Data: 2022-09-02T05:37:35.198 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.199 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.200 DEBUG:boto:Params: {} 2022-09-02T05:37:35.200 DEBUG:boto:Token: None 2022-09-02T05:37:35.201 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LRVVOwTvOOVvAwLOlvdbCztsNFM= 2022-09-02T05:37:35.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LRVVOwTvOOVvAwLOlvdbCztsNFM=', 'Content-Length': '0'} 2022-09-02T05:37:35.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2383137a5715ab8-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.205 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.206 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.206 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.207 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.207 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.208 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.208 DEBUG:boto:Data: 2022-09-02T05:37:35.209 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.210 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.210 DEBUG:boto:Params: {} 2022-09-02T05:37:35.211 DEBUG:boto:Token: None 2022-09-02T05:37:35.211 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LRVVOwTvOOVvAwLOlvdbCztsNFM= 2022-09-02T05:37:35.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LRVVOwTvOOVvAwLOlvdbCztsNFM=', 'Content-Length': '0'} 2022-09-02T05:37:35.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7ec3cde0d71fc04-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:35.217 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.218 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.218 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.219 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.220 DEBUG:boto:Method: GET 2022-09-02T05:37:35.220 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.221 DEBUG:boto:Data: 2022-09-02T05:37:35.221 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.222 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.223 DEBUG:boto:Params: {} 2022-09-02T05:37:35.223 DEBUG:boto:Token: None 2022-09-02T05:37:35.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg= 2022-09-02T05:37:35.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg=', 'Content-Length': '0'} 2022-09-02T05:37:35.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9072d4308401f87-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.230 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.231 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.232 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.233 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:35.234 DEBUG:boto:Method: GET 2022-09-02T05:37:35.235 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.236 DEBUG:boto:Data: 2022-09-02T05:37:35.237 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.238 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.239 DEBUG:boto:Params: {} 2022-09-02T05:37:35.239 DEBUG:boto:Token: None 2022-09-02T05:37:35.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:35.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg= 2022-09-02T05:37:35.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FR3RnfDaklVNa0lOLucDsu15ACg=', 'Content-Length': '0'} 2022-09-02T05:37:35.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ecc0dcab20cfa0c-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.286 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.287 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.288 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.288 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.289 DEBUG:boto:Method: GET 2022-09-02T05:37:35.289 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:35.290 DEBUG:boto:Data: 2022-09-02T05:37:35.291 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.292 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.293 DEBUG:boto:Params: {} 2022-09-02T05:37:35.293 DEBUG:boto:Token: None 2022-09-02T05:37:35.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04be9b02e8a5514-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.299 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.300 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.301 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.302 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.302 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.303 DEBUG:boto:Method: GET 2022-09-02T05:37:35.303 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:35.304 DEBUG:boto:Data: 2022-09-02T05:37:35.305 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.306 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.306 DEBUG:boto:Params: {} 2022-09-02T05:37:35.307 DEBUG:boto:Token: None 2022-09-02T05:37:35.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006284251f993386e6-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.312 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.313 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.314 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.314 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.315 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:35.315 DEBUG:boto:Data: 2022-09-02T05:37:35.316 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.317 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.317 DEBUG:boto:Params: {} 2022-09-02T05:37:35.318 DEBUG:boto:Token: None 2022-09-02T05:37:35.319 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f 2022-09-02T05:37:35.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VgdSf/oicc4u7PIyEg93F4YgEG4= 2022-09-02T05:37:35.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VgdSf/oicc4u7PIyEg93F4YgEG4=', 'Content-Length': '0'} 2022-09-02T05:37:35.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd49fdcd94479d7e-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.323 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.323 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.324 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.324 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:35.325 DEBUG:boto:Data: 2022-09-02T05:37:35.325 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.326 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.327 DEBUG:boto:Params: {} 2022-09-02T05:37:35.328 DEBUG:boto:Token: None 2022-09-02T05:37:35.328 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f 2022-09-02T05:37:35.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VgdSf/oicc4u7PIyEg93F4YgEG4= 2022-09-02T05:37:35.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VgdSf/oicc4u7PIyEg93F4YgEG4=', 'Content-Length': '0'} 2022-09-02T05:37:35.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095817172f47b0923-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:35.333 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.334 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.334 DEBUG:boto:Method: GET 2022-09-02T05:37:35.335 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:35.335 DEBUG:boto:Data: 2022-09-02T05:37:35.336 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.337 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.337 DEBUG:boto:Params: {} 2022-09-02T05:37:35.338 DEBUG:boto:Token: None 2022-09-02T05:37:35.339 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f 2022-09-02T05:37:35.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qG4Y9VvjDO38CVavFnZFqO5J5b0= 2022-09-02T05:37:35.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qG4Y9VvjDO38CVavFnZFqO5J5b0=', 'Content-Length': '0'} 2022-09-02T05:37:35.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b19639dd351031e-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.386 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:35.387 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:35.388 DEBUG:boto:Method: GET 2022-09-02T05:37:35.388 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:35.389 DEBUG:boto:Data: 2022-09-02T05:37:35.390 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.391 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.391 DEBUG:boto:Params: {} 2022-09-02T05:37:35.392 DEBUG:boto:Token: None 2022-09-02T05:37:35.393 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/f 2022-09-02T05:37:35.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qG4Y9VvjDO38CVavFnZFqO5J5b0= 2022-09-02T05:37:35.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qG4Y9VvjDO38CVavFnZFqO5J5b0=', 'Content-Length': '0'} 2022-09-02T05:37:35.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6570028fa3e7332-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.397 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.397 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.399 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.400 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.400 DEBUG:boto:Method: GET 2022-09-02T05:37:35.401 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:35.401 DEBUG:boto:Data: 2022-09-02T05:37:35.402 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.403 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.404 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.404 DEBUG:boto:Params: {} 2022-09-02T05:37:35.405 DEBUG:boto:Token: None 2022-09-02T05:37:35.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab7d898acfe0601-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.411 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.412 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.412 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.413 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.413 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:35.414 DEBUG:boto:Method: GET 2022-09-02T05:37:35.415 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:35.415 DEBUG:boto:Data: 2022-09-02T05:37:35.416 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.417 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.417 DEBUG:boto:Params: {} 2022-09-02T05:37:35.418 DEBUG:boto:Token: None 2022-09-02T05:37:35.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680be3a0a8eca0dc-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.423 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:35.424 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.425 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.425 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.426 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.427 DEBUG:boto:Method: GET 2022-09-02T05:37:35.428 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.429 DEBUG:boto:Data: 2022-09-02T05:37:35.430 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.432 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.432 DEBUG:boto:Params: {} 2022-09-02T05:37:35.433 DEBUG:boto:Token: None 2022-09-02T05:37:35.434 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8= 2022-09-02T05:37:35.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8=', 'Content-Length': '0'} 2022-09-02T05:37:35.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cdbb3b1c47666b4-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.439 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.439 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.440 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.441 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.441 DEBUG:boto:Method: GET 2022-09-02T05:37:35.442 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.442 DEBUG:boto:Data: 2022-09-02T05:37:35.443 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.444 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.445 DEBUG:boto:Params: {} 2022-09-02T05:37:35.445 DEBUG:boto:Token: None 2022-09-02T05:37:35.446 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8= 2022-09-02T05:37:35.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8=', 'Content-Length': '0'} 2022-09-02T05:37:35.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4e5880e1efb3460-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.450 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.451 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.451 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.452 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.452 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.453 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.453 DEBUG:boto:Data: 2022-09-02T05:37:35.454 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.455 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.455 DEBUG:boto:Params: {} 2022-09-02T05:37:35.456 DEBUG:boto:Token: None 2022-09-02T05:37:35.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVDOihprYH93dXD1UV1hdfQuhug= 2022-09-02T05:37:35.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVDOihprYH93dXD1UV1hdfQuhug=', 'Content-Length': '0'} 2022-09-02T05:37:35.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e91f46e66468aa1-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.460 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.461 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.461 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.462 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.462 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.463 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.463 DEBUG:boto:Data: 2022-09-02T05:37:35.464 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.465 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.465 DEBUG:boto:Params: {} 2022-09-02T05:37:35.466 DEBUG:boto:Token: None 2022-09-02T05:37:35.466 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVDOihprYH93dXD1UV1hdfQuhug= 2022-09-02T05:37:35.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVDOihprYH93dXD1UV1hdfQuhug=', 'Content-Length': '0'} 2022-09-02T05:37:35.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a99d8202a11da4d-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:35.471 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.471 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.472 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.472 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.473 DEBUG:boto:Method: GET 2022-09-02T05:37:35.474 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.474 DEBUG:boto:Data: 2022-09-02T05:37:35.475 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.476 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.476 DEBUG:boto:Params: {} 2022-09-02T05:37:35.477 DEBUG:boto:Token: None 2022-09-02T05:37:35.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8= 2022-09-02T05:37:35.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8=', 'Content-Length': '0'} 2022-09-02T05:37:35.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d60ef38589b28a-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.481 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.482 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.483 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.483 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:35.484 DEBUG:boto:Method: GET 2022-09-02T05:37:35.484 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.485 DEBUG:boto:Data: 2022-09-02T05:37:35.485 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.487 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.487 DEBUG:boto:Params: {} 2022-09-02T05:37:35.488 DEBUG:boto:Token: None 2022-09-02T05:37:35.488 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:35.489 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8= 2022-09-02T05:37:35.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KMTXPHroMo1z8iUuNnzoxhtqUt8=', 'Content-Length': '0'} 2022-09-02T05:37:35.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009321fa93eb0e8d3e-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.493 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.493 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.494 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.495 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.495 DEBUG:boto:Method: GET 2022-09-02T05:37:35.496 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:35.497 DEBUG:boto:Data: 2022-09-02T05:37:35.498 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.500 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.500 DEBUG:boto:Params: {} 2022-09-02T05:37:35.501 DEBUG:boto:Token: None 2022-09-02T05:37:35.501 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa5cbf8eb4fbabc-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.507 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.507 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.508 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.509 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.509 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.510 DEBUG:boto:Method: GET 2022-09-02T05:37:35.511 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:35.511 DEBUG:boto:Data: 2022-09-02T05:37:35.512 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.513 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.513 DEBUG:boto:Params: {} 2022-09-02T05:37:35.514 DEBUG:boto:Token: None 2022-09-02T05:37:35.515 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000374fbab4499711b5-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.520 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.521 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.521 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.522 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.522 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:35.523 DEBUG:boto:Data: 2022-09-02T05:37:35.524 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.525 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.526 DEBUG:boto:Params: {} 2022-09-02T05:37:35.526 DEBUG:boto:Token: None 2022-09-02T05:37:35.527 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g 2022-09-02T05:37:35.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XvbsF6Oz/4eZU5wJ1CIdPxKr1Q8= 2022-09-02T05:37:35.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XvbsF6Oz/4eZU5wJ1CIdPxKr1Q8=', 'Content-Length': '0'} 2022-09-02T05:37:35.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b826f2d3298735b8-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.531 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.531 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.532 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.532 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:35.533 DEBUG:boto:Data: 2022-09-02T05:37:35.533 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.535 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.536 DEBUG:boto:Params: {} 2022-09-02T05:37:35.537 DEBUG:boto:Token: None 2022-09-02T05:37:35.538 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g 2022-09-02T05:37:35.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XvbsF6Oz/4eZU5wJ1CIdPxKr1Q8= 2022-09-02T05:37:35.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XvbsF6Oz/4eZU5wJ1CIdPxKr1Q8=', 'Content-Length': '0'} 2022-09-02T05:37:35.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056878ca5e028f956-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:35.545 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.546 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.546 DEBUG:boto:Method: GET 2022-09-02T05:37:35.547 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:35.548 DEBUG:boto:Data: 2022-09-02T05:37:35.549 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.550 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.551 DEBUG:boto:Params: {} 2022-09-02T05:37:35.551 DEBUG:boto:Token: None 2022-09-02T05:37:35.552 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g 2022-09-02T05:37:35.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vaxii7hb/QnhSyKEJia5/bxH/WI= 2022-09-02T05:37:35.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vaxii7hb/QnhSyKEJia5/bxH/WI=', 'Content-Length': '0'} 2022-09-02T05:37:35.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c67757a67191e8-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.602 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:35.603 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:35.603 DEBUG:boto:Method: GET 2022-09-02T05:37:35.604 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:35.605 DEBUG:boto:Data: 2022-09-02T05:37:35.605 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.606 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.607 DEBUG:boto:Params: {} 2022-09-02T05:37:35.607 DEBUG:boto:Token: None 2022-09-02T05:37:35.608 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/g 2022-09-02T05:37:35.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vaxii7hb/QnhSyKEJia5/bxH/WI= 2022-09-02T05:37:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vaxii7hb/QnhSyKEJia5/bxH/WI=', 'Content-Length': '0'} 2022-09-02T05:37:35.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e964cd320c4cd7-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.613 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.613 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.614 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.614 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.615 DEBUG:boto:Method: GET 2022-09-02T05:37:35.615 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:35.616 DEBUG:boto:Data: 2022-09-02T05:37:35.616 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.617 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.618 DEBUG:boto:Params: {} 2022-09-02T05:37:35.619 DEBUG:boto:Token: None 2022-09-02T05:37:35.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f640df3660fd37fb-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.624 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.625 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.625 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.626 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.626 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:35.627 DEBUG:boto:Method: GET 2022-09-02T05:37:35.627 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:35.628 DEBUG:boto:Data: 2022-09-02T05:37:35.628 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.629 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.630 DEBUG:boto:Params: {} 2022-09-02T05:37:35.631 DEBUG:boto:Token: None 2022-09-02T05:37:35.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a80a28a9c75abf-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.635 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:35.637 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.637 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.638 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.638 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.639 DEBUG:boto:Method: GET 2022-09-02T05:37:35.640 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.640 DEBUG:boto:Data: 2022-09-02T05:37:35.641 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.642 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.642 DEBUG:boto:Params: {} 2022-09-02T05:37:35.643 DEBUG:boto:Token: None 2022-09-02T05:37:35.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0= 2022-09-02T05:37:35.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0=', 'Content-Length': '0'} 2022-09-02T05:37:35.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f637dc5dc1efeab-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.648 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.648 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.649 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.649 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.650 DEBUG:boto:Method: GET 2022-09-02T05:37:35.650 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.651 DEBUG:boto:Data: 2022-09-02T05:37:35.651 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.652 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.653 DEBUG:boto:Params: {} 2022-09-02T05:37:35.653 DEBUG:boto:Token: None 2022-09-02T05:37:35.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0= 2022-09-02T05:37:35.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0=', 'Content-Length': '0'} 2022-09-02T05:37:35.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c43337d2efdede-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.658 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.659 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.659 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.660 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.660 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.661 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.661 DEBUG:boto:Data: 2022-09-02T05:37:35.662 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.663 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.663 DEBUG:boto:Params: {} 2022-09-02T05:37:35.664 DEBUG:boto:Token: None 2022-09-02T05:37:35.664 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:77vttpQpHL44XMLNAcbGAdwfBZ0= 2022-09-02T05:37:35.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:77vttpQpHL44XMLNAcbGAdwfBZ0=', 'Content-Length': '0'} 2022-09-02T05:37:35.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13321e2632175ec-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.669 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.669 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.670 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.670 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.671 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.671 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.672 DEBUG:boto:Data: 2022-09-02T05:37:35.672 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.673 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.674 DEBUG:boto:Params: {} 2022-09-02T05:37:35.674 DEBUG:boto:Token: None 2022-09-02T05:37:35.675 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:77vttpQpHL44XMLNAcbGAdwfBZ0= 2022-09-02T05:37:35.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:77vttpQpHL44XMLNAcbGAdwfBZ0=', 'Content-Length': '0'} 2022-09-02T05:37:35.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000312de804b500148a-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:35.680 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.680 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.681 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.681 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.682 DEBUG:boto:Method: GET 2022-09-02T05:37:35.682 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.683 DEBUG:boto:Data: 2022-09-02T05:37:35.683 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.684 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.685 DEBUG:boto:Params: {} 2022-09-02T05:37:35.686 DEBUG:boto:Token: None 2022-09-02T05:37:35.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0= 2022-09-02T05:37:35.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0=', 'Content-Length': '0'} 2022-09-02T05:37:35.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18b91155d990615-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.738 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.739 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.740 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.741 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:35.742 DEBUG:boto:Method: GET 2022-09-02T05:37:35.743 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.744 DEBUG:boto:Data: 2022-09-02T05:37:35.744 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.745 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.746 DEBUG:boto:Params: {} 2022-09-02T05:37:35.747 DEBUG:boto:Token: None 2022-09-02T05:37:35.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:35.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0= 2022-09-02T05:37:35.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nyxs2DgrQoxeVJRi6ySM/YLhTd0=', 'Content-Length': '0'} 2022-09-02T05:37:35.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092eed3b12f5f266b-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.752 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.753 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.753 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.754 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.754 DEBUG:boto:Method: GET 2022-09-02T05:37:35.755 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:35.755 DEBUG:boto:Data: 2022-09-02T05:37:35.756 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.757 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.757 DEBUG:boto:Params: {} 2022-09-02T05:37:35.758 DEBUG:boto:Token: None 2022-09-02T05:37:35.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013515474b3ba9e49-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.763 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.764 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.765 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.765 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.766 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.767 DEBUG:boto:Method: GET 2022-09-02T05:37:35.768 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:35.769 DEBUG:boto:Data: 2022-09-02T05:37:35.770 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.771 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.772 DEBUG:boto:Params: {} 2022-09-02T05:37:35.773 DEBUG:boto:Token: None 2022-09-02T05:37:35.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8491c8b8ec05ca1-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.780 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.781 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.782 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.782 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.783 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:35.783 DEBUG:boto:Data: 2022-09-02T05:37:35.784 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.785 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.785 DEBUG:boto:Params: {} 2022-09-02T05:37:35.786 DEBUG:boto:Token: None 2022-09-02T05:37:35.787 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h 2022-09-02T05:37:35.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AP/jYKtEqA8+SdiDaix7g5S1FhA= 2022-09-02T05:37:35.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AP/jYKtEqA8+SdiDaix7g5S1FhA=', 'Content-Length': '0'} 2022-09-02T05:37:35.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34bc7e99b3811b0-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.791 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.791 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.792 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.792 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:35.793 DEBUG:boto:Data: 2022-09-02T05:37:35.793 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.794 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.795 DEBUG:boto:Params: {} 2022-09-02T05:37:35.795 DEBUG:boto:Token: None 2022-09-02T05:37:35.796 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h 2022-09-02T05:37:35.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AP/jYKtEqA8+SdiDaix7g5S1FhA= 2022-09-02T05:37:35.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AP/jYKtEqA8+SdiDaix7g5S1FhA=', 'Content-Length': '0'} 2022-09-02T05:37:35.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f8e53a033f367d-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:35.803 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.804 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.805 DEBUG:boto:Method: GET 2022-09-02T05:37:35.806 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:35.807 DEBUG:boto:Data: 2022-09-02T05:37:35.808 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.809 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.809 DEBUG:boto:Params: {} 2022-09-02T05:37:35.810 DEBUG:boto:Token: None 2022-09-02T05:37:35.811 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h 2022-09-02T05:37:35.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XXiSHSjZe34VXK+be53XOt+++wY= 2022-09-02T05:37:35.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XXiSHSjZe34VXK+be53XOt+++wY=', 'Content-Length': '0'} 2022-09-02T05:37:35.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea2c4d145c6d961-006311969f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.815 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:35.816 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:35.816 DEBUG:boto:Method: GET 2022-09-02T05:37:35.817 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:35.817 DEBUG:boto:Data: 2022-09-02T05:37:35.818 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.819 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.820 DEBUG:boto:Params: {} 2022-09-02T05:37:35.820 DEBUG:boto:Token: None 2022-09-02T05:37:35.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/h 2022-09-02T05:37:35.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XXiSHSjZe34VXK+be53XOt+++wY= 2022-09-02T05:37:35.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XXiSHSjZe34VXK+be53XOt+++wY=', 'Content-Length': '0'} 2022-09-02T05:37:35.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041bc9c28846f273b-006311969f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.825 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.826 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.827 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.828 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.829 DEBUG:boto:Method: GET 2022-09-02T05:37:35.830 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:35.831 DEBUG:boto:Data: 2022-09-02T05:37:35.831 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:35.833 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.834 DEBUG:boto:Params: {} 2022-09-02T05:37:35.835 DEBUG:boto:Token: None 2022-09-02T05:37:35.836 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4dcc945e2f7be2-006311969f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.882 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.883 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.884 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.884 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.885 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:35.885 DEBUG:boto:Method: GET 2022-09-02T05:37:35.886 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:35.887 DEBUG:boto:Data: 2022-09-02T05:37:35.887 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.888 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.889 DEBUG:boto:Params: {} 2022-09-02T05:37:35.890 DEBUG:boto:Token: None 2022-09-02T05:37:35.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw= 2022-09-02T05:37:35.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJkoxHOPUobkixYzNLrtDZ4hvsw=', 'Content-Length': '0'} 2022-09-02T05:37:35.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003714376fc437122b-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.895 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.896 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-72 zones={test-zone2, test-zone1} 2022-09-02T05:37:35.896 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-72 zones={test-zone3, test-zone1} 2022-09-02T05:37:35.897 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.897 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.898 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.898 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:37:35.899 DEBUG:boto:Data: 2022-09-02T05:37:35.899 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:35.901 DEBUG:boto:Port: 8001 2022-09-02T05:37:35.901 DEBUG:boto:Params: {} 2022-09-02T05:37:35.902 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:37:35.903 DEBUG:boto:Token: None 2022-09-02T05:37:35.903 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2l9S1Cii10lOZFukygHOwxuuM8I= 2022-09-02T05:37:35.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2l9S1Cii10lOZFukygHOwxuuM8I=', 'Content-Length': '0'} 2022-09-02T05:37:35.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000790baa9675ac1768-006311969f-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.910 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.911 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.911 DEBUG:boto:Method: HEAD 2022-09-02T05:37:35.912 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:37:35.912 DEBUG:boto:Data: 2022-09-02T05:37:35.913 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.914 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.914 DEBUG:boto:Params: {} 2022-09-02T05:37:35.915 DEBUG:boto:Token: None 2022-09-02T05:37:35.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/ 2022-09-02T05:37:35.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2l9S1Cii10lOZFukygHOwxuuM8I= 2022-09-02T05:37:35.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2l9S1Cii10lOZFukygHOwxuuM8I=', 'Content-Length': '0'} 2022-09-02T05:37:35.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068efc1cb057a998f-006311969f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:37:35.920 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.921 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.921 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.922 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.922 DEBUG:boto:Method: GET 2022-09-02T05:37:35.923 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.923 DEBUG:boto:Data: 2022-09-02T05:37:35.924 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:35.925 DEBUG:boto:Port: 8001 2022-09-02T05:37:35.925 DEBUG:boto:Params: {} 2022-09-02T05:37:35.926 DEBUG:boto:Token: None 2022-09-02T05:37:35.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/?versions 2022-09-02T05:37:35.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY= 2022-09-02T05:37:35.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY=', 'Content-Length': '0'} 2022-09-02T05:37:35.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f51bdacb9b072ee0-006311969f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.931 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.933 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-02T05:37:35.934 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-02T05:37:35.935 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-02T05:37:35.935 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-02T05:37:35.936 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-02T05:37:35.936 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-02T05:37:35.937 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-02T05:37:35.938 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-02T05:37:35.938 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:37:35.939 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.939 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.940 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.940 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.941 DEBUG:boto:Method: GET 2022-09-02T05:37:35.941 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.942 DEBUG:boto:Data: 2022-09-02T05:37:35.942 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.943 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.944 DEBUG:boto:Params: {} 2022-09-02T05:37:35.945 DEBUG:boto:Token: None 2022-09-02T05:37:35.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/?versions 2022-09-02T05:37:35.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY= 2022-09-02T05:37:35.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY=', 'Content-Length': '0'} 2022-09-02T05:37:35.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b7f5843fd02d1d-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.950 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.952 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-02T05:37:35.952 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-02T05:37:35.953 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-02T05:37:35.953 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-02T05:37:35.954 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-02T05:37:35.954 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-02T05:37:35.954 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-02T05:37:35.955 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-02T05:37:35.955 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.956 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.956 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.957 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.958 DEBUG:boto:Method: GET 2022-09-02T05:37:35.959 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.959 DEBUG:boto:Data: 2022-09-02T05:37:35.960 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:35.961 DEBUG:boto:Port: 8001 2022-09-02T05:37:35.961 DEBUG:boto:Params: {} 2022-09-02T05:37:35.962 DEBUG:boto:Token: None 2022-09-02T05:37:35.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/?versions 2022-09-02T05:37:35.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY= 2022-09-02T05:37:35.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY=', 'Content-Length': '0'} 2022-09-02T05:37:35.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100978b7fb835f49-006311969f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.968 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.969 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:35.970 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:35.970 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.971 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.971 DEBUG:boto:Method: GET 2022-09-02T05:37:35.972 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:35.972 DEBUG:boto:Data: 2022-09-02T05:37:35.973 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.974 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.974 DEBUG:boto:Params: {} 2022-09-02T05:37:35.975 DEBUG:boto:Token: None 2022-09-02T05:37:35.975 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/?versions 2022-09-02T05:37:35.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY= 2022-09-02T05:37:35.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TsQirsYsWxcJYIsVaWgrnPL6BaY=', 'Content-Length': '0'} 2022-09-02T05:37:35.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6b777c8b36f5ea-006311969f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.979 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:35.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:35.981 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:35.982 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:35.982 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:35.983 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:35.983 DEBUG:boto:Method: GET 2022-09-02T05:37:35.984 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:35.984 DEBUG:boto:Data: 2022-09-02T05:37:35.985 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:35.986 DEBUG:boto:Port: 8001 2022-09-02T05:37:35.987 DEBUG:boto:Params: {} 2022-09-02T05:37:35.987 DEBUG:boto:Token: None 2022-09-02T05:37:35.988 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:35.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xfWrthdxmtkEAzNGStFRB0OwSU8= 2022-09-02T05:37:35.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xfWrthdxmtkEAzNGStFRB0OwSU8=', 'Content-Length': '0'} 2022-09-02T05:37:35.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b057a875d498181-006311969f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:35.993 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:35.993 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:35.994 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:35.994 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:35.995 DEBUG:boto:Method: GET 2022-09-02T05:37:35.995 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:35.996 DEBUG:boto:Data: 2022-09-02T05:37:35.997 DEBUG:boto:Headers: {} 2022-09-02T05:37:35.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:35.998 DEBUG:boto:Port: 8000 2022-09-02T05:37:35.998 DEBUG:boto:Params: {} 2022-09-02T05:37:35.999 DEBUG:boto:Token: None 2022-09-02T05:37:35.999 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:35 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xfWrthdxmtkEAzNGStFRB0OwSU8= 2022-09-02T05:37:36.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xfWrthdxmtkEAzNGStFRB0OwSU8=', 'Content-Length': '0'} 2022-09-02T05:37:36.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f2de68c8056836-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.050 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.051 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.051 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.052 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.053 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.053 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.054 DEBUG:boto:Data: 2022-09-02T05:37:36.054 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.055 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.056 DEBUG:boto:Params: {} 2022-09-02T05:37:36.056 DEBUG:boto:Token: None 2022-09-02T05:37:36.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pe/j9oEcDQl87KOZsQXggpKxh8w= 2022-09-02T05:37:36.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pe/j9oEcDQl87KOZsQXggpKxh8w=', 'Content-Length': '0'} 2022-09-02T05:37:36.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9c646daa9bc10f-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.061 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.061 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.062 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.062 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.063 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.063 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.064 DEBUG:boto:Data: 2022-09-02T05:37:36.064 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.065 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.066 DEBUG:boto:Params: {} 2022-09-02T05:37:36.067 DEBUG:boto:Token: None 2022-09-02T05:37:36.067 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pe/j9oEcDQl87KOZsQXggpKxh8w= 2022-09-02T05:37:36.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pe/j9oEcDQl87KOZsQXggpKxh8w=', 'Content-Length': '0'} 2022-09-02T05:37:36.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d375ba5b5e681a57-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:36.072 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.072 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.073 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.073 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.074 DEBUG:boto:Method: GET 2022-09-02T05:37:36.075 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.075 DEBUG:boto:Data: 2022-09-02T05:37:36.076 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.077 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.077 DEBUG:boto:Params: {} 2022-09-02T05:37:36.078 DEBUG:boto:Token: None 2022-09-02T05:37:36.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yfFiOWC39IYfSMRGYK//8mHKcb0= 2022-09-02T05:37:36.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yfFiOWC39IYfSMRGYK//8mHKcb0=', 'Content-Length': '0'} 2022-09-02T05:37:36.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29b557f1daa7eb2-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.083 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.083 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.084 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.084 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:36.085 DEBUG:boto:Method: GET 2022-09-02T05:37:36.085 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.086 DEBUG:boto:Data: 2022-09-02T05:37:36.086 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.088 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.088 DEBUG:boto:Params: {} 2022-09-02T05:37:36.089 DEBUG:boto:Token: None 2022-09-02T05:37:36.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:36.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yfFiOWC39IYfSMRGYK//8mHKcb0= 2022-09-02T05:37:36.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yfFiOWC39IYfSMRGYK//8mHKcb0=', 'Content-Length': '0'} 2022-09-02T05:37:36.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcf2207f04dc6d7-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.094 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.094 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.095 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.095 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.096 DEBUG:boto:Method: GET 2022-09-02T05:37:36.096 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:36.097 DEBUG:boto:Data: 2022-09-02T05:37:36.097 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.098 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.099 DEBUG:boto:Params: {} 2022-09-02T05:37:36.100 DEBUG:boto:Token: None 2022-09-02T05:37:36.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f09e942eb3d14a1-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.146 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.147 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.148 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.148 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.149 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.150 DEBUG:boto:Method: GET 2022-09-02T05:37:36.151 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:36.151 DEBUG:boto:Data: 2022-09-02T05:37:36.152 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.153 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.154 DEBUG:boto:Params: {} 2022-09-02T05:37:36.155 DEBUG:boto:Token: None 2022-09-02T05:37:36.155 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3ca6f3d0f73243d-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.159 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.160 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.161 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.161 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.162 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:36.162 DEBUG:boto:Data: 2022-09-02T05:37:36.163 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.164 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.165 DEBUG:boto:Params: {} 2022-09-02T05:37:36.165 DEBUG:boto:Token: None 2022-09-02T05:37:36.166 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a 2022-09-02T05:37:36.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bYFYyi+cpsxVRkGR2Urf3nlreo8= 2022-09-02T05:37:36.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bYFYyi+cpsxVRkGR2Urf3nlreo8=', 'Content-Length': '0'} 2022-09-02T05:37:36.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068abeae0bae47a00-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.170 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.171 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.172 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.172 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:36.173 DEBUG:boto:Data: 2022-09-02T05:37:36.173 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.175 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.175 DEBUG:boto:Params: {} 2022-09-02T05:37:36.176 DEBUG:boto:Token: None 2022-09-02T05:37:36.177 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a 2022-09-02T05:37:36.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bYFYyi+cpsxVRkGR2Urf3nlreo8= 2022-09-02T05:37:36.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bYFYyi+cpsxVRkGR2Urf3nlreo8=', 'Content-Length': '0'} 2022-09-02T05:37:36.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097dd86c342016654-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:36.182 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.183 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.184 DEBUG:boto:Method: GET 2022-09-02T05:37:36.185 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:36.185 DEBUG:boto:Data: 2022-09-02T05:37:36.186 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.187 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.188 DEBUG:boto:Params: {} 2022-09-02T05:37:36.189 DEBUG:boto:Token: None 2022-09-02T05:37:36.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a 2022-09-02T05:37:36.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l+UBDWponGDqcUjwrFXkIKm8VhE= 2022-09-02T05:37:36.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l+UBDWponGDqcUjwrFXkIKm8VhE=', 'Content-Length': '0'} 2022-09-02T05:37:36.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5761d28e51436c4-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.195 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:36.195 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:36.196 DEBUG:boto:Method: GET 2022-09-02T05:37:36.196 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:36.197 DEBUG:boto:Data: 2022-09-02T05:37:36.198 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.199 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.199 DEBUG:boto:Params: {} 2022-09-02T05:37:36.200 DEBUG:boto:Token: None 2022-09-02T05:37:36.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/a 2022-09-02T05:37:36.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l+UBDWponGDqcUjwrFXkIKm8VhE= 2022-09-02T05:37:36.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l+UBDWponGDqcUjwrFXkIKm8VhE=', 'Content-Length': '0'} 2022-09-02T05:37:36.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e677ab0bdbc6b4-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.250 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.251 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.251 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.252 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.252 DEBUG:boto:Method: GET 2022-09-02T05:37:36.253 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:36.253 DEBUG:boto:Data: 2022-09-02T05:37:36.254 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.255 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.256 DEBUG:boto:Params: {} 2022-09-02T05:37:36.256 DEBUG:boto:Token: None 2022-09-02T05:37:36.257 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059236738e20f6381-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.262 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.263 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.263 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.264 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.264 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:36.265 DEBUG:boto:Method: GET 2022-09-02T05:37:36.265 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:36.266 DEBUG:boto:Data: 2022-09-02T05:37:36.266 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.267 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.268 DEBUG:boto:Params: {} 2022-09-02T05:37:36.268 DEBUG:boto:Token: None 2022-09-02T05:37:36.269 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048308bd635983e80-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.273 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:36.275 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.275 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.276 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.276 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.277 DEBUG:boto:Method: GET 2022-09-02T05:37:36.277 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.278 DEBUG:boto:Data: 2022-09-02T05:37:36.279 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.280 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.280 DEBUG:boto:Params: {} 2022-09-02T05:37:36.281 DEBUG:boto:Token: None 2022-09-02T05:37:36.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU= 2022-09-02T05:37:36.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU=', 'Content-Length': '0'} 2022-09-02T05:37:36.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038aa02876074a7d-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.286 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.286 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.287 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.287 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.288 DEBUG:boto:Method: GET 2022-09-02T05:37:36.288 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.289 DEBUG:boto:Data: 2022-09-02T05:37:36.289 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.290 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.291 DEBUG:boto:Params: {} 2022-09-02T05:37:36.292 DEBUG:boto:Token: None 2022-09-02T05:37:36.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU= 2022-09-02T05:37:36.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU=', 'Content-Length': '0'} 2022-09-02T05:37:36.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de3f4c48a258dc8f-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.297 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.297 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.298 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.299 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.299 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.300 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.300 DEBUG:boto:Data: 2022-09-02T05:37:36.301 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.302 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.303 DEBUG:boto:Params: {} 2022-09-02T05:37:36.303 DEBUG:boto:Token: None 2022-09-02T05:37:36.304 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/s2FIB1SCPAk1c7xBvWsNomlgSE= 2022-09-02T05:37:36.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/s2FIB1SCPAk1c7xBvWsNomlgSE=', 'Content-Length': '0'} 2022-09-02T05:37:36.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e2af4fa73d59bf-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.308 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.309 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.309 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.310 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.310 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.311 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.311 DEBUG:boto:Data: 2022-09-02T05:37:36.312 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.313 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.313 DEBUG:boto:Params: {} 2022-09-02T05:37:36.314 DEBUG:boto:Token: None 2022-09-02T05:37:36.315 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/s2FIB1SCPAk1c7xBvWsNomlgSE= 2022-09-02T05:37:36.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/s2FIB1SCPAk1c7xBvWsNomlgSE=', 'Content-Length': '0'} 2022-09-02T05:37:36.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000392773d435adf6d9-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:36.320 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.320 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.321 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.322 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.323 DEBUG:boto:Method: GET 2022-09-02T05:37:36.324 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.325 DEBUG:boto:Data: 2022-09-02T05:37:36.325 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.327 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.328 DEBUG:boto:Params: {} 2022-09-02T05:37:36.328 DEBUG:boto:Token: None 2022-09-02T05:37:36.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU= 2022-09-02T05:37:36.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU=', 'Content-Length': '0'} 2022-09-02T05:37:36.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebd90eb63f5ce59-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.333 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.334 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.335 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.335 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:36.336 DEBUG:boto:Method: GET 2022-09-02T05:37:36.336 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.337 DEBUG:boto:Data: 2022-09-02T05:37:36.337 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.338 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.339 DEBUG:boto:Params: {} 2022-09-02T05:37:36.340 DEBUG:boto:Token: None 2022-09-02T05:37:36.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:36.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU= 2022-09-02T05:37:36.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhyPpZpwdCnNY2VOH9lYIUBRxCU=', 'Content-Length': '0'} 2022-09-02T05:37:36.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d71e5972b6aff46-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.344 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.345 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.345 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.346 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.346 DEBUG:boto:Method: GET 2022-09-02T05:37:36.347 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:36.348 DEBUG:boto:Data: 2022-09-02T05:37:36.348 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.349 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.350 DEBUG:boto:Params: {} 2022-09-02T05:37:36.350 DEBUG:boto:Token: None 2022-09-02T05:37:36.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a9c955471ae16ee-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.355 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.356 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.357 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.358 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.358 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.359 DEBUG:boto:Method: GET 2022-09-02T05:37:36.359 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:36.361 DEBUG:boto:Data: 2022-09-02T05:37:36.361 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.362 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.363 DEBUG:boto:Params: {} 2022-09-02T05:37:36.364 DEBUG:boto:Token: None 2022-09-02T05:37:36.364 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044c880ca7bde272f-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.369 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.370 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.370 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.371 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.371 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:36.372 DEBUG:boto:Data: 2022-09-02T05:37:36.372 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.373 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.374 DEBUG:boto:Params: {} 2022-09-02T05:37:36.375 DEBUG:boto:Token: None 2022-09-02T05:37:36.375 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b 2022-09-02T05:37:36.376 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BDaA6QUpdAXG2Z3fkrA7UIdrcYc= 2022-09-02T05:37:36.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BDaA6QUpdAXG2Z3fkrA7UIdrcYc=', 'Content-Length': '0'} 2022-09-02T05:37:36.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf41bed10da9807e-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.379 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.380 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.380 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.381 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:36.381 DEBUG:boto:Data: 2022-09-02T05:37:36.382 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.383 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.383 DEBUG:boto:Params: {} 2022-09-02T05:37:36.384 DEBUG:boto:Token: None 2022-09-02T05:37:36.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b 2022-09-02T05:37:36.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BDaA6QUpdAXG2Z3fkrA7UIdrcYc= 2022-09-02T05:37:36.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BDaA6QUpdAXG2Z3fkrA7UIdrcYc=', 'Content-Length': '0'} 2022-09-02T05:37:36.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006631ad6f761de1cc-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:36.389 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.390 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.390 DEBUG:boto:Method: GET 2022-09-02T05:37:36.391 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:36.391 DEBUG:boto:Data: 2022-09-02T05:37:36.392 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.393 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.393 DEBUG:boto:Params: {} 2022-09-02T05:37:36.394 DEBUG:boto:Token: None 2022-09-02T05:37:36.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b 2022-09-02T05:37:36.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PT1VXlGrr+vnNURnoq9q9cNC/pU= 2022-09-02T05:37:36.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PT1VXlGrr+vnNURnoq9q9cNC/pU=', 'Content-Length': '0'} 2022-09-02T05:37:36.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022112e7f23c313f3-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.442 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:36.443 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:36.444 DEBUG:boto:Method: GET 2022-09-02T05:37:36.444 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:36.445 DEBUG:boto:Data: 2022-09-02T05:37:36.445 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.446 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.447 DEBUG:boto:Params: {} 2022-09-02T05:37:36.448 DEBUG:boto:Token: None 2022-09-02T05:37:36.448 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/b 2022-09-02T05:37:36.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PT1VXlGrr+vnNURnoq9q9cNC/pU= 2022-09-02T05:37:36.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PT1VXlGrr+vnNURnoq9q9cNC/pU=', 'Content-Length': '0'} 2022-09-02T05:37:36.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb2ac18c3b959ff-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.453 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.453 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.454 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.455 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.455 DEBUG:boto:Method: GET 2022-09-02T05:37:36.456 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:36.456 DEBUG:boto:Data: 2022-09-02T05:37:36.457 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.460 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.461 DEBUG:boto:Params: {} 2022-09-02T05:37:36.462 DEBUG:boto:Token: None 2022-09-02T05:37:36.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b153ea89335a9d1-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.470 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.471 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.472 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.473 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.474 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:36.475 DEBUG:boto:Method: GET 2022-09-02T05:37:36.476 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:36.477 DEBUG:boto:Data: 2022-09-02T05:37:36.487 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.489 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.490 DEBUG:boto:Params: {} 2022-09-02T05:37:36.491 DEBUG:boto:Token: None 2022-09-02T05:37:36.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047960641f692e51a-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.503 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:36.505 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.505 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.506 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.506 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.507 DEBUG:boto:Method: GET 2022-09-02T05:37:36.508 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.508 DEBUG:boto:Data: 2022-09-02T05:37:36.509 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.510 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.511 DEBUG:boto:Params: {} 2022-09-02T05:37:36.512 DEBUG:boto:Token: None 2022-09-02T05:37:36.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA= 2022-09-02T05:37:36.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA=', 'Content-Length': '0'} 2022-09-02T05:37:36.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330c50d50eed143a-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.516 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.517 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.518 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.518 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.519 DEBUG:boto:Method: GET 2022-09-02T05:37:36.520 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.520 DEBUG:boto:Data: 2022-09-02T05:37:36.521 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.522 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.523 DEBUG:boto:Params: {} 2022-09-02T05:37:36.524 DEBUG:boto:Token: None 2022-09-02T05:37:36.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA= 2022-09-02T05:37:36.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA=', 'Content-Length': '0'} 2022-09-02T05:37:36.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e01b67fece49e34-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.570 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.571 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.571 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.572 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.572 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.573 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.574 DEBUG:boto:Data: 2022-09-02T05:37:36.574 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.576 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.576 DEBUG:boto:Params: {} 2022-09-02T05:37:36.577 DEBUG:boto:Token: None 2022-09-02T05:37:36.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/bV9ZOIEVoGno3xfCXdg111g2Ww= 2022-09-02T05:37:36.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/bV9ZOIEVoGno3xfCXdg111g2Ww=', 'Content-Length': '0'} 2022-09-02T05:37:36.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb23b5b9de5382e9-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.582 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.582 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.583 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.583 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.584 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.584 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.585 DEBUG:boto:Data: 2022-09-02T05:37:36.585 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.586 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.587 DEBUG:boto:Params: {} 2022-09-02T05:37:36.588 DEBUG:boto:Token: None 2022-09-02T05:37:36.589 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/bV9ZOIEVoGno3xfCXdg111g2Ww= 2022-09-02T05:37:36.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/bV9ZOIEVoGno3xfCXdg111g2Ww=', 'Content-Length': '0'} 2022-09-02T05:37:36.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2d83ba65945730-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:36.594 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.595 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.595 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.596 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.596 DEBUG:boto:Method: GET 2022-09-02T05:37:36.597 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.597 DEBUG:boto:Data: 2022-09-02T05:37:36.598 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.600 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.601 DEBUG:boto:Params: {} 2022-09-02T05:37:36.601 DEBUG:boto:Token: None 2022-09-02T05:37:36.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA= 2022-09-02T05:37:36.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA=', 'Content-Length': '0'} 2022-09-02T05:37:36.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddbeb3b12f1739c0-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.607 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.607 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.608 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.609 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:36.609 DEBUG:boto:Method: GET 2022-09-02T05:37:36.610 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.611 DEBUG:boto:Data: 2022-09-02T05:37:36.611 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.612 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.613 DEBUG:boto:Params: {} 2022-09-02T05:37:36.614 DEBUG:boto:Token: None 2022-09-02T05:37:36.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:36.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA= 2022-09-02T05:37:36.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y659f7PnU5kRccIYSPvGJddKTZA=', 'Content-Length': '0'} 2022-09-02T05:37:36.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa6572f32b393c0-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.619 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.619 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.620 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.620 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.621 DEBUG:boto:Method: GET 2022-09-02T05:37:36.622 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:36.622 DEBUG:boto:Data: 2022-09-02T05:37:36.623 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.624 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.624 DEBUG:boto:Params: {} 2022-09-02T05:37:36.625 DEBUG:boto:Token: None 2022-09-02T05:37:36.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf9777f0845fadc-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.631 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.632 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.632 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.633 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.634 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.635 DEBUG:boto:Method: GET 2022-09-02T05:37:36.636 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:36.637 DEBUG:boto:Data: 2022-09-02T05:37:36.638 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.640 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.641 DEBUG:boto:Params: {} 2022-09-02T05:37:36.642 DEBUG:boto:Token: None 2022-09-02T05:37:36.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7419def6ce6169-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.650 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.651 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.652 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.653 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.654 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:36.655 DEBUG:boto:Data: 2022-09-02T05:37:36.656 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.658 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.658 DEBUG:boto:Params: {} 2022-09-02T05:37:36.659 DEBUG:boto:Token: None 2022-09-02T05:37:36.660 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c 2022-09-02T05:37:36.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C79AVHlTNrxQa5P/fOmHU4OuqiI= 2022-09-02T05:37:36.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C79AVHlTNrxQa5P/fOmHU4OuqiI=', 'Content-Length': '0'} 2022-09-02T05:37:36.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dabe091779991827-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.665 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.666 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.667 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.668 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:36.669 DEBUG:boto:Data: 2022-09-02T05:37:36.670 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.672 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.673 DEBUG:boto:Params: {} 2022-09-02T05:37:36.674 DEBUG:boto:Token: None 2022-09-02T05:37:36.675 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c 2022-09-02T05:37:36.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C79AVHlTNrxQa5P/fOmHU4OuqiI= 2022-09-02T05:37:36.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C79AVHlTNrxQa5P/fOmHU4OuqiI=', 'Content-Length': '0'} 2022-09-02T05:37:36.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3fc522a24ff0a8a-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:36.679 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.680 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.681 DEBUG:boto:Method: GET 2022-09-02T05:37:36.681 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:36.682 DEBUG:boto:Data: 2022-09-02T05:37:36.682 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.683 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.684 DEBUG:boto:Params: {} 2022-09-02T05:37:36.684 DEBUG:boto:Token: None 2022-09-02T05:37:36.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c 2022-09-02T05:37:36.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VZxc1OCN0EnqQDN427NIaV24tQY= 2022-09-02T05:37:36.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VZxc1OCN0EnqQDN427NIaV24tQY=', 'Content-Length': '0'} 2022-09-02T05:37:36.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081cc218b7a2e6581-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.689 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:36.690 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:36.691 DEBUG:boto:Method: GET 2022-09-02T05:37:36.691 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:36.692 DEBUG:boto:Data: 2022-09-02T05:37:36.692 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.694 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.694 DEBUG:boto:Params: {} 2022-09-02T05:37:36.695 DEBUG:boto:Token: None 2022-09-02T05:37:36.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/c 2022-09-02T05:37:36.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VZxc1OCN0EnqQDN427NIaV24tQY= 2022-09-02T05:37:36.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VZxc1OCN0EnqQDN427NIaV24tQY=', 'Content-Length': '0'} 2022-09-02T05:37:36.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbeb647bc6a0485-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.742 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.743 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.743 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.744 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.745 DEBUG:boto:Method: GET 2022-09-02T05:37:36.745 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:36.746 DEBUG:boto:Data: 2022-09-02T05:37:36.746 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.747 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.748 DEBUG:boto:Params: {} 2022-09-02T05:37:36.748 DEBUG:boto:Token: None 2022-09-02T05:37:36.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e94a0f5255432521-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.755 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.756 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.757 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.758 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.759 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:36.760 DEBUG:boto:Method: GET 2022-09-02T05:37:36.761 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:36.762 DEBUG:boto:Data: 2022-09-02T05:37:36.763 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.764 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.765 DEBUG:boto:Params: {} 2022-09-02T05:37:36.766 DEBUG:boto:Token: None 2022-09-02T05:37:36.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebeb7d27b334220-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.771 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:36.774 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.775 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.776 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.777 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.778 DEBUG:boto:Method: GET 2022-09-02T05:37:36.779 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.779 DEBUG:boto:Data: 2022-09-02T05:37:36.780 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.782 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.782 DEBUG:boto:Params: {} 2022-09-02T05:37:36.783 DEBUG:boto:Token: None 2022-09-02T05:37:36.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY= 2022-09-02T05:37:36.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY=', 'Content-Length': '0'} 2022-09-02T05:37:36.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070748daea632fb88-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.789 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.789 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.790 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.791 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.791 DEBUG:boto:Method: GET 2022-09-02T05:37:36.792 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.793 DEBUG:boto:Data: 2022-09-02T05:37:36.793 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.794 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.795 DEBUG:boto:Params: {} 2022-09-02T05:37:36.796 DEBUG:boto:Token: None 2022-09-02T05:37:36.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY= 2022-09-02T05:37:36.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY=', 'Content-Length': '0'} 2022-09-02T05:37:36.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f30f2b310586408-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.801 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.801 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.802 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.802 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.803 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.803 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.804 DEBUG:boto:Data: 2022-09-02T05:37:36.804 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.805 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.806 DEBUG:boto:Params: {} 2022-09-02T05:37:36.807 DEBUG:boto:Token: None 2022-09-02T05:37:36.808 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XvD6tA/WabzJHUCwr+b/ipKKDK4= 2022-09-02T05:37:36.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XvD6tA/WabzJHUCwr+b/ipKKDK4=', 'Content-Length': '0'} 2022-09-02T05:37:36.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfbc074f441b310d-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.815 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.816 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.817 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.817 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.818 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.819 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.820 DEBUG:boto:Data: 2022-09-02T05:37:36.821 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.822 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.822 DEBUG:boto:Params: {} 2022-09-02T05:37:36.823 DEBUG:boto:Token: None 2022-09-02T05:37:36.824 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XvD6tA/WabzJHUCwr+b/ipKKDK4= 2022-09-02T05:37:36.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XvD6tA/WabzJHUCwr+b/ipKKDK4=', 'Content-Length': '0'} 2022-09-02T05:37:36.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000250a280ee76feaa-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:36.829 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.830 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.830 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.831 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.831 DEBUG:boto:Method: GET 2022-09-02T05:37:36.832 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.833 DEBUG:boto:Data: 2022-09-02T05:37:36.833 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.835 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.835 DEBUG:boto:Params: {} 2022-09-02T05:37:36.836 DEBUG:boto:Token: None 2022-09-02T05:37:36.836 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY= 2022-09-02T05:37:36.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY=', 'Content-Length': '0'} 2022-09-02T05:37:36.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04bbd9477eb3092-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.841 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.842 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.843 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.843 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:36.844 DEBUG:boto:Method: GET 2022-09-02T05:37:36.845 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.845 DEBUG:boto:Data: 2022-09-02T05:37:36.846 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.847 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.848 DEBUG:boto:Params: {} 2022-09-02T05:37:36.849 DEBUG:boto:Token: None 2022-09-02T05:37:36.850 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:36.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY= 2022-09-02T05:37:36.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRolweP0XgORVWwlNkJuCVfD9SY=', 'Content-Length': '0'} 2022-09-02T05:37:36.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001061cff449b08ca1-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.856 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.856 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.857 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:36.858 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:36.858 DEBUG:boto:Method: GET 2022-09-02T05:37:36.859 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:36.859 DEBUG:boto:Data: 2022-09-02T05:37:36.860 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.861 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.862 DEBUG:boto:Params: {} 2022-09-02T05:37:36.863 DEBUG:boto:Token: None 2022-09-02T05:37:36.864 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094ce5d191d4d1867-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.871 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.872 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:36.873 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:36.874 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:36.874 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:36.875 DEBUG:boto:Method: GET 2022-09-02T05:37:36.875 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:36.876 DEBUG:boto:Data: 2022-09-02T05:37:36.877 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.878 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.878 DEBUG:boto:Params: {} 2022-09-02T05:37:36.879 DEBUG:boto:Token: None 2022-09-02T05:37:36.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/ 2022-09-02T05:37:36.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w= 2022-09-02T05:37:36.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SECUlWE2nS59xJYFyUL06+E491w=', 'Content-Length': '0'} 2022-09-02T05:37:36.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a99bf40454f55051-00631196a0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.926 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:36.928 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.929 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.930 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.931 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:36.932 DEBUG:boto:Data: 2022-09-02T05:37:36.933 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.935 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.935 DEBUG:boto:Params: {} 2022-09-02T05:37:36.937 DEBUG:boto:Token: None 2022-09-02T05:37:36.938 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d 2022-09-02T05:37:36.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uDpgqWBgnKcQPBpk5IsOjFiwa8U= 2022-09-02T05:37:36.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uDpgqWBgnKcQPBpk5IsOjFiwa8U=', 'Content-Length': '0'} 2022-09-02T05:37:36.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a3f1fb7e3a0621-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.943 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.944 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.944 DEBUG:boto:Method: HEAD 2022-09-02T05:37:36.945 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:36.945 DEBUG:boto:Data: 2022-09-02T05:37:36.946 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:36.947 DEBUG:boto:Port: 8000 2022-09-02T05:37:36.948 DEBUG:boto:Params: {} 2022-09-02T05:37:36.949 DEBUG:boto:Token: None 2022-09-02T05:37:36.949 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d 2022-09-02T05:37:36.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uDpgqWBgnKcQPBpk5IsOjFiwa8U= 2022-09-02T05:37:36.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uDpgqWBgnKcQPBpk5IsOjFiwa8U=', 'Content-Length': '0'} 2022-09-02T05:37:36.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187991f848d6b616-00631196a0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:36.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:36.954 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:36.955 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:36.955 DEBUG:boto:Method: GET 2022-09-02T05:37:36.956 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:36.956 DEBUG:boto:Data: 2022-09-02T05:37:36.957 DEBUG:boto:Headers: {} 2022-09-02T05:37:36.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:36.958 DEBUG:boto:Port: 8001 2022-09-02T05:37:36.959 DEBUG:boto:Params: {} 2022-09-02T05:37:36.959 DEBUG:boto:Token: None 2022-09-02T05:37:36.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:36 GMT /xljbkf-72/d 2022-09-02T05:37:36.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c0UHBePrdLfwqtqQbqJ3Xep62Mk= 2022-09-02T05:37:36.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c0UHBePrdLfwqtqQbqJ3Xep62Mk=', 'Content-Length': '0'} 2022-09-02T05:37:36.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d1f1e7b37c6bba-00631196a0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.006 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:37.007 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:37.007 DEBUG:boto:Method: GET 2022-09-02T05:37:37.008 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:37.009 DEBUG:boto:Data: 2022-09-02T05:37:37.009 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.010 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.011 DEBUG:boto:Params: {} 2022-09-02T05:37:37.011 DEBUG:boto:Token: None 2022-09-02T05:37:37.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/d 2022-09-02T05:37:37.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vCFvkNIEJRDOcoYcKiLp3QdsSWI= 2022-09-02T05:37:37.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vCFvkNIEJRDOcoYcKiLp3QdsSWI=', 'Content-Length': '0'} 2022-09-02T05:37:37.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b0bab076a0fc96-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.017 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.017 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.018 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:37.018 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:37.019 DEBUG:boto:Method: GET 2022-09-02T05:37:37.019 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:37.020 DEBUG:boto:Data: 2022-09-02T05:37:37.020 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.022 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.022 DEBUG:boto:Params: {} 2022-09-02T05:37:37.023 DEBUG:boto:Token: None 2022-09-02T05:37:37.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be257d8ea6194891-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.030 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.032 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.033 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.034 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:37.035 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:37.036 DEBUG:boto:Method: GET 2022-09-02T05:37:37.036 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:37.037 DEBUG:boto:Data: 2022-09-02T05:37:37.038 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.039 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.040 DEBUG:boto:Params: {} 2022-09-02T05:37:37.040 DEBUG:boto:Token: None 2022-09-02T05:37:37.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c1db953d67b427-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.046 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:37.047 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.048 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.049 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.049 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.050 DEBUG:boto:Method: GET 2022-09-02T05:37:37.051 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.051 DEBUG:boto:Data: 2022-09-02T05:37:37.052 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.053 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.053 DEBUG:boto:Params: {} 2022-09-02T05:37:37.054 DEBUG:boto:Token: None 2022-09-02T05:37:37.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4= 2022-09-02T05:37:37.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4=', 'Content-Length': '0'} 2022-09-02T05:37:37.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d9efa9d6a5aa64-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.060 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.061 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.061 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.062 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.063 DEBUG:boto:Method: GET 2022-09-02T05:37:37.063 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.064 DEBUG:boto:Data: 2022-09-02T05:37:37.065 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.066 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.066 DEBUG:boto:Params: {} 2022-09-02T05:37:37.067 DEBUG:boto:Token: None 2022-09-02T05:37:37.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4= 2022-09-02T05:37:37.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4=', 'Content-Length': '0'} 2022-09-02T05:37:37.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d728b15f048a4fa5-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.072 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.073 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.073 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.074 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.074 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.075 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.076 DEBUG:boto:Data: 2022-09-02T05:37:37.076 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.077 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.078 DEBUG:boto:Params: {} 2022-09-02T05:37:37.078 DEBUG:boto:Token: None 2022-09-02T05:37:37.079 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NsokTcdEcLirUbpnrJ4AiYmrN70= 2022-09-02T05:37:37.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NsokTcdEcLirUbpnrJ4AiYmrN70=', 'Content-Length': '0'} 2022-09-02T05:37:37.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008694d2540ea81dde-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.083 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.084 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.084 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.085 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.085 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.086 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.086 DEBUG:boto:Data: 2022-09-02T05:37:37.087 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.088 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.088 DEBUG:boto:Params: {} 2022-09-02T05:37:37.089 DEBUG:boto:Token: None 2022-09-02T05:37:37.089 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NsokTcdEcLirUbpnrJ4AiYmrN70= 2022-09-02T05:37:37.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NsokTcdEcLirUbpnrJ4AiYmrN70=', 'Content-Length': '0'} 2022-09-02T05:37:37.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b933ff1fa497ea69-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:37.096 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.097 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.097 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.098 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.098 DEBUG:boto:Method: GET 2022-09-02T05:37:37.099 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.099 DEBUG:boto:Data: 2022-09-02T05:37:37.100 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.101 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.102 DEBUG:boto:Params: {} 2022-09-02T05:37:37.103 DEBUG:boto:Token: None 2022-09-02T05:37:37.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4= 2022-09-02T05:37:37.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4=', 'Content-Length': '0'} 2022-09-02T05:37:37.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f877c8b79e59fe30-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.155 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.156 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.158 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.159 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:37.160 DEBUG:boto:Method: GET 2022-09-02T05:37:37.160 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.161 DEBUG:boto:Data: 2022-09-02T05:37:37.161 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.163 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.163 DEBUG:boto:Params: {} 2022-09-02T05:37:37.164 DEBUG:boto:Token: None 2022-09-02T05:37:37.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:37.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4= 2022-09-02T05:37:37.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UWfYELLDjs4lGFjxI6muKIHZFI4=', 'Content-Length': '0'} 2022-09-02T05:37:37.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d9899bba4c90af2-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.169 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.170 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.171 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.171 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.172 DEBUG:boto:Method: GET 2022-09-02T05:37:37.173 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:37.174 DEBUG:boto:Data: 2022-09-02T05:37:37.175 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.177 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.177 DEBUG:boto:Params: {} 2022-09-02T05:37:37.179 DEBUG:boto:Token: None 2022-09-02T05:37:37.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca0b5c9a37c6069-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.185 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.187 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.187 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.188 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.189 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.190 DEBUG:boto:Method: GET 2022-09-02T05:37:37.191 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:37.192 DEBUG:boto:Data: 2022-09-02T05:37:37.193 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.196 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.196 DEBUG:boto:Params: {} 2022-09-02T05:37:37.197 DEBUG:boto:Token: None 2022-09-02T05:37:37.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d525a1e9dec49b0-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.202 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.203 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.204 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.204 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.205 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:37.205 DEBUG:boto:Data: 2022-09-02T05:37:37.206 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.207 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.207 DEBUG:boto:Params: {} 2022-09-02T05:37:37.208 DEBUG:boto:Token: None 2022-09-02T05:37:37.209 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e 2022-09-02T05:37:37.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gcIV0WCfZV9IaftCeZ7laEmG7kE= 2022-09-02T05:37:37.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gcIV0WCfZV9IaftCeZ7laEmG7kE=', 'Content-Length': '0'} 2022-09-02T05:37:37.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccfa5f435f2276c1-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.213 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.214 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.215 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.216 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:37.217 DEBUG:boto:Data: 2022-09-02T05:37:37.218 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.219 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.219 DEBUG:boto:Params: {} 2022-09-02T05:37:37.220 DEBUG:boto:Token: None 2022-09-02T05:37:37.220 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e 2022-09-02T05:37:37.221 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gcIV0WCfZV9IaftCeZ7laEmG7kE= 2022-09-02T05:37:37.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gcIV0WCfZV9IaftCeZ7laEmG7kE=', 'Content-Length': '0'} 2022-09-02T05:37:37.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078bf65bbd8646071-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:37.225 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.226 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.226 DEBUG:boto:Method: GET 2022-09-02T05:37:37.227 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:37.227 DEBUG:boto:Data: 2022-09-02T05:37:37.228 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.229 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.230 DEBUG:boto:Params: {} 2022-09-02T05:37:37.231 DEBUG:boto:Token: None 2022-09-02T05:37:37.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e 2022-09-02T05:37:37.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uwxRMWpPVYzrR1GSwOaY0naQ28= 2022-09-02T05:37:37.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uwxRMWpPVYzrR1GSwOaY0naQ28=', 'Content-Length': '0'} 2022-09-02T05:37:37.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd23e57a8ad1dab3-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.236 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:37.237 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:37.238 DEBUG:boto:Method: GET 2022-09-02T05:37:37.238 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:37.239 DEBUG:boto:Data: 2022-09-02T05:37:37.239 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.241 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.242 DEBUG:boto:Params: {} 2022-09-02T05:37:37.242 DEBUG:boto:Token: None 2022-09-02T05:37:37.243 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/e 2022-09-02T05:37:37.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uwxRMWpPVYzrR1GSwOaY0naQ28= 2022-09-02T05:37:37.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uwxRMWpPVYzrR1GSwOaY0naQ28=', 'Content-Length': '0'} 2022-09-02T05:37:37.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad37542fe215a4bd-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.248 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.249 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.249 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.250 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.251 DEBUG:boto:Method: GET 2022-09-02T05:37:37.251 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:37.252 DEBUG:boto:Data: 2022-09-02T05:37:37.252 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.253 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.254 DEBUG:boto:Params: {} 2022-09-02T05:37:37.254 DEBUG:boto:Token: None 2022-09-02T05:37:37.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd3d11e994070e7-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.302 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.304 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.305 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.305 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.307 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:37.308 DEBUG:boto:Method: GET 2022-09-02T05:37:37.309 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:37.309 DEBUG:boto:Data: 2022-09-02T05:37:37.310 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.312 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.313 DEBUG:boto:Params: {} 2022-09-02T05:37:37.313 DEBUG:boto:Token: None 2022-09-02T05:37:37.314 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac45ac35735fcc5-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.320 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:37.323 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.323 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.324 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.325 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.326 DEBUG:boto:Method: GET 2022-09-02T05:37:37.327 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.328 DEBUG:boto:Data: 2022-09-02T05:37:37.329 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.331 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.332 DEBUG:boto:Params: {} 2022-09-02T05:37:37.333 DEBUG:boto:Token: None 2022-09-02T05:37:37.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis= 2022-09-02T05:37:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis=', 'Content-Length': '0'} 2022-09-02T05:37:37.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ee2df77a5f972b-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.340 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.340 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.341 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.342 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.342 DEBUG:boto:Method: GET 2022-09-02T05:37:37.343 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.343 DEBUG:boto:Data: 2022-09-02T05:37:37.344 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.345 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.346 DEBUG:boto:Params: {} 2022-09-02T05:37:37.346 DEBUG:boto:Token: None 2022-09-02T05:37:37.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis= 2022-09-02T05:37:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis=', 'Content-Length': '0'} 2022-09-02T05:37:37.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011daf24633442e32-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.352 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.353 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.354 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.355 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.356 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.357 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.359 DEBUG:boto:Data: 2022-09-02T05:37:37.360 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.362 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.363 DEBUG:boto:Params: {} 2022-09-02T05:37:37.364 DEBUG:boto:Token: None 2022-09-02T05:37:37.365 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ljzNe1Bk8Duz2iUjdFs7y/rJlfY= 2022-09-02T05:37:37.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ljzNe1Bk8Duz2iUjdFs7y/rJlfY=', 'Content-Length': '0'} 2022-09-02T05:37:37.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd1049d2fc8025b4-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.371 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.372 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.372 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.373 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.373 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.374 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.374 DEBUG:boto:Data: 2022-09-02T05:37:37.375 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.376 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.376 DEBUG:boto:Params: {} 2022-09-02T05:37:37.377 DEBUG:boto:Token: None 2022-09-02T05:37:37.378 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ljzNe1Bk8Duz2iUjdFs7y/rJlfY= 2022-09-02T05:37:37.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ljzNe1Bk8Duz2iUjdFs7y/rJlfY=', 'Content-Length': '0'} 2022-09-02T05:37:37.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e89b6dcec62a5ea-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:37.383 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.384 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.384 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.385 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.385 DEBUG:boto:Method: GET 2022-09-02T05:37:37.386 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.386 DEBUG:boto:Data: 2022-09-02T05:37:37.387 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.388 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.389 DEBUG:boto:Params: {} 2022-09-02T05:37:37.389 DEBUG:boto:Token: None 2022-09-02T05:37:37.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis= 2022-09-02T05:37:37.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis=', 'Content-Length': '0'} 2022-09-02T05:37:37.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab63f94db39f9e5-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.395 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.395 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.396 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.396 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:37.397 DEBUG:boto:Method: GET 2022-09-02T05:37:37.397 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.398 DEBUG:boto:Data: 2022-09-02T05:37:37.399 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.400 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.400 DEBUG:boto:Params: {} 2022-09-02T05:37:37.401 DEBUG:boto:Token: None 2022-09-02T05:37:37.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:37.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis= 2022-09-02T05:37:37.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RaQLMy19ejet8q7ecLBvG2iUqis=', 'Content-Length': '0'} 2022-09-02T05:37:37.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b9f0b1c9ea9cdd6-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.407 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.407 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.408 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.408 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.409 DEBUG:boto:Method: GET 2022-09-02T05:37:37.410 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:37.410 DEBUG:boto:Data: 2022-09-02T05:37:37.411 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.412 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.412 DEBUG:boto:Params: {} 2022-09-02T05:37:37.414 DEBUG:boto:Token: None 2022-09-02T05:37:37.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe2a98ab44a9951-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.420 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.421 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.421 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.422 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.422 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.423 DEBUG:boto:Method: GET 2022-09-02T05:37:37.423 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:37.424 DEBUG:boto:Data: 2022-09-02T05:37:37.424 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.425 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.426 DEBUG:boto:Params: {} 2022-09-02T05:37:37.427 DEBUG:boto:Token: None 2022-09-02T05:37:37.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f164fcbc748c618-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.474 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.475 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.476 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.476 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.477 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:37.477 DEBUG:boto:Data: 2022-09-02T05:37:37.478 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.479 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.479 DEBUG:boto:Params: {} 2022-09-02T05:37:37.480 DEBUG:boto:Token: None 2022-09-02T05:37:37.480 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f 2022-09-02T05:37:37.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xEi0+lnQq+UtfDJTv697EVcXjXM= 2022-09-02T05:37:37.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xEi0+lnQq+UtfDJTv697EVcXjXM=', 'Content-Length': '0'} 2022-09-02T05:37:37.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cbb5a94ad84da61-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.485 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.485 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.486 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.486 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:37.487 DEBUG:boto:Data: 2022-09-02T05:37:37.487 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.488 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.489 DEBUG:boto:Params: {} 2022-09-02T05:37:37.489 DEBUG:boto:Token: None 2022-09-02T05:37:37.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f 2022-09-02T05:37:37.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xEi0+lnQq+UtfDJTv697EVcXjXM= 2022-09-02T05:37:37.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xEi0+lnQq+UtfDJTv697EVcXjXM=', 'Content-Length': '0'} 2022-09-02T05:37:37.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002264156c27e51bca-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:37.495 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.495 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.496 DEBUG:boto:Method: GET 2022-09-02T05:37:37.496 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:37.497 DEBUG:boto:Data: 2022-09-02T05:37:37.497 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.498 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.499 DEBUG:boto:Params: {} 2022-09-02T05:37:37.500 DEBUG:boto:Token: None 2022-09-02T05:37:37.500 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f 2022-09-02T05:37:37.501 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1zky1AFvpdIRML3RVo0e1cSQF/o= 2022-09-02T05:37:37.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1zky1AFvpdIRML3RVo0e1cSQF/o=', 'Content-Length': '0'} 2022-09-02T05:37:37.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5c036fa65ec3f9-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.504 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:37.505 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:37.505 DEBUG:boto:Method: GET 2022-09-02T05:37:37.506 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:37.507 DEBUG:boto:Data: 2022-09-02T05:37:37.507 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.508 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.509 DEBUG:boto:Params: {} 2022-09-02T05:37:37.509 DEBUG:boto:Token: None 2022-09-02T05:37:37.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/f 2022-09-02T05:37:37.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1zky1AFvpdIRML3RVo0e1cSQF/o= 2022-09-02T05:37:37.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1zky1AFvpdIRML3RVo0e1cSQF/o=', 'Content-Length': '0'} 2022-09-02T05:37:37.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f89e92bd030a55f-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.514 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.515 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.515 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.516 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.516 DEBUG:boto:Method: GET 2022-09-02T05:37:37.517 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:37.517 DEBUG:boto:Data: 2022-09-02T05:37:37.518 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.519 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.520 DEBUG:boto:Params: {} 2022-09-02T05:37:37.520 DEBUG:boto:Token: None 2022-09-02T05:37:37.521 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000962ab3c342576696-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.525 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.526 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.527 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.527 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.528 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:37.529 DEBUG:boto:Method: GET 2022-09-02T05:37:37.529 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:37.530 DEBUG:boto:Data: 2022-09-02T05:37:37.530 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.531 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.532 DEBUG:boto:Params: {} 2022-09-02T05:37:37.532 DEBUG:boto:Token: None 2022-09-02T05:37:37.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003deadcbc14d906cc-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.537 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:37.539 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.539 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.540 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.540 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.541 DEBUG:boto:Method: GET 2022-09-02T05:37:37.541 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.542 DEBUG:boto:Data: 2022-09-02T05:37:37.542 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.543 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.544 DEBUG:boto:Params: {} 2022-09-02T05:37:37.545 DEBUG:boto:Token: None 2022-09-02T05:37:37.545 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig= 2022-09-02T05:37:37.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig=', 'Content-Length': '0'} 2022-09-02T05:37:37.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c33c853e693c04-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.550 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.550 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.551 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.551 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.552 DEBUG:boto:Method: GET 2022-09-02T05:37:37.552 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.553 DEBUG:boto:Data: 2022-09-02T05:37:37.554 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.555 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.555 DEBUG:boto:Params: {} 2022-09-02T05:37:37.556 DEBUG:boto:Token: None 2022-09-02T05:37:37.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig= 2022-09-02T05:37:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig=', 'Content-Length': '0'} 2022-09-02T05:37:37.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099051f225a6c0ff9-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.561 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.561 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.562 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.562 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.563 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.563 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.564 DEBUG:boto:Data: 2022-09-02T05:37:37.564 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.565 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.566 DEBUG:boto:Params: {} 2022-09-02T05:37:37.567 DEBUG:boto:Token: None 2022-09-02T05:37:37.567 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AuB6wchVNceOY9etifZjUnMsBjw= 2022-09-02T05:37:37.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AuB6wchVNceOY9etifZjUnMsBjw=', 'Content-Length': '0'} 2022-09-02T05:37:37.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e74fef9ac4c0328-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.571 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.572 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.572 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.573 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.573 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.574 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.574 DEBUG:boto:Data: 2022-09-02T05:37:37.575 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.576 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.576 DEBUG:boto:Params: {} 2022-09-02T05:37:37.577 DEBUG:boto:Token: None 2022-09-02T05:37:37.578 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AuB6wchVNceOY9etifZjUnMsBjw= 2022-09-02T05:37:37.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AuB6wchVNceOY9etifZjUnMsBjw=', 'Content-Length': '0'} 2022-09-02T05:37:37.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad4126086b77e38-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:37.582 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.583 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.584 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.584 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.585 DEBUG:boto:Method: GET 2022-09-02T05:37:37.585 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.586 DEBUG:boto:Data: 2022-09-02T05:37:37.586 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.587 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.588 DEBUG:boto:Params: {} 2022-09-02T05:37:37.589 DEBUG:boto:Token: None 2022-09-02T05:37:37.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig= 2022-09-02T05:37:37.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig=', 'Content-Length': '0'} 2022-09-02T05:37:37.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fed2a1717199a49-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.634 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.635 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.635 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.636 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:37.637 DEBUG:boto:Method: GET 2022-09-02T05:37:37.637 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.638 DEBUG:boto:Data: 2022-09-02T05:37:37.638 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.639 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.640 DEBUG:boto:Params: {} 2022-09-02T05:37:37.640 DEBUG:boto:Token: None 2022-09-02T05:37:37.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:37.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig= 2022-09-02T05:37:37.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a3jODoS4kUkxRDCed14A/EF1Tig=', 'Content-Length': '0'} 2022-09-02T05:37:37.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087fd70447fb7d855-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.646 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.646 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.647 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.647 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.648 DEBUG:boto:Method: GET 2022-09-02T05:37:37.648 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:37.649 DEBUG:boto:Data: 2022-09-02T05:37:37.649 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.651 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.651 DEBUG:boto:Params: {} 2022-09-02T05:37:37.652 DEBUG:boto:Token: None 2022-09-02T05:37:37.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2868d18351b795e-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.657 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.658 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.658 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.659 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.659 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.660 DEBUG:boto:Method: GET 2022-09-02T05:37:37.661 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:37.661 DEBUG:boto:Data: 2022-09-02T05:37:37.662 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.663 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.663 DEBUG:boto:Params: {} 2022-09-02T05:37:37.664 DEBUG:boto:Token: None 2022-09-02T05:37:37.664 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f517af7ef7aa853f-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.669 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.670 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.670 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.671 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.671 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:37.672 DEBUG:boto:Data: 2022-09-02T05:37:37.672 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.673 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.674 DEBUG:boto:Params: {} 2022-09-02T05:37:37.674 DEBUG:boto:Token: None 2022-09-02T05:37:37.675 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g 2022-09-02T05:37:37.676 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BeLEtrt46GKznxsXGYG7RlOLct4= 2022-09-02T05:37:37.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BeLEtrt46GKznxsXGYG7RlOLct4=', 'Content-Length': '0'} 2022-09-02T05:37:37.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d299a71200ebe9b6-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.679 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.680 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.680 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.681 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:37.681 DEBUG:boto:Data: 2022-09-02T05:37:37.682 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.684 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.685 DEBUG:boto:Params: {} 2022-09-02T05:37:37.686 DEBUG:boto:Token: None 2022-09-02T05:37:37.687 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g 2022-09-02T05:37:37.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BeLEtrt46GKznxsXGYG7RlOLct4= 2022-09-02T05:37:37.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BeLEtrt46GKznxsXGYG7RlOLct4=', 'Content-Length': '0'} 2022-09-02T05:37:37.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c12ed4a6ce5e0245-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:37.694 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.695 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.696 DEBUG:boto:Method: GET 2022-09-02T05:37:37.697 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:37.698 DEBUG:boto:Data: 2022-09-02T05:37:37.699 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.701 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.702 DEBUG:boto:Params: {} 2022-09-02T05:37:37.703 DEBUG:boto:Token: None 2022-09-02T05:37:37.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g 2022-09-02T05:37:37.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZEzJr5I/GXcIgHoNTbUDqISUy5Y= 2022-09-02T05:37:37.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZEzJr5I/GXcIgHoNTbUDqISUy5Y=', 'Content-Length': '0'} 2022-09-02T05:37:37.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c3805e81363c65-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.708 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:37.709 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:37.709 DEBUG:boto:Method: GET 2022-09-02T05:37:37.710 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:37.711 DEBUG:boto:Data: 2022-09-02T05:37:37.711 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.712 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.713 DEBUG:boto:Params: {} 2022-09-02T05:37:37.714 DEBUG:boto:Token: None 2022-09-02T05:37:37.714 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/g 2022-09-02T05:37:37.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZEzJr5I/GXcIgHoNTbUDqISUy5Y= 2022-09-02T05:37:37.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZEzJr5I/GXcIgHoNTbUDqISUy5Y=', 'Content-Length': '0'} 2022-09-02T05:37:37.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007653d4e81a3004ad-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.719 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.720 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.720 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.721 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.722 DEBUG:boto:Method: GET 2022-09-02T05:37:37.723 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:37.723 DEBUG:boto:Data: 2022-09-02T05:37:37.724 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.725 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.725 DEBUG:boto:Params: {} 2022-09-02T05:37:37.726 DEBUG:boto:Token: None 2022-09-02T05:37:37.727 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ffc80deb4fb6ae-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.734 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.735 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.736 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.736 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.737 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:37.738 DEBUG:boto:Method: GET 2022-09-02T05:37:37.738 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:37.739 DEBUG:boto:Data: 2022-09-02T05:37:37.740 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.741 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.742 DEBUG:boto:Params: {} 2022-09-02T05:37:37.743 DEBUG:boto:Token: None 2022-09-02T05:37:37.745 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6864fd09dae1fe4-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.750 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:37.752 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.752 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.753 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.753 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.754 DEBUG:boto:Method: GET 2022-09-02T05:37:37.754 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.755 DEBUG:boto:Data: 2022-09-02T05:37:37.756 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.757 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.757 DEBUG:boto:Params: {} 2022-09-02T05:37:37.758 DEBUG:boto:Token: None 2022-09-02T05:37:37.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY= 2022-09-02T05:37:37.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY=', 'Content-Length': '0'} 2022-09-02T05:37:37.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004584e14897b2865c-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.763 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.764 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.764 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.765 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.765 DEBUG:boto:Method: GET 2022-09-02T05:37:37.766 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.767 DEBUG:boto:Data: 2022-09-02T05:37:37.767 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.768 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.769 DEBUG:boto:Params: {} 2022-09-02T05:37:37.770 DEBUG:boto:Token: None 2022-09-02T05:37:37.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY= 2022-09-02T05:37:37.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY=', 'Content-Length': '0'} 2022-09-02T05:37:37.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1da7a7a15f382b-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.775 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.775 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.776 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.776 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.777 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.777 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.778 DEBUG:boto:Data: 2022-09-02T05:37:37.778 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.780 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.780 DEBUG:boto:Params: {} 2022-09-02T05:37:37.781 DEBUG:boto:Token: None 2022-09-02T05:37:37.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kAEGJ6a0fk2Aq+9OHpdkiYN0CBc= 2022-09-02T05:37:37.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kAEGJ6a0fk2Aq+9OHpdkiYN0CBc=', 'Content-Length': '0'} 2022-09-02T05:37:37.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71f0a9c116a523d-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.786 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.786 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.787 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.788 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.788 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.789 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.789 DEBUG:boto:Data: 2022-09-02T05:37:37.790 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.791 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.791 DEBUG:boto:Params: {} 2022-09-02T05:37:37.792 DEBUG:boto:Token: None 2022-09-02T05:37:37.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kAEGJ6a0fk2Aq+9OHpdkiYN0CBc= 2022-09-02T05:37:37.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kAEGJ6a0fk2Aq+9OHpdkiYN0CBc=', 'Content-Length': '0'} 2022-09-02T05:37:37.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9938da4a62632d5-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:37.798 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.798 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.799 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.799 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.800 DEBUG:boto:Method: GET 2022-09-02T05:37:37.800 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.801 DEBUG:boto:Data: 2022-09-02T05:37:37.801 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.803 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.803 DEBUG:boto:Params: {} 2022-09-02T05:37:37.804 DEBUG:boto:Token: None 2022-09-02T05:37:37.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY= 2022-09-02T05:37:37.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY=', 'Content-Length': '0'} 2022-09-02T05:37:37.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ca3aa83e7aee3f-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.854 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.855 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.855 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.856 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:37.856 DEBUG:boto:Method: GET 2022-09-02T05:37:37.857 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.858 DEBUG:boto:Data: 2022-09-02T05:37:37.858 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.859 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.860 DEBUG:boto:Params: {} 2022-09-02T05:37:37.860 DEBUG:boto:Token: None 2022-09-02T05:37:37.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:37.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY= 2022-09-02T05:37:37.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:suB6Qnb8hvF/7RNGdKSyVw2a4JY=', 'Content-Length': '0'} 2022-09-02T05:37:37.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009479d524c34bef89-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.866 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.867 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.867 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.868 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.868 DEBUG:boto:Method: GET 2022-09-02T05:37:37.869 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:37.869 DEBUG:boto:Data: 2022-09-02T05:37:37.870 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.871 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.871 DEBUG:boto:Params: {} 2022-09-02T05:37:37.872 DEBUG:boto:Token: None 2022-09-02T05:37:37.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac199e3cdb63b25a-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.877 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.878 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.878 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.879 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.880 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.880 DEBUG:boto:Method: GET 2022-09-02T05:37:37.881 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:37.881 DEBUG:boto:Data: 2022-09-02T05:37:37.882 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.883 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.883 DEBUG:boto:Params: {} 2022-09-02T05:37:37.884 DEBUG:boto:Token: None 2022-09-02T05:37:37.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000282c667d554843a2-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.889 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.890 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.890 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.891 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.891 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:37.892 DEBUG:boto:Data: 2022-09-02T05:37:37.892 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.893 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.894 DEBUG:boto:Params: {} 2022-09-02T05:37:37.895 DEBUG:boto:Token: None 2022-09-02T05:37:37.895 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h 2022-09-02T05:37:37.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:quyIhXDkjLYu4ufBUYnKXvLcg1Y= 2022-09-02T05:37:37.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:quyIhXDkjLYu4ufBUYnKXvLcg1Y=', 'Content-Length': '0'} 2022-09-02T05:37:37.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23a85ad5754de99-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.899 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.900 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.900 DEBUG:boto:Method: HEAD 2022-09-02T05:37:37.901 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:37.901 DEBUG:boto:Data: 2022-09-02T05:37:37.902 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.903 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.904 DEBUG:boto:Params: {} 2022-09-02T05:37:37.904 DEBUG:boto:Token: None 2022-09-02T05:37:37.905 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h 2022-09-02T05:37:37.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:quyIhXDkjLYu4ufBUYnKXvLcg1Y= 2022-09-02T05:37:37.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:quyIhXDkjLYu4ufBUYnKXvLcg1Y=', 'Content-Length': '0'} 2022-09-02T05:37:37.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d42970f088cb6ff-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:37.910 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.910 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.911 DEBUG:boto:Method: GET 2022-09-02T05:37:37.911 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:37.912 DEBUG:boto:Data: 2022-09-02T05:37:37.913 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.914 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.914 DEBUG:boto:Params: {} 2022-09-02T05:37:37.915 DEBUG:boto:Token: None 2022-09-02T05:37:37.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h 2022-09-02T05:37:37.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cAupwAAofDVzsbv8YKhRNwh8Eu4= 2022-09-02T05:37:37.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cAupwAAofDVzsbv8YKhRNwh8Eu4=', 'Content-Length': '0'} 2022-09-02T05:37:37.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007252fad1025730ec-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.920 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:37.920 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:37.921 DEBUG:boto:Method: GET 2022-09-02T05:37:37.921 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:37.922 DEBUG:boto:Data: 2022-09-02T05:37:37.923 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.924 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.924 DEBUG:boto:Params: {} 2022-09-02T05:37:37.925 DEBUG:boto:Token: None 2022-09-02T05:37:37.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/h 2022-09-02T05:37:37.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cAupwAAofDVzsbv8YKhRNwh8Eu4= 2022-09-02T05:37:37.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cAupwAAofDVzsbv8YKhRNwh8Eu4=', 'Content-Length': '0'} 2022-09-02T05:37:37.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e196ba6dd6fba5ae-00631196a1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.930 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.930 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.931 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.931 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.932 DEBUG:boto:Method: GET 2022-09-02T05:37:37.933 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:37.933 DEBUG:boto:Data: 2022-09-02T05:37:37.934 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:37.935 DEBUG:boto:Port: 8001 2022-09-02T05:37:37.935 DEBUG:boto:Params: {} 2022-09-02T05:37:37.936 DEBUG:boto:Token: None 2022-09-02T05:37:37.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0794b6c94ab1511-00631196a1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.982 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.984 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:37.984 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:37.985 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.986 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:37.986 DEBUG:boto:Method: GET 2022-09-02T05:37:37.987 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:37.988 DEBUG:boto:Data: 2022-09-02T05:37:37.988 DEBUG:boto:Headers: {} 2022-09-02T05:37:37.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:37.989 DEBUG:boto:Port: 8000 2022-09-02T05:37:37.990 DEBUG:boto:Params: {} 2022-09-02T05:37:37.991 DEBUG:boto:Token: None 2022-09-02T05:37:37.991 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:37 GMT /xljbkf-72/ 2022-09-02T05:37:37.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4= 2022-09-02T05:37:37.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:22tfr4f5p7iw15jLYIYHvsBuxk4=', 'Content-Length': '0'} 2022-09-02T05:37:37.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d32ebc49f66176-00631196a1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:37.996 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:37.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-72 zones={test-zone3, test-zone1} 2022-09-02T05:37:37.999 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-72 zones={test-zone3, test-zone2} 2022-09-02T05:37:38.000 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.001 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.002 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.003 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:37:38.004 DEBUG:boto:Data: 2022-09-02T05:37:38.005 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.006 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.007 DEBUG:boto:Params: {} 2022-09-02T05:37:38.008 DEBUG:boto:Token: None 2022-09-02T05:37:38.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wGGdFmKhrf6wfZVqTJ0zWs0DDTA= 2022-09-02T05:37:38.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wGGdFmKhrf6wfZVqTJ0zWs0DDTA=', 'Content-Length': '0'} 2022-09-02T05:37:38.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000308e072536b6b4cb-00631196a2-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.013 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.014 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.015 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.015 DEBUG:boto:Path: /xljbkf-72/ 2022-09-02T05:37:38.016 DEBUG:boto:Data: 2022-09-02T05:37:38.016 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.017 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.017 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.018 DEBUG:boto:Params: {} 2022-09-02T05:37:38.018 DEBUG:boto:Token: None 2022-09-02T05:37:38.019 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wGGdFmKhrf6wfZVqTJ0zWs0DDTA= 2022-09-02T05:37:38.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wGGdFmKhrf6wfZVqTJ0zWs0DDTA=', 'Content-Length': '0'} 2022-09-02T05:37:38.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019cb64ef897217d6-00631196a2-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.024 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:37:38.024 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.025 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.025 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.026 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.026 DEBUG:boto:Method: GET 2022-09-02T05:37:38.027 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.027 DEBUG:boto:Data: 2022-09-02T05:37:38.028 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.029 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.029 DEBUG:boto:Params: {} 2022-09-02T05:37:38.030 DEBUG:boto:Token: None 2022-09-02T05:37:38.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/?versions 2022-09-02T05:37:38.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI= 2022-09-02T05:37:38.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI=', 'Content-Length': '0'} 2022-09-02T05:37:38.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c775b3dee31cad4-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.035 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.036 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-02T05:37:38.037 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-02T05:37:38.037 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-02T05:37:38.038 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-02T05:37:38.038 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-02T05:37:38.039 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-02T05:37:38.039 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-02T05:37:38.040 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-02T05:37:38.040 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:37:38.041 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.041 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.042 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.042 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.043 DEBUG:boto:Method: GET 2022-09-02T05:37:38.043 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.044 DEBUG:boto:Data: 2022-09-02T05:37:38.044 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.045 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.046 DEBUG:boto:Params: {} 2022-09-02T05:37:38.046 DEBUG:boto:Token: None 2022-09-02T05:37:38.047 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/?versions 2022-09-02T05:37:38.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI= 2022-09-02T05:37:38.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI=', 'Content-Length': '0'} 2022-09-02T05:37:38.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a55316444e6bf26c-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.052 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.053 DEBUG:tasks.rgw_multi.zone_rados:o=a 2022-09-02T05:37:38.053 DEBUG:tasks.rgw_multi.zone_rados:o=b 2022-09-02T05:37:38.054 DEBUG:tasks.rgw_multi.zone_rados:o=c 2022-09-02T05:37:38.054 DEBUG:tasks.rgw_multi.zone_rados:o=d 2022-09-02T05:37:38.055 DEBUG:tasks.rgw_multi.zone_rados:o=e 2022-09-02T05:37:38.055 DEBUG:tasks.rgw_multi.zone_rados:o=f 2022-09-02T05:37:38.056 DEBUG:tasks.rgw_multi.zone_rados:o=g 2022-09-02T05:37:38.056 DEBUG:tasks.rgw_multi.zone_rados:o=h 2022-09-02T05:37:38.057 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.057 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.058 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.058 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.059 DEBUG:boto:Method: GET 2022-09-02T05:37:38.059 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.060 DEBUG:boto:Data: 2022-09-02T05:37:38.060 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.061 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.062 DEBUG:boto:Params: {} 2022-09-02T05:37:38.062 DEBUG:boto:Token: None 2022-09-02T05:37:38.063 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/?versions 2022-09-02T05:37:38.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI= 2022-09-02T05:37:38.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI=', 'Content-Length': '0'} 2022-09-02T05:37:38.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001340f62ccec08568-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.068 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.069 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.069 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.070 DEBUG:boto:path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.070 DEBUG:boto:auth_path=/xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.071 DEBUG:boto:Method: GET 2022-09-02T05:37:38.071 DEBUG:boto:Path: /xljbkf-72/?versions&max-keys=999 2022-09-02T05:37:38.072 DEBUG:boto:Data: 2022-09-02T05:37:38.072 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.074 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.074 DEBUG:boto:Params: {} 2022-09-02T05:37:38.075 DEBUG:boto:Token: None 2022-09-02T05:37:38.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/?versions 2022-09-02T05:37:38.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI= 2022-09-02T05:37:38.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uy5ycVk8iczVwHZK4SD/E0LH+QI=', 'Content-Length': '0'} 2022-09-02T05:37:38.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb0ccfe8203dbff-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.080 DEBUG:boto:b'xljbkf-72999falseanulltrue2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:38.082 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.082 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.083 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.083 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.084 DEBUG:boto:Method: GET 2022-09-02T05:37:38.085 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.085 DEBUG:boto:Data: 2022-09-02T05:37:38.086 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.087 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.087 DEBUG:boto:Params: {} 2022-09-02T05:37:38.088 DEBUG:boto:Token: None 2022-09-02T05:37:38.088 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4= 2022-09-02T05:37:38.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4=', 'Content-Length': '0'} 2022-09-02T05:37:38.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e07961730166b3-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.134 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.135 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.135 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.136 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.136 DEBUG:boto:Method: GET 2022-09-02T05:37:38.137 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.138 DEBUG:boto:Data: 2022-09-02T05:37:38.138 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.139 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.140 DEBUG:boto:Params: {} 2022-09-02T05:37:38.140 DEBUG:boto:Token: None 2022-09-02T05:37:38.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4= 2022-09-02T05:37:38.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4=', 'Content-Length': '0'} 2022-09-02T05:37:38.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2de6e5806fa440-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.145 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.146 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.146 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.147 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.147 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.148 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.149 DEBUG:boto:Data: 2022-09-02T05:37:38.149 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.150 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.151 DEBUG:boto:Params: {} 2022-09-02T05:37:38.151 DEBUG:boto:Token: None 2022-09-02T05:37:38.152 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lu3Uu18L7iyOCp9Jo+vzUu+hE0w= 2022-09-02T05:37:38.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lu3Uu18L7iyOCp9Jo+vzUu+hE0w=', 'Content-Length': '0'} 2022-09-02T05:37:38.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000356fe29f334fdab7-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.156 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.156 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.157 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.158 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.158 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.159 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.159 DEBUG:boto:Data: 2022-09-02T05:37:38.160 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.161 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.161 DEBUG:boto:Params: {} 2022-09-02T05:37:38.162 DEBUG:boto:Token: None 2022-09-02T05:37:38.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lu3Uu18L7iyOCp9Jo+vzUu+hE0w= 2022-09-02T05:37:38.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lu3Uu18L7iyOCp9Jo+vzUu+hE0w=', 'Content-Length': '0'} 2022-09-02T05:37:38.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5577378ab61132d-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:38.167 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.168 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.168 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.169 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.169 DEBUG:boto:Method: GET 2022-09-02T05:37:38.170 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.170 DEBUG:boto:Data: 2022-09-02T05:37:38.171 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.172 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.172 DEBUG:boto:Params: {} 2022-09-02T05:37:38.173 DEBUG:boto:Token: None 2022-09-02T05:37:38.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4= 2022-09-02T05:37:38.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4=', 'Content-Length': '0'} 2022-09-02T05:37:38.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b8ddfe6909710a2-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.178 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.178 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.179 DEBUG:boto:path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.179 DEBUG:boto:auth_path=/xljbkf-72/a?versionId=null 2022-09-02T05:37:38.180 DEBUG:boto:Method: GET 2022-09-02T05:37:38.180 DEBUG:boto:Path: /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.181 DEBUG:boto:Data: 2022-09-02T05:37:38.181 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.183 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.183 DEBUG:boto:Params: {} 2022-09-02T05:37:38.184 DEBUG:boto:Token: None 2022-09-02T05:37:38.184 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a?versionId=null 2022-09-02T05:37:38.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4= 2022-09-02T05:37:38.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Za1b1nUthIj/wedFVkLQ0Lx5zy4=', 'Content-Length': '0'} 2022-09-02T05:37:38.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b555dd4d5b2e552-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.189 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.189 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.190 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.190 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.191 DEBUG:boto:Method: GET 2022-09-02T05:37:38.191 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:38.192 DEBUG:boto:Data: 2022-09-02T05:37:38.192 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.193 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.194 DEBUG:boto:Params: {} 2022-09-02T05:37:38.194 DEBUG:boto:Token: None 2022-09-02T05:37:38.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000830c2fed1e0e7378-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.199 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.200 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.201 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.201 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.202 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.202 DEBUG:boto:Method: GET 2022-09-02T05:37:38.203 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:38.204 DEBUG:boto:Data: 2022-09-02T05:37:38.204 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.205 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.206 DEBUG:boto:Params: {} 2022-09-02T05:37:38.207 DEBUG:boto:Token: None 2022-09-02T05:37:38.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024248e44275e9406-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.212 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.213 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.213 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.214 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.214 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:38.215 DEBUG:boto:Data: 2022-09-02T05:37:38.215 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.216 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.217 DEBUG:boto:Params: {} 2022-09-02T05:37:38.217 DEBUG:boto:Token: None 2022-09-02T05:37:38.218 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a 2022-09-02T05:37:38.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:flzNmgAyJWQkoeXPhpferSwMI1M= 2022-09-02T05:37:38.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:flzNmgAyJWQkoeXPhpferSwMI1M=', 'Content-Length': '0'} 2022-09-02T05:37:38.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07f0e3d89075f19-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.222 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.223 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.223 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.224 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:38.224 DEBUG:boto:Data: 2022-09-02T05:37:38.225 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.226 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.226 DEBUG:boto:Params: {} 2022-09-02T05:37:38.227 DEBUG:boto:Token: None 2022-09-02T05:37:38.227 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a 2022-09-02T05:37:38.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:flzNmgAyJWQkoeXPhpferSwMI1M= 2022-09-02T05:37:38.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:flzNmgAyJWQkoeXPhpferSwMI1M=', 'Content-Length': '0'} 2022-09-02T05:37:38.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a8c77f5cdfe4e76-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2022-09-02T05:37:38.232 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.233 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.233 DEBUG:boto:Method: GET 2022-09-02T05:37:38.234 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:38.234 DEBUG:boto:Data: 2022-09-02T05:37:38.235 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.236 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.236 DEBUG:boto:Params: {} 2022-09-02T05:37:38.237 DEBUG:boto:Token: None 2022-09-02T05:37:38.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a 2022-09-02T05:37:38.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XMj01awCiO83/GUO1CTRP43lUxI= 2022-09-02T05:37:38.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XMj01awCiO83/GUO1CTRP43lUxI=', 'Content-Length': '0'} 2022-09-02T05:37:38.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255b339cf7c6ebff-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.242 DEBUG:boto:path=/xljbkf-72/a 2022-09-02T05:37:38.242 DEBUG:boto:auth_path=/xljbkf-72/a 2022-09-02T05:37:38.243 DEBUG:boto:Method: GET 2022-09-02T05:37:38.243 DEBUG:boto:Path: /xljbkf-72/a 2022-09-02T05:37:38.244 DEBUG:boto:Data: 2022-09-02T05:37:38.244 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.245 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.246 DEBUG:boto:Params: {} 2022-09-02T05:37:38.246 DEBUG:boto:Token: None 2022-09-02T05:37:38.247 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/a 2022-09-02T05:37:38.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XMj01awCiO83/GUO1CTRP43lUxI= 2022-09-02T05:37:38.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XMj01awCiO83/GUO1CTRP43lUxI=', 'Content-Length': '0'} 2022-09-02T05:37:38.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd63fe238a22ddc-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.294 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.295 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.296 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.296 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.297 DEBUG:boto:Method: GET 2022-09-02T05:37:38.297 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:38.298 DEBUG:boto:Data: 2022-09-02T05:37:38.298 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.301 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.301 DEBUG:boto:Params: {} 2022-09-02T05:37:38.302 DEBUG:boto:Token: None 2022-09-02T05:37:38.303 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.303 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b504cf93cd5e98-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.307 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.308 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.309 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.309 DEBUG:boto:path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.310 DEBUG:boto:auth_path=/xljbkf-72/?prefix=a 2022-09-02T05:37:38.311 DEBUG:boto:Method: GET 2022-09-02T05:37:38.311 DEBUG:boto:Path: /xljbkf-72/?prefix=a 2022-09-02T05:37:38.312 DEBUG:boto:Data: 2022-09-02T05:37:38.312 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.313 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.314 DEBUG:boto:Params: {} 2022-09-02T05:37:38.315 DEBUG:boto:Token: None 2022-09-02T05:37:38.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d47fe7ace0aed6d-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.320 DEBUG:boto:b'xljbkf-72a1000falsea2022-09-02T05:33:19.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:38.321 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.322 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.322 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.323 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.324 DEBUG:boto:Method: GET 2022-09-02T05:37:38.324 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.325 DEBUG:boto:Data: 2022-09-02T05:37:38.325 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.326 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.327 DEBUG:boto:Params: {} 2022-09-02T05:37:38.328 DEBUG:boto:Token: None 2022-09-02T05:37:38.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs= 2022-09-02T05:37:38.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs=', 'Content-Length': '0'} 2022-09-02T05:37:38.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024d3833ce14c8e0e-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.374 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.375 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.375 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.376 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.377 DEBUG:boto:Method: GET 2022-09-02T05:37:38.377 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.378 DEBUG:boto:Data: 2022-09-02T05:37:38.378 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.379 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.380 DEBUG:boto:Params: {} 2022-09-02T05:37:38.381 DEBUG:boto:Token: None 2022-09-02T05:37:38.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs= 2022-09-02T05:37:38.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs=', 'Content-Length': '0'} 2022-09-02T05:37:38.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f27e93e6de4c92c-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.387 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.388 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.389 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.389 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.390 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.391 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.392 DEBUG:boto:Data: 2022-09-02T05:37:38.393 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.395 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.396 DEBUG:boto:Params: {} 2022-09-02T05:37:38.397 DEBUG:boto:Token: None 2022-09-02T05:37:38.397 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dPbzCoXD5ujShqHKzaBybWN3DZ8= 2022-09-02T05:37:38.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dPbzCoXD5ujShqHKzaBybWN3DZ8=', 'Content-Length': '0'} 2022-09-02T05:37:38.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072026b273189ab75-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.402 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.402 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.403 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.403 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.404 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.404 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.405 DEBUG:boto:Data: 2022-09-02T05:37:38.405 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.406 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.407 DEBUG:boto:Params: {} 2022-09-02T05:37:38.408 DEBUG:boto:Token: None 2022-09-02T05:37:38.409 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dPbzCoXD5ujShqHKzaBybWN3DZ8= 2022-09-02T05:37:38.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dPbzCoXD5ujShqHKzaBybWN3DZ8=', 'Content-Length': '0'} 2022-09-02T05:37:38.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aec90e9c689b0f36-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:38.414 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.414 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.415 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.415 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.416 DEBUG:boto:Method: GET 2022-09-02T05:37:38.416 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.417 DEBUG:boto:Data: 2022-09-02T05:37:38.417 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.419 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.420 DEBUG:boto:Params: {} 2022-09-02T05:37:38.420 DEBUG:boto:Token: None 2022-09-02T05:37:38.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs= 2022-09-02T05:37:38.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs=', 'Content-Length': '0'} 2022-09-02T05:37:38.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801dd796cdf4a3f5-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.425 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.426 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.426 DEBUG:boto:path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.427 DEBUG:boto:auth_path=/xljbkf-72/b?versionId=null 2022-09-02T05:37:38.428 DEBUG:boto:Method: GET 2022-09-02T05:37:38.428 DEBUG:boto:Path: /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.429 DEBUG:boto:Data: 2022-09-02T05:37:38.429 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.431 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.431 DEBUG:boto:Params: {} 2022-09-02T05:37:38.432 DEBUG:boto:Token: None 2022-09-02T05:37:38.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b?versionId=null 2022-09-02T05:37:38.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs= 2022-09-02T05:37:38.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jdBXZfmWsdTAn9zbjcVFjpkIWDs=', 'Content-Length': '0'} 2022-09-02T05:37:38.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a749f76f350ac3e-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.437 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.438 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.439 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.439 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.440 DEBUG:boto:Method: GET 2022-09-02T05:37:38.440 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:38.441 DEBUG:boto:Data: 2022-09-02T05:37:38.442 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.443 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.443 DEBUG:boto:Params: {} 2022-09-02T05:37:38.444 DEBUG:boto:Token: None 2022-09-02T05:37:38.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e383bfa9af2cdf00-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.450 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.452 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.453 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.454 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.455 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.456 DEBUG:boto:Method: GET 2022-09-02T05:37:38.457 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:38.458 DEBUG:boto:Data: 2022-09-02T05:37:38.459 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.461 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.462 DEBUG:boto:Params: {} 2022-09-02T05:37:38.463 DEBUG:boto:Token: None 2022-09-02T05:37:38.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a1046365cdfd59-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.468 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.469 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.470 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.470 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.471 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:38.471 DEBUG:boto:Data: 2022-09-02T05:37:38.472 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.473 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.474 DEBUG:boto:Params: {} 2022-09-02T05:37:38.474 DEBUG:boto:Token: None 2022-09-02T05:37:38.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b 2022-09-02T05:37:38.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nviQzQ1WOlSdIMkHg2BxE+pg898= 2022-09-02T05:37:38.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nviQzQ1WOlSdIMkHg2BxE+pg898=', 'Content-Length': '0'} 2022-09-02T05:37:38.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f102ca14983ab0c6-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.479 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.480 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.480 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.481 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:38.481 DEBUG:boto:Data: 2022-09-02T05:37:38.482 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.484 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.484 DEBUG:boto:Params: {} 2022-09-02T05:37:38.485 DEBUG:boto:Token: None 2022-09-02T05:37:38.486 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b 2022-09-02T05:37:38.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nviQzQ1WOlSdIMkHg2BxE+pg898= 2022-09-02T05:37:38.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nviQzQ1WOlSdIMkHg2BxE+pg898=', 'Content-Length': '0'} 2022-09-02T05:37:38.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e62749aba211cc49-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2022-09-02T05:37:38.498 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.499 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.499 DEBUG:boto:Method: GET 2022-09-02T05:37:38.500 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:38.500 DEBUG:boto:Data: 2022-09-02T05:37:38.501 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.502 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.503 DEBUG:boto:Params: {} 2022-09-02T05:37:38.503 DEBUG:boto:Token: None 2022-09-02T05:37:38.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b 2022-09-02T05:37:38.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YyQa4KbPbBTmeRDXQfgk1VZF+Us= 2022-09-02T05:37:38.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YyQa4KbPbBTmeRDXQfgk1VZF+Us=', 'Content-Length': '0'} 2022-09-02T05:37:38.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2ea1617610e179-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.508 DEBUG:boto:path=/xljbkf-72/b 2022-09-02T05:37:38.509 DEBUG:boto:auth_path=/xljbkf-72/b 2022-09-02T05:37:38.509 DEBUG:boto:Method: GET 2022-09-02T05:37:38.510 DEBUG:boto:Path: /xljbkf-72/b 2022-09-02T05:37:38.510 DEBUG:boto:Data: 2022-09-02T05:37:38.511 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.512 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.513 DEBUG:boto:Params: {} 2022-09-02T05:37:38.513 DEBUG:boto:Token: None 2022-09-02T05:37:38.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/b 2022-09-02T05:37:38.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YyQa4KbPbBTmeRDXQfgk1VZF+Us= 2022-09-02T05:37:38.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YyQa4KbPbBTmeRDXQfgk1VZF+Us=', 'Content-Length': '0'} 2022-09-02T05:37:38.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d707d8292be5258d-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.562 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.563 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.564 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.564 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.565 DEBUG:boto:Method: GET 2022-09-02T05:37:38.566 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:38.567 DEBUG:boto:Data: 2022-09-02T05:37:38.568 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.570 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.572 DEBUG:boto:Params: {} 2022-09-02T05:37:38.573 DEBUG:boto:Token: None 2022-09-02T05:37:38.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c9dc1a23fb1e88a-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.579 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.580 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.581 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.581 DEBUG:boto:path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.582 DEBUG:boto:auth_path=/xljbkf-72/?prefix=b 2022-09-02T05:37:38.582 DEBUG:boto:Method: GET 2022-09-02T05:37:38.583 DEBUG:boto:Path: /xljbkf-72/?prefix=b 2022-09-02T05:37:38.583 DEBUG:boto:Data: 2022-09-02T05:37:38.584 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.585 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.586 DEBUG:boto:Params: {} 2022-09-02T05:37:38.586 DEBUG:boto:Token: None 2022-09-02T05:37:38.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f423c722e15db60-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.592 DEBUG:boto:b'xljbkf-72b1000falseb2022-09-02T05:33:19.293Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:38.593 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.594 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.594 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.595 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.595 DEBUG:boto:Method: GET 2022-09-02T05:37:38.596 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.596 DEBUG:boto:Data: 2022-09-02T05:37:38.597 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.598 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.599 DEBUG:boto:Params: {} 2022-09-02T05:37:38.599 DEBUG:boto:Token: None 2022-09-02T05:37:38.600 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c= 2022-09-02T05:37:38.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c=', 'Content-Length': '0'} 2022-09-02T05:37:38.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b50cdab3ed4884-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.650 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.651 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.652 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.653 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.654 DEBUG:boto:Method: GET 2022-09-02T05:37:38.655 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.656 DEBUG:boto:Data: 2022-09-02T05:37:38.657 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.658 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.659 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.660 DEBUG:boto:Params: {} 2022-09-02T05:37:38.661 DEBUG:boto:Token: None 2022-09-02T05:37:38.662 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c= 2022-09-02T05:37:38.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c=', 'Content-Length': '0'} 2022-09-02T05:37:38.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c05b59205929dc8-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.667 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.668 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.668 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.669 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.669 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.670 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.671 DEBUG:boto:Data: 2022-09-02T05:37:38.671 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.672 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.673 DEBUG:boto:Params: {} 2022-09-02T05:37:38.673 DEBUG:boto:Token: None 2022-09-02T05:37:38.674 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ba5oGOQ/Vaszre7m5IBHzXb5lQQ= 2022-09-02T05:37:38.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ba5oGOQ/Vaszre7m5IBHzXb5lQQ=', 'Content-Length': '0'} 2022-09-02T05:37:38.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584601b9571fcd43-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.678 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.678 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.679 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.679 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.680 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.680 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.681 DEBUG:boto:Data: 2022-09-02T05:37:38.681 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.682 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.683 DEBUG:boto:Params: {} 2022-09-02T05:37:38.684 DEBUG:boto:Token: None 2022-09-02T05:37:38.684 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ba5oGOQ/Vaszre7m5IBHzXb5lQQ= 2022-09-02T05:37:38.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ba5oGOQ/Vaszre7m5IBHzXb5lQQ=', 'Content-Length': '0'} 2022-09-02T05:37:38.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752e0fccf16838b2-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:38.689 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.690 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.690 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.691 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.691 DEBUG:boto:Method: GET 2022-09-02T05:37:38.692 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.693 DEBUG:boto:Data: 2022-09-02T05:37:38.693 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.694 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.695 DEBUG:boto:Params: {} 2022-09-02T05:37:38.696 DEBUG:boto:Token: None 2022-09-02T05:37:38.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c= 2022-09-02T05:37:38.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c=', 'Content-Length': '0'} 2022-09-02T05:37:38.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007540becb865b09d6-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.701 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.702 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.703 DEBUG:boto:path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.703 DEBUG:boto:auth_path=/xljbkf-72/c?versionId=null 2022-09-02T05:37:38.704 DEBUG:boto:Method: GET 2022-09-02T05:37:38.704 DEBUG:boto:Path: /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.705 DEBUG:boto:Data: 2022-09-02T05:37:38.705 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.707 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.707 DEBUG:boto:Params: {} 2022-09-02T05:37:38.708 DEBUG:boto:Token: None 2022-09-02T05:37:38.709 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c?versionId=null 2022-09-02T05:37:38.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c= 2022-09-02T05:37:38.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r6nJh6DM9cmSCSSXKvNB8bqxO5c=', 'Content-Length': '0'} 2022-09-02T05:37:38.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005025cff384590fb1-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.714 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.714 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.715 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.715 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.716 DEBUG:boto:Method: GET 2022-09-02T05:37:38.717 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:38.717 DEBUG:boto:Data: 2022-09-02T05:37:38.717 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.719 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.719 DEBUG:boto:Params: {} 2022-09-02T05:37:38.720 DEBUG:boto:Token: None 2022-09-02T05:37:38.720 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd58ecdfdd10d8c-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.725 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.726 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.726 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.727 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.728 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.728 DEBUG:boto:Method: GET 2022-09-02T05:37:38.729 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:38.729 DEBUG:boto:Data: 2022-09-02T05:37:38.730 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.731 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.732 DEBUG:boto:Params: {} 2022-09-02T05:37:38.733 DEBUG:boto:Token: None 2022-09-02T05:37:38.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aa70b514f146775-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.740 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.741 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.741 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.742 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.742 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:38.743 DEBUG:boto:Data: 2022-09-02T05:37:38.744 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.744 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.745 DEBUG:boto:Params: {} 2022-09-02T05:37:38.746 DEBUG:boto:Token: None 2022-09-02T05:37:38.746 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c 2022-09-02T05:37:38.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:75WyBygEcjXorTNKTWjFwVarPcY= 2022-09-02T05:37:38.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:75WyBygEcjXorTNKTWjFwVarPcY=', 'Content-Length': '0'} 2022-09-02T05:37:38.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690c38301e4a81de-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.750 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.751 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.752 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.752 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:38.753 DEBUG:boto:Data: 2022-09-02T05:37:38.753 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.754 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.755 DEBUG:boto:Params: {} 2022-09-02T05:37:38.755 DEBUG:boto:Token: None 2022-09-02T05:37:38.756 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c 2022-09-02T05:37:38.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:75WyBygEcjXorTNKTWjFwVarPcY= 2022-09-02T05:37:38.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:75WyBygEcjXorTNKTWjFwVarPcY=', 'Content-Length': '0'} 2022-09-02T05:37:38.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075eb855c1b5461ca-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2022-09-02T05:37:38.761 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.762 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.762 DEBUG:boto:Method: GET 2022-09-02T05:37:38.763 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:38.763 DEBUG:boto:Data: 2022-09-02T05:37:38.764 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.765 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.766 DEBUG:boto:Params: {} 2022-09-02T05:37:38.766 DEBUG:boto:Token: None 2022-09-02T05:37:38.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c 2022-09-02T05:37:38.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PvXPlk5rYn26NbgB2BoBQkQrOxc= 2022-09-02T05:37:38.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PvXPlk5rYn26NbgB2BoBQkQrOxc=', 'Content-Length': '0'} 2022-09-02T05:37:38.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e96d39f25f9019-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.771 DEBUG:boto:path=/xljbkf-72/c 2022-09-02T05:37:38.772 DEBUG:boto:auth_path=/xljbkf-72/c 2022-09-02T05:37:38.772 DEBUG:boto:Method: GET 2022-09-02T05:37:38.773 DEBUG:boto:Path: /xljbkf-72/c 2022-09-02T05:37:38.773 DEBUG:boto:Data: 2022-09-02T05:37:38.774 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.775 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.775 DEBUG:boto:Params: {} 2022-09-02T05:37:38.776 DEBUG:boto:Token: None 2022-09-02T05:37:38.777 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/c 2022-09-02T05:37:38.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PvXPlk5rYn26NbgB2BoBQkQrOxc= 2022-09-02T05:37:38.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PvXPlk5rYn26NbgB2BoBQkQrOxc=', 'Content-Length': '0'} 2022-09-02T05:37:38.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a1944ddb991e55-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.822 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.823 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.824 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.824 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.825 DEBUG:boto:Method: GET 2022-09-02T05:37:38.825 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:38.826 DEBUG:boto:Data: 2022-09-02T05:37:38.826 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.827 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.828 DEBUG:boto:Params: {} 2022-09-02T05:37:38.829 DEBUG:boto:Token: None 2022-09-02T05:37:38.829 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e8c2e05303fb19-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.834 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.835 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.836 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.836 DEBUG:boto:path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.837 DEBUG:boto:auth_path=/xljbkf-72/?prefix=c 2022-09-02T05:37:38.837 DEBUG:boto:Method: GET 2022-09-02T05:37:38.838 DEBUG:boto:Path: /xljbkf-72/?prefix=c 2022-09-02T05:37:38.838 DEBUG:boto:Data: 2022-09-02T05:37:38.839 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.840 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.841 DEBUG:boto:Params: {} 2022-09-02T05:37:38.841 DEBUG:boto:Token: None 2022-09-02T05:37:38.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d2a7da78cbbcf5-00631196a2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.847 DEBUG:boto:b'xljbkf-72c1000falsec2022-09-02T05:33:19.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:38.848 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:38.848 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:38.849 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.849 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.850 DEBUG:boto:Method: GET 2022-09-02T05:37:38.851 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.851 DEBUG:boto:Data: 2022-09-02T05:37:38.852 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.853 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.853 DEBUG:boto:Params: {} 2022-09-02T05:37:38.854 DEBUG:boto:Token: None 2022-09-02T05:37:38.854 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes= 2022-09-02T05:37:38.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes=', 'Content-Length': '0'} 2022-09-02T05:37:38.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db9648696099b2b-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.858 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:38.859 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:38.860 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.860 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.861 DEBUG:boto:Method: GET 2022-09-02T05:37:38.861 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.862 DEBUG:boto:Data: 2022-09-02T05:37:38.862 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.863 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.864 DEBUG:boto:Params: {} 2022-09-02T05:37:38.864 DEBUG:boto:Token: None 2022-09-02T05:37:38.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes= 2022-09-02T05:37:38.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes=', 'Content-Length': '0'} 2022-09-02T05:37:38.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c998583d3b489bc-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.869 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:38.870 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:38.870 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.871 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.872 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.872 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.873 DEBUG:boto:Data: 2022-09-02T05:37:38.873 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.874 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.875 DEBUG:boto:Params: {} 2022-09-02T05:37:38.875 DEBUG:boto:Token: None 2022-09-02T05:37:38.876 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IJobhITYXAJGRvO1q9HVGL20w+0= 2022-09-02T05:37:38.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IJobhITYXAJGRvO1q9HVGL20w+0=', 'Content-Length': '0'} 2022-09-02T05:37:38.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf9ea24cd127458-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.882 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:38.883 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:38.884 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.885 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.886 DEBUG:boto:Method: HEAD 2022-09-02T05:37:38.887 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.888 DEBUG:boto:Data: 2022-09-02T05:37:38.889 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.891 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.892 DEBUG:boto:Params: {} 2022-09-02T05:37:38.893 DEBUG:boto:Token: None 2022-09-02T05:37:38.894 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IJobhITYXAJGRvO1q9HVGL20w+0= 2022-09-02T05:37:38.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IJobhITYXAJGRvO1q9HVGL20w+0=', 'Content-Length': '0'} 2022-09-02T05:37:38.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0fb81db1c3549a-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:38.902 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:38.902 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:38.903 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.904 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.905 DEBUG:boto:Method: GET 2022-09-02T05:37:38.906 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.906 DEBUG:boto:Data: 2022-09-02T05:37:38.907 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.908 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.909 DEBUG:boto:Params: {} 2022-09-02T05:37:38.910 DEBUG:boto:Token: None 2022-09-02T05:37:38.911 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes= 2022-09-02T05:37:38.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes=', 'Content-Length': '0'} 2022-09-02T05:37:38.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001df88fba5b665bcd-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.954 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:38.955 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:38.956 DEBUG:boto:path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.957 DEBUG:boto:auth_path=/xljbkf-72/d?versionId=null 2022-09-02T05:37:38.958 DEBUG:boto:Method: GET 2022-09-02T05:37:38.959 DEBUG:boto:Path: /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.959 DEBUG:boto:Data: 2022-09-02T05:37:38.960 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.961 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.962 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.963 DEBUG:boto:Params: {} 2022-09-02T05:37:38.964 DEBUG:boto:Token: None 2022-09-02T05:37:38.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/d?versionId=null 2022-09-02T05:37:38.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes= 2022-09-02T05:37:38.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+N/KWbSQULO7kvlw+FYWNuSPes=', 'Content-Length': '0'} 2022-09-02T05:37:38.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000977aacce8fe6061c-00631196a2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.972 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.972 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.973 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:38.974 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:38.974 DEBUG:boto:Method: GET 2022-09-02T05:37:38.975 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:38.976 DEBUG:boto:Data: 2022-09-02T05:37:38.977 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:38.979 DEBUG:boto:Port: 8001 2022-09-02T05:37:38.980 DEBUG:boto:Params: {} 2022-09-02T05:37:38.981 DEBUG:boto:Token: None 2022-09-02T05:37:38.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:38.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000014fdaf43e206986-00631196a2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:38.990 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:38.991 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:38.991 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:38.992 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:38.992 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:38.993 DEBUG:boto:Method: GET 2022-09-02T05:37:38.993 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:38.994 DEBUG:boto:Data: 2022-09-02T05:37:38.995 DEBUG:boto:Headers: {} 2022-09-02T05:37:38.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:38.996 DEBUG:boto:Port: 8000 2022-09-02T05:37:38.996 DEBUG:boto:Params: {} 2022-09-02T05:37:38.997 DEBUG:boto:Token: None 2022-09-02T05:37:38.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:38 GMT /xljbkf-72/ 2022-09-02T05:37:38.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs= 2022-09-02T05:37:38.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTz0mQEBMkL6KqnIWozSgNVFqQs=', 'Content-Length': '0'} 2022-09-02T05:37:39.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025cd07dad849beba-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.050 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.051 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:39.052 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:39.052 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.053 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:39.053 DEBUG:boto:Data: 2022-09-02T05:37:39.054 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.055 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.055 DEBUG:boto:Params: {} 2022-09-02T05:37:39.056 DEBUG:boto:Token: None 2022-09-02T05:37:39.056 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/d 2022-09-02T05:37:39.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vV0tObq+ZPHWFkRmf2FNVxp9/Nw= 2022-09-02T05:37:39.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vV0tObq+ZPHWFkRmf2FNVxp9/Nw=', 'Content-Length': '0'} 2022-09-02T05:37:39.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd7e0c8f87138bb-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.061 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:39.062 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:39.062 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.063 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:39.063 DEBUG:boto:Data: 2022-09-02T05:37:39.064 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.065 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.065 DEBUG:boto:Params: {} 2022-09-02T05:37:39.066 DEBUG:boto:Token: None 2022-09-02T05:37:39.066 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/d 2022-09-02T05:37:39.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vV0tObq+ZPHWFkRmf2FNVxp9/Nw= 2022-09-02T05:37:39.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vV0tObq+ZPHWFkRmf2FNVxp9/Nw=', 'Content-Length': '0'} 2022-09-02T05:37:39.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf6d9bfd0c8e2eb-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2022-09-02T05:37:39.071 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:39.072 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:39.072 DEBUG:boto:Method: GET 2022-09-02T05:37:39.073 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:39.073 DEBUG:boto:Data: 2022-09-02T05:37:39.074 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.075 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.075 DEBUG:boto:Params: {} 2022-09-02T05:37:39.076 DEBUG:boto:Token: None 2022-09-02T05:37:39.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/d 2022-09-02T05:37:39.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZSyrvggYlq4jJEsgY5EB7aNu9A= 2022-09-02T05:37:39.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZSyrvggYlq4jJEsgY5EB7aNu9A=', 'Content-Length': '0'} 2022-09-02T05:37:39.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c43d6018220583a-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.081 DEBUG:boto:path=/xljbkf-72/d 2022-09-02T05:37:39.081 DEBUG:boto:auth_path=/xljbkf-72/d 2022-09-02T05:37:39.082 DEBUG:boto:Method: GET 2022-09-02T05:37:39.082 DEBUG:boto:Path: /xljbkf-72/d 2022-09-02T05:37:39.083 DEBUG:boto:Data: 2022-09-02T05:37:39.083 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.084 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.085 DEBUG:boto:Params: {} 2022-09-02T05:37:39.086 DEBUG:boto:Token: None 2022-09-02T05:37:39.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/d 2022-09-02T05:37:39.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZSyrvggYlq4jJEsgY5EB7aNu9A= 2022-09-02T05:37:39.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZSyrvggYlq4jJEsgY5EB7aNu9A=', 'Content-Length': '0'} 2022-09-02T05:37:39.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702196c919e02c5c-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.091 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.092 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.093 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:39.094 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:39.095 DEBUG:boto:Method: GET 2022-09-02T05:37:39.096 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:39.097 DEBUG:boto:Data: 2022-09-02T05:37:39.098 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.100 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.101 DEBUG:boto:Params: {} 2022-09-02T05:37:39.102 DEBUG:boto:Token: None 2022-09-02T05:37:39.102 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf8b3856dc4a943-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.109 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.110 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.111 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.111 DEBUG:boto:path=/xljbkf-72/?prefix=d 2022-09-02T05:37:39.112 DEBUG:boto:auth_path=/xljbkf-72/?prefix=d 2022-09-02T05:37:39.112 DEBUG:boto:Method: GET 2022-09-02T05:37:39.113 DEBUG:boto:Path: /xljbkf-72/?prefix=d 2022-09-02T05:37:39.113 DEBUG:boto:Data: 2022-09-02T05:37:39.114 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.115 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.116 DEBUG:boto:Params: {} 2022-09-02T05:37:39.116 DEBUG:boto:Token: None 2022-09-02T05:37:39.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046f66bb41c598d6a-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.122 DEBUG:boto:b'xljbkf-72d1000falsed2022-09-02T05:33:19.391Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:39.123 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.124 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.125 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.125 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.126 DEBUG:boto:Method: GET 2022-09-02T05:37:39.126 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.127 DEBUG:boto:Data: 2022-09-02T05:37:39.127 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.129 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.129 DEBUG:boto:Params: {} 2022-09-02T05:37:39.130 DEBUG:boto:Token: None 2022-09-02T05:37:39.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU= 2022-09-02T05:37:39.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU=', 'Content-Length': '0'} 2022-09-02T05:37:39.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6847884abf9ffea-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.136 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.137 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.137 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.138 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.138 DEBUG:boto:Method: GET 2022-09-02T05:37:39.139 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.140 DEBUG:boto:Data: 2022-09-02T05:37:39.141 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.142 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.143 DEBUG:boto:Params: {} 2022-09-02T05:37:39.143 DEBUG:boto:Token: None 2022-09-02T05:37:39.144 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU= 2022-09-02T05:37:39.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU=', 'Content-Length': '0'} 2022-09-02T05:37:39.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d087f323d8bb1acb-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.151 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.152 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.153 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.154 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.155 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.156 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.157 DEBUG:boto:Data: 2022-09-02T05:37:39.158 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.160 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.161 DEBUG:boto:Params: {} 2022-09-02T05:37:39.162 DEBUG:boto:Token: None 2022-09-02T05:37:39.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:twg2IcnXei4Qa4gUP+7JKJLYjO0= 2022-09-02T05:37:39.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:twg2IcnXei4Qa4gUP+7JKJLYjO0=', 'Content-Length': '0'} 2022-09-02T05:37:39.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbda189ba7a66318-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.166 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.167 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.167 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.168 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.168 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.169 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.169 DEBUG:boto:Data: 2022-09-02T05:37:39.170 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.171 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.171 DEBUG:boto:Params: {} 2022-09-02T05:37:39.172 DEBUG:boto:Token: None 2022-09-02T05:37:39.172 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:twg2IcnXei4Qa4gUP+7JKJLYjO0= 2022-09-02T05:37:39.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:twg2IcnXei4Qa4gUP+7JKJLYjO0=', 'Content-Length': '0'} 2022-09-02T05:37:39.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f346d4319112767b-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:39.177 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.177 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.178 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.179 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.179 DEBUG:boto:Method: GET 2022-09-02T05:37:39.180 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.180 DEBUG:boto:Data: 2022-09-02T05:37:39.181 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.182 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.182 DEBUG:boto:Params: {} 2022-09-02T05:37:39.183 DEBUG:boto:Token: None 2022-09-02T05:37:39.184 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU= 2022-09-02T05:37:39.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU=', 'Content-Length': '0'} 2022-09-02T05:37:39.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823c126fe2ceb3db-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.188 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.189 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.189 DEBUG:boto:path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.190 DEBUG:boto:auth_path=/xljbkf-72/e?versionId=null 2022-09-02T05:37:39.190 DEBUG:boto:Method: GET 2022-09-02T05:37:39.191 DEBUG:boto:Path: /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.191 DEBUG:boto:Data: 2022-09-02T05:37:39.192 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.193 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.193 DEBUG:boto:Params: {} 2022-09-02T05:37:39.194 DEBUG:boto:Token: None 2022-09-02T05:37:39.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e?versionId=null 2022-09-02T05:37:39.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU= 2022-09-02T05:37:39.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RwAw72WyFlz4wdtZD5S3gwPuPMU=', 'Content-Length': '0'} 2022-09-02T05:37:39.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3638d7e8c320b62-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.199 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.200 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.200 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.201 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.201 DEBUG:boto:Method: GET 2022-09-02T05:37:39.202 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:39.202 DEBUG:boto:Data: 2022-09-02T05:37:39.203 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.204 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.205 DEBUG:boto:Params: {} 2022-09-02T05:37:39.206 DEBUG:boto:Token: None 2022-09-02T05:37:39.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ba0fe9fbf1187e-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.211 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.212 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.213 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.213 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.214 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.214 DEBUG:boto:Method: GET 2022-09-02T05:37:39.215 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:39.215 DEBUG:boto:Data: 2022-09-02T05:37:39.216 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.217 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.217 DEBUG:boto:Params: {} 2022-09-02T05:37:39.218 DEBUG:boto:Token: None 2022-09-02T05:37:39.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008603efc9c0d9c1c0-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.223 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.224 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.225 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.225 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.226 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:39.226 DEBUG:boto:Data: 2022-09-02T05:37:39.227 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.228 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.228 DEBUG:boto:Params: {} 2022-09-02T05:37:39.229 DEBUG:boto:Token: None 2022-09-02T05:37:39.229 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e 2022-09-02T05:37:39.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OwgbcOgfZSPqWJYFpzxVZtMJ5as= 2022-09-02T05:37:39.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OwgbcOgfZSPqWJYFpzxVZtMJ5as=', 'Content-Length': '0'} 2022-09-02T05:37:39.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000884fbcf634c0d878-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.233 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.234 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.235 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.235 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:39.236 DEBUG:boto:Data: 2022-09-02T05:37:39.236 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.237 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.238 DEBUG:boto:Params: {} 2022-09-02T05:37:39.239 DEBUG:boto:Token: None 2022-09-02T05:37:39.239 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e 2022-09-02T05:37:39.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OwgbcOgfZSPqWJYFpzxVZtMJ5as= 2022-09-02T05:37:39.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OwgbcOgfZSPqWJYFpzxVZtMJ5as=', 'Content-Length': '0'} 2022-09-02T05:37:39.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000913e40e6573716a7-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2022-09-02T05:37:39.245 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.245 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.246 DEBUG:boto:Method: GET 2022-09-02T05:37:39.247 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:39.247 DEBUG:boto:Data: 2022-09-02T05:37:39.248 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.249 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.250 DEBUG:boto:Params: {} 2022-09-02T05:37:39.250 DEBUG:boto:Token: None 2022-09-02T05:37:39.251 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e 2022-09-02T05:37:39.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GBEXcyONXfqWE/qGjZrPltqgb8I= 2022-09-02T05:37:39.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GBEXcyONXfqWE/qGjZrPltqgb8I=', 'Content-Length': '0'} 2022-09-02T05:37:39.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d28df6cd827c941-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.302 DEBUG:boto:path=/xljbkf-72/e 2022-09-02T05:37:39.303 DEBUG:boto:auth_path=/xljbkf-72/e 2022-09-02T05:37:39.304 DEBUG:boto:Method: GET 2022-09-02T05:37:39.304 DEBUG:boto:Path: /xljbkf-72/e 2022-09-02T05:37:39.305 DEBUG:boto:Data: 2022-09-02T05:37:39.305 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.306 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.307 DEBUG:boto:Params: {} 2022-09-02T05:37:39.308 DEBUG:boto:Token: None 2022-09-02T05:37:39.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/e 2022-09-02T05:37:39.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GBEXcyONXfqWE/qGjZrPltqgb8I= 2022-09-02T05:37:39.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GBEXcyONXfqWE/qGjZrPltqgb8I=', 'Content-Length': '0'} 2022-09-02T05:37:39.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a0efcf333c71bdc-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.312 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.313 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.314 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.315 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.315 DEBUG:boto:Method: GET 2022-09-02T05:37:39.316 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:39.317 DEBUG:boto:Data: 2022-09-02T05:37:39.317 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.318 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.319 DEBUG:boto:Params: {} 2022-09-02T05:37:39.319 DEBUG:boto:Token: None 2022-09-02T05:37:39.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03c84108915f3f1-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.325 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.325 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.326 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.326 DEBUG:boto:path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.327 DEBUG:boto:auth_path=/xljbkf-72/?prefix=e 2022-09-02T05:37:39.328 DEBUG:boto:Method: GET 2022-09-02T05:37:39.328 DEBUG:boto:Path: /xljbkf-72/?prefix=e 2022-09-02T05:37:39.329 DEBUG:boto:Data: 2022-09-02T05:37:39.329 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.330 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.331 DEBUG:boto:Params: {} 2022-09-02T05:37:39.331 DEBUG:boto:Token: None 2022-09-02T05:37:39.332 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b709c162e6d1036-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.337 DEBUG:boto:b'xljbkf-72e1000falsee2022-09-02T05:33:19.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:39.339 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.339 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.340 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.340 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.341 DEBUG:boto:Method: GET 2022-09-02T05:37:39.341 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.342 DEBUG:boto:Data: 2022-09-02T05:37:39.342 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.343 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.344 DEBUG:boto:Params: {} 2022-09-02T05:37:39.344 DEBUG:boto:Token: None 2022-09-02T05:37:39.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80= 2022-09-02T05:37:39.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80=', 'Content-Length': '0'} 2022-09-02T05:37:39.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006761157d1010df6a-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.349 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.350 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.351 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.351 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.352 DEBUG:boto:Method: GET 2022-09-02T05:37:39.352 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.353 DEBUG:boto:Data: 2022-09-02T05:37:39.353 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.354 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.355 DEBUG:boto:Params: {} 2022-09-02T05:37:39.355 DEBUG:boto:Token: None 2022-09-02T05:37:39.356 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80= 2022-09-02T05:37:39.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80=', 'Content-Length': '0'} 2022-09-02T05:37:39.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99e3bf0206d3733-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.360 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.361 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.361 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.362 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.362 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.363 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.363 DEBUG:boto:Data: 2022-09-02T05:37:39.364 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.365 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.365 DEBUG:boto:Params: {} 2022-09-02T05:37:39.366 DEBUG:boto:Token: None 2022-09-02T05:37:39.366 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUsuBF9Zxb0LYTIruvmVwJi2spA= 2022-09-02T05:37:39.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUsuBF9Zxb0LYTIruvmVwJi2spA=', 'Content-Length': '0'} 2022-09-02T05:37:39.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311487e0b177de08-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.371 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.371 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.372 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.372 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.373 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.373 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.374 DEBUG:boto:Data: 2022-09-02T05:37:39.374 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.375 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.376 DEBUG:boto:Params: {} 2022-09-02T05:37:39.376 DEBUG:boto:Token: None 2022-09-02T05:37:39.377 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUsuBF9Zxb0LYTIruvmVwJi2spA= 2022-09-02T05:37:39.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUsuBF9Zxb0LYTIruvmVwJi2spA=', 'Content-Length': '0'} 2022-09-02T05:37:39.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004335e37f80ff4ce4-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:39.381 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.382 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.382 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.383 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.383 DEBUG:boto:Method: GET 2022-09-02T05:37:39.384 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.384 DEBUG:boto:Data: 2022-09-02T05:37:39.385 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.386 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.387 DEBUG:boto:Params: {} 2022-09-02T05:37:39.387 DEBUG:boto:Token: None 2022-09-02T05:37:39.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80= 2022-09-02T05:37:39.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80=', 'Content-Length': '0'} 2022-09-02T05:37:39.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb45a753a6c5dc19-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.392 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.393 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.393 DEBUG:boto:path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.394 DEBUG:boto:auth_path=/xljbkf-72/f?versionId=null 2022-09-02T05:37:39.394 DEBUG:boto:Method: GET 2022-09-02T05:37:39.395 DEBUG:boto:Path: /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.395 DEBUG:boto:Data: 2022-09-02T05:37:39.396 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.397 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.397 DEBUG:boto:Params: {} 2022-09-02T05:37:39.398 DEBUG:boto:Token: None 2022-09-02T05:37:39.399 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f?versionId=null 2022-09-02T05:37:39.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80= 2022-09-02T05:37:39.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ajb4VLa82KbRNIXZJKNhYuYnb80=', 'Content-Length': '0'} 2022-09-02T05:37:39.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0446fd219041e68-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.403 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.403 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.404 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.404 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.405 DEBUG:boto:Method: GET 2022-09-02T05:37:39.405 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:39.406 DEBUG:boto:Data: 2022-09-02T05:37:39.407 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.408 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.408 DEBUG:boto:Params: {} 2022-09-02T05:37:39.409 DEBUG:boto:Token: None 2022-09-02T05:37:39.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c05ac9e80f95fb5-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.415 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.416 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.416 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.417 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.417 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.418 DEBUG:boto:Method: GET 2022-09-02T05:37:39.419 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:39.420 DEBUG:boto:Data: 2022-09-02T05:37:39.421 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.422 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.422 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.423 DEBUG:boto:Params: {} 2022-09-02T05:37:39.424 DEBUG:boto:Token: None 2022-09-02T05:37:39.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c6b9d8048b5ae0a-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.432 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.432 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.433 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.434 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.434 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:39.435 DEBUG:boto:Data: 2022-09-02T05:37:39.436 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.437 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.437 DEBUG:boto:Params: {} 2022-09-02T05:37:39.438 DEBUG:boto:Token: None 2022-09-02T05:37:39.438 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f 2022-09-02T05:37:39.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LseJYQX6/43B2iwJl1sRzxeS6iI= 2022-09-02T05:37:39.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LseJYQX6/43B2iwJl1sRzxeS6iI=', 'Content-Length': '0'} 2022-09-02T05:37:39.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca32eb4601203420-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.443 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.444 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.444 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.445 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:39.445 DEBUG:boto:Data: 2022-09-02T05:37:39.446 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.447 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.447 DEBUG:boto:Params: {} 2022-09-02T05:37:39.448 DEBUG:boto:Token: None 2022-09-02T05:37:39.448 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f 2022-09-02T05:37:39.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LseJYQX6/43B2iwJl1sRzxeS6iI= 2022-09-02T05:37:39.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LseJYQX6/43B2iwJl1sRzxeS6iI=', 'Content-Length': '0'} 2022-09-02T05:37:39.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c0ed82034e58683-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2022-09-02T05:37:39.453 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.453 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.454 DEBUG:boto:Method: GET 2022-09-02T05:37:39.455 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:39.455 DEBUG:boto:Data: 2022-09-02T05:37:39.456 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.457 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.457 DEBUG:boto:Params: {} 2022-09-02T05:37:39.458 DEBUG:boto:Token: None 2022-09-02T05:37:39.458 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f 2022-09-02T05:37:39.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cU0vSnSLDtD5PCdus2T/e3ELWDc= 2022-09-02T05:37:39.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cU0vSnSLDtD5PCdus2T/e3ELWDc=', 'Content-Length': '0'} 2022-09-02T05:37:39.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c0ede8b9845de2-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.463 DEBUG:boto:path=/xljbkf-72/f 2022-09-02T05:37:39.464 DEBUG:boto:auth_path=/xljbkf-72/f 2022-09-02T05:37:39.465 DEBUG:boto:Method: GET 2022-09-02T05:37:39.466 DEBUG:boto:Path: /xljbkf-72/f 2022-09-02T05:37:39.466 DEBUG:boto:Data: 2022-09-02T05:37:39.467 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.469 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.470 DEBUG:boto:Params: {} 2022-09-02T05:37:39.471 DEBUG:boto:Token: None 2022-09-02T05:37:39.472 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/f 2022-09-02T05:37:39.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cU0vSnSLDtD5PCdus2T/e3ELWDc= 2022-09-02T05:37:39.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cU0vSnSLDtD5PCdus2T/e3ELWDc=', 'Content-Length': '0'} 2022-09-02T05:37:39.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093ea6cb3b7662cf-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.477 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.478 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.478 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.479 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.479 DEBUG:boto:Method: GET 2022-09-02T05:37:39.480 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:39.480 DEBUG:boto:Data: 2022-09-02T05:37:39.482 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.483 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.484 DEBUG:boto:Params: {} 2022-09-02T05:37:39.485 DEBUG:boto:Token: None 2022-09-02T05:37:39.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095c8ef981eb26f1e-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.538 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.540 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.541 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.541 DEBUG:boto:path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.542 DEBUG:boto:auth_path=/xljbkf-72/?prefix=f 2022-09-02T05:37:39.543 DEBUG:boto:Method: GET 2022-09-02T05:37:39.544 DEBUG:boto:Path: /xljbkf-72/?prefix=f 2022-09-02T05:37:39.545 DEBUG:boto:Data: 2022-09-02T05:37:39.546 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.546 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.547 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.547 DEBUG:boto:Params: {} 2022-09-02T05:37:39.548 DEBUG:boto:Token: None 2022-09-02T05:37:39.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000132a79d0469d3d14-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.554 DEBUG:boto:b'xljbkf-72f1000falsef2022-09-02T05:33:19.628Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:39.555 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.556 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.557 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.558 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.559 DEBUG:boto:Method: GET 2022-09-02T05:37:39.560 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.561 DEBUG:boto:Data: 2022-09-02T05:37:39.562 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.564 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.565 DEBUG:boto:Params: {} 2022-09-02T05:37:39.566 DEBUG:boto:Token: None 2022-09-02T05:37:39.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08= 2022-09-02T05:37:39.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08=', 'Content-Length': '0'} 2022-09-02T05:37:39.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872b299928c5a76a-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.572 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.573 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.573 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.574 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.574 DEBUG:boto:Method: GET 2022-09-02T05:37:39.575 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.576 DEBUG:boto:Data: 2022-09-02T05:37:39.576 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.577 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.578 DEBUG:boto:Params: {} 2022-09-02T05:37:39.578 DEBUG:boto:Token: None 2022-09-02T05:37:39.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08= 2022-09-02T05:37:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08=', 'Content-Length': '0'} 2022-09-02T05:37:39.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc49d510d58aad1-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.584 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.585 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.586 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.586 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.587 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.587 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.588 DEBUG:boto:Data: 2022-09-02T05:37:39.588 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.589 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.590 DEBUG:boto:Params: {} 2022-09-02T05:37:39.590 DEBUG:boto:Token: None 2022-09-02T05:37:39.591 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AIuSUivKmSYwka4CSlK5Y9qAf0s= 2022-09-02T05:37:39.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AIuSUivKmSYwka4CSlK5Y9qAf0s=', 'Content-Length': '0'} 2022-09-02T05:37:39.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d022accfe9eba0-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.595 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.596 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.596 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.597 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.597 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.598 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.598 DEBUG:boto:Data: 2022-09-02T05:37:39.599 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.600 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.600 DEBUG:boto:Params: {} 2022-09-02T05:37:39.601 DEBUG:boto:Token: None 2022-09-02T05:37:39.601 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AIuSUivKmSYwka4CSlK5Y9qAf0s= 2022-09-02T05:37:39.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AIuSUivKmSYwka4CSlK5Y9qAf0s=', 'Content-Length': '0'} 2022-09-02T05:37:39.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa54f58cb6389c04-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:39.606 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.607 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.607 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.608 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.608 DEBUG:boto:Method: GET 2022-09-02T05:37:39.609 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.609 DEBUG:boto:Data: 2022-09-02T05:37:39.610 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.611 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.611 DEBUG:boto:Params: {} 2022-09-02T05:37:39.612 DEBUG:boto:Token: None 2022-09-02T05:37:39.613 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08= 2022-09-02T05:37:39.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08=', 'Content-Length': '0'} 2022-09-02T05:37:39.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002271c2ea2ba0bc44-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.617 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.618 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.618 DEBUG:boto:path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.619 DEBUG:boto:auth_path=/xljbkf-72/g?versionId=null 2022-09-02T05:37:39.619 DEBUG:boto:Method: GET 2022-09-02T05:37:39.620 DEBUG:boto:Path: /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.620 DEBUG:boto:Data: 2022-09-02T05:37:39.621 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.621 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.622 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.622 DEBUG:boto:Params: {} 2022-09-02T05:37:39.623 DEBUG:boto:Token: None 2022-09-02T05:37:39.624 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g?versionId=null 2022-09-02T05:37:39.624 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08= 2022-09-02T05:37:39.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2b2KjPOKV2thaJzhiYIkYFOai08=', 'Content-Length': '0'} 2022-09-02T05:37:39.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c39dcc084b11237-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.674 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.675 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.675 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.676 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.676 DEBUG:boto:Method: GET 2022-09-02T05:37:39.677 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:39.677 DEBUG:boto:Data: 2022-09-02T05:37:39.678 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.679 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.680 DEBUG:boto:Params: {} 2022-09-02T05:37:39.680 DEBUG:boto:Token: None 2022-09-02T05:37:39.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036427784b2605666-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.687 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.688 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.689 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.690 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.691 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.692 DEBUG:boto:Method: GET 2022-09-02T05:37:39.693 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:39.694 DEBUG:boto:Data: 2022-09-02T05:37:39.695 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.697 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.698 DEBUG:boto:Params: {} 2022-09-02T05:37:39.699 DEBUG:boto:Token: None 2022-09-02T05:37:39.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea81c36e30878a79-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.708 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.709 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.710 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.710 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.711 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:39.711 DEBUG:boto:Data: 2022-09-02T05:37:39.712 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.713 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.714 DEBUG:boto:Params: {} 2022-09-02T05:37:39.715 DEBUG:boto:Token: None 2022-09-02T05:37:39.716 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g 2022-09-02T05:37:39.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dhyFSpTLXmxHBbSbEGCDgeGOSSI= 2022-09-02T05:37:39.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dhyFSpTLXmxHBbSbEGCDgeGOSSI=', 'Content-Length': '0'} 2022-09-02T05:37:39.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4195bd2a9549bfb-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.723 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.723 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.724 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.724 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:39.725 DEBUG:boto:Data: 2022-09-02T05:37:39.725 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.726 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.727 DEBUG:boto:Params: {} 2022-09-02T05:37:39.727 DEBUG:boto:Token: None 2022-09-02T05:37:39.728 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g 2022-09-02T05:37:39.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dhyFSpTLXmxHBbSbEGCDgeGOSSI= 2022-09-02T05:37:39.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dhyFSpTLXmxHBbSbEGCDgeGOSSI=', 'Content-Length': '0'} 2022-09-02T05:37:39.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006062d2a4ee5cbc28-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2022-09-02T05:37:39.734 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.735 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.735 DEBUG:boto:Method: GET 2022-09-02T05:37:39.736 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:39.736 DEBUG:boto:Data: 2022-09-02T05:37:39.737 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.738 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.738 DEBUG:boto:Params: {} 2022-09-02T05:37:39.739 DEBUG:boto:Token: None 2022-09-02T05:37:39.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g 2022-09-02T05:37:39.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:adKu5jCUenCt70vNp1G7aSzmWBM= 2022-09-02T05:37:39.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:adKu5jCUenCt70vNp1G7aSzmWBM=', 'Content-Length': '0'} 2022-09-02T05:37:39.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003611ce1abda02cc-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.744 DEBUG:boto:path=/xljbkf-72/g 2022-09-02T05:37:39.745 DEBUG:boto:auth_path=/xljbkf-72/g 2022-09-02T05:37:39.745 DEBUG:boto:Method: GET 2022-09-02T05:37:39.746 DEBUG:boto:Path: /xljbkf-72/g 2022-09-02T05:37:39.746 DEBUG:boto:Data: 2022-09-02T05:37:39.747 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.748 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.748 DEBUG:boto:Params: {} 2022-09-02T05:37:39.749 DEBUG:boto:Token: None 2022-09-02T05:37:39.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/g 2022-09-02T05:37:39.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:adKu5jCUenCt70vNp1G7aSzmWBM= 2022-09-02T05:37:39.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:adKu5jCUenCt70vNp1G7aSzmWBM=', 'Content-Length': '0'} 2022-09-02T05:37:39.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cd71001837ffd0-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.754 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.754 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.755 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.755 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.756 DEBUG:boto:Method: GET 2022-09-02T05:37:39.757 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:39.757 DEBUG:boto:Data: 2022-09-02T05:37:39.758 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.759 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.759 DEBUG:boto:Params: {} 2022-09-02T05:37:39.760 DEBUG:boto:Token: None 2022-09-02T05:37:39.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d4aea164d7d9b2-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.765 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.766 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.767 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.767 DEBUG:boto:path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.768 DEBUG:boto:auth_path=/xljbkf-72/?prefix=g 2022-09-02T05:37:39.768 DEBUG:boto:Method: GET 2022-09-02T05:37:39.769 DEBUG:boto:Path: /xljbkf-72/?prefix=g 2022-09-02T05:37:39.769 DEBUG:boto:Data: 2022-09-02T05:37:39.770 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.771 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.772 DEBUG:boto:Params: {} 2022-09-02T05:37:39.772 DEBUG:boto:Token: None 2022-09-02T05:37:39.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d97064555ddc097-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.778 DEBUG:boto:b'xljbkf-72g1000falseg2022-09-02T05:33:19.658Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:39.779 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.780 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.780 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.781 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.781 DEBUG:boto:Method: GET 2022-09-02T05:37:39.782 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.783 DEBUG:boto:Data: 2022-09-02T05:37:39.783 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.784 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.785 DEBUG:boto:Params: {} 2022-09-02T05:37:39.785 DEBUG:boto:Token: None 2022-09-02T05:37:39.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q= 2022-09-02T05:37:39.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q=', 'Content-Length': '0'} 2022-09-02T05:37:39.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d764d0a4d9ec2c1f-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.790 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.791 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.791 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.792 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.792 DEBUG:boto:Method: GET 2022-09-02T05:37:39.793 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.794 DEBUG:boto:Data: 2022-09-02T05:37:39.794 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.795 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.795 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.796 DEBUG:boto:Params: {} 2022-09-02T05:37:39.796 DEBUG:boto:Token: None 2022-09-02T05:37:39.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q= 2022-09-02T05:37:39.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q=', 'Content-Length': '0'} 2022-09-02T05:37:39.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000214bdc37f19eb3fd-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.805 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.806 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.806 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.807 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.807 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.808 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.808 DEBUG:boto:Data: 2022-09-02T05:37:39.809 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.810 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.810 DEBUG:boto:Params: {} 2022-09-02T05:37:39.811 DEBUG:boto:Token: None 2022-09-02T05:37:39.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FqrxK6Jq9murZcID3QyFfmXr1cc= 2022-09-02T05:37:39.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FqrxK6Jq9murZcID3QyFfmXr1cc=', 'Content-Length': '0'} 2022-09-02T05:37:39.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cda8e0b4ddda321-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.816 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.816 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.817 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.817 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.818 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.818 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.819 DEBUG:boto:Data: 2022-09-02T05:37:39.819 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.820 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.820 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.821 DEBUG:boto:Params: {} 2022-09-02T05:37:39.821 DEBUG:boto:Token: None 2022-09-02T05:37:39.822 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FqrxK6Jq9murZcID3QyFfmXr1cc= 2022-09-02T05:37:39.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FqrxK6Jq9murZcID3QyFfmXr1cc=', 'Content-Length': '0'} 2022-09-02T05:37:39.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb78f8da588ad5f-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:39.827 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.827 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.828 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.828 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.829 DEBUG:boto:Method: GET 2022-09-02T05:37:39.829 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.830 DEBUG:boto:Data: 2022-09-02T05:37:39.830 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.831 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.832 DEBUG:boto:Params: {} 2022-09-02T05:37:39.833 DEBUG:boto:Token: None 2022-09-02T05:37:39.833 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q= 2022-09-02T05:37:39.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q=', 'Content-Length': '0'} 2022-09-02T05:37:39.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621c58125d445814-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.882 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.883 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.884 DEBUG:boto:path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.885 DEBUG:boto:auth_path=/xljbkf-72/h?versionId=null 2022-09-02T05:37:39.885 DEBUG:boto:Method: GET 2022-09-02T05:37:39.886 DEBUG:boto:Path: /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.886 DEBUG:boto:Data: 2022-09-02T05:37:39.887 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.888 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.888 DEBUG:boto:Params: {} 2022-09-02T05:37:39.889 DEBUG:boto:Token: None 2022-09-02T05:37:39.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h?versionId=null 2022-09-02T05:37:39.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q= 2022-09-02T05:37:39.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pPmX0wrPdDR6Z4y/whd2IUZGX1Q=', 'Content-Length': '0'} 2022-09-02T05:37:39.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5278f461bdce9b-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.894 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.895 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.896 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.896 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.897 DEBUG:boto:Method: GET 2022-09-02T05:37:39.897 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:39.898 DEBUG:boto:Data: 2022-09-02T05:37:39.899 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.900 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.900 DEBUG:boto:Params: {} 2022-09-02T05:37:39.901 DEBUG:boto:Token: None 2022-09-02T05:37:39.901 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5ab9967d351e98-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.906 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.908 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.908 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.909 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.910 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.910 DEBUG:boto:Method: GET 2022-09-02T05:37:39.911 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:39.911 DEBUG:boto:Data: 2022-09-02T05:37:39.912 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.913 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.913 DEBUG:boto:Params: {} 2022-09-02T05:37:39.914 DEBUG:boto:Token: None 2022-09-02T05:37:39.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d20f44a08c47a2-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.920 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.921 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.922 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.922 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.923 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:39.923 DEBUG:boto:Data: 2022-09-02T05:37:39.924 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.925 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.925 DEBUG:boto:Params: {} 2022-09-02T05:37:39.926 DEBUG:boto:Token: None 2022-09-02T05:37:39.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h 2022-09-02T05:37:39.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o3jzDWTsVGr1H4tNABFgZX3nz08= 2022-09-02T05:37:39.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o3jzDWTsVGr1H4tNABFgZX3nz08=', 'Content-Length': '0'} 2022-09-02T05:37:39.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30011c2271f230e-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.931 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.931 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.932 DEBUG:boto:Method: HEAD 2022-09-02T05:37:39.932 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:39.933 DEBUG:boto:Data: 2022-09-02T05:37:39.933 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.934 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.934 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.935 DEBUG:boto:Params: {} 2022-09-02T05:37:39.935 DEBUG:boto:Token: None 2022-09-02T05:37:39.936 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h 2022-09-02T05:37:39.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o3jzDWTsVGr1H4tNABFgZX3nz08= 2022-09-02T05:37:39.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o3jzDWTsVGr1H4tNABFgZX3nz08=', 'Content-Length': '0'} 2022-09-02T05:37:39.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007175e5022045f533-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2022-09-02T05:37:39.940 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.941 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.942 DEBUG:boto:Method: GET 2022-09-02T05:37:39.942 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:39.943 DEBUG:boto:Data: 2022-09-02T05:37:39.943 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.944 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.945 DEBUG:boto:Params: {} 2022-09-02T05:37:39.945 DEBUG:boto:Token: None 2022-09-02T05:37:39.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h 2022-09-02T05:37:39.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:akFMmwt6ZwDbIQRS5FBS1TFuvS4= 2022-09-02T05:37:39.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:akFMmwt6ZwDbIQRS5FBS1TFuvS4=', 'Content-Length': '0'} 2022-09-02T05:37:39.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c85948df9a1e98a-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.951 DEBUG:boto:path=/xljbkf-72/h 2022-09-02T05:37:39.951 DEBUG:boto:auth_path=/xljbkf-72/h 2022-09-02T05:37:39.952 DEBUG:boto:Method: GET 2022-09-02T05:37:39.952 DEBUG:boto:Path: /xljbkf-72/h 2022-09-02T05:37:39.953 DEBUG:boto:Data: 2022-09-02T05:37:39.954 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.955 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.955 DEBUG:boto:Params: {} 2022-09-02T05:37:39.956 DEBUG:boto:Token: None 2022-09-02T05:37:39.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/h 2022-09-02T05:37:39.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:akFMmwt6ZwDbIQRS5FBS1TFuvS4= 2022-09-02T05:37:39.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:akFMmwt6ZwDbIQRS5FBS1TFuvS4=', 'Content-Length': '0'} 2022-09-02T05:37:39.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:33:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30df1ec44d666a8-00631196a3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.961 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.962 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.963 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.964 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.964 DEBUG:boto:Method: GET 2022-09-02T05:37:39.965 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:39.966 DEBUG:boto:Data: 2022-09-02T05:37:39.966 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:39.968 DEBUG:boto:Port: 8001 2022-09-02T05:37:39.968 DEBUG:boto:Params: {} 2022-09-02T05:37:39.969 DEBUG:boto:Token: None 2022-09-02T05:37:39.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c018ae0d28364c49-00631196a3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.975 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.976 DEBUG:boto:path=/xljbkf-72/ 2022-09-02T05:37:39.976 DEBUG:boto:auth_path=/xljbkf-72/ 2022-09-02T05:37:39.977 DEBUG:boto:path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.978 DEBUG:boto:auth_path=/xljbkf-72/?prefix=h 2022-09-02T05:37:39.978 DEBUG:boto:Method: GET 2022-09-02T05:37:39.979 DEBUG:boto:Path: /xljbkf-72/?prefix=h 2022-09-02T05:37:39.980 DEBUG:boto:Data: 2022-09-02T05:37:39.980 DEBUG:boto:Headers: {} 2022-09-02T05:37:39.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:39.982 DEBUG:boto:Port: 8000 2022-09-02T05:37:39.983 DEBUG:boto:Params: {} 2022-09-02T05:37:39.985 DEBUG:boto:Token: None 2022-09-02T05:37:39.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:39 GMT /xljbkf-72/ 2022-09-02T05:37:39.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA= 2022-09-02T05:37:39.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U9gSpIkDBoamUX2H8575FZqmcAA=', 'Content-Length': '0'} 2022-09-02T05:37:39.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb73ce2d71fdaf1e-00631196a3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:39.993 DEBUG:boto:b'xljbkf-72h1000falseh2022-09-02T05:33:19.722Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:37:39.994 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-72 zones={test-zone3, test-zone2} 2022-09-02T05:37:39.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2022-09-02T05:37:39.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.001 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.058 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.059 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:37:40.114 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:37:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:37:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:37:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096799.531739_388.1", 2022-09-02T05:37:40.120 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:33:19.531739Z" 2022-09-02T05:37:40.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.122 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:37:40.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.123 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:37:40.125 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.126 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:37:40.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.129 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.186 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.187 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:40.220 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:40.221+0000 7f7cdb3ae600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.221 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:40.222+0000 7f7cdb3ae600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.237 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:37:40.237 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:37:40.238 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:37:40.238 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:37:40.239 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:37:40.239 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:37:40.239 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:40.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.240 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:37:40.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.240 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:37:40.241 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.241 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.241 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:37:40.241 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.242 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:37:40.242 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.242 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:37:40.243 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.243 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.244 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:37:40.244 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.244 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.244 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:37:40.244 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.245 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.245 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.245 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:37:40.245 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.246 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.246 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:37:40.247 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.247 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.247 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096799.531739_388.1", 2022-09-02T05:37:40.247 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.248 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.248 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.248 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:33:19.531739Z", 2022-09-02T05:37:40.248 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:40.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:37:40.250 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:37:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.251 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:37:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.252 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:37:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.253 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:37:40.254 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:37:40.254 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:37:40.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.254 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:37:40Z" 2022-09-02T05:37:40.255 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:37:40.256 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096799.531739_388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:19.531739Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:37:40Z" } 2022-09-02T05:37:40.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096799.531739_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:19.531739Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:37:40.257 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.258 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.259 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:37:40.259 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:37:40.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.261 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:40.367 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:40.369+0000 7fe95fedb600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.368 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:40.370+0000 7fe95fedb600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.386 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:37:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:37:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:37:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:37:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:37:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:37:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:37:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:37:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:37:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:37:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:37:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:37:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:37:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:37:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:37:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:37:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:37:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:37:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662096799.531739_388.1", 2022-09-02T05:37:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:40.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:37:40.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:40.396 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:33:19.531739Z", 2022-09-02T05:37:40.396 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:37:40.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.397 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:37:40.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:40.398 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:40.398 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:37:40.398 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:40.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:37:40.399 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:40.399 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:37:40.399 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:37:40.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:37:40.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.400 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:37:40.401 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:37:40.401 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:37:40.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.401 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:37:40Z" 2022-09-02T05:37:40.402 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:37:40.403 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096799.531739_388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:19.531739Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:37:40Z" } 2022-09-02T05:37:40.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096799.531739_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:19.531739Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:37:40.404 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.405 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:37:40.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:37:40.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.407 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.458 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.459 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:40.498 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:40.499+0000 7fef1adf3600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.498 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:40.500+0000 7fef1adf3600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.519 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:37:40.519 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:37:40.520 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:37:40.520 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:37:40.520 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:37:40.520 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:37:40.521 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:40.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.521 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:37:40.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.522 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:37:40.522 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.523 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.523 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:37:40.523 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.523 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:37:40.524 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.524 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:37:40.524 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.524 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.525 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:37:40.526 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.526 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.526 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:37:40.527 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.527 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.528 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.529 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:37:40.530 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.532 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:37:40.532 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.533 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.533 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662096799.531739_388.1", 2022-09-02T05:37:40.534 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.534 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.535 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.535 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:33:19.531739Z", 2022-09-02T05:37:40.536 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:40.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.537 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:37:40.538 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.538 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.539 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:37:40.540 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.540 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.540 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.540 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:37:40.541 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.542 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:37:40.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.542 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:37:40.543 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:37:40.543 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:37:40.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.543 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:37:40Z" 2022-09-02T05:37:40.544 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:37:40.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662096799.531739_388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:33:19.531739Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:37:40Z" } 2022-09-02T05:37:40.546 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662096799.531739_388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:33:19.531739Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:37:40.546 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.547 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662096799.531739_388.1', 3: ''} 2022-09-02T05:37:40.547 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:37:40.548 DEBUG:boto:path=/ 2022-09-02T05:37:40.548 DEBUG:boto:auth_path=/ 2022-09-02T05:37:40.549 DEBUG:boto:Method: GET 2022-09-02T05:37:40.550 DEBUG:boto:Path: / 2022-09-02T05:37:40.550 DEBUG:boto:Data: 2022-09-02T05:37:40.551 DEBUG:boto:Headers: {} 2022-09-02T05:37:40.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:40.552 DEBUG:boto:Port: 8000 2022-09-02T05:37:40.552 DEBUG:boto:Params: {} 2022-09-02T05:37:40.553 DEBUG:boto:Token: None 2022-09-02T05:37:40.554 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:40 GMT / 2022-09-02T05:37:40.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KlOukxq67yjRDj3JI4HEdIikP6U= 2022-09-02T05:37:40.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KlOukxq67yjRDj3JI4HEdIikP6U=', 'Content-Length': '0'} 2022-09-02T05:37:40.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aee003772e9c9ff-00631196a4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:40.561 DEBUG:boto:path=/ 2022-09-02T05:37:40.561 DEBUG:boto:auth_path=/ 2022-09-02T05:37:40.562 DEBUG:boto:Method: GET 2022-09-02T05:37:40.562 DEBUG:boto:Path: / 2022-09-02T05:37:40.563 DEBUG:boto:Data: 2022-09-02T05:37:40.563 DEBUG:boto:Headers: {} 2022-09-02T05:37:40.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:37:40.564 DEBUG:boto:Port: 8000 2022-09-02T05:37:40.565 DEBUG:boto:Params: {} 2022-09-02T05:37:40.565 DEBUG:boto:Token: None 2022-09-02T05:37:40.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:40 GMT / 2022-09-02T05:37:40.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KlOukxq67yjRDj3JI4HEdIikP6U= 2022-09-02T05:37:40.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KlOukxq67yjRDj3JI4HEdIikP6U=', 'Content-Length': '0'} 2022-09-02T05:37:40.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a45b98066290fe-00631196a4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:40.573 DEBUG:boto:path=/ 2022-09-02T05:37:40.574 DEBUG:boto:auth_path=/ 2022-09-02T05:37:40.574 DEBUG:boto:Method: GET 2022-09-02T05:37:40.575 DEBUG:boto:Path: / 2022-09-02T05:37:40.576 DEBUG:boto:Data: 2022-09-02T05:37:40.576 DEBUG:boto:Headers: {} 2022-09-02T05:37:40.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:37:40.578 DEBUG:boto:Port: 8001 2022-09-02T05:37:40.579 DEBUG:boto:Params: {} 2022-09-02T05:37:40.579 DEBUG:boto:Token: None 2022-09-02T05:37:40.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:37:40 GMT / 2022-09-02T05:37:40.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KlOukxq67yjRDj3JI4HEdIikP6U= 2022-09-02T05:37:40.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KlOukxq67yjRDj3JI4HEdIikP6U=', 'Content-Length': '0'} 2022-09-02T05:37:40.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048b9100ab8b553b9-00631196a4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:37:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:40.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2022-09-02T05:37:40.586 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:40.587 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:40.588 DEBUG:boto:Method: PUT 2022-09-02T05:37:40.588 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:40.589 DEBUG:boto:Data: 2022-09-02T05:37:40.589 DEBUG:boto:Headers: {} 2022-09-02T05:37:40.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:40.590 DEBUG:boto:Port: 8000 2022-09-02T05:37:40.591 DEBUG:boto:Params: {} 2022-09-02T05:37:40.592 DEBUG:boto:Token: None 2022-09-02T05:37:40.592 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:37:40 GMT /xljbkf-73/ 2022-09-02T05:37:40.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:somKptZ8thwPNfTuP4yaa6m3MRA= 2022-09-02T05:37:40.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:somKptZ8thwPNfTuP4yaa6m3MRA=', 'Content-Length': '0'} 2022-09-02T05:37:40.618 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008a5a84b9d017a393-00631196a4-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:40.620 DEBUG:rgw_multi.tests:created bucket=xljbkf-73 2022-09-02T05:37:40.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.622 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:40.716 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:37:40.736 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:37:40.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:37:40.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.739 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:37:40.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.739 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662097060.616370_397.1", 2022-09-02T05:37:40.740 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:37:40.616370Z" 2022-09-02T05:37:40.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:40.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:40.741 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:40.741 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:37:40.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:40.742 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:37:40.744 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:40.745 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:37:40.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.747 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.797 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.797 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:40.827 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:40.828+0000 7f20f1649600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.828 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:40.829+0000 7f20f1649600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:40.876 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:37:40.877 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:37:40.877 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:37:40.877 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:37:40.878 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:37:40.878 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:37:40.878 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:40.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.879 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:37:40.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.879 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:37:40.879 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.880 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.880 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:37:40.880 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.881 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:37:40.881 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.881 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:37:40.881 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.882 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.882 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:37:40.883 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.883 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.883 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:37:40.883 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.884 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.884 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.884 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:37:40.884 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.885 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:37:40.886 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.886 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.886 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662097060.616370_397.1", 2022-09-02T05:37:40.886 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.887 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.887 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.887 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:37:40.616370Z", 2022-09-02T05:37:40.887 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:40.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:40.888 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:37:40.889 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:40.889 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:40.889 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:37:40.889 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:40.890 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:40.890 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:40.890 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:37:40.890 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:40.891 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.891 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:40.891 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:37:40.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.892 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:37:40.892 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:37:40.892 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:37:40.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:40.892 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:37:40Z" 2022-09-02T05:37:40.893 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:37:40.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662097060.616370_397.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:37:40.616370Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:37:40Z" } 2022-09-02T05:37:40.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662097060.616370_397.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:37:40.616370Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:37:40.895 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:40.896 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:40.896 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:37:40.897 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:37:40.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:40.898 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:40.947 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:40.948 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:41.002+0000 7f83811e7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:37:41.003+0000 7f83811e7600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:41.013 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:37:41.014 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:37:41.015 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:37:41.015 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:37:41.016 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:37:41.016 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:37:41.017 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:37:41.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:41.018 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:37:41.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:41.019 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:37:41.019 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:41.020 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:41.020 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:37:41.020 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:41.021 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:37:41.021 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:41.022 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:37:41.022 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:37:41.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:41.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:41.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:41.024 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:37:41.024 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:41.024 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:41.025 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:37:41.025 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:41.025 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:37:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:37:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:37:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:37:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662097060.616370_397.1", 2022-09-02T05:37:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:37:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:37:40.616370Z", 2022-09-02T05:37:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:37:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:37:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:37:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:37:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:37:41.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:37:41.035 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:37:41.036 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:37:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:37:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:37:41.037 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:37:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:37:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:37:41.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:37:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:37:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:37:41.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:37:41.040 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:37:41Z" 2022-09-02T05:37:41.040 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:37:41.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662097060.616370_397.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:37:40.616370Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:37:41Z" } 2022-09-02T05:37:41.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662097060.616370_397.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:37:40.616370Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:37:41.043 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:41.043 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:41.044 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:37:41.044 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:37:41.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:41.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:41.046 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:41.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:41.096 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:41.141 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:41.141+0000 7f1935770600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:41.142 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:41.142+0000 7f1935770600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:37:41.154 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:37:41.155 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:37:41.155 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:37:41.155 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:37:41.156 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:37:41.156 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:37:41.156 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:41.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:41.157 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:37:41.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:41.158 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:37:41.158 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:41.159 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:41.159 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:37:41.160 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:41.160 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:37:41.161 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:41.161 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:37:41.162 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:41.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:41.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:41.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:41.163 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:37:41.164 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:41.164 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:41.165 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:37:41.165 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:41.166 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:41.166 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:41.166 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:37:41.166 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:41.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:41.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:41.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:41.167 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:37:41.168 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:41.168 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:41.168 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662097060.616370_397.1", 2022-09-02T05:37:41.168 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:41.169 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:41.169 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:41.169 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:37:40.616370Z", 2022-09-02T05:37:41.169 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:37:41.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:41.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:41.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:37:41.170 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:37:41.171 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:37:41.171 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:37:41.171 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:37:41.171 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:37:41.171 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:37:41.172 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:37:41.172 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:37:41.172 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:37:41.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:41.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:37:41.173 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:37:41.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:41.174 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:37:41.174 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:37:41.174 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:37:41.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:37:41.175 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:37:41Z" 2022-09-02T05:37:41.175 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:37:41.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662097060.616370_397.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:37:40.616370Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:37:41Z" } 2022-09-02T05:37:41.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662097060.616370_397.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:37:40.616370Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:37:41.177 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:41.178 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662097060.616370_397.1', 3: ''} 2022-09-02T05:37:41.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:37:41.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:41.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:37:41.180 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:37:41.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:41.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:41.291 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:37:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2022-09-02T05:37:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2022-09-02T05:37:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2022-09-02T05:37:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2022-09-02T05:37:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2022-09-02T05:37:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2022-09-02T05:37:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2022-09-02T05:37:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2022-09-02T05:37:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2022-09-02T05:37:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2022-09-02T05:37:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2022-09-02T05:37:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006" 2022-09-02T05:37:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:37:41.295 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:37:41.297 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.298 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.299 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.300 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.301 DEBUG:boto:Data: 2022-09-02T05:37:41.302 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.304 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.305 DEBUG:boto:Params: {} 2022-09-02T05:37:41.306 DEBUG:boto:Token: None 2022-09-02T05:37:41.306 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030ae8d5764c95706-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.312 DEBUG:boto:path=/xljbkf-73/obj00000000000000000006 2022-09-02T05:37:41.313 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000006 2022-09-02T05:37:41.313 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.314 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000006 2022-09-02T05:37:41.314 DEBUG:boto:Data: 2022-09-02T05:37:41.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.316 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.317 DEBUG:boto:Params: {} 2022-09-02T05:37:41.318 DEBUG:boto:Token: None 2022-09-02T05:37:41.319 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000006 2022-09-02T05:37:41.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7EpfBd1XxaydMqG96YBmq7wKoB8= 2022-09-02T05:37:41.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7EpfBd1XxaydMqG96YBmq7wKoB8='} 2022-09-02T05:37:41.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000875576cfda856d5-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.371 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.372 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.372 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.373 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.374 DEBUG:boto:Data: 2022-09-02T05:37:41.374 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.375 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.376 DEBUG:boto:Params: {} 2022-09-02T05:37:41.377 DEBUG:boto:Token: None 2022-09-02T05:37:41.377 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b46cf553e90bac8a-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.382 DEBUG:boto:path=/xljbkf-73/obj00000000000000000012 2022-09-02T05:37:41.383 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000012 2022-09-02T05:37:41.383 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.384 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000012 2022-09-02T05:37:41.384 DEBUG:boto:Data: 2022-09-02T05:37:41.385 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.386 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.386 DEBUG:boto:Params: {} 2022-09-02T05:37:41.387 DEBUG:boto:Token: None 2022-09-02T05:37:41.387 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000012 2022-09-02T05:37:41.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HdOw9gVul0CDbdZaywTwOfEzZLk= 2022-09-02T05:37:41.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HdOw9gVul0CDbdZaywTwOfEzZLk='} 2022-09-02T05:37:41.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c894a3060a69207-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.399 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.399 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.400 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.400 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.401 DEBUG:boto:Data: 2022-09-02T05:37:41.401 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.402 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.403 DEBUG:boto:Params: {} 2022-09-02T05:37:41.403 DEBUG:boto:Token: None 2022-09-02T05:37:41.404 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000253ce4caecaead2c-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.408 DEBUG:boto:path=/xljbkf-73/obj00000000000000000002 2022-09-02T05:37:41.409 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000002 2022-09-02T05:37:41.409 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.410 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000002 2022-09-02T05:37:41.410 DEBUG:boto:Data: 2022-09-02T05:37:41.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.412 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.413 DEBUG:boto:Params: {} 2022-09-02T05:37:41.413 DEBUG:boto:Token: None 2022-09-02T05:37:41.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000002 2022-09-02T05:37:41.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wl+J2cOYMdcKiTAGtLGknZhpgjo= 2022-09-02T05:37:41.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wl+J2cOYMdcKiTAGtLGknZhpgjo='} 2022-09-02T05:37:41.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053b9fa655ee5ae51-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.428 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.428 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.429 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.429 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.430 DEBUG:boto:Data: 2022-09-02T05:37:41.430 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.431 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.432 DEBUG:boto:Params: {} 2022-09-02T05:37:41.432 DEBUG:boto:Token: None 2022-09-02T05:37:41.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fe87b1ad0331b3e-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.438 DEBUG:boto:path=/xljbkf-73/obj00000000000000000005 2022-09-02T05:37:41.438 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000005 2022-09-02T05:37:41.439 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.439 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000005 2022-09-02T05:37:41.440 DEBUG:boto:Data: 2022-09-02T05:37:41.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.441 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.442 DEBUG:boto:Params: {} 2022-09-02T05:37:41.442 DEBUG:boto:Token: None 2022-09-02T05:37:41.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000005 2022-09-02T05:37:41.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iq13lMF9lU66zY4XLx8X0t9VkiQ= 2022-09-02T05:37:41.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iq13lMF9lU66zY4XLx8X0t9VkiQ='} 2022-09-02T05:37:41.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027c0ea35359d7e64-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.491 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.492 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.492 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.493 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.493 DEBUG:boto:Data: 2022-09-02T05:37:41.494 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.495 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.495 DEBUG:boto:Params: {} 2022-09-02T05:37:41.496 DEBUG:boto:Token: None 2022-09-02T05:37:41.496 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003856765d07908af7-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.500 DEBUG:boto:path=/xljbkf-73/obj00000000000000000004 2022-09-02T05:37:41.501 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000004 2022-09-02T05:37:41.501 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.502 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000004 2022-09-02T05:37:41.502 DEBUG:boto:Data: 2022-09-02T05:37:41.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.504 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.504 DEBUG:boto:Params: {} 2022-09-02T05:37:41.505 DEBUG:boto:Token: None 2022-09-02T05:37:41.506 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000004 2022-09-02T05:37:41.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Bo9MMrieroUP3qIHB4nTaZa+u0= 2022-09-02T05:37:41.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Bo9MMrieroUP3qIHB4nTaZa+u0='} 2022-09-02T05:37:41.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000098e65cda141e15a-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.519 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.520 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.520 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.521 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.521 DEBUG:boto:Data: 2022-09-02T05:37:41.522 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.523 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.523 DEBUG:boto:Params: {} 2022-09-02T05:37:41.524 DEBUG:boto:Token: None 2022-09-02T05:37:41.524 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025188ed0ca2ea31d-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.528 DEBUG:boto:path=/xljbkf-73/obj00000000000000000011 2022-09-02T05:37:41.529 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000011 2022-09-02T05:37:41.529 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.530 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000011 2022-09-02T05:37:41.530 DEBUG:boto:Data: 2022-09-02T05:37:41.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.532 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.532 DEBUG:boto:Params: {} 2022-09-02T05:37:41.533 DEBUG:boto:Token: None 2022-09-02T05:37:41.534 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000011 2022-09-02T05:37:41.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G5c4cNU2vh1WtU2w1eY9/9VkwbI= 2022-09-02T05:37:41.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G5c4cNU2vh1WtU2w1eY9/9VkwbI='} 2022-09-02T05:37:41.543 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fee670ce9a5a606f-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.543 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.544 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.544 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.545 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.545 DEBUG:boto:Data: 2022-09-02T05:37:41.546 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.547 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.547 DEBUG:boto:Params: {} 2022-09-02T05:37:41.548 DEBUG:boto:Token: None 2022-09-02T05:37:41.548 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ac03233b53f8c88-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.553 DEBUG:boto:path=/xljbkf-73/obj00000000000000000013 2022-09-02T05:37:41.553 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000013 2022-09-02T05:37:41.554 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.554 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000013 2022-09-02T05:37:41.555 DEBUG:boto:Data: 2022-09-02T05:37:41.555 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.556 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.557 DEBUG:boto:Params: {} 2022-09-02T05:37:41.557 DEBUG:boto:Token: None 2022-09-02T05:37:41.558 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000013 2022-09-02T05:37:41.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGfab/4VKzBdJhMah+5KQwTY/8o= 2022-09-02T05:37:41.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGfab/4VKzBdJhMah+5KQwTY/8o='} 2022-09-02T05:37:41.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d027eea79aae18eb-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.603 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.604 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.604 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.605 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.605 DEBUG:boto:Data: 2022-09-02T05:37:41.606 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.607 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.607 DEBUG:boto:Params: {} 2022-09-02T05:37:41.608 DEBUG:boto:Token: None 2022-09-02T05:37:41.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b805d2122c36e0b-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.613 DEBUG:boto:path=/xljbkf-73/obj00000000000000000001 2022-09-02T05:37:41.614 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000001 2022-09-02T05:37:41.615 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.615 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000001 2022-09-02T05:37:41.616 DEBUG:boto:Data: 2022-09-02T05:37:41.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.617 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.618 DEBUG:boto:Params: {} 2022-09-02T05:37:41.618 DEBUG:boto:Token: None 2022-09-02T05:37:41.619 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000001 2022-09-02T05:37:41.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sd2FWxrYGJZSk4OWVKFd8MbRwoc= 2022-09-02T05:37:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sd2FWxrYGJZSk4OWVKFd8MbRwoc='} 2022-09-02T05:37:41.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea9934b93647ab3f-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.630 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.631 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.631 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.632 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.632 DEBUG:boto:Data: 2022-09-02T05:37:41.633 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.634 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.634 DEBUG:boto:Params: {} 2022-09-02T05:37:41.635 DEBUG:boto:Token: None 2022-09-02T05:37:41.635 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090182bcc8568adf5-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.640 DEBUG:boto:path=/xljbkf-73/obj00000000000000000003 2022-09-02T05:37:41.640 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000003 2022-09-02T05:37:41.641 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.641 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000003 2022-09-02T05:37:41.642 DEBUG:boto:Data: 2022-09-02T05:37:41.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.643 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.644 DEBUG:boto:Params: {} 2022-09-02T05:37:41.645 DEBUG:boto:Token: None 2022-09-02T05:37:41.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000003 2022-09-02T05:37:41.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ra5TaaI6hqt64rDy9S9Rn3vENsI= 2022-09-02T05:37:41.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ra5TaaI6hqt64rDy9S9Rn3vENsI='} 2022-09-02T05:37:41.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033d3e5e3e9e4d85d-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.695 DEBUG:boto:path=/xljbkf-73/ 2022-09-02T05:37:41.696 DEBUG:boto:auth_path=/xljbkf-73/ 2022-09-02T05:37:41.697 DEBUG:boto:Method: HEAD 2022-09-02T05:37:41.697 DEBUG:boto:Path: /xljbkf-73/ 2022-09-02T05:37:41.698 DEBUG:boto:Data: 2022-09-02T05:37:41.698 DEBUG:boto:Headers: {} 2022-09-02T05:37:41.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.699 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.700 DEBUG:boto:Params: {} 2022-09-02T05:37:41.700 DEBUG:boto:Token: None 2022-09-02T05:37:41.701 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/ 2022-09-02T05:37:41.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ= 2022-09-02T05:37:41.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFRO7DjnGp+yN6wKJ2TvF/AU1QQ=', 'Content-Length': '0'} 2022-09-02T05:37:41.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a82151fd2698e3c7-00631196a5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.706 DEBUG:boto:path=/xljbkf-73/obj00000000000000000010 2022-09-02T05:37:41.707 DEBUG:boto:auth_path=/xljbkf-73/obj00000000000000000010 2022-09-02T05:37:41.707 DEBUG:boto:Method: PUT 2022-09-02T05:37:41.708 DEBUG:boto:Path: /xljbkf-73/obj00000000000000000010 2022-09-02T05:37:41.709 DEBUG:boto:Data: 2022-09-02T05:37:41.709 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:37:41.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:37:41.710 DEBUG:boto:Port: 8000 2022-09-02T05:37:41.711 DEBUG:boto:Params: {} 2022-09-02T05:37:41.711 DEBUG:boto:Token: None 2022-09-02T05:37:41.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:37:41 GMT /xljbkf-73/obj00000000000000000010 2022-09-02T05:37:41.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6zXG9Xb07dbLJxPwSDcX64TIqx8= 2022-09-02T05:37:41.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:37:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6zXG9Xb07dbLJxPwSDcX64TIqx8='} 2022-09-02T05:37:41.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b837827ba7e9807-00631196a5-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:37:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:37:41.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-73 2022-09-02T05:37:41.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:37:41.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:37:41.723 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:37:41.779 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:37:41.780 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:37:41.815 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:41.816+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:37:46.817 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:46.817+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:37:51.817 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:51.818+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:37:56.817 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:37:56.819+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:01.818 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:01.819+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:06.819 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:06.820+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:11.819 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:11.820+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:16.820 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:16.822+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:21.821 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:21.823+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:26.821 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:26.823+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:31.822 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:31.823+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:36.822 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:36.824+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:41.823 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:41.824+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:46.824 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:46.825+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:51.825 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:51.825+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:38:56.826 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:38:56.826+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:39:01.826 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:01.827+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:39:06.827 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:06.828+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:39:11.827 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:11.828+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:39:16.828 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:16.829+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:39:21.829 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:21.830+0000 7faaaf761600 1 waiting to reach incremental sync.. 2022-09-02T05:39:26.830 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:26.832+0000 7faaaf761600 1 bucket sync caught up with source: 2022-09-02T05:39:26.831 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:39:26.831 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:39:26.832 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:26.832+0000 7faaaf761600 0 bucket checkpoint complete 2022-09-02T05:39:26.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-73 2022-09-02T05:39:26.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:39:26.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:39:26.853 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:39:26.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:39:26.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:39:26.958 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:26.959+0000 7f8b1b897600 1 waiting to reach incremental sync.. 2022-09-02T05:39:31.959 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:31.960+0000 7f8b1b897600 1 waiting to reach incremental sync.. 2022-09-02T05:39:36.960 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:36.961+0000 7f8b1b897600 1 waiting to reach incremental sync.. 2022-09-02T05:39:41.961 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:41.961+0000 7f8b1b897600 1 waiting to reach incremental sync.. 2022-09-02T05:39:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:46.962+0000 7f8b1b897600 1 waiting to reach incremental sync.. 2022-09-02T05:39:51.963 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:51.964+0000 7f8b1b897600 1 bucket sync caught up with source: 2022-09-02T05:39:51.964 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:39:51.965 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:39:51.965 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:39:51.964+0000 7f8b1b897600 0 bucket checkpoint complete 2022-09-02T05:39:51.990 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-73 2022-09-02T05:39:51.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:39:51.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:39:51.993 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:39:52.057 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:39:52.058 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:39:52.096 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:52.096+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:39:52.096 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:39:52.097 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:39:57.098 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:39:57.098+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:39:57.098 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:39:57.099 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:02.099 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:02.100+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:02.100 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:02.100 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:07.101 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:07.101+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:07.102 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:07.102 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:12.102 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:12.103+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:12.103 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:12.103 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:17.104 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:17.104+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:17.104 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:17.104 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:22.105 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:22.105+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:22.106 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:22.107 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:27.106 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:27.107+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:27.107 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:27.107 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:32.108 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:32.109+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:32.109 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:32.109 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:37.109 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:37.110+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:37.110 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:37.110 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:42.112 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:42.111+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:42.113 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:42.113 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:47.112 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:47.112+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:47.112 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:47.112 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:52.114 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:52.114+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:52.115 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:52.115 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:40:57.114 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:40:57.115+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:40:57.115 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:40:57.115 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:02.116 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:02.116+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:02.117 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:02.118 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:07.118 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:07.118+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:07.119 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:07.119 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:12.119 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:12.120+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:12.120 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:12.121 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:17.121+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:17.122 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:17.122 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:22.123 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:22.122+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:22.123 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:22.124 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:27.124 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:27.124+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:27.125 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:27.125 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:32.125 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:32.125+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:32.126 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:32.126 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:37.127 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:37.128+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:37.127 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:37.128 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:42.128 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:42.129+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:42.128 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:42.129 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:47.129 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:47.130+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:47.130 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:47.130 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:52.131 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:52.131+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:52.131 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:52.132 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:41:57.132 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:41:57.132+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:41:57.133 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:41:57.133 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:02.134 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:02.134+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:02.134 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:02.135 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:07.136 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:07.136+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:07.136 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:07.136 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:12.137 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:12.137+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:12.138 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:12.138 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:17.138 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:17.138+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:17.139 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:17.139 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:22.140 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:22.140+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:22.140 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:22.141 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:27.141 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:27.141+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:27.142 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:27.143 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:32.143 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:32.143+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:32.143 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:32.144 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:37.144 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:37.144+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:37.144 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:37.145 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:42.145 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:42.146+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:42.146 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:42.146 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:47.147 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:47.148+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:47.147 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:47.148 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:52.148 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:52.148+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:52.149 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:52.149 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:42:57.150 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:42:57.150+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:42:57.150 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:42:57.151 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:02.151 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:02.151+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:02.152 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:02.152 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:07.153 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:07.153+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:07.153 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:07.153 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:12.154 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:12.154+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:12.155 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:12.155 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:17.156 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:17.155+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:17.156 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:17.156 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:22.158 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:22.157+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:22.158 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:22.159 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:27.159 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:27.158+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:27.159 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:27.159 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:32.160 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:32.160+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:32.161 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:32.162 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:37.162 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:37.161+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:37.163 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:37.163 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:42.163 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:42.162+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:42.163 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:42.163 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:47.164 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:47.165+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:47.164 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:47.165 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:52.166 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:52.166+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:52.166 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:52.167 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:43:57.167 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:43:57.167+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:43:57.168 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:43:57.168 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:02.169 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:02.168+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:02.169 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:02.171 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:07.170 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:07.169+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:07.171 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:07.171 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:12.171 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:12.171+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:12.172 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:12.172 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:17.173 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:17.172+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:17.173 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:17.174 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:22.174 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:22.174+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:22.175 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:22.175 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:27.175 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:27.175+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:27.176 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:27.176 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:32.177 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:32.177+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:32.178 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:32.178 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:37.178 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:37.178+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:37.179 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:37.179 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:42.180 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:42.180+0000 7f26c4515600 1 waiting for incremental sync to catch up: 2022-09-02T05:44:42.180 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, , 00000000001.49.5] 2022-09-02T05:44:42.181 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.51.5, 00000000001.46.5, 00000000001.59.5, 00000000001.24.5, 00000000001.69.5, , 00000000001.31.5, 00000000001.46.5, 00000000001.61.5, 00000000001.29.5, 00000000001.49.5] 2022-09-02T05:44:47.182 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.181+0000 7f26c4515600 0 bucket checkpoint timed out waiting for incremental sync to catch up 2022-09-02T05:44:47.182 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.181+0000 7f26c4515600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:44:47.183 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.181+0000 7f26c4515600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T05:44:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T05:44:47.207 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2022-09-02T05:44:47.210 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:44:47.211 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:44:47.212 DEBUG:boto:Method: PUT 2022-09-02T05:44:47.213 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:44:47.214 DEBUG:boto:Data: 2022-09-02T05:44:47.215 DEBUG:boto:Headers: {} 2022-09-02T05:44:47.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.217 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.217 DEBUG:boto:Params: {} 2022-09-02T05:44:47.218 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:44:47.219 DEBUG:boto:Token: None 2022-09-02T05:44:47.220 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/ 2022-09-02T05:44:47.221 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aWvOcBOHZ+9TyjbKqgzbwOvguLc= 2022-09-02T05:44:47.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aWvOcBOHZ+9TyjbKqgzbwOvguLc=', 'Content-Length': '0'} 2022-09-02T05:44:47.253 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023aa5ab57ba9c0d8-006311984f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.254 DEBUG:rgw_multi.tests:created bucket=xljbkf-74 2022-09-02T05:44:47.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.257 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:44:47.313 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:44:47.314 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:44:47.353 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T05:44:47.367 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T05:44:47.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.368 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:44:47.368 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:44:47.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.369 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:44:47.370 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:44:47.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.371 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662097487.249007_403.1", 2022-09-02T05:44:47.371 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T05:44:47.249007Z" 2022-09-02T05:44:47.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.372 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:44:47.372 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T05:44:47.372 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:44:47.373 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T05:44:47.374 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T05:44:47.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.379 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:44:47.434 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:44:47.435 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:44:47.481 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.481+0000 7f93c0c00600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:44:47.482 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.482+0000 7f93c0c00600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:44:47.506 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:44:47.506 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:44:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:44:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:44:47.507 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:44:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:44:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:44:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.508 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:44:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:44:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.509 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:44:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.510 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:44:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:44:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:44:47.511 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:44:47.512 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:44:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:44:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:44:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:44:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:44:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.516 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.516 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662097487.249007_403.1", 2022-09-02T05:44:47.516 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.516 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:44:47.517 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.517 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:44:47.249007Z", 2022-09-02T05:44:47.517 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:44:47.518 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.518 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:44:47.519 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.519 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.519 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:44:47.519 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.520 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:44:47.520 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.520 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:44:47.521 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:44:47.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.521 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:44:47.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.522 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:44:47.522 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:44:47.522 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:44:47.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.523 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:44:47Z" 2022-09-02T05:44:47.523 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:44:47.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662097487.249007_403.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:44:47.249007Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:44:47Z" } 2022-09-02T05:44:47.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662097487.249007_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:44:47.249007Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:44:47.526 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.526 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T05:44:47.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T05:44:47.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:44:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:44:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:44:47.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:44:47.631+0000 7fa295f4c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:44:47.632 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:44:47.632+0000 7fa295f4c600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:44:47.652 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T05:44:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T05:44:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T05:44:47.653 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T05:44:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T05:44:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:44:47.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:44:47.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.655 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T05:44:47.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.655 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T05:44:47.656 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:44:47.656 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:44:47.656 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:44:47.656 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:44:47.657 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T05:44:47.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:44:47.657 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:44:47.657 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:44:47.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:44:47.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.658 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T05:44:47.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:44:47.659 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:44:47.659 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T05:44:47.659 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:44:47.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:44:47.660 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:44:47.660 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2022-09-02T05:44:47.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:44:47.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:44:47.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.661 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T05:44:47.662 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:44:47.662 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:44:47.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662097487.249007_403.1", 2022-09-02T05:44:47.662 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:44:47.663 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:44:47.663 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:44:47.663 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T05:44:47.249007Z", 2022-09-02T05:44:47.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T05:44:47.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:44:47.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T05:44:47.664 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T05:44:47.665 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T05:44:47.665 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T05:44:47.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:44:47.665 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T05:44:47.666 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T05:44:47.666 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T05:44:47.666 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:44:47.666 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T05:44:47.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:44:47.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T05:44:47.667 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T05:44:47.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.668 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T05:44:47.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T05:44:47.668 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T05:44:47.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T05:44:47.669 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T05:44:47Z" 2022-09-02T05:44:47.669 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T05:44:47.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662097487.249007_403.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:44:47.249007Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:44:47Z" } 2022-09-02T05:44:47.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662097487.249007_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:44:47.249007Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:44:47.672 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.672 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T05:44:47.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T05:44:47.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:44:47.675 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:44:47.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:44:47.725 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:44:47.760 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.760+0000 7f2c87b95600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:44:47.761 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:47.761+0000 7f2c87b95600 20 RGW-SYNC:meta: read sync status 2022-09-02T05:44:47.776 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T05:44:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T05:44:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T05:44:47.777 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T05:44:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T05:44:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T05:44:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:44:47.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T05:44:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T05:44:47.779 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T05:44:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.780 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T05:44:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T05:44:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:44:47.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T05:44:47.782 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T05:44:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:44:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T05:44:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:44:47.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T05:44:47.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662097487.249007_403.1", 2022-09-02T05:44:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:44:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:44:47.249007Z", 2022-09-02T05:44:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T05:44:47.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:44:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T05:44:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T05:44:47.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T05:44:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T05:44:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T05:44:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T05:44:47.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T05:44:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T05:44:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T05:44:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:44:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T05:44:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T05:44:47.791 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T05:44:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T05:44:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:44:47.792 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T05:44:47Z" 2022-09-02T05:44:47.792 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T05:44:47.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662097487.249007_403.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:44:47.249007Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T05:44:47Z" } 2022-09-02T05:44:47.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662097487.249007_403.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:44:47.249007Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T05:44:47.795 DEBUG:rgw_multi.tests:log_status={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.795 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662097487.249007_403.1', 3: ''} 2022-09-02T05:44:47.796 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T05:44:47.796 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:44:47.797 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:44:47.797 DEBUG:boto:Method: HEAD 2022-09-02T05:44:47.798 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:44:47.798 DEBUG:boto:Data: 2022-09-02T05:44:47.799 DEBUG:boto:Headers: {} 2022-09-02T05:44:47.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.800 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.800 DEBUG:boto:Params: {} 2022-09-02T05:44:47.801 DEBUG:boto:Token: None 2022-09-02T05:44:47.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/ 2022-09-02T05:44:47.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg= 2022-09-02T05:44:47.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg=', 'Content-Length': '0'} 2022-09-02T05:44:47.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d28d75bb4046f48-006311984f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.806 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:44:47.806 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:44:47.807 DEBUG:boto:Method: PUT 2022-09-02T05:44:47.807 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:44:47.808 DEBUG:boto:Data: 2022-09-02T05:44:47.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:44:47.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.809 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.810 DEBUG:boto:Params: {} 2022-09-02T05:44:47.810 DEBUG:boto:Token: None 2022-09-02T05:44:47.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/obj11 2022-09-02T05:44:47.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BOjwpAOLE4c7aB2/LYyqgbg5/os= 2022-09-02T05:44:47.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BOjwpAOLE4c7aB2/LYyqgbg5/os='} 2022-09-02T05:44:47.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050caa2f38bf70e5a-006311984f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.824 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:44:47.825 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:44:47.826 DEBUG:boto:Method: HEAD 2022-09-02T05:44:47.826 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:44:47.827 DEBUG:boto:Data: 2022-09-02T05:44:47.827 DEBUG:boto:Headers: {} 2022-09-02T05:44:47.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.829 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.829 DEBUG:boto:Params: {} 2022-09-02T05:44:47.830 DEBUG:boto:Token: None 2022-09-02T05:44:47.830 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/ 2022-09-02T05:44:47.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg= 2022-09-02T05:44:47.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg=', 'Content-Length': '0'} 2022-09-02T05:44:47.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7398d674a014d7d-006311984f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.835 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:44:47.835 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:44:47.836 DEBUG:boto:Method: PUT 2022-09-02T05:44:47.836 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:44:47.837 DEBUG:boto:Data: 2022-09-02T05:44:47.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:44:47.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.838 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.839 DEBUG:boto:Params: {} 2022-09-02T05:44:47.839 DEBUG:boto:Token: None 2022-09-02T05:44:47.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/obj22 2022-09-02T05:44:47.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3aLZTAZIFnqiLH7Za/b0xvfTYE4= 2022-09-02T05:44:47.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3aLZTAZIFnqiLH7Za/b0xvfTYE4='} 2022-09-02T05:44:47.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036ed9e6e78c31314-006311984f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.848 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:44:47.849 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:44:47.849 DEBUG:boto:Method: HEAD 2022-09-02T05:44:47.850 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:44:47.850 DEBUG:boto:Data: 2022-09-02T05:44:47.851 DEBUG:boto:Headers: {} 2022-09-02T05:44:47.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.852 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.852 DEBUG:boto:Params: {} 2022-09-02T05:44:47.853 DEBUG:boto:Token: None 2022-09-02T05:44:47.853 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/ 2022-09-02T05:44:47.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg= 2022-09-02T05:44:47.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg=', 'Content-Length': '0'} 2022-09-02T05:44:47.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afa65fe5e92d2bd8-006311984f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.857 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:44:47.858 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:44:47.858 DEBUG:boto:Method: PUT 2022-09-02T05:44:47.859 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:44:47.859 DEBUG:boto:Data: 2022-09-02T05:44:47.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:44:47.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.861 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.861 DEBUG:boto:Params: {} 2022-09-02T05:44:47.862 DEBUG:boto:Token: None 2022-09-02T05:44:47.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/obj33 2022-09-02T05:44:47.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nc1yn/PZaYWs0DEUJBE+CxM3rvg= 2022-09-02T05:44:47.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nc1yn/PZaYWs0DEUJBE+CxM3rvg='} 2022-09-02T05:44:47.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007509ad3ec8751181-006311984f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.907 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:44:47.908 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:44:47.908 DEBUG:boto:Method: HEAD 2022-09-02T05:44:47.909 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:44:47.909 DEBUG:boto:Data: 2022-09-02T05:44:47.910 DEBUG:boto:Headers: {} 2022-09-02T05:44:47.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.911 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.911 DEBUG:boto:Params: {} 2022-09-02T05:44:47.912 DEBUG:boto:Token: None 2022-09-02T05:44:47.912 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/ 2022-09-02T05:44:47.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg= 2022-09-02T05:44:47.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg=', 'Content-Length': '0'} 2022-09-02T05:44:47.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009639445cba8bbf84-006311984f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.917 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:44:47.917 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:44:47.918 DEBUG:boto:Method: PUT 2022-09-02T05:44:47.918 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:44:47.919 DEBUG:boto:Data: 2022-09-02T05:44:47.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:44:47.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.920 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.921 DEBUG:boto:Params: {} 2022-09-02T05:44:47.922 DEBUG:boto:Token: None 2022-09-02T05:44:47.922 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/obj44 2022-09-02T05:44:47.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTWDzU2VB4ukmjUXrXtHffQs4Ss= 2022-09-02T05:44:47.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTWDzU2VB4ukmjUXrXtHffQs4Ss='} 2022-09-02T05:44:47.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003be7edb97fb164dd-006311984f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.935 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:44:47.936 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:44:47.936 DEBUG:boto:Method: HEAD 2022-09-02T05:44:47.937 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:44:47.937 DEBUG:boto:Data: 2022-09-02T05:44:47.938 DEBUG:boto:Headers: {} 2022-09-02T05:44:47.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.939 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.939 DEBUG:boto:Params: {} 2022-09-02T05:44:47.940 DEBUG:boto:Token: None 2022-09-02T05:44:47.940 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/ 2022-09-02T05:44:47.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg= 2022-09-02T05:44:47.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8wramdpvH+qqmqLuF+SDEGPQLwg=', 'Content-Length': '0'} 2022-09-02T05:44:47.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db1ca76d7ff8df64-006311984f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.945 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:44:47.945 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:44:47.946 DEBUG:boto:Method: PUT 2022-09-02T05:44:47.946 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:44:47.947 DEBUG:boto:Data: 2022-09-02T05:44:47.948 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:44:47.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:44:47.949 DEBUG:boto:Port: 8000 2022-09-02T05:44:47.949 DEBUG:boto:Params: {} 2022-09-02T05:44:47.950 DEBUG:boto:Token: None 2022-09-02T05:44:47.950 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:44:47 GMT /xljbkf-74/obj55 2022-09-02T05:44:47.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:14y1wsqYotFbXoWg+WlhI5lG4+A= 2022-09-02T05:44:47.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:44:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:14y1wsqYotFbXoWg+WlhI5lG4+A='} 2022-09-02T05:44:47.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee4891450354137f-006311984f-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:44:47.958 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:44:47.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:44:47.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:44:47.960 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:44:48.017 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:44:48.017 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:44:48.074 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:48.075+0000 7fbcadbb0600 1 waiting to reach incremental sync.. 2022-09-02T05:44:53.075 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:53.075+0000 7fbcadbb0600 1 waiting to reach incremental sync.. 2022-09-02T05:44:58.075 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:44:58.076+0000 7fbcadbb0600 1 waiting to reach incremental sync.. 2022-09-02T05:45:03.076 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:03.075+0000 7fbcadbb0600 1 waiting to reach incremental sync.. 2022-09-02T05:45:08.077 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:08.077+0000 7fbcadbb0600 1 bucket sync caught up with source: 2022-09-02T05:45:08.078 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:08.078 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:08.079 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:08.077+0000 7fbcadbb0600 0 bucket checkpoint complete 2022-09-02T05:45:08.096 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:45:08.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:08.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:08.098 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:08.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:08.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:08.215 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:08.215+0000 7f5bf4118600 1 waiting to reach incremental sync.. 2022-09-02T05:45:13.217 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:13.217+0000 7f5bf4118600 1 bucket sync caught up with source: 2022-09-02T05:45:13.218 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:13.218 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:13.217+0000 7f5bf4118600 0 bucket checkpoint complete 2022-09-02T05:45:13.240 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:45:13.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:13.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:13.242 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:13.305 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:13.306 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:13.343 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:13.343+0000 7fcdd2a3a600 1 waiting for incremental sync to catch up: 2022-09-02T05:45:13.344 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.42.5, , 00000000002.42.5] 2022-09-02T05:45:13.344 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:18.345 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:18.344+0000 7fcdd2a3a600 1 waiting for incremental sync to catch up: 2022-09-02T05:45:18.346 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.42.5, , 00000000002.42.5] 2022-09-02T05:45:18.346 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:23.346 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:23.346+0000 7fcdd2a3a600 1 bucket sync caught up with source: 2022-09-02T05:45:23.347 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:23.347 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.42.5, 00000000002.58.5, 00000000002.42.5] 2022-09-02T05:45:23.348 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:23.346+0000 7fcdd2a3a600 0 bucket checkpoint complete 2022-09-02T05:45:23.369 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:45:23.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.372 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:23.436 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:23.437 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:23.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:23.487+0000 7efc80a85600 1 bucket sync caught up with source: 2022-09-02T05:45:23.488 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.538.5, 00000000002.1192.5, 00000000002.823.5] 2022-09-02T05:45:23.488 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.538.5, 00000000002.1192.5, 00000000002.823.5] 2022-09-02T05:45:23.488 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:23.487+0000 7efc80a85600 0 bucket checkpoint complete 2022-09-02T05:45:23.504 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:45:23.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.506 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:23.622 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:23.622+0000 7f1c548f3600 1 bucket sync caught up with source: 2022-09-02T05:45:23.623 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.538.5, 00000000002.1192.5, 00000000002.823.5] 2022-09-02T05:45:23.623 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.538.5, 00000000002.1192.5, 00000000002.823.5] 2022-09-02T05:45:23.623 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:23.622+0000 7f1c548f3600 0 bucket checkpoint complete 2022-09-02T05:45:23.647 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:45:23.648 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:45:23.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.650 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:23.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:23.706 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:23.752+0000 7fcb5bf1e600 1 bucket sync caught up with source: 2022-09-02T05:45:23.752 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.193.5, 00000000002.321.5, 00000000002.510.5] 2022-09-02T05:45:23.753 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.193.5, 00000000002.321.5, 00000000002.510.5] 2022-09-02T05:45:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:45:23.752+0000 7fcb5bf1e600 0 bucket checkpoint complete 2022-09-02T05:45:23.766 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:45:23.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:23.767 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:23.833 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:23.834 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:23.891 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:23.891+0000 7fc1463b5600 1 bucket sync caught up with source: 2022-09-02T05:45:23.892 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000001.193.5, 00000000002.321.5, 00000000002.510.5] 2022-09-02T05:45:23.892 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , , , 00000000001.193.5, 00000000002.321.5, 00000000002.510.5] 2022-09-02T05:45:23.892 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:23.891+0000 7fc1463b5600 0 bucket checkpoint complete 2022-09-02T05:45:23.915 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:45:23.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:45:23.916 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:23.917 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:23.917 DEBUG:boto:Method: HEAD 2022-09-02T05:45:23.918 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:23.918 DEBUG:boto:Data: 2022-09-02T05:45:23.919 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:23.920 DEBUG:boto:Port: 8000 2022-09-02T05:45:23.920 DEBUG:boto:Params: {} 2022-09-02T05:45:23.921 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:45:23.922 DEBUG:boto:Token: None 2022-09-02T05:45:23.923 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:23 GMT /xljbkf-74/ 2022-09-02T05:45:23.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ES736hoc2mjVeRIm2ieNu5dYO7M= 2022-09-02T05:45:23.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ES736hoc2mjVeRIm2ieNu5dYO7M=', 'Content-Length': '0'} 2022-09-02T05:45:23.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087ef5b4946e91513-0063119873-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:23.928 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:23.929 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:23.929 DEBUG:boto:Method: HEAD 2022-09-02T05:45:23.930 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:23.930 DEBUG:boto:Data: 2022-09-02T05:45:23.931 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:23.932 DEBUG:boto:Port: 8000 2022-09-02T05:45:23.933 DEBUG:boto:Params: {} 2022-09-02T05:45:23.933 DEBUG:boto:Token: None 2022-09-02T05:45:23.934 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:23 GMT /xljbkf-74/ 2022-09-02T05:45:23.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ES736hoc2mjVeRIm2ieNu5dYO7M= 2022-09-02T05:45:23.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ES736hoc2mjVeRIm2ieNu5dYO7M=', 'Content-Length': '0'} 2022-09-02T05:45:23.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c296ed39fb0898da-0063119873-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:23.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:45:23.939 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:23.942 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:23.942 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.943 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.943 DEBUG:boto:Method: GET 2022-09-02T05:45:23.944 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.944 DEBUG:boto:Data: 2022-09-02T05:45:23.945 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.946 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:23.946 DEBUG:boto:Port: 8000 2022-09-02T05:45:23.947 DEBUG:boto:Params: {} 2022-09-02T05:45:23.947 DEBUG:boto:Token: None 2022-09-02T05:45:23.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:23 GMT /xljbkf-74/?versions 2022-09-02T05:45:23.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw= 2022-09-02T05:45:23.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw=', 'Content-Length': '0'} 2022-09-02T05:45:23.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ecf08c3597e3da-0063119873-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:23.952 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:23.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:45:23.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:45:23.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:45:23.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:45:23.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:45:23.956 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:45:23.957 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:23.958 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:23.958 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.959 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.959 DEBUG:boto:Method: GET 2022-09-02T05:45:23.960 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.960 DEBUG:boto:Data: 2022-09-02T05:45:23.961 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:23.962 DEBUG:boto:Port: 8000 2022-09-02T05:45:23.962 DEBUG:boto:Params: {} 2022-09-02T05:45:23.963 DEBUG:boto:Token: None 2022-09-02T05:45:23.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:23 GMT /xljbkf-74/?versions 2022-09-02T05:45:23.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw= 2022-09-02T05:45:23.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw=', 'Content-Length': '0'} 2022-09-02T05:45:23.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c69812a339a81c4-0063119873-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:23.967 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:23.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:45:23.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:45:23.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:45:23.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:45:23.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:45:23.971 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:23.971 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:23.972 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.973 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.973 DEBUG:boto:Method: GET 2022-09-02T05:45:23.974 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.974 DEBUG:boto:Data: 2022-09-02T05:45:23.975 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.975 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:23.976 DEBUG:boto:Port: 8000 2022-09-02T05:45:23.976 DEBUG:boto:Params: {} 2022-09-02T05:45:23.977 DEBUG:boto:Token: None 2022-09-02T05:45:23.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:23 GMT /xljbkf-74/?versions 2022-09-02T05:45:23.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw= 2022-09-02T05:45:23.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw=', 'Content-Length': '0'} 2022-09-02T05:45:23.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d23b68e32be103b-0063119873-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:23.981 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:23.982 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:23.983 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:23.983 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.984 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.985 DEBUG:boto:Method: GET 2022-09-02T05:45:23.985 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:23.986 DEBUG:boto:Data: 2022-09-02T05:45:23.986 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:23.987 DEBUG:boto:Port: 8000 2022-09-02T05:45:23.988 DEBUG:boto:Params: {} 2022-09-02T05:45:23.988 DEBUG:boto:Token: None 2022-09-02T05:45:23.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:23 GMT /xljbkf-74/?versions 2022-09-02T05:45:23.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw= 2022-09-02T05:45:23.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:23 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:25OG8fqqACCilWHaJ6urWDRpeJw=', 'Content-Length': '0'} 2022-09-02T05:45:23.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf5138f87ddeb95-0063119873-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:23 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:23.993 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:23.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:23.995 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:23.996 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:23.996 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:23.997 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:23.997 DEBUG:boto:Method: GET 2022-09-02T05:45:23.998 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:23.998 DEBUG:boto:Data: 2022-09-02T05:45:23.999 DEBUG:boto:Headers: {} 2022-09-02T05:45:23.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.000 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.000 DEBUG:boto:Params: {} 2022-09-02T05:45:24.001 DEBUG:boto:Token: None 2022-09-02T05:45:24.002 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A= 2022-09-02T05:45:24.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A=', 'Content-Length': '0'} 2022-09-02T05:45:24.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321e3baaf5467722-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.006 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.006 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.007 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.008 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.008 DEBUG:boto:Method: GET 2022-09-02T05:45:24.009 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.009 DEBUG:boto:Data: 2022-09-02T05:45:24.010 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.011 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.012 DEBUG:boto:Params: {} 2022-09-02T05:45:24.012 DEBUG:boto:Token: None 2022-09-02T05:45:24.013 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A= 2022-09-02T05:45:24.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A=', 'Content-Length': '0'} 2022-09-02T05:45:24.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca9b1365ebde55e-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.058 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.059 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.059 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.060 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.060 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.061 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.061 DEBUG:boto:Data: 2022-09-02T05:45:24.062 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.062 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.063 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.064 DEBUG:boto:Params: {} 2022-09-02T05:45:24.064 DEBUG:boto:Token: None 2022-09-02T05:45:24.065 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ilk+mLX+uEaPHT3LzcD+H+gbW0M= 2022-09-02T05:45:24.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ilk+mLX+uEaPHT3LzcD+H+gbW0M=', 'Content-Length': '0'} 2022-09-02T05:45:24.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee65b3237414b96e-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.069 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.069 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.070 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.070 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.071 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.072 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.072 DEBUG:boto:Data: 2022-09-02T05:45:24.073 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.074 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.075 DEBUG:boto:Params: {} 2022-09-02T05:45:24.075 DEBUG:boto:Token: None 2022-09-02T05:45:24.076 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ilk+mLX+uEaPHT3LzcD+H+gbW0M= 2022-09-02T05:45:24.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ilk+mLX+uEaPHT3LzcD+H+gbW0M=', 'Content-Length': '0'} 2022-09-02T05:45:24.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c401450e4eb77995-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:24.080 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.081 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.081 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.082 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.082 DEBUG:boto:Method: GET 2022-09-02T05:45:24.083 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.084 DEBUG:boto:Data: 2022-09-02T05:45:24.084 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.085 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.086 DEBUG:boto:Params: {} 2022-09-02T05:45:24.086 DEBUG:boto:Token: None 2022-09-02T05:45:24.087 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A= 2022-09-02T05:45:24.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A=', 'Content-Length': '0'} 2022-09-02T05:45:24.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e1533f0ebb8fc3-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.092 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.092 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.093 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.093 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.094 DEBUG:boto:Method: GET 2022-09-02T05:45:24.095 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.095 DEBUG:boto:Data: 2022-09-02T05:45:24.096 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.097 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.097 DEBUG:boto:Params: {} 2022-09-02T05:45:24.099 DEBUG:boto:Token: None 2022-09-02T05:45:24.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:24.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A= 2022-09-02T05:45:24.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G+GRiRVUkoRnPkMqmRcYxmj/79A=', 'Content-Length': '0'} 2022-09-02T05:45:24.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f70a80a7c8eb6e7-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.104 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.105 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.105 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.106 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.107 DEBUG:boto:Method: GET 2022-09-02T05:45:24.108 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.109 DEBUG:boto:Data: 2022-09-02T05:45:24.110 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.112 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.112 DEBUG:boto:Params: {} 2022-09-02T05:45:24.113 DEBUG:boto:Token: None 2022-09-02T05:45:24.114 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000314a3aa4bb78188d-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.119 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.120 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.122 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.123 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.123 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.124 DEBUG:boto:Method: GET 2022-09-02T05:45:24.125 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.126 DEBUG:boto:Data: 2022-09-02T05:45:24.127 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.128 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.128 DEBUG:boto:Params: {} 2022-09-02T05:45:24.129 DEBUG:boto:Token: None 2022-09-02T05:45:24.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e35d42e172323363-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.136 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.137 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.137 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.138 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.139 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:24.140 DEBUG:boto:Data: 2022-09-02T05:45:24.140 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.141 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.142 DEBUG:boto:Params: {} 2022-09-02T05:45:24.143 DEBUG:boto:Token: None 2022-09-02T05:45:24.144 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11 2022-09-02T05:45:24.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xRK5/KBzh33EEDbHI6XLsA/ag70= 2022-09-02T05:45:24.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xRK5/KBzh33EEDbHI6XLsA/ag70=', 'Content-Length': '0'} 2022-09-02T05:45:24.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046675a2abd051e98-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.148 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.148 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.149 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.149 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:24.150 DEBUG:boto:Data: 2022-09-02T05:45:24.150 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.151 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.152 DEBUG:boto:Params: {} 2022-09-02T05:45:24.153 DEBUG:boto:Token: None 2022-09-02T05:45:24.154 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11 2022-09-02T05:45:24.155 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xRK5/KBzh33EEDbHI6XLsA/ag70= 2022-09-02T05:45:24.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xRK5/KBzh33EEDbHI6XLsA/ag70=', 'Content-Length': '0'} 2022-09-02T05:45:24.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007296f6dc5ff7ef05-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:24.160 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.161 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.162 DEBUG:boto:Method: GET 2022-09-02T05:45:24.163 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:24.164 DEBUG:boto:Data: 2022-09-02T05:45:24.164 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.165 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.166 DEBUG:boto:Params: {} 2022-09-02T05:45:24.167 DEBUG:boto:Token: None 2022-09-02T05:45:24.168 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11 2022-09-02T05:45:24.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+m8v37si1BqWir4ubGeTIbwjHs= 2022-09-02T05:45:24.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+m8v37si1BqWir4ubGeTIbwjHs=', 'Content-Length': '0'} 2022-09-02T05:45:24.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9d3402f712671e-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.173 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:24.174 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:24.174 DEBUG:boto:Method: GET 2022-09-02T05:45:24.175 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:24.175 DEBUG:boto:Data: 2022-09-02T05:45:24.176 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.177 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.179 DEBUG:boto:Params: {} 2022-09-02T05:45:24.180 DEBUG:boto:Token: None 2022-09-02T05:45:24.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj11 2022-09-02T05:45:24.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+m8v37si1BqWir4ubGeTIbwjHs= 2022-09-02T05:45:24.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+m8v37si1BqWir4ubGeTIbwjHs=', 'Content-Length': '0'} 2022-09-02T05:45:24.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8696c4bc123c2f-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.226 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.227 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.228 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.228 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.229 DEBUG:boto:Method: GET 2022-09-02T05:45:24.229 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.230 DEBUG:boto:Data: 2022-09-02T05:45:24.230 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.232 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.232 DEBUG:boto:Params: {} 2022-09-02T05:45:24.233 DEBUG:boto:Token: None 2022-09-02T05:45:24.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c49bb9a685aa91-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.239 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.240 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.241 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.241 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.242 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.243 DEBUG:boto:Method: GET 2022-09-02T05:45:24.243 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:24.244 DEBUG:boto:Data: 2022-09-02T05:45:24.244 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.245 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.246 DEBUG:boto:Params: {} 2022-09-02T05:45:24.246 DEBUG:boto:Token: None 2022-09-02T05:45:24.247 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096f3d9ddeafbfa8d-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.252 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:24.253 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.254 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.254 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.255 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.255 DEBUG:boto:Method: GET 2022-09-02T05:45:24.256 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.256 DEBUG:boto:Data: 2022-09-02T05:45:24.257 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.258 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.258 DEBUG:boto:Params: {} 2022-09-02T05:45:24.259 DEBUG:boto:Token: None 2022-09-02T05:45:24.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE= 2022-09-02T05:45:24.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE=', 'Content-Length': '0'} 2022-09-02T05:45:24.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007137c0e941d6fc5f-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.306 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.307 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.307 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.308 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.308 DEBUG:boto:Method: GET 2022-09-02T05:45:24.309 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.310 DEBUG:boto:Data: 2022-09-02T05:45:24.310 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.311 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.312 DEBUG:boto:Params: {} 2022-09-02T05:45:24.312 DEBUG:boto:Token: None 2022-09-02T05:45:24.313 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE= 2022-09-02T05:45:24.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE=', 'Content-Length': '0'} 2022-09-02T05:45:24.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce5e6eb2f9f4fce9-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.317 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.318 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.318 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.319 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.319 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.320 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.320 DEBUG:boto:Data: 2022-09-02T05:45:24.321 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.322 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.322 DEBUG:boto:Params: {} 2022-09-02T05:45:24.323 DEBUG:boto:Token: None 2022-09-02T05:45:24.323 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2PeMmqz83jqESzaKU9uKfwQJY/I= 2022-09-02T05:45:24.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2PeMmqz83jqESzaKU9uKfwQJY/I=', 'Content-Length': '0'} 2022-09-02T05:45:24.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeebb0a9ab8b94b3-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.327 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.328 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.328 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.329 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.329 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.330 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.330 DEBUG:boto:Data: 2022-09-02T05:45:24.331 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.332 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.332 DEBUG:boto:Params: {} 2022-09-02T05:45:24.333 DEBUG:boto:Token: None 2022-09-02T05:45:24.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2PeMmqz83jqESzaKU9uKfwQJY/I= 2022-09-02T05:45:24.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2PeMmqz83jqESzaKU9uKfwQJY/I=', 'Content-Length': '0'} 2022-09-02T05:45:24.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bf3d8f9dc850496-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:24.338 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.338 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.339 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.339 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.340 DEBUG:boto:Method: GET 2022-09-02T05:45:24.340 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.341 DEBUG:boto:Data: 2022-09-02T05:45:24.341 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.342 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.343 DEBUG:boto:Params: {} 2022-09-02T05:45:24.344 DEBUG:boto:Token: None 2022-09-02T05:45:24.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE= 2022-09-02T05:45:24.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE=', 'Content-Length': '0'} 2022-09-02T05:45:24.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609fdec0bd34907f-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.348 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.349 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.349 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.350 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.350 DEBUG:boto:Method: GET 2022-09-02T05:45:24.351 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.351 DEBUG:boto:Data: 2022-09-02T05:45:24.352 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.353 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.354 DEBUG:boto:Params: {} 2022-09-02T05:45:24.354 DEBUG:boto:Token: None 2022-09-02T05:45:24.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:24.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE= 2022-09-02T05:45:24.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zNKk/z56Li3zPkEH5lB2FcEe1zE=', 'Content-Length': '0'} 2022-09-02T05:45:24.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b542e4bda06933d-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.359 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.359 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.360 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.360 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.361 DEBUG:boto:Method: GET 2022-09-02T05:45:24.361 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.362 DEBUG:boto:Data: 2022-09-02T05:45:24.362 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.363 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.364 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.364 DEBUG:boto:Params: {} 2022-09-02T05:45:24.365 DEBUG:boto:Token: None 2022-09-02T05:45:24.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f846e97a116a03-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.370 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.371 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.371 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.372 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.372 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.373 DEBUG:boto:Method: GET 2022-09-02T05:45:24.373 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.374 DEBUG:boto:Data: 2022-09-02T05:45:24.374 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.376 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.376 DEBUG:boto:Params: {} 2022-09-02T05:45:24.377 DEBUG:boto:Token: None 2022-09-02T05:45:24.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a2ca88c4b29af25-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.383 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.384 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.385 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.385 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.386 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:24.386 DEBUG:boto:Data: 2022-09-02T05:45:24.387 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.388 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.388 DEBUG:boto:Params: {} 2022-09-02T05:45:24.389 DEBUG:boto:Token: None 2022-09-02T05:45:24.389 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22 2022-09-02T05:45:24.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FANGNV4a4wDlXPS/UFvniOqp8f0= 2022-09-02T05:45:24.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FANGNV4a4wDlXPS/UFvniOqp8f0=', 'Content-Length': '0'} 2022-09-02T05:45:24.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078be1d8a30142da1-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.393 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.393 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.394 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.395 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:24.395 DEBUG:boto:Data: 2022-09-02T05:45:24.396 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.397 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.397 DEBUG:boto:Params: {} 2022-09-02T05:45:24.398 DEBUG:boto:Token: None 2022-09-02T05:45:24.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22 2022-09-02T05:45:24.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FANGNV4a4wDlXPS/UFvniOqp8f0= 2022-09-02T05:45:24.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FANGNV4a4wDlXPS/UFvniOqp8f0=', 'Content-Length': '0'} 2022-09-02T05:45:24.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b611eeb1ce6d6e7-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:24.403 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.404 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.404 DEBUG:boto:Method: GET 2022-09-02T05:45:24.405 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:24.405 DEBUG:boto:Data: 2022-09-02T05:45:24.406 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.407 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.407 DEBUG:boto:Params: {} 2022-09-02T05:45:24.408 DEBUG:boto:Token: None 2022-09-02T05:45:24.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22 2022-09-02T05:45:24.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GVHf+BDqIz1FdDKq/gy7Xe9/Oyk= 2022-09-02T05:45:24.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GVHf+BDqIz1FdDKq/gy7Xe9/Oyk=', 'Content-Length': '0'} 2022-09-02T05:45:24.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e49bb8cbc4a20f2-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.413 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:24.413 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:24.414 DEBUG:boto:Method: GET 2022-09-02T05:45:24.414 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:24.415 DEBUG:boto:Data: 2022-09-02T05:45:24.415 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.417 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.417 DEBUG:boto:Params: {} 2022-09-02T05:45:24.418 DEBUG:boto:Token: None 2022-09-02T05:45:24.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj22 2022-09-02T05:45:24.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GVHf+BDqIz1FdDKq/gy7Xe9/Oyk= 2022-09-02T05:45:24.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GVHf+BDqIz1FdDKq/gy7Xe9/Oyk=', 'Content-Length': '0'} 2022-09-02T05:45:24.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097790215350e01a1-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.462 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.463 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.463 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.464 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.464 DEBUG:boto:Method: GET 2022-09-02T05:45:24.465 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.466 DEBUG:boto:Data: 2022-09-02T05:45:24.466 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.467 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.468 DEBUG:boto:Params: {} 2022-09-02T05:45:24.468 DEBUG:boto:Token: None 2022-09-02T05:45:24.469 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecbdaa00fcd56c42-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.473 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.475 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.475 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.476 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.476 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.477 DEBUG:boto:Method: GET 2022-09-02T05:45:24.477 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:24.478 DEBUG:boto:Data: 2022-09-02T05:45:24.478 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.479 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.480 DEBUG:boto:Params: {} 2022-09-02T05:45:24.481 DEBUG:boto:Token: None 2022-09-02T05:45:24.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004489c4b5cf7d09dc-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.486 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:24.488 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.488 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.489 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.489 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.490 DEBUG:boto:Method: GET 2022-09-02T05:45:24.490 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.491 DEBUG:boto:Data: 2022-09-02T05:45:24.492 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.493 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.493 DEBUG:boto:Params: {} 2022-09-02T05:45:24.494 DEBUG:boto:Token: None 2022-09-02T05:45:24.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q= 2022-09-02T05:45:24.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q=', 'Content-Length': '0'} 2022-09-02T05:45:24.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8e97ba98dc8da11-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.499 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.500 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.500 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.501 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.501 DEBUG:boto:Method: GET 2022-09-02T05:45:24.502 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.503 DEBUG:boto:Data: 2022-09-02T05:45:24.503 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.505 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.505 DEBUG:boto:Params: {} 2022-09-02T05:45:24.506 DEBUG:boto:Token: None 2022-09-02T05:45:24.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q= 2022-09-02T05:45:24.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q=', 'Content-Length': '0'} 2022-09-02T05:45:24.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658103645c08d743-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.510 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.511 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.511 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.512 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.512 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.513 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.514 DEBUG:boto:Data: 2022-09-02T05:45:24.515 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.516 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.516 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.517 DEBUG:boto:Params: {} 2022-09-02T05:45:24.518 DEBUG:boto:Token: None 2022-09-02T05:45:24.519 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pw+vTKO8bvWePWTkY4k2jvOg51o= 2022-09-02T05:45:24.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pw+vTKO8bvWePWTkY4k2jvOg51o=', 'Content-Length': '0'} 2022-09-02T05:45:24.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004953ca0e929f2580-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.523 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.524 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.524 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.525 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.525 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.526 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.526 DEBUG:boto:Data: 2022-09-02T05:45:24.527 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.528 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.528 DEBUG:boto:Params: {} 2022-09-02T05:45:24.529 DEBUG:boto:Token: None 2022-09-02T05:45:24.530 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pw+vTKO8bvWePWTkY4k2jvOg51o= 2022-09-02T05:45:24.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pw+vTKO8bvWePWTkY4k2jvOg51o=', 'Content-Length': '0'} 2022-09-02T05:45:24.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd31da11d303a22b-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:24.534 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.535 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.535 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.536 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.536 DEBUG:boto:Method: GET 2022-09-02T05:45:24.537 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.537 DEBUG:boto:Data: 2022-09-02T05:45:24.538 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.539 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.540 DEBUG:boto:Params: {} 2022-09-02T05:45:24.541 DEBUG:boto:Token: None 2022-09-02T05:45:24.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q= 2022-09-02T05:45:24.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q=', 'Content-Length': '0'} 2022-09-02T05:45:24.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e541c0d9eab4c8a9-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.586 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.587 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.587 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.588 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.588 DEBUG:boto:Method: GET 2022-09-02T05:45:24.589 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.590 DEBUG:boto:Data: 2022-09-02T05:45:24.590 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.591 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.592 DEBUG:boto:Params: {} 2022-09-02T05:45:24.592 DEBUG:boto:Token: None 2022-09-02T05:45:24.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:24.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q= 2022-09-02T05:45:24.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pigcnn92KjKG/vLX4VxGzG1Qk5Q=', 'Content-Length': '0'} 2022-09-02T05:45:24.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bb635d7c0397683-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.597 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.598 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.598 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.599 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.599 DEBUG:boto:Method: GET 2022-09-02T05:45:24.600 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.600 DEBUG:boto:Data: 2022-09-02T05:45:24.601 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.602 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.603 DEBUG:boto:Params: {} 2022-09-02T05:45:24.603 DEBUG:boto:Token: None 2022-09-02T05:45:24.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001756f31cbd8a76f9-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.608 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.609 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.610 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.610 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.611 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.611 DEBUG:boto:Method: GET 2022-09-02T05:45:24.612 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.612 DEBUG:boto:Data: 2022-09-02T05:45:24.613 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.614 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.615 DEBUG:boto:Params: {} 2022-09-02T05:45:24.615 DEBUG:boto:Token: None 2022-09-02T05:45:24.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000617f24773e6b1ba5-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.620 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.621 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.621 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.622 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.623 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:24.623 DEBUG:boto:Data: 2022-09-02T05:45:24.624 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.624 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.624 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.625 DEBUG:boto:Params: {} 2022-09-02T05:45:24.626 DEBUG:boto:Token: None 2022-09-02T05:45:24.626 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33 2022-09-02T05:45:24.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ZOLUXIvQM7h8wNLdef5PjlGSsk= 2022-09-02T05:45:24.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ZOLUXIvQM7h8wNLdef5PjlGSsk=', 'Content-Length': '0'} 2022-09-02T05:45:24.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6758d40c55804c-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.630 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.630 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.631 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.631 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:24.632 DEBUG:boto:Data: 2022-09-02T05:45:24.632 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.633 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.634 DEBUG:boto:Params: {} 2022-09-02T05:45:24.634 DEBUG:boto:Token: None 2022-09-02T05:45:24.635 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33 2022-09-02T05:45:24.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ZOLUXIvQM7h8wNLdef5PjlGSsk= 2022-09-02T05:45:24.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ZOLUXIvQM7h8wNLdef5PjlGSsk=', 'Content-Length': '0'} 2022-09-02T05:45:24.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003175ae4b8435ba5a-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:24.640 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.640 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.641 DEBUG:boto:Method: GET 2022-09-02T05:45:24.641 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:24.642 DEBUG:boto:Data: 2022-09-02T05:45:24.642 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.643 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.644 DEBUG:boto:Params: {} 2022-09-02T05:45:24.644 DEBUG:boto:Token: None 2022-09-02T05:45:24.645 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33 2022-09-02T05:45:24.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK4JGswARG5IS2R5VkhGYo4jQLY= 2022-09-02T05:45:24.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK4JGswARG5IS2R5VkhGYo4jQLY=', 'Content-Length': '0'} 2022-09-02T05:45:24.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63414ed51c65d3c-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.649 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:24.650 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:24.650 DEBUG:boto:Method: GET 2022-09-02T05:45:24.651 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:24.651 DEBUG:boto:Data: 2022-09-02T05:45:24.652 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.653 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.653 DEBUG:boto:Params: {} 2022-09-02T05:45:24.654 DEBUG:boto:Token: None 2022-09-02T05:45:24.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj33 2022-09-02T05:45:24.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tK4JGswARG5IS2R5VkhGYo4jQLY= 2022-09-02T05:45:24.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tK4JGswARG5IS2R5VkhGYo4jQLY=', 'Content-Length': '0'} 2022-09-02T05:45:24.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbbf899548c91a14-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.698 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.699 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.699 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.700 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.700 DEBUG:boto:Method: GET 2022-09-02T05:45:24.701 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.702 DEBUG:boto:Data: 2022-09-02T05:45:24.702 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.703 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.704 DEBUG:boto:Params: {} 2022-09-02T05:45:24.704 DEBUG:boto:Token: None 2022-09-02T05:45:24.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003681c01726ce4880-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.709 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.710 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.711 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.711 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.712 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.712 DEBUG:boto:Method: GET 2022-09-02T05:45:24.713 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:24.713 DEBUG:boto:Data: 2022-09-02T05:45:24.714 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.715 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.715 DEBUG:boto:Params: {} 2022-09-02T05:45:24.716 DEBUG:boto:Token: None 2022-09-02T05:45:24.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c361368539032a57-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.721 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:24.723 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.723 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.724 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.724 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.725 DEBUG:boto:Method: GET 2022-09-02T05:45:24.725 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.726 DEBUG:boto:Data: 2022-09-02T05:45:24.726 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.727 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.727 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.728 DEBUG:boto:Params: {} 2022-09-02T05:45:24.729 DEBUG:boto:Token: None 2022-09-02T05:45:24.729 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k= 2022-09-02T05:45:24.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k=', 'Content-Length': '0'} 2022-09-02T05:45:24.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a01ac44dfab28b-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.733 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.734 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.734 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.735 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.735 DEBUG:boto:Method: GET 2022-09-02T05:45:24.736 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.736 DEBUG:boto:Data: 2022-09-02T05:45:24.737 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.738 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.738 DEBUG:boto:Params: {} 2022-09-02T05:45:24.739 DEBUG:boto:Token: None 2022-09-02T05:45:24.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k= 2022-09-02T05:45:24.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k=', 'Content-Length': '0'} 2022-09-02T05:45:24.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090550472b9dc402f-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.744 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.744 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.745 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.745 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.746 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.746 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.747 DEBUG:boto:Data: 2022-09-02T05:45:24.747 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.748 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.749 DEBUG:boto:Params: {} 2022-09-02T05:45:24.749 DEBUG:boto:Token: None 2022-09-02T05:45:24.750 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bhzcBqZ7GG+CyOZH9oHzPSOl/zE= 2022-09-02T05:45:24.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bhzcBqZ7GG+CyOZH9oHzPSOl/zE=', 'Content-Length': '0'} 2022-09-02T05:45:24.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a205e3b3dd2f06-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.754 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.755 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.755 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.756 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.757 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.757 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.758 DEBUG:boto:Data: 2022-09-02T05:45:24.758 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.759 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.760 DEBUG:boto:Params: {} 2022-09-02T05:45:24.760 DEBUG:boto:Token: None 2022-09-02T05:45:24.761 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bhzcBqZ7GG+CyOZH9oHzPSOl/zE= 2022-09-02T05:45:24.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bhzcBqZ7GG+CyOZH9oHzPSOl/zE=', 'Content-Length': '0'} 2022-09-02T05:45:24.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee721c54d17f764a-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:24.765 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.766 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.766 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.767 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.767 DEBUG:boto:Method: GET 2022-09-02T05:45:24.768 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.768 DEBUG:boto:Data: 2022-09-02T05:45:24.769 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.770 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.771 DEBUG:boto:Params: {} 2022-09-02T05:45:24.771 DEBUG:boto:Token: None 2022-09-02T05:45:24.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k= 2022-09-02T05:45:24.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k=', 'Content-Length': '0'} 2022-09-02T05:45:24.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71303ef86856a4b-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.776 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.776 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.777 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.778 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.778 DEBUG:boto:Method: GET 2022-09-02T05:45:24.779 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.779 DEBUG:boto:Data: 2022-09-02T05:45:24.780 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.781 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.782 DEBUG:boto:Params: {} 2022-09-02T05:45:24.782 DEBUG:boto:Token: None 2022-09-02T05:45:24.783 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:24.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k= 2022-09-02T05:45:24.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8aUgoQuNq1awXVEXDoWqjHc/5k=', 'Content-Length': '0'} 2022-09-02T05:45:24.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b833b481b46a0ec-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.787 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.788 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.788 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.789 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.789 DEBUG:boto:Method: GET 2022-09-02T05:45:24.790 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.790 DEBUG:boto:Data: 2022-09-02T05:45:24.791 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.792 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.792 DEBUG:boto:Params: {} 2022-09-02T05:45:24.793 DEBUG:boto:Token: None 2022-09-02T05:45:24.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e1d1c1c19df4b5-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.798 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.799 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.799 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.800 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.800 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.801 DEBUG:boto:Method: GET 2022-09-02T05:45:24.801 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.802 DEBUG:boto:Data: 2022-09-02T05:45:24.803 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.804 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.804 DEBUG:boto:Params: {} 2022-09-02T05:45:24.805 DEBUG:boto:Token: None 2022-09-02T05:45:24.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cb8c8465be75d95-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.854 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.855 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.856 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.857 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.857 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:24.858 DEBUG:boto:Data: 2022-09-02T05:45:24.858 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.859 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.860 DEBUG:boto:Params: {} 2022-09-02T05:45:24.860 DEBUG:boto:Token: None 2022-09-02T05:45:24.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44 2022-09-02T05:45:24.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Foteqpe5fUKGZoi+t2is/LH6mbc= 2022-09-02T05:45:24.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Foteqpe5fUKGZoi+t2is/LH6mbc=', 'Content-Length': '0'} 2022-09-02T05:45:24.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e34c6c09cf63d287-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.865 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.866 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.866 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.867 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:24.867 DEBUG:boto:Data: 2022-09-02T05:45:24.868 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.869 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.869 DEBUG:boto:Params: {} 2022-09-02T05:45:24.870 DEBUG:boto:Token: None 2022-09-02T05:45:24.870 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44 2022-09-02T05:45:24.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Foteqpe5fUKGZoi+t2is/LH6mbc= 2022-09-02T05:45:24.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Foteqpe5fUKGZoi+t2is/LH6mbc=', 'Content-Length': '0'} 2022-09-02T05:45:24.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004223788844efb9bf-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:24.875 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.876 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.876 DEBUG:boto:Method: GET 2022-09-02T05:45:24.877 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:24.877 DEBUG:boto:Data: 2022-09-02T05:45:24.878 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.879 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.879 DEBUG:boto:Params: {} 2022-09-02T05:45:24.880 DEBUG:boto:Token: None 2022-09-02T05:45:24.880 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44 2022-09-02T05:45:24.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:miLAecYOnU3ej+UP1xo2fcLBbRM= 2022-09-02T05:45:24.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:miLAecYOnU3ej+UP1xo2fcLBbRM=', 'Content-Length': '0'} 2022-09-02T05:45:24.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011211b5be4924fd0-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.884 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:24.885 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:24.885 DEBUG:boto:Method: GET 2022-09-02T05:45:24.886 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:24.887 DEBUG:boto:Data: 2022-09-02T05:45:24.887 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.888 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.889 DEBUG:boto:Params: {} 2022-09-02T05:45:24.889 DEBUG:boto:Token: None 2022-09-02T05:45:24.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj44 2022-09-02T05:45:24.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:miLAecYOnU3ej+UP1xo2fcLBbRM= 2022-09-02T05:45:24.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:miLAecYOnU3ej+UP1xo2fcLBbRM=', 'Content-Length': '0'} 2022-09-02T05:45:24.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674a4e6a619c9ad4-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.895 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.895 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.896 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.896 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.897 DEBUG:boto:Method: GET 2022-09-02T05:45:24.897 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.898 DEBUG:boto:Data: 2022-09-02T05:45:24.899 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.900 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.900 DEBUG:boto:Params: {} 2022-09-02T05:45:24.901 DEBUG:boto:Token: None 2022-09-02T05:45:24.901 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004faf649e493189a6-0063119874-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.906 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.906 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.907 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.908 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.908 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.909 DEBUG:boto:Method: GET 2022-09-02T05:45:24.909 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:24.910 DEBUG:boto:Data: 2022-09-02T05:45:24.910 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.911 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.912 DEBUG:boto:Params: {} 2022-09-02T05:45:24.912 DEBUG:boto:Token: None 2022-09-02T05:45:24.913 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:24.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b2695dd97d849ac-0063119874-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.917 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:24.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:24.919 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:24.919 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:24.920 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.920 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.921 DEBUG:boto:Method: GET 2022-09-02T05:45:24.921 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.922 DEBUG:boto:Data: 2022-09-02T05:45:24.922 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.923 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.924 DEBUG:boto:Params: {} 2022-09-02T05:45:24.925 DEBUG:boto:Token: None 2022-09-02T05:45:24.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60= 2022-09-02T05:45:24.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60=', 'Content-Length': '0'} 2022-09-02T05:45:24.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000448a73cea949307-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.929 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:24.930 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:24.930 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.931 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.931 DEBUG:boto:Method: GET 2022-09-02T05:45:24.932 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.932 DEBUG:boto:Data: 2022-09-02T05:45:24.933 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.934 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.934 DEBUG:boto:Params: {} 2022-09-02T05:45:24.935 DEBUG:boto:Token: None 2022-09-02T05:45:24.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60= 2022-09-02T05:45:24.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60=', 'Content-Length': '0'} 2022-09-02T05:45:24.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03c7dff85f98007-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.939 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:24.940 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:24.940 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.941 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.941 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.942 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.942 DEBUG:boto:Data: 2022-09-02T05:45:24.943 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.944 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.944 DEBUG:boto:Params: {} 2022-09-02T05:45:24.945 DEBUG:boto:Token: None 2022-09-02T05:45:24.946 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UTOHA1IOqTaZpeIW+Hy6snzcrIs= 2022-09-02T05:45:24.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UTOHA1IOqTaZpeIW+Hy6snzcrIs=', 'Content-Length': '0'} 2022-09-02T05:45:24.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d866e7b03651b8fb-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.950 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:24.950 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:24.951 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.951 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.952 DEBUG:boto:Method: HEAD 2022-09-02T05:45:24.952 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.953 DEBUG:boto:Data: 2022-09-02T05:45:24.953 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.954 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.955 DEBUG:boto:Params: {} 2022-09-02T05:45:24.955 DEBUG:boto:Token: None 2022-09-02T05:45:24.956 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UTOHA1IOqTaZpeIW+Hy6snzcrIs= 2022-09-02T05:45:24.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UTOHA1IOqTaZpeIW+Hy6snzcrIs=', 'Content-Length': '0'} 2022-09-02T05:45:24.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3fdc1c96188707-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:24.970 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:24.970 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:24.971 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.971 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.972 DEBUG:boto:Method: GET 2022-09-02T05:45:24.972 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.973 DEBUG:boto:Data: 2022-09-02T05:45:24.973 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.974 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.975 DEBUG:boto:Params: {} 2022-09-02T05:45:24.976 DEBUG:boto:Token: None 2022-09-02T05:45:24.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60= 2022-09-02T05:45:24.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60=', 'Content-Length': '0'} 2022-09-02T05:45:24.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008433066b530aef12-0063119874-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.980 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:24.981 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:24.981 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.982 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.982 DEBUG:boto:Method: GET 2022-09-02T05:45:24.983 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.984 DEBUG:boto:Data: 2022-09-02T05:45:24.984 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:24.985 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.986 DEBUG:boto:Params: {} 2022-09-02T05:45:24.986 DEBUG:boto:Token: None 2022-09-02T05:45:24.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:24.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60= 2022-09-02T05:45:24.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AVRnPgd3utERjBI2qVJ4dAWLi60=', 'Content-Length': '0'} 2022-09-02T05:45:24.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8378a676be466b0-0063119874-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:24.991 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:24.991 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:24.992 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:24.993 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:24.993 DEBUG:boto:Method: GET 2022-09-02T05:45:24.994 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:24.994 DEBUG:boto:Data: 2022-09-02T05:45:24.995 DEBUG:boto:Headers: {} 2022-09-02T05:45:24.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:24.996 DEBUG:boto:Port: 8000 2022-09-02T05:45:24.996 DEBUG:boto:Params: {} 2022-09-02T05:45:24.997 DEBUG:boto:Token: None 2022-09-02T05:45:24.997 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:24 GMT /xljbkf-74/ 2022-09-02T05:45:24.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js= 2022-09-02T05:45:24.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NjMn6e+3zElxRmvtTjqW67jH0js=', 'Content-Length': '0'} 2022-09-02T05:45:25.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa12da65ef63761-0063119875-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.002 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.003 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.003 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.004 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.004 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.005 DEBUG:boto:Method: GET 2022-09-02T05:45:25.005 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.006 DEBUG:boto:Data: 2022-09-02T05:45:25.006 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.007 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.008 DEBUG:boto:Params: {} 2022-09-02T05:45:25.008 DEBUG:boto:Token: None 2022-09-02T05:45:25.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb261beb9950a360-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.054 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.055 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:25.056 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:25.056 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.057 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:25.057 DEBUG:boto:Data: 2022-09-02T05:45:25.058 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:25.059 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.059 DEBUG:boto:Params: {} 2022-09-02T05:45:25.060 DEBUG:boto:Token: None 2022-09-02T05:45:25.060 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj55 2022-09-02T05:45:25.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GMVxhFbIdqGFMKH/yieViVd7W98= 2022-09-02T05:45:25.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GMVxhFbIdqGFMKH/yieViVd7W98=', 'Content-Length': '0'} 2022-09-02T05:45:25.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006304ecf824b1157f-0063119875-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.064 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:25.065 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:25.065 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.066 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:25.066 DEBUG:boto:Data: 2022-09-02T05:45:25.067 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.068 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.068 DEBUG:boto:Params: {} 2022-09-02T05:45:25.069 DEBUG:boto:Token: None 2022-09-02T05:45:25.070 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj55 2022-09-02T05:45:25.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GMVxhFbIdqGFMKH/yieViVd7W98= 2022-09-02T05:45:25.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GMVxhFbIdqGFMKH/yieViVd7W98=', 'Content-Length': '0'} 2022-09-02T05:45:25.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ff0b58928943ff-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:25.074 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:25.075 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:25.075 DEBUG:boto:Method: GET 2022-09-02T05:45:25.076 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:25.076 DEBUG:boto:Data: 2022-09-02T05:45:25.077 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:25.078 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.078 DEBUG:boto:Params: {} 2022-09-02T05:45:25.079 DEBUG:boto:Token: None 2022-09-02T05:45:25.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj55 2022-09-02T05:45:25.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4lBErbyChBq72GeYayHuc4OdNT0= 2022-09-02T05:45:25.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4lBErbyChBq72GeYayHuc4OdNT0=', 'Content-Length': '0'} 2022-09-02T05:45:25.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d6d7e618956e6e-0063119875-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.084 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:25.084 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:25.085 DEBUG:boto:Method: GET 2022-09-02T05:45:25.085 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:25.086 DEBUG:boto:Data: 2022-09-02T05:45:25.086 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.088 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.088 DEBUG:boto:Params: {} 2022-09-02T05:45:25.089 DEBUG:boto:Token: None 2022-09-02T05:45:25.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj55 2022-09-02T05:45:25.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4lBErbyChBq72GeYayHuc4OdNT0= 2022-09-02T05:45:25.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4lBErbyChBq72GeYayHuc4OdNT0=', 'Content-Length': '0'} 2022-09-02T05:45:25.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f62dd4e8a05771-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.093 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.094 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.094 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.095 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.095 DEBUG:boto:Method: GET 2022-09-02T05:45:25.096 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.096 DEBUG:boto:Data: 2022-09-02T05:45:25.097 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.097 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:25.098 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.098 DEBUG:boto:Params: {} 2022-09-02T05:45:25.099 DEBUG:boto:Token: None 2022-09-02T05:45:25.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004850628309a6d3a3-0063119875-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.104 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.105 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.105 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.106 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.107 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.107 DEBUG:boto:Method: GET 2022-09-02T05:45:25.108 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:25.108 DEBUG:boto:Data: 2022-09-02T05:45:25.109 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.110 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.110 DEBUG:boto:Params: {} 2022-09-02T05:45:25.111 DEBUG:boto:Token: None 2022-09-02T05:45:25.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db942115c5c34414-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.116 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.117 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:45:25.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:45:25.118 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.118 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.119 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.119 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:25.120 DEBUG:boto:Data: 2022-09-02T05:45:25.120 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.121 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.122 DEBUG:boto:Params: {} 2022-09-02T05:45:25.122 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:45:25.123 DEBUG:boto:Token: None 2022-09-02T05:45:25.124 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2wkQfpg/Nsa03c2N3ZsMpfzaeGE= 2022-09-02T05:45:25.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2wkQfpg/Nsa03c2N3ZsMpfzaeGE=', 'Content-Length': '0'} 2022-09-02T05:45:25.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000563c85c6e5f4df93-0063119875-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.129 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.129 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.130 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.130 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:25.131 DEBUG:boto:Data: 2022-09-02T05:45:25.131 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.132 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.133 DEBUG:boto:Params: {} 2022-09-02T05:45:25.134 DEBUG:boto:Token: None 2022-09-02T05:45:25.134 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2wkQfpg/Nsa03c2N3ZsMpfzaeGE= 2022-09-02T05:45:25.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2wkQfpg/Nsa03c2N3ZsMpfzaeGE=', 'Content-Length': '0'} 2022-09-02T05:45:25.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f52aa6e3b1bdf29-0063119875-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.138 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:45:25.139 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.139 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.140 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.140 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.141 DEBUG:boto:Method: GET 2022-09-02T05:45:25.141 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.142 DEBUG:boto:Data: 2022-09-02T05:45:25.142 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.144 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.144 DEBUG:boto:Params: {} 2022-09-02T05:45:25.145 DEBUG:boto:Token: None 2022-09-02T05:45:25.145 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/?versions 2022-09-02T05:45:25.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E= 2022-09-02T05:45:25.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E=', 'Content-Length': '0'} 2022-09-02T05:45:25.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1445620d5d6dd9f-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.149 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:45:25.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:45:25.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:45:25.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:45:25.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:45:25.153 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:45:25.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.154 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.154 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.155 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.155 DEBUG:boto:Method: GET 2022-09-02T05:45:25.156 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.156 DEBUG:boto:Data: 2022-09-02T05:45:25.157 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.158 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.159 DEBUG:boto:Params: {} 2022-09-02T05:45:25.159 DEBUG:boto:Token: None 2022-09-02T05:45:25.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/?versions 2022-09-02T05:45:25.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E= 2022-09-02T05:45:25.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E=', 'Content-Length': '0'} 2022-09-02T05:45:25.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000238ceaea62f24d66-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.164 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:45:25.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:45:25.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:45:25.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:45:25.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:45:25.168 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.168 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.169 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.169 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.170 DEBUG:boto:Method: GET 2022-09-02T05:45:25.170 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.171 DEBUG:boto:Data: 2022-09-02T05:45:25.171 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.172 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.173 DEBUG:boto:Params: {} 2022-09-02T05:45:25.173 DEBUG:boto:Token: None 2022-09-02T05:45:25.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/?versions 2022-09-02T05:45:25.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E= 2022-09-02T05:45:25.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E=', 'Content-Length': '0'} 2022-09-02T05:45:25.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000018a9dee60e2b406-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.178 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.179 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.180 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.180 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.181 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.181 DEBUG:boto:Method: GET 2022-09-02T05:45:25.182 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:25.182 DEBUG:boto:Data: 2022-09-02T05:45:25.183 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.184 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.184 DEBUG:boto:Params: {} 2022-09-02T05:45:25.185 DEBUG:boto:Token: None 2022-09-02T05:45:25.186 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/?versions 2022-09-02T05:45:25.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E= 2022-09-02T05:45:25.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1rKtFEg93PjgZxhjN8kmo4K2n6E=', 'Content-Length': '0'} 2022-09-02T05:45:25.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ffb62f36275da2-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.190 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:25.192 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.192 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.193 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.193 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.194 DEBUG:boto:Method: GET 2022-09-02T05:45:25.194 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.195 DEBUG:boto:Data: 2022-09-02T05:45:25.195 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.197 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.197 DEBUG:boto:Params: {} 2022-09-02T05:45:25.198 DEBUG:boto:Token: None 2022-09-02T05:45:25.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA= 2022-09-02T05:45:25.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA=', 'Content-Length': '0'} 2022-09-02T05:45:25.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001405dec54ab1cbc7-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.202 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.203 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.204 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.204 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.205 DEBUG:boto:Method: GET 2022-09-02T05:45:25.205 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.206 DEBUG:boto:Data: 2022-09-02T05:45:25.207 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.208 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.208 DEBUG:boto:Params: {} 2022-09-02T05:45:25.209 DEBUG:boto:Token: None 2022-09-02T05:45:25.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA= 2022-09-02T05:45:25.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA=', 'Content-Length': '0'} 2022-09-02T05:45:25.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b717016fb265835e-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.254 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.255 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.255 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.256 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.256 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.257 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.257 DEBUG:boto:Data: 2022-09-02T05:45:25.258 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.259 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.259 DEBUG:boto:Params: {} 2022-09-02T05:45:25.260 DEBUG:boto:Token: None 2022-09-02T05:45:25.261 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BTYORSyuoE0yAoytFoDC6MLffWo= 2022-09-02T05:45:25.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BTYORSyuoE0yAoytFoDC6MLffWo=', 'Content-Length': '0'} 2022-09-02T05:45:25.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d204691ef7388ea-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.265 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.265 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.266 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.266 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.267 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.267 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.268 DEBUG:boto:Data: 2022-09-02T05:45:25.268 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.269 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.270 DEBUG:boto:Params: {} 2022-09-02T05:45:25.270 DEBUG:boto:Token: None 2022-09-02T05:45:25.271 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BTYORSyuoE0yAoytFoDC6MLffWo= 2022-09-02T05:45:25.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BTYORSyuoE0yAoytFoDC6MLffWo=', 'Content-Length': '0'} 2022-09-02T05:45:25.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d4136ebd3f71110-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:25.275 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.276 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.276 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.277 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.277 DEBUG:boto:Method: GET 2022-09-02T05:45:25.278 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.278 DEBUG:boto:Data: 2022-09-02T05:45:25.279 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.280 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.281 DEBUG:boto:Params: {} 2022-09-02T05:45:25.281 DEBUG:boto:Token: None 2022-09-02T05:45:25.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA= 2022-09-02T05:45:25.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA=', 'Content-Length': '0'} 2022-09-02T05:45:25.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2c16d9f27180d2-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.286 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.286 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.287 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.288 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.288 DEBUG:boto:Method: GET 2022-09-02T05:45:25.289 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.289 DEBUG:boto:Data: 2022-09-02T05:45:25.290 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.291 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.291 DEBUG:boto:Params: {} 2022-09-02T05:45:25.292 DEBUG:boto:Token: None 2022-09-02T05:45:25.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:25.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA= 2022-09-02T05:45:25.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/0mG/5lGkDX7S3XR0W+IQjm9yFA=', 'Content-Length': '0'} 2022-09-02T05:45:25.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006db339ca6c6496b-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.296 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.297 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.298 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.298 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.299 DEBUG:boto:Method: GET 2022-09-02T05:45:25.299 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.300 DEBUG:boto:Data: 2022-09-02T05:45:25.300 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.301 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.302 DEBUG:boto:Params: {} 2022-09-02T05:45:25.303 DEBUG:boto:Token: None 2022-09-02T05:45:25.303 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd62679e19b61b8-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.307 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.308 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.309 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.309 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.310 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.311 DEBUG:boto:Method: GET 2022-09-02T05:45:25.311 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.312 DEBUG:boto:Data: 2022-09-02T05:45:25.312 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.313 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.314 DEBUG:boto:Params: {} 2022-09-02T05:45:25.314 DEBUG:boto:Token: None 2022-09-02T05:45:25.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a8f5cf0ea007d25-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.320 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.320 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.321 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.322 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.322 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:25.323 DEBUG:boto:Data: 2022-09-02T05:45:25.323 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.324 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.325 DEBUG:boto:Params: {} 2022-09-02T05:45:25.325 DEBUG:boto:Token: None 2022-09-02T05:45:25.326 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11 2022-09-02T05:45:25.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwPDtePMy7dFxALnK4liSqK4L2M= 2022-09-02T05:45:25.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwPDtePMy7dFxALnK4liSqK4L2M=', 'Content-Length': '0'} 2022-09-02T05:45:25.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870375174becf3ad-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.329 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.330 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.331 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.331 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:25.332 DEBUG:boto:Data: 2022-09-02T05:45:25.332 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.333 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.334 DEBUG:boto:Params: {} 2022-09-02T05:45:25.334 DEBUG:boto:Token: None 2022-09-02T05:45:25.335 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11 2022-09-02T05:45:25.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwPDtePMy7dFxALnK4liSqK4L2M= 2022-09-02T05:45:25.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwPDtePMy7dFxALnK4liSqK4L2M=', 'Content-Length': '0'} 2022-09-02T05:45:25.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7bd2033c42862a1-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:25.339 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.340 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.340 DEBUG:boto:Method: GET 2022-09-02T05:45:25.341 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:25.341 DEBUG:boto:Data: 2022-09-02T05:45:25.342 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.343 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.344 DEBUG:boto:Params: {} 2022-09-02T05:45:25.344 DEBUG:boto:Token: None 2022-09-02T05:45:25.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11 2022-09-02T05:45:25.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fbRp5xhXWRsqwDU+Cmm2APDYy0Q= 2022-09-02T05:45:25.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fbRp5xhXWRsqwDU+Cmm2APDYy0Q=', 'Content-Length': '0'} 2022-09-02T05:45:25.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28d27c7b82b4a17-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.349 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:25.349 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:25.350 DEBUG:boto:Method: GET 2022-09-02T05:45:25.351 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:25.351 DEBUG:boto:Data: 2022-09-02T05:45:25.352 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.353 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.353 DEBUG:boto:Params: {} 2022-09-02T05:45:25.354 DEBUG:boto:Token: None 2022-09-02T05:45:25.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj11 2022-09-02T05:45:25.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fbRp5xhXWRsqwDU+Cmm2APDYy0Q= 2022-09-02T05:45:25.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fbRp5xhXWRsqwDU+Cmm2APDYy0Q=', 'Content-Length': '0'} 2022-09-02T05:45:25.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f81ecb977b2b245-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.359 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.359 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.360 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.360 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.361 DEBUG:boto:Method: GET 2022-09-02T05:45:25.361 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.362 DEBUG:boto:Data: 2022-09-02T05:45:25.363 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.364 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.364 DEBUG:boto:Params: {} 2022-09-02T05:45:25.365 DEBUG:boto:Token: None 2022-09-02T05:45:25.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000626adb80a137e1d6-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.370 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.371 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.372 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.372 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.373 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.373 DEBUG:boto:Method: GET 2022-09-02T05:45:25.374 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:25.374 DEBUG:boto:Data: 2022-09-02T05:45:25.375 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.376 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.377 DEBUG:boto:Params: {} 2022-09-02T05:45:25.377 DEBUG:boto:Token: None 2022-09-02T05:45:25.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbbd1d0b8a8b1c2d-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.382 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:25.384 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.384 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.385 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.385 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.386 DEBUG:boto:Method: GET 2022-09-02T05:45:25.386 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.387 DEBUG:boto:Data: 2022-09-02T05:45:25.387 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.389 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.389 DEBUG:boto:Params: {} 2022-09-02T05:45:25.390 DEBUG:boto:Token: None 2022-09-02T05:45:25.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ= 2022-09-02T05:45:25.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ=', 'Content-Length': '0'} 2022-09-02T05:45:25.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ed692e0176aa26-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.395 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.395 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.396 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.396 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.397 DEBUG:boto:Method: GET 2022-09-02T05:45:25.397 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.398 DEBUG:boto:Data: 2022-09-02T05:45:25.398 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.399 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.400 DEBUG:boto:Params: {} 2022-09-02T05:45:25.401 DEBUG:boto:Token: None 2022-09-02T05:45:25.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ= 2022-09-02T05:45:25.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ=', 'Content-Length': '0'} 2022-09-02T05:45:25.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c06572e787230341-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.405 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.406 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.406 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.407 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.407 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.408 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.408 DEBUG:boto:Data: 2022-09-02T05:45:25.409 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.410 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.410 DEBUG:boto:Params: {} 2022-09-02T05:45:25.411 DEBUG:boto:Token: None 2022-09-02T05:45:25.412 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SaaGaP7vgUGd1xAmIp9jRduP8kg= 2022-09-02T05:45:25.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SaaGaP7vgUGd1xAmIp9jRduP8kg=', 'Content-Length': '0'} 2022-09-02T05:45:25.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004284e2763a4d22ef-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.415 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.416 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.416 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.417 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.417 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.418 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.418 DEBUG:boto:Data: 2022-09-02T05:45:25.419 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.420 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.420 DEBUG:boto:Params: {} 2022-09-02T05:45:25.421 DEBUG:boto:Token: None 2022-09-02T05:45:25.422 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.422 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SaaGaP7vgUGd1xAmIp9jRduP8kg= 2022-09-02T05:45:25.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SaaGaP7vgUGd1xAmIp9jRduP8kg=', 'Content-Length': '0'} 2022-09-02T05:45:25.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020463ba444d2cdce-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:25.426 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.426 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.427 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.428 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.428 DEBUG:boto:Method: GET 2022-09-02T05:45:25.429 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.429 DEBUG:boto:Data: 2022-09-02T05:45:25.430 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.431 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.431 DEBUG:boto:Params: {} 2022-09-02T05:45:25.432 DEBUG:boto:Token: None 2022-09-02T05:45:25.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ= 2022-09-02T05:45:25.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ=', 'Content-Length': '0'} 2022-09-02T05:45:25.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083aba5e86585ffa9-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.478 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.479 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.480 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.480 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.481 DEBUG:boto:Method: GET 2022-09-02T05:45:25.482 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.482 DEBUG:boto:Data: 2022-09-02T05:45:25.483 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.484 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.484 DEBUG:boto:Params: {} 2022-09-02T05:45:25.485 DEBUG:boto:Token: None 2022-09-02T05:45:25.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:25.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ= 2022-09-02T05:45:25.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yO7j/eWh5VfYcZSm39369T33pLQ=', 'Content-Length': '0'} 2022-09-02T05:45:25.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9011ac76b088eb6-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.491 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.492 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.492 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.493 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.493 DEBUG:boto:Method: GET 2022-09-02T05:45:25.494 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.495 DEBUG:boto:Data: 2022-09-02T05:45:25.495 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.496 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.497 DEBUG:boto:Params: {} 2022-09-02T05:45:25.497 DEBUG:boto:Token: None 2022-09-02T05:45:25.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d95e720c29d1fb9-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.502 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.503 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.504 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.504 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.505 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.506 DEBUG:boto:Method: GET 2022-09-02T05:45:25.506 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.507 DEBUG:boto:Data: 2022-09-02T05:45:25.507 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.508 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.509 DEBUG:boto:Params: {} 2022-09-02T05:45:25.510 DEBUG:boto:Token: None 2022-09-02T05:45:25.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542b87669adff40f-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.558 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.559 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.560 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.560 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.561 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:25.561 DEBUG:boto:Data: 2022-09-02T05:45:25.562 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.563 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.563 DEBUG:boto:Params: {} 2022-09-02T05:45:25.564 DEBUG:boto:Token: None 2022-09-02T05:45:25.564 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22 2022-09-02T05:45:25.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:texJqy6sBOunwvJ6hJXPISVSttU= 2022-09-02T05:45:25.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:texJqy6sBOunwvJ6hJXPISVSttU=', 'Content-Length': '0'} 2022-09-02T05:45:25.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307ede23705915cd-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.569 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.569 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.570 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.570 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:25.571 DEBUG:boto:Data: 2022-09-02T05:45:25.571 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.573 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.573 DEBUG:boto:Params: {} 2022-09-02T05:45:25.574 DEBUG:boto:Token: None 2022-09-02T05:45:25.575 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22 2022-09-02T05:45:25.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:texJqy6sBOunwvJ6hJXPISVSttU= 2022-09-02T05:45:25.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:texJqy6sBOunwvJ6hJXPISVSttU=', 'Content-Length': '0'} 2022-09-02T05:45:25.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f58f2b473e9dda9-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:25.583 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.584 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.585 DEBUG:boto:Method: GET 2022-09-02T05:45:25.586 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:25.587 DEBUG:boto:Data: 2022-09-02T05:45:25.588 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.589 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.589 DEBUG:boto:Params: {} 2022-09-02T05:45:25.590 DEBUG:boto:Token: None 2022-09-02T05:45:25.591 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22 2022-09-02T05:45:25.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A1xwFDnTrmZQQUA54+qWGHR3HXo= 2022-09-02T05:45:25.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A1xwFDnTrmZQQUA54+qWGHR3HXo=', 'Content-Length': '0'} 2022-09-02T05:45:25.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e05ca9934592f5c3-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.595 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:25.596 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:25.597 DEBUG:boto:Method: GET 2022-09-02T05:45:25.597 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:25.598 DEBUG:boto:Data: 2022-09-02T05:45:25.598 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.599 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.600 DEBUG:boto:Params: {} 2022-09-02T05:45:25.601 DEBUG:boto:Token: None 2022-09-02T05:45:25.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj22 2022-09-02T05:45:25.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A1xwFDnTrmZQQUA54+qWGHR3HXo= 2022-09-02T05:45:25.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A1xwFDnTrmZQQUA54+qWGHR3HXo=', 'Content-Length': '0'} 2022-09-02T05:45:25.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037d602f2a761fd0-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.607 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.608 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.608 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.609 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.609 DEBUG:boto:Method: GET 2022-09-02T05:45:25.610 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.611 DEBUG:boto:Data: 2022-09-02T05:45:25.611 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.612 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.613 DEBUG:boto:Params: {} 2022-09-02T05:45:25.613 DEBUG:boto:Token: None 2022-09-02T05:45:25.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d66e8e9c8f08874-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.618 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.619 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.620 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.620 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.621 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.621 DEBUG:boto:Method: GET 2022-09-02T05:45:25.622 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:25.622 DEBUG:boto:Data: 2022-09-02T05:45:25.623 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.624 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.625 DEBUG:boto:Params: {} 2022-09-02T05:45:25.625 DEBUG:boto:Token: None 2022-09-02T05:45:25.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b548f6a2bef7a1-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.630 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:25.632 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.632 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.633 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.633 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.634 DEBUG:boto:Method: GET 2022-09-02T05:45:25.634 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.635 DEBUG:boto:Data: 2022-09-02T05:45:25.635 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.637 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.637 DEBUG:boto:Params: {} 2022-09-02T05:45:25.638 DEBUG:boto:Token: None 2022-09-02T05:45:25.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4= 2022-09-02T05:45:25.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4=', 'Content-Length': '0'} 2022-09-02T05:45:25.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd1b242ccd62031-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.682 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.683 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.684 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.685 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.686 DEBUG:boto:Method: GET 2022-09-02T05:45:25.687 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.688 DEBUG:boto:Data: 2022-09-02T05:45:25.689 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.691 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.691 DEBUG:boto:Params: {} 2022-09-02T05:45:25.692 DEBUG:boto:Token: None 2022-09-02T05:45:25.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4= 2022-09-02T05:45:25.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4=', 'Content-Length': '0'} 2022-09-02T05:45:25.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092b45a5b32611153-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.697 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.697 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.698 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.698 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.699 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.699 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.700 DEBUG:boto:Data: 2022-09-02T05:45:25.700 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.701 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.702 DEBUG:boto:Params: {} 2022-09-02T05:45:25.703 DEBUG:boto:Token: None 2022-09-02T05:45:25.703 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x3SeMk9PmOQ+27QTE5wfIUs1UKI= 2022-09-02T05:45:25.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x3SeMk9PmOQ+27QTE5wfIUs1UKI=', 'Content-Length': '0'} 2022-09-02T05:45:25.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b274f0af6cfa2578-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.707 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.707 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.708 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.708 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.709 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.709 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.710 DEBUG:boto:Data: 2022-09-02T05:45:25.710 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.712 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.712 DEBUG:boto:Params: {} 2022-09-02T05:45:25.713 DEBUG:boto:Token: None 2022-09-02T05:45:25.713 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x3SeMk9PmOQ+27QTE5wfIUs1UKI= 2022-09-02T05:45:25.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x3SeMk9PmOQ+27QTE5wfIUs1UKI=', 'Content-Length': '0'} 2022-09-02T05:45:25.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f4f38c0b6f2a38f-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:25.718 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.718 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.719 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.719 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.720 DEBUG:boto:Method: GET 2022-09-02T05:45:25.721 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.721 DEBUG:boto:Data: 2022-09-02T05:45:25.722 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.723 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.723 DEBUG:boto:Params: {} 2022-09-02T05:45:25.724 DEBUG:boto:Token: None 2022-09-02T05:45:25.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4= 2022-09-02T05:45:25.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4=', 'Content-Length': '0'} 2022-09-02T05:45:25.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000159c18171f731376-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.729 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.729 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.730 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.730 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.731 DEBUG:boto:Method: GET 2022-09-02T05:45:25.731 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.732 DEBUG:boto:Data: 2022-09-02T05:45:25.732 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.733 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.734 DEBUG:boto:Params: {} 2022-09-02T05:45:25.735 DEBUG:boto:Token: None 2022-09-02T05:45:25.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:25.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4= 2022-09-02T05:45:25.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hVFPXKKENiApMFk11b3M9Yl36Z4=', 'Content-Length': '0'} 2022-09-02T05:45:25.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc20100969b4038-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.739 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.740 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.741 DEBUG:boto:Method: GET 2022-09-02T05:45:25.742 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.742 DEBUG:boto:Data: 2022-09-02T05:45:25.743 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.744 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.744 DEBUG:boto:Params: {} 2022-09-02T05:45:25.745 DEBUG:boto:Token: None 2022-09-02T05:45:25.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006330c45420916a44-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.750 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.751 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.752 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.752 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.753 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.753 DEBUG:boto:Method: GET 2022-09-02T05:45:25.754 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.755 DEBUG:boto:Data: 2022-09-02T05:45:25.756 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.757 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.757 DEBUG:boto:Params: {} 2022-09-02T05:45:25.758 DEBUG:boto:Token: None 2022-09-02T05:45:25.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637870c3e2865148-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.806 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.807 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.808 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.808 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.809 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:25.809 DEBUG:boto:Data: 2022-09-02T05:45:25.810 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.811 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.811 DEBUG:boto:Params: {} 2022-09-02T05:45:25.812 DEBUG:boto:Token: None 2022-09-02T05:45:25.813 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33 2022-09-02T05:45:25.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:afITQ2VHgUovAfr31levY8l4hUo= 2022-09-02T05:45:25.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:afITQ2VHgUovAfr31levY8l4hUo=', 'Content-Length': '0'} 2022-09-02T05:45:25.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3e5d2fa7b68e7cf-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.817 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.817 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.818 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.818 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:25.819 DEBUG:boto:Data: 2022-09-02T05:45:25.819 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.820 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.821 DEBUG:boto:Params: {} 2022-09-02T05:45:25.821 DEBUG:boto:Token: None 2022-09-02T05:45:25.822 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33 2022-09-02T05:45:25.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:afITQ2VHgUovAfr31levY8l4hUo= 2022-09-02T05:45:25.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:afITQ2VHgUovAfr31levY8l4hUo=', 'Content-Length': '0'} 2022-09-02T05:45:25.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9009c824a46d02d-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:25.827 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.827 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.828 DEBUG:boto:Method: GET 2022-09-02T05:45:25.828 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:25.829 DEBUG:boto:Data: 2022-09-02T05:45:25.829 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.830 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.831 DEBUG:boto:Params: {} 2022-09-02T05:45:25.832 DEBUG:boto:Token: None 2022-09-02T05:45:25.832 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33 2022-09-02T05:45:25.833 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RFkdQPO/OgGPSr0GoZO+FJilM2o= 2022-09-02T05:45:25.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RFkdQPO/OgGPSr0GoZO+FJilM2o=', 'Content-Length': '0'} 2022-09-02T05:45:25.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009907d7c7bfb22053-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.836 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:25.837 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:25.837 DEBUG:boto:Method: GET 2022-09-02T05:45:25.838 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:25.838 DEBUG:boto:Data: 2022-09-02T05:45:25.839 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.840 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.840 DEBUG:boto:Params: {} 2022-09-02T05:45:25.841 DEBUG:boto:Token: None 2022-09-02T05:45:25.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj33 2022-09-02T05:45:25.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RFkdQPO/OgGPSr0GoZO+FJilM2o= 2022-09-02T05:45:25.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RFkdQPO/OgGPSr0GoZO+FJilM2o=', 'Content-Length': '0'} 2022-09-02T05:45:25.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be03069d7945521e-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.846 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.846 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.847 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.847 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.848 DEBUG:boto:Method: GET 2022-09-02T05:45:25.848 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.849 DEBUG:boto:Data: 2022-09-02T05:45:25.849 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.850 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.851 DEBUG:boto:Params: {} 2022-09-02T05:45:25.852 DEBUG:boto:Token: None 2022-09-02T05:45:25.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a12f8a25eb7b4ed-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.856 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.857 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.858 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.858 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.859 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.860 DEBUG:boto:Method: GET 2022-09-02T05:45:25.860 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:25.861 DEBUG:boto:Data: 2022-09-02T05:45:25.861 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.862 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.863 DEBUG:boto:Params: {} 2022-09-02T05:45:25.863 DEBUG:boto:Token: None 2022-09-02T05:45:25.864 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698ae865171490b2-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.868 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:25.869 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.870 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:25.870 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.871 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.872 DEBUG:boto:Method: GET 2022-09-02T05:45:25.872 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.873 DEBUG:boto:Data: 2022-09-02T05:45:25.873 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.874 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.875 DEBUG:boto:Params: {} 2022-09-02T05:45:25.875 DEBUG:boto:Token: None 2022-09-02T05:45:25.876 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0= 2022-09-02T05:45:25.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0=', 'Content-Length': '0'} 2022-09-02T05:45:25.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a5574205756ed0-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.880 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.881 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:25.881 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.882 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.883 DEBUG:boto:Method: GET 2022-09-02T05:45:25.883 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.884 DEBUG:boto:Data: 2022-09-02T05:45:25.884 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.885 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.886 DEBUG:boto:Params: {} 2022-09-02T05:45:25.886 DEBUG:boto:Token: None 2022-09-02T05:45:25.887 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0= 2022-09-02T05:45:25.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0=', 'Content-Length': '0'} 2022-09-02T05:45:25.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a5d3e5956063d6-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.891 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.892 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:25.892 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.893 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.893 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.894 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.894 DEBUG:boto:Data: 2022-09-02T05:45:25.895 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.896 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.896 DEBUG:boto:Params: {} 2022-09-02T05:45:25.897 DEBUG:boto:Token: None 2022-09-02T05:45:25.898 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D5ML6tTXpCcy+DOsZ+vJihb9/L0= 2022-09-02T05:45:25.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D5ML6tTXpCcy+DOsZ+vJihb9/L0=', 'Content-Length': '0'} 2022-09-02T05:45:25.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011198e858551d647-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.901 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.902 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:25.902 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.903 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.903 DEBUG:boto:Method: HEAD 2022-09-02T05:45:25.904 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.904 DEBUG:boto:Data: 2022-09-02T05:45:25.905 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.906 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.906 DEBUG:boto:Params: {} 2022-09-02T05:45:25.907 DEBUG:boto:Token: None 2022-09-02T05:45:25.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D5ML6tTXpCcy+DOsZ+vJihb9/L0= 2022-09-02T05:45:25.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D5ML6tTXpCcy+DOsZ+vJihb9/L0=', 'Content-Length': '0'} 2022-09-02T05:45:25.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe9bfecd6ca3114-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:25.912 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.913 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:25.913 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.914 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.914 DEBUG:boto:Method: GET 2022-09-02T05:45:25.915 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.915 DEBUG:boto:Data: 2022-09-02T05:45:25.916 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.917 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.918 DEBUG:boto:Params: {} 2022-09-02T05:45:25.918 DEBUG:boto:Token: None 2022-09-02T05:45:25.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0= 2022-09-02T05:45:25.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0=', 'Content-Length': '0'} 2022-09-02T05:45:25.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1dd9f7a32ae3a7c-0063119875-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.923 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.923 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:25.924 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.924 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.925 DEBUG:boto:Method: GET 2022-09-02T05:45:25.925 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.926 DEBUG:boto:Data: 2022-09-02T05:45:25.926 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.927 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.928 DEBUG:boto:Params: {} 2022-09-02T05:45:25.929 DEBUG:boto:Token: None 2022-09-02T05:45:25.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:25.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0= 2022-09-02T05:45:25.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ekKcJvbDD6iRBjdfXfhzNeG+Q/0=', 'Content-Length': '0'} 2022-09-02T05:45:25.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0d68cace83ad45-0063119875-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.974 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.975 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.975 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:25.976 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:25.976 DEBUG:boto:Method: GET 2022-09-02T05:45:25.977 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:25.977 DEBUG:boto:Data: 2022-09-02T05:45:25.978 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:25.979 DEBUG:boto:Port: 8001 2022-09-02T05:45:25.979 DEBUG:boto:Params: {} 2022-09-02T05:45:25.980 DEBUG:boto:Token: None 2022-09-02T05:45:25.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5da75457b2bd206-0063119875-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.985 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:25.986 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:25.987 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:25.987 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:25.988 DEBUG:boto:Method: GET 2022-09-02T05:45:25.988 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:25.989 DEBUG:boto:Data: 2022-09-02T05:45:25.989 DEBUG:boto:Headers: {} 2022-09-02T05:45:25.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:25.990 DEBUG:boto:Port: 8000 2022-09-02T05:45:25.991 DEBUG:boto:Params: {} 2022-09-02T05:45:25.991 DEBUG:boto:Token: None 2022-09-02T05:45:25.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:25 GMT /xljbkf-74/ 2022-09-02T05:45:25.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4= 2022-09-02T05:45:25.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ebI4Wb3wbTYS634d4Os+YAci+e4=', 'Content-Length': '0'} 2022-09-02T05:45:25.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071772bd847e432b6-0063119875-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:25.997 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:25.998 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:25.999 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:26.000 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.001 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:26.002 DEBUG:boto:Data: 2022-09-02T05:45:26.003 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.005 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.006 DEBUG:boto:Params: {} 2022-09-02T05:45:26.007 DEBUG:boto:Token: None 2022-09-02T05:45:26.008 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj44 2022-09-02T05:45:26.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ez1N/Uz/2X422mSZzXc1LrAjuEg= 2022-09-02T05:45:26.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ez1N/Uz/2X422mSZzXc1LrAjuEg=', 'Content-Length': '0'} 2022-09-02T05:45:26.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170f849b1b57ff1b-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.014 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:26.015 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:26.016 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.017 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:26.018 DEBUG:boto:Data: 2022-09-02T05:45:26.019 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.021 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.022 DEBUG:boto:Params: {} 2022-09-02T05:45:26.023 DEBUG:boto:Token: None 2022-09-02T05:45:26.024 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj44 2022-09-02T05:45:26.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ez1N/Uz/2X422mSZzXc1LrAjuEg= 2022-09-02T05:45:26.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ez1N/Uz/2X422mSZzXc1LrAjuEg=', 'Content-Length': '0'} 2022-09-02T05:45:26.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a573bd458d51fd7d-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:26.031 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:26.032 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:26.033 DEBUG:boto:Method: GET 2022-09-02T05:45:26.034 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:26.035 DEBUG:boto:Data: 2022-09-02T05:45:26.036 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.038 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.039 DEBUG:boto:Params: {} 2022-09-02T05:45:26.040 DEBUG:boto:Token: None 2022-09-02T05:45:26.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj44 2022-09-02T05:45:26.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TbR0j3bplOSOG395cmQ3ZpOahH0= 2022-09-02T05:45:26.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TbR0j3bplOSOG395cmQ3ZpOahH0=', 'Content-Length': '0'} 2022-09-02T05:45:26.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e07615fd4f79c62-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.047 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:26.048 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:26.049 DEBUG:boto:Method: GET 2022-09-02T05:45:26.050 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:26.051 DEBUG:boto:Data: 2022-09-02T05:45:26.052 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.054 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.055 DEBUG:boto:Params: {} 2022-09-02T05:45:26.056 DEBUG:boto:Token: None 2022-09-02T05:45:26.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj44 2022-09-02T05:45:26.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TbR0j3bplOSOG395cmQ3ZpOahH0= 2022-09-02T05:45:26.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TbR0j3bplOSOG395cmQ3ZpOahH0=', 'Content-Length': '0'} 2022-09-02T05:45:26.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10d689a4a6ab315-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.063 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.064 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.064 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:26.065 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:26.065 DEBUG:boto:Method: GET 2022-09-02T05:45:26.066 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:26.066 DEBUG:boto:Data: 2022-09-02T05:45:26.067 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.068 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.069 DEBUG:boto:Params: {} 2022-09-02T05:45:26.069 DEBUG:boto:Token: None 2022-09-02T05:45:26.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e88d41140125cc74-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.074 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.075 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.076 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.076 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:26.077 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:26.077 DEBUG:boto:Method: GET 2022-09-02T05:45:26.078 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:26.078 DEBUG:boto:Data: 2022-09-02T05:45:26.079 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.080 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.080 DEBUG:boto:Params: {} 2022-09-02T05:45:26.081 DEBUG:boto:Token: None 2022-09-02T05:45:26.081 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb2a2df0afdeec4-0063119876-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.086 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:26.087 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.088 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.088 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.089 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.089 DEBUG:boto:Method: GET 2022-09-02T05:45:26.090 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.090 DEBUG:boto:Data: 2022-09-02T05:45:26.091 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.092 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.092 DEBUG:boto:Params: {} 2022-09-02T05:45:26.093 DEBUG:boto:Token: None 2022-09-02T05:45:26.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs= 2022-09-02T05:45:26.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs=', 'Content-Length': '0'} 2022-09-02T05:45:26.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ae922d6546a136-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.098 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.098 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.099 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.099 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.100 DEBUG:boto:Method: GET 2022-09-02T05:45:26.100 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.101 DEBUG:boto:Data: 2022-09-02T05:45:26.101 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.102 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.103 DEBUG:boto:Params: {} 2022-09-02T05:45:26.103 DEBUG:boto:Token: None 2022-09-02T05:45:26.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs= 2022-09-02T05:45:26.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs=', 'Content-Length': '0'} 2022-09-02T05:45:26.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098fbdbe81a57b186-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.108 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.109 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.109 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.110 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.110 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.111 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.111 DEBUG:boto:Data: 2022-09-02T05:45:26.112 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.113 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.113 DEBUG:boto:Params: {} 2022-09-02T05:45:26.114 DEBUG:boto:Token: None 2022-09-02T05:45:26.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v89FugJli6lJFdjKvQT2lTO8nPM= 2022-09-02T05:45:26.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v89FugJli6lJFdjKvQT2lTO8nPM=', 'Content-Length': '0'} 2022-09-02T05:45:26.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6f8065c6f1bb7d-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.118 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.119 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.119 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.120 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.120 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.121 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.121 DEBUG:boto:Data: 2022-09-02T05:45:26.122 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.123 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.123 DEBUG:boto:Params: {} 2022-09-02T05:45:26.124 DEBUG:boto:Token: None 2022-09-02T05:45:26.124 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v89FugJli6lJFdjKvQT2lTO8nPM= 2022-09-02T05:45:26.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v89FugJli6lJFdjKvQT2lTO8nPM=', 'Content-Length': '0'} 2022-09-02T05:45:26.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ed9ac298e30978-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:26.129 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.129 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.130 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.130 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.131 DEBUG:boto:Method: GET 2022-09-02T05:45:26.131 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.132 DEBUG:boto:Data: 2022-09-02T05:45:26.132 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.134 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.134 DEBUG:boto:Params: {} 2022-09-02T05:45:26.135 DEBUG:boto:Token: None 2022-09-02T05:45:26.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs= 2022-09-02T05:45:26.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs=', 'Content-Length': '0'} 2022-09-02T05:45:26.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007167dc87ede113e1-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.139 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.140 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.140 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.141 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.141 DEBUG:boto:Method: GET 2022-09-02T05:45:26.142 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.142 DEBUG:boto:Data: 2022-09-02T05:45:26.143 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.144 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.144 DEBUG:boto:Params: {} 2022-09-02T05:45:26.145 DEBUG:boto:Token: None 2022-09-02T05:45:26.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:26.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs= 2022-09-02T05:45:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uirbKz8P5eo3CBnB+wbFwKSCHFs=', 'Content-Length': '0'} 2022-09-02T05:45:26.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef7fad6310a58f6a-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.150 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.150 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.151 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.151 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.152 DEBUG:boto:Method: GET 2022-09-02T05:45:26.152 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.153 DEBUG:boto:Data: 2022-09-02T05:45:26.153 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.154 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.155 DEBUG:boto:Params: {} 2022-09-02T05:45:26.155 DEBUG:boto:Token: None 2022-09-02T05:45:26.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007821b108f7568b11-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.202 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.203 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.204 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.205 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.205 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.206 DEBUG:boto:Method: GET 2022-09-02T05:45:26.206 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.207 DEBUG:boto:Data: 2022-09-02T05:45:26.208 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.209 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.209 DEBUG:boto:Params: {} 2022-09-02T05:45:26.210 DEBUG:boto:Token: None 2022-09-02T05:45:26.211 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000917895e9419432c1-0063119876-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.215 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.216 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.217 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.217 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.218 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:26.218 DEBUG:boto:Data: 2022-09-02T05:45:26.219 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.220 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.220 DEBUG:boto:Params: {} 2022-09-02T05:45:26.221 DEBUG:boto:Token: None 2022-09-02T05:45:26.221 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55 2022-09-02T05:45:26.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xaW+S77YJEVB64xwRqLAcbvoC0A= 2022-09-02T05:45:26.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xaW+S77YJEVB64xwRqLAcbvoC0A=', 'Content-Length': '0'} 2022-09-02T05:45:26.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1cff818b23cf4e-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.225 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.226 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.226 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.227 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:26.227 DEBUG:boto:Data: 2022-09-02T05:45:26.228 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.229 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.229 DEBUG:boto:Params: {} 2022-09-02T05:45:26.230 DEBUG:boto:Token: None 2022-09-02T05:45:26.230 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55 2022-09-02T05:45:26.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xaW+S77YJEVB64xwRqLAcbvoC0A= 2022-09-02T05:45:26.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xaW+S77YJEVB64xwRqLAcbvoC0A=', 'Content-Length': '0'} 2022-09-02T05:45:26.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf78de1fd29dfa8d-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:26.235 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.235 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.236 DEBUG:boto:Method: GET 2022-09-02T05:45:26.236 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:26.237 DEBUG:boto:Data: 2022-09-02T05:45:26.238 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.239 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.239 DEBUG:boto:Params: {} 2022-09-02T05:45:26.240 DEBUG:boto:Token: None 2022-09-02T05:45:26.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55 2022-09-02T05:45:26.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+kWt2/17nt9/NYeR17i60Z40+Y= 2022-09-02T05:45:26.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+kWt2/17nt9/NYeR17i60Z40+Y=', 'Content-Length': '0'} 2022-09-02T05:45:26.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f89b829770cca15-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.244 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:26.245 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:26.245 DEBUG:boto:Method: GET 2022-09-02T05:45:26.246 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:26.246 DEBUG:boto:Data: 2022-09-02T05:45:26.247 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.248 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.249 DEBUG:boto:Params: {} 2022-09-02T05:45:26.249 DEBUG:boto:Token: None 2022-09-02T05:45:26.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj55 2022-09-02T05:45:26.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+kWt2/17nt9/NYeR17i60Z40+Y= 2022-09-02T05:45:26.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+kWt2/17nt9/NYeR17i60Z40+Y=', 'Content-Length': '0'} 2022-09-02T05:45:26.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b58e0c3c2b656e4-0063119876-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.294 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.295 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.295 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.296 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.297 DEBUG:boto:Method: GET 2022-09-02T05:45:26.297 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.298 DEBUG:boto:Data: 2022-09-02T05:45:26.298 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.299 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.300 DEBUG:boto:Params: {} 2022-09-02T05:45:26.300 DEBUG:boto:Token: None 2022-09-02T05:45:26.301 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.302 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f53ca986ee4ce3b5-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.306 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.307 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.307 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.308 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.308 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.309 DEBUG:boto:Method: GET 2022-09-02T05:45:26.310 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:26.310 DEBUG:boto:Data: 2022-09-02T05:45:26.311 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:26.312 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.312 DEBUG:boto:Params: {} 2022-09-02T05:45:26.313 DEBUG:boto:Token: None 2022-09-02T05:45:26.313 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc7401c5d674aa52-0063119876-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.318 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.319 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:45:26.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:45:26.320 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.320 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.321 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.321 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:26.322 DEBUG:boto:Data: 2022-09-02T05:45:26.323 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.324 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.324 DEBUG:boto:Params: {} 2022-09-02T05:45:26.325 DEBUG:boto:Token: None 2022-09-02T05:45:26.325 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6ony7APpystZ0Huswmn6z7XHyWQ= 2022-09-02T05:45:26.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6ony7APpystZ0Huswmn6z7XHyWQ=', 'Content-Length': '0'} 2022-09-02T05:45:26.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000931e58ff0e529928-0063119876-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.330 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.330 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.331 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.331 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:26.332 DEBUG:boto:Data: 2022-09-02T05:45:26.332 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.334 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.334 DEBUG:boto:Params: {} 2022-09-02T05:45:26.335 DEBUG:boto:Token: None 2022-09-02T05:45:26.335 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6ony7APpystZ0Huswmn6z7XHyWQ= 2022-09-02T05:45:26.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6ony7APpystZ0Huswmn6z7XHyWQ=', 'Content-Length': '0'} 2022-09-02T05:45:26.339 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082c9f68ea78e8263-0063119876-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.340 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:45:26.340 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.341 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.341 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.342 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.342 DEBUG:boto:Method: GET 2022-09-02T05:45:26.343 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.344 DEBUG:boto:Data: 2022-09-02T05:45:26.344 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.345 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.346 DEBUG:boto:Params: {} 2022-09-02T05:45:26.346 DEBUG:boto:Token: None 2022-09-02T05:45:26.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/?versions 2022-09-02T05:45:26.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io= 2022-09-02T05:45:26.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io=', 'Content-Length': '0'} 2022-09-02T05:45:26.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057bc45f9b43c88db-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.351 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:45:26.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:45:26.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:45:26.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:45:26.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:45:26.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:45:26.355 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.356 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.357 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.357 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.358 DEBUG:boto:Method: GET 2022-09-02T05:45:26.358 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.359 DEBUG:boto:Data: 2022-09-02T05:45:26.359 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.360 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.361 DEBUG:boto:Params: {} 2022-09-02T05:45:26.361 DEBUG:boto:Token: None 2022-09-02T05:45:26.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/?versions 2022-09-02T05:45:26.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io= 2022-09-02T05:45:26.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io=', 'Content-Length': '0'} 2022-09-02T05:45:26.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004f8bf83b9e7ed42-0063119876-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.367 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:45:26.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:45:26.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:45:26.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:45:26.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:45:26.370 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.371 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.372 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.372 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.373 DEBUG:boto:Method: GET 2022-09-02T05:45:26.373 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.374 DEBUG:boto:Data: 2022-09-02T05:45:26.374 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.375 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.376 DEBUG:boto:Params: {} 2022-09-02T05:45:26.377 DEBUG:boto:Token: None 2022-09-02T05:45:26.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/?versions 2022-09-02T05:45:26.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io= 2022-09-02T05:45:26.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io=', 'Content-Length': '0'} 2022-09-02T05:45:26.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dbd4c0b16a24e64-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.382 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.383 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.383 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.384 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.384 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.385 DEBUG:boto:Method: GET 2022-09-02T05:45:26.385 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:45:26.386 DEBUG:boto:Data: 2022-09-02T05:45:26.386 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.387 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.388 DEBUG:boto:Params: {} 2022-09-02T05:45:26.388 DEBUG:boto:Token: None 2022-09-02T05:45:26.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/?versions 2022-09-02T05:45:26.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io= 2022-09-02T05:45:26.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KD1ssPCBnx1SuOe+zPP4GXxo0io=', 'Content-Length': '0'} 2022-09-02T05:45:26.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afdb75dda2c32c19-0063119876-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.394 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:26.395 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.396 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.396 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.397 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.397 DEBUG:boto:Method: GET 2022-09-02T05:45:26.398 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.398 DEBUG:boto:Data: 2022-09-02T05:45:26.399 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.400 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.400 DEBUG:boto:Params: {} 2022-09-02T05:45:26.401 DEBUG:boto:Token: None 2022-09-02T05:45:26.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc= 2022-09-02T05:45:26.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc=', 'Content-Length': '0'} 2022-09-02T05:45:26.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d856c1e6ef1187-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.405 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.406 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.407 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.407 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.408 DEBUG:boto:Method: GET 2022-09-02T05:45:26.408 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.409 DEBUG:boto:Data: 2022-09-02T05:45:26.409 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.410 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.411 DEBUG:boto:Params: {} 2022-09-02T05:45:26.411 DEBUG:boto:Token: None 2022-09-02T05:45:26.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc= 2022-09-02T05:45:26.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc=', 'Content-Length': '0'} 2022-09-02T05:45:26.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000458fcdf1b1f56a41-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.458 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.459 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.465 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.465 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.466 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.466 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.467 DEBUG:boto:Data: 2022-09-02T05:45:26.467 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.468 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.469 DEBUG:boto:Params: {} 2022-09-02T05:45:26.469 DEBUG:boto:Token: None 2022-09-02T05:45:26.470 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TRqZgUQgkq638qIDGrkd6sURQJM= 2022-09-02T05:45:26.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TRqZgUQgkq638qIDGrkd6sURQJM=', 'Content-Length': '0'} 2022-09-02T05:45:26.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a82a0c6420e707-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.474 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.474 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.475 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.475 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.476 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.476 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.477 DEBUG:boto:Data: 2022-09-02T05:45:26.477 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.478 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.479 DEBUG:boto:Params: {} 2022-09-02T05:45:26.479 DEBUG:boto:Token: None 2022-09-02T05:45:26.480 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TRqZgUQgkq638qIDGrkd6sURQJM= 2022-09-02T05:45:26.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TRqZgUQgkq638qIDGrkd6sURQJM=', 'Content-Length': '0'} 2022-09-02T05:45:26.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832140f2ebdb2fa4-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:26.484 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.485 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.485 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.486 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.486 DEBUG:boto:Method: GET 2022-09-02T05:45:26.487 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.487 DEBUG:boto:Data: 2022-09-02T05:45:26.488 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.489 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.490 DEBUG:boto:Params: {} 2022-09-02T05:45:26.490 DEBUG:boto:Token: None 2022-09-02T05:45:26.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc= 2022-09-02T05:45:26.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc=', 'Content-Length': '0'} 2022-09-02T05:45:26.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025f58b151f269d4d-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.495 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.495 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.496 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.496 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.497 DEBUG:boto:Method: GET 2022-09-02T05:45:26.498 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.498 DEBUG:boto:Data: 2022-09-02T05:45:26.499 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.500 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.500 DEBUG:boto:Params: {} 2022-09-02T05:45:26.501 DEBUG:boto:Token: None 2022-09-02T05:45:26.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:45:26.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc= 2022-09-02T05:45:26.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ILBdhrDJUNc1mrGmSI3Cf1wulNc=', 'Content-Length': '0'} 2022-09-02T05:45:26.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000621b8798725c78f-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.506 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.507 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.507 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.508 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.508 DEBUG:boto:Method: GET 2022-09-02T05:45:26.509 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.509 DEBUG:boto:Data: 2022-09-02T05:45:26.510 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.511 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.511 DEBUG:boto:Params: {} 2022-09-02T05:45:26.512 DEBUG:boto:Token: None 2022-09-02T05:45:26.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db40650fd457d802-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.562 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.563 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.564 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.564 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.565 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.565 DEBUG:boto:Method: GET 2022-09-02T05:45:26.566 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.566 DEBUG:boto:Data: 2022-09-02T05:45:26.567 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.568 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.569 DEBUG:boto:Params: {} 2022-09-02T05:45:26.569 DEBUG:boto:Token: None 2022-09-02T05:45:26.570 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.571 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bc87749fcfeadb8-0063119876-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.575 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.575 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.576 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.576 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.577 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:26.577 DEBUG:boto:Data: 2022-09-02T05:45:26.578 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.579 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.580 DEBUG:boto:Params: {} 2022-09-02T05:45:26.580 DEBUG:boto:Token: None 2022-09-02T05:45:26.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11 2022-09-02T05:45:26.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HsFkanUrNW7pcYrtMk0rfsc0uoA= 2022-09-02T05:45:26.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HsFkanUrNW7pcYrtMk0rfsc0uoA=', 'Content-Length': '0'} 2022-09-02T05:45:26.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f1ee868f018567-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.585 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.585 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.586 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.586 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:26.587 DEBUG:boto:Data: 2022-09-02T05:45:26.587 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.588 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.589 DEBUG:boto:Params: {} 2022-09-02T05:45:26.589 DEBUG:boto:Token: None 2022-09-02T05:45:26.590 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11 2022-09-02T05:45:26.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HsFkanUrNW7pcYrtMk0rfsc0uoA= 2022-09-02T05:45:26.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HsFkanUrNW7pcYrtMk0rfsc0uoA=', 'Content-Length': '0'} 2022-09-02T05:45:26.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009100db6c5a8a5324-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:45:26.594 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.595 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.596 DEBUG:boto:Method: GET 2022-09-02T05:45:26.596 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:26.597 DEBUG:boto:Data: 2022-09-02T05:45:26.597 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.598 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.599 DEBUG:boto:Params: {} 2022-09-02T05:45:26.599 DEBUG:boto:Token: None 2022-09-02T05:45:26.600 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11 2022-09-02T05:45:26.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hk6gxPoXTgSMEejn/pNnQeMA1t0= 2022-09-02T05:45:26.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hk6gxPoXTgSMEejn/pNnQeMA1t0=', 'Content-Length': '0'} 2022-09-02T05:45:26.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002592f0bb0da9f0e4-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.604 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:45:26.604 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:45:26.605 DEBUG:boto:Method: GET 2022-09-02T05:45:26.605 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:45:26.606 DEBUG:boto:Data: 2022-09-02T05:45:26.607 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.608 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.608 DEBUG:boto:Params: {} 2022-09-02T05:45:26.609 DEBUG:boto:Token: None 2022-09-02T05:45:26.609 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj11 2022-09-02T05:45:26.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hk6gxPoXTgSMEejn/pNnQeMA1t0= 2022-09-02T05:45:26.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hk6gxPoXTgSMEejn/pNnQeMA1t0=', 'Content-Length': '0'} 2022-09-02T05:45:26.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec3dfab481389a22-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.658 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.659 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.659 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.660 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.661 DEBUG:boto:Method: GET 2022-09-02T05:45:26.661 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.662 DEBUG:boto:Data: 2022-09-02T05:45:26.663 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.664 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.664 DEBUG:boto:Params: {} 2022-09-02T05:45:26.665 DEBUG:boto:Token: None 2022-09-02T05:45:26.666 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000510843ab27056a24-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.670 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.671 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.672 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.672 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.673 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.674 DEBUG:boto:Method: GET 2022-09-02T05:45:26.674 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:45:26.675 DEBUG:boto:Data: 2022-09-02T05:45:26.675 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.677 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.678 DEBUG:boto:Params: {} 2022-09-02T05:45:26.678 DEBUG:boto:Token: None 2022-09-02T05:45:26.679 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010dfbf00df6b0c3f-0063119876-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.684 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:26.685 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.686 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.687 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.687 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.688 DEBUG:boto:Method: GET 2022-09-02T05:45:26.688 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.689 DEBUG:boto:Data: 2022-09-02T05:45:26.689 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.690 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.691 DEBUG:boto:Params: {} 2022-09-02T05:45:26.692 DEBUG:boto:Token: None 2022-09-02T05:45:26.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ= 2022-09-02T05:45:26.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ=', 'Content-Length': '0'} 2022-09-02T05:45:26.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004292f7d8f350b8f6-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.697 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.697 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.698 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.698 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.699 DEBUG:boto:Method: GET 2022-09-02T05:45:26.699 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.700 DEBUG:boto:Data: 2022-09-02T05:45:26.700 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.701 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.702 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.703 DEBUG:boto:Params: {} 2022-09-02T05:45:26.703 DEBUG:boto:Token: None 2022-09-02T05:45:26.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ= 2022-09-02T05:45:26.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ=', 'Content-Length': '0'} 2022-09-02T05:45:26.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114f04106febf1ac-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.708 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.708 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.709 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.709 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.710 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.711 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.711 DEBUG:boto:Data: 2022-09-02T05:45:26.712 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.713 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.714 DEBUG:boto:Params: {} 2022-09-02T05:45:26.714 DEBUG:boto:Token: None 2022-09-02T05:45:26.715 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:on3W4vQzKYIMU6Xj5M55pyvNH6g= 2022-09-02T05:45:26.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:on3W4vQzKYIMU6Xj5M55pyvNH6g=', 'Content-Length': '0'} 2022-09-02T05:45:26.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222c7962a2f92452-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.719 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.720 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.720 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.721 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.722 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.722 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.723 DEBUG:boto:Data: 2022-09-02T05:45:26.723 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.724 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.725 DEBUG:boto:Params: {} 2022-09-02T05:45:26.725 DEBUG:boto:Token: None 2022-09-02T05:45:26.726 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:on3W4vQzKYIMU6Xj5M55pyvNH6g= 2022-09-02T05:45:26.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:on3W4vQzKYIMU6Xj5M55pyvNH6g=', 'Content-Length': '0'} 2022-09-02T05:45:26.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6e7b7f2722012fe-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:26.731 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.731 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.732 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.732 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.733 DEBUG:boto:Method: GET 2022-09-02T05:45:26.734 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.734 DEBUG:boto:Data: 2022-09-02T05:45:26.735 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.736 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.737 DEBUG:boto:Params: {} 2022-09-02T05:45:26.738 DEBUG:boto:Token: None 2022-09-02T05:45:26.738 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ= 2022-09-02T05:45:26.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ=', 'Content-Length': '0'} 2022-09-02T05:45:26.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a20ea5de5733c37-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.743 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.743 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.744 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.744 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.745 DEBUG:boto:Method: GET 2022-09-02T05:45:26.745 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.746 DEBUG:boto:Data: 2022-09-02T05:45:26.747 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.748 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.748 DEBUG:boto:Params: {} 2022-09-02T05:45:26.749 DEBUG:boto:Token: None 2022-09-02T05:45:26.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:45:26.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ= 2022-09-02T05:45:26.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POmhsSS7K8Z82SnyxuyDPhPAWcQ=', 'Content-Length': '0'} 2022-09-02T05:45:26.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64a5130b90b59ee-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.754 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.755 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.755 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.756 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.756 DEBUG:boto:Method: GET 2022-09-02T05:45:26.757 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.757 DEBUG:boto:Data: 2022-09-02T05:45:26.758 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.759 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.760 DEBUG:boto:Params: {} 2022-09-02T05:45:26.760 DEBUG:boto:Token: None 2022-09-02T05:45:26.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e4d12eb72f6bdc6-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.765 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.766 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.767 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.767 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.768 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.769 DEBUG:boto:Method: GET 2022-09-02T05:45:26.770 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.770 DEBUG:boto:Data: 2022-09-02T05:45:26.771 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.772 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.772 DEBUG:boto:Params: {} 2022-09-02T05:45:26.773 DEBUG:boto:Token: None 2022-09-02T05:45:26.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8195c57d198d587-0063119876-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.779 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.779 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.780 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.781 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.781 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:26.782 DEBUG:boto:Data: 2022-09-02T05:45:26.782 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.783 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.784 DEBUG:boto:Params: {} 2022-09-02T05:45:26.785 DEBUG:boto:Token: None 2022-09-02T05:45:26.785 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22 2022-09-02T05:45:26.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9otSp0KtzLwx0R9ss8JEKM+BmY4= 2022-09-02T05:45:26.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9otSp0KtzLwx0R9ss8JEKM+BmY4=', 'Content-Length': '0'} 2022-09-02T05:45:26.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016299c19b2e8c5e6-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.789 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.790 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.790 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.791 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:26.791 DEBUG:boto:Data: 2022-09-02T05:45:26.792 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.793 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.794 DEBUG:boto:Params: {} 2022-09-02T05:45:26.794 DEBUG:boto:Token: None 2022-09-02T05:45:26.795 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22 2022-09-02T05:45:26.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9otSp0KtzLwx0R9ss8JEKM+BmY4= 2022-09-02T05:45:26.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9otSp0KtzLwx0R9ss8JEKM+BmY4=', 'Content-Length': '0'} 2022-09-02T05:45:26.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7c3540212761842-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:45:26.799 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.800 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.800 DEBUG:boto:Method: GET 2022-09-02T05:45:26.801 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:26.801 DEBUG:boto:Data: 2022-09-02T05:45:26.802 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.803 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.804 DEBUG:boto:Params: {} 2022-09-02T05:45:26.804 DEBUG:boto:Token: None 2022-09-02T05:45:26.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22 2022-09-02T05:45:26.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J0d3WuvROZuztFi/iheiGlzsCrs= 2022-09-02T05:45:26.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J0d3WuvROZuztFi/iheiGlzsCrs=', 'Content-Length': '0'} 2022-09-02T05:45:26.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000085c2630922e25be-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.809 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:45:26.810 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:45:26.810 DEBUG:boto:Method: GET 2022-09-02T05:45:26.811 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:45:26.811 DEBUG:boto:Data: 2022-09-02T05:45:26.812 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.813 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.814 DEBUG:boto:Params: {} 2022-09-02T05:45:26.814 DEBUG:boto:Token: None 2022-09-02T05:45:26.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj22 2022-09-02T05:45:26.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J0d3WuvROZuztFi/iheiGlzsCrs= 2022-09-02T05:45:26.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J0d3WuvROZuztFi/iheiGlzsCrs=', 'Content-Length': '0'} 2022-09-02T05:45:26.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a37fac669822fe06-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.862 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.863 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.863 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.864 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.865 DEBUG:boto:Method: GET 2022-09-02T05:45:26.865 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.866 DEBUG:boto:Data: 2022-09-02T05:45:26.866 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.867 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.868 DEBUG:boto:Params: {} 2022-09-02T05:45:26.868 DEBUG:boto:Token: None 2022-09-02T05:45:26.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca40a9d08d9a27b7-0063119876-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.874 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.874 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.875 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.876 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.876 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.877 DEBUG:boto:Method: GET 2022-09-02T05:45:26.877 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:45:26.878 DEBUG:boto:Data: 2022-09-02T05:45:26.879 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.880 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.880 DEBUG:boto:Params: {} 2022-09-02T05:45:26.881 DEBUG:boto:Token: None 2022-09-02T05:45:26.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/ 2022-09-02T05:45:26.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI= 2022-09-02T05:45:26.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wGfNqy8KwkU7v8hx20X28glLzI=', 'Content-Length': '0'} 2022-09-02T05:45:26.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b69d2b473696e1ac-0063119876-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.887 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:26.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:26.888 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:26.888 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:26.889 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.889 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.890 DEBUG:boto:Method: GET 2022-09-02T05:45:26.891 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.891 DEBUG:boto:Data: 2022-09-02T05:45:26.892 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.893 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.893 DEBUG:boto:Params: {} 2022-09-02T05:45:26.894 DEBUG:boto:Token: None 2022-09-02T05:45:26.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI= 2022-09-02T05:45:26.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI=', 'Content-Length': '0'} 2022-09-02T05:45:26.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add2c92d1be2ed7e-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.899 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:26.899 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:26.900 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.901 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.901 DEBUG:boto:Method: GET 2022-09-02T05:45:26.902 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.902 DEBUG:boto:Data: 2022-09-02T05:45:26.903 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.904 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.904 DEBUG:boto:Params: {} 2022-09-02T05:45:26.905 DEBUG:boto:Token: None 2022-09-02T05:45:26.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI= 2022-09-02T05:45:26.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI=', 'Content-Length': '0'} 2022-09-02T05:45:26.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c32979b0a348260-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.910 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:26.911 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:26.911 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.912 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.912 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.913 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.913 DEBUG:boto:Data: 2022-09-02T05:45:26.914 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.915 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.915 DEBUG:boto:Params: {} 2022-09-02T05:45:26.916 DEBUG:boto:Token: None 2022-09-02T05:45:26.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZhzF+GLHSRkPcN9y3K7+13AfrXw= 2022-09-02T05:45:26.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZhzF+GLHSRkPcN9y3K7+13AfrXw=', 'Content-Length': '0'} 2022-09-02T05:45:26.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57b924bb198fbe8-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.920 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:26.921 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:26.921 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.922 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.922 DEBUG:boto:Method: HEAD 2022-09-02T05:45:26.923 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.923 DEBUG:boto:Data: 2022-09-02T05:45:26.924 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.925 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.925 DEBUG:boto:Params: {} 2022-09-02T05:45:26.926 DEBUG:boto:Token: None 2022-09-02T05:45:26.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZhzF+GLHSRkPcN9y3K7+13AfrXw= 2022-09-02T05:45:26.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZhzF+GLHSRkPcN9y3K7+13AfrXw=', 'Content-Length': '0'} 2022-09-02T05:45:26.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aa537a5df3cb706-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:26.931 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:26.932 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:26.932 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.933 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.933 DEBUG:boto:Method: GET 2022-09-02T05:45:26.934 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.934 DEBUG:boto:Data: 2022-09-02T05:45:26.935 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.936 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.937 DEBUG:boto:Params: {} 2022-09-02T05:45:26.937 DEBUG:boto:Token: None 2022-09-02T05:45:26.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI= 2022-09-02T05:45:26.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI=', 'Content-Length': '0'} 2022-09-02T05:45:26.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df12e686a7712d54-0063119876-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.982 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:26.983 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:26.983 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.984 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.984 DEBUG:boto:Method: GET 2022-09-02T05:45:26.985 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.985 DEBUG:boto:Data: 2022-09-02T05:45:26.986 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:26.987 DEBUG:boto:Port: 8000 2022-09-02T05:45:26.988 DEBUG:boto:Params: {} 2022-09-02T05:45:26.988 DEBUG:boto:Token: None 2022-09-02T05:45:26.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:26 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:45:26.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI= 2022-09-02T05:45:26.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iUbjVqtUMzW+YDZP4jG6CVWJeuI=', 'Content-Length': '0'} 2022-09-02T05:45:26.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaccf3c3bffe3d65-0063119876-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:26.993 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:26.994 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:26.994 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:26.995 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:26.995 DEBUG:boto:Method: GET 2022-09-02T05:45:26.996 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:26.996 DEBUG:boto:Data: 2022-09-02T05:45:26.997 DEBUG:boto:Headers: {} 2022-09-02T05:45:26.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:26.998 DEBUG:boto:Port: 8001 2022-09-02T05:45:26.999 DEBUG:boto:Params: {} 2022-09-02T05:45:26.999 DEBUG:boto:Token: None 2022-09-02T05:45:27.000 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bb11841209ae57d-0063119877-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.004 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.005 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.006 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.006 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.007 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.007 DEBUG:boto:Method: GET 2022-09-02T05:45:27.008 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.008 DEBUG:boto:Data: 2022-09-02T05:45:27.009 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.010 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.010 DEBUG:boto:Params: {} 2022-09-02T05:45:27.011 DEBUG:boto:Token: None 2022-09-02T05:45:27.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002800f1830bdc8532-0063119877-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.016 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.017 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:27.017 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:27.018 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.018 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:27.019 DEBUG:boto:Data: 2022-09-02T05:45:27.019 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.020 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.021 DEBUG:boto:Params: {} 2022-09-02T05:45:27.021 DEBUG:boto:Token: None 2022-09-02T05:45:27.022 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj33 2022-09-02T05:45:27.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XNfvXK4Itvxs1lwd7aNU74YudkQ= 2022-09-02T05:45:27.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XNfvXK4Itvxs1lwd7aNU74YudkQ=', 'Content-Length': '0'} 2022-09-02T05:45:27.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000188158b745c9e654-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.026 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:27.026 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:27.027 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.027 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:27.028 DEBUG:boto:Data: 2022-09-02T05:45:27.028 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.029 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.029 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.030 DEBUG:boto:Params: {} 2022-09-02T05:45:27.030 DEBUG:boto:Token: None 2022-09-02T05:45:27.031 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj33 2022-09-02T05:45:27.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XNfvXK4Itvxs1lwd7aNU74YudkQ= 2022-09-02T05:45:27.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XNfvXK4Itvxs1lwd7aNU74YudkQ=', 'Content-Length': '0'} 2022-09-02T05:45:27.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0129bab6b0c83de-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:45:27.035 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:27.036 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:27.036 DEBUG:boto:Method: GET 2022-09-02T05:45:27.037 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:27.038 DEBUG:boto:Data: 2022-09-02T05:45:27.038 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.039 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.040 DEBUG:boto:Params: {} 2022-09-02T05:45:27.040 DEBUG:boto:Token: None 2022-09-02T05:45:27.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj33 2022-09-02T05:45:27.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C2SiHZSyiHeYhspFoNR3/fosqcI= 2022-09-02T05:45:27.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C2SiHZSyiHeYhspFoNR3/fosqcI=', 'Content-Length': '0'} 2022-09-02T05:45:27.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c57692e0c54ec0-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.045 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:45:27.045 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:45:27.046 DEBUG:boto:Method: GET 2022-09-02T05:45:27.046 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:45:27.047 DEBUG:boto:Data: 2022-09-02T05:45:27.047 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.048 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.049 DEBUG:boto:Params: {} 2022-09-02T05:45:27.049 DEBUG:boto:Token: None 2022-09-02T05:45:27.050 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj33 2022-09-02T05:45:27.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C2SiHZSyiHeYhspFoNR3/fosqcI= 2022-09-02T05:45:27.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C2SiHZSyiHeYhspFoNR3/fosqcI=', 'Content-Length': '0'} 2022-09-02T05:45:27.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6671285584927e9-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.098 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.099 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.099 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.100 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.101 DEBUG:boto:Method: GET 2022-09-02T05:45:27.101 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.102 DEBUG:boto:Data: 2022-09-02T05:45:27.102 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.103 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.104 DEBUG:boto:Params: {} 2022-09-02T05:45:27.104 DEBUG:boto:Token: None 2022-09-02T05:45:27.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f31f2186b52b918a-0063119877-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.109 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.110 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.111 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.111 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.112 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.113 DEBUG:boto:Method: GET 2022-09-02T05:45:27.113 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:45:27.114 DEBUG:boto:Data: 2022-09-02T05:45:27.114 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.115 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.116 DEBUG:boto:Params: {} 2022-09-02T05:45:27.116 DEBUG:boto:Token: None 2022-09-02T05:45:27.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016503fda03df5b8d-0063119877-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.122 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:27.123 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.124 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.125 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.125 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.126 DEBUG:boto:Method: GET 2022-09-02T05:45:27.126 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.127 DEBUG:boto:Data: 2022-09-02T05:45:27.127 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.129 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.129 DEBUG:boto:Params: {} 2022-09-02T05:45:27.130 DEBUG:boto:Token: None 2022-09-02T05:45:27.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8= 2022-09-02T05:45:27.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8=', 'Content-Length': '0'} 2022-09-02T05:45:27.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f86a4616345bbf-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.135 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.135 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.136 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.137 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.138 DEBUG:boto:Method: GET 2022-09-02T05:45:27.139 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.140 DEBUG:boto:Data: 2022-09-02T05:45:27.141 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.143 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.144 DEBUG:boto:Params: {} 2022-09-02T05:45:27.145 DEBUG:boto:Token: None 2022-09-02T05:45:27.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8= 2022-09-02T05:45:27.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8=', 'Content-Length': '0'} 2022-09-02T05:45:27.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c5bf95a834a0b5-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.151 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.152 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.153 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.153 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.154 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.154 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.155 DEBUG:boto:Data: 2022-09-02T05:45:27.155 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.157 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.158 DEBUG:boto:Params: {} 2022-09-02T05:45:27.158 DEBUG:boto:Token: None 2022-09-02T05:45:27.159 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H6iBYxh1FR8YxoLOYRFp9l35kl4= 2022-09-02T05:45:27.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H6iBYxh1FR8YxoLOYRFp9l35kl4=', 'Content-Length': '0'} 2022-09-02T05:45:27.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e523c6c40b117dd4-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.163 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.163 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.164 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.165 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.165 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.166 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.167 DEBUG:boto:Data: 2022-09-02T05:45:27.167 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.168 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.168 DEBUG:boto:Params: {} 2022-09-02T05:45:27.169 DEBUG:boto:Token: None 2022-09-02T05:45:27.170 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H6iBYxh1FR8YxoLOYRFp9l35kl4= 2022-09-02T05:45:27.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H6iBYxh1FR8YxoLOYRFp9l35kl4=', 'Content-Length': '0'} 2022-09-02T05:45:27.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db379702505b5f92-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:27.174 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.175 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.176 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.176 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.177 DEBUG:boto:Method: GET 2022-09-02T05:45:27.177 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.178 DEBUG:boto:Data: 2022-09-02T05:45:27.178 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.179 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.180 DEBUG:boto:Params: {} 2022-09-02T05:45:27.180 DEBUG:boto:Token: None 2022-09-02T05:45:27.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8= 2022-09-02T05:45:27.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8=', 'Content-Length': '0'} 2022-09-02T05:45:27.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e04a73a8eebe87f-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.185 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.186 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.186 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.187 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.187 DEBUG:boto:Method: GET 2022-09-02T05:45:27.188 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.188 DEBUG:boto:Data: 2022-09-02T05:45:27.189 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.190 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.190 DEBUG:boto:Params: {} 2022-09-02T05:45:27.191 DEBUG:boto:Token: None 2022-09-02T05:45:27.192 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:45:27.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8= 2022-09-02T05:45:27.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:92VsYG16epZGJ/mdAUjhfaTlTT8=', 'Content-Length': '0'} 2022-09-02T05:45:27.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e85ea16d980d733-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.196 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.196 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.197 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.197 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.198 DEBUG:boto:Method: GET 2022-09-02T05:45:27.199 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.199 DEBUG:boto:Data: 2022-09-02T05:45:27.200 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.201 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.201 DEBUG:boto:Params: {} 2022-09-02T05:45:27.202 DEBUG:boto:Token: None 2022-09-02T05:45:27.202 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e8d55ffa8139a7-0063119877-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.207 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.208 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.208 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.209 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.209 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.210 DEBUG:boto:Method: GET 2022-09-02T05:45:27.210 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.211 DEBUG:boto:Data: 2022-09-02T05:45:27.211 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.212 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.213 DEBUG:boto:Params: {} 2022-09-02T05:45:27.213 DEBUG:boto:Token: None 2022-09-02T05:45:27.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33089c042886a33-0063119877-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.219 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.220 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.220 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.221 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.221 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:27.222 DEBUG:boto:Data: 2022-09-02T05:45:27.222 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.223 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.224 DEBUG:boto:Params: {} 2022-09-02T05:45:27.224 DEBUG:boto:Token: None 2022-09-02T05:45:27.225 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44 2022-09-02T05:45:27.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a5Ea9lAWEVBb5RQNfC+QElR8MTw= 2022-09-02T05:45:27.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a5Ea9lAWEVBb5RQNfC+QElR8MTw=', 'Content-Length': '0'} 2022-09-02T05:45:27.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb49a577530bc25-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.228 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.229 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.230 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.230 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:27.231 DEBUG:boto:Data: 2022-09-02T05:45:27.231 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.232 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.233 DEBUG:boto:Params: {} 2022-09-02T05:45:27.233 DEBUG:boto:Token: None 2022-09-02T05:45:27.234 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44 2022-09-02T05:45:27.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a5Ea9lAWEVBb5RQNfC+QElR8MTw= 2022-09-02T05:45:27.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a5Ea9lAWEVBb5RQNfC+QElR8MTw=', 'Content-Length': '0'} 2022-09-02T05:45:27.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d732cefb34b964c-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:45:27.238 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.239 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.239 DEBUG:boto:Method: GET 2022-09-02T05:45:27.240 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:27.240 DEBUG:boto:Data: 2022-09-02T05:45:27.241 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.242 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.242 DEBUG:boto:Params: {} 2022-09-02T05:45:27.243 DEBUG:boto:Token: None 2022-09-02T05:45:27.244 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44 2022-09-02T05:45:27.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f9W7AuIbcSYja8ZL7NiT8SGS7wU= 2022-09-02T05:45:27.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f9W7AuIbcSYja8ZL7NiT8SGS7wU=', 'Content-Length': '0'} 2022-09-02T05:45:27.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d097d7b87f4b2b48-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.290 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:45:27.291 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:45:27.291 DEBUG:boto:Method: GET 2022-09-02T05:45:27.292 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:45:27.292 DEBUG:boto:Data: 2022-09-02T05:45:27.293 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.293 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.294 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.295 DEBUG:boto:Params: {} 2022-09-02T05:45:27.295 DEBUG:boto:Token: None 2022-09-02T05:45:27.296 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj44 2022-09-02T05:45:27.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f9W7AuIbcSYja8ZL7NiT8SGS7wU= 2022-09-02T05:45:27.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f9W7AuIbcSYja8ZL7NiT8SGS7wU=', 'Content-Length': '0'} 2022-09-02T05:45:27.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008448e7670b7eec01-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.300 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.301 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.301 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.302 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.302 DEBUG:boto:Method: GET 2022-09-02T05:45:27.303 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.303 DEBUG:boto:Data: 2022-09-02T05:45:27.304 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.305 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.305 DEBUG:boto:Params: {} 2022-09-02T05:45:27.306 DEBUG:boto:Token: None 2022-09-02T05:45:27.307 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002be8b94ac8175f5e-0063119877-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.311 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.312 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.312 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.313 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.313 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.314 DEBUG:boto:Method: GET 2022-09-02T05:45:27.315 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:45:27.315 DEBUG:boto:Data: 2022-09-02T05:45:27.316 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.317 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.317 DEBUG:boto:Params: {} 2022-09-02T05:45:27.318 DEBUG:boto:Token: None 2022-09-02T05:45:27.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007819910bcc9ed4e7-0063119877-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.323 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:27.324 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.325 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.325 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.326 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.327 DEBUG:boto:Method: GET 2022-09-02T05:45:27.327 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.328 DEBUG:boto:Data: 2022-09-02T05:45:27.328 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.329 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.330 DEBUG:boto:Params: {} 2022-09-02T05:45:27.331 DEBUG:boto:Token: None 2022-09-02T05:45:27.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI= 2022-09-02T05:45:27.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI=', 'Content-Length': '0'} 2022-09-02T05:45:27.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005c1018eed96e7e3-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.335 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.336 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.336 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.337 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.338 DEBUG:boto:Method: GET 2022-09-02T05:45:27.338 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.339 DEBUG:boto:Data: 2022-09-02T05:45:27.339 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.341 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.341 DEBUG:boto:Params: {} 2022-09-02T05:45:27.342 DEBUG:boto:Token: None 2022-09-02T05:45:27.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI= 2022-09-02T05:45:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI=', 'Content-Length': '0'} 2022-09-02T05:45:27.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049487e8dc7b84f2a-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.346 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.347 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.347 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.348 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.348 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.349 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.349 DEBUG:boto:Data: 2022-09-02T05:45:27.350 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.351 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.351 DEBUG:boto:Params: {} 2022-09-02T05:45:27.352 DEBUG:boto:Token: None 2022-09-02T05:45:27.353 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ywz6P48f7GehTtpyllMDjF8RjGc= 2022-09-02T05:45:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ywz6P48f7GehTtpyllMDjF8RjGc=', 'Content-Length': '0'} 2022-09-02T05:45:27.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce370a1fc6443c4-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.357 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.357 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.358 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.358 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.359 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.359 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.360 DEBUG:boto:Data: 2022-09-02T05:45:27.361 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.362 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.362 DEBUG:boto:Params: {} 2022-09-02T05:45:27.363 DEBUG:boto:Token: None 2022-09-02T05:45:27.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ywz6P48f7GehTtpyllMDjF8RjGc= 2022-09-02T05:45:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ywz6P48f7GehTtpyllMDjF8RjGc=', 'Content-Length': '0'} 2022-09-02T05:45:27.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b85c850aa6371a-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:27.368 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.369 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.369 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.370 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.371 DEBUG:boto:Method: GET 2022-09-02T05:45:27.371 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.372 DEBUG:boto:Data: 2022-09-02T05:45:27.372 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.374 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.374 DEBUG:boto:Params: {} 2022-09-02T05:45:27.375 DEBUG:boto:Token: None 2022-09-02T05:45:27.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.376 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI= 2022-09-02T05:45:27.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI=', 'Content-Length': '0'} 2022-09-02T05:45:27.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030aab69d6ccc173e-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.422 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.423 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.424 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.424 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.425 DEBUG:boto:Method: GET 2022-09-02T05:45:27.425 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.426 DEBUG:boto:Data: 2022-09-02T05:45:27.427 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.428 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.429 DEBUG:boto:Params: {} 2022-09-02T05:45:27.429 DEBUG:boto:Token: None 2022-09-02T05:45:27.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:45:27.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI= 2022-09-02T05:45:27.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mpwk28P72o63g7V7rtQo2WaKFiI=', 'Content-Length': '0'} 2022-09-02T05:45:27.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0c2deeeb0901be-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.435 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.436 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.437 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.438 DEBUG:boto:Method: GET 2022-09-02T05:45:27.438 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.439 DEBUG:boto:Data: 2022-09-02T05:45:27.440 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.441 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.441 DEBUG:boto:Params: {} 2022-09-02T05:45:27.442 DEBUG:boto:Token: None 2022-09-02T05:45:27.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0f35e9c4ff73cb-0063119877-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.447 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.448 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.448 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.449 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.449 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.450 DEBUG:boto:Method: GET 2022-09-02T05:45:27.450 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.451 DEBUG:boto:Data: 2022-09-02T05:45:27.451 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.453 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.453 DEBUG:boto:Params: {} 2022-09-02T05:45:27.454 DEBUG:boto:Token: None 2022-09-02T05:45:27.454 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053292d6c1c26c11d-0063119877-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.459 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.460 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.460 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.461 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.461 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:27.462 DEBUG:boto:Data: 2022-09-02T05:45:27.462 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.463 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.464 DEBUG:boto:Params: {} 2022-09-02T05:45:27.464 DEBUG:boto:Token: None 2022-09-02T05:45:27.465 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55 2022-09-02T05:45:27.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n1as9Rv4O/O0cO7r1429O9U67Zk= 2022-09-02T05:45:27.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n1as9Rv4O/O0cO7r1429O9U67Zk=', 'Content-Length': '0'} 2022-09-02T05:45:27.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb1e39472e48b59-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.469 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.469 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.470 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.470 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:27.471 DEBUG:boto:Data: 2022-09-02T05:45:27.471 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.472 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.473 DEBUG:boto:Params: {} 2022-09-02T05:45:27.474 DEBUG:boto:Token: None 2022-09-02T05:45:27.474 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55 2022-09-02T05:45:27.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n1as9Rv4O/O0cO7r1429O9U67Zk= 2022-09-02T05:45:27.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n1as9Rv4O/O0cO7r1429O9U67Zk=', 'Content-Length': '0'} 2022-09-02T05:45:27.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb884c8d94af4dd-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:45:27.480 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.480 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.481 DEBUG:boto:Method: GET 2022-09-02T05:45:27.481 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:27.482 DEBUG:boto:Data: 2022-09-02T05:45:27.483 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.484 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.485 DEBUG:boto:Params: {} 2022-09-02T05:45:27.485 DEBUG:boto:Token: None 2022-09-02T05:45:27.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55 2022-09-02T05:45:27.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NLbOnU86JJ745BUD559hesypj0s= 2022-09-02T05:45:27.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NLbOnU86JJ745BUD559hesypj0s=', 'Content-Length': '0'} 2022-09-02T05:45:27.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c9f0f642af39c2-0063119877-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.491 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:45:27.491 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:45:27.492 DEBUG:boto:Method: GET 2022-09-02T05:45:27.492 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:45:27.493 DEBUG:boto:Data: 2022-09-02T05:45:27.493 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.495 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.495 DEBUG:boto:Params: {} 2022-09-02T05:45:27.496 DEBUG:boto:Token: None 2022-09-02T05:45:27.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj55 2022-09-02T05:45:27.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NLbOnU86JJ745BUD559hesypj0s= 2022-09-02T05:45:27.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NLbOnU86JJ745BUD559hesypj0s=', 'Content-Length': '0'} 2022-09-02T05:45:27.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1ae62d01905408-0063119877-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.542 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.543 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.543 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.544 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.545 DEBUG:boto:Method: GET 2022-09-02T05:45:27.545 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.546 DEBUG:boto:Data: 2022-09-02T05:45:27.546 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:45:27.548 DEBUG:boto:Port: 8001 2022-09-02T05:45:27.548 DEBUG:boto:Params: {} 2022-09-02T05:45:27.549 DEBUG:boto:Token: None 2022-09-02T05:45:27.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005abcbb9d4e9137b2-0063119877-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.554 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.555 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.556 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.556 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.557 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.557 DEBUG:boto:Method: GET 2022-09-02T05:45:27.558 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:45:27.559 DEBUG:boto:Data: 2022-09-02T05:45:27.559 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:45:27.560 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.561 DEBUG:boto:Params: {} 2022-09-02T05:45:27.561 DEBUG:boto:Token: None 2022-09-02T05:45:27.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY= 2022-09-02T05:45:27.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Ek0zECHYturWEeL56jA5VxYlhY=', 'Content-Length': '0'} 2022-09-02T05:45:27.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2ccd7c9e1a63784-0063119877-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.567 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:45:27.567 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:45:27.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:45:27.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:45:27.569 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:45:27.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:27.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:27.691 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:45:27.691 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-74 2022-09-02T05:45:27.695 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2022-09-02T05:45:27.762 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.763 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.763 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.764 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:27.765 DEBUG:boto:Data: 2022-09-02T05:45:27.765 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.766 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.767 DEBUG:boto:Params: {} 2022-09-02T05:45:27.767 DEBUG:boto:Token: None 2022-09-02T05:45:27.768 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k= 2022-09-02T05:45:27.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k=', 'Content-Length': '0'} 2022-09-02T05:45:27.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002514207c04730b1-0063119877-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.772 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:45:27.773 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:45:27.774 DEBUG:boto:Method: PUT 2022-09-02T05:45:27.774 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:45:27.775 DEBUG:boto:Data: 2022-09-02T05:45:27.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:45:27.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.776 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.777 DEBUG:boto:Params: {} 2022-09-02T05:45:27.777 DEBUG:boto:Token: None 2022-09-02T05:45:27.778 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj17 2022-09-02T05:45:27.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jcxtFJJ6KfBS/3EXbElWe/Gd+38= 2022-09-02T05:45:27.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jcxtFJJ6KfBS/3EXbElWe/Gd+38='} 2022-09-02T05:45:27.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008440de136c663db0-0063119877-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.787 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.788 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:27.789 DEBUG:boto:Data: 2022-09-02T05:45:27.789 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.790 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.791 DEBUG:boto:Params: {} 2022-09-02T05:45:27.791 DEBUG:boto:Token: None 2022-09-02T05:45:27.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k= 2022-09-02T05:45:27.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k=', 'Content-Length': '0'} 2022-09-02T05:45:27.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050be8d77597a6476-0063119877-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.796 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:45:27.797 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:45:27.797 DEBUG:boto:Method: PUT 2022-09-02T05:45:27.798 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:45:27.799 DEBUG:boto:Data: 2022-09-02T05:45:27.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:45:27.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.800 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.801 DEBUG:boto:Params: {} 2022-09-02T05:45:27.801 DEBUG:boto:Token: None 2022-09-02T05:45:27.802 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj34 2022-09-02T05:45:27.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aSibCr+hbMbBTE5Jgk2w3msnkb0= 2022-09-02T05:45:27.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aSibCr+hbMbBTE5Jgk2w3msnkb0='} 2022-09-02T05:45:27.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3807b5182410ef5-0063119877-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.811 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.811 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.812 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.812 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:27.813 DEBUG:boto:Data: 2022-09-02T05:45:27.813 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.814 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.815 DEBUG:boto:Params: {} 2022-09-02T05:45:27.815 DEBUG:boto:Token: None 2022-09-02T05:45:27.816 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.816 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k= 2022-09-02T05:45:27.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k=', 'Content-Length': '0'} 2022-09-02T05:45:27.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1340ce53ba79bbc-0063119877-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.820 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:45:27.821 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:45:27.821 DEBUG:boto:Method: PUT 2022-09-02T05:45:27.822 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:45:27.822 DEBUG:boto:Data: 2022-09-02T05:45:27.823 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:45:27.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.824 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.824 DEBUG:boto:Params: {} 2022-09-02T05:45:27.825 DEBUG:boto:Token: None 2022-09-02T05:45:27.825 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj51 2022-09-02T05:45:27.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eHoBcmO+W+YupI6DgZ+wDJk3QoU= 2022-09-02T05:45:27.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eHoBcmO+W+YupI6DgZ+wDJk3QoU='} 2022-09-02T05:45:27.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d20e98f2309060b-0063119877-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.875 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.876 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.876 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.877 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:27.877 DEBUG:boto:Data: 2022-09-02T05:45:27.878 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.879 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.879 DEBUG:boto:Params: {} 2022-09-02T05:45:27.880 DEBUG:boto:Token: None 2022-09-02T05:45:27.880 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k= 2022-09-02T05:45:27.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k=', 'Content-Length': '0'} 2022-09-02T05:45:27.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021a0f64b96907a1d-0063119877-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.885 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:45:27.886 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:45:27.886 DEBUG:boto:Method: PUT 2022-09-02T05:45:27.887 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:45:27.887 DEBUG:boto:Data: 2022-09-02T05:45:27.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:45:27.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.889 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.889 DEBUG:boto:Params: {} 2022-09-02T05:45:27.890 DEBUG:boto:Token: None 2022-09-02T05:45:27.890 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj68 2022-09-02T05:45:27.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yW4pOT6pN7F+tLCP3g2eDteHHl8= 2022-09-02T05:45:27.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yW4pOT6pN7F+tLCP3g2eDteHHl8='} 2022-09-02T05:45:27.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b3fd643eb1a0f70-0063119877-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.898 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:45:27.899 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:45:27.899 DEBUG:boto:Method: HEAD 2022-09-02T05:45:27.900 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:45:27.900 DEBUG:boto:Data: 2022-09-02T05:45:27.901 DEBUG:boto:Headers: {} 2022-09-02T05:45:27.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.902 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.902 DEBUG:boto:Params: {} 2022-09-02T05:45:27.903 DEBUG:boto:Token: None 2022-09-02T05:45:27.904 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/ 2022-09-02T05:45:27.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k= 2022-09-02T05:45:27.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EeA446C6C4BLyVWBjnl+0xScu9k=', 'Content-Length': '0'} 2022-09-02T05:45:27.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f721c911b9c2f78-0063119877-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.908 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:45:27.909 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:45:27.909 DEBUG:boto:Method: PUT 2022-09-02T05:45:27.910 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:45:27.910 DEBUG:boto:Data: 2022-09-02T05:45:27.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:45:27.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:45:27.912 DEBUG:boto:Port: 8000 2022-09-02T05:45:27.912 DEBUG:boto:Params: {} 2022-09-02T05:45:27.913 DEBUG:boto:Token: None 2022-09-02T05:45:27.914 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:45:27 GMT /xljbkf-74/obj85 2022-09-02T05:45:27.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCxtaiOJE32swawlaKxlQepyyVs= 2022-09-02T05:45:27.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:45:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCxtaiOJE32swawlaKxlQepyyVs='} 2022-09-02T05:45:27.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae01bf42b1631065-0063119877-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:45:27.927 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:45:27.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:27.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:45:27.928 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:45:27.992 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:45:27.993 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:45:28.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:28.039+0000 7fae705df600 1 waiting to reach latest gen 1, on 0.. 2022-09-02T05:45:33.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:33.039+0000 7fae705df600 1 waiting to reach latest gen 1, on 0.. 2022-09-02T05:45:38.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:38.040+0000 7fae705df600 1 waiting to reach latest gen 1, on 0.. 2022-09-02T05:45:43.041 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:43.041+0000 7fae705df600 1 waiting to reach latest gen 1, on 0.. 2022-09-02T05:45:48.043 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:48.043+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:45:48.044 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-02T05:45:48.044 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:45:53.045 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:53.045+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:45:53.046 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-02T05:45:53.046 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:45:58.047 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:45:58.047+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:45:58.047 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-02T05:45:58.047 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:03.049 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:03.049+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:03.050 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , ] 2022-09-02T05:46:03.051 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:08.050 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:08.049+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:08.050 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:08.051 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:13.052 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:13.051+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:13.053 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:13.053 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:18.053 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:18.053+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:18.054 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:18.054 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:23.055 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:23.054+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:23.055 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:23.056 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:28.057 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:28.056+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:28.058 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:28.058 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:33.058 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:33.058+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:33.059 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:33.059 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:38.060 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:38.059+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:38.060 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:38.061 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:43.062 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:43.061+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:43.062 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , , , , , , 00000000001.57.5, , , ] 2022-09-02T05:46:43.063 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:48.064 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:48.063+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:48.064 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:48.064 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:53.065 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:53.064+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:53.066 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:53.067 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:58.067 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:46:58.067+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:46:58.067 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:46:58.068 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:03.068 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:03.069+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:03.069 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:03.069 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:08.070 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:08.070+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:08.071 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:08.071 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:13.071 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:13.071+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:13.072 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:13.072 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:18.074 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:18.072+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:18.075 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:18.075 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:23.075 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:23.074+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:23.076 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:23.076 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:28.077 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:28.076+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:28.078 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:28.079 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:33.078 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:33.077+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:33.079 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:33.079 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:38.080 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:38.079+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:38.081 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:38.081 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:43.082 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:43.081+0000 7fae705df600 1 waiting for incremental sync to catch up: 2022-09-02T05:47:43.083 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:43.083 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.083 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:48.082+0000 7fae705df600 1 bucket sync caught up with source: 2022-09-02T05:47:48.084 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.084 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.084 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:48.082+0000 7fae705df600 0 bucket checkpoint complete 2022-09-02T05:47:48.106 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:47:48.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.109 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:48.229 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.228+0000 7ff907a65600 1 bucket sync caught up with source: 2022-09-02T05:47:48.229 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.230 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.230 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.228+0000 7ff907a65600 0 bucket checkpoint complete 2022-09-02T05:47:48.244 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:47:48.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.247 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:48.311 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:48.311 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:48.363 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:48.362+0000 7efea3b70600 1 bucket sync caught up with source: 2022-09-02T05:47:48.364 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.364 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000001.50.5, , , , , , 00000000001.63.5, , 00000000001.52.5, , , , 00000000001.55.5, 00000000001.57.5, , , ] 2022-09-02T05:47:48.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:48.362+0000 7efea3b70600 0 bucket checkpoint complete 2022-09-02T05:47:48.380 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:47:48.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.382 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:48.440 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:48.440 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.475+0000 7f71d86cc600 1 bucket sync caught up with source: 2022-09-02T05:47:48.476 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.466.5, 00000000001.399.5, , , 00000000001.401.5, , , 00000000001.538.5, 00000000003.1196.5, 00000000003.829.5] 2022-09-02T05:47:48.476 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.466.5, 00000000001.399.5, , , 00000000001.401.5, , , 00000000001.538.5, 00000000003.1196.5, 00000000003.829.5] 2022-09-02T05:47:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.475+0000 7f71d86cc600 0 bucket checkpoint complete 2022-09-02T05:47:48.489 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:47:48.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.491 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:48.546 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:48.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:48.584 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.583+0000 7f4dede2d600 1 bucket sync caught up with source: 2022-09-02T05:47:48.585 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.466.5, 00000000001.399.5, , , 00000000001.401.5, , , 00000000001.538.5, 00000000003.1196.5, 00000000003.829.5] 2022-09-02T05:47:48.585 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.466.5, 00000000001.399.5, , , 00000000001.401.5, , , 00000000001.538.5, 00000000003.1196.5, 00000000003.829.5] 2022-09-02T05:47:48.585 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.583+0000 7f4dede2d600 0 bucket checkpoint complete 2022-09-02T05:47:48.605 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:47:48.606 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:47:48.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:48.665 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:48.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:48.716 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.715+0000 7ff7f08d4600 1 bucket sync caught up with source: 2022-09-02T05:47:48.717 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.88.5, 00000000001.378.5, , , 00000000001.380.5, , , 00000000001.193.5, 00000000003.326.5, 00000000003.518.5] 2022-09-02T05:47:48.717 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.88.5, 00000000001.378.5, , , 00000000001.380.5, , , 00000000001.193.5, 00000000003.326.5, 00000000003.518.5] 2022-09-02T05:47:48.717 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:47:48.715+0000 7ff7f08d4600 0 bucket checkpoint complete 2022-09-02T05:47:48.731 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:47:48.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:48.733 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:48.790 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:48.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:48.838 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:48.837+0000 7f2981a93600 1 bucket sync caught up with source: 2022-09-02T05:47:48.838 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000001.88.5, 00000000001.378.5, , , 00000000001.380.5, , , 00000000001.193.5, 00000000003.326.5, 00000000003.518.5] 2022-09-02T05:47:48.839 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000001.88.5, 00000000001.378.5, , , 00000000001.380.5, , , 00000000001.193.5, 00000000003.326.5, 00000000003.518.5] 2022-09-02T05:47:48.839 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:48.837+0000 7f2981a93600 0 bucket checkpoint complete 2022-09-02T05:47:48.857 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:47:48.858 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:47:48.860 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:48.861 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:48.862 DEBUG:boto:Method: HEAD 2022-09-02T05:47:48.863 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:48.864 DEBUG:boto:Data: 2022-09-02T05:47:48.865 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:48.867 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.868 DEBUG:boto:Params: {} 2022-09-02T05:47:48.870 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:47:48.871 DEBUG:boto:Token: None 2022-09-02T05:47:48.872 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/ 2022-09-02T05:47:48.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rd9xk3Gv6LHi7KdEprWFuqeVGLc= 2022-09-02T05:47:48.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rd9xk3Gv6LHi7KdEprWFuqeVGLc=', 'Content-Length': '0'} 2022-09-02T05:47:48.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c01ce9d45421978-0063119904-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.879 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:48.879 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:48.880 DEBUG:boto:Method: HEAD 2022-09-02T05:47:48.881 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:48.881 DEBUG:boto:Data: 2022-09-02T05:47:48.882 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:48.883 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.884 DEBUG:boto:Params: {} 2022-09-02T05:47:48.885 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:47:48.885 DEBUG:boto:Token: None 2022-09-02T05:47:48.886 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/ 2022-09-02T05:47:48.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rd9xk3Gv6LHi7KdEprWFuqeVGLc= 2022-09-02T05:47:48.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rd9xk3Gv6LHi7KdEprWFuqeVGLc=', 'Content-Length': '0'} 2022-09-02T05:47:48.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e956f38ef1e0b26-0063119904-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.892 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:47:48.892 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:48.893 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:48.893 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.894 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.895 DEBUG:boto:Method: GET 2022-09-02T05:47:48.895 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.896 DEBUG:boto:Data: 2022-09-02T05:47:48.897 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.897 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:48.898 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.898 DEBUG:boto:Params: {} 2022-09-02T05:47:48.899 DEBUG:boto:Token: None 2022-09-02T05:47:48.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/?versions 2022-09-02T05:47:48.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs= 2022-09-02T05:47:48.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs=', 'Content-Length': '0'} 2022-09-02T05:47:48.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb811d3f4577105-0063119904-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.905 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:48.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:47:48.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:47:48.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:47:48.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:47:48.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:47:48.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:47:48.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:47:48.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:47:48.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:47:48.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:47:48.912 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:47:48.913 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:48.914 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:48.914 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.915 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.915 DEBUG:boto:Method: GET 2022-09-02T05:47:48.916 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.917 DEBUG:boto:Data: 2022-09-02T05:47:48.917 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:48.918 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.919 DEBUG:boto:Params: {} 2022-09-02T05:47:48.919 DEBUG:boto:Token: None 2022-09-02T05:47:48.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/?versions 2022-09-02T05:47:48.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs= 2022-09-02T05:47:48.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs=', 'Content-Length': '0'} 2022-09-02T05:47:48.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000054a37a181ec0042-0063119904-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.925 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:48.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:47:48.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:47:48.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:47:48.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:47:48.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:47:48.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:47:48.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:47:48.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:47:48.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:47:48.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:47:48.931 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:48.932 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:48.933 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.933 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.934 DEBUG:boto:Method: GET 2022-09-02T05:47:48.934 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.935 DEBUG:boto:Data: 2022-09-02T05:47:48.935 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:48.936 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.937 DEBUG:boto:Params: {} 2022-09-02T05:47:48.938 DEBUG:boto:Token: None 2022-09-02T05:47:48.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/?versions 2022-09-02T05:47:48.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs= 2022-09-02T05:47:48.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs=', 'Content-Length': '0'} 2022-09-02T05:47:48.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030157f066637bfff-0063119904-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.943 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:48.945 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:48.945 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:48.946 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.946 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.947 DEBUG:boto:Method: GET 2022-09-02T05:47:48.948 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:48.948 DEBUG:boto:Data: 2022-09-02T05:47:48.949 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:48.950 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.950 DEBUG:boto:Params: {} 2022-09-02T05:47:48.951 DEBUG:boto:Token: None 2022-09-02T05:47:48.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/?versions 2022-09-02T05:47:48.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs= 2022-09-02T05:47:48.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ygYQQkxtBuGIjLqAZ9ocRP4GFzs=', 'Content-Length': '0'} 2022-09-02T05:47:48.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8599b0e8441f790-0063119904-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.957 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:48.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:48.959 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:48.960 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:48.960 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.961 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.962 DEBUG:boto:Method: GET 2022-09-02T05:47:48.962 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.963 DEBUG:boto:Data: 2022-09-02T05:47:48.963 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:48.964 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.965 DEBUG:boto:Params: {} 2022-09-02T05:47:48.965 DEBUG:boto:Token: None 2022-09-02T05:47:48.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KhGIw1YCRHDWa8063FLx41YNtNI= 2022-09-02T05:47:48.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KhGIw1YCRHDWa8063FLx41YNtNI=', 'Content-Length': '0'} 2022-09-02T05:47:48.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a6d69ea010e617-0063119904-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.970 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:48.971 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:48.971 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.972 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.973 DEBUG:boto:Method: GET 2022-09-02T05:47:48.973 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.974 DEBUG:boto:Data: 2022-09-02T05:47:48.974 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:48.975 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.976 DEBUG:boto:Params: {} 2022-09-02T05:47:48.977 DEBUG:boto:Token: None 2022-09-02T05:47:48.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KhGIw1YCRHDWa8063FLx41YNtNI= 2022-09-02T05:47:48.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KhGIw1YCRHDWa8063FLx41YNtNI=', 'Content-Length': '0'} 2022-09-02T05:47:48.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9017dec7e0709f-0063119904-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.982 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:48.982 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:48.983 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.983 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.984 DEBUG:boto:Method: HEAD 2022-09-02T05:47:48.985 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.985 DEBUG:boto:Data: 2022-09-02T05:47:48.986 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:48.988 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.988 DEBUG:boto:Params: {} 2022-09-02T05:47:48.989 DEBUG:boto:Token: None 2022-09-02T05:47:48.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:48 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hPKWt2eoru0hXgq+087ZlB93LDM= 2022-09-02T05:47:48.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hPKWt2eoru0hXgq+087ZlB93LDM=', 'Content-Length': '0'} 2022-09-02T05:47:48.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbf51f04c3cd3aa-0063119904-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:48.993 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:48.994 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:48.994 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.995 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.995 DEBUG:boto:Method: HEAD 2022-09-02T05:47:48.996 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:48.996 DEBUG:boto:Data: 2022-09-02T05:47:48.997 DEBUG:boto:Headers: {} 2022-09-02T05:47:48.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:48.999 DEBUG:boto:Port: 8000 2022-09-02T05:47:48.999 DEBUG:boto:Params: {} 2022-09-02T05:47:49.000 DEBUG:boto:Token: None 2022-09-02T05:47:49.000 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BL1MwkRWC3BSDBlnNvT6Xy1sNNI= 2022-09-02T05:47:49.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BL1MwkRWC3BSDBlnNvT6Xy1sNNI=', 'Content-Length': '0'} 2022-09-02T05:47:49.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f7edd5025f17c7-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:49.005 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:49.005 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:49.006 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.006 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.007 DEBUG:boto:Method: GET 2022-09-02T05:47:49.008 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.008 DEBUG:boto:Data: 2022-09-02T05:47:49.009 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.010 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.010 DEBUG:boto:Params: {} 2022-09-02T05:47:49.011 DEBUG:boto:Token: None 2022-09-02T05:47:49.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrQFVBZ65pp4f5qzeM9TFgZci10= 2022-09-02T05:47:49.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrQFVBZ65pp4f5qzeM9TFgZci10=', 'Content-Length': '0'} 2022-09-02T05:47:49.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f41ebf44504dce-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.016 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:49.016 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:49.017 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.017 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.018 DEBUG:boto:Method: GET 2022-09-02T05:47:49.018 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.019 DEBUG:boto:Data: 2022-09-02T05:47:49.019 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.020 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.021 DEBUG:boto:Params: {} 2022-09-02T05:47:49.022 DEBUG:boto:Token: None 2022-09-02T05:47:49.022 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:49.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrQFVBZ65pp4f5qzeM9TFgZci10= 2022-09-02T05:47:49.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrQFVBZ65pp4f5qzeM9TFgZci10=', 'Content-Length': '0'} 2022-09-02T05:47:49.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69c597d0b0041c1-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.026 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.027 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.027 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.028 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.028 DEBUG:boto:Method: GET 2022-09-02T05:47:49.029 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.029 DEBUG:boto:Data: 2022-09-02T05:47:49.030 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.031 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.032 DEBUG:boto:Params: {} 2022-09-02T05:47:49.032 DEBUG:boto:Token: None 2022-09-02T05:47:49.033 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fac3c6192afa233a-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.038 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.039 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.039 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.040 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.040 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.041 DEBUG:boto:Method: GET 2022-09-02T05:47:49.041 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.042 DEBUG:boto:Data: 2022-09-02T05:47:49.042 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.044 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.044 DEBUG:boto:Params: {} 2022-09-02T05:47:49.045 DEBUG:boto:Token: None 2022-09-02T05:47:49.046 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091fa8507647fabf2-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.051 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.051 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:49.052 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:49.053 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.053 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:49.054 DEBUG:boto:Data: 2022-09-02T05:47:49.054 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.055 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.056 DEBUG:boto:Params: {} 2022-09-02T05:47:49.056 DEBUG:boto:Token: None 2022-09-02T05:47:49.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11 2022-09-02T05:47:49.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fgcu8MXKQBRDz1g7FNWiFI2a01s= 2022-09-02T05:47:49.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fgcu8MXKQBRDz1g7FNWiFI2a01s=', 'Content-Length': '0'} 2022-09-02T05:47:49.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad6297cf925b18e-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.061 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:49.061 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:49.062 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.063 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:49.063 DEBUG:boto:Data: 2022-09-02T05:47:49.064 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.065 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.066 DEBUG:boto:Params: {} 2022-09-02T05:47:49.067 DEBUG:boto:Token: None 2022-09-02T05:47:49.068 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11 2022-09-02T05:47:49.069 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fgcu8MXKQBRDz1g7FNWiFI2a01s= 2022-09-02T05:47:49.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fgcu8MXKQBRDz1g7FNWiFI2a01s=', 'Content-Length': '0'} 2022-09-02T05:47:49.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f8d1f45b5f3ed5-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:49.075 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:49.076 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:49.077 DEBUG:boto:Method: GET 2022-09-02T05:47:49.078 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:49.079 DEBUG:boto:Data: 2022-09-02T05:47:49.080 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.081 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.082 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.083 DEBUG:boto:Params: {} 2022-09-02T05:47:49.085 DEBUG:boto:Token: None 2022-09-02T05:47:49.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11 2022-09-02T05:47:49.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Sl6lirbXz3zbTO1eeDCicK6aDE= 2022-09-02T05:47:49.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Sl6lirbXz3zbTO1eeDCicK6aDE=', 'Content-Length': '0'} 2022-09-02T05:47:49.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0fd7bf4bfbbc57-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.089 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:49.090 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:49.091 DEBUG:boto:Method: GET 2022-09-02T05:47:49.091 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:49.092 DEBUG:boto:Data: 2022-09-02T05:47:49.092 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.093 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.094 DEBUG:boto:Params: {} 2022-09-02T05:47:49.095 DEBUG:boto:Token: None 2022-09-02T05:47:49.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj11 2022-09-02T05:47:49.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1Sl6lirbXz3zbTO1eeDCicK6aDE= 2022-09-02T05:47:49.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1Sl6lirbXz3zbTO1eeDCicK6aDE=', 'Content-Length': '0'} 2022-09-02T05:47:49.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470d6d2a68bf9d43-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.100 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.101 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.101 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.102 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.103 DEBUG:boto:Method: GET 2022-09-02T05:47:49.103 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.104 DEBUG:boto:Data: 2022-09-02T05:47:49.105 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.106 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.106 DEBUG:boto:Params: {} 2022-09-02T05:47:49.107 DEBUG:boto:Token: None 2022-09-02T05:47:49.108 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084b326f108b8e852-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.113 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.114 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.114 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.115 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.116 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.117 DEBUG:boto:Method: GET 2022-09-02T05:47:49.118 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:49.119 DEBUG:boto:Data: 2022-09-02T05:47:49.120 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.121 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.122 DEBUG:boto:Params: {} 2022-09-02T05:47:49.123 DEBUG:boto:Token: None 2022-09-02T05:47:49.124 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c471c5b0810fd082-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.129 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:49.131 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.132 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.132 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.133 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.134 DEBUG:boto:Method: GET 2022-09-02T05:47:49.134 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.135 DEBUG:boto:Data: 2022-09-02T05:47:49.135 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.137 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.137 DEBUG:boto:Params: {} 2022-09-02T05:47:49.138 DEBUG:boto:Token: None 2022-09-02T05:47:49.138 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U= 2022-09-02T05:47:49.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U=', 'Content-Length': '0'} 2022-09-02T05:47:49.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f83406425e41a77-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.142 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.143 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.144 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.144 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.145 DEBUG:boto:Method: GET 2022-09-02T05:47:49.145 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.146 DEBUG:boto:Data: 2022-09-02T05:47:49.146 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.147 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.148 DEBUG:boto:Params: {} 2022-09-02T05:47:49.148 DEBUG:boto:Token: None 2022-09-02T05:47:49.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U= 2022-09-02T05:47:49.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U=', 'Content-Length': '0'} 2022-09-02T05:47:49.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b089c3f18adb5e-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.155 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.155 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.156 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.157 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.158 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.159 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.160 DEBUG:boto:Data: 2022-09-02T05:47:49.161 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.162 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.163 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.164 DEBUG:boto:Params: {} 2022-09-02T05:47:49.165 DEBUG:boto:Token: None 2022-09-02T05:47:49.166 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cn0CJ93C59G0avQ3iwA70p6r1pI= 2022-09-02T05:47:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cn0CJ93C59G0avQ3iwA70p6r1pI=', 'Content-Length': '0'} 2022-09-02T05:47:49.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd811eeffa2b24a8-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.172 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.173 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.174 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.175 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.176 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.177 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.178 DEBUG:boto:Data: 2022-09-02T05:47:49.179 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.181 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.182 DEBUG:boto:Params: {} 2022-09-02T05:47:49.183 DEBUG:boto:Token: None 2022-09-02T05:47:49.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cn0CJ93C59G0avQ3iwA70p6r1pI= 2022-09-02T05:47:49.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cn0CJ93C59G0avQ3iwA70p6r1pI=', 'Content-Length': '0'} 2022-09-02T05:47:49.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeffcfc6fd47a537-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:49.191 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.192 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.193 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.193 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.194 DEBUG:boto:Method: GET 2022-09-02T05:47:49.194 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.195 DEBUG:boto:Data: 2022-09-02T05:47:49.195 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.196 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.197 DEBUG:boto:Params: {} 2022-09-02T05:47:49.198 DEBUG:boto:Token: None 2022-09-02T05:47:49.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U= 2022-09-02T05:47:49.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U=', 'Content-Length': '0'} 2022-09-02T05:47:49.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b48348a75c50e1-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.202 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.203 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.204 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.204 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.205 DEBUG:boto:Method: GET 2022-09-02T05:47:49.205 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.206 DEBUG:boto:Data: 2022-09-02T05:47:49.206 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.207 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.208 DEBUG:boto:Params: {} 2022-09-02T05:47:49.209 DEBUG:boto:Token: None 2022-09-02T05:47:49.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:49.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U= 2022-09-02T05:47:49.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZbpALgIvkBswsIurYh9nxy7Qq8U=', 'Content-Length': '0'} 2022-09-02T05:47:49.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eacb6aabf68bcbb-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.213 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.214 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.214 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.215 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.215 DEBUG:boto:Method: GET 2022-09-02T05:47:49.216 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.216 DEBUG:boto:Data: 2022-09-02T05:47:49.217 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.217 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.218 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.219 DEBUG:boto:Params: {} 2022-09-02T05:47:49.219 DEBUG:boto:Token: None 2022-09-02T05:47:49.220 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070cacd9fe1da0c30-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.224 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.225 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.226 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.226 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.227 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.227 DEBUG:boto:Method: GET 2022-09-02T05:47:49.228 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.228 DEBUG:boto:Data: 2022-09-02T05:47:49.229 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.230 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.230 DEBUG:boto:Params: {} 2022-09-02T05:47:49.231 DEBUG:boto:Token: None 2022-09-02T05:47:49.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfbb80fbff37a269-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.236 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.237 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.237 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.238 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.238 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:49.239 DEBUG:boto:Data: 2022-09-02T05:47:49.239 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.240 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.241 DEBUG:boto:Params: {} 2022-09-02T05:47:49.241 DEBUG:boto:Token: None 2022-09-02T05:47:49.242 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17 2022-09-02T05:47:49.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wxR82TbOGgxBDK8NeDBszm3WG2s= 2022-09-02T05:47:49.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wxR82TbOGgxBDK8NeDBszm3WG2s=', 'Content-Length': '0'} 2022-09-02T05:47:49.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004156affc3f1e1c75-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.246 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.246 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.247 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.247 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:49.248 DEBUG:boto:Data: 2022-09-02T05:47:49.248 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.249 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.250 DEBUG:boto:Params: {} 2022-09-02T05:47:49.250 DEBUG:boto:Token: None 2022-09-02T05:47:49.251 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17 2022-09-02T05:47:49.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wxR82TbOGgxBDK8NeDBszm3WG2s= 2022-09-02T05:47:49.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wxR82TbOGgxBDK8NeDBszm3WG2s=', 'Content-Length': '0'} 2022-09-02T05:47:49.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c708d5b00dea3185-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:49.255 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.256 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.256 DEBUG:boto:Method: GET 2022-09-02T05:47:49.257 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:49.257 DEBUG:boto:Data: 2022-09-02T05:47:49.258 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.259 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.259 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.260 DEBUG:boto:Params: {} 2022-09-02T05:47:49.260 DEBUG:boto:Token: None 2022-09-02T05:47:49.261 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17 2022-09-02T05:47:49.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LpFrPUWsII+kqFSHU6Tpo3LE28E= 2022-09-02T05:47:49.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LpFrPUWsII+kqFSHU6Tpo3LE28E=', 'Content-Length': '0'} 2022-09-02T05:47:49.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea18e9da6ae8527-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.266 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:49.266 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:49.267 DEBUG:boto:Method: GET 2022-09-02T05:47:49.268 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:49.268 DEBUG:boto:Data: 2022-09-02T05:47:49.269 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.270 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.270 DEBUG:boto:Params: {} 2022-09-02T05:47:49.271 DEBUG:boto:Token: None 2022-09-02T05:47:49.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj17 2022-09-02T05:47:49.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LpFrPUWsII+kqFSHU6Tpo3LE28E= 2022-09-02T05:47:49.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LpFrPUWsII+kqFSHU6Tpo3LE28E=', 'Content-Length': '0'} 2022-09-02T05:47:49.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039884ac15fa75ebe-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.318 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.319 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.320 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.320 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.321 DEBUG:boto:Method: GET 2022-09-02T05:47:49.321 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.322 DEBUG:boto:Data: 2022-09-02T05:47:49.323 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.324 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.324 DEBUG:boto:Params: {} 2022-09-02T05:47:49.325 DEBUG:boto:Token: None 2022-09-02T05:47:49.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000654983346565392a-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.330 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.331 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.332 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.332 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.333 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.333 DEBUG:boto:Method: GET 2022-09-02T05:47:49.334 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:49.334 DEBUG:boto:Data: 2022-09-02T05:47:49.335 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.336 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.337 DEBUG:boto:Params: {} 2022-09-02T05:47:49.337 DEBUG:boto:Token: None 2022-09-02T05:47:49.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aba35860337368fe-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.344 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:49.346 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.347 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.348 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.349 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.350 DEBUG:boto:Method: GET 2022-09-02T05:47:49.351 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.352 DEBUG:boto:Data: 2022-09-02T05:47:49.353 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.354 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.355 DEBUG:boto:Params: {} 2022-09-02T05:47:49.356 DEBUG:boto:Token: None 2022-09-02T05:47:49.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso= 2022-09-02T05:47:49.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso=', 'Content-Length': '0'} 2022-09-02T05:47:49.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6c1b8a9f5a3016-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.363 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.364 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.365 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.366 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.367 DEBUG:boto:Method: GET 2022-09-02T05:47:49.368 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.369 DEBUG:boto:Data: 2022-09-02T05:47:49.370 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.371 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.372 DEBUG:boto:Params: {} 2022-09-02T05:47:49.373 DEBUG:boto:Token: None 2022-09-02T05:47:49.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso= 2022-09-02T05:47:49.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso=', 'Content-Length': '0'} 2022-09-02T05:47:49.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb03f88fb75d7ad-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.378 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.379 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.379 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.380 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.381 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.381 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.382 DEBUG:boto:Data: 2022-09-02T05:47:49.382 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.383 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.384 DEBUG:boto:Params: {} 2022-09-02T05:47:49.384 DEBUG:boto:Token: None 2022-09-02T05:47:49.385 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bo42dLqh9dZZJya+BML1iakat+A= 2022-09-02T05:47:49.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bo42dLqh9dZZJya+BML1iakat+A=', 'Content-Length': '0'} 2022-09-02T05:47:49.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086554ff07c93825a-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.389 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.390 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.390 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.391 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.391 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.392 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.393 DEBUG:boto:Data: 2022-09-02T05:47:49.393 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.394 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.395 DEBUG:boto:Params: {} 2022-09-02T05:47:49.396 DEBUG:boto:Token: None 2022-09-02T05:47:49.397 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bo42dLqh9dZZJya+BML1iakat+A= 2022-09-02T05:47:49.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bo42dLqh9dZZJya+BML1iakat+A=', 'Content-Length': '0'} 2022-09-02T05:47:49.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c79c3f31a783f60b-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:49.401 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.402 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.402 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.403 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.404 DEBUG:boto:Method: GET 2022-09-02T05:47:49.404 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.405 DEBUG:boto:Data: 2022-09-02T05:47:49.405 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.407 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.408 DEBUG:boto:Params: {} 2022-09-02T05:47:49.409 DEBUG:boto:Token: None 2022-09-02T05:47:49.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso= 2022-09-02T05:47:49.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso=', 'Content-Length': '0'} 2022-09-02T05:47:49.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce1fdb3ce268fa8d-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.458 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.459 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.460 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.460 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.461 DEBUG:boto:Method: GET 2022-09-02T05:47:49.461 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.462 DEBUG:boto:Data: 2022-09-02T05:47:49.463 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.465 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.466 DEBUG:boto:Params: {} 2022-09-02T05:47:49.466 DEBUG:boto:Token: None 2022-09-02T05:47:49.467 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:49.468 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso= 2022-09-02T05:47:49.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4Jv3Vg2hgmJsbnCRRUMnI9Hndso=', 'Content-Length': '0'} 2022-09-02T05:47:49.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd30e46ff53d922-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.472 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.473 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.474 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.475 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.476 DEBUG:boto:Method: GET 2022-09-02T05:47:49.477 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.478 DEBUG:boto:Data: 2022-09-02T05:47:49.479 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.481 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.482 DEBUG:boto:Params: {} 2022-09-02T05:47:49.483 DEBUG:boto:Token: None 2022-09-02T05:47:49.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe544f44a8f30ff5-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.492 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.493 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.494 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.495 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.496 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.497 DEBUG:boto:Method: GET 2022-09-02T05:47:49.498 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.499 DEBUG:boto:Data: 2022-09-02T05:47:49.500 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.502 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.503 DEBUG:boto:Params: {} 2022-09-02T05:47:49.504 DEBUG:boto:Token: None 2022-09-02T05:47:49.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bece0e3398d787f0-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.558 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.560 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.561 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.562 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.563 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:49.563 DEBUG:boto:Data: 2022-09-02T05:47:49.564 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.565 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.565 DEBUG:boto:Params: {} 2022-09-02T05:47:49.566 DEBUG:boto:Token: None 2022-09-02T05:47:49.567 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22 2022-09-02T05:47:49.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WIl/vqL7/36Rjzy+QinnlObhRZ8= 2022-09-02T05:47:49.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WIl/vqL7/36Rjzy+QinnlObhRZ8=', 'Content-Length': '0'} 2022-09-02T05:47:49.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c56fdd34508ae809-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.571 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.571 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.572 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.572 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:49.573 DEBUG:boto:Data: 2022-09-02T05:47:49.573 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.574 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.575 DEBUG:boto:Params: {} 2022-09-02T05:47:49.576 DEBUG:boto:Token: None 2022-09-02T05:47:49.576 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22 2022-09-02T05:47:49.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WIl/vqL7/36Rjzy+QinnlObhRZ8= 2022-09-02T05:47:49.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WIl/vqL7/36Rjzy+QinnlObhRZ8=', 'Content-Length': '0'} 2022-09-02T05:47:49.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c858608725518c-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:49.581 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.581 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.582 DEBUG:boto:Method: GET 2022-09-02T05:47:49.582 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:49.583 DEBUG:boto:Data: 2022-09-02T05:47:49.583 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.584 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.585 DEBUG:boto:Params: {} 2022-09-02T05:47:49.586 DEBUG:boto:Token: None 2022-09-02T05:47:49.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22 2022-09-02T05:47:49.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RPxnFEb9DTqilzuU6Iy0oUseyQM= 2022-09-02T05:47:49.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RPxnFEb9DTqilzuU6Iy0oUseyQM=', 'Content-Length': '0'} 2022-09-02T05:47:49.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f5f9c64feb7d6c-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.591 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:49.592 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:49.592 DEBUG:boto:Method: GET 2022-09-02T05:47:49.593 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:49.593 DEBUG:boto:Data: 2022-09-02T05:47:49.594 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.595 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.596 DEBUG:boto:Params: {} 2022-09-02T05:47:49.596 DEBUG:boto:Token: None 2022-09-02T05:47:49.597 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj22 2022-09-02T05:47:49.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RPxnFEb9DTqilzuU6Iy0oUseyQM= 2022-09-02T05:47:49.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RPxnFEb9DTqilzuU6Iy0oUseyQM=', 'Content-Length': '0'} 2022-09-02T05:47:49.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475e71e77597b338-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.603 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.604 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.604 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.605 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.605 DEBUG:boto:Method: GET 2022-09-02T05:47:49.606 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.607 DEBUG:boto:Data: 2022-09-02T05:47:49.607 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.608 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.609 DEBUG:boto:Params: {} 2022-09-02T05:47:49.609 DEBUG:boto:Token: None 2022-09-02T05:47:49.610 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9010df244748ead-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.614 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.615 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.616 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.616 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.617 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.617 DEBUG:boto:Method: GET 2022-09-02T05:47:49.618 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:49.619 DEBUG:boto:Data: 2022-09-02T05:47:49.619 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.620 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.621 DEBUG:boto:Params: {} 2022-09-02T05:47:49.621 DEBUG:boto:Token: None 2022-09-02T05:47:49.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf302864889a7f8f-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.627 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:49.629 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.629 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.630 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.631 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.631 DEBUG:boto:Method: GET 2022-09-02T05:47:49.632 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.633 DEBUG:boto:Data: 2022-09-02T05:47:49.633 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.634 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.634 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.635 DEBUG:boto:Params: {} 2022-09-02T05:47:49.636 DEBUG:boto:Token: None 2022-09-02T05:47:49.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc= 2022-09-02T05:47:49.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc=', 'Content-Length': '0'} 2022-09-02T05:47:49.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4461655f9c7c0d-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.640 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.641 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.641 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.642 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.643 DEBUG:boto:Method: GET 2022-09-02T05:47:49.643 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.644 DEBUG:boto:Data: 2022-09-02T05:47:49.644 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.645 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.646 DEBUG:boto:Params: {} 2022-09-02T05:47:49.646 DEBUG:boto:Token: None 2022-09-02T05:47:49.647 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc= 2022-09-02T05:47:49.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc=', 'Content-Length': '0'} 2022-09-02T05:47:49.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dadf13436b6072e-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.694 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.695 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.695 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.696 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.696 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.697 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.698 DEBUG:boto:Data: 2022-09-02T05:47:49.698 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.699 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.700 DEBUG:boto:Params: {} 2022-09-02T05:47:49.701 DEBUG:boto:Token: None 2022-09-02T05:47:49.702 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UEX1ZVIMDJQI7pWX6jZ6ui4QBv4= 2022-09-02T05:47:49.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UEX1ZVIMDJQI7pWX6jZ6ui4QBv4=', 'Content-Length': '0'} 2022-09-02T05:47:49.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000123e6a13365505b-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.705 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.706 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.706 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.707 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.707 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.708 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.708 DEBUG:boto:Data: 2022-09-02T05:47:49.709 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.710 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.711 DEBUG:boto:Params: {} 2022-09-02T05:47:49.711 DEBUG:boto:Token: None 2022-09-02T05:47:49.712 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UEX1ZVIMDJQI7pWX6jZ6ui4QBv4= 2022-09-02T05:47:49.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UEX1ZVIMDJQI7pWX6jZ6ui4QBv4=', 'Content-Length': '0'} 2022-09-02T05:47:49.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004622230ebdcfdc39-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:49.716 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.717 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.718 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.718 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.719 DEBUG:boto:Method: GET 2022-09-02T05:47:49.720 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.720 DEBUG:boto:Data: 2022-09-02T05:47:49.721 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.722 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.722 DEBUG:boto:Params: {} 2022-09-02T05:47:49.723 DEBUG:boto:Token: None 2022-09-02T05:47:49.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc= 2022-09-02T05:47:49.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc=', 'Content-Length': '0'} 2022-09-02T05:47:49.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009acad1b487f265fe-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.728 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.728 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.729 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.729 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.730 DEBUG:boto:Method: GET 2022-09-02T05:47:49.730 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.731 DEBUG:boto:Data: 2022-09-02T05:47:49.731 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.732 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.733 DEBUG:boto:Params: {} 2022-09-02T05:47:49.734 DEBUG:boto:Token: None 2022-09-02T05:47:49.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:49.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc= 2022-09-02T05:47:49.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UGAiHxVILc78H/xw8t22lCZaPqc=', 'Content-Length': '0'} 2022-09-02T05:47:49.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963f54b4d6a980fa-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.739 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.740 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.741 DEBUG:boto:Method: GET 2022-09-02T05:47:49.742 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.742 DEBUG:boto:Data: 2022-09-02T05:47:49.743 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.745 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.745 DEBUG:boto:Params: {} 2022-09-02T05:47:49.746 DEBUG:boto:Token: None 2022-09-02T05:47:49.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d76ac726e8c9f3b9-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.798 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.799 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.800 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.801 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.802 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.803 DEBUG:boto:Method: GET 2022-09-02T05:47:49.804 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.804 DEBUG:boto:Data: 2022-09-02T05:47:49.805 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.807 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.808 DEBUG:boto:Params: {} 2022-09-02T05:47:49.808 DEBUG:boto:Token: None 2022-09-02T05:47:49.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057744fe94c84f64c-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.814 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.815 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.816 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.816 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.817 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:49.817 DEBUG:boto:Data: 2022-09-02T05:47:49.818 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.819 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.819 DEBUG:boto:Params: {} 2022-09-02T05:47:49.820 DEBUG:boto:Token: None 2022-09-02T05:47:49.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33 2022-09-02T05:47:49.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TYl+UDCZBjijECj7pE8+GBRr1qY= 2022-09-02T05:47:49.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TYl+UDCZBjijECj7pE8+GBRr1qY=', 'Content-Length': '0'} 2022-09-02T05:47:49.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f2a695396612a2-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.824 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.825 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.825 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.826 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:49.826 DEBUG:boto:Data: 2022-09-02T05:47:49.827 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.828 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.828 DEBUG:boto:Params: {} 2022-09-02T05:47:49.829 DEBUG:boto:Token: None 2022-09-02T05:47:49.829 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33 2022-09-02T05:47:49.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TYl+UDCZBjijECj7pE8+GBRr1qY= 2022-09-02T05:47:49.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TYl+UDCZBjijECj7pE8+GBRr1qY=', 'Content-Length': '0'} 2022-09-02T05:47:49.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a26c5f62684224b-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:49.834 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.834 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.835 DEBUG:boto:Method: GET 2022-09-02T05:47:49.835 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:49.836 DEBUG:boto:Data: 2022-09-02T05:47:49.837 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.837 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.838 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.838 DEBUG:boto:Params: {} 2022-09-02T05:47:49.839 DEBUG:boto:Token: None 2022-09-02T05:47:49.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33 2022-09-02T05:47:49.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qUcJ/dTSlIzsyB8+FPnh+vvlOoQ= 2022-09-02T05:47:49.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qUcJ/dTSlIzsyB8+FPnh+vvlOoQ=', 'Content-Length': '0'} 2022-09-02T05:47:49.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075cee5f7ed7e22c5-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.844 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:49.844 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:49.845 DEBUG:boto:Method: GET 2022-09-02T05:47:49.845 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:49.846 DEBUG:boto:Data: 2022-09-02T05:47:49.846 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.847 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.848 DEBUG:boto:Params: {} 2022-09-02T05:47:49.849 DEBUG:boto:Token: None 2022-09-02T05:47:49.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj33 2022-09-02T05:47:49.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qUcJ/dTSlIzsyB8+FPnh+vvlOoQ= 2022-09-02T05:47:49.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qUcJ/dTSlIzsyB8+FPnh+vvlOoQ=', 'Content-Length': '0'} 2022-09-02T05:47:49.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5932020062ddb3-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.854 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.854 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.855 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.855 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.856 DEBUG:boto:Method: GET 2022-09-02T05:47:49.856 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.857 DEBUG:boto:Data: 2022-09-02T05:47:49.857 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.858 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.859 DEBUG:boto:Params: {} 2022-09-02T05:47:49.860 DEBUG:boto:Token: None 2022-09-02T05:47:49.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa6e183aba42eaf-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.865 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.866 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.866 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.867 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.867 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.868 DEBUG:boto:Method: GET 2022-09-02T05:47:49.868 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:49.869 DEBUG:boto:Data: 2022-09-02T05:47:49.869 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.870 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.871 DEBUG:boto:Params: {} 2022-09-02T05:47:49.872 DEBUG:boto:Token: None 2022-09-02T05:47:49.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e782a87947fe5d94-0063119905-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.877 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:49.879 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:49.879 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:49.880 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.881 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.881 DEBUG:boto:Method: GET 2022-09-02T05:47:49.882 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.882 DEBUG:boto:Data: 2022-09-02T05:47:49.883 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.884 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.885 DEBUG:boto:Params: {} 2022-09-02T05:47:49.885 DEBUG:boto:Token: None 2022-09-02T05:47:49.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8= 2022-09-02T05:47:49.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8=', 'Content-Length': '0'} 2022-09-02T05:47:49.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006800f8db0fa1b33a-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.890 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:49.891 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:49.891 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.892 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.893 DEBUG:boto:Method: GET 2022-09-02T05:47:49.893 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.894 DEBUG:boto:Data: 2022-09-02T05:47:49.894 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.895 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.896 DEBUG:boto:Params: {} 2022-09-02T05:47:49.896 DEBUG:boto:Token: None 2022-09-02T05:47:49.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8= 2022-09-02T05:47:49.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8=', 'Content-Length': '0'} 2022-09-02T05:47:49.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e0b148d9ac008d-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.901 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:49.902 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:49.902 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.903 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.903 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.904 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.904 DEBUG:boto:Data: 2022-09-02T05:47:49.905 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.906 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.906 DEBUG:boto:Params: {} 2022-09-02T05:47:49.907 DEBUG:boto:Token: None 2022-09-02T05:47:49.908 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:koLctnAjjS5IqyoXlGw9tjILTCw= 2022-09-02T05:47:49.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:koLctnAjjS5IqyoXlGw9tjILTCw=', 'Content-Length': '0'} 2022-09-02T05:47:49.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c90ed650b97634-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.911 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:49.912 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:49.912 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.913 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.913 DEBUG:boto:Method: HEAD 2022-09-02T05:47:49.914 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.914 DEBUG:boto:Data: 2022-09-02T05:47:49.915 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.916 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.916 DEBUG:boto:Params: {} 2022-09-02T05:47:49.917 DEBUG:boto:Token: None 2022-09-02T05:47:49.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:koLctnAjjS5IqyoXlGw9tjILTCw= 2022-09-02T05:47:49.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:koLctnAjjS5IqyoXlGw9tjILTCw=', 'Content-Length': '0'} 2022-09-02T05:47:49.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4335979bddc6b8-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:49.922 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:49.922 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:49.923 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.923 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.924 DEBUG:boto:Method: GET 2022-09-02T05:47:49.924 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.925 DEBUG:boto:Data: 2022-09-02T05:47:49.925 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.926 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.927 DEBUG:boto:Params: {} 2022-09-02T05:47:49.928 DEBUG:boto:Token: None 2022-09-02T05:47:49.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8= 2022-09-02T05:47:49.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8=', 'Content-Length': '0'} 2022-09-02T05:47:49.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003940597e7a665816-0063119905-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.974 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:49.975 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:49.976 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.976 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.977 DEBUG:boto:Method: GET 2022-09-02T05:47:49.977 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.978 DEBUG:boto:Data: 2022-09-02T05:47:49.978 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:49.979 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.980 DEBUG:boto:Params: {} 2022-09-02T05:47:49.981 DEBUG:boto:Token: None 2022-09-02T05:47:49.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:49.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8= 2022-09-02T05:47:49.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BL75gc/UkMsshtk/xetotB93Lj8=', 'Content-Length': '0'} 2022-09-02T05:47:49.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004836cabde9972597-0063119905-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.986 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:49.987 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:49.987 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:49.988 DEBUG:boto:Method: GET 2022-09-02T05:47:49.988 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:49.989 DEBUG:boto:Data: 2022-09-02T05:47:49.990 DEBUG:boto:Headers: {} 2022-09-02T05:47:49.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:49.991 DEBUG:boto:Port: 8000 2022-09-02T05:47:49.991 DEBUG:boto:Params: {} 2022-09-02T05:47:49.992 DEBUG:boto:Token: None 2022-09-02T05:47:49.993 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:49 GMT /xljbkf-74/ 2022-09-02T05:47:49.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo= 2022-09-02T05:47:49.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LkR77aFQe5M1Po5lytq1lWidpZo=', 'Content-Length': '0'} 2022-09-02T05:47:49.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b93ba55c90deed6d-0063119905-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:49.997 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:49.998 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:49.999 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.000 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.000 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.001 DEBUG:boto:Method: GET 2022-09-02T05:47:50.001 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.002 DEBUG:boto:Data: 2022-09-02T05:47:50.003 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.004 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.004 DEBUG:boto:Params: {} 2022-09-02T05:47:50.005 DEBUG:boto:Token: None 2022-09-02T05:47:50.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8466fcfef97200b-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.010 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.011 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:50.012 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:50.012 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.013 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:50.013 DEBUG:boto:Data: 2022-09-02T05:47:50.014 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.016 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.017 DEBUG:boto:Params: {} 2022-09-02T05:47:50.018 DEBUG:boto:Token: None 2022-09-02T05:47:50.019 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj34 2022-09-02T05:47:50.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rydw6dDYmbT36RzLqkjA40+igMw= 2022-09-02T05:47:50.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rydw6dDYmbT36RzLqkjA40+igMw=', 'Content-Length': '0'} 2022-09-02T05:47:50.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21adfa21c373e4b-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.025 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:50.026 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:50.027 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.028 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:50.029 DEBUG:boto:Data: 2022-09-02T05:47:50.030 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.032 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.033 DEBUG:boto:Params: {} 2022-09-02T05:47:50.034 DEBUG:boto:Token: None 2022-09-02T05:47:50.035 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj34 2022-09-02T05:47:50.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rydw6dDYmbT36RzLqkjA40+igMw= 2022-09-02T05:47:50.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rydw6dDYmbT36RzLqkjA40+igMw=', 'Content-Length': '0'} 2022-09-02T05:47:50.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007093459e6733f708-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:50.040 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:50.041 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:50.041 DEBUG:boto:Method: GET 2022-09-02T05:47:50.042 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:50.042 DEBUG:boto:Data: 2022-09-02T05:47:50.043 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.044 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.044 DEBUG:boto:Params: {} 2022-09-02T05:47:50.045 DEBUG:boto:Token: None 2022-09-02T05:47:50.046 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj34 2022-09-02T05:47:50.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jfa9LKXImZFZq/5dZW5uvQIKg8c= 2022-09-02T05:47:50.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jfa9LKXImZFZq/5dZW5uvQIKg8c=', 'Content-Length': '0'} 2022-09-02T05:47:50.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483fa11571d3e16b-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.050 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:50.050 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:50.051 DEBUG:boto:Method: GET 2022-09-02T05:47:50.051 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:50.052 DEBUG:boto:Data: 2022-09-02T05:47:50.052 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.053 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.054 DEBUG:boto:Params: {} 2022-09-02T05:47:50.054 DEBUG:boto:Token: None 2022-09-02T05:47:50.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj34 2022-09-02T05:47:50.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jfa9LKXImZFZq/5dZW5uvQIKg8c= 2022-09-02T05:47:50.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jfa9LKXImZFZq/5dZW5uvQIKg8c=', 'Content-Length': '0'} 2022-09-02T05:47:50.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa22ffe03743ee3-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.059 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.060 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.060 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.061 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.061 DEBUG:boto:Method: GET 2022-09-02T05:47:50.062 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.063 DEBUG:boto:Data: 2022-09-02T05:47:50.063 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.064 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.065 DEBUG:boto:Params: {} 2022-09-02T05:47:50.066 DEBUG:boto:Token: None 2022-09-02T05:47:50.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a6eac71a660207-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.072 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.074 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.075 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.076 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.076 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.077 DEBUG:boto:Method: GET 2022-09-02T05:47:50.078 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:50.079 DEBUG:boto:Data: 2022-09-02T05:47:50.080 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.082 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.082 DEBUG:boto:Params: {} 2022-09-02T05:47:50.083 DEBUG:boto:Token: None 2022-09-02T05:47:50.084 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.084 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000683c605906c317e3-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.130 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:50.132 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.133 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.134 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.135 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.135 DEBUG:boto:Method: GET 2022-09-02T05:47:50.136 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.136 DEBUG:boto:Data: 2022-09-02T05:47:50.137 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.138 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.138 DEBUG:boto:Params: {} 2022-09-02T05:47:50.139 DEBUG:boto:Token: None 2022-09-02T05:47:50.140 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA= 2022-09-02T05:47:50.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA=', 'Content-Length': '0'} 2022-09-02T05:47:50.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002787eb9a7adad601-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.144 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.145 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.145 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.146 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.146 DEBUG:boto:Method: GET 2022-09-02T05:47:50.147 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.147 DEBUG:boto:Data: 2022-09-02T05:47:50.148 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.149 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.149 DEBUG:boto:Params: {} 2022-09-02T05:47:50.150 DEBUG:boto:Token: None 2022-09-02T05:47:50.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA= 2022-09-02T05:47:50.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA=', 'Content-Length': '0'} 2022-09-02T05:47:50.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000588d95310c469497-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.155 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.156 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.156 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.157 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.157 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.158 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.159 DEBUG:boto:Data: 2022-09-02T05:47:50.160 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.161 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.162 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.163 DEBUG:boto:Params: {} 2022-09-02T05:47:50.164 DEBUG:boto:Token: None 2022-09-02T05:47:50.165 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Szkd1ILppJMVdWr5lNzebZfT6zQ= 2022-09-02T05:47:50.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Szkd1ILppJMVdWr5lNzebZfT6zQ=', 'Content-Length': '0'} 2022-09-02T05:47:50.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a44cd120cd3aca9c-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.171 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.172 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.173 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.174 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.175 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.176 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.177 DEBUG:boto:Data: 2022-09-02T05:47:50.178 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.180 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.181 DEBUG:boto:Params: {} 2022-09-02T05:47:50.182 DEBUG:boto:Token: None 2022-09-02T05:47:50.183 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Szkd1ILppJMVdWr5lNzebZfT6zQ= 2022-09-02T05:47:50.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Szkd1ILppJMVdWr5lNzebZfT6zQ=', 'Content-Length': '0'} 2022-09-02T05:47:50.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076005737c8bc9f78-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:50.189 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.190 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.191 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.192 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.192 DEBUG:boto:Method: GET 2022-09-02T05:47:50.193 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.193 DEBUG:boto:Data: 2022-09-02T05:47:50.194 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.195 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.195 DEBUG:boto:Params: {} 2022-09-02T05:47:50.196 DEBUG:boto:Token: None 2022-09-02T05:47:50.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA= 2022-09-02T05:47:50.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA=', 'Content-Length': '0'} 2022-09-02T05:47:50.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006433b264eb3cad92-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.201 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.202 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.202 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.203 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.204 DEBUG:boto:Method: GET 2022-09-02T05:47:50.204 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.205 DEBUG:boto:Data: 2022-09-02T05:47:50.205 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.207 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.207 DEBUG:boto:Params: {} 2022-09-02T05:47:50.208 DEBUG:boto:Token: None 2022-09-02T05:47:50.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:50.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA= 2022-09-02T05:47:50.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qP2D/jlBFoROEu68eqbyGgGQQvA=', 'Content-Length': '0'} 2022-09-02T05:47:50.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c374fb6edc7d926a-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.214 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.215 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.215 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.216 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.217 DEBUG:boto:Method: GET 2022-09-02T05:47:50.218 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.219 DEBUG:boto:Data: 2022-09-02T05:47:50.220 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.221 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.230 DEBUG:boto:Params: {} 2022-09-02T05:47:50.231 DEBUG:boto:Token: None 2022-09-02T05:47:50.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153f639f944dc20c-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.236 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.237 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.238 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.239 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.240 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.241 DEBUG:boto:Method: GET 2022-09-02T05:47:50.242 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.243 DEBUG:boto:Data: 2022-09-02T05:47:50.244 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.245 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.246 DEBUG:boto:Params: {} 2022-09-02T05:47:50.248 DEBUG:boto:Token: None 2022-09-02T05:47:50.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006596ce69893b64a-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.256 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.257 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.258 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.259 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.260 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:50.261 DEBUG:boto:Data: 2022-09-02T05:47:50.262 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.264 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.265 DEBUG:boto:Params: {} 2022-09-02T05:47:50.266 DEBUG:boto:Token: None 2022-09-02T05:47:50.267 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44 2022-09-02T05:47:50.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gHt1R+f+eJ8oQTYl+cHN24Fx6PE= 2022-09-02T05:47:50.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gHt1R+f+eJ8oQTYl+cHN24Fx6PE=', 'Content-Length': '0'} 2022-09-02T05:47:50.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6946f093bd4d8a-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.273 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.274 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.275 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.276 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:50.277 DEBUG:boto:Data: 2022-09-02T05:47:50.278 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.280 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.281 DEBUG:boto:Params: {} 2022-09-02T05:47:50.282 DEBUG:boto:Token: None 2022-09-02T05:47:50.283 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44 2022-09-02T05:47:50.284 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gHt1R+f+eJ8oQTYl+cHN24Fx6PE= 2022-09-02T05:47:50.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gHt1R+f+eJ8oQTYl+cHN24Fx6PE=', 'Content-Length': '0'} 2022-09-02T05:47:50.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d56b13b17deed7-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:50.290 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.290 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.291 DEBUG:boto:Method: GET 2022-09-02T05:47:50.292 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:50.293 DEBUG:boto:Data: 2022-09-02T05:47:50.293 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.295 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.295 DEBUG:boto:Params: {} 2022-09-02T05:47:50.296 DEBUG:boto:Token: None 2022-09-02T05:47:50.296 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44 2022-09-02T05:47:50.297 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJye+KHLeCNxIVkEDGXl6w9LaaY= 2022-09-02T05:47:50.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJye+KHLeCNxIVkEDGXl6w9LaaY=', 'Content-Length': '0'} 2022-09-02T05:47:50.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007816ca0b5ffc0d3f-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.300 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:50.301 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:50.301 DEBUG:boto:Method: GET 2022-09-02T05:47:50.302 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:50.302 DEBUG:boto:Data: 2022-09-02T05:47:50.303 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.304 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.304 DEBUG:boto:Params: {} 2022-09-02T05:47:50.305 DEBUG:boto:Token: None 2022-09-02T05:47:50.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj44 2022-09-02T05:47:50.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hJye+KHLeCNxIVkEDGXl6w9LaaY= 2022-09-02T05:47:50.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hJye+KHLeCNxIVkEDGXl6w9LaaY=', 'Content-Length': '0'} 2022-09-02T05:47:50.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb66224f6f1fdd4-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.354 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.355 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.355 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.356 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.356 DEBUG:boto:Method: GET 2022-09-02T05:47:50.357 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.357 DEBUG:boto:Data: 2022-09-02T05:47:50.358 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.359 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.360 DEBUG:boto:Params: {} 2022-09-02T05:47:50.360 DEBUG:boto:Token: None 2022-09-02T05:47:50.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe8c937c041eed14-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.365 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.366 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.367 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.367 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.368 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.368 DEBUG:boto:Method: GET 2022-09-02T05:47:50.369 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:50.369 DEBUG:boto:Data: 2022-09-02T05:47:50.370 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.371 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.372 DEBUG:boto:Params: {} 2022-09-02T05:47:50.372 DEBUG:boto:Token: None 2022-09-02T05:47:50.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed78e25380eb5d0-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.379 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:50.381 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.381 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.382 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.382 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.383 DEBUG:boto:Method: GET 2022-09-02T05:47:50.384 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.384 DEBUG:boto:Data: 2022-09-02T05:47:50.385 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.386 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.386 DEBUG:boto:Params: {} 2022-09-02T05:47:50.387 DEBUG:boto:Token: None 2022-09-02T05:47:50.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4= 2022-09-02T05:47:50.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4=', 'Content-Length': '0'} 2022-09-02T05:47:50.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000610423312935eaeb-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.393 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.394 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.394 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.395 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.396 DEBUG:boto:Method: GET 2022-09-02T05:47:50.396 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.397 DEBUG:boto:Data: 2022-09-02T05:47:50.397 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.398 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.399 DEBUG:boto:Params: {} 2022-09-02T05:47:50.400 DEBUG:boto:Token: None 2022-09-02T05:47:50.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4= 2022-09-02T05:47:50.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4=', 'Content-Length': '0'} 2022-09-02T05:47:50.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ecf5c5424322b2-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.405 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.406 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.406 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.407 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.407 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.408 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.409 DEBUG:boto:Data: 2022-09-02T05:47:50.409 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.410 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.411 DEBUG:boto:Params: {} 2022-09-02T05:47:50.412 DEBUG:boto:Token: None 2022-09-02T05:47:50.412 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ILCSZ1IurbYtjJmLzjZ7ckRudas= 2022-09-02T05:47:50.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ILCSZ1IurbYtjJmLzjZ7ckRudas=', 'Content-Length': '0'} 2022-09-02T05:47:50.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f240953c130ca1f7-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.417 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.417 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.418 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.418 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.419 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.419 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.420 DEBUG:boto:Data: 2022-09-02T05:47:50.420 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.421 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.422 DEBUG:boto:Params: {} 2022-09-02T05:47:50.422 DEBUG:boto:Token: None 2022-09-02T05:47:50.423 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ILCSZ1IurbYtjJmLzjZ7ckRudas= 2022-09-02T05:47:50.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ILCSZ1IurbYtjJmLzjZ7ckRudas=', 'Content-Length': '0'} 2022-09-02T05:47:50.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d30f5f61e05b35a-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:50.428 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.428 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.429 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.430 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.430 DEBUG:boto:Method: GET 2022-09-02T05:47:50.431 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.432 DEBUG:boto:Data: 2022-09-02T05:47:50.432 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.434 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.434 DEBUG:boto:Params: {} 2022-09-02T05:47:50.435 DEBUG:boto:Token: None 2022-09-02T05:47:50.436 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4= 2022-09-02T05:47:50.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4=', 'Content-Length': '0'} 2022-09-02T05:47:50.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000327f58f5c356d38-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.441 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.441 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.442 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.442 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.443 DEBUG:boto:Method: GET 2022-09-02T05:47:50.444 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.445 DEBUG:boto:Data: 2022-09-02T05:47:50.445 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.447 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.448 DEBUG:boto:Params: {} 2022-09-02T05:47:50.448 DEBUG:boto:Token: None 2022-09-02T05:47:50.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:50.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4= 2022-09-02T05:47:50.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLDJuVZ5b5lgV/dQrKMX685kzn4=', 'Content-Length': '0'} 2022-09-02T05:47:50.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab44a3f521c080af-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.498 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.499 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.499 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.500 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.501 DEBUG:boto:Method: GET 2022-09-02T05:47:50.502 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.503 DEBUG:boto:Data: 2022-09-02T05:47:50.503 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.504 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.505 DEBUG:boto:Params: {} 2022-09-02T05:47:50.506 DEBUG:boto:Token: None 2022-09-02T05:47:50.507 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5a1d411304fbd97-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.512 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.512 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.513 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.513 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.514 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.514 DEBUG:boto:Method: GET 2022-09-02T05:47:50.515 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.516 DEBUG:boto:Data: 2022-09-02T05:47:50.516 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.517 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.518 DEBUG:boto:Params: {} 2022-09-02T05:47:50.519 DEBUG:boto:Token: None 2022-09-02T05:47:50.519 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aea3631f16f64448-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.525 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.526 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.527 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.527 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.528 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:50.528 DEBUG:boto:Data: 2022-09-02T05:47:50.529 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.529 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.530 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.531 DEBUG:boto:Params: {} 2022-09-02T05:47:50.531 DEBUG:boto:Token: None 2022-09-02T05:47:50.532 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51 2022-09-02T05:47:50.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6NRGMUnPJKJjbXYLk5/nVF8kmQU= 2022-09-02T05:47:50.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6NRGMUnPJKJjbXYLk5/nVF8kmQU=', 'Content-Length': '0'} 2022-09-02T05:47:50.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9214376c94fdc5-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.538 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.539 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.539 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.540 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:50.540 DEBUG:boto:Data: 2022-09-02T05:47:50.541 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.542 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.543 DEBUG:boto:Params: {} 2022-09-02T05:47:50.543 DEBUG:boto:Token: None 2022-09-02T05:47:50.544 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51 2022-09-02T05:47:50.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6NRGMUnPJKJjbXYLk5/nVF8kmQU= 2022-09-02T05:47:50.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6NRGMUnPJKJjbXYLk5/nVF8kmQU=', 'Content-Length': '0'} 2022-09-02T05:47:50.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a323f16f8cf28d42-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:50.549 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.549 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.550 DEBUG:boto:Method: GET 2022-09-02T05:47:50.550 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:50.551 DEBUG:boto:Data: 2022-09-02T05:47:50.551 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.553 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.554 DEBUG:boto:Params: {} 2022-09-02T05:47:50.554 DEBUG:boto:Token: None 2022-09-02T05:47:50.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51 2022-09-02T05:47:50.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/WG7c/65MKPJ156FwjJdZth/So= 2022-09-02T05:47:50.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/WG7c/65MKPJ156FwjJdZth/So=', 'Content-Length': '0'} 2022-09-02T05:47:50.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd89fa49a510bf7-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.610 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:50.612 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:50.613 DEBUG:boto:Method: GET 2022-09-02T05:47:50.615 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:50.616 DEBUG:boto:Data: 2022-09-02T05:47:50.617 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.618 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.618 DEBUG:boto:Params: {} 2022-09-02T05:47:50.619 DEBUG:boto:Token: None 2022-09-02T05:47:50.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj51 2022-09-02T05:47:50.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/WG7c/65MKPJ156FwjJdZth/So= 2022-09-02T05:47:50.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/WG7c/65MKPJ156FwjJdZth/So=', 'Content-Length': '0'} 2022-09-02T05:47:50.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b5a3800b2bd2168-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.624 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.625 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.625 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.626 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.626 DEBUG:boto:Method: GET 2022-09-02T05:47:50.627 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.627 DEBUG:boto:Data: 2022-09-02T05:47:50.628 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.629 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.629 DEBUG:boto:Params: {} 2022-09-02T05:47:50.630 DEBUG:boto:Token: None 2022-09-02T05:47:50.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d382a2f17d2817c0-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.636 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.637 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.637 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.638 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.642 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.643 DEBUG:boto:Method: GET 2022-09-02T05:47:50.644 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:50.645 DEBUG:boto:Data: 2022-09-02T05:47:50.646 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.648 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.649 DEBUG:boto:Params: {} 2022-09-02T05:47:50.650 DEBUG:boto:Token: None 2022-09-02T05:47:50.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5926208447e2bed-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.698 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:50.700 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.701 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.701 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.702 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.703 DEBUG:boto:Method: GET 2022-09-02T05:47:50.703 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.704 DEBUG:boto:Data: 2022-09-02T05:47:50.705 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.706 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.707 DEBUG:boto:Params: {} 2022-09-02T05:47:50.707 DEBUG:boto:Token: None 2022-09-02T05:47:50.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE= 2022-09-02T05:47:50.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE=', 'Content-Length': '0'} 2022-09-02T05:47:50.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55c048602edae8b-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.713 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.714 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.715 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.716 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.717 DEBUG:boto:Method: GET 2022-09-02T05:47:50.717 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.718 DEBUG:boto:Data: 2022-09-02T05:47:50.718 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.719 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.720 DEBUG:boto:Params: {} 2022-09-02T05:47:50.721 DEBUG:boto:Token: None 2022-09-02T05:47:50.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE= 2022-09-02T05:47:50.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE=', 'Content-Length': '0'} 2022-09-02T05:47:50.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5d379a5a66728b-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.727 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.728 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.729 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.730 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.731 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.732 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.733 DEBUG:boto:Data: 2022-09-02T05:47:50.734 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.735 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.736 DEBUG:boto:Params: {} 2022-09-02T05:47:50.737 DEBUG:boto:Token: None 2022-09-02T05:47:50.737 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:01MwNcBk9foBi4S2H87IWTHTlIE= 2022-09-02T05:47:50.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:01MwNcBk9foBi4S2H87IWTHTlIE=', 'Content-Length': '0'} 2022-09-02T05:47:50.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a713c9676bcd35d-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.742 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.743 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.743 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.744 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.744 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.745 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.745 DEBUG:boto:Data: 2022-09-02T05:47:50.746 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.747 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.747 DEBUG:boto:Params: {} 2022-09-02T05:47:50.748 DEBUG:boto:Token: None 2022-09-02T05:47:50.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:01MwNcBk9foBi4S2H87IWTHTlIE= 2022-09-02T05:47:50.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:01MwNcBk9foBi4S2H87IWTHTlIE=', 'Content-Length': '0'} 2022-09-02T05:47:50.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac057d4888b6d9a-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:50.753 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.753 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.754 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.754 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.755 DEBUG:boto:Method: GET 2022-09-02T05:47:50.756 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.756 DEBUG:boto:Data: 2022-09-02T05:47:50.757 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.758 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.758 DEBUG:boto:Params: {} 2022-09-02T05:47:50.759 DEBUG:boto:Token: None 2022-09-02T05:47:50.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE= 2022-09-02T05:47:50.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE=', 'Content-Length': '0'} 2022-09-02T05:47:50.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70e5678206825d3-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.764 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.764 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.765 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.765 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.766 DEBUG:boto:Method: GET 2022-09-02T05:47:50.766 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.767 DEBUG:boto:Data: 2022-09-02T05:47:50.767 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.768 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.769 DEBUG:boto:Params: {} 2022-09-02T05:47:50.769 DEBUG:boto:Token: None 2022-09-02T05:47:50.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:50.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE= 2022-09-02T05:47:50.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iiOR3w7/UQ6Av35TUBXdkMlUebE=', 'Content-Length': '0'} 2022-09-02T05:47:50.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd990c639f979093-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.775 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.776 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.776 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.777 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.777 DEBUG:boto:Method: GET 2022-09-02T05:47:50.778 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.779 DEBUG:boto:Data: 2022-09-02T05:47:50.779 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.780 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.780 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.781 DEBUG:boto:Params: {} 2022-09-02T05:47:50.781 DEBUG:boto:Token: None 2022-09-02T05:47:50.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4feead0a3dd556-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.787 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.788 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.788 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.789 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.789 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.790 DEBUG:boto:Method: GET 2022-09-02T05:47:50.790 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.791 DEBUG:boto:Data: 2022-09-02T05:47:50.791 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.792 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.793 DEBUG:boto:Params: {} 2022-09-02T05:47:50.793 DEBUG:boto:Token: None 2022-09-02T05:47:50.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000131a0250d4b473a8-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.799 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.800 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.801 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.801 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.802 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:50.802 DEBUG:boto:Data: 2022-09-02T05:47:50.803 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.804 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.804 DEBUG:boto:Params: {} 2022-09-02T05:47:50.805 DEBUG:boto:Token: None 2022-09-02T05:47:50.806 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55 2022-09-02T05:47:50.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RLpEVCD0pB9PslPhRe4oicyIdjM= 2022-09-02T05:47:50.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RLpEVCD0pB9PslPhRe4oicyIdjM=', 'Content-Length': '0'} 2022-09-02T05:47:50.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc133dd6f6385760-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.810 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.810 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.811 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.811 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:50.812 DEBUG:boto:Data: 2022-09-02T05:47:50.812 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.813 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.814 DEBUG:boto:Params: {} 2022-09-02T05:47:50.814 DEBUG:boto:Token: None 2022-09-02T05:47:50.815 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55 2022-09-02T05:47:50.816 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RLpEVCD0pB9PslPhRe4oicyIdjM= 2022-09-02T05:47:50.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RLpEVCD0pB9PslPhRe4oicyIdjM=', 'Content-Length': '0'} 2022-09-02T05:47:50.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013cfdd6718620abe-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:50.819 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.820 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.821 DEBUG:boto:Method: GET 2022-09-02T05:47:50.821 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:50.822 DEBUG:boto:Data: 2022-09-02T05:47:50.822 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.823 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.824 DEBUG:boto:Params: {} 2022-09-02T05:47:50.824 DEBUG:boto:Token: None 2022-09-02T05:47:50.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55 2022-09-02T05:47:50.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:erGdVn72mXk0CoGINdO8Ml1eN3Q= 2022-09-02T05:47:50.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:erGdVn72mXk0CoGINdO8Ml1eN3Q=', 'Content-Length': '0'} 2022-09-02T05:47:50.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a931f022f6540ddb-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.870 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:50.871 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:50.871 DEBUG:boto:Method: GET 2022-09-02T05:47:50.872 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:50.873 DEBUG:boto:Data: 2022-09-02T05:47:50.873 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.874 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.875 DEBUG:boto:Params: {} 2022-09-02T05:47:50.876 DEBUG:boto:Token: None 2022-09-02T05:47:50.876 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj55 2022-09-02T05:47:50.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:erGdVn72mXk0CoGINdO8Ml1eN3Q= 2022-09-02T05:47:50.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:erGdVn72mXk0CoGINdO8Ml1eN3Q=', 'Content-Length': '0'} 2022-09-02T05:47:50.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c2638ea32400d7-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.881 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.881 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.882 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.882 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.883 DEBUG:boto:Method: GET 2022-09-02T05:47:50.883 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.884 DEBUG:boto:Data: 2022-09-02T05:47:50.884 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.885 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.886 DEBUG:boto:Params: {} 2022-09-02T05:47:50.887 DEBUG:boto:Token: None 2022-09-02T05:47:50.887 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a61f89f82967db-0063119906-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.893 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.895 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:50.896 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:50.897 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.898 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.898 DEBUG:boto:Method: GET 2022-09-02T05:47:50.899 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:50.899 DEBUG:boto:Data: 2022-09-02T05:47:50.900 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.901 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.901 DEBUG:boto:Params: {} 2022-09-02T05:47:50.902 DEBUG:boto:Token: None 2022-09-02T05:47:50.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/ 2022-09-02T05:47:50.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI= 2022-09-02T05:47:50.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fSQpn2m3ImdjZG2hA5+kCm5GGZI=', 'Content-Length': '0'} 2022-09-02T05:47:50.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f68c796926e13a5-0063119906-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.909 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:50.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:50.910 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:50.911 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:50.912 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.913 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.914 DEBUG:boto:Method: GET 2022-09-02T05:47:50.915 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.916 DEBUG:boto:Data: 2022-09-02T05:47:50.917 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.918 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.919 DEBUG:boto:Params: {} 2022-09-02T05:47:50.920 DEBUG:boto:Token: None 2022-09-02T05:47:50.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P9tiF8dbAhIvoQ9TZAwHj6PzvJo= 2022-09-02T05:47:50.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P9tiF8dbAhIvoQ9TZAwHj6PzvJo=', 'Content-Length': '0'} 2022-09-02T05:47:50.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b3a041c8efd5f3-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.925 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:50.925 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:50.926 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.927 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.927 DEBUG:boto:Method: GET 2022-09-02T05:47:50.928 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.928 DEBUG:boto:Data: 2022-09-02T05:47:50.929 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:50.930 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.930 DEBUG:boto:Params: {} 2022-09-02T05:47:50.931 DEBUG:boto:Token: None 2022-09-02T05:47:50.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P9tiF8dbAhIvoQ9TZAwHj6PzvJo= 2022-09-02T05:47:50.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P9tiF8dbAhIvoQ9TZAwHj6PzvJo=', 'Content-Length': '0'} 2022-09-02T05:47:50.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f573cf0f367b4ee-0063119906-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.985 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:50.986 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:50.986 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.987 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.988 DEBUG:boto:Method: HEAD 2022-09-02T05:47:50.988 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.989 DEBUG:boto:Data: 2022-09-02T05:47:50.989 DEBUG:boto:Headers: {} 2022-09-02T05:47:50.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:50.991 DEBUG:boto:Port: 8000 2022-09-02T05:47:50.991 DEBUG:boto:Params: {} 2022-09-02T05:47:50.992 DEBUG:boto:Token: None 2022-09-02T05:47:50.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xWWROE9p37dK4BHqrK+T4ytyBwE= 2022-09-02T05:47:50.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xWWROE9p37dK4BHqrK+T4ytyBwE=', 'Content-Length': '0'} 2022-09-02T05:47:50.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000631938b08d4e6269-0063119906-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:50.997 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:50.997 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:50.998 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.999 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:50.999 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.000 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.000 DEBUG:boto:Data: 2022-09-02T05:47:51.001 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.002 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.002 DEBUG:boto:Params: {} 2022-09-02T05:47:51.003 DEBUG:boto:Token: None 2022-09-02T05:47:51.004 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O60bMxUeSI4auChUcDENlNmB8zk= 2022-09-02T05:47:51.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O60bMxUeSI4auChUcDENlNmB8zk=', 'Content-Length': '0'} 2022-09-02T05:47:51.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefa381c9f0e9500-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:51.009 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:51.009 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:51.010 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.010 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.011 DEBUG:boto:Method: GET 2022-09-02T05:47:51.012 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.012 DEBUG:boto:Data: 2022-09-02T05:47:51.013 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.014 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.015 DEBUG:boto:Params: {} 2022-09-02T05:47:51.015 DEBUG:boto:Token: None 2022-09-02T05:47:51.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CvnpXncIdusOusPS4dVEi6MTrE= 2022-09-02T05:47:51.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CvnpXncIdusOusPS4dVEi6MTrE=', 'Content-Length': '0'} 2022-09-02T05:47:51.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b04bf4a9d43be94-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.020 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:51.021 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:51.022 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.022 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.023 DEBUG:boto:Method: GET 2022-09-02T05:47:51.024 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.024 DEBUG:boto:Data: 2022-09-02T05:47:51.025 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.026 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.026 DEBUG:boto:Params: {} 2022-09-02T05:47:51.027 DEBUG:boto:Token: None 2022-09-02T05:47:51.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:51.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5CvnpXncIdusOusPS4dVEi6MTrE= 2022-09-02T05:47:51.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5CvnpXncIdusOusPS4dVEi6MTrE=', 'Content-Length': '0'} 2022-09-02T05:47:51.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286ca2e1c6c12671-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.032 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.033 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.033 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.034 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.035 DEBUG:boto:Method: GET 2022-09-02T05:47:51.035 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.036 DEBUG:boto:Data: 2022-09-02T05:47:51.036 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.038 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.038 DEBUG:boto:Params: {} 2022-09-02T05:47:51.039 DEBUG:boto:Token: None 2022-09-02T05:47:51.040 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb9bd97f5b5ce12a-0063119907-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.045 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.046 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.047 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.047 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.048 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.048 DEBUG:boto:Method: GET 2022-09-02T05:47:51.049 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.050 DEBUG:boto:Data: 2022-09-02T05:47:51.050 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.051 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.052 DEBUG:boto:Params: {} 2022-09-02T05:47:51.053 DEBUG:boto:Token: None 2022-09-02T05:47:51.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b89945b5cafd1326-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.059 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.059 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:51.060 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:51.061 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.061 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:51.062 DEBUG:boto:Data: 2022-09-02T05:47:51.062 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.063 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.064 DEBUG:boto:Params: {} 2022-09-02T05:47:51.064 DEBUG:boto:Token: None 2022-09-02T05:47:51.065 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68 2022-09-02T05:47:51.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N5LzVaiC5CBborH/ktJQU/nMQ9s= 2022-09-02T05:47:51.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N5LzVaiC5CBborH/ktJQU/nMQ9s=', 'Content-Length': '0'} 2022-09-02T05:47:51.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b0c2d660cc2bc10-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.069 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:51.070 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:51.070 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.071 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:51.071 DEBUG:boto:Data: 2022-09-02T05:47:51.072 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.073 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.073 DEBUG:boto:Params: {} 2022-09-02T05:47:51.074 DEBUG:boto:Token: None 2022-09-02T05:47:51.075 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68 2022-09-02T05:47:51.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N5LzVaiC5CBborH/ktJQU/nMQ9s= 2022-09-02T05:47:51.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N5LzVaiC5CBborH/ktJQU/nMQ9s=', 'Content-Length': '0'} 2022-09-02T05:47:51.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a6153c36e390ef-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:51.080 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:51.080 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:51.081 DEBUG:boto:Method: GET 2022-09-02T05:47:51.082 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:51.082 DEBUG:boto:Data: 2022-09-02T05:47:51.083 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.084 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.085 DEBUG:boto:Params: {} 2022-09-02T05:47:51.086 DEBUG:boto:Token: None 2022-09-02T05:47:51.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68 2022-09-02T05:47:51.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TUw5tjg+hy1ZjvZlzyl8u2OwPd4= 2022-09-02T05:47:51.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TUw5tjg+hy1ZjvZlzyl8u2OwPd4=', 'Content-Length': '0'} 2022-09-02T05:47:51.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453ae550a65fc860-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.091 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:51.091 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:51.092 DEBUG:boto:Method: GET 2022-09-02T05:47:51.093 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:51.093 DEBUG:boto:Data: 2022-09-02T05:47:51.094 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.095 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.096 DEBUG:boto:Params: {} 2022-09-02T05:47:51.096 DEBUG:boto:Token: None 2022-09-02T05:47:51.097 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj68 2022-09-02T05:47:51.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TUw5tjg+hy1ZjvZlzyl8u2OwPd4= 2022-09-02T05:47:51.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TUw5tjg+hy1ZjvZlzyl8u2OwPd4=', 'Content-Length': '0'} 2022-09-02T05:47:51.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009234e0c3524f6486-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.142 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.143 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.143 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.144 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.145 DEBUG:boto:Method: GET 2022-09-02T05:47:51.145 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.146 DEBUG:boto:Data: 2022-09-02T05:47:51.147 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.149 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.149 DEBUG:boto:Params: {} 2022-09-02T05:47:51.150 DEBUG:boto:Token: None 2022-09-02T05:47:51.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278d271c29d14df5-0063119907-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.156 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.157 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.157 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.158 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.158 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.159 DEBUG:boto:Method: GET 2022-09-02T05:47:51.160 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:51.160 DEBUG:boto:Data: 2022-09-02T05:47:51.161 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.162 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.162 DEBUG:boto:Params: {} 2022-09-02T05:47:51.164 DEBUG:boto:Token: None 2022-09-02T05:47:51.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009712f87be0a5ae59-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.170 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:51.172 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.172 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.173 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.173 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.174 DEBUG:boto:Method: GET 2022-09-02T05:47:51.174 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.175 DEBUG:boto:Data: 2022-09-02T05:47:51.176 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.176 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.177 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.177 DEBUG:boto:Params: {} 2022-09-02T05:47:51.178 DEBUG:boto:Token: None 2022-09-02T05:47:51.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE= 2022-09-02T05:47:51.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE=', 'Content-Length': '0'} 2022-09-02T05:47:51.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0aa9c70bf62829a-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.234 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.235 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.236 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.237 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.238 DEBUG:boto:Method: GET 2022-09-02T05:47:51.239 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.240 DEBUG:boto:Data: 2022-09-02T05:47:51.241 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.243 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.244 DEBUG:boto:Params: {} 2022-09-02T05:47:51.245 DEBUG:boto:Token: None 2022-09-02T05:47:51.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE= 2022-09-02T05:47:51.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE=', 'Content-Length': '0'} 2022-09-02T05:47:51.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fbce78917170eec-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.258 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.260 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.262 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.263 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.264 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.265 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.266 DEBUG:boto:Data: 2022-09-02T05:47:51.267 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.269 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.270 DEBUG:boto:Params: {} 2022-09-02T05:47:51.475 DEBUG:boto:Token: None 2022-09-02T05:47:51.476 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TUv+25SB043wKUFY7GUYoLCDHU4= 2022-09-02T05:47:51.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TUv+25SB043wKUFY7GUYoLCDHU4=', 'Content-Length': '0'} 2022-09-02T05:47:51.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b797246169579b6-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.481 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.481 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.482 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.482 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.483 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.484 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.484 DEBUG:boto:Data: 2022-09-02T05:47:51.485 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.486 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.486 DEBUG:boto:Params: {} 2022-09-02T05:47:51.487 DEBUG:boto:Token: None 2022-09-02T05:47:51.488 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.489 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TUv+25SB043wKUFY7GUYoLCDHU4= 2022-09-02T05:47:51.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TUv+25SB043wKUFY7GUYoLCDHU4=', 'Content-Length': '0'} 2022-09-02T05:47:51.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2cafa670cd2e1f3-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:51.494 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.495 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.495 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.496 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.497 DEBUG:boto:Method: GET 2022-09-02T05:47:51.497 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.498 DEBUG:boto:Data: 2022-09-02T05:47:51.499 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.500 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.501 DEBUG:boto:Params: {} 2022-09-02T05:47:51.502 DEBUG:boto:Token: None 2022-09-02T05:47:51.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE= 2022-09-02T05:47:51.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE=', 'Content-Length': '0'} 2022-09-02T05:47:51.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001229db332b44ca05-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.511 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.512 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.513 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.514 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.515 DEBUG:boto:Method: GET 2022-09-02T05:47:51.516 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.516 DEBUG:boto:Data: 2022-09-02T05:47:51.517 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.518 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.519 DEBUG:boto:Params: {} 2022-09-02T05:47:51.520 DEBUG:boto:Token: None 2022-09-02T05:47:51.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:51.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE= 2022-09-02T05:47:51.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CINJ/21SudeNhHARZpK+qNaVLaE=', 'Content-Length': '0'} 2022-09-02T05:47:51.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b664d8b6d40f122d-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.527 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.528 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.529 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.529 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.530 DEBUG:boto:Method: GET 2022-09-02T05:47:51.531 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.531 DEBUG:boto:Data: 2022-09-02T05:47:51.532 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.533 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.534 DEBUG:boto:Params: {} 2022-09-02T05:47:51.535 DEBUG:boto:Token: None 2022-09-02T05:47:51.536 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005625f21651da9b5-0063119907-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.541 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.542 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.543 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.544 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.545 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.545 DEBUG:boto:Method: GET 2022-09-02T05:47:51.546 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.547 DEBUG:boto:Data: 2022-09-02T05:47:51.547 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.548 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.549 DEBUG:boto:Params: {} 2022-09-02T05:47:51.550 DEBUG:boto:Token: None 2022-09-02T05:47:51.551 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007862a1f145632809-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.557 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.558 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.559 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.560 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.561 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:51.561 DEBUG:boto:Data: 2022-09-02T05:47:51.562 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.563 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.564 DEBUG:boto:Params: {} 2022-09-02T05:47:51.564 DEBUG:boto:Token: None 2022-09-02T05:47:51.565 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85 2022-09-02T05:47:51.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T4pEIS+RGsB73nM0L/oIVY5QNVk= 2022-09-02T05:47:51.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T4pEIS+RGsB73nM0L/oIVY5QNVk=', 'Content-Length': '0'} 2022-09-02T05:47:51.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374870578c6dfbcb-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.570 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.571 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.572 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.572 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:51.573 DEBUG:boto:Data: 2022-09-02T05:47:51.573 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.575 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.575 DEBUG:boto:Params: {} 2022-09-02T05:47:51.576 DEBUG:boto:Token: None 2022-09-02T05:47:51.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85 2022-09-02T05:47:51.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T4pEIS+RGsB73nM0L/oIVY5QNVk= 2022-09-02T05:47:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T4pEIS+RGsB73nM0L/oIVY5QNVk=', 'Content-Length': '0'} 2022-09-02T05:47:51.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdaad2c3dd53a34-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:51.582 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.582 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.583 DEBUG:boto:Method: GET 2022-09-02T05:47:51.584 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:51.585 DEBUG:boto:Data: 2022-09-02T05:47:51.585 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.587 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.587 DEBUG:boto:Params: {} 2022-09-02T05:47:51.588 DEBUG:boto:Token: None 2022-09-02T05:47:51.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85 2022-09-02T05:47:51.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3FvHdwvHeSpDIjWzu2PJxfh2oeo= 2022-09-02T05:47:51.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3FvHdwvHeSpDIjWzu2PJxfh2oeo=', 'Content-Length': '0'} 2022-09-02T05:47:51.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017eec1a3981afdf5-0063119907-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.593 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:51.594 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:51.595 DEBUG:boto:Method: GET 2022-09-02T05:47:51.595 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:51.596 DEBUG:boto:Data: 2022-09-02T05:47:51.597 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.598 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.598 DEBUG:boto:Params: {} 2022-09-02T05:47:51.599 DEBUG:boto:Token: None 2022-09-02T05:47:51.599 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj85 2022-09-02T05:47:51.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3FvHdwvHeSpDIjWzu2PJxfh2oeo= 2022-09-02T05:47:51.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3FvHdwvHeSpDIjWzu2PJxfh2oeo=', 'Content-Length': '0'} 2022-09-02T05:47:51.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006945920087d429de-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.658 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.659 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.660 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.661 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.662 DEBUG:boto:Method: GET 2022-09-02T05:47:51.663 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.664 DEBUG:boto:Data: 2022-09-02T05:47:51.665 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:51.668 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.669 DEBUG:boto:Params: {} 2022-09-02T05:47:51.670 DEBUG:boto:Token: None 2022-09-02T05:47:51.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98e1566b80964d4-0063119907-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.675 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.676 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.677 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.677 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.678 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.679 DEBUG:boto:Method: GET 2022-09-02T05:47:51.679 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:51.680 DEBUG:boto:Data: 2022-09-02T05:47:51.681 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.682 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.683 DEBUG:boto:Params: {} 2022-09-02T05:47:51.683 DEBUG:boto:Token: None 2022-09-02T05:47:51.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b1569d6495abf87-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.690 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.691 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:47:51.692 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:47:51.692 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.693 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.693 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.694 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:51.694 DEBUG:boto:Data: 2022-09-02T05:47:51.695 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.696 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.697 DEBUG:boto:Params: {} 2022-09-02T05:47:51.698 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:47:51.698 DEBUG:boto:Token: None 2022-09-02T05:47:51.699 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDAXwT5/hRuUXcrDoFPljXJ+vJw= 2022-09-02T05:47:51.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDAXwT5/hRuUXcrDoFPljXJ+vJw=', 'Content-Length': '0'} 2022-09-02T05:47:51.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f6d893082dfd82e-0063119907-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.705 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.705 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.706 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.707 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:51.708 DEBUG:boto:Data: 2022-09-02T05:47:51.708 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.709 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.710 DEBUG:boto:Params: {} 2022-09-02T05:47:51.710 DEBUG:boto:Token: None 2022-09-02T05:47:51.711 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDAXwT5/hRuUXcrDoFPljXJ+vJw= 2022-09-02T05:47:51.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDAXwT5/hRuUXcrDoFPljXJ+vJw=', 'Content-Length': '0'} 2022-09-02T05:47:51.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9a754390a7af338-0063119907-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.716 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:47:51.717 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.717 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.718 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.718 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.719 DEBUG:boto:Method: GET 2022-09-02T05:47:51.720 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.720 DEBUG:boto:Data: 2022-09-02T05:47:51.721 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.722 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.722 DEBUG:boto:Params: {} 2022-09-02T05:47:51.723 DEBUG:boto:Token: None 2022-09-02T05:47:51.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/?versions 2022-09-02T05:47:51.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck= 2022-09-02T05:47:51.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck=', 'Content-Length': '0'} 2022-09-02T05:47:51.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d374b38bd7b4fb0-0063119907-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.728 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:47:51.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:47:51.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:47:51.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:47:51.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:47:51.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:47:51.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:47:51.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:47:51.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:47:51.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:47:51.734 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:47:51.735 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.735 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.736 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.736 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.737 DEBUG:boto:Method: GET 2022-09-02T05:47:51.737 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.738 DEBUG:boto:Data: 2022-09-02T05:47:51.738 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.739 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.740 DEBUG:boto:Params: {} 2022-09-02T05:47:51.740 DEBUG:boto:Token: None 2022-09-02T05:47:51.741 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/?versions 2022-09-02T05:47:51.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck= 2022-09-02T05:47:51.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck=', 'Content-Length': '0'} 2022-09-02T05:47:51.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001085acfa2e2840dc-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.747 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:47:51.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:47:51.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:47:51.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:47:51.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:47:51.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:47:51.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:47:51.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:47:51.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:47:51.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:47:51.753 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.753 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.754 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.754 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.755 DEBUG:boto:Method: GET 2022-09-02T05:47:51.755 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.756 DEBUG:boto:Data: 2022-09-02T05:47:51.756 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.757 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.758 DEBUG:boto:Params: {} 2022-09-02T05:47:51.759 DEBUG:boto:Token: None 2022-09-02T05:47:51.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/?versions 2022-09-02T05:47:51.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck= 2022-09-02T05:47:51.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck=', 'Content-Length': '0'} 2022-09-02T05:47:51.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011705c7209c9627d-0063119907-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.764 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.765 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.766 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.766 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.767 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.767 DEBUG:boto:Method: GET 2022-09-02T05:47:51.768 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:51.768 DEBUG:boto:Data: 2022-09-02T05:47:51.769 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.770 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.770 DEBUG:boto:Params: {} 2022-09-02T05:47:51.771 DEBUG:boto:Token: None 2022-09-02T05:47:51.771 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/?versions 2022-09-02T05:47:51.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck= 2022-09-02T05:47:51.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mqfmQEwQAfFI8woBhu5ODcNb5Ck=', 'Content-Length': '0'} 2022-09-02T05:47:51.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6178553ac0a54d8-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.776 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:51.778 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.779 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.779 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.780 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.781 DEBUG:boto:Method: GET 2022-09-02T05:47:51.781 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.782 DEBUG:boto:Data: 2022-09-02T05:47:51.782 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.784 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.784 DEBUG:boto:Params: {} 2022-09-02T05:47:51.785 DEBUG:boto:Token: None 2022-09-02T05:47:51.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8= 2022-09-02T05:47:51.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8=', 'Content-Length': '0'} 2022-09-02T05:47:51.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df01aef253061ed5-0063119907-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.790 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.790 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.791 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.792 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.793 DEBUG:boto:Method: GET 2022-09-02T05:47:51.794 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.795 DEBUG:boto:Data: 2022-09-02T05:47:51.796 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.798 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.799 DEBUG:boto:Params: {} 2022-09-02T05:47:51.800 DEBUG:boto:Token: None 2022-09-02T05:47:51.801 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8= 2022-09-02T05:47:51.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8=', 'Content-Length': '0'} 2022-09-02T05:47:51.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8518500bea363aa-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.846 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.847 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.847 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.848 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.848 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.849 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.849 DEBUG:boto:Data: 2022-09-02T05:47:51.850 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.851 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.852 DEBUG:boto:Params: {} 2022-09-02T05:47:51.852 DEBUG:boto:Token: None 2022-09-02T05:47:51.853 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTM+l/iGCnMMz4cDWjUoCytWpPo= 2022-09-02T05:47:51.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTM+l/iGCnMMz4cDWjUoCytWpPo=', 'Content-Length': '0'} 2022-09-02T05:47:51.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deaeb4ad44f8778e-0063119907-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.857 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.857 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.858 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.859 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.859 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.860 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.861 DEBUG:boto:Data: 2022-09-02T05:47:51.862 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.863 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.864 DEBUG:boto:Params: {} 2022-09-02T05:47:51.865 DEBUG:boto:Token: None 2022-09-02T05:47:51.866 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tTM+l/iGCnMMz4cDWjUoCytWpPo= 2022-09-02T05:47:51.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tTM+l/iGCnMMz4cDWjUoCytWpPo=', 'Content-Length': '0'} 2022-09-02T05:47:51.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095177dfb9e4481ab-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:51.871 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.872 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.873 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.873 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.874 DEBUG:boto:Method: GET 2022-09-02T05:47:51.874 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.875 DEBUG:boto:Data: 2022-09-02T05:47:51.876 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.877 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.877 DEBUG:boto:Params: {} 2022-09-02T05:47:51.878 DEBUG:boto:Token: None 2022-09-02T05:47:51.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.879 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8= 2022-09-02T05:47:51.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8=', 'Content-Length': '0'} 2022-09-02T05:47:51.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052951f39ae16ce6c-0063119907-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.883 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.884 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.884 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.885 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.886 DEBUG:boto:Method: GET 2022-09-02T05:47:51.886 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.887 DEBUG:boto:Data: 2022-09-02T05:47:51.887 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.889 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.889 DEBUG:boto:Params: {} 2022-09-02T05:47:51.890 DEBUG:boto:Token: None 2022-09-02T05:47:51.891 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:51.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8= 2022-09-02T05:47:51.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c48yVIL/bf9lorgGxh2vCSy1MQ8=', 'Content-Length': '0'} 2022-09-02T05:47:51.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656adfc1816662b4-0063119907-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.895 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.896 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.896 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:51.897 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:51.897 DEBUG:boto:Method: GET 2022-09-02T05:47:51.898 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:51.899 DEBUG:boto:Data: 2022-09-02T05:47:51.899 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.900 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.901 DEBUG:boto:Params: {} 2022-09-02T05:47:51.901 DEBUG:boto:Token: None 2022-09-02T05:47:51.902 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8999231f2535b9-0063119907-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.950 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.952 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:51.952 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:51.953 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:51.954 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:51.955 DEBUG:boto:Method: GET 2022-09-02T05:47:51.956 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:51.957 DEBUG:boto:Data: 2022-09-02T05:47:51.958 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.960 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.961 DEBUG:boto:Params: {} 2022-09-02T05:47:51.962 DEBUG:boto:Token: None 2022-09-02T05:47:51.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/ 2022-09-02T05:47:51.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0= 2022-09-02T05:47:51.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KzRcflJXZXNdiGE2pi+2VfNJLd0=', 'Content-Length': '0'} 2022-09-02T05:47:51.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e85b1a17c8bc09c-0063119907-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.971 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:51.972 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.973 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.974 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.975 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:51.976 DEBUG:boto:Data: 2022-09-02T05:47:51.977 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:51.979 DEBUG:boto:Port: 8001 2022-09-02T05:47:51.980 DEBUG:boto:Params: {} 2022-09-02T05:47:51.981 DEBUG:boto:Token: None 2022-09-02T05:47:51.982 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11 2022-09-02T05:47:51.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iZ/CQomr6VDdRDnM0K9NXTR4kkk= 2022-09-02T05:47:51.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iZ/CQomr6VDdRDnM0K9NXTR4kkk=', 'Content-Length': '0'} 2022-09-02T05:47:51.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede7b488565fcf1f-0063119907-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:51.988 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:51.989 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:51.990 DEBUG:boto:Method: HEAD 2022-09-02T05:47:51.991 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:51.992 DEBUG:boto:Data: 2022-09-02T05:47:51.993 DEBUG:boto:Headers: {} 2022-09-02T05:47:51.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:51.995 DEBUG:boto:Port: 8000 2022-09-02T05:47:51.996 DEBUG:boto:Params: {} 2022-09-02T05:47:51.998 DEBUG:boto:Token: None 2022-09-02T05:47:51.999 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:51 GMT /xljbkf-74/obj11 2022-09-02T05:47:52.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iZ/CQomr6VDdRDnM0K9NXTR4kkk= 2022-09-02T05:47:52.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iZ/CQomr6VDdRDnM0K9NXTR4kkk=', 'Content-Length': '0'} 2022-09-02T05:47:52.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c54eb04accf86fe8-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:52.006 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:52.007 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:52.008 DEBUG:boto:Method: GET 2022-09-02T05:47:52.009 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:52.010 DEBUG:boto:Data: 2022-09-02T05:47:52.011 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.013 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.014 DEBUG:boto:Params: {} 2022-09-02T05:47:52.015 DEBUG:boto:Token: None 2022-09-02T05:47:52.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj11 2022-09-02T05:47:52.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NmqHfXMNN/YH+vwhpoIzmpPmg4A= 2022-09-02T05:47:52.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NmqHfXMNN/YH+vwhpoIzmpPmg4A=', 'Content-Length': '0'} 2022-09-02T05:47:52.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000853943e7504f29f7-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.022 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:52.022 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:52.023 DEBUG:boto:Method: GET 2022-09-02T05:47:52.024 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:52.024 DEBUG:boto:Data: 2022-09-02T05:47:52.025 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.026 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.027 DEBUG:boto:Params: {} 2022-09-02T05:47:52.027 DEBUG:boto:Token: None 2022-09-02T05:47:52.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj11 2022-09-02T05:47:52.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NmqHfXMNN/YH+vwhpoIzmpPmg4A= 2022-09-02T05:47:52.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NmqHfXMNN/YH+vwhpoIzmpPmg4A=', 'Content-Length': '0'} 2022-09-02T05:47:52.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c27e73ce6dd33b88-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.033 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.033 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.034 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:52.034 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:52.035 DEBUG:boto:Method: GET 2022-09-02T05:47:52.036 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:52.036 DEBUG:boto:Data: 2022-09-02T05:47:52.037 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.038 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.039 DEBUG:boto:Params: {} 2022-09-02T05:47:52.040 DEBUG:boto:Token: None 2022-09-02T05:47:52.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d848b503b9b8b3b-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.046 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.047 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.048 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.048 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:52.049 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:52.049 DEBUG:boto:Method: GET 2022-09-02T05:47:52.050 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:52.050 DEBUG:boto:Data: 2022-09-02T05:47:52.051 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.052 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.052 DEBUG:boto:Params: {} 2022-09-02T05:47:52.053 DEBUG:boto:Token: None 2022-09-02T05:47:52.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000614e4c71b1c1d95c-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.059 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:52.061 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.062 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.062 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.063 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.063 DEBUG:boto:Method: GET 2022-09-02T05:47:52.064 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.064 DEBUG:boto:Data: 2022-09-02T05:47:52.065 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.066 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.067 DEBUG:boto:Params: {} 2022-09-02T05:47:52.067 DEBUG:boto:Token: None 2022-09-02T05:47:52.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw= 2022-09-02T05:47:52.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw=', 'Content-Length': '0'} 2022-09-02T05:47:52.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b4cfc1a294487f1-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.072 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.073 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.073 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.074 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.075 DEBUG:boto:Method: GET 2022-09-02T05:47:52.075 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.076 DEBUG:boto:Data: 2022-09-02T05:47:52.076 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.077 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.078 DEBUG:boto:Params: {} 2022-09-02T05:47:52.078 DEBUG:boto:Token: None 2022-09-02T05:47:52.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw= 2022-09-02T05:47:52.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw=', 'Content-Length': '0'} 2022-09-02T05:47:52.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e22c0ee5ec82dc7-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.084 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.084 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.085 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.085 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.086 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.086 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.087 DEBUG:boto:Data: 2022-09-02T05:47:52.087 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.089 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.089 DEBUG:boto:Params: {} 2022-09-02T05:47:52.090 DEBUG:boto:Token: None 2022-09-02T05:47:52.091 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CkkBMU8nG5ZTV+AlG+Cofu13EUo= 2022-09-02T05:47:52.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CkkBMU8nG5ZTV+AlG+Cofu13EUo=', 'Content-Length': '0'} 2022-09-02T05:47:52.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023370f5f5ac8b5e0-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.095 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.096 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.096 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.097 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.097 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.098 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.099 DEBUG:boto:Data: 2022-09-02T05:47:52.099 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.100 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.101 DEBUG:boto:Params: {} 2022-09-02T05:47:52.102 DEBUG:boto:Token: None 2022-09-02T05:47:52.103 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CkkBMU8nG5ZTV+AlG+Cofu13EUo= 2022-09-02T05:47:52.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CkkBMU8nG5ZTV+AlG+Cofu13EUo=', 'Content-Length': '0'} 2022-09-02T05:47:52.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a62652c9a38b9ba-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:52.110 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.111 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.112 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.113 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.114 DEBUG:boto:Method: GET 2022-09-02T05:47:52.116 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.117 DEBUG:boto:Data: 2022-09-02T05:47:52.118 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.121 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.122 DEBUG:boto:Params: {} 2022-09-02T05:47:52.124 DEBUG:boto:Token: None 2022-09-02T05:47:52.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw= 2022-09-02T05:47:52.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw=', 'Content-Length': '0'} 2022-09-02T05:47:52.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e00ead96471f244-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.174 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.175 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.176 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.177 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.177 DEBUG:boto:Method: GET 2022-09-02T05:47:52.178 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.179 DEBUG:boto:Data: 2022-09-02T05:47:52.180 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.182 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.183 DEBUG:boto:Params: {} 2022-09-02T05:47:52.184 DEBUG:boto:Token: None 2022-09-02T05:47:52.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:52.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw= 2022-09-02T05:47:52.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H2tP0S5kc7EgEkC/I88Rri5oHZw=', 'Content-Length': '0'} 2022-09-02T05:47:52.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c2744e8dbd84ec-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.191 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.192 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.193 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.193 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.194 DEBUG:boto:Method: GET 2022-09-02T05:47:52.195 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.196 DEBUG:boto:Data: 2022-09-02T05:47:52.197 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.198 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.199 DEBUG:boto:Params: {} 2022-09-02T05:47:52.200 DEBUG:boto:Token: None 2022-09-02T05:47:52.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003230b34b6dd42d1a-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.207 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.208 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.209 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.210 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.211 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.212 DEBUG:boto:Method: GET 2022-09-02T05:47:52.212 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.213 DEBUG:boto:Data: 2022-09-02T05:47:52.214 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.215 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.216 DEBUG:boto:Params: {} 2022-09-02T05:47:52.216 DEBUG:boto:Token: None 2022-09-02T05:47:52.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000866b33b53674e6bd-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.224 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.225 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.225 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.226 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.227 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:52.228 DEBUG:boto:Data: 2022-09-02T05:47:52.228 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.230 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.230 DEBUG:boto:Params: {} 2022-09-02T05:47:52.231 DEBUG:boto:Token: None 2022-09-02T05:47:52.232 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17 2022-09-02T05:47:52.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nx9wwwP7WQSUseX3x/zIJCLTEk8= 2022-09-02T05:47:52.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nx9wwwP7WQSUseX3x/zIJCLTEk8=', 'Content-Length': '0'} 2022-09-02T05:47:52.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c73f8f1f21363c-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.236 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.237 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.238 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.239 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:52.239 DEBUG:boto:Data: 2022-09-02T05:47:52.240 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.241 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.242 DEBUG:boto:Params: {} 2022-09-02T05:47:52.243 DEBUG:boto:Token: None 2022-09-02T05:47:52.243 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17 2022-09-02T05:47:52.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nx9wwwP7WQSUseX3x/zIJCLTEk8= 2022-09-02T05:47:52.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nx9wwwP7WQSUseX3x/zIJCLTEk8=', 'Content-Length': '0'} 2022-09-02T05:47:52.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e053d6f3bfea45-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:52.249 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.250 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.250 DEBUG:boto:Method: GET 2022-09-02T05:47:52.251 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:52.252 DEBUG:boto:Data: 2022-09-02T05:47:52.252 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.254 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.255 DEBUG:boto:Params: {} 2022-09-02T05:47:52.256 DEBUG:boto:Token: None 2022-09-02T05:47:52.257 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17 2022-09-02T05:47:52.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mJRJaNyXNVV5vJBmhAHi1XAGaZc= 2022-09-02T05:47:52.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mJRJaNyXNVV5vJBmhAHi1XAGaZc=', 'Content-Length': '0'} 2022-09-02T05:47:52.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8d118d64c4d92a-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.261 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:52.262 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:52.262 DEBUG:boto:Method: GET 2022-09-02T05:47:52.263 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:52.264 DEBUG:boto:Data: 2022-09-02T05:47:52.264 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.266 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.266 DEBUG:boto:Params: {} 2022-09-02T05:47:52.267 DEBUG:boto:Token: None 2022-09-02T05:47:52.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj17 2022-09-02T05:47:52.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mJRJaNyXNVV5vJBmhAHi1XAGaZc= 2022-09-02T05:47:52.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mJRJaNyXNVV5vJBmhAHi1XAGaZc=', 'Content-Length': '0'} 2022-09-02T05:47:52.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006432e6ac02c260bd-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.272 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.272 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.273 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.274 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.274 DEBUG:boto:Method: GET 2022-09-02T05:47:52.275 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.276 DEBUG:boto:Data: 2022-09-02T05:47:52.276 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.277 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.278 DEBUG:boto:Params: {} 2022-09-02T05:47:52.278 DEBUG:boto:Token: None 2022-09-02T05:47:52.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099f9aac6c4919c01-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.284 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.285 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.285 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.286 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.286 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.287 DEBUG:boto:Method: GET 2022-09-02T05:47:52.288 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:52.289 DEBUG:boto:Data: 2022-09-02T05:47:52.289 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.290 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.291 DEBUG:boto:Params: {} 2022-09-02T05:47:52.292 DEBUG:boto:Token: None 2022-09-02T05:47:52.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ee54f8a65fe9787-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.298 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:52.299 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.300 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.301 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.301 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.302 DEBUG:boto:Method: GET 2022-09-02T05:47:52.302 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.303 DEBUG:boto:Data: 2022-09-02T05:47:52.303 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.304 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.305 DEBUG:boto:Params: {} 2022-09-02T05:47:52.306 DEBUG:boto:Token: None 2022-09-02T05:47:52.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY= 2022-09-02T05:47:52.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY=', 'Content-Length': '0'} 2022-09-02T05:47:52.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004feb17d4e7283b24-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.351 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.351 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.352 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.352 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.353 DEBUG:boto:Method: GET 2022-09-02T05:47:52.354 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.354 DEBUG:boto:Data: 2022-09-02T05:47:52.355 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.356 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.357 DEBUG:boto:Params: {} 2022-09-02T05:47:52.357 DEBUG:boto:Token: None 2022-09-02T05:47:52.358 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY= 2022-09-02T05:47:52.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY=', 'Content-Length': '0'} 2022-09-02T05:47:52.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464ea58c91740ed3-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.364 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.365 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.366 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.367 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.367 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.368 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.368 DEBUG:boto:Data: 2022-09-02T05:47:52.369 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.370 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.371 DEBUG:boto:Params: {} 2022-09-02T05:47:52.371 DEBUG:boto:Token: None 2022-09-02T05:47:52.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yLBGJXgZVOyfVHATi9OjktgCjUY= 2022-09-02T05:47:52.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yLBGJXgZVOyfVHATi9OjktgCjUY=', 'Content-Length': '0'} 2022-09-02T05:47:52.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041db089b9eb95998-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.378 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.379 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.379 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.380 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.380 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.381 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.381 DEBUG:boto:Data: 2022-09-02T05:47:52.382 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.383 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.383 DEBUG:boto:Params: {} 2022-09-02T05:47:52.384 DEBUG:boto:Token: None 2022-09-02T05:47:52.385 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yLBGJXgZVOyfVHATi9OjktgCjUY= 2022-09-02T05:47:52.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yLBGJXgZVOyfVHATi9OjktgCjUY=', 'Content-Length': '0'} 2022-09-02T05:47:52.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e468ac6d96959d2-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:52.390 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.390 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.391 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.391 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.392 DEBUG:boto:Method: GET 2022-09-02T05:47:52.393 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.393 DEBUG:boto:Data: 2022-09-02T05:47:52.394 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.396 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.396 DEBUG:boto:Params: {} 2022-09-02T05:47:52.397 DEBUG:boto:Token: None 2022-09-02T05:47:52.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY= 2022-09-02T05:47:52.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY=', 'Content-Length': '0'} 2022-09-02T05:47:52.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9a07f6e299feec-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.402 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.403 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.403 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.404 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.404 DEBUG:boto:Method: GET 2022-09-02T05:47:52.405 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.405 DEBUG:boto:Data: 2022-09-02T05:47:52.406 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.407 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.408 DEBUG:boto:Params: {} 2022-09-02T05:47:52.408 DEBUG:boto:Token: None 2022-09-02T05:47:52.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:52.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY= 2022-09-02T05:47:52.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7sxHy62KBdVaafGf/oVfXd4inZY=', 'Content-Length': '0'} 2022-09-02T05:47:52.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec4cc34f9658ae6-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.454 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.455 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.456 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.456 DEBUG:boto:Method: GET 2022-09-02T05:47:52.457 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.458 DEBUG:boto:Data: 2022-09-02T05:47:52.459 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.460 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.460 DEBUG:boto:Params: {} 2022-09-02T05:47:52.461 DEBUG:boto:Token: None 2022-09-02T05:47:52.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a24e1228dde69ff-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.466 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.467 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.468 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.468 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.469 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.469 DEBUG:boto:Method: GET 2022-09-02T05:47:52.470 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.470 DEBUG:boto:Data: 2022-09-02T05:47:52.471 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.472 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.473 DEBUG:boto:Params: {} 2022-09-02T05:47:52.473 DEBUG:boto:Token: None 2022-09-02T05:47:52.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019672ede24cfc386-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.479 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.480 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.480 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.481 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.482 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:52.482 DEBUG:boto:Data: 2022-09-02T05:47:52.483 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.484 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.484 DEBUG:boto:Params: {} 2022-09-02T05:47:52.485 DEBUG:boto:Token: None 2022-09-02T05:47:52.485 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22 2022-09-02T05:47:52.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TyT/o9Wo3VnAKL4N1IUHtUn3do= 2022-09-02T05:47:52.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TyT/o9Wo3VnAKL4N1IUHtUn3do=', 'Content-Length': '0'} 2022-09-02T05:47:52.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d51c947d3bf11f-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.489 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.490 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.490 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.491 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:52.491 DEBUG:boto:Data: 2022-09-02T05:47:52.492 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.493 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.493 DEBUG:boto:Params: {} 2022-09-02T05:47:52.494 DEBUG:boto:Token: None 2022-09-02T05:47:52.495 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22 2022-09-02T05:47:52.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2TyT/o9Wo3VnAKL4N1IUHtUn3do= 2022-09-02T05:47:52.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2TyT/o9Wo3VnAKL4N1IUHtUn3do=', 'Content-Length': '0'} 2022-09-02T05:47:52.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c28874e20fb726-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:52.499 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.500 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.500 DEBUG:boto:Method: GET 2022-09-02T05:47:52.501 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:52.501 DEBUG:boto:Data: 2022-09-02T05:47:52.502 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.503 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.503 DEBUG:boto:Params: {} 2022-09-02T05:47:52.505 DEBUG:boto:Token: None 2022-09-02T05:47:52.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22 2022-09-02T05:47:52.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ewlj1MsdCCUfPPU373RQ/KtDeOc= 2022-09-02T05:47:52.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ewlj1MsdCCUfPPU373RQ/KtDeOc=', 'Content-Length': '0'} 2022-09-02T05:47:52.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000859c4b071994999d-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.511 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:52.513 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:52.514 DEBUG:boto:Method: GET 2022-09-02T05:47:52.515 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:52.516 DEBUG:boto:Data: 2022-09-02T05:47:52.517 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.519 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.520 DEBUG:boto:Params: {} 2022-09-02T05:47:52.521 DEBUG:boto:Token: None 2022-09-02T05:47:52.524 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj22 2022-09-02T05:47:52.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ewlj1MsdCCUfPPU373RQ/KtDeOc= 2022-09-02T05:47:52.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ewlj1MsdCCUfPPU373RQ/KtDeOc=', 'Content-Length': '0'} 2022-09-02T05:47:52.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000443f7501b338e388-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.531 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.532 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.533 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.534 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.535 DEBUG:boto:Method: GET 2022-09-02T05:47:52.536 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.537 DEBUG:boto:Data: 2022-09-02T05:47:52.538 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.540 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.541 DEBUG:boto:Params: {} 2022-09-02T05:47:52.542 DEBUG:boto:Token: None 2022-09-02T05:47:52.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2479194c8472fb-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.549 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.551 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.552 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.553 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.554 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.555 DEBUG:boto:Method: GET 2022-09-02T05:47:52.556 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:52.557 DEBUG:boto:Data: 2022-09-02T05:47:52.558 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.560 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.561 DEBUG:boto:Params: {} 2022-09-02T05:47:52.563 DEBUG:boto:Token: None 2022-09-02T05:47:52.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003daf408c02b10f09-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.571 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:52.573 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.574 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.575 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.576 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.577 DEBUG:boto:Method: GET 2022-09-02T05:47:52.578 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.579 DEBUG:boto:Data: 2022-09-02T05:47:52.580 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.582 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.583 DEBUG:boto:Params: {} 2022-09-02T05:47:52.584 DEBUG:boto:Token: None 2022-09-02T05:47:52.586 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU= 2022-09-02T05:47:52.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU=', 'Content-Length': '0'} 2022-09-02T05:47:52.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003b9e9be365a1064-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.592 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.594 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.595 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.596 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.597 DEBUG:boto:Method: GET 2022-09-02T05:47:52.598 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.599 DEBUG:boto:Data: 2022-09-02T05:47:52.600 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.602 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.603 DEBUG:boto:Params: {} 2022-09-02T05:47:52.604 DEBUG:boto:Token: None 2022-09-02T05:47:52.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU= 2022-09-02T05:47:52.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU=', 'Content-Length': '0'} 2022-09-02T05:47:52.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22da394dbe94c06-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.611 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.612 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.613 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.614 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.614 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.615 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.615 DEBUG:boto:Data: 2022-09-02T05:47:52.616 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.617 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.617 DEBUG:boto:Params: {} 2022-09-02T05:47:52.618 DEBUG:boto:Token: None 2022-09-02T05:47:52.619 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UQGpI9k+5N9WzRUQtYE+XKwzpmA= 2022-09-02T05:47:52.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UQGpI9k+5N9WzRUQtYE+XKwzpmA=', 'Content-Length': '0'} 2022-09-02T05:47:52.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005669566361a9427e-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.623 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.624 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.624 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.625 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.625 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.626 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.627 DEBUG:boto:Data: 2022-09-02T05:47:52.627 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.628 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.629 DEBUG:boto:Params: {} 2022-09-02T05:47:52.629 DEBUG:boto:Token: None 2022-09-02T05:47:52.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UQGpI9k+5N9WzRUQtYE+XKwzpmA= 2022-09-02T05:47:52.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UQGpI9k+5N9WzRUQtYE+XKwzpmA=', 'Content-Length': '0'} 2022-09-02T05:47:52.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007986c73b18ebd53-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:52.635 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.636 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.636 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.637 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.637 DEBUG:boto:Method: GET 2022-09-02T05:47:52.638 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.638 DEBUG:boto:Data: 2022-09-02T05:47:52.639 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.640 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.640 DEBUG:boto:Params: {} 2022-09-02T05:47:52.641 DEBUG:boto:Token: None 2022-09-02T05:47:52.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU= 2022-09-02T05:47:52.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU=', 'Content-Length': '0'} 2022-09-02T05:47:52.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e264d9d48ca26f48-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.686 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.687 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.688 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.689 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.690 DEBUG:boto:Method: GET 2022-09-02T05:47:52.691 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.692 DEBUG:boto:Data: 2022-09-02T05:47:52.693 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.695 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.696 DEBUG:boto:Params: {} 2022-09-02T05:47:52.697 DEBUG:boto:Token: None 2022-09-02T05:47:52.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:52.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU= 2022-09-02T05:47:52.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rsg8pqP+YtFKFoI3UZS6K22INXU=', 'Content-Length': '0'} 2022-09-02T05:47:52.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e8875ea6808ee3-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.703 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.704 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.704 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.705 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.705 DEBUG:boto:Method: GET 2022-09-02T05:47:52.706 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.707 DEBUG:boto:Data: 2022-09-02T05:47:52.707 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.709 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.709 DEBUG:boto:Params: {} 2022-09-02T05:47:52.711 DEBUG:boto:Token: None 2022-09-02T05:47:52.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000189205b0531f515f-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.716 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.717 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.718 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.719 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.719 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.720 DEBUG:boto:Method: GET 2022-09-02T05:47:52.720 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.721 DEBUG:boto:Data: 2022-09-02T05:47:52.721 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.722 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.723 DEBUG:boto:Params: {} 2022-09-02T05:47:52.724 DEBUG:boto:Token: None 2022-09-02T05:47:52.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5293ab2d6f705b2-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.730 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.731 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.732 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.732 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.733 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:52.733 DEBUG:boto:Data: 2022-09-02T05:47:52.734 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.735 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.736 DEBUG:boto:Params: {} 2022-09-02T05:47:52.736 DEBUG:boto:Token: None 2022-09-02T05:47:52.737 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33 2022-09-02T05:47:52.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Epc8hc6BxSuXhCAYIq5Ro3keSMo= 2022-09-02T05:47:52.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Epc8hc6BxSuXhCAYIq5Ro3keSMo=', 'Content-Length': '0'} 2022-09-02T05:47:52.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003180f3eb4c18baa6-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.741 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.742 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.743 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.743 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:52.744 DEBUG:boto:Data: 2022-09-02T05:47:52.745 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.746 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.746 DEBUG:boto:Params: {} 2022-09-02T05:47:52.747 DEBUG:boto:Token: None 2022-09-02T05:47:52.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33 2022-09-02T05:47:52.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Epc8hc6BxSuXhCAYIq5Ro3keSMo= 2022-09-02T05:47:52.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Epc8hc6BxSuXhCAYIq5Ro3keSMo=', 'Content-Length': '0'} 2022-09-02T05:47:52.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92a23c95681e3fd-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:52.754 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.755 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.756 DEBUG:boto:Method: GET 2022-09-02T05:47:52.757 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:52.758 DEBUG:boto:Data: 2022-09-02T05:47:52.759 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.761 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.762 DEBUG:boto:Params: {} 2022-09-02T05:47:52.762 DEBUG:boto:Token: None 2022-09-02T05:47:52.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33 2022-09-02T05:47:52.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYdrEbpsm1wRisM4DcGfC0FrXnc= 2022-09-02T05:47:52.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYdrEbpsm1wRisM4DcGfC0FrXnc=', 'Content-Length': '0'} 2022-09-02T05:47:52.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf09841f975f6265-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.767 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:52.768 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:52.768 DEBUG:boto:Method: GET 2022-09-02T05:47:52.769 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:52.770 DEBUG:boto:Data: 2022-09-02T05:47:52.771 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.772 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.772 DEBUG:boto:Params: {} 2022-09-02T05:47:52.773 DEBUG:boto:Token: None 2022-09-02T05:47:52.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj33 2022-09-02T05:47:52.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYdrEbpsm1wRisM4DcGfC0FrXnc= 2022-09-02T05:47:52.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYdrEbpsm1wRisM4DcGfC0FrXnc=', 'Content-Length': '0'} 2022-09-02T05:47:52.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4b2dcdf22cfa59-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.778 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.779 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.780 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.780 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.781 DEBUG:boto:Method: GET 2022-09-02T05:47:52.781 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.782 DEBUG:boto:Data: 2022-09-02T05:47:52.782 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.783 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.784 DEBUG:boto:Params: {} 2022-09-02T05:47:52.785 DEBUG:boto:Token: None 2022-09-02T05:47:52.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc351e53960eaa0-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.830 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.831 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.832 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.833 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.835 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.836 DEBUG:boto:Method: GET 2022-09-02T05:47:52.837 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:52.839 DEBUG:boto:Data: 2022-09-02T05:47:52.840 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.842 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.843 DEBUG:boto:Params: {} 2022-09-02T05:47:52.845 DEBUG:boto:Token: None 2022-09-02T05:47:52.846 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.847 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf83d75fd378139-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.854 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:52.857 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.858 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.859 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.860 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.861 DEBUG:boto:Method: GET 2022-09-02T05:47:52.862 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.863 DEBUG:boto:Data: 2022-09-02T05:47:52.864 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.866 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.867 DEBUG:boto:Params: {} 2022-09-02T05:47:52.868 DEBUG:boto:Token: None 2022-09-02T05:47:52.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k= 2022-09-02T05:47:52.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k=', 'Content-Length': '0'} 2022-09-02T05:47:52.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc831fd6973507b4-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.876 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.877 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.878 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.880 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.881 DEBUG:boto:Method: GET 2022-09-02T05:47:52.882 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.883 DEBUG:boto:Data: 2022-09-02T05:47:52.884 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.886 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.887 DEBUG:boto:Params: {} 2022-09-02T05:47:52.889 DEBUG:boto:Token: None 2022-09-02T05:47:52.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k= 2022-09-02T05:47:52.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k=', 'Content-Length': '0'} 2022-09-02T05:47:52.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568301550b8e40e7-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.897 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.898 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.899 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.900 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.901 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.901 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.902 DEBUG:boto:Data: 2022-09-02T05:47:52.902 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.904 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.904 DEBUG:boto:Params: {} 2022-09-02T05:47:52.905 DEBUG:boto:Token: None 2022-09-02T05:47:52.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aqIxySQO3RAyn5qefn0qdO+Afzc= 2022-09-02T05:47:52.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aqIxySQO3RAyn5qefn0qdO+Afzc=', 'Content-Length': '0'} 2022-09-02T05:47:52.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e31bf8def73d99-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.910 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.911 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.911 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.912 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.912 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.913 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.914 DEBUG:boto:Data: 2022-09-02T05:47:52.914 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.915 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.916 DEBUG:boto:Params: {} 2022-09-02T05:47:52.916 DEBUG:boto:Token: None 2022-09-02T05:47:52.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aqIxySQO3RAyn5qefn0qdO+Afzc= 2022-09-02T05:47:52.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aqIxySQO3RAyn5qefn0qdO+Afzc=', 'Content-Length': '0'} 2022-09-02T05:47:52.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e49e1d0701f63e-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:52.921 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.922 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.923 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.923 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.924 DEBUG:boto:Method: GET 2022-09-02T05:47:52.925 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.925 DEBUG:boto:Data: 2022-09-02T05:47:52.926 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.927 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.928 DEBUG:boto:Params: {} 2022-09-02T05:47:52.928 DEBUG:boto:Token: None 2022-09-02T05:47:52.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k= 2022-09-02T05:47:52.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k=', 'Content-Length': '0'} 2022-09-02T05:47:52.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0630f722d8c8f11-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.933 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.934 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.935 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.935 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.936 DEBUG:boto:Method: GET 2022-09-02T05:47:52.936 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.937 DEBUG:boto:Data: 2022-09-02T05:47:52.937 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.938 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.939 DEBUG:boto:Params: {} 2022-09-02T05:47:52.940 DEBUG:boto:Token: None 2022-09-02T05:47:52.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:52.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k= 2022-09-02T05:47:52.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9LrEJjGwXxHjFLZQf6waRfEfr5k=', 'Content-Length': '0'} 2022-09-02T05:47:52.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000822ed32597978a6a-0063119908-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.945 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.946 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.946 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:52.947 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:52.947 DEBUG:boto:Method: GET 2022-09-02T05:47:52.948 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:52.948 DEBUG:boto:Data: 2022-09-02T05:47:52.949 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.950 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.951 DEBUG:boto:Params: {} 2022-09-02T05:47:52.952 DEBUG:boto:Token: None 2022-09-02T05:47:52.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ef523816422117-0063119908-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.958 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.959 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:52.959 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:52.960 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:52.961 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:52.961 DEBUG:boto:Method: GET 2022-09-02T05:47:52.962 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:52.962 DEBUG:boto:Data: 2022-09-02T05:47:52.963 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.964 DEBUG:boto:Port: 8000 2022-09-02T05:47:52.964 DEBUG:boto:Params: {} 2022-09-02T05:47:52.965 DEBUG:boto:Token: None 2022-09-02T05:47:52.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/ 2022-09-02T05:47:52.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc= 2022-09-02T05:47:52.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bDVi1VSZcLRn7EnfyEhpLXY42Tc=', 'Content-Length': '0'} 2022-09-02T05:47:52.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e2d9aa40f524840-0063119908-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.972 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:52.974 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.975 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.976 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.977 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:52.978 DEBUG:boto:Data: 2022-09-02T05:47:52.979 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:52.981 DEBUG:boto:Port: 8001 2022-09-02T05:47:52.982 DEBUG:boto:Params: {} 2022-09-02T05:47:52.984 DEBUG:boto:Token: None 2022-09-02T05:47:52.985 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:52 GMT /xljbkf-74/obj34 2022-09-02T05:47:52.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:clMQxWZw9wG9osaYJSrAv74kU/M= 2022-09-02T05:47:52.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:clMQxWZw9wG9osaYJSrAv74kU/M=', 'Content-Length': '0'} 2022-09-02T05:47:52.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9555f4be274cef9-0063119908-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:52.991 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:52.992 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:52.993 DEBUG:boto:Method: HEAD 2022-09-02T05:47:52.994 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:52.996 DEBUG:boto:Data: 2022-09-02T05:47:52.997 DEBUG:boto:Headers: {} 2022-09-02T05:47:52.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:52.999 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.000 DEBUG:boto:Params: {} 2022-09-02T05:47:53.001 DEBUG:boto:Token: None 2022-09-02T05:47:53.002 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj34 2022-09-02T05:47:53.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y4ripnTqYwNjC+RDC8gwkEj4s10= 2022-09-02T05:47:53.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y4ripnTqYwNjC+RDC8gwkEj4s10=', 'Content-Length': '0'} 2022-09-02T05:47:53.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b45f34e1db97b588-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:53.010 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:53.011 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:53.012 DEBUG:boto:Method: GET 2022-09-02T05:47:53.013 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:53.014 DEBUG:boto:Data: 2022-09-02T05:47:53.015 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.017 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.018 DEBUG:boto:Params: {} 2022-09-02T05:47:53.020 DEBUG:boto:Token: None 2022-09-02T05:47:53.021 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj34 2022-09-02T05:47:53.022 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gVNgy3rDG97Xa1ORPQYiszeIDQk= 2022-09-02T05:47:53.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gVNgy3rDG97Xa1ORPQYiszeIDQk=', 'Content-Length': '0'} 2022-09-02T05:47:53.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41cbb52442d29e1-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.028 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:53.029 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:53.030 DEBUG:boto:Method: GET 2022-09-02T05:47:53.031 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:53.032 DEBUG:boto:Data: 2022-09-02T05:47:53.033 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.035 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.035 DEBUG:boto:Params: {} 2022-09-02T05:47:53.036 DEBUG:boto:Token: None 2022-09-02T05:47:53.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj34 2022-09-02T05:47:53.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gVNgy3rDG97Xa1ORPQYiszeIDQk= 2022-09-02T05:47:53.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gVNgy3rDG97Xa1ORPQYiszeIDQk=', 'Content-Length': '0'} 2022-09-02T05:47:53.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4dfd0fb50d280b-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.082 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.083 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.084 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:53.084 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:53.085 DEBUG:boto:Method: GET 2022-09-02T05:47:53.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:53.086 DEBUG:boto:Data: 2022-09-02T05:47:53.087 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.088 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.088 DEBUG:boto:Params: {} 2022-09-02T05:47:53.089 DEBUG:boto:Token: None 2022-09-02T05:47:53.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a08bbf27eb4ce8b-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.095 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.096 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.097 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.098 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:53.099 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:53.100 DEBUG:boto:Method: GET 2022-09-02T05:47:53.100 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:53.101 DEBUG:boto:Data: 2022-09-02T05:47:53.101 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.103 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.103 DEBUG:boto:Params: {} 2022-09-02T05:47:53.104 DEBUG:boto:Token: None 2022-09-02T05:47:53.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003604d5c296c90c6d-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.110 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:53.112 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.112 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.113 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.113 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.114 DEBUG:boto:Method: GET 2022-09-02T05:47:53.114 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.115 DEBUG:boto:Data: 2022-09-02T05:47:53.115 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.116 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.117 DEBUG:boto:Params: {} 2022-09-02T05:47:53.117 DEBUG:boto:Token: None 2022-09-02T05:47:53.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk= 2022-09-02T05:47:53.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk=', 'Content-Length': '0'} 2022-09-02T05:47:53.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088ba7f62751a91c3-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.123 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.124 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.125 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.126 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.127 DEBUG:boto:Method: GET 2022-09-02T05:47:53.128 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.129 DEBUG:boto:Data: 2022-09-02T05:47:53.130 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.132 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.133 DEBUG:boto:Params: {} 2022-09-02T05:47:53.134 DEBUG:boto:Token: None 2022-09-02T05:47:53.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk= 2022-09-02T05:47:53.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk=', 'Content-Length': '0'} 2022-09-02T05:47:53.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286255ef7e671273-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.141 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.141 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.142 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.142 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.143 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.143 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.144 DEBUG:boto:Data: 2022-09-02T05:47:53.144 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.145 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.146 DEBUG:boto:Params: {} 2022-09-02T05:47:53.146 DEBUG:boto:Token: None 2022-09-02T05:47:53.147 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3vmWM7u6uB3YdcCmd/6uZXtPzC8= 2022-09-02T05:47:53.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3vmWM7u6uB3YdcCmd/6uZXtPzC8=', 'Content-Length': '0'} 2022-09-02T05:47:53.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea5eed4852f9113-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.151 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.152 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.152 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.153 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.153 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.154 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.154 DEBUG:boto:Data: 2022-09-02T05:47:53.155 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.156 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.156 DEBUG:boto:Params: {} 2022-09-02T05:47:53.157 DEBUG:boto:Token: None 2022-09-02T05:47:53.158 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3vmWM7u6uB3YdcCmd/6uZXtPzC8= 2022-09-02T05:47:53.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3vmWM7u6uB3YdcCmd/6uZXtPzC8=', 'Content-Length': '0'} 2022-09-02T05:47:53.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c264d671cd59f1-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:53.163 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.163 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.164 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.164 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.165 DEBUG:boto:Method: GET 2022-09-02T05:47:53.165 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.166 DEBUG:boto:Data: 2022-09-02T05:47:53.167 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.168 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.168 DEBUG:boto:Params: {} 2022-09-02T05:47:53.169 DEBUG:boto:Token: None 2022-09-02T05:47:53.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk= 2022-09-02T05:47:53.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk=', 'Content-Length': '0'} 2022-09-02T05:47:53.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6efa68ee3a874d-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.214 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.215 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.215 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.216 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.217 DEBUG:boto:Method: GET 2022-09-02T05:47:53.218 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.220 DEBUG:boto:Data: 2022-09-02T05:47:53.221 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.223 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.224 DEBUG:boto:Params: {} 2022-09-02T05:47:53.226 DEBUG:boto:Token: None 2022-09-02T05:47:53.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:53.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk= 2022-09-02T05:47:53.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HyvLCttXdftumZulUgWWIA7TsGk=', 'Content-Length': '0'} 2022-09-02T05:47:53.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e814159adce00e58-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.235 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.236 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.237 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.238 DEBUG:boto:Method: GET 2022-09-02T05:47:53.239 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.240 DEBUG:boto:Data: 2022-09-02T05:47:53.241 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.243 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.244 DEBUG:boto:Params: {} 2022-09-02T05:47:53.245 DEBUG:boto:Token: None 2022-09-02T05:47:53.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5ad9970d9b6fbcd-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.254 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.255 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.256 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.257 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.258 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.259 DEBUG:boto:Method: GET 2022-09-02T05:47:53.260 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.262 DEBUG:boto:Data: 2022-09-02T05:47:53.263 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.265 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.266 DEBUG:boto:Params: {} 2022-09-02T05:47:53.267 DEBUG:boto:Token: None 2022-09-02T05:47:53.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000932ceb0387411c04-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.276 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.277 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.278 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.280 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.281 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:53.281 DEBUG:boto:Data: 2022-09-02T05:47:53.282 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.284 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.285 DEBUG:boto:Params: {} 2022-09-02T05:47:53.286 DEBUG:boto:Token: None 2022-09-02T05:47:53.287 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44 2022-09-02T05:47:53.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B0sTH72Mt5KKfQDLmqz8wenTt0Q= 2022-09-02T05:47:53.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B0sTH72Mt5KKfQDLmqz8wenTt0Q=', 'Content-Length': '0'} 2022-09-02T05:47:53.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf68e4a72156f282-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.294 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.295 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.296 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.297 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:53.298 DEBUG:boto:Data: 2022-09-02T05:47:53.299 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.301 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.302 DEBUG:boto:Params: {} 2022-09-02T05:47:53.303 DEBUG:boto:Token: None 2022-09-02T05:47:53.304 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44 2022-09-02T05:47:53.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B0sTH72Mt5KKfQDLmqz8wenTt0Q= 2022-09-02T05:47:53.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B0sTH72Mt5KKfQDLmqz8wenTt0Q=', 'Content-Length': '0'} 2022-09-02T05:47:53.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84f998a4297afe7-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:53.311 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.313 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.314 DEBUG:boto:Method: GET 2022-09-02T05:47:53.315 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:53.316 DEBUG:boto:Data: 2022-09-02T05:47:53.317 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.319 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.320 DEBUG:boto:Params: {} 2022-09-02T05:47:53.321 DEBUG:boto:Token: None 2022-09-02T05:47:53.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44 2022-09-02T05:47:53.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E8Op6XOGg/THyyfWwqH7MYGAs5A= 2022-09-02T05:47:53.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E8Op6XOGg/THyyfWwqH7MYGAs5A=', 'Content-Length': '0'} 2022-09-02T05:47:53.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470896b06d992ffa-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.329 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:53.330 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:53.331 DEBUG:boto:Method: GET 2022-09-02T05:47:53.332 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:53.333 DEBUG:boto:Data: 2022-09-02T05:47:53.334 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.336 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.337 DEBUG:boto:Params: {} 2022-09-02T05:47:53.338 DEBUG:boto:Token: None 2022-09-02T05:47:53.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj44 2022-09-02T05:47:53.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E8Op6XOGg/THyyfWwqH7MYGAs5A= 2022-09-02T05:47:53.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E8Op6XOGg/THyyfWwqH7MYGAs5A=', 'Content-Length': '0'} 2022-09-02T05:47:53.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098488cc725052fea-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.347 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.347 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.348 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.349 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.349 DEBUG:boto:Method: GET 2022-09-02T05:47:53.350 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.350 DEBUG:boto:Data: 2022-09-02T05:47:53.351 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.352 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.353 DEBUG:boto:Params: {} 2022-09-02T05:47:53.353 DEBUG:boto:Token: None 2022-09-02T05:47:53.354 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012192fec1ecdf26e-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.359 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.360 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.360 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.361 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.361 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.362 DEBUG:boto:Method: GET 2022-09-02T05:47:53.363 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:53.363 DEBUG:boto:Data: 2022-09-02T05:47:53.364 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.365 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.365 DEBUG:boto:Params: {} 2022-09-02T05:47:53.366 DEBUG:boto:Token: None 2022-09-02T05:47:53.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c0463cf1859cfef-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.372 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:53.373 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.374 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.374 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.375 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.375 DEBUG:boto:Method: GET 2022-09-02T05:47:53.376 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.376 DEBUG:boto:Data: 2022-09-02T05:47:53.377 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.378 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.379 DEBUG:boto:Params: {} 2022-09-02T05:47:53.379 DEBUG:boto:Token: None 2022-09-02T05:47:53.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4= 2022-09-02T05:47:53.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4=', 'Content-Length': '0'} 2022-09-02T05:47:53.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001075a344748d91d5-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.384 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.385 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.385 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.386 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.386 DEBUG:boto:Method: GET 2022-09-02T05:47:53.387 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.388 DEBUG:boto:Data: 2022-09-02T05:47:53.388 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.389 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.390 DEBUG:boto:Params: {} 2022-09-02T05:47:53.391 DEBUG:boto:Token: None 2022-09-02T05:47:53.391 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4= 2022-09-02T05:47:53.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4=', 'Content-Length': '0'} 2022-09-02T05:47:53.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d367639603dbf4b1-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.438 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.439 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.439 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.440 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.440 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.441 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.441 DEBUG:boto:Data: 2022-09-02T05:47:53.442 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.443 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.444 DEBUG:boto:Params: {} 2022-09-02T05:47:53.444 DEBUG:boto:Token: None 2022-09-02T05:47:53.445 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xdQysH1J94pchLlq7PclV3Gweg8= 2022-09-02T05:47:53.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xdQysH1J94pchLlq7PclV3Gweg8=', 'Content-Length': '0'} 2022-09-02T05:47:53.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de2e984c116efc0-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.449 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.450 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.450 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.451 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.451 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.452 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.452 DEBUG:boto:Data: 2022-09-02T05:47:53.453 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.454 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.455 DEBUG:boto:Params: {} 2022-09-02T05:47:53.455 DEBUG:boto:Token: None 2022-09-02T05:47:53.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xdQysH1J94pchLlq7PclV3Gweg8= 2022-09-02T05:47:53.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xdQysH1J94pchLlq7PclV3Gweg8=', 'Content-Length': '0'} 2022-09-02T05:47:53.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d762ef65e330cce-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:53.460 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.461 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.462 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.462 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.463 DEBUG:boto:Method: GET 2022-09-02T05:47:53.463 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.464 DEBUG:boto:Data: 2022-09-02T05:47:53.464 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.465 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.466 DEBUG:boto:Params: {} 2022-09-02T05:47:53.467 DEBUG:boto:Token: None 2022-09-02T05:47:53.467 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.468 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4= 2022-09-02T05:47:53.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4=', 'Content-Length': '0'} 2022-09-02T05:47:53.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005145e0e8c04b9850-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.472 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.472 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.473 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.473 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.474 DEBUG:boto:Method: GET 2022-09-02T05:47:53.474 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.475 DEBUG:boto:Data: 2022-09-02T05:47:53.476 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.477 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.477 DEBUG:boto:Params: {} 2022-09-02T05:47:53.478 DEBUG:boto:Token: None 2022-09-02T05:47:53.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:53.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4= 2022-09-02T05:47:53.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5NVS7LrvwWlecOw4pxrsvQQ5i+4=', 'Content-Length': '0'} 2022-09-02T05:47:53.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f2ef7f90c93eaf-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.484 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.485 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.485 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.486 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.487 DEBUG:boto:Method: GET 2022-09-02T05:47:53.488 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.488 DEBUG:boto:Data: 2022-09-02T05:47:53.489 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.490 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.491 DEBUG:boto:Params: {} 2022-09-02T05:47:53.491 DEBUG:boto:Token: None 2022-09-02T05:47:53.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c70ca91d0459fcd-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.538 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.539 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.540 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.540 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.541 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.542 DEBUG:boto:Method: GET 2022-09-02T05:47:53.542 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.543 DEBUG:boto:Data: 2022-09-02T05:47:53.543 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.544 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.545 DEBUG:boto:Params: {} 2022-09-02T05:47:53.545 DEBUG:boto:Token: None 2022-09-02T05:47:53.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc62ac4172ae1ddb-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.551 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.552 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.552 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.553 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.553 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:53.554 DEBUG:boto:Data: 2022-09-02T05:47:53.554 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.555 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.556 DEBUG:boto:Params: {} 2022-09-02T05:47:53.557 DEBUG:boto:Token: None 2022-09-02T05:47:53.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51 2022-09-02T05:47:53.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RfzJGgKgNQTlktm+zaN85Qeg9EA= 2022-09-02T05:47:53.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RfzJGgKgNQTlktm+zaN85Qeg9EA=', 'Content-Length': '0'} 2022-09-02T05:47:53.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030423fffd9e734cc-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.562 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.563 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.563 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.564 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:53.564 DEBUG:boto:Data: 2022-09-02T05:47:53.565 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.566 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.567 DEBUG:boto:Params: {} 2022-09-02T05:47:53.567 DEBUG:boto:Token: None 2022-09-02T05:47:53.568 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51 2022-09-02T05:47:53.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RfzJGgKgNQTlktm+zaN85Qeg9EA= 2022-09-02T05:47:53.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RfzJGgKgNQTlktm+zaN85Qeg9EA=', 'Content-Length': '0'} 2022-09-02T05:47:53.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7abb96cd788d809-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:53.574 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.574 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.575 DEBUG:boto:Method: GET 2022-09-02T05:47:53.576 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:53.577 DEBUG:boto:Data: 2022-09-02T05:47:53.577 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.579 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.580 DEBUG:boto:Params: {} 2022-09-02T05:47:53.581 DEBUG:boto:Token: None 2022-09-02T05:47:53.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51 2022-09-02T05:47:53.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SN0F+7v7xcNpR2QN4BADNSI/tYk= 2022-09-02T05:47:53.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SN0F+7v7xcNpR2QN4BADNSI/tYk=', 'Content-Length': '0'} 2022-09-02T05:47:53.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fec0e60c5988fe7-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.588 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:53.589 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:53.589 DEBUG:boto:Method: GET 2022-09-02T05:47:53.590 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:53.591 DEBUG:boto:Data: 2022-09-02T05:47:53.591 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.592 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.593 DEBUG:boto:Params: {} 2022-09-02T05:47:53.593 DEBUG:boto:Token: None 2022-09-02T05:47:53.594 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj51 2022-09-02T05:47:53.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SN0F+7v7xcNpR2QN4BADNSI/tYk= 2022-09-02T05:47:53.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SN0F+7v7xcNpR2QN4BADNSI/tYk=', 'Content-Length': '0'} 2022-09-02T05:47:53.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f3b7eb9790befce-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.599 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.599 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.600 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.600 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.601 DEBUG:boto:Method: GET 2022-09-02T05:47:53.601 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.602 DEBUG:boto:Data: 2022-09-02T05:47:53.602 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.603 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.604 DEBUG:boto:Params: {} 2022-09-02T05:47:53.604 DEBUG:boto:Token: None 2022-09-02T05:47:53.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d037dcf7c63c2dc1-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.610 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.611 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.612 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.612 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.613 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.613 DEBUG:boto:Method: GET 2022-09-02T05:47:53.614 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:53.614 DEBUG:boto:Data: 2022-09-02T05:47:53.615 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.616 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.616 DEBUG:boto:Params: {} 2022-09-02T05:47:53.617 DEBUG:boto:Token: None 2022-09-02T05:47:53.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7a314c008c60fa5-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.623 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:53.625 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.625 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.626 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.626 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.627 DEBUG:boto:Method: GET 2022-09-02T05:47:53.627 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.628 DEBUG:boto:Data: 2022-09-02T05:47:53.628 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.630 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.630 DEBUG:boto:Params: {} 2022-09-02T05:47:53.631 DEBUG:boto:Token: None 2022-09-02T05:47:53.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ= 2022-09-02T05:47:53.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ=', 'Content-Length': '0'} 2022-09-02T05:47:53.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b218b590216790-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.636 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.636 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.637 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.637 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.638 DEBUG:boto:Method: GET 2022-09-02T05:47:53.639 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.639 DEBUG:boto:Data: 2022-09-02T05:47:53.640 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.641 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.641 DEBUG:boto:Params: {} 2022-09-02T05:47:53.642 DEBUG:boto:Token: None 2022-09-02T05:47:53.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ= 2022-09-02T05:47:53.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ=', 'Content-Length': '0'} 2022-09-02T05:47:53.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca0ab22de06bce8-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.690 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.691 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.691 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.692 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.692 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.693 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.693 DEBUG:boto:Data: 2022-09-02T05:47:53.694 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.695 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.695 DEBUG:boto:Params: {} 2022-09-02T05:47:53.696 DEBUG:boto:Token: None 2022-09-02T05:47:53.696 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7QW4B9TSgjy1mtDe82gPraGTXM= 2022-09-02T05:47:53.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7QW4B9TSgjy1mtDe82gPraGTXM=', 'Content-Length': '0'} 2022-09-02T05:47:53.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4c0082e83ce52b-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.701 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.701 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.702 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.702 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.703 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.703 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.704 DEBUG:boto:Data: 2022-09-02T05:47:53.704 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.705 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.706 DEBUG:boto:Params: {} 2022-09-02T05:47:53.706 DEBUG:boto:Token: None 2022-09-02T05:47:53.707 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7QW4B9TSgjy1mtDe82gPraGTXM= 2022-09-02T05:47:53.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7QW4B9TSgjy1mtDe82gPraGTXM=', 'Content-Length': '0'} 2022-09-02T05:47:53.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d2b7f9c30e84b6-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:53.712 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.713 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.713 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.714 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.714 DEBUG:boto:Method: GET 2022-09-02T05:47:53.715 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.715 DEBUG:boto:Data: 2022-09-02T05:47:53.716 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.717 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.718 DEBUG:boto:Params: {} 2022-09-02T05:47:53.719 DEBUG:boto:Token: None 2022-09-02T05:47:53.719 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ= 2022-09-02T05:47:53.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ=', 'Content-Length': '0'} 2022-09-02T05:47:53.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a89b1c101c477d7-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.724 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.725 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.726 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.726 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.727 DEBUG:boto:Method: GET 2022-09-02T05:47:53.727 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.728 DEBUG:boto:Data: 2022-09-02T05:47:53.728 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.729 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.730 DEBUG:boto:Params: {} 2022-09-02T05:47:53.730 DEBUG:boto:Token: None 2022-09-02T05:47:53.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:53.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ= 2022-09-02T05:47:53.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C0u5nSKnnaQ0Q8BQGh7CRd4ucpQ=', 'Content-Length': '0'} 2022-09-02T05:47:53.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c54cccb5572f6b-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.735 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.736 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.736 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.737 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.737 DEBUG:boto:Method: GET 2022-09-02T05:47:53.738 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.739 DEBUG:boto:Data: 2022-09-02T05:47:53.739 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.740 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.741 DEBUG:boto:Params: {} 2022-09-02T05:47:53.742 DEBUG:boto:Token: None 2022-09-02T05:47:53.742 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d02fcd5ea54c36-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.747 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.748 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.748 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.749 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.749 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.750 DEBUG:boto:Method: GET 2022-09-02T05:47:53.750 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.751 DEBUG:boto:Data: 2022-09-02T05:47:53.751 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.752 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.753 DEBUG:boto:Params: {} 2022-09-02T05:47:53.753 DEBUG:boto:Token: None 2022-09-02T05:47:53.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c13828ff973bba9-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.802 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.804 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.805 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.806 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.807 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:53.808 DEBUG:boto:Data: 2022-09-02T05:47:53.809 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.810 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.810 DEBUG:boto:Params: {} 2022-09-02T05:47:53.811 DEBUG:boto:Token: None 2022-09-02T05:47:53.812 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55 2022-09-02T05:47:53.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vVAFF6R5BhC4augS+GlEw9UG+hQ= 2022-09-02T05:47:53.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vVAFF6R5BhC4augS+GlEw9UG+hQ=', 'Content-Length': '0'} 2022-09-02T05:47:53.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b81b57da35defeb2-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.816 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.817 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.817 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.818 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:53.818 DEBUG:boto:Data: 2022-09-02T05:47:53.819 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.820 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.820 DEBUG:boto:Params: {} 2022-09-02T05:47:53.821 DEBUG:boto:Token: None 2022-09-02T05:47:53.821 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55 2022-09-02T05:47:53.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vVAFF6R5BhC4augS+GlEw9UG+hQ= 2022-09-02T05:47:53.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vVAFF6R5BhC4augS+GlEw9UG+hQ=', 'Content-Length': '0'} 2022-09-02T05:47:53.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e420c062b404512-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:53.827 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.827 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.828 DEBUG:boto:Method: GET 2022-09-02T05:47:53.829 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:53.829 DEBUG:boto:Data: 2022-09-02T05:47:53.830 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.831 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.832 DEBUG:boto:Params: {} 2022-09-02T05:47:53.833 DEBUG:boto:Token: None 2022-09-02T05:47:53.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55 2022-09-02T05:47:53.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ahJ4oBIX/+w0PgV2dtztjz8r3A4= 2022-09-02T05:47:53.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ahJ4oBIX/+w0PgV2dtztjz8r3A4=', 'Content-Length': '0'} 2022-09-02T05:47:53.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f866807cc164ec10-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.841 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:53.842 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:53.843 DEBUG:boto:Method: GET 2022-09-02T05:47:53.845 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:53.846 DEBUG:boto:Data: 2022-09-02T05:47:53.847 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.849 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.850 DEBUG:boto:Params: {} 2022-09-02T05:47:53.851 DEBUG:boto:Token: None 2022-09-02T05:47:53.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj55 2022-09-02T05:47:53.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ahJ4oBIX/+w0PgV2dtztjz8r3A4= 2022-09-02T05:47:53.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ahJ4oBIX/+w0PgV2dtztjz8r3A4=', 'Content-Length': '0'} 2022-09-02T05:47:53.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5779afc08e477c-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.865 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.866 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.867 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.868 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.869 DEBUG:boto:Method: GET 2022-09-02T05:47:53.870 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.870 DEBUG:boto:Data: 2022-09-02T05:47:53.871 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.872 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.872 DEBUG:boto:Params: {} 2022-09-02T05:47:53.873 DEBUG:boto:Token: None 2022-09-02T05:47:53.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b20144569a58c847-0063119909-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.879 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.880 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:53.880 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:53.881 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.882 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.882 DEBUG:boto:Method: GET 2022-09-02T05:47:53.883 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:53.884 DEBUG:boto:Data: 2022-09-02T05:47:53.884 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.885 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.886 DEBUG:boto:Params: {} 2022-09-02T05:47:53.886 DEBUG:boto:Token: None 2022-09-02T05:47:53.887 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/ 2022-09-02T05:47:53.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE= 2022-09-02T05:47:53.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MLZiMwAOE1zcg3PTMIju4W6xbiE=', 'Content-Length': '0'} 2022-09-02T05:47:53.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078eff2f6beb264b1-0063119909-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.892 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:53.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:53.893 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:53.894 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:53.895 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.895 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.896 DEBUG:boto:Method: GET 2022-09-02T05:47:53.896 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.897 DEBUG:boto:Data: 2022-09-02T05:47:53.897 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.898 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.899 DEBUG:boto:Params: {} 2022-09-02T05:47:53.900 DEBUG:boto:Token: None 2022-09-02T05:47:53.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U+HcaoT+Ncqt3PD5iv+quNpaYt8= 2022-09-02T05:47:53.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U+HcaoT+Ncqt3PD5iv+quNpaYt8=', 'Content-Length': '0'} 2022-09-02T05:47:53.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bf88b1f651e89e-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.905 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:53.905 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:53.906 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.906 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.907 DEBUG:boto:Method: GET 2022-09-02T05:47:53.907 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.908 DEBUG:boto:Data: 2022-09-02T05:47:53.909 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.910 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.911 DEBUG:boto:Params: {} 2022-09-02T05:47:53.911 DEBUG:boto:Token: None 2022-09-02T05:47:53.912 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U+HcaoT+Ncqt3PD5iv+quNpaYt8= 2022-09-02T05:47:53.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U+HcaoT+Ncqt3PD5iv+quNpaYt8=', 'Content-Length': '0'} 2022-09-02T05:47:53.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c854b4c5412454b-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.958 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:53.959 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:53.960 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.961 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.961 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.962 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.963 DEBUG:boto:Data: 2022-09-02T05:47:53.963 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.964 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.965 DEBUG:boto:Params: {} 2022-09-02T05:47:53.966 DEBUG:boto:Token: None 2022-09-02T05:47:53.966 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n6pxu/q/BOsGrdg2P2HpGvR/lJE= 2022-09-02T05:47:53.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n6pxu/q/BOsGrdg2P2HpGvR/lJE=', 'Content-Length': '0'} 2022-09-02T05:47:53.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed69049a5dba29d1-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.971 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:53.971 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:53.972 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.972 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.973 DEBUG:boto:Method: HEAD 2022-09-02T05:47:53.973 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.974 DEBUG:boto:Data: 2022-09-02T05:47:53.974 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.975 DEBUG:boto:Port: 8000 2022-09-02T05:47:53.976 DEBUG:boto:Params: {} 2022-09-02T05:47:53.976 DEBUG:boto:Token: None 2022-09-02T05:47:53.977 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n6pxu/q/BOsGrdg2P2HpGvR/lJE= 2022-09-02T05:47:53.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n6pxu/q/BOsGrdg2P2HpGvR/lJE=', 'Content-Length': '0'} 2022-09-02T05:47:53.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454bc6f7ceb977b7-0063119909-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:53.982 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:53.982 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:53.983 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.984 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.984 DEBUG:boto:Method: GET 2022-09-02T05:47:53.985 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.985 DEBUG:boto:Data: 2022-09-02T05:47:53.986 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:53.987 DEBUG:boto:Port: 8001 2022-09-02T05:47:53.988 DEBUG:boto:Params: {} 2022-09-02T05:47:53.988 DEBUG:boto:Token: None 2022-09-02T05:47:53.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:53 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U+HcaoT+Ncqt3PD5iv+quNpaYt8= 2022-09-02T05:47:53.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U+HcaoT+Ncqt3PD5iv+quNpaYt8=', 'Content-Length': '0'} 2022-09-02T05:47:53.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076ffc83ac967d45c-0063119909-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:53.994 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:53.995 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:53.995 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.996 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.996 DEBUG:boto:Method: GET 2022-09-02T05:47:53.997 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:53.998 DEBUG:boto:Data: 2022-09-02T05:47:53.998 DEBUG:boto:Headers: {} 2022-09-02T05:47:53.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:53.999 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.000 DEBUG:boto:Params: {} 2022-09-02T05:47:54.000 DEBUG:boto:Token: None 2022-09-02T05:47:54.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:54.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaopnloH5fHfSgNzkhSHVhti2dQ= 2022-09-02T05:47:54.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaopnloH5fHfSgNzkhSHVhti2dQ=', 'Content-Length': '0'} 2022-09-02T05:47:54.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ed5483eada4f38-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.006 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.007 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.007 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.008 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.009 DEBUG:boto:Method: GET 2022-09-02T05:47:54.009 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.010 DEBUG:boto:Data: 2022-09-02T05:47:54.010 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.012 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.012 DEBUG:boto:Params: {} 2022-09-02T05:47:54.013 DEBUG:boto:Token: None 2022-09-02T05:47:54.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.014 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e97571d895af70-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.019 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.020 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.020 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.021 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.022 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.022 DEBUG:boto:Method: GET 2022-09-02T05:47:54.023 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.023 DEBUG:boto:Data: 2022-09-02T05:47:54.024 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.025 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.026 DEBUG:boto:Params: {} 2022-09-02T05:47:54.026 DEBUG:boto:Token: None 2022-09-02T05:47:54.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7fcf4cbc06642f2-006311990a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.032 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.033 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:54.034 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:54.034 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.035 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:54.036 DEBUG:boto:Data: 2022-09-02T05:47:54.036 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.037 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.038 DEBUG:boto:Params: {} 2022-09-02T05:47:54.038 DEBUG:boto:Token: None 2022-09-02T05:47:54.039 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj68 2022-09-02T05:47:54.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AfJb9EscCjldPx6D3Cnzdqq7LaA= 2022-09-02T05:47:54.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AfJb9EscCjldPx6D3Cnzdqq7LaA=', 'Content-Length': '0'} 2022-09-02T05:47:54.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee99d8b4326086e8-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.043 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:54.044 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:54.044 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.045 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:54.045 DEBUG:boto:Data: 2022-09-02T05:47:54.046 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.047 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.047 DEBUG:boto:Params: {} 2022-09-02T05:47:54.048 DEBUG:boto:Token: None 2022-09-02T05:47:54.049 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj68 2022-09-02T05:47:54.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AfJb9EscCjldPx6D3Cnzdqq7LaA= 2022-09-02T05:47:54.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AfJb9EscCjldPx6D3Cnzdqq7LaA=', 'Content-Length': '0'} 2022-09-02T05:47:54.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3b6e9788c5bf274-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:54.054 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:54.055 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:54.055 DEBUG:boto:Method: GET 2022-09-02T05:47:54.056 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:54.056 DEBUG:boto:Data: 2022-09-02T05:47:54.057 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.058 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.059 DEBUG:boto:Params: {} 2022-09-02T05:47:54.059 DEBUG:boto:Token: None 2022-09-02T05:47:54.060 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj68 2022-09-02T05:47:54.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Atj0WHZvx3AlCajirn2ADdH0mZk= 2022-09-02T05:47:54.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Atj0WHZvx3AlCajirn2ADdH0mZk=', 'Content-Length': '0'} 2022-09-02T05:47:54.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c3d49d4c6b25ac-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.106 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:54.107 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:54.107 DEBUG:boto:Method: GET 2022-09-02T05:47:54.108 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:54.108 DEBUG:boto:Data: 2022-09-02T05:47:54.109 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.110 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.111 DEBUG:boto:Params: {} 2022-09-02T05:47:54.112 DEBUG:boto:Token: None 2022-09-02T05:47:54.112 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj68 2022-09-02T05:47:54.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Atj0WHZvx3AlCajirn2ADdH0mZk= 2022-09-02T05:47:54.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Atj0WHZvx3AlCajirn2ADdH0mZk=', 'Content-Length': '0'} 2022-09-02T05:47:54.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882629a4e259cfe4-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.117 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.117 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.118 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.118 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.119 DEBUG:boto:Method: GET 2022-09-02T05:47:54.119 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.120 DEBUG:boto:Data: 2022-09-02T05:47:54.121 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.122 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.122 DEBUG:boto:Params: {} 2022-09-02T05:47:54.123 DEBUG:boto:Token: None 2022-09-02T05:47:54.124 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c1ab58b2ad9a2b-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.129 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.129 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.130 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.131 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.131 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.132 DEBUG:boto:Method: GET 2022-09-02T05:47:54.132 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:54.133 DEBUG:boto:Data: 2022-09-02T05:47:54.133 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.134 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.135 DEBUG:boto:Params: {} 2022-09-02T05:47:54.136 DEBUG:boto:Token: None 2022-09-02T05:47:54.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb3cdcd8c49f7a8-006311990a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.142 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:54.143 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.144 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.145 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.145 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.146 DEBUG:boto:Method: GET 2022-09-02T05:47:54.146 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.147 DEBUG:boto:Data: 2022-09-02T05:47:54.147 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.149 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.149 DEBUG:boto:Params: {} 2022-09-02T05:47:54.150 DEBUG:boto:Token: None 2022-09-02T05:47:54.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w= 2022-09-02T05:47:54.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w=', 'Content-Length': '0'} 2022-09-02T05:47:54.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014665d77c76bc343-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.157 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.158 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.159 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.159 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.160 DEBUG:boto:Method: GET 2022-09-02T05:47:54.160 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.161 DEBUG:boto:Data: 2022-09-02T05:47:54.162 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.163 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.163 DEBUG:boto:Params: {} 2022-09-02T05:47:54.164 DEBUG:boto:Token: None 2022-09-02T05:47:54.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w= 2022-09-02T05:47:54.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w=', 'Content-Length': '0'} 2022-09-02T05:47:54.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a28355a82ee0e52-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.169 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.170 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.171 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.171 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.172 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.172 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.173 DEBUG:boto:Data: 2022-09-02T05:47:54.173 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.174 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.175 DEBUG:boto:Params: {} 2022-09-02T05:47:54.175 DEBUG:boto:Token: None 2022-09-02T05:47:54.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X1UcCY1aNpgzW34U5zSSbVYrZew= 2022-09-02T05:47:54.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X1UcCY1aNpgzW34U5zSSbVYrZew=', 'Content-Length': '0'} 2022-09-02T05:47:54.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fb3770bdfe47d7-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.181 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.182 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.183 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.183 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.184 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.185 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.186 DEBUG:boto:Data: 2022-09-02T05:47:54.187 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.188 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.189 DEBUG:boto:Params: {} 2022-09-02T05:47:54.190 DEBUG:boto:Token: None 2022-09-02T05:47:54.190 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X1UcCY1aNpgzW34U5zSSbVYrZew= 2022-09-02T05:47:54.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X1UcCY1aNpgzW34U5zSSbVYrZew=', 'Content-Length': '0'} 2022-09-02T05:47:54.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a86a9de93f1670-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:54.195 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.196 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.196 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.198 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.199 DEBUG:boto:Method: GET 2022-09-02T05:47:54.200 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.200 DEBUG:boto:Data: 2022-09-02T05:47:54.201 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.203 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.204 DEBUG:boto:Params: {} 2022-09-02T05:47:54.205 DEBUG:boto:Token: None 2022-09-02T05:47:54.206 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w= 2022-09-02T05:47:54.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w=', 'Content-Length': '0'} 2022-09-02T05:47:54.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da4762e1a061020-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.212 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.213 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.213 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.214 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.214 DEBUG:boto:Method: GET 2022-09-02T05:47:54.215 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.216 DEBUG:boto:Data: 2022-09-02T05:47:54.216 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.217 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.218 DEBUG:boto:Params: {} 2022-09-02T05:47:54.219 DEBUG:boto:Token: None 2022-09-02T05:47:54.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:54.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w= 2022-09-02T05:47:54.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2fazQHWpd8u8/z6tPs6Gt3J4c8w=', 'Content-Length': '0'} 2022-09-02T05:47:54.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9e78c4899f4981-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.224 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.224 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.225 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.225 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.226 DEBUG:boto:Method: GET 2022-09-02T05:47:54.226 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.227 DEBUG:boto:Data: 2022-09-02T05:47:54.227 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.228 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.229 DEBUG:boto:Params: {} 2022-09-02T05:47:54.230 DEBUG:boto:Token: None 2022-09-02T05:47:54.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2f6b29f78ba055d-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.278 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.279 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.280 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.281 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.281 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.282 DEBUG:boto:Method: GET 2022-09-02T05:47:54.282 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.283 DEBUG:boto:Data: 2022-09-02T05:47:54.284 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.285 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.286 DEBUG:boto:Params: {} 2022-09-02T05:47:54.286 DEBUG:boto:Token: None 2022-09-02T05:47:54.287 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec3a9fe148c61e7b-006311990a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.293 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.293 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.294 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.295 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.295 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:54.296 DEBUG:boto:Data: 2022-09-02T05:47:54.296 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.297 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.298 DEBUG:boto:Params: {} 2022-09-02T05:47:54.299 DEBUG:boto:Token: None 2022-09-02T05:47:54.299 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85 2022-09-02T05:47:54.300 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2HC5xhojoRaC9CjYx6L+E9KAX6A= 2022-09-02T05:47:54.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2HC5xhojoRaC9CjYx6L+E9KAX6A=', 'Content-Length': '0'} 2022-09-02T05:47:54.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000506b756abc26c95e-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.303 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.304 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.304 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.305 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:54.305 DEBUG:boto:Data: 2022-09-02T05:47:54.306 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.308 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.309 DEBUG:boto:Params: {} 2022-09-02T05:47:54.310 DEBUG:boto:Token: None 2022-09-02T05:47:54.311 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85 2022-09-02T05:47:54.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2HC5xhojoRaC9CjYx6L+E9KAX6A= 2022-09-02T05:47:54.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2HC5xhojoRaC9CjYx6L+E9KAX6A=', 'Content-Length': '0'} 2022-09-02T05:47:54.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e224bdec2bd6c05-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:54.317 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.317 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.318 DEBUG:boto:Method: GET 2022-09-02T05:47:54.319 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:54.320 DEBUG:boto:Data: 2022-09-02T05:47:54.320 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.321 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.322 DEBUG:boto:Params: {} 2022-09-02T05:47:54.323 DEBUG:boto:Token: None 2022-09-02T05:47:54.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85 2022-09-02T05:47:54.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l0dFstuF0eikuaM67M3o525B9e8= 2022-09-02T05:47:54.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l0dFstuF0eikuaM67M3o525B9e8=', 'Content-Length': '0'} 2022-09-02T05:47:54.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a395749ec3f7c88-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.328 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:54.329 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:54.330 DEBUG:boto:Method: GET 2022-09-02T05:47:54.330 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:54.331 DEBUG:boto:Data: 2022-09-02T05:47:54.332 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.333 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.334 DEBUG:boto:Params: {} 2022-09-02T05:47:54.334 DEBUG:boto:Token: None 2022-09-02T05:47:54.335 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj85 2022-09-02T05:47:54.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l0dFstuF0eikuaM67M3o525B9e8= 2022-09-02T05:47:54.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l0dFstuF0eikuaM67M3o525B9e8=', 'Content-Length': '0'} 2022-09-02T05:47:54.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a78d332db19520-006311990a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.382 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.383 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.384 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.384 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.385 DEBUG:boto:Method: GET 2022-09-02T05:47:54.386 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.386 DEBUG:boto:Data: 2022-09-02T05:47:54.387 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.388 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.389 DEBUG:boto:Params: {} 2022-09-02T05:47:54.390 DEBUG:boto:Token: None 2022-09-02T05:47:54.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c200af306dfeeed-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.396 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.397 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.397 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.398 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.399 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.399 DEBUG:boto:Method: GET 2022-09-02T05:47:54.400 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:54.400 DEBUG:boto:Data: 2022-09-02T05:47:54.401 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:54.402 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.403 DEBUG:boto:Params: {} 2022-09-02T05:47:54.404 DEBUG:boto:Token: None 2022-09-02T05:47:54.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada56d8858e14243-006311990a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.411 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.412 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:47:54.412 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:47:54.413 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.414 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.414 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.415 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:54.416 DEBUG:boto:Data: 2022-09-02T05:47:54.416 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.418 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.418 DEBUG:boto:Params: {} 2022-09-02T05:47:54.419 DEBUG:boto:Token: None 2022-09-02T05:47:54.420 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QoNebX0+At/G17Ptud2xQB6SkDs= 2022-09-02T05:47:54.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QoNebX0+At/G17Ptud2xQB6SkDs=', 'Content-Length': '0'} 2022-09-02T05:47:54.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091ea246c7e546e13-006311990a-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.425 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.426 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.426 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.427 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:54.428 DEBUG:boto:Data: 2022-09-02T05:47:54.428 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.429 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.430 DEBUG:boto:Params: {} 2022-09-02T05:47:54.431 DEBUG:boto:Token: None 2022-09-02T05:47:54.431 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QoNebX0+At/G17Ptud2xQB6SkDs= 2022-09-02T05:47:54.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QoNebX0+At/G17Ptud2xQB6SkDs=', 'Content-Length': '0'} 2022-09-02T05:47:54.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be34d3b1839bd619-006311990a-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.437 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:47:54.437 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.438 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.438 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.439 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.440 DEBUG:boto:Method: GET 2022-09-02T05:47:54.440 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.441 DEBUG:boto:Data: 2022-09-02T05:47:54.441 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.443 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.443 DEBUG:boto:Params: {} 2022-09-02T05:47:54.444 DEBUG:boto:Token: None 2022-09-02T05:47:54.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/?versions 2022-09-02T05:47:54.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE= 2022-09-02T05:47:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE=', 'Content-Length': '0'} 2022-09-02T05:47:54.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a82429ab1c4d15-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.450 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:47:54.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:47:54.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:47:54.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:47:54.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:47:54.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:47:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:47:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:47:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:47:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:47:54.456 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:47:54.457 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.458 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.458 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.459 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.459 DEBUG:boto:Method: GET 2022-09-02T05:47:54.460 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.461 DEBUG:boto:Data: 2022-09-02T05:47:54.461 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.463 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.463 DEBUG:boto:Params: {} 2022-09-02T05:47:54.464 DEBUG:boto:Token: None 2022-09-02T05:47:54.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/?versions 2022-09-02T05:47:54.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE= 2022-09-02T05:47:54.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE=', 'Content-Length': '0'} 2022-09-02T05:47:54.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2cba1f462c3cd2-006311990a-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.470 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:47:54.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:47:54.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:47:54.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:47:54.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:47:54.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:47:54.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:47:54.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:47:54.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:47:54.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:47:54.477 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.478 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.478 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.479 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.480 DEBUG:boto:Method: GET 2022-09-02T05:47:54.480 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.481 DEBUG:boto:Data: 2022-09-02T05:47:54.481 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.483 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.483 DEBUG:boto:Params: {} 2022-09-02T05:47:54.484 DEBUG:boto:Token: None 2022-09-02T05:47:54.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/?versions 2022-09-02T05:47:54.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE= 2022-09-02T05:47:54.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE=', 'Content-Length': '0'} 2022-09-02T05:47:54.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8abd4f0cfa6b1df-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.495 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.497 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.498 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.499 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.499 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.500 DEBUG:boto:Method: GET 2022-09-02T05:47:54.501 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:47:54.503 DEBUG:boto:Data: 2022-09-02T05:47:54.505 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.509 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.510 DEBUG:boto:Params: {} 2022-09-02T05:47:54.512 DEBUG:boto:Token: None 2022-09-02T05:47:54.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/?versions 2022-09-02T05:47:54.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE= 2022-09-02T05:47:54.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zF/ofiCJ1vH4qlOi34ZL4AejrYE=', 'Content-Length': '0'} 2022-09-02T05:47:54.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c445d9777a7c5cdb-006311990a-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.525 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:54.530 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.532 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.533 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.534 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.536 DEBUG:boto:Method: GET 2022-09-02T05:47:54.538 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.539 DEBUG:boto:Data: 2022-09-02T05:47:54.540 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.543 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.544 DEBUG:boto:Params: {} 2022-09-02T05:47:54.545 DEBUG:boto:Token: None 2022-09-02T05:47:54.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8= 2022-09-02T05:47:54.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8=', 'Content-Length': '0'} 2022-09-02T05:47:54.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001664791f63067a39-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.552 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.553 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.553 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.554 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.555 DEBUG:boto:Method: GET 2022-09-02T05:47:54.556 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.557 DEBUG:boto:Data: 2022-09-02T05:47:54.557 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.559 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.559 DEBUG:boto:Params: {} 2022-09-02T05:47:54.560 DEBUG:boto:Token: None 2022-09-02T05:47:54.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8= 2022-09-02T05:47:54.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8=', 'Content-Length': '0'} 2022-09-02T05:47:54.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a529f9e5390521cc-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.567 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.567 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.568 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.569 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.570 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.571 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.572 DEBUG:boto:Data: 2022-09-02T05:47:54.573 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.574 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.575 DEBUG:boto:Params: {} 2022-09-02T05:47:54.576 DEBUG:boto:Token: None 2022-09-02T05:47:54.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4vhqlVxSpSc6pfBRWPqjlnwBuEI= 2022-09-02T05:47:54.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4vhqlVxSpSc6pfBRWPqjlnwBuEI=', 'Content-Length': '0'} 2022-09-02T05:47:54.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c6cd39a47bd46e8-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.583 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.584 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.585 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.586 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.586 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.587 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.588 DEBUG:boto:Data: 2022-09-02T05:47:54.589 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.591 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.591 DEBUG:boto:Params: {} 2022-09-02T05:47:54.592 DEBUG:boto:Token: None 2022-09-02T05:47:54.593 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4vhqlVxSpSc6pfBRWPqjlnwBuEI= 2022-09-02T05:47:54.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4vhqlVxSpSc6pfBRWPqjlnwBuEI=', 'Content-Length': '0'} 2022-09-02T05:47:54.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009abcc2e5524e311a-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:54.600 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.600 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.601 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.601 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.602 DEBUG:boto:Method: GET 2022-09-02T05:47:54.602 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.603 DEBUG:boto:Data: 2022-09-02T05:47:54.604 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.605 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.605 DEBUG:boto:Params: {} 2022-09-02T05:47:54.606 DEBUG:boto:Token: None 2022-09-02T05:47:54.606 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8= 2022-09-02T05:47:54.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8=', 'Content-Length': '0'} 2022-09-02T05:47:54.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb9d94745e9cdf4-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.611 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.611 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.612 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.613 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.613 DEBUG:boto:Method: GET 2022-09-02T05:47:54.614 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.615 DEBUG:boto:Data: 2022-09-02T05:47:54.616 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.617 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.617 DEBUG:boto:Params: {} 2022-09-02T05:47:54.618 DEBUG:boto:Token: None 2022-09-02T05:47:54.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:47:54.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8= 2022-09-02T05:47:54.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xQrMhOi5Nk7msqb2/qEv18MC6H8=', 'Content-Length': '0'} 2022-09-02T05:47:54.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000652aff5fbf1d601e-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.623 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.624 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.625 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.626 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.627 DEBUG:boto:Method: GET 2022-09-02T05:47:54.628 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.628 DEBUG:boto:Data: 2022-09-02T05:47:54.629 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.630 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.630 DEBUG:boto:Params: {} 2022-09-02T05:47:54.631 DEBUG:boto:Token: None 2022-09-02T05:47:54.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d1a086bfbe3479-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.637 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.638 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.639 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.639 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.640 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.641 DEBUG:boto:Method: GET 2022-09-02T05:47:54.641 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.642 DEBUG:boto:Data: 2022-09-02T05:47:54.643 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.643 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.644 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.644 DEBUG:boto:Params: {} 2022-09-02T05:47:54.645 DEBUG:boto:Token: None 2022-09-02T05:47:54.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b18281db18f9f3-006311990a-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.694 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.696 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.696 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.697 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.698 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:54.699 DEBUG:boto:Data: 2022-09-02T05:47:54.700 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.702 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.704 DEBUG:boto:Params: {} 2022-09-02T05:47:54.705 DEBUG:boto:Token: None 2022-09-02T05:47:54.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11 2022-09-02T05:47:54.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p1QoNtet5gw7o2Ib/Lr6IeEm0js= 2022-09-02T05:47:54.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p1QoNtet5gw7o2Ib/Lr6IeEm0js=', 'Content-Length': '0'} 2022-09-02T05:47:54.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f16d2d7aa7ee292-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.712 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.713 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.713 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.714 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:54.715 DEBUG:boto:Data: 2022-09-02T05:47:54.716 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.717 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.718 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.719 DEBUG:boto:Params: {} 2022-09-02T05:47:54.721 DEBUG:boto:Token: None 2022-09-02T05:47:54.722 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11 2022-09-02T05:47:54.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p1QoNtet5gw7o2Ib/Lr6IeEm0js= 2022-09-02T05:47:54.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p1QoNtet5gw7o2Ib/Lr6IeEm0js=', 'Content-Length': '0'} 2022-09-02T05:47:54.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5fcfc911063035e-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:47:54.729 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.730 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.731 DEBUG:boto:Method: GET 2022-09-02T05:47:54.731 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:54.732 DEBUG:boto:Data: 2022-09-02T05:47:54.732 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.733 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.734 DEBUG:boto:Params: {} 2022-09-02T05:47:54.735 DEBUG:boto:Token: None 2022-09-02T05:47:54.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11 2022-09-02T05:47:54.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7LtY7ihG2L1kXLvurmnQDKmHM9Q= 2022-09-02T05:47:54.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7LtY7ihG2L1kXLvurmnQDKmHM9Q=', 'Content-Length': '0'} 2022-09-02T05:47:54.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b8f6cef7c40043-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.740 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:47:54.741 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:47:54.741 DEBUG:boto:Method: GET 2022-09-02T05:47:54.742 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:47:54.742 DEBUG:boto:Data: 2022-09-02T05:47:54.743 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.744 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.744 DEBUG:boto:Params: {} 2022-09-02T05:47:54.745 DEBUG:boto:Token: None 2022-09-02T05:47:54.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj11 2022-09-02T05:47:54.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7LtY7ihG2L1kXLvurmnQDKmHM9Q= 2022-09-02T05:47:54.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7LtY7ihG2L1kXLvurmnQDKmHM9Q=', 'Content-Length': '0'} 2022-09-02T05:47:54.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f7b48632fc7de0-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.750 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.751 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.751 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.752 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.752 DEBUG:boto:Method: GET 2022-09-02T05:47:54.753 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.754 DEBUG:boto:Data: 2022-09-02T05:47:54.754 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.755 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.756 DEBUG:boto:Params: {} 2022-09-02T05:47:54.757 DEBUG:boto:Token: None 2022-09-02T05:47:54.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d137081da312d81d-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.802 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.804 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.804 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.805 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.806 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.806 DEBUG:boto:Method: GET 2022-09-02T05:47:54.807 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:47:54.808 DEBUG:boto:Data: 2022-09-02T05:47:54.809 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.811 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.812 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.813 DEBUG:boto:Params: {} 2022-09-02T05:47:54.814 DEBUG:boto:Token: None 2022-09-02T05:47:54.816 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.817 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f0f7c63a75a0de-006311990a-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.822 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:54.824 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.825 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.826 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.827 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.829 DEBUG:boto:Method: GET 2022-09-02T05:47:54.830 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.831 DEBUG:boto:Data: 2022-09-02T05:47:54.831 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.832 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.833 DEBUG:boto:Params: {} 2022-09-02T05:47:54.833 DEBUG:boto:Token: None 2022-09-02T05:47:54.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc= 2022-09-02T05:47:54.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc=', 'Content-Length': '0'} 2022-09-02T05:47:54.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7cc64cb38d8add-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.839 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.841 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.843 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.844 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.845 DEBUG:boto:Method: GET 2022-09-02T05:47:54.846 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.846 DEBUG:boto:Data: 2022-09-02T05:47:54.847 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.848 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.849 DEBUG:boto:Params: {} 2022-09-02T05:47:54.850 DEBUG:boto:Token: None 2022-09-02T05:47:54.850 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc= 2022-09-02T05:47:54.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc=', 'Content-Length': '0'} 2022-09-02T05:47:54.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df9be79e6a9a852b-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.855 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.855 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.856 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.856 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.857 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.858 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.858 DEBUG:boto:Data: 2022-09-02T05:47:54.859 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.860 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.860 DEBUG:boto:Params: {} 2022-09-02T05:47:54.861 DEBUG:boto:Token: None 2022-09-02T05:47:54.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EmZ/uww5LU+YrNwQIzLsENKakaY= 2022-09-02T05:47:54.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EmZ/uww5LU+YrNwQIzLsENKakaY=', 'Content-Length': '0'} 2022-09-02T05:47:54.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05aa3d6dc0e29d3-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.866 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.867 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.867 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.868 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.868 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.869 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.869 DEBUG:boto:Data: 2022-09-02T05:47:54.870 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.871 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.871 DEBUG:boto:Params: {} 2022-09-02T05:47:54.872 DEBUG:boto:Token: None 2022-09-02T05:47:54.873 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EmZ/uww5LU+YrNwQIzLsENKakaY= 2022-09-02T05:47:54.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EmZ/uww5LU+YrNwQIzLsENKakaY=', 'Content-Length': '0'} 2022-09-02T05:47:54.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989544f07d6f6712-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:54.878 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.878 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.879 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.879 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.880 DEBUG:boto:Method: GET 2022-09-02T05:47:54.880 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.881 DEBUG:boto:Data: 2022-09-02T05:47:54.881 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.883 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.883 DEBUG:boto:Params: {} 2022-09-02T05:47:54.884 DEBUG:boto:Token: None 2022-09-02T05:47:54.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc= 2022-09-02T05:47:54.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc=', 'Content-Length': '0'} 2022-09-02T05:47:54.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffe4e44819061d3-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.888 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.889 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.890 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.890 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.891 DEBUG:boto:Method: GET 2022-09-02T05:47:54.891 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.892 DEBUG:boto:Data: 2022-09-02T05:47:54.892 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.893 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.894 DEBUG:boto:Params: {} 2022-09-02T05:47:54.895 DEBUG:boto:Token: None 2022-09-02T05:47:54.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:47:54.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc= 2022-09-02T05:47:54.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tyxf4XXJmNpW7E1LACM/Lvv+GQc=', 'Content-Length': '0'} 2022-09-02T05:47:54.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af27d9fc06348ab-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.900 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.900 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.901 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:54.901 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:54.902 DEBUG:boto:Method: GET 2022-09-02T05:47:54.902 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:54.903 DEBUG:boto:Data: 2022-09-02T05:47:54.903 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.904 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.905 DEBUG:boto:Params: {} 2022-09-02T05:47:54.906 DEBUG:boto:Token: None 2022-09-02T05:47:54.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079e89993c3d4e612-006311990a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.911 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.912 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:54.912 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:54.913 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:54.913 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:54.914 DEBUG:boto:Method: GET 2022-09-02T05:47:54.914 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:54.915 DEBUG:boto:Data: 2022-09-02T05:47:54.915 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.916 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.916 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.917 DEBUG:boto:Params: {} 2022-09-02T05:47:54.918 DEBUG:boto:Token: None 2022-09-02T05:47:54.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/ 2022-09-02T05:47:54.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM= 2022-09-02T05:47:54.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kUq0XIkrKyyuaQjfN51P23HRzTM=', 'Content-Length': '0'} 2022-09-02T05:47:54.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c215f54d56ebe22-006311990a-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.966 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:54.967 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.968 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.968 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.969 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:54.969 DEBUG:boto:Data: 2022-09-02T05:47:54.970 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.971 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.971 DEBUG:boto:Params: {} 2022-09-02T05:47:54.972 DEBUG:boto:Token: None 2022-09-02T05:47:54.973 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17 2022-09-02T05:47:54.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MdVSdrMew24eVDK+mmwtpULdRKs= 2022-09-02T05:47:54.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MdVSdrMew24eVDK+mmwtpULdRKs=', 'Content-Length': '0'} 2022-09-02T05:47:54.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e9883cf838b022a-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.977 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.978 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.978 DEBUG:boto:Method: HEAD 2022-09-02T05:47:54.979 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:54.979 DEBUG:boto:Data: 2022-09-02T05:47:54.980 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:54.981 DEBUG:boto:Port: 8000 2022-09-02T05:47:54.981 DEBUG:boto:Params: {} 2022-09-02T05:47:54.982 DEBUG:boto:Token: None 2022-09-02T05:47:54.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17 2022-09-02T05:47:54.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MdVSdrMew24eVDK+mmwtpULdRKs= 2022-09-02T05:47:54.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MdVSdrMew24eVDK+mmwtpULdRKs=', 'Content-Length': '0'} 2022-09-02T05:47:54.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6acd96ea13d1d98-006311990a-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:54.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:47:54.990 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:54.990 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:54.991 DEBUG:boto:Method: GET 2022-09-02T05:47:54.992 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:54.992 DEBUG:boto:Data: 2022-09-02T05:47:54.993 DEBUG:boto:Headers: {} 2022-09-02T05:47:54.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:54.994 DEBUG:boto:Port: 8001 2022-09-02T05:47:54.994 DEBUG:boto:Params: {} 2022-09-02T05:47:54.995 DEBUG:boto:Token: None 2022-09-02T05:47:54.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:54 GMT /xljbkf-74/obj17 2022-09-02T05:47:54.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wHcVP9PKm/o5Js9Um8Fz1oYi9MI= 2022-09-02T05:47:54.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wHcVP9PKm/o5Js9Um8Fz1oYi9MI=', 'Content-Length': '0'} 2022-09-02T05:47:55.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe256d481de946c-006311990a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.000 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:47:55.001 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:47:55.002 DEBUG:boto:Method: GET 2022-09-02T05:47:55.002 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:47:55.003 DEBUG:boto:Data: 2022-09-02T05:47:55.003 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.004 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.004 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.005 DEBUG:boto:Params: {} 2022-09-02T05:47:55.005 DEBUG:boto:Token: None 2022-09-02T05:47:55.006 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj17 2022-09-02T05:47:55.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CCsJxn+ikrvxYNLeBAdg2vj4U9c= 2022-09-02T05:47:55.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CCsJxn+ikrvxYNLeBAdg2vj4U9c=', 'Content-Length': '0'} 2022-09-02T05:47:55.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ddf293e749ddb9-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.011 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.011 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.012 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:55.012 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:55.013 DEBUG:boto:Method: GET 2022-09-02T05:47:55.013 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:55.014 DEBUG:boto:Data: 2022-09-02T05:47:55.014 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.015 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.016 DEBUG:boto:Params: {} 2022-09-02T05:47:55.017 DEBUG:boto:Token: None 2022-09-02T05:47:55.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10965479b8e33a9-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.022 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.023 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.024 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.024 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:55.025 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:47:55.025 DEBUG:boto:Method: GET 2022-09-02T05:47:55.026 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:47:55.026 DEBUG:boto:Data: 2022-09-02T05:47:55.027 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.028 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.029 DEBUG:boto:Params: {} 2022-09-02T05:47:55.029 DEBUG:boto:Token: None 2022-09-02T05:47:55.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000700a7c14c088583d-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.035 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:55.036 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.037 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.037 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.038 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.039 DEBUG:boto:Method: GET 2022-09-02T05:47:55.040 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.041 DEBUG:boto:Data: 2022-09-02T05:47:55.041 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.043 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.043 DEBUG:boto:Params: {} 2022-09-02T05:47:55.044 DEBUG:boto:Token: None 2022-09-02T05:47:55.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I= 2022-09-02T05:47:55.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I=', 'Content-Length': '0'} 2022-09-02T05:47:55.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bef6c1577859333-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.049 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.050 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.051 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.051 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.052 DEBUG:boto:Method: GET 2022-09-02T05:47:55.052 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.053 DEBUG:boto:Data: 2022-09-02T05:47:55.054 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.055 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.055 DEBUG:boto:Params: {} 2022-09-02T05:47:55.056 DEBUG:boto:Token: None 2022-09-02T05:47:55.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I= 2022-09-02T05:47:55.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I=', 'Content-Length': '0'} 2022-09-02T05:47:55.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000385215acc8d28813-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.061 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.062 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.063 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.064 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.065 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.066 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.067 DEBUG:boto:Data: 2022-09-02T05:47:55.068 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.070 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.071 DEBUG:boto:Params: {} 2022-09-02T05:47:55.071 DEBUG:boto:Token: None 2022-09-02T05:47:55.072 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.073 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G/vUxfFuDJjG0wj86FJxgLGzsfY= 2022-09-02T05:47:55.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G/vUxfFuDJjG0wj86FJxgLGzsfY=', 'Content-Length': '0'} 2022-09-02T05:47:55.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb5afd52b7f286f4-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.078 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.079 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.080 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.080 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.081 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.082 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.083 DEBUG:boto:Data: 2022-09-02T05:47:55.084 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.086 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.087 DEBUG:boto:Params: {} 2022-09-02T05:47:55.087 DEBUG:boto:Token: None 2022-09-02T05:47:55.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G/vUxfFuDJjG0wj86FJxgLGzsfY= 2022-09-02T05:47:55.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G/vUxfFuDJjG0wj86FJxgLGzsfY=', 'Content-Length': '0'} 2022-09-02T05:47:55.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eafda8893a9495c6-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:55.093 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.093 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.094 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.094 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.095 DEBUG:boto:Method: GET 2022-09-02T05:47:55.095 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.096 DEBUG:boto:Data: 2022-09-02T05:47:55.096 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.098 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.098 DEBUG:boto:Params: {} 2022-09-02T05:47:55.099 DEBUG:boto:Token: None 2022-09-02T05:47:55.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I= 2022-09-02T05:47:55.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I=', 'Content-Length': '0'} 2022-09-02T05:47:55.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004348e1dabea56445-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.104 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.104 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.105 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.106 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.106 DEBUG:boto:Method: GET 2022-09-02T05:47:55.107 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.107 DEBUG:boto:Data: 2022-09-02T05:47:55.108 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.109 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.109 DEBUG:boto:Params: {} 2022-09-02T05:47:55.110 DEBUG:boto:Token: None 2022-09-02T05:47:55.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:47:55.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I= 2022-09-02T05:47:55.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h/fc+ibsAVQEXHp9fZ6pVQOsq+I=', 'Content-Length': '0'} 2022-09-02T05:47:55.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0657d482c97d1f-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.115 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.115 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.116 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.116 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.117 DEBUG:boto:Method: GET 2022-09-02T05:47:55.117 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.118 DEBUG:boto:Data: 2022-09-02T05:47:55.119 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.120 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.121 DEBUG:boto:Params: {} 2022-09-02T05:47:55.121 DEBUG:boto:Token: None 2022-09-02T05:47:55.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3491061368365c6-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.174 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.176 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.177 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.178 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.179 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.180 DEBUG:boto:Method: GET 2022-09-02T05:47:55.181 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.182 DEBUG:boto:Data: 2022-09-02T05:47:55.183 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.185 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.186 DEBUG:boto:Params: {} 2022-09-02T05:47:55.187 DEBUG:boto:Token: None 2022-09-02T05:47:55.188 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea8a57c016e86036-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.195 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.196 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.197 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.198 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.199 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:55.200 DEBUG:boto:Data: 2022-09-02T05:47:55.201 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.203 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.204 DEBUG:boto:Params: {} 2022-09-02T05:47:55.205 DEBUG:boto:Token: None 2022-09-02T05:47:55.206 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22 2022-09-02T05:47:55.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7CCfuYpv3wxYxwkzOz9BQ+gEamw= 2022-09-02T05:47:55.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7CCfuYpv3wxYxwkzOz9BQ+gEamw=', 'Content-Length': '0'} 2022-09-02T05:47:55.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7a795811817819-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.210 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.211 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.211 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.212 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:55.212 DEBUG:boto:Data: 2022-09-02T05:47:55.213 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.214 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.214 DEBUG:boto:Params: {} 2022-09-02T05:47:55.215 DEBUG:boto:Token: None 2022-09-02T05:47:55.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22 2022-09-02T05:47:55.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7CCfuYpv3wxYxwkzOz9BQ+gEamw= 2022-09-02T05:47:55.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7CCfuYpv3wxYxwkzOz9BQ+gEamw=', 'Content-Length': '0'} 2022-09-02T05:47:55.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d432fd256e4ac77d-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:47:55.220 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.220 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.221 DEBUG:boto:Method: GET 2022-09-02T05:47:55.222 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:55.222 DEBUG:boto:Data: 2022-09-02T05:47:55.223 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.224 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.224 DEBUG:boto:Params: {} 2022-09-02T05:47:55.225 DEBUG:boto:Token: None 2022-09-02T05:47:55.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22 2022-09-02T05:47:55.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uzYlGGW6LrNA5HMLxYzd/1VxOnM= 2022-09-02T05:47:55.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uzYlGGW6LrNA5HMLxYzd/1VxOnM=', 'Content-Length': '0'} 2022-09-02T05:47:55.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6827e4b685bdc4f-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.230 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:47:55.230 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:47:55.239 DEBUG:boto:Method: GET 2022-09-02T05:47:55.240 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:47:55.241 DEBUG:boto:Data: 2022-09-02T05:47:55.241 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.243 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.243 DEBUG:boto:Params: {} 2022-09-02T05:47:55.244 DEBUG:boto:Token: None 2022-09-02T05:47:55.244 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj22 2022-09-02T05:47:55.245 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uzYlGGW6LrNA5HMLxYzd/1VxOnM= 2022-09-02T05:47:55.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uzYlGGW6LrNA5HMLxYzd/1VxOnM=', 'Content-Length': '0'} 2022-09-02T05:47:55.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f59a928517ac3781-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.290 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.291 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.292 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.293 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.294 DEBUG:boto:Method: GET 2022-09-02T05:47:55.295 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.296 DEBUG:boto:Data: 2022-09-02T05:47:55.296 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.298 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.299 DEBUG:boto:Params: {} 2022-09-02T05:47:55.300 DEBUG:boto:Token: None 2022-09-02T05:47:55.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005daf14a62c2e9252-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.305 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.306 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.306 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.307 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.307 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.308 DEBUG:boto:Method: GET 2022-09-02T05:47:55.308 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:47:55.309 DEBUG:boto:Data: 2022-09-02T05:47:55.310 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.311 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.311 DEBUG:boto:Params: {} 2022-09-02T05:47:55.312 DEBUG:boto:Token: None 2022-09-02T05:47:55.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7bf8980c0bf264-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.317 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:55.318 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.319 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.319 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.320 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.321 DEBUG:boto:Method: GET 2022-09-02T05:47:55.321 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.322 DEBUG:boto:Data: 2022-09-02T05:47:55.322 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.323 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.324 DEBUG:boto:Params: {} 2022-09-02T05:47:55.324 DEBUG:boto:Token: None 2022-09-02T05:47:55.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc= 2022-09-02T05:47:55.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc=', 'Content-Length': '0'} 2022-09-02T05:47:55.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093524b8f327666fd-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.329 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.330 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.330 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.331 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.331 DEBUG:boto:Method: GET 2022-09-02T05:47:55.332 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.332 DEBUG:boto:Data: 2022-09-02T05:47:55.333 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.334 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.334 DEBUG:boto:Params: {} 2022-09-02T05:47:55.335 DEBUG:boto:Token: None 2022-09-02T05:47:55.335 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc= 2022-09-02T05:47:55.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc=', 'Content-Length': '0'} 2022-09-02T05:47:55.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdefd5a721ab48ef-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.340 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.340 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.341 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.341 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.342 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.342 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.343 DEBUG:boto:Data: 2022-09-02T05:47:55.343 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.344 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.345 DEBUG:boto:Params: {} 2022-09-02T05:47:55.345 DEBUG:boto:Token: None 2022-09-02T05:47:55.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:prVvTCmapsrW5hAym69TkFA1VNY= 2022-09-02T05:47:55.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:prVvTCmapsrW5hAym69TkFA1VNY=', 'Content-Length': '0'} 2022-09-02T05:47:55.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341b245af373cc0c-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.350 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.351 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.351 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.352 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.352 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.353 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.353 DEBUG:boto:Data: 2022-09-02T05:47:55.354 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.355 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.355 DEBUG:boto:Params: {} 2022-09-02T05:47:55.356 DEBUG:boto:Token: None 2022-09-02T05:47:55.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:prVvTCmapsrW5hAym69TkFA1VNY= 2022-09-02T05:47:55.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:prVvTCmapsrW5hAym69TkFA1VNY=', 'Content-Length': '0'} 2022-09-02T05:47:55.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef5d3397e274597-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:55.361 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.361 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.362 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.362 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.363 DEBUG:boto:Method: GET 2022-09-02T05:47:55.363 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.364 DEBUG:boto:Data: 2022-09-02T05:47:55.364 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.365 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.366 DEBUG:boto:Params: {} 2022-09-02T05:47:55.367 DEBUG:boto:Token: None 2022-09-02T05:47:55.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc= 2022-09-02T05:47:55.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc=', 'Content-Length': '0'} 2022-09-02T05:47:55.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418610818b3453bd-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.371 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.371 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.372 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.372 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.373 DEBUG:boto:Method: GET 2022-09-02T05:47:55.374 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.374 DEBUG:boto:Data: 2022-09-02T05:47:55.375 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.375 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.376 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.376 DEBUG:boto:Params: {} 2022-09-02T05:47:55.377 DEBUG:boto:Token: None 2022-09-02T05:47:55.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:47:55.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc= 2022-09-02T05:47:55.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yGlMtGnOa2iomfxOLJ5iLLp5poc=', 'Content-Length': '0'} 2022-09-02T05:47:55.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8d924848c97b7f-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.381 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.382 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.383 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.383 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.384 DEBUG:boto:Method: GET 2022-09-02T05:47:55.384 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.385 DEBUG:boto:Data: 2022-09-02T05:47:55.385 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.386 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.387 DEBUG:boto:Params: {} 2022-09-02T05:47:55.387 DEBUG:boto:Token: None 2022-09-02T05:47:55.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050fc2a5cf2932f8c-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.394 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.394 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.395 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.395 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.396 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.396 DEBUG:boto:Method: GET 2022-09-02T05:47:55.397 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.397 DEBUG:boto:Data: 2022-09-02T05:47:55.398 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.399 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.400 DEBUG:boto:Params: {} 2022-09-02T05:47:55.400 DEBUG:boto:Token: None 2022-09-02T05:47:55.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0de9670c756b666-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.405 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.406 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.407 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.407 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.408 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:55.408 DEBUG:boto:Data: 2022-09-02T05:47:55.409 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.410 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.410 DEBUG:boto:Params: {} 2022-09-02T05:47:55.411 DEBUG:boto:Token: None 2022-09-02T05:47:55.411 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33 2022-09-02T05:47:55.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Kn6rMPQpajcNvNHp+QFqL0C91M= 2022-09-02T05:47:55.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Kn6rMPQpajcNvNHp+QFqL0C91M=', 'Content-Length': '0'} 2022-09-02T05:47:55.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e02bbe5caf121450-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.415 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.416 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.416 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.417 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:55.417 DEBUG:boto:Data: 2022-09-02T05:47:55.418 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.418 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.419 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.419 DEBUG:boto:Params: {} 2022-09-02T05:47:55.420 DEBUG:boto:Token: None 2022-09-02T05:47:55.420 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33 2022-09-02T05:47:55.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Kn6rMPQpajcNvNHp+QFqL0C91M= 2022-09-02T05:47:55.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Kn6rMPQpajcNvNHp+QFqL0C91M=', 'Content-Length': '0'} 2022-09-02T05:47:55.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe139e0a7b8ee549-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:47:55.425 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.425 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.426 DEBUG:boto:Method: GET 2022-09-02T05:47:55.426 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:55.427 DEBUG:boto:Data: 2022-09-02T05:47:55.427 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.428 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.429 DEBUG:boto:Params: {} 2022-09-02T05:47:55.430 DEBUG:boto:Token: None 2022-09-02T05:47:55.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33 2022-09-02T05:47:55.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dlT7tbi8vfQFUEAoCPImxeViVKI= 2022-09-02T05:47:55.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dlT7tbi8vfQFUEAoCPImxeViVKI=', 'Content-Length': '0'} 2022-09-02T05:47:55.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e0d283f2b8cd7c-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.478 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:47:55.479 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:47:55.479 DEBUG:boto:Method: GET 2022-09-02T05:47:55.480 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:47:55.480 DEBUG:boto:Data: 2022-09-02T05:47:55.481 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.482 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.483 DEBUG:boto:Params: {} 2022-09-02T05:47:55.483 DEBUG:boto:Token: None 2022-09-02T05:47:55.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj33 2022-09-02T05:47:55.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dlT7tbi8vfQFUEAoCPImxeViVKI= 2022-09-02T05:47:55.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dlT7tbi8vfQFUEAoCPImxeViVKI=', 'Content-Length': '0'} 2022-09-02T05:47:55.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9071c2e9f7fcfb-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.488 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.488 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.489 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.489 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.490 DEBUG:boto:Method: GET 2022-09-02T05:47:55.491 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.491 DEBUG:boto:Data: 2022-09-02T05:47:55.492 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.493 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.493 DEBUG:boto:Params: {} 2022-09-02T05:47:55.494 DEBUG:boto:Token: None 2022-09-02T05:47:55.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba14e70ffc4ac6af-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.499 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.500 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.500 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.501 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.501 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.502 DEBUG:boto:Method: GET 2022-09-02T05:47:55.502 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:47:55.503 DEBUG:boto:Data: 2022-09-02T05:47:55.503 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.505 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.505 DEBUG:boto:Params: {} 2022-09-02T05:47:55.506 DEBUG:boto:Token: None 2022-09-02T05:47:55.507 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8278f9c39f006f2-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.511 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:55.512 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.513 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.513 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.514 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.515 DEBUG:boto:Method: GET 2022-09-02T05:47:55.515 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.516 DEBUG:boto:Data: 2022-09-02T05:47:55.516 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.517 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.518 DEBUG:boto:Params: {} 2022-09-02T05:47:55.518 DEBUG:boto:Token: None 2022-09-02T05:47:55.519 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0= 2022-09-02T05:47:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0=', 'Content-Length': '0'} 2022-09-02T05:47:55.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9a9250eb8de2ef7-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.523 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.524 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.524 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.525 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.525 DEBUG:boto:Method: GET 2022-09-02T05:47:55.526 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.526 DEBUG:boto:Data: 2022-09-02T05:47:55.527 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.528 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.528 DEBUG:boto:Params: {} 2022-09-02T05:47:55.529 DEBUG:boto:Token: None 2022-09-02T05:47:55.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0= 2022-09-02T05:47:55.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0=', 'Content-Length': '0'} 2022-09-02T05:47:55.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c71493c593942448-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.578 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.579 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.580 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.581 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.581 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.582 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.582 DEBUG:boto:Data: 2022-09-02T05:47:55.583 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.584 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.584 DEBUG:boto:Params: {} 2022-09-02T05:47:55.585 DEBUG:boto:Token: None 2022-09-02T05:47:55.586 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5FVFVZYuthshUF2kxyBTQ5CZZ4s= 2022-09-02T05:47:55.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5FVFVZYuthshUF2kxyBTQ5CZZ4s=', 'Content-Length': '0'} 2022-09-02T05:47:55.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be47d40cc248c320-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.591 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.591 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.592 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.592 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.593 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.594 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.594 DEBUG:boto:Data: 2022-09-02T05:47:55.595 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.597 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.597 DEBUG:boto:Params: {} 2022-09-02T05:47:55.598 DEBUG:boto:Token: None 2022-09-02T05:47:55.600 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5FVFVZYuthshUF2kxyBTQ5CZZ4s= 2022-09-02T05:47:55.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5FVFVZYuthshUF2kxyBTQ5CZZ4s=', 'Content-Length': '0'} 2022-09-02T05:47:55.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084680307ffa04bab-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:55.608 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.609 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.610 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.611 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.612 DEBUG:boto:Method: GET 2022-09-02T05:47:55.613 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.614 DEBUG:boto:Data: 2022-09-02T05:47:55.615 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.617 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.618 DEBUG:boto:Params: {} 2022-09-02T05:47:55.619 DEBUG:boto:Token: None 2022-09-02T05:47:55.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0= 2022-09-02T05:47:55.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0=', 'Content-Length': '0'} 2022-09-02T05:47:55.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b072784795c8a9-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.628 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.629 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.629 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.630 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.630 DEBUG:boto:Method: GET 2022-09-02T05:47:55.631 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.632 DEBUG:boto:Data: 2022-09-02T05:47:55.632 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.633 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.633 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.634 DEBUG:boto:Params: {} 2022-09-02T05:47:55.635 DEBUG:boto:Token: None 2022-09-02T05:47:55.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:47:55.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0= 2022-09-02T05:47:55.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEUiS/+PH8GZ1HH41jg8u9u2ye0=', 'Content-Length': '0'} 2022-09-02T05:47:55.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bfd379e8aa983db-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.652 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.653 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.657 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.658 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.659 DEBUG:boto:Method: GET 2022-09-02T05:47:55.662 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.662 DEBUG:boto:Data: 2022-09-02T05:47:55.663 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.665 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.665 DEBUG:boto:Params: {} 2022-09-02T05:47:55.666 DEBUG:boto:Token: None 2022-09-02T05:47:55.667 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aa0ca535a0489c8-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.672 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.673 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.674 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.675 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.675 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.676 DEBUG:boto:Method: GET 2022-09-02T05:47:55.677 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.677 DEBUG:boto:Data: 2022-09-02T05:47:55.678 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.679 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.680 DEBUG:boto:Params: {} 2022-09-02T05:47:55.680 DEBUG:boto:Token: None 2022-09-02T05:47:55.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c22da2947b665417-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.686 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.687 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.688 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.688 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.689 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:55.690 DEBUG:boto:Data: 2022-09-02T05:47:55.690 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.691 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.692 DEBUG:boto:Params: {} 2022-09-02T05:47:55.693 DEBUG:boto:Token: None 2022-09-02T05:47:55.693 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34 2022-09-02T05:47:55.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m45DwgEjkF2C01ejisZKnbs7jaY= 2022-09-02T05:47:55.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m45DwgEjkF2C01ejisZKnbs7jaY=', 'Content-Length': '0'} 2022-09-02T05:47:55.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967a9a45bc64d021-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.698 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.699 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.700 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.700 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:55.701 DEBUG:boto:Data: 2022-09-02T05:47:55.701 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.703 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.703 DEBUG:boto:Params: {} 2022-09-02T05:47:55.704 DEBUG:boto:Token: None 2022-09-02T05:47:55.705 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34 2022-09-02T05:47:55.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m45DwgEjkF2C01ejisZKnbs7jaY= 2022-09-02T05:47:55.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m45DwgEjkF2C01ejisZKnbs7jaY=', 'Content-Length': '0'} 2022-09-02T05:47:55.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f03d1fdcb8fcd3cb-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:47:55.710 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.710 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.711 DEBUG:boto:Method: GET 2022-09-02T05:47:55.712 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:55.712 DEBUG:boto:Data: 2022-09-02T05:47:55.713 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.714 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.714 DEBUG:boto:Params: {} 2022-09-02T05:47:55.715 DEBUG:boto:Token: None 2022-09-02T05:47:55.716 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34 2022-09-02T05:47:55.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Snn7gkHxChoT6AudnVhcKWkcDgM= 2022-09-02T05:47:55.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Snn7gkHxChoT6AudnVhcKWkcDgM=', 'Content-Length': '0'} 2022-09-02T05:47:55.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008706a4bf6e5b3d-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.766 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:47:55.767 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:47:55.768 DEBUG:boto:Method: GET 2022-09-02T05:47:55.769 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:47:55.771 DEBUG:boto:Data: 2022-09-02T05:47:55.771 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.774 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.775 DEBUG:boto:Params: {} 2022-09-02T05:47:55.776 DEBUG:boto:Token: None 2022-09-02T05:47:55.777 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj34 2022-09-02T05:47:55.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Snn7gkHxChoT6AudnVhcKWkcDgM= 2022-09-02T05:47:55.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Snn7gkHxChoT6AudnVhcKWkcDgM=', 'Content-Length': '0'} 2022-09-02T05:47:55.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add264c96866fe3b-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.806 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.806 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.807 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.808 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.809 DEBUG:boto:Method: GET 2022-09-02T05:47:55.810 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.810 DEBUG:boto:Data: 2022-09-02T05:47:55.811 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.812 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.813 DEBUG:boto:Params: {} 2022-09-02T05:47:55.813 DEBUG:boto:Token: None 2022-09-02T05:47:55.814 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ee99bec5a5375ff-006311990b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.829 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.830 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:55.831 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:55.832 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.833 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.834 DEBUG:boto:Method: GET 2022-09-02T05:47:55.834 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:47:55.835 DEBUG:boto:Data: 2022-09-02T05:47:55.835 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.837 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.837 DEBUG:boto:Params: {} 2022-09-02T05:47:55.838 DEBUG:boto:Token: None 2022-09-02T05:47:55.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/ 2022-09-02T05:47:55.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc= 2022-09-02T05:47:55.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zlRha6Jj0wi5J1XniaPWCgLdmXc=', 'Content-Length': '0'} 2022-09-02T05:47:55.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b1d98941ba38ce5-006311990b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.853 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:55.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:55.856 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:55.857 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:55.857 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.858 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.859 DEBUG:boto:Method: GET 2022-09-02T05:47:55.860 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.861 DEBUG:boto:Data: 2022-09-02T05:47:55.862 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.864 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.864 DEBUG:boto:Params: {} 2022-09-02T05:47:55.865 DEBUG:boto:Token: None 2022-09-02T05:47:55.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80= 2022-09-02T05:47:55.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80=', 'Content-Length': '0'} 2022-09-02T05:47:55.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95fa921493677a4-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.888 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:55.889 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:55.890 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.890 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.891 DEBUG:boto:Method: GET 2022-09-02T05:47:55.892 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.892 DEBUG:boto:Data: 2022-09-02T05:47:55.893 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.894 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.895 DEBUG:boto:Params: {} 2022-09-02T05:47:55.895 DEBUG:boto:Token: None 2022-09-02T05:47:55.896 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80= 2022-09-02T05:47:55.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80=', 'Content-Length': '0'} 2022-09-02T05:47:55.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a604b05263a3f44-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.906 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:55.907 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:55.907 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.908 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.908 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.909 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.909 DEBUG:boto:Data: 2022-09-02T05:47:55.910 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.911 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.912 DEBUG:boto:Params: {} 2022-09-02T05:47:55.912 DEBUG:boto:Token: None 2022-09-02T05:47:55.913 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uLzctkBBTvPqlOVPKi7xTXSY5AU= 2022-09-02T05:47:55.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uLzctkBBTvPqlOVPKi7xTXSY5AU=', 'Content-Length': '0'} 2022-09-02T05:47:55.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3715e4d220a12a1-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.919 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:55.920 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:55.920 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.921 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.921 DEBUG:boto:Method: HEAD 2022-09-02T05:47:55.922 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.922 DEBUG:boto:Data: 2022-09-02T05:47:55.923 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.924 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.924 DEBUG:boto:Params: {} 2022-09-02T05:47:55.925 DEBUG:boto:Token: None 2022-09-02T05:47:55.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uLzctkBBTvPqlOVPKi7xTXSY5AU= 2022-09-02T05:47:55.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uLzctkBBTvPqlOVPKi7xTXSY5AU=', 'Content-Length': '0'} 2022-09-02T05:47:55.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004880d83f63e851d2-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:55.932 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:55.933 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:55.933 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.934 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.934 DEBUG:boto:Method: GET 2022-09-02T05:47:55.935 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.935 DEBUG:boto:Data: 2022-09-02T05:47:55.936 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:55.937 DEBUG:boto:Port: 8001 2022-09-02T05:47:55.938 DEBUG:boto:Params: {} 2022-09-02T05:47:55.938 DEBUG:boto:Token: None 2022-09-02T05:47:55.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80= 2022-09-02T05:47:55.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80=', 'Content-Length': '0'} 2022-09-02T05:47:55.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d45350ba0273b9d-006311990b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:55.946 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:55.947 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:55.948 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.948 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.949 DEBUG:boto:Method: GET 2022-09-02T05:47:55.949 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.950 DEBUG:boto:Data: 2022-09-02T05:47:55.951 DEBUG:boto:Headers: {} 2022-09-02T05:47:55.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:55.952 DEBUG:boto:Port: 8000 2022-09-02T05:47:55.952 DEBUG:boto:Params: {} 2022-09-02T05:47:55.953 DEBUG:boto:Token: None 2022-09-02T05:47:55.953 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:55 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:47:55.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80= 2022-09-02T05:47:55.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R3KCMObPY0/4lx0mg4fXa/Fv/80=', 'Content-Length': '0'} 2022-09-02T05:47:55.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf57522b9e89ab9-006311990b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.005 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.006 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.007 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.007 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.008 DEBUG:boto:Method: GET 2022-09-02T05:47:56.008 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.009 DEBUG:boto:Data: 2022-09-02T05:47:56.009 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.010 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.011 DEBUG:boto:Params: {} 2022-09-02T05:47:56.012 DEBUG:boto:Token: None 2022-09-02T05:47:56.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d3d9aefba185b7d-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.020 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.020 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.021 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.022 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.022 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.023 DEBUG:boto:Method: GET 2022-09-02T05:47:56.023 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.024 DEBUG:boto:Data: 2022-09-02T05:47:56.024 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.025 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.025 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.026 DEBUG:boto:Params: {} 2022-09-02T05:47:56.027 DEBUG:boto:Token: None 2022-09-02T05:47:56.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4a7ed88a73d29a-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.037 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.038 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:56.038 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:56.039 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.040 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:56.040 DEBUG:boto:Data: 2022-09-02T05:47:56.041 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.042 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.042 DEBUG:boto:Params: {} 2022-09-02T05:47:56.043 DEBUG:boto:Token: None 2022-09-02T05:47:56.043 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj44 2022-09-02T05:47:56.044 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:weB5PnQ3vkxWs4kray75BhyOBbM= 2022-09-02T05:47:56.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:weB5PnQ3vkxWs4kray75BhyOBbM=', 'Content-Length': '0'} 2022-09-02T05:47:56.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ea3f6dac2c043c-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.052 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:56.052 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:56.053 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.054 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:56.054 DEBUG:boto:Data: 2022-09-02T05:47:56.055 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.056 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.056 DEBUG:boto:Params: {} 2022-09-02T05:47:56.057 DEBUG:boto:Token: None 2022-09-02T05:47:56.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj44 2022-09-02T05:47:56.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:weB5PnQ3vkxWs4kray75BhyOBbM= 2022-09-02T05:47:56.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:weB5PnQ3vkxWs4kray75BhyOBbM=', 'Content-Length': '0'} 2022-09-02T05:47:56.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cc5801de1bd52e-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:47:56.066 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:56.066 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:56.067 DEBUG:boto:Method: GET 2022-09-02T05:47:56.067 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:56.068 DEBUG:boto:Data: 2022-09-02T05:47:56.069 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.070 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.070 DEBUG:boto:Params: {} 2022-09-02T05:47:56.071 DEBUG:boto:Token: None 2022-09-02T05:47:56.072 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj44 2022-09-02T05:47:56.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zaX82+AGt6PNYP1fHULgbFfl8F8= 2022-09-02T05:47:56.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zaX82+AGt6PNYP1fHULgbFfl8F8=', 'Content-Length': '0'} 2022-09-02T05:47:56.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592fee1cceb50d76-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.122 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:47:56.123 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:47:56.124 DEBUG:boto:Method: GET 2022-09-02T05:47:56.124 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:47:56.125 DEBUG:boto:Data: 2022-09-02T05:47:56.126 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.127 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.127 DEBUG:boto:Params: {} 2022-09-02T05:47:56.128 DEBUG:boto:Token: None 2022-09-02T05:47:56.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj44 2022-09-02T05:47:56.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zaX82+AGt6PNYP1fHULgbFfl8F8= 2022-09-02T05:47:56.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zaX82+AGt6PNYP1fHULgbFfl8F8=', 'Content-Length': '0'} 2022-09-02T05:47:56.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d6307c52b9bb5fa-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.136 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.136 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.137 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.137 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.138 DEBUG:boto:Method: GET 2022-09-02T05:47:56.139 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.139 DEBUG:boto:Data: 2022-09-02T05:47:56.140 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.141 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.141 DEBUG:boto:Params: {} 2022-09-02T05:47:56.142 DEBUG:boto:Token: None 2022-09-02T05:47:56.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d070b4786cf13aa0-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.150 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.151 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.151 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.152 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.152 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.153 DEBUG:boto:Method: GET 2022-09-02T05:47:56.153 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:47:56.154 DEBUG:boto:Data: 2022-09-02T05:47:56.155 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.156 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.156 DEBUG:boto:Params: {} 2022-09-02T05:47:56.157 DEBUG:boto:Token: None 2022-09-02T05:47:56.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000bd788a2b73b84c-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.205 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:56.207 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.208 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.208 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.209 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.209 DEBUG:boto:Method: GET 2022-09-02T05:47:56.210 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.210 DEBUG:boto:Data: 2022-09-02T05:47:56.211 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.212 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.212 DEBUG:boto:Params: {} 2022-09-02T05:47:56.213 DEBUG:boto:Token: None 2022-09-02T05:47:56.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644= 2022-09-02T05:47:56.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644=', 'Content-Length': '0'} 2022-09-02T05:47:56.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c7e3f6bd6fa67e-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.218 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.219 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.220 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.220 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.221 DEBUG:boto:Method: GET 2022-09-02T05:47:56.221 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.222 DEBUG:boto:Data: 2022-09-02T05:47:56.223 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.223 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.224 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.224 DEBUG:boto:Params: {} 2022-09-02T05:47:56.225 DEBUG:boto:Token: None 2022-09-02T05:47:56.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644= 2022-09-02T05:47:56.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644=', 'Content-Length': '0'} 2022-09-02T05:47:56.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3dda05f4724b453-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.233 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.233 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.234 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.234 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.235 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.235 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.236 DEBUG:boto:Data: 2022-09-02T05:47:56.237 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.238 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.238 DEBUG:boto:Params: {} 2022-09-02T05:47:56.239 DEBUG:boto:Token: None 2022-09-02T05:47:56.239 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ULbdHpHw8+y19pmmfu5Aqbl0ZTM= 2022-09-02T05:47:56.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ULbdHpHw8+y19pmmfu5Aqbl0ZTM=', 'Content-Length': '0'} 2022-09-02T05:47:56.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d873f29ee6aeed79-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.244 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.244 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.245 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.245 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.246 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.247 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.247 DEBUG:boto:Data: 2022-09-02T05:47:56.248 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.249 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.249 DEBUG:boto:Params: {} 2022-09-02T05:47:56.250 DEBUG:boto:Token: None 2022-09-02T05:47:56.250 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ULbdHpHw8+y19pmmfu5Aqbl0ZTM= 2022-09-02T05:47:56.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ULbdHpHw8+y19pmmfu5Aqbl0ZTM=', 'Content-Length': '0'} 2022-09-02T05:47:56.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e0a38c10a14a2d-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:56.255 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.256 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.256 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.257 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.257 DEBUG:boto:Method: GET 2022-09-02T05:47:56.258 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.259 DEBUG:boto:Data: 2022-09-02T05:47:56.259 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.260 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.261 DEBUG:boto:Params: {} 2022-09-02T05:47:56.261 DEBUG:boto:Token: None 2022-09-02T05:47:56.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644= 2022-09-02T05:47:56.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644=', 'Content-Length': '0'} 2022-09-02T05:47:56.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016bd794807d53fc9-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.314 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.315 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.316 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.316 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.317 DEBUG:boto:Method: GET 2022-09-02T05:47:56.318 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.318 DEBUG:boto:Data: 2022-09-02T05:47:56.319 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.320 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.320 DEBUG:boto:Params: {} 2022-09-02T05:47:56.321 DEBUG:boto:Token: None 2022-09-02T05:47:56.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:47:56.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644= 2022-09-02T05:47:56.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ax5vg2KlNpZDpHh1d/aQXEsb644=', 'Content-Length': '0'} 2022-09-02T05:47:56.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076636cfe99857ebe-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.331 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.331 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.332 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.333 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.333 DEBUG:boto:Method: GET 2022-09-02T05:47:56.334 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.334 DEBUG:boto:Data: 2022-09-02T05:47:56.335 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.336 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.336 DEBUG:boto:Params: {} 2022-09-02T05:47:56.337 DEBUG:boto:Token: None 2022-09-02T05:47:56.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b037cb7f7d48222-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.344 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.345 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.345 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.346 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.346 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.347 DEBUG:boto:Method: GET 2022-09-02T05:47:56.348 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.348 DEBUG:boto:Data: 2022-09-02T05:47:56.349 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.350 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.351 DEBUG:boto:Params: {} 2022-09-02T05:47:56.352 DEBUG:boto:Token: None 2022-09-02T05:47:56.352 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c46aff61880662-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.409 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.412 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.413 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.414 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.414 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:56.415 DEBUG:boto:Data: 2022-09-02T05:47:56.415 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.416 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.417 DEBUG:boto:Params: {} 2022-09-02T05:47:56.418 DEBUG:boto:Token: None 2022-09-02T05:47:56.418 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51 2022-09-02T05:47:56.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bk5hqeP1Pfo7UsIILR7YaC6d2Zc= 2022-09-02T05:47:56.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bk5hqeP1Pfo7UsIILR7YaC6d2Zc=', 'Content-Length': '0'} 2022-09-02T05:47:56.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3401f01006a3486-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.423 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.423 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.424 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.424 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:56.425 DEBUG:boto:Data: 2022-09-02T05:47:56.426 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.427 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.428 DEBUG:boto:Params: {} 2022-09-02T05:47:56.428 DEBUG:boto:Token: None 2022-09-02T05:47:56.429 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51 2022-09-02T05:47:56.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bk5hqeP1Pfo7UsIILR7YaC6d2Zc= 2022-09-02T05:47:56.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bk5hqeP1Pfo7UsIILR7YaC6d2Zc=', 'Content-Length': '0'} 2022-09-02T05:47:56.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000024e9727e2112bee-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:47:56.436 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.437 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.438 DEBUG:boto:Method: GET 2022-09-02T05:47:56.439 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:56.441 DEBUG:boto:Data: 2022-09-02T05:47:56.441 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.442 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.444 DEBUG:boto:Params: {} 2022-09-02T05:47:56.446 DEBUG:boto:Token: None 2022-09-02T05:47:56.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51 2022-09-02T05:47:56.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2K6PD+rhpwFYE1QVZXdRzNcBuuc= 2022-09-02T05:47:56.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2K6PD+rhpwFYE1QVZXdRzNcBuuc=', 'Content-Length': '0'} 2022-09-02T05:47:56.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d79a0819a4af1328-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.455 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:47:56.457 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:47:56.459 DEBUG:boto:Method: GET 2022-09-02T05:47:56.460 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:47:56.461 DEBUG:boto:Data: 2022-09-02T05:47:56.461 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.462 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.463 DEBUG:boto:Params: {} 2022-09-02T05:47:56.464 DEBUG:boto:Token: None 2022-09-02T05:47:56.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj51 2022-09-02T05:47:56.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2K6PD+rhpwFYE1QVZXdRzNcBuuc= 2022-09-02T05:47:56.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2K6PD+rhpwFYE1QVZXdRzNcBuuc=', 'Content-Length': '0'} 2022-09-02T05:47:56.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c94b11af99cb3d4-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.470 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.471 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.472 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.472 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.473 DEBUG:boto:Method: GET 2022-09-02T05:47:56.473 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.474 DEBUG:boto:Data: 2022-09-02T05:47:56.475 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.476 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.477 DEBUG:boto:Params: {} 2022-09-02T05:47:56.478 DEBUG:boto:Token: None 2022-09-02T05:47:56.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ef90272e03dfcd9-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.484 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.485 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.486 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.487 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.487 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.488 DEBUG:boto:Method: GET 2022-09-02T05:47:56.489 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:47:56.490 DEBUG:boto:Data: 2022-09-02T05:47:56.491 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.493 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.493 DEBUG:boto:Params: {} 2022-09-02T05:47:56.494 DEBUG:boto:Token: None 2022-09-02T05:47:56.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d4db0e65a00659-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.546 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:56.549 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.550 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.551 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.552 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.552 DEBUG:boto:Method: GET 2022-09-02T05:47:56.553 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.553 DEBUG:boto:Data: 2022-09-02T05:47:56.554 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.555 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.556 DEBUG:boto:Params: {} 2022-09-02T05:47:56.556 DEBUG:boto:Token: None 2022-09-02T05:47:56.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c= 2022-09-02T05:47:56.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c=', 'Content-Length': '0'} 2022-09-02T05:47:56.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af772eacb568ac6-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.562 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.562 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.563 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.564 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.565 DEBUG:boto:Method: GET 2022-09-02T05:47:56.565 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.566 DEBUG:boto:Data: 2022-09-02T05:47:56.566 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.568 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.568 DEBUG:boto:Params: {} 2022-09-02T05:47:56.569 DEBUG:boto:Token: None 2022-09-02T05:47:56.570 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c= 2022-09-02T05:47:56.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c=', 'Content-Length': '0'} 2022-09-02T05:47:56.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf234033ea0e8ffd-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.575 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.576 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.577 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.578 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.578 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.579 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.580 DEBUG:boto:Data: 2022-09-02T05:47:56.581 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.583 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.584 DEBUG:boto:Params: {} 2022-09-02T05:47:56.585 DEBUG:boto:Token: None 2022-09-02T05:47:56.586 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OvDAc7YnQ9TIqr1zwL+xFCzv7Vw= 2022-09-02T05:47:56.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OvDAc7YnQ9TIqr1zwL+xFCzv7Vw=', 'Content-Length': '0'} 2022-09-02T05:47:56.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b241dc133aaa5c2c-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.592 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.592 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.593 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.593 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.594 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.595 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.596 DEBUG:boto:Data: 2022-09-02T05:47:56.597 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.598 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.599 DEBUG:boto:Params: {} 2022-09-02T05:47:56.600 DEBUG:boto:Token: None 2022-09-02T05:47:56.601 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OvDAc7YnQ9TIqr1zwL+xFCzv7Vw= 2022-09-02T05:47:56.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OvDAc7YnQ9TIqr1zwL+xFCzv7Vw=', 'Content-Length': '0'} 2022-09-02T05:47:56.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdde9b14ebb7f6ba-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:56.609 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.609 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.610 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.611 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.611 DEBUG:boto:Method: GET 2022-09-02T05:47:56.612 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.612 DEBUG:boto:Data: 2022-09-02T05:47:56.613 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.614 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.614 DEBUG:boto:Params: {} 2022-09-02T05:47:56.615 DEBUG:boto:Token: None 2022-09-02T05:47:56.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c= 2022-09-02T05:47:56.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c=', 'Content-Length': '0'} 2022-09-02T05:47:56.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e95c4d0fb033f07-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.622 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.623 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.624 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.625 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.626 DEBUG:boto:Method: GET 2022-09-02T05:47:56.627 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.628 DEBUG:boto:Data: 2022-09-02T05:47:56.628 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.629 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.629 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.630 DEBUG:boto:Params: {} 2022-09-02T05:47:56.630 DEBUG:boto:Token: None 2022-09-02T05:47:56.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:47:56.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c= 2022-09-02T05:47:56.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:flapiZ9rbSGV6UNeZuOndGLlt5c=', 'Content-Length': '0'} 2022-09-02T05:47:56.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006088d659ed3f6da1-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.636 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.636 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.637 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.637 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.638 DEBUG:boto:Method: GET 2022-09-02T05:47:56.638 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.639 DEBUG:boto:Data: 2022-09-02T05:47:56.639 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.640 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.641 DEBUG:boto:Params: {} 2022-09-02T05:47:56.641 DEBUG:boto:Token: None 2022-09-02T05:47:56.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8cada1778408027-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.692 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.693 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.694 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.695 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.696 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.697 DEBUG:boto:Method: GET 2022-09-02T05:47:56.697 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.698 DEBUG:boto:Data: 2022-09-02T05:47:56.699 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.700 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.701 DEBUG:boto:Params: {} 2022-09-02T05:47:56.702 DEBUG:boto:Token: None 2022-09-02T05:47:56.702 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072da3ba9df9efb31-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.708 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.709 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.709 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.710 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.711 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:56.711 DEBUG:boto:Data: 2022-09-02T05:47:56.712 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.713 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.714 DEBUG:boto:Params: {} 2022-09-02T05:47:56.714 DEBUG:boto:Token: None 2022-09-02T05:47:56.715 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55 2022-09-02T05:47:56.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pq+w3aiI8JmTT9ZQRcV2Ww+y6Rs= 2022-09-02T05:47:56.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pq+w3aiI8JmTT9ZQRcV2Ww+y6Rs=', 'Content-Length': '0'} 2022-09-02T05:47:56.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710ef8a6d793117c-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.720 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.720 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.721 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.721 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:56.722 DEBUG:boto:Data: 2022-09-02T05:47:56.722 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.723 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.724 DEBUG:boto:Params: {} 2022-09-02T05:47:56.724 DEBUG:boto:Token: None 2022-09-02T05:47:56.725 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55 2022-09-02T05:47:56.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pq+w3aiI8JmTT9ZQRcV2Ww+y6Rs= 2022-09-02T05:47:56.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pq+w3aiI8JmTT9ZQRcV2Ww+y6Rs=', 'Content-Length': '0'} 2022-09-02T05:47:56.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009131e0c17e705ce8-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:47:56.730 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.730 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.731 DEBUG:boto:Method: GET 2022-09-02T05:47:56.731 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:56.732 DEBUG:boto:Data: 2022-09-02T05:47:56.733 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.734 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.735 DEBUG:boto:Params: {} 2022-09-02T05:47:56.735 DEBUG:boto:Token: None 2022-09-02T05:47:56.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55 2022-09-02T05:47:56.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1sXBy8gnFMl3EWBb4+snzSSS0i8= 2022-09-02T05:47:56.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1sXBy8gnFMl3EWBb4+snzSSS0i8=', 'Content-Length': '0'} 2022-09-02T05:47:56.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a767683650c3c8-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.741 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:47:56.742 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:47:56.743 DEBUG:boto:Method: GET 2022-09-02T05:47:56.744 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:47:56.745 DEBUG:boto:Data: 2022-09-02T05:47:56.746 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.748 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.749 DEBUG:boto:Params: {} 2022-09-02T05:47:56.751 DEBUG:boto:Token: None 2022-09-02T05:47:56.752 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj55 2022-09-02T05:47:56.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1sXBy8gnFMl3EWBb4+snzSSS0i8= 2022-09-02T05:47:56.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1sXBy8gnFMl3EWBb4+snzSSS0i8=', 'Content-Length': '0'} 2022-09-02T05:47:56.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008761dd71369d2c12-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.798 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.799 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.800 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.801 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.801 DEBUG:boto:Method: GET 2022-09-02T05:47:56.802 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.803 DEBUG:boto:Data: 2022-09-02T05:47:56.804 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.805 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.806 DEBUG:boto:Params: {} 2022-09-02T05:47:56.806 DEBUG:boto:Token: None 2022-09-02T05:47:56.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd15093bebd12e7b-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.812 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.813 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.814 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.815 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.815 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.816 DEBUG:boto:Method: GET 2022-09-02T05:47:56.816 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:47:56.817 DEBUG:boto:Data: 2022-09-02T05:47:56.817 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.818 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.818 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.819 DEBUG:boto:Params: {} 2022-09-02T05:47:56.819 DEBUG:boto:Token: None 2022-09-02T05:47:56.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f2004874194c25f-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.825 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:56.827 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.828 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.829 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.830 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.831 DEBUG:boto:Method: GET 2022-09-02T05:47:56.832 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.833 DEBUG:boto:Data: 2022-09-02T05:47:56.834 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.836 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.836 DEBUG:boto:Params: {} 2022-09-02T05:47:56.837 DEBUG:boto:Token: None 2022-09-02T05:47:56.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU= 2022-09-02T05:47:56.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU=', 'Content-Length': '0'} 2022-09-02T05:47:56.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a711b9365fd29d-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.882 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.883 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.883 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.884 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.884 DEBUG:boto:Method: GET 2022-09-02T05:47:56.885 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.886 DEBUG:boto:Data: 2022-09-02T05:47:56.886 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.887 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.888 DEBUG:boto:Params: {} 2022-09-02T05:47:56.888 DEBUG:boto:Token: None 2022-09-02T05:47:56.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU= 2022-09-02T05:47:56.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU=', 'Content-Length': '0'} 2022-09-02T05:47:56.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef857d220ba259a6-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.893 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.894 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.894 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.895 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.895 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.896 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.896 DEBUG:boto:Data: 2022-09-02T05:47:56.897 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.898 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.898 DEBUG:boto:Params: {} 2022-09-02T05:47:56.899 DEBUG:boto:Token: None 2022-09-02T05:47:56.900 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G5o630BjWGBjDsLjjzejrQh4grc= 2022-09-02T05:47:56.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G5o630BjWGBjDsLjjzejrQh4grc=', 'Content-Length': '0'} 2022-09-02T05:47:56.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d7533e6fd719c3b-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.904 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.904 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.905 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.905 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.906 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.906 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.907 DEBUG:boto:Data: 2022-09-02T05:47:56.907 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.908 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.909 DEBUG:boto:Params: {} 2022-09-02T05:47:56.909 DEBUG:boto:Token: None 2022-09-02T05:47:56.910 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G5o630BjWGBjDsLjjzejrQh4grc= 2022-09-02T05:47:56.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G5o630BjWGBjDsLjjzejrQh4grc=', 'Content-Length': '0'} 2022-09-02T05:47:56.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c044a2a5b3150a0b-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:56.915 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.915 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.916 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.916 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.917 DEBUG:boto:Method: GET 2022-09-02T05:47:56.918 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.918 DEBUG:boto:Data: 2022-09-02T05:47:56.919 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.920 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.920 DEBUG:boto:Params: {} 2022-09-02T05:47:56.921 DEBUG:boto:Token: None 2022-09-02T05:47:56.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU= 2022-09-02T05:47:56.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU=', 'Content-Length': '0'} 2022-09-02T05:47:56.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009456d637d0baf7e3-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.926 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.927 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.927 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.928 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.928 DEBUG:boto:Method: GET 2022-09-02T05:47:56.929 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.929 DEBUG:boto:Data: 2022-09-02T05:47:56.930 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.931 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.931 DEBUG:boto:Params: {} 2022-09-02T05:47:56.932 DEBUG:boto:Token: None 2022-09-02T05:47:56.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:47:56.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU= 2022-09-02T05:47:56.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bSfPocdkMhmX1sCWLQvKsknD4GU=', 'Content-Length': '0'} 2022-09-02T05:47:56.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004593e341ca0b8365-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.937 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.938 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.938 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:56.939 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:56.939 DEBUG:boto:Method: GET 2022-09-02T05:47:56.940 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:56.941 DEBUG:boto:Data: 2022-09-02T05:47:56.941 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.942 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.943 DEBUG:boto:Params: {} 2022-09-02T05:47:56.944 DEBUG:boto:Token: None 2022-09-02T05:47:56.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c96f844a4b088ed-006311990c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.950 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.950 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:56.951 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:56.951 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:56.952 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:56.953 DEBUG:boto:Method: GET 2022-09-02T05:47:56.953 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:56.955 DEBUG:boto:Data: 2022-09-02T05:47:56.955 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.958 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.959 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.959 DEBUG:boto:Params: {} 2022-09-02T05:47:56.960 DEBUG:boto:Token: None 2022-09-02T05:47:56.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/ 2022-09-02T05:47:56.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY= 2022-09-02T05:47:56.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6cFfGwH9YqEjLjDAwSC/1GC7lmY=', 'Content-Length': '0'} 2022-09-02T05:47:56.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14671774a6fb612-006311990c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.968 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:56.969 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.970 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.970 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.971 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:56.971 DEBUG:boto:Data: 2022-09-02T05:47:56.972 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.973 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.973 DEBUG:boto:Params: {} 2022-09-02T05:47:56.974 DEBUG:boto:Token: None 2022-09-02T05:47:56.975 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68 2022-09-02T05:47:56.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NQzulAL4MnYZLThnU9y/CwvLkHM= 2022-09-02T05:47:56.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NQzulAL4MnYZLThnU9y/CwvLkHM=', 'Content-Length': '0'} 2022-09-02T05:47:56.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c828d850c2831691-006311990c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.979 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.980 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.980 DEBUG:boto:Method: HEAD 2022-09-02T05:47:56.981 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:56.981 DEBUG:boto:Data: 2022-09-02T05:47:56.982 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.982 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:56.983 DEBUG:boto:Port: 8000 2022-09-02T05:47:56.983 DEBUG:boto:Params: {} 2022-09-02T05:47:56.984 DEBUG:boto:Token: None 2022-09-02T05:47:56.984 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68 2022-09-02T05:47:56.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NQzulAL4MnYZLThnU9y/CwvLkHM= 2022-09-02T05:47:56.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NQzulAL4MnYZLThnU9y/CwvLkHM=', 'Content-Length': '0'} 2022-09-02T05:47:56.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000716f35e5b2bbc16f-006311990c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:56.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:47:56.990 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:56.991 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:56.992 DEBUG:boto:Method: GET 2022-09-02T05:47:56.992 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:56.993 DEBUG:boto:Data: 2022-09-02T05:47:56.994 DEBUG:boto:Headers: {} 2022-09-02T05:47:56.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:56.995 DEBUG:boto:Port: 8001 2022-09-02T05:47:56.996 DEBUG:boto:Params: {} 2022-09-02T05:47:56.996 DEBUG:boto:Token: None 2022-09-02T05:47:56.997 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:56 GMT /xljbkf-74/obj68 2022-09-02T05:47:56.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WZrAliFWS+nAc3RvRVEjaHacJNI= 2022-09-02T05:47:56.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WZrAliFWS+nAc3RvRVEjaHacJNI=', 'Content-Length': '0'} 2022-09-02T05:47:57.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098b71a6d57761457-006311990d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.046 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:47:57.047 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:47:57.048 DEBUG:boto:Method: GET 2022-09-02T05:47:57.050 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:47:57.051 DEBUG:boto:Data: 2022-09-02T05:47:57.052 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.054 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.055 DEBUG:boto:Params: {} 2022-09-02T05:47:57.056 DEBUG:boto:Token: None 2022-09-02T05:47:57.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj68 2022-09-02T05:47:57.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H9/4RZr8lVf2wgF8iP5NII3edws= 2022-09-02T05:47:57.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H9/4RZr8lVf2wgF8iP5NII3edws=', 'Content-Length': '0'} 2022-09-02T05:47:57.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae728582876f1f59-006311990d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.062 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.063 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.063 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:57.064 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:57.065 DEBUG:boto:Method: GET 2022-09-02T05:47:57.065 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:57.066 DEBUG:boto:Data: 2022-09-02T05:47:57.067 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.068 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.069 DEBUG:boto:Params: {} 2022-09-02T05:47:57.069 DEBUG:boto:Token: None 2022-09-02T05:47:57.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs= 2022-09-02T05:47:57.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs=', 'Content-Length': '0'} 2022-09-02T05:47:57.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef3d77349bc81a35-006311990d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.076 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:57.077 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.078 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.079 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:57.079 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:47:57.080 DEBUG:boto:Method: GET 2022-09-02T05:47:57.080 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:47:57.081 DEBUG:boto:Data: 2022-09-02T05:47:57.082 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.083 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.084 DEBUG:boto:Params: {} 2022-09-02T05:47:57.085 DEBUG:boto:Token: None 2022-09-02T05:47:57.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs= 2022-09-02T05:47:57.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs=', 'Content-Length': '0'} 2022-09-02T05:47:57.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000894393b842223386-006311990d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.091 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:57.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:57.093 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.094 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.094 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.095 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.096 DEBUG:boto:Method: GET 2022-09-02T05:47:57.096 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.097 DEBUG:boto:Data: 2022-09-02T05:47:57.098 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.099 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.100 DEBUG:boto:Params: {} 2022-09-02T05:47:57.101 DEBUG:boto:Token: None 2022-09-02T05:47:57.101 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM= 2022-09-02T05:47:57.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM=', 'Content-Length': '0'} 2022-09-02T05:47:57.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5695e8b5bda84b0-006311990d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.108 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.108 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.109 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.110 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.111 DEBUG:boto:Method: GET 2022-09-02T05:47:57.111 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.112 DEBUG:boto:Data: 2022-09-02T05:47:57.113 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.115 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.115 DEBUG:boto:Params: {} 2022-09-02T05:47:57.116 DEBUG:boto:Token: None 2022-09-02T05:47:57.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM= 2022-09-02T05:47:57.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM=', 'Content-Length': '0'} 2022-09-02T05:47:57.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee9cb7cd2b7938e-006311990d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.124 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.125 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.126 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.127 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.128 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.130 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.131 DEBUG:boto:Data: 2022-09-02T05:47:57.132 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.135 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.136 DEBUG:boto:Params: {} 2022-09-02T05:47:57.137 DEBUG:boto:Token: None 2022-09-02T05:47:57.137 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.138 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KF0FEm4GnRv18YwLhrUnhA70t9I= 2022-09-02T05:47:57.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KF0FEm4GnRv18YwLhrUnhA70t9I=', 'Content-Length': '0'} 2022-09-02T05:47:57.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f3e7cddc21e35d-006311990d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.143 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.143 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.144 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.145 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.146 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.146 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.147 DEBUG:boto:Data: 2022-09-02T05:47:57.148 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.149 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.150 DEBUG:boto:Params: {} 2022-09-02T05:47:57.151 DEBUG:boto:Token: None 2022-09-02T05:47:57.152 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KF0FEm4GnRv18YwLhrUnhA70t9I= 2022-09-02T05:47:57.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KF0FEm4GnRv18YwLhrUnhA70t9I=', 'Content-Length': '0'} 2022-09-02T05:47:57.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f268b16ea512a039-006311990d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:57.160 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.161 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.162 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.162 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.163 DEBUG:boto:Method: GET 2022-09-02T05:47:57.165 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.165 DEBUG:boto:Data: 2022-09-02T05:47:57.166 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.168 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.169 DEBUG:boto:Params: {} 2022-09-02T05:47:57.170 DEBUG:boto:Token: None 2022-09-02T05:47:57.171 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.172 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM= 2022-09-02T05:47:57.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM=', 'Content-Length': '0'} 2022-09-02T05:47:57.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f636df4343a503-006311990d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.222 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.223 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.224 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.225 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.226 DEBUG:boto:Method: GET 2022-09-02T05:47:57.227 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.228 DEBUG:boto:Data: 2022-09-02T05:47:57.230 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.233 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.235 DEBUG:boto:Params: {} 2022-09-02T05:47:57.237 DEBUG:boto:Token: None 2022-09-02T05:47:57.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:47:57.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM= 2022-09-02T05:47:57.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xLS2JF/HVCnKC6Corj2tw+dusDM=', 'Content-Length': '0'} 2022-09-02T05:47:57.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e515e74e7199c7-006311990d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.245 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.245 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.246 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.247 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.248 DEBUG:boto:Method: GET 2022-09-02T05:47:57.249 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.249 DEBUG:boto:Data: 2022-09-02T05:47:57.250 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.251 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.252 DEBUG:boto:Params: {} 2022-09-02T05:47:57.252 DEBUG:boto:Token: None 2022-09-02T05:47:57.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs= 2022-09-02T05:47:57.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs=', 'Content-Length': '0'} 2022-09-02T05:47:57.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce79a5f1ef23fe0-006311990d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.258 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:57.259 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.259 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.260 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.260 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.261 DEBUG:boto:Method: GET 2022-09-02T05:47:57.261 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.262 DEBUG:boto:Data: 2022-09-02T05:47:57.263 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.263 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.264 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.264 DEBUG:boto:Params: {} 2022-09-02T05:47:57.265 DEBUG:boto:Token: None 2022-09-02T05:47:57.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs= 2022-09-02T05:47:57.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs=', 'Content-Length': '0'} 2022-09-02T05:47:57.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05463f96790f8cd-006311990d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.270 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:57.271 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.272 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.272 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.273 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:57.273 DEBUG:boto:Data: 2022-09-02T05:47:57.274 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.275 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.275 DEBUG:boto:Params: {} 2022-09-02T05:47:57.276 DEBUG:boto:Token: None 2022-09-02T05:47:57.277 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85 2022-09-02T05:47:57.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mXW7LDAZxledMhUm5m+sWQTlJYk= 2022-09-02T05:47:57.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mXW7LDAZxledMhUm5m+sWQTlJYk=', 'Content-Length': '0'} 2022-09-02T05:47:57.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044479b18e111ab36-006311990d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.281 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.282 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.282 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.283 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:57.283 DEBUG:boto:Data: 2022-09-02T05:47:57.284 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.285 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.285 DEBUG:boto:Params: {} 2022-09-02T05:47:57.286 DEBUG:boto:Token: None 2022-09-02T05:47:57.286 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85 2022-09-02T05:47:57.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mXW7LDAZxledMhUm5m+sWQTlJYk= 2022-09-02T05:47:57.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mXW7LDAZxledMhUm5m+sWQTlJYk=', 'Content-Length': '0'} 2022-09-02T05:47:57.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028706ec53930875-006311990d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:47:57.291 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.292 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.293 DEBUG:boto:Method: GET 2022-09-02T05:47:57.293 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:57.294 DEBUG:boto:Data: 2022-09-02T05:47:57.294 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.296 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.296 DEBUG:boto:Params: {} 2022-09-02T05:47:57.297 DEBUG:boto:Token: None 2022-09-02T05:47:57.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85 2022-09-02T05:47:57.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DZo8yYSXaKv9KloW+rp7te/TDnY= 2022-09-02T05:47:57.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DZo8yYSXaKv9KloW+rp7te/TDnY=', 'Content-Length': '0'} 2022-09-02T05:47:57.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091dd68f776b95f34-006311990d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.301 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:47:57.302 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:47:57.303 DEBUG:boto:Method: GET 2022-09-02T05:47:57.304 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:47:57.304 DEBUG:boto:Data: 2022-09-02T05:47:57.305 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.305 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.306 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.306 DEBUG:boto:Params: {} 2022-09-02T05:47:57.307 DEBUG:boto:Token: None 2022-09-02T05:47:57.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj85 2022-09-02T05:47:57.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DZo8yYSXaKv9KloW+rp7te/TDnY= 2022-09-02T05:47:57.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DZo8yYSXaKv9KloW+rp7te/TDnY=', 'Content-Length': '0'} 2022-09-02T05:47:57.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7bd6241a6406b3-006311990d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.312 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.313 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.313 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.314 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.314 DEBUG:boto:Method: GET 2022-09-02T05:47:57.315 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.315 DEBUG:boto:Data: 2022-09-02T05:47:57.316 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:47:57.317 DEBUG:boto:Port: 8001 2022-09-02T05:47:57.317 DEBUG:boto:Params: {} 2022-09-02T05:47:57.318 DEBUG:boto:Token: None 2022-09-02T05:47:57.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs= 2022-09-02T05:47:57.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs=', 'Content-Length': '0'} 2022-09-02T05:47:57.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000799a4561181a7603-006311990d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.323 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:57.324 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.325 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.325 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.326 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.327 DEBUG:boto:Method: GET 2022-09-02T05:47:57.328 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:47:57.328 DEBUG:boto:Data: 2022-09-02T05:47:57.329 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:47:57.330 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.331 DEBUG:boto:Params: {} 2022-09-02T05:47:57.332 DEBUG:boto:Token: None 2022-09-02T05:47:57.332 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs= 2022-09-02T05:47:57.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KaXTlBOs5/aFEueBDSQ3tsaKnqs=', 'Content-Length': '0'} 2022-09-02T05:47:57.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dd1651d65627c4-006311990d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.337 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:47:57.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:47:57.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:47:57.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:47:57.340 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:47:57.400 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:57.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:57.456 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:47:57.457 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-74 2022-09-02T05:47:57.463 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2022-09-02T05:47:57.541 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.542 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.543 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.543 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:57.544 DEBUG:boto:Data: 2022-09-02T05:47:57.544 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.545 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.546 DEBUG:boto:Params: {} 2022-09-02T05:47:57.547 DEBUG:boto:Token: None 2022-09-02T05:47:57.548 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q= 2022-09-02T05:47:57.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q=', 'Content-Length': '0'} 2022-09-02T05:47:57.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fd03306dc4d0ff5-006311990d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.553 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:47:57.554 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:47:57.554 DEBUG:boto:Method: PUT 2022-09-02T05:47:57.555 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:47:57.555 DEBUG:boto:Data: 2022-09-02T05:47:57.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:47:57.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.557 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.558 DEBUG:boto:Params: {} 2022-09-02T05:47:57.559 DEBUG:boto:Token: None 2022-09-02T05:47:57.559 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj19 2022-09-02T05:47:57.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FkWwA5LQL5o/U97oLGflqMIGgjo= 2022-09-02T05:47:57.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FkWwA5LQL5o/U97oLGflqMIGgjo='} 2022-09-02T05:47:57.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007842d9d378cbf53c-006311990d-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.571 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.572 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.572 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.573 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:57.573 DEBUG:boto:Data: 2022-09-02T05:47:57.574 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.575 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.576 DEBUG:boto:Params: {} 2022-09-02T05:47:57.576 DEBUG:boto:Token: None 2022-09-02T05:47:57.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q= 2022-09-02T05:47:57.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q=', 'Content-Length': '0'} 2022-09-02T05:47:57.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000088e3c8d4219f22e-006311990d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.582 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:47:57.583 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:47:57.583 DEBUG:boto:Method: PUT 2022-09-02T05:47:57.584 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:47:57.584 DEBUG:boto:Data: 2022-09-02T05:47:57.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:47:57.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.586 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.586 DEBUG:boto:Params: {} 2022-09-02T05:47:57.587 DEBUG:boto:Token: None 2022-09-02T05:47:57.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj38 2022-09-02T05:47:57.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eXPZK19tVnOYvZmp2WDwleerBnQ= 2022-09-02T05:47:57.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eXPZK19tVnOYvZmp2WDwleerBnQ='} 2022-09-02T05:47:57.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a82852f53e2df9b-006311990d-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.594 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.595 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.596 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.597 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:57.598 DEBUG:boto:Data: 2022-09-02T05:47:57.599 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.601 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.602 DEBUG:boto:Params: {} 2022-09-02T05:47:57.602 DEBUG:boto:Token: None 2022-09-02T05:47:57.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q= 2022-09-02T05:47:57.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q=', 'Content-Length': '0'} 2022-09-02T05:47:57.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022968f640ae645ae-006311990d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.608 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:47:57.609 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:47:57.609 DEBUG:boto:Method: PUT 2022-09-02T05:47:57.610 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:47:57.610 DEBUG:boto:Data: 2022-09-02T05:47:57.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:47:57.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.612 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.613 DEBUG:boto:Params: {} 2022-09-02T05:47:57.614 DEBUG:boto:Token: None 2022-09-02T05:47:57.615 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj57 2022-09-02T05:47:57.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KM5GGAVrynQkvaEjML3XSiqqzio= 2022-09-02T05:47:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KM5GGAVrynQkvaEjML3XSiqqzio='} 2022-09-02T05:47:57.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009921ecbe1ff6532f-006311990d-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.663 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.664 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.664 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.665 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:57.665 DEBUG:boto:Data: 2022-09-02T05:47:57.666 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.667 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.668 DEBUG:boto:Params: {} 2022-09-02T05:47:57.668 DEBUG:boto:Token: None 2022-09-02T05:47:57.669 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q= 2022-09-02T05:47:57.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q=', 'Content-Length': '0'} 2022-09-02T05:47:57.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b92c8f8ed45a7777-006311990d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.674 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:47:57.675 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:47:57.675 DEBUG:boto:Method: PUT 2022-09-02T05:47:57.676 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:47:57.677 DEBUG:boto:Data: 2022-09-02T05:47:57.677 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:47:57.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.678 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.679 DEBUG:boto:Params: {} 2022-09-02T05:47:57.680 DEBUG:boto:Token: None 2022-09-02T05:47:57.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj76 2022-09-02T05:47:57.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8FJ15ZjT8NlngbhvvxIy20W3E3s= 2022-09-02T05:47:57.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8FJ15ZjT8NlngbhvvxIy20W3E3s='} 2022-09-02T05:47:57.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2a3316945785b7c-006311990d-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.690 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:47:57.691 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:47:57.691 DEBUG:boto:Method: HEAD 2022-09-02T05:47:57.692 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:47:57.693 DEBUG:boto:Data: 2022-09-02T05:47:57.694 DEBUG:boto:Headers: {} 2022-09-02T05:47:57.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.696 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.697 DEBUG:boto:Params: {} 2022-09-02T05:47:57.697 DEBUG:boto:Token: None 2022-09-02T05:47:57.698 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/ 2022-09-02T05:47:57.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q= 2022-09-02T05:47:57.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AilBJLYXi5mIBfOfgsEhmSmIr0Q=', 'Content-Length': '0'} 2022-09-02T05:47:57.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091634df7d79cd25c-006311990d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.704 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:47:57.705 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:47:57.705 DEBUG:boto:Method: PUT 2022-09-02T05:47:57.706 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:47:57.706 DEBUG:boto:Data: 2022-09-02T05:47:57.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:47:57.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:47:57.708 DEBUG:boto:Port: 8000 2022-09-02T05:47:57.709 DEBUG:boto:Params: {} 2022-09-02T05:47:57.709 DEBUG:boto:Token: None 2022-09-02T05:47:57.710 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:47:57 GMT /xljbkf-74/obj95 2022-09-02T05:47:57.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3x4mHia4IxGRndujJeBJbj68CB0= 2022-09-02T05:47:57.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:47:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3x4mHia4IxGRndujJeBJbj68CB0='} 2022-09-02T05:47:57.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c6d3b76fd0370fb-006311990d-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:47:57.755 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:47:57.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:57.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:47:57.757 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:47:57.821 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:47:57.822 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:47:57.877 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:47:57.876+0000 7f011c839600 1 waiting to reach latest gen 2, on 1.. 2022-09-02T05:48:02.878 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:02.877+0000 7f011c839600 1 waiting to reach latest gen 2, on 1.. 2022-09-02T05:48:07.880 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:07.880+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:07.881 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-02T05:48:07.881 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:12.882 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:12.882+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:12.883 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-02T05:48:12.883 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:17.884 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:17.883+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:17.884 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-02T05:48:17.884 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:22.886 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:22.885+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:22.886 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2022-09-02T05:48:22.886 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:27.887 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:27.886+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:27.888 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:27.889 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:32.889 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:32.888+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:32.889 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:32.890 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:37.891 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:37.890+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:37.891 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:37.891 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:42.893 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:42.892+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:42.893 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:42.894 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:47.894 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:47.894+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:47.895 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:47.895 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:52.896 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:52.895+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:52.897 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:52.897 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:48:57.898 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:48:57.897+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:48:57.899 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:48:57.899 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:02.899 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:02.898+0000 7f011c839600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:02.900 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:02.900 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:07.900+0000 7f011c839600 1 bucket sync caught up with source: 2022-09-02T05:49:07.902 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:07.903 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:07.903 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:07.900+0000 7f011c839600 0 bucket checkpoint complete 2022-09-02T05:49:07.923 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:49:07.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:49:07.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:49:07.926 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:49:07.989 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:49:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:49:08.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:49:08.033+0000 7f704e448600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:08.034 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:08.035 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:13.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:49:13.035+0000 7f704e448600 1 bucket sync caught up with source: 2022-09-02T05:49:13.037 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:13.038 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:13.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:49:13.035+0000 7f704e448600 0 bucket checkpoint complete 2022-09-02T05:49:13.055 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:49:13.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:49:13.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:49:13.057 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:49:13.121 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:49:13.122 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:49:13.162 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:13.162+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:13.163 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:13.163 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:18.164 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:18.164+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:18.165 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:18.165 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:23.166 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:23.165+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:23.166 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:23.167 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:28.169 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:28.166+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:28.169 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:28.170 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:33.169 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:33.168+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:33.170 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:33.170 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:38.171 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:38.170+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:38.172 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:38.173 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:43.173 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:43.172+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:43.173 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:43.174 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:48.175 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:48.174+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:48.175 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:48.175 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:53.176 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:53.176+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:53.177 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:53.177 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:49:58.179 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:49:58.177+0000 7ff6cbcb9600 1 waiting for incremental sync to catch up: 2022-09-02T05:49:58.179 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, , , , , , , , , , , , ] 2022-09-02T05:49:58.180 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:50:03.180 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:03.179+0000 7ff6cbcb9600 1 bucket sync caught up with source: 2022-09-02T05:50:03.181 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:50:03.181 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.63.5, 00000000004.56.5, , , , , , , , , , , ] 2022-09-02T05:50:03.181 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:03.179+0000 7ff6cbcb9600 0 bucket checkpoint complete 2022-09-02T05:50:03.201 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:50:03.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.205 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:50:03.270 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:50:03.271 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:50:03.322 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:50:03.321+0000 7feba0872600 1 bucket sync caught up with source: 2022-09-02T05:50:03.323 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.470.5, 00000000001.399.5, 00000000001.1063.5, 00000000001.587.5, 00000000001.401.5, , , 00000000002.544.5, 00000000004.1200.5, 00000000003.829.5] 2022-09-02T05:50:03.323 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.470.5, 00000000001.399.5, 00000000001.1063.5, 00000000001.587.5, 00000000001.401.5, , , 00000000002.544.5, 00000000004.1200.5, 00000000003.829.5] 2022-09-02T05:50:03.324 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:50:03.321+0000 7feba0872600 0 bucket checkpoint complete 2022-09-02T05:50:03.340 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:50:03.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.344 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:50:03.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:50:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:50:03.450 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:50:03.448+0000 7fc077473600 1 bucket sync caught up with source: 2022-09-02T05:50:03.450 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.470.5, 00000000001.399.5, 00000000001.1063.5, 00000000001.587.5, 00000000001.401.5, , , 00000000002.544.5, 00000000004.1200.5, 00000000003.829.5] 2022-09-02T05:50:03.450 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.470.5, 00000000001.399.5, 00000000001.1063.5, 00000000001.587.5, 00000000001.401.5, , , 00000000002.544.5, 00000000004.1200.5, 00000000003.829.5] 2022-09-02T05:50:03.451 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:50:03.449+0000 7fc077473600 0 bucket checkpoint complete 2022-09-02T05:50:03.462 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:50:03.463 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:50:03.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.465 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:50:03.521 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:50:03.521 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:50:03.557 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:50:03.556+0000 7f6d9b961600 1 bucket sync caught up with source: 2022-09-02T05:50:03.557 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.92.5, 00000000001.378.5, 00000000001.524.5, 00000000001.347.5, 00000000001.380.5, , , 00000000002.200.5, 00000000004.330.5, 00000000003.518.5] 2022-09-02T05:50:03.558 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.92.5, 00000000001.378.5, 00000000001.524.5, 00000000001.347.5, 00000000001.380.5, , , 00000000002.200.5, 00000000004.330.5, 00000000003.518.5] 2022-09-02T05:50:03.558 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:50:03.556+0000 7f6d9b961600 0 bucket checkpoint complete 2022-09-02T05:50:03.573 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:50:03.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:03.575 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:50:03.639 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:50:03.640 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:50:03.683 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:03.682+0000 7f85a37c1600 1 bucket sync caught up with source: 2022-09-02T05:50:03.683 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.92.5, 00000000001.378.5, 00000000001.524.5, 00000000001.347.5, 00000000001.380.5, , , 00000000002.200.5, 00000000004.330.5, 00000000003.518.5] 2022-09-02T05:50:03.684 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.92.5, 00000000001.378.5, 00000000001.524.5, 00000000001.347.5, 00000000001.380.5, , , 00000000002.200.5, 00000000004.330.5, 00000000003.518.5] 2022-09-02T05:50:03.684 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:03.682+0000 7f85a37c1600 0 bucket checkpoint complete 2022-09-02T05:50:03.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:50:03.700 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:50:03.701 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.702 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.704 DEBUG:boto:Method: HEAD 2022-09-02T05:50:03.705 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:03.705 DEBUG:boto:Data: 2022-09-02T05:50:03.706 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.707 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.707 DEBUG:boto:Params: {} 2022-09-02T05:50:03.708 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:50:03.709 DEBUG:boto:Token: None 2022-09-02T05:50:03.710 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/ 2022-09-02T05:50:03.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SeSfwhsgWPgC94P4+5EpsaH2G2A= 2022-09-02T05:50:03.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SeSfwhsgWPgC94P4+5EpsaH2G2A=', 'Content-Length': '0'} 2022-09-02T05:50:03.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056dbf326a9029df7-006311998b-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.716 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.716 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.717 DEBUG:boto:Method: HEAD 2022-09-02T05:50:03.717 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:03.718 DEBUG:boto:Data: 2022-09-02T05:50:03.718 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.719 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.720 DEBUG:boto:Params: {} 2022-09-02T05:50:03.720 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:50:03.721 DEBUG:boto:Token: None 2022-09-02T05:50:03.722 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/ 2022-09-02T05:50:03.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SeSfwhsgWPgC94P4+5EpsaH2G2A= 2022-09-02T05:50:03.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SeSfwhsgWPgC94P4+5EpsaH2G2A=', 'Content-Length': '0'} 2022-09-02T05:50:03.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b3e4ddf21dcf33c-006311998b-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.728 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:50:03.729 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.729 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.730 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.730 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.731 DEBUG:boto:Method: GET 2022-09-02T05:50:03.731 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.732 DEBUG:boto:Data: 2022-09-02T05:50:03.732 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.733 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.734 DEBUG:boto:Params: {} 2022-09-02T05:50:03.735 DEBUG:boto:Token: None 2022-09-02T05:50:03.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/?versions 2022-09-02T05:50:03.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As= 2022-09-02T05:50:03.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As=', 'Content-Length': '0'} 2022-09-02T05:50:03.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007e984cfc876dc3e-006311998b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.740 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:50:03.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:50:03.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:50:03.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:50:03.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:50:03.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:50:03.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:50:03.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:50:03.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:50:03.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:50:03.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:50:03.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:50:03.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:50:03.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:50:03.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:50:03.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:50:03.750 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.750 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.751 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.751 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.752 DEBUG:boto:Method: GET 2022-09-02T05:50:03.752 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.753 DEBUG:boto:Data: 2022-09-02T05:50:03.753 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.754 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.755 DEBUG:boto:Params: {} 2022-09-02T05:50:03.756 DEBUG:boto:Token: None 2022-09-02T05:50:03.756 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/?versions 2022-09-02T05:50:03.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As= 2022-09-02T05:50:03.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As=', 'Content-Length': '0'} 2022-09-02T05:50:03.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62ac97dad7cd288-006311998b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.761 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:50:03.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:50:03.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:50:03.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:50:03.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:50:03.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:50:03.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:50:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:50:03.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:50:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:50:03.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:50:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:50:03.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:50:03.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:50:03.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:50:03.770 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.770 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.771 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.771 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.772 DEBUG:boto:Method: GET 2022-09-02T05:50:03.772 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.773 DEBUG:boto:Data: 2022-09-02T05:50:03.773 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.774 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.775 DEBUG:boto:Params: {} 2022-09-02T05:50:03.776 DEBUG:boto:Token: None 2022-09-02T05:50:03.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/?versions 2022-09-02T05:50:03.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As= 2022-09-02T05:50:03.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As=', 'Content-Length': '0'} 2022-09-02T05:50:03.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006510843b2a7219f6-006311998b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.781 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.783 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.783 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.784 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.784 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.785 DEBUG:boto:Method: GET 2022-09-02T05:50:03.785 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:03.786 DEBUG:boto:Data: 2022-09-02T05:50:03.786 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.787 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.788 DEBUG:boto:Params: {} 2022-09-02T05:50:03.788 DEBUG:boto:Token: None 2022-09-02T05:50:03.789 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/?versions 2022-09-02T05:50:03.790 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As= 2022-09-02T05:50:03.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y3H6aoPArIgbX2YPYMk8QQyt6As=', 'Content-Length': '0'} 2022-09-02T05:50:03.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e1acdb1cb1c455-006311998b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.794 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:03.796 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.797 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.797 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.798 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.798 DEBUG:boto:Method: GET 2022-09-02T05:50:03.799 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.799 DEBUG:boto:Data: 2022-09-02T05:50:03.800 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.801 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.801 DEBUG:boto:Params: {} 2022-09-02T05:50:03.802 DEBUG:boto:Token: None 2022-09-02T05:50:03.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4= 2022-09-02T05:50:03.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4=', 'Content-Length': '0'} 2022-09-02T05:50:03.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060ad6974af1947e-006311998b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.807 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.807 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.808 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.809 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.809 DEBUG:boto:Method: GET 2022-09-02T05:50:03.810 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.810 DEBUG:boto:Data: 2022-09-02T05:50:03.811 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.812 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.812 DEBUG:boto:Params: {} 2022-09-02T05:50:03.813 DEBUG:boto:Token: None 2022-09-02T05:50:03.814 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4= 2022-09-02T05:50:03.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4=', 'Content-Length': '0'} 2022-09-02T05:50:03.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b81bc201ec4fe49d-006311998b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.818 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.819 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.819 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.820 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.820 DEBUG:boto:Method: HEAD 2022-09-02T05:50:03.821 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.822 DEBUG:boto:Data: 2022-09-02T05:50:03.822 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.823 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.824 DEBUG:boto:Params: {} 2022-09-02T05:50:03.824 DEBUG:boto:Token: None 2022-09-02T05:50:03.825 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W7OrqSSW44t3SJOEW7jnDjJxrjc= 2022-09-02T05:50:03.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W7OrqSSW44t3SJOEW7jnDjJxrjc=', 'Content-Length': '0'} 2022-09-02T05:50:03.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518341bcba9611d7-006311998b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.829 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.829 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.830 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.830 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.831 DEBUG:boto:Method: HEAD 2022-09-02T05:50:03.832 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.832 DEBUG:boto:Data: 2022-09-02T05:50:03.833 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.834 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.834 DEBUG:boto:Params: {} 2022-09-02T05:50:03.835 DEBUG:boto:Token: None 2022-09-02T05:50:03.836 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W7OrqSSW44t3SJOEW7jnDjJxrjc= 2022-09-02T05:50:03.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W7OrqSSW44t3SJOEW7jnDjJxrjc=', 'Content-Length': '0'} 2022-09-02T05:50:03.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3cf7b73e6297548-006311998b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:03.842 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.843 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.843 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.844 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.845 DEBUG:boto:Method: GET 2022-09-02T05:50:03.845 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.846 DEBUG:boto:Data: 2022-09-02T05:50:03.846 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.847 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.848 DEBUG:boto:Params: {} 2022-09-02T05:50:03.848 DEBUG:boto:Token: None 2022-09-02T05:50:03.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4= 2022-09-02T05:50:03.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4=', 'Content-Length': '0'} 2022-09-02T05:50:03.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bcd75fa4cc3b87d-006311998b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.854 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.854 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.855 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.856 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.856 DEBUG:boto:Method: GET 2022-09-02T05:50:03.857 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.857 DEBUG:boto:Data: 2022-09-02T05:50:03.858 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.859 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.859 DEBUG:boto:Params: {} 2022-09-02T05:50:03.860 DEBUG:boto:Token: None 2022-09-02T05:50:03.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:03.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4= 2022-09-02T05:50:03.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ny+LQfeelJcLxhvBAjkEnK4vCd4=', 'Content-Length': '0'} 2022-09-02T05:50:03.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005933a676371e23d9-006311998b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.865 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.866 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.866 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.867 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.867 DEBUG:boto:Method: GET 2022-09-02T05:50:03.868 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.868 DEBUG:boto:Data: 2022-09-02T05:50:03.869 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.869 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.870 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.871 DEBUG:boto:Params: {} 2022-09-02T05:50:03.871 DEBUG:boto:Token: None 2022-09-02T05:50:03.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/ 2022-09-02T05:50:03.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yEepXAB9S//06557lOMjLxvYyMs= 2022-09-02T05:50:03.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yEepXAB9S//06557lOMjLxvYyMs=', 'Content-Length': '0'} 2022-09-02T05:50:03.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009844ffdb52e1afae-006311998b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.876 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.877 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.878 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.878 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.879 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.879 DEBUG:boto:Method: GET 2022-09-02T05:50:03.880 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.880 DEBUG:boto:Data: 2022-09-02T05:50:03.881 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.882 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.883 DEBUG:boto:Params: {} 2022-09-02T05:50:03.883 DEBUG:boto:Token: None 2022-09-02T05:50:03.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/ 2022-09-02T05:50:03.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yEepXAB9S//06557lOMjLxvYyMs= 2022-09-02T05:50:03.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yEepXAB9S//06557lOMjLxvYyMs=', 'Content-Length': '0'} 2022-09-02T05:50:03.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad3a6df4f9567ee-006311998b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.889 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.890 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.890 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.891 DEBUG:boto:Method: HEAD 2022-09-02T05:50:03.891 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:03.892 DEBUG:boto:Data: 2022-09-02T05:50:03.892 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.893 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.894 DEBUG:boto:Params: {} 2022-09-02T05:50:03.895 DEBUG:boto:Token: None 2022-09-02T05:50:03.895 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11 2022-09-02T05:50:03.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k8GZNM4shwVgIBzYEmIwpIno/SA= 2022-09-02T05:50:03.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k8GZNM4shwVgIBzYEmIwpIno/SA=', 'Content-Length': '0'} 2022-09-02T05:50:03.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a08a300f1eb0ad4-006311998b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.899 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.900 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.900 DEBUG:boto:Method: HEAD 2022-09-02T05:50:03.901 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:03.901 DEBUG:boto:Data: 2022-09-02T05:50:03.902 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.903 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.903 DEBUG:boto:Params: {} 2022-09-02T05:50:03.904 DEBUG:boto:Token: None 2022-09-02T05:50:03.904 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11 2022-09-02T05:50:03.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k8GZNM4shwVgIBzYEmIwpIno/SA= 2022-09-02T05:50:03.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k8GZNM4shwVgIBzYEmIwpIno/SA=', 'Content-Length': '0'} 2022-09-02T05:50:03.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004644a3596f6f530b-006311998b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:03.909 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.910 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.910 DEBUG:boto:Method: GET 2022-09-02T05:50:03.911 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:03.911 DEBUG:boto:Data: 2022-09-02T05:50:03.912 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.912 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.913 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.913 DEBUG:boto:Params: {} 2022-09-02T05:50:03.914 DEBUG:boto:Token: None 2022-09-02T05:50:03.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11 2022-09-02T05:50:03.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WNmuLhLIRzcK10yR0jHqD37z4mU= 2022-09-02T05:50:03.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WNmuLhLIRzcK10yR0jHqD37z4mU=', 'Content-Length': '0'} 2022-09-02T05:50:03.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21bf16aa89ffbc1-006311998b-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.962 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:03.963 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:03.963 DEBUG:boto:Method: GET 2022-09-02T05:50:03.964 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:03.965 DEBUG:boto:Data: 2022-09-02T05:50:03.965 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:03.966 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.967 DEBUG:boto:Params: {} 2022-09-02T05:50:03.968 DEBUG:boto:Token: None 2022-09-02T05:50:03.968 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/obj11 2022-09-02T05:50:03.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WNmuLhLIRzcK10yR0jHqD37z4mU= 2022-09-02T05:50:03.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WNmuLhLIRzcK10yR0jHqD37z4mU=', 'Content-Length': '0'} 2022-09-02T05:50:03.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a74354f9d0acad-006311998b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.974 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.974 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.975 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.975 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.976 DEBUG:boto:Method: GET 2022-09-02T05:50:03.977 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.978 DEBUG:boto:Data: 2022-09-02T05:50:03.979 DEBUG:boto:Headers: {} 2022-09-02T05:50:03.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:03.981 DEBUG:boto:Port: 8000 2022-09-02T05:50:03.982 DEBUG:boto:Params: {} 2022-09-02T05:50:03.983 DEBUG:boto:Token: None 2022-09-02T05:50:03.985 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:03 GMT /xljbkf-74/ 2022-09-02T05:50:03.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yEepXAB9S//06557lOMjLxvYyMs= 2022-09-02T05:50:03.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yEepXAB9S//06557lOMjLxvYyMs=', 'Content-Length': '0'} 2022-09-02T05:50:03.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d0b4f9c0f04334f-006311998b-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:03.991 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:03.993 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:03.994 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:03.995 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.996 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.997 DEBUG:boto:Method: GET 2022-09-02T05:50:03.998 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:03.999 DEBUG:boto:Data: 2022-09-02T05:50:04.000 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.002 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.003 DEBUG:boto:Params: {} 2022-09-02T05:50:04.004 DEBUG:boto:Token: None 2022-09-02T05:50:04.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a9e7a15ca4af680-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.012 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:04.014 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.015 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.016 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.017 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.018 DEBUG:boto:Method: GET 2022-09-02T05:50:04.019 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.020 DEBUG:boto:Data: 2022-09-02T05:50:04.021 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.023 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.024 DEBUG:boto:Params: {} 2022-09-02T05:50:04.025 DEBUG:boto:Token: None 2022-09-02T05:50:04.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0= 2022-09-02T05:50:04.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0=', 'Content-Length': '0'} 2022-09-02T05:50:04.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a347d02e724c6bf7-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.029 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.030 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.030 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.031 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.031 DEBUG:boto:Method: GET 2022-09-02T05:50:04.032 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.033 DEBUG:boto:Data: 2022-09-02T05:50:04.033 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.034 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.035 DEBUG:boto:Params: {} 2022-09-02T05:50:04.036 DEBUG:boto:Token: None 2022-09-02T05:50:04.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0= 2022-09-02T05:50:04.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0=', 'Content-Length': '0'} 2022-09-02T05:50:04.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f882855f9ffe7ecb-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.082 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.083 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.084 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.085 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.086 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.087 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.088 DEBUG:boto:Data: 2022-09-02T05:50:04.089 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.090 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.091 DEBUG:boto:Params: {} 2022-09-02T05:50:04.093 DEBUG:boto:Token: None 2022-09-02T05:50:04.094 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mijzL85o0x93cId6LNpl5po7GpM= 2022-09-02T05:50:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mijzL85o0x93cId6LNpl5po7GpM=', 'Content-Length': '0'} 2022-09-02T05:50:04.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8f3c42ad3f2a51-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.100 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.101 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.101 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.102 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.103 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.104 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.105 DEBUG:boto:Data: 2022-09-02T05:50:04.106 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.108 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.109 DEBUG:boto:Params: {} 2022-09-02T05:50:04.110 DEBUG:boto:Token: None 2022-09-02T05:50:04.111 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mijzL85o0x93cId6LNpl5po7GpM= 2022-09-02T05:50:04.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mijzL85o0x93cId6LNpl5po7GpM=', 'Content-Length': '0'} 2022-09-02T05:50:04.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249a153d6d690378-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:04.118 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.119 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.120 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.121 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.122 DEBUG:boto:Method: GET 2022-09-02T05:50:04.123 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.124 DEBUG:boto:Data: 2022-09-02T05:50:04.125 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.126 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.126 DEBUG:boto:Params: {} 2022-09-02T05:50:04.127 DEBUG:boto:Token: None 2022-09-02T05:50:04.128 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0= 2022-09-02T05:50:04.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0=', 'Content-Length': '0'} 2022-09-02T05:50:04.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465b5de097745328-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.174 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.175 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.176 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.177 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.178 DEBUG:boto:Method: GET 2022-09-02T05:50:04.179 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.180 DEBUG:boto:Data: 2022-09-02T05:50:04.181 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.183 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.184 DEBUG:boto:Params: {} 2022-09-02T05:50:04.185 DEBUG:boto:Token: None 2022-09-02T05:50:04.186 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:04.187 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0= 2022-09-02T05:50:04.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xYeSKcpuSlsJC1W9hvFhcMVTuz0=', 'Content-Length': '0'} 2022-09-02T05:50:04.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8eb9d22a587b3b-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.192 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.192 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.193 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.193 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.194 DEBUG:boto:Method: GET 2022-09-02T05:50:04.194 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.195 DEBUG:boto:Data: 2022-09-02T05:50:04.195 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.197 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.197 DEBUG:boto:Params: {} 2022-09-02T05:50:04.198 DEBUG:boto:Token: None 2022-09-02T05:50:04.199 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fc96ed523f26101-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.204 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.205 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.205 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.206 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.207 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.209 DEBUG:boto:Method: GET 2022-09-02T05:50:04.210 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.211 DEBUG:boto:Data: 2022-09-02T05:50:04.212 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.214 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.215 DEBUG:boto:Params: {} 2022-09-02T05:50:04.216 DEBUG:boto:Token: None 2022-09-02T05:50:04.218 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000518b7c80f3095a87-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.224 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.225 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.225 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.226 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.227 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:04.227 DEBUG:boto:Data: 2022-09-02T05:50:04.228 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.229 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.229 DEBUG:boto:Params: {} 2022-09-02T05:50:04.230 DEBUG:boto:Token: None 2022-09-02T05:50:04.231 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17 2022-09-02T05:50:04.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cZ3+wvN7fyBaskteNHvAAoaazBE= 2022-09-02T05:50:04.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cZ3+wvN7fyBaskteNHvAAoaazBE=', 'Content-Length': '0'} 2022-09-02T05:50:04.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c831bb855158a147-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.235 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.235 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.236 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.236 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:04.237 DEBUG:boto:Data: 2022-09-02T05:50:04.237 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.239 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.239 DEBUG:boto:Params: {} 2022-09-02T05:50:04.240 DEBUG:boto:Token: None 2022-09-02T05:50:04.241 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17 2022-09-02T05:50:04.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cZ3+wvN7fyBaskteNHvAAoaazBE= 2022-09-02T05:50:04.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cZ3+wvN7fyBaskteNHvAAoaazBE=', 'Content-Length': '0'} 2022-09-02T05:50:04.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2f536ca960fc25-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:04.246 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.247 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.248 DEBUG:boto:Method: GET 2022-09-02T05:50:04.249 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:04.250 DEBUG:boto:Data: 2022-09-02T05:50:04.251 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.253 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.254 DEBUG:boto:Params: {} 2022-09-02T05:50:04.255 DEBUG:boto:Token: None 2022-09-02T05:50:04.256 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17 2022-09-02T05:50:04.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGvV/xp3qu5luJKZj4F1dyWO0cM= 2022-09-02T05:50:04.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGvV/xp3qu5luJKZj4F1dyWO0cM=', 'Content-Length': '0'} 2022-09-02T05:50:04.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad8547db5c8b9dc-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.302 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:04.303 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:04.303 DEBUG:boto:Method: GET 2022-09-02T05:50:04.304 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:04.305 DEBUG:boto:Data: 2022-09-02T05:50:04.305 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.306 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.307 DEBUG:boto:Params: {} 2022-09-02T05:50:04.307 DEBUG:boto:Token: None 2022-09-02T05:50:04.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj17 2022-09-02T05:50:04.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGvV/xp3qu5luJKZj4F1dyWO0cM= 2022-09-02T05:50:04.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGvV/xp3qu5luJKZj4F1dyWO0cM=', 'Content-Length': '0'} 2022-09-02T05:50:04.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f54d2d9d68efad-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.312 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.313 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.313 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.314 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.315 DEBUG:boto:Method: GET 2022-09-02T05:50:04.315 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.316 DEBUG:boto:Data: 2022-09-02T05:50:04.316 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.317 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.318 DEBUG:boto:Params: {} 2022-09-02T05:50:04.318 DEBUG:boto:Token: None 2022-09-02T05:50:04.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1f7fff659ade47e-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.323 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.324 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.325 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.326 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.326 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.327 DEBUG:boto:Method: GET 2022-09-02T05:50:04.327 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:04.328 DEBUG:boto:Data: 2022-09-02T05:50:04.328 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.329 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.330 DEBUG:boto:Params: {} 2022-09-02T05:50:04.331 DEBUG:boto:Token: None 2022-09-02T05:50:04.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074734823a73ec473-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.336 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:04.338 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.338 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.339 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.340 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.341 DEBUG:boto:Method: GET 2022-09-02T05:50:04.342 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.343 DEBUG:boto:Data: 2022-09-02T05:50:04.344 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.345 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.346 DEBUG:boto:Params: {} 2022-09-02T05:50:04.346 DEBUG:boto:Token: None 2022-09-02T05:50:04.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo= 2022-09-02T05:50:04.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo=', 'Content-Length': '0'} 2022-09-02T05:50:04.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077cafbe8ff96cf78-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.351 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.351 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.352 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.352 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.353 DEBUG:boto:Method: GET 2022-09-02T05:50:04.354 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.354 DEBUG:boto:Data: 2022-09-02T05:50:04.355 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.356 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.356 DEBUG:boto:Params: {} 2022-09-02T05:50:04.357 DEBUG:boto:Token: None 2022-09-02T05:50:04.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo= 2022-09-02T05:50:04.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo=', 'Content-Length': '0'} 2022-09-02T05:50:04.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edfe08f3ca0738c9-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.362 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.362 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.363 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.363 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.364 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.364 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.365 DEBUG:boto:Data: 2022-09-02T05:50:04.365 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.366 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.367 DEBUG:boto:Params: {} 2022-09-02T05:50:04.367 DEBUG:boto:Token: None 2022-09-02T05:50:04.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GqLbZp/qz7mngtXaYM95mIJPaqs= 2022-09-02T05:50:04.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GqLbZp/qz7mngtXaYM95mIJPaqs=', 'Content-Length': '0'} 2022-09-02T05:50:04.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000475ec31a1f6f4d-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.372 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.372 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.373 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.373 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.374 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.374 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.375 DEBUG:boto:Data: 2022-09-02T05:50:04.375 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.376 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.377 DEBUG:boto:Params: {} 2022-09-02T05:50:04.378 DEBUG:boto:Token: None 2022-09-02T05:50:04.378 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GqLbZp/qz7mngtXaYM95mIJPaqs= 2022-09-02T05:50:04.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GqLbZp/qz7mngtXaYM95mIJPaqs=', 'Content-Length': '0'} 2022-09-02T05:50:04.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c61396f147ea5053-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:04.383 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.383 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.384 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.384 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.385 DEBUG:boto:Method: GET 2022-09-02T05:50:04.385 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.386 DEBUG:boto:Data: 2022-09-02T05:50:04.386 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.387 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.388 DEBUG:boto:Params: {} 2022-09-02T05:50:04.389 DEBUG:boto:Token: None 2022-09-02T05:50:04.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo= 2022-09-02T05:50:04.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo=', 'Content-Length': '0'} 2022-09-02T05:50:04.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ebc29dbac0ea0f-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.434 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.435 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.435 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.436 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.437 DEBUG:boto:Method: GET 2022-09-02T05:50:04.437 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.438 DEBUG:boto:Data: 2022-09-02T05:50:04.438 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.439 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.440 DEBUG:boto:Params: {} 2022-09-02T05:50:04.440 DEBUG:boto:Token: None 2022-09-02T05:50:04.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:04.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo= 2022-09-02T05:50:04.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:spgrUHfyBRF9/tXygryrJ3Jydqo=', 'Content-Length': '0'} 2022-09-02T05:50:04.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe004aa712766af-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.445 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.446 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.447 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.447 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.448 DEBUG:boto:Method: GET 2022-09-02T05:50:04.448 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.449 DEBUG:boto:Data: 2022-09-02T05:50:04.449 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.450 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.451 DEBUG:boto:Params: {} 2022-09-02T05:50:04.452 DEBUG:boto:Token: None 2022-09-02T05:50:04.452 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000279420e33da65682-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.457 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.458 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.458 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.459 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.459 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.460 DEBUG:boto:Method: GET 2022-09-02T05:50:04.460 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.461 DEBUG:boto:Data: 2022-09-02T05:50:04.462 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.463 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.463 DEBUG:boto:Params: {} 2022-09-02T05:50:04.464 DEBUG:boto:Token: None 2022-09-02T05:50:04.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e13ee7a04cc0fd3-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.469 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.470 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.470 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.471 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.471 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:04.472 DEBUG:boto:Data: 2022-09-02T05:50:04.473 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.474 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.474 DEBUG:boto:Params: {} 2022-09-02T05:50:04.475 DEBUG:boto:Token: None 2022-09-02T05:50:04.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19 2022-09-02T05:50:04.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Yjy9OnFFqkhde71DTZX+k34dt4= 2022-09-02T05:50:04.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Yjy9OnFFqkhde71DTZX+k34dt4=', 'Content-Length': '0'} 2022-09-02T05:50:04.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec0d89ad299c540-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.479 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.479 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.480 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.481 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:04.481 DEBUG:boto:Data: 2022-09-02T05:50:04.482 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.483 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.483 DEBUG:boto:Params: {} 2022-09-02T05:50:04.484 DEBUG:boto:Token: None 2022-09-02T05:50:04.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19 2022-09-02T05:50:04.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Yjy9OnFFqkhde71DTZX+k34dt4= 2022-09-02T05:50:04.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Yjy9OnFFqkhde71DTZX+k34dt4=', 'Content-Length': '0'} 2022-09-02T05:50:04.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4869ad01023f67c-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:04.488 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.489 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.490 DEBUG:boto:Method: GET 2022-09-02T05:50:04.490 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:04.491 DEBUG:boto:Data: 2022-09-02T05:50:04.491 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.492 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.492 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.493 DEBUG:boto:Params: {} 2022-09-02T05:50:04.493 DEBUG:boto:Token: None 2022-09-02T05:50:04.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19 2022-09-02T05:50:04.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1tHYTUYu7uNZ53VMSUquAmbZh9Y= 2022-09-02T05:50:04.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1tHYTUYu7uNZ53VMSUquAmbZh9Y=', 'Content-Length': '0'} 2022-09-02T05:50:04.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e646be86c3f5a4b7-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.498 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:04.498 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:04.499 DEBUG:boto:Method: GET 2022-09-02T05:50:04.500 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:04.500 DEBUG:boto:Data: 2022-09-02T05:50:04.501 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.502 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.502 DEBUG:boto:Params: {} 2022-09-02T05:50:04.503 DEBUG:boto:Token: None 2022-09-02T05:50:04.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj19 2022-09-02T05:50:04.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1tHYTUYu7uNZ53VMSUquAmbZh9Y= 2022-09-02T05:50:04.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1tHYTUYu7uNZ53VMSUquAmbZh9Y=', 'Content-Length': '0'} 2022-09-02T05:50:04.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa8a42ad2c1c724-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.507 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.508 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.508 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.509 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.509 DEBUG:boto:Method: GET 2022-09-02T05:50:04.510 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.511 DEBUG:boto:Data: 2022-09-02T05:50:04.511 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.512 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.513 DEBUG:boto:Params: {} 2022-09-02T05:50:04.513 DEBUG:boto:Token: None 2022-09-02T05:50:04.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a1d6fc87cd1a5d-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.518 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.519 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.520 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.520 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.521 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.521 DEBUG:boto:Method: GET 2022-09-02T05:50:04.522 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:04.523 DEBUG:boto:Data: 2022-09-02T05:50:04.523 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.524 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.525 DEBUG:boto:Params: {} 2022-09-02T05:50:04.525 DEBUG:boto:Token: None 2022-09-02T05:50:04.526 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d177397f8ee644b3-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.531 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:04.532 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.533 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.533 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.534 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.534 DEBUG:boto:Method: GET 2022-09-02T05:50:04.535 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.535 DEBUG:boto:Data: 2022-09-02T05:50:04.536 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.537 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.537 DEBUG:boto:Params: {} 2022-09-02T05:50:04.538 DEBUG:boto:Token: None 2022-09-02T05:50:04.539 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8= 2022-09-02T05:50:04.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8=', 'Content-Length': '0'} 2022-09-02T05:50:04.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b979dfae7ab3bf61-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.543 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.543 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.544 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.544 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.545 DEBUG:boto:Method: GET 2022-09-02T05:50:04.545 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.546 DEBUG:boto:Data: 2022-09-02T05:50:04.546 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.548 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.548 DEBUG:boto:Params: {} 2022-09-02T05:50:04.549 DEBUG:boto:Token: None 2022-09-02T05:50:04.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8= 2022-09-02T05:50:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8=', 'Content-Length': '0'} 2022-09-02T05:50:04.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae85d8f8f0ccd69-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.594 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.595 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.596 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.597 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.598 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.599 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.600 DEBUG:boto:Data: 2022-09-02T05:50:04.601 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.603 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.604 DEBUG:boto:Params: {} 2022-09-02T05:50:04.605 DEBUG:boto:Token: None 2022-09-02T05:50:04.608 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0hSQhrOPBCFDjxIxeDM0AoAqb58= 2022-09-02T05:50:04.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0hSQhrOPBCFDjxIxeDM0AoAqb58=', 'Content-Length': '0'} 2022-09-02T05:50:04.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce8e93c9aada391b-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.614 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.614 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.615 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.615 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.616 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.617 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.617 DEBUG:boto:Data: 2022-09-02T05:50:04.618 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.619 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.619 DEBUG:boto:Params: {} 2022-09-02T05:50:04.620 DEBUG:boto:Token: None 2022-09-02T05:50:04.620 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0hSQhrOPBCFDjxIxeDM0AoAqb58= 2022-09-02T05:50:04.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0hSQhrOPBCFDjxIxeDM0AoAqb58=', 'Content-Length': '0'} 2022-09-02T05:50:04.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0abdb3882de0a3d-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:04.625 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.626 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.627 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.627 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.628 DEBUG:boto:Method: GET 2022-09-02T05:50:04.628 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.629 DEBUG:boto:Data: 2022-09-02T05:50:04.630 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.631 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.632 DEBUG:boto:Params: {} 2022-09-02T05:50:04.633 DEBUG:boto:Token: None 2022-09-02T05:50:04.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8= 2022-09-02T05:50:04.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8=', 'Content-Length': '0'} 2022-09-02T05:50:04.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e46bb6ad288ea3-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.638 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.639 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.639 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.640 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.641 DEBUG:boto:Method: GET 2022-09-02T05:50:04.641 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.642 DEBUG:boto:Data: 2022-09-02T05:50:04.643 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.644 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.645 DEBUG:boto:Params: {} 2022-09-02T05:50:04.646 DEBUG:boto:Token: None 2022-09-02T05:50:04.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:04.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8= 2022-09-02T05:50:04.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:teJzd2aGVAQNjJlhNFarbG+PDm8=', 'Content-Length': '0'} 2022-09-02T05:50:04.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4eb5268974b99b-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.652 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.652 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.653 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.653 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.654 DEBUG:boto:Method: GET 2022-09-02T05:50:04.655 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.655 DEBUG:boto:Data: 2022-09-02T05:50:04.656 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.657 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.658 DEBUG:boto:Params: {} 2022-09-02T05:50:04.659 DEBUG:boto:Token: None 2022-09-02T05:50:04.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397b00168e9a4842-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.665 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.666 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.667 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.667 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.668 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.668 DEBUG:boto:Method: GET 2022-09-02T05:50:04.669 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.669 DEBUG:boto:Data: 2022-09-02T05:50:04.670 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.671 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.672 DEBUG:boto:Params: {} 2022-09-02T05:50:04.672 DEBUG:boto:Token: None 2022-09-02T05:50:04.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b423990449af979-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.679 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.679 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.680 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.681 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.681 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:04.682 DEBUG:boto:Data: 2022-09-02T05:50:04.683 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.685 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.685 DEBUG:boto:Params: {} 2022-09-02T05:50:04.686 DEBUG:boto:Token: None 2022-09-02T05:50:04.686 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22 2022-09-02T05:50:04.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xkT5iWyB4UJh7dNSwbj8qYmFfc0= 2022-09-02T05:50:04.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xkT5iWyB4UJh7dNSwbj8qYmFfc0=', 'Content-Length': '0'} 2022-09-02T05:50:04.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81eeb29eb4931c7-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.690 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.691 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.692 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.693 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:04.693 DEBUG:boto:Data: 2022-09-02T05:50:04.694 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.696 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.696 DEBUG:boto:Params: {} 2022-09-02T05:50:04.697 DEBUG:boto:Token: None 2022-09-02T05:50:04.698 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22 2022-09-02T05:50:04.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xkT5iWyB4UJh7dNSwbj8qYmFfc0= 2022-09-02T05:50:04.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xkT5iWyB4UJh7dNSwbj8qYmFfc0=', 'Content-Length': '0'} 2022-09-02T05:50:04.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e3e09ad7d00313-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:04.703 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.703 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.704 DEBUG:boto:Method: GET 2022-09-02T05:50:04.704 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:04.705 DEBUG:boto:Data: 2022-09-02T05:50:04.705 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.707 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.708 DEBUG:boto:Params: {} 2022-09-02T05:50:04.709 DEBUG:boto:Token: None 2022-09-02T05:50:04.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22 2022-09-02T05:50:04.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b/AnaXck65KovQiSLDEEVg9qq+Y= 2022-09-02T05:50:04.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b/AnaXck65KovQiSLDEEVg9qq+Y=', 'Content-Length': '0'} 2022-09-02T05:50:04.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f17b25c9d317ad-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.715 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:04.715 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:04.716 DEBUG:boto:Method: GET 2022-09-02T05:50:04.716 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:04.717 DEBUG:boto:Data: 2022-09-02T05:50:04.718 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.720 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.721 DEBUG:boto:Params: {} 2022-09-02T05:50:04.722 DEBUG:boto:Token: None 2022-09-02T05:50:04.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj22 2022-09-02T05:50:04.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b/AnaXck65KovQiSLDEEVg9qq+Y= 2022-09-02T05:50:04.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b/AnaXck65KovQiSLDEEVg9qq+Y=', 'Content-Length': '0'} 2022-09-02T05:50:04.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022832859178fdb6c-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.770 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.771 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.772 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.773 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.774 DEBUG:boto:Method: GET 2022-09-02T05:50:04.775 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.776 DEBUG:boto:Data: 2022-09-02T05:50:04.777 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.778 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.779 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.780 DEBUG:boto:Params: {} 2022-09-02T05:50:04.781 DEBUG:boto:Token: None 2022-09-02T05:50:04.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000765f8be89c382c5e-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.789 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.790 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.791 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.792 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.793 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.794 DEBUG:boto:Method: GET 2022-09-02T05:50:04.795 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:04.796 DEBUG:boto:Data: 2022-09-02T05:50:04.797 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.799 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.800 DEBUG:boto:Params: {} 2022-09-02T05:50:04.801 DEBUG:boto:Token: None 2022-09-02T05:50:04.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9703ee8aaa687d1-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.809 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:04.812 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.813 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.814 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.815 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.816 DEBUG:boto:Method: GET 2022-09-02T05:50:04.816 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.817 DEBUG:boto:Data: 2022-09-02T05:50:04.818 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.820 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.821 DEBUG:boto:Params: {} 2022-09-02T05:50:04.822 DEBUG:boto:Token: None 2022-09-02T05:50:04.824 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA= 2022-09-02T05:50:04.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA=', 'Content-Length': '0'} 2022-09-02T05:50:04.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fe4781fa1a96f9-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.830 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.831 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.832 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.833 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.834 DEBUG:boto:Method: GET 2022-09-02T05:50:04.835 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.836 DEBUG:boto:Data: 2022-09-02T05:50:04.837 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.839 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.839 DEBUG:boto:Params: {} 2022-09-02T05:50:04.841 DEBUG:boto:Token: None 2022-09-02T05:50:04.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA= 2022-09-02T05:50:04.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA=', 'Content-Length': '0'} 2022-09-02T05:50:04.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908e82ad3612acd3-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.848 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.849 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.850 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.851 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.852 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.853 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.854 DEBUG:boto:Data: 2022-09-02T05:50:04.855 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.856 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.857 DEBUG:boto:Params: {} 2022-09-02T05:50:04.858 DEBUG:boto:Token: None 2022-09-02T05:50:04.859 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EuaPUuLOn4g4/k60XDW4r5EBJbA= 2022-09-02T05:50:04.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EuaPUuLOn4g4/k60XDW4r5EBJbA=', 'Content-Length': '0'} 2022-09-02T05:50:04.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483a6991c4e9493d-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.865 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.866 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.867 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.868 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.869 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.870 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.871 DEBUG:boto:Data: 2022-09-02T05:50:04.872 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.873 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.874 DEBUG:boto:Params: {} 2022-09-02T05:50:04.875 DEBUG:boto:Token: None 2022-09-02T05:50:04.875 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EuaPUuLOn4g4/k60XDW4r5EBJbA= 2022-09-02T05:50:04.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EuaPUuLOn4g4/k60XDW4r5EBJbA=', 'Content-Length': '0'} 2022-09-02T05:50:04.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002774122b4cd9ed31-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:04.880 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.880 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.881 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.881 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.882 DEBUG:boto:Method: GET 2022-09-02T05:50:04.883 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.883 DEBUG:boto:Data: 2022-09-02T05:50:04.884 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.884 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.885 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.885 DEBUG:boto:Params: {} 2022-09-02T05:50:04.886 DEBUG:boto:Token: None 2022-09-02T05:50:04.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA= 2022-09-02T05:50:04.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA=', 'Content-Length': '0'} 2022-09-02T05:50:04.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663b1f3ace13a783-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.891 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.892 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.893 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.893 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.894 DEBUG:boto:Method: GET 2022-09-02T05:50:04.895 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.895 DEBUG:boto:Data: 2022-09-02T05:50:04.896 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.897 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.897 DEBUG:boto:Params: {} 2022-09-02T05:50:04.898 DEBUG:boto:Token: None 2022-09-02T05:50:04.899 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:04.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA= 2022-09-02T05:50:04.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U6QoDJ2quMqtPigHIEQLazEPIxA=', 'Content-Length': '0'} 2022-09-02T05:50:04.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c17de7775d2a97-006311998c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.946 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.947 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.948 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:04.948 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:04.949 DEBUG:boto:Method: GET 2022-09-02T05:50:04.950 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:04.950 DEBUG:boto:Data: 2022-09-02T05:50:04.951 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.953 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.953 DEBUG:boto:Params: {} 2022-09-02T05:50:04.954 DEBUG:boto:Token: None 2022-09-02T05:50:04.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de904169a72f9b6c-006311998c-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.960 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.961 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:04.962 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:04.963 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:04.964 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:04.965 DEBUG:boto:Method: GET 2022-09-02T05:50:04.965 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:04.966 DEBUG:boto:Data: 2022-09-02T05:50:04.967 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.968 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.968 DEBUG:boto:Params: {} 2022-09-02T05:50:04.969 DEBUG:boto:Token: None 2022-09-02T05:50:04.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/ 2022-09-02T05:50:04.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc= 2022-09-02T05:50:04.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:daBOPMH01Vevrj0V7Fe4ZkQJrxc=', 'Content-Length': '0'} 2022-09-02T05:50:04.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0eedb773f65015-006311998c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.976 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:04.977 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.977 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.978 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.979 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:04.979 DEBUG:boto:Data: 2022-09-02T05:50:04.980 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.980 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:04.981 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.981 DEBUG:boto:Params: {} 2022-09-02T05:50:04.982 DEBUG:boto:Token: None 2022-09-02T05:50:04.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33 2022-09-02T05:50:04.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n9bZyj+0fhWvhzfToLFIER49vjQ= 2022-09-02T05:50:04.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n9bZyj+0fhWvhzfToLFIER49vjQ=', 'Content-Length': '0'} 2022-09-02T05:50:04.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f74dd45053ba53-006311998c-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:04.991 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:04.991 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:04.992 DEBUG:boto:Method: HEAD 2022-09-02T05:50:04.992 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:04.993 DEBUG:boto:Data: 2022-09-02T05:50:04.993 DEBUG:boto:Headers: {} 2022-09-02T05:50:04.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:04.994 DEBUG:boto:Port: 8000 2022-09-02T05:50:04.995 DEBUG:boto:Params: {} 2022-09-02T05:50:04.996 DEBUG:boto:Token: None 2022-09-02T05:50:04.997 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:04 GMT /xljbkf-74/obj33 2022-09-02T05:50:04.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n9bZyj+0fhWvhzfToLFIER49vjQ= 2022-09-02T05:50:04.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n9bZyj+0fhWvhzfToLFIER49vjQ=', 'Content-Length': '0'} 2022-09-02T05:50:05.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7306483f7a0c09-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:05.002 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:05.003 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:05.004 DEBUG:boto:Method: GET 2022-09-02T05:50:05.004 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:05.005 DEBUG:boto:Data: 2022-09-02T05:50:05.005 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.007 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.007 DEBUG:boto:Params: {} 2022-09-02T05:50:05.008 DEBUG:boto:Token: None 2022-09-02T05:50:05.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj33 2022-09-02T05:50:05.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+5XVwTkk9CraQYva6xh2AKFY94s= 2022-09-02T05:50:05.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+5XVwTkk9CraQYva6xh2AKFY94s=', 'Content-Length': '0'} 2022-09-02T05:50:05.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbef2a2435b6b09b-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.054 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:05.055 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:05.055 DEBUG:boto:Method: GET 2022-09-02T05:50:05.056 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:05.057 DEBUG:boto:Data: 2022-09-02T05:50:05.057 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.058 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.059 DEBUG:boto:Params: {} 2022-09-02T05:50:05.060 DEBUG:boto:Token: None 2022-09-02T05:50:05.060 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj33 2022-09-02T05:50:05.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+5XVwTkk9CraQYva6xh2AKFY94s= 2022-09-02T05:50:05.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+5XVwTkk9CraQYva6xh2AKFY94s=', 'Content-Length': '0'} 2022-09-02T05:50:05.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006770d58d23e6941a-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.066 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.067 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.067 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:05.068 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:05.068 DEBUG:boto:Method: GET 2022-09-02T05:50:05.069 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:05.070 DEBUG:boto:Data: 2022-09-02T05:50:05.070 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.071 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.071 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.072 DEBUG:boto:Params: {} 2022-09-02T05:50:05.073 DEBUG:boto:Token: None 2022-09-02T05:50:05.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5a700be37e4d11-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.078 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.079 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.080 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.081 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:05.081 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:05.082 DEBUG:boto:Method: GET 2022-09-02T05:50:05.082 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:05.092 DEBUG:boto:Data: 2022-09-02T05:50:05.093 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.095 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.097 DEBUG:boto:Params: {} 2022-09-02T05:50:05.098 DEBUG:boto:Token: None 2022-09-02T05:50:05.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b2a3e37b82164c-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.106 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:05.109 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.110 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.111 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.112 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.113 DEBUG:boto:Method: GET 2022-09-02T05:50:05.114 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.115 DEBUG:boto:Data: 2022-09-02T05:50:05.116 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.119 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.120 DEBUG:boto:Params: {} 2022-09-02T05:50:05.121 DEBUG:boto:Token: None 2022-09-02T05:50:05.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg= 2022-09-02T05:50:05.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg=', 'Content-Length': '0'} 2022-09-02T05:50:05.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097677f40ba0b72a2-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.129 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.130 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.132 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.133 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.134 DEBUG:boto:Method: GET 2022-09-02T05:50:05.135 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.136 DEBUG:boto:Data: 2022-09-02T05:50:05.137 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.138 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.138 DEBUG:boto:Params: {} 2022-09-02T05:50:05.139 DEBUG:boto:Token: None 2022-09-02T05:50:05.140 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg= 2022-09-02T05:50:05.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg=', 'Content-Length': '0'} 2022-09-02T05:50:05.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6d6152e46f61b1-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.144 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.145 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.145 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.146 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.146 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.147 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.147 DEBUG:boto:Data: 2022-09-02T05:50:05.148 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.149 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.149 DEBUG:boto:Params: {} 2022-09-02T05:50:05.150 DEBUG:boto:Token: None 2022-09-02T05:50:05.151 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nD97LQXeuTC1sHVbCKVZFLklmWM= 2022-09-02T05:50:05.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nD97LQXeuTC1sHVbCKVZFLklmWM=', 'Content-Length': '0'} 2022-09-02T05:50:05.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d733d5fe3b80844c-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.155 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.155 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.156 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.156 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.157 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.157 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.158 DEBUG:boto:Data: 2022-09-02T05:50:05.159 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.160 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.160 DEBUG:boto:Params: {} 2022-09-02T05:50:05.161 DEBUG:boto:Token: None 2022-09-02T05:50:05.162 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nD97LQXeuTC1sHVbCKVZFLklmWM= 2022-09-02T05:50:05.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nD97LQXeuTC1sHVbCKVZFLklmWM=', 'Content-Length': '0'} 2022-09-02T05:50:05.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007766fe7db801cd24-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:05.167 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.167 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.168 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.169 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.169 DEBUG:boto:Method: GET 2022-09-02T05:50:05.170 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.171 DEBUG:boto:Data: 2022-09-02T05:50:05.171 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.173 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.174 DEBUG:boto:Params: {} 2022-09-02T05:50:05.175 DEBUG:boto:Token: None 2022-09-02T05:50:05.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg= 2022-09-02T05:50:05.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg=', 'Content-Length': '0'} 2022-09-02T05:50:05.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb3ad733db29cfa-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.180 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.180 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.181 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.181 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.182 DEBUG:boto:Method: GET 2022-09-02T05:50:05.183 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.183 DEBUG:boto:Data: 2022-09-02T05:50:05.184 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.185 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.186 DEBUG:boto:Params: {} 2022-09-02T05:50:05.187 DEBUG:boto:Token: None 2022-09-02T05:50:05.188 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:05.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg= 2022-09-02T05:50:05.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dX5KYy/QdGa2gb978MX6sGtFPtg=', 'Content-Length': '0'} 2022-09-02T05:50:05.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051aa2681ace4c04b-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.195 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.195 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.196 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.197 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.198 DEBUG:boto:Method: GET 2022-09-02T05:50:05.199 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.201 DEBUG:boto:Data: 2022-09-02T05:50:05.202 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.205 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.206 DEBUG:boto:Params: {} 2022-09-02T05:50:05.208 DEBUG:boto:Token: None 2022-09-02T05:50:05.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f3a62eac78030cb-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.258 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.260 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.261 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.263 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.263 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.264 DEBUG:boto:Method: GET 2022-09-02T05:50:05.265 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.266 DEBUG:boto:Data: 2022-09-02T05:50:05.267 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.268 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.269 DEBUG:boto:Params: {} 2022-09-02T05:50:05.270 DEBUG:boto:Token: None 2022-09-02T05:50:05.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ae5d76b63aeb20-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.278 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.279 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.280 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.281 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.281 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:05.282 DEBUG:boto:Data: 2022-09-02T05:50:05.283 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.285 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.286 DEBUG:boto:Params: {} 2022-09-02T05:50:05.286 DEBUG:boto:Token: None 2022-09-02T05:50:05.287 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34 2022-09-02T05:50:05.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JKcx88xIkE1qTUoelUzUaeqZcNc= 2022-09-02T05:50:05.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JKcx88xIkE1qTUoelUzUaeqZcNc=', 'Content-Length': '0'} 2022-09-02T05:50:05.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008faeada96b5e4e6b-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.293 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.294 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.295 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.295 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:05.296 DEBUG:boto:Data: 2022-09-02T05:50:05.297 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.298 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.299 DEBUG:boto:Params: {} 2022-09-02T05:50:05.299 DEBUG:boto:Token: None 2022-09-02T05:50:05.300 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34 2022-09-02T05:50:05.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JKcx88xIkE1qTUoelUzUaeqZcNc= 2022-09-02T05:50:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JKcx88xIkE1qTUoelUzUaeqZcNc=', 'Content-Length': '0'} 2022-09-02T05:50:05.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950b5bfc655d30e9-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:05.305 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.306 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.307 DEBUG:boto:Method: GET 2022-09-02T05:50:05.308 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:05.308 DEBUG:boto:Data: 2022-09-02T05:50:05.309 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.309 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.310 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.311 DEBUG:boto:Params: {} 2022-09-02T05:50:05.311 DEBUG:boto:Token: None 2022-09-02T05:50:05.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34 2022-09-02T05:50:05.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XL/fQJ03bQF66qB9T3B29RqUs94= 2022-09-02T05:50:05.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XL/fQJ03bQF66qB9T3B29RqUs94=', 'Content-Length': '0'} 2022-09-02T05:50:05.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcb3ea881d0873b5-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.316 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:05.317 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:05.317 DEBUG:boto:Method: GET 2022-09-02T05:50:05.318 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:05.319 DEBUG:boto:Data: 2022-09-02T05:50:05.320 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.321 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.321 DEBUG:boto:Params: {} 2022-09-02T05:50:05.322 DEBUG:boto:Token: None 2022-09-02T05:50:05.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj34 2022-09-02T05:50:05.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XL/fQJ03bQF66qB9T3B29RqUs94= 2022-09-02T05:50:05.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XL/fQJ03bQF66qB9T3B29RqUs94=', 'Content-Length': '0'} 2022-09-02T05:50:05.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967e34a6156e8fe4-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.327 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.328 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.328 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.329 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.329 DEBUG:boto:Method: GET 2022-09-02T05:50:05.330 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.331 DEBUG:boto:Data: 2022-09-02T05:50:05.331 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.333 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.334 DEBUG:boto:Params: {} 2022-09-02T05:50:05.335 DEBUG:boto:Token: None 2022-09-02T05:50:05.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ac26c08997be0a4-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.341 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.342 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.343 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.344 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.345 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.346 DEBUG:boto:Method: GET 2022-09-02T05:50:05.347 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:05.347 DEBUG:boto:Data: 2022-09-02T05:50:05.348 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.349 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.350 DEBUG:boto:Params: {} 2022-09-02T05:50:05.351 DEBUG:boto:Token: None 2022-09-02T05:50:05.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9c5232573ffb25-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.402 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:05.404 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.404 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.405 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.405 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.406 DEBUG:boto:Method: GET 2022-09-02T05:50:05.406 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.407 DEBUG:boto:Data: 2022-09-02T05:50:05.407 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.408 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.409 DEBUG:boto:Params: {} 2022-09-02T05:50:05.410 DEBUG:boto:Token: None 2022-09-02T05:50:05.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U= 2022-09-02T05:50:05.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U=', 'Content-Length': '0'} 2022-09-02T05:50:05.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b18269910bbc041-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.415 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.416 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.416 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.417 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.417 DEBUG:boto:Method: GET 2022-09-02T05:50:05.418 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.418 DEBUG:boto:Data: 2022-09-02T05:50:05.419 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.420 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.421 DEBUG:boto:Params: {} 2022-09-02T05:50:05.421 DEBUG:boto:Token: None 2022-09-02T05:50:05.422 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U= 2022-09-02T05:50:05.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U=', 'Content-Length': '0'} 2022-09-02T05:50:05.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0cdf0da7dca3134-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.427 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.427 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.428 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.428 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.429 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.429 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.430 DEBUG:boto:Data: 2022-09-02T05:50:05.430 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.431 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.432 DEBUG:boto:Params: {} 2022-09-02T05:50:05.432 DEBUG:boto:Token: None 2022-09-02T05:50:05.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BhdArXowxvPGnhEWXDEJDwAWE2E= 2022-09-02T05:50:05.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BhdArXowxvPGnhEWXDEJDwAWE2E=', 'Content-Length': '0'} 2022-09-02T05:50:05.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5f6e415d0226b5-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.437 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.438 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.438 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.439 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.440 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.440 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.441 DEBUG:boto:Data: 2022-09-02T05:50:05.441 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.442 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.442 DEBUG:boto:Params: {} 2022-09-02T05:50:05.443 DEBUG:boto:Token: None 2022-09-02T05:50:05.444 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BhdArXowxvPGnhEWXDEJDwAWE2E= 2022-09-02T05:50:05.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BhdArXowxvPGnhEWXDEJDwAWE2E=', 'Content-Length': '0'} 2022-09-02T05:50:05.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e659e3121531979e-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:05.449 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.449 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.450 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.450 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.451 DEBUG:boto:Method: GET 2022-09-02T05:50:05.451 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.452 DEBUG:boto:Data: 2022-09-02T05:50:05.452 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.453 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.453 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.454 DEBUG:boto:Params: {} 2022-09-02T05:50:05.455 DEBUG:boto:Token: None 2022-09-02T05:50:05.455 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.456 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U= 2022-09-02T05:50:05.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U=', 'Content-Length': '0'} 2022-09-02T05:50:05.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a2888df4dc6e94-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.459 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.460 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.461 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.461 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.469 DEBUG:boto:Method: GET 2022-09-02T05:50:05.470 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.470 DEBUG:boto:Data: 2022-09-02T05:50:05.471 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.472 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.472 DEBUG:boto:Params: {} 2022-09-02T05:50:05.473 DEBUG:boto:Token: None 2022-09-02T05:50:05.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:05.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U= 2022-09-02T05:50:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h+CdO06Z+9hqB78wsa7ife3hc9U=', 'Content-Length': '0'} 2022-09-02T05:50:05.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d4e2c9bd9a6bb4-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.478 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.478 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.479 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.480 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.480 DEBUG:boto:Method: GET 2022-09-02T05:50:05.481 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.481 DEBUG:boto:Data: 2022-09-02T05:50:05.482 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.483 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.483 DEBUG:boto:Params: {} 2022-09-02T05:50:05.484 DEBUG:boto:Token: None 2022-09-02T05:50:05.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc5d53bc34db57b-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.534 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.535 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.536 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.537 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.537 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.538 DEBUG:boto:Method: GET 2022-09-02T05:50:05.538 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.539 DEBUG:boto:Data: 2022-09-02T05:50:05.539 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.540 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.541 DEBUG:boto:Params: {} 2022-09-02T05:50:05.542 DEBUG:boto:Token: None 2022-09-02T05:50:05.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003739d3d78e1f3ebc-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.549 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.550 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.551 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.552 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.553 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:05.554 DEBUG:boto:Data: 2022-09-02T05:50:05.555 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.557 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.558 DEBUG:boto:Params: {} 2022-09-02T05:50:05.559 DEBUG:boto:Token: None 2022-09-02T05:50:05.560 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38 2022-09-02T05:50:05.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYX6kgMzC+qDn3h0HV84ayBEpXg= 2022-09-02T05:50:05.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYX6kgMzC+qDn3h0HV84ayBEpXg=', 'Content-Length': '0'} 2022-09-02T05:50:05.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e7effd90dccd17-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.566 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.567 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.568 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.569 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:05.569 DEBUG:boto:Data: 2022-09-02T05:50:05.570 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.571 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.571 DEBUG:boto:Params: {} 2022-09-02T05:50:05.572 DEBUG:boto:Token: None 2022-09-02T05:50:05.573 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38 2022-09-02T05:50:05.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nYX6kgMzC+qDn3h0HV84ayBEpXg= 2022-09-02T05:50:05.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nYX6kgMzC+qDn3h0HV84ayBEpXg=', 'Content-Length': '0'} 2022-09-02T05:50:05.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db3da4b9eb46b46-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:05.583 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.583 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.584 DEBUG:boto:Method: GET 2022-09-02T05:50:05.585 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:05.585 DEBUG:boto:Data: 2022-09-02T05:50:05.586 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.587 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.587 DEBUG:boto:Params: {} 2022-09-02T05:50:05.588 DEBUG:boto:Token: None 2022-09-02T05:50:05.588 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38 2022-09-02T05:50:05.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GQU/tXbi4TKzno2E6fAYzJCFkOo= 2022-09-02T05:50:05.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GQU/tXbi4TKzno2E6fAYzJCFkOo=', 'Content-Length': '0'} 2022-09-02T05:50:05.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31ac6b0b96d6b31-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.593 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:05.594 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:05.595 DEBUG:boto:Method: GET 2022-09-02T05:50:05.595 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:05.596 DEBUG:boto:Data: 2022-09-02T05:50:05.596 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.597 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.598 DEBUG:boto:Params: {} 2022-09-02T05:50:05.599 DEBUG:boto:Token: None 2022-09-02T05:50:05.600 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj38 2022-09-02T05:50:05.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GQU/tXbi4TKzno2E6fAYzJCFkOo= 2022-09-02T05:50:05.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GQU/tXbi4TKzno2E6fAYzJCFkOo=', 'Content-Length': '0'} 2022-09-02T05:50:05.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1626eef453bb4ef-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.646 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.648 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.649 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.650 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.651 DEBUG:boto:Method: GET 2022-09-02T05:50:05.652 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.653 DEBUG:boto:Data: 2022-09-02T05:50:05.654 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.656 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.657 DEBUG:boto:Params: {} 2022-09-02T05:50:05.658 DEBUG:boto:Token: None 2022-09-02T05:50:05.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ddf82eb02c445e-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.665 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.665 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.666 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.666 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.667 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.668 DEBUG:boto:Method: GET 2022-09-02T05:50:05.668 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:05.669 DEBUG:boto:Data: 2022-09-02T05:50:05.669 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.670 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.671 DEBUG:boto:Params: {} 2022-09-02T05:50:05.671 DEBUG:boto:Token: None 2022-09-02T05:50:05.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbca45698a007eee-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.677 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:05.678 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.679 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.679 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.680 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.680 DEBUG:boto:Method: GET 2022-09-02T05:50:05.681 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.682 DEBUG:boto:Data: 2022-09-02T05:50:05.682 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.683 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.683 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.684 DEBUG:boto:Params: {} 2022-09-02T05:50:05.684 DEBUG:boto:Token: None 2022-09-02T05:50:05.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg= 2022-09-02T05:50:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg=', 'Content-Length': '0'} 2022-09-02T05:50:05.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d07030b5aa90cd-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.689 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.690 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.690 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.691 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.691 DEBUG:boto:Method: GET 2022-09-02T05:50:05.692 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.692 DEBUG:boto:Data: 2022-09-02T05:50:05.693 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.694 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.695 DEBUG:boto:Params: {} 2022-09-02T05:50:05.696 DEBUG:boto:Token: None 2022-09-02T05:50:05.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg= 2022-09-02T05:50:05.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg=', 'Content-Length': '0'} 2022-09-02T05:50:05.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff698d8d57f62b86-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.701 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.701 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.702 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.702 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.703 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.703 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.704 DEBUG:boto:Data: 2022-09-02T05:50:05.704 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.705 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.705 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.706 DEBUG:boto:Params: {} 2022-09-02T05:50:05.706 DEBUG:boto:Token: None 2022-09-02T05:50:05.707 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KLuoWPggmMrWKaTDqni8zDtW/Lk= 2022-09-02T05:50:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KLuoWPggmMrWKaTDqni8zDtW/Lk=', 'Content-Length': '0'} 2022-09-02T05:50:05.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6874217f9ef31f2-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.711 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.711 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.712 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.712 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.713 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.713 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.714 DEBUG:boto:Data: 2022-09-02T05:50:05.715 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.716 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.716 DEBUG:boto:Params: {} 2022-09-02T05:50:05.717 DEBUG:boto:Token: None 2022-09-02T05:50:05.717 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KLuoWPggmMrWKaTDqni8zDtW/Lk= 2022-09-02T05:50:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KLuoWPggmMrWKaTDqni8zDtW/Lk=', 'Content-Length': '0'} 2022-09-02T05:50:05.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006207e3955fbeb4dd-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:05.722 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.722 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.723 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.723 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.724 DEBUG:boto:Method: GET 2022-09-02T05:50:05.725 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.725 DEBUG:boto:Data: 2022-09-02T05:50:05.726 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.727 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.727 DEBUG:boto:Params: {} 2022-09-02T05:50:05.728 DEBUG:boto:Token: None 2022-09-02T05:50:05.728 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.729 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg= 2022-09-02T05:50:05.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg=', 'Content-Length': '0'} 2022-09-02T05:50:05.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000412e92592c5b222e-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.733 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.733 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.734 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.734 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.735 DEBUG:boto:Method: GET 2022-09-02T05:50:05.735 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.736 DEBUG:boto:Data: 2022-09-02T05:50:05.736 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.737 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.738 DEBUG:boto:Params: {} 2022-09-02T05:50:05.739 DEBUG:boto:Token: None 2022-09-02T05:50:05.739 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:05.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg= 2022-09-02T05:50:05.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c2GEkNp3h/dZ0bIeFl6LeU1pnqg=', 'Content-Length': '0'} 2022-09-02T05:50:05.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360e3be968c688bd-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.743 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.744 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.744 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.745 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.745 DEBUG:boto:Method: GET 2022-09-02T05:50:05.746 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.747 DEBUG:boto:Data: 2022-09-02T05:50:05.747 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.749 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.749 DEBUG:boto:Params: {} 2022-09-02T05:50:05.750 DEBUG:boto:Token: None 2022-09-02T05:50:05.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9336029d312154c-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.755 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.756 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.757 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.757 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.758 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.758 DEBUG:boto:Method: GET 2022-09-02T05:50:05.759 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.759 DEBUG:boto:Data: 2022-09-02T05:50:05.760 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.761 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.762 DEBUG:boto:Params: {} 2022-09-02T05:50:05.762 DEBUG:boto:Token: None 2022-09-02T05:50:05.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf44ab5a6d8ba90a-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.768 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.769 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.769 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.770 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.770 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:05.771 DEBUG:boto:Data: 2022-09-02T05:50:05.771 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.772 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.773 DEBUG:boto:Params: {} 2022-09-02T05:50:05.773 DEBUG:boto:Token: None 2022-09-02T05:50:05.774 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44 2022-09-02T05:50:05.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n93+3m3o3FTv4wCoW3m1Nwh0IQk= 2022-09-02T05:50:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n93+3m3o3FTv4wCoW3m1Nwh0IQk=', 'Content-Length': '0'} 2022-09-02T05:50:05.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003057cbd9e6b22fe6-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.778 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.779 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.779 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.780 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:05.780 DEBUG:boto:Data: 2022-09-02T05:50:05.781 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.782 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.782 DEBUG:boto:Params: {} 2022-09-02T05:50:05.783 DEBUG:boto:Token: None 2022-09-02T05:50:05.783 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44 2022-09-02T05:50:05.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n93+3m3o3FTv4wCoW3m1Nwh0IQk= 2022-09-02T05:50:05.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n93+3m3o3FTv4wCoW3m1Nwh0IQk=', 'Content-Length': '0'} 2022-09-02T05:50:05.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004068a34329cbc307-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:05.788 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.789 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.789 DEBUG:boto:Method: GET 2022-09-02T05:50:05.790 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:05.791 DEBUG:boto:Data: 2022-09-02T05:50:05.792 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.794 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.794 DEBUG:boto:Params: {} 2022-09-02T05:50:05.795 DEBUG:boto:Token: None 2022-09-02T05:50:05.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44 2022-09-02T05:50:05.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i4jYTyICr0ZvSQ4h9WyORcjys54= 2022-09-02T05:50:05.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i4jYTyICr0ZvSQ4h9WyORcjys54=', 'Content-Length': '0'} 2022-09-02T05:50:05.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067522890e8839f23-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.842 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:05.843 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:05.843 DEBUG:boto:Method: GET 2022-09-02T05:50:05.844 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:05.845 DEBUG:boto:Data: 2022-09-02T05:50:05.845 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.846 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.847 DEBUG:boto:Params: {} 2022-09-02T05:50:05.847 DEBUG:boto:Token: None 2022-09-02T05:50:05.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj44 2022-09-02T05:50:05.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i4jYTyICr0ZvSQ4h9WyORcjys54= 2022-09-02T05:50:05.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i4jYTyICr0ZvSQ4h9WyORcjys54=', 'Content-Length': '0'} 2022-09-02T05:50:05.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003feadf22aa142dfa-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.852 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.853 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.853 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.854 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.854 DEBUG:boto:Method: GET 2022-09-02T05:50:05.855 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.855 DEBUG:boto:Data: 2022-09-02T05:50:05.856 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.857 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.857 DEBUG:boto:Params: {} 2022-09-02T05:50:05.858 DEBUG:boto:Token: None 2022-09-02T05:50:05.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1272b47407be2aa-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.863 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.864 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.865 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.865 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.866 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.867 DEBUG:boto:Method: GET 2022-09-02T05:50:05.867 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:05.868 DEBUG:boto:Data: 2022-09-02T05:50:05.868 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.869 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.870 DEBUG:boto:Params: {} 2022-09-02T05:50:05.870 DEBUG:boto:Token: None 2022-09-02T05:50:05.871 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a29f266bf2acd191-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.876 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:05.877 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:05.878 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:05.878 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.879 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.879 DEBUG:boto:Method: GET 2022-09-02T05:50:05.880 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.881 DEBUG:boto:Data: 2022-09-02T05:50:05.881 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.882 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.883 DEBUG:boto:Params: {} 2022-09-02T05:50:05.883 DEBUG:boto:Token: None 2022-09-02T05:50:05.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g= 2022-09-02T05:50:05.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g=', 'Content-Length': '0'} 2022-09-02T05:50:05.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9ef57f6796a92b-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.888 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:05.889 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:05.889 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.890 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.890 DEBUG:boto:Method: GET 2022-09-02T05:50:05.891 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.891 DEBUG:boto:Data: 2022-09-02T05:50:05.892 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.893 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.893 DEBUG:boto:Params: {} 2022-09-02T05:50:05.894 DEBUG:boto:Token: None 2022-09-02T05:50:05.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g= 2022-09-02T05:50:05.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g=', 'Content-Length': '0'} 2022-09-02T05:50:05.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6272fefb4df7a00-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.899 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:05.899 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:05.900 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.900 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.901 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.902 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.902 DEBUG:boto:Data: 2022-09-02T05:50:05.903 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.904 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.904 DEBUG:boto:Params: {} 2022-09-02T05:50:05.905 DEBUG:boto:Token: None 2022-09-02T05:50:05.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h1Bufb4GsMlukW7bvjK0n6qLMGA= 2022-09-02T05:50:05.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h1Bufb4GsMlukW7bvjK0n6qLMGA=', 'Content-Length': '0'} 2022-09-02T05:50:05.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001013327d786322ea-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.909 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:05.910 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:05.911 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.911 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.912 DEBUG:boto:Method: HEAD 2022-09-02T05:50:05.912 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.913 DEBUG:boto:Data: 2022-09-02T05:50:05.913 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.914 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.915 DEBUG:boto:Params: {} 2022-09-02T05:50:05.915 DEBUG:boto:Token: None 2022-09-02T05:50:05.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h1Bufb4GsMlukW7bvjK0n6qLMGA= 2022-09-02T05:50:05.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h1Bufb4GsMlukW7bvjK0n6qLMGA=', 'Content-Length': '0'} 2022-09-02T05:50:05.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f346414045b6a9d1-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:05.921 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:05.921 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:05.922 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.922 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.923 DEBUG:boto:Method: GET 2022-09-02T05:50:05.924 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.924 DEBUG:boto:Data: 2022-09-02T05:50:05.925 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.926 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.926 DEBUG:boto:Params: {} 2022-09-02T05:50:05.927 DEBUG:boto:Token: None 2022-09-02T05:50:05.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g= 2022-09-02T05:50:05.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g=', 'Content-Length': '0'} 2022-09-02T05:50:05.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afaeeab2caac3223-006311998d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.932 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:05.932 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:05.933 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.933 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.934 DEBUG:boto:Method: GET 2022-09-02T05:50:05.935 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.935 DEBUG:boto:Data: 2022-09-02T05:50:05.936 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.937 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.937 DEBUG:boto:Params: {} 2022-09-02T05:50:05.938 DEBUG:boto:Token: None 2022-09-02T05:50:05.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:05.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g= 2022-09-02T05:50:05.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4AK2PnQOlEr3Rl5o75f1eRVM68g=', 'Content-Length': '0'} 2022-09-02T05:50:05.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2a1b844643e109-006311998d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.943 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.943 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.944 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:05.945 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:05.945 DEBUG:boto:Method: GET 2022-09-02T05:50:05.946 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:05.946 DEBUG:boto:Data: 2022-09-02T05:50:05.947 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:05.948 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.949 DEBUG:boto:Params: {} 2022-09-02T05:50:05.949 DEBUG:boto:Token: None 2022-09-02T05:50:05.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d8ff6d5172ab70-006311998d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:05.955 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:05.955 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:05.956 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:05.957 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:05.957 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:05.958 DEBUG:boto:Method: GET 2022-09-02T05:50:05.958 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:05.959 DEBUG:boto:Data: 2022-09-02T05:50:05.960 DEBUG:boto:Headers: {} 2022-09-02T05:50:05.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:05.962 DEBUG:boto:Port: 8000 2022-09-02T05:50:05.963 DEBUG:boto:Params: {} 2022-09-02T05:50:05.964 DEBUG:boto:Token: None 2022-09-02T05:50:05.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:05 GMT /xljbkf-74/ 2022-09-02T05:50:05.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk= 2022-09-02T05:50:05.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fE83RXVpv6kyJ/xeVwG0ZJJcUXk=', 'Content-Length': '0'} 2022-09-02T05:50:05.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d21ba852e3903d-006311998d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.014 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.015 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:06.016 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:06.017 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.017 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:06.018 DEBUG:boto:Data: 2022-09-02T05:50:06.018 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.020 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.020 DEBUG:boto:Params: {} 2022-09-02T05:50:06.021 DEBUG:boto:Token: None 2022-09-02T05:50:06.022 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj51 2022-09-02T05:50:06.022 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YuM552mmJSOchl+JISWcLmh48Co= 2022-09-02T05:50:06.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YuM552mmJSOchl+JISWcLmh48Co=', 'Content-Length': '0'} 2022-09-02T05:50:06.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017cc3c26e3d1b255-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.026 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:06.027 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:06.028 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.030 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:06.031 DEBUG:boto:Data: 2022-09-02T05:50:06.032 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.034 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.035 DEBUG:boto:Params: {} 2022-09-02T05:50:06.036 DEBUG:boto:Token: None 2022-09-02T05:50:06.037 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj51 2022-09-02T05:50:06.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YuM552mmJSOchl+JISWcLmh48Co= 2022-09-02T05:50:06.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YuM552mmJSOchl+JISWcLmh48Co=', 'Content-Length': '0'} 2022-09-02T05:50:06.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada9336c483be089-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:06.047 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:06.048 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:06.049 DEBUG:boto:Method: GET 2022-09-02T05:50:06.050 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:06.051 DEBUG:boto:Data: 2022-09-02T05:50:06.052 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.054 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.055 DEBUG:boto:Params: {} 2022-09-02T05:50:06.055 DEBUG:boto:Token: None 2022-09-02T05:50:06.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj51 2022-09-02T05:50:06.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7xddE/WY37jwtn4EjpxOZDesFms= 2022-09-02T05:50:06.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7xddE/WY37jwtn4EjpxOZDesFms=', 'Content-Length': '0'} 2022-09-02T05:50:06.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e15aa2528567afa-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.060 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:06.060 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:06.061 DEBUG:boto:Method: GET 2022-09-02T05:50:06.062 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:06.062 DEBUG:boto:Data: 2022-09-02T05:50:06.063 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.064 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.064 DEBUG:boto:Params: {} 2022-09-02T05:50:06.065 DEBUG:boto:Token: None 2022-09-02T05:50:06.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj51 2022-09-02T05:50:06.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7xddE/WY37jwtn4EjpxOZDesFms= 2022-09-02T05:50:06.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7xddE/WY37jwtn4EjpxOZDesFms=', 'Content-Length': '0'} 2022-09-02T05:50:06.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db33a2b44932d702-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.072 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.073 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.074 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:06.075 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:06.075 DEBUG:boto:Method: GET 2022-09-02T05:50:06.076 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:06.076 DEBUG:boto:Data: 2022-09-02T05:50:06.077 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.078 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.078 DEBUG:boto:Params: {} 2022-09-02T05:50:06.079 DEBUG:boto:Token: None 2022-09-02T05:50:06.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738cf52508328a76-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.085 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.086 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.087 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.087 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:06.088 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:06.089 DEBUG:boto:Method: GET 2022-09-02T05:50:06.089 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:06.090 DEBUG:boto:Data: 2022-09-02T05:50:06.090 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.091 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.092 DEBUG:boto:Params: {} 2022-09-02T05:50:06.092 DEBUG:boto:Token: None 2022-09-02T05:50:06.093 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ed4647ef8d467c2-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.098 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:06.100 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.100 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.101 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.101 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.102 DEBUG:boto:Method: GET 2022-09-02T05:50:06.103 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.104 DEBUG:boto:Data: 2022-09-02T05:50:06.104 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.105 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.106 DEBUG:boto:Params: {} 2022-09-02T05:50:06.107 DEBUG:boto:Token: None 2022-09-02T05:50:06.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog= 2022-09-02T05:50:06.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog=', 'Content-Length': '0'} 2022-09-02T05:50:06.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9be93bcb6e84c9-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.112 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.112 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.113 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.113 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.114 DEBUG:boto:Method: GET 2022-09-02T05:50:06.115 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.115 DEBUG:boto:Data: 2022-09-02T05:50:06.116 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.117 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.117 DEBUG:boto:Params: {} 2022-09-02T05:50:06.118 DEBUG:boto:Token: None 2022-09-02T05:50:06.119 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog= 2022-09-02T05:50:06.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog=', 'Content-Length': '0'} 2022-09-02T05:50:06.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9854536332edd1-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.123 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.124 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.124 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.125 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.125 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.126 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.126 DEBUG:boto:Data: 2022-09-02T05:50:06.127 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.128 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.128 DEBUG:boto:Params: {} 2022-09-02T05:50:06.129 DEBUG:boto:Token: None 2022-09-02T05:50:06.129 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lnYcIfkFMZeqr6eJhSEGMgS4C8I= 2022-09-02T05:50:06.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lnYcIfkFMZeqr6eJhSEGMgS4C8I=', 'Content-Length': '0'} 2022-09-02T05:50:06.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fc51a2a8b5df3ab-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.133 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.134 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.135 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.135 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.136 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.136 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.137 DEBUG:boto:Data: 2022-09-02T05:50:06.137 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.138 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.139 DEBUG:boto:Params: {} 2022-09-02T05:50:06.139 DEBUG:boto:Token: None 2022-09-02T05:50:06.140 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lnYcIfkFMZeqr6eJhSEGMgS4C8I= 2022-09-02T05:50:06.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lnYcIfkFMZeqr6eJhSEGMgS4C8I=', 'Content-Length': '0'} 2022-09-02T05:50:06.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed159f7c991c133-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:06.145 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.145 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.146 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.146 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.147 DEBUG:boto:Method: GET 2022-09-02T05:50:06.147 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.148 DEBUG:boto:Data: 2022-09-02T05:50:06.148 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.149 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.150 DEBUG:boto:Params: {} 2022-09-02T05:50:06.151 DEBUG:boto:Token: None 2022-09-02T05:50:06.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog= 2022-09-02T05:50:06.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog=', 'Content-Length': '0'} 2022-09-02T05:50:06.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0227be36f60b0a7-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.198 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.199 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.199 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.200 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.201 DEBUG:boto:Method: GET 2022-09-02T05:50:06.201 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.202 DEBUG:boto:Data: 2022-09-02T05:50:06.202 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.203 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.204 DEBUG:boto:Params: {} 2022-09-02T05:50:06.205 DEBUG:boto:Token: None 2022-09-02T05:50:06.205 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:06.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog= 2022-09-02T05:50:06.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+NR9N94k8PSuRi/BRo/Bt90jog=', 'Content-Length': '0'} 2022-09-02T05:50:06.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f3adc3745dfd73-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.210 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.211 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.212 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.213 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.214 DEBUG:boto:Method: GET 2022-09-02T05:50:06.215 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.216 DEBUG:boto:Data: 2022-09-02T05:50:06.217 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.218 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.219 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.220 DEBUG:boto:Params: {} 2022-09-02T05:50:06.221 DEBUG:boto:Token: None 2022-09-02T05:50:06.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da815433577c2201-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.228 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.229 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.230 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.230 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.231 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.231 DEBUG:boto:Method: GET 2022-09-02T05:50:06.232 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.232 DEBUG:boto:Data: 2022-09-02T05:50:06.233 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.234 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.234 DEBUG:boto:Params: {} 2022-09-02T05:50:06.235 DEBUG:boto:Token: None 2022-09-02T05:50:06.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000772db78e5826bd2e-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.241 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.242 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.242 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.243 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.243 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:06.244 DEBUG:boto:Data: 2022-09-02T05:50:06.245 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.246 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.246 DEBUG:boto:Params: {} 2022-09-02T05:50:06.247 DEBUG:boto:Token: None 2022-09-02T05:50:06.247 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55 2022-09-02T05:50:06.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w0P9yPcD1Ga7c9iNhLjgXWnoPlQ= 2022-09-02T05:50:06.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w0P9yPcD1Ga7c9iNhLjgXWnoPlQ=', 'Content-Length': '0'} 2022-09-02T05:50:06.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f88bc8d5603adde-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.252 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.253 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.253 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.254 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:06.255 DEBUG:boto:Data: 2022-09-02T05:50:06.255 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.256 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.257 DEBUG:boto:Params: {} 2022-09-02T05:50:06.258 DEBUG:boto:Token: None 2022-09-02T05:50:06.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55 2022-09-02T05:50:06.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w0P9yPcD1Ga7c9iNhLjgXWnoPlQ= 2022-09-02T05:50:06.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w0P9yPcD1Ga7c9iNhLjgXWnoPlQ=', 'Content-Length': '0'} 2022-09-02T05:50:06.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed32d26d9286051-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:06.263 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.264 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.264 DEBUG:boto:Method: GET 2022-09-02T05:50:06.265 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:06.265 DEBUG:boto:Data: 2022-09-02T05:50:06.266 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.267 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.268 DEBUG:boto:Params: {} 2022-09-02T05:50:06.268 DEBUG:boto:Token: None 2022-09-02T05:50:06.269 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55 2022-09-02T05:50:06.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zqPYTkDwPv8I84GH/WLIM1FURos= 2022-09-02T05:50:06.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zqPYTkDwPv8I84GH/WLIM1FURos=', 'Content-Length': '0'} 2022-09-02T05:50:06.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b3047dcb0c2fc4-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.273 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:06.274 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:06.274 DEBUG:boto:Method: GET 2022-09-02T05:50:06.275 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:06.275 DEBUG:boto:Data: 2022-09-02T05:50:06.276 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.278 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.279 DEBUG:boto:Params: {} 2022-09-02T05:50:06.280 DEBUG:boto:Token: None 2022-09-02T05:50:06.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj55 2022-09-02T05:50:06.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zqPYTkDwPv8I84GH/WLIM1FURos= 2022-09-02T05:50:06.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zqPYTkDwPv8I84GH/WLIM1FURos=', 'Content-Length': '0'} 2022-09-02T05:50:06.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b390a94aefe8ec-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.285 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.285 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.286 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.287 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.288 DEBUG:boto:Method: GET 2022-09-02T05:50:06.289 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.290 DEBUG:boto:Data: 2022-09-02T05:50:06.291 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.293 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.294 DEBUG:boto:Params: {} 2022-09-02T05:50:06.295 DEBUG:boto:Token: None 2022-09-02T05:50:06.296 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.297 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003705ec856fbf6ad3-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.303 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.305 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.306 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.307 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.308 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.309 DEBUG:boto:Method: GET 2022-09-02T05:50:06.310 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:06.311 DEBUG:boto:Data: 2022-09-02T05:50:06.312 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.314 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.315 DEBUG:boto:Params: {} 2022-09-02T05:50:06.316 DEBUG:boto:Token: None 2022-09-02T05:50:06.317 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f913557d6459c066-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.325 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:06.327 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.328 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.329 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.330 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.331 DEBUG:boto:Method: GET 2022-09-02T05:50:06.332 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.333 DEBUG:boto:Data: 2022-09-02T05:50:06.334 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.336 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.337 DEBUG:boto:Params: {} 2022-09-02T05:50:06.338 DEBUG:boto:Token: None 2022-09-02T05:50:06.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM= 2022-09-02T05:50:06.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM=', 'Content-Length': '0'} 2022-09-02T05:50:06.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ee7fa250431a9e-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.386 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.387 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.387 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.388 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.388 DEBUG:boto:Method: GET 2022-09-02T05:50:06.389 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.390 DEBUG:boto:Data: 2022-09-02T05:50:06.390 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.392 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.392 DEBUG:boto:Params: {} 2022-09-02T05:50:06.393 DEBUG:boto:Token: None 2022-09-02T05:50:06.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM= 2022-09-02T05:50:06.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM=', 'Content-Length': '0'} 2022-09-02T05:50:06.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17bd25fea75571d-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.400 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.401 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.402 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.403 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.404 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.405 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.406 DEBUG:boto:Data: 2022-09-02T05:50:06.407 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.409 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.409 DEBUG:boto:Params: {} 2022-09-02T05:50:06.411 DEBUG:boto:Token: None 2022-09-02T05:50:06.412 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W09NJPT4e1ccZvJmc8oVrfCaq5g= 2022-09-02T05:50:06.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W09NJPT4e1ccZvJmc8oVrfCaq5g=', 'Content-Length': '0'} 2022-09-02T05:50:06.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001685e88a08b42ea4-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.418 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.419 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.419 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.420 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.420 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.421 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.421 DEBUG:boto:Data: 2022-09-02T05:50:06.422 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.423 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.423 DEBUG:boto:Params: {} 2022-09-02T05:50:06.424 DEBUG:boto:Token: None 2022-09-02T05:50:06.424 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W09NJPT4e1ccZvJmc8oVrfCaq5g= 2022-09-02T05:50:06.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W09NJPT4e1ccZvJmc8oVrfCaq5g=', 'Content-Length': '0'} 2022-09-02T05:50:06.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001826325d86d0c1f2-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:06.429 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.430 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.430 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.431 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.431 DEBUG:boto:Method: GET 2022-09-02T05:50:06.432 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.432 DEBUG:boto:Data: 2022-09-02T05:50:06.433 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.434 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.435 DEBUG:boto:Params: {} 2022-09-02T05:50:06.435 DEBUG:boto:Token: None 2022-09-02T05:50:06.436 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM= 2022-09-02T05:50:06.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM=', 'Content-Length': '0'} 2022-09-02T05:50:06.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003159e6cd451895d2-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.440 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.441 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.441 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.442 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.442 DEBUG:boto:Method: GET 2022-09-02T05:50:06.443 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.444 DEBUG:boto:Data: 2022-09-02T05:50:06.444 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.446 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.446 DEBUG:boto:Params: {} 2022-09-02T05:50:06.447 DEBUG:boto:Token: None 2022-09-02T05:50:06.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:06.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM= 2022-09-02T05:50:06.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7C9s9BGS5QpjyJnhW3FyXQewTyM=', 'Content-Length': '0'} 2022-09-02T05:50:06.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6c39e236eb7b1b-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.452 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.452 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.453 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.453 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.454 DEBUG:boto:Method: GET 2022-09-02T05:50:06.455 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.455 DEBUG:boto:Data: 2022-09-02T05:50:06.456 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.457 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.457 DEBUG:boto:Params: {} 2022-09-02T05:50:06.458 DEBUG:boto:Token: None 2022-09-02T05:50:06.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1af0cf3503cca46-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.463 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.464 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.465 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.465 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.466 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.466 DEBUG:boto:Method: GET 2022-09-02T05:50:06.467 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.467 DEBUG:boto:Data: 2022-09-02T05:50:06.468 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.469 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.469 DEBUG:boto:Params: {} 2022-09-02T05:50:06.470 DEBUG:boto:Token: None 2022-09-02T05:50:06.471 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b69d9b4f702649-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.476 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.477 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.477 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.478 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.478 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:06.479 DEBUG:boto:Data: 2022-09-02T05:50:06.479 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.480 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.480 DEBUG:boto:Params: {} 2022-09-02T05:50:06.481 DEBUG:boto:Token: None 2022-09-02T05:50:06.482 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57 2022-09-02T05:50:06.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fGw9aj6ZTeIzZUndUEMEFgndLqs= 2022-09-02T05:50:06.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fGw9aj6ZTeIzZUndUEMEFgndLqs=', 'Content-Length': '0'} 2022-09-02T05:50:06.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fdb913030c43c42-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.486 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.486 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.487 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.487 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:06.488 DEBUG:boto:Data: 2022-09-02T05:50:06.488 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.489 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.490 DEBUG:boto:Params: {} 2022-09-02T05:50:06.490 DEBUG:boto:Token: None 2022-09-02T05:50:06.491 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57 2022-09-02T05:50:06.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fGw9aj6ZTeIzZUndUEMEFgndLqs= 2022-09-02T05:50:06.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fGw9aj6ZTeIzZUndUEMEFgndLqs=', 'Content-Length': '0'} 2022-09-02T05:50:06.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0427da8f9c17dc-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:06.495 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.496 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.496 DEBUG:boto:Method: GET 2022-09-02T05:50:06.497 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:06.497 DEBUG:boto:Data: 2022-09-02T05:50:06.498 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.498 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.499 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.499 DEBUG:boto:Params: {} 2022-09-02T05:50:06.500 DEBUG:boto:Token: None 2022-09-02T05:50:06.500 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57 2022-09-02T05:50:06.501 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gGlnfyGdYGPvnUEtV55wm78LzY= 2022-09-02T05:50:06.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gGlnfyGdYGPvnUEtV55wm78LzY=', 'Content-Length': '0'} 2022-09-02T05:50:06.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14030af957baad3-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.504 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:06.505 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:06.505 DEBUG:boto:Method: GET 2022-09-02T05:50:06.506 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:06.506 DEBUG:boto:Data: 2022-09-02T05:50:06.507 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.508 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.508 DEBUG:boto:Params: {} 2022-09-02T05:50:06.509 DEBUG:boto:Token: None 2022-09-02T05:50:06.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj57 2022-09-02T05:50:06.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gGlnfyGdYGPvnUEtV55wm78LzY= 2022-09-02T05:50:06.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gGlnfyGdYGPvnUEtV55wm78LzY=', 'Content-Length': '0'} 2022-09-02T05:50:06.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099aa5f439a01bed2-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.554 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.555 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.555 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.556 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.557 DEBUG:boto:Method: GET 2022-09-02T05:50:06.557 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.558 DEBUG:boto:Data: 2022-09-02T05:50:06.558 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.559 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.560 DEBUG:boto:Params: {} 2022-09-02T05:50:06.561 DEBUG:boto:Token: None 2022-09-02T05:50:06.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef46cfe766df56fd-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.566 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.567 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.567 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.568 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.569 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.569 DEBUG:boto:Method: GET 2022-09-02T05:50:06.570 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:06.570 DEBUG:boto:Data: 2022-09-02T05:50:06.571 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.572 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.572 DEBUG:boto:Params: {} 2022-09-02T05:50:06.573 DEBUG:boto:Token: None 2022-09-02T05:50:06.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072c0dcac5cadcd50-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.579 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:06.580 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.581 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.581 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.582 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.582 DEBUG:boto:Method: GET 2022-09-02T05:50:06.583 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.583 DEBUG:boto:Data: 2022-09-02T05:50:06.584 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.585 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.586 DEBUG:boto:Params: {} 2022-09-02T05:50:06.586 DEBUG:boto:Token: None 2022-09-02T05:50:06.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4= 2022-09-02T05:50:06.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4=', 'Content-Length': '0'} 2022-09-02T05:50:06.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003efdcc869506f7ef-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.591 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.592 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.592 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.593 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.593 DEBUG:boto:Method: GET 2022-09-02T05:50:06.594 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.594 DEBUG:boto:Data: 2022-09-02T05:50:06.595 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.596 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.596 DEBUG:boto:Params: {} 2022-09-02T05:50:06.597 DEBUG:boto:Token: None 2022-09-02T05:50:06.598 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4= 2022-09-02T05:50:06.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4=', 'Content-Length': '0'} 2022-09-02T05:50:06.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f29a4f731909ae-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.602 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.602 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.603 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.603 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.604 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.604 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.605 DEBUG:boto:Data: 2022-09-02T05:50:06.605 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.606 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.607 DEBUG:boto:Params: {} 2022-09-02T05:50:06.607 DEBUG:boto:Token: None 2022-09-02T05:50:06.608 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yWV6Lj4PLl3+WiPopdwYo2nyV+o= 2022-09-02T05:50:06.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yWV6Lj4PLl3+WiPopdwYo2nyV+o=', 'Content-Length': '0'} 2022-09-02T05:50:06.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fca035eb01f0921-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.612 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.612 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.613 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.613 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.614 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.614 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.615 DEBUG:boto:Data: 2022-09-02T05:50:06.615 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.616 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.617 DEBUG:boto:Params: {} 2022-09-02T05:50:06.617 DEBUG:boto:Token: None 2022-09-02T05:50:06.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yWV6Lj4PLl3+WiPopdwYo2nyV+o= 2022-09-02T05:50:06.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yWV6Lj4PLl3+WiPopdwYo2nyV+o=', 'Content-Length': '0'} 2022-09-02T05:50:06.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b56ca2a90c4a2d-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:06.623 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.623 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.624 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.624 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.625 DEBUG:boto:Method: GET 2022-09-02T05:50:06.625 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.626 DEBUG:boto:Data: 2022-09-02T05:50:06.626 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.627 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.628 DEBUG:boto:Params: {} 2022-09-02T05:50:06.628 DEBUG:boto:Token: None 2022-09-02T05:50:06.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4= 2022-09-02T05:50:06.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4=', 'Content-Length': '0'} 2022-09-02T05:50:06.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196a532d414caf5f-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.674 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.675 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.676 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.676 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.677 DEBUG:boto:Method: GET 2022-09-02T05:50:06.677 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.678 DEBUG:boto:Data: 2022-09-02T05:50:06.678 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.679 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.680 DEBUG:boto:Params: {} 2022-09-02T05:50:06.680 DEBUG:boto:Token: None 2022-09-02T05:50:06.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:06.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4= 2022-09-02T05:50:06.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ac5OW0xNDOW3iR0tRBQYaHOZe4=', 'Content-Length': '0'} 2022-09-02T05:50:06.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004255fcc62d35addf-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.686 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.687 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.687 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.688 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.688 DEBUG:boto:Method: GET 2022-09-02T05:50:06.689 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.689 DEBUG:boto:Data: 2022-09-02T05:50:06.690 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.691 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.691 DEBUG:boto:Params: {} 2022-09-02T05:50:06.692 DEBUG:boto:Token: None 2022-09-02T05:50:06.693 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d348d49657d9e037-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.697 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.698 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.699 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.699 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.700 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.700 DEBUG:boto:Method: GET 2022-09-02T05:50:06.701 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.701 DEBUG:boto:Data: 2022-09-02T05:50:06.702 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.703 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.704 DEBUG:boto:Params: {} 2022-09-02T05:50:06.704 DEBUG:boto:Token: None 2022-09-02T05:50:06.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042386f0b052404b3-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.710 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.711 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.712 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.712 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.713 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:06.713 DEBUG:boto:Data: 2022-09-02T05:50:06.714 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.715 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.715 DEBUG:boto:Params: {} 2022-09-02T05:50:06.716 DEBUG:boto:Token: None 2022-09-02T05:50:06.717 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68 2022-09-02T05:50:06.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fB/HDv8Wc1k7MYAKToOX9AIQwS0= 2022-09-02T05:50:06.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fB/HDv8Wc1k7MYAKToOX9AIQwS0=', 'Content-Length': '0'} 2022-09-02T05:50:06.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ea8f67dd3317f9-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.720 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.721 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.722 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.722 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:06.723 DEBUG:boto:Data: 2022-09-02T05:50:06.723 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.724 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.725 DEBUG:boto:Params: {} 2022-09-02T05:50:06.725 DEBUG:boto:Token: None 2022-09-02T05:50:06.726 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68 2022-09-02T05:50:06.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fB/HDv8Wc1k7MYAKToOX9AIQwS0= 2022-09-02T05:50:06.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fB/HDv8Wc1k7MYAKToOX9AIQwS0=', 'Content-Length': '0'} 2022-09-02T05:50:06.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034000f268fc5cd43-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:06.731 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.731 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.732 DEBUG:boto:Method: GET 2022-09-02T05:50:06.733 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:06.733 DEBUG:boto:Data: 2022-09-02T05:50:06.734 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.735 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.736 DEBUG:boto:Params: {} 2022-09-02T05:50:06.736 DEBUG:boto:Token: None 2022-09-02T05:50:06.737 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68 2022-09-02T05:50:06.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IYqi4yyNhgGKXsgyO+/lrDJonKQ= 2022-09-02T05:50:06.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IYqi4yyNhgGKXsgyO+/lrDJonKQ=', 'Content-Length': '0'} 2022-09-02T05:50:06.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91c484a9b2e63cf-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.741 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:06.742 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:06.743 DEBUG:boto:Method: GET 2022-09-02T05:50:06.743 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:06.744 DEBUG:boto:Data: 2022-09-02T05:50:06.745 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.746 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.746 DEBUG:boto:Params: {} 2022-09-02T05:50:06.747 DEBUG:boto:Token: None 2022-09-02T05:50:06.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj68 2022-09-02T05:50:06.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IYqi4yyNhgGKXsgyO+/lrDJonKQ= 2022-09-02T05:50:06.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IYqi4yyNhgGKXsgyO+/lrDJonKQ=', 'Content-Length': '0'} 2022-09-02T05:50:06.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b896d99c8cfda69-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.753 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.753 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.754 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.754 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.755 DEBUG:boto:Method: GET 2022-09-02T05:50:06.755 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.756 DEBUG:boto:Data: 2022-09-02T05:50:06.757 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.758 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.758 DEBUG:boto:Params: {} 2022-09-02T05:50:06.759 DEBUG:boto:Token: None 2022-09-02T05:50:06.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082b0057a59b154db-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.765 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.765 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.766 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.767 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.768 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.769 DEBUG:boto:Method: GET 2022-09-02T05:50:06.770 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:06.771 DEBUG:boto:Data: 2022-09-02T05:50:06.772 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.774 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.775 DEBUG:boto:Params: {} 2022-09-02T05:50:06.776 DEBUG:boto:Token: None 2022-09-02T05:50:06.778 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4dd19baac27f56-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.785 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:06.788 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.789 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.790 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.791 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.791 DEBUG:boto:Method: GET 2022-09-02T05:50:06.792 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.792 DEBUG:boto:Data: 2022-09-02T05:50:06.793 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.794 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.795 DEBUG:boto:Params: {} 2022-09-02T05:50:06.796 DEBUG:boto:Token: None 2022-09-02T05:50:06.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg= 2022-09-02T05:50:06.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg=', 'Content-Length': '0'} 2022-09-02T05:50:06.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e208830bb1d1d5-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.801 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.802 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.802 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.803 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.804 DEBUG:boto:Method: GET 2022-09-02T05:50:06.804 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.805 DEBUG:boto:Data: 2022-09-02T05:50:06.805 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.806 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.807 DEBUG:boto:Params: {} 2022-09-02T05:50:06.808 DEBUG:boto:Token: None 2022-09-02T05:50:06.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg= 2022-09-02T05:50:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg=', 'Content-Length': '0'} 2022-09-02T05:50:06.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d9935aa14c03335-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.813 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.813 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.814 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.814 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.815 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.815 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.816 DEBUG:boto:Data: 2022-09-02T05:50:06.816 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.817 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.818 DEBUG:boto:Params: {} 2022-09-02T05:50:06.819 DEBUG:boto:Token: None 2022-09-02T05:50:06.819 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yxZ/hwoITiBmboqAMDuvIVpYEpo= 2022-09-02T05:50:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yxZ/hwoITiBmboqAMDuvIVpYEpo=', 'Content-Length': '0'} 2022-09-02T05:50:06.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009858699d479c87d6-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.824 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.824 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.825 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.825 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.826 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.826 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.827 DEBUG:boto:Data: 2022-09-02T05:50:06.827 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.828 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.829 DEBUG:boto:Params: {} 2022-09-02T05:50:06.830 DEBUG:boto:Token: None 2022-09-02T05:50:06.831 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yxZ/hwoITiBmboqAMDuvIVpYEpo= 2022-09-02T05:50:06.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yxZ/hwoITiBmboqAMDuvIVpYEpo=', 'Content-Length': '0'} 2022-09-02T05:50:06.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034b5df0b4db61530-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:06.836 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.836 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.837 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.838 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.838 DEBUG:boto:Method: GET 2022-09-02T05:50:06.839 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.839 DEBUG:boto:Data: 2022-09-02T05:50:06.840 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.841 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.842 DEBUG:boto:Params: {} 2022-09-02T05:50:06.842 DEBUG:boto:Token: None 2022-09-02T05:50:06.843 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.844 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg= 2022-09-02T05:50:06.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg=', 'Content-Length': '0'} 2022-09-02T05:50:06.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119d7d692eb34f11-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.847 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.848 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.848 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.849 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.849 DEBUG:boto:Method: GET 2022-09-02T05:50:06.850 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.850 DEBUG:boto:Data: 2022-09-02T05:50:06.851 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.852 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.853 DEBUG:boto:Params: {} 2022-09-02T05:50:06.853 DEBUG:boto:Token: None 2022-09-02T05:50:06.854 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:06.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg= 2022-09-02T05:50:06.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zCd2k3wd4W8nXGThB9VONb8/mJg=', 'Content-Length': '0'} 2022-09-02T05:50:06.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004decbb30aa39440b-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.902 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.903 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.904 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:06.906 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:06.907 DEBUG:boto:Method: GET 2022-09-02T05:50:06.908 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:06.908 DEBUG:boto:Data: 2022-09-02T05:50:06.909 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.910 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.911 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.912 DEBUG:boto:Params: {} 2022-09-02T05:50:06.914 DEBUG:boto:Token: None 2022-09-02T05:50:06.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760365d5da19b327-006311998e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.922 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.924 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:06.925 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:06.925 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:06.926 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:06.927 DEBUG:boto:Method: GET 2022-09-02T05:50:06.928 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:06.929 DEBUG:boto:Data: 2022-09-02T05:50:06.931 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.933 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.934 DEBUG:boto:Params: {} 2022-09-02T05:50:06.935 DEBUG:boto:Token: None 2022-09-02T05:50:06.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/ 2022-09-02T05:50:06.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY= 2022-09-02T05:50:06.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9+WRJaM/pPPwuGqasDJjABbNrZY=', 'Content-Length': '0'} 2022-09-02T05:50:06.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6da717ca33b52d4-006311998e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.943 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:06.944 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.945 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.945 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.946 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:06.947 DEBUG:boto:Data: 2022-09-02T05:50:06.947 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.948 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.949 DEBUG:boto:Params: {} 2022-09-02T05:50:06.949 DEBUG:boto:Token: None 2022-09-02T05:50:06.950 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76 2022-09-02T05:50:06.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c/BiwviCIUcOnzhtNKp3sf83X0k= 2022-09-02T05:50:06.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c/BiwviCIUcOnzhtNKp3sf83X0k=', 'Content-Length': '0'} 2022-09-02T05:50:06.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c1fe893753f20d2-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.954 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.954 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.955 DEBUG:boto:Method: HEAD 2022-09-02T05:50:06.956 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:06.956 DEBUG:boto:Data: 2022-09-02T05:50:06.957 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.958 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.958 DEBUG:boto:Params: {} 2022-09-02T05:50:06.959 DEBUG:boto:Token: None 2022-09-02T05:50:06.959 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76 2022-09-02T05:50:06.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c/BiwviCIUcOnzhtNKp3sf83X0k= 2022-09-02T05:50:06.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c/BiwviCIUcOnzhtNKp3sf83X0k=', 'Content-Length': '0'} 2022-09-02T05:50:06.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000355f82d3cd8f0753-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:06.964 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.965 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.965 DEBUG:boto:Method: GET 2022-09-02T05:50:06.966 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:06.966 DEBUG:boto:Data: 2022-09-02T05:50:06.967 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:06.968 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.968 DEBUG:boto:Params: {} 2022-09-02T05:50:06.969 DEBUG:boto:Token: None 2022-09-02T05:50:06.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76 2022-09-02T05:50:06.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EpeBBgYA9bXuO4XolXH3gROPbGA= 2022-09-02T05:50:06.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EpeBBgYA9bXuO4XolXH3gROPbGA=', 'Content-Length': '0'} 2022-09-02T05:50:06.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc20893407a01330-006311998e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:06.974 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:06.975 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:06.975 DEBUG:boto:Method: GET 2022-09-02T05:50:06.976 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:06.976 DEBUG:boto:Data: 2022-09-02T05:50:06.977 DEBUG:boto:Headers: {} 2022-09-02T05:50:06.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:06.978 DEBUG:boto:Port: 8000 2022-09-02T05:50:06.978 DEBUG:boto:Params: {} 2022-09-02T05:50:06.979 DEBUG:boto:Token: None 2022-09-02T05:50:06.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:06 GMT /xljbkf-74/obj76 2022-09-02T05:50:06.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EpeBBgYA9bXuO4XolXH3gROPbGA= 2022-09-02T05:50:06.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EpeBBgYA9bXuO4XolXH3gROPbGA=', 'Content-Length': '0'} 2022-09-02T05:50:06.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5156579cb1f8a33-006311998e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.030 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.031 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.031 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:07.032 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:07.033 DEBUG:boto:Method: GET 2022-09-02T05:50:07.033 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:07.034 DEBUG:boto:Data: 2022-09-02T05:50:07.034 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.036 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.036 DEBUG:boto:Params: {} 2022-09-02T05:50:07.037 DEBUG:boto:Token: None 2022-09-02T05:50:07.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090d7d2724f15ccb6-006311998f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.042 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.043 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.044 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.044 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:07.045 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:07.045 DEBUG:boto:Method: GET 2022-09-02T05:50:07.046 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:07.046 DEBUG:boto:Data: 2022-09-02T05:50:07.047 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.048 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.048 DEBUG:boto:Params: {} 2022-09-02T05:50:07.049 DEBUG:boto:Token: None 2022-09-02T05:50:07.050 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ee811701001ab50-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.055 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:07.056 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.057 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.057 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.058 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.059 DEBUG:boto:Method: GET 2022-09-02T05:50:07.059 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.060 DEBUG:boto:Data: 2022-09-02T05:50:07.060 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.061 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.062 DEBUG:boto:Params: {} 2022-09-02T05:50:07.063 DEBUG:boto:Token: None 2022-09-02T05:50:07.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU= 2022-09-02T05:50:07.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU=', 'Content-Length': '0'} 2022-09-02T05:50:07.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008379210d7cefc8cd-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.068 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.069 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.069 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.070 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.070 DEBUG:boto:Method: GET 2022-09-02T05:50:07.071 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.071 DEBUG:boto:Data: 2022-09-02T05:50:07.072 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.073 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.073 DEBUG:boto:Params: {} 2022-09-02T05:50:07.074 DEBUG:boto:Token: None 2022-09-02T05:50:07.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU= 2022-09-02T05:50:07.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU=', 'Content-Length': '0'} 2022-09-02T05:50:07.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3f2b15db501ea7-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.079 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.080 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.081 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.081 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.082 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.083 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.083 DEBUG:boto:Data: 2022-09-02T05:50:07.084 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.085 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.085 DEBUG:boto:Params: {} 2022-09-02T05:50:07.086 DEBUG:boto:Token: None 2022-09-02T05:50:07.086 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JvXqj/qMte87ZAng+AOyK6EyIlg= 2022-09-02T05:50:07.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JvXqj/qMte87ZAng+AOyK6EyIlg=', 'Content-Length': '0'} 2022-09-02T05:50:07.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a61a2913669dfae-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.091 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.091 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.092 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.092 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.093 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.093 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.094 DEBUG:boto:Data: 2022-09-02T05:50:07.094 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.095 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.096 DEBUG:boto:Params: {} 2022-09-02T05:50:07.097 DEBUG:boto:Token: None 2022-09-02T05:50:07.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JvXqj/qMte87ZAng+AOyK6EyIlg= 2022-09-02T05:50:07.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JvXqj/qMte87ZAng+AOyK6EyIlg=', 'Content-Length': '0'} 2022-09-02T05:50:07.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a446c463b30d37a-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:07.106 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.107 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.108 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.108 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.109 DEBUG:boto:Method: GET 2022-09-02T05:50:07.109 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.110 DEBUG:boto:Data: 2022-09-02T05:50:07.110 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.111 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.111 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.112 DEBUG:boto:Params: {} 2022-09-02T05:50:07.112 DEBUG:boto:Token: None 2022-09-02T05:50:07.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU= 2022-09-02T05:50:07.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU=', 'Content-Length': '0'} 2022-09-02T05:50:07.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472d06f99d970039-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.117 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.118 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.119 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.119 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.120 DEBUG:boto:Method: GET 2022-09-02T05:50:07.120 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.121 DEBUG:boto:Data: 2022-09-02T05:50:07.122 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.123 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.123 DEBUG:boto:Params: {} 2022-09-02T05:50:07.124 DEBUG:boto:Token: None 2022-09-02T05:50:07.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:07.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU= 2022-09-02T05:50:07.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4cB0WaPhdYvdGE+LrEphapTnEEU=', 'Content-Length': '0'} 2022-09-02T05:50:07.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360144f6c2a90f17-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.170 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.171 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.171 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.172 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.172 DEBUG:boto:Method: GET 2022-09-02T05:50:07.173 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.174 DEBUG:boto:Data: 2022-09-02T05:50:07.174 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.175 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.176 DEBUG:boto:Params: {} 2022-09-02T05:50:07.176 DEBUG:boto:Token: None 2022-09-02T05:50:07.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab4e129478161954-006311998f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.182 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.183 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.183 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.184 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.184 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.185 DEBUG:boto:Method: GET 2022-09-02T05:50:07.185 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.186 DEBUG:boto:Data: 2022-09-02T05:50:07.186 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.187 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.188 DEBUG:boto:Params: {} 2022-09-02T05:50:07.188 DEBUG:boto:Token: None 2022-09-02T05:50:07.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15ca52bcd5057e6-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.195 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.196 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.196 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.197 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.198 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:07.198 DEBUG:boto:Data: 2022-09-02T05:50:07.199 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.199 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.200 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.201 DEBUG:boto:Params: {} 2022-09-02T05:50:07.201 DEBUG:boto:Token: None 2022-09-02T05:50:07.202 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85 2022-09-02T05:50:07.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xKQ3+80bJWaQhYYYt63Jd5jgAUc= 2022-09-02T05:50:07.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xKQ3+80bJWaQhYYYt63Jd5jgAUc=', 'Content-Length': '0'} 2022-09-02T05:50:07.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c71a2f25f7584f-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.206 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.206 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.207 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.208 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:07.208 DEBUG:boto:Data: 2022-09-02T05:50:07.209 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.210 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.211 DEBUG:boto:Params: {} 2022-09-02T05:50:07.211 DEBUG:boto:Token: None 2022-09-02T05:50:07.212 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85 2022-09-02T05:50:07.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xKQ3+80bJWaQhYYYt63Jd5jgAUc= 2022-09-02T05:50:07.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xKQ3+80bJWaQhYYYt63Jd5jgAUc=', 'Content-Length': '0'} 2022-09-02T05:50:07.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083bf04d354b1b0ab-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:07.218 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.219 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.219 DEBUG:boto:Method: GET 2022-09-02T05:50:07.220 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:07.220 DEBUG:boto:Data: 2022-09-02T05:50:07.221 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.222 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.222 DEBUG:boto:Params: {} 2022-09-02T05:50:07.223 DEBUG:boto:Token: None 2022-09-02T05:50:07.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85 2022-09-02T05:50:07.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g5fyydw6t/cHV7KW/3V6xjV4iM= 2022-09-02T05:50:07.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g5fyydw6t/cHV7KW/3V6xjV4iM=', 'Content-Length': '0'} 2022-09-02T05:50:07.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2de56fabb07835-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.228 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:07.228 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:07.229 DEBUG:boto:Method: GET 2022-09-02T05:50:07.230 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:07.230 DEBUG:boto:Data: 2022-09-02T05:50:07.231 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.232 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.232 DEBUG:boto:Params: {} 2022-09-02T05:50:07.233 DEBUG:boto:Token: None 2022-09-02T05:50:07.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj85 2022-09-02T05:50:07.235 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7g5fyydw6t/cHV7KW/3V6xjV4iM= 2022-09-02T05:50:07.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7g5fyydw6t/cHV7KW/3V6xjV4iM=', 'Content-Length': '0'} 2022-09-02T05:50:07.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32a0659cb69af93-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.240 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.240 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.241 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.241 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.242 DEBUG:boto:Method: GET 2022-09-02T05:50:07.243 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.243 DEBUG:boto:Data: 2022-09-02T05:50:07.244 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.244 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.245 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.246 DEBUG:boto:Params: {} 2022-09-02T05:50:07.246 DEBUG:boto:Token: None 2022-09-02T05:50:07.247 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f2beb91bc7886d-006311998f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.294 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.295 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.296 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.297 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.297 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.298 DEBUG:boto:Method: GET 2022-09-02T05:50:07.298 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:07.299 DEBUG:boto:Data: 2022-09-02T05:50:07.299 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.301 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.303 DEBUG:boto:Params: {} 2022-09-02T05:50:07.304 DEBUG:boto:Token: None 2022-09-02T05:50:07.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000059b7bb82f450b5-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.310 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:07.311 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.312 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.312 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.313 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.314 DEBUG:boto:Method: GET 2022-09-02T05:50:07.315 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.315 DEBUG:boto:Data: 2022-09-02T05:50:07.316 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.317 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.318 DEBUG:boto:Params: {} 2022-09-02T05:50:07.319 DEBUG:boto:Token: None 2022-09-02T05:50:07.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA= 2022-09-02T05:50:07.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA=', 'Content-Length': '0'} 2022-09-02T05:50:07.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017ba21ab1b0ae54-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.325 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.326 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.327 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.327 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.328 DEBUG:boto:Method: GET 2022-09-02T05:50:07.329 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.329 DEBUG:boto:Data: 2022-09-02T05:50:07.330 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.331 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.332 DEBUG:boto:Params: {} 2022-09-02T05:50:07.332 DEBUG:boto:Token: None 2022-09-02T05:50:07.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA= 2022-09-02T05:50:07.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA=', 'Content-Length': '0'} 2022-09-02T05:50:07.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5de5cc35bdd966-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.338 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.339 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.339 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.340 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.341 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.341 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.342 DEBUG:boto:Data: 2022-09-02T05:50:07.343 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.345 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.346 DEBUG:boto:Params: {} 2022-09-02T05:50:07.347 DEBUG:boto:Token: None 2022-09-02T05:50:07.348 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.349 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NkBYUXmMekUfW49rbH5ryNQgdtw= 2022-09-02T05:50:07.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NkBYUXmMekUfW49rbH5ryNQgdtw=', 'Content-Length': '0'} 2022-09-02T05:50:07.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2b5e6b1fdcba55-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.353 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.354 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.355 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.355 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.356 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.357 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.357 DEBUG:boto:Data: 2022-09-02T05:50:07.358 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.360 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.360 DEBUG:boto:Params: {} 2022-09-02T05:50:07.361 DEBUG:boto:Token: None 2022-09-02T05:50:07.362 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NkBYUXmMekUfW49rbH5ryNQgdtw= 2022-09-02T05:50:07.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NkBYUXmMekUfW49rbH5ryNQgdtw=', 'Content-Length': '0'} 2022-09-02T05:50:07.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037444f33ee473d04-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:07.368 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.370 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.370 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.371 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.372 DEBUG:boto:Method: GET 2022-09-02T05:50:07.373 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.374 DEBUG:boto:Data: 2022-09-02T05:50:07.375 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.376 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.377 DEBUG:boto:Params: {} 2022-09-02T05:50:07.378 DEBUG:boto:Token: None 2022-09-02T05:50:07.379 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA= 2022-09-02T05:50:07.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA=', 'Content-Length': '0'} 2022-09-02T05:50:07.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2b9e5c0ca83b7b-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.384 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.385 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.386 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.386 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.387 DEBUG:boto:Method: GET 2022-09-02T05:50:07.388 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.389 DEBUG:boto:Data: 2022-09-02T05:50:07.390 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.392 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.393 DEBUG:boto:Params: {} 2022-09-02T05:50:07.394 DEBUG:boto:Token: None 2022-09-02T05:50:07.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:07.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA= 2022-09-02T05:50:07.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZC330TuD4H6ErctpoH9bbmIawaA=', 'Content-Length': '0'} 2022-09-02T05:50:07.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8d2b393bee01ab-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.401 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.402 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.403 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.404 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.404 DEBUG:boto:Method: GET 2022-09-02T05:50:07.405 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.406 DEBUG:boto:Data: 2022-09-02T05:50:07.407 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.408 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.409 DEBUG:boto:Params: {} 2022-09-02T05:50:07.410 DEBUG:boto:Token: None 2022-09-02T05:50:07.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b67fa63692f6b7a3-006311998f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.462 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.464 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.465 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.466 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.467 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.467 DEBUG:boto:Method: GET 2022-09-02T05:50:07.468 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.469 DEBUG:boto:Data: 2022-09-02T05:50:07.470 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.471 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.472 DEBUG:boto:Params: {} 2022-09-02T05:50:07.473 DEBUG:boto:Token: None 2022-09-02T05:50:07.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a2d7a7bf706927-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.484 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.485 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.485 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.486 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.487 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:07.487 DEBUG:boto:Data: 2022-09-02T05:50:07.488 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.489 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.490 DEBUG:boto:Params: {} 2022-09-02T05:50:07.490 DEBUG:boto:Token: None 2022-09-02T05:50:07.491 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95 2022-09-02T05:50:07.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yh047ruJ3H7VdXQTN2RJpKujtoY= 2022-09-02T05:50:07.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yh047ruJ3H7VdXQTN2RJpKujtoY=', 'Content-Length': '0'} 2022-09-02T05:50:07.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac93846f5f6c14f0-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.497 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.497 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.498 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.499 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:07.500 DEBUG:boto:Data: 2022-09-02T05:50:07.501 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.502 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.503 DEBUG:boto:Params: {} 2022-09-02T05:50:07.504 DEBUG:boto:Token: None 2022-09-02T05:50:07.506 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95 2022-09-02T05:50:07.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yh047ruJ3H7VdXQTN2RJpKujtoY= 2022-09-02T05:50:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yh047ruJ3H7VdXQTN2RJpKujtoY=', 'Content-Length': '0'} 2022-09-02T05:50:07.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4985f91ccf66eb3-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:07.515 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.516 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.517 DEBUG:boto:Method: GET 2022-09-02T05:50:07.518 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:07.520 DEBUG:boto:Data: 2022-09-02T05:50:07.520 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.522 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.524 DEBUG:boto:Params: {} 2022-09-02T05:50:07.525 DEBUG:boto:Token: None 2022-09-02T05:50:07.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95 2022-09-02T05:50:07.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:af+KIBDrZTNSZrhRY5lYzXkEzDs= 2022-09-02T05:50:07.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:af+KIBDrZTNSZrhRY5lYzXkEzDs=', 'Content-Length': '0'} 2022-09-02T05:50:07.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001352e65a81d73b21-006311998f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.537 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:07.538 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:07.539 DEBUG:boto:Method: GET 2022-09-02T05:50:07.540 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:07.542 DEBUG:boto:Data: 2022-09-02T05:50:07.542 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.545 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.546 DEBUG:boto:Params: {} 2022-09-02T05:50:07.547 DEBUG:boto:Token: None 2022-09-02T05:50:07.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj95 2022-09-02T05:50:07.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:af+KIBDrZTNSZrhRY5lYzXkEzDs= 2022-09-02T05:50:07.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:af+KIBDrZTNSZrhRY5lYzXkEzDs=', 'Content-Length': '0'} 2022-09-02T05:50:07.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2dd639afe2ba04-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.601 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.602 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.603 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.604 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.605 DEBUG:boto:Method: GET 2022-09-02T05:50:07.606 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.607 DEBUG:boto:Data: 2022-09-02T05:50:07.608 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:07.611 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.612 DEBUG:boto:Params: {} 2022-09-02T05:50:07.613 DEBUG:boto:Token: None 2022-09-02T05:50:07.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcac4aecb428c4d2-006311998f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.624 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.625 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.626 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.627 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.629 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.630 DEBUG:boto:Method: GET 2022-09-02T05:50:07.631 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:07.632 DEBUG:boto:Data: 2022-09-02T05:50:07.633 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.635 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.636 DEBUG:boto:Params: {} 2022-09-02T05:50:07.637 DEBUG:boto:Token: None 2022-09-02T05:50:07.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d40acde6d49bb0-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.653 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.655 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:50:07.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:50:07.657 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.658 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.659 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.660 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:07.661 DEBUG:boto:Data: 2022-09-02T05:50:07.663 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.665 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.666 DEBUG:boto:Params: {} 2022-09-02T05:50:07.667 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:50:07.668 DEBUG:boto:Token: None 2022-09-02T05:50:07.669 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HtU41BYxssu7iNJaHuDuoPoay2w= 2022-09-02T05:50:07.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HtU41BYxssu7iNJaHuDuoPoay2w=', 'Content-Length': '0'} 2022-09-02T05:50:07.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000415913c578decf91-006311998f-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.678 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.680 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.680 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.681 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:07.681 DEBUG:boto:Data: 2022-09-02T05:50:07.682 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.684 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.684 DEBUG:boto:Params: {} 2022-09-02T05:50:07.685 DEBUG:boto:Token: None 2022-09-02T05:50:07.685 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HtU41BYxssu7iNJaHuDuoPoay2w= 2022-09-02T05:50:07.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HtU41BYxssu7iNJaHuDuoPoay2w=', 'Content-Length': '0'} 2022-09-02T05:50:07.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2174144c122a558-006311998f-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:50:07.691 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.692 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.693 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.694 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.695 DEBUG:boto:Method: GET 2022-09-02T05:50:07.695 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.696 DEBUG:boto:Data: 2022-09-02T05:50:07.696 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.697 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.698 DEBUG:boto:Params: {} 2022-09-02T05:50:07.699 DEBUG:boto:Token: None 2022-09-02T05:50:07.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/?versions 2022-09-02T05:50:07.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo= 2022-09-02T05:50:07.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo=', 'Content-Length': '0'} 2022-09-02T05:50:07.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000649f434c957b4911-006311998f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.714 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:50:07.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:50:07.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:50:07.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:50:07.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:50:07.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:50:07.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:50:07.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:50:07.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:50:07.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:50:07.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:50:07.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:50:07.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:50:07.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:50:07.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:50:07.728 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:50:07.728 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.729 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.729 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.730 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.730 DEBUG:boto:Method: GET 2022-09-02T05:50:07.731 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.731 DEBUG:boto:Data: 2022-09-02T05:50:07.734 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.736 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.737 DEBUG:boto:Params: {} 2022-09-02T05:50:07.738 DEBUG:boto:Token: None 2022-09-02T05:50:07.741 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/?versions 2022-09-02T05:50:07.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo= 2022-09-02T05:50:07.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo=', 'Content-Length': '0'} 2022-09-02T05:50:07.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000323576e2df4b3d6a-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.753 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:50:07.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:50:07.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:50:07.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:50:07.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:50:07.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:50:07.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:50:07.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:50:07.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:50:07.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:50:07.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:50:07.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:50:07.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:50:07.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:50:07.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:50:07.775 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.776 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.778 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.779 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.780 DEBUG:boto:Method: GET 2022-09-02T05:50:07.781 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.782 DEBUG:boto:Data: 2022-09-02T05:50:07.783 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.786 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.787 DEBUG:boto:Params: {} 2022-09-02T05:50:07.788 DEBUG:boto:Token: None 2022-09-02T05:50:07.789 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/?versions 2022-09-02T05:50:07.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo= 2022-09-02T05:50:07.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo=', 'Content-Length': '0'} 2022-09-02T05:50:07.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f633c25ba76f1145-006311998f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.797 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.800 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.801 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.803 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.803 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.805 DEBUG:boto:Method: GET 2022-09-02T05:50:07.806 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:07.807 DEBUG:boto:Data: 2022-09-02T05:50:07.808 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.810 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.810 DEBUG:boto:Params: {} 2022-09-02T05:50:07.812 DEBUG:boto:Token: None 2022-09-02T05:50:07.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/?versions 2022-09-02T05:50:07.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo= 2022-09-02T05:50:07.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rfUmrabSD/c0/Ef4T+x1SywsEpo=', 'Content-Length': '0'} 2022-09-02T05:50:07.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c25bb1931a6e2dd-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.824 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:07.827 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.828 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.829 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.830 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.831 DEBUG:boto:Method: GET 2022-09-02T05:50:07.832 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.833 DEBUG:boto:Data: 2022-09-02T05:50:07.833 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.835 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.835 DEBUG:boto:Params: {} 2022-09-02T05:50:07.837 DEBUG:boto:Token: None 2022-09-02T05:50:07.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s= 2022-09-02T05:50:07.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s=', 'Content-Length': '0'} 2022-09-02T05:50:07.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014b44c063f05a444-006311998f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.847 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.848 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.848 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.849 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.849 DEBUG:boto:Method: GET 2022-09-02T05:50:07.850 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.851 DEBUG:boto:Data: 2022-09-02T05:50:07.851 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.852 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.853 DEBUG:boto:Params: {} 2022-09-02T05:50:07.854 DEBUG:boto:Token: None 2022-09-02T05:50:07.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s= 2022-09-02T05:50:07.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s=', 'Content-Length': '0'} 2022-09-02T05:50:07.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052aab5fbc079ea3c-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.863 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.864 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.864 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.865 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.865 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.866 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.866 DEBUG:boto:Data: 2022-09-02T05:50:07.867 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.868 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.868 DEBUG:boto:Params: {} 2022-09-02T05:50:07.869 DEBUG:boto:Token: None 2022-09-02T05:50:07.869 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O8fA7+FXBNG9RMmEKGRC7e8kJCw= 2022-09-02T05:50:07.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O8fA7+FXBNG9RMmEKGRC7e8kJCw=', 'Content-Length': '0'} 2022-09-02T05:50:07.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002777134cd74c6563-006311998f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.874 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.874 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.875 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.875 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.876 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.876 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.877 DEBUG:boto:Data: 2022-09-02T05:50:07.877 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.878 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.879 DEBUG:boto:Params: {} 2022-09-02T05:50:07.879 DEBUG:boto:Token: None 2022-09-02T05:50:07.880 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O8fA7+FXBNG9RMmEKGRC7e8kJCw= 2022-09-02T05:50:07.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O8fA7+FXBNG9RMmEKGRC7e8kJCw=', 'Content-Length': '0'} 2022-09-02T05:50:07.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92817a949b602bb-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:07.885 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.886 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.886 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.887 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.888 DEBUG:boto:Method: GET 2022-09-02T05:50:07.888 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.889 DEBUG:boto:Data: 2022-09-02T05:50:07.889 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.890 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.891 DEBUG:boto:Params: {} 2022-09-02T05:50:07.891 DEBUG:boto:Token: None 2022-09-02T05:50:07.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s= 2022-09-02T05:50:07.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s=', 'Content-Length': '0'} 2022-09-02T05:50:07.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056293033fb93ea01-006311998f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.899 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.900 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.900 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.901 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.901 DEBUG:boto:Method: GET 2022-09-02T05:50:07.902 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.903 DEBUG:boto:Data: 2022-09-02T05:50:07.903 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.904 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.905 DEBUG:boto:Params: {} 2022-09-02T05:50:07.906 DEBUG:boto:Token: None 2022-09-02T05:50:07.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:07.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s= 2022-09-02T05:50:07.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NKqYqyxSJGj7FNsXmHXcqMidl4s=', 'Content-Length': '0'} 2022-09-02T05:50:07.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009979013d489f3d3e-006311998f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.954 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.955 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.956 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:07.956 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:07.957 DEBUG:boto:Method: GET 2022-09-02T05:50:07.957 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:07.958 DEBUG:boto:Data: 2022-09-02T05:50:07.958 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.959 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.960 DEBUG:boto:Params: {} 2022-09-02T05:50:07.961 DEBUG:boto:Token: None 2022-09-02T05:50:07.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ce2a56f546320c-006311998f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.966 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.967 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:07.968 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:07.969 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:07.969 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:07.970 DEBUG:boto:Method: GET 2022-09-02T05:50:07.970 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:07.971 DEBUG:boto:Data: 2022-09-02T05:50:07.972 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.973 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.973 DEBUG:boto:Params: {} 2022-09-02T05:50:07.974 DEBUG:boto:Token: None 2022-09-02T05:50:07.975 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/ 2022-09-02T05:50:07.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU= 2022-09-02T05:50:07.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bjrYlif85xVeMrw1wvNgljuCjrU=', 'Content-Length': '0'} 2022-09-02T05:50:07.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858eef0bbfa6505c-006311998f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.980 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:07.982 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.982 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.983 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.983 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:07.984 DEBUG:boto:Data: 2022-09-02T05:50:07.984 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:07.986 DEBUG:boto:Port: 8001 2022-09-02T05:50:07.986 DEBUG:boto:Params: {} 2022-09-02T05:50:07.987 DEBUG:boto:Token: None 2022-09-02T05:50:07.987 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11 2022-09-02T05:50:07.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yTsgz8KjGUsNnVUuBb2Ypmxmd+A= 2022-09-02T05:50:07.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yTsgz8KjGUsNnVUuBb2Ypmxmd+A=', 'Content-Length': '0'} 2022-09-02T05:50:07.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc242d05859b992-006311998f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:07.992 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:07.993 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:07.993 DEBUG:boto:Method: HEAD 2022-09-02T05:50:07.994 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:07.994 DEBUG:boto:Data: 2022-09-02T05:50:07.995 DEBUG:boto:Headers: {} 2022-09-02T05:50:07.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:07.996 DEBUG:boto:Port: 8000 2022-09-02T05:50:07.996 DEBUG:boto:Params: {} 2022-09-02T05:50:07.997 DEBUG:boto:Token: None 2022-09-02T05:50:07.998 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:07 GMT /xljbkf-74/obj11 2022-09-02T05:50:07.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yTsgz8KjGUsNnVUuBb2Ypmxmd+A= 2022-09-02T05:50:08.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yTsgz8KjGUsNnVUuBb2Ypmxmd+A=', 'Content-Length': '0'} 2022-09-02T05:50:08.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55a8be541c6aea5-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:08.012 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:08.014 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:08.015 DEBUG:boto:Method: GET 2022-09-02T05:50:08.019 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:08.019 DEBUG:boto:Data: 2022-09-02T05:50:08.020 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.021 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.022 DEBUG:boto:Params: {} 2022-09-02T05:50:08.022 DEBUG:boto:Token: None 2022-09-02T05:50:08.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj11 2022-09-02T05:50:08.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4bvmdjR1aEAXrwrdfWJgKt4ja88= 2022-09-02T05:50:08.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4bvmdjR1aEAXrwrdfWJgKt4ja88=', 'Content-Length': '0'} 2022-09-02T05:50:08.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ee773105edea01-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.070 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:08.071 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:08.071 DEBUG:boto:Method: GET 2022-09-02T05:50:08.072 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:08.072 DEBUG:boto:Data: 2022-09-02T05:50:08.073 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.074 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.075 DEBUG:boto:Params: {} 2022-09-02T05:50:08.075 DEBUG:boto:Token: None 2022-09-02T05:50:08.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj11 2022-09-02T05:50:08.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4bvmdjR1aEAXrwrdfWJgKt4ja88= 2022-09-02T05:50:08.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4bvmdjR1aEAXrwrdfWJgKt4ja88=', 'Content-Length': '0'} 2022-09-02T05:50:08.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002960fe989855a4c-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.085 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.086 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.086 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:08.087 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:08.087 DEBUG:boto:Method: GET 2022-09-02T05:50:08.088 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:08.089 DEBUG:boto:Data: 2022-09-02T05:50:08.089 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.090 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.091 DEBUG:boto:Params: {} 2022-09-02T05:50:08.091 DEBUG:boto:Token: None 2022-09-02T05:50:08.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.092 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1057400f6152af-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.103 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.103 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.104 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.105 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:08.105 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:08.106 DEBUG:boto:Method: GET 2022-09-02T05:50:08.106 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:08.107 DEBUG:boto:Data: 2022-09-02T05:50:08.107 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.108 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.109 DEBUG:boto:Params: {} 2022-09-02T05:50:08.110 DEBUG:boto:Token: None 2022-09-02T05:50:08.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a2d4a87fe407ff-0063119990-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.158 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:08.160 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.160 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.161 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.161 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.162 DEBUG:boto:Method: GET 2022-09-02T05:50:08.162 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.163 DEBUG:boto:Data: 2022-09-02T05:50:08.163 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.164 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.165 DEBUG:boto:Params: {} 2022-09-02T05:50:08.165 DEBUG:boto:Token: None 2022-09-02T05:50:08.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI= 2022-09-02T05:50:08.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI=', 'Content-Length': '0'} 2022-09-02T05:50:08.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf63bd22fca0fa55-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.177 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.178 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.179 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.180 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.180 DEBUG:boto:Method: GET 2022-09-02T05:50:08.181 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.182 DEBUG:boto:Data: 2022-09-02T05:50:08.183 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.185 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.186 DEBUG:boto:Params: {} 2022-09-02T05:50:08.189 DEBUG:boto:Token: None 2022-09-02T05:50:08.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI= 2022-09-02T05:50:08.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI=', 'Content-Length': '0'} 2022-09-02T05:50:08.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c9d439d1f21660-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.199 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.203 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.204 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.204 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.205 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.206 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.207 DEBUG:boto:Data: 2022-09-02T05:50:08.208 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.209 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.210 DEBUG:boto:Params: {} 2022-09-02T05:50:08.211 DEBUG:boto:Token: None 2022-09-02T05:50:08.212 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WAm1azR4wShwEapvino3r/bcJM8= 2022-09-02T05:50:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WAm1azR4wShwEapvino3r/bcJM8=', 'Content-Length': '0'} 2022-09-02T05:50:08.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3eb6ea4bbeb52f-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.218 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.218 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.219 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.220 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.221 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.221 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.222 DEBUG:boto:Data: 2022-09-02T05:50:08.223 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.225 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.226 DEBUG:boto:Params: {} 2022-09-02T05:50:08.227 DEBUG:boto:Token: None 2022-09-02T05:50:08.228 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WAm1azR4wShwEapvino3r/bcJM8= 2022-09-02T05:50:08.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WAm1azR4wShwEapvino3r/bcJM8=', 'Content-Length': '0'} 2022-09-02T05:50:08.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fadfebc29a62c2d6-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:08.234 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.235 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.236 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.237 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.238 DEBUG:boto:Method: GET 2022-09-02T05:50:08.239 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.240 DEBUG:boto:Data: 2022-09-02T05:50:08.241 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.243 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.243 DEBUG:boto:Params: {} 2022-09-02T05:50:08.244 DEBUG:boto:Token: None 2022-09-02T05:50:08.245 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI= 2022-09-02T05:50:08.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI=', 'Content-Length': '0'} 2022-09-02T05:50:08.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8136147c09e427-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.251 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.252 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.253 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.254 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.255 DEBUG:boto:Method: GET 2022-09-02T05:50:08.256 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.257 DEBUG:boto:Data: 2022-09-02T05:50:08.258 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.259 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.260 DEBUG:boto:Params: {} 2022-09-02T05:50:08.261 DEBUG:boto:Token: None 2022-09-02T05:50:08.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:08.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI= 2022-09-02T05:50:08.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkesaeLrBHb2/UvEEGIB8qyrcoI=', 'Content-Length': '0'} 2022-09-02T05:50:08.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009370a5cd436a78ae-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.269 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.270 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.271 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.271 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.272 DEBUG:boto:Method: GET 2022-09-02T05:50:08.273 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.273 DEBUG:boto:Data: 2022-09-02T05:50:08.274 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.275 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.276 DEBUG:boto:Params: {} 2022-09-02T05:50:08.276 DEBUG:boto:Token: None 2022-09-02T05:50:08.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449d07ad30530bfb-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.282 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.283 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.284 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.284 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.285 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.286 DEBUG:boto:Method: GET 2022-09-02T05:50:08.286 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.287 DEBUG:boto:Data: 2022-09-02T05:50:08.287 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.289 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.289 DEBUG:boto:Params: {} 2022-09-02T05:50:08.290 DEBUG:boto:Token: None 2022-09-02T05:50:08.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250da1312f6d5477-0063119990-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.338 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.339 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.340 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.340 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.341 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:08.342 DEBUG:boto:Data: 2022-09-02T05:50:08.342 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.343 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.344 DEBUG:boto:Params: {} 2022-09-02T05:50:08.345 DEBUG:boto:Token: None 2022-09-02T05:50:08.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17 2022-09-02T05:50:08.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:urRPYBuDb24ucrxcVt3OMrsdafg= 2022-09-02T05:50:08.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:urRPYBuDb24ucrxcVt3OMrsdafg=', 'Content-Length': '0'} 2022-09-02T05:50:08.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a14e050b8edd58-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.351 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.352 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.352 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.353 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:08.353 DEBUG:boto:Data: 2022-09-02T05:50:08.354 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.355 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.356 DEBUG:boto:Params: {} 2022-09-02T05:50:08.356 DEBUG:boto:Token: None 2022-09-02T05:50:08.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17 2022-09-02T05:50:08.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:urRPYBuDb24ucrxcVt3OMrsdafg= 2022-09-02T05:50:08.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:urRPYBuDb24ucrxcVt3OMrsdafg=', 'Content-Length': '0'} 2022-09-02T05:50:08.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3521bad59e9945f-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:08.364 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.364 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.365 DEBUG:boto:Method: GET 2022-09-02T05:50:08.366 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:08.366 DEBUG:boto:Data: 2022-09-02T05:50:08.367 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.368 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.369 DEBUG:boto:Params: {} 2022-09-02T05:50:08.370 DEBUG:boto:Token: None 2022-09-02T05:50:08.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17 2022-09-02T05:50:08.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NCuhsU1csyKYHad8SFt4/lp6nE4= 2022-09-02T05:50:08.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NCuhsU1csyKYHad8SFt4/lp6nE4=', 'Content-Length': '0'} 2022-09-02T05:50:08.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099fbf8adf31955c9-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.376 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:08.378 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:08.379 DEBUG:boto:Method: GET 2022-09-02T05:50:08.379 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:08.380 DEBUG:boto:Data: 2022-09-02T05:50:08.380 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.381 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.382 DEBUG:boto:Params: {} 2022-09-02T05:50:08.383 DEBUG:boto:Token: None 2022-09-02T05:50:08.383 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj17 2022-09-02T05:50:08.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NCuhsU1csyKYHad8SFt4/lp6nE4= 2022-09-02T05:50:08.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NCuhsU1csyKYHad8SFt4/lp6nE4=', 'Content-Length': '0'} 2022-09-02T05:50:08.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da8ddd37c53b9b5-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.389 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.390 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.390 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.391 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.391 DEBUG:boto:Method: GET 2022-09-02T05:50:08.392 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.392 DEBUG:boto:Data: 2022-09-02T05:50:08.393 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.394 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.395 DEBUG:boto:Params: {} 2022-09-02T05:50:08.395 DEBUG:boto:Token: None 2022-09-02T05:50:08.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ecc8caa3340c48-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.448 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.449 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.450 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.450 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.451 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.452 DEBUG:boto:Method: GET 2022-09-02T05:50:08.452 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:08.453 DEBUG:boto:Data: 2022-09-02T05:50:08.453 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.454 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.455 DEBUG:boto:Params: {} 2022-09-02T05:50:08.456 DEBUG:boto:Token: None 2022-09-02T05:50:08.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ca021173754250-0063119990-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.462 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:08.463 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.464 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.464 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.465 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.465 DEBUG:boto:Method: GET 2022-09-02T05:50:08.466 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.467 DEBUG:boto:Data: 2022-09-02T05:50:08.467 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.468 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.469 DEBUG:boto:Params: {} 2022-09-02T05:50:08.469 DEBUG:boto:Token: None 2022-09-02T05:50:08.470 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8= 2022-09-02T05:50:08.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8=', 'Content-Length': '0'} 2022-09-02T05:50:08.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea645c1d1c5d489-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.475 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.475 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.476 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.476 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.477 DEBUG:boto:Method: GET 2022-09-02T05:50:08.477 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.478 DEBUG:boto:Data: 2022-09-02T05:50:08.479 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.480 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.480 DEBUG:boto:Params: {} 2022-09-02T05:50:08.481 DEBUG:boto:Token: None 2022-09-02T05:50:08.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8= 2022-09-02T05:50:08.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8=', 'Content-Length': '0'} 2022-09-02T05:50:08.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c049d1ff5bf36d3-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.486 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.487 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.487 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.488 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.488 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.489 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.489 DEBUG:boto:Data: 2022-09-02T05:50:08.490 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.491 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.491 DEBUG:boto:Params: {} 2022-09-02T05:50:08.492 DEBUG:boto:Token: None 2022-09-02T05:50:08.492 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:90AsZRDJpZQSnbQzev9osssfmhc= 2022-09-02T05:50:08.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:90AsZRDJpZQSnbQzev9osssfmhc=', 'Content-Length': '0'} 2022-09-02T05:50:08.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b5aa03e28335fcf-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.497 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.497 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.498 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.498 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.499 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.500 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.500 DEBUG:boto:Data: 2022-09-02T05:50:08.501 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.502 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.503 DEBUG:boto:Params: {} 2022-09-02T05:50:08.503 DEBUG:boto:Token: None 2022-09-02T05:50:08.504 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:90AsZRDJpZQSnbQzev9osssfmhc= 2022-09-02T05:50:08.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:90AsZRDJpZQSnbQzev9osssfmhc=', 'Content-Length': '0'} 2022-09-02T05:50:08.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f71f728c06b89016-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:08.512 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.513 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.514 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.515 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.516 DEBUG:boto:Method: GET 2022-09-02T05:50:08.516 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.517 DEBUG:boto:Data: 2022-09-02T05:50:08.518 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.519 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.519 DEBUG:boto:Params: {} 2022-09-02T05:50:08.520 DEBUG:boto:Token: None 2022-09-02T05:50:08.521 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8= 2022-09-02T05:50:08.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8=', 'Content-Length': '0'} 2022-09-02T05:50:08.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008659a704b663ea26-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.526 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.526 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.527 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.527 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.528 DEBUG:boto:Method: GET 2022-09-02T05:50:08.529 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.529 DEBUG:boto:Data: 2022-09-02T05:50:08.530 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.532 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.533 DEBUG:boto:Params: {} 2022-09-02T05:50:08.534 DEBUG:boto:Token: None 2022-09-02T05:50:08.536 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:08.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8= 2022-09-02T05:50:08.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MvzFVyvJcLygVHsOhMTOkrlX4F8=', 'Content-Length': '0'} 2022-09-02T05:50:08.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c03dab6720af76a-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.587 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.588 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.590 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.591 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.593 DEBUG:boto:Method: GET 2022-09-02T05:50:08.595 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.596 DEBUG:boto:Data: 2022-09-02T05:50:08.597 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.599 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.599 DEBUG:boto:Params: {} 2022-09-02T05:50:08.600 DEBUG:boto:Token: None 2022-09-02T05:50:08.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a303e8f59f59abdd-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.607 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.608 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.609 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.609 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.610 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.611 DEBUG:boto:Method: GET 2022-09-02T05:50:08.611 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.612 DEBUG:boto:Data: 2022-09-02T05:50:08.612 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.613 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.614 DEBUG:boto:Params: {} 2022-09-02T05:50:08.615 DEBUG:boto:Token: None 2022-09-02T05:50:08.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2bb0b8ea10721b-0063119990-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.623 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.624 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.625 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.625 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.626 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:08.626 DEBUG:boto:Data: 2022-09-02T05:50:08.627 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.629 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.630 DEBUG:boto:Params: {} 2022-09-02T05:50:08.630 DEBUG:boto:Token: None 2022-09-02T05:50:08.631 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19 2022-09-02T05:50:08.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wa0sPWM/h6cQn/DNhbBJ8YQT7uY= 2022-09-02T05:50:08.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wa0sPWM/h6cQn/DNhbBJ8YQT7uY=', 'Content-Length': '0'} 2022-09-02T05:50:08.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee97f2af391f7a74-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.636 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.637 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.637 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.638 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:08.638 DEBUG:boto:Data: 2022-09-02T05:50:08.639 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.641 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.642 DEBUG:boto:Params: {} 2022-09-02T05:50:08.642 DEBUG:boto:Token: None 2022-09-02T05:50:08.643 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19 2022-09-02T05:50:08.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wa0sPWM/h6cQn/DNhbBJ8YQT7uY= 2022-09-02T05:50:08.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wa0sPWM/h6cQn/DNhbBJ8YQT7uY=', 'Content-Length': '0'} 2022-09-02T05:50:08.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000666f64dc1cd16ce8-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:08.648 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.649 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.650 DEBUG:boto:Method: GET 2022-09-02T05:50:08.650 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:08.651 DEBUG:boto:Data: 2022-09-02T05:50:08.652 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.653 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.654 DEBUG:boto:Params: {} 2022-09-02T05:50:08.655 DEBUG:boto:Token: None 2022-09-02T05:50:08.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19 2022-09-02T05:50:08.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oKmski7wt2o7t4EJ5RAHBtQydUY= 2022-09-02T05:50:08.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oKmski7wt2o7t4EJ5RAHBtQydUY=', 'Content-Length': '0'} 2022-09-02T05:50:08.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e88063b768277a2a-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.702 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:08.703 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:08.704 DEBUG:boto:Method: GET 2022-09-02T05:50:08.705 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:08.706 DEBUG:boto:Data: 2022-09-02T05:50:08.707 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.709 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.710 DEBUG:boto:Params: {} 2022-09-02T05:50:08.711 DEBUG:boto:Token: None 2022-09-02T05:50:08.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj19 2022-09-02T05:50:08.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oKmski7wt2o7t4EJ5RAHBtQydUY= 2022-09-02T05:50:08.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oKmski7wt2o7t4EJ5RAHBtQydUY=', 'Content-Length': '0'} 2022-09-02T05:50:08.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b8b6a704dc6c11c-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.719 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.719 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.720 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.721 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.721 DEBUG:boto:Method: GET 2022-09-02T05:50:08.722 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.722 DEBUG:boto:Data: 2022-09-02T05:50:08.723 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.724 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.724 DEBUG:boto:Params: {} 2022-09-02T05:50:08.725 DEBUG:boto:Token: None 2022-09-02T05:50:08.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de18c74d6129b934-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.732 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.733 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.734 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.735 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.735 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.736 DEBUG:boto:Method: GET 2022-09-02T05:50:08.737 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:08.738 DEBUG:boto:Data: 2022-09-02T05:50:08.739 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.740 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.741 DEBUG:boto:Params: {} 2022-09-02T05:50:08.741 DEBUG:boto:Token: None 2022-09-02T05:50:08.742 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdead1b70f355c22-0063119990-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.791 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:08.792 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.793 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.794 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.795 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.795 DEBUG:boto:Method: GET 2022-09-02T05:50:08.796 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.796 DEBUG:boto:Data: 2022-09-02T05:50:08.797 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.798 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.798 DEBUG:boto:Params: {} 2022-09-02T05:50:08.799 DEBUG:boto:Token: None 2022-09-02T05:50:08.800 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE= 2022-09-02T05:50:08.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE=', 'Content-Length': '0'} 2022-09-02T05:50:08.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d052846f71a2bce7-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.807 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.807 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.808 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.809 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.809 DEBUG:boto:Method: GET 2022-09-02T05:50:08.810 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.810 DEBUG:boto:Data: 2022-09-02T05:50:08.811 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.812 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.813 DEBUG:boto:Params: {} 2022-09-02T05:50:08.813 DEBUG:boto:Token: None 2022-09-02T05:50:08.814 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE= 2022-09-02T05:50:08.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE=', 'Content-Length': '0'} 2022-09-02T05:50:08.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70da743c68c9c8d-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.820 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.820 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.821 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.821 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.822 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.823 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.823 DEBUG:boto:Data: 2022-09-02T05:50:08.824 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.825 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.826 DEBUG:boto:Params: {} 2022-09-02T05:50:08.826 DEBUG:boto:Token: None 2022-09-02T05:50:08.827 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pu38XTS7ILjorcMZWK1FunSXIb0= 2022-09-02T05:50:08.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pu38XTS7ILjorcMZWK1FunSXIb0=', 'Content-Length': '0'} 2022-09-02T05:50:08.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036996d954b6de904-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.831 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.831 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.832 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.832 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.833 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.834 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.834 DEBUG:boto:Data: 2022-09-02T05:50:08.835 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.836 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.836 DEBUG:boto:Params: {} 2022-09-02T05:50:08.837 DEBUG:boto:Token: None 2022-09-02T05:50:08.837 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pu38XTS7ILjorcMZWK1FunSXIb0= 2022-09-02T05:50:08.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pu38XTS7ILjorcMZWK1FunSXIb0=', 'Content-Length': '0'} 2022-09-02T05:50:08.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053dc74cc8d7fc53a-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:08.842 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.843 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.843 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.844 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.844 DEBUG:boto:Method: GET 2022-09-02T05:50:08.845 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.845 DEBUG:boto:Data: 2022-09-02T05:50:08.846 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.847 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.848 DEBUG:boto:Params: {} 2022-09-02T05:50:08.848 DEBUG:boto:Token: None 2022-09-02T05:50:08.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE= 2022-09-02T05:50:08.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE=', 'Content-Length': '0'} 2022-09-02T05:50:08.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011dc45db616bd7eb-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.853 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.854 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.854 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.855 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.855 DEBUG:boto:Method: GET 2022-09-02T05:50:08.856 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.856 DEBUG:boto:Data: 2022-09-02T05:50:08.857 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.858 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.858 DEBUG:boto:Params: {} 2022-09-02T05:50:08.859 DEBUG:boto:Token: None 2022-09-02T05:50:08.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:08.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE= 2022-09-02T05:50:08.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cFJn+IVUpXHkGKj3V1QBQ2wDDbE=', 'Content-Length': '0'} 2022-09-02T05:50:08.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007693cefe69a7d2f0-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.864 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.864 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.865 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.865 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.866 DEBUG:boto:Method: GET 2022-09-02T05:50:08.867 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.867 DEBUG:boto:Data: 2022-09-02T05:50:08.868 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.869 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.869 DEBUG:boto:Params: {} 2022-09-02T05:50:08.870 DEBUG:boto:Token: None 2022-09-02T05:50:08.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a9b7b3b413be67-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.875 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.875 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.876 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.876 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.877 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.877 DEBUG:boto:Method: GET 2022-09-02T05:50:08.878 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.879 DEBUG:boto:Data: 2022-09-02T05:50:08.879 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.880 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.881 DEBUG:boto:Params: {} 2022-09-02T05:50:08.881 DEBUG:boto:Token: None 2022-09-02T05:50:08.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f43e8a9e9b4cba-0063119990-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.930 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:08.931 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.932 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.933 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.934 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:08.934 DEBUG:boto:Data: 2022-09-02T05:50:08.935 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.936 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.937 DEBUG:boto:Params: {} 2022-09-02T05:50:08.938 DEBUG:boto:Token: None 2022-09-02T05:50:08.939 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22 2022-09-02T05:50:08.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8cTbaISFBU9gQNiK7zLi7FDA1M= 2022-09-02T05:50:08.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8cTbaISFBU9gQNiK7zLi7FDA1M=', 'Content-Length': '0'} 2022-09-02T05:50:08.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7c78a219b7cc5b-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.944 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.945 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.946 DEBUG:boto:Method: HEAD 2022-09-02T05:50:08.947 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:08.947 DEBUG:boto:Data: 2022-09-02T05:50:08.948 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.950 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.951 DEBUG:boto:Params: {} 2022-09-02T05:50:08.952 DEBUG:boto:Token: None 2022-09-02T05:50:08.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22 2022-09-02T05:50:08.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8cTbaISFBU9gQNiK7zLi7FDA1M= 2022-09-02T05:50:08.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8cTbaISFBU9gQNiK7zLi7FDA1M=', 'Content-Length': '0'} 2022-09-02T05:50:08.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bafb23fd9005ecc2-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:08.958 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.958 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.959 DEBUG:boto:Method: GET 2022-09-02T05:50:08.959 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:08.960 DEBUG:boto:Data: 2022-09-02T05:50:08.961 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.962 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.962 DEBUG:boto:Params: {} 2022-09-02T05:50:08.963 DEBUG:boto:Token: None 2022-09-02T05:50:08.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22 2022-09-02T05:50:08.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KG3S0Z2Pluqf3c6cEiR/QJff0fI= 2022-09-02T05:50:08.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KG3S0Z2Pluqf3c6cEiR/QJff0fI=', 'Content-Length': '0'} 2022-09-02T05:50:08.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000694dab102efe56e1-0063119990-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.967 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:08.968 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:08.969 DEBUG:boto:Method: GET 2022-09-02T05:50:08.969 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:08.970 DEBUG:boto:Data: 2022-09-02T05:50:08.970 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:08.971 DEBUG:boto:Port: 8000 2022-09-02T05:50:08.972 DEBUG:boto:Params: {} 2022-09-02T05:50:08.972 DEBUG:boto:Token: None 2022-09-02T05:50:08.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/obj22 2022-09-02T05:50:08.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KG3S0Z2Pluqf3c6cEiR/QJff0fI= 2022-09-02T05:50:08.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KG3S0Z2Pluqf3c6cEiR/QJff0fI=', 'Content-Length': '0'} 2022-09-02T05:50:08.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3752984d420db4e-0063119990-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:08.977 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:08.978 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:08.978 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.979 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.979 DEBUG:boto:Method: GET 2022-09-02T05:50:08.980 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:08.980 DEBUG:boto:Data: 2022-09-02T05:50:08.981 DEBUG:boto:Headers: {} 2022-09-02T05:50:08.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:08.982 DEBUG:boto:Port: 8001 2022-09-02T05:50:08.983 DEBUG:boto:Params: {} 2022-09-02T05:50:08.983 DEBUG:boto:Token: None 2022-09-02T05:50:08.984 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:08 GMT /xljbkf-74/ 2022-09-02T05:50:08.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA= 2022-09-02T05:50:08.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:chBTrfLyPIyNJgENYU7c/D4boFA=', 'Content-Length': '0'} 2022-09-02T05:50:08.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006756dea75310d8f-0063119990-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.034 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.035 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.036 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.036 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:09.037 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:09.037 DEBUG:boto:Method: GET 2022-09-02T05:50:09.038 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:09.038 DEBUG:boto:Data: 2022-09-02T05:50:09.039 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.040 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.040 DEBUG:boto:Params: {} 2022-09-02T05:50:09.041 DEBUG:boto:Token: None 2022-09-02T05:50:09.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4edc0dd36a398a-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.047 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:09.048 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.049 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.050 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.050 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.051 DEBUG:boto:Method: GET 2022-09-02T05:50:09.051 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.052 DEBUG:boto:Data: 2022-09-02T05:50:09.052 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.053 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.054 DEBUG:boto:Params: {} 2022-09-02T05:50:09.054 DEBUG:boto:Token: None 2022-09-02T05:50:09.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA= 2022-09-02T05:50:09.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA=', 'Content-Length': '0'} 2022-09-02T05:50:09.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbbdc45b0058b4c7-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.059 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.060 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.060 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.061 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.061 DEBUG:boto:Method: GET 2022-09-02T05:50:09.062 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.062 DEBUG:boto:Data: 2022-09-02T05:50:09.063 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.064 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.065 DEBUG:boto:Params: {} 2022-09-02T05:50:09.065 DEBUG:boto:Token: None 2022-09-02T05:50:09.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA= 2022-09-02T05:50:09.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA=', 'Content-Length': '0'} 2022-09-02T05:50:09.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000588bc6c89f21b286-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.071 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.072 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.072 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.073 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.073 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.074 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.074 DEBUG:boto:Data: 2022-09-02T05:50:09.075 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.076 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.077 DEBUG:boto:Params: {} 2022-09-02T05:50:09.077 DEBUG:boto:Token: None 2022-09-02T05:50:09.078 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4d+JCjl7x4vt7lSA1nPSIRmIseU= 2022-09-02T05:50:09.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4d+JCjl7x4vt7lSA1nPSIRmIseU=', 'Content-Length': '0'} 2022-09-02T05:50:09.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c92a6e78dcebf16-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.082 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.082 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.083 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.083 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.084 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.085 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.085 DEBUG:boto:Data: 2022-09-02T05:50:09.086 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.087 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.087 DEBUG:boto:Params: {} 2022-09-02T05:50:09.088 DEBUG:boto:Token: None 2022-09-02T05:50:09.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4d+JCjl7x4vt7lSA1nPSIRmIseU= 2022-09-02T05:50:09.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4d+JCjl7x4vt7lSA1nPSIRmIseU=', 'Content-Length': '0'} 2022-09-02T05:50:09.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0949254f20b92b-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:09.093 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.094 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.094 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.095 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.095 DEBUG:boto:Method: GET 2022-09-02T05:50:09.096 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.096 DEBUG:boto:Data: 2022-09-02T05:50:09.097 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.098 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.098 DEBUG:boto:Params: {} 2022-09-02T05:50:09.099 DEBUG:boto:Token: None 2022-09-02T05:50:09.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA= 2022-09-02T05:50:09.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA=', 'Content-Length': '0'} 2022-09-02T05:50:09.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9871062be7137b4-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.104 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.105 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.105 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.106 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.106 DEBUG:boto:Method: GET 2022-09-02T05:50:09.107 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.107 DEBUG:boto:Data: 2022-09-02T05:50:09.108 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.109 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.110 DEBUG:boto:Params: {} 2022-09-02T05:50:09.110 DEBUG:boto:Token: None 2022-09-02T05:50:09.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:09.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA= 2022-09-02T05:50:09.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pmtcSkNl7UHmR9m00Z6/7gBH0GA=', 'Content-Length': '0'} 2022-09-02T05:50:09.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b19308ed927487-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.162 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.163 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.164 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.164 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.165 DEBUG:boto:Method: GET 2022-09-02T05:50:09.165 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.166 DEBUG:boto:Data: 2022-09-02T05:50:09.166 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.167 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.168 DEBUG:boto:Params: {} 2022-09-02T05:50:09.169 DEBUG:boto:Token: None 2022-09-02T05:50:09.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b2800b87219e00-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.174 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.175 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.176 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.176 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.177 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.177 DEBUG:boto:Method: GET 2022-09-02T05:50:09.178 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.179 DEBUG:boto:Data: 2022-09-02T05:50:09.180 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.182 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.183 DEBUG:boto:Params: {} 2022-09-02T05:50:09.184 DEBUG:boto:Token: None 2022-09-02T05:50:09.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001673672f0b293943-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.192 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.193 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.194 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.195 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.196 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:09.196 DEBUG:boto:Data: 2022-09-02T05:50:09.197 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.198 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.199 DEBUG:boto:Params: {} 2022-09-02T05:50:09.199 DEBUG:boto:Token: None 2022-09-02T05:50:09.200 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33 2022-09-02T05:50:09.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VTr2OKh9Fibnhxncz8/hG2Lr23c= 2022-09-02T05:50:09.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VTr2OKh9Fibnhxncz8/hG2Lr23c=', 'Content-Length': '0'} 2022-09-02T05:50:09.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036356efb033d4074-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.204 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.204 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.205 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.205 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:09.206 DEBUG:boto:Data: 2022-09-02T05:50:09.206 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.207 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.208 DEBUG:boto:Params: {} 2022-09-02T05:50:09.208 DEBUG:boto:Token: None 2022-09-02T05:50:09.209 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33 2022-09-02T05:50:09.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VTr2OKh9Fibnhxncz8/hG2Lr23c= 2022-09-02T05:50:09.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VTr2OKh9Fibnhxncz8/hG2Lr23c=', 'Content-Length': '0'} 2022-09-02T05:50:09.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552e3bd2c7fba1f6-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:09.214 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.214 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.215 DEBUG:boto:Method: GET 2022-09-02T05:50:09.215 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:09.216 DEBUG:boto:Data: 2022-09-02T05:50:09.217 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.218 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.218 DEBUG:boto:Params: {} 2022-09-02T05:50:09.219 DEBUG:boto:Token: None 2022-09-02T05:50:09.220 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33 2022-09-02T05:50:09.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2x9yFIyMrZe0pGeyp8GKMffFt2M= 2022-09-02T05:50:09.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2x9yFIyMrZe0pGeyp8GKMffFt2M=', 'Content-Length': '0'} 2022-09-02T05:50:09.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097914667a707f913-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.224 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:09.224 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:09.225 DEBUG:boto:Method: GET 2022-09-02T05:50:09.225 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:09.226 DEBUG:boto:Data: 2022-09-02T05:50:09.227 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.228 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.228 DEBUG:boto:Params: {} 2022-09-02T05:50:09.229 DEBUG:boto:Token: None 2022-09-02T05:50:09.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj33 2022-09-02T05:50:09.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2x9yFIyMrZe0pGeyp8GKMffFt2M= 2022-09-02T05:50:09.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2x9yFIyMrZe0pGeyp8GKMffFt2M=', 'Content-Length': '0'} 2022-09-02T05:50:09.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186f2391ef5458ea-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.234 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.235 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.235 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.236 DEBUG:boto:Method: GET 2022-09-02T05:50:09.236 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.237 DEBUG:boto:Data: 2022-09-02T05:50:09.237 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.238 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.239 DEBUG:boto:Params: {} 2022-09-02T05:50:09.239 DEBUG:boto:Token: None 2022-09-02T05:50:09.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a5b20577fe4def2-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.245 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.245 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.246 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.247 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.247 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.248 DEBUG:boto:Method: GET 2022-09-02T05:50:09.248 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:09.249 DEBUG:boto:Data: 2022-09-02T05:50:09.249 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.250 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.251 DEBUG:boto:Params: {} 2022-09-02T05:50:09.251 DEBUG:boto:Token: None 2022-09-02T05:50:09.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d08f65faaaa61053-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.257 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:09.259 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.259 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.260 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.260 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.261 DEBUG:boto:Method: GET 2022-09-02T05:50:09.261 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.262 DEBUG:boto:Data: 2022-09-02T05:50:09.262 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.263 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.264 DEBUG:boto:Params: {} 2022-09-02T05:50:09.265 DEBUG:boto:Token: None 2022-09-02T05:50:09.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA= 2022-09-02T05:50:09.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA=', 'Content-Length': '0'} 2022-09-02T05:50:09.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f653908bf743495-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.269 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.270 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.270 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.271 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.272 DEBUG:boto:Method: GET 2022-09-02T05:50:09.272 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.273 DEBUG:boto:Data: 2022-09-02T05:50:09.273 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.274 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.275 DEBUG:boto:Params: {} 2022-09-02T05:50:09.275 DEBUG:boto:Token: None 2022-09-02T05:50:09.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA= 2022-09-02T05:50:09.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA=', 'Content-Length': '0'} 2022-09-02T05:50:09.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861c7594d4674f87-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.280 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.281 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.281 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.282 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.282 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.283 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.283 DEBUG:boto:Data: 2022-09-02T05:50:09.284 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.285 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.285 DEBUG:boto:Params: {} 2022-09-02T05:50:09.286 DEBUG:boto:Token: None 2022-09-02T05:50:09.287 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j4TRsHnlHn2/fMgtTohPU/vNwtY= 2022-09-02T05:50:09.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j4TRsHnlHn2/fMgtTohPU/vNwtY=', 'Content-Length': '0'} 2022-09-02T05:50:09.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f6d10c2e52e3ee-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.291 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.292 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.293 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.294 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.295 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.296 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.297 DEBUG:boto:Data: 2022-09-02T05:50:09.298 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.300 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.301 DEBUG:boto:Params: {} 2022-09-02T05:50:09.302 DEBUG:boto:Token: None 2022-09-02T05:50:09.303 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j4TRsHnlHn2/fMgtTohPU/vNwtY= 2022-09-02T05:50:09.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j4TRsHnlHn2/fMgtTohPU/vNwtY=', 'Content-Length': '0'} 2022-09-02T05:50:09.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1f1cab20aa6c9f-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:09.308 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.308 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.309 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.309 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.310 DEBUG:boto:Method: GET 2022-09-02T05:50:09.311 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.311 DEBUG:boto:Data: 2022-09-02T05:50:09.312 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.313 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.314 DEBUG:boto:Params: {} 2022-09-02T05:50:09.314 DEBUG:boto:Token: None 2022-09-02T05:50:09.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA= 2022-09-02T05:50:09.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA=', 'Content-Length': '0'} 2022-09-02T05:50:09.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000073b26b1dbe64cdf-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.366 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.367 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.368 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.368 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.369 DEBUG:boto:Method: GET 2022-09-02T05:50:09.370 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.371 DEBUG:boto:Data: 2022-09-02T05:50:09.371 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.373 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.373 DEBUG:boto:Params: {} 2022-09-02T05:50:09.374 DEBUG:boto:Token: None 2022-09-02T05:50:09.375 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:09.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA= 2022-09-02T05:50:09.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3SURvx4bKlwjYGoax3QkLaRc4PA=', 'Content-Length': '0'} 2022-09-02T05:50:09.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009106d80c6c8b396a-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.379 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.380 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.380 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.381 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.381 DEBUG:boto:Method: GET 2022-09-02T05:50:09.382 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.382 DEBUG:boto:Data: 2022-09-02T05:50:09.383 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.384 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.384 DEBUG:boto:Params: {} 2022-09-02T05:50:09.385 DEBUG:boto:Token: None 2022-09-02T05:50:09.386 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e316a2a1b9c76240-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.391 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.392 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.392 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.393 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.393 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.394 DEBUG:boto:Method: GET 2022-09-02T05:50:09.395 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.395 DEBUG:boto:Data: 2022-09-02T05:50:09.396 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.397 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.398 DEBUG:boto:Params: {} 2022-09-02T05:50:09.399 DEBUG:boto:Token: None 2022-09-02T05:50:09.399 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.400 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e7216a60e93dba3-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.450 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.452 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.453 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.454 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.455 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:09.457 DEBUG:boto:Data: 2022-09-02T05:50:09.458 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.460 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.461 DEBUG:boto:Params: {} 2022-09-02T05:50:09.462 DEBUG:boto:Token: None 2022-09-02T05:50:09.463 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34 2022-09-02T05:50:09.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EkWCQ6q/f+x6bwjQMcDxFrJGTwA= 2022-09-02T05:50:09.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EkWCQ6q/f+x6bwjQMcDxFrJGTwA=', 'Content-Length': '0'} 2022-09-02T05:50:09.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99ac489d953400e-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.469 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.469 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.470 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.470 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:09.471 DEBUG:boto:Data: 2022-09-02T05:50:09.471 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.472 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.473 DEBUG:boto:Params: {} 2022-09-02T05:50:09.474 DEBUG:boto:Token: None 2022-09-02T05:50:09.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34 2022-09-02T05:50:09.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EkWCQ6q/f+x6bwjQMcDxFrJGTwA= 2022-09-02T05:50:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EkWCQ6q/f+x6bwjQMcDxFrJGTwA=', 'Content-Length': '0'} 2022-09-02T05:50:09.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331582f3463b2c95-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:09.480 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.480 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.481 DEBUG:boto:Method: GET 2022-09-02T05:50:09.482 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:09.482 DEBUG:boto:Data: 2022-09-02T05:50:09.483 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.484 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.484 DEBUG:boto:Params: {} 2022-09-02T05:50:09.485 DEBUG:boto:Token: None 2022-09-02T05:50:09.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34 2022-09-02T05:50:09.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaNGf+c6cL3f8akrMBJjFQTy5iE= 2022-09-02T05:50:09.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaNGf+c6cL3f8akrMBJjFQTy5iE=', 'Content-Length': '0'} 2022-09-02T05:50:09.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dc7b61eee03005-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.490 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:09.491 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:09.491 DEBUG:boto:Method: GET 2022-09-02T05:50:09.492 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:09.492 DEBUG:boto:Data: 2022-09-02T05:50:09.493 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.494 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.494 DEBUG:boto:Params: {} 2022-09-02T05:50:09.495 DEBUG:boto:Token: None 2022-09-02T05:50:09.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj34 2022-09-02T05:50:09.496 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaNGf+c6cL3f8akrMBJjFQTy5iE= 2022-09-02T05:50:09.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaNGf+c6cL3f8akrMBJjFQTy5iE=', 'Content-Length': '0'} 2022-09-02T05:50:09.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d33bececb9932cab-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.501 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.502 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.503 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.503 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.504 DEBUG:boto:Method: GET 2022-09-02T05:50:09.506 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.506 DEBUG:boto:Data: 2022-09-02T05:50:09.507 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.508 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.509 DEBUG:boto:Params: {} 2022-09-02T05:50:09.509 DEBUG:boto:Token: None 2022-09-02T05:50:09.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072da7bd9008995c7-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.515 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.516 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.517 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.517 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.518 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.518 DEBUG:boto:Method: GET 2022-09-02T05:50:09.519 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:09.520 DEBUG:boto:Data: 2022-09-02T05:50:09.520 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.521 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.522 DEBUG:boto:Params: {} 2022-09-02T05:50:09.523 DEBUG:boto:Token: None 2022-09-02T05:50:09.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2ecbd8a1d9c6f8-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.529 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:09.531 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.531 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.532 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.533 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.533 DEBUG:boto:Method: GET 2022-09-02T05:50:09.534 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.534 DEBUG:boto:Data: 2022-09-02T05:50:09.535 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.536 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.537 DEBUG:boto:Params: {} 2022-09-02T05:50:09.538 DEBUG:boto:Token: None 2022-09-02T05:50:09.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk= 2022-09-02T05:50:09.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk=', 'Content-Length': '0'} 2022-09-02T05:50:09.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921c5400bb2f2cb4-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.544 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.545 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.546 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.546 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.547 DEBUG:boto:Method: GET 2022-09-02T05:50:09.548 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.549 DEBUG:boto:Data: 2022-09-02T05:50:09.549 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.551 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.551 DEBUG:boto:Params: {} 2022-09-02T05:50:09.552 DEBUG:boto:Token: None 2022-09-02T05:50:09.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk= 2022-09-02T05:50:09.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk=', 'Content-Length': '0'} 2022-09-02T05:50:09.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c1a6bcfc4d00f9b-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.559 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.560 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.560 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.561 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.561 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.562 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.563 DEBUG:boto:Data: 2022-09-02T05:50:09.563 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.565 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.565 DEBUG:boto:Params: {} 2022-09-02T05:50:09.566 DEBUG:boto:Token: None 2022-09-02T05:50:09.567 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZHsymqJEcrDE/teZ0J9RB2vEKxk= 2022-09-02T05:50:09.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZHsymqJEcrDE/teZ0J9RB2vEKxk=', 'Content-Length': '0'} 2022-09-02T05:50:09.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a13d8639195c41-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.571 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.572 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.573 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.573 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.574 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.575 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.575 DEBUG:boto:Data: 2022-09-02T05:50:09.576 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.577 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.578 DEBUG:boto:Params: {} 2022-09-02T05:50:09.578 DEBUG:boto:Token: None 2022-09-02T05:50:09.579 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZHsymqJEcrDE/teZ0J9RB2vEKxk= 2022-09-02T05:50:09.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZHsymqJEcrDE/teZ0J9RB2vEKxk=', 'Content-Length': '0'} 2022-09-02T05:50:09.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007572b41f2555aeac-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:09.585 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.585 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.586 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.587 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.588 DEBUG:boto:Method: GET 2022-09-02T05:50:09.588 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.589 DEBUG:boto:Data: 2022-09-02T05:50:09.590 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.591 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.592 DEBUG:boto:Params: {} 2022-09-02T05:50:09.592 DEBUG:boto:Token: None 2022-09-02T05:50:09.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk= 2022-09-02T05:50:09.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk=', 'Content-Length': '0'} 2022-09-02T05:50:09.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005731bffe29a21b1a-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.638 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.639 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.639 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.640 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.641 DEBUG:boto:Method: GET 2022-09-02T05:50:09.641 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.642 DEBUG:boto:Data: 2022-09-02T05:50:09.642 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.643 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.644 DEBUG:boto:Params: {} 2022-09-02T05:50:09.645 DEBUG:boto:Token: None 2022-09-02T05:50:09.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:09.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk= 2022-09-02T05:50:09.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LaxgKNtZ6gPVmROJ3L4ZYWzRFwk=', 'Content-Length': '0'} 2022-09-02T05:50:09.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025186f06b9e114b9-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.650 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.651 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.651 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.652 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.652 DEBUG:boto:Method: GET 2022-09-02T05:50:09.653 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.654 DEBUG:boto:Data: 2022-09-02T05:50:09.654 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.655 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.656 DEBUG:boto:Params: {} 2022-09-02T05:50:09.657 DEBUG:boto:Token: None 2022-09-02T05:50:09.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd5d1c096245c6ac-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.663 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.663 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.664 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.664 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.665 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.665 DEBUG:boto:Method: GET 2022-09-02T05:50:09.667 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.667 DEBUG:boto:Data: 2022-09-02T05:50:09.668 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.669 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.669 DEBUG:boto:Params: {} 2022-09-02T05:50:09.670 DEBUG:boto:Token: None 2022-09-02T05:50:09.670 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7db4980dc0412a9-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.676 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.676 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.677 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.677 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.678 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:09.678 DEBUG:boto:Data: 2022-09-02T05:50:09.679 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.680 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.681 DEBUG:boto:Params: {} 2022-09-02T05:50:09.682 DEBUG:boto:Token: None 2022-09-02T05:50:09.683 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38 2022-09-02T05:50:09.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SuC68Deonf1/rsiR1BH9PbVe2TM= 2022-09-02T05:50:09.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SuC68Deonf1/rsiR1BH9PbVe2TM=', 'Content-Length': '0'} 2022-09-02T05:50:09.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abfbdfdd9a5fcc20-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.687 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.687 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.688 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.688 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:09.689 DEBUG:boto:Data: 2022-09-02T05:50:09.689 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.690 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.691 DEBUG:boto:Params: {} 2022-09-02T05:50:09.691 DEBUG:boto:Token: None 2022-09-02T05:50:09.692 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38 2022-09-02T05:50:09.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SuC68Deonf1/rsiR1BH9PbVe2TM= 2022-09-02T05:50:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SuC68Deonf1/rsiR1BH9PbVe2TM=', 'Content-Length': '0'} 2022-09-02T05:50:09.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2e1c6649dd160b-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:09.697 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.698 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.698 DEBUG:boto:Method: GET 2022-09-02T05:50:09.699 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:09.699 DEBUG:boto:Data: 2022-09-02T05:50:09.700 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.701 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.702 DEBUG:boto:Params: {} 2022-09-02T05:50:09.702 DEBUG:boto:Token: None 2022-09-02T05:50:09.703 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38 2022-09-02T05:50:09.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tEjYQwi0LmDQw/QUmoFuoeaQ4CY= 2022-09-02T05:50:09.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tEjYQwi0LmDQw/QUmoFuoeaQ4CY=', 'Content-Length': '0'} 2022-09-02T05:50:09.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca9b133bebc4b16-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.707 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:09.708 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:09.709 DEBUG:boto:Method: GET 2022-09-02T05:50:09.710 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:09.711 DEBUG:boto:Data: 2022-09-02T05:50:09.712 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.713 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.714 DEBUG:boto:Params: {} 2022-09-02T05:50:09.714 DEBUG:boto:Token: None 2022-09-02T05:50:09.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj38 2022-09-02T05:50:09.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tEjYQwi0LmDQw/QUmoFuoeaQ4CY= 2022-09-02T05:50:09.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tEjYQwi0LmDQw/QUmoFuoeaQ4CY=', 'Content-Length': '0'} 2022-09-02T05:50:09.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d974f58fb71c13-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.719 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.720 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.720 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.721 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.721 DEBUG:boto:Method: GET 2022-09-02T05:50:09.722 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.723 DEBUG:boto:Data: 2022-09-02T05:50:09.723 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.724 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.725 DEBUG:boto:Params: {} 2022-09-02T05:50:09.725 DEBUG:boto:Token: None 2022-09-02T05:50:09.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076faa5f22760d1e2-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.731 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.732 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.732 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.733 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.733 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.734 DEBUG:boto:Method: GET 2022-09-02T05:50:09.735 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:09.736 DEBUG:boto:Data: 2022-09-02T05:50:09.737 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.738 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.739 DEBUG:boto:Params: {} 2022-09-02T05:50:09.740 DEBUG:boto:Token: None 2022-09-02T05:50:09.741 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030fcbd18fdd329c5-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.746 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:09.748 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.748 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.749 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.750 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.750 DEBUG:boto:Method: GET 2022-09-02T05:50:09.751 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.751 DEBUG:boto:Data: 2022-09-02T05:50:09.752 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.753 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.754 DEBUG:boto:Params: {} 2022-09-02T05:50:09.754 DEBUG:boto:Token: None 2022-09-02T05:50:09.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk= 2022-09-02T05:50:09.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk=', 'Content-Length': '0'} 2022-09-02T05:50:09.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c77a646f4aed18f-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.802 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.803 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.804 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.804 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.805 DEBUG:boto:Method: GET 2022-09-02T05:50:09.805 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.806 DEBUG:boto:Data: 2022-09-02T05:50:09.806 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.807 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.808 DEBUG:boto:Params: {} 2022-09-02T05:50:09.809 DEBUG:boto:Token: None 2022-09-02T05:50:09.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk= 2022-09-02T05:50:09.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk=', 'Content-Length': '0'} 2022-09-02T05:50:09.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaab00f8d6f175d2-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.819 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.820 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.820 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.821 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.821 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.822 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.822 DEBUG:boto:Data: 2022-09-02T05:50:09.823 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.824 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.824 DEBUG:boto:Params: {} 2022-09-02T05:50:09.825 DEBUG:boto:Token: None 2022-09-02T05:50:09.826 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fwda8v0wg9vcvlpqwLmRI+3ueoQ= 2022-09-02T05:50:09.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fwda8v0wg9vcvlpqwLmRI+3ueoQ=', 'Content-Length': '0'} 2022-09-02T05:50:09.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003573777a0984b163-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.830 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.830 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.831 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.832 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.833 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.833 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.834 DEBUG:boto:Data: 2022-09-02T05:50:09.835 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.836 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.836 DEBUG:boto:Params: {} 2022-09-02T05:50:09.837 DEBUG:boto:Token: None 2022-09-02T05:50:09.838 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fwda8v0wg9vcvlpqwLmRI+3ueoQ= 2022-09-02T05:50:09.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fwda8v0wg9vcvlpqwLmRI+3ueoQ=', 'Content-Length': '0'} 2022-09-02T05:50:09.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7e6c051bd205787-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:09.844 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.845 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.845 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.846 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.846 DEBUG:boto:Method: GET 2022-09-02T05:50:09.847 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.847 DEBUG:boto:Data: 2022-09-02T05:50:09.848 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.849 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.849 DEBUG:boto:Params: {} 2022-09-02T05:50:09.850 DEBUG:boto:Token: None 2022-09-02T05:50:09.851 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk= 2022-09-02T05:50:09.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk=', 'Content-Length': '0'} 2022-09-02T05:50:09.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ecfd862a404186-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.856 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.857 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.858 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.858 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.859 DEBUG:boto:Method: GET 2022-09-02T05:50:09.860 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.861 DEBUG:boto:Data: 2022-09-02T05:50:09.862 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.863 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.864 DEBUG:boto:Params: {} 2022-09-02T05:50:09.864 DEBUG:boto:Token: None 2022-09-02T05:50:09.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:09.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk= 2022-09-02T05:50:09.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3wwy1lvWWzPwHCN/CiAhTWyggyk=', 'Content-Length': '0'} 2022-09-02T05:50:09.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000995bd2d0516e0a9e-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.910 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.911 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.912 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.913 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.914 DEBUG:boto:Method: GET 2022-09-02T05:50:09.915 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.915 DEBUG:boto:Data: 2022-09-02T05:50:09.916 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.917 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.917 DEBUG:boto:Params: {} 2022-09-02T05:50:09.918 DEBUG:boto:Token: None 2022-09-02T05:50:09.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ca0376e42bedd4-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.924 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.925 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.926 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.927 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.928 DEBUG:boto:Method: GET 2022-09-02T05:50:09.928 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.929 DEBUG:boto:Data: 2022-09-02T05:50:09.929 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.931 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.932 DEBUG:boto:Params: {} 2022-09-02T05:50:09.932 DEBUG:boto:Token: None 2022-09-02T05:50:09.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:09.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d153a08c4df7fb-0063119991-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.939 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:09.940 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.941 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.941 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.942 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:09.942 DEBUG:boto:Data: 2022-09-02T05:50:09.943 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.945 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.946 DEBUG:boto:Params: {} 2022-09-02T05:50:09.947 DEBUG:boto:Token: None 2022-09-02T05:50:09.948 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44 2022-09-02T05:50:09.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IzgH4J8VEGZXunsGqNFEishMi5Q= 2022-09-02T05:50:09.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IzgH4J8VEGZXunsGqNFEishMi5Q=', 'Content-Length': '0'} 2022-09-02T05:50:09.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e344cd5a66fc2544-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.953 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.954 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.954 DEBUG:boto:Method: HEAD 2022-09-02T05:50:09.955 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:09.956 DEBUG:boto:Data: 2022-09-02T05:50:09.956 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.957 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.958 DEBUG:boto:Params: {} 2022-09-02T05:50:09.959 DEBUG:boto:Token: None 2022-09-02T05:50:09.959 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44 2022-09-02T05:50:09.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IzgH4J8VEGZXunsGqNFEishMi5Q= 2022-09-02T05:50:09.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IzgH4J8VEGZXunsGqNFEishMi5Q=', 'Content-Length': '0'} 2022-09-02T05:50:09.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000981474526d723ee2-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:09.965 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.965 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.966 DEBUG:boto:Method: GET 2022-09-02T05:50:09.966 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:09.967 DEBUG:boto:Data: 2022-09-02T05:50:09.968 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.969 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.970 DEBUG:boto:Params: {} 2022-09-02T05:50:09.971 DEBUG:boto:Token: None 2022-09-02T05:50:09.972 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44 2022-09-02T05:50:09.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ansbuxx2b4FLBN15ZJGSgThYydY= 2022-09-02T05:50:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ansbuxx2b4FLBN15ZJGSgThYydY=', 'Content-Length': '0'} 2022-09-02T05:50:09.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000270bbcbe4afa7444-0063119991-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.976 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:09.977 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:09.978 DEBUG:boto:Method: GET 2022-09-02T05:50:09.978 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:09.979 DEBUG:boto:Data: 2022-09-02T05:50:09.979 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:09.980 DEBUG:boto:Port: 8000 2022-09-02T05:50:09.981 DEBUG:boto:Params: {} 2022-09-02T05:50:09.981 DEBUG:boto:Token: None 2022-09-02T05:50:09.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/obj44 2022-09-02T05:50:09.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ansbuxx2b4FLBN15ZJGSgThYydY= 2022-09-02T05:50:09.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ansbuxx2b4FLBN15ZJGSgThYydY=', 'Content-Length': '0'} 2022-09-02T05:50:09.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0f607ce4d499d6-0063119991-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:09.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:09.987 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:09.988 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.989 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.990 DEBUG:boto:Method: GET 2022-09-02T05:50:09.991 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:09.992 DEBUG:boto:Data: 2022-09-02T05:50:09.993 DEBUG:boto:Headers: {} 2022-09-02T05:50:09.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:09.994 DEBUG:boto:Port: 8001 2022-09-02T05:50:09.995 DEBUG:boto:Params: {} 2022-09-02T05:50:09.995 DEBUG:boto:Token: None 2022-09-02T05:50:09.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:09 GMT /xljbkf-74/ 2022-09-02T05:50:09.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk= 2022-09-02T05:50:09.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+yUXsplJUZRX0Fop5E9iQjpVWk=', 'Content-Length': '0'} 2022-09-02T05:50:10.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8e5a4cb75f5987-0063119991-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.001 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.002 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.003 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.003 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:10.004 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:10.004 DEBUG:boto:Method: GET 2022-09-02T05:50:10.005 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:10.006 DEBUG:boto:Data: 2022-09-02T05:50:10.006 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.008 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.008 DEBUG:boto:Params: {} 2022-09-02T05:50:10.009 DEBUG:boto:Token: None 2022-09-02T05:50:10.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001411039081be4f80-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.016 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:10.018 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.019 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.020 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.021 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.022 DEBUG:boto:Method: GET 2022-09-02T05:50:10.023 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.024 DEBUG:boto:Data: 2022-09-02T05:50:10.025 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.027 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.028 DEBUG:boto:Params: {} 2022-09-02T05:50:10.029 DEBUG:boto:Token: None 2022-09-02T05:50:10.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww= 2022-09-02T05:50:10.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww=', 'Content-Length': '0'} 2022-09-02T05:50:10.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008737a1240d8c56a5-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.036 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.037 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.038 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.038 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.039 DEBUG:boto:Method: GET 2022-09-02T05:50:10.040 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.041 DEBUG:boto:Data: 2022-09-02T05:50:10.041 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.043 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.043 DEBUG:boto:Params: {} 2022-09-02T05:50:10.044 DEBUG:boto:Token: None 2022-09-02T05:50:10.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww= 2022-09-02T05:50:10.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww=', 'Content-Length': '0'} 2022-09-02T05:50:10.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429a6847c4000614-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.051 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.051 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.052 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.053 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.053 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.054 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.055 DEBUG:boto:Data: 2022-09-02T05:50:10.055 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.057 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.058 DEBUG:boto:Params: {} 2022-09-02T05:50:10.060 DEBUG:boto:Token: None 2022-09-02T05:50:10.061 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+RdpSAIElWdDonTWDLmiVm6EQIQ= 2022-09-02T05:50:10.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+RdpSAIElWdDonTWDLmiVm6EQIQ=', 'Content-Length': '0'} 2022-09-02T05:50:10.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ae0c274a8a8070-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.067 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.068 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.068 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.069 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.069 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.070 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.071 DEBUG:boto:Data: 2022-09-02T05:50:10.071 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.072 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.073 DEBUG:boto:Params: {} 2022-09-02T05:50:10.074 DEBUG:boto:Token: None 2022-09-02T05:50:10.074 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+RdpSAIElWdDonTWDLmiVm6EQIQ= 2022-09-02T05:50:10.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+RdpSAIElWdDonTWDLmiVm6EQIQ=', 'Content-Length': '0'} 2022-09-02T05:50:10.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce779f60e542d1e-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:10.079 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.080 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.081 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.081 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.082 DEBUG:boto:Method: GET 2022-09-02T05:50:10.082 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.083 DEBUG:boto:Data: 2022-09-02T05:50:10.084 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.085 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.086 DEBUG:boto:Params: {} 2022-09-02T05:50:10.086 DEBUG:boto:Token: None 2022-09-02T05:50:10.087 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww= 2022-09-02T05:50:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww=', 'Content-Length': '0'} 2022-09-02T05:50:10.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000666745747266e5c-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.134 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.136 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.137 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.138 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.140 DEBUG:boto:Method: GET 2022-09-02T05:50:10.141 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.143 DEBUG:boto:Data: 2022-09-02T05:50:10.144 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.147 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.148 DEBUG:boto:Params: {} 2022-09-02T05:50:10.150 DEBUG:boto:Token: None 2022-09-02T05:50:10.152 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:10.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww= 2022-09-02T05:50:10.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q3oO2lK1hUkYqcyeDDvWAdDS0Ww=', 'Content-Length': '0'} 2022-09-02T05:50:10.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f730985752049f7b-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.160 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.160 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.161 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.162 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.163 DEBUG:boto:Method: GET 2022-09-02T05:50:10.164 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.164 DEBUG:boto:Data: 2022-09-02T05:50:10.165 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.167 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.168 DEBUG:boto:Params: {} 2022-09-02T05:50:10.169 DEBUG:boto:Token: None 2022-09-02T05:50:10.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d592f2b8eeb24009-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.176 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.177 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.177 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.178 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.179 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.180 DEBUG:boto:Method: GET 2022-09-02T05:50:10.182 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.182 DEBUG:boto:Data: 2022-09-02T05:50:10.183 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.185 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.185 DEBUG:boto:Params: {} 2022-09-02T05:50:10.186 DEBUG:boto:Token: None 2022-09-02T05:50:10.187 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078bef2831d4d289c-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.194 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.195 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.195 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.196 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.197 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:10.198 DEBUG:boto:Data: 2022-09-02T05:50:10.199 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.200 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.201 DEBUG:boto:Params: {} 2022-09-02T05:50:10.201 DEBUG:boto:Token: None 2022-09-02T05:50:10.202 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51 2022-09-02T05:50:10.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZrSoeUYRzzG55/E6mZuJpllG/BI= 2022-09-02T05:50:10.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZrSoeUYRzzG55/E6mZuJpllG/BI=', 'Content-Length': '0'} 2022-09-02T05:50:10.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad7da3a7711c410-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.207 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.207 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.208 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.209 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:10.210 DEBUG:boto:Data: 2022-09-02T05:50:10.210 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.212 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.212 DEBUG:boto:Params: {} 2022-09-02T05:50:10.213 DEBUG:boto:Token: None 2022-09-02T05:50:10.214 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51 2022-09-02T05:50:10.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZrSoeUYRzzG55/E6mZuJpllG/BI= 2022-09-02T05:50:10.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZrSoeUYRzzG55/E6mZuJpllG/BI=', 'Content-Length': '0'} 2022-09-02T05:50:10.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d76b79dbc4703b0c-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:10.220 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.220 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.221 DEBUG:boto:Method: GET 2022-09-02T05:50:10.222 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:10.223 DEBUG:boto:Data: 2022-09-02T05:50:10.223 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.225 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.226 DEBUG:boto:Params: {} 2022-09-02T05:50:10.227 DEBUG:boto:Token: None 2022-09-02T05:50:10.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51 2022-09-02T05:50:10.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eU8kgRQ6fuao1zETayiG16SaYs8= 2022-09-02T05:50:10.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eU8kgRQ6fuao1zETayiG16SaYs8=', 'Content-Length': '0'} 2022-09-02T05:50:10.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a27ec5e7d81187b-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.278 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:10.279 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:10.279 DEBUG:boto:Method: GET 2022-09-02T05:50:10.280 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:10.280 DEBUG:boto:Data: 2022-09-02T05:50:10.281 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.282 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.283 DEBUG:boto:Params: {} 2022-09-02T05:50:10.284 DEBUG:boto:Token: None 2022-09-02T05:50:10.284 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj51 2022-09-02T05:50:10.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eU8kgRQ6fuao1zETayiG16SaYs8= 2022-09-02T05:50:10.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eU8kgRQ6fuao1zETayiG16SaYs8=', 'Content-Length': '0'} 2022-09-02T05:50:10.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5082bc1a2c40506-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.288 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.289 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.290 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.290 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.291 DEBUG:boto:Method: GET 2022-09-02T05:50:10.291 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.292 DEBUG:boto:Data: 2022-09-02T05:50:10.293 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.294 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.294 DEBUG:boto:Params: {} 2022-09-02T05:50:10.295 DEBUG:boto:Token: None 2022-09-02T05:50:10.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000731923fda2b0de14-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.300 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.301 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.302 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.302 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.303 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.304 DEBUG:boto:Method: GET 2022-09-02T05:50:10.304 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:10.305 DEBUG:boto:Data: 2022-09-02T05:50:10.305 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.306 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.307 DEBUG:boto:Params: {} 2022-09-02T05:50:10.307 DEBUG:boto:Token: None 2022-09-02T05:50:10.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005704542cde54bbdc-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.313 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:10.315 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.315 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.316 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.316 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.317 DEBUG:boto:Method: GET 2022-09-02T05:50:10.318 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.318 DEBUG:boto:Data: 2022-09-02T05:50:10.319 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.320 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.321 DEBUG:boto:Params: {} 2022-09-02T05:50:10.321 DEBUG:boto:Token: None 2022-09-02T05:50:10.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08= 2022-09-02T05:50:10.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08=', 'Content-Length': '0'} 2022-09-02T05:50:10.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d6cf6e3a3d617ae-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.326 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.327 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.327 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.328 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.328 DEBUG:boto:Method: GET 2022-09-02T05:50:10.329 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.330 DEBUG:boto:Data: 2022-09-02T05:50:10.330 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.331 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.332 DEBUG:boto:Params: {} 2022-09-02T05:50:10.332 DEBUG:boto:Token: None 2022-09-02T05:50:10.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08= 2022-09-02T05:50:10.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08=', 'Content-Length': '0'} 2022-09-02T05:50:10.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf95391cdd5fbecc-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.337 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.338 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.338 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.339 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.339 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.340 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.340 DEBUG:boto:Data: 2022-09-02T05:50:10.341 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.342 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.342 DEBUG:boto:Params: {} 2022-09-02T05:50:10.343 DEBUG:boto:Token: None 2022-09-02T05:50:10.343 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fodfXlg0zzOLAYg8mglgFui9r4w= 2022-09-02T05:50:10.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fodfXlg0zzOLAYg8mglgFui9r4w=', 'Content-Length': '0'} 2022-09-02T05:50:10.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003952005787e19de2-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.347 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.348 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.348 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.349 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.350 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.350 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.351 DEBUG:boto:Data: 2022-09-02T05:50:10.352 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.353 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.354 DEBUG:boto:Params: {} 2022-09-02T05:50:10.354 DEBUG:boto:Token: None 2022-09-02T05:50:10.355 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fodfXlg0zzOLAYg8mglgFui9r4w= 2022-09-02T05:50:10.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fodfXlg0zzOLAYg8mglgFui9r4w=', 'Content-Length': '0'} 2022-09-02T05:50:10.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4a607d14e018f9c-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:10.361 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.362 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.363 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.363 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.364 DEBUG:boto:Method: GET 2022-09-02T05:50:10.365 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.366 DEBUG:boto:Data: 2022-09-02T05:50:10.367 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.368 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.369 DEBUG:boto:Params: {} 2022-09-02T05:50:10.370 DEBUG:boto:Token: None 2022-09-02T05:50:10.371 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08= 2022-09-02T05:50:10.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08=', 'Content-Length': '0'} 2022-09-02T05:50:10.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000340b970e79a6ce5f-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.418 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.419 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.420 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.421 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.421 DEBUG:boto:Method: GET 2022-09-02T05:50:10.423 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.424 DEBUG:boto:Data: 2022-09-02T05:50:10.425 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.427 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.428 DEBUG:boto:Params: {} 2022-09-02T05:50:10.429 DEBUG:boto:Token: None 2022-09-02T05:50:10.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:10.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08= 2022-09-02T05:50:10.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9xhlrP4NfndeRvk0l/aw2nz/w08=', 'Content-Length': '0'} 2022-09-02T05:50:10.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8dcc80daa5efe9-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.437 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.438 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.439 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.440 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.441 DEBUG:boto:Method: GET 2022-09-02T05:50:10.442 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.443 DEBUG:boto:Data: 2022-09-02T05:50:10.444 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.445 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.446 DEBUG:boto:Params: {} 2022-09-02T05:50:10.448 DEBUG:boto:Token: None 2022-09-02T05:50:10.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d806a456d8077439-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.454 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.455 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.456 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.456 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.457 DEBUG:boto:Method: GET 2022-09-02T05:50:10.457 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.458 DEBUG:boto:Data: 2022-09-02T05:50:10.459 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.460 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.460 DEBUG:boto:Params: {} 2022-09-02T05:50:10.461 DEBUG:boto:Token: None 2022-09-02T05:50:10.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a96dffc1c3aead69-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.467 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.468 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.468 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.475 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.476 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:10.477 DEBUG:boto:Data: 2022-09-02T05:50:10.478 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.479 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.479 DEBUG:boto:Params: {} 2022-09-02T05:50:10.480 DEBUG:boto:Token: None 2022-09-02T05:50:10.480 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55 2022-09-02T05:50:10.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lMeuBGJ/hbeqxKTxt4AqwQMuWTE= 2022-09-02T05:50:10.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lMeuBGJ/hbeqxKTxt4AqwQMuWTE=', 'Content-Length': '0'} 2022-09-02T05:50:10.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047aabfd71787c941-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.484 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.485 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.485 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.486 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:10.486 DEBUG:boto:Data: 2022-09-02T05:50:10.487 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.488 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.489 DEBUG:boto:Params: {} 2022-09-02T05:50:10.489 DEBUG:boto:Token: None 2022-09-02T05:50:10.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55 2022-09-02T05:50:10.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lMeuBGJ/hbeqxKTxt4AqwQMuWTE= 2022-09-02T05:50:10.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lMeuBGJ/hbeqxKTxt4AqwQMuWTE=', 'Content-Length': '0'} 2022-09-02T05:50:10.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639ca9f0f6412115-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:10.494 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.495 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.495 DEBUG:boto:Method: GET 2022-09-02T05:50:10.496 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:10.496 DEBUG:boto:Data: 2022-09-02T05:50:10.497 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.498 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.498 DEBUG:boto:Params: {} 2022-09-02T05:50:10.499 DEBUG:boto:Token: None 2022-09-02T05:50:10.500 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55 2022-09-02T05:50:10.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ch5k/yn9jCIa8EqJ1fQWq6TCiXs= 2022-09-02T05:50:10.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ch5k/yn9jCIa8EqJ1fQWq6TCiXs=', 'Content-Length': '0'} 2022-09-02T05:50:10.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fcbba62b14654f2-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.504 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:10.505 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:10.505 DEBUG:boto:Method: GET 2022-09-02T05:50:10.506 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:10.506 DEBUG:boto:Data: 2022-09-02T05:50:10.507 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.508 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.508 DEBUG:boto:Params: {} 2022-09-02T05:50:10.509 DEBUG:boto:Token: None 2022-09-02T05:50:10.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj55 2022-09-02T05:50:10.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ch5k/yn9jCIa8EqJ1fQWq6TCiXs= 2022-09-02T05:50:10.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ch5k/yn9jCIa8EqJ1fQWq6TCiXs=', 'Content-Length': '0'} 2022-09-02T05:50:10.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26e608acf527184-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.515 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.515 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.516 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.516 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.517 DEBUG:boto:Method: GET 2022-09-02T05:50:10.517 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.518 DEBUG:boto:Data: 2022-09-02T05:50:10.518 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.520 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.520 DEBUG:boto:Params: {} 2022-09-02T05:50:10.521 DEBUG:boto:Token: None 2022-09-02T05:50:10.521 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.522 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a564dfe06dc448e-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.526 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.527 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.528 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.528 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.529 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.529 DEBUG:boto:Method: GET 2022-09-02T05:50:10.530 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:10.531 DEBUG:boto:Data: 2022-09-02T05:50:10.531 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.532 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.533 DEBUG:boto:Params: {} 2022-09-02T05:50:10.533 DEBUG:boto:Token: None 2022-09-02T05:50:10.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6af0011bba2b850-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.545 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:10.546 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.547 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.547 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.548 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.548 DEBUG:boto:Method: GET 2022-09-02T05:50:10.549 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.550 DEBUG:boto:Data: 2022-09-02T05:50:10.550 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.551 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.552 DEBUG:boto:Params: {} 2022-09-02T05:50:10.552 DEBUG:boto:Token: None 2022-09-02T05:50:10.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g= 2022-09-02T05:50:10.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g=', 'Content-Length': '0'} 2022-09-02T05:50:10.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8d342682822c28-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.557 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.558 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.558 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.559 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.560 DEBUG:boto:Method: GET 2022-09-02T05:50:10.560 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.561 DEBUG:boto:Data: 2022-09-02T05:50:10.561 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.562 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.563 DEBUG:boto:Params: {} 2022-09-02T05:50:10.563 DEBUG:boto:Token: None 2022-09-02T05:50:10.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g= 2022-09-02T05:50:10.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g=', 'Content-Length': '0'} 2022-09-02T05:50:10.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005af268bc55ace4d-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.568 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.569 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.569 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.570 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.570 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.571 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.571 DEBUG:boto:Data: 2022-09-02T05:50:10.572 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.573 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.573 DEBUG:boto:Params: {} 2022-09-02T05:50:10.574 DEBUG:boto:Token: None 2022-09-02T05:50:10.574 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QeepPgYBXzmMW4PEgX98PIWFTkM= 2022-09-02T05:50:10.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QeepPgYBXzmMW4PEgX98PIWFTkM=', 'Content-Length': '0'} 2022-09-02T05:50:10.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd67c3d81951b2f-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.578 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.579 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.579 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.580 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.580 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.581 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.581 DEBUG:boto:Data: 2022-09-02T05:50:10.582 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.583 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.583 DEBUG:boto:Params: {} 2022-09-02T05:50:10.584 DEBUG:boto:Token: None 2022-09-02T05:50:10.585 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QeepPgYBXzmMW4PEgX98PIWFTkM= 2022-09-02T05:50:10.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QeepPgYBXzmMW4PEgX98PIWFTkM=', 'Content-Length': '0'} 2022-09-02T05:50:10.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cef7c9149f98a6c-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:10.591 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.591 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.592 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.592 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.593 DEBUG:boto:Method: GET 2022-09-02T05:50:10.593 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.594 DEBUG:boto:Data: 2022-09-02T05:50:10.594 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.595 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.596 DEBUG:boto:Params: {} 2022-09-02T05:50:10.596 DEBUG:boto:Token: None 2022-09-02T05:50:10.597 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g= 2022-09-02T05:50:10.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g=', 'Content-Length': '0'} 2022-09-02T05:50:10.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e03fd3a77bdebe9-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.601 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.602 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.602 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.603 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.603 DEBUG:boto:Method: GET 2022-09-02T05:50:10.604 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.604 DEBUG:boto:Data: 2022-09-02T05:50:10.605 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.606 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.607 DEBUG:boto:Params: {} 2022-09-02T05:50:10.607 DEBUG:boto:Token: None 2022-09-02T05:50:10.608 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:10.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g= 2022-09-02T05:50:10.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dSOIrVnF2c6SWRc9H/fllKTnM4g=', 'Content-Length': '0'} 2022-09-02T05:50:10.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ffe8fc7c344db7-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.612 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.613 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.613 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.614 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.614 DEBUG:boto:Method: GET 2022-09-02T05:50:10.615 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.615 DEBUG:boto:Data: 2022-09-02T05:50:10.616 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.617 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.617 DEBUG:boto:Params: {} 2022-09-02T05:50:10.619 DEBUG:boto:Token: None 2022-09-02T05:50:10.620 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000432ab1f3436814bb-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.627 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.628 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.630 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.630 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.631 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.631 DEBUG:boto:Method: GET 2022-09-02T05:50:10.632 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.633 DEBUG:boto:Data: 2022-09-02T05:50:10.633 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.635 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.635 DEBUG:boto:Params: {} 2022-09-02T05:50:10.636 DEBUG:boto:Token: None 2022-09-02T05:50:10.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ec25e6f5a84c91e-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.652 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.653 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.654 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.655 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.655 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:10.656 DEBUG:boto:Data: 2022-09-02T05:50:10.656 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.657 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.658 DEBUG:boto:Params: {} 2022-09-02T05:50:10.659 DEBUG:boto:Token: None 2022-09-02T05:50:10.659 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57 2022-09-02T05:50:10.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xeVhAzOjUp5nk0YS0gF30S8yD6g= 2022-09-02T05:50:10.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xeVhAzOjUp5nk0YS0gF30S8yD6g=', 'Content-Length': '0'} 2022-09-02T05:50:10.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b71a57ad690fd3-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.663 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.663 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.664 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.665 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:10.665 DEBUG:boto:Data: 2022-09-02T05:50:10.666 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.667 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.667 DEBUG:boto:Params: {} 2022-09-02T05:50:10.668 DEBUG:boto:Token: None 2022-09-02T05:50:10.668 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57 2022-09-02T05:50:10.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xeVhAzOjUp5nk0YS0gF30S8yD6g= 2022-09-02T05:50:10.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xeVhAzOjUp5nk0YS0gF30S8yD6g=', 'Content-Length': '0'} 2022-09-02T05:50:10.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b8b8926bdc231b-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:10.674 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.675 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.675 DEBUG:boto:Method: GET 2022-09-02T05:50:10.676 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:10.676 DEBUG:boto:Data: 2022-09-02T05:50:10.677 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.678 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.678 DEBUG:boto:Params: {} 2022-09-02T05:50:10.679 DEBUG:boto:Token: None 2022-09-02T05:50:10.680 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57 2022-09-02T05:50:10.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8lQGBJOkZ8deQe2eexVxbWA9E0s= 2022-09-02T05:50:10.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8lQGBJOkZ8deQe2eexVxbWA9E0s=', 'Content-Length': '0'} 2022-09-02T05:50:10.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f3ac60eba62e99-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.730 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:10.731 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:10.732 DEBUG:boto:Method: GET 2022-09-02T05:50:10.733 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:10.733 DEBUG:boto:Data: 2022-09-02T05:50:10.734 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.735 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.736 DEBUG:boto:Params: {} 2022-09-02T05:50:10.736 DEBUG:boto:Token: None 2022-09-02T05:50:10.737 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj57 2022-09-02T05:50:10.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8lQGBJOkZ8deQe2eexVxbWA9E0s= 2022-09-02T05:50:10.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8lQGBJOkZ8deQe2eexVxbWA9E0s=', 'Content-Length': '0'} 2022-09-02T05:50:10.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8de0a84cf948862-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.741 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.742 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.742 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.743 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.743 DEBUG:boto:Method: GET 2022-09-02T05:50:10.744 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.745 DEBUG:boto:Data: 2022-09-02T05:50:10.745 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.747 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.748 DEBUG:boto:Params: {} 2022-09-02T05:50:10.749 DEBUG:boto:Token: None 2022-09-02T05:50:10.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6dbcde0b5e61cbc-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.754 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.755 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.755 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.756 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.756 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.757 DEBUG:boto:Method: GET 2022-09-02T05:50:10.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:10.758 DEBUG:boto:Data: 2022-09-02T05:50:10.759 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.760 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.761 DEBUG:boto:Params: {} 2022-09-02T05:50:10.762 DEBUG:boto:Token: None 2022-09-02T05:50:10.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000438f1e4343dc5442-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.770 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:10.773 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.773 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.774 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.775 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.777 DEBUG:boto:Method: GET 2022-09-02T05:50:10.778 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.779 DEBUG:boto:Data: 2022-09-02T05:50:10.780 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.782 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.783 DEBUG:boto:Params: {} 2022-09-02T05:50:10.784 DEBUG:boto:Token: None 2022-09-02T05:50:10.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds= 2022-09-02T05:50:10.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds=', 'Content-Length': '0'} 2022-09-02T05:50:10.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003749d9dac5e73d-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.791 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.792 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.792 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.793 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.793 DEBUG:boto:Method: GET 2022-09-02T05:50:10.794 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.794 DEBUG:boto:Data: 2022-09-02T05:50:10.795 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.796 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.796 DEBUG:boto:Params: {} 2022-09-02T05:50:10.797 DEBUG:boto:Token: None 2022-09-02T05:50:10.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds= 2022-09-02T05:50:10.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds=', 'Content-Length': '0'} 2022-09-02T05:50:10.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078455cc9c289f400-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.802 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.802 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.803 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.803 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.804 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.804 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.805 DEBUG:boto:Data: 2022-09-02T05:50:10.805 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.806 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.807 DEBUG:boto:Params: {} 2022-09-02T05:50:10.807 DEBUG:boto:Token: None 2022-09-02T05:50:10.808 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Uz+Q0AQxcZbdL2fegid6urCbXfc= 2022-09-02T05:50:10.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Uz+Q0AQxcZbdL2fegid6urCbXfc=', 'Content-Length': '0'} 2022-09-02T05:50:10.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f3bcb20c03c133b-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.811 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.812 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.812 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.813 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.813 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.814 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.814 DEBUG:boto:Data: 2022-09-02T05:50:10.815 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.816 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.816 DEBUG:boto:Params: {} 2022-09-02T05:50:10.817 DEBUG:boto:Token: None 2022-09-02T05:50:10.817 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Uz+Q0AQxcZbdL2fegid6urCbXfc= 2022-09-02T05:50:10.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Uz+Q0AQxcZbdL2fegid6urCbXfc=', 'Content-Length': '0'} 2022-09-02T05:50:10.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314942f4bc716fac-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:10.822 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.822 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.823 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.823 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.824 DEBUG:boto:Method: GET 2022-09-02T05:50:10.825 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.825 DEBUG:boto:Data: 2022-09-02T05:50:10.826 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.827 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.828 DEBUG:boto:Params: {} 2022-09-02T05:50:10.828 DEBUG:boto:Token: None 2022-09-02T05:50:10.829 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds= 2022-09-02T05:50:10.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds=', 'Content-Length': '0'} 2022-09-02T05:50:10.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a570d5228fc749c-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.874 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.875 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.875 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.876 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.876 DEBUG:boto:Method: GET 2022-09-02T05:50:10.877 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.877 DEBUG:boto:Data: 2022-09-02T05:50:10.878 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.880 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.880 DEBUG:boto:Params: {} 2022-09-02T05:50:10.881 DEBUG:boto:Token: None 2022-09-02T05:50:10.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:10.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds= 2022-09-02T05:50:10.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dx/CqZQxEylEDr0tGIHlIWa/9ds=', 'Content-Length': '0'} 2022-09-02T05:50:10.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74db5b400750f5d-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.886 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.886 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.887 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.887 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.888 DEBUG:boto:Method: GET 2022-09-02T05:50:10.888 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.889 DEBUG:boto:Data: 2022-09-02T05:50:10.890 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.891 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.891 DEBUG:boto:Params: {} 2022-09-02T05:50:10.892 DEBUG:boto:Token: None 2022-09-02T05:50:10.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ed3de4e74012185-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.897 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.898 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.899 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.899 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.900 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.900 DEBUG:boto:Method: GET 2022-09-02T05:50:10.901 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.901 DEBUG:boto:Data: 2022-09-02T05:50:10.902 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.903 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.904 DEBUG:boto:Params: {} 2022-09-02T05:50:10.904 DEBUG:boto:Token: None 2022-09-02T05:50:10.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a8704c4427a5859-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.910 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.911 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.912 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.913 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.914 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:10.915 DEBUG:boto:Data: 2022-09-02T05:50:10.915 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.917 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.918 DEBUG:boto:Params: {} 2022-09-02T05:50:10.919 DEBUG:boto:Token: None 2022-09-02T05:50:10.920 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68 2022-09-02T05:50:10.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y95aD1GA+EF5JbIqYOfGga7d264= 2022-09-02T05:50:10.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y95aD1GA+EF5JbIqYOfGga7d264=', 'Content-Length': '0'} 2022-09-02T05:50:10.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000738783c2bd9a9e1d-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.927 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.927 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.928 DEBUG:boto:Method: HEAD 2022-09-02T05:50:10.929 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:10.930 DEBUG:boto:Data: 2022-09-02T05:50:10.931 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.933 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.934 DEBUG:boto:Params: {} 2022-09-02T05:50:10.935 DEBUG:boto:Token: None 2022-09-02T05:50:10.935 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68 2022-09-02T05:50:10.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y95aD1GA+EF5JbIqYOfGga7d264= 2022-09-02T05:50:10.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y95aD1GA+EF5JbIqYOfGga7d264=', 'Content-Length': '0'} 2022-09-02T05:50:10.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe3aaedd8475bb6f-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:10.940 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.940 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.941 DEBUG:boto:Method: GET 2022-09-02T05:50:10.942 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:10.942 DEBUG:boto:Data: 2022-09-02T05:50:10.943 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.944 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.944 DEBUG:boto:Params: {} 2022-09-02T05:50:10.945 DEBUG:boto:Token: None 2022-09-02T05:50:10.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68 2022-09-02T05:50:10.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sZFj7Oi/0qxhyItq+1ehw1MFoeU= 2022-09-02T05:50:10.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sZFj7Oi/0qxhyItq+1ehw1MFoeU=', 'Content-Length': '0'} 2022-09-02T05:50:10.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048733aa6e853ba2f-0063119992-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.950 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:10.950 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:10.951 DEBUG:boto:Method: GET 2022-09-02T05:50:10.952 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:10.952 DEBUG:boto:Data: 2022-09-02T05:50:10.953 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.954 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.954 DEBUG:boto:Params: {} 2022-09-02T05:50:10.955 DEBUG:boto:Token: None 2022-09-02T05:50:10.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/obj68 2022-09-02T05:50:10.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sZFj7Oi/0qxhyItq+1ehw1MFoeU= 2022-09-02T05:50:10.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sZFj7Oi/0qxhyItq+1ehw1MFoeU=', 'Content-Length': '0'} 2022-09-02T05:50:10.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aee8d8cbb641cf9-0063119992-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.960 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.960 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.961 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.962 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.962 DEBUG:boto:Method: GET 2022-09-02T05:50:10.963 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.963 DEBUG:boto:Data: 2022-09-02T05:50:10.964 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:10.965 DEBUG:boto:Port: 8001 2022-09-02T05:50:10.966 DEBUG:boto:Params: {} 2022-09-02T05:50:10.966 DEBUG:boto:Token: None 2022-09-02T05:50:10.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070df7c9c7d5fd6c2-0063119992-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.972 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.974 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:10.974 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:10.975 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.975 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.976 DEBUG:boto:Method: GET 2022-09-02T05:50:10.977 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:10.978 DEBUG:boto:Data: 2022-09-02T05:50:10.979 DEBUG:boto:Headers: {} 2022-09-02T05:50:10.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:10.982 DEBUG:boto:Port: 8000 2022-09-02T05:50:10.983 DEBUG:boto:Params: {} 2022-09-02T05:50:10.984 DEBUG:boto:Token: None 2022-09-02T05:50:10.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:10 GMT /xljbkf-74/ 2022-09-02T05:50:10.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0= 2022-09-02T05:50:10.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k+MVbxq+ZT8VdsbsFnkkkoO/4Z0=', 'Content-Length': '0'} 2022-09-02T05:50:10.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e1e96931be5c427-0063119992-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:10.995 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:10.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:10.998 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:10.999 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.000 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.001 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.002 DEBUG:boto:Method: GET 2022-09-02T05:50:11.003 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.004 DEBUG:boto:Data: 2022-09-02T05:50:11.005 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.007 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.008 DEBUG:boto:Params: {} 2022-09-02T05:50:11.010 DEBUG:boto:Token: None 2022-09-02T05:50:11.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4= 2022-09-02T05:50:11.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4=', 'Content-Length': '0'} 2022-09-02T05:50:11.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061dd04aee34a5290-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.018 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.019 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.020 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.021 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.024 DEBUG:boto:Method: GET 2022-09-02T05:50:11.025 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.026 DEBUG:boto:Data: 2022-09-02T05:50:11.027 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.030 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.030 DEBUG:boto:Params: {} 2022-09-02T05:50:11.031 DEBUG:boto:Token: None 2022-09-02T05:50:11.032 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.032 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4= 2022-09-02T05:50:11.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4=', 'Content-Length': '0'} 2022-09-02T05:50:11.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022694f0ed5e5ec48-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.036 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.037 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.038 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.038 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.039 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.039 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.040 DEBUG:boto:Data: 2022-09-02T05:50:11.041 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.042 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.043 DEBUG:boto:Params: {} 2022-09-02T05:50:11.043 DEBUG:boto:Token: None 2022-09-02T05:50:11.044 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kYkenwTF6M4K7AisPEUmJxIqh68= 2022-09-02T05:50:11.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kYkenwTF6M4K7AisPEUmJxIqh68=', 'Content-Length': '0'} 2022-09-02T05:50:11.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff77f36e3a65a86-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.049 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.050 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.051 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.051 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.052 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.052 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.053 DEBUG:boto:Data: 2022-09-02T05:50:11.054 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.055 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.056 DEBUG:boto:Params: {} 2022-09-02T05:50:11.057 DEBUG:boto:Token: None 2022-09-02T05:50:11.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kYkenwTF6M4K7AisPEUmJxIqh68= 2022-09-02T05:50:11.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kYkenwTF6M4K7AisPEUmJxIqh68=', 'Content-Length': '0'} 2022-09-02T05:50:11.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219f3f97b5938e13-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:11.062 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.063 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.064 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.064 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.065 DEBUG:boto:Method: GET 2022-09-02T05:50:11.065 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.066 DEBUG:boto:Data: 2022-09-02T05:50:11.066 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.068 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.068 DEBUG:boto:Params: {} 2022-09-02T05:50:11.069 DEBUG:boto:Token: None 2022-09-02T05:50:11.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4= 2022-09-02T05:50:11.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4=', 'Content-Length': '0'} 2022-09-02T05:50:11.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0000e9d87acfab-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.075 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.076 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.077 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.077 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.078 DEBUG:boto:Method: GET 2022-09-02T05:50:11.079 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.079 DEBUG:boto:Data: 2022-09-02T05:50:11.080 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.082 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.083 DEBUG:boto:Params: {} 2022-09-02T05:50:11.083 DEBUG:boto:Token: None 2022-09-02T05:50:11.084 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:11.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4= 2022-09-02T05:50:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+rWggMLDWd1sO0ntf63iPyOgsI4=', 'Content-Length': '0'} 2022-09-02T05:50:11.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857cce41f3a1b98b-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.088 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.089 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.089 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.090 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.091 DEBUG:boto:Method: GET 2022-09-02T05:50:11.091 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.092 DEBUG:boto:Data: 2022-09-02T05:50:11.092 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.093 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.094 DEBUG:boto:Params: {} 2022-09-02T05:50:11.094 DEBUG:boto:Token: None 2022-09-02T05:50:11.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000649b2bc7fc43af40-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.100 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.101 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.102 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.102 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.103 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.104 DEBUG:boto:Method: GET 2022-09-02T05:50:11.104 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.105 DEBUG:boto:Data: 2022-09-02T05:50:11.106 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.107 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.108 DEBUG:boto:Params: {} 2022-09-02T05:50:11.108 DEBUG:boto:Token: None 2022-09-02T05:50:11.109 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3f64b6d0df2caf-0063119993-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.115 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.116 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.117 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.117 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.118 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:11.119 DEBUG:boto:Data: 2022-09-02T05:50:11.120 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.121 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.122 DEBUG:boto:Params: {} 2022-09-02T05:50:11.123 DEBUG:boto:Token: None 2022-09-02T05:50:11.123 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76 2022-09-02T05:50:11.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hfzi3Jv6CeEyGQ+zboowCRd9yVI= 2022-09-02T05:50:11.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hfzi3Jv6CeEyGQ+zboowCRd9yVI=', 'Content-Length': '0'} 2022-09-02T05:50:11.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b541a91f023fea-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.127 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.128 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.129 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.129 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:11.143 DEBUG:boto:Data: 2022-09-02T05:50:11.144 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.145 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.146 DEBUG:boto:Params: {} 2022-09-02T05:50:11.147 DEBUG:boto:Token: None 2022-09-02T05:50:11.147 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76 2022-09-02T05:50:11.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hfzi3Jv6CeEyGQ+zboowCRd9yVI= 2022-09-02T05:50:11.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hfzi3Jv6CeEyGQ+zboowCRd9yVI=', 'Content-Length': '0'} 2022-09-02T05:50:11.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca5a0c669276218-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:11.153 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.154 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.155 DEBUG:boto:Method: GET 2022-09-02T05:50:11.155 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:11.156 DEBUG:boto:Data: 2022-09-02T05:50:11.156 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.158 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.158 DEBUG:boto:Params: {} 2022-09-02T05:50:11.159 DEBUG:boto:Token: None 2022-09-02T05:50:11.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76 2022-09-02T05:50:11.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SAb71S5kC3LzCLta+mQ746KXzWA= 2022-09-02T05:50:11.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SAb71S5kC3LzCLta+mQ746KXzWA=', 'Content-Length': '0'} 2022-09-02T05:50:11.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff6a8aa559c52cf-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.206 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:11.207 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:11.208 DEBUG:boto:Method: GET 2022-09-02T05:50:11.209 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:11.209 DEBUG:boto:Data: 2022-09-02T05:50:11.210 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.212 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.213 DEBUG:boto:Params: {} 2022-09-02T05:50:11.214 DEBUG:boto:Token: None 2022-09-02T05:50:11.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj76 2022-09-02T05:50:11.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SAb71S5kC3LzCLta+mQ746KXzWA= 2022-09-02T05:50:11.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SAb71S5kC3LzCLta+mQ746KXzWA=', 'Content-Length': '0'} 2022-09-02T05:50:11.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008391ef95b5cb1436-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.220 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.221 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.222 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.222 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.223 DEBUG:boto:Method: GET 2022-09-02T05:50:11.224 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.225 DEBUG:boto:Data: 2022-09-02T05:50:11.225 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.226 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.227 DEBUG:boto:Params: {} 2022-09-02T05:50:11.228 DEBUG:boto:Token: None 2022-09-02T05:50:11.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004752e1c7077f6847-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.233 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.235 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.236 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.237 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.238 DEBUG:boto:Method: GET 2022-09-02T05:50:11.239 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:11.239 DEBUG:boto:Data: 2022-09-02T05:50:11.240 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.241 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.241 DEBUG:boto:Params: {} 2022-09-02T05:50:11.242 DEBUG:boto:Token: None 2022-09-02T05:50:11.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014207476ecfae5b1-0063119993-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.248 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:11.250 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.251 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.252 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.253 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.254 DEBUG:boto:Method: GET 2022-09-02T05:50:11.256 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.257 DEBUG:boto:Data: 2022-09-02T05:50:11.258 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.260 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.261 DEBUG:boto:Params: {} 2022-09-02T05:50:11.262 DEBUG:boto:Token: None 2022-09-02T05:50:11.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY= 2022-09-02T05:50:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY=', 'Content-Length': '0'} 2022-09-02T05:50:11.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006447005651b3c9f5-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.269 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.270 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.271 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.271 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.272 DEBUG:boto:Method: GET 2022-09-02T05:50:11.272 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.273 DEBUG:boto:Data: 2022-09-02T05:50:11.274 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.275 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.275 DEBUG:boto:Params: {} 2022-09-02T05:50:11.276 DEBUG:boto:Token: None 2022-09-02T05:50:11.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY= 2022-09-02T05:50:11.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY=', 'Content-Length': '0'} 2022-09-02T05:50:11.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e05ea5b54a44a0c-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.322 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.323 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.323 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.324 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.325 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.325 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.326 DEBUG:boto:Data: 2022-09-02T05:50:11.326 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.327 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.328 DEBUG:boto:Params: {} 2022-09-02T05:50:11.328 DEBUG:boto:Token: None 2022-09-02T05:50:11.329 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nkYCkl7M43B/nThxe31VueWkiHU= 2022-09-02T05:50:11.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nkYCkl7M43B/nThxe31VueWkiHU=', 'Content-Length': '0'} 2022-09-02T05:50:11.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b2e4ffe8e8fc0e-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.333 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.334 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.334 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.335 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.335 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.336 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.336 DEBUG:boto:Data: 2022-09-02T05:50:11.337 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.338 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.338 DEBUG:boto:Params: {} 2022-09-02T05:50:11.339 DEBUG:boto:Token: None 2022-09-02T05:50:11.340 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nkYCkl7M43B/nThxe31VueWkiHU= 2022-09-02T05:50:11.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nkYCkl7M43B/nThxe31VueWkiHU=', 'Content-Length': '0'} 2022-09-02T05:50:11.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb474a821dc0251b-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:11.345 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.345 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.346 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.347 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.347 DEBUG:boto:Method: GET 2022-09-02T05:50:11.348 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.348 DEBUG:boto:Data: 2022-09-02T05:50:11.349 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.350 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.350 DEBUG:boto:Params: {} 2022-09-02T05:50:11.351 DEBUG:boto:Token: None 2022-09-02T05:50:11.352 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY= 2022-09-02T05:50:11.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY=', 'Content-Length': '0'} 2022-09-02T05:50:11.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86a05462f5bb64b-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.356 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.356 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.357 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.358 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.359 DEBUG:boto:Method: GET 2022-09-02T05:50:11.359 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.360 DEBUG:boto:Data: 2022-09-02T05:50:11.360 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.361 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.362 DEBUG:boto:Params: {} 2022-09-02T05:50:11.363 DEBUG:boto:Token: None 2022-09-02T05:50:11.363 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:11.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY= 2022-09-02T05:50:11.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0AhT3p7V4+wKp62mSnly7yuMrSY=', 'Content-Length': '0'} 2022-09-02T05:50:11.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8fdf7928f1ce680-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.369 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.369 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.370 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.371 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.371 DEBUG:boto:Method: GET 2022-09-02T05:50:11.372 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.373 DEBUG:boto:Data: 2022-09-02T05:50:11.374 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.375 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.375 DEBUG:boto:Params: {} 2022-09-02T05:50:11.376 DEBUG:boto:Token: None 2022-09-02T05:50:11.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035eb6e898f9f0ad9-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.382 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.383 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.384 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.384 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.385 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.385 DEBUG:boto:Method: GET 2022-09-02T05:50:11.386 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.386 DEBUG:boto:Data: 2022-09-02T05:50:11.387 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.388 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.388 DEBUG:boto:Params: {} 2022-09-02T05:50:11.390 DEBUG:boto:Token: None 2022-09-02T05:50:11.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f8c346ddd7d6bc-0063119993-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.396 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.397 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.398 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.399 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.400 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:11.400 DEBUG:boto:Data: 2022-09-02T05:50:11.401 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.402 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.402 DEBUG:boto:Params: {} 2022-09-02T05:50:11.403 DEBUG:boto:Token: None 2022-09-02T05:50:11.403 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85 2022-09-02T05:50:11.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wUFqVhXAoEO2PZzGaTENFmiP0uM= 2022-09-02T05:50:11.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wUFqVhXAoEO2PZzGaTENFmiP0uM=', 'Content-Length': '0'} 2022-09-02T05:50:11.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3cbdcea6afcf2f6-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.408 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.408 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.409 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.412 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:11.412 DEBUG:boto:Data: 2022-09-02T05:50:11.413 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.414 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.415 DEBUG:boto:Params: {} 2022-09-02T05:50:11.415 DEBUG:boto:Token: None 2022-09-02T05:50:11.416 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85 2022-09-02T05:50:11.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wUFqVhXAoEO2PZzGaTENFmiP0uM= 2022-09-02T05:50:11.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wUFqVhXAoEO2PZzGaTENFmiP0uM=', 'Content-Length': '0'} 2022-09-02T05:50:11.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026dd53d87de2b13e-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:11.421 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.422 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.422 DEBUG:boto:Method: GET 2022-09-02T05:50:11.423 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:11.423 DEBUG:boto:Data: 2022-09-02T05:50:11.424 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.425 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.426 DEBUG:boto:Params: {} 2022-09-02T05:50:11.426 DEBUG:boto:Token: None 2022-09-02T05:50:11.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85 2022-09-02T05:50:11.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNHGonDeBS186DSi2dx6uVSE1Rw= 2022-09-02T05:50:11.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNHGonDeBS186DSi2dx6uVSE1Rw=', 'Content-Length': '0'} 2022-09-02T05:50:11.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005011857a9c5f9da2-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.474 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:11.475 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:11.476 DEBUG:boto:Method: GET 2022-09-02T05:50:11.477 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:11.477 DEBUG:boto:Data: 2022-09-02T05:50:11.478 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.479 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.479 DEBUG:boto:Params: {} 2022-09-02T05:50:11.480 DEBUG:boto:Token: None 2022-09-02T05:50:11.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj85 2022-09-02T05:50:11.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNHGonDeBS186DSi2dx6uVSE1Rw= 2022-09-02T05:50:11.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNHGonDeBS186DSi2dx6uVSE1Rw=', 'Content-Length': '0'} 2022-09-02T05:50:11.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cbf9db2bd163bd2-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.486 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.486 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.487 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.488 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.488 DEBUG:boto:Method: GET 2022-09-02T05:50:11.489 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.489 DEBUG:boto:Data: 2022-09-02T05:50:11.490 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.491 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.491 DEBUG:boto:Params: {} 2022-09-02T05:50:11.492 DEBUG:boto:Token: None 2022-09-02T05:50:11.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c932b91b1d6bd93-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.497 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.498 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.499 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.499 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.500 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.500 DEBUG:boto:Method: GET 2022-09-02T05:50:11.501 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:11.501 DEBUG:boto:Data: 2022-09-02T05:50:11.502 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.503 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.504 DEBUG:boto:Params: {} 2022-09-02T05:50:11.504 DEBUG:boto:Token: None 2022-09-02T05:50:11.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca87a269f35490df-0063119993-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.510 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:11.511 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.512 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.512 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.513 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.513 DEBUG:boto:Method: GET 2022-09-02T05:50:11.514 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.514 DEBUG:boto:Data: 2022-09-02T05:50:11.515 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.516 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.516 DEBUG:boto:Params: {} 2022-09-02T05:50:11.517 DEBUG:boto:Token: None 2022-09-02T05:50:11.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50= 2022-09-02T05:50:11.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50=', 'Content-Length': '0'} 2022-09-02T05:50:11.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa1c6325b32826f-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.522 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.522 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.523 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.523 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.524 DEBUG:boto:Method: GET 2022-09-02T05:50:11.525 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.525 DEBUG:boto:Data: 2022-09-02T05:50:11.526 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.527 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.527 DEBUG:boto:Params: {} 2022-09-02T05:50:11.528 DEBUG:boto:Token: None 2022-09-02T05:50:11.529 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50= 2022-09-02T05:50:11.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50=', 'Content-Length': '0'} 2022-09-02T05:50:11.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4ca1a524e138ad-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.533 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.533 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.534 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.534 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.535 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.535 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.536 DEBUG:boto:Data: 2022-09-02T05:50:11.536 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.537 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.538 DEBUG:boto:Params: {} 2022-09-02T05:50:11.539 DEBUG:boto:Token: None 2022-09-02T05:50:11.539 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EhKT654MBBJftQmNVPAi9ICa4uI= 2022-09-02T05:50:11.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EhKT654MBBJftQmNVPAi9ICa4uI=', 'Content-Length': '0'} 2022-09-02T05:50:11.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee8f38f43f947bf-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.543 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.544 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.544 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.545 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.545 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.546 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.546 DEBUG:boto:Data: 2022-09-02T05:50:11.547 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.548 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.549 DEBUG:boto:Params: {} 2022-09-02T05:50:11.549 DEBUG:boto:Token: None 2022-09-02T05:50:11.550 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EhKT654MBBJftQmNVPAi9ICa4uI= 2022-09-02T05:50:11.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EhKT654MBBJftQmNVPAi9ICa4uI=', 'Content-Length': '0'} 2022-09-02T05:50:11.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009600f7b05b094fed-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:11.554 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.555 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.555 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.556 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.556 DEBUG:boto:Method: GET 2022-09-02T05:50:11.557 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.558 DEBUG:boto:Data: 2022-09-02T05:50:11.558 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.559 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.560 DEBUG:boto:Params: {} 2022-09-02T05:50:11.560 DEBUG:boto:Token: None 2022-09-02T05:50:11.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50= 2022-09-02T05:50:11.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50=', 'Content-Length': '0'} 2022-09-02T05:50:11.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351c614fe000cae6-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.565 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.566 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.566 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.567 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.567 DEBUG:boto:Method: GET 2022-09-02T05:50:11.568 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.568 DEBUG:boto:Data: 2022-09-02T05:50:11.569 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.570 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.571 DEBUG:boto:Params: {} 2022-09-02T05:50:11.571 DEBUG:boto:Token: None 2022-09-02T05:50:11.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:11.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50= 2022-09-02T05:50:11.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MrDXn00LjLhgB55fo0aaWoOXC50=', 'Content-Length': '0'} 2022-09-02T05:50:11.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7989fc80be440e9-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.577 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.577 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.578 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.578 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.579 DEBUG:boto:Method: GET 2022-09-02T05:50:11.579 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.580 DEBUG:boto:Data: 2022-09-02T05:50:11.580 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.581 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.582 DEBUG:boto:Params: {} 2022-09-02T05:50:11.583 DEBUG:boto:Token: None 2022-09-02T05:50:11.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178d0c357c0541c5-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.588 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.589 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.590 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.590 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.591 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.591 DEBUG:boto:Method: GET 2022-09-02T05:50:11.592 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.592 DEBUG:boto:Data: 2022-09-02T05:50:11.593 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.594 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.594 DEBUG:boto:Params: {} 2022-09-02T05:50:11.595 DEBUG:boto:Token: None 2022-09-02T05:50:11.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000407f1f45b1c0e357-0063119993-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.601 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.602 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.602 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.603 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.603 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:11.604 DEBUG:boto:Data: 2022-09-02T05:50:11.604 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.605 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.606 DEBUG:boto:Params: {} 2022-09-02T05:50:11.607 DEBUG:boto:Token: None 2022-09-02T05:50:11.607 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95 2022-09-02T05:50:11.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gibx0lKYP8UPTvMNXZ7nqA/rg7k= 2022-09-02T05:50:11.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gibx0lKYP8UPTvMNXZ7nqA/rg7k=', 'Content-Length': '0'} 2022-09-02T05:50:11.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c36537dd46dea99-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.611 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.612 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.612 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.612 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:11.613 DEBUG:boto:Data: 2022-09-02T05:50:11.614 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.615 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.615 DEBUG:boto:Params: {} 2022-09-02T05:50:11.616 DEBUG:boto:Token: None 2022-09-02T05:50:11.616 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95 2022-09-02T05:50:11.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gibx0lKYP8UPTvMNXZ7nqA/rg7k= 2022-09-02T05:50:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gibx0lKYP8UPTvMNXZ7nqA/rg7k=', 'Content-Length': '0'} 2022-09-02T05:50:11.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e501bee21732d20-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:11.621 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.621 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.622 DEBUG:boto:Method: GET 2022-09-02T05:50:11.622 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:11.623 DEBUG:boto:Data: 2022-09-02T05:50:11.624 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.625 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.625 DEBUG:boto:Params: {} 2022-09-02T05:50:11.626 DEBUG:boto:Token: None 2022-09-02T05:50:11.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95 2022-09-02T05:50:11.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pi3yTBXEXhrhB6aCK5gcTMOf0Rs= 2022-09-02T05:50:11.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pi3yTBXEXhrhB6aCK5gcTMOf0Rs=', 'Content-Length': '0'} 2022-09-02T05:50:11.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae2433f6dd13bab-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.674 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:11.675 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:11.676 DEBUG:boto:Method: GET 2022-09-02T05:50:11.676 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:11.677 DEBUG:boto:Data: 2022-09-02T05:50:11.677 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.678 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.679 DEBUG:boto:Params: {} 2022-09-02T05:50:11.680 DEBUG:boto:Token: None 2022-09-02T05:50:11.680 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj95 2022-09-02T05:50:11.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pi3yTBXEXhrhB6aCK5gcTMOf0Rs= 2022-09-02T05:50:11.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pi3yTBXEXhrhB6aCK5gcTMOf0Rs=', 'Content-Length': '0'} 2022-09-02T05:50:11.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1dab27a5295f70-0063119993-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.686 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.686 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.687 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.687 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.688 DEBUG:boto:Method: GET 2022-09-02T05:50:11.689 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.689 DEBUG:boto:Data: 2022-09-02T05:50:11.690 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.691 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.691 DEBUG:boto:Params: {} 2022-09-02T05:50:11.692 DEBUG:boto:Token: None 2022-09-02T05:50:11.693 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b746c512318861-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.697 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.698 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.699 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.699 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.700 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.700 DEBUG:boto:Method: GET 2022-09-02T05:50:11.701 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:11.701 DEBUG:boto:Data: 2022-09-02T05:50:11.702 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:11.703 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.703 DEBUG:boto:Params: {} 2022-09-02T05:50:11.704 DEBUG:boto:Token: None 2022-09-02T05:50:11.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b9cf59aad0f68e8-0063119993-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.710 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:50:11.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:50:11.712 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.712 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.713 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.713 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:11.714 DEBUG:boto:Data: 2022-09-02T05:50:11.714 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.715 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.716 DEBUG:boto:Params: {} 2022-09-02T05:50:11.716 DEBUG:boto:Token: None 2022-09-02T05:50:11.717 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rZlchJAhFfsQVNc9ZU0Uj0ZpYhs= 2022-09-02T05:50:11.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rZlchJAhFfsQVNc9ZU0Uj0ZpYhs=', 'Content-Length': '0'} 2022-09-02T05:50:11.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093ba2b37f7f08942-0063119993-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.721 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.722 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.722 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.723 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:11.723 DEBUG:boto:Data: 2022-09-02T05:50:11.724 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.724 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.725 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.725 DEBUG:boto:Params: {} 2022-09-02T05:50:11.726 DEBUG:boto:Token: None 2022-09-02T05:50:11.727 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rZlchJAhFfsQVNc9ZU0Uj0ZpYhs= 2022-09-02T05:50:11.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rZlchJAhFfsQVNc9ZU0Uj0ZpYhs=', 'Content-Length': '0'} 2022-09-02T05:50:11.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bc8e4f2353a51b0-0063119993-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:50:11.732 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.732 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.733 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.733 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.734 DEBUG:boto:Method: GET 2022-09-02T05:50:11.734 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.735 DEBUG:boto:Data: 2022-09-02T05:50:11.735 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.736 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.737 DEBUG:boto:Params: {} 2022-09-02T05:50:11.737 DEBUG:boto:Token: None 2022-09-02T05:50:11.738 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/?versions 2022-09-02T05:50:11.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M= 2022-09-02T05:50:11.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M=', 'Content-Length': '0'} 2022-09-02T05:50:11.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f45bdf5aa5ddac71-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.743 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:50:11.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:50:11.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:50:11.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:50:11.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:50:11.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:50:11.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:50:11.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:50:11.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:50:11.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:50:11.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:50:11.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:50:11.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:50:11.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:50:11.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:50:11.753 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:50:11.754 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.755 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.755 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.756 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.756 DEBUG:boto:Method: GET 2022-09-02T05:50:11.757 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.757 DEBUG:boto:Data: 2022-09-02T05:50:11.758 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.758 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.759 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.759 DEBUG:boto:Params: {} 2022-09-02T05:50:11.760 DEBUG:boto:Token: None 2022-09-02T05:50:11.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/?versions 2022-09-02T05:50:11.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M= 2022-09-02T05:50:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M=', 'Content-Length': '0'} 2022-09-02T05:50:11.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de8ffec50501e8b9-0063119993-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.765 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:50:11.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:50:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:50:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:50:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:50:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:50:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:50:11.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:50:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:50:11.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:50:11.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:50:11.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:50:11.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:50:11.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:50:11.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:50:11.774 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.775 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.775 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.776 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.776 DEBUG:boto:Method: GET 2022-09-02T05:50:11.777 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.777 DEBUG:boto:Data: 2022-09-02T05:50:11.778 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.779 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.779 DEBUG:boto:Params: {} 2022-09-02T05:50:11.780 DEBUG:boto:Token: None 2022-09-02T05:50:11.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/?versions 2022-09-02T05:50:11.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M= 2022-09-02T05:50:11.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M=', 'Content-Length': '0'} 2022-09-02T05:50:11.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003749a9f9ad46cc46-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.785 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.787 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.788 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.788 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.789 DEBUG:boto:Method: GET 2022-09-02T05:50:11.789 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:50:11.790 DEBUG:boto:Data: 2022-09-02T05:50:11.790 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.792 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.792 DEBUG:boto:Params: {} 2022-09-02T05:50:11.793 DEBUG:boto:Token: None 2022-09-02T05:50:11.793 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/?versions 2022-09-02T05:50:11.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M= 2022-09-02T05:50:11.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pix6jGBz/jhMCJfu6CrkL/JK68M=', 'Content-Length': '0'} 2022-09-02T05:50:11.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4ff8dafb785bc5-0063119993-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.798 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:11.800 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.801 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.801 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.802 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.802 DEBUG:boto:Method: GET 2022-09-02T05:50:11.803 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.804 DEBUG:boto:Data: 2022-09-02T05:50:11.804 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.805 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.806 DEBUG:boto:Params: {} 2022-09-02T05:50:11.807 DEBUG:boto:Token: None 2022-09-02T05:50:11.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0= 2022-09-02T05:50:11.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0=', 'Content-Length': '0'} 2022-09-02T05:50:11.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c810c715eabd2c10-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.811 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.812 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.813 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.813 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.814 DEBUG:boto:Method: GET 2022-09-02T05:50:11.814 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.815 DEBUG:boto:Data: 2022-09-02T05:50:11.815 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.817 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.817 DEBUG:boto:Params: {} 2022-09-02T05:50:11.818 DEBUG:boto:Token: None 2022-09-02T05:50:11.818 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0= 2022-09-02T05:50:11.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0=', 'Content-Length': '0'} 2022-09-02T05:50:11.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000510c3655ad56cef1-0063119993-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.866 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.867 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.867 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.868 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.868 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.869 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.869 DEBUG:boto:Data: 2022-09-02T05:50:11.870 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.871 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.871 DEBUG:boto:Params: {} 2022-09-02T05:50:11.872 DEBUG:boto:Token: None 2022-09-02T05:50:11.873 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:54mf6uzgtS8ptKwxQzsMwL5n6uA= 2022-09-02T05:50:11.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:54mf6uzgtS8ptKwxQzsMwL5n6uA=', 'Content-Length': '0'} 2022-09-02T05:50:11.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0943f46477845f-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.877 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.877 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.878 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.878 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.879 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.879 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.880 DEBUG:boto:Data: 2022-09-02T05:50:11.880 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.881 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.882 DEBUG:boto:Params: {} 2022-09-02T05:50:11.883 DEBUG:boto:Token: None 2022-09-02T05:50:11.883 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:54mf6uzgtS8ptKwxQzsMwL5n6uA= 2022-09-02T05:50:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:54mf6uzgtS8ptKwxQzsMwL5n6uA=', 'Content-Length': '0'} 2022-09-02T05:50:11.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdbe5fe2e41c4425-0063119993-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:11.892 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.893 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.893 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.894 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.894 DEBUG:boto:Method: GET 2022-09-02T05:50:11.895 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.895 DEBUG:boto:Data: 2022-09-02T05:50:11.896 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.898 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.898 DEBUG:boto:Params: {} 2022-09-02T05:50:11.899 DEBUG:boto:Token: None 2022-09-02T05:50:11.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0= 2022-09-02T05:50:11.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0=', 'Content-Length': '0'} 2022-09-02T05:50:11.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000792dd6737bfedc6d-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.904 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.904 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.905 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.905 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.906 DEBUG:boto:Method: GET 2022-09-02T05:50:11.906 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.907 DEBUG:boto:Data: 2022-09-02T05:50:11.907 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.908 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.909 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.909 DEBUG:boto:Params: {} 2022-09-02T05:50:11.910 DEBUG:boto:Token: None 2022-09-02T05:50:11.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:50:11.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0= 2022-09-02T05:50:11.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FYB3rDZIBWIJoVxIQQd/CCS8mv0=', 'Content-Length': '0'} 2022-09-02T05:50:11.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb2d675e372ce6f-0063119993-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.915 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.915 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.916 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:11.917 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:11.917 DEBUG:boto:Method: GET 2022-09-02T05:50:11.918 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:11.918 DEBUG:boto:Data: 2022-09-02T05:50:11.919 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.920 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.921 DEBUG:boto:Params: {} 2022-09-02T05:50:11.921 DEBUG:boto:Token: None 2022-09-02T05:50:11.922 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab9564490ba678c-0063119993-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.926 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.927 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:11.928 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:11.928 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:11.929 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:11.929 DEBUG:boto:Method: GET 2022-09-02T05:50:11.930 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:11.931 DEBUG:boto:Data: 2022-09-02T05:50:11.931 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:11.932 DEBUG:boto:Port: 8000 2022-09-02T05:50:11.933 DEBUG:boto:Params: {} 2022-09-02T05:50:11.933 DEBUG:boto:Token: None 2022-09-02T05:50:11.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/ 2022-09-02T05:50:11.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU= 2022-09-02T05:50:11.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MX1Ty0o7x/tEBSnosp0Q/fCMHiU=', 'Content-Length': '0'} 2022-09-02T05:50:11.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cc77b8985e2ca11-0063119993-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:11.982 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:11.984 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:11.985 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:11.987 DEBUG:boto:Method: HEAD 2022-09-02T05:50:11.988 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:11.989 DEBUG:boto:Data: 2022-09-02T05:50:11.990 DEBUG:boto:Headers: {} 2022-09-02T05:50:11.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:11.992 DEBUG:boto:Port: 8001 2022-09-02T05:50:11.993 DEBUG:boto:Params: {} 2022-09-02T05:50:11.994 DEBUG:boto:Token: None 2022-09-02T05:50:11.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:11 GMT /xljbkf-74/obj11 2022-09-02T05:50:11.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BtFI8L/q6erHbvr/udUCanI5IE0= 2022-09-02T05:50:11.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BtFI8L/q6erHbvr/udUCanI5IE0=', 'Content-Length': '0'} 2022-09-02T05:50:12.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a0df5c5ecffdbe-0063119993-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.002 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:12.003 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:12.004 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.005 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:12.006 DEBUG:boto:Data: 2022-09-02T05:50:12.007 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.009 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.010 DEBUG:boto:Params: {} 2022-09-02T05:50:12.011 DEBUG:boto:Token: None 2022-09-02T05:50:12.012 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj11 2022-09-02T05:50:12.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a6SQtZF3fArGcqERftvwdZB27nk= 2022-09-02T05:50:12.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a6SQtZF3fArGcqERftvwdZB27nk=', 'Content-Length': '0'} 2022-09-02T05:50:12.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b95cd92297f636cb-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:50:12.019 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:12.020 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:12.021 DEBUG:boto:Method: GET 2022-09-02T05:50:12.023 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:12.024 DEBUG:boto:Data: 2022-09-02T05:50:12.025 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.027 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.028 DEBUG:boto:Params: {} 2022-09-02T05:50:12.029 DEBUG:boto:Token: None 2022-09-02T05:50:12.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj11 2022-09-02T05:50:12.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/c9uspXM0ZrNIr+WIcMTN+6UskE= 2022-09-02T05:50:12.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/c9uspXM0ZrNIr+WIcMTN+6UskE=', 'Content-Length': '0'} 2022-09-02T05:50:12.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f41360e67857a96-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.037 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:50:12.038 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:50:12.039 DEBUG:boto:Method: GET 2022-09-02T05:50:12.040 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:50:12.041 DEBUG:boto:Data: 2022-09-02T05:50:12.042 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.043 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.044 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.045 DEBUG:boto:Params: {} 2022-09-02T05:50:12.046 DEBUG:boto:Token: None 2022-09-02T05:50:12.047 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj11 2022-09-02T05:50:12.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/c9uspXM0ZrNIr+WIcMTN+6UskE= 2022-09-02T05:50:12.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/c9uspXM0ZrNIr+WIcMTN+6UskE=', 'Content-Length': '0'} 2022-09-02T05:50:12.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b9c98797d4b93c6-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.054 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.055 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.055 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:12.056 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:12.057 DEBUG:boto:Method: GET 2022-09-02T05:50:12.057 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:12.058 DEBUG:boto:Data: 2022-09-02T05:50:12.059 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.060 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.061 DEBUG:boto:Params: {} 2022-09-02T05:50:12.062 DEBUG:boto:Token: None 2022-09-02T05:50:12.062 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061b0f1de48ec309e-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.067 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.068 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.069 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.069 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:12.070 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:50:12.071 DEBUG:boto:Method: GET 2022-09-02T05:50:12.071 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:50:12.072 DEBUG:boto:Data: 2022-09-02T05:50:12.073 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.074 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.074 DEBUG:boto:Params: {} 2022-09-02T05:50:12.075 DEBUG:boto:Token: None 2022-09-02T05:50:12.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1c69fd467366268-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.080 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:12.082 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.082 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.083 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.083 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.084 DEBUG:boto:Method: GET 2022-09-02T05:50:12.084 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.085 DEBUG:boto:Data: 2022-09-02T05:50:12.085 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.086 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.087 DEBUG:boto:Params: {} 2022-09-02T05:50:12.088 DEBUG:boto:Token: None 2022-09-02T05:50:12.088 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE= 2022-09-02T05:50:12.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE=', 'Content-Length': '0'} 2022-09-02T05:50:12.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000275b2b4f07849d80-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.134 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.135 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.136 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.137 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.138 DEBUG:boto:Method: GET 2022-09-02T05:50:12.139 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.140 DEBUG:boto:Data: 2022-09-02T05:50:12.141 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.143 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.144 DEBUG:boto:Params: {} 2022-09-02T05:50:12.145 DEBUG:boto:Token: None 2022-09-02T05:50:12.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE= 2022-09-02T05:50:12.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE=', 'Content-Length': '0'} 2022-09-02T05:50:12.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f7df44f8e6ffe44-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.153 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.154 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.155 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.156 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.157 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.158 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.159 DEBUG:boto:Data: 2022-09-02T05:50:12.160 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.162 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.163 DEBUG:boto:Params: {} 2022-09-02T05:50:12.164 DEBUG:boto:Token: None 2022-09-02T05:50:12.166 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O6W9+vmop6FtjmNk+uaL58TB16A= 2022-09-02T05:50:12.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O6W9+vmop6FtjmNk+uaL58TB16A=', 'Content-Length': '0'} 2022-09-02T05:50:12.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3578abae34f3453-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.172 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.172 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.173 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.174 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.175 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.175 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.176 DEBUG:boto:Data: 2022-09-02T05:50:12.176 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.177 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.178 DEBUG:boto:Params: {} 2022-09-02T05:50:12.179 DEBUG:boto:Token: None 2022-09-02T05:50:12.179 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O6W9+vmop6FtjmNk+uaL58TB16A= 2022-09-02T05:50:12.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O6W9+vmop6FtjmNk+uaL58TB16A=', 'Content-Length': '0'} 2022-09-02T05:50:12.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4d36f47172c7eb-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:12.184 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.185 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.185 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.186 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.187 DEBUG:boto:Method: GET 2022-09-02T05:50:12.187 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.188 DEBUG:boto:Data: 2022-09-02T05:50:12.188 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.190 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.190 DEBUG:boto:Params: {} 2022-09-02T05:50:12.191 DEBUG:boto:Token: None 2022-09-02T05:50:12.192 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.193 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE= 2022-09-02T05:50:12.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE=', 'Content-Length': '0'} 2022-09-02T05:50:12.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648ab49e0a86bf0b-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.196 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.197 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.198 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.198 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.199 DEBUG:boto:Method: GET 2022-09-02T05:50:12.200 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.200 DEBUG:boto:Data: 2022-09-02T05:50:12.201 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.202 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.203 DEBUG:boto:Params: {} 2022-09-02T05:50:12.203 DEBUG:boto:Token: None 2022-09-02T05:50:12.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:50:12.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE= 2022-09-02T05:50:12.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P4afXO71tk5TyHbFqKPDzJEp6XE=', 'Content-Length': '0'} 2022-09-02T05:50:12.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e411f8e61f93608f-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.209 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.209 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.210 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.210 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.211 DEBUG:boto:Method: GET 2022-09-02T05:50:12.212 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.212 DEBUG:boto:Data: 2022-09-02T05:50:12.213 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.214 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.214 DEBUG:boto:Params: {} 2022-09-02T05:50:12.215 DEBUG:boto:Token: None 2022-09-02T05:50:12.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50b4aeb1c9db66a-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.221 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.222 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.222 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.223 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.224 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.224 DEBUG:boto:Method: GET 2022-09-02T05:50:12.225 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.226 DEBUG:boto:Data: 2022-09-02T05:50:12.226 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.227 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.228 DEBUG:boto:Params: {} 2022-09-02T05:50:12.229 DEBUG:boto:Token: None 2022-09-02T05:50:12.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2f4aaafee142ba0-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.234 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.235 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.236 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.236 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.237 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:12.238 DEBUG:boto:Data: 2022-09-02T05:50:12.238 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.239 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.240 DEBUG:boto:Params: {} 2022-09-02T05:50:12.241 DEBUG:boto:Token: None 2022-09-02T05:50:12.241 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17 2022-09-02T05:50:12.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z3Hx9GtYobKxoF4iYQxah5YyTIE= 2022-09-02T05:50:12.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z3Hx9GtYobKxoF4iYQxah5YyTIE=', 'Content-Length': '0'} 2022-09-02T05:50:12.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20ad1982064b576-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.246 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.246 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.247 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.247 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:12.248 DEBUG:boto:Data: 2022-09-02T05:50:12.248 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.249 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.249 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.250 DEBUG:boto:Params: {} 2022-09-02T05:50:12.251 DEBUG:boto:Token: None 2022-09-02T05:50:12.251 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17 2022-09-02T05:50:12.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z3Hx9GtYobKxoF4iYQxah5YyTIE= 2022-09-02T05:50:12.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z3Hx9GtYobKxoF4iYQxah5YyTIE=', 'Content-Length': '0'} 2022-09-02T05:50:12.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea1e329b20a8936-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:50:12.256 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.257 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.258 DEBUG:boto:Method: GET 2022-09-02T05:50:12.261 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:12.262 DEBUG:boto:Data: 2022-09-02T05:50:12.263 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.264 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.265 DEBUG:boto:Params: {} 2022-09-02T05:50:12.266 DEBUG:boto:Token: None 2022-09-02T05:50:12.267 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17 2022-09-02T05:50:12.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AOvWYEcCPinKMVoREj0M6Sp1Lkk= 2022-09-02T05:50:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AOvWYEcCPinKMVoREj0M6Sp1Lkk=', 'Content-Length': '0'} 2022-09-02T05:50:12.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008071765a305e934e-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.271 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:50:12.272 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:50:12.273 DEBUG:boto:Method: GET 2022-09-02T05:50:12.273 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:50:12.274 DEBUG:boto:Data: 2022-09-02T05:50:12.275 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.276 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.277 DEBUG:boto:Params: {} 2022-09-02T05:50:12.278 DEBUG:boto:Token: None 2022-09-02T05:50:12.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj17 2022-09-02T05:50:12.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AOvWYEcCPinKMVoREj0M6Sp1Lkk= 2022-09-02T05:50:12.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AOvWYEcCPinKMVoREj0M6Sp1Lkk=', 'Content-Length': '0'} 2022-09-02T05:50:12.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5205ccecadc3c59-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.326 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.327 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.328 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.328 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.329 DEBUG:boto:Method: GET 2022-09-02T05:50:12.330 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.330 DEBUG:boto:Data: 2022-09-02T05:50:12.331 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.333 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.334 DEBUG:boto:Params: {} 2022-09-02T05:50:12.335 DEBUG:boto:Token: None 2022-09-02T05:50:12.335 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001984d31dd86dbc7f-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.341 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.343 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.343 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.344 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.345 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.346 DEBUG:boto:Method: GET 2022-09-02T05:50:12.347 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:50:12.348 DEBUG:boto:Data: 2022-09-02T05:50:12.349 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.350 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.351 DEBUG:boto:Params: {} 2022-09-02T05:50:12.352 DEBUG:boto:Token: None 2022-09-02T05:50:12.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a49ff12b6f46ee-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.358 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:12.360 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.362 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.363 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.363 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.364 DEBUG:boto:Method: GET 2022-09-02T05:50:12.365 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.365 DEBUG:boto:Data: 2022-09-02T05:50:12.366 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.367 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.368 DEBUG:boto:Params: {} 2022-09-02T05:50:12.369 DEBUG:boto:Token: None 2022-09-02T05:50:12.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.370 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo= 2022-09-02T05:50:12.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo=', 'Content-Length': '0'} 2022-09-02T05:50:12.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e0026775ff00cc-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.374 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.375 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.375 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.376 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.376 DEBUG:boto:Method: GET 2022-09-02T05:50:12.377 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.378 DEBUG:boto:Data: 2022-09-02T05:50:12.378 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.379 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.380 DEBUG:boto:Params: {} 2022-09-02T05:50:12.380 DEBUG:boto:Token: None 2022-09-02T05:50:12.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo= 2022-09-02T05:50:12.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo=', 'Content-Length': '0'} 2022-09-02T05:50:12.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1cbcbd54e6f98b-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.385 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.386 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.387 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.388 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.389 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.390 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.390 DEBUG:boto:Data: 2022-09-02T05:50:12.391 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.393 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.394 DEBUG:boto:Params: {} 2022-09-02T05:50:12.395 DEBUG:boto:Token: None 2022-09-02T05:50:12.396 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2t5hwTENXKGPtzMgL96sDQhd6A= 2022-09-02T05:50:12.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2t5hwTENXKGPtzMgL96sDQhd6A=', 'Content-Length': '0'} 2022-09-02T05:50:12.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc9c80749b21402-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.400 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.401 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.402 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.402 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.403 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.404 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.404 DEBUG:boto:Data: 2022-09-02T05:50:12.405 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.406 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.406 DEBUG:boto:Params: {} 2022-09-02T05:50:12.407 DEBUG:boto:Token: None 2022-09-02T05:50:12.408 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2t5hwTENXKGPtzMgL96sDQhd6A= 2022-09-02T05:50:12.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2t5hwTENXKGPtzMgL96sDQhd6A=', 'Content-Length': '0'} 2022-09-02T05:50:12.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03e334dc63e8084-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:12.416 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.417 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.418 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.419 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.420 DEBUG:boto:Method: GET 2022-09-02T05:50:12.421 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.421 DEBUG:boto:Data: 2022-09-02T05:50:12.422 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.423 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.424 DEBUG:boto:Params: {} 2022-09-02T05:50:12.425 DEBUG:boto:Token: None 2022-09-02T05:50:12.426 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo= 2022-09-02T05:50:12.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo=', 'Content-Length': '0'} 2022-09-02T05:50:12.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa9b76dedefb02a-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.430 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.431 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.431 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.432 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.432 DEBUG:boto:Method: GET 2022-09-02T05:50:12.433 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.433 DEBUG:boto:Data: 2022-09-02T05:50:12.434 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.435 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.435 DEBUG:boto:Params: {} 2022-09-02T05:50:12.436 DEBUG:boto:Token: None 2022-09-02T05:50:12.437 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:50:12.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo= 2022-09-02T05:50:12.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yKpxRQgpEvPolSAwGsgEPomtGMo=', 'Content-Length': '0'} 2022-09-02T05:50:12.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078dfa59b626635ad-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.441 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.441 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.442 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.443 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.443 DEBUG:boto:Method: GET 2022-09-02T05:50:12.444 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.444 DEBUG:boto:Data: 2022-09-02T05:50:12.445 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.446 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.446 DEBUG:boto:Params: {} 2022-09-02T05:50:12.447 DEBUG:boto:Token: None 2022-09-02T05:50:12.448 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b227776e318893b9-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.452 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.453 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.454 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.454 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.455 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.455 DEBUG:boto:Method: GET 2022-09-02T05:50:12.456 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.457 DEBUG:boto:Data: 2022-09-02T05:50:12.457 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.458 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.458 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.459 DEBUG:boto:Params: {} 2022-09-02T05:50:12.459 DEBUG:boto:Token: None 2022-09-02T05:50:12.460 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42a02ac3386236b-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.465 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.466 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.467 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.468 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.469 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:12.470 DEBUG:boto:Data: 2022-09-02T05:50:12.471 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.473 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.474 DEBUG:boto:Params: {} 2022-09-02T05:50:12.474 DEBUG:boto:Token: None 2022-09-02T05:50:12.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19 2022-09-02T05:50:12.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+Ex55vrJoVJgZDB0WvK6ECtyV0U= 2022-09-02T05:50:12.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+Ex55vrJoVJgZDB0WvK6ECtyV0U=', 'Content-Length': '0'} 2022-09-02T05:50:12.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000634c7429d38c7e97-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.479 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.479 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.480 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.480 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:12.481 DEBUG:boto:Data: 2022-09-02T05:50:12.481 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.482 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.483 DEBUG:boto:Params: {} 2022-09-02T05:50:12.483 DEBUG:boto:Token: None 2022-09-02T05:50:12.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19 2022-09-02T05:50:12.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+Ex55vrJoVJgZDB0WvK6ECtyV0U= 2022-09-02T05:50:12.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+Ex55vrJoVJgZDB0WvK6ECtyV0U=', 'Content-Length': '0'} 2022-09-02T05:50:12.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfa318331aa0181-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:50:12.488 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.489 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.489 DEBUG:boto:Method: GET 2022-09-02T05:50:12.490 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:12.491 DEBUG:boto:Data: 2022-09-02T05:50:12.491 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.492 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.493 DEBUG:boto:Params: {} 2022-09-02T05:50:12.493 DEBUG:boto:Token: None 2022-09-02T05:50:12.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19 2022-09-02T05:50:12.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+/DTrc4tDlll/BraVs+k0r4Br98= 2022-09-02T05:50:12.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+/DTrc4tDlll/BraVs+k0r4Br98=', 'Content-Length': '0'} 2022-09-02T05:50:12.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d48c7e7b571d46-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.538 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:50:12.539 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:50:12.539 DEBUG:boto:Method: GET 2022-09-02T05:50:12.540 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:50:12.540 DEBUG:boto:Data: 2022-09-02T05:50:12.541 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.542 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.543 DEBUG:boto:Params: {} 2022-09-02T05:50:12.543 DEBUG:boto:Token: None 2022-09-02T05:50:12.544 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj19 2022-09-02T05:50:12.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+/DTrc4tDlll/BraVs+k0r4Br98= 2022-09-02T05:50:12.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+/DTrc4tDlll/BraVs+k0r4Br98=', 'Content-Length': '0'} 2022-09-02T05:50:12.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a779ddfc45d4d7d1-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.548 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.548 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.549 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.550 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.550 DEBUG:boto:Method: GET 2022-09-02T05:50:12.551 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.551 DEBUG:boto:Data: 2022-09-02T05:50:12.552 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.553 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.553 DEBUG:boto:Params: {} 2022-09-02T05:50:12.554 DEBUG:boto:Token: None 2022-09-02T05:50:12.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e127259e1d9defb-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.559 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.560 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.560 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.561 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.561 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.562 DEBUG:boto:Method: GET 2022-09-02T05:50:12.562 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:50:12.563 DEBUG:boto:Data: 2022-09-02T05:50:12.564 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.565 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.565 DEBUG:boto:Params: {} 2022-09-02T05:50:12.566 DEBUG:boto:Token: None 2022-09-02T05:50:12.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000581bf78347bf377c-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.571 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:12.573 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.574 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.574 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.575 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.576 DEBUG:boto:Method: GET 2022-09-02T05:50:12.576 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.577 DEBUG:boto:Data: 2022-09-02T05:50:12.577 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.578 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.579 DEBUG:boto:Params: {} 2022-09-02T05:50:12.580 DEBUG:boto:Token: None 2022-09-02T05:50:12.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4= 2022-09-02T05:50:12.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4=', 'Content-Length': '0'} 2022-09-02T05:50:12.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df6d1dd36bdc562-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.585 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.585 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.586 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.587 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.587 DEBUG:boto:Method: GET 2022-09-02T05:50:12.588 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.588 DEBUG:boto:Data: 2022-09-02T05:50:12.589 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.590 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.590 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.591 DEBUG:boto:Params: {} 2022-09-02T05:50:12.592 DEBUG:boto:Token: None 2022-09-02T05:50:12.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4= 2022-09-02T05:50:12.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4=', 'Content-Length': '0'} 2022-09-02T05:50:12.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4bb0b42f4d85788-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.596 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.597 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.598 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.598 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.599 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.599 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.600 DEBUG:boto:Data: 2022-09-02T05:50:12.600 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.601 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.602 DEBUG:boto:Params: {} 2022-09-02T05:50:12.602 DEBUG:boto:Token: None 2022-09-02T05:50:12.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OC4fw0sDEq7o8WgIKpYLNoySboU= 2022-09-02T05:50:12.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OC4fw0sDEq7o8WgIKpYLNoySboU=', 'Content-Length': '0'} 2022-09-02T05:50:12.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004522273d75f51dc9-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.608 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.608 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.609 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.610 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.610 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.611 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.611 DEBUG:boto:Data: 2022-09-02T05:50:12.612 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.613 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.613 DEBUG:boto:Params: {} 2022-09-02T05:50:12.614 DEBUG:boto:Token: None 2022-09-02T05:50:12.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OC4fw0sDEq7o8WgIKpYLNoySboU= 2022-09-02T05:50:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OC4fw0sDEq7o8WgIKpYLNoySboU=', 'Content-Length': '0'} 2022-09-02T05:50:12.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f59e4a507fcc7ad8-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:12.621 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.622 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.623 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.623 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.624 DEBUG:boto:Method: GET 2022-09-02T05:50:12.625 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.626 DEBUG:boto:Data: 2022-09-02T05:50:12.627 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.628 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.629 DEBUG:boto:Params: {} 2022-09-02T05:50:12.630 DEBUG:boto:Token: None 2022-09-02T05:50:12.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4= 2022-09-02T05:50:12.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4=', 'Content-Length': '0'} 2022-09-02T05:50:12.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a281723736e26d4-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.678 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.679 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.679 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.680 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.681 DEBUG:boto:Method: GET 2022-09-02T05:50:12.681 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.682 DEBUG:boto:Data: 2022-09-02T05:50:12.682 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.683 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.683 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.684 DEBUG:boto:Params: {} 2022-09-02T05:50:12.684 DEBUG:boto:Token: None 2022-09-02T05:50:12.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:50:12.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4= 2022-09-02T05:50:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OIoSQUCFEMu4V64BZn1gIVy3/y4=', 'Content-Length': '0'} 2022-09-02T05:50:12.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f3e1229cb56830-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.689 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.690 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.690 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.691 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.691 DEBUG:boto:Method: GET 2022-09-02T05:50:12.692 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.692 DEBUG:boto:Data: 2022-09-02T05:50:12.693 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.694 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.695 DEBUG:boto:Params: {} 2022-09-02T05:50:12.695 DEBUG:boto:Token: None 2022-09-02T05:50:12.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ba6b2bcebde75c-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.700 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.701 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.702 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.702 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.703 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.703 DEBUG:boto:Method: GET 2022-09-02T05:50:12.704 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.705 DEBUG:boto:Data: 2022-09-02T05:50:12.705 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.706 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.707 DEBUG:boto:Params: {} 2022-09-02T05:50:12.707 DEBUG:boto:Token: None 2022-09-02T05:50:12.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dea9054adaa9e80-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.713 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.713 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.714 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.715 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.715 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:12.716 DEBUG:boto:Data: 2022-09-02T05:50:12.716 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.717 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.718 DEBUG:boto:Params: {} 2022-09-02T05:50:12.718 DEBUG:boto:Token: None 2022-09-02T05:50:12.719 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22 2022-09-02T05:50:12.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TEmqM26RQ2U2chu3wBoTxbpYJC8= 2022-09-02T05:50:12.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TEmqM26RQ2U2chu3wBoTxbpYJC8=', 'Content-Length': '0'} 2022-09-02T05:50:12.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005014a6a0373d62fc-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.723 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.723 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.724 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.724 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:12.725 DEBUG:boto:Data: 2022-09-02T05:50:12.725 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.726 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.726 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.727 DEBUG:boto:Params: {} 2022-09-02T05:50:12.728 DEBUG:boto:Token: None 2022-09-02T05:50:12.728 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22 2022-09-02T05:50:12.729 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TEmqM26RQ2U2chu3wBoTxbpYJC8= 2022-09-02T05:50:12.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TEmqM26RQ2U2chu3wBoTxbpYJC8=', 'Content-Length': '0'} 2022-09-02T05:50:12.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda3b7b2f85778a2-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:50:12.732 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.733 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.734 DEBUG:boto:Method: GET 2022-09-02T05:50:12.734 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:12.735 DEBUG:boto:Data: 2022-09-02T05:50:12.735 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.736 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.737 DEBUG:boto:Params: {} 2022-09-02T05:50:12.737 DEBUG:boto:Token: None 2022-09-02T05:50:12.738 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22 2022-09-02T05:50:12.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u7iKG58PZ6iHDe5v3mepSiYD5VM= 2022-09-02T05:50:12.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u7iKG58PZ6iHDe5v3mepSiYD5VM=', 'Content-Length': '0'} 2022-09-02T05:50:12.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009839c878124bc25f-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.742 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:50:12.743 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:50:12.743 DEBUG:boto:Method: GET 2022-09-02T05:50:12.744 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:50:12.744 DEBUG:boto:Data: 2022-09-02T05:50:12.745 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.746 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.746 DEBUG:boto:Params: {} 2022-09-02T05:50:12.747 DEBUG:boto:Token: None 2022-09-02T05:50:12.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj22 2022-09-02T05:50:12.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u7iKG58PZ6iHDe5v3mepSiYD5VM= 2022-09-02T05:50:12.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u7iKG58PZ6iHDe5v3mepSiYD5VM=', 'Content-Length': '0'} 2022-09-02T05:50:12.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000599e0f6c49965d62-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.753 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.754 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.755 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.756 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.757 DEBUG:boto:Method: GET 2022-09-02T05:50:12.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.759 DEBUG:boto:Data: 2022-09-02T05:50:12.760 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.762 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.762 DEBUG:boto:Params: {} 2022-09-02T05:50:12.763 DEBUG:boto:Token: None 2022-09-02T05:50:12.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002187f18641e4086c-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.768 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.769 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.770 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.771 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.771 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.772 DEBUG:boto:Method: GET 2022-09-02T05:50:12.772 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:50:12.773 DEBUG:boto:Data: 2022-09-02T05:50:12.773 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.774 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.775 DEBUG:boto:Params: {} 2022-09-02T05:50:12.775 DEBUG:boto:Token: None 2022-09-02T05:50:12.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000204d8fd887080283-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.822 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:12.824 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.825 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.825 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.826 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.826 DEBUG:boto:Method: GET 2022-09-02T05:50:12.827 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.828 DEBUG:boto:Data: 2022-09-02T05:50:12.828 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.829 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.830 DEBUG:boto:Params: {} 2022-09-02T05:50:12.831 DEBUG:boto:Token: None 2022-09-02T05:50:12.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE= 2022-09-02T05:50:12.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE=', 'Content-Length': '0'} 2022-09-02T05:50:12.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002411080f240ca9a0-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.836 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.837 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.837 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.838 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.838 DEBUG:boto:Method: GET 2022-09-02T05:50:12.839 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.839 DEBUG:boto:Data: 2022-09-02T05:50:12.840 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.841 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.841 DEBUG:boto:Params: {} 2022-09-02T05:50:12.842 DEBUG:boto:Token: None 2022-09-02T05:50:12.843 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE= 2022-09-02T05:50:12.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE=', 'Content-Length': '0'} 2022-09-02T05:50:12.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682b6c87d51868c2-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.847 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.848 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.848 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.849 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.849 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.850 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.850 DEBUG:boto:Data: 2022-09-02T05:50:12.851 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.852 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.852 DEBUG:boto:Params: {} 2022-09-02T05:50:12.853 DEBUG:boto:Token: None 2022-09-02T05:50:12.854 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i010rZg5GeEeO0ANPyAOSWaOErE= 2022-09-02T05:50:12.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i010rZg5GeEeO0ANPyAOSWaOErE=', 'Content-Length': '0'} 2022-09-02T05:50:12.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab479cb12807fe8-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.857 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.858 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.858 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.859 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.859 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.860 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.860 DEBUG:boto:Data: 2022-09-02T05:50:12.861 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.862 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.863 DEBUG:boto:Params: {} 2022-09-02T05:50:12.863 DEBUG:boto:Token: None 2022-09-02T05:50:12.864 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i010rZg5GeEeO0ANPyAOSWaOErE= 2022-09-02T05:50:12.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i010rZg5GeEeO0ANPyAOSWaOErE=', 'Content-Length': '0'} 2022-09-02T05:50:12.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e72463026f53787-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:12.868 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.869 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.869 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.870 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.870 DEBUG:boto:Method: GET 2022-09-02T05:50:12.871 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.872 DEBUG:boto:Data: 2022-09-02T05:50:12.872 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.873 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.874 DEBUG:boto:Params: {} 2022-09-02T05:50:12.874 DEBUG:boto:Token: None 2022-09-02T05:50:12.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE= 2022-09-02T05:50:12.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE=', 'Content-Length': '0'} 2022-09-02T05:50:12.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0169c9918ec705-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.879 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.880 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.880 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.881 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.881 DEBUG:boto:Method: GET 2022-09-02T05:50:12.882 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.882 DEBUG:boto:Data: 2022-09-02T05:50:12.883 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.884 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.884 DEBUG:boto:Params: {} 2022-09-02T05:50:12.885 DEBUG:boto:Token: None 2022-09-02T05:50:12.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:50:12.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE= 2022-09-02T05:50:12.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vxziG3q5h/QrnZY2Kbic+niAaKE=', 'Content-Length': '0'} 2022-09-02T05:50:12.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008087bdae1d4f2dad-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.930 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.931 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.932 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:12.933 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:12.934 DEBUG:boto:Method: GET 2022-09-02T05:50:12.935 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:12.936 DEBUG:boto:Data: 2022-09-02T05:50:12.937 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.939 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.940 DEBUG:boto:Params: {} 2022-09-02T05:50:12.941 DEBUG:boto:Token: None 2022-09-02T05:50:12.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2d11cb6922f9fc-0063119994-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.949 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.950 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:12.951 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:12.952 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:12.953 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:12.954 DEBUG:boto:Method: GET 2022-09-02T05:50:12.955 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:12.956 DEBUG:boto:Data: 2022-09-02T05:50:12.957 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.958 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.959 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.960 DEBUG:boto:Params: {} 2022-09-02T05:50:12.961 DEBUG:boto:Token: None 2022-09-02T05:50:12.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/ 2022-09-02T05:50:12.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k= 2022-09-02T05:50:12.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Vu0EiDNRQ6XimmbiH03Wrf4/5k=', 'Content-Length': '0'} 2022-09-02T05:50:12.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004605e77d432b2f8d-0063119994-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.968 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:12.970 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.971 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.972 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.973 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:12.974 DEBUG:boto:Data: 2022-09-02T05:50:12.974 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:12.976 DEBUG:boto:Port: 8001 2022-09-02T05:50:12.977 DEBUG:boto:Params: {} 2022-09-02T05:50:12.979 DEBUG:boto:Token: None 2022-09-02T05:50:12.980 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33 2022-09-02T05:50:12.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PQtZ74uOzQHz6NaNFIHLbUJ6EGg= 2022-09-02T05:50:12.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PQtZ74uOzQHz6NaNFIHLbUJ6EGg=', 'Content-Length': '0'} 2022-09-02T05:50:12.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096846477c1b0d89f-0063119994-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:12.986 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:12.987 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:12.988 DEBUG:boto:Method: HEAD 2022-09-02T05:50:12.989 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:12.990 DEBUG:boto:Data: 2022-09-02T05:50:12.991 DEBUG:boto:Headers: {} 2022-09-02T05:50:12.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:12.993 DEBUG:boto:Port: 8000 2022-09-02T05:50:12.994 DEBUG:boto:Params: {} 2022-09-02T05:50:12.995 DEBUG:boto:Token: None 2022-09-02T05:50:12.996 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:12 GMT /xljbkf-74/obj33 2022-09-02T05:50:12.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PQtZ74uOzQHz6NaNFIHLbUJ6EGg= 2022-09-02T05:50:12.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PQtZ74uOzQHz6NaNFIHLbUJ6EGg=', 'Content-Length': '0'} 2022-09-02T05:50:13.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdad08877f653120-0063119994-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:50:13.001 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:13.002 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:13.002 DEBUG:boto:Method: GET 2022-09-02T05:50:13.003 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:13.003 DEBUG:boto:Data: 2022-09-02T05:50:13.004 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.005 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.006 DEBUG:boto:Params: {} 2022-09-02T05:50:13.006 DEBUG:boto:Token: None 2022-09-02T05:50:13.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj33 2022-09-02T05:50:13.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iWs3PBlh+TxwIA1bhrS+KQ3Wklc= 2022-09-02T05:50:13.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iWs3PBlh+TxwIA1bhrS+KQ3Wklc=', 'Content-Length': '0'} 2022-09-02T05:50:13.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000574a3d5438da1d0-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.055 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:50:13.056 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:50:13.056 DEBUG:boto:Method: GET 2022-09-02T05:50:13.057 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:50:13.058 DEBUG:boto:Data: 2022-09-02T05:50:13.059 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.061 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.062 DEBUG:boto:Params: {} 2022-09-02T05:50:13.063 DEBUG:boto:Token: None 2022-09-02T05:50:13.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj33 2022-09-02T05:50:13.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iWs3PBlh+TxwIA1bhrS+KQ3Wklc= 2022-09-02T05:50:13.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iWs3PBlh+TxwIA1bhrS+KQ3Wklc=', 'Content-Length': '0'} 2022-09-02T05:50:13.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6dda7e0c8fd72cd-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.070 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.071 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.072 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:13.073 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:13.074 DEBUG:boto:Method: GET 2022-09-02T05:50:13.075 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:13.076 DEBUG:boto:Data: 2022-09-02T05:50:13.077 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.079 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.080 DEBUG:boto:Params: {} 2022-09-02T05:50:13.081 DEBUG:boto:Token: None 2022-09-02T05:50:13.082 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.084 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d3030c6990fe65-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.089 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.091 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.092 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.093 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:13.094 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:50:13.095 DEBUG:boto:Method: GET 2022-09-02T05:50:13.096 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:50:13.097 DEBUG:boto:Data: 2022-09-02T05:50:13.098 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.099 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.100 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.101 DEBUG:boto:Params: {} 2022-09-02T05:50:13.102 DEBUG:boto:Token: None 2022-09-02T05:50:13.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000338263761c96da3d-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.110 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:13.112 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.113 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.114 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.115 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.116 DEBUG:boto:Method: GET 2022-09-02T05:50:13.117 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.117 DEBUG:boto:Data: 2022-09-02T05:50:13.118 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.119 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.120 DEBUG:boto:Params: {} 2022-09-02T05:50:13.121 DEBUG:boto:Token: None 2022-09-02T05:50:13.121 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc= 2022-09-02T05:50:13.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc=', 'Content-Length': '0'} 2022-09-02T05:50:13.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e9204822e87172-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.126 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.127 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.127 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.128 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.128 DEBUG:boto:Method: GET 2022-09-02T05:50:13.129 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.129 DEBUG:boto:Data: 2022-09-02T05:50:13.130 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.131 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.131 DEBUG:boto:Params: {} 2022-09-02T05:50:13.132 DEBUG:boto:Token: None 2022-09-02T05:50:13.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc= 2022-09-02T05:50:13.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc=', 'Content-Length': '0'} 2022-09-02T05:50:13.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000421ef471f40a07b9-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.137 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.138 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.138 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.139 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.139 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.140 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.141 DEBUG:boto:Data: 2022-09-02T05:50:13.141 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.142 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.143 DEBUG:boto:Params: {} 2022-09-02T05:50:13.143 DEBUG:boto:Token: None 2022-09-02T05:50:13.144 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lMzRXsDAzJKFrwxTwXTtG/TCjIo= 2022-09-02T05:50:13.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lMzRXsDAzJKFrwxTwXTtG/TCjIo=', 'Content-Length': '0'} 2022-09-02T05:50:13.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081ac10308809dd4-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.161 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.161 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.162 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.162 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.163 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.163 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.164 DEBUG:boto:Data: 2022-09-02T05:50:13.164 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.165 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.165 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.166 DEBUG:boto:Params: {} 2022-09-02T05:50:13.167 DEBUG:boto:Token: None 2022-09-02T05:50:13.168 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lMzRXsDAzJKFrwxTwXTtG/TCjIo= 2022-09-02T05:50:13.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lMzRXsDAzJKFrwxTwXTtG/TCjIo=', 'Content-Length': '0'} 2022-09-02T05:50:13.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191f7a9a5ed6db94-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:13.172 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.173 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.173 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.174 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.175 DEBUG:boto:Method: GET 2022-09-02T05:50:13.175 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.176 DEBUG:boto:Data: 2022-09-02T05:50:13.176 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.178 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.178 DEBUG:boto:Params: {} 2022-09-02T05:50:13.179 DEBUG:boto:Token: None 2022-09-02T05:50:13.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc= 2022-09-02T05:50:13.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc=', 'Content-Length': '0'} 2022-09-02T05:50:13.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cd26f1683e5283-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.183 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.184 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.185 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.185 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.186 DEBUG:boto:Method: GET 2022-09-02T05:50:13.187 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.187 DEBUG:boto:Data: 2022-09-02T05:50:13.188 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.189 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.189 DEBUG:boto:Params: {} 2022-09-02T05:50:13.190 DEBUG:boto:Token: None 2022-09-02T05:50:13.191 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:50:13.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc= 2022-09-02T05:50:13.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2UYxloYtQ6zqRq/aYJ301qpprc=', 'Content-Length': '0'} 2022-09-02T05:50:13.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009031f91640eae374-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.238 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.239 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.240 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.241 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.242 DEBUG:boto:Method: GET 2022-09-02T05:50:13.243 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.244 DEBUG:boto:Data: 2022-09-02T05:50:13.245 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.247 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.248 DEBUG:boto:Params: {} 2022-09-02T05:50:13.249 DEBUG:boto:Token: None 2022-09-02T05:50:13.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92a9cb3b483e118-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.257 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.258 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.259 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.260 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.261 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.262 DEBUG:boto:Method: GET 2022-09-02T05:50:13.263 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.264 DEBUG:boto:Data: 2022-09-02T05:50:13.265 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.267 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.268 DEBUG:boto:Params: {} 2022-09-02T05:50:13.269 DEBUG:boto:Token: None 2022-09-02T05:50:13.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c10f4a30a16e106-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.278 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.279 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.280 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.281 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.282 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:13.283 DEBUG:boto:Data: 2022-09-02T05:50:13.284 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.286 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.288 DEBUG:boto:Params: {} 2022-09-02T05:50:13.289 DEBUG:boto:Token: None 2022-09-02T05:50:13.290 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34 2022-09-02T05:50:13.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VBFW0RSq8Qnz3we3iD4XL42nd5A= 2022-09-02T05:50:13.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VBFW0RSq8Qnz3we3iD4XL42nd5A=', 'Content-Length': '0'} 2022-09-02T05:50:13.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad18b5252dd90a6f-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.294 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.295 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.296 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.296 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:13.297 DEBUG:boto:Data: 2022-09-02T05:50:13.297 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.298 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.299 DEBUG:boto:Params: {} 2022-09-02T05:50:13.300 DEBUG:boto:Token: None 2022-09-02T05:50:13.300 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34 2022-09-02T05:50:13.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VBFW0RSq8Qnz3we3iD4XL42nd5A= 2022-09-02T05:50:13.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VBFW0RSq8Qnz3we3iD4XL42nd5A=', 'Content-Length': '0'} 2022-09-02T05:50:13.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba7796b318b7997-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:50:13.305 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.306 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.307 DEBUG:boto:Method: GET 2022-09-02T05:50:13.307 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:13.308 DEBUG:boto:Data: 2022-09-02T05:50:13.308 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.309 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.310 DEBUG:boto:Params: {} 2022-09-02T05:50:13.310 DEBUG:boto:Token: None 2022-09-02T05:50:13.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34 2022-09-02T05:50:13.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pZ6cL4eUbEluV6Vzb5zMMQsFQc4= 2022-09-02T05:50:13.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pZ6cL4eUbEluV6Vzb5zMMQsFQc4=', 'Content-Length': '0'} 2022-09-02T05:50:13.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0418ba9bd6d8199-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.316 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:50:13.316 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:50:13.317 DEBUG:boto:Method: GET 2022-09-02T05:50:13.318 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:50:13.318 DEBUG:boto:Data: 2022-09-02T05:50:13.319 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.320 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.320 DEBUG:boto:Params: {} 2022-09-02T05:50:13.321 DEBUG:boto:Token: None 2022-09-02T05:50:13.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj34 2022-09-02T05:50:13.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pZ6cL4eUbEluV6Vzb5zMMQsFQc4= 2022-09-02T05:50:13.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pZ6cL4eUbEluV6Vzb5zMMQsFQc4=', 'Content-Length': '0'} 2022-09-02T05:50:13.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f868488fab4a7a90-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.326 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.327 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.327 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.328 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.328 DEBUG:boto:Method: GET 2022-09-02T05:50:13.329 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.329 DEBUG:boto:Data: 2022-09-02T05:50:13.330 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.331 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.332 DEBUG:boto:Params: {} 2022-09-02T05:50:13.332 DEBUG:boto:Token: None 2022-09-02T05:50:13.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064882c5c3e1195f4-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.338 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.339 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.339 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.340 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.340 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.341 DEBUG:boto:Method: GET 2022-09-02T05:50:13.342 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:50:13.342 DEBUG:boto:Data: 2022-09-02T05:50:13.343 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.344 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.345 DEBUG:boto:Params: {} 2022-09-02T05:50:13.345 DEBUG:boto:Token: None 2022-09-02T05:50:13.346 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d20052f7de904b95-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.351 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:13.352 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.353 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.353 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.354 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.354 DEBUG:boto:Method: GET 2022-09-02T05:50:13.355 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.355 DEBUG:boto:Data: 2022-09-02T05:50:13.356 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.357 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.357 DEBUG:boto:Params: {} 2022-09-02T05:50:13.358 DEBUG:boto:Token: None 2022-09-02T05:50:13.359 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc= 2022-09-02T05:50:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc=', 'Content-Length': '0'} 2022-09-02T05:50:13.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1d4623e75afe6e-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.363 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.363 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.364 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.364 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.365 DEBUG:boto:Method: GET 2022-09-02T05:50:13.365 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.366 DEBUG:boto:Data: 2022-09-02T05:50:13.367 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.368 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.369 DEBUG:boto:Params: {} 2022-09-02T05:50:13.369 DEBUG:boto:Token: None 2022-09-02T05:50:13.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc= 2022-09-02T05:50:13.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc=', 'Content-Length': '0'} 2022-09-02T05:50:13.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faef02d1578bd896-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.375 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.375 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.376 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.376 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.377 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.378 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.378 DEBUG:boto:Data: 2022-09-02T05:50:13.379 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.381 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.382 DEBUG:boto:Params: {} 2022-09-02T05:50:13.383 DEBUG:boto:Token: None 2022-09-02T05:50:13.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wLUo7k1U+GWAXx1T2SuFqT3i/Bo= 2022-09-02T05:50:13.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wLUo7k1U+GWAXx1T2SuFqT3i/Bo=', 'Content-Length': '0'} 2022-09-02T05:50:13.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece898408128c949-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.389 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.389 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.390 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.391 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.391 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.392 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.392 DEBUG:boto:Data: 2022-09-02T05:50:13.393 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.394 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.394 DEBUG:boto:Params: {} 2022-09-02T05:50:13.395 DEBUG:boto:Token: None 2022-09-02T05:50:13.396 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wLUo7k1U+GWAXx1T2SuFqT3i/Bo= 2022-09-02T05:50:13.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wLUo7k1U+GWAXx1T2SuFqT3i/Bo=', 'Content-Length': '0'} 2022-09-02T05:50:13.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e3d283e6945edd2-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:13.401 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.401 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.402 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.403 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.403 DEBUG:boto:Method: GET 2022-09-02T05:50:13.404 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.404 DEBUG:boto:Data: 2022-09-02T05:50:13.405 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.406 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.406 DEBUG:boto:Params: {} 2022-09-02T05:50:13.407 DEBUG:boto:Token: None 2022-09-02T05:50:13.408 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc= 2022-09-02T05:50:13.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc=', 'Content-Length': '0'} 2022-09-02T05:50:13.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849f5223dd183c56-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.454 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.455 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.455 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.456 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.456 DEBUG:boto:Method: GET 2022-09-02T05:50:13.457 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.458 DEBUG:boto:Data: 2022-09-02T05:50:13.458 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.459 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.460 DEBUG:boto:Params: {} 2022-09-02T05:50:13.460 DEBUG:boto:Token: None 2022-09-02T05:50:13.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:50:13.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc= 2022-09-02T05:50:13.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pwbmdsIDTZJHjI/JPnhefI4uhJc=', 'Content-Length': '0'} 2022-09-02T05:50:13.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b21bfc21ce1f0bb6-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.466 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.467 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.467 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.468 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.468 DEBUG:boto:Method: GET 2022-09-02T05:50:13.469 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.470 DEBUG:boto:Data: 2022-09-02T05:50:13.470 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.471 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.472 DEBUG:boto:Params: {} 2022-09-02T05:50:13.473 DEBUG:boto:Token: None 2022-09-02T05:50:13.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004af023e90bcbefd0-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.479 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.480 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.480 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.481 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.481 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.482 DEBUG:boto:Method: GET 2022-09-02T05:50:13.482 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.483 DEBUG:boto:Data: 2022-09-02T05:50:13.483 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.485 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.485 DEBUG:boto:Params: {} 2022-09-02T05:50:13.486 DEBUG:boto:Token: None 2022-09-02T05:50:13.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c41bc903e7e83f27-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.491 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.492 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.493 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.493 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.494 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:13.494 DEBUG:boto:Data: 2022-09-02T05:50:13.495 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.496 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.496 DEBUG:boto:Params: {} 2022-09-02T05:50:13.497 DEBUG:boto:Token: None 2022-09-02T05:50:13.498 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38 2022-09-02T05:50:13.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vH45GYnpfYHuMkt3KVQXH0xqUkA= 2022-09-02T05:50:13.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vH45GYnpfYHuMkt3KVQXH0xqUkA=', 'Content-Length': '0'} 2022-09-02T05:50:13.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008714b1d422fd3961-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.504 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.505 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.506 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.507 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:13.508 DEBUG:boto:Data: 2022-09-02T05:50:13.509 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.511 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.512 DEBUG:boto:Params: {} 2022-09-02T05:50:13.513 DEBUG:boto:Token: None 2022-09-02T05:50:13.514 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38 2022-09-02T05:50:13.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vH45GYnpfYHuMkt3KVQXH0xqUkA= 2022-09-02T05:50:13.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vH45GYnpfYHuMkt3KVQXH0xqUkA=', 'Content-Length': '0'} 2022-09-02T05:50:13.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18219a0c2bc4357-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:50:13.518 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.519 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.519 DEBUG:boto:Method: GET 2022-09-02T05:50:13.520 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:13.520 DEBUG:boto:Data: 2022-09-02T05:50:13.521 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.522 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.522 DEBUG:boto:Params: {} 2022-09-02T05:50:13.523 DEBUG:boto:Token: None 2022-09-02T05:50:13.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38 2022-09-02T05:50:13.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0h1dujtXIFSpgWxe/WW9jOQIG/Y= 2022-09-02T05:50:13.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0h1dujtXIFSpgWxe/WW9jOQIG/Y=', 'Content-Length': '0'} 2022-09-02T05:50:13.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcdc4ccfd30079a1-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.570 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:50:13.571 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:50:13.572 DEBUG:boto:Method: GET 2022-09-02T05:50:13.572 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:50:13.573 DEBUG:boto:Data: 2022-09-02T05:50:13.573 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.575 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.575 DEBUG:boto:Params: {} 2022-09-02T05:50:13.576 DEBUG:boto:Token: None 2022-09-02T05:50:13.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj38 2022-09-02T05:50:13.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0h1dujtXIFSpgWxe/WW9jOQIG/Y= 2022-09-02T05:50:13.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0h1dujtXIFSpgWxe/WW9jOQIG/Y=', 'Content-Length': '0'} 2022-09-02T05:50:13.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150a3afad123d4c9-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.581 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.581 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.582 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.582 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.583 DEBUG:boto:Method: GET 2022-09-02T05:50:13.583 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.584 DEBUG:boto:Data: 2022-09-02T05:50:13.584 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.585 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.586 DEBUG:boto:Params: {} 2022-09-02T05:50:13.587 DEBUG:boto:Token: None 2022-09-02T05:50:13.587 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048bf71d1fe96adeb-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.592 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.593 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.594 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.594 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.595 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.595 DEBUG:boto:Method: GET 2022-09-02T05:50:13.596 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:50:13.596 DEBUG:boto:Data: 2022-09-02T05:50:13.597 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.597 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.598 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.598 DEBUG:boto:Params: {} 2022-09-02T05:50:13.599 DEBUG:boto:Token: None 2022-09-02T05:50:13.600 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815177ea91ebf979-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.605 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:13.606 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.607 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.607 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.608 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.609 DEBUG:boto:Method: GET 2022-09-02T05:50:13.609 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.610 DEBUG:boto:Data: 2022-09-02T05:50:13.610 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.611 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.612 DEBUG:boto:Params: {} 2022-09-02T05:50:13.612 DEBUG:boto:Token: None 2022-09-02T05:50:13.613 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU= 2022-09-02T05:50:13.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU=', 'Content-Length': '0'} 2022-09-02T05:50:13.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eddc560381b86da1-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.618 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.618 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.619 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.619 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.620 DEBUG:boto:Method: GET 2022-09-02T05:50:13.620 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.621 DEBUG:boto:Data: 2022-09-02T05:50:13.621 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.622 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.623 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.623 DEBUG:boto:Params: {} 2022-09-02T05:50:13.624 DEBUG:boto:Token: None 2022-09-02T05:50:13.624 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU= 2022-09-02T05:50:13.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU=', 'Content-Length': '0'} 2022-09-02T05:50:13.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca91aac718ab9a8-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.629 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.630 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.631 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.632 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.633 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.634 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.635 DEBUG:boto:Data: 2022-09-02T05:50:13.635 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.636 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.637 DEBUG:boto:Params: {} 2022-09-02T05:50:13.638 DEBUG:boto:Token: None 2022-09-02T05:50:13.638 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j6DWpsL0M5zmgGRcuKKRW7JWVIM= 2022-09-02T05:50:13.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j6DWpsL0M5zmgGRcuKKRW7JWVIM=', 'Content-Length': '0'} 2022-09-02T05:50:13.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006597a89c79208330-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.643 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.643 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.644 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.644 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.645 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.645 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.646 DEBUG:boto:Data: 2022-09-02T05:50:13.647 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.648 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.648 DEBUG:boto:Params: {} 2022-09-02T05:50:13.649 DEBUG:boto:Token: None 2022-09-02T05:50:13.649 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j6DWpsL0M5zmgGRcuKKRW7JWVIM= 2022-09-02T05:50:13.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j6DWpsL0M5zmgGRcuKKRW7JWVIM=', 'Content-Length': '0'} 2022-09-02T05:50:13.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb80a6b0287fa01a-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:13.654 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.655 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.655 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.656 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.656 DEBUG:boto:Method: GET 2022-09-02T05:50:13.657 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.657 DEBUG:boto:Data: 2022-09-02T05:50:13.658 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.659 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.660 DEBUG:boto:Params: {} 2022-09-02T05:50:13.660 DEBUG:boto:Token: None 2022-09-02T05:50:13.661 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU= 2022-09-02T05:50:13.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU=', 'Content-Length': '0'} 2022-09-02T05:50:13.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa0221eabbfb88db-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.665 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.666 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.666 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.667 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.667 DEBUG:boto:Method: GET 2022-09-02T05:50:13.668 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.669 DEBUG:boto:Data: 2022-09-02T05:50:13.669 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.670 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.671 DEBUG:boto:Params: {} 2022-09-02T05:50:13.672 DEBUG:boto:Token: None 2022-09-02T05:50:13.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:50:13.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU= 2022-09-02T05:50:13.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lJS4RIIa7BFMh9Dz+X6hsIiMRnU=', 'Content-Length': '0'} 2022-09-02T05:50:13.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006522b198fcea5135-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.676 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.677 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.678 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.678 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.679 DEBUG:boto:Method: GET 2022-09-02T05:50:13.679 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.680 DEBUG:boto:Data: 2022-09-02T05:50:13.680 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.681 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.682 DEBUG:boto:Params: {} 2022-09-02T05:50:13.682 DEBUG:boto:Token: None 2022-09-02T05:50:13.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd5ae8dfc175c6c3-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.688 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.689 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.689 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.690 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.690 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.691 DEBUG:boto:Method: GET 2022-09-02T05:50:13.691 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.692 DEBUG:boto:Data: 2022-09-02T05:50:13.692 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.693 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.694 DEBUG:boto:Params: {} 2022-09-02T05:50:13.695 DEBUG:boto:Token: None 2022-09-02T05:50:13.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5f3a5ba2677797-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.700 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.701 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.701 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.702 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.703 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:13.703 DEBUG:boto:Data: 2022-09-02T05:50:13.704 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.705 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.705 DEBUG:boto:Params: {} 2022-09-02T05:50:13.706 DEBUG:boto:Token: None 2022-09-02T05:50:13.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44 2022-09-02T05:50:13.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+nng13h0F/4o2H+fGG68rjnm48Q= 2022-09-02T05:50:13.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+nng13h0F/4o2H+fGG68rjnm48Q=', 'Content-Length': '0'} 2022-09-02T05:50:13.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119ea24dc4e92845-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.710 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.711 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.711 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.712 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:13.712 DEBUG:boto:Data: 2022-09-02T05:50:13.713 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.714 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.714 DEBUG:boto:Params: {} 2022-09-02T05:50:13.715 DEBUG:boto:Token: None 2022-09-02T05:50:13.715 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44 2022-09-02T05:50:13.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+nng13h0F/4o2H+fGG68rjnm48Q= 2022-09-02T05:50:13.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+nng13h0F/4o2H+fGG68rjnm48Q=', 'Content-Length': '0'} 2022-09-02T05:50:13.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e1273cf22d6114-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:50:13.720 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.720 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.721 DEBUG:boto:Method: GET 2022-09-02T05:50:13.721 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:13.722 DEBUG:boto:Data: 2022-09-02T05:50:13.723 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.724 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.724 DEBUG:boto:Params: {} 2022-09-02T05:50:13.725 DEBUG:boto:Token: None 2022-09-02T05:50:13.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44 2022-09-02T05:50:13.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mn1M/yw3wRAef8xMGjiKjZXA7dk= 2022-09-02T05:50:13.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mn1M/yw3wRAef8xMGjiKjZXA7dk=', 'Content-Length': '0'} 2022-09-02T05:50:13.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639d5bf5518ca23d-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.770 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:50:13.771 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:50:13.771 DEBUG:boto:Method: GET 2022-09-02T05:50:13.772 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:50:13.772 DEBUG:boto:Data: 2022-09-02T05:50:13.773 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.774 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.774 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.775 DEBUG:boto:Params: {} 2022-09-02T05:50:13.775 DEBUG:boto:Token: None 2022-09-02T05:50:13.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj44 2022-09-02T05:50:13.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mn1M/yw3wRAef8xMGjiKjZXA7dk= 2022-09-02T05:50:13.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mn1M/yw3wRAef8xMGjiKjZXA7dk=', 'Content-Length': '0'} 2022-09-02T05:50:13.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba42a774f2f81310-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.780 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.781 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.781 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.782 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.783 DEBUG:boto:Method: GET 2022-09-02T05:50:13.783 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.784 DEBUG:boto:Data: 2022-09-02T05:50:13.784 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.785 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.786 DEBUG:boto:Params: {} 2022-09-02T05:50:13.787 DEBUG:boto:Token: None 2022-09-02T05:50:13.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb4c990e367d80f-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.792 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.793 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.794 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.794 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.795 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.796 DEBUG:boto:Method: GET 2022-09-02T05:50:13.796 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:50:13.797 DEBUG:boto:Data: 2022-09-02T05:50:13.797 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.799 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.799 DEBUG:boto:Params: {} 2022-09-02T05:50:13.800 DEBUG:boto:Token: None 2022-09-02T05:50:13.800 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231ba1fec9e2b65f-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.805 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:13.807 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.807 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.808 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.808 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.809 DEBUG:boto:Method: GET 2022-09-02T05:50:13.810 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.810 DEBUG:boto:Data: 2022-09-02T05:50:13.811 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.812 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.812 DEBUG:boto:Params: {} 2022-09-02T05:50:13.813 DEBUG:boto:Token: None 2022-09-02T05:50:13.814 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94= 2022-09-02T05:50:13.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94=', 'Content-Length': '0'} 2022-09-02T05:50:13.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed0465e7c371412-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.819 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.819 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.820 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.821 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.821 DEBUG:boto:Method: GET 2022-09-02T05:50:13.822 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.822 DEBUG:boto:Data: 2022-09-02T05:50:13.823 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.824 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.825 DEBUG:boto:Params: {} 2022-09-02T05:50:13.825 DEBUG:boto:Token: None 2022-09-02T05:50:13.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94= 2022-09-02T05:50:13.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94=', 'Content-Length': '0'} 2022-09-02T05:50:13.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92a8ff42dfb39d5-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.830 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.831 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.831 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.832 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.832 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.833 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.833 DEBUG:boto:Data: 2022-09-02T05:50:13.834 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.835 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.836 DEBUG:boto:Params: {} 2022-09-02T05:50:13.837 DEBUG:boto:Token: None 2022-09-02T05:50:13.838 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GjsUv2Ux0orVf2zV/m1ivtbcBu0= 2022-09-02T05:50:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GjsUv2Ux0orVf2zV/m1ivtbcBu0=', 'Content-Length': '0'} 2022-09-02T05:50:13.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da830cebb981f0d8-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.844 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.845 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.846 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.847 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.848 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.849 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.850 DEBUG:boto:Data: 2022-09-02T05:50:13.851 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.852 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.853 DEBUG:boto:Params: {} 2022-09-02T05:50:13.854 DEBUG:boto:Token: None 2022-09-02T05:50:13.855 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GjsUv2Ux0orVf2zV/m1ivtbcBu0= 2022-09-02T05:50:13.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GjsUv2Ux0orVf2zV/m1ivtbcBu0=', 'Content-Length': '0'} 2022-09-02T05:50:13.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d03980cf9eb0e6-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:13.862 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.863 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.864 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.865 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.866 DEBUG:boto:Method: GET 2022-09-02T05:50:13.867 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.868 DEBUG:boto:Data: 2022-09-02T05:50:13.869 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.871 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.872 DEBUG:boto:Params: {} 2022-09-02T05:50:13.873 DEBUG:boto:Token: None 2022-09-02T05:50:13.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94= 2022-09-02T05:50:13.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94=', 'Content-Length': '0'} 2022-09-02T05:50:13.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007ffb51da998bce1-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.881 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.882 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.883 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.884 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.884 DEBUG:boto:Method: GET 2022-09-02T05:50:13.885 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.885 DEBUG:boto:Data: 2022-09-02T05:50:13.886 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.887 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.887 DEBUG:boto:Params: {} 2022-09-02T05:50:13.888 DEBUG:boto:Token: None 2022-09-02T05:50:13.888 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:50:13.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94= 2022-09-02T05:50:13.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q18WD7J1c1b3V5pFq9yww9C1M94=', 'Content-Length': '0'} 2022-09-02T05:50:13.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ee87369c24505f-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.893 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.893 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.894 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:13.894 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:13.895 DEBUG:boto:Method: GET 2022-09-02T05:50:13.895 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:13.896 DEBUG:boto:Data: 2022-09-02T05:50:13.896 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.897 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.898 DEBUG:boto:Params: {} 2022-09-02T05:50:13.899 DEBUG:boto:Token: None 2022-09-02T05:50:13.899 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000daff04cfcd4753-0063119995-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.904 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.905 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:13.906 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:13.907 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:13.908 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:13.909 DEBUG:boto:Method: GET 2022-09-02T05:50:13.910 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:13.911 DEBUG:boto:Data: 2022-09-02T05:50:13.912 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.914 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.915 DEBUG:boto:Params: {} 2022-09-02T05:50:13.916 DEBUG:boto:Token: None 2022-09-02T05:50:13.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/ 2022-09-02T05:50:13.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go= 2022-09-02T05:50:13.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EG2M9qt4xrdxRdSXM/+aP368/go=', 'Content-Length': '0'} 2022-09-02T05:50:13.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037254357023bb802-0063119995-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.923 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:13.925 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.926 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.927 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.927 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:13.928 DEBUG:boto:Data: 2022-09-02T05:50:13.929 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.931 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.931 DEBUG:boto:Params: {} 2022-09-02T05:50:13.932 DEBUG:boto:Token: None 2022-09-02T05:50:13.933 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51 2022-09-02T05:50:13.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxyveB3h+IPANZiP6egntrG0CVg= 2022-09-02T05:50:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxyveB3h+IPANZiP6egntrG0CVg=', 'Content-Length': '0'} 2022-09-02T05:50:13.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359150ebd86ae276-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.937 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.937 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.938 DEBUG:boto:Method: HEAD 2022-09-02T05:50:13.938 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:13.939 DEBUG:boto:Data: 2022-09-02T05:50:13.939 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:13.940 DEBUG:boto:Port: 8000 2022-09-02T05:50:13.941 DEBUG:boto:Params: {} 2022-09-02T05:50:13.941 DEBUG:boto:Token: None 2022-09-02T05:50:13.942 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51 2022-09-02T05:50:13.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxyveB3h+IPANZiP6egntrG0CVg= 2022-09-02T05:50:13.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxyveB3h+IPANZiP6egntrG0CVg=', 'Content-Length': '0'} 2022-09-02T05:50:13.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042aa43efe5fc7cb4-0063119995-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:50:13.946 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.946 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.947 DEBUG:boto:Method: GET 2022-09-02T05:50:13.948 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:13.948 DEBUG:boto:Data: 2022-09-02T05:50:13.949 DEBUG:boto:Headers: {} 2022-09-02T05:50:13.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:13.950 DEBUG:boto:Port: 8001 2022-09-02T05:50:13.950 DEBUG:boto:Params: {} 2022-09-02T05:50:13.951 DEBUG:boto:Token: None 2022-09-02T05:50:13.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:13 GMT /xljbkf-74/obj51 2022-09-02T05:50:13.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8Uq4xRLMaFa2CqmNIXfvZGbazhc= 2022-09-02T05:50:13.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8Uq4xRLMaFa2CqmNIXfvZGbazhc=', 'Content-Length': '0'} 2022-09-02T05:50:13.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f33d696cbcb70c25-0063119995-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:13.998 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:50:13.999 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:50:13.999 DEBUG:boto:Method: GET 2022-09-02T05:50:14.000 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:50:14.000 DEBUG:boto:Data: 2022-09-02T05:50:14.001 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.002 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.003 DEBUG:boto:Params: {} 2022-09-02T05:50:14.003 DEBUG:boto:Token: None 2022-09-02T05:50:14.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj51 2022-09-02T05:50:14.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AowoLlzmjqDlzTBcwxcXPFku5xo= 2022-09-02T05:50:14.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AowoLlzmjqDlzTBcwxcXPFku5xo=', 'Content-Length': '0'} 2022-09-02T05:50:14.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5fab4a6e4ed7bb8-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.008 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.008 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.009 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:14.009 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:14.010 DEBUG:boto:Method: GET 2022-09-02T05:50:14.010 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:14.011 DEBUG:boto:Data: 2022-09-02T05:50:14.011 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.013 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.013 DEBUG:boto:Params: {} 2022-09-02T05:50:14.014 DEBUG:boto:Token: None 2022-09-02T05:50:14.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cafd15ce2f54183f-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.019 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.019 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.020 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.020 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:14.021 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:50:14.022 DEBUG:boto:Method: GET 2022-09-02T05:50:14.022 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:50:14.023 DEBUG:boto:Data: 2022-09-02T05:50:14.023 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.024 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.025 DEBUG:boto:Params: {} 2022-09-02T05:50:14.025 DEBUG:boto:Token: None 2022-09-02T05:50:14.026 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006614dc422ec0ed9-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.030 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:14.032 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.032 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.033 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.033 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.034 DEBUG:boto:Method: GET 2022-09-02T05:50:14.034 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.035 DEBUG:boto:Data: 2022-09-02T05:50:14.035 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.036 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.037 DEBUG:boto:Params: {} 2022-09-02T05:50:14.037 DEBUG:boto:Token: None 2022-09-02T05:50:14.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8= 2022-09-02T05:50:14.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8=', 'Content-Length': '0'} 2022-09-02T05:50:14.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d681117e0d4eacf-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.042 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.043 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.043 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.044 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.044 DEBUG:boto:Method: GET 2022-09-02T05:50:14.045 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.045 DEBUG:boto:Data: 2022-09-02T05:50:14.046 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.047 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.047 DEBUG:boto:Params: {} 2022-09-02T05:50:14.048 DEBUG:boto:Token: None 2022-09-02T05:50:14.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8= 2022-09-02T05:50:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8=', 'Content-Length': '0'} 2022-09-02T05:50:14.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6415b422099ef36-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.053 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.054 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.054 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.055 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.055 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.056 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.056 DEBUG:boto:Data: 2022-09-02T05:50:14.057 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.058 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.058 DEBUG:boto:Params: {} 2022-09-02T05:50:14.059 DEBUG:boto:Token: None 2022-09-02T05:50:14.059 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6hTpa4f4aFi6ocgQDOSZxZMSYtM= 2022-09-02T05:50:14.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6hTpa4f4aFi6ocgQDOSZxZMSYtM=', 'Content-Length': '0'} 2022-09-02T05:50:14.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af132275e62d18e1-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.063 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.064 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.064 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.065 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.065 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.066 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.066 DEBUG:boto:Data: 2022-09-02T05:50:14.067 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.068 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.068 DEBUG:boto:Params: {} 2022-09-02T05:50:14.069 DEBUG:boto:Token: None 2022-09-02T05:50:14.069 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6hTpa4f4aFi6ocgQDOSZxZMSYtM= 2022-09-02T05:50:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6hTpa4f4aFi6ocgQDOSZxZMSYtM=', 'Content-Length': '0'} 2022-09-02T05:50:14.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9b8092f5c13538-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:14.074 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.074 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.075 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.075 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.076 DEBUG:boto:Method: GET 2022-09-02T05:50:14.076 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.077 DEBUG:boto:Data: 2022-09-02T05:50:14.077 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.078 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.079 DEBUG:boto:Params: {} 2022-09-02T05:50:14.079 DEBUG:boto:Token: None 2022-09-02T05:50:14.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8= 2022-09-02T05:50:14.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8=', 'Content-Length': '0'} 2022-09-02T05:50:14.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbce6316e67a5f9-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.130 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.131 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.131 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.132 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.132 DEBUG:boto:Method: GET 2022-09-02T05:50:14.133 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.134 DEBUG:boto:Data: 2022-09-02T05:50:14.134 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.135 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.136 DEBUG:boto:Params: {} 2022-09-02T05:50:14.137 DEBUG:boto:Token: None 2022-09-02T05:50:14.137 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:50:14.138 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8= 2022-09-02T05:50:14.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hmr+IgaXAYFVff+qGCBf/nKX7V8=', 'Content-Length': '0'} 2022-09-02T05:50:14.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8401dc751fc288-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.141 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.142 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.142 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.143 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.144 DEBUG:boto:Method: GET 2022-09-02T05:50:14.145 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.146 DEBUG:boto:Data: 2022-09-02T05:50:14.147 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.149 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.150 DEBUG:boto:Params: {} 2022-09-02T05:50:14.151 DEBUG:boto:Token: None 2022-09-02T05:50:14.152 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f531a1a8c616037e-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.159 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.161 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.162 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.163 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.164 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.165 DEBUG:boto:Method: GET 2022-09-02T05:50:14.166 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.167 DEBUG:boto:Data: 2022-09-02T05:50:14.168 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.169 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.170 DEBUG:boto:Params: {} 2022-09-02T05:50:14.172 DEBUG:boto:Token: None 2022-09-02T05:50:14.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e013d1c7ea177225-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.179 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.180 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.181 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.181 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.182 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:14.182 DEBUG:boto:Data: 2022-09-02T05:50:14.183 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.184 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.184 DEBUG:boto:Params: {} 2022-09-02T05:50:14.185 DEBUG:boto:Token: None 2022-09-02T05:50:14.186 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55 2022-09-02T05:50:14.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mzEeQ9BA1ATfFhqEzp63a6nQrHY= 2022-09-02T05:50:14.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mzEeQ9BA1ATfFhqEzp63a6nQrHY=', 'Content-Length': '0'} 2022-09-02T05:50:14.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8085060b3a905b-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.189 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.190 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.190 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.191 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:14.191 DEBUG:boto:Data: 2022-09-02T05:50:14.192 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.193 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.193 DEBUG:boto:Params: {} 2022-09-02T05:50:14.194 DEBUG:boto:Token: None 2022-09-02T05:50:14.194 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55 2022-09-02T05:50:14.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mzEeQ9BA1ATfFhqEzp63a6nQrHY= 2022-09-02T05:50:14.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mzEeQ9BA1ATfFhqEzp63a6nQrHY=', 'Content-Length': '0'} 2022-09-02T05:50:14.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc4be72bc570a33-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:50:14.199 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.199 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.200 DEBUG:boto:Method: GET 2022-09-02T05:50:14.201 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:14.201 DEBUG:boto:Data: 2022-09-02T05:50:14.202 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.203 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.203 DEBUG:boto:Params: {} 2022-09-02T05:50:14.204 DEBUG:boto:Token: None 2022-09-02T05:50:14.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55 2022-09-02T05:50:14.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jTUP5NTypMCCkdx9MRq1kDSkImw= 2022-09-02T05:50:14.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jTUP5NTypMCCkdx9MRq1kDSkImw=', 'Content-Length': '0'} 2022-09-02T05:50:14.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46fea8b0d624729-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.208 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:50:14.209 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:50:14.209 DEBUG:boto:Method: GET 2022-09-02T05:50:14.210 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:50:14.211 DEBUG:boto:Data: 2022-09-02T05:50:14.211 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.212 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.213 DEBUG:boto:Params: {} 2022-09-02T05:50:14.213 DEBUG:boto:Token: None 2022-09-02T05:50:14.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj55 2022-09-02T05:50:14.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jTUP5NTypMCCkdx9MRq1kDSkImw= 2022-09-02T05:50:14.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jTUP5NTypMCCkdx9MRq1kDSkImw=', 'Content-Length': '0'} 2022-09-02T05:50:14.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793fdb4539abb734-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.218 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.219 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.219 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.220 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.220 DEBUG:boto:Method: GET 2022-09-02T05:50:14.221 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.221 DEBUG:boto:Data: 2022-09-02T05:50:14.222 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.223 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.223 DEBUG:boto:Params: {} 2022-09-02T05:50:14.224 DEBUG:boto:Token: None 2022-09-02T05:50:14.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a237f64507353287-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.229 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.229 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.230 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.231 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.231 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.232 DEBUG:boto:Method: GET 2022-09-02T05:50:14.232 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:50:14.233 DEBUG:boto:Data: 2022-09-02T05:50:14.233 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.234 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.235 DEBUG:boto:Params: {} 2022-09-02T05:50:14.235 DEBUG:boto:Token: None 2022-09-02T05:50:14.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e03ccba5f05e4c-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.240 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:14.242 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.242 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.243 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.243 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.244 DEBUG:boto:Method: GET 2022-09-02T05:50:14.244 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.245 DEBUG:boto:Data: 2022-09-02T05:50:14.245 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.246 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.247 DEBUG:boto:Params: {} 2022-09-02T05:50:14.248 DEBUG:boto:Token: None 2022-09-02T05:50:14.248 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.249 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY= 2022-09-02T05:50:14.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY=', 'Content-Length': '0'} 2022-09-02T05:50:14.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a58d10df0a6bb476-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.252 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.253 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.254 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.254 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.255 DEBUG:boto:Method: GET 2022-09-02T05:50:14.255 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.256 DEBUG:boto:Data: 2022-09-02T05:50:14.256 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.257 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.258 DEBUG:boto:Params: {} 2022-09-02T05:50:14.259 DEBUG:boto:Token: None 2022-09-02T05:50:14.259 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY= 2022-09-02T05:50:14.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY=', 'Content-Length': '0'} 2022-09-02T05:50:14.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006294dac12af9ac26-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.263 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.264 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.264 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.265 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.265 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.266 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.266 DEBUG:boto:Data: 2022-09-02T05:50:14.267 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.268 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.268 DEBUG:boto:Params: {} 2022-09-02T05:50:14.269 DEBUG:boto:Token: None 2022-09-02T05:50:14.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IJVQ7uCPuZJZnqJvKw9DmpPZ2h8= 2022-09-02T05:50:14.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IJVQ7uCPuZJZnqJvKw9DmpPZ2h8=', 'Content-Length': '0'} 2022-09-02T05:50:14.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001afcf845402bb1ec-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.273 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.274 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.275 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.275 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.276 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.277 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.278 DEBUG:boto:Data: 2022-09-02T05:50:14.279 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.281 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.282 DEBUG:boto:Params: {} 2022-09-02T05:50:14.283 DEBUG:boto:Token: None 2022-09-02T05:50:14.284 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IJVQ7uCPuZJZnqJvKw9DmpPZ2h8= 2022-09-02T05:50:14.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IJVQ7uCPuZJZnqJvKw9DmpPZ2h8=', 'Content-Length': '0'} 2022-09-02T05:50:14.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646713264499aa36-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:14.291 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.292 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.293 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.294 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.295 DEBUG:boto:Method: GET 2022-09-02T05:50:14.296 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.297 DEBUG:boto:Data: 2022-09-02T05:50:14.298 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.300 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.301 DEBUG:boto:Params: {} 2022-09-02T05:50:14.303 DEBUG:boto:Token: None 2022-09-02T05:50:14.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY= 2022-09-02T05:50:14.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY=', 'Content-Length': '0'} 2022-09-02T05:50:14.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb4c3a16a5f0303-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.311 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.312 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.312 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.313 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.314 DEBUG:boto:Method: GET 2022-09-02T05:50:14.315 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.317 DEBUG:boto:Data: 2022-09-02T05:50:14.318 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.320 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.321 DEBUG:boto:Params: {} 2022-09-02T05:50:14.322 DEBUG:boto:Token: None 2022-09-02T05:50:14.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:50:14.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY= 2022-09-02T05:50:14.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XpT8K+q2R0LHBJhXgqRhL0WIrJY=', 'Content-Length': '0'} 2022-09-02T05:50:14.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018c20d8ff508f6ab-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.330 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.330 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.331 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.331 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.332 DEBUG:boto:Method: GET 2022-09-02T05:50:14.332 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.333 DEBUG:boto:Data: 2022-09-02T05:50:14.333 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.334 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.335 DEBUG:boto:Params: {} 2022-09-02T05:50:14.335 DEBUG:boto:Token: None 2022-09-02T05:50:14.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000052633b789822079-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.341 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.342 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.343 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.343 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.344 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.344 DEBUG:boto:Method: GET 2022-09-02T05:50:14.345 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.345 DEBUG:boto:Data: 2022-09-02T05:50:14.346 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.346 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.347 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.347 DEBUG:boto:Params: {} 2022-09-02T05:50:14.349 DEBUG:boto:Token: None 2022-09-02T05:50:14.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a51a13ee11d3c73-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.354 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.355 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.356 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.356 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.357 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:14.357 DEBUG:boto:Data: 2022-09-02T05:50:14.358 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.359 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.359 DEBUG:boto:Params: {} 2022-09-02T05:50:14.360 DEBUG:boto:Token: None 2022-09-02T05:50:14.360 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57 2022-09-02T05:50:14.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHPJ3CAU1vtmMks9vlgSC2T8nlY= 2022-09-02T05:50:14.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHPJ3CAU1vtmMks9vlgSC2T8nlY=', 'Content-Length': '0'} 2022-09-02T05:50:14.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde34f5ad553e4fb-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.365 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.365 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.366 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.366 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:14.367 DEBUG:boto:Data: 2022-09-02T05:50:14.367 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.368 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.369 DEBUG:boto:Params: {} 2022-09-02T05:50:14.369 DEBUG:boto:Token: None 2022-09-02T05:50:14.370 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57 2022-09-02T05:50:14.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHPJ3CAU1vtmMks9vlgSC2T8nlY= 2022-09-02T05:50:14.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHPJ3CAU1vtmMks9vlgSC2T8nlY=', 'Content-Length': '0'} 2022-09-02T05:50:14.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d98abc6d6089efc8-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:50:14.375 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.376 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.376 DEBUG:boto:Method: GET 2022-09-02T05:50:14.377 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:14.378 DEBUG:boto:Data: 2022-09-02T05:50:14.378 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.380 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.380 DEBUG:boto:Params: {} 2022-09-02T05:50:14.381 DEBUG:boto:Token: None 2022-09-02T05:50:14.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57 2022-09-02T05:50:14.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SLacnD3AbaHg7tA3nPLstJa1/fw= 2022-09-02T05:50:14.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SLacnD3AbaHg7tA3nPLstJa1/fw=', 'Content-Length': '0'} 2022-09-02T05:50:14.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4eed43883ae024-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.386 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:50:14.387 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:50:14.387 DEBUG:boto:Method: GET 2022-09-02T05:50:14.388 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:50:14.389 DEBUG:boto:Data: 2022-09-02T05:50:14.389 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.391 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.391 DEBUG:boto:Params: {} 2022-09-02T05:50:14.392 DEBUG:boto:Token: None 2022-09-02T05:50:14.393 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj57 2022-09-02T05:50:14.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SLacnD3AbaHg7tA3nPLstJa1/fw= 2022-09-02T05:50:14.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SLacnD3AbaHg7tA3nPLstJa1/fw=', 'Content-Length': '0'} 2022-09-02T05:50:14.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eccd515a83fe98b1-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.397 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.398 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.398 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.399 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.400 DEBUG:boto:Method: GET 2022-09-02T05:50:14.400 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.401 DEBUG:boto:Data: 2022-09-02T05:50:14.401 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.403 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.403 DEBUG:boto:Params: {} 2022-09-02T05:50:14.404 DEBUG:boto:Token: None 2022-09-02T05:50:14.405 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af73662dc4a7d474-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.410 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.410 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.411 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.412 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.412 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.413 DEBUG:boto:Method: GET 2022-09-02T05:50:14.413 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:50:14.414 DEBUG:boto:Data: 2022-09-02T05:50:14.415 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.416 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.417 DEBUG:boto:Params: {} 2022-09-02T05:50:14.417 DEBUG:boto:Token: None 2022-09-02T05:50:14.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004614dc38eebcb27-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.423 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:14.424 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.425 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.425 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.426 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.427 DEBUG:boto:Method: GET 2022-09-02T05:50:14.427 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.428 DEBUG:boto:Data: 2022-09-02T05:50:14.429 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.430 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.430 DEBUG:boto:Params: {} 2022-09-02T05:50:14.431 DEBUG:boto:Token: None 2022-09-02T05:50:14.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4= 2022-09-02T05:50:14.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4=', 'Content-Length': '0'} 2022-09-02T05:50:14.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c54d8521b3a24a7-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.436 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.437 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.437 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.438 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.439 DEBUG:boto:Method: GET 2022-09-02T05:50:14.439 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.440 DEBUG:boto:Data: 2022-09-02T05:50:14.441 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.442 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.442 DEBUG:boto:Params: {} 2022-09-02T05:50:14.443 DEBUG:boto:Token: None 2022-09-02T05:50:14.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4= 2022-09-02T05:50:14.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4=', 'Content-Length': '0'} 2022-09-02T05:50:14.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f43e3b0a738bca-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.494 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.495 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.495 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.496 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.497 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.497 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.498 DEBUG:boto:Data: 2022-09-02T05:50:14.499 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.500 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.501 DEBUG:boto:Params: {} 2022-09-02T05:50:14.502 DEBUG:boto:Token: None 2022-09-02T05:50:14.502 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M08HlYyxzlKAvUtuPZcGzO1Ze8s= 2022-09-02T05:50:14.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M08HlYyxzlKAvUtuPZcGzO1Ze8s=', 'Content-Length': '0'} 2022-09-02T05:50:14.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011171b5943ad078d-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.507 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.508 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.509 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.509 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.510 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.511 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.512 DEBUG:boto:Data: 2022-09-02T05:50:14.512 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.514 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.514 DEBUG:boto:Params: {} 2022-09-02T05:50:14.515 DEBUG:boto:Token: None 2022-09-02T05:50:14.515 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M08HlYyxzlKAvUtuPZcGzO1Ze8s= 2022-09-02T05:50:14.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M08HlYyxzlKAvUtuPZcGzO1Ze8s=', 'Content-Length': '0'} 2022-09-02T05:50:14.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000766a6e39b12d4c79-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:14.520 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.521 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.522 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.523 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.523 DEBUG:boto:Method: GET 2022-09-02T05:50:14.524 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.525 DEBUG:boto:Data: 2022-09-02T05:50:14.525 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.526 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.527 DEBUG:boto:Params: {} 2022-09-02T05:50:14.528 DEBUG:boto:Token: None 2022-09-02T05:50:14.529 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4= 2022-09-02T05:50:14.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4=', 'Content-Length': '0'} 2022-09-02T05:50:14.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea2cadf00defd9b6-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.574 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.576 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.577 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.578 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.580 DEBUG:boto:Method: GET 2022-09-02T05:50:14.581 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.582 DEBUG:boto:Data: 2022-09-02T05:50:14.584 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.586 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.588 DEBUG:boto:Params: {} 2022-09-02T05:50:14.589 DEBUG:boto:Token: None 2022-09-02T05:50:14.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:50:14.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4= 2022-09-02T05:50:14.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7h83a+aaI46kfe131hmcNfMh4+4=', 'Content-Length': '0'} 2022-09-02T05:50:14.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d74ec7838d01f53-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.598 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.599 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.600 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.601 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.602 DEBUG:boto:Method: GET 2022-09-02T05:50:14.604 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.605 DEBUG:boto:Data: 2022-09-02T05:50:14.606 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.608 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.609 DEBUG:boto:Params: {} 2022-09-02T05:50:14.611 DEBUG:boto:Token: None 2022-09-02T05:50:14.612 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b64bb88f716b4f1-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.620 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.622 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.623 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.624 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.625 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.627 DEBUG:boto:Method: GET 2022-09-02T05:50:14.628 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.629 DEBUG:boto:Data: 2022-09-02T05:50:14.630 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.632 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.633 DEBUG:boto:Params: {} 2022-09-02T05:50:14.634 DEBUG:boto:Token: None 2022-09-02T05:50:14.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ef407e252a6cc0-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.640 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.641 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.642 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.642 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.643 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:14.644 DEBUG:boto:Data: 2022-09-02T05:50:14.644 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.645 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.646 DEBUG:boto:Params: {} 2022-09-02T05:50:14.647 DEBUG:boto:Token: None 2022-09-02T05:50:14.648 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68 2022-09-02T05:50:14.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k/RFgHxy4FhKYbuS9Mh2Nxfg58o= 2022-09-02T05:50:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k/RFgHxy4FhKYbuS9Mh2Nxfg58o=', 'Content-Length': '0'} 2022-09-02T05:50:14.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f33fbe5deee469c-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.652 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.653 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.654 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.654 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:14.655 DEBUG:boto:Data: 2022-09-02T05:50:14.656 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.657 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.658 DEBUG:boto:Params: {} 2022-09-02T05:50:14.658 DEBUG:boto:Token: None 2022-09-02T05:50:14.659 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68 2022-09-02T05:50:14.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k/RFgHxy4FhKYbuS9Mh2Nxfg58o= 2022-09-02T05:50:14.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k/RFgHxy4FhKYbuS9Mh2Nxfg58o=', 'Content-Length': '0'} 2022-09-02T05:50:14.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990168c4f307806e-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:50:14.664 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.665 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.666 DEBUG:boto:Method: GET 2022-09-02T05:50:14.666 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:14.667 DEBUG:boto:Data: 2022-09-02T05:50:14.668 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.670 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.671 DEBUG:boto:Params: {} 2022-09-02T05:50:14.672 DEBUG:boto:Token: None 2022-09-02T05:50:14.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68 2022-09-02T05:50:14.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xcP8aLvd/Sawl2dNTDLM91T+sHM= 2022-09-02T05:50:14.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xcP8aLvd/Sawl2dNTDLM91T+sHM=', 'Content-Length': '0'} 2022-09-02T05:50:14.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006405186784f7b341-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.677 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:50:14.678 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:50:14.678 DEBUG:boto:Method: GET 2022-09-02T05:50:14.679 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:50:14.679 DEBUG:boto:Data: 2022-09-02T05:50:14.680 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.681 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.682 DEBUG:boto:Params: {} 2022-09-02T05:50:14.682 DEBUG:boto:Token: None 2022-09-02T05:50:14.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj68 2022-09-02T05:50:14.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xcP8aLvd/Sawl2dNTDLM91T+sHM= 2022-09-02T05:50:14.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xcP8aLvd/Sawl2dNTDLM91T+sHM=', 'Content-Length': '0'} 2022-09-02T05:50:14.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5ceb48554476f3-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.687 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.688 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.688 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.689 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.689 DEBUG:boto:Method: GET 2022-09-02T05:50:14.690 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.690 DEBUG:boto:Data: 2022-09-02T05:50:14.691 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.692 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.693 DEBUG:boto:Params: {} 2022-09-02T05:50:14.693 DEBUG:boto:Token: None 2022-09-02T05:50:14.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a4c793c5c26b644-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.698 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.699 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.700 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.700 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.701 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.701 DEBUG:boto:Method: GET 2022-09-02T05:50:14.702 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:50:14.703 DEBUG:boto:Data: 2022-09-02T05:50:14.703 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.704 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.704 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.705 DEBUG:boto:Params: {} 2022-09-02T05:50:14.705 DEBUG:boto:Token: None 2022-09-02T05:50:14.706 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de9249f8b22741c4-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.750 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:14.752 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.752 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.753 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.753 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.754 DEBUG:boto:Method: GET 2022-09-02T05:50:14.754 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.755 DEBUG:boto:Data: 2022-09-02T05:50:14.756 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.757 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.757 DEBUG:boto:Params: {} 2022-09-02T05:50:14.758 DEBUG:boto:Token: None 2022-09-02T05:50:14.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w= 2022-09-02T05:50:14.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w=', 'Content-Length': '0'} 2022-09-02T05:50:14.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000467d3c1e75f84b37-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.763 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.763 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.764 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.764 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.765 DEBUG:boto:Method: GET 2022-09-02T05:50:14.766 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.766 DEBUG:boto:Data: 2022-09-02T05:50:14.767 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.768 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.768 DEBUG:boto:Params: {} 2022-09-02T05:50:14.769 DEBUG:boto:Token: None 2022-09-02T05:50:14.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w= 2022-09-02T05:50:14.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w=', 'Content-Length': '0'} 2022-09-02T05:50:14.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100ee92bc5a13cd7-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.774 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.775 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.776 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.776 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.777 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.778 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.778 DEBUG:boto:Data: 2022-09-02T05:50:14.779 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.780 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.780 DEBUG:boto:Params: {} 2022-09-02T05:50:14.781 DEBUG:boto:Token: None 2022-09-02T05:50:14.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CtdrYMN/YE8BLKwKAytJ6Jap+Qs= 2022-09-02T05:50:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CtdrYMN/YE8BLKwKAytJ6Jap+Qs=', 'Content-Length': '0'} 2022-09-02T05:50:14.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006000daf03049c417-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.785 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.786 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.787 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.788 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.788 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.789 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.789 DEBUG:boto:Data: 2022-09-02T05:50:14.790 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.791 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.791 DEBUG:boto:Params: {} 2022-09-02T05:50:14.792 DEBUG:boto:Token: None 2022-09-02T05:50:14.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CtdrYMN/YE8BLKwKAytJ6Jap+Qs= 2022-09-02T05:50:14.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CtdrYMN/YE8BLKwKAytJ6Jap+Qs=', 'Content-Length': '0'} 2022-09-02T05:50:14.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072abc1498990399c-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:14.797 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.797 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.798 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.798 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.799 DEBUG:boto:Method: GET 2022-09-02T05:50:14.799 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.800 DEBUG:boto:Data: 2022-09-02T05:50:14.800 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.801 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.802 DEBUG:boto:Params: {} 2022-09-02T05:50:14.803 DEBUG:boto:Token: None 2022-09-02T05:50:14.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w= 2022-09-02T05:50:14.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w=', 'Content-Length': '0'} 2022-09-02T05:50:14.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c60d8ccc2f0f79c-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.808 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.808 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.809 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.809 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.810 DEBUG:boto:Method: GET 2022-09-02T05:50:14.810 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.811 DEBUG:boto:Data: 2022-09-02T05:50:14.811 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.812 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.813 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.813 DEBUG:boto:Params: {} 2022-09-02T05:50:14.814 DEBUG:boto:Token: None 2022-09-02T05:50:14.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:50:14.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w= 2022-09-02T05:50:14.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xg9zDJIcC28/BQ3/QPVfjFUco6w=', 'Content-Length': '0'} 2022-09-02T05:50:14.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5c77d2ce0f0f8a-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.819 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.819 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.820 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.821 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.821 DEBUG:boto:Method: GET 2022-09-02T05:50:14.822 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.822 DEBUG:boto:Data: 2022-09-02T05:50:14.823 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.824 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.824 DEBUG:boto:Params: {} 2022-09-02T05:50:14.825 DEBUG:boto:Token: None 2022-09-02T05:50:14.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006bfd70d1e801954-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.830 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.831 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.831 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.832 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.832 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.833 DEBUG:boto:Method: GET 2022-09-02T05:50:14.833 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.834 DEBUG:boto:Data: 2022-09-02T05:50:14.834 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.835 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.836 DEBUG:boto:Params: {} 2022-09-02T05:50:14.836 DEBUG:boto:Token: None 2022-09-02T05:50:14.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037bf44737d51a05d-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.842 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.842 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.843 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.843 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.844 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:14.844 DEBUG:boto:Data: 2022-09-02T05:50:14.845 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.846 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.846 DEBUG:boto:Params: {} 2022-09-02T05:50:14.847 DEBUG:boto:Token: None 2022-09-02T05:50:14.847 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76 2022-09-02T05:50:14.848 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NHz+eWsGMxD5YDsU0FnsUQr9NXE= 2022-09-02T05:50:14.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NHz+eWsGMxD5YDsU0FnsUQr9NXE=', 'Content-Length': '0'} 2022-09-02T05:50:14.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f502bb7bb8be734c-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.852 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.852 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.853 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.853 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:14.854 DEBUG:boto:Data: 2022-09-02T05:50:14.854 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.855 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.856 DEBUG:boto:Params: {} 2022-09-02T05:50:14.857 DEBUG:boto:Token: None 2022-09-02T05:50:14.858 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76 2022-09-02T05:50:14.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NHz+eWsGMxD5YDsU0FnsUQr9NXE= 2022-09-02T05:50:14.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NHz+eWsGMxD5YDsU0FnsUQr9NXE=', 'Content-Length': '0'} 2022-09-02T05:50:14.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e143af88c07de13-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:50:14.862 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.863 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.863 DEBUG:boto:Method: GET 2022-09-02T05:50:14.864 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:14.864 DEBUG:boto:Data: 2022-09-02T05:50:14.865 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.866 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.866 DEBUG:boto:Params: {} 2022-09-02T05:50:14.867 DEBUG:boto:Token: None 2022-09-02T05:50:14.868 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76 2022-09-02T05:50:14.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K9X4GXx2E4Metwqpki+XAvV8Mf0= 2022-09-02T05:50:14.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K9X4GXx2E4Metwqpki+XAvV8Mf0=', 'Content-Length': '0'} 2022-09-02T05:50:14.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ca77d0c9d74ccf-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.914 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:50:14.915 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:50:14.916 DEBUG:boto:Method: GET 2022-09-02T05:50:14.916 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:50:14.917 DEBUG:boto:Data: 2022-09-02T05:50:14.918 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.918 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.919 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.920 DEBUG:boto:Params: {} 2022-09-02T05:50:14.921 DEBUG:boto:Token: None 2022-09-02T05:50:14.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj76 2022-09-02T05:50:14.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K9X4GXx2E4Metwqpki+XAvV8Mf0= 2022-09-02T05:50:14.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K9X4GXx2E4Metwqpki+XAvV8Mf0=', 'Content-Length': '0'} 2022-09-02T05:50:14.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42c4f01c13fa294-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.926 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.928 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.928 DEBUG:boto:Method: GET 2022-09-02T05:50:14.929 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.929 DEBUG:boto:Data: 2022-09-02T05:50:14.930 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.931 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.931 DEBUG:boto:Params: {} 2022-09-02T05:50:14.932 DEBUG:boto:Token: None 2022-09-02T05:50:14.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028c0fdd189ac4028-0063119996-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.937 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.938 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:14.939 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:14.939 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.940 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.940 DEBUG:boto:Method: GET 2022-09-02T05:50:14.941 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:50:14.941 DEBUG:boto:Data: 2022-09-02T05:50:14.942 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.943 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.944 DEBUG:boto:Params: {} 2022-09-02T05:50:14.945 DEBUG:boto:Token: None 2022-09-02T05:50:14.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/ 2022-09-02T05:50:14.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno= 2022-09-02T05:50:14.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0bl5ZrvuSEqB+vMpc2Op2CELno=', 'Content-Length': '0'} 2022-09-02T05:50:14.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356277b351b82e71-0063119996-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.950 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:14.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:14.952 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:14.953 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:14.953 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.954 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.955 DEBUG:boto:Method: GET 2022-09-02T05:50:14.956 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.956 DEBUG:boto:Data: 2022-09-02T05:50:14.957 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.958 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.959 DEBUG:boto:Params: {} 2022-09-02T05:50:14.959 DEBUG:boto:Token: None 2022-09-02T05:50:14.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.961 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IUn1Nj4MXznM8DVBXfe3x2HYC9w= 2022-09-02T05:50:14.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IUn1Nj4MXznM8DVBXfe3x2HYC9w=', 'Content-Length': '0'} 2022-09-02T05:50:14.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb4c03db29f58f52-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.964 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:14.965 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:14.966 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.966 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.967 DEBUG:boto:Method: GET 2022-09-02T05:50:14.967 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.968 DEBUG:boto:Data: 2022-09-02T05:50:14.968 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.970 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.971 DEBUG:boto:Params: {} 2022-09-02T05:50:14.971 DEBUG:boto:Token: None 2022-09-02T05:50:14.972 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IUn1Nj4MXznM8DVBXfe3x2HYC9w= 2022-09-02T05:50:14.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IUn1Nj4MXznM8DVBXfe3x2HYC9w=', 'Content-Length': '0'} 2022-09-02T05:50:14.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001798d22205b47dae-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.976 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:14.976 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:14.977 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.977 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.978 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.979 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.979 DEBUG:boto:Data: 2022-09-02T05:50:14.980 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:14.981 DEBUG:boto:Port: 8001 2022-09-02T05:50:14.981 DEBUG:boto:Params: {} 2022-09-02T05:50:14.982 DEBUG:boto:Token: None 2022-09-02T05:50:14.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8QlKb4cCo7L07cBlcNN6HatyvM= 2022-09-02T05:50:14.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8QlKb4cCo7L07cBlcNN6HatyvM=', 'Content-Length': '0'} 2022-09-02T05:50:14.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f12a0ccc3ab7e079-0063119996-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.987 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:14.988 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:14.988 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.989 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.989 DEBUG:boto:Method: HEAD 2022-09-02T05:50:14.990 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.990 DEBUG:boto:Data: 2022-09-02T05:50:14.991 DEBUG:boto:Headers: {} 2022-09-02T05:50:14.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:14.992 DEBUG:boto:Port: 8000 2022-09-02T05:50:14.992 DEBUG:boto:Params: {} 2022-09-02T05:50:14.993 DEBUG:boto:Token: None 2022-09-02T05:50:14.994 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:14 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:14.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8QlKb4cCo7L07cBlcNN6HatyvM= 2022-09-02T05:50:14.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8QlKb4cCo7L07cBlcNN6HatyvM=', 'Content-Length': '0'} 2022-09-02T05:50:14.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076435648249f8a5c-0063119996-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:14.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:14.999 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:14.999 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:15.000 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.000 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.001 DEBUG:boto:Method: GET 2022-09-02T05:50:15.001 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.002 DEBUG:boto:Data: 2022-09-02T05:50:15.002 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.003 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.004 DEBUG:boto:Params: {} 2022-09-02T05:50:15.005 DEBUG:boto:Token: None 2022-09-02T05:50:15.006 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yUEosoYtZ/Rho4aOpJZHsZj4obg= 2022-09-02T05:50:15.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yUEosoYtZ/Rho4aOpJZHsZj4obg=', 'Content-Length': '0'} 2022-09-02T05:50:15.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008988d1d0c4eebd9-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.010 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:15.011 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:15.011 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.012 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.012 DEBUG:boto:Method: GET 2022-09-02T05:50:15.013 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.014 DEBUG:boto:Data: 2022-09-02T05:50:15.014 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.016 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.016 DEBUG:boto:Params: {} 2022-09-02T05:50:15.017 DEBUG:boto:Token: None 2022-09-02T05:50:15.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:50:15.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yUEosoYtZ/Rho4aOpJZHsZj4obg= 2022-09-02T05:50:15.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yUEosoYtZ/Rho4aOpJZHsZj4obg=', 'Content-Length': '0'} 2022-09-02T05:50:15.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fe0dd84485c3ebe-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.022 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.023 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.024 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.024 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.025 DEBUG:boto:Method: GET 2022-09-02T05:50:15.025 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.026 DEBUG:boto:Data: 2022-09-02T05:50:15.026 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.027 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.028 DEBUG:boto:Params: {} 2022-09-02T05:50:15.028 DEBUG:boto:Token: None 2022-09-02T05:50:15.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9d75e44cd958c9-0063119997-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.034 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.035 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.036 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.036 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.037 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.037 DEBUG:boto:Method: GET 2022-09-02T05:50:15.038 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.039 DEBUG:boto:Data: 2022-09-02T05:50:15.039 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.041 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.041 DEBUG:boto:Params: {} 2022-09-02T05:50:15.042 DEBUG:boto:Token: None 2022-09-02T05:50:15.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246c1d99da66c386-0063119997-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.048 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.048 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:15.049 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:15.049 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.050 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:15.051 DEBUG:boto:Data: 2022-09-02T05:50:15.051 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.052 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.053 DEBUG:boto:Params: {} 2022-09-02T05:50:15.053 DEBUG:boto:Token: None 2022-09-02T05:50:15.054 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj85 2022-09-02T05:50:15.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oUD5BFHTdkO70kanVHAx9lRzN1k= 2022-09-02T05:50:15.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oUD5BFHTdkO70kanVHAx9lRzN1k=', 'Content-Length': '0'} 2022-09-02T05:50:15.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede14f118e9a95b2-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.058 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:15.059 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:15.059 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.060 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:15.060 DEBUG:boto:Data: 2022-09-02T05:50:15.061 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.062 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.062 DEBUG:boto:Params: {} 2022-09-02T05:50:15.063 DEBUG:boto:Token: None 2022-09-02T05:50:15.063 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj85 2022-09-02T05:50:15.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oUD5BFHTdkO70kanVHAx9lRzN1k= 2022-09-02T05:50:15.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oUD5BFHTdkO70kanVHAx9lRzN1k=', 'Content-Length': '0'} 2022-09-02T05:50:15.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dadda91ab2ed89-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:50:15.068 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:15.069 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:15.069 DEBUG:boto:Method: GET 2022-09-02T05:50:15.070 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:15.070 DEBUG:boto:Data: 2022-09-02T05:50:15.071 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.072 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.073 DEBUG:boto:Params: {} 2022-09-02T05:50:15.073 DEBUG:boto:Token: None 2022-09-02T05:50:15.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj85 2022-09-02T05:50:15.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AKFLh2bj+UOyhmL0D04uyPmdLUg= 2022-09-02T05:50:15.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AKFLh2bj+UOyhmL0D04uyPmdLUg=', 'Content-Length': '0'} 2022-09-02T05:50:15.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022be91638f7c9832-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.078 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:50:15.079 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:50:15.080 DEBUG:boto:Method: GET 2022-09-02T05:50:15.080 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:50:15.081 DEBUG:boto:Data: 2022-09-02T05:50:15.081 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.083 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.083 DEBUG:boto:Params: {} 2022-09-02T05:50:15.084 DEBUG:boto:Token: None 2022-09-02T05:50:15.084 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj85 2022-09-02T05:50:15.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AKFLh2bj+UOyhmL0D04uyPmdLUg= 2022-09-02T05:50:15.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AKFLh2bj+UOyhmL0D04uyPmdLUg=', 'Content-Length': '0'} 2022-09-02T05:50:15.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698e19733a1ae8c1-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.134 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.135 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.135 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.136 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.136 DEBUG:boto:Method: GET 2022-09-02T05:50:15.137 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.138 DEBUG:boto:Data: 2022-09-02T05:50:15.138 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.139 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.140 DEBUG:boto:Params: {} 2022-09-02T05:50:15.141 DEBUG:boto:Token: None 2022-09-02T05:50:15.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a8926be5a9aa36-0063119997-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.146 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.147 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.148 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.148 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.149 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.149 DEBUG:boto:Method: GET 2022-09-02T05:50:15.150 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:50:15.151 DEBUG:boto:Data: 2022-09-02T05:50:15.151 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.152 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.153 DEBUG:boto:Params: {} 2022-09-02T05:50:15.153 DEBUG:boto:Token: None 2022-09-02T05:50:15.154 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.155 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f95d20aebcc12e1-0063119997-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.159 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:15.160 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.161 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.161 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.162 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.162 DEBUG:boto:Method: GET 2022-09-02T05:50:15.163 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.163 DEBUG:boto:Data: 2022-09-02T05:50:15.164 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.173 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.173 DEBUG:boto:Params: {} 2022-09-02T05:50:15.174 DEBUG:boto:Token: None 2022-09-02T05:50:15.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E= 2022-09-02T05:50:15.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E=', 'Content-Length': '0'} 2022-09-02T05:50:15.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f983589e5667021-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.179 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.179 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.180 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.180 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.181 DEBUG:boto:Method: GET 2022-09-02T05:50:15.182 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.182 DEBUG:boto:Data: 2022-09-02T05:50:15.183 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.184 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.184 DEBUG:boto:Params: {} 2022-09-02T05:50:15.185 DEBUG:boto:Token: None 2022-09-02T05:50:15.186 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E= 2022-09-02T05:50:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E=', 'Content-Length': '0'} 2022-09-02T05:50:15.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe84c8c5852caf6-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.190 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.191 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.191 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.192 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.192 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.193 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.193 DEBUG:boto:Data: 2022-09-02T05:50:15.194 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.195 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.195 DEBUG:boto:Params: {} 2022-09-02T05:50:15.196 DEBUG:boto:Token: None 2022-09-02T05:50:15.197 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zEUsPgDxBbho1TvyiQdg1GOzpDE= 2022-09-02T05:50:15.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zEUsPgDxBbho1TvyiQdg1GOzpDE=', 'Content-Length': '0'} 2022-09-02T05:50:15.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090195aeda0de3080-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.201 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.201 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.202 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.202 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.203 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.203 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.204 DEBUG:boto:Data: 2022-09-02T05:50:15.204 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.205 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.206 DEBUG:boto:Params: {} 2022-09-02T05:50:15.206 DEBUG:boto:Token: None 2022-09-02T05:50:15.207 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zEUsPgDxBbho1TvyiQdg1GOzpDE= 2022-09-02T05:50:15.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zEUsPgDxBbho1TvyiQdg1GOzpDE=', 'Content-Length': '0'} 2022-09-02T05:50:15.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fea6ef9488de576a-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:15.212 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.213 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.213 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.214 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.214 DEBUG:boto:Method: GET 2022-09-02T05:50:15.215 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.215 DEBUG:boto:Data: 2022-09-02T05:50:15.216 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.217 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.217 DEBUG:boto:Params: {} 2022-09-02T05:50:15.218 DEBUG:boto:Token: None 2022-09-02T05:50:15.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E= 2022-09-02T05:50:15.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E=', 'Content-Length': '0'} 2022-09-02T05:50:15.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002207d94eda78a264-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.270 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.271 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.271 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.272 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.273 DEBUG:boto:Method: GET 2022-09-02T05:50:15.273 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.274 DEBUG:boto:Data: 2022-09-02T05:50:15.275 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.276 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.276 DEBUG:boto:Params: {} 2022-09-02T05:50:15.277 DEBUG:boto:Token: None 2022-09-02T05:50:15.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:50:15.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E= 2022-09-02T05:50:15.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TMd5lpgmdbKAc1KsZvrfUWdXe8E=', 'Content-Length': '0'} 2022-09-02T05:50:15.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e5cc3e3bf0eacb3-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.282 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.283 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.283 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.284 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.284 DEBUG:boto:Method: GET 2022-09-02T05:50:15.285 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.285 DEBUG:boto:Data: 2022-09-02T05:50:15.286 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.287 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.288 DEBUG:boto:Params: {} 2022-09-02T05:50:15.288 DEBUG:boto:Token: None 2022-09-02T05:50:15.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090810c95cec0c520-0063119997-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.294 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.295 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.295 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.296 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.296 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.297 DEBUG:boto:Method: GET 2022-09-02T05:50:15.298 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.298 DEBUG:boto:Data: 2022-09-02T05:50:15.299 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.300 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.300 DEBUG:boto:Params: {} 2022-09-02T05:50:15.301 DEBUG:boto:Token: None 2022-09-02T05:50:15.301 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.302 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fcb3c5f10a568e7-0063119997-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.306 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.307 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.307 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.308 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.308 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:15.309 DEBUG:boto:Data: 2022-09-02T05:50:15.309 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.310 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.311 DEBUG:boto:Params: {} 2022-09-02T05:50:15.312 DEBUG:boto:Token: None 2022-09-02T05:50:15.312 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95 2022-09-02T05:50:15.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3a5EH0dJeXycpDbbUtKSCK473fI= 2022-09-02T05:50:15.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3a5EH0dJeXycpDbbUtKSCK473fI=', 'Content-Length': '0'} 2022-09-02T05:50:15.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a069dce60aa63e37-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.316 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.317 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.317 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.318 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:15.318 DEBUG:boto:Data: 2022-09-02T05:50:15.319 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.320 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.320 DEBUG:boto:Params: {} 2022-09-02T05:50:15.321 DEBUG:boto:Token: None 2022-09-02T05:50:15.321 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95 2022-09-02T05:50:15.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3a5EH0dJeXycpDbbUtKSCK473fI= 2022-09-02T05:50:15.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3a5EH0dJeXycpDbbUtKSCK473fI=', 'Content-Length': '0'} 2022-09-02T05:50:15.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007500c761b8428896-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:50:15.326 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.326 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.327 DEBUG:boto:Method: GET 2022-09-02T05:50:15.327 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:15.328 DEBUG:boto:Data: 2022-09-02T05:50:15.328 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.330 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.330 DEBUG:boto:Params: {} 2022-09-02T05:50:15.331 DEBUG:boto:Token: None 2022-09-02T05:50:15.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95 2022-09-02T05:50:15.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QDmIlisUKiEZgZQr+gGU5luhhms= 2022-09-02T05:50:15.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QDmIlisUKiEZgZQr+gGU5luhhms=', 'Content-Length': '0'} 2022-09-02T05:50:15.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6165f147521b0fa-0063119997-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.336 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:50:15.336 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:50:15.337 DEBUG:boto:Method: GET 2022-09-02T05:50:15.337 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:50:15.338 DEBUG:boto:Data: 2022-09-02T05:50:15.339 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.340 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.341 DEBUG:boto:Params: {} 2022-09-02T05:50:15.341 DEBUG:boto:Token: None 2022-09-02T05:50:15.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj95 2022-09-02T05:50:15.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QDmIlisUKiEZgZQr+gGU5luhhms= 2022-09-02T05:50:15.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QDmIlisUKiEZgZQr+gGU5luhhms=', 'Content-Length': '0'} 2022-09-02T05:50:15.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e21507ab1d047277-0063119997-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.346 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.347 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.347 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.348 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.349 DEBUG:boto:Method: GET 2022-09-02T05:50:15.349 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.350 DEBUG:boto:Data: 2022-09-02T05:50:15.350 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:50:15.351 DEBUG:boto:Port: 8001 2022-09-02T05:50:15.352 DEBUG:boto:Params: {} 2022-09-02T05:50:15.353 DEBUG:boto:Token: None 2022-09-02T05:50:15.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb914c329201f3ec-0063119997-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.358 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.359 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.360 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.360 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.361 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.361 DEBUG:boto:Method: GET 2022-09-02T05:50:15.362 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:50:15.362 DEBUG:boto:Data: 2022-09-02T05:50:15.363 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:50:15.364 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.365 DEBUG:boto:Params: {} 2022-09-02T05:50:15.365 DEBUG:boto:Token: None 2022-09-02T05:50:15.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA= 2022-09-02T05:50:15.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shrkICatsSb38SBHEwmjZ83iTRA=', 'Content-Length': '0'} 2022-09-02T05:50:15.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1ca61c5d48b06bb-0063119997-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.372 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:50:15.374 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:50:15.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:50:15.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:50:15.377 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:50:15.436 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:50:15.437 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:50:15.498 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:50:15.498 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-74 2022-09-02T05:50:15.505 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2022-09-02T05:50:15.567 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.568 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.568 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.569 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:15.569 DEBUG:boto:Data: 2022-09-02T05:50:15.570 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.571 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.572 DEBUG:boto:Params: {} 2022-09-02T05:50:15.572 DEBUG:boto:Token: None 2022-09-02T05:50:15.573 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY= 2022-09-02T05:50:15.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY=', 'Content-Length': '0'} 2022-09-02T05:50:15.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d11cd5d1b3c36a0a-0063119997-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.579 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:50:15.579 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:50:15.580 DEBUG:boto:Method: PUT 2022-09-02T05:50:15.580 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:50:15.581 DEBUG:boto:Data: 2022-09-02T05:50:15.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:50:15.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.582 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.583 DEBUG:boto:Params: {} 2022-09-02T05:50:15.584 DEBUG:boto:Token: None 2022-09-02T05:50:15.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj23 2022-09-02T05:50:15.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xJzyBOJxYNy49D6SkndFAIG+pok= 2022-09-02T05:50:15.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xJzyBOJxYNy49D6SkndFAIG+pok='} 2022-09-02T05:50:15.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000aa7199e966a533-0063119997-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.592 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.593 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.593 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.594 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:15.595 DEBUG:boto:Data: 2022-09-02T05:50:15.595 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.596 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.597 DEBUG:boto:Params: {} 2022-09-02T05:50:15.597 DEBUG:boto:Token: None 2022-09-02T05:50:15.598 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY= 2022-09-02T05:50:15.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY=', 'Content-Length': '0'} 2022-09-02T05:50:15.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fcc160a80481d93-0063119997-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.603 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:50:15.604 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:50:15.605 DEBUG:boto:Method: PUT 2022-09-02T05:50:15.605 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:50:15.606 DEBUG:boto:Data: 2022-09-02T05:50:15.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:50:15.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.607 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.608 DEBUG:boto:Params: {} 2022-09-02T05:50:15.608 DEBUG:boto:Token: None 2022-09-02T05:50:15.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj46 2022-09-02T05:50:15.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QNgoj0iWr5y7ajYv818PPevP1Rs= 2022-09-02T05:50:15.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QNgoj0iWr5y7ajYv818PPevP1Rs='} 2022-09-02T05:50:15.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a285d16474d766cc-0063119997-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.617 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.617 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.618 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.618 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:15.619 DEBUG:boto:Data: 2022-09-02T05:50:15.619 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.621 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.621 DEBUG:boto:Params: {} 2022-09-02T05:50:15.622 DEBUG:boto:Token: None 2022-09-02T05:50:15.622 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY= 2022-09-02T05:50:15.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY=', 'Content-Length': '0'} 2022-09-02T05:50:15.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9fd9a28e9344668-0063119997-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.627 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:50:15.627 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:50:15.628 DEBUG:boto:Method: PUT 2022-09-02T05:50:15.629 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:50:15.629 DEBUG:boto:Data: 2022-09-02T05:50:15.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:50:15.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.631 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.631 DEBUG:boto:Params: {} 2022-09-02T05:50:15.632 DEBUG:boto:Token: None 2022-09-02T05:50:15.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj69 2022-09-02T05:50:15.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BE8UVhCWYPsRjGVWqDW9ET1+DNg= 2022-09-02T05:50:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BE8UVhCWYPsRjGVWqDW9ET1+DNg='} 2022-09-02T05:50:15.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095632540eabb6c90-0063119997-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.679 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.680 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.681 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.681 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:15.682 DEBUG:boto:Data: 2022-09-02T05:50:15.682 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.683 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.684 DEBUG:boto:Params: {} 2022-09-02T05:50:15.684 DEBUG:boto:Token: None 2022-09-02T05:50:15.685 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.686 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY= 2022-09-02T05:50:15.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY=', 'Content-Length': '0'} 2022-09-02T05:50:15.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4e912cd500202c8-0063119997-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.690 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:50:15.691 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:50:15.691 DEBUG:boto:Method: PUT 2022-09-02T05:50:15.692 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:50:15.692 DEBUG:boto:Data: 2022-09-02T05:50:15.693 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:50:15.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.694 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.694 DEBUG:boto:Params: {} 2022-09-02T05:50:15.695 DEBUG:boto:Token: None 2022-09-02T05:50:15.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj92 2022-09-02T05:50:15.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6VWbQiYbd9vxkwZWeA9XY/hbVuQ= 2022-09-02T05:50:15.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6VWbQiYbd9vxkwZWeA9XY/hbVuQ='} 2022-09-02T05:50:15.704 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073e011a5a7ce8970-0063119997-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.705 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:50:15.706 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:50:15.706 DEBUG:boto:Method: HEAD 2022-09-02T05:50:15.707 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:50:15.707 DEBUG:boto:Data: 2022-09-02T05:50:15.708 DEBUG:boto:Headers: {} 2022-09-02T05:50:15.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.709 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.709 DEBUG:boto:Params: {} 2022-09-02T05:50:15.710 DEBUG:boto:Token: None 2022-09-02T05:50:15.711 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/ 2022-09-02T05:50:15.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY= 2022-09-02T05:50:15.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mISgfYQK5PkwGLRM7luzt4sozhY=', 'Content-Length': '0'} 2022-09-02T05:50:15.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fa71f058b3126fb-0063119997-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.715 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:50:15.716 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:50:15.716 DEBUG:boto:Method: PUT 2022-09-02T05:50:15.717 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:50:15.717 DEBUG:boto:Data: 2022-09-02T05:50:15.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:50:15.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:50:15.719 DEBUG:boto:Port: 8000 2022-09-02T05:50:15.720 DEBUG:boto:Params: {} 2022-09-02T05:50:15.720 DEBUG:boto:Token: None 2022-09-02T05:50:15.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:50:15 GMT /xljbkf-74/obj115 2022-09-02T05:50:15.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9pvqcYVE2k/ZJIUIRB24OCCwhH8= 2022-09-02T05:50:15.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:50:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9pvqcYVE2k/ZJIUIRB24OCCwhH8='} 2022-09-02T05:50:15.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000221e67ecebf8b627-0063119997-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:50:15.772 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:50:15.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:15.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:50:15.774 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:50:15.841 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:50:15.842 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:50:15.902 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:15.902+0000 7f920d3b8600 1 waiting to reach latest gen 3, on 2.. 2022-09-02T05:50:20.902 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:20.902+0000 7f920d3b8600 1 waiting to reach latest gen 3, on 2.. 2022-09-02T05:50:25.903 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:25.903+0000 7f920d3b8600 1 waiting to reach latest gen 3, on 2.. 2022-09-02T05:50:30.906 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:30.904+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:50:30.906 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:50:30.907 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:50:35.908 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:35.906+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:50:35.909 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:50:35.909 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:50:40.910 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:40.908+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:50:40.910 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:50:40.910 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:50:45.911 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:45.910+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:50:45.912 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:50:45.912 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:50:50.914 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:50.913+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:50:50.915 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:50:50.915 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:50:55.916 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:50:55.915+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:50:55.916 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:50:55.917 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:00.918 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:00.917+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:00.918 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:00.919 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:05.920 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:05.919+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:05.920 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:05.921 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:10.922 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:10.921+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:10.922 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:10.923 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:15.924 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:15.923+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:15.924 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:15.925 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:20.926 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:20.926+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:20.926 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:20.927 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:25.928 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:25.928+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:25.929 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , , , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:25.929 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:30.930 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:30.930+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:30.931 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:30.931 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:35.932 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:35.931+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:35.932 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:35.933 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:40.934 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:40.933+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:40.935 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:40.935 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:45.936 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:45.935+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:45.937 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:45.938 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:50.938 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:50.937+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:50.938 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:50.939 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:51:55.940 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:51:55.939+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:51:55.941 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:51:55.941 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:00.942 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:00.941+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:00.943 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:52:00.943 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:05.944 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:05.943+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:05.945 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:52:05.946 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:10.946 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:10.945+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:10.947 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:52:10.947 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:15.948 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:15.947+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:15.949 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:52:15.949 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:20.950 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:20.949+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:20.951 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:52:20.951 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:25.952 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:25.951+0000 7f920d3b8600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:25.953 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , , , , , ] 2022-09-02T05:52:25.953 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:30.954 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:30.954+0000 7f920d3b8600 1 bucket sync caught up with source: 2022-09-02T05:52:30.955 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:30.955 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:30.956 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:30.954+0000 7f920d3b8600 0 bucket checkpoint complete 2022-09-02T05:52:30.980 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:52:30.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:52:30.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:52:30.983 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:52:31.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:52:31.049 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:52:31.094 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:52:31.093+0000 7fe669ee0600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:31.095 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.54.5, , , , , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:31.095 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:36.096 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:52:36.095+0000 7fe669ee0600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:36.097 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.54.5, , , , , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:36.098 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:41.098 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:52:41.097+0000 7fe669ee0600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:41.098 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.54.5, , , , , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:41.099 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:52:46.099+0000 7fe669ee0600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:46.101 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.54.5, , , , , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:46.101 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:51.103 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:52:51.101+0000 7fe669ee0600 1 bucket sync caught up with source: 2022-09-02T05:52:51.103 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:51.104 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:51.104 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:52:51.101+0000 7fe669ee0600 0 bucket checkpoint complete 2022-09-02T05:52:51.122 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:52:51.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:52:51.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:52:51.124 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:52:51.190 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:52:51.190 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:52:51.240 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:51.239+0000 7fdc1385a600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:51.240 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , , , 00000000001.57.5, , , , ] 2022-09-02T05:52:51.241 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:52:56.242 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:52:56.241+0000 7fdc1385a600 1 waiting for incremental sync to catch up: 2022-09-02T05:52:56.243 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , , , 00000000001.57.5, , , , ] 2022-09-02T05:52:56.243 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:53:01.244 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:01.243+0000 7fdc1385a600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:01.245 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , , , 00000000001.57.5, , , , ] 2022-09-02T05:53:01.245 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:53:06.246 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:06.245+0000 7fdc1385a600 1 bucket sync caught up with source: 2022-09-02T05:53:06.247 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:53:06.248 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , 00000000001.54.5, , , 00000000002.37.5, , 00000000002.32.5, , , , , , 00000000001.55.5, , 00000000001.57.5, , , , ] 2022-09-02T05:53:06.248 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:06.245+0000 7fdc1385a600 0 bucket checkpoint complete 2022-09-02T05:53:06.267 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:53:06.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.269 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:53:06.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:53:06.334 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:53:06.379 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:53:06.378+0000 7fdedef70600 1 bucket sync caught up with source: 2022-09-02T05:53:06.379 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000002.591.5, 00000000002.412.5, , , 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:53:06.380 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000002.591.5, 00000000002.412.5, , , 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:53:06.380 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:53:06.378+0000 7fdedef70600 0 bucket checkpoint complete 2022-09-02T05:53:06.393 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:53:06.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.395 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:53:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:53:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:53:06.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:53:06.489+0000 7f2f412d1600 1 bucket sync caught up with source: 2022-09-02T05:53:06.492 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000002.591.5, 00000000002.412.5, , , 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:53:06.492 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000002.591.5, 00000000002.412.5, , , 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:53:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:53:06.489+0000 7f2f412d1600 0 bucket checkpoint complete 2022-09-02T05:53:06.507 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:53:06.508 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:53:06.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.509 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:53:06.567 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:53:06.567 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:53:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:53:06.616+0000 7f610e251600 1 bucket sync caught up with source: 2022-09-02T05:53:06.618 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000002.351.5, 00000000002.391.5, , , 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:53:06.619 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000002.351.5, 00000000002.391.5, , , 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:53:06.619 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:53:06.616+0000 7f610e251600 0 bucket checkpoint complete 2022-09-02T05:53:06.638 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:53:06.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:06.640 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:53:06.705 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:53:06.705 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:53:06.749 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:06.748+0000 7f54eb8d0600 1 bucket sync caught up with source: 2022-09-02T05:53:06.750 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000002.351.5, 00000000002.391.5, , , 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:53:06.750 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000002.351.5, 00000000002.391.5, , , 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:53:06.750 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:06.748+0000 7f54eb8d0600 0 bucket checkpoint complete 2022-09-02T05:53:06.766 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:53:06.767 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:53:06.768 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.769 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.770 DEBUG:boto:Method: HEAD 2022-09-02T05:53:06.770 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:06.771 DEBUG:boto:Data: 2022-09-02T05:53:06.771 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.772 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.773 DEBUG:boto:Params: {} 2022-09-02T05:53:06.774 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:53:06.775 DEBUG:boto:Token: None 2022-09-02T05:53:06.776 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/ 2022-09-02T05:53:06.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m9fNoTgzS6Ds5dHCkTV0baBqp4U= 2022-09-02T05:53:06.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m9fNoTgzS6Ds5dHCkTV0baBqp4U=', 'Content-Length': '0'} 2022-09-02T05:53:06.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f13aa865bf24a90f-0063119a42-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.785 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.786 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.786 DEBUG:boto:Method: HEAD 2022-09-02T05:53:06.787 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:06.787 DEBUG:boto:Data: 2022-09-02T05:53:06.788 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:06.789 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.790 DEBUG:boto:Params: {} 2022-09-02T05:53:06.790 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:53:06.791 DEBUG:boto:Token: None 2022-09-02T05:53:06.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/ 2022-09-02T05:53:06.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m9fNoTgzS6Ds5dHCkTV0baBqp4U= 2022-09-02T05:53:06.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m9fNoTgzS6Ds5dHCkTV0baBqp4U=', 'Content-Length': '0'} 2022-09-02T05:53:06.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0e591ef7c0c2e4f-0063119a42-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.797 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:53:06.798 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.799 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.799 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.800 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.800 DEBUG:boto:Method: GET 2022-09-02T05:53:06.801 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.801 DEBUG:boto:Data: 2022-09-02T05:53:06.802 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.802 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.803 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.803 DEBUG:boto:Params: {} 2022-09-02T05:53:06.804 DEBUG:boto:Token: None 2022-09-02T05:53:06.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/?versions 2022-09-02T05:53:06.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw= 2022-09-02T05:53:06.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw=', 'Content-Length': '0'} 2022-09-02T05:53:06.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003606d73b1e1e22c4-0063119a42-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.810 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:06.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:53:06.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:53:06.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:53:06.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:53:06.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:53:06.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:53:06.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:53:06.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:53:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:53:06.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:53:06.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:53:06.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:53:06.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:53:06.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:53:06.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:53:06.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:53:06.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:53:06.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:53:06.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:53:06.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:53:06.825 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:53:06.825 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.826 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.826 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.827 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.827 DEBUG:boto:Method: GET 2022-09-02T05:53:06.828 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.829 DEBUG:boto:Data: 2022-09-02T05:53:06.829 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:06.830 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.831 DEBUG:boto:Params: {} 2022-09-02T05:53:06.831 DEBUG:boto:Token: None 2022-09-02T05:53:06.832 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/?versions 2022-09-02T05:53:06.833 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw= 2022-09-02T05:53:06.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw=', 'Content-Length': '0'} 2022-09-02T05:53:06.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a62616c984b5da2-0063119a42-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.838 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:06.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:53:06.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:53:06.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:53:06.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:53:06.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:53:06.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:53:06.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:53:06.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:53:06.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:53:06.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:53:06.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:53:06.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:53:06.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:53:06.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:53:06.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:53:06.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:53:06.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:53:06.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:53:06.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:53:06.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:53:06.850 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.850 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.851 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.851 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.852 DEBUG:boto:Method: GET 2022-09-02T05:53:06.852 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.853 DEBUG:boto:Data: 2022-09-02T05:53:06.853 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.855 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.855 DEBUG:boto:Params: {} 2022-09-02T05:53:06.856 DEBUG:boto:Token: None 2022-09-02T05:53:06.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/?versions 2022-09-02T05:53:06.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw= 2022-09-02T05:53:06.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw=', 'Content-Length': '0'} 2022-09-02T05:53:06.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000791d2f9571cb0630-0063119a42-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.861 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:06.864 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.865 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.866 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.867 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.868 DEBUG:boto:Method: GET 2022-09-02T05:53:06.869 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:06.869 DEBUG:boto:Data: 2022-09-02T05:53:06.870 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:06.871 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.871 DEBUG:boto:Params: {} 2022-09-02T05:53:06.872 DEBUG:boto:Token: None 2022-09-02T05:53:06.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/?versions 2022-09-02T05:53:06.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw= 2022-09-02T05:53:06.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2xmbi/5tr5FLBMP9ypo9ZyTu9qw=', 'Content-Length': '0'} 2022-09-02T05:53:06.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022d9a052eff33663-0063119a42-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.878 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:06.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:06.880 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:06.881 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:06.881 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.882 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.882 DEBUG:boto:Method: GET 2022-09-02T05:53:06.883 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.884 DEBUG:boto:Data: 2022-09-02T05:53:06.885 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.887 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.888 DEBUG:boto:Params: {} 2022-09-02T05:53:06.889 DEBUG:boto:Token: None 2022-09-02T05:53:06.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE= 2022-09-02T05:53:06.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE=', 'Content-Length': '0'} 2022-09-02T05:53:06.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004219816da89bbc8c-0063119a42-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.896 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:06.897 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:06.898 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.899 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.900 DEBUG:boto:Method: GET 2022-09-02T05:53:06.901 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.902 DEBUG:boto:Data: 2022-09-02T05:53:06.903 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:06.905 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.906 DEBUG:boto:Params: {} 2022-09-02T05:53:06.907 DEBUG:boto:Token: None 2022-09-02T05:53:06.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE= 2022-09-02T05:53:06.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE=', 'Content-Length': '0'} 2022-09-02T05:53:06.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002623a6ca44a83ece-0063119a42-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.914 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:06.914 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:06.915 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.916 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.917 DEBUG:boto:Method: HEAD 2022-09-02T05:53:06.917 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.918 DEBUG:boto:Data: 2022-09-02T05:53:06.919 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.920 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.920 DEBUG:boto:Params: {} 2022-09-02T05:53:06.921 DEBUG:boto:Token: None 2022-09-02T05:53:06.921 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udfYNK5S2KMuvZg7H13g/BirC38= 2022-09-02T05:53:06.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udfYNK5S2KMuvZg7H13g/BirC38=', 'Content-Length': '0'} 2022-09-02T05:53:06.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64f421021ba0067-0063119a42-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.926 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:06.926 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:06.927 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.927 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.928 DEBUG:boto:Method: HEAD 2022-09-02T05:53:06.928 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.929 DEBUG:boto:Data: 2022-09-02T05:53:06.929 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:06.930 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.931 DEBUG:boto:Params: {} 2022-09-02T05:53:06.932 DEBUG:boto:Token: None 2022-09-02T05:53:06.932 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:udfYNK5S2KMuvZg7H13g/BirC38= 2022-09-02T05:53:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:udfYNK5S2KMuvZg7H13g/BirC38=', 'Content-Length': '0'} 2022-09-02T05:53:06.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd7b746832574bb-0063119a42-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:06.936 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:06.937 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:06.938 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.938 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.939 DEBUG:boto:Method: GET 2022-09-02T05:53:06.939 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.940 DEBUG:boto:Data: 2022-09-02T05:53:06.940 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.941 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.942 DEBUG:boto:Params: {} 2022-09-02T05:53:06.943 DEBUG:boto:Token: None 2022-09-02T05:53:06.943 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE= 2022-09-02T05:53:06.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE=', 'Content-Length': '0'} 2022-09-02T05:53:06.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f9d77fdd81e5f0-0063119a42-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.947 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:06.948 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:06.948 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.949 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.949 DEBUG:boto:Method: GET 2022-09-02T05:53:06.950 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.951 DEBUG:boto:Data: 2022-09-02T05:53:06.952 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:06.953 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.954 DEBUG:boto:Params: {} 2022-09-02T05:53:06.955 DEBUG:boto:Token: None 2022-09-02T05:53:06.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:06.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE= 2022-09-02T05:53:06.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oTUFMoRGodQGoyiWjlzfql1lGrE=', 'Content-Length': '0'} 2022-09-02T05:53:06.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a87017898c74e435-0063119a42-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:06.962 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:06.963 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:06.964 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:06.964 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:06.965 DEBUG:boto:Method: GET 2022-09-02T05:53:06.965 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:06.966 DEBUG:boto:Data: 2022-09-02T05:53:06.966 DEBUG:boto:Headers: {} 2022-09-02T05:53:06.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:06.967 DEBUG:boto:Port: 8000 2022-09-02T05:53:06.968 DEBUG:boto:Params: {} 2022-09-02T05:53:06.968 DEBUG:boto:Token: None 2022-09-02T05:53:06.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:06 GMT /xljbkf-74/ 2022-09-02T05:53:06.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FTrhuz+UzL5guyC+8wTXfXGlFc8= 2022-09-02T05:53:06.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FTrhuz+UzL5guyC+8wTXfXGlFc8=', 'Content-Length': '0'} 2022-09-02T05:53:06.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979aad047941b7d9-0063119a42-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.014 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.015 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.016 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.016 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.017 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.018 DEBUG:boto:Method: GET 2022-09-02T05:53:07.018 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.019 DEBUG:boto:Data: 2022-09-02T05:53:07.019 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.020 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.021 DEBUG:boto:Params: {} 2022-09-02T05:53:07.022 DEBUG:boto:Token: None 2022-09-02T05:53:07.022 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca7b6f0fb1989e3-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.027 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.028 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:07.029 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:07.029 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.030 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:07.030 DEBUG:boto:Data: 2022-09-02T05:53:07.031 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.031 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.032 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.033 DEBUG:boto:Params: {} 2022-09-02T05:53:07.033 DEBUG:boto:Token: None 2022-09-02T05:53:07.034 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj11 2022-09-02T05:53:07.035 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QzpFG7Yr5SaB0+We2+ZRsJ+5ji0= 2022-09-02T05:53:07.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QzpFG7Yr5SaB0+We2+ZRsJ+5ji0=', 'Content-Length': '0'} 2022-09-02T05:53:07.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371f264bdf9ab1ce-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.040 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:07.041 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:07.041 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.042 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:07.043 DEBUG:boto:Data: 2022-09-02T05:53:07.044 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.046 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.047 DEBUG:boto:Params: {} 2022-09-02T05:53:07.048 DEBUG:boto:Token: None 2022-09-02T05:53:07.048 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj11 2022-09-02T05:53:07.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QzpFG7Yr5SaB0+We2+ZRsJ+5ji0= 2022-09-02T05:53:07.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QzpFG7Yr5SaB0+We2+ZRsJ+5ji0=', 'Content-Length': '0'} 2022-09-02T05:53:07.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa84affa508cb40-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:07.053 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:07.054 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:07.054 DEBUG:boto:Method: GET 2022-09-02T05:53:07.055 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:07.055 DEBUG:boto:Data: 2022-09-02T05:53:07.056 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.057 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.057 DEBUG:boto:Params: {} 2022-09-02T05:53:07.058 DEBUG:boto:Token: None 2022-09-02T05:53:07.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj11 2022-09-02T05:53:07.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P8yWwwIkuo7uiuhVCCgF3/HD3ro= 2022-09-02T05:53:07.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P8yWwwIkuo7uiuhVCCgF3/HD3ro=', 'Content-Length': '0'} 2022-09-02T05:53:07.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca7d7b8ebe9ce95-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.062 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:07.063 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:07.064 DEBUG:boto:Method: GET 2022-09-02T05:53:07.064 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:07.065 DEBUG:boto:Data: 2022-09-02T05:53:07.065 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.066 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.067 DEBUG:boto:Params: {} 2022-09-02T05:53:07.067 DEBUG:boto:Token: None 2022-09-02T05:53:07.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj11 2022-09-02T05:53:07.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P8yWwwIkuo7uiuhVCCgF3/HD3ro= 2022-09-02T05:53:07.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P8yWwwIkuo7uiuhVCCgF3/HD3ro=', 'Content-Length': '0'} 2022-09-02T05:53:07.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f3bfde49a5af8d-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.072 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.072 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.073 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.073 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.074 DEBUG:boto:Method: GET 2022-09-02T05:53:07.075 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.075 DEBUG:boto:Data: 2022-09-02T05:53:07.076 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.077 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.077 DEBUG:boto:Params: {} 2022-09-02T05:53:07.078 DEBUG:boto:Token: None 2022-09-02T05:53:07.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f08d282fc541d5-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.082 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.083 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.084 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.084 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.085 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.085 DEBUG:boto:Method: GET 2022-09-02T05:53:07.086 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:07.086 DEBUG:boto:Data: 2022-09-02T05:53:07.087 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.088 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.088 DEBUG:boto:Params: {} 2022-09-02T05:53:07.089 DEBUG:boto:Token: None 2022-09-02T05:53:07.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c84a3a5de3cba4-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.095 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:07.096 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.097 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.097 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.098 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.098 DEBUG:boto:Method: GET 2022-09-02T05:53:07.099 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.100 DEBUG:boto:Data: 2022-09-02T05:53:07.100 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.101 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.102 DEBUG:boto:Params: {} 2022-09-02T05:53:07.102 DEBUG:boto:Token: None 2022-09-02T05:53:07.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc= 2022-09-02T05:53:07.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc=', 'Content-Length': '0'} 2022-09-02T05:53:07.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd9a86c381b9ab3-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.150 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.151 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.152 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.153 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.153 DEBUG:boto:Method: GET 2022-09-02T05:53:07.154 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.154 DEBUG:boto:Data: 2022-09-02T05:53:07.155 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.156 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.156 DEBUG:boto:Params: {} 2022-09-02T05:53:07.157 DEBUG:boto:Token: None 2022-09-02T05:53:07.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc= 2022-09-02T05:53:07.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc=', 'Content-Length': '0'} 2022-09-02T05:53:07.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b335a6e442609e-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.163 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.163 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.164 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.164 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.165 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.165 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.166 DEBUG:boto:Data: 2022-09-02T05:53:07.166 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.168 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.168 DEBUG:boto:Params: {} 2022-09-02T05:53:07.169 DEBUG:boto:Token: None 2022-09-02T05:53:07.169 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:INyfS6ElsCaii5tdASH/T0V14x8= 2022-09-02T05:53:07.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:INyfS6ElsCaii5tdASH/T0V14x8=', 'Content-Length': '0'} 2022-09-02T05:53:07.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08612ed2274885b-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.173 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.174 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.174 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.175 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.176 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.176 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.177 DEBUG:boto:Data: 2022-09-02T05:53:07.177 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.178 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.179 DEBUG:boto:Params: {} 2022-09-02T05:53:07.179 DEBUG:boto:Token: None 2022-09-02T05:53:07.180 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:INyfS6ElsCaii5tdASH/T0V14x8= 2022-09-02T05:53:07.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:INyfS6ElsCaii5tdASH/T0V14x8=', 'Content-Length': '0'} 2022-09-02T05:53:07.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000936ad2b1c6b46035-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:07.184 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.185 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.186 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.186 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.187 DEBUG:boto:Method: GET 2022-09-02T05:53:07.187 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.188 DEBUG:boto:Data: 2022-09-02T05:53:07.188 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.189 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.189 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.190 DEBUG:boto:Params: {} 2022-09-02T05:53:07.191 DEBUG:boto:Token: None 2022-09-02T05:53:07.192 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc= 2022-09-02T05:53:07.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc=', 'Content-Length': '0'} 2022-09-02T05:53:07.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307953a298df097d-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.197 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.198 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.199 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.200 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.201 DEBUG:boto:Method: GET 2022-09-02T05:53:07.201 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.202 DEBUG:boto:Data: 2022-09-02T05:53:07.202 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.203 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.204 DEBUG:boto:Params: {} 2022-09-02T05:53:07.205 DEBUG:boto:Token: None 2022-09-02T05:53:07.205 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:07.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc= 2022-09-02T05:53:07.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AUs8Wj0Wzf90bitzeqgoGJ4QFJc=', 'Content-Length': '0'} 2022-09-02T05:53:07.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173d6b6325b54ec0-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.209 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.210 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.211 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.211 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.212 DEBUG:boto:Method: GET 2022-09-02T05:53:07.212 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.213 DEBUG:boto:Data: 2022-09-02T05:53:07.213 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.214 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.214 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.215 DEBUG:boto:Params: {} 2022-09-02T05:53:07.215 DEBUG:boto:Token: None 2022-09-02T05:53:07.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb106af9d9809d84-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.220 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.221 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.222 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.223 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.223 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.224 DEBUG:boto:Method: GET 2022-09-02T05:53:07.224 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.225 DEBUG:boto:Data: 2022-09-02T05:53:07.225 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.226 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.227 DEBUG:boto:Params: {} 2022-09-02T05:53:07.227 DEBUG:boto:Token: None 2022-09-02T05:53:07.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ec8efc6d8e4cb7-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.233 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.234 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.234 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.235 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.235 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:07.236 DEBUG:boto:Data: 2022-09-02T05:53:07.236 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.237 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.238 DEBUG:boto:Params: {} 2022-09-02T05:53:07.239 DEBUG:boto:Token: None 2022-09-02T05:53:07.240 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115 2022-09-02T05:53:07.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E1rtzK0OfTWFCSsP7HrPaYr+KQA= 2022-09-02T05:53:07.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E1rtzK0OfTWFCSsP7HrPaYr+KQA=', 'Content-Length': '0'} 2022-09-02T05:53:07.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bea6a2be6178297-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.246 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.247 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.248 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.248 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:07.249 DEBUG:boto:Data: 2022-09-02T05:53:07.250 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.252 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.253 DEBUG:boto:Params: {} 2022-09-02T05:53:07.254 DEBUG:boto:Token: None 2022-09-02T05:53:07.255 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115 2022-09-02T05:53:07.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E1rtzK0OfTWFCSsP7HrPaYr+KQA= 2022-09-02T05:53:07.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E1rtzK0OfTWFCSsP7HrPaYr+KQA=', 'Content-Length': '0'} 2022-09-02T05:53:07.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089bf4a76346840f1-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:07.262 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.263 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.264 DEBUG:boto:Method: GET 2022-09-02T05:53:07.265 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:07.265 DEBUG:boto:Data: 2022-09-02T05:53:07.266 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.267 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.267 DEBUG:boto:Params: {} 2022-09-02T05:53:07.268 DEBUG:boto:Token: None 2022-09-02T05:53:07.269 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115 2022-09-02T05:53:07.269 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RQnxS+yihLyE9h+1ZRBjyeaFte8= 2022-09-02T05:53:07.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RQnxS+yihLyE9h+1ZRBjyeaFte8=', 'Content-Length': '0'} 2022-09-02T05:53:07.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbcbd003345f0b65-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.274 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:07.274 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:07.275 DEBUG:boto:Method: GET 2022-09-02T05:53:07.275 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:07.276 DEBUG:boto:Data: 2022-09-02T05:53:07.276 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.277 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.278 DEBUG:boto:Params: {} 2022-09-02T05:53:07.278 DEBUG:boto:Token: None 2022-09-02T05:53:07.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj115 2022-09-02T05:53:07.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RQnxS+yihLyE9h+1ZRBjyeaFte8= 2022-09-02T05:53:07.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RQnxS+yihLyE9h+1ZRBjyeaFte8=', 'Content-Length': '0'} 2022-09-02T05:53:07.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03b3e62020f9468-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.283 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.283 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.284 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.284 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.285 DEBUG:boto:Method: GET 2022-09-02T05:53:07.285 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.286 DEBUG:boto:Data: 2022-09-02T05:53:07.287 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.288 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.288 DEBUG:boto:Params: {} 2022-09-02T05:53:07.289 DEBUG:boto:Token: None 2022-09-02T05:53:07.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f21501e0bd889f-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.294 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.295 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.296 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.296 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.297 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.297 DEBUG:boto:Method: GET 2022-09-02T05:53:07.298 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:07.298 DEBUG:boto:Data: 2022-09-02T05:53:07.299 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.300 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.301 DEBUG:boto:Params: {} 2022-09-02T05:53:07.301 DEBUG:boto:Token: None 2022-09-02T05:53:07.302 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.302 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb35326df6323eb9-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.307 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:07.308 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.309 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.309 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.310 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.310 DEBUG:boto:Method: GET 2022-09-02T05:53:07.311 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.312 DEBUG:boto:Data: 2022-09-02T05:53:07.312 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.313 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.314 DEBUG:boto:Params: {} 2022-09-02T05:53:07.315 DEBUG:boto:Token: None 2022-09-02T05:53:07.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA= 2022-09-02T05:53:07.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA=', 'Content-Length': '0'} 2022-09-02T05:53:07.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005504939beadcdce5-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.321 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.322 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.322 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.323 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.323 DEBUG:boto:Method: GET 2022-09-02T05:53:07.324 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.325 DEBUG:boto:Data: 2022-09-02T05:53:07.325 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.326 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.327 DEBUG:boto:Params: {} 2022-09-02T05:53:07.328 DEBUG:boto:Token: None 2022-09-02T05:53:07.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA= 2022-09-02T05:53:07.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA=', 'Content-Length': '0'} 2022-09-02T05:53:07.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fa6b08df2ba584-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.374 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.375 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.375 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.376 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.377 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.378 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.379 DEBUG:boto:Data: 2022-09-02T05:53:07.379 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.380 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.381 DEBUG:boto:Params: {} 2022-09-02T05:53:07.381 DEBUG:boto:Token: None 2022-09-02T05:53:07.382 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qw1TBll2MuX91qJ6wHBiSwFadjg= 2022-09-02T05:53:07.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qw1TBll2MuX91qJ6wHBiSwFadjg=', 'Content-Length': '0'} 2022-09-02T05:53:07.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea5b059b5231b253-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.387 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.387 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.388 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.388 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.389 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.389 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.390 DEBUG:boto:Data: 2022-09-02T05:53:07.390 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.392 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.392 DEBUG:boto:Params: {} 2022-09-02T05:53:07.393 DEBUG:boto:Token: None 2022-09-02T05:53:07.394 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qw1TBll2MuX91qJ6wHBiSwFadjg= 2022-09-02T05:53:07.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qw1TBll2MuX91qJ6wHBiSwFadjg=', 'Content-Length': '0'} 2022-09-02T05:53:07.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab601241b33c82a-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:07.399 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.399 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.400 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.401 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.402 DEBUG:boto:Method: GET 2022-09-02T05:53:07.403 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.405 DEBUG:boto:Data: 2022-09-02T05:53:07.406 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.409 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.410 DEBUG:boto:Params: {} 2022-09-02T05:53:07.412 DEBUG:boto:Token: None 2022-09-02T05:53:07.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA= 2022-09-02T05:53:07.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA=', 'Content-Length': '0'} 2022-09-02T05:53:07.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c828f084ceecb199-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.421 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.422 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.423 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.425 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.426 DEBUG:boto:Method: GET 2022-09-02T05:53:07.427 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.428 DEBUG:boto:Data: 2022-09-02T05:53:07.430 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.432 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.433 DEBUG:boto:Params: {} 2022-09-02T05:53:07.435 DEBUG:boto:Token: None 2022-09-02T05:53:07.436 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:07.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA= 2022-09-02T05:53:07.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IWumsODskwE0T3nLEZoMw/op8WA=', 'Content-Length': '0'} 2022-09-02T05:53:07.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090057b746fc4ac72-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.441 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.441 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.442 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.443 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.443 DEBUG:boto:Method: GET 2022-09-02T05:53:07.444 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.445 DEBUG:boto:Data: 2022-09-02T05:53:07.446 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.447 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.448 DEBUG:boto:Params: {} 2022-09-02T05:53:07.449 DEBUG:boto:Token: None 2022-09-02T05:53:07.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9cd4a80c27b5ea8-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.456 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.457 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.457 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.458 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.459 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.459 DEBUG:boto:Method: GET 2022-09-02T05:53:07.460 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.461 DEBUG:boto:Data: 2022-09-02T05:53:07.462 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.463 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.464 DEBUG:boto:Params: {} 2022-09-02T05:53:07.465 DEBUG:boto:Token: None 2022-09-02T05:53:07.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001184a103eebbeac6-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.471 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.472 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.472 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.473 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.474 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:07.474 DEBUG:boto:Data: 2022-09-02T05:53:07.475 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.476 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.477 DEBUG:boto:Params: {} 2022-09-02T05:53:07.478 DEBUG:boto:Token: None 2022-09-02T05:53:07.479 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17 2022-09-02T05:53:07.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZmcKYR2mbLUOBkqPikM2VC3UVw4= 2022-09-02T05:53:07.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZmcKYR2mbLUOBkqPikM2VC3UVw4=', 'Content-Length': '0'} 2022-09-02T05:53:07.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b436840ffd61576-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.483 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.484 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.485 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.485 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:07.486 DEBUG:boto:Data: 2022-09-02T05:53:07.487 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.488 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.488 DEBUG:boto:Params: {} 2022-09-02T05:53:07.489 DEBUG:boto:Token: None 2022-09-02T05:53:07.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17 2022-09-02T05:53:07.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZmcKYR2mbLUOBkqPikM2VC3UVw4= 2022-09-02T05:53:07.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZmcKYR2mbLUOBkqPikM2VC3UVw4=', 'Content-Length': '0'} 2022-09-02T05:53:07.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38531176ca2b128-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:07.495 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.496 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.496 DEBUG:boto:Method: GET 2022-09-02T05:53:07.497 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:07.498 DEBUG:boto:Data: 2022-09-02T05:53:07.498 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.500 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.501 DEBUG:boto:Params: {} 2022-09-02T05:53:07.501 DEBUG:boto:Token: None 2022-09-02T05:53:07.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17 2022-09-02T05:53:07.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zhesSDIpW/tttR1z4RyVVu8Hl8I= 2022-09-02T05:53:07.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zhesSDIpW/tttR1z4RyVVu8Hl8I=', 'Content-Length': '0'} 2022-09-02T05:53:07.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028272947a5760a6-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.550 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:07.551 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:07.551 DEBUG:boto:Method: GET 2022-09-02T05:53:07.552 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:07.552 DEBUG:boto:Data: 2022-09-02T05:53:07.553 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.554 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.555 DEBUG:boto:Params: {} 2022-09-02T05:53:07.556 DEBUG:boto:Token: None 2022-09-02T05:53:07.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj17 2022-09-02T05:53:07.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zhesSDIpW/tttR1z4RyVVu8Hl8I= 2022-09-02T05:53:07.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zhesSDIpW/tttR1z4RyVVu8Hl8I=', 'Content-Length': '0'} 2022-09-02T05:53:07.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098db741a7bdb877e-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.561 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.561 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.562 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.563 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.563 DEBUG:boto:Method: GET 2022-09-02T05:53:07.564 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.565 DEBUG:boto:Data: 2022-09-02T05:53:07.565 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.567 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.568 DEBUG:boto:Params: {} 2022-09-02T05:53:07.569 DEBUG:boto:Token: None 2022-09-02T05:53:07.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d77139b33760663-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.575 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.577 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.578 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.579 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.581 DEBUG:boto:Method: GET 2022-09-02T05:53:07.582 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:07.583 DEBUG:boto:Data: 2022-09-02T05:53:07.584 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.587 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.588 DEBUG:boto:Params: {} 2022-09-02T05:53:07.589 DEBUG:boto:Token: None 2022-09-02T05:53:07.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a7749701111e3f2-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.598 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:07.600 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.601 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.601 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.602 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.602 DEBUG:boto:Method: GET 2022-09-02T05:53:07.603 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.604 DEBUG:boto:Data: 2022-09-02T05:53:07.604 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.605 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.605 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.606 DEBUG:boto:Params: {} 2022-09-02T05:53:07.606 DEBUG:boto:Token: None 2022-09-02T05:53:07.607 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48= 2022-09-02T05:53:07.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48=', 'Content-Length': '0'} 2022-09-02T05:53:07.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094578d40ae9939d8-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.614 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.614 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.615 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.616 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.616 DEBUG:boto:Method: GET 2022-09-02T05:53:07.617 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.618 DEBUG:boto:Data: 2022-09-02T05:53:07.619 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.620 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.621 DEBUG:boto:Params: {} 2022-09-02T05:53:07.621 DEBUG:boto:Token: None 2022-09-02T05:53:07.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48= 2022-09-02T05:53:07.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48=', 'Content-Length': '0'} 2022-09-02T05:53:07.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906fc7abf37f1b6a-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.627 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.628 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.628 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.629 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.629 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.630 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.630 DEBUG:boto:Data: 2022-09-02T05:53:07.631 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.631 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.632 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.632 DEBUG:boto:Params: {} 2022-09-02T05:53:07.633 DEBUG:boto:Token: None 2022-09-02T05:53:07.633 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.634 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sp+IZpaNp2GU8z32Edk6NZcG4zY= 2022-09-02T05:53:07.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sp+IZpaNp2GU8z32Edk6NZcG4zY=', 'Content-Length': '0'} 2022-09-02T05:53:07.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a35fb28735cd24-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.642 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.643 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.644 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.644 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.645 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.646 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.646 DEBUG:boto:Data: 2022-09-02T05:53:07.647 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.648 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.648 DEBUG:boto:Params: {} 2022-09-02T05:53:07.649 DEBUG:boto:Token: None 2022-09-02T05:53:07.650 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sp+IZpaNp2GU8z32Edk6NZcG4zY= 2022-09-02T05:53:07.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sp+IZpaNp2GU8z32Edk6NZcG4zY=', 'Content-Length': '0'} 2022-09-02T05:53:07.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8276d556895dec-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:07.654 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.655 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.655 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.656 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.656 DEBUG:boto:Method: GET 2022-09-02T05:53:07.657 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.657 DEBUG:boto:Data: 2022-09-02T05:53:07.658 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.658 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.659 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.659 DEBUG:boto:Params: {} 2022-09-02T05:53:07.660 DEBUG:boto:Token: None 2022-09-02T05:53:07.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48= 2022-09-02T05:53:07.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48=', 'Content-Length': '0'} 2022-09-02T05:53:07.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b00fb752ef5d38c6-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.665 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.665 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.666 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.666 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.667 DEBUG:boto:Method: GET 2022-09-02T05:53:07.667 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.668 DEBUG:boto:Data: 2022-09-02T05:53:07.668 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.669 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.670 DEBUG:boto:Params: {} 2022-09-02T05:53:07.670 DEBUG:boto:Token: None 2022-09-02T05:53:07.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:07.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48= 2022-09-02T05:53:07.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4+UsDeBdg8T4U8ZvhawYLnLCe48=', 'Content-Length': '0'} 2022-09-02T05:53:07.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e77398fd120c9c-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.675 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.676 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.676 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.677 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.677 DEBUG:boto:Method: GET 2022-09-02T05:53:07.678 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.678 DEBUG:boto:Data: 2022-09-02T05:53:07.679 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.680 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.680 DEBUG:boto:Params: {} 2022-09-02T05:53:07.681 DEBUG:boto:Token: None 2022-09-02T05:53:07.682 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000223afd03948693eb-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.686 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.687 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.688 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.688 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.689 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.689 DEBUG:boto:Method: GET 2022-09-02T05:53:07.690 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.690 DEBUG:boto:Data: 2022-09-02T05:53:07.691 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.692 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.693 DEBUG:boto:Params: {} 2022-09-02T05:53:07.693 DEBUG:boto:Token: None 2022-09-02T05:53:07.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bdd5ba8d3e55ead-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.746 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.747 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.748 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.748 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.749 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:07.749 DEBUG:boto:Data: 2022-09-02T05:53:07.750 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.750 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.751 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.751 DEBUG:boto:Params: {} 2022-09-02T05:53:07.752 DEBUG:boto:Token: None 2022-09-02T05:53:07.752 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19 2022-09-02T05:53:07.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3pIuHTk5Jup6IL/LLDno4w1eR7Y= 2022-09-02T05:53:07.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3pIuHTk5Jup6IL/LLDno4w1eR7Y=', 'Content-Length': '0'} 2022-09-02T05:53:07.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a620182471a45fa4-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.757 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.757 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.758 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.758 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:07.759 DEBUG:boto:Data: 2022-09-02T05:53:07.759 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.760 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.761 DEBUG:boto:Params: {} 2022-09-02T05:53:07.761 DEBUG:boto:Token: None 2022-09-02T05:53:07.762 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19 2022-09-02T05:53:07.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3pIuHTk5Jup6IL/LLDno4w1eR7Y= 2022-09-02T05:53:07.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3pIuHTk5Jup6IL/LLDno4w1eR7Y=', 'Content-Length': '0'} 2022-09-02T05:53:07.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000126af4b68e2474cb-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:07.766 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.767 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.767 DEBUG:boto:Method: GET 2022-09-02T05:53:07.768 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:07.768 DEBUG:boto:Data: 2022-09-02T05:53:07.769 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.770 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.771 DEBUG:boto:Params: {} 2022-09-02T05:53:07.771 DEBUG:boto:Token: None 2022-09-02T05:53:07.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19 2022-09-02T05:53:07.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R8RvXYHyZvIWfzqT9nKhfkZpcdg= 2022-09-02T05:53:07.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R8RvXYHyZvIWfzqT9nKhfkZpcdg=', 'Content-Length': '0'} 2022-09-02T05:53:07.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a90102b25ebae71-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.776 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:07.777 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:07.777 DEBUG:boto:Method: GET 2022-09-02T05:53:07.778 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:07.778 DEBUG:boto:Data: 2022-09-02T05:53:07.779 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.780 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.780 DEBUG:boto:Params: {} 2022-09-02T05:53:07.781 DEBUG:boto:Token: None 2022-09-02T05:53:07.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj19 2022-09-02T05:53:07.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R8RvXYHyZvIWfzqT9nKhfkZpcdg= 2022-09-02T05:53:07.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R8RvXYHyZvIWfzqT9nKhfkZpcdg=', 'Content-Length': '0'} 2022-09-02T05:53:07.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c638bd245539187-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.786 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.787 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.788 DEBUG:boto:Method: GET 2022-09-02T05:53:07.788 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.790 DEBUG:boto:Data: 2022-09-02T05:53:07.791 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.793 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.793 DEBUG:boto:Params: {} 2022-09-02T05:53:07.794 DEBUG:boto:Token: None 2022-09-02T05:53:07.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006fbbfa4d7d6dbdc-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.803 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.804 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.805 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.806 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.806 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.807 DEBUG:boto:Method: GET 2022-09-02T05:53:07.807 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:07.808 DEBUG:boto:Data: 2022-09-02T05:53:07.809 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.810 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.811 DEBUG:boto:Params: {} 2022-09-02T05:53:07.812 DEBUG:boto:Token: None 2022-09-02T05:53:07.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000641fd4253ac3f0a3-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.818 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:07.819 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.820 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.821 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.821 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.822 DEBUG:boto:Method: GET 2022-09-02T05:53:07.822 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.823 DEBUG:boto:Data: 2022-09-02T05:53:07.824 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.825 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.826 DEBUG:boto:Params: {} 2022-09-02T05:53:07.827 DEBUG:boto:Token: None 2022-09-02T05:53:07.828 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo= 2022-09-02T05:53:07.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo=', 'Content-Length': '0'} 2022-09-02T05:53:07.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267060821ddf164d-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.832 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.833 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.834 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.834 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.835 DEBUG:boto:Method: GET 2022-09-02T05:53:07.836 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.836 DEBUG:boto:Data: 2022-09-02T05:53:07.837 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.838 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.839 DEBUG:boto:Params: {} 2022-09-02T05:53:07.840 DEBUG:boto:Token: None 2022-09-02T05:53:07.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo= 2022-09-02T05:53:07.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo=', 'Content-Length': '0'} 2022-09-02T05:53:07.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4f8069f1b2c3f9-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.845 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.845 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.846 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.846 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.847 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.847 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.848 DEBUG:boto:Data: 2022-09-02T05:53:07.848 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.849 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.850 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.850 DEBUG:boto:Params: {} 2022-09-02T05:53:07.851 DEBUG:boto:Token: None 2022-09-02T05:53:07.851 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JtcdWMOYiTwyVN6V2AOnigripYY= 2022-09-02T05:53:07.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JtcdWMOYiTwyVN6V2AOnigripYY=', 'Content-Length': '0'} 2022-09-02T05:53:07.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d2d55a941f1f614-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.856 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.856 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.857 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.857 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.858 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.858 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.859 DEBUG:boto:Data: 2022-09-02T05:53:07.859 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.860 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.861 DEBUG:boto:Params: {} 2022-09-02T05:53:07.861 DEBUG:boto:Token: None 2022-09-02T05:53:07.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JtcdWMOYiTwyVN6V2AOnigripYY= 2022-09-02T05:53:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JtcdWMOYiTwyVN6V2AOnigripYY=', 'Content-Length': '0'} 2022-09-02T05:53:07.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0d087271185e3a-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:07.866 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.867 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.867 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.868 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.868 DEBUG:boto:Method: GET 2022-09-02T05:53:07.869 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.869 DEBUG:boto:Data: 2022-09-02T05:53:07.870 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.871 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.871 DEBUG:boto:Params: {} 2022-09-02T05:53:07.872 DEBUG:boto:Token: None 2022-09-02T05:53:07.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo= 2022-09-02T05:53:07.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo=', 'Content-Length': '0'} 2022-09-02T05:53:07.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edfd18174e80c38b-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.878 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.878 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.879 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.879 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.880 DEBUG:boto:Method: GET 2022-09-02T05:53:07.880 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.881 DEBUG:boto:Data: 2022-09-02T05:53:07.881 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.882 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.883 DEBUG:boto:Params: {} 2022-09-02T05:53:07.884 DEBUG:boto:Token: None 2022-09-02T05:53:07.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:07.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo= 2022-09-02T05:53:07.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Juus/J6UH1VRal43Ew90eGcMAUo=', 'Content-Length': '0'} 2022-09-02T05:53:07.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fe5064b0a6dedb2-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.889 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.889 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.890 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:07.891 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:07.891 DEBUG:boto:Method: GET 2022-09-02T05:53:07.892 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:07.892 DEBUG:boto:Data: 2022-09-02T05:53:07.893 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.894 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.895 DEBUG:boto:Params: {} 2022-09-02T05:53:07.895 DEBUG:boto:Token: None 2022-09-02T05:53:07.896 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e38956c50d3702a-0063119a43-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.901 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.902 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:07.902 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:07.903 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:07.903 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:07.904 DEBUG:boto:Method: GET 2022-09-02T05:53:07.904 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:07.905 DEBUG:boto:Data: 2022-09-02T05:53:07.905 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.906 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.907 DEBUG:boto:Params: {} 2022-09-02T05:53:07.908 DEBUG:boto:Token: None 2022-09-02T05:53:07.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/ 2022-09-02T05:53:07.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q= 2022-09-02T05:53:07.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:goIdSPFwoyKYKdANi7AcA/eHW0Q=', 'Content-Length': '0'} 2022-09-02T05:53:07.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df06a4fade818ac-0063119a43-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.913 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:07.914 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.915 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.915 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.916 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:07.916 DEBUG:boto:Data: 2022-09-02T05:53:07.917 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.918 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.918 DEBUG:boto:Params: {} 2022-09-02T05:53:07.919 DEBUG:boto:Token: None 2022-09-02T05:53:07.920 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22 2022-09-02T05:53:07.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KgB0obVaBmG36yiZjPG258H5ac4= 2022-09-02T05:53:07.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KgB0obVaBmG36yiZjPG258H5ac4=', 'Content-Length': '0'} 2022-09-02T05:53:07.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e14bf05c7bf68978-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.924 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.924 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.925 DEBUG:boto:Method: HEAD 2022-09-02T05:53:07.925 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:07.926 DEBUG:boto:Data: 2022-09-02T05:53:07.926 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.928 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.929 DEBUG:boto:Params: {} 2022-09-02T05:53:07.930 DEBUG:boto:Token: None 2022-09-02T05:53:07.931 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22 2022-09-02T05:53:07.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KgB0obVaBmG36yiZjPG258H5ac4= 2022-09-02T05:53:07.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KgB0obVaBmG36yiZjPG258H5ac4=', 'Content-Length': '0'} 2022-09-02T05:53:07.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004635565a969c9445-0063119a43-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:07.938 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.939 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.940 DEBUG:boto:Method: GET 2022-09-02T05:53:07.940 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:07.941 DEBUG:boto:Data: 2022-09-02T05:53:07.941 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.942 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:07.943 DEBUG:boto:Port: 8000 2022-09-02T05:53:07.943 DEBUG:boto:Params: {} 2022-09-02T05:53:07.944 DEBUG:boto:Token: None 2022-09-02T05:53:07.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:07 GMT /xljbkf-74/obj22 2022-09-02T05:53:07.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6FGTTNSHuXKzaAPXpUVqflKcPfs= 2022-09-02T05:53:07.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6FGTTNSHuXKzaAPXpUVqflKcPfs=', 'Content-Length': '0'} 2022-09-02T05:53:07.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c16a4de14ea96d0-0063119a43-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:07.994 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:07.995 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:07.996 DEBUG:boto:Method: GET 2022-09-02T05:53:07.996 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:07.997 DEBUG:boto:Data: 2022-09-02T05:53:07.998 DEBUG:boto:Headers: {} 2022-09-02T05:53:07.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:07.999 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.000 DEBUG:boto:Params: {} 2022-09-02T05:53:08.000 DEBUG:boto:Token: None 2022-09-02T05:53:08.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj22 2022-09-02T05:53:08.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a0U71izuDKLrouE/TUjt9JetYWo= 2022-09-02T05:53:08.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a0U71izuDKLrouE/TUjt9JetYWo=', 'Content-Length': '0'} 2022-09-02T05:53:08.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004609ad50b27d9934-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.005 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.006 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.006 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:08.007 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:08.007 DEBUG:boto:Method: GET 2022-09-02T05:53:08.008 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:08.008 DEBUG:boto:Data: 2022-09-02T05:53:08.009 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.010 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.010 DEBUG:boto:Params: {} 2022-09-02T05:53:08.011 DEBUG:boto:Token: None 2022-09-02T05:53:08.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc29f2ad9b52a29e-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.016 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.017 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.018 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.018 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:08.019 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:08.019 DEBUG:boto:Method: GET 2022-09-02T05:53:08.020 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:08.021 DEBUG:boto:Data: 2022-09-02T05:53:08.021 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.022 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.023 DEBUG:boto:Params: {} 2022-09-02T05:53:08.024 DEBUG:boto:Token: None 2022-09-02T05:53:08.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000844e1d3eb78dc2d1-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.029 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:08.031 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.031 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.032 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.033 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.033 DEBUG:boto:Method: GET 2022-09-02T05:53:08.034 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.034 DEBUG:boto:Data: 2022-09-02T05:53:08.035 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.036 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.037 DEBUG:boto:Params: {} 2022-09-02T05:53:08.037 DEBUG:boto:Token: None 2022-09-02T05:53:08.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y= 2022-09-02T05:53:08.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y=', 'Content-Length': '0'} 2022-09-02T05:53:08.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049678f54b648fdd0-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.042 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.043 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.043 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.044 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.044 DEBUG:boto:Method: GET 2022-09-02T05:53:08.045 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.045 DEBUG:boto:Data: 2022-09-02T05:53:08.046 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.047 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.047 DEBUG:boto:Params: {} 2022-09-02T05:53:08.048 DEBUG:boto:Token: None 2022-09-02T05:53:08.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y= 2022-09-02T05:53:08.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y=', 'Content-Length': '0'} 2022-09-02T05:53:08.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030a97ac64ad1c617-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.053 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.053 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.054 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.054 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.055 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.055 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.056 DEBUG:boto:Data: 2022-09-02T05:53:08.056 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.057 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.057 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.058 DEBUG:boto:Params: {} 2022-09-02T05:53:08.058 DEBUG:boto:Token: None 2022-09-02T05:53:08.059 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OtwviukPjqZ6EZibFe28upyXUow= 2022-09-02T05:53:08.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OtwviukPjqZ6EZibFe28upyXUow=', 'Content-Length': '0'} 2022-09-02T05:53:08.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db81d424e33ea0b-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.063 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.064 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.064 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.065 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.065 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.066 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.066 DEBUG:boto:Data: 2022-09-02T05:53:08.067 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.068 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.068 DEBUG:boto:Params: {} 2022-09-02T05:53:08.069 DEBUG:boto:Token: None 2022-09-02T05:53:08.069 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OtwviukPjqZ6EZibFe28upyXUow= 2022-09-02T05:53:08.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OtwviukPjqZ6EZibFe28upyXUow=', 'Content-Length': '0'} 2022-09-02T05:53:08.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11100f7a26c54fa-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:08.074 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.074 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.075 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.075 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.076 DEBUG:boto:Method: GET 2022-09-02T05:53:08.076 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.077 DEBUG:boto:Data: 2022-09-02T05:53:08.077 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.078 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.079 DEBUG:boto:Params: {} 2022-09-02T05:53:08.080 DEBUG:boto:Token: None 2022-09-02T05:53:08.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y= 2022-09-02T05:53:08.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y=', 'Content-Length': '0'} 2022-09-02T05:53:08.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000909e7aa2e69400f-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.126 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.127 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.128 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.128 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.129 DEBUG:boto:Method: GET 2022-09-02T05:53:08.129 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.130 DEBUG:boto:Data: 2022-09-02T05:53:08.131 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.132 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.132 DEBUG:boto:Params: {} 2022-09-02T05:53:08.133 DEBUG:boto:Token: None 2022-09-02T05:53:08.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:08.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y= 2022-09-02T05:53:08.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IBDXCL0o46oQZvwe4bhCa7H577Y=', 'Content-Length': '0'} 2022-09-02T05:53:08.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3ee1c183b612a9-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.137 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.138 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.139 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.139 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.140 DEBUG:boto:Method: GET 2022-09-02T05:53:08.140 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.141 DEBUG:boto:Data: 2022-09-02T05:53:08.141 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.142 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.143 DEBUG:boto:Params: {} 2022-09-02T05:53:08.143 DEBUG:boto:Token: None 2022-09-02T05:53:08.144 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfdefedf52e5813a-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.149 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.150 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.150 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.151 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.151 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.152 DEBUG:boto:Method: GET 2022-09-02T05:53:08.152 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.153 DEBUG:boto:Data: 2022-09-02T05:53:08.153 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.155 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.155 DEBUG:boto:Params: {} 2022-09-02T05:53:08.156 DEBUG:boto:Token: None 2022-09-02T05:53:08.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000792aa98d5b1a2fee-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.162 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.163 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.163 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.164 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.164 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:08.165 DEBUG:boto:Data: 2022-09-02T05:53:08.165 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.166 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.167 DEBUG:boto:Params: {} 2022-09-02T05:53:08.167 DEBUG:boto:Token: None 2022-09-02T05:53:08.168 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23 2022-09-02T05:53:08.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bw5DgDaMcbQvBMuBWopYBgNz7Pg= 2022-09-02T05:53:08.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bw5DgDaMcbQvBMuBWopYBgNz7Pg=', 'Content-Length': '0'} 2022-09-02T05:53:08.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d2903206a7ea71-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.172 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.172 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.173 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.173 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:08.174 DEBUG:boto:Data: 2022-09-02T05:53:08.174 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.175 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.176 DEBUG:boto:Params: {} 2022-09-02T05:53:08.176 DEBUG:boto:Token: None 2022-09-02T05:53:08.177 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23 2022-09-02T05:53:08.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bw5DgDaMcbQvBMuBWopYBgNz7Pg= 2022-09-02T05:53:08.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bw5DgDaMcbQvBMuBWopYBgNz7Pg=', 'Content-Length': '0'} 2022-09-02T05:53:08.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef009eda677b81f-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:08.182 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.182 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.183 DEBUG:boto:Method: GET 2022-09-02T05:53:08.184 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:08.184 DEBUG:boto:Data: 2022-09-02T05:53:08.185 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.186 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.186 DEBUG:boto:Params: {} 2022-09-02T05:53:08.187 DEBUG:boto:Token: None 2022-09-02T05:53:08.188 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23 2022-09-02T05:53:08.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZvcyggmBKJf8lOH+7kouoYI2hnk= 2022-09-02T05:53:08.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZvcyggmBKJf8lOH+7kouoYI2hnk=', 'Content-Length': '0'} 2022-09-02T05:53:08.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c414c9fb51612d2b-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.192 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:08.192 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:08.193 DEBUG:boto:Method: GET 2022-09-02T05:53:08.193 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:08.194 DEBUG:boto:Data: 2022-09-02T05:53:08.194 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.195 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.196 DEBUG:boto:Params: {} 2022-09-02T05:53:08.197 DEBUG:boto:Token: None 2022-09-02T05:53:08.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj23 2022-09-02T05:53:08.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZvcyggmBKJf8lOH+7kouoYI2hnk= 2022-09-02T05:53:08.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZvcyggmBKJf8lOH+7kouoYI2hnk=', 'Content-Length': '0'} 2022-09-02T05:53:08.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0ae652c303174b9-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.201 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.202 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.202 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.203 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.203 DEBUG:boto:Method: GET 2022-09-02T05:53:08.204 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.204 DEBUG:boto:Data: 2022-09-02T05:53:08.205 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.206 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.207 DEBUG:boto:Params: {} 2022-09-02T05:53:08.207 DEBUG:boto:Token: None 2022-09-02T05:53:08.208 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fefb566e51e55f1f-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.212 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.213 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.214 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.214 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.215 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.215 DEBUG:boto:Method: GET 2022-09-02T05:53:08.216 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:08.216 DEBUG:boto:Data: 2022-09-02T05:53:08.217 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.218 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.219 DEBUG:boto:Params: {} 2022-09-02T05:53:08.219 DEBUG:boto:Token: None 2022-09-02T05:53:08.220 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.221 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022095fd622c0dc11-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.225 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:08.227 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.227 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.228 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.229 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.229 DEBUG:boto:Method: GET 2022-09-02T05:53:08.230 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.230 DEBUG:boto:Data: 2022-09-02T05:53:08.231 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.232 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.232 DEBUG:boto:Params: {} 2022-09-02T05:53:08.233 DEBUG:boto:Token: None 2022-09-02T05:53:08.233 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw= 2022-09-02T05:53:08.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw=', 'Content-Length': '0'} 2022-09-02T05:53:08.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b80f629d13fb889-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.238 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.239 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.239 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.240 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.240 DEBUG:boto:Method: GET 2022-09-02T05:53:08.241 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.242 DEBUG:boto:Data: 2022-09-02T05:53:08.242 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.243 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.244 DEBUG:boto:Params: {} 2022-09-02T05:53:08.246 DEBUG:boto:Token: None 2022-09-02T05:53:08.247 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw= 2022-09-02T05:53:08.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw=', 'Content-Length': '0'} 2022-09-02T05:53:08.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8b6078a4a7d7c50-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.252 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.252 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.253 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.253 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.254 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.255 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.255 DEBUG:boto:Data: 2022-09-02T05:53:08.256 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.257 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.257 DEBUG:boto:Params: {} 2022-09-02T05:53:08.258 DEBUG:boto:Token: None 2022-09-02T05:53:08.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pupnEv4I0CuGqAZ75uJerUTKbuI= 2022-09-02T05:53:08.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pupnEv4I0CuGqAZ75uJerUTKbuI=', 'Content-Length': '0'} 2022-09-02T05:53:08.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000782f0f8a5290210e-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.264 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.265 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.265 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.266 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.267 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.268 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.268 DEBUG:boto:Data: 2022-09-02T05:53:08.269 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.270 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.270 DEBUG:boto:Params: {} 2022-09-02T05:53:08.271 DEBUG:boto:Token: None 2022-09-02T05:53:08.272 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pupnEv4I0CuGqAZ75uJerUTKbuI= 2022-09-02T05:53:08.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pupnEv4I0CuGqAZ75uJerUTKbuI=', 'Content-Length': '0'} 2022-09-02T05:53:08.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db9c09abd4114178-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:08.276 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.277 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.277 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.278 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.278 DEBUG:boto:Method: GET 2022-09-02T05:53:08.279 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.280 DEBUG:boto:Data: 2022-09-02T05:53:08.280 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.281 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.282 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.283 DEBUG:boto:Params: {} 2022-09-02T05:53:08.284 DEBUG:boto:Token: None 2022-09-02T05:53:08.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw= 2022-09-02T05:53:08.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw=', 'Content-Length': '0'} 2022-09-02T05:53:08.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000100a8f498a6059d9-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.343 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.344 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.345 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.346 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.347 DEBUG:boto:Method: GET 2022-09-02T05:53:08.348 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.349 DEBUG:boto:Data: 2022-09-02T05:53:08.350 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.352 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.353 DEBUG:boto:Params: {} 2022-09-02T05:53:08.354 DEBUG:boto:Token: None 2022-09-02T05:53:08.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:08.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw= 2022-09-02T05:53:08.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ICDYXAUHPoahc47pvO8AzVm/ZMw=', 'Content-Length': '0'} 2022-09-02T05:53:08.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c117f5031367437-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.361 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.362 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.363 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.364 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.365 DEBUG:boto:Method: GET 2022-09-02T05:53:08.366 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.367 DEBUG:boto:Data: 2022-09-02T05:53:08.367 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.369 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.370 DEBUG:boto:Params: {} 2022-09-02T05:53:08.371 DEBUG:boto:Token: None 2022-09-02T05:53:08.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4092862fa53bf99-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.377 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.377 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.378 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.378 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.379 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.379 DEBUG:boto:Method: GET 2022-09-02T05:53:08.380 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.381 DEBUG:boto:Data: 2022-09-02T05:53:08.382 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.383 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.383 DEBUG:boto:Params: {} 2022-09-02T05:53:08.384 DEBUG:boto:Token: None 2022-09-02T05:53:08.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdef75acf0725b05-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.389 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.390 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.391 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.391 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.392 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:08.392 DEBUG:boto:Data: 2022-09-02T05:53:08.393 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.393 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.394 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.394 DEBUG:boto:Params: {} 2022-09-02T05:53:08.395 DEBUG:boto:Token: None 2022-09-02T05:53:08.396 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33 2022-09-02T05:53:08.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B0G0IBHQYmm/6AGJaKbbrjbDUBM= 2022-09-02T05:53:08.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B0G0IBHQYmm/6AGJaKbbrjbDUBM=', 'Content-Length': '0'} 2022-09-02T05:53:08.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f50ac07dd59febe8-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.400 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.400 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.401 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.402 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:08.402 DEBUG:boto:Data: 2022-09-02T05:53:08.403 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.403 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.404 DEBUG:boto:Params: {} 2022-09-02T05:53:08.405 DEBUG:boto:Token: None 2022-09-02T05:53:08.405 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33 2022-09-02T05:53:08.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B0G0IBHQYmm/6AGJaKbbrjbDUBM= 2022-09-02T05:53:08.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B0G0IBHQYmm/6AGJaKbbrjbDUBM=', 'Content-Length': '0'} 2022-09-02T05:53:08.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536bdf01504d9719-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:08.410 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.411 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.411 DEBUG:boto:Method: GET 2022-09-02T05:53:08.412 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:08.412 DEBUG:boto:Data: 2022-09-02T05:53:08.413 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.414 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.414 DEBUG:boto:Params: {} 2022-09-02T05:53:08.415 DEBUG:boto:Token: None 2022-09-02T05:53:08.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33 2022-09-02T05:53:08.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6lqNhmfBErPvLekiLVZK9zFTvZw= 2022-09-02T05:53:08.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6lqNhmfBErPvLekiLVZK9zFTvZw=', 'Content-Length': '0'} 2022-09-02T05:53:08.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d8156bc36f8eff-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.419 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:08.420 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:08.420 DEBUG:boto:Method: GET 2022-09-02T05:53:08.421 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:08.421 DEBUG:boto:Data: 2022-09-02T05:53:08.422 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.423 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.423 DEBUG:boto:Params: {} 2022-09-02T05:53:08.424 DEBUG:boto:Token: None 2022-09-02T05:53:08.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj33 2022-09-02T05:53:08.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6lqNhmfBErPvLekiLVZK9zFTvZw= 2022-09-02T05:53:08.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6lqNhmfBErPvLekiLVZK9zFTvZw=', 'Content-Length': '0'} 2022-09-02T05:53:08.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1673806f9853eb7-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.429 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.429 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.430 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.430 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.431 DEBUG:boto:Method: GET 2022-09-02T05:53:08.431 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.432 DEBUG:boto:Data: 2022-09-02T05:53:08.432 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.433 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.433 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.434 DEBUG:boto:Params: {} 2022-09-02T05:53:08.434 DEBUG:boto:Token: None 2022-09-02T05:53:08.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e80b84eaf85c6d15-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.439 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.440 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.441 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.441 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.442 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.442 DEBUG:boto:Method: GET 2022-09-02T05:53:08.443 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:08.443 DEBUG:boto:Data: 2022-09-02T05:53:08.444 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.445 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.445 DEBUG:boto:Params: {} 2022-09-02T05:53:08.446 DEBUG:boto:Token: None 2022-09-02T05:53:08.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51a1e3977079432-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.451 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:08.453 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.453 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.454 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.454 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.455 DEBUG:boto:Method: GET 2022-09-02T05:53:08.455 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.456 DEBUG:boto:Data: 2022-09-02T05:53:08.456 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.457 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.458 DEBUG:boto:Params: {} 2022-09-02T05:53:08.459 DEBUG:boto:Token: None 2022-09-02T05:53:08.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY= 2022-09-02T05:53:08.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY=', 'Content-Length': '0'} 2022-09-02T05:53:08.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000847429f1bb0d07b6-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.506 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.507 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.508 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.509 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.509 DEBUG:boto:Method: GET 2022-09-02T05:53:08.510 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.511 DEBUG:boto:Data: 2022-09-02T05:53:08.511 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.512 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.513 DEBUG:boto:Params: {} 2022-09-02T05:53:08.514 DEBUG:boto:Token: None 2022-09-02T05:53:08.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY= 2022-09-02T05:53:08.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY=', 'Content-Length': '0'} 2022-09-02T05:53:08.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047081a9d2df757d6-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.518 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.519 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.519 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.520 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.520 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.521 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.521 DEBUG:boto:Data: 2022-09-02T05:53:08.522 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.523 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.523 DEBUG:boto:Params: {} 2022-09-02T05:53:08.524 DEBUG:boto:Token: None 2022-09-02T05:53:08.525 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SNUyjyrrvS8xSF28lEwXxlw4OTw= 2022-09-02T05:53:08.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SNUyjyrrvS8xSF28lEwXxlw4OTw=', 'Content-Length': '0'} 2022-09-02T05:53:08.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7ad087b236a3b9-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.529 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.529 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.530 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.531 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.531 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.532 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.532 DEBUG:boto:Data: 2022-09-02T05:53:08.533 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.534 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.535 DEBUG:boto:Params: {} 2022-09-02T05:53:08.535 DEBUG:boto:Token: None 2022-09-02T05:53:08.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SNUyjyrrvS8xSF28lEwXxlw4OTw= 2022-09-02T05:53:08.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SNUyjyrrvS8xSF28lEwXxlw4OTw=', 'Content-Length': '0'} 2022-09-02T05:53:08.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa0f7af2346d9bc-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:08.542 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.543 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.544 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.545 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.546 DEBUG:boto:Method: GET 2022-09-02T05:53:08.546 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.547 DEBUG:boto:Data: 2022-09-02T05:53:08.547 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.548 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.549 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.549 DEBUG:boto:Params: {} 2022-09-02T05:53:08.550 DEBUG:boto:Token: None 2022-09-02T05:53:08.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY= 2022-09-02T05:53:08.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY=', 'Content-Length': '0'} 2022-09-02T05:53:08.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec1c6714c92422a-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.555 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.555 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.556 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.556 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.557 DEBUG:boto:Method: GET 2022-09-02T05:53:08.558 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.558 DEBUG:boto:Data: 2022-09-02T05:53:08.559 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.560 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.560 DEBUG:boto:Params: {} 2022-09-02T05:53:08.561 DEBUG:boto:Token: None 2022-09-02T05:53:08.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:08.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY= 2022-09-02T05:53:08.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iO75bxkP9AtULjtdjDaHFlQ/imY=', 'Content-Length': '0'} 2022-09-02T05:53:08.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f556b3d5c91a4ba6-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.566 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.567 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.567 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.568 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.568 DEBUG:boto:Method: GET 2022-09-02T05:53:08.569 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.570 DEBUG:boto:Data: 2022-09-02T05:53:08.570 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.571 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.571 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.572 DEBUG:boto:Params: {} 2022-09-02T05:53:08.572 DEBUG:boto:Token: None 2022-09-02T05:53:08.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a37f9a978faa6c-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.577 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.578 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.579 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.581 DEBUG:boto:Method: GET 2022-09-02T05:53:08.581 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.582 DEBUG:boto:Data: 2022-09-02T05:53:08.582 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.583 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.584 DEBUG:boto:Params: {} 2022-09-02T05:53:08.585 DEBUG:boto:Token: None 2022-09-02T05:53:08.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edfcd3b22b6d0db3-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.591 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.591 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.592 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.592 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.593 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:08.593 DEBUG:boto:Data: 2022-09-02T05:53:08.594 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.595 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.595 DEBUG:boto:Params: {} 2022-09-02T05:53:08.596 DEBUG:boto:Token: None 2022-09-02T05:53:08.597 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34 2022-09-02T05:53:08.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2wes0fpt6Dk4OPf7C0QAjH4OVSY= 2022-09-02T05:53:08.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2wes0fpt6Dk4OPf7C0QAjH4OVSY=', 'Content-Length': '0'} 2022-09-02T05:53:08.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7783039696f1caf-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.601 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.601 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.602 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.602 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:08.603 DEBUG:boto:Data: 2022-09-02T05:53:08.603 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.604 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.605 DEBUG:boto:Params: {} 2022-09-02T05:53:08.605 DEBUG:boto:Token: None 2022-09-02T05:53:08.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34 2022-09-02T05:53:08.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2wes0fpt6Dk4OPf7C0QAjH4OVSY= 2022-09-02T05:53:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2wes0fpt6Dk4OPf7C0QAjH4OVSY=', 'Content-Length': '0'} 2022-09-02T05:53:08.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003292b24c2fa52523-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:08.611 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.611 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.612 DEBUG:boto:Method: GET 2022-09-02T05:53:08.613 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:08.613 DEBUG:boto:Data: 2022-09-02T05:53:08.614 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.614 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.615 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.615 DEBUG:boto:Params: {} 2022-09-02T05:53:08.616 DEBUG:boto:Token: None 2022-09-02T05:53:08.617 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34 2022-09-02T05:53:08.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ms0yfQT0Uwhk6xCqFCQu+95gxa4= 2022-09-02T05:53:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ms0yfQT0Uwhk6xCqFCQu+95gxa4=', 'Content-Length': '0'} 2022-09-02T05:53:08.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c419a0af4ca55938-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.621 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:08.622 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:08.622 DEBUG:boto:Method: GET 2022-09-02T05:53:08.623 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:08.623 DEBUG:boto:Data: 2022-09-02T05:53:08.624 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.625 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.626 DEBUG:boto:Params: {} 2022-09-02T05:53:08.626 DEBUG:boto:Token: None 2022-09-02T05:53:08.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj34 2022-09-02T05:53:08.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ms0yfQT0Uwhk6xCqFCQu+95gxa4= 2022-09-02T05:53:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ms0yfQT0Uwhk6xCqFCQu+95gxa4=', 'Content-Length': '0'} 2022-09-02T05:53:08.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a5b1cd21e29944-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.631 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.631 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.632 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.632 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.633 DEBUG:boto:Method: GET 2022-09-02T05:53:08.633 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.634 DEBUG:boto:Data: 2022-09-02T05:53:08.635 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.636 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.636 DEBUG:boto:Params: {} 2022-09-02T05:53:08.637 DEBUG:boto:Token: None 2022-09-02T05:53:08.637 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087eff615964c79b1-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.642 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.643 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.644 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.645 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.645 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.646 DEBUG:boto:Method: GET 2022-09-02T05:53:08.646 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:08.647 DEBUG:boto:Data: 2022-09-02T05:53:08.647 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.648 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.649 DEBUG:boto:Params: {} 2022-09-02T05:53:08.650 DEBUG:boto:Token: None 2022-09-02T05:53:08.650 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513552aa42b87eae-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.702 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:08.705 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.706 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.707 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.708 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.709 DEBUG:boto:Method: GET 2022-09-02T05:53:08.710 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.711 DEBUG:boto:Data: 2022-09-02T05:53:08.712 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.714 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.715 DEBUG:boto:Params: {} 2022-09-02T05:53:08.716 DEBUG:boto:Token: None 2022-09-02T05:53:08.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I= 2022-09-02T05:53:08.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I=', 'Content-Length': '0'} 2022-09-02T05:53:08.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7da4e8f5b50edc1-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.723 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.723 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.724 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.724 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.725 DEBUG:boto:Method: GET 2022-09-02T05:53:08.725 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.726 DEBUG:boto:Data: 2022-09-02T05:53:08.726 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.728 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.728 DEBUG:boto:Params: {} 2022-09-02T05:53:08.729 DEBUG:boto:Token: None 2022-09-02T05:53:08.730 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I= 2022-09-02T05:53:08.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I=', 'Content-Length': '0'} 2022-09-02T05:53:08.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cd0ec0d3d4c3c3-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.741 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.742 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.742 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.743 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.743 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.744 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.744 DEBUG:boto:Data: 2022-09-02T05:53:08.745 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.745 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.746 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.747 DEBUG:boto:Params: {} 2022-09-02T05:53:08.748 DEBUG:boto:Token: None 2022-09-02T05:53:08.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/c2TYh3kKHhlkwWoxJ+GYXljEQw= 2022-09-02T05:53:08.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/c2TYh3kKHhlkwWoxJ+GYXljEQw=', 'Content-Length': '0'} 2022-09-02T05:53:08.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b103d519212a26fc-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.753 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.754 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.754 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.755 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.755 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.756 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.756 DEBUG:boto:Data: 2022-09-02T05:53:08.757 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.758 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.759 DEBUG:boto:Params: {} 2022-09-02T05:53:08.760 DEBUG:boto:Token: None 2022-09-02T05:53:08.761 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/c2TYh3kKHhlkwWoxJ+GYXljEQw= 2022-09-02T05:53:08.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/c2TYh3kKHhlkwWoxJ+GYXljEQw=', 'Content-Length': '0'} 2022-09-02T05:53:08.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c45ab08b2a91831-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:08.767 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.768 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.768 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.769 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.769 DEBUG:boto:Method: GET 2022-09-02T05:53:08.770 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.770 DEBUG:boto:Data: 2022-09-02T05:53:08.771 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.772 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.773 DEBUG:boto:Params: {} 2022-09-02T05:53:08.773 DEBUG:boto:Token: None 2022-09-02T05:53:08.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I= 2022-09-02T05:53:08.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I=', 'Content-Length': '0'} 2022-09-02T05:53:08.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffcaabe0a68daa8d-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.778 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.779 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.780 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.780 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.781 DEBUG:boto:Method: GET 2022-09-02T05:53:08.783 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.784 DEBUG:boto:Data: 2022-09-02T05:53:08.784 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.785 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.786 DEBUG:boto:Params: {} 2022-09-02T05:53:08.786 DEBUG:boto:Token: None 2022-09-02T05:53:08.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:08.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I= 2022-09-02T05:53:08.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MXuyheTdWq7OrqJPA9pfd95TZ8I=', 'Content-Length': '0'} 2022-09-02T05:53:08.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2e9b69e900dc48-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.792 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.792 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.793 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.794 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.794 DEBUG:boto:Method: GET 2022-09-02T05:53:08.795 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.795 DEBUG:boto:Data: 2022-09-02T05:53:08.796 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.797 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.798 DEBUG:boto:Params: {} 2022-09-02T05:53:08.798 DEBUG:boto:Token: None 2022-09-02T05:53:08.799 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a519c4e3e9364c7-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.804 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.805 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.805 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.806 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.807 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.807 DEBUG:boto:Method: GET 2022-09-02T05:53:08.808 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.809 DEBUG:boto:Data: 2022-09-02T05:53:08.809 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.811 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.811 DEBUG:boto:Params: {} 2022-09-02T05:53:08.812 DEBUG:boto:Token: None 2022-09-02T05:53:08.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a65ac63089ac9ee-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.817 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.819 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.819 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.820 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.820 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:08.821 DEBUG:boto:Data: 2022-09-02T05:53:08.821 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.822 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.823 DEBUG:boto:Params: {} 2022-09-02T05:53:08.823 DEBUG:boto:Token: None 2022-09-02T05:53:08.824 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38 2022-09-02T05:53:08.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kd6skOhqgjE+ReaIEoGQizbPw74= 2022-09-02T05:53:08.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kd6skOhqgjE+ReaIEoGQizbPw74=', 'Content-Length': '0'} 2022-09-02T05:53:08.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003518ca8688ce1823-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.828 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.829 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.830 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.830 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:08.831 DEBUG:boto:Data: 2022-09-02T05:53:08.831 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.832 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.833 DEBUG:boto:Params: {} 2022-09-02T05:53:08.833 DEBUG:boto:Token: None 2022-09-02T05:53:08.834 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38 2022-09-02T05:53:08.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kd6skOhqgjE+ReaIEoGQizbPw74= 2022-09-02T05:53:08.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kd6skOhqgjE+ReaIEoGQizbPw74=', 'Content-Length': '0'} 2022-09-02T05:53:08.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb409a70cfdfd9c-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:08.838 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.839 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.840 DEBUG:boto:Method: GET 2022-09-02T05:53:08.841 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:08.841 DEBUG:boto:Data: 2022-09-02T05:53:08.842 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.843 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.844 DEBUG:boto:Params: {} 2022-09-02T05:53:08.844 DEBUG:boto:Token: None 2022-09-02T05:53:08.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38 2022-09-02T05:53:08.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eyEtJrib+p/H+dAzzYucQ6ujAyk= 2022-09-02T05:53:08.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eyEtJrib+p/H+dAzzYucQ6ujAyk=', 'Content-Length': '0'} 2022-09-02T05:53:08.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000845adac7403965d3-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.890 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:08.891 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:08.891 DEBUG:boto:Method: GET 2022-09-02T05:53:08.892 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:08.893 DEBUG:boto:Data: 2022-09-02T05:53:08.894 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.896 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.897 DEBUG:boto:Params: {} 2022-09-02T05:53:08.899 DEBUG:boto:Token: None 2022-09-02T05:53:08.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj38 2022-09-02T05:53:08.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eyEtJrib+p/H+dAzzYucQ6ujAyk= 2022-09-02T05:53:08.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eyEtJrib+p/H+dAzzYucQ6ujAyk=', 'Content-Length': '0'} 2022-09-02T05:53:08.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad63d48034a3e0c-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.906 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.907 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.907 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.908 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.909 DEBUG:boto:Method: GET 2022-09-02T05:53:08.909 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.910 DEBUG:boto:Data: 2022-09-02T05:53:08.910 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.911 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.911 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.912 DEBUG:boto:Params: {} 2022-09-02T05:53:08.912 DEBUG:boto:Token: None 2022-09-02T05:53:08.913 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000210ce55927177bec-0063119a44-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.918 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.919 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:08.920 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:08.920 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.921 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.922 DEBUG:boto:Method: GET 2022-09-02T05:53:08.922 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:08.923 DEBUG:boto:Data: 2022-09-02T05:53:08.924 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.925 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.926 DEBUG:boto:Params: {} 2022-09-02T05:53:08.926 DEBUG:boto:Token: None 2022-09-02T05:53:08.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/ 2022-09-02T05:53:08.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI= 2022-09-02T05:53:08.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB://EBS4uGXYtuoRqk3CLM5rgOwEI=', 'Content-Length': '0'} 2022-09-02T05:53:08.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001477be70b111364f-0063119a44-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.934 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:08.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:08.937 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:08.938 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:08.939 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.940 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.941 DEBUG:boto:Method: GET 2022-09-02T05:53:08.941 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.942 DEBUG:boto:Data: 2022-09-02T05:53:08.943 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.944 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.944 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.946 DEBUG:boto:Params: {} 2022-09-02T05:53:08.947 DEBUG:boto:Token: None 2022-09-02T05:53:08.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nWavJPaEsz62bRddsZ5sEB4Dc2c= 2022-09-02T05:53:08.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nWavJPaEsz62bRddsZ5sEB4Dc2c=', 'Content-Length': '0'} 2022-09-02T05:53:08.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d6b7dd3a2b4ddb-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.954 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:08.955 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:08.955 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.956 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.957 DEBUG:boto:Method: GET 2022-09-02T05:53:08.958 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.958 DEBUG:boto:Data: 2022-09-02T05:53:08.959 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.960 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.961 DEBUG:boto:Params: {} 2022-09-02T05:53:08.962 DEBUG:boto:Token: None 2022-09-02T05:53:08.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nWavJPaEsz62bRddsZ5sEB4Dc2c= 2022-09-02T05:53:08.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nWavJPaEsz62bRddsZ5sEB4Dc2c=', 'Content-Length': '0'} 2022-09-02T05:53:08.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f7f1cc3a69effc-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.970 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:08.970 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:08.971 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.971 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.972 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.973 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.973 DEBUG:boto:Data: 2022-09-02T05:53:08.974 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:08.975 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.976 DEBUG:boto:Params: {} 2022-09-02T05:53:08.977 DEBUG:boto:Token: None 2022-09-02T05:53:08.977 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkewMN/mrkcHR6EnAfDpMqXWwQ4= 2022-09-02T05:53:08.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkewMN/mrkcHR6EnAfDpMqXWwQ4=', 'Content-Length': '0'} 2022-09-02T05:53:08.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018695d5af93ae7fe-0063119a44-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:08.984 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:08.985 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:08.985 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.986 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.987 DEBUG:boto:Method: HEAD 2022-09-02T05:53:08.988 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.989 DEBUG:boto:Data: 2022-09-02T05:53:08.990 DEBUG:boto:Headers: {} 2022-09-02T05:53:08.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:08.991 DEBUG:boto:Port: 8000 2022-09-02T05:53:08.992 DEBUG:boto:Params: {} 2022-09-02T05:53:08.993 DEBUG:boto:Token: None 2022-09-02T05:53:08.994 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:08 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:08.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkewMN/mrkcHR6EnAfDpMqXWwQ4= 2022-09-02T05:53:08.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkewMN/mrkcHR6EnAfDpMqXWwQ4=', 'Content-Length': '0'} 2022-09-02T05:53:09.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a078735385d7c26d-0063119a44-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:09.003 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:09.003 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:09.004 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.005 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.005 DEBUG:boto:Method: GET 2022-09-02T05:53:09.006 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.006 DEBUG:boto:Data: 2022-09-02T05:53:09.007 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.009 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.010 DEBUG:boto:Params: {} 2022-09-02T05:53:09.011 DEBUG:boto:Token: None 2022-09-02T05:53:09.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bj7TNh9wP+YcCmbnYBsJdde6Q3A= 2022-09-02T05:53:09.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bj7TNh9wP+YcCmbnYBsJdde6Q3A=', 'Content-Length': '0'} 2022-09-02T05:53:09.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6663b3d8a15342-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.023 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:09.024 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:09.025 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.026 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.026 DEBUG:boto:Method: GET 2022-09-02T05:53:09.027 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.028 DEBUG:boto:Data: 2022-09-02T05:53:09.029 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.031 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.031 DEBUG:boto:Params: {} 2022-09-02T05:53:09.032 DEBUG:boto:Token: None 2022-09-02T05:53:09.033 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:09.034 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bj7TNh9wP+YcCmbnYBsJdde6Q3A= 2022-09-02T05:53:09.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bj7TNh9wP+YcCmbnYBsJdde6Q3A=', 'Content-Length': '0'} 2022-09-02T05:53:09.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c91213225f0df76-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.078 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.079 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.080 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.081 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.082 DEBUG:boto:Method: GET 2022-09-02T05:53:09.083 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.084 DEBUG:boto:Data: 2022-09-02T05:53:09.085 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.087 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.087 DEBUG:boto:Params: {} 2022-09-02T05:53:09.088 DEBUG:boto:Token: None 2022-09-02T05:53:09.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a8d9738a15144f5-0063119a45-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.094 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.096 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.096 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.097 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.097 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.098 DEBUG:boto:Method: GET 2022-09-02T05:53:09.099 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.099 DEBUG:boto:Data: 2022-09-02T05:53:09.100 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.102 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.102 DEBUG:boto:Params: {} 2022-09-02T05:53:09.103 DEBUG:boto:Token: None 2022-09-02T05:53:09.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ce2b6fa5d2c79c8-0063119a45-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.118 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.119 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:09.120 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:09.121 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.121 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:09.122 DEBUG:boto:Data: 2022-09-02T05:53:09.123 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.124 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.125 DEBUG:boto:Params: {} 2022-09-02T05:53:09.126 DEBUG:boto:Token: None 2022-09-02T05:53:09.127 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj44 2022-09-02T05:53:09.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eeE+yg/X4FVOm59313kWMegfm0o= 2022-09-02T05:53:09.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eeE+yg/X4FVOm59313kWMegfm0o=', 'Content-Length': '0'} 2022-09-02T05:53:09.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568fcc6e9b2f02d3-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.139 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:09.139 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:09.140 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.140 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:09.141 DEBUG:boto:Data: 2022-09-02T05:53:09.142 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.143 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.143 DEBUG:boto:Params: {} 2022-09-02T05:53:09.144 DEBUG:boto:Token: None 2022-09-02T05:53:09.144 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj44 2022-09-02T05:53:09.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eeE+yg/X4FVOm59313kWMegfm0o= 2022-09-02T05:53:09.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eeE+yg/X4FVOm59313kWMegfm0o=', 'Content-Length': '0'} 2022-09-02T05:53:09.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017dbbf33585cf982-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:09.149 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:09.150 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:09.150 DEBUG:boto:Method: GET 2022-09-02T05:53:09.151 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:09.152 DEBUG:boto:Data: 2022-09-02T05:53:09.152 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.153 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.154 DEBUG:boto:Params: {} 2022-09-02T05:53:09.155 DEBUG:boto:Token: None 2022-09-02T05:53:09.155 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj44 2022-09-02T05:53:09.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1OFH0tagoh1c269YQ19Sh9pUGT0= 2022-09-02T05:53:09.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1OFH0tagoh1c269YQ19Sh9pUGT0=', 'Content-Length': '0'} 2022-09-02T05:53:09.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f17c51f26bf81be-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.206 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:09.207 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:09.208 DEBUG:boto:Method: GET 2022-09-02T05:53:09.209 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:09.210 DEBUG:boto:Data: 2022-09-02T05:53:09.212 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.213 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.214 DEBUG:boto:Params: {} 2022-09-02T05:53:09.215 DEBUG:boto:Token: None 2022-09-02T05:53:09.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj44 2022-09-02T05:53:09.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1OFH0tagoh1c269YQ19Sh9pUGT0= 2022-09-02T05:53:09.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1OFH0tagoh1c269YQ19Sh9pUGT0=', 'Content-Length': '0'} 2022-09-02T05:53:09.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e415c48511de9989-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.220 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.221 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.221 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.222 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.222 DEBUG:boto:Method: GET 2022-09-02T05:53:09.223 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.223 DEBUG:boto:Data: 2022-09-02T05:53:09.224 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.225 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.225 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.226 DEBUG:boto:Params: {} 2022-09-02T05:53:09.226 DEBUG:boto:Token: None 2022-09-02T05:53:09.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeae9de4d9489390-0063119a45-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.232 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.233 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.234 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.234 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.235 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.236 DEBUG:boto:Method: GET 2022-09-02T05:53:09.236 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:09.237 DEBUG:boto:Data: 2022-09-02T05:53:09.238 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.239 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.240 DEBUG:boto:Params: {} 2022-09-02T05:53:09.241 DEBUG:boto:Token: None 2022-09-02T05:53:09.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b44243fa03f460f-0063119a45-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.294 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:09.296 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.297 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.298 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.299 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.300 DEBUG:boto:Method: GET 2022-09-02T05:53:09.301 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.302 DEBUG:boto:Data: 2022-09-02T05:53:09.303 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.304 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.305 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.306 DEBUG:boto:Params: {} 2022-09-02T05:53:09.307 DEBUG:boto:Token: None 2022-09-02T05:53:09.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo= 2022-09-02T05:53:09.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo=', 'Content-Length': '0'} 2022-09-02T05:53:09.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4817c006ffe432-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.315 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.316 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.317 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.318 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.319 DEBUG:boto:Method: GET 2022-09-02T05:53:09.320 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.321 DEBUG:boto:Data: 2022-09-02T05:53:09.322 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.324 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.324 DEBUG:boto:Params: {} 2022-09-02T05:53:09.326 DEBUG:boto:Token: None 2022-09-02T05:53:09.327 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo= 2022-09-02T05:53:09.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo=', 'Content-Length': '0'} 2022-09-02T05:53:09.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476ac644c853ad26-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.335 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.335 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.336 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.337 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.338 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.339 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.340 DEBUG:boto:Data: 2022-09-02T05:53:09.341 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.343 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.344 DEBUG:boto:Params: {} 2022-09-02T05:53:09.345 DEBUG:boto:Token: None 2022-09-02T05:53:09.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Tdpr3z/+9Ca2xh6vATv6zKSIA/0= 2022-09-02T05:53:09.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Tdpr3z/+9Ca2xh6vATv6zKSIA/0=', 'Content-Length': '0'} 2022-09-02T05:53:09.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcc5269db3c3769d-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.353 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.354 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.355 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.355 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.356 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.357 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.358 DEBUG:boto:Data: 2022-09-02T05:53:09.359 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.361 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.362 DEBUG:boto:Params: {} 2022-09-02T05:53:09.363 DEBUG:boto:Token: None 2022-09-02T05:53:09.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Tdpr3z/+9Ca2xh6vATv6zKSIA/0= 2022-09-02T05:53:09.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Tdpr3z/+9Ca2xh6vATv6zKSIA/0=', 'Content-Length': '0'} 2022-09-02T05:53:09.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5be2ee37b316f6-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:09.371 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.372 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.373 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.374 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.375 DEBUG:boto:Method: GET 2022-09-02T05:53:09.376 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.377 DEBUG:boto:Data: 2022-09-02T05:53:09.377 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.378 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.379 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.380 DEBUG:boto:Params: {} 2022-09-02T05:53:09.381 DEBUG:boto:Token: None 2022-09-02T05:53:09.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo= 2022-09-02T05:53:09.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo=', 'Content-Length': '0'} 2022-09-02T05:53:09.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aa538fc9febfb0e-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.389 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.390 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.391 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.392 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.393 DEBUG:boto:Method: GET 2022-09-02T05:53:09.394 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.395 DEBUG:boto:Data: 2022-09-02T05:53:09.396 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.398 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.398 DEBUG:boto:Params: {} 2022-09-02T05:53:09.399 DEBUG:boto:Token: None 2022-09-02T05:53:09.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:09.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo= 2022-09-02T05:53:09.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RLBW3cU/FNl05MZMjDQHQAJrZRo=', 'Content-Length': '0'} 2022-09-02T05:53:09.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041cecf2559c919ca-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.405 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.406 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.406 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.407 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.407 DEBUG:boto:Method: GET 2022-09-02T05:53:09.408 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.409 DEBUG:boto:Data: 2022-09-02T05:53:09.409 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.410 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.411 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.411 DEBUG:boto:Params: {} 2022-09-02T05:53:09.412 DEBUG:boto:Token: None 2022-09-02T05:53:09.414 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee77dadac3a6d966-0063119a45-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.420 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.421 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.422 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.423 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.423 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.424 DEBUG:boto:Method: GET 2022-09-02T05:53:09.425 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.425 DEBUG:boto:Data: 2022-09-02T05:53:09.426 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.427 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.428 DEBUG:boto:Params: {} 2022-09-02T05:53:09.429 DEBUG:boto:Token: None 2022-09-02T05:53:09.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa689a36e92a8be3-0063119a45-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.437 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.438 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.439 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.440 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.440 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:09.441 DEBUG:boto:Data: 2022-09-02T05:53:09.441 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.443 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.443 DEBUG:boto:Params: {} 2022-09-02T05:53:09.444 DEBUG:boto:Token: None 2022-09-02T05:53:09.445 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46 2022-09-02T05:53:09.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U1Ddx1yESCESM/YUsz5VuThFq2k= 2022-09-02T05:53:09.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U1Ddx1yESCESM/YUsz5VuThFq2k=', 'Content-Length': '0'} 2022-09-02T05:53:09.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2cda58ceb61a08-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.450 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.450 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.451 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.451 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:09.452 DEBUG:boto:Data: 2022-09-02T05:53:09.453 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.454 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.455 DEBUG:boto:Params: {} 2022-09-02T05:53:09.456 DEBUG:boto:Token: None 2022-09-02T05:53:09.457 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46 2022-09-02T05:53:09.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U1Ddx1yESCESM/YUsz5VuThFq2k= 2022-09-02T05:53:09.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U1Ddx1yESCESM/YUsz5VuThFq2k=', 'Content-Length': '0'} 2022-09-02T05:53:09.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000968eb7d80d571269-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:09.463 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.464 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.465 DEBUG:boto:Method: GET 2022-09-02T05:53:09.466 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:09.466 DEBUG:boto:Data: 2022-09-02T05:53:09.467 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.468 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.468 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.469 DEBUG:boto:Params: {} 2022-09-02T05:53:09.470 DEBUG:boto:Token: None 2022-09-02T05:53:09.470 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46 2022-09-02T05:53:09.472 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9LZo7AA2FLGkhSeX+NZp/NtLNXI= 2022-09-02T05:53:09.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9LZo7AA2FLGkhSeX+NZp/NtLNXI=', 'Content-Length': '0'} 2022-09-02T05:53:09.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006769e06f12755f67-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.528 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:09.529 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:09.530 DEBUG:boto:Method: GET 2022-09-02T05:53:09.531 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:09.532 DEBUG:boto:Data: 2022-09-02T05:53:09.533 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.534 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.535 DEBUG:boto:Params: {} 2022-09-02T05:53:09.537 DEBUG:boto:Token: None 2022-09-02T05:53:09.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj46 2022-09-02T05:53:09.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9LZo7AA2FLGkhSeX+NZp/NtLNXI= 2022-09-02T05:53:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9LZo7AA2FLGkhSeX+NZp/NtLNXI=', 'Content-Length': '0'} 2022-09-02T05:53:09.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6abbf2eb0a07ca-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.543 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.544 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.545 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.545 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.546 DEBUG:boto:Method: GET 2022-09-02T05:53:09.546 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.547 DEBUG:boto:Data: 2022-09-02T05:53:09.548 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.549 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.550 DEBUG:boto:Params: {} 2022-09-02T05:53:09.551 DEBUG:boto:Token: None 2022-09-02T05:53:09.552 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a72680e306e11c-0063119a45-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.559 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.560 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.561 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.562 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.563 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.564 DEBUG:boto:Method: GET 2022-09-02T05:53:09.565 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:09.566 DEBUG:boto:Data: 2022-09-02T05:53:09.566 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.568 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.569 DEBUG:boto:Params: {} 2022-09-02T05:53:09.570 DEBUG:boto:Token: None 2022-09-02T05:53:09.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066c4499ca66a0e64-0063119a45-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.577 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:09.579 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.580 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.580 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.581 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.582 DEBUG:boto:Method: GET 2022-09-02T05:53:09.583 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.584 DEBUG:boto:Data: 2022-09-02T05:53:09.585 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.586 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.586 DEBUG:boto:Params: {} 2022-09-02T05:53:09.587 DEBUG:boto:Token: None 2022-09-02T05:53:09.588 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8= 2022-09-02T05:53:09.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8=', 'Content-Length': '0'} 2022-09-02T05:53:09.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e118faa2d97d537f-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.593 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.594 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.595 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.595 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.596 DEBUG:boto:Method: GET 2022-09-02T05:53:09.596 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.597 DEBUG:boto:Data: 2022-09-02T05:53:09.598 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.599 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.599 DEBUG:boto:Params: {} 2022-09-02T05:53:09.600 DEBUG:boto:Token: None 2022-09-02T05:53:09.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8= 2022-09-02T05:53:09.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8=', 'Content-Length': '0'} 2022-09-02T05:53:09.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057888d322dbb087b-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.606 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.606 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.607 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.608 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.608 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.609 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.609 DEBUG:boto:Data: 2022-09-02T05:53:09.610 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.611 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.611 DEBUG:boto:Params: {} 2022-09-02T05:53:09.612 DEBUG:boto:Token: None 2022-09-02T05:53:09.613 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4C0wocHkK7Dk5WOREjPxP4yD+CI= 2022-09-02T05:53:09.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4C0wocHkK7Dk5WOREjPxP4yD+CI=', 'Content-Length': '0'} 2022-09-02T05:53:09.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008faf6ec34e666df0-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.619 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.620 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.621 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.623 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.624 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.625 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.626 DEBUG:boto:Data: 2022-09-02T05:53:09.626 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.628 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.629 DEBUG:boto:Params: {} 2022-09-02T05:53:09.630 DEBUG:boto:Token: None 2022-09-02T05:53:09.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4C0wocHkK7Dk5WOREjPxP4yD+CI= 2022-09-02T05:53:09.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4C0wocHkK7Dk5WOREjPxP4yD+CI=', 'Content-Length': '0'} 2022-09-02T05:53:09.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1ed49385b1f2d3-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:09.636 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.637 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.638 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.639 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.640 DEBUG:boto:Method: GET 2022-09-02T05:53:09.640 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.641 DEBUG:boto:Data: 2022-09-02T05:53:09.642 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.642 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.643 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.644 DEBUG:boto:Params: {} 2022-09-02T05:53:09.645 DEBUG:boto:Token: None 2022-09-02T05:53:09.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8= 2022-09-02T05:53:09.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8=', 'Content-Length': '0'} 2022-09-02T05:53:09.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c53e0b0fc3adb0d-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.657 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.657 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.658 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.659 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.659 DEBUG:boto:Method: GET 2022-09-02T05:53:09.660 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.660 DEBUG:boto:Data: 2022-09-02T05:53:09.661 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.663 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.663 DEBUG:boto:Params: {} 2022-09-02T05:53:09.664 DEBUG:boto:Token: None 2022-09-02T05:53:09.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:09.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8= 2022-09-02T05:53:09.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aNvDABOVBnJFmf3W4KJ8hQ+3WO8=', 'Content-Length': '0'} 2022-09-02T05:53:09.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d2f5440c801e7f-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.714 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.715 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.715 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.716 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.716 DEBUG:boto:Method: GET 2022-09-02T05:53:09.717 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.717 DEBUG:boto:Data: 2022-09-02T05:53:09.718 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.719 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.720 DEBUG:boto:Params: {} 2022-09-02T05:53:09.720 DEBUG:boto:Token: None 2022-09-02T05:53:09.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c951b9f10477d2-0063119a45-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.727 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.727 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.728 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.729 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.729 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.730 DEBUG:boto:Method: GET 2022-09-02T05:53:09.731 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.731 DEBUG:boto:Data: 2022-09-02T05:53:09.732 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.733 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.733 DEBUG:boto:Params: {} 2022-09-02T05:53:09.734 DEBUG:boto:Token: None 2022-09-02T05:53:09.734 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002344460b219487a6-0063119a45-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.740 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.741 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.742 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.742 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.743 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:09.743 DEBUG:boto:Data: 2022-09-02T05:53:09.744 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.745 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.745 DEBUG:boto:Params: {} 2022-09-02T05:53:09.746 DEBUG:boto:Token: None 2022-09-02T05:53:09.747 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51 2022-09-02T05:53:09.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G2wZf75F6YKYbYF8420HUZiOtJw= 2022-09-02T05:53:09.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G2wZf75F6YKYbYF8420HUZiOtJw=', 'Content-Length': '0'} 2022-09-02T05:53:09.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091f08d22c7c4165b-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.776 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.777 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.777 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.778 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:09.778 DEBUG:boto:Data: 2022-09-02T05:53:09.779 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.780 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.781 DEBUG:boto:Params: {} 2022-09-02T05:53:09.781 DEBUG:boto:Token: None 2022-09-02T05:53:09.782 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51 2022-09-02T05:53:09.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G2wZf75F6YKYbYF8420HUZiOtJw= 2022-09-02T05:53:09.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G2wZf75F6YKYbYF8420HUZiOtJw=', 'Content-Length': '0'} 2022-09-02T05:53:09.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021a7a70404edf8f9-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:09.793 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.794 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.794 DEBUG:boto:Method: GET 2022-09-02T05:53:09.795 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:09.796 DEBUG:boto:Data: 2022-09-02T05:53:09.796 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.798 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.798 DEBUG:boto:Params: {} 2022-09-02T05:53:09.799 DEBUG:boto:Token: None 2022-09-02T05:53:09.800 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51 2022-09-02T05:53:09.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uWTHHXE0zyJcqGvfFcZJ8FzEgE= 2022-09-02T05:53:09.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uWTHHXE0zyJcqGvfFcZJ8FzEgE=', 'Content-Length': '0'} 2022-09-02T05:53:09.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b90eabc8003a418-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.850 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:09.851 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:09.852 DEBUG:boto:Method: GET 2022-09-02T05:53:09.853 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:09.853 DEBUG:boto:Data: 2022-09-02T05:53:09.854 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.856 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.856 DEBUG:boto:Params: {} 2022-09-02T05:53:09.857 DEBUG:boto:Token: None 2022-09-02T05:53:09.858 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj51 2022-09-02T05:53:09.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/uWTHHXE0zyJcqGvfFcZJ8FzEgE= 2022-09-02T05:53:09.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/uWTHHXE0zyJcqGvfFcZJ8FzEgE=', 'Content-Length': '0'} 2022-09-02T05:53:09.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051151d0abdf85ff2-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.863 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.864 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.864 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.865 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.865 DEBUG:boto:Method: GET 2022-09-02T05:53:09.866 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.866 DEBUG:boto:Data: 2022-09-02T05:53:09.867 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.868 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.869 DEBUG:boto:Params: {} 2022-09-02T05:53:09.869 DEBUG:boto:Token: None 2022-09-02T05:53:09.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30eb96cab26d34e-0063119a45-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.875 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.876 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:09.877 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:09.877 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.878 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.879 DEBUG:boto:Method: GET 2022-09-02T05:53:09.879 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:09.880 DEBUG:boto:Data: 2022-09-02T05:53:09.881 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.882 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.882 DEBUG:boto:Params: {} 2022-09-02T05:53:09.883 DEBUG:boto:Token: None 2022-09-02T05:53:09.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/ 2022-09-02T05:53:09.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM= 2022-09-02T05:53:09.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IruXF5qnYrEe/VMenW0oqfurnKM=', 'Content-Length': '0'} 2022-09-02T05:53:09.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e18b91079d9a2d9-0063119a45-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.892 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:09.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:09.895 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:09.897 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:09.899 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.900 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.901 DEBUG:boto:Method: GET 2022-09-02T05:53:09.901 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.902 DEBUG:boto:Data: 2022-09-02T05:53:09.903 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.904 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.904 DEBUG:boto:Params: {} 2022-09-02T05:53:09.905 DEBUG:boto:Token: None 2022-09-02T05:53:09.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S+OlHKiTG4o7XSe33gzShEN3XJk= 2022-09-02T05:53:09.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S+OlHKiTG4o7XSe33gzShEN3XJk=', 'Content-Length': '0'} 2022-09-02T05:53:09.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da43ac1b38a542d2-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.910 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:09.911 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:09.912 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.912 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.913 DEBUG:boto:Method: GET 2022-09-02T05:53:09.913 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.914 DEBUG:boto:Data: 2022-09-02T05:53:09.914 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.915 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.916 DEBUG:boto:Params: {} 2022-09-02T05:53:09.917 DEBUG:boto:Token: None 2022-09-02T05:53:09.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S+OlHKiTG4o7XSe33gzShEN3XJk= 2022-09-02T05:53:09.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S+OlHKiTG4o7XSe33gzShEN3XJk=', 'Content-Length': '0'} 2022-09-02T05:53:09.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8963dd41d4bbb1-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.966 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:09.967 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:09.967 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.968 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.969 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.970 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.970 DEBUG:boto:Data: 2022-09-02T05:53:09.971 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.972 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.973 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.973 DEBUG:boto:Params: {} 2022-09-02T05:53:09.974 DEBUG:boto:Token: None 2022-09-02T05:53:09.975 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CyVLzIsD2TzQzqdovwlmxlBL8D0= 2022-09-02T05:53:09.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CyVLzIsD2TzQzqdovwlmxlBL8D0=', 'Content-Length': '0'} 2022-09-02T05:53:09.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fee59fa12f34f71-0063119a45-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.979 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:09.980 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:09.981 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.982 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.982 DEBUG:boto:Method: HEAD 2022-09-02T05:53:09.983 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.983 DEBUG:boto:Data: 2022-09-02T05:53:09.984 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:09.985 DEBUG:boto:Port: 8000 2022-09-02T05:53:09.985 DEBUG:boto:Params: {} 2022-09-02T05:53:09.986 DEBUG:boto:Token: None 2022-09-02T05:53:09.987 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:09 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CyVLzIsD2TzQzqdovwlmxlBL8D0= 2022-09-02T05:53:09.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CyVLzIsD2TzQzqdovwlmxlBL8D0=', 'Content-Length': '0'} 2022-09-02T05:53:09.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031e635ccba142abd-0063119a45-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:09.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:09.994 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:09.995 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:09.995 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.996 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.996 DEBUG:boto:Method: GET 2022-09-02T05:53:09.997 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:09.998 DEBUG:boto:Data: 2022-09-02T05:53:09.998 DEBUG:boto:Headers: {} 2022-09-02T05:53:09.999 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:09.999 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.000 DEBUG:boto:Params: {} 2022-09-02T05:53:10.000 DEBUG:boto:Token: None 2022-09-02T05:53:10.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:10.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+P66eau5cpTOjfjT47otMEJgEFs= 2022-09-02T05:53:10.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+P66eau5cpTOjfjT47otMEJgEFs=', 'Content-Length': '0'} 2022-09-02T05:53:10.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c07286ae3c226fb-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.009 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:10.010 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:10.010 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:10.011 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:10.011 DEBUG:boto:Method: GET 2022-09-02T05:53:10.012 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:10.013 DEBUG:boto:Data: 2022-09-02T05:53:10.013 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.015 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.015 DEBUG:boto:Params: {} 2022-09-02T05:53:10.016 DEBUG:boto:Token: None 2022-09-02T05:53:10.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:10.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+P66eau5cpTOjfjT47otMEJgEFs= 2022-09-02T05:53:10.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+P66eau5cpTOjfjT47otMEJgEFs=', 'Content-Length': '0'} 2022-09-02T05:53:10.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a36d9ed6571b64-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.022 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.023 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.024 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.024 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.025 DEBUG:boto:Method: GET 2022-09-02T05:53:10.025 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.026 DEBUG:boto:Data: 2022-09-02T05:53:10.026 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.027 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.028 DEBUG:boto:Params: {} 2022-09-02T05:53:10.028 DEBUG:boto:Token: None 2022-09-02T05:53:10.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5abe7cffb544041-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.078 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.080 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.080 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.081 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.081 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.082 DEBUG:boto:Method: GET 2022-09-02T05:53:10.082 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.083 DEBUG:boto:Data: 2022-09-02T05:53:10.084 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.085 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.085 DEBUG:boto:Params: {} 2022-09-02T05:53:10.086 DEBUG:boto:Token: None 2022-09-02T05:53:10.087 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d86af307496e2a4-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.094 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.095 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:10.095 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:10.096 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.096 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:10.097 DEBUG:boto:Data: 2022-09-02T05:53:10.097 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.098 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.099 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.100 DEBUG:boto:Params: {} 2022-09-02T05:53:10.101 DEBUG:boto:Token: None 2022-09-02T05:53:10.102 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj55 2022-09-02T05:53:10.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cnezdWGV00fFckVee6StudofnYk= 2022-09-02T05:53:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cnezdWGV00fFckVee6StudofnYk=', 'Content-Length': '0'} 2022-09-02T05:53:10.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfda3da422c4626-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.107 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:10.107 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:10.108 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.109 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:10.109 DEBUG:boto:Data: 2022-09-02T05:53:10.110 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.111 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.112 DEBUG:boto:Params: {} 2022-09-02T05:53:10.113 DEBUG:boto:Token: None 2022-09-02T05:53:10.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj55 2022-09-02T05:53:10.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cnezdWGV00fFckVee6StudofnYk= 2022-09-02T05:53:10.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cnezdWGV00fFckVee6StudofnYk=', 'Content-Length': '0'} 2022-09-02T05:53:10.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073cd27d1f4afb7ba-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:10.121 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:10.121 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:10.122 DEBUG:boto:Method: GET 2022-09-02T05:53:10.123 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:10.123 DEBUG:boto:Data: 2022-09-02T05:53:10.124 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.125 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.125 DEBUG:boto:Params: {} 2022-09-02T05:53:10.126 DEBUG:boto:Token: None 2022-09-02T05:53:10.127 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj55 2022-09-02T05:53:10.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTcSKTESxJQj2upFAwlL9vZRJe8= 2022-09-02T05:53:10.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTcSKTESxJQj2upFAwlL9vZRJe8=', 'Content-Length': '0'} 2022-09-02T05:53:10.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5f7e65ef4c3118-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.134 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:10.135 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:10.136 DEBUG:boto:Method: GET 2022-09-02T05:53:10.136 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:10.137 DEBUG:boto:Data: 2022-09-02T05:53:10.137 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.139 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.139 DEBUG:boto:Params: {} 2022-09-02T05:53:10.140 DEBUG:boto:Token: None 2022-09-02T05:53:10.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj55 2022-09-02T05:53:10.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mTcSKTESxJQj2upFAwlL9vZRJe8= 2022-09-02T05:53:10.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mTcSKTESxJQj2upFAwlL9vZRJe8=', 'Content-Length': '0'} 2022-09-02T05:53:10.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003076c0a946eb2281-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.193 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.195 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.196 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.198 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.199 DEBUG:boto:Method: GET 2022-09-02T05:53:10.200 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.201 DEBUG:boto:Data: 2022-09-02T05:53:10.202 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.204 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.205 DEBUG:boto:Params: {} 2022-09-02T05:53:10.206 DEBUG:boto:Token: None 2022-09-02T05:53:10.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001d7c34e714fa215-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.215 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.217 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.218 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.218 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.219 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.219 DEBUG:boto:Method: GET 2022-09-02T05:53:10.220 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:10.221 DEBUG:boto:Data: 2022-09-02T05:53:10.221 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.222 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.223 DEBUG:boto:Params: {} 2022-09-02T05:53:10.224 DEBUG:boto:Token: None 2022-09-02T05:53:10.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea962c0aa7fc04d-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.232 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:10.233 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.234 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.234 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.235 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.235 DEBUG:boto:Method: GET 2022-09-02T05:53:10.236 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.237 DEBUG:boto:Data: 2022-09-02T05:53:10.237 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.238 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.239 DEBUG:boto:Params: {} 2022-09-02T05:53:10.239 DEBUG:boto:Token: None 2022-09-02T05:53:10.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q= 2022-09-02T05:53:10.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q=', 'Content-Length': '0'} 2022-09-02T05:53:10.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000771c1a144f79c9d8-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.246 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.246 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.247 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.247 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.248 DEBUG:boto:Method: GET 2022-09-02T05:53:10.248 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.249 DEBUG:boto:Data: 2022-09-02T05:53:10.249 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.251 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.251 DEBUG:boto:Params: {} 2022-09-02T05:53:10.252 DEBUG:boto:Token: None 2022-09-02T05:53:10.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q= 2022-09-02T05:53:10.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q=', 'Content-Length': '0'} 2022-09-02T05:53:10.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062020d5ce16fdf02-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.262 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.263 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.263 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.264 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.264 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.265 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.265 DEBUG:boto:Data: 2022-09-02T05:53:10.266 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.267 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.267 DEBUG:boto:Params: {} 2022-09-02T05:53:10.268 DEBUG:boto:Token: None 2022-09-02T05:53:10.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.269 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5y7AEhpCk3BO5O6af1nmPS2uvqI= 2022-09-02T05:53:10.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5y7AEhpCk3BO5O6af1nmPS2uvqI=', 'Content-Length': '0'} 2022-09-02T05:53:10.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001267ba67e65f6e97-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.274 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.274 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.275 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.275 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.276 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.276 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.277 DEBUG:boto:Data: 2022-09-02T05:53:10.277 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.279 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.279 DEBUG:boto:Params: {} 2022-09-02T05:53:10.280 DEBUG:boto:Token: None 2022-09-02T05:53:10.280 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5y7AEhpCk3BO5O6af1nmPS2uvqI= 2022-09-02T05:53:10.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5y7AEhpCk3BO5O6af1nmPS2uvqI=', 'Content-Length': '0'} 2022-09-02T05:53:10.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a11c1067f1f8dc-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:10.286 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.286 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.287 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.288 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.289 DEBUG:boto:Method: GET 2022-09-02T05:53:10.289 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.290 DEBUG:boto:Data: 2022-09-02T05:53:10.290 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.291 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.292 DEBUG:boto:Params: {} 2022-09-02T05:53:10.293 DEBUG:boto:Token: None 2022-09-02T05:53:10.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q= 2022-09-02T05:53:10.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q=', 'Content-Length': '0'} 2022-09-02T05:53:10.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000986d7a8f155f023a-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.300 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.301 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.301 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.302 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.302 DEBUG:boto:Method: GET 2022-09-02T05:53:10.303 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.303 DEBUG:boto:Data: 2022-09-02T05:53:10.304 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.305 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.305 DEBUG:boto:Params: {} 2022-09-02T05:53:10.306 DEBUG:boto:Token: None 2022-09-02T05:53:10.307 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:10.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q= 2022-09-02T05:53:10.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/HsUqmNZEbbwTDb/x54PClX089Q=', 'Content-Length': '0'} 2022-09-02T05:53:10.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311741f8177baf56-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.354 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.356 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.357 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.358 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.359 DEBUG:boto:Method: GET 2022-09-02T05:53:10.360 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.361 DEBUG:boto:Data: 2022-09-02T05:53:10.362 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.363 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.364 DEBUG:boto:Params: {} 2022-09-02T05:53:10.366 DEBUG:boto:Token: None 2022-09-02T05:53:10.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009789792226bc1274-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.374 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.375 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.376 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.376 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.377 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.377 DEBUG:boto:Method: GET 2022-09-02T05:53:10.378 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.378 DEBUG:boto:Data: 2022-09-02T05:53:10.379 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.380 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.380 DEBUG:boto:Params: {} 2022-09-02T05:53:10.381 DEBUG:boto:Token: None 2022-09-02T05:53:10.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f2882c33ca13f37-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.387 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.388 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.389 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.389 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.390 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:10.390 DEBUG:boto:Data: 2022-09-02T05:53:10.391 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.392 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.392 DEBUG:boto:Params: {} 2022-09-02T05:53:10.393 DEBUG:boto:Token: None 2022-09-02T05:53:10.393 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57 2022-09-02T05:53:10.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1mgwJvAiTc77iv0h2CyUA0GFYEM= 2022-09-02T05:53:10.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1mgwJvAiTc77iv0h2CyUA0GFYEM=', 'Content-Length': '0'} 2022-09-02T05:53:10.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f480b1b88abc25-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.399 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.399 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.400 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.400 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:10.401 DEBUG:boto:Data: 2022-09-02T05:53:10.402 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.403 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.403 DEBUG:boto:Params: {} 2022-09-02T05:53:10.404 DEBUG:boto:Token: None 2022-09-02T05:53:10.405 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57 2022-09-02T05:53:10.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1mgwJvAiTc77iv0h2CyUA0GFYEM= 2022-09-02T05:53:10.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1mgwJvAiTc77iv0h2CyUA0GFYEM=', 'Content-Length': '0'} 2022-09-02T05:53:10.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f2e30f2a666e0e0-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:10.412 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.413 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.413 DEBUG:boto:Method: GET 2022-09-02T05:53:10.414 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:10.415 DEBUG:boto:Data: 2022-09-02T05:53:10.415 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.416 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.417 DEBUG:boto:Params: {} 2022-09-02T05:53:10.417 DEBUG:boto:Token: None 2022-09-02T05:53:10.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57 2022-09-02T05:53:10.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5za9lD2nJX/upNFchZVEiEzAuk= 2022-09-02T05:53:10.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5za9lD2nJX/upNFchZVEiEzAuk=', 'Content-Length': '0'} 2022-09-02T05:53:10.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000012ecb3a4985afa0-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.423 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:10.424 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:10.424 DEBUG:boto:Method: GET 2022-09-02T05:53:10.425 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:10.426 DEBUG:boto:Data: 2022-09-02T05:53:10.426 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.427 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.428 DEBUG:boto:Params: {} 2022-09-02T05:53:10.428 DEBUG:boto:Token: None 2022-09-02T05:53:10.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj57 2022-09-02T05:53:10.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5za9lD2nJX/upNFchZVEiEzAuk= 2022-09-02T05:53:10.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5za9lD2nJX/upNFchZVEiEzAuk=', 'Content-Length': '0'} 2022-09-02T05:53:10.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe631f2e41c2863-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.435 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.436 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.436 DEBUG:boto:Method: GET 2022-09-02T05:53:10.437 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.437 DEBUG:boto:Data: 2022-09-02T05:53:10.438 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.438 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.439 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.439 DEBUG:boto:Params: {} 2022-09-02T05:53:10.440 DEBUG:boto:Token: None 2022-09-02T05:53:10.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.441 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000494c957660a02989-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.486 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.488 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.488 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.489 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.489 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.490 DEBUG:boto:Method: GET 2022-09-02T05:53:10.490 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:10.491 DEBUG:boto:Data: 2022-09-02T05:53:10.492 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.493 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.493 DEBUG:boto:Params: {} 2022-09-02T05:53:10.494 DEBUG:boto:Token: None 2022-09-02T05:53:10.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081ffe9f908b65ddd-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.501 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:10.502 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.502 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.503 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.504 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.504 DEBUG:boto:Method: GET 2022-09-02T05:53:10.505 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.505 DEBUG:boto:Data: 2022-09-02T05:53:10.506 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.507 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.507 DEBUG:boto:Params: {} 2022-09-02T05:53:10.508 DEBUG:boto:Token: None 2022-09-02T05:53:10.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA= 2022-09-02T05:53:10.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA=', 'Content-Length': '0'} 2022-09-02T05:53:10.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efec4361c0ca6890-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.514 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.514 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.515 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.515 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.516 DEBUG:boto:Method: GET 2022-09-02T05:53:10.516 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.517 DEBUG:boto:Data: 2022-09-02T05:53:10.517 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.518 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.519 DEBUG:boto:Params: {} 2022-09-02T05:53:10.519 DEBUG:boto:Token: None 2022-09-02T05:53:10.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA= 2022-09-02T05:53:10.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA=', 'Content-Length': '0'} 2022-09-02T05:53:10.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000315b5d2b0516a55a-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.574 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.575 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.575 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.576 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.576 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.577 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.577 DEBUG:boto:Data: 2022-09-02T05:53:10.578 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.579 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.580 DEBUG:boto:Params: {} 2022-09-02T05:53:10.580 DEBUG:boto:Token: None 2022-09-02T05:53:10.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zzu6+W03I55aVzCbF95JDxfgNAk= 2022-09-02T05:53:10.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zzu6+W03I55aVzCbF95JDxfgNAk=', 'Content-Length': '0'} 2022-09-02T05:53:10.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351d03d5335a15fe-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.585 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.586 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.587 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.587 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.588 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.589 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.589 DEBUG:boto:Data: 2022-09-02T05:53:10.589 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.590 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.591 DEBUG:boto:Params: {} 2022-09-02T05:53:10.592 DEBUG:boto:Token: None 2022-09-02T05:53:10.592 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zzu6+W03I55aVzCbF95JDxfgNAk= 2022-09-02T05:53:10.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zzu6+W03I55aVzCbF95JDxfgNAk=', 'Content-Length': '0'} 2022-09-02T05:53:10.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5434607b24bd1f7-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:10.597 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.597 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.598 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.598 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.599 DEBUG:boto:Method: GET 2022-09-02T05:53:10.600 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.600 DEBUG:boto:Data: 2022-09-02T05:53:10.601 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.602 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.602 DEBUG:boto:Params: {} 2022-09-02T05:53:10.603 DEBUG:boto:Token: None 2022-09-02T05:53:10.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA= 2022-09-02T05:53:10.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA=', 'Content-Length': '0'} 2022-09-02T05:53:10.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f316148890e7946f-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.608 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.608 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.609 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.610 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.610 DEBUG:boto:Method: GET 2022-09-02T05:53:10.611 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.611 DEBUG:boto:Data: 2022-09-02T05:53:10.612 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.613 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.613 DEBUG:boto:Params: {} 2022-09-02T05:53:10.614 DEBUG:boto:Token: None 2022-09-02T05:53:10.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:10.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA= 2022-09-02T05:53:10.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ijUL2HWOIlyGi2f7D/zvm1tTRA=', 'Content-Length': '0'} 2022-09-02T05:53:10.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5505a9c679abbd9-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.619 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.620 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.621 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.623 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.624 DEBUG:boto:Method: GET 2022-09-02T05:53:10.624 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.625 DEBUG:boto:Data: 2022-09-02T05:53:10.625 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.626 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.627 DEBUG:boto:Params: {} 2022-09-02T05:53:10.628 DEBUG:boto:Token: None 2022-09-02T05:53:10.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000288996c21feb86f4-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.633 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.634 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.635 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.635 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.636 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.637 DEBUG:boto:Method: GET 2022-09-02T05:53:10.638 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.638 DEBUG:boto:Data: 2022-09-02T05:53:10.639 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.640 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.641 DEBUG:boto:Params: {} 2022-09-02T05:53:10.641 DEBUG:boto:Token: None 2022-09-02T05:53:10.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000623cb69df7010f08-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.648 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.649 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.649 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.650 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.651 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:10.651 DEBUG:boto:Data: 2022-09-02T05:53:10.652 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.653 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.653 DEBUG:boto:Params: {} 2022-09-02T05:53:10.654 DEBUG:boto:Token: None 2022-09-02T05:53:10.655 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68 2022-09-02T05:53:10.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H0OcjdL9frA75ndv4yaD0NQc/zs= 2022-09-02T05:53:10.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H0OcjdL9frA75ndv4yaD0NQc/zs=', 'Content-Length': '0'} 2022-09-02T05:53:10.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ff14b3895400705-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.659 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.659 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.660 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.660 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:10.661 DEBUG:boto:Data: 2022-09-02T05:53:10.661 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.662 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.663 DEBUG:boto:Params: {} 2022-09-02T05:53:10.664 DEBUG:boto:Token: None 2022-09-02T05:53:10.664 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68 2022-09-02T05:53:10.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H0OcjdL9frA75ndv4yaD0NQc/zs= 2022-09-02T05:53:10.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H0OcjdL9frA75ndv4yaD0NQc/zs=', 'Content-Length': '0'} 2022-09-02T05:53:10.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000172498796a9a9902-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:10.669 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.670 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.670 DEBUG:boto:Method: GET 2022-09-02T05:53:10.671 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:10.671 DEBUG:boto:Data: 2022-09-02T05:53:10.672 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.673 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.674 DEBUG:boto:Params: {} 2022-09-02T05:53:10.674 DEBUG:boto:Token: None 2022-09-02T05:53:10.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68 2022-09-02T05:53:10.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7G38UdZwqsSMihq6PNSGojrdbQc= 2022-09-02T05:53:10.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7G38UdZwqsSMihq6PNSGojrdbQc=', 'Content-Length': '0'} 2022-09-02T05:53:10.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd6f22ebe1185d9-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.722 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:10.723 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:10.724 DEBUG:boto:Method: GET 2022-09-02T05:53:10.725 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:10.727 DEBUG:boto:Data: 2022-09-02T05:53:10.727 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.729 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.730 DEBUG:boto:Params: {} 2022-09-02T05:53:10.732 DEBUG:boto:Token: None 2022-09-02T05:53:10.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj68 2022-09-02T05:53:10.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7G38UdZwqsSMihq6PNSGojrdbQc= 2022-09-02T05:53:10.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7G38UdZwqsSMihq6PNSGojrdbQc=', 'Content-Length': '0'} 2022-09-02T05:53:10.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abc434288d6a8a95-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.739 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.741 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.742 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.743 DEBUG:boto:Method: GET 2022-09-02T05:53:10.744 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.745 DEBUG:boto:Data: 2022-09-02T05:53:10.745 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.746 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.756 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.757 DEBUG:boto:Params: {} 2022-09-02T05:53:10.757 DEBUG:boto:Token: None 2022-09-02T05:53:10.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f3e036403aa2cb8-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.763 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.764 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.764 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.765 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.765 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.766 DEBUG:boto:Method: GET 2022-09-02T05:53:10.766 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:10.767 DEBUG:boto:Data: 2022-09-02T05:53:10.767 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.768 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.769 DEBUG:boto:Params: {} 2022-09-02T05:53:10.770 DEBUG:boto:Token: None 2022-09-02T05:53:10.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48c03cd366ebcd2-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.776 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:10.777 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.777 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.778 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.779 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.779 DEBUG:boto:Method: GET 2022-09-02T05:53:10.780 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.780 DEBUG:boto:Data: 2022-09-02T05:53:10.781 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.782 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.782 DEBUG:boto:Params: {} 2022-09-02T05:53:10.783 DEBUG:boto:Token: None 2022-09-02T05:53:10.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU= 2022-09-02T05:53:10.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU=', 'Content-Length': '0'} 2022-09-02T05:53:10.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53ee73c94869aae-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.788 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.788 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.789 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.790 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.790 DEBUG:boto:Method: GET 2022-09-02T05:53:10.791 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.791 DEBUG:boto:Data: 2022-09-02T05:53:10.792 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.793 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.794 DEBUG:boto:Params: {} 2022-09-02T05:53:10.794 DEBUG:boto:Token: None 2022-09-02T05:53:10.795 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU= 2022-09-02T05:53:10.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU=', 'Content-Length': '0'} 2022-09-02T05:53:10.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097de0ced06cf6ad3-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.800 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.801 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.802 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.802 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.803 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.804 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.805 DEBUG:boto:Data: 2022-09-02T05:53:10.806 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.807 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.808 DEBUG:boto:Params: {} 2022-09-02T05:53:10.808 DEBUG:boto:Token: None 2022-09-02T05:53:10.809 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tbVpDDPR9Rms7/MMlwSOWf8cErI= 2022-09-02T05:53:10.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tbVpDDPR9Rms7/MMlwSOWf8cErI=', 'Content-Length': '0'} 2022-09-02T05:53:10.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000144b5d6bee979baf-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.813 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.813 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.814 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.814 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.815 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.815 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.816 DEBUG:boto:Data: 2022-09-02T05:53:10.816 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.817 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.818 DEBUG:boto:Params: {} 2022-09-02T05:53:10.818 DEBUG:boto:Token: None 2022-09-02T05:53:10.819 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tbVpDDPR9Rms7/MMlwSOWf8cErI= 2022-09-02T05:53:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tbVpDDPR9Rms7/MMlwSOWf8cErI=', 'Content-Length': '0'} 2022-09-02T05:53:10.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084695bfab0881945-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:10.823 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.824 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.825 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.825 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.826 DEBUG:boto:Method: GET 2022-09-02T05:53:10.826 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.827 DEBUG:boto:Data: 2022-09-02T05:53:10.827 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.828 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.829 DEBUG:boto:Params: {} 2022-09-02T05:53:10.829 DEBUG:boto:Token: None 2022-09-02T05:53:10.830 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU= 2022-09-02T05:53:10.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU=', 'Content-Length': '0'} 2022-09-02T05:53:10.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1d6f084e56432e-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.874 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.875 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.875 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.876 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.876 DEBUG:boto:Method: GET 2022-09-02T05:53:10.877 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.877 DEBUG:boto:Data: 2022-09-02T05:53:10.878 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.879 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.880 DEBUG:boto:Params: {} 2022-09-02T05:53:10.880 DEBUG:boto:Token: None 2022-09-02T05:53:10.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:10.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU= 2022-09-02T05:53:10.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e7H1OYoLBadwc/E10NvKXk67AFU=', 'Content-Length': '0'} 2022-09-02T05:53:10.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0d7c8eaea32c574-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.885 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.886 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.886 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.887 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.887 DEBUG:boto:Method: GET 2022-09-02T05:53:10.888 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.888 DEBUG:boto:Data: 2022-09-02T05:53:10.889 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.890 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.890 DEBUG:boto:Params: {} 2022-09-02T05:53:10.891 DEBUG:boto:Token: None 2022-09-02T05:53:10.891 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.892 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a0dd8805dffe91-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.896 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.897 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.897 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.898 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.898 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.899 DEBUG:boto:Method: GET 2022-09-02T05:53:10.900 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.900 DEBUG:boto:Data: 2022-09-02T05:53:10.901 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.902 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.902 DEBUG:boto:Params: {} 2022-09-02T05:53:10.903 DEBUG:boto:Token: None 2022-09-02T05:53:10.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb838578815d058-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.910 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.911 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.911 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.912 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.912 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:10.913 DEBUG:boto:Data: 2022-09-02T05:53:10.913 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.914 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.915 DEBUG:boto:Params: {} 2022-09-02T05:53:10.915 DEBUG:boto:Token: None 2022-09-02T05:53:10.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69 2022-09-02T05:53:10.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CPE78YmlLYyUthZfxUfJGPj4tes= 2022-09-02T05:53:10.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CPE78YmlLYyUthZfxUfJGPj4tes=', 'Content-Length': '0'} 2022-09-02T05:53:10.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6cccc3223b36525-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.921 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.922 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.923 DEBUG:boto:Method: HEAD 2022-09-02T05:53:10.923 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:10.924 DEBUG:boto:Data: 2022-09-02T05:53:10.925 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.926 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.927 DEBUG:boto:Params: {} 2022-09-02T05:53:10.928 DEBUG:boto:Token: None 2022-09-02T05:53:10.929 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69 2022-09-02T05:53:10.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CPE78YmlLYyUthZfxUfJGPj4tes= 2022-09-02T05:53:10.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CPE78YmlLYyUthZfxUfJGPj4tes=', 'Content-Length': '0'} 2022-09-02T05:53:10.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a3480bcf1255e8-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:10.935 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.935 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.936 DEBUG:boto:Method: GET 2022-09-02T05:53:10.937 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:10.937 DEBUG:boto:Data: 2022-09-02T05:53:10.938 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.938 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.939 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.940 DEBUG:boto:Params: {} 2022-09-02T05:53:10.940 DEBUG:boto:Token: None 2022-09-02T05:53:10.941 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69 2022-09-02T05:53:10.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1WOqfeLkX4tx+kS/52emzyCelSE= 2022-09-02T05:53:10.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1WOqfeLkX4tx+kS/52emzyCelSE=', 'Content-Length': '0'} 2022-09-02T05:53:10.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076c7145ab4f83562-0063119a46-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.945 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:10.945 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:10.946 DEBUG:boto:Method: GET 2022-09-02T05:53:10.947 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:10.948 DEBUG:boto:Data: 2022-09-02T05:53:10.948 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.949 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.950 DEBUG:boto:Params: {} 2022-09-02T05:53:10.950 DEBUG:boto:Token: None 2022-09-02T05:53:10.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/obj69 2022-09-02T05:53:10.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1WOqfeLkX4tx+kS/52emzyCelSE= 2022-09-02T05:53:10.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1WOqfeLkX4tx+kS/52emzyCelSE=', 'Content-Length': '0'} 2022-09-02T05:53:10.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091e44aad7e36bfd7-0063119a46-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.955 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.956 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.956 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.958 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.958 DEBUG:boto:Method: GET 2022-09-02T05:53:10.959 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.959 DEBUG:boto:Data: 2022-09-02T05:53:10.960 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.960 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:10.961 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.961 DEBUG:boto:Params: {} 2022-09-02T05:53:10.962 DEBUG:boto:Token: None 2022-09-02T05:53:10.963 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a27ce988639ecd2b-0063119a46-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:10.967 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:10.968 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:10.968 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:10.969 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.969 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.970 DEBUG:boto:Method: GET 2022-09-02T05:53:10.971 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:10.971 DEBUG:boto:Data: 2022-09-02T05:53:10.972 DEBUG:boto:Headers: {} 2022-09-02T05:53:10.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:10.973 DEBUG:boto:Port: 8000 2022-09-02T05:53:10.973 DEBUG:boto:Params: {} 2022-09-02T05:53:10.974 DEBUG:boto:Token: None 2022-09-02T05:53:10.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:10 GMT /xljbkf-74/ 2022-09-02T05:53:10.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM= 2022-09-02T05:53:10.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gkzulDPsk6B2nHaFlMTRJmBKcpM=', 'Content-Length': '0'} 2022-09-02T05:53:10.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b2a4f50410de07-0063119a46-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.026 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:11.029 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.030 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.031 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.032 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.033 DEBUG:boto:Method: GET 2022-09-02T05:53:11.034 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.035 DEBUG:boto:Data: 2022-09-02T05:53:11.036 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.038 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.039 DEBUG:boto:Params: {} 2022-09-02T05:53:11.040 DEBUG:boto:Token: None 2022-09-02T05:53:11.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M= 2022-09-02T05:53:11.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M=', 'Content-Length': '0'} 2022-09-02T05:53:11.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5852dbffeeb3078-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.048 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.048 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.049 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.050 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.050 DEBUG:boto:Method: GET 2022-09-02T05:53:11.051 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.051 DEBUG:boto:Data: 2022-09-02T05:53:11.052 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.053 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.053 DEBUG:boto:Params: {} 2022-09-02T05:53:11.054 DEBUG:boto:Token: None 2022-09-02T05:53:11.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M= 2022-09-02T05:53:11.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M=', 'Content-Length': '0'} 2022-09-02T05:53:11.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052749646fe9b20b2-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.059 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.059 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.060 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.060 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.061 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.061 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.062 DEBUG:boto:Data: 2022-09-02T05:53:11.062 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.063 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.064 DEBUG:boto:Params: {} 2022-09-02T05:53:11.064 DEBUG:boto:Token: None 2022-09-02T05:53:11.065 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R8Qlr4rgmFNwvQbxMZSrh/lzqwM= 2022-09-02T05:53:11.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R8Qlr4rgmFNwvQbxMZSrh/lzqwM=', 'Content-Length': '0'} 2022-09-02T05:53:11.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a640641b01d9fbd-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.070 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.071 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.072 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.073 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.073 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.074 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.075 DEBUG:boto:Data: 2022-09-02T05:53:11.076 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.078 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.079 DEBUG:boto:Params: {} 2022-09-02T05:53:11.080 DEBUG:boto:Token: None 2022-09-02T05:53:11.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R8Qlr4rgmFNwvQbxMZSrh/lzqwM= 2022-09-02T05:53:11.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R8Qlr4rgmFNwvQbxMZSrh/lzqwM=', 'Content-Length': '0'} 2022-09-02T05:53:11.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b1131bcbf9509e-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:11.088 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.089 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.090 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.091 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.092 DEBUG:boto:Method: GET 2022-09-02T05:53:11.093 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.094 DEBUG:boto:Data: 2022-09-02T05:53:11.095 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.097 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.097 DEBUG:boto:Params: {} 2022-09-02T05:53:11.099 DEBUG:boto:Token: None 2022-09-02T05:53:11.100 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M= 2022-09-02T05:53:11.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M=', 'Content-Length': '0'} 2022-09-02T05:53:11.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff5ffe8bf10a732-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.146 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.147 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.147 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.148 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.149 DEBUG:boto:Method: GET 2022-09-02T05:53:11.149 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.150 DEBUG:boto:Data: 2022-09-02T05:53:11.151 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.153 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.154 DEBUG:boto:Params: {} 2022-09-02T05:53:11.155 DEBUG:boto:Token: None 2022-09-02T05:53:11.155 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:11.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M= 2022-09-02T05:53:11.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o6Yhi1Ywd9q0bWJDye+oBCOA61M=', 'Content-Length': '0'} 2022-09-02T05:53:11.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020a8292f7dafdb06-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.159 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.160 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.161 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.161 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.162 DEBUG:boto:Method: GET 2022-09-02T05:53:11.162 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.163 DEBUG:boto:Data: 2022-09-02T05:53:11.163 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.164 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.165 DEBUG:boto:Params: {} 2022-09-02T05:53:11.166 DEBUG:boto:Token: None 2022-09-02T05:53:11.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9a90672092bf4c3-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.171 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.172 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.172 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.173 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.173 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.174 DEBUG:boto:Method: GET 2022-09-02T05:53:11.175 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.175 DEBUG:boto:Data: 2022-09-02T05:53:11.176 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.177 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.177 DEBUG:boto:Params: {} 2022-09-02T05:53:11.178 DEBUG:boto:Token: None 2022-09-02T05:53:11.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008761000e45138473-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.226 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.227 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.228 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.228 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.229 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:11.229 DEBUG:boto:Data: 2022-09-02T05:53:11.230 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.231 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.231 DEBUG:boto:Params: {} 2022-09-02T05:53:11.232 DEBUG:boto:Token: None 2022-09-02T05:53:11.233 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76 2022-09-02T05:53:11.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D3k8dvkvJfM9gLcb/ufsnHNNblg= 2022-09-02T05:53:11.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D3k8dvkvJfM9gLcb/ufsnHNNblg=', 'Content-Length': '0'} 2022-09-02T05:53:11.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f399204bc7f623-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.237 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.237 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.238 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.238 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:11.239 DEBUG:boto:Data: 2022-09-02T05:53:11.239 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.240 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.241 DEBUG:boto:Params: {} 2022-09-02T05:53:11.241 DEBUG:boto:Token: None 2022-09-02T05:53:11.242 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76 2022-09-02T05:53:11.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D3k8dvkvJfM9gLcb/ufsnHNNblg= 2022-09-02T05:53:11.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D3k8dvkvJfM9gLcb/ufsnHNNblg=', 'Content-Length': '0'} 2022-09-02T05:53:11.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2966b15386ef88-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:11.247 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.248 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.248 DEBUG:boto:Method: GET 2022-09-02T05:53:11.249 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:11.249 DEBUG:boto:Data: 2022-09-02T05:53:11.250 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.251 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.252 DEBUG:boto:Params: {} 2022-09-02T05:53:11.252 DEBUG:boto:Token: None 2022-09-02T05:53:11.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76 2022-09-02T05:53:11.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oq3oAuIU9MYFE0odD9YFbnkkkqc= 2022-09-02T05:53:11.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oq3oAuIU9MYFE0odD9YFbnkkkqc=', 'Content-Length': '0'} 2022-09-02T05:53:11.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db80c9355a33fb9b-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.258 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:11.259 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:11.259 DEBUG:boto:Method: GET 2022-09-02T05:53:11.260 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:11.260 DEBUG:boto:Data: 2022-09-02T05:53:11.261 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.262 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.263 DEBUG:boto:Params: {} 2022-09-02T05:53:11.263 DEBUG:boto:Token: None 2022-09-02T05:53:11.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj76 2022-09-02T05:53:11.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oq3oAuIU9MYFE0odD9YFbnkkkqc= 2022-09-02T05:53:11.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oq3oAuIU9MYFE0odD9YFbnkkkqc=', 'Content-Length': '0'} 2022-09-02T05:53:11.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab75680a242fedf3-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.269 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.270 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.271 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.272 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.273 DEBUG:boto:Method: GET 2022-09-02T05:53:11.274 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.275 DEBUG:boto:Data: 2022-09-02T05:53:11.275 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.277 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.277 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.278 DEBUG:boto:Params: {} 2022-09-02T05:53:11.279 DEBUG:boto:Token: None 2022-09-02T05:53:11.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005046246a98ea5f38-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.285 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.286 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.287 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.288 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.288 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.289 DEBUG:boto:Method: GET 2022-09-02T05:53:11.289 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:11.290 DEBUG:boto:Data: 2022-09-02T05:53:11.291 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.292 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.292 DEBUG:boto:Params: {} 2022-09-02T05:53:11.293 DEBUG:boto:Token: None 2022-09-02T05:53:11.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007896a48e71c240a0-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.300 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:11.302 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.302 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.303 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.304 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.305 DEBUG:boto:Method: GET 2022-09-02T05:53:11.306 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.306 DEBUG:boto:Data: 2022-09-02T05:53:11.307 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.308 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.309 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.310 DEBUG:boto:Params: {} 2022-09-02T05:53:11.311 DEBUG:boto:Token: None 2022-09-02T05:53:11.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us= 2022-09-02T05:53:11.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us=', 'Content-Length': '0'} 2022-09-02T05:53:11.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb024ae83b016db8-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.316 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.317 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.317 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.318 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.319 DEBUG:boto:Method: GET 2022-09-02T05:53:11.319 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.320 DEBUG:boto:Data: 2022-09-02T05:53:11.321 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.324 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.324 DEBUG:boto:Params: {} 2022-09-02T05:53:11.325 DEBUG:boto:Token: None 2022-09-02T05:53:11.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us= 2022-09-02T05:53:11.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us=', 'Content-Length': '0'} 2022-09-02T05:53:11.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a328d1ec8d3b6869-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.330 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.331 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.332 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.332 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.333 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.333 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.334 DEBUG:boto:Data: 2022-09-02T05:53:11.335 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.337 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.338 DEBUG:boto:Params: {} 2022-09-02T05:53:11.338 DEBUG:boto:Token: None 2022-09-02T05:53:11.339 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4oQPJtRcPqTv35rWte2wXB5Tkvg= 2022-09-02T05:53:11.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4oQPJtRcPqTv35rWte2wXB5Tkvg=', 'Content-Length': '0'} 2022-09-02T05:53:11.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18c748f5242355a-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.343 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.344 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.344 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.345 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.345 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.346 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.347 DEBUG:boto:Data: 2022-09-02T05:53:11.347 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.349 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.350 DEBUG:boto:Params: {} 2022-09-02T05:53:11.351 DEBUG:boto:Token: None 2022-09-02T05:53:11.352 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4oQPJtRcPqTv35rWte2wXB5Tkvg= 2022-09-02T05:53:11.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4oQPJtRcPqTv35rWte2wXB5Tkvg=', 'Content-Length': '0'} 2022-09-02T05:53:11.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091064e8da6077db3-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:11.357 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.358 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.358 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.359 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.359 DEBUG:boto:Method: GET 2022-09-02T05:53:11.360 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.360 DEBUG:boto:Data: 2022-09-02T05:53:11.361 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.362 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.363 DEBUG:boto:Params: {} 2022-09-02T05:53:11.364 DEBUG:boto:Token: None 2022-09-02T05:53:11.364 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us= 2022-09-02T05:53:11.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us=', 'Content-Length': '0'} 2022-09-02T05:53:11.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ca304aed289f4b-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.369 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.369 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.370 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.370 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.371 DEBUG:boto:Method: GET 2022-09-02T05:53:11.372 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.372 DEBUG:boto:Data: 2022-09-02T05:53:11.373 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.374 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.374 DEBUG:boto:Params: {} 2022-09-02T05:53:11.375 DEBUG:boto:Token: None 2022-09-02T05:53:11.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:11.376 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us= 2022-09-02T05:53:11.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nQItdTnuZoYASONDi85TjKm12Us=', 'Content-Length': '0'} 2022-09-02T05:53:11.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8fd6baaa75d71f7-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.380 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.381 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.381 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.382 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.382 DEBUG:boto:Method: GET 2022-09-02T05:53:11.383 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.383 DEBUG:boto:Data: 2022-09-02T05:53:11.384 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.385 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.386 DEBUG:boto:Params: {} 2022-09-02T05:53:11.387 DEBUG:boto:Token: None 2022-09-02T05:53:11.387 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a8a678c9eff830b-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.392 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.393 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.394 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.394 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.395 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.395 DEBUG:boto:Method: GET 2022-09-02T05:53:11.396 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.396 DEBUG:boto:Data: 2022-09-02T05:53:11.397 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.398 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.399 DEBUG:boto:Params: {} 2022-09-02T05:53:11.399 DEBUG:boto:Token: None 2022-09-02T05:53:11.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.400 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000387b06d3c80f3283-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.405 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.406 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.407 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.407 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.408 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:11.408 DEBUG:boto:Data: 2022-09-02T05:53:11.409 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.410 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.410 DEBUG:boto:Params: {} 2022-09-02T05:53:11.411 DEBUG:boto:Token: None 2022-09-02T05:53:11.412 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85 2022-09-02T05:53:11.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yrlYSNT6DnRbNpBrRaq7Ud9pEHU= 2022-09-02T05:53:11.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yrlYSNT6DnRbNpBrRaq7Ud9pEHU=', 'Content-Length': '0'} 2022-09-02T05:53:11.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7966563875e02c3-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.417 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.417 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.418 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.418 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:11.419 DEBUG:boto:Data: 2022-09-02T05:53:11.419 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.420 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.421 DEBUG:boto:Params: {} 2022-09-02T05:53:11.422 DEBUG:boto:Token: None 2022-09-02T05:53:11.422 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85 2022-09-02T05:53:11.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yrlYSNT6DnRbNpBrRaq7Ud9pEHU= 2022-09-02T05:53:11.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yrlYSNT6DnRbNpBrRaq7Ud9pEHU=', 'Content-Length': '0'} 2022-09-02T05:53:11.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68bbd62a650ddfe-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:11.427 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.428 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.428 DEBUG:boto:Method: GET 2022-09-02T05:53:11.429 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:11.429 DEBUG:boto:Data: 2022-09-02T05:53:11.430 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.431 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.432 DEBUG:boto:Params: {} 2022-09-02T05:53:11.432 DEBUG:boto:Token: None 2022-09-02T05:53:11.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85 2022-09-02T05:53:11.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eA47RJp46YL0369qAZz+XXILY0w= 2022-09-02T05:53:11.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eA47RJp46YL0369qAZz+XXILY0w=', 'Content-Length': '0'} 2022-09-02T05:53:11.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4e5bfb6fb876a5a-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.478 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:11.479 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:11.479 DEBUG:boto:Method: GET 2022-09-02T05:53:11.480 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:11.480 DEBUG:boto:Data: 2022-09-02T05:53:11.481 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.482 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.482 DEBUG:boto:Params: {} 2022-09-02T05:53:11.483 DEBUG:boto:Token: None 2022-09-02T05:53:11.484 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj85 2022-09-02T05:53:11.485 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eA47RJp46YL0369qAZz+XXILY0w= 2022-09-02T05:53:11.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eA47RJp46YL0369qAZz+XXILY0w=', 'Content-Length': '0'} 2022-09-02T05:53:11.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6991ce3224a536-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.489 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.489 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.490 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.491 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.491 DEBUG:boto:Method: GET 2022-09-02T05:53:11.492 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.493 DEBUG:boto:Data: 2022-09-02T05:53:11.493 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.495 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.496 DEBUG:boto:Params: {} 2022-09-02T05:53:11.496 DEBUG:boto:Token: None 2022-09-02T05:53:11.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7843b6a56a44b1-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.502 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.504 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.505 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.505 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.506 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.507 DEBUG:boto:Method: GET 2022-09-02T05:53:11.507 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:11.508 DEBUG:boto:Data: 2022-09-02T05:53:11.509 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.510 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.511 DEBUG:boto:Params: {} 2022-09-02T05:53:11.511 DEBUG:boto:Token: None 2022-09-02T05:53:11.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e89481373b9a52-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.518 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:11.520 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.520 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.521 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.521 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.522 DEBUG:boto:Method: GET 2022-09-02T05:53:11.523 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.523 DEBUG:boto:Data: 2022-09-02T05:53:11.524 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.525 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.526 DEBUG:boto:Params: {} 2022-09-02T05:53:11.527 DEBUG:boto:Token: None 2022-09-02T05:53:11.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU= 2022-09-02T05:53:11.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU=', 'Content-Length': '0'} 2022-09-02T05:53:11.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f55c97d15102bc-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.532 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.533 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.534 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.534 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.535 DEBUG:boto:Method: GET 2022-09-02T05:53:11.535 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.536 DEBUG:boto:Data: 2022-09-02T05:53:11.536 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.537 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.538 DEBUG:boto:Params: {} 2022-09-02T05:53:11.538 DEBUG:boto:Token: None 2022-09-02T05:53:11.539 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU= 2022-09-02T05:53:11.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU=', 'Content-Length': '0'} 2022-09-02T05:53:11.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058815e74c8bc8e33-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.543 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.544 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.544 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.545 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.545 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.546 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.546 DEBUG:boto:Data: 2022-09-02T05:53:11.547 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.547 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.548 DEBUG:boto:Params: {} 2022-09-02T05:53:11.549 DEBUG:boto:Token: None 2022-09-02T05:53:11.549 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52JIGj06uDZ61ITZGL552QMaWtQ= 2022-09-02T05:53:11.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52JIGj06uDZ61ITZGL552QMaWtQ=', 'Content-Length': '0'} 2022-09-02T05:53:11.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6f46cc73657c74-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.553 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.554 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.554 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.555 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.555 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.556 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.556 DEBUG:boto:Data: 2022-09-02T05:53:11.557 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.558 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.559 DEBUG:boto:Params: {} 2022-09-02T05:53:11.559 DEBUG:boto:Token: None 2022-09-02T05:53:11.560 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52JIGj06uDZ61ITZGL552QMaWtQ= 2022-09-02T05:53:11.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52JIGj06uDZ61ITZGL552QMaWtQ=', 'Content-Length': '0'} 2022-09-02T05:53:11.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd46bf0fea3b39f6-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:11.565 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.565 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.566 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.566 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.567 DEBUG:boto:Method: GET 2022-09-02T05:53:11.568 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.568 DEBUG:boto:Data: 2022-09-02T05:53:11.569 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.569 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.570 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.571 DEBUG:boto:Params: {} 2022-09-02T05:53:11.571 DEBUG:boto:Token: None 2022-09-02T05:53:11.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU= 2022-09-02T05:53:11.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU=', 'Content-Length': '0'} 2022-09-02T05:53:11.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c863ed0d0f47081-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.576 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.577 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.577 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.578 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.578 DEBUG:boto:Method: GET 2022-09-02T05:53:11.579 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.579 DEBUG:boto:Data: 2022-09-02T05:53:11.580 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.581 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.582 DEBUG:boto:Params: {} 2022-09-02T05:53:11.583 DEBUG:boto:Token: None 2022-09-02T05:53:11.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:11.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU= 2022-09-02T05:53:11.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T6csRtWQL/+t6wyiWSAH3+DLMYU=', 'Content-Length': '0'} 2022-09-02T05:53:11.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe5d7dcd5e6df40-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.630 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.631 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.632 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.633 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.634 DEBUG:boto:Method: GET 2022-09-02T05:53:11.634 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.635 DEBUG:boto:Data: 2022-09-02T05:53:11.635 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.637 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.638 DEBUG:boto:Params: {} 2022-09-02T05:53:11.639 DEBUG:boto:Token: None 2022-09-02T05:53:11.640 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010aad56f3c99d1d4-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.647 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.649 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.650 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.650 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.651 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.653 DEBUG:boto:Method: GET 2022-09-02T05:53:11.654 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.655 DEBUG:boto:Data: 2022-09-02T05:53:11.655 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.657 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.658 DEBUG:boto:Params: {} 2022-09-02T05:53:11.659 DEBUG:boto:Token: None 2022-09-02T05:53:11.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d5cc64a364d88b-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.668 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.670 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.670 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.671 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.672 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:11.673 DEBUG:boto:Data: 2022-09-02T05:53:11.674 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.676 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.677 DEBUG:boto:Params: {} 2022-09-02T05:53:11.678 DEBUG:boto:Token: None 2022-09-02T05:53:11.679 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92 2022-09-02T05:53:11.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fFFMe2izsCdkeY3utSE150ibCss= 2022-09-02T05:53:11.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fFFMe2izsCdkeY3utSE150ibCss=', 'Content-Length': '0'} 2022-09-02T05:53:11.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872fa8d4f72e22c7-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.685 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.686 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.687 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.688 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:11.688 DEBUG:boto:Data: 2022-09-02T05:53:11.689 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.690 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.690 DEBUG:boto:Params: {} 2022-09-02T05:53:11.691 DEBUG:boto:Token: None 2022-09-02T05:53:11.692 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92 2022-09-02T05:53:11.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fFFMe2izsCdkeY3utSE150ibCss= 2022-09-02T05:53:11.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fFFMe2izsCdkeY3utSE150ibCss=', 'Content-Length': '0'} 2022-09-02T05:53:11.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da3d23785eee33f-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:11.697 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.698 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.698 DEBUG:boto:Method: GET 2022-09-02T05:53:11.699 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:11.699 DEBUG:boto:Data: 2022-09-02T05:53:11.700 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.701 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.701 DEBUG:boto:Params: {} 2022-09-02T05:53:11.702 DEBUG:boto:Token: None 2022-09-02T05:53:11.703 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92 2022-09-02T05:53:11.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d8Ops1Pv766BvecXTHHJ7zmTLfc= 2022-09-02T05:53:11.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d8Ops1Pv766BvecXTHHJ7zmTLfc=', 'Content-Length': '0'} 2022-09-02T05:53:11.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ee7cb34ec0edf0-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.707 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:11.707 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:11.708 DEBUG:boto:Method: GET 2022-09-02T05:53:11.708 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:11.709 DEBUG:boto:Data: 2022-09-02T05:53:11.709 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.711 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.711 DEBUG:boto:Params: {} 2022-09-02T05:53:11.712 DEBUG:boto:Token: None 2022-09-02T05:53:11.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj92 2022-09-02T05:53:11.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d8Ops1Pv766BvecXTHHJ7zmTLfc= 2022-09-02T05:53:11.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d8Ops1Pv766BvecXTHHJ7zmTLfc=', 'Content-Length': '0'} 2022-09-02T05:53:11.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad8148574f6ee55-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.716 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.717 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.718 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.718 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.719 DEBUG:boto:Method: GET 2022-09-02T05:53:11.719 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.720 DEBUG:boto:Data: 2022-09-02T05:53:11.720 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.721 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.722 DEBUG:boto:Params: {} 2022-09-02T05:53:11.722 DEBUG:boto:Token: None 2022-09-02T05:53:11.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003261f416b038c8fa-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.774 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.775 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.776 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.776 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.777 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.777 DEBUG:boto:Method: GET 2022-09-02T05:53:11.778 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:11.778 DEBUG:boto:Data: 2022-09-02T05:53:11.779 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.780 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.781 DEBUG:boto:Params: {} 2022-09-02T05:53:11.781 DEBUG:boto:Token: None 2022-09-02T05:53:11.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002836d257ad80191-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.787 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:11.789 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.789 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.790 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.791 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.791 DEBUG:boto:Method: GET 2022-09-02T05:53:11.792 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.792 DEBUG:boto:Data: 2022-09-02T05:53:11.793 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.794 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.794 DEBUG:boto:Params: {} 2022-09-02T05:53:11.795 DEBUG:boto:Token: None 2022-09-02T05:53:11.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw= 2022-09-02T05:53:11.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw=', 'Content-Length': '0'} 2022-09-02T05:53:11.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a949f5a3c81bc93-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.800 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.801 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.801 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.802 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.802 DEBUG:boto:Method: GET 2022-09-02T05:53:11.803 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.803 DEBUG:boto:Data: 2022-09-02T05:53:11.804 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.805 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.805 DEBUG:boto:Params: {} 2022-09-02T05:53:11.806 DEBUG:boto:Token: None 2022-09-02T05:53:11.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw= 2022-09-02T05:53:11.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw=', 'Content-Length': '0'} 2022-09-02T05:53:11.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008512434ecfbe080-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.854 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.855 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.855 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.856 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.856 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.857 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.857 DEBUG:boto:Data: 2022-09-02T05:53:11.858 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.859 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.859 DEBUG:boto:Params: {} 2022-09-02T05:53:11.860 DEBUG:boto:Token: None 2022-09-02T05:53:11.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cN1qTLsahZUwGDlV99VR5ZhfMVc= 2022-09-02T05:53:11.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cN1qTLsahZUwGDlV99VR5ZhfMVc=', 'Content-Length': '0'} 2022-09-02T05:53:11.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002921b93fa58bb857-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.865 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.865 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.866 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.866 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.867 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.867 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.868 DEBUG:boto:Data: 2022-09-02T05:53:11.868 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.869 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.870 DEBUG:boto:Params: {} 2022-09-02T05:53:11.870 DEBUG:boto:Token: None 2022-09-02T05:53:11.871 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cN1qTLsahZUwGDlV99VR5ZhfMVc= 2022-09-02T05:53:11.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cN1qTLsahZUwGDlV99VR5ZhfMVc=', 'Content-Length': '0'} 2022-09-02T05:53:11.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d6a9bf52ea16df-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:11.876 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.877 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.877 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.878 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.878 DEBUG:boto:Method: GET 2022-09-02T05:53:11.879 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.879 DEBUG:boto:Data: 2022-09-02T05:53:11.880 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.880 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.881 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.881 DEBUG:boto:Params: {} 2022-09-02T05:53:11.882 DEBUG:boto:Token: None 2022-09-02T05:53:11.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw= 2022-09-02T05:53:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw=', 'Content-Length': '0'} 2022-09-02T05:53:11.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a376cb798a78b27b-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.887 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.887 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.888 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.888 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.889 DEBUG:boto:Method: GET 2022-09-02T05:53:11.890 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.890 DEBUG:boto:Data: 2022-09-02T05:53:11.891 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.892 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.892 DEBUG:boto:Params: {} 2022-09-02T05:53:11.893 DEBUG:boto:Token: None 2022-09-02T05:53:11.894 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:11.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw= 2022-09-02T05:53:11.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cw6YAkRPpIiX/Qf4VhU9LZnK9Bw=', 'Content-Length': '0'} 2022-09-02T05:53:11.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9bfa3eb625029e-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.898 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.899 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.899 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:11.900 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:11.900 DEBUG:boto:Method: GET 2022-09-02T05:53:11.901 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:11.901 DEBUG:boto:Data: 2022-09-02T05:53:11.902 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.903 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.904 DEBUG:boto:Params: {} 2022-09-02T05:53:11.904 DEBUG:boto:Token: None 2022-09-02T05:53:11.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045971e1f721a2195-0063119a47-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.910 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.910 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:11.911 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:11.912 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:11.912 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:11.913 DEBUG:boto:Method: GET 2022-09-02T05:53:11.913 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:11.914 DEBUG:boto:Data: 2022-09-02T05:53:11.914 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.915 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.916 DEBUG:boto:Params: {} 2022-09-02T05:53:11.916 DEBUG:boto:Token: None 2022-09-02T05:53:11.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/ 2022-09-02T05:53:11.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0= 2022-09-02T05:53:11.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ESoDtnsVKMIBAo+XXEKokpwquI0=', 'Content-Length': '0'} 2022-09-02T05:53:11.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c222aec45d782846-0063119a47-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.923 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:11.924 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.925 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.925 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.926 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:11.926 DEBUG:boto:Data: 2022-09-02T05:53:11.927 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.928 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.928 DEBUG:boto:Params: {} 2022-09-02T05:53:11.929 DEBUG:boto:Token: None 2022-09-02T05:53:11.929 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95 2022-09-02T05:53:11.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j3d/uWxGFJO3ZV7eg8iVLJZUb3I= 2022-09-02T05:53:11.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j3d/uWxGFJO3ZV7eg8iVLJZUb3I=', 'Content-Length': '0'} 2022-09-02T05:53:11.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8201686e4f3e66-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.934 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.934 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.935 DEBUG:boto:Method: HEAD 2022-09-02T05:53:11.936 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:11.936 DEBUG:boto:Data: 2022-09-02T05:53:11.937 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.938 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.938 DEBUG:boto:Params: {} 2022-09-02T05:53:11.939 DEBUG:boto:Token: None 2022-09-02T05:53:11.939 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95 2022-09-02T05:53:11.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j3d/uWxGFJO3ZV7eg8iVLJZUb3I= 2022-09-02T05:53:11.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j3d/uWxGFJO3ZV7eg8iVLJZUb3I=', 'Content-Length': '0'} 2022-09-02T05:53:11.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969b7013134a8484-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:11.944 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.945 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.945 DEBUG:boto:Method: GET 2022-09-02T05:53:11.946 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:11.947 DEBUG:boto:Data: 2022-09-02T05:53:11.947 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:11.948 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.949 DEBUG:boto:Params: {} 2022-09-02T05:53:11.950 DEBUG:boto:Token: None 2022-09-02T05:53:11.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95 2022-09-02T05:53:11.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J/Cja4XJwCqzyR8NqU3CnzC/qvI= 2022-09-02T05:53:11.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J/Cja4XJwCqzyR8NqU3CnzC/qvI=', 'Content-Length': '0'} 2022-09-02T05:53:11.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb3e1decc9486f4b-0063119a47-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:11.955 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:11.956 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:11.956 DEBUG:boto:Method: GET 2022-09-02T05:53:11.957 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:11.957 DEBUG:boto:Data: 2022-09-02T05:53:11.958 DEBUG:boto:Headers: {} 2022-09-02T05:53:11.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:11.959 DEBUG:boto:Port: 8000 2022-09-02T05:53:11.960 DEBUG:boto:Params: {} 2022-09-02T05:53:11.960 DEBUG:boto:Token: None 2022-09-02T05:53:11.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:11 GMT /xljbkf-74/obj95 2022-09-02T05:53:11.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J/Cja4XJwCqzyR8NqU3CnzC/qvI= 2022-09-02T05:53:11.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J/Cja4XJwCqzyR8NqU3CnzC/qvI=', 'Content-Length': '0'} 2022-09-02T05:53:11.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008caebc6214eaf62c-0063119a47-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.006 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.007 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.007 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:12.008 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:12.008 DEBUG:boto:Method: GET 2022-09-02T05:53:12.009 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:12.010 DEBUG:boto:Data: 2022-09-02T05:53:12.010 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.011 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:12.011 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.012 DEBUG:boto:Params: {} 2022-09-02T05:53:12.012 DEBUG:boto:Token: None 2022-09-02T05:53:12.013 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.014 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a20654b616cd8d-0063119a48-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.018 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.019 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.020 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.020 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:12.021 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:12.022 DEBUG:boto:Method: GET 2022-09-02T05:53:12.022 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:12.023 DEBUG:boto:Data: 2022-09-02T05:53:12.024 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.025 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.025 DEBUG:boto:Params: {} 2022-09-02T05:53:12.026 DEBUG:boto:Token: None 2022-09-02T05:53:12.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d2ffee1df4da281-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.033 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:53:12.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:53:12.035 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.036 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.036 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.037 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:12.038 DEBUG:boto:Data: 2022-09-02T05:53:12.038 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.039 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.040 DEBUG:boto:Params: {} 2022-09-02T05:53:12.041 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:53:12.041 DEBUG:boto:Token: None 2022-09-02T05:53:12.042 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OCeDyYSf0MjyOtjzG4OvLqk/9Rk= 2022-09-02T05:53:12.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OCeDyYSf0MjyOtjzG4OvLqk/9Rk=', 'Content-Length': '0'} 2022-09-02T05:53:12.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e54b733963c936ce-0063119a48-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.049 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.049 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.050 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.050 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:12.051 DEBUG:boto:Data: 2022-09-02T05:53:12.051 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.052 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.053 DEBUG:boto:Params: {} 2022-09-02T05:53:12.053 DEBUG:boto:Token: None 2022-09-02T05:53:12.054 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OCeDyYSf0MjyOtjzG4OvLqk/9Rk= 2022-09-02T05:53:12.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OCeDyYSf0MjyOtjzG4OvLqk/9Rk=', 'Content-Length': '0'} 2022-09-02T05:53:12.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000735d1da77f19077e-0063119a48-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.059 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:53:12.060 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.061 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.061 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.062 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.063 DEBUG:boto:Method: GET 2022-09-02T05:53:12.063 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.064 DEBUG:boto:Data: 2022-09-02T05:53:12.064 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.065 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.066 DEBUG:boto:Params: {} 2022-09-02T05:53:12.067 DEBUG:boto:Token: None 2022-09-02T05:53:12.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/?versions 2022-09-02T05:53:12.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s= 2022-09-02T05:53:12.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s=', 'Content-Length': '0'} 2022-09-02T05:53:12.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004658fbf0dea633c5-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.073 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:53:12.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:53:12.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:53:12.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:53:12.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:53:12.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:53:12.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:53:12.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:53:12.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:53:12.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:53:12.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:53:12.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:53:12.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:53:12.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:53:12.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:53:12.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:53:12.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:53:12.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:53:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:53:12.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:53:12.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:53:12.087 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.087 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.088 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.088 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.089 DEBUG:boto:Method: GET 2022-09-02T05:53:12.089 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.090 DEBUG:boto:Data: 2022-09-02T05:53:12.090 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.091 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.092 DEBUG:boto:Params: {} 2022-09-02T05:53:12.093 DEBUG:boto:Token: None 2022-09-02T05:53:12.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/?versions 2022-09-02T05:53:12.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s= 2022-09-02T05:53:12.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s=', 'Content-Length': '0'} 2022-09-02T05:53:12.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48e2fc9d391a951-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.099 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:53:12.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:53:12.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:53:12.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:53:12.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:53:12.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:53:12.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:53:12.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:53:12.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:53:12.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:53:12.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:53:12.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:53:12.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:53:12.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:53:12.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:53:12.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:53:12.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:53:12.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:53:12.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:53:12.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:53:12.111 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.111 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.112 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.112 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.113 DEBUG:boto:Method: GET 2022-09-02T05:53:12.113 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.114 DEBUG:boto:Data: 2022-09-02T05:53:12.114 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.115 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.116 DEBUG:boto:Params: {} 2022-09-02T05:53:12.117 DEBUG:boto:Token: None 2022-09-02T05:53:12.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/?versions 2022-09-02T05:53:12.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s= 2022-09-02T05:53:12.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s=', 'Content-Length': '0'} 2022-09-02T05:53:12.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d60b24ff132a9ac0-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.123 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.125 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.125 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.126 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.127 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.127 DEBUG:boto:Method: GET 2022-09-02T05:53:12.128 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:12.128 DEBUG:boto:Data: 2022-09-02T05:53:12.129 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.130 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.130 DEBUG:boto:Params: {} 2022-09-02T05:53:12.131 DEBUG:boto:Token: None 2022-09-02T05:53:12.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/?versions 2022-09-02T05:53:12.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s= 2022-09-02T05:53:12.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PYkBsgxDsuPrrT6bhRB+6+yg64s=', 'Content-Length': '0'} 2022-09-02T05:53:12.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000409ef4aa852741aa-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.138 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:12.141 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.141 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.142 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.142 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.143 DEBUG:boto:Method: GET 2022-09-02T05:53:12.144 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.144 DEBUG:boto:Data: 2022-09-02T05:53:12.145 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.146 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.147 DEBUG:boto:Params: {} 2022-09-02T05:53:12.148 DEBUG:boto:Token: None 2022-09-02T05:53:12.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U= 2022-09-02T05:53:12.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U=', 'Content-Length': '0'} 2022-09-02T05:53:12.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c89314c13e3db8a-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.154 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.155 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.155 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.156 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.156 DEBUG:boto:Method: GET 2022-09-02T05:53:12.157 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.158 DEBUG:boto:Data: 2022-09-02T05:53:12.159 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.161 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.161 DEBUG:boto:Params: {} 2022-09-02T05:53:12.162 DEBUG:boto:Token: None 2022-09-02T05:53:12.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U= 2022-09-02T05:53:12.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U=', 'Content-Length': '0'} 2022-09-02T05:53:12.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d41d02af28c5bda5-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.210 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.211 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.212 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.212 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.213 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.213 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.214 DEBUG:boto:Data: 2022-09-02T05:53:12.214 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.215 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.216 DEBUG:boto:Params: {} 2022-09-02T05:53:12.216 DEBUG:boto:Token: None 2022-09-02T05:53:12.217 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vZv2ieMlE5UKsKcDEOdEPdHK9E= 2022-09-02T05:53:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vZv2ieMlE5UKsKcDEOdEPdHK9E=', 'Content-Length': '0'} 2022-09-02T05:53:12.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005962169287e8e020-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.221 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.222 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.222 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.223 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.223 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.224 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.224 DEBUG:boto:Data: 2022-09-02T05:53:12.225 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.226 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.226 DEBUG:boto:Params: {} 2022-09-02T05:53:12.227 DEBUG:boto:Token: None 2022-09-02T05:53:12.228 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2vZv2ieMlE5UKsKcDEOdEPdHK9E= 2022-09-02T05:53:12.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2vZv2ieMlE5UKsKcDEOdEPdHK9E=', 'Content-Length': '0'} 2022-09-02T05:53:12.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8609e5f7cfc148a-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:12.232 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.233 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.233 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.234 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.235 DEBUG:boto:Method: GET 2022-09-02T05:53:12.235 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.236 DEBUG:boto:Data: 2022-09-02T05:53:12.236 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.237 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.238 DEBUG:boto:Params: {} 2022-09-02T05:53:12.239 DEBUG:boto:Token: None 2022-09-02T05:53:12.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U= 2022-09-02T05:53:12.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U=', 'Content-Length': '0'} 2022-09-02T05:53:12.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009adc67830cb05f41-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.244 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.244 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.245 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.245 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.246 DEBUG:boto:Method: GET 2022-09-02T05:53:12.246 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.247 DEBUG:boto:Data: 2022-09-02T05:53:12.247 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.248 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.249 DEBUG:boto:Params: {} 2022-09-02T05:53:12.250 DEBUG:boto:Token: None 2022-09-02T05:53:12.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:12.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U= 2022-09-02T05:53:12.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GmhIgv54gUHx2np3Vk9fgiqRT5U=', 'Content-Length': '0'} 2022-09-02T05:53:12.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983c37a351e32775-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.255 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.255 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.256 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.257 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.257 DEBUG:boto:Method: GET 2022-09-02T05:53:12.258 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.259 DEBUG:boto:Data: 2022-09-02T05:53:12.259 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.260 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.261 DEBUG:boto:Params: {} 2022-09-02T05:53:12.261 DEBUG:boto:Token: None 2022-09-02T05:53:12.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0eb0c1e6b28e592-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.310 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.311 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.312 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.312 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.313 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.313 DEBUG:boto:Method: GET 2022-09-02T05:53:12.314 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.315 DEBUG:boto:Data: 2022-09-02T05:53:12.315 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.316 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.317 DEBUG:boto:Params: {} 2022-09-02T05:53:12.317 DEBUG:boto:Token: None 2022-09-02T05:53:12.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009c7a1407b38e9d3-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.323 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.324 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.325 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.325 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.326 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:12.327 DEBUG:boto:Data: 2022-09-02T05:53:12.328 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.330 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.331 DEBUG:boto:Params: {} 2022-09-02T05:53:12.332 DEBUG:boto:Token: None 2022-09-02T05:53:12.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11 2022-09-02T05:53:12.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F4zO9qHUT2xKPRRED6eoBCrmcoc= 2022-09-02T05:53:12.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F4zO9qHUT2xKPRRED6eoBCrmcoc=', 'Content-Length': '0'} 2022-09-02T05:53:12.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daeaa1dfd8abdbca-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.339 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.339 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.340 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.340 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:12.341 DEBUG:boto:Data: 2022-09-02T05:53:12.342 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.343 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.343 DEBUG:boto:Params: {} 2022-09-02T05:53:12.344 DEBUG:boto:Token: None 2022-09-02T05:53:12.344 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11 2022-09-02T05:53:12.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F4zO9qHUT2xKPRRED6eoBCrmcoc= 2022-09-02T05:53:12.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F4zO9qHUT2xKPRRED6eoBCrmcoc=', 'Content-Length': '0'} 2022-09-02T05:53:12.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c937cf6bb96f08c-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:12.349 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.349 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.350 DEBUG:boto:Method: GET 2022-09-02T05:53:12.351 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:12.351 DEBUG:boto:Data: 2022-09-02T05:53:12.352 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.353 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.354 DEBUG:boto:Params: {} 2022-09-02T05:53:12.355 DEBUG:boto:Token: None 2022-09-02T05:53:12.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11 2022-09-02T05:53:12.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nf0D/iSC9nmThbGZgAs9PlqXFZI= 2022-09-02T05:53:12.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nf0D/iSC9nmThbGZgAs9PlqXFZI=', 'Content-Length': '0'} 2022-09-02T05:53:12.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff89e8c37028c37-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.360 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:12.360 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:12.361 DEBUG:boto:Method: GET 2022-09-02T05:53:12.361 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:12.362 DEBUG:boto:Data: 2022-09-02T05:53:12.363 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.364 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.365 DEBUG:boto:Params: {} 2022-09-02T05:53:12.366 DEBUG:boto:Token: None 2022-09-02T05:53:12.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj11 2022-09-02T05:53:12.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nf0D/iSC9nmThbGZgAs9PlqXFZI= 2022-09-02T05:53:12.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nf0D/iSC9nmThbGZgAs9PlqXFZI=', 'Content-Length': '0'} 2022-09-02T05:53:12.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c10b38548ea0e9-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.410 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.411 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.411 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.412 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.413 DEBUG:boto:Method: GET 2022-09-02T05:53:12.413 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.414 DEBUG:boto:Data: 2022-09-02T05:53:12.414 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.415 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.416 DEBUG:boto:Params: {} 2022-09-02T05:53:12.417 DEBUG:boto:Token: None 2022-09-02T05:53:12.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfaf47221a140948-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.422 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.423 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.424 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.424 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.425 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.425 DEBUG:boto:Method: GET 2022-09-02T05:53:12.426 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:12.427 DEBUG:boto:Data: 2022-09-02T05:53:12.427 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.429 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.429 DEBUG:boto:Params: {} 2022-09-02T05:53:12.430 DEBUG:boto:Token: None 2022-09-02T05:53:12.431 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3350f9f7457e79d-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.436 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:12.438 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.438 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.439 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.439 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.440 DEBUG:boto:Method: GET 2022-09-02T05:53:12.440 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.441 DEBUG:boto:Data: 2022-09-02T05:53:12.441 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.443 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.443 DEBUG:boto:Params: {} 2022-09-02T05:53:12.444 DEBUG:boto:Token: None 2022-09-02T05:53:12.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8= 2022-09-02T05:53:12.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8=', 'Content-Length': '0'} 2022-09-02T05:53:12.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a2bf9269167169-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.449 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.450 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.451 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.451 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.452 DEBUG:boto:Method: GET 2022-09-02T05:53:12.453 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.453 DEBUG:boto:Data: 2022-09-02T05:53:12.454 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.455 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.456 DEBUG:boto:Params: {} 2022-09-02T05:53:12.456 DEBUG:boto:Token: None 2022-09-02T05:53:12.457 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8= 2022-09-02T05:53:12.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8=', 'Content-Length': '0'} 2022-09-02T05:53:12.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002900321c73b3bf17-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.462 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.462 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.463 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.463 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.464 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.464 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.465 DEBUG:boto:Data: 2022-09-02T05:53:12.465 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.467 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.467 DEBUG:boto:Params: {} 2022-09-02T05:53:12.468 DEBUG:boto:Token: None 2022-09-02T05:53:12.468 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7EjnfXKNRIMuWeYEuK4/0Ai10W8= 2022-09-02T05:53:12.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7EjnfXKNRIMuWeYEuK4/0Ai10W8=', 'Content-Length': '0'} 2022-09-02T05:53:12.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079888fd3031450e5-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.473 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.474 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.475 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.476 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.477 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.478 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.479 DEBUG:boto:Data: 2022-09-02T05:53:12.480 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.482 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.483 DEBUG:boto:Params: {} 2022-09-02T05:53:12.484 DEBUG:boto:Token: None 2022-09-02T05:53:12.485 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7EjnfXKNRIMuWeYEuK4/0Ai10W8= 2022-09-02T05:53:12.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7EjnfXKNRIMuWeYEuK4/0Ai10W8=', 'Content-Length': '0'} 2022-09-02T05:53:12.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ace30f7e2d4508-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:12.491 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.491 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.492 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.492 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.493 DEBUG:boto:Method: GET 2022-09-02T05:53:12.493 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.494 DEBUG:boto:Data: 2022-09-02T05:53:12.494 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.495 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.496 DEBUG:boto:Params: {} 2022-09-02T05:53:12.497 DEBUG:boto:Token: None 2022-09-02T05:53:12.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8= 2022-09-02T05:53:12.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8=', 'Content-Length': '0'} 2022-09-02T05:53:12.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008983a50d9d85f6ff-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.502 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.503 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.503 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.504 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.504 DEBUG:boto:Method: GET 2022-09-02T05:53:12.505 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.505 DEBUG:boto:Data: 2022-09-02T05:53:12.506 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.507 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.508 DEBUG:boto:Params: {} 2022-09-02T05:53:12.508 DEBUG:boto:Token: None 2022-09-02T05:53:12.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:12.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8= 2022-09-02T05:53:12.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R6qzi9shr0SbBQwWKRK8FJ3YNH8=', 'Content-Length': '0'} 2022-09-02T05:53:12.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656b2317a3eedb90-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.514 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.515 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.515 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.516 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.516 DEBUG:boto:Method: GET 2022-09-02T05:53:12.517 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.517 DEBUG:boto:Data: 2022-09-02T05:53:12.518 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.519 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.520 DEBUG:boto:Params: {} 2022-09-02T05:53:12.521 DEBUG:boto:Token: None 2022-09-02T05:53:12.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.522 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc93b7081960172c-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.527 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.528 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.528 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.529 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.529 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.530 DEBUG:boto:Method: GET 2022-09-02T05:53:12.530 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.531 DEBUG:boto:Data: 2022-09-02T05:53:12.532 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.533 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.533 DEBUG:boto:Params: {} 2022-09-02T05:53:12.534 DEBUG:boto:Token: None 2022-09-02T05:53:12.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af88ef0dd00fec1-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.540 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.541 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.541 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.542 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.543 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:12.543 DEBUG:boto:Data: 2022-09-02T05:53:12.544 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.545 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.545 DEBUG:boto:Params: {} 2022-09-02T05:53:12.546 DEBUG:boto:Token: None 2022-09-02T05:53:12.546 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115 2022-09-02T05:53:12.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ioRKVcuG7OwpKJxv95Yo5EvGYKo= 2022-09-02T05:53:12.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ioRKVcuG7OwpKJxv95Yo5EvGYKo=', 'Content-Length': '0'} 2022-09-02T05:53:12.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b605d8321b54dc59-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.551 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.551 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.552 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.552 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:12.553 DEBUG:boto:Data: 2022-09-02T05:53:12.553 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.554 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.555 DEBUG:boto:Params: {} 2022-09-02T05:53:12.555 DEBUG:boto:Token: None 2022-09-02T05:53:12.556 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115 2022-09-02T05:53:12.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ioRKVcuG7OwpKJxv95Yo5EvGYKo= 2022-09-02T05:53:12.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ioRKVcuG7OwpKJxv95Yo5EvGYKo=', 'Content-Length': '0'} 2022-09-02T05:53:12.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045324f2cbf0ff5ef-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:12.561 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.561 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.562 DEBUG:boto:Method: GET 2022-09-02T05:53:12.562 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:12.563 DEBUG:boto:Data: 2022-09-02T05:53:12.563 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.564 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.565 DEBUG:boto:Params: {} 2022-09-02T05:53:12.566 DEBUG:boto:Token: None 2022-09-02T05:53:12.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115 2022-09-02T05:53:12.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XgwpQaRormttKgFAaf1ynHa0jrI= 2022-09-02T05:53:12.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XgwpQaRormttKgFAaf1ynHa0jrI=', 'Content-Length': '0'} 2022-09-02T05:53:12.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ff4b73076a0155-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.571 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:12.571 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:12.572 DEBUG:boto:Method: GET 2022-09-02T05:53:12.573 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:12.573 DEBUG:boto:Data: 2022-09-02T05:53:12.574 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.575 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.576 DEBUG:boto:Params: {} 2022-09-02T05:53:12.576 DEBUG:boto:Token: None 2022-09-02T05:53:12.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj115 2022-09-02T05:53:12.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XgwpQaRormttKgFAaf1ynHa0jrI= 2022-09-02T05:53:12.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XgwpQaRormttKgFAaf1ynHa0jrI=', 'Content-Length': '0'} 2022-09-02T05:53:12.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005c32e0a0afab473-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.622 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.623 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.623 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.624 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.625 DEBUG:boto:Method: GET 2022-09-02T05:53:12.626 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.627 DEBUG:boto:Data: 2022-09-02T05:53:12.627 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.629 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.629 DEBUG:boto:Params: {} 2022-09-02T05:53:12.630 DEBUG:boto:Token: None 2022-09-02T05:53:12.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc8a35f039917e32-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.636 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.637 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.638 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.638 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.639 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.639 DEBUG:boto:Method: GET 2022-09-02T05:53:12.640 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:12.640 DEBUG:boto:Data: 2022-09-02T05:53:12.641 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.642 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.642 DEBUG:boto:Params: {} 2022-09-02T05:53:12.643 DEBUG:boto:Token: None 2022-09-02T05:53:12.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9c10f3b7cce239-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.649 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:12.651 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.651 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.652 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.653 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.654 DEBUG:boto:Method: GET 2022-09-02T05:53:12.655 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.656 DEBUG:boto:Data: 2022-09-02T05:53:12.656 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.658 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.659 DEBUG:boto:Params: {} 2022-09-02T05:53:12.660 DEBUG:boto:Token: None 2022-09-02T05:53:12.662 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4= 2022-09-02T05:53:12.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4=', 'Content-Length': '0'} 2022-09-02T05:53:12.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efef12753190fd10-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.710 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.712 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.713 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.714 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.715 DEBUG:boto:Method: GET 2022-09-02T05:53:12.715 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.716 DEBUG:boto:Data: 2022-09-02T05:53:12.717 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.718 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.719 DEBUG:boto:Params: {} 2022-09-02T05:53:12.721 DEBUG:boto:Token: None 2022-09-02T05:53:12.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4= 2022-09-02T05:53:12.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4=', 'Content-Length': '0'} 2022-09-02T05:53:12.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054610f988f0ae8ba-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.729 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.730 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.731 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.732 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.733 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.734 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.735 DEBUG:boto:Data: 2022-09-02T05:53:12.736 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.738 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.739 DEBUG:boto:Params: {} 2022-09-02T05:53:12.740 DEBUG:boto:Token: None 2022-09-02T05:53:12.741 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QCKjs4moHUCGIrm66eIG5nWUmjQ= 2022-09-02T05:53:12.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QCKjs4moHUCGIrm66eIG5nWUmjQ=', 'Content-Length': '0'} 2022-09-02T05:53:12.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab29d37de15d9b1-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.747 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.748 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.749 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.750 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.751 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.752 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.753 DEBUG:boto:Data: 2022-09-02T05:53:12.753 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.755 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.756 DEBUG:boto:Params: {} 2022-09-02T05:53:12.757 DEBUG:boto:Token: None 2022-09-02T05:53:12.758 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QCKjs4moHUCGIrm66eIG5nWUmjQ= 2022-09-02T05:53:12.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QCKjs4moHUCGIrm66eIG5nWUmjQ=', 'Content-Length': '0'} 2022-09-02T05:53:12.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e61dd3c3c80fe56-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:12.764 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.764 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.765 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.766 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.766 DEBUG:boto:Method: GET 2022-09-02T05:53:12.778 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.779 DEBUG:boto:Data: 2022-09-02T05:53:12.779 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.780 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.781 DEBUG:boto:Params: {} 2022-09-02T05:53:12.782 DEBUG:boto:Token: None 2022-09-02T05:53:12.782 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4= 2022-09-02T05:53:12.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4=', 'Content-Length': '0'} 2022-09-02T05:53:12.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006489778f67ec6bb3-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.787 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.788 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.788 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.789 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.789 DEBUG:boto:Method: GET 2022-09-02T05:53:12.790 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.790 DEBUG:boto:Data: 2022-09-02T05:53:12.791 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.792 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.792 DEBUG:boto:Params: {} 2022-09-02T05:53:12.793 DEBUG:boto:Token: None 2022-09-02T05:53:12.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:12.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4= 2022-09-02T05:53:12.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs57K/Ny5iLtGMkd1ohtG9gR7s4=', 'Content-Length': '0'} 2022-09-02T05:53:12.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd25cd3e84f8255d-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.798 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.799 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.799 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.800 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.800 DEBUG:boto:Method: GET 2022-09-02T05:53:12.801 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.801 DEBUG:boto:Data: 2022-09-02T05:53:12.802 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.803 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.804 DEBUG:boto:Params: {} 2022-09-02T05:53:12.804 DEBUG:boto:Token: None 2022-09-02T05:53:12.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000537aa917e5fd8529-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.810 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.811 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.811 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.812 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.812 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.813 DEBUG:boto:Method: GET 2022-09-02T05:53:12.814 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.814 DEBUG:boto:Data: 2022-09-02T05:53:12.815 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.816 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.816 DEBUG:boto:Params: {} 2022-09-02T05:53:12.817 DEBUG:boto:Token: None 2022-09-02T05:53:12.817 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef54d79b98f9dbc-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.823 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.824 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.824 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.825 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.825 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:12.826 DEBUG:boto:Data: 2022-09-02T05:53:12.826 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.827 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.828 DEBUG:boto:Params: {} 2022-09-02T05:53:12.828 DEBUG:boto:Token: None 2022-09-02T05:53:12.829 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17 2022-09-02T05:53:12.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KcV9qjL7vRnOyP5nr8Q5D+57zNg= 2022-09-02T05:53:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KcV9qjL7vRnOyP5nr8Q5D+57zNg=', 'Content-Length': '0'} 2022-09-02T05:53:12.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3aa8fa2d6d105a9-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.833 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.834 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.834 DEBUG:boto:Method: HEAD 2022-09-02T05:53:12.835 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:12.835 DEBUG:boto:Data: 2022-09-02T05:53:12.836 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.837 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.838 DEBUG:boto:Params: {} 2022-09-02T05:53:12.838 DEBUG:boto:Token: None 2022-09-02T05:53:12.839 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17 2022-09-02T05:53:12.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KcV9qjL7vRnOyP5nr8Q5D+57zNg= 2022-09-02T05:53:12.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KcV9qjL7vRnOyP5nr8Q5D+57zNg=', 'Content-Length': '0'} 2022-09-02T05:53:12.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f513eefa0b37056-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:12.844 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.844 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.845 DEBUG:boto:Method: GET 2022-09-02T05:53:12.845 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:12.846 DEBUG:boto:Data: 2022-09-02T05:53:12.846 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.847 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.848 DEBUG:boto:Params: {} 2022-09-02T05:53:12.848 DEBUG:boto:Token: None 2022-09-02T05:53:12.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17 2022-09-02T05:53:12.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GfllvkEm42/xssyH+fLW8ltaBt4= 2022-09-02T05:53:12.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GfllvkEm42/xssyH+fLW8ltaBt4=', 'Content-Length': '0'} 2022-09-02T05:53:12.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f8d5fa9d6f59be-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.854 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:12.854 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:12.855 DEBUG:boto:Method: GET 2022-09-02T05:53:12.855 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:12.856 DEBUG:boto:Data: 2022-09-02T05:53:12.856 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.857 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.858 DEBUG:boto:Params: {} 2022-09-02T05:53:12.859 DEBUG:boto:Token: None 2022-09-02T05:53:12.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj17 2022-09-02T05:53:12.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GfllvkEm42/xssyH+fLW8ltaBt4= 2022-09-02T05:53:12.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GfllvkEm42/xssyH+fLW8ltaBt4=', 'Content-Length': '0'} 2022-09-02T05:53:12.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e48267d2c8fc6ad-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.906 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.907 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.907 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.908 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.909 DEBUG:boto:Method: GET 2022-09-02T05:53:12.909 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.910 DEBUG:boto:Data: 2022-09-02T05:53:12.911 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.912 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.912 DEBUG:boto:Params: {} 2022-09-02T05:53:12.913 DEBUG:boto:Token: None 2022-09-02T05:53:12.913 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8bfcd04ed622a6-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.919 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.920 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:12.920 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:12.921 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.921 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.922 DEBUG:boto:Method: GET 2022-09-02T05:53:12.923 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:12.923 DEBUG:boto:Data: 2022-09-02T05:53:12.924 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.925 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.925 DEBUG:boto:Params: {} 2022-09-02T05:53:12.926 DEBUG:boto:Token: None 2022-09-02T05:53:12.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/ 2022-09-02T05:53:12.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw= 2022-09-02T05:53:12.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lc9kYrouBohvjjCNiM//bl3PjUw=', 'Content-Length': '0'} 2022-09-02T05:53:12.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e5aa1e1c70b2ac4-0063119a48-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.932 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:12.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:12.934 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:12.934 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:12.935 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.935 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.936 DEBUG:boto:Method: GET 2022-09-02T05:53:12.936 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.937 DEBUG:boto:Data: 2022-09-02T05:53:12.937 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:12.938 DEBUG:boto:Port: 8001 2022-09-02T05:53:12.939 DEBUG:boto:Params: {} 2022-09-02T05:53:12.940 DEBUG:boto:Token: None 2022-09-02T05:53:12.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1SLmxErK4WC4JAn78Rk4uL1TesM= 2022-09-02T05:53:12.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1SLmxErK4WC4JAn78Rk4uL1TesM=', 'Content-Length': '0'} 2022-09-02T05:53:12.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019be882b70c1fb21-0063119a48-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.986 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:12.987 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:12.988 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.989 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.989 DEBUG:boto:Method: GET 2022-09-02T05:53:12.990 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.990 DEBUG:boto:Data: 2022-09-02T05:53:12.991 DEBUG:boto:Headers: {} 2022-09-02T05:53:12.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:12.992 DEBUG:boto:Port: 8000 2022-09-02T05:53:12.993 DEBUG:boto:Params: {} 2022-09-02T05:53:12.993 DEBUG:boto:Token: None 2022-09-02T05:53:12.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:12 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:12.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1SLmxErK4WC4JAn78Rk4uL1TesM= 2022-09-02T05:53:12.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1SLmxErK4WC4JAn78Rk4uL1TesM=', 'Content-Length': '0'} 2022-09-02T05:53:12.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd4413510bb0554-0063119a48-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:12.999 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:12.999 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.000 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.000 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.001 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.002 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.002 DEBUG:boto:Data: 2022-09-02T05:53:13.003 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.004 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.004 DEBUG:boto:Params: {} 2022-09-02T05:53:13.005 DEBUG:boto:Token: None 2022-09-02T05:53:13.006 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d614AHO24U6OFLsDSxKTR2PUTP8= 2022-09-02T05:53:13.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d614AHO24U6OFLsDSxKTR2PUTP8=', 'Content-Length': '0'} 2022-09-02T05:53:13.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b1a981db1891930-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.010 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.010 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.011 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.011 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.012 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.013 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.013 DEBUG:boto:Data: 2022-09-02T05:53:13.014 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.015 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.015 DEBUG:boto:Params: {} 2022-09-02T05:53:13.016 DEBUG:boto:Token: None 2022-09-02T05:53:13.016 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d614AHO24U6OFLsDSxKTR2PUTP8= 2022-09-02T05:53:13.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d614AHO24U6OFLsDSxKTR2PUTP8=', 'Content-Length': '0'} 2022-09-02T05:53:13.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b7902cf5d6c62e7-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:13.021 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.022 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.022 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.023 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.024 DEBUG:boto:Method: GET 2022-09-02T05:53:13.024 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.025 DEBUG:boto:Data: 2022-09-02T05:53:13.025 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.027 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.027 DEBUG:boto:Params: {} 2022-09-02T05:53:13.028 DEBUG:boto:Token: None 2022-09-02T05:53:13.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j58+jLQWAyfvr4PKzJlfbej3NCU= 2022-09-02T05:53:13.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j58+jLQWAyfvr4PKzJlfbej3NCU=', 'Content-Length': '0'} 2022-09-02T05:53:13.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3486f6883f8555-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.033 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.034 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.034 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.035 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.035 DEBUG:boto:Method: GET 2022-09-02T05:53:13.036 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.036 DEBUG:boto:Data: 2022-09-02T05:53:13.037 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.038 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.039 DEBUG:boto:Params: {} 2022-09-02T05:53:13.040 DEBUG:boto:Token: None 2022-09-02T05:53:13.040 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:13.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j58+jLQWAyfvr4PKzJlfbej3NCU= 2022-09-02T05:53:13.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j58+jLQWAyfvr4PKzJlfbej3NCU=', 'Content-Length': '0'} 2022-09-02T05:53:13.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df53156a12235b7-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.045 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.045 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.046 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.047 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.048 DEBUG:boto:Method: GET 2022-09-02T05:53:13.049 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.050 DEBUG:boto:Data: 2022-09-02T05:53:13.051 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.053 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.054 DEBUG:boto:Params: {} 2022-09-02T05:53:13.055 DEBUG:boto:Token: None 2022-09-02T05:53:13.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31e416042360c50-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.060 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.061 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.061 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.062 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.062 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.063 DEBUG:boto:Method: GET 2022-09-02T05:53:13.063 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.064 DEBUG:boto:Data: 2022-09-02T05:53:13.064 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.065 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.066 DEBUG:boto:Params: {} 2022-09-02T05:53:13.066 DEBUG:boto:Token: None 2022-09-02T05:53:13.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ba633c70ab6046-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.073 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.073 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.074 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.074 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.075 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:13.075 DEBUG:boto:Data: 2022-09-02T05:53:13.076 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.077 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.077 DEBUG:boto:Params: {} 2022-09-02T05:53:13.078 DEBUG:boto:Token: None 2022-09-02T05:53:13.079 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19 2022-09-02T05:53:13.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SnmaN4ELeUxYPbJqJOLBhWKFYfk= 2022-09-02T05:53:13.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SnmaN4ELeUxYPbJqJOLBhWKFYfk=', 'Content-Length': '0'} 2022-09-02T05:53:13.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000788f14ab10f13e55-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.082 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.083 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.084 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.084 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:13.085 DEBUG:boto:Data: 2022-09-02T05:53:13.085 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.086 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.086 DEBUG:boto:Params: {} 2022-09-02T05:53:13.087 DEBUG:boto:Token: None 2022-09-02T05:53:13.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19 2022-09-02T05:53:13.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SnmaN4ELeUxYPbJqJOLBhWKFYfk= 2022-09-02T05:53:13.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SnmaN4ELeUxYPbJqJOLBhWKFYfk=', 'Content-Length': '0'} 2022-09-02T05:53:13.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71cc017649efa6e-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:13.092 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.093 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.093 DEBUG:boto:Method: GET 2022-09-02T05:53:13.094 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:13.094 DEBUG:boto:Data: 2022-09-02T05:53:13.095 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.096 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.097 DEBUG:boto:Params: {} 2022-09-02T05:53:13.097 DEBUG:boto:Token: None 2022-09-02T05:53:13.098 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19 2022-09-02T05:53:13.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TxHUMmgM81SJFSjjjOLrQ9k3an8= 2022-09-02T05:53:13.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TxHUMmgM81SJFSjjjOLrQ9k3an8=', 'Content-Length': '0'} 2022-09-02T05:53:13.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed02ad23e1633da7-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.102 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:13.102 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:13.103 DEBUG:boto:Method: GET 2022-09-02T05:53:13.104 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:13.104 DEBUG:boto:Data: 2022-09-02T05:53:13.105 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.106 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.106 DEBUG:boto:Params: {} 2022-09-02T05:53:13.107 DEBUG:boto:Token: None 2022-09-02T05:53:13.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj19 2022-09-02T05:53:13.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TxHUMmgM81SJFSjjjOLrQ9k3an8= 2022-09-02T05:53:13.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TxHUMmgM81SJFSjjjOLrQ9k3an8=', 'Content-Length': '0'} 2022-09-02T05:53:13.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a7d3d519e95c10f-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.158 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.159 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.160 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.160 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.161 DEBUG:boto:Method: GET 2022-09-02T05:53:13.161 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.162 DEBUG:boto:Data: 2022-09-02T05:53:13.163 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.164 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.165 DEBUG:boto:Params: {} 2022-09-02T05:53:13.165 DEBUG:boto:Token: None 2022-09-02T05:53:13.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efe173bdb5693577-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.171 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.172 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.172 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.173 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.173 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.174 DEBUG:boto:Method: GET 2022-09-02T05:53:13.174 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:13.175 DEBUG:boto:Data: 2022-09-02T05:53:13.175 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.176 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.177 DEBUG:boto:Params: {} 2022-09-02T05:53:13.178 DEBUG:boto:Token: None 2022-09-02T05:53:13.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4b245bdb53bcd95-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.185 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:13.187 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.187 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.188 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.189 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.190 DEBUG:boto:Method: GET 2022-09-02T05:53:13.191 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.192 DEBUG:boto:Data: 2022-09-02T05:53:13.193 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.196 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.197 DEBUG:boto:Params: {} 2022-09-02T05:53:13.198 DEBUG:boto:Token: None 2022-09-02T05:53:13.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk= 2022-09-02T05:53:13.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk=', 'Content-Length': '0'} 2022-09-02T05:53:13.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8225fb8312804a-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.250 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.251 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.251 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.252 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.252 DEBUG:boto:Method: GET 2022-09-02T05:53:13.253 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.253 DEBUG:boto:Data: 2022-09-02T05:53:13.254 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.255 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.255 DEBUG:boto:Params: {} 2022-09-02T05:53:13.256 DEBUG:boto:Token: None 2022-09-02T05:53:13.258 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk= 2022-09-02T05:53:13.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk=', 'Content-Length': '0'} 2022-09-02T05:53:13.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0463cab858d9ca-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.263 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.264 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.264 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.265 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.265 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.266 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.266 DEBUG:boto:Data: 2022-09-02T05:53:13.267 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.268 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.268 DEBUG:boto:Params: {} 2022-09-02T05:53:13.269 DEBUG:boto:Token: None 2022-09-02T05:53:13.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MGlRLsRTKtUz7t9lflDwnILL9wE= 2022-09-02T05:53:13.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MGlRLsRTKtUz7t9lflDwnILL9wE=', 'Content-Length': '0'} 2022-09-02T05:53:13.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f13052577227155-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.273 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.274 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.274 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.275 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.276 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.276 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.277 DEBUG:boto:Data: 2022-09-02T05:53:13.277 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.278 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.279 DEBUG:boto:Params: {} 2022-09-02T05:53:13.279 DEBUG:boto:Token: None 2022-09-02T05:53:13.280 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MGlRLsRTKtUz7t9lflDwnILL9wE= 2022-09-02T05:53:13.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MGlRLsRTKtUz7t9lflDwnILL9wE=', 'Content-Length': '0'} 2022-09-02T05:53:13.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3cf334da128d63d-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:13.286 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.287 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.287 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.288 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.289 DEBUG:boto:Method: GET 2022-09-02T05:53:13.290 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.291 DEBUG:boto:Data: 2022-09-02T05:53:13.292 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.294 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.295 DEBUG:boto:Params: {} 2022-09-02T05:53:13.296 DEBUG:boto:Token: None 2022-09-02T05:53:13.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk= 2022-09-02T05:53:13.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk=', 'Content-Length': '0'} 2022-09-02T05:53:13.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a357a8d512be814e-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.303 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.304 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.304 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.305 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.306 DEBUG:boto:Method: GET 2022-09-02T05:53:13.306 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.307 DEBUG:boto:Data: 2022-09-02T05:53:13.307 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.309 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.310 DEBUG:boto:Params: {} 2022-09-02T05:53:13.311 DEBUG:boto:Token: None 2022-09-02T05:53:13.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:13.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk= 2022-09-02T05:53:13.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YrxUnUPA0Ln5EJvvquPXGTFMXqk=', 'Content-Length': '0'} 2022-09-02T05:53:13.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948fc98605b236a9-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.317 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.317 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.318 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.319 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.320 DEBUG:boto:Method: GET 2022-09-02T05:53:13.321 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.322 DEBUG:boto:Data: 2022-09-02T05:53:13.323 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.326 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.327 DEBUG:boto:Params: {} 2022-09-02T05:53:13.328 DEBUG:boto:Token: None 2022-09-02T05:53:13.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbbcf7a3175b5f96-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.335 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.336 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.337 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.337 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.338 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.338 DEBUG:boto:Method: GET 2022-09-02T05:53:13.339 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.339 DEBUG:boto:Data: 2022-09-02T05:53:13.340 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.341 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.342 DEBUG:boto:Params: {} 2022-09-02T05:53:13.342 DEBUG:boto:Token: None 2022-09-02T05:53:13.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baee51427c7b51a7-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.348 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.349 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.350 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.351 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.351 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:13.352 DEBUG:boto:Data: 2022-09-02T05:53:13.352 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.353 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.354 DEBUG:boto:Params: {} 2022-09-02T05:53:13.355 DEBUG:boto:Token: None 2022-09-02T05:53:13.355 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22 2022-09-02T05:53:13.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8JnU30r8fwrYF+qXLcAeccO9Qic= 2022-09-02T05:53:13.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8JnU30r8fwrYF+qXLcAeccO9Qic=', 'Content-Length': '0'} 2022-09-02T05:53:13.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99e52356bd893ac-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.361 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.362 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.363 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.364 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:13.364 DEBUG:boto:Data: 2022-09-02T05:53:13.365 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.367 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.368 DEBUG:boto:Params: {} 2022-09-02T05:53:13.369 DEBUG:boto:Token: None 2022-09-02T05:53:13.370 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22 2022-09-02T05:53:13.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8JnU30r8fwrYF+qXLcAeccO9Qic= 2022-09-02T05:53:13.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8JnU30r8fwrYF+qXLcAeccO9Qic=', 'Content-Length': '0'} 2022-09-02T05:53:13.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58e0e3b4c5ba92e-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:13.379 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.380 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.380 DEBUG:boto:Method: GET 2022-09-02T05:53:13.381 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:13.382 DEBUG:boto:Data: 2022-09-02T05:53:13.382 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.383 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.384 DEBUG:boto:Params: {} 2022-09-02T05:53:13.385 DEBUG:boto:Token: None 2022-09-02T05:53:13.386 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22 2022-09-02T05:53:13.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8kS5OBGzbUs6QZvcua6MhDLe0lQ= 2022-09-02T05:53:13.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8kS5OBGzbUs6QZvcua6MhDLe0lQ=', 'Content-Length': '0'} 2022-09-02T05:53:13.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843ce1a6e113c0cf-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.434 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:13.435 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:13.436 DEBUG:boto:Method: GET 2022-09-02T05:53:13.437 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:13.438 DEBUG:boto:Data: 2022-09-02T05:53:13.439 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.441 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.442 DEBUG:boto:Params: {} 2022-09-02T05:53:13.443 DEBUG:boto:Token: None 2022-09-02T05:53:13.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj22 2022-09-02T05:53:13.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8kS5OBGzbUs6QZvcua6MhDLe0lQ= 2022-09-02T05:53:13.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8kS5OBGzbUs6QZvcua6MhDLe0lQ=', 'Content-Length': '0'} 2022-09-02T05:53:13.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bffe6133ab488728-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.453 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.454 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.455 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.455 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.456 DEBUG:boto:Method: GET 2022-09-02T05:53:13.457 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.458 DEBUG:boto:Data: 2022-09-02T05:53:13.459 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.460 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.461 DEBUG:boto:Params: {} 2022-09-02T05:53:13.462 DEBUG:boto:Token: None 2022-09-02T05:53:13.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5e0634dc5b11ad4-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.470 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.471 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.472 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.472 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.473 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.474 DEBUG:boto:Method: GET 2022-09-02T05:53:13.474 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:13.475 DEBUG:boto:Data: 2022-09-02T05:53:13.475 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.476 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.477 DEBUG:boto:Params: {} 2022-09-02T05:53:13.478 DEBUG:boto:Token: None 2022-09-02T05:53:13.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4040331570bf4b-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.484 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:13.485 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.486 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.487 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.487 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.488 DEBUG:boto:Method: GET 2022-09-02T05:53:13.488 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.489 DEBUG:boto:Data: 2022-09-02T05:53:13.489 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.490 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.491 DEBUG:boto:Params: {} 2022-09-02T05:53:13.491 DEBUG:boto:Token: None 2022-09-02T05:53:13.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY= 2022-09-02T05:53:13.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY=', 'Content-Length': '0'} 2022-09-02T05:53:13.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c306ad20cd3da9-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.497 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.498 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.499 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.499 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.500 DEBUG:boto:Method: GET 2022-09-02T05:53:13.500 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.501 DEBUG:boto:Data: 2022-09-02T05:53:13.501 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.503 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.503 DEBUG:boto:Params: {} 2022-09-02T05:53:13.504 DEBUG:boto:Token: None 2022-09-02T05:53:13.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY= 2022-09-02T05:53:13.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY=', 'Content-Length': '0'} 2022-09-02T05:53:13.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f20f3e5445510c5a-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.511 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.512 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.513 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.514 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.514 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.515 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.516 DEBUG:boto:Data: 2022-09-02T05:53:13.516 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.517 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.518 DEBUG:boto:Params: {} 2022-09-02T05:53:13.519 DEBUG:boto:Token: None 2022-09-02T05:53:13.519 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L6JgDTguSVUttM2Z1OXE2LV+faA= 2022-09-02T05:53:13.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L6JgDTguSVUttM2Z1OXE2LV+faA=', 'Content-Length': '0'} 2022-09-02T05:53:13.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cacefee28c57a0a-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.524 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.524 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.525 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.525 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.526 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.526 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.527 DEBUG:boto:Data: 2022-09-02T05:53:13.527 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.528 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.529 DEBUG:boto:Params: {} 2022-09-02T05:53:13.529 DEBUG:boto:Token: None 2022-09-02T05:53:13.530 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L6JgDTguSVUttM2Z1OXE2LV+faA= 2022-09-02T05:53:13.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L6JgDTguSVUttM2Z1OXE2LV+faA=', 'Content-Length': '0'} 2022-09-02T05:53:13.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000428e8f78b0a16d32-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:13.535 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.535 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.536 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.536 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.537 DEBUG:boto:Method: GET 2022-09-02T05:53:13.537 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.538 DEBUG:boto:Data: 2022-09-02T05:53:13.538 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.539 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.540 DEBUG:boto:Params: {} 2022-09-02T05:53:13.541 DEBUG:boto:Token: None 2022-09-02T05:53:13.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY= 2022-09-02T05:53:13.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY=', 'Content-Length': '0'} 2022-09-02T05:53:13.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c79b9c32d3ec5b54-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.545 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.546 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.547 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.547 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.548 DEBUG:boto:Method: GET 2022-09-02T05:53:13.548 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.549 DEBUG:boto:Data: 2022-09-02T05:53:13.549 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.550 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.551 DEBUG:boto:Params: {} 2022-09-02T05:53:13.551 DEBUG:boto:Token: None 2022-09-02T05:53:13.552 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:13.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY= 2022-09-02T05:53:13.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KItBv4FfeIO++D++844xbDHCtpY=', 'Content-Length': '0'} 2022-09-02T05:53:13.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea82886c9591cb5-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.557 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.557 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.558 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.558 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.559 DEBUG:boto:Method: GET 2022-09-02T05:53:13.559 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.560 DEBUG:boto:Data: 2022-09-02T05:53:13.560 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.561 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.562 DEBUG:boto:Params: {} 2022-09-02T05:53:13.563 DEBUG:boto:Token: None 2022-09-02T05:53:13.563 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35a0b1c8bba249b-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.568 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.569 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.569 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.570 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.570 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.571 DEBUG:boto:Method: GET 2022-09-02T05:53:13.572 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.572 DEBUG:boto:Data: 2022-09-02T05:53:13.573 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.574 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.574 DEBUG:boto:Params: {} 2022-09-02T05:53:13.575 DEBUG:boto:Token: None 2022-09-02T05:53:13.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a791d5cc1091ea0a-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.581 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.582 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.583 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.583 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.584 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:13.584 DEBUG:boto:Data: 2022-09-02T05:53:13.585 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.586 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.586 DEBUG:boto:Params: {} 2022-09-02T05:53:13.587 DEBUG:boto:Token: None 2022-09-02T05:53:13.588 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23 2022-09-02T05:53:13.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3p4kGllDGrSUeC6hdLpcLxXqqNA= 2022-09-02T05:53:13.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3p4kGllDGrSUeC6hdLpcLxXqqNA=', 'Content-Length': '0'} 2022-09-02T05:53:13.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3916c77d2a76e8-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.592 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.592 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.593 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.593 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:13.594 DEBUG:boto:Data: 2022-09-02T05:53:13.594 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.595 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.596 DEBUG:boto:Params: {} 2022-09-02T05:53:13.596 DEBUG:boto:Token: None 2022-09-02T05:53:13.597 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23 2022-09-02T05:53:13.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3p4kGllDGrSUeC6hdLpcLxXqqNA= 2022-09-02T05:53:13.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3p4kGllDGrSUeC6hdLpcLxXqqNA=', 'Content-Length': '0'} 2022-09-02T05:53:13.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000026275391275a70-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:13.603 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.605 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.605 DEBUG:boto:Method: GET 2022-09-02T05:53:13.606 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:13.606 DEBUG:boto:Data: 2022-09-02T05:53:13.607 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.608 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.609 DEBUG:boto:Params: {} 2022-09-02T05:53:13.610 DEBUG:boto:Token: None 2022-09-02T05:53:13.611 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23 2022-09-02T05:53:13.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EFIygdd8CUay2RMJzHJrPkCs/DQ= 2022-09-02T05:53:13.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EFIygdd8CUay2RMJzHJrPkCs/DQ=', 'Content-Length': '0'} 2022-09-02T05:53:13.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0d00b0ad1d3606-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.658 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:13.659 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:13.660 DEBUG:boto:Method: GET 2022-09-02T05:53:13.661 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:13.661 DEBUG:boto:Data: 2022-09-02T05:53:13.662 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.663 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.664 DEBUG:boto:Params: {} 2022-09-02T05:53:13.665 DEBUG:boto:Token: None 2022-09-02T05:53:13.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj23 2022-09-02T05:53:13.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EFIygdd8CUay2RMJzHJrPkCs/DQ= 2022-09-02T05:53:13.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EFIygdd8CUay2RMJzHJrPkCs/DQ=', 'Content-Length': '0'} 2022-09-02T05:53:13.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ccbbc78546ae69-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.671 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.672 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.672 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.673 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.673 DEBUG:boto:Method: GET 2022-09-02T05:53:13.674 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.674 DEBUG:boto:Data: 2022-09-02T05:53:13.675 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.677 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.678 DEBUG:boto:Params: {} 2022-09-02T05:53:13.679 DEBUG:boto:Token: None 2022-09-02T05:53:13.680 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d39b158f86e2e5a-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.686 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.687 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.688 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.688 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.689 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.689 DEBUG:boto:Method: GET 2022-09-02T05:53:13.690 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:13.690 DEBUG:boto:Data: 2022-09-02T05:53:13.691 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.692 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.692 DEBUG:boto:Params: {} 2022-09-02T05:53:13.693 DEBUG:boto:Token: None 2022-09-02T05:53:13.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58ebf8ce74c01b6-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.700 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:13.702 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.702 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.703 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.703 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.704 DEBUG:boto:Method: GET 2022-09-02T05:53:13.704 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.705 DEBUG:boto:Data: 2022-09-02T05:53:13.705 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.706 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.707 DEBUG:boto:Params: {} 2022-09-02T05:53:13.708 DEBUG:boto:Token: None 2022-09-02T05:53:13.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8= 2022-09-02T05:53:13.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8=', 'Content-Length': '0'} 2022-09-02T05:53:13.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f188ab44c30759-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.712 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.713 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.713 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.714 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.714 DEBUG:boto:Method: GET 2022-09-02T05:53:13.715 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.715 DEBUG:boto:Data: 2022-09-02T05:53:13.716 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.717 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.717 DEBUG:boto:Params: {} 2022-09-02T05:53:13.718 DEBUG:boto:Token: None 2022-09-02T05:53:13.719 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8= 2022-09-02T05:53:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8=', 'Content-Length': '0'} 2022-09-02T05:53:13.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008134cb325d3fa286-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.723 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.724 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.724 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.725 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.725 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.726 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.726 DEBUG:boto:Data: 2022-09-02T05:53:13.727 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.728 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.728 DEBUG:boto:Params: {} 2022-09-02T05:53:13.729 DEBUG:boto:Token: None 2022-09-02T05:53:13.730 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v/7eGUN381MlDrrl5TtSrooZOVM= 2022-09-02T05:53:13.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v/7eGUN381MlDrrl5TtSrooZOVM=', 'Content-Length': '0'} 2022-09-02T05:53:13.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003773e888577fcd22-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.734 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.734 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.735 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.735 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.736 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.736 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.737 DEBUG:boto:Data: 2022-09-02T05:53:13.737 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.739 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.739 DEBUG:boto:Params: {} 2022-09-02T05:53:13.740 DEBUG:boto:Token: None 2022-09-02T05:53:13.741 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v/7eGUN381MlDrrl5TtSrooZOVM= 2022-09-02T05:53:13.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v/7eGUN381MlDrrl5TtSrooZOVM=', 'Content-Length': '0'} 2022-09-02T05:53:13.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eacf80128e883fd6-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:13.746 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.747 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.747 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.748 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.748 DEBUG:boto:Method: GET 2022-09-02T05:53:13.749 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.750 DEBUG:boto:Data: 2022-09-02T05:53:13.750 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.752 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.752 DEBUG:boto:Params: {} 2022-09-02T05:53:13.753 DEBUG:boto:Token: None 2022-09-02T05:53:13.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8= 2022-09-02T05:53:13.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8=', 'Content-Length': '0'} 2022-09-02T05:53:13.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a0c26b01f50916c-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.759 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.759 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.760 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.761 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.761 DEBUG:boto:Method: GET 2022-09-02T05:53:13.762 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.762 DEBUG:boto:Data: 2022-09-02T05:53:13.763 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.764 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.764 DEBUG:boto:Params: {} 2022-09-02T05:53:13.765 DEBUG:boto:Token: None 2022-09-02T05:53:13.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:13.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8= 2022-09-02T05:53:13.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UDZ2fIVrMjPC+x/wPF+c1kKnkJ8=', 'Content-Length': '0'} 2022-09-02T05:53:13.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb056d171690c68-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.810 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.811 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.812 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.812 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.813 DEBUG:boto:Method: GET 2022-09-02T05:53:13.813 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.814 DEBUG:boto:Data: 2022-09-02T05:53:13.814 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.815 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.816 DEBUG:boto:Params: {} 2022-09-02T05:53:13.817 DEBUG:boto:Token: None 2022-09-02T05:53:13.817 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e477cc6f5859fe2-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.831 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.832 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.833 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.833 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.834 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.834 DEBUG:boto:Method: GET 2022-09-02T05:53:13.835 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.836 DEBUG:boto:Data: 2022-09-02T05:53:13.837 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.838 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.839 DEBUG:boto:Params: {} 2022-09-02T05:53:13.840 DEBUG:boto:Token: None 2022-09-02T05:53:13.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250b6e280eacf8db-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.846 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.847 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.848 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.848 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.849 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:13.849 DEBUG:boto:Data: 2022-09-02T05:53:13.850 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.851 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.851 DEBUG:boto:Params: {} 2022-09-02T05:53:13.852 DEBUG:boto:Token: None 2022-09-02T05:53:13.852 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33 2022-09-02T05:53:13.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cJtFq/5LLTROqNr4k9ZzD7GXEjg= 2022-09-02T05:53:13.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cJtFq/5LLTROqNr4k9ZzD7GXEjg=', 'Content-Length': '0'} 2022-09-02T05:53:13.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdbc8c57d4556fd7-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.856 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.857 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.858 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.858 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:13.859 DEBUG:boto:Data: 2022-09-02T05:53:13.859 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.860 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.861 DEBUG:boto:Params: {} 2022-09-02T05:53:13.861 DEBUG:boto:Token: None 2022-09-02T05:53:13.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33 2022-09-02T05:53:13.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cJtFq/5LLTROqNr4k9ZzD7GXEjg= 2022-09-02T05:53:13.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cJtFq/5LLTROqNr4k9ZzD7GXEjg=', 'Content-Length': '0'} 2022-09-02T05:53:13.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7ab128521f074ad-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:13.866 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.867 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.867 DEBUG:boto:Method: GET 2022-09-02T05:53:13.868 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:13.868 DEBUG:boto:Data: 2022-09-02T05:53:13.869 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.870 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.871 DEBUG:boto:Params: {} 2022-09-02T05:53:13.871 DEBUG:boto:Token: None 2022-09-02T05:53:13.872 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33 2022-09-02T05:53:13.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A4w/hVXaMsiUe0SyN834L7pqDDc= 2022-09-02T05:53:13.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A4w/hVXaMsiUe0SyN834L7pqDDc=', 'Content-Length': '0'} 2022-09-02T05:53:13.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098581d84a095ff41-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.876 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:13.877 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:13.877 DEBUG:boto:Method: GET 2022-09-02T05:53:13.878 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:13.878 DEBUG:boto:Data: 2022-09-02T05:53:13.879 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.880 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.880 DEBUG:boto:Params: {} 2022-09-02T05:53:13.881 DEBUG:boto:Token: None 2022-09-02T05:53:13.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj33 2022-09-02T05:53:13.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A4w/hVXaMsiUe0SyN834L7pqDDc= 2022-09-02T05:53:13.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A4w/hVXaMsiUe0SyN834L7pqDDc=', 'Content-Length': '0'} 2022-09-02T05:53:13.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a4a4e759fa16cba-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.886 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.887 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.888 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.888 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.889 DEBUG:boto:Method: GET 2022-09-02T05:53:13.889 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.890 DEBUG:boto:Data: 2022-09-02T05:53:13.890 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.891 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.892 DEBUG:boto:Params: {} 2022-09-02T05:53:13.893 DEBUG:boto:Token: None 2022-09-02T05:53:13.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d08fbe1357d4528e-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.898 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.899 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:13.899 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:13.900 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.901 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.901 DEBUG:boto:Method: GET 2022-09-02T05:53:13.902 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:13.902 DEBUG:boto:Data: 2022-09-02T05:53:13.903 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.904 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.904 DEBUG:boto:Params: {} 2022-09-02T05:53:13.905 DEBUG:boto:Token: None 2022-09-02T05:53:13.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/ 2022-09-02T05:53:13.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc= 2022-09-02T05:53:13.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8KcN9fkikYgwP4QJ7Ps7a1Y6Xrc=', 'Content-Length': '0'} 2022-09-02T05:53:13.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd6f9b9d5ef1222-0063119a49-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.911 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:13.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:13.912 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:13.913 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:13.913 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.914 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.915 DEBUG:boto:Method: GET 2022-09-02T05:53:13.915 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.916 DEBUG:boto:Data: 2022-09-02T05:53:13.917 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.918 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.918 DEBUG:boto:Params: {} 2022-09-02T05:53:13.919 DEBUG:boto:Token: None 2022-09-02T05:53:13.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UKJ0qoKgOVYWcDLbVy3q8olHCog= 2022-09-02T05:53:13.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UKJ0qoKgOVYWcDLbVy3q8olHCog=', 'Content-Length': '0'} 2022-09-02T05:53:13.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d127457b8dfbd71-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.925 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:13.925 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:13.926 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.926 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.927 DEBUG:boto:Method: GET 2022-09-02T05:53:13.927 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.928 DEBUG:boto:Data: 2022-09-02T05:53:13.928 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.930 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.930 DEBUG:boto:Params: {} 2022-09-02T05:53:13.931 DEBUG:boto:Token: None 2022-09-02T05:53:13.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UKJ0qoKgOVYWcDLbVy3q8olHCog= 2022-09-02T05:53:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UKJ0qoKgOVYWcDLbVy3q8olHCog=', 'Content-Length': '0'} 2022-09-02T05:53:13.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003019cdff1bdfbd58-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.937 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:13.938 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:13.938 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.939 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.939 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.940 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.941 DEBUG:boto:Data: 2022-09-02T05:53:13.941 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.942 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.943 DEBUG:boto:Params: {} 2022-09-02T05:53:13.943 DEBUG:boto:Token: None 2022-09-02T05:53:13.944 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DRpVYpbZvejX7U6m+I5+DJ+ERMM= 2022-09-02T05:53:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DRpVYpbZvejX7U6m+I5+DJ+ERMM=', 'Content-Length': '0'} 2022-09-02T05:53:13.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411ccda78f91724d-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.948 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:13.948 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:13.949 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.949 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.950 DEBUG:boto:Method: HEAD 2022-09-02T05:53:13.950 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.951 DEBUG:boto:Data: 2022-09-02T05:53:13.951 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:13.952 DEBUG:boto:Port: 8000 2022-09-02T05:53:13.953 DEBUG:boto:Params: {} 2022-09-02T05:53:13.953 DEBUG:boto:Token: None 2022-09-02T05:53:13.954 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DRpVYpbZvejX7U6m+I5+DJ+ERMM= 2022-09-02T05:53:13.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DRpVYpbZvejX7U6m+I5+DJ+ERMM=', 'Content-Length': '0'} 2022-09-02T05:53:13.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699d8d5226dd8447-0063119a49-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:13.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:13.959 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:13.960 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:13.960 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.961 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.961 DEBUG:boto:Method: GET 2022-09-02T05:53:13.962 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.962 DEBUG:boto:Data: 2022-09-02T05:53:13.963 DEBUG:boto:Headers: {} 2022-09-02T05:53:13.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:13.964 DEBUG:boto:Port: 8001 2022-09-02T05:53:13.964 DEBUG:boto:Params: {} 2022-09-02T05:53:13.965 DEBUG:boto:Token: None 2022-09-02T05:53:13.966 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:13 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:13.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UKJ0qoKgOVYWcDLbVy3q8olHCog= 2022-09-02T05:53:13.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UKJ0qoKgOVYWcDLbVy3q8olHCog=', 'Content-Length': '0'} 2022-09-02T05:53:13.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad3f6bfc0899e9bd-0063119a49-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.014 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:14.015 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:14.015 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:14.016 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:14.016 DEBUG:boto:Method: GET 2022-09-02T05:53:14.017 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:14.018 DEBUG:boto:Data: 2022-09-02T05:53:14.018 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.019 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.020 DEBUG:boto:Params: {} 2022-09-02T05:53:14.020 DEBUG:boto:Token: None 2022-09-02T05:53:14.021 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:14.022 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xSO5iBhMOEKxC8oiWC78BsBw7II= 2022-09-02T05:53:14.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xSO5iBhMOEKxC8oiWC78BsBw7II=', 'Content-Length': '0'} 2022-09-02T05:53:14.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab732f1b8074f7a-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.026 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.026 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.029 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.029 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.030 DEBUG:boto:Method: GET 2022-09-02T05:53:14.031 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.032 DEBUG:boto:Data: 2022-09-02T05:53:14.033 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.034 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.035 DEBUG:boto:Params: {} 2022-09-02T05:53:14.036 DEBUG:boto:Token: None 2022-09-02T05:53:14.036 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d9087cd416805f-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.041 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.043 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.043 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.044 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.044 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.045 DEBUG:boto:Method: GET 2022-09-02T05:53:14.045 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.046 DEBUG:boto:Data: 2022-09-02T05:53:14.046 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.047 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.048 DEBUG:boto:Params: {} 2022-09-02T05:53:14.048 DEBUG:boto:Token: None 2022-09-02T05:53:14.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcefb1c605452858-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.055 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.056 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:14.056 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:14.057 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.057 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:14.058 DEBUG:boto:Data: 2022-09-02T05:53:14.058 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.059 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.060 DEBUG:boto:Params: {} 2022-09-02T05:53:14.060 DEBUG:boto:Token: None 2022-09-02T05:53:14.061 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj34 2022-09-02T05:53:14.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E3+Okm2f0/ghJHVfShp7FeutZ6Y= 2022-09-02T05:53:14.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E3+Okm2f0/ghJHVfShp7FeutZ6Y=', 'Content-Length': '0'} 2022-09-02T05:53:14.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ddad59dfc9e7a4-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.066 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:14.067 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:14.068 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.069 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:14.070 DEBUG:boto:Data: 2022-09-02T05:53:14.071 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.073 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.073 DEBUG:boto:Params: {} 2022-09-02T05:53:14.074 DEBUG:boto:Token: None 2022-09-02T05:53:14.075 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj34 2022-09-02T05:53:14.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E3+Okm2f0/ghJHVfShp7FeutZ6Y= 2022-09-02T05:53:14.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E3+Okm2f0/ghJHVfShp7FeutZ6Y=', 'Content-Length': '0'} 2022-09-02T05:53:14.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c58f29cf604342-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:14.079 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:14.080 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:14.080 DEBUG:boto:Method: GET 2022-09-02T05:53:14.081 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:14.081 DEBUG:boto:Data: 2022-09-02T05:53:14.082 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.083 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.083 DEBUG:boto:Params: {} 2022-09-02T05:53:14.084 DEBUG:boto:Token: None 2022-09-02T05:53:14.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj34 2022-09-02T05:53:14.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zqd6e3IaFjW+vEHqn937N08wA+U= 2022-09-02T05:53:14.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zqd6e3IaFjW+vEHqn937N08wA+U=', 'Content-Length': '0'} 2022-09-02T05:53:14.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44406282ad3a9c7-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.089 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:14.089 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:14.090 DEBUG:boto:Method: GET 2022-09-02T05:53:14.090 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:14.091 DEBUG:boto:Data: 2022-09-02T05:53:14.092 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.093 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.093 DEBUG:boto:Params: {} 2022-09-02T05:53:14.094 DEBUG:boto:Token: None 2022-09-02T05:53:14.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj34 2022-09-02T05:53:14.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zqd6e3IaFjW+vEHqn937N08wA+U= 2022-09-02T05:53:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zqd6e3IaFjW+vEHqn937N08wA+U=', 'Content-Length': '0'} 2022-09-02T05:53:14.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40418bfa1dbebb1-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.142 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.143 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.143 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.144 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.144 DEBUG:boto:Method: GET 2022-09-02T05:53:14.145 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.146 DEBUG:boto:Data: 2022-09-02T05:53:14.146 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.147 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.148 DEBUG:boto:Params: {} 2022-09-02T05:53:14.148 DEBUG:boto:Token: None 2022-09-02T05:53:14.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a51016517214de8f-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.154 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.155 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.156 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.157 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.158 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.159 DEBUG:boto:Method: GET 2022-09-02T05:53:14.160 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:14.161 DEBUG:boto:Data: 2022-09-02T05:53:14.162 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.164 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.165 DEBUG:boto:Params: {} 2022-09-02T05:53:14.166 DEBUG:boto:Token: None 2022-09-02T05:53:14.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f7414b27e4fc35-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.174 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:14.177 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.178 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.179 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.180 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.180 DEBUG:boto:Method: GET 2022-09-02T05:53:14.181 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.182 DEBUG:boto:Data: 2022-09-02T05:53:14.182 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.183 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.184 DEBUG:boto:Params: {} 2022-09-02T05:53:14.184 DEBUG:boto:Token: None 2022-09-02T05:53:14.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs= 2022-09-02T05:53:14.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs=', 'Content-Length': '0'} 2022-09-02T05:53:14.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f72fb5bedfb1f5-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.191 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.192 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.193 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.194 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.194 DEBUG:boto:Method: GET 2022-09-02T05:53:14.195 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.196 DEBUG:boto:Data: 2022-09-02T05:53:14.197 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.199 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.200 DEBUG:boto:Params: {} 2022-09-02T05:53:14.201 DEBUG:boto:Token: None 2022-09-02T05:53:14.202 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs= 2022-09-02T05:53:14.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs=', 'Content-Length': '0'} 2022-09-02T05:53:14.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003970c217cdeb2127-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.209 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.210 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.211 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.212 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.213 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.214 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.215 DEBUG:boto:Data: 2022-09-02T05:53:14.216 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.218 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.219 DEBUG:boto:Params: {} 2022-09-02T05:53:14.220 DEBUG:boto:Token: None 2022-09-02T05:53:14.221 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yUgWpWA8zNcyulJt+DgiCTZ+9gA= 2022-09-02T05:53:14.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yUgWpWA8zNcyulJt+DgiCTZ+9gA=', 'Content-Length': '0'} 2022-09-02T05:53:14.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8d6717e8a47531-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.227 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.228 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.229 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.230 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.231 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.232 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.233 DEBUG:boto:Data: 2022-09-02T05:53:14.234 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.236 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.236 DEBUG:boto:Params: {} 2022-09-02T05:53:14.237 DEBUG:boto:Token: None 2022-09-02T05:53:14.238 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yUgWpWA8zNcyulJt+DgiCTZ+9gA= 2022-09-02T05:53:14.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yUgWpWA8zNcyulJt+DgiCTZ+9gA=', 'Content-Length': '0'} 2022-09-02T05:53:14.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219f5a2ec33682b9-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:14.244 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.245 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.246 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.247 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.248 DEBUG:boto:Method: GET 2022-09-02T05:53:14.248 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.249 DEBUG:boto:Data: 2022-09-02T05:53:14.249 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.250 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.251 DEBUG:boto:Params: {} 2022-09-02T05:53:14.251 DEBUG:boto:Token: None 2022-09-02T05:53:14.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs= 2022-09-02T05:53:14.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs=', 'Content-Length': '0'} 2022-09-02T05:53:14.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b933aabef951e35e-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.257 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.258 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.258 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.259 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.260 DEBUG:boto:Method: GET 2022-09-02T05:53:14.260 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.261 DEBUG:boto:Data: 2022-09-02T05:53:14.262 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.263 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.264 DEBUG:boto:Params: {} 2022-09-02T05:53:14.264 DEBUG:boto:Token: None 2022-09-02T05:53:14.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:14.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs= 2022-09-02T05:53:14.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JrNY/vuMMp+VLcM1Av/ycZ0x/qs=', 'Content-Length': '0'} 2022-09-02T05:53:14.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eedd4aa2492c819f-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.271 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.271 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.272 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.273 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.274 DEBUG:boto:Method: GET 2022-09-02T05:53:14.274 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.275 DEBUG:boto:Data: 2022-09-02T05:53:14.276 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.277 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.277 DEBUG:boto:Params: {} 2022-09-02T05:53:14.278 DEBUG:boto:Token: None 2022-09-02T05:53:14.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce915260132c541f-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.283 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.284 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.285 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.285 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.286 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.286 DEBUG:boto:Method: GET 2022-09-02T05:53:14.287 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.288 DEBUG:boto:Data: 2022-09-02T05:53:14.288 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.289 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.290 DEBUG:boto:Params: {} 2022-09-02T05:53:14.290 DEBUG:boto:Token: None 2022-09-02T05:53:14.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d2e0fb9e15e0f87-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.297 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.297 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.298 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.299 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.299 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:14.300 DEBUG:boto:Data: 2022-09-02T05:53:14.300 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.301 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.302 DEBUG:boto:Params: {} 2022-09-02T05:53:14.302 DEBUG:boto:Token: None 2022-09-02T05:53:14.303 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38 2022-09-02T05:53:14.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rvPG1zLISh0hbEojsTZFUHhHT9M= 2022-09-02T05:53:14.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rvPG1zLISh0hbEojsTZFUHhHT9M=', 'Content-Length': '0'} 2022-09-02T05:53:14.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969d96d52dbf4040-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.307 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.308 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.309 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.309 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:14.310 DEBUG:boto:Data: 2022-09-02T05:53:14.310 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.311 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.312 DEBUG:boto:Params: {} 2022-09-02T05:53:14.312 DEBUG:boto:Token: None 2022-09-02T05:53:14.313 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38 2022-09-02T05:53:14.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rvPG1zLISh0hbEojsTZFUHhHT9M= 2022-09-02T05:53:14.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rvPG1zLISh0hbEojsTZFUHhHT9M=', 'Content-Length': '0'} 2022-09-02T05:53:14.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a1fe2903e3d45c-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:14.318 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.319 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.320 DEBUG:boto:Method: GET 2022-09-02T05:53:14.320 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:14.321 DEBUG:boto:Data: 2022-09-02T05:53:14.322 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.323 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.324 DEBUG:boto:Params: {} 2022-09-02T05:53:14.325 DEBUG:boto:Token: None 2022-09-02T05:53:14.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38 2022-09-02T05:53:14.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5cw8Ic9vzVVmYtufIwLOpyFLrhk= 2022-09-02T05:53:14.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5cw8Ic9vzVVmYtufIwLOpyFLrhk=', 'Content-Length': '0'} 2022-09-02T05:53:14.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30f1a4a1cd4cff5-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.370 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:14.372 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:14.373 DEBUG:boto:Method: GET 2022-09-02T05:53:14.375 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:14.376 DEBUG:boto:Data: 2022-09-02T05:53:14.377 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.378 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.379 DEBUG:boto:Params: {} 2022-09-02T05:53:14.380 DEBUG:boto:Token: None 2022-09-02T05:53:14.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj38 2022-09-02T05:53:14.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5cw8Ic9vzVVmYtufIwLOpyFLrhk= 2022-09-02T05:53:14.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5cw8Ic9vzVVmYtufIwLOpyFLrhk=', 'Content-Length': '0'} 2022-09-02T05:53:14.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f984344cc708579-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.387 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.387 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.388 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.389 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.389 DEBUG:boto:Method: GET 2022-09-02T05:53:14.390 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.391 DEBUG:boto:Data: 2022-09-02T05:53:14.392 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.393 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.394 DEBUG:boto:Params: {} 2022-09-02T05:53:14.395 DEBUG:boto:Token: None 2022-09-02T05:53:14.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003818b24bda612ace-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.401 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.402 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.403 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.404 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.405 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.406 DEBUG:boto:Method: GET 2022-09-02T05:53:14.407 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:14.408 DEBUG:boto:Data: 2022-09-02T05:53:14.409 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.412 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.413 DEBUG:boto:Params: {} 2022-09-02T05:53:14.414 DEBUG:boto:Token: None 2022-09-02T05:53:14.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d3b45f44807126-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.421 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:14.422 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.423 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.424 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.424 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.425 DEBUG:boto:Method: GET 2022-09-02T05:53:14.425 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.426 DEBUG:boto:Data: 2022-09-02T05:53:14.427 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.428 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.429 DEBUG:boto:Params: {} 2022-09-02T05:53:14.430 DEBUG:boto:Token: None 2022-09-02T05:53:14.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw= 2022-09-02T05:53:14.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw=', 'Content-Length': '0'} 2022-09-02T05:53:14.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd8223619fee93d-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.435 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.436 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.436 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.437 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.437 DEBUG:boto:Method: GET 2022-09-02T05:53:14.438 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.438 DEBUG:boto:Data: 2022-09-02T05:53:14.439 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.440 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.441 DEBUG:boto:Params: {} 2022-09-02T05:53:14.441 DEBUG:boto:Token: None 2022-09-02T05:53:14.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw= 2022-09-02T05:53:14.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw=', 'Content-Length': '0'} 2022-09-02T05:53:14.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9c3a3dccf69bd5-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.446 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.447 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.447 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.448 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.448 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.449 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.449 DEBUG:boto:Data: 2022-09-02T05:53:14.450 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.451 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.451 DEBUG:boto:Params: {} 2022-09-02T05:53:14.452 DEBUG:boto:Token: None 2022-09-02T05:53:14.452 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jwmSkhRaVzYPjdDjSu1G6k/dgTQ= 2022-09-02T05:53:14.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jwmSkhRaVzYPjdDjSu1G6k/dgTQ=', 'Content-Length': '0'} 2022-09-02T05:53:14.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160a40629de95c3a-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.457 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.457 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.458 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.458 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.459 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.459 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.460 DEBUG:boto:Data: 2022-09-02T05:53:14.460 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.461 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.462 DEBUG:boto:Params: {} 2022-09-02T05:53:14.462 DEBUG:boto:Token: None 2022-09-02T05:53:14.463 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jwmSkhRaVzYPjdDjSu1G6k/dgTQ= 2022-09-02T05:53:14.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jwmSkhRaVzYPjdDjSu1G6k/dgTQ=', 'Content-Length': '0'} 2022-09-02T05:53:14.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab780e020008ca3-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:14.468 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.468 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.469 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.469 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.470 DEBUG:boto:Method: GET 2022-09-02T05:53:14.470 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.471 DEBUG:boto:Data: 2022-09-02T05:53:14.471 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.472 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.473 DEBUG:boto:Params: {} 2022-09-02T05:53:14.473 DEBUG:boto:Token: None 2022-09-02T05:53:14.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw= 2022-09-02T05:53:14.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw=', 'Content-Length': '0'} 2022-09-02T05:53:14.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce298f785c71864a-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.478 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.479 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.479 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.480 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.480 DEBUG:boto:Method: GET 2022-09-02T05:53:14.481 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.481 DEBUG:boto:Data: 2022-09-02T05:53:14.482 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.483 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.484 DEBUG:boto:Params: {} 2022-09-02T05:53:14.484 DEBUG:boto:Token: None 2022-09-02T05:53:14.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:14.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw= 2022-09-02T05:53:14.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7cKdFxI77fN0atJlrxDWMtWuBkw=', 'Content-Length': '0'} 2022-09-02T05:53:14.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648ffb823f53e8f8-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.490 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.490 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.491 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.491 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.492 DEBUG:boto:Method: GET 2022-09-02T05:53:14.492 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.493 DEBUG:boto:Data: 2022-09-02T05:53:14.493 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.495 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.495 DEBUG:boto:Params: {} 2022-09-02T05:53:14.496 DEBUG:boto:Token: None 2022-09-02T05:53:14.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4afbfe2aa2e479-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.501 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.502 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.502 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.503 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.503 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.504 DEBUG:boto:Method: GET 2022-09-02T05:53:14.504 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.505 DEBUG:boto:Data: 2022-09-02T05:53:14.506 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.507 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.507 DEBUG:boto:Params: {} 2022-09-02T05:53:14.508 DEBUG:boto:Token: None 2022-09-02T05:53:14.508 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eb89713b6abfd7a-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.514 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.515 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.515 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.516 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.516 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:14.517 DEBUG:boto:Data: 2022-09-02T05:53:14.517 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.518 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.519 DEBUG:boto:Params: {} 2022-09-02T05:53:14.519 DEBUG:boto:Token: None 2022-09-02T05:53:14.520 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44 2022-09-02T05:53:14.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkNJuMMbdYV95iINJPfeAFqMPoo= 2022-09-02T05:53:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkNJuMMbdYV95iINJPfeAFqMPoo=', 'Content-Length': '0'} 2022-09-02T05:53:14.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a089ac90dc6067-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.524 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.525 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.525 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.526 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:14.526 DEBUG:boto:Data: 2022-09-02T05:53:14.527 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.528 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.529 DEBUG:boto:Params: {} 2022-09-02T05:53:14.529 DEBUG:boto:Token: None 2022-09-02T05:53:14.530 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44 2022-09-02T05:53:14.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hkNJuMMbdYV95iINJPfeAFqMPoo= 2022-09-02T05:53:14.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hkNJuMMbdYV95iINJPfeAFqMPoo=', 'Content-Length': '0'} 2022-09-02T05:53:14.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bad0e170e3c35af-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:14.535 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.535 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.536 DEBUG:boto:Method: GET 2022-09-02T05:53:14.536 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:14.537 DEBUG:boto:Data: 2022-09-02T05:53:14.537 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.539 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.539 DEBUG:boto:Params: {} 2022-09-02T05:53:14.540 DEBUG:boto:Token: None 2022-09-02T05:53:14.540 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44 2022-09-02T05:53:14.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l5xzlLG/TpTdA/JMs8gxA2JJBzs= 2022-09-02T05:53:14.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l5xzlLG/TpTdA/JMs8gxA2JJBzs=', 'Content-Length': '0'} 2022-09-02T05:53:14.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9e754631d16845-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.545 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:14.545 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:14.546 DEBUG:boto:Method: GET 2022-09-02T05:53:14.546 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:14.547 DEBUG:boto:Data: 2022-09-02T05:53:14.548 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.549 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.549 DEBUG:boto:Params: {} 2022-09-02T05:53:14.550 DEBUG:boto:Token: None 2022-09-02T05:53:14.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj44 2022-09-02T05:53:14.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l5xzlLG/TpTdA/JMs8gxA2JJBzs= 2022-09-02T05:53:14.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l5xzlLG/TpTdA/JMs8gxA2JJBzs=', 'Content-Length': '0'} 2022-09-02T05:53:14.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f0998b9420891e-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.555 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.556 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.557 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.558 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.559 DEBUG:boto:Method: GET 2022-09-02T05:53:14.559 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.560 DEBUG:boto:Data: 2022-09-02T05:53:14.561 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.562 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.563 DEBUG:boto:Params: {} 2022-09-02T05:53:14.564 DEBUG:boto:Token: None 2022-09-02T05:53:14.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d43bd2817606881e-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.569 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.570 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.571 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.571 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.572 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.572 DEBUG:boto:Method: GET 2022-09-02T05:53:14.573 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:14.573 DEBUG:boto:Data: 2022-09-02T05:53:14.574 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.575 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.576 DEBUG:boto:Params: {} 2022-09-02T05:53:14.577 DEBUG:boto:Token: None 2022-09-02T05:53:14.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc825d01d37c0aa-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.630 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:14.632 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.633 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.634 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.635 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.636 DEBUG:boto:Method: GET 2022-09-02T05:53:14.637 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.638 DEBUG:boto:Data: 2022-09-02T05:53:14.639 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.641 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.642 DEBUG:boto:Params: {} 2022-09-02T05:53:14.643 DEBUG:boto:Token: None 2022-09-02T05:53:14.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs= 2022-09-02T05:53:14.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs=', 'Content-Length': '0'} 2022-09-02T05:53:14.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c79f5f924212c13-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.650 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.651 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.651 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.652 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.652 DEBUG:boto:Method: GET 2022-09-02T05:53:14.653 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.654 DEBUG:boto:Data: 2022-09-02T05:53:14.654 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.655 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.656 DEBUG:boto:Params: {} 2022-09-02T05:53:14.656 DEBUG:boto:Token: None 2022-09-02T05:53:14.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs= 2022-09-02T05:53:14.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs=', 'Content-Length': '0'} 2022-09-02T05:53:14.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d516cb3642a0894-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.662 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.663 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.663 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.664 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.664 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.665 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.665 DEBUG:boto:Data: 2022-09-02T05:53:14.666 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.667 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.667 DEBUG:boto:Params: {} 2022-09-02T05:53:14.668 DEBUG:boto:Token: None 2022-09-02T05:53:14.668 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:67Ppu4xek4DHBWumDIQfuWwTYg0= 2022-09-02T05:53:14.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:67Ppu4xek4DHBWumDIQfuWwTYg0=', 'Content-Length': '0'} 2022-09-02T05:53:14.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab61a5f08226c600-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.672 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.673 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.673 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.674 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.674 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.675 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.676 DEBUG:boto:Data: 2022-09-02T05:53:14.676 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.677 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.677 DEBUG:boto:Params: {} 2022-09-02T05:53:14.678 DEBUG:boto:Token: None 2022-09-02T05:53:14.679 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:67Ppu4xek4DHBWumDIQfuWwTYg0= 2022-09-02T05:53:14.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:67Ppu4xek4DHBWumDIQfuWwTYg0=', 'Content-Length': '0'} 2022-09-02T05:53:14.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d44ddfd1381f7fc-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:14.683 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.684 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.685 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.685 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.686 DEBUG:boto:Method: GET 2022-09-02T05:53:14.686 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.687 DEBUG:boto:Data: 2022-09-02T05:53:14.687 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.688 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.689 DEBUG:boto:Params: {} 2022-09-02T05:53:14.689 DEBUG:boto:Token: None 2022-09-02T05:53:14.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs= 2022-09-02T05:53:14.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs=', 'Content-Length': '0'} 2022-09-02T05:53:14.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38ce8b8c1d559f6-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.694 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.695 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.695 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.696 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.696 DEBUG:boto:Method: GET 2022-09-02T05:53:14.697 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.697 DEBUG:boto:Data: 2022-09-02T05:53:14.698 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.699 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.700 DEBUG:boto:Params: {} 2022-09-02T05:53:14.700 DEBUG:boto:Token: None 2022-09-02T05:53:14.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:14.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs= 2022-09-02T05:53:14.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ByMrgKbrrl76nBtoV3sI+XkBovs=', 'Content-Length': '0'} 2022-09-02T05:53:14.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225ddab07609ed6b-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.705 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.706 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.706 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.707 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.707 DEBUG:boto:Method: GET 2022-09-02T05:53:14.708 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.708 DEBUG:boto:Data: 2022-09-02T05:53:14.709 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.710 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.710 DEBUG:boto:Params: {} 2022-09-02T05:53:14.711 DEBUG:boto:Token: None 2022-09-02T05:53:14.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000864be7ec29a12383-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.716 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.717 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.718 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.718 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.719 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.719 DEBUG:boto:Method: GET 2022-09-02T05:53:14.720 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.720 DEBUG:boto:Data: 2022-09-02T05:53:14.721 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.722 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.722 DEBUG:boto:Params: {} 2022-09-02T05:53:14.723 DEBUG:boto:Token: None 2022-09-02T05:53:14.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecdf01691632e874-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.729 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.730 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.731 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.731 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.732 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:14.732 DEBUG:boto:Data: 2022-09-02T05:53:14.733 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.734 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.734 DEBUG:boto:Params: {} 2022-09-02T05:53:14.735 DEBUG:boto:Token: None 2022-09-02T05:53:14.735 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46 2022-09-02T05:53:14.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2kf6NdBf+NjbacRxiQWWfM/JtU= 2022-09-02T05:53:14.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2kf6NdBf+NjbacRxiQWWfM/JtU=', 'Content-Length': '0'} 2022-09-02T05:53:14.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0100242f5f25aa-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.740 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.740 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.741 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.741 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:14.742 DEBUG:boto:Data: 2022-09-02T05:53:14.742 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.743 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.744 DEBUG:boto:Params: {} 2022-09-02T05:53:14.744 DEBUG:boto:Token: None 2022-09-02T05:53:14.745 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46 2022-09-02T05:53:14.745 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2kf6NdBf+NjbacRxiQWWfM/JtU= 2022-09-02T05:53:14.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2kf6NdBf+NjbacRxiQWWfM/JtU=', 'Content-Length': '0'} 2022-09-02T05:53:14.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e840796620c95f4-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:14.750 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.750 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.751 DEBUG:boto:Method: GET 2022-09-02T05:53:14.751 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:14.752 DEBUG:boto:Data: 2022-09-02T05:53:14.753 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.754 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.754 DEBUG:boto:Params: {} 2022-09-02T05:53:14.755 DEBUG:boto:Token: None 2022-09-02T05:53:14.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46 2022-09-02T05:53:14.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o1ztOSokzgfW1tMHE0vO3A3klIQ= 2022-09-02T05:53:14.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o1ztOSokzgfW1tMHE0vO3A3klIQ=', 'Content-Length': '0'} 2022-09-02T05:53:14.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a49f4194149a66d4-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.806 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:14.807 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:14.808 DEBUG:boto:Method: GET 2022-09-02T05:53:14.808 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:14.809 DEBUG:boto:Data: 2022-09-02T05:53:14.809 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.811 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.811 DEBUG:boto:Params: {} 2022-09-02T05:53:14.812 DEBUG:boto:Token: None 2022-09-02T05:53:14.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj46 2022-09-02T05:53:14.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o1ztOSokzgfW1tMHE0vO3A3klIQ= 2022-09-02T05:53:14.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o1ztOSokzgfW1tMHE0vO3A3klIQ=', 'Content-Length': '0'} 2022-09-02T05:53:14.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb15507c7c5bc693-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.828 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.829 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.829 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.830 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.830 DEBUG:boto:Method: GET 2022-09-02T05:53:14.831 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.831 DEBUG:boto:Data: 2022-09-02T05:53:14.832 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.833 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.833 DEBUG:boto:Params: {} 2022-09-02T05:53:14.834 DEBUG:boto:Token: None 2022-09-02T05:53:14.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000dd94d62336ca65-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.840 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.841 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.841 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.842 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.843 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.843 DEBUG:boto:Method: GET 2022-09-02T05:53:14.844 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:14.844 DEBUG:boto:Data: 2022-09-02T05:53:14.845 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.846 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.847 DEBUG:boto:Params: {} 2022-09-02T05:53:14.847 DEBUG:boto:Token: None 2022-09-02T05:53:14.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d455ed2f595ceda-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.854 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:14.856 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:14.857 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:14.858 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.859 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.860 DEBUG:boto:Method: GET 2022-09-02T05:53:14.861 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.861 DEBUG:boto:Data: 2022-09-02T05:53:14.862 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.863 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.864 DEBUG:boto:Params: {} 2022-09-02T05:53:14.864 DEBUG:boto:Token: None 2022-09-02T05:53:14.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0= 2022-09-02T05:53:14.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0=', 'Content-Length': '0'} 2022-09-02T05:53:14.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91c09a15b0b383f-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.870 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:14.870 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:14.871 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.872 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.872 DEBUG:boto:Method: GET 2022-09-02T05:53:14.873 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.874 DEBUG:boto:Data: 2022-09-02T05:53:14.876 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.878 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.879 DEBUG:boto:Params: {} 2022-09-02T05:53:14.881 DEBUG:boto:Token: None 2022-09-02T05:53:14.882 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0= 2022-09-02T05:53:14.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0=', 'Content-Length': '0'} 2022-09-02T05:53:14.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa00777981498831-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.888 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:14.889 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:14.889 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.890 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.890 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.891 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.892 DEBUG:boto:Data: 2022-09-02T05:53:14.892 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.894 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.894 DEBUG:boto:Params: {} 2022-09-02T05:53:14.895 DEBUG:boto:Token: None 2022-09-02T05:53:14.895 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1j6KDk3w//pSjSIOnCsGlguWnJM= 2022-09-02T05:53:14.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1j6KDk3w//pSjSIOnCsGlguWnJM=', 'Content-Length': '0'} 2022-09-02T05:53:14.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001145f8184c2e93f3-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.900 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:14.900 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:14.901 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.901 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.902 DEBUG:boto:Method: HEAD 2022-09-02T05:53:14.903 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.903 DEBUG:boto:Data: 2022-09-02T05:53:14.904 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.905 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.905 DEBUG:boto:Params: {} 2022-09-02T05:53:14.906 DEBUG:boto:Token: None 2022-09-02T05:53:14.907 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1j6KDk3w//pSjSIOnCsGlguWnJM= 2022-09-02T05:53:14.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1j6KDk3w//pSjSIOnCsGlguWnJM=', 'Content-Length': '0'} 2022-09-02T05:53:14.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd6b994f9091b4bc-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:14.911 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:14.912 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:14.912 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.913 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.914 DEBUG:boto:Method: GET 2022-09-02T05:53:14.914 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.915 DEBUG:boto:Data: 2022-09-02T05:53:14.915 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.916 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.917 DEBUG:boto:Params: {} 2022-09-02T05:53:14.918 DEBUG:boto:Token: None 2022-09-02T05:53:14.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0= 2022-09-02T05:53:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0=', 'Content-Length': '0'} 2022-09-02T05:53:14.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26421ffee5aea3c-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.966 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:14.967 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:14.967 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.968 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.968 DEBUG:boto:Method: GET 2022-09-02T05:53:14.969 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.969 DEBUG:boto:Data: 2022-09-02T05:53:14.970 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.971 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.972 DEBUG:boto:Params: {} 2022-09-02T05:53:14.972 DEBUG:boto:Token: None 2022-09-02T05:53:14.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:14.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0= 2022-09-02T05:53:14.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qs8XdCdBXsp247s3aw8uqSOFHn0=', 'Content-Length': '0'} 2022-09-02T05:53:14.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f268fd45cdcfed-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.977 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.978 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.979 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:14.979 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:14.980 DEBUG:boto:Method: GET 2022-09-02T05:53:14.980 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:14.981 DEBUG:boto:Data: 2022-09-02T05:53:14.981 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:14.982 DEBUG:boto:Port: 8001 2022-09-02T05:53:14.983 DEBUG:boto:Params: {} 2022-09-02T05:53:14.983 DEBUG:boto:Token: None 2022-09-02T05:53:14.984 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:14.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027fc27ea97513598-0063119a4a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:14 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:14.988 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:14.989 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:14.990 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:14.990 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:14.991 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:14.992 DEBUG:boto:Method: GET 2022-09-02T05:53:14.992 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:14.993 DEBUG:boto:Data: 2022-09-02T05:53:14.993 DEBUG:boto:Headers: {} 2022-09-02T05:53:14.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:14.994 DEBUG:boto:Port: 8000 2022-09-02T05:53:14.995 DEBUG:boto:Params: {} 2022-09-02T05:53:14.995 DEBUG:boto:Token: None 2022-09-02T05:53:14.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:14 GMT /xljbkf-74/ 2022-09-02T05:53:14.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY= 2022-09-02T05:53:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:14 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDpvrynvwKMqB6FTtwvbsYo+WVY=', 'Content-Length': '0'} 2022-09-02T05:53:15.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648b70c3eb045262-0063119a4a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.002 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.003 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:15.003 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:15.005 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.005 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:15.006 DEBUG:boto:Data: 2022-09-02T05:53:15.006 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.007 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.008 DEBUG:boto:Params: {} 2022-09-02T05:53:15.008 DEBUG:boto:Token: None 2022-09-02T05:53:15.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj51 2022-09-02T05:53:15.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9eR0mBJ37i4dI1iRgj4zjWLxiEU= 2022-09-02T05:53:15.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9eR0mBJ37i4dI1iRgj4zjWLxiEU=', 'Content-Length': '0'} 2022-09-02T05:53:15.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea50b5768833da3c-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.013 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:15.013 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:15.014 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.014 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:15.015 DEBUG:boto:Data: 2022-09-02T05:53:15.015 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.016 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.017 DEBUG:boto:Params: {} 2022-09-02T05:53:15.017 DEBUG:boto:Token: None 2022-09-02T05:53:15.018 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj51 2022-09-02T05:53:15.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9eR0mBJ37i4dI1iRgj4zjWLxiEU= 2022-09-02T05:53:15.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9eR0mBJ37i4dI1iRgj4zjWLxiEU=', 'Content-Length': '0'} 2022-09-02T05:53:15.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec2e244261e8471e-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:15.023 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:15.024 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:15.024 DEBUG:boto:Method: GET 2022-09-02T05:53:15.025 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:15.026 DEBUG:boto:Data: 2022-09-02T05:53:15.027 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.028 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.029 DEBUG:boto:Params: {} 2022-09-02T05:53:15.029 DEBUG:boto:Token: None 2022-09-02T05:53:15.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj51 2022-09-02T05:53:15.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9G+wypGwZRAgfNVJ0byOpKZ8LJE= 2022-09-02T05:53:15.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9G+wypGwZRAgfNVJ0byOpKZ8LJE=', 'Content-Length': '0'} 2022-09-02T05:53:15.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a238a2cbdfc5b10-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.041 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:15.042 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:15.042 DEBUG:boto:Method: GET 2022-09-02T05:53:15.043 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:15.044 DEBUG:boto:Data: 2022-09-02T05:53:15.044 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.045 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.046 DEBUG:boto:Params: {} 2022-09-02T05:53:15.047 DEBUG:boto:Token: None 2022-09-02T05:53:15.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj51 2022-09-02T05:53:15.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9G+wypGwZRAgfNVJ0byOpKZ8LJE= 2022-09-02T05:53:15.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9G+wypGwZRAgfNVJ0byOpKZ8LJE=', 'Content-Length': '0'} 2022-09-02T05:53:15.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab6207a4b45247e-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.053 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.053 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.054 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:15.054 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:15.055 DEBUG:boto:Method: GET 2022-09-02T05:53:15.056 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:15.056 DEBUG:boto:Data: 2022-09-02T05:53:15.057 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.058 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.058 DEBUG:boto:Params: {} 2022-09-02T05:53:15.059 DEBUG:boto:Token: None 2022-09-02T05:53:15.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061adec20bc1c8f54-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.065 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.066 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.067 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.067 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:15.068 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:15.068 DEBUG:boto:Method: GET 2022-09-02T05:53:15.069 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:15.069 DEBUG:boto:Data: 2022-09-02T05:53:15.070 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.071 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.072 DEBUG:boto:Params: {} 2022-09-02T05:53:15.073 DEBUG:boto:Token: None 2022-09-02T05:53:15.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e21fb03397955d-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.080 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:15.081 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.082 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.083 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.083 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.084 DEBUG:boto:Method: GET 2022-09-02T05:53:15.084 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.085 DEBUG:boto:Data: 2022-09-02T05:53:15.085 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.087 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.087 DEBUG:boto:Params: {} 2022-09-02T05:53:15.088 DEBUG:boto:Token: None 2022-09-02T05:53:15.088 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co= 2022-09-02T05:53:15.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co=', 'Content-Length': '0'} 2022-09-02T05:53:15.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000394a42df0fb02f85-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.094 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.095 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.096 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.096 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.097 DEBUG:boto:Method: GET 2022-09-02T05:53:15.098 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.099 DEBUG:boto:Data: 2022-09-02T05:53:15.100 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.102 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.103 DEBUG:boto:Params: {} 2022-09-02T05:53:15.105 DEBUG:boto:Token: None 2022-09-02T05:53:15.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co= 2022-09-02T05:53:15.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co=', 'Content-Length': '0'} 2022-09-02T05:53:15.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd4c05cd5ef54962-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.113 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.114 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.115 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.116 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.116 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.117 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.118 DEBUG:boto:Data: 2022-09-02T05:53:15.119 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.121 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.122 DEBUG:boto:Params: {} 2022-09-02T05:53:15.123 DEBUG:boto:Token: None 2022-09-02T05:53:15.124 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pQLH11VmhS4lKxhYzLlZDyjZ8qI= 2022-09-02T05:53:15.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pQLH11VmhS4lKxhYzLlZDyjZ8qI=', 'Content-Length': '0'} 2022-09-02T05:53:15.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6654eb89094f9b4-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.131 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.132 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.132 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.133 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.134 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.135 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.135 DEBUG:boto:Data: 2022-09-02T05:53:15.136 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.137 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.138 DEBUG:boto:Params: {} 2022-09-02T05:53:15.138 DEBUG:boto:Token: None 2022-09-02T05:53:15.139 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pQLH11VmhS4lKxhYzLlZDyjZ8qI= 2022-09-02T05:53:15.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pQLH11VmhS4lKxhYzLlZDyjZ8qI=', 'Content-Length': '0'} 2022-09-02T05:53:15.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043bef664cd49f2a2-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:15.144 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.145 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.145 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.146 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.147 DEBUG:boto:Method: GET 2022-09-02T05:53:15.147 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.148 DEBUG:boto:Data: 2022-09-02T05:53:15.149 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.150 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.150 DEBUG:boto:Params: {} 2022-09-02T05:53:15.151 DEBUG:boto:Token: None 2022-09-02T05:53:15.152 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co= 2022-09-02T05:53:15.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co=', 'Content-Length': '0'} 2022-09-02T05:53:15.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5a319d5376123d7-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.156 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.157 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.158 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.158 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.159 DEBUG:boto:Method: GET 2022-09-02T05:53:15.160 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.160 DEBUG:boto:Data: 2022-09-02T05:53:15.161 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.162 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.163 DEBUG:boto:Params: {} 2022-09-02T05:53:15.163 DEBUG:boto:Token: None 2022-09-02T05:53:15.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:15.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co= 2022-09-02T05:53:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CeDRT66nMsc5yu+sTkRJ7M7J0co=', 'Content-Length': '0'} 2022-09-02T05:53:15.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b13e4bed9cdf3e25-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.214 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.215 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.216 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.216 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.217 DEBUG:boto:Method: GET 2022-09-02T05:53:15.217 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.218 DEBUG:boto:Data: 2022-09-02T05:53:15.218 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.220 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.220 DEBUG:boto:Params: {} 2022-09-02T05:53:15.221 DEBUG:boto:Token: None 2022-09-02T05:53:15.221 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000692fa508c93ac217-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.226 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.227 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.228 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.228 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.229 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.229 DEBUG:boto:Method: GET 2022-09-02T05:53:15.230 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.231 DEBUG:boto:Data: 2022-09-02T05:53:15.231 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.232 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.233 DEBUG:boto:Params: {} 2022-09-02T05:53:15.233 DEBUG:boto:Token: None 2022-09-02T05:53:15.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d18ef4853a766688-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.239 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.240 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.241 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.241 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.242 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:15.242 DEBUG:boto:Data: 2022-09-02T05:53:15.243 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.244 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.244 DEBUG:boto:Params: {} 2022-09-02T05:53:15.245 DEBUG:boto:Token: None 2022-09-02T05:53:15.245 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55 2022-09-02T05:53:15.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ac2b69IWtuSXq1Jjr7L1muzE+nk= 2022-09-02T05:53:15.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ac2b69IWtuSXq1Jjr7L1muzE+nk=', 'Content-Length': '0'} 2022-09-02T05:53:15.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc9a2819e717b429-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.250 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.250 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.251 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.251 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:15.252 DEBUG:boto:Data: 2022-09-02T05:53:15.252 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.253 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.254 DEBUG:boto:Params: {} 2022-09-02T05:53:15.254 DEBUG:boto:Token: None 2022-09-02T05:53:15.255 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55 2022-09-02T05:53:15.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ac2b69IWtuSXq1Jjr7L1muzE+nk= 2022-09-02T05:53:15.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ac2b69IWtuSXq1Jjr7L1muzE+nk=', 'Content-Length': '0'} 2022-09-02T05:53:15.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfbcce894a47863b-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:15.259 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.260 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.260 DEBUG:boto:Method: GET 2022-09-02T05:53:15.261 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:15.262 DEBUG:boto:Data: 2022-09-02T05:53:15.262 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.263 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.264 DEBUG:boto:Params: {} 2022-09-02T05:53:15.264 DEBUG:boto:Token: None 2022-09-02T05:53:15.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55 2022-09-02T05:53:15.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t/MZwOVD/Be5lQcI2PpoId8KPrE= 2022-09-02T05:53:15.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t/MZwOVD/Be5lQcI2PpoId8KPrE=', 'Content-Length': '0'} 2022-09-02T05:53:15.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d69be649d001e9c-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.269 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:15.270 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:15.270 DEBUG:boto:Method: GET 2022-09-02T05:53:15.271 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:15.271 DEBUG:boto:Data: 2022-09-02T05:53:15.272 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.273 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.273 DEBUG:boto:Params: {} 2022-09-02T05:53:15.274 DEBUG:boto:Token: None 2022-09-02T05:53:15.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj55 2022-09-02T05:53:15.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t/MZwOVD/Be5lQcI2PpoId8KPrE= 2022-09-02T05:53:15.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t/MZwOVD/Be5lQcI2PpoId8KPrE=', 'Content-Length': '0'} 2022-09-02T05:53:15.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf352b7cdad8243-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.322 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.322 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.323 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.323 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.324 DEBUG:boto:Method: GET 2022-09-02T05:53:15.325 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.325 DEBUG:boto:Data: 2022-09-02T05:53:15.326 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.327 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.327 DEBUG:boto:Params: {} 2022-09-02T05:53:15.328 DEBUG:boto:Token: None 2022-09-02T05:53:15.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483116887cb9ad9a-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.333 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.334 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.334 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.335 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.335 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.336 DEBUG:boto:Method: GET 2022-09-02T05:53:15.336 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:15.337 DEBUG:boto:Data: 2022-09-02T05:53:15.337 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.338 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.339 DEBUG:boto:Params: {} 2022-09-02T05:53:15.339 DEBUG:boto:Token: None 2022-09-02T05:53:15.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c582df551045560-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.345 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:15.346 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.347 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.347 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.348 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.348 DEBUG:boto:Method: GET 2022-09-02T05:53:15.349 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.349 DEBUG:boto:Data: 2022-09-02T05:53:15.350 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.351 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.351 DEBUG:boto:Params: {} 2022-09-02T05:53:15.352 DEBUG:boto:Token: None 2022-09-02T05:53:15.352 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk= 2022-09-02T05:53:15.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk=', 'Content-Length': '0'} 2022-09-02T05:53:15.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce2050731e857e29-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.356 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.357 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.357 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.358 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.358 DEBUG:boto:Method: GET 2022-09-02T05:53:15.359 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.359 DEBUG:boto:Data: 2022-09-02T05:53:15.360 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.361 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.361 DEBUG:boto:Params: {} 2022-09-02T05:53:15.362 DEBUG:boto:Token: None 2022-09-02T05:53:15.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk= 2022-09-02T05:53:15.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk=', 'Content-Length': '0'} 2022-09-02T05:53:15.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bd9c2f9dbd301d-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.366 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.367 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.367 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.368 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.368 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.369 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.369 DEBUG:boto:Data: 2022-09-02T05:53:15.370 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.371 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.371 DEBUG:boto:Params: {} 2022-09-02T05:53:15.372 DEBUG:boto:Token: None 2022-09-02T05:53:15.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kDI0cyGxB9bp2XEV4aP5e2NLSfk= 2022-09-02T05:53:15.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kDI0cyGxB9bp2XEV4aP5e2NLSfk=', 'Content-Length': '0'} 2022-09-02T05:53:15.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1bfcd35e7766510-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.376 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.376 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.377 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.377 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.378 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.378 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.379 DEBUG:boto:Data: 2022-09-02T05:53:15.379 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.380 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.381 DEBUG:boto:Params: {} 2022-09-02T05:53:15.381 DEBUG:boto:Token: None 2022-09-02T05:53:15.382 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kDI0cyGxB9bp2XEV4aP5e2NLSfk= 2022-09-02T05:53:15.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kDI0cyGxB9bp2XEV4aP5e2NLSfk=', 'Content-Length': '0'} 2022-09-02T05:53:15.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b59a31aa6620fa3-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:15.386 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.387 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.387 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.388 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.388 DEBUG:boto:Method: GET 2022-09-02T05:53:15.389 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.389 DEBUG:boto:Data: 2022-09-02T05:53:15.390 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.391 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.391 DEBUG:boto:Params: {} 2022-09-02T05:53:15.392 DEBUG:boto:Token: None 2022-09-02T05:53:15.393 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk= 2022-09-02T05:53:15.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk=', 'Content-Length': '0'} 2022-09-02T05:53:15.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e8911ea37dda91-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.397 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.398 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.398 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.399 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.399 DEBUG:boto:Method: GET 2022-09-02T05:53:15.400 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.400 DEBUG:boto:Data: 2022-09-02T05:53:15.401 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.402 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.402 DEBUG:boto:Params: {} 2022-09-02T05:53:15.403 DEBUG:boto:Token: None 2022-09-02T05:53:15.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:15.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk= 2022-09-02T05:53:15.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ncUy1XkYcqJD5sHCq5H670dQmgk=', 'Content-Length': '0'} 2022-09-02T05:53:15.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000912f0eed0c3ff7f8-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.408 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.409 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.409 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.410 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.410 DEBUG:boto:Method: GET 2022-09-02T05:53:15.411 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.411 DEBUG:boto:Data: 2022-09-02T05:53:15.412 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.413 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.413 DEBUG:boto:Params: {} 2022-09-02T05:53:15.414 DEBUG:boto:Token: None 2022-09-02T05:53:15.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e851b418f97ebb-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.419 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.420 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.421 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.422 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.422 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.423 DEBUG:boto:Method: GET 2022-09-02T05:53:15.423 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.424 DEBUG:boto:Data: 2022-09-02T05:53:15.424 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.425 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.426 DEBUG:boto:Params: {} 2022-09-02T05:53:15.426 DEBUG:boto:Token: None 2022-09-02T05:53:15.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ce1a9e8532d75fb-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.432 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.433 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.434 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.435 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.435 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:15.436 DEBUG:boto:Data: 2022-09-02T05:53:15.436 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.437 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.437 DEBUG:boto:Params: {} 2022-09-02T05:53:15.438 DEBUG:boto:Token: None 2022-09-02T05:53:15.439 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57 2022-09-02T05:53:15.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y3T87azlmTsu8acL6sdf+t1CrIE= 2022-09-02T05:53:15.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y3T87azlmTsu8acL6sdf+t1CrIE=', 'Content-Length': '0'} 2022-09-02T05:53:15.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046706e85650f504a-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.443 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.444 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.445 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.446 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:15.447 DEBUG:boto:Data: 2022-09-02T05:53:15.448 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.450 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.451 DEBUG:boto:Params: {} 2022-09-02T05:53:15.452 DEBUG:boto:Token: None 2022-09-02T05:53:15.453 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57 2022-09-02T05:53:15.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y3T87azlmTsu8acL6sdf+t1CrIE= 2022-09-02T05:53:15.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y3T87azlmTsu8acL6sdf+t1CrIE=', 'Content-Length': '0'} 2022-09-02T05:53:15.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c81413661bc9f7-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:15.459 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.459 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.460 DEBUG:boto:Method: GET 2022-09-02T05:53:15.461 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:15.461 DEBUG:boto:Data: 2022-09-02T05:53:15.462 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.463 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.463 DEBUG:boto:Params: {} 2022-09-02T05:53:15.464 DEBUG:boto:Token: None 2022-09-02T05:53:15.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57 2022-09-02T05:53:15.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2LJmGQ/ERjp7+cIobDvyFsff6Y8= 2022-09-02T05:53:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2LJmGQ/ERjp7+cIobDvyFsff6Y8=', 'Content-Length': '0'} 2022-09-02T05:53:15.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022d6d75f0250857e-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.510 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:15.511 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:15.512 DEBUG:boto:Method: GET 2022-09-02T05:53:15.513 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:15.513 DEBUG:boto:Data: 2022-09-02T05:53:15.514 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.515 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.516 DEBUG:boto:Params: {} 2022-09-02T05:53:15.517 DEBUG:boto:Token: None 2022-09-02T05:53:15.519 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj57 2022-09-02T05:53:15.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2LJmGQ/ERjp7+cIobDvyFsff6Y8= 2022-09-02T05:53:15.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2LJmGQ/ERjp7+cIobDvyFsff6Y8=', 'Content-Length': '0'} 2022-09-02T05:53:15.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000132a8a7fcf2bde4f-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.525 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.525 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.526 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.526 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.527 DEBUG:boto:Method: GET 2022-09-02T05:53:15.528 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.528 DEBUG:boto:Data: 2022-09-02T05:53:15.529 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.530 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.530 DEBUG:boto:Params: {} 2022-09-02T05:53:15.531 DEBUG:boto:Token: None 2022-09-02T05:53:15.532 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5c1df2b5f61b1c-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.537 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.538 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.539 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.540 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.540 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.541 DEBUG:boto:Method: GET 2022-09-02T05:53:15.541 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:15.542 DEBUG:boto:Data: 2022-09-02T05:53:15.543 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.544 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.544 DEBUG:boto:Params: {} 2022-09-02T05:53:15.545 DEBUG:boto:Token: None 2022-09-02T05:53:15.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078cae36f3d0990e0-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.551 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:15.553 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.553 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.554 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.555 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.555 DEBUG:boto:Method: GET 2022-09-02T05:53:15.556 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.556 DEBUG:boto:Data: 2022-09-02T05:53:15.557 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.558 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.559 DEBUG:boto:Params: {} 2022-09-02T05:53:15.559 DEBUG:boto:Token: None 2022-09-02T05:53:15.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc= 2022-09-02T05:53:15.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc=', 'Content-Length': '0'} 2022-09-02T05:53:15.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc968a0a0c2b1bc-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.565 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.565 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.566 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.566 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.567 DEBUG:boto:Method: GET 2022-09-02T05:53:15.567 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.568 DEBUG:boto:Data: 2022-09-02T05:53:15.568 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.569 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.570 DEBUG:boto:Params: {} 2022-09-02T05:53:15.571 DEBUG:boto:Token: None 2022-09-02T05:53:15.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc= 2022-09-02T05:53:15.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc=', 'Content-Length': '0'} 2022-09-02T05:53:15.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065a7864c1c27667b-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.575 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.576 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.576 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.577 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.577 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.578 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.578 DEBUG:boto:Data: 2022-09-02T05:53:15.579 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.580 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.580 DEBUG:boto:Params: {} 2022-09-02T05:53:15.581 DEBUG:boto:Token: None 2022-09-02T05:53:15.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qEO4HRfuM/Ddd/bB1wgl/oiRwTU= 2022-09-02T05:53:15.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qEO4HRfuM/Ddd/bB1wgl/oiRwTU=', 'Content-Length': '0'} 2022-09-02T05:53:15.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809f607c3c73a751-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.585 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.586 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.586 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.587 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.587 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.588 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.588 DEBUG:boto:Data: 2022-09-02T05:53:15.589 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.590 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.590 DEBUG:boto:Params: {} 2022-09-02T05:53:15.591 DEBUG:boto:Token: None 2022-09-02T05:53:15.591 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qEO4HRfuM/Ddd/bB1wgl/oiRwTU= 2022-09-02T05:53:15.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qEO4HRfuM/Ddd/bB1wgl/oiRwTU=', 'Content-Length': '0'} 2022-09-02T05:53:15.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047121679ccbbfb1d-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:15.596 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.596 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.597 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.597 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.598 DEBUG:boto:Method: GET 2022-09-02T05:53:15.598 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.599 DEBUG:boto:Data: 2022-09-02T05:53:15.599 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.600 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.601 DEBUG:boto:Params: {} 2022-09-02T05:53:15.602 DEBUG:boto:Token: None 2022-09-02T05:53:15.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc= 2022-09-02T05:53:15.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc=', 'Content-Length': '0'} 2022-09-02T05:53:15.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027a4dbe1d22c9fa4-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.606 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.607 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.607 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.608 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.609 DEBUG:boto:Method: GET 2022-09-02T05:53:15.609 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.610 DEBUG:boto:Data: 2022-09-02T05:53:15.610 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.611 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.612 DEBUG:boto:Params: {} 2022-09-02T05:53:15.612 DEBUG:boto:Token: None 2022-09-02T05:53:15.613 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:15.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc= 2022-09-02T05:53:15.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAVqFjsNgSeWxW1DvEeZC5XzUlc=', 'Content-Length': '0'} 2022-09-02T05:53:15.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008837a48fd94d573e-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.662 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.663 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.664 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.664 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.665 DEBUG:boto:Method: GET 2022-09-02T05:53:15.666 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.666 DEBUG:boto:Data: 2022-09-02T05:53:15.667 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.668 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.668 DEBUG:boto:Params: {} 2022-09-02T05:53:15.669 DEBUG:boto:Token: None 2022-09-02T05:53:15.670 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ba3e78f7ceb14e-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.675 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.676 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.676 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.677 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.677 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.678 DEBUG:boto:Method: GET 2022-09-02T05:53:15.678 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.679 DEBUG:boto:Data: 2022-09-02T05:53:15.680 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.681 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.681 DEBUG:boto:Params: {} 2022-09-02T05:53:15.682 DEBUG:boto:Token: None 2022-09-02T05:53:15.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd4158226526ca8-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.688 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.689 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.690 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.690 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.691 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:15.691 DEBUG:boto:Data: 2022-09-02T05:53:15.692 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.693 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.693 DEBUG:boto:Params: {} 2022-09-02T05:53:15.694 DEBUG:boto:Token: None 2022-09-02T05:53:15.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68 2022-09-02T05:53:15.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aej0MQbzZrjBCvly0n1gWlM9HoY= 2022-09-02T05:53:15.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aej0MQbzZrjBCvly0n1gWlM9HoY=', 'Content-Length': '0'} 2022-09-02T05:53:15.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdb2f12de75ca99-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.700 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.701 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.702 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.702 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:15.703 DEBUG:boto:Data: 2022-09-02T05:53:15.703 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.704 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.705 DEBUG:boto:Params: {} 2022-09-02T05:53:15.706 DEBUG:boto:Token: None 2022-09-02T05:53:15.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68 2022-09-02T05:53:15.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aej0MQbzZrjBCvly0n1gWlM9HoY= 2022-09-02T05:53:15.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aej0MQbzZrjBCvly0n1gWlM9HoY=', 'Content-Length': '0'} 2022-09-02T05:53:15.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502fa7a590917f2f-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:15.711 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.711 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.712 DEBUG:boto:Method: GET 2022-09-02T05:53:15.713 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:15.713 DEBUG:boto:Data: 2022-09-02T05:53:15.714 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.715 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.715 DEBUG:boto:Params: {} 2022-09-02T05:53:15.716 DEBUG:boto:Token: None 2022-09-02T05:53:15.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68 2022-09-02T05:53:15.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M7uhe1buwS5vzQw5y/RxYuoLHtg= 2022-09-02T05:53:15.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M7uhe1buwS5vzQw5y/RxYuoLHtg=', 'Content-Length': '0'} 2022-09-02T05:53:15.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c7e9d3ce73776f-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.762 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:15.763 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:15.764 DEBUG:boto:Method: GET 2022-09-02T05:53:15.764 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:15.765 DEBUG:boto:Data: 2022-09-02T05:53:15.766 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.767 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.767 DEBUG:boto:Params: {} 2022-09-02T05:53:15.768 DEBUG:boto:Token: None 2022-09-02T05:53:15.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj68 2022-09-02T05:53:15.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M7uhe1buwS5vzQw5y/RxYuoLHtg= 2022-09-02T05:53:15.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M7uhe1buwS5vzQw5y/RxYuoLHtg=', 'Content-Length': '0'} 2022-09-02T05:53:15.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000415788eeae60ea0f-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.773 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.774 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.774 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.775 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.775 DEBUG:boto:Method: GET 2022-09-02T05:53:15.776 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.777 DEBUG:boto:Data: 2022-09-02T05:53:15.777 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.778 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.779 DEBUG:boto:Params: {} 2022-09-02T05:53:15.780 DEBUG:boto:Token: None 2022-09-02T05:53:15.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b05219307aa1db91-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.785 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.789 DEBUG:boto:Method: GET 2022-09-02T05:53:15.789 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:15.790 DEBUG:boto:Data: 2022-09-02T05:53:15.790 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.791 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.792 DEBUG:boto:Params: {} 2022-09-02T05:53:15.792 DEBUG:boto:Token: None 2022-09-02T05:53:15.793 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5c82e085b6e1d8c-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.807 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:15.808 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.809 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.809 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.810 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.810 DEBUG:boto:Method: GET 2022-09-02T05:53:15.811 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.812 DEBUG:boto:Data: 2022-09-02T05:53:15.812 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.813 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.814 DEBUG:boto:Params: {} 2022-09-02T05:53:15.814 DEBUG:boto:Token: None 2022-09-02T05:53:15.815 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.816 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M= 2022-09-02T05:53:15.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M=', 'Content-Length': '0'} 2022-09-02T05:53:15.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8f6406aba28cef7-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.819 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.820 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.821 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.821 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.822 DEBUG:boto:Method: GET 2022-09-02T05:53:15.822 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.823 DEBUG:boto:Data: 2022-09-02T05:53:15.823 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.824 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.825 DEBUG:boto:Params: {} 2022-09-02T05:53:15.826 DEBUG:boto:Token: None 2022-09-02T05:53:15.827 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M= 2022-09-02T05:53:15.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M=', 'Content-Length': '0'} 2022-09-02T05:53:15.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001defb49150b106ea-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.874 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.875 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.875 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.876 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.877 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.877 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.878 DEBUG:boto:Data: 2022-09-02T05:53:15.879 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.880 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.880 DEBUG:boto:Params: {} 2022-09-02T05:53:15.881 DEBUG:boto:Token: None 2022-09-02T05:53:15.882 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e74RppdGYW0NzEp09loTXymc71E= 2022-09-02T05:53:15.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e74RppdGYW0NzEp09loTXymc71E=', 'Content-Length': '0'} 2022-09-02T05:53:15.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004780901774e6fd3b-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.887 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.887 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.888 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.888 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.889 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.889 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.890 DEBUG:boto:Data: 2022-09-02T05:53:15.891 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.892 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.892 DEBUG:boto:Params: {} 2022-09-02T05:53:15.893 DEBUG:boto:Token: None 2022-09-02T05:53:15.894 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e74RppdGYW0NzEp09loTXymc71E= 2022-09-02T05:53:15.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e74RppdGYW0NzEp09loTXymc71E=', 'Content-Length': '0'} 2022-09-02T05:53:15.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2079f1bdca215e5-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:15.899 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.900 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.900 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.901 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.901 DEBUG:boto:Method: GET 2022-09-02T05:53:15.902 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.902 DEBUG:boto:Data: 2022-09-02T05:53:15.903 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.904 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.904 DEBUG:boto:Params: {} 2022-09-02T05:53:15.905 DEBUG:boto:Token: None 2022-09-02T05:53:15.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M= 2022-09-02T05:53:15.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M=', 'Content-Length': '0'} 2022-09-02T05:53:15.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d786417a01a1ab-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.910 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.911 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.911 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.912 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.912 DEBUG:boto:Method: GET 2022-09-02T05:53:15.913 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.913 DEBUG:boto:Data: 2022-09-02T05:53:15.914 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.915 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.915 DEBUG:boto:Params: {} 2022-09-02T05:53:15.916 DEBUG:boto:Token: None 2022-09-02T05:53:15.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:15.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M= 2022-09-02T05:53:15.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7P4COfcryBYTJV1BpeIpRmzJR4M=', 'Content-Length': '0'} 2022-09-02T05:53:15.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d1f45b2abb580bc-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.921 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.921 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.922 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:15.922 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:15.923 DEBUG:boto:Method: GET 2022-09-02T05:53:15.923 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:15.924 DEBUG:boto:Data: 2022-09-02T05:53:15.924 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.925 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.926 DEBUG:boto:Params: {} 2022-09-02T05:53:15.926 DEBUG:boto:Token: None 2022-09-02T05:53:15.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ee3dbd4e75bdb7e-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.932 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.933 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:15.934 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:15.935 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:15.936 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:15.937 DEBUG:boto:Method: GET 2022-09-02T05:53:15.937 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:15.938 DEBUG:boto:Data: 2022-09-02T05:53:15.938 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.940 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.940 DEBUG:boto:Params: {} 2022-09-02T05:53:15.941 DEBUG:boto:Token: None 2022-09-02T05:53:15.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/ 2022-09-02T05:53:15.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o= 2022-09-02T05:53:15.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5Pf+UJjso3EtBpHCX76Mn3ba9o=', 'Content-Length': '0'} 2022-09-02T05:53:15.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000333bbaa6dfc8211f-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.947 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:15.948 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.949 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.949 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.950 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:15.950 DEBUG:boto:Data: 2022-09-02T05:53:15.951 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.952 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.952 DEBUG:boto:Params: {} 2022-09-02T05:53:15.953 DEBUG:boto:Token: None 2022-09-02T05:53:15.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69 2022-09-02T05:53:15.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J0ilWnK/jfFBFVZ5mOdgpvQGydc= 2022-09-02T05:53:15.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J0ilWnK/jfFBFVZ5mOdgpvQGydc=', 'Content-Length': '0'} 2022-09-02T05:53:15.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002996df95ba06cfab-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.957 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.958 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.958 DEBUG:boto:Method: HEAD 2022-09-02T05:53:15.959 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:15.959 DEBUG:boto:Data: 2022-09-02T05:53:15.960 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.961 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.961 DEBUG:boto:Params: {} 2022-09-02T05:53:15.962 DEBUG:boto:Token: None 2022-09-02T05:53:15.963 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69 2022-09-02T05:53:15.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J0ilWnK/jfFBFVZ5mOdgpvQGydc= 2022-09-02T05:53:15.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J0ilWnK/jfFBFVZ5mOdgpvQGydc=', 'Content-Length': '0'} 2022-09-02T05:53:15.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233d3c410589553b-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:15.968 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.968 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.969 DEBUG:boto:Method: GET 2022-09-02T05:53:15.970 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:15.971 DEBUG:boto:Data: 2022-09-02T05:53:15.972 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:15.974 DEBUG:boto:Port: 8001 2022-09-02T05:53:15.975 DEBUG:boto:Params: {} 2022-09-02T05:53:15.976 DEBUG:boto:Token: None 2022-09-02T05:53:15.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69 2022-09-02T05:53:15.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xlfrTt1Z5JsxL3t/Rb0IHw32FfI= 2022-09-02T05:53:15.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xlfrTt1Z5JsxL3t/Rb0IHw32FfI=', 'Content-Length': '0'} 2022-09-02T05:53:15.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003da474d4f9548db-0063119a4b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:15.981 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:15.982 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:15.982 DEBUG:boto:Method: GET 2022-09-02T05:53:15.983 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:15.983 DEBUG:boto:Data: 2022-09-02T05:53:15.984 DEBUG:boto:Headers: {} 2022-09-02T05:53:15.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:15.985 DEBUG:boto:Port: 8000 2022-09-02T05:53:15.985 DEBUG:boto:Params: {} 2022-09-02T05:53:15.986 DEBUG:boto:Token: None 2022-09-02T05:53:15.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:15 GMT /xljbkf-74/obj69 2022-09-02T05:53:15.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xlfrTt1Z5JsxL3t/Rb0IHw32FfI= 2022-09-02T05:53:15.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:15 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xlfrTt1Z5JsxL3t/Rb0IHw32FfI=', 'Content-Length': '0'} 2022-09-02T05:53:15.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f10c8d6c1a02ee9b-0063119a4b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:15 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.034 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.035 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.035 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:16.036 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:16.036 DEBUG:boto:Method: GET 2022-09-02T05:53:16.037 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:16.037 DEBUG:boto:Data: 2022-09-02T05:53:16.038 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.039 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.039 DEBUG:boto:Params: {} 2022-09-02T05:53:16.040 DEBUG:boto:Token: None 2022-09-02T05:53:16.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dbda5a1a49426fc-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.046 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.046 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.047 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.048 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:16.048 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:16.049 DEBUG:boto:Method: GET 2022-09-02T05:53:16.049 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:16.050 DEBUG:boto:Data: 2022-09-02T05:53:16.050 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.051 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.052 DEBUG:boto:Params: {} 2022-09-02T05:53:16.052 DEBUG:boto:Token: None 2022-09-02T05:53:16.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a749295a4b51bacb-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.058 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:16.059 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.060 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.060 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.061 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.061 DEBUG:boto:Method: GET 2022-09-02T05:53:16.062 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.063 DEBUG:boto:Data: 2022-09-02T05:53:16.063 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.064 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.065 DEBUG:boto:Params: {} 2022-09-02T05:53:16.065 DEBUG:boto:Token: None 2022-09-02T05:53:16.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA= 2022-09-02T05:53:16.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA=', 'Content-Length': '0'} 2022-09-02T05:53:16.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a74a02133a778e9-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.070 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.070 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.071 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.071 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.072 DEBUG:boto:Method: GET 2022-09-02T05:53:16.072 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.073 DEBUG:boto:Data: 2022-09-02T05:53:16.074 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.075 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.075 DEBUG:boto:Params: {} 2022-09-02T05:53:16.076 DEBUG:boto:Token: None 2022-09-02T05:53:16.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA= 2022-09-02T05:53:16.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA=', 'Content-Length': '0'} 2022-09-02T05:53:16.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a9c02e72696c15e-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.081 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.081 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.082 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.082 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.083 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.083 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.084 DEBUG:boto:Data: 2022-09-02T05:53:16.084 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.085 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.086 DEBUG:boto:Params: {} 2022-09-02T05:53:16.086 DEBUG:boto:Token: None 2022-09-02T05:53:16.087 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9K8jC7/3tSEyeJ4cM7WQAVpug8g= 2022-09-02T05:53:16.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9K8jC7/3tSEyeJ4cM7WQAVpug8g=', 'Content-Length': '0'} 2022-09-02T05:53:16.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3597c3de2750d33-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.090 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.091 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.091 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.092 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.092 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.093 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.093 DEBUG:boto:Data: 2022-09-02T05:53:16.094 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.095 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.095 DEBUG:boto:Params: {} 2022-09-02T05:53:16.096 DEBUG:boto:Token: None 2022-09-02T05:53:16.096 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9K8jC7/3tSEyeJ4cM7WQAVpug8g= 2022-09-02T05:53:16.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9K8jC7/3tSEyeJ4cM7WQAVpug8g=', 'Content-Length': '0'} 2022-09-02T05:53:16.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18b61c0a5528672-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:16.101 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.102 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.102 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.103 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.103 DEBUG:boto:Method: GET 2022-09-02T05:53:16.104 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.104 DEBUG:boto:Data: 2022-09-02T05:53:16.105 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.106 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.106 DEBUG:boto:Params: {} 2022-09-02T05:53:16.107 DEBUG:boto:Token: None 2022-09-02T05:53:16.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA= 2022-09-02T05:53:16.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA=', 'Content-Length': '0'} 2022-09-02T05:53:16.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001834b4ea77d0a5c3-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.112 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.112 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.113 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.113 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.114 DEBUG:boto:Method: GET 2022-09-02T05:53:16.114 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.115 DEBUG:boto:Data: 2022-09-02T05:53:16.115 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.116 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.117 DEBUG:boto:Params: {} 2022-09-02T05:53:16.117 DEBUG:boto:Token: None 2022-09-02T05:53:16.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:16.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA= 2022-09-02T05:53:16.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a9Yz6mnuYMYaFgopzxENWzrUsiA=', 'Content-Length': '0'} 2022-09-02T05:53:16.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2f53a7c77d968f-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.122 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.123 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.123 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.124 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.124 DEBUG:boto:Method: GET 2022-09-02T05:53:16.125 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.125 DEBUG:boto:Data: 2022-09-02T05:53:16.126 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.127 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.128 DEBUG:boto:Params: {} 2022-09-02T05:53:16.128 DEBUG:boto:Token: None 2022-09-02T05:53:16.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc288aefc30d493-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.178 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.180 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.181 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.181 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.182 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.183 DEBUG:boto:Method: GET 2022-09-02T05:53:16.184 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.184 DEBUG:boto:Data: 2022-09-02T05:53:16.185 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.186 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.186 DEBUG:boto:Params: {} 2022-09-02T05:53:16.187 DEBUG:boto:Token: None 2022-09-02T05:53:16.188 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007141ad8258c7aba3-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.193 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.194 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.195 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.195 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.196 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:16.196 DEBUG:boto:Data: 2022-09-02T05:53:16.197 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.198 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.199 DEBUG:boto:Params: {} 2022-09-02T05:53:16.199 DEBUG:boto:Token: None 2022-09-02T05:53:16.200 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76 2022-09-02T05:53:16.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hhhnV5hr4blgCHCKYnJOWBHOrCc= 2022-09-02T05:53:16.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hhhnV5hr4blgCHCKYnJOWBHOrCc=', 'Content-Length': '0'} 2022-09-02T05:53:16.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e9206ba0fcc6e2-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.204 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.204 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.205 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.205 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:16.206 DEBUG:boto:Data: 2022-09-02T05:53:16.206 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.208 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.208 DEBUG:boto:Params: {} 2022-09-02T05:53:16.209 DEBUG:boto:Token: None 2022-09-02T05:53:16.209 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76 2022-09-02T05:53:16.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hhhnV5hr4blgCHCKYnJOWBHOrCc= 2022-09-02T05:53:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hhhnV5hr4blgCHCKYnJOWBHOrCc=', 'Content-Length': '0'} 2022-09-02T05:53:16.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02ff3b53795775a-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:16.215 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.216 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.217 DEBUG:boto:Method: GET 2022-09-02T05:53:16.218 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:16.219 DEBUG:boto:Data: 2022-09-02T05:53:16.220 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.222 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.223 DEBUG:boto:Params: {} 2022-09-02T05:53:16.224 DEBUG:boto:Token: None 2022-09-02T05:53:16.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76 2022-09-02T05:53:16.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RxCm4VF/VsTHCeznbrNF8Nwy/G4= 2022-09-02T05:53:16.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RxCm4VF/VsTHCeznbrNF8Nwy/G4=', 'Content-Length': '0'} 2022-09-02T05:53:16.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f6fc5d9488ee5dc-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.231 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:16.232 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:16.232 DEBUG:boto:Method: GET 2022-09-02T05:53:16.233 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:16.234 DEBUG:boto:Data: 2022-09-02T05:53:16.235 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.237 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.237 DEBUG:boto:Params: {} 2022-09-02T05:53:16.238 DEBUG:boto:Token: None 2022-09-02T05:53:16.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj76 2022-09-02T05:53:16.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RxCm4VF/VsTHCeznbrNF8Nwy/G4= 2022-09-02T05:53:16.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RxCm4VF/VsTHCeznbrNF8Nwy/G4=', 'Content-Length': '0'} 2022-09-02T05:53:16.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fcf612e70ad9747-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.243 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.243 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.244 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.244 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.245 DEBUG:boto:Method: GET 2022-09-02T05:53:16.245 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.246 DEBUG:boto:Data: 2022-09-02T05:53:16.246 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.247 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.248 DEBUG:boto:Params: {} 2022-09-02T05:53:16.249 DEBUG:boto:Token: None 2022-09-02T05:53:16.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da8e99f62607644-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.254 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.255 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.255 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.256 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.256 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.257 DEBUG:boto:Method: GET 2022-09-02T05:53:16.257 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:16.258 DEBUG:boto:Data: 2022-09-02T05:53:16.258 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.259 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.260 DEBUG:boto:Params: {} 2022-09-02T05:53:16.261 DEBUG:boto:Token: None 2022-09-02T05:53:16.261 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.262 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60c599ef1710ce2-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.267 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:16.269 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.270 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.271 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.272 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.273 DEBUG:boto:Method: GET 2022-09-02T05:53:16.274 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.275 DEBUG:boto:Data: 2022-09-02T05:53:16.276 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.278 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.279 DEBUG:boto:Params: {} 2022-09-02T05:53:16.280 DEBUG:boto:Token: None 2022-09-02T05:53:16.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA= 2022-09-02T05:53:16.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA=', 'Content-Length': '0'} 2022-09-02T05:53:16.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071362dc2f315bfbc-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.285 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.285 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.286 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.286 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.287 DEBUG:boto:Method: GET 2022-09-02T05:53:16.287 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.288 DEBUG:boto:Data: 2022-09-02T05:53:16.289 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.290 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.291 DEBUG:boto:Params: {} 2022-09-02T05:53:16.291 DEBUG:boto:Token: None 2022-09-02T05:53:16.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA= 2022-09-02T05:53:16.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA=', 'Content-Length': '0'} 2022-09-02T05:53:16.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1630c86360d97ff-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.343 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.344 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.344 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.345 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.345 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.346 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.346 DEBUG:boto:Data: 2022-09-02T05:53:16.347 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.348 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.349 DEBUG:boto:Params: {} 2022-09-02T05:53:16.349 DEBUG:boto:Token: None 2022-09-02T05:53:16.350 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2arXDqZLltD3pa7mViO3ks0V97I= 2022-09-02T05:53:16.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2arXDqZLltD3pa7mViO3ks0V97I=', 'Content-Length': '0'} 2022-09-02T05:53:16.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f76d9eac9e09da1-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.354 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.356 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.357 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.358 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.359 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.360 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.361 DEBUG:boto:Data: 2022-09-02T05:53:16.361 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.362 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.363 DEBUG:boto:Params: {} 2022-09-02T05:53:16.364 DEBUG:boto:Token: None 2022-09-02T05:53:16.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2arXDqZLltD3pa7mViO3ks0V97I= 2022-09-02T05:53:16.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2arXDqZLltD3pa7mViO3ks0V97I=', 'Content-Length': '0'} 2022-09-02T05:53:16.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000966590e47283355c-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:16.370 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.371 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.372 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.373 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.373 DEBUG:boto:Method: GET 2022-09-02T05:53:16.374 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.374 DEBUG:boto:Data: 2022-09-02T05:53:16.375 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.377 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.379 DEBUG:boto:Params: {} 2022-09-02T05:53:16.379 DEBUG:boto:Token: None 2022-09-02T05:53:16.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA= 2022-09-02T05:53:16.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA=', 'Content-Length': '0'} 2022-09-02T05:53:16.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058becdf34a8c8483-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.387 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.387 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.388 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.389 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.389 DEBUG:boto:Method: GET 2022-09-02T05:53:16.390 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.391 DEBUG:boto:Data: 2022-09-02T05:53:16.391 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.392 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.393 DEBUG:boto:Params: {} 2022-09-02T05:53:16.394 DEBUG:boto:Token: None 2022-09-02T05:53:16.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:16.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA= 2022-09-02T05:53:16.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/rytXVYPhUmpHN2+gzk/eEgfAKA=', 'Content-Length': '0'} 2022-09-02T05:53:16.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3a81c14553593d-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.399 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.400 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.400 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.401 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.401 DEBUG:boto:Method: GET 2022-09-02T05:53:16.402 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.402 DEBUG:boto:Data: 2022-09-02T05:53:16.403 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.404 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.405 DEBUG:boto:Params: {} 2022-09-02T05:53:16.405 DEBUG:boto:Token: None 2022-09-02T05:53:16.406 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006548a25e7216faf3-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.411 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.412 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.413 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.413 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.414 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.415 DEBUG:boto:Method: GET 2022-09-02T05:53:16.415 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.416 DEBUG:boto:Data: 2022-09-02T05:53:16.417 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.418 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.419 DEBUG:boto:Params: {} 2022-09-02T05:53:16.420 DEBUG:boto:Token: None 2022-09-02T05:53:16.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.422 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021d1b49e160bb5a8-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.427 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.428 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.429 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.429 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.430 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:16.431 DEBUG:boto:Data: 2022-09-02T05:53:16.432 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.433 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.434 DEBUG:boto:Params: {} 2022-09-02T05:53:16.434 DEBUG:boto:Token: None 2022-09-02T05:53:16.435 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85 2022-09-02T05:53:16.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W3cEHfdY2zo770Id3MsaPEH6CtM= 2022-09-02T05:53:16.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W3cEHfdY2zo770Id3MsaPEH6CtM=', 'Content-Length': '0'} 2022-09-02T05:53:16.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a79b0fcccbfb179-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.441 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.442 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.442 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.443 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:16.444 DEBUG:boto:Data: 2022-09-02T05:53:16.445 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.446 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.447 DEBUG:boto:Params: {} 2022-09-02T05:53:16.448 DEBUG:boto:Token: None 2022-09-02T05:53:16.448 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85 2022-09-02T05:53:16.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W3cEHfdY2zo770Id3MsaPEH6CtM= 2022-09-02T05:53:16.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W3cEHfdY2zo770Id3MsaPEH6CtM=', 'Content-Length': '0'} 2022-09-02T05:53:16.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009095ab0ce1cd789b-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:16.454 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.455 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.455 DEBUG:boto:Method: GET 2022-09-02T05:53:16.456 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:16.457 DEBUG:boto:Data: 2022-09-02T05:53:16.457 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.459 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.460 DEBUG:boto:Params: {} 2022-09-02T05:53:16.461 DEBUG:boto:Token: None 2022-09-02T05:53:16.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85 2022-09-02T05:53:16.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vr0U3Uye0P9N3eA2bg8ncF8EwZ8= 2022-09-02T05:53:16.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vr0U3Uye0P9N3eA2bg8ncF8EwZ8=', 'Content-Length': '0'} 2022-09-02T05:53:16.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000863ff70009e4bb3f-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.506 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:16.507 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:16.508 DEBUG:boto:Method: GET 2022-09-02T05:53:16.509 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:16.509 DEBUG:boto:Data: 2022-09-02T05:53:16.510 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.512 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.513 DEBUG:boto:Params: {} 2022-09-02T05:53:16.513 DEBUG:boto:Token: None 2022-09-02T05:53:16.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj85 2022-09-02T05:53:16.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vr0U3Uye0P9N3eA2bg8ncF8EwZ8= 2022-09-02T05:53:16.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vr0U3Uye0P9N3eA2bg8ncF8EwZ8=', 'Content-Length': '0'} 2022-09-02T05:53:16.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbc262ddbecafbf4-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.519 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.520 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.521 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.522 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.523 DEBUG:boto:Method: GET 2022-09-02T05:53:16.523 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.524 DEBUG:boto:Data: 2022-09-02T05:53:16.525 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.526 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.527 DEBUG:boto:Params: {} 2022-09-02T05:53:16.528 DEBUG:boto:Token: None 2022-09-02T05:53:16.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3aaebf166278dca-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.533 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.535 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.535 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.536 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.537 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.537 DEBUG:boto:Method: GET 2022-09-02T05:53:16.538 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:16.539 DEBUG:boto:Data: 2022-09-02T05:53:16.539 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.541 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.541 DEBUG:boto:Params: {} 2022-09-02T05:53:16.542 DEBUG:boto:Token: None 2022-09-02T05:53:16.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f74928586acec2de-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.549 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:16.551 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.552 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.553 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.553 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.554 DEBUG:boto:Method: GET 2022-09-02T05:53:16.555 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.556 DEBUG:boto:Data: 2022-09-02T05:53:16.556 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.558 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.558 DEBUG:boto:Params: {} 2022-09-02T05:53:16.559 DEBUG:boto:Token: None 2022-09-02T05:53:16.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI= 2022-09-02T05:53:16.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI=', 'Content-Length': '0'} 2022-09-02T05:53:16.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ea99c05881c080-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.565 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.565 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.566 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.567 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.568 DEBUG:boto:Method: GET 2022-09-02T05:53:16.569 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.569 DEBUG:boto:Data: 2022-09-02T05:53:16.570 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.571 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.572 DEBUG:boto:Params: {} 2022-09-02T05:53:16.572 DEBUG:boto:Token: None 2022-09-02T05:53:16.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI= 2022-09-02T05:53:16.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI=', 'Content-Length': '0'} 2022-09-02T05:53:16.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bf98b7c288313c6-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.579 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.579 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.580 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.581 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.581 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.582 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.583 DEBUG:boto:Data: 2022-09-02T05:53:16.583 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.584 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.585 DEBUG:boto:Params: {} 2022-09-02T05:53:16.585 DEBUG:boto:Token: None 2022-09-02T05:53:16.586 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+hO3rrtPFNDGERHVyXtE5uV0SSU= 2022-09-02T05:53:16.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+hO3rrtPFNDGERHVyXtE5uV0SSU=', 'Content-Length': '0'} 2022-09-02T05:53:16.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec9475d85b74098-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.591 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.591 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.592 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.592 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.593 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.593 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.594 DEBUG:boto:Data: 2022-09-02T05:53:16.595 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.596 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.596 DEBUG:boto:Params: {} 2022-09-02T05:53:16.597 DEBUG:boto:Token: None 2022-09-02T05:53:16.598 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+hO3rrtPFNDGERHVyXtE5uV0SSU= 2022-09-02T05:53:16.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+hO3rrtPFNDGERHVyXtE5uV0SSU=', 'Content-Length': '0'} 2022-09-02T05:53:16.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008954f79fad488b03-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:16.603 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.603 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.604 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.605 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.605 DEBUG:boto:Method: GET 2022-09-02T05:53:16.606 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.607 DEBUG:boto:Data: 2022-09-02T05:53:16.607 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.609 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.609 DEBUG:boto:Params: {} 2022-09-02T05:53:16.610 DEBUG:boto:Token: None 2022-09-02T05:53:16.611 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI= 2022-09-02T05:53:16.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI=', 'Content-Length': '0'} 2022-09-02T05:53:16.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097020c61b2f020c4-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.616 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.616 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.617 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.618 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.618 DEBUG:boto:Method: GET 2022-09-02T05:53:16.619 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.620 DEBUG:boto:Data: 2022-09-02T05:53:16.620 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.622 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.622 DEBUG:boto:Params: {} 2022-09-02T05:53:16.623 DEBUG:boto:Token: None 2022-09-02T05:53:16.624 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:16.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI= 2022-09-02T05:53:16.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q30TlXwmj4f+Sqjv3FF0zWE1mmI=', 'Content-Length': '0'} 2022-09-02T05:53:16.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ae1cb0fb06af61-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.670 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.671 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.672 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.673 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.673 DEBUG:boto:Method: GET 2022-09-02T05:53:16.674 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.675 DEBUG:boto:Data: 2022-09-02T05:53:16.675 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.677 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.677 DEBUG:boto:Params: {} 2022-09-02T05:53:16.678 DEBUG:boto:Token: None 2022-09-02T05:53:16.679 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebbee827d5ff5783-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.685 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.686 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.687 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.688 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.688 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.689 DEBUG:boto:Method: GET 2022-09-02T05:53:16.690 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.691 DEBUG:boto:Data: 2022-09-02T05:53:16.691 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.692 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.693 DEBUG:boto:Params: {} 2022-09-02T05:53:16.694 DEBUG:boto:Token: None 2022-09-02T05:53:16.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a2aad32072c289-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.700 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.701 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.702 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.702 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.703 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:16.703 DEBUG:boto:Data: 2022-09-02T05:53:16.704 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.705 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.705 DEBUG:boto:Params: {} 2022-09-02T05:53:16.706 DEBUG:boto:Token: None 2022-09-02T05:53:16.707 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92 2022-09-02T05:53:16.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+fTrsk8khgJcaXNp309dOcex70I= 2022-09-02T05:53:16.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+fTrsk8khgJcaXNp309dOcex70I=', 'Content-Length': '0'} 2022-09-02T05:53:16.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93e1b4053e9b3f0-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.711 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.712 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.713 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.714 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:16.715 DEBUG:boto:Data: 2022-09-02T05:53:16.715 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.717 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.718 DEBUG:boto:Params: {} 2022-09-02T05:53:16.719 DEBUG:boto:Token: None 2022-09-02T05:53:16.720 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92 2022-09-02T05:53:16.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+fTrsk8khgJcaXNp309dOcex70I= 2022-09-02T05:53:16.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+fTrsk8khgJcaXNp309dOcex70I=', 'Content-Length': '0'} 2022-09-02T05:53:16.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb75451b125bd74-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:16.727 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.727 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.728 DEBUG:boto:Method: GET 2022-09-02T05:53:16.728 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:16.729 DEBUG:boto:Data: 2022-09-02T05:53:16.729 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.730 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.731 DEBUG:boto:Params: {} 2022-09-02T05:53:16.732 DEBUG:boto:Token: None 2022-09-02T05:53:16.732 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92 2022-09-02T05:53:16.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYSK0yplwP6shE71ZihZuFeJ3dw= 2022-09-02T05:53:16.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYSK0yplwP6shE71ZihZuFeJ3dw=', 'Content-Length': '0'} 2022-09-02T05:53:16.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bfedfa2641d1a6d-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.736 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:16.737 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:16.738 DEBUG:boto:Method: GET 2022-09-02T05:53:16.738 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:16.739 DEBUG:boto:Data: 2022-09-02T05:53:16.740 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.741 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.741 DEBUG:boto:Params: {} 2022-09-02T05:53:16.742 DEBUG:boto:Token: None 2022-09-02T05:53:16.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj92 2022-09-02T05:53:16.743 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYSK0yplwP6shE71ZihZuFeJ3dw= 2022-09-02T05:53:16.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYSK0yplwP6shE71ZihZuFeJ3dw=', 'Content-Length': '0'} 2022-09-02T05:53:16.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069801daf0e3ad60b-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.748 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.748 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.749 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.749 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.750 DEBUG:boto:Method: GET 2022-09-02T05:53:16.751 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.751 DEBUG:boto:Data: 2022-09-02T05:53:16.752 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.753 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.754 DEBUG:boto:Params: {} 2022-09-02T05:53:16.754 DEBUG:boto:Token: None 2022-09-02T05:53:16.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000687eb4470ae57a6-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.761 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.761 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.762 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.763 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.763 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.764 DEBUG:boto:Method: GET 2022-09-02T05:53:16.764 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:16.765 DEBUG:boto:Data: 2022-09-02T05:53:16.766 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.767 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.767 DEBUG:boto:Params: {} 2022-09-02T05:53:16.768 DEBUG:boto:Token: None 2022-09-02T05:53:16.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066ec686ab9c66019-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.814 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:16.816 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.816 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.817 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.817 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.818 DEBUG:boto:Method: GET 2022-09-02T05:53:16.819 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.820 DEBUG:boto:Data: 2022-09-02T05:53:16.820 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.822 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.823 DEBUG:boto:Params: {} 2022-09-02T05:53:16.824 DEBUG:boto:Token: None 2022-09-02T05:53:16.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs= 2022-09-02T05:53:16.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs=', 'Content-Length': '0'} 2022-09-02T05:53:16.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823c2c82f02df659-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.843 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.843 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.844 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.845 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.845 DEBUG:boto:Method: GET 2022-09-02T05:53:16.846 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.847 DEBUG:boto:Data: 2022-09-02T05:53:16.847 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.848 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.849 DEBUG:boto:Params: {} 2022-09-02T05:53:16.849 DEBUG:boto:Token: None 2022-09-02T05:53:16.850 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs= 2022-09-02T05:53:16.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs=', 'Content-Length': '0'} 2022-09-02T05:53:16.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f351f17b444a699-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.855 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.855 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.856 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.856 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.857 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.857 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.858 DEBUG:boto:Data: 2022-09-02T05:53:16.858 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.859 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.860 DEBUG:boto:Params: {} 2022-09-02T05:53:16.860 DEBUG:boto:Token: None 2022-09-02T05:53:16.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vk64UXvCF1m0sL5K3TFZTMxtKM0= 2022-09-02T05:53:16.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vk64UXvCF1m0sL5K3TFZTMxtKM0=', 'Content-Length': '0'} 2022-09-02T05:53:16.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628f9219f544f535-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.865 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.865 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.866 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.866 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.867 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.867 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.868 DEBUG:boto:Data: 2022-09-02T05:53:16.868 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.869 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.870 DEBUG:boto:Params: {} 2022-09-02T05:53:16.871 DEBUG:boto:Token: None 2022-09-02T05:53:16.871 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vk64UXvCF1m0sL5K3TFZTMxtKM0= 2022-09-02T05:53:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vk64UXvCF1m0sL5K3TFZTMxtKM0=', 'Content-Length': '0'} 2022-09-02T05:53:16.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003faa9b8c34eaa0b7-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:16.876 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.877 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.877 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.878 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.879 DEBUG:boto:Method: GET 2022-09-02T05:53:16.879 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.880 DEBUG:boto:Data: 2022-09-02T05:53:16.880 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.881 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.882 DEBUG:boto:Params: {} 2022-09-02T05:53:16.883 DEBUG:boto:Token: None 2022-09-02T05:53:16.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs= 2022-09-02T05:53:16.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs=', 'Content-Length': '0'} 2022-09-02T05:53:16.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38b787344f7b7d6-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.930 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.931 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.931 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.932 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.933 DEBUG:boto:Method: GET 2022-09-02T05:53:16.933 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.934 DEBUG:boto:Data: 2022-09-02T05:53:16.934 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.936 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.936 DEBUG:boto:Params: {} 2022-09-02T05:53:16.937 DEBUG:boto:Token: None 2022-09-02T05:53:16.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:16.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs= 2022-09-02T05:53:16.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQYSr+s37B5i0A16BRpzooyRlOs=', 'Content-Length': '0'} 2022-09-02T05:53:16.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd3c42848998687-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.942 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.943 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.944 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:16.944 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:16.945 DEBUG:boto:Method: GET 2022-09-02T05:53:16.945 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:16.946 DEBUG:boto:Data: 2022-09-02T05:53:16.947 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.948 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.948 DEBUG:boto:Params: {} 2022-09-02T05:53:16.949 DEBUG:boto:Token: None 2022-09-02T05:53:16.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d4edc14737c7ef-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.955 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.956 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:16.956 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:16.957 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:16.958 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:16.959 DEBUG:boto:Method: GET 2022-09-02T05:53:16.959 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:16.960 DEBUG:boto:Data: 2022-09-02T05:53:16.960 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.962 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.962 DEBUG:boto:Params: {} 2022-09-02T05:53:16.963 DEBUG:boto:Token: None 2022-09-02T05:53:16.964 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/ 2022-09-02T05:53:16.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI= 2022-09-02T05:53:16.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9n7QC5VKvwwaz3pIk9lcSf2yhkI=', 'Content-Length': '0'} 2022-09-02T05:53:16.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053300e6fa34896a7-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.969 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:16.970 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.971 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.971 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.972 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:16.972 DEBUG:boto:Data: 2022-09-02T05:53:16.973 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.974 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.974 DEBUG:boto:Params: {} 2022-09-02T05:53:16.975 DEBUG:boto:Token: None 2022-09-02T05:53:16.975 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95 2022-09-02T05:53:16.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a2vW11wqZfKwMc+rq/HzV/5Yq4M= 2022-09-02T05:53:16.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a2vW11wqZfKwMc+rq/HzV/5Yq4M=', 'Content-Length': '0'} 2022-09-02T05:53:16.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c09ded62fe239dae-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.980 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.980 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.981 DEBUG:boto:Method: HEAD 2022-09-02T05:53:16.981 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:16.982 DEBUG:boto:Data: 2022-09-02T05:53:16.982 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:16.983 DEBUG:boto:Port: 8000 2022-09-02T05:53:16.984 DEBUG:boto:Params: {} 2022-09-02T05:53:16.984 DEBUG:boto:Token: None 2022-09-02T05:53:16.985 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95 2022-09-02T05:53:16.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a2vW11wqZfKwMc+rq/HzV/5Yq4M= 2022-09-02T05:53:16.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a2vW11wqZfKwMc+rq/HzV/5Yq4M=', 'Content-Length': '0'} 2022-09-02T05:53:16.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036884627ff4b7acb-0063119a4c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:16.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:16.989 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:16.990 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:16.991 DEBUG:boto:Method: GET 2022-09-02T05:53:16.991 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:16.992 DEBUG:boto:Data: 2022-09-02T05:53:16.993 DEBUG:boto:Headers: {} 2022-09-02T05:53:16.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:16.994 DEBUG:boto:Port: 8001 2022-09-02T05:53:16.994 DEBUG:boto:Params: {} 2022-09-02T05:53:16.995 DEBUG:boto:Token: None 2022-09-02T05:53:16.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:16 GMT /xljbkf-74/obj95 2022-09-02T05:53:16.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wSiPI+UEuxfMqP+/OZ2D51QIf6E= 2022-09-02T05:53:16.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:16 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wSiPI+UEuxfMqP+/OZ2D51QIf6E=', 'Content-Length': '0'} 2022-09-02T05:53:16.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f945493a8283ec3b-0063119a4c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:16 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.042 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:17.043 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:17.044 DEBUG:boto:Method: GET 2022-09-02T05:53:17.044 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:17.045 DEBUG:boto:Data: 2022-09-02T05:53:17.045 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:17.046 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.047 DEBUG:boto:Params: {} 2022-09-02T05:53:17.047 DEBUG:boto:Token: None 2022-09-02T05:53:17.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj95 2022-09-02T05:53:17.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D+VjmbC0eICSBuF4ShUKJXbFvp8= 2022-09-02T05:53:17.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D+VjmbC0eICSBuF4ShUKJXbFvp8=', 'Content-Length': '0'} 2022-09-02T05:53:17.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be95006d564cf8d4-0063119a4d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.053 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.053 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.054 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:17.054 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:17.055 DEBUG:boto:Method: GET 2022-09-02T05:53:17.055 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:17.056 DEBUG:boto:Data: 2022-09-02T05:53:17.056 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.057 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.058 DEBUG:boto:Params: {} 2022-09-02T05:53:17.058 DEBUG:boto:Token: None 2022-09-02T05:53:17.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d638f0880f8e8b89-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.064 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.064 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.065 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.065 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:17.066 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:17.067 DEBUG:boto:Method: GET 2022-09-02T05:53:17.067 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:17.068 DEBUG:boto:Data: 2022-09-02T05:53:17.068 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:17.070 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.070 DEBUG:boto:Params: {} 2022-09-02T05:53:17.071 DEBUG:boto:Token: None 2022-09-02T05:53:17.071 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6f6f34241e1282e-0063119a4d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.077 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.079 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:53:17.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:53:17.080 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.081 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.081 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.082 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:17.083 DEBUG:boto:Data: 2022-09-02T05:53:17.083 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.084 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.085 DEBUG:boto:Params: {} 2022-09-02T05:53:17.085 DEBUG:boto:Token: None 2022-09-02T05:53:17.086 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NyOHk5BaETPcVTVkntBq8PKvyHM= 2022-09-02T05:53:17.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NyOHk5BaETPcVTVkntBq8PKvyHM=', 'Content-Length': '0'} 2022-09-02T05:53:17.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3c60be50f5908e8-0063119a4d-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.090 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.091 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.092 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.092 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:17.093 DEBUG:boto:Data: 2022-09-02T05:53:17.093 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.094 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.095 DEBUG:boto:Params: {} 2022-09-02T05:53:17.096 DEBUG:boto:Token: None 2022-09-02T05:53:17.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NyOHk5BaETPcVTVkntBq8PKvyHM= 2022-09-02T05:53:17.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NyOHk5BaETPcVTVkntBq8PKvyHM=', 'Content-Length': '0'} 2022-09-02T05:53:17.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e921f92fc65aa16-0063119a4d-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.103 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:53:17.104 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.105 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.105 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.106 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.107 DEBUG:boto:Method: GET 2022-09-02T05:53:17.107 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.108 DEBUG:boto:Data: 2022-09-02T05:53:17.108 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.109 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.110 DEBUG:boto:Params: {} 2022-09-02T05:53:17.111 DEBUG:boto:Token: None 2022-09-02T05:53:17.112 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/?versions 2022-09-02T05:53:17.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA= 2022-09-02T05:53:17.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA=', 'Content-Length': '0'} 2022-09-02T05:53:17.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0cce408b3edaa9e-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.119 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:53:17.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:53:17.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:53:17.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:53:17.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:53:17.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:53:17.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:53:17.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:53:17.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:53:17.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:53:17.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:53:17.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:53:17.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:53:17.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:53:17.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:53:17.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:53:17.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:53:17.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:53:17.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:53:17.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:53:17.130 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:53:17.131 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.131 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.132 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.132 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.133 DEBUG:boto:Method: GET 2022-09-02T05:53:17.133 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.134 DEBUG:boto:Data: 2022-09-02T05:53:17.134 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.135 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.136 DEBUG:boto:Params: {} 2022-09-02T05:53:17.136 DEBUG:boto:Token: None 2022-09-02T05:53:17.137 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/?versions 2022-09-02T05:53:17.138 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA= 2022-09-02T05:53:17.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA=', 'Content-Length': '0'} 2022-09-02T05:53:17.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b76d572bf93ea10-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.142 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:53:17.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:53:17.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:53:17.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:53:17.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:53:17.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:53:17.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:53:17.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:53:17.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:53:17.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:53:17.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:53:17.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:53:17.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:53:17.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:53:17.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:53:17.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:53:17.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:53:17.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:53:17.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:53:17.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:53:17.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.153 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.154 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.154 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.155 DEBUG:boto:Method: GET 2022-09-02T05:53:17.155 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.156 DEBUG:boto:Data: 2022-09-02T05:53:17.156 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.157 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.158 DEBUG:boto:Params: {} 2022-09-02T05:53:17.159 DEBUG:boto:Token: None 2022-09-02T05:53:17.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/?versions 2022-09-02T05:53:17.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA= 2022-09-02T05:53:17.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA=', 'Content-Length': '0'} 2022-09-02T05:53:17.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f1ad6b2399fc8a7-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.164 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.165 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.166 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.167 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.167 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.168 DEBUG:boto:Method: GET 2022-09-02T05:53:17.168 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:53:17.169 DEBUG:boto:Data: 2022-09-02T05:53:17.169 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.170 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.171 DEBUG:boto:Params: {} 2022-09-02T05:53:17.171 DEBUG:boto:Token: None 2022-09-02T05:53:17.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/?versions 2022-09-02T05:53:17.172 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA= 2022-09-02T05:53:17.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dd5WN+h99s7iVTvyqDzdtfW7IcA=', 'Content-Length': '0'} 2022-09-02T05:53:17.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea72978468af703-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.177 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:17.179 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.179 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.180 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.180 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.181 DEBUG:boto:Method: GET 2022-09-02T05:53:17.181 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.182 DEBUG:boto:Data: 2022-09-02T05:53:17.183 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.184 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.184 DEBUG:boto:Params: {} 2022-09-02T05:53:17.185 DEBUG:boto:Token: None 2022-09-02T05:53:17.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo= 2022-09-02T05:53:17.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo=', 'Content-Length': '0'} 2022-09-02T05:53:17.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000504ad95d82df482-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.230 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.231 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.232 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.233 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.234 DEBUG:boto:Method: GET 2022-09-02T05:53:17.234 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.235 DEBUG:boto:Data: 2022-09-02T05:53:17.236 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.238 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.239 DEBUG:boto:Params: {} 2022-09-02T05:53:17.240 DEBUG:boto:Token: None 2022-09-02T05:53:17.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo= 2022-09-02T05:53:17.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo=', 'Content-Length': '0'} 2022-09-02T05:53:17.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92bdac5f7a460f4-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.247 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.248 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.248 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.249 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.249 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.250 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.250 DEBUG:boto:Data: 2022-09-02T05:53:17.251 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.252 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.252 DEBUG:boto:Params: {} 2022-09-02T05:53:17.253 DEBUG:boto:Token: None 2022-09-02T05:53:17.253 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o3c7E2G7LVh+amEuok28o2lkOUQ= 2022-09-02T05:53:17.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o3c7E2G7LVh+amEuok28o2lkOUQ=', 'Content-Length': '0'} 2022-09-02T05:53:17.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b28dc8322bad2e-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.257 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.258 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.258 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.259 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.259 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.260 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.260 DEBUG:boto:Data: 2022-09-02T05:53:17.261 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.262 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.262 DEBUG:boto:Params: {} 2022-09-02T05:53:17.263 DEBUG:boto:Token: None 2022-09-02T05:53:17.263 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o3c7E2G7LVh+amEuok28o2lkOUQ= 2022-09-02T05:53:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o3c7E2G7LVh+amEuok28o2lkOUQ=', 'Content-Length': '0'} 2022-09-02T05:53:17.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073fa1178fd3bb72e-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:17.268 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.269 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.269 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.270 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.270 DEBUG:boto:Method: GET 2022-09-02T05:53:17.271 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.271 DEBUG:boto:Data: 2022-09-02T05:53:17.272 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.273 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.274 DEBUG:boto:Params: {} 2022-09-02T05:53:17.274 DEBUG:boto:Token: None 2022-09-02T05:53:17.275 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo= 2022-09-02T05:53:17.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo=', 'Content-Length': '0'} 2022-09-02T05:53:17.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003952142e120e1d63-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.279 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.280 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.280 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.281 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.281 DEBUG:boto:Method: GET 2022-09-02T05:53:17.282 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.282 DEBUG:boto:Data: 2022-09-02T05:53:17.283 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.284 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.285 DEBUG:boto:Params: {} 2022-09-02T05:53:17.285 DEBUG:boto:Token: None 2022-09-02T05:53:17.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:53:17.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo= 2022-09-02T05:53:17.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EVvy4EFEtfAc1h6kM4jBdC8dGKo=', 'Content-Length': '0'} 2022-09-02T05:53:17.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b9bb114a6b7265-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.290 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.291 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.292 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.292 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.293 DEBUG:boto:Method: GET 2022-09-02T05:53:17.293 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.294 DEBUG:boto:Data: 2022-09-02T05:53:17.294 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.295 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.296 DEBUG:boto:Params: {} 2022-09-02T05:53:17.297 DEBUG:boto:Token: None 2022-09-02T05:53:17.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097bca84de6aaf7bb-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.302 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.303 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.304 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.304 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.305 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.305 DEBUG:boto:Method: GET 2022-09-02T05:53:17.306 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.306 DEBUG:boto:Data: 2022-09-02T05:53:17.307 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.308 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.309 DEBUG:boto:Params: {} 2022-09-02T05:53:17.309 DEBUG:boto:Token: None 2022-09-02T05:53:17.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc538c3e70edf80d-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.315 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.316 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.316 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.317 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.317 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:17.318 DEBUG:boto:Data: 2022-09-02T05:53:17.319 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.320 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.320 DEBUG:boto:Params: {} 2022-09-02T05:53:17.321 DEBUG:boto:Token: None 2022-09-02T05:53:17.321 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11 2022-09-02T05:53:17.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S+C1+wqUPnkP7Wtd6HzSeD4F5xE= 2022-09-02T05:53:17.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S+C1+wqUPnkP7Wtd6HzSeD4F5xE=', 'Content-Length': '0'} 2022-09-02T05:53:17.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb12d6a9b74dd838-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.326 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.326 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.327 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.328 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:17.329 DEBUG:boto:Data: 2022-09-02T05:53:17.330 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.332 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.332 DEBUG:boto:Params: {} 2022-09-02T05:53:17.333 DEBUG:boto:Token: None 2022-09-02T05:53:17.334 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11 2022-09-02T05:53:17.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S+C1+wqUPnkP7Wtd6HzSeD4F5xE= 2022-09-02T05:53:17.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S+C1+wqUPnkP7Wtd6HzSeD4F5xE=', 'Content-Length': '0'} 2022-09-02T05:53:17.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6e7405294ec40a-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:53:17.339 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.340 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.340 DEBUG:boto:Method: GET 2022-09-02T05:53:17.341 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:17.341 DEBUG:boto:Data: 2022-09-02T05:53:17.342 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.343 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.344 DEBUG:boto:Params: {} 2022-09-02T05:53:17.345 DEBUG:boto:Token: None 2022-09-02T05:53:17.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11 2022-09-02T05:53:17.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mxd2g3s50iIje5rR2RKAUJ8VQkE= 2022-09-02T05:53:17.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mxd2g3s50iIje5rR2RKAUJ8VQkE=', 'Content-Length': '0'} 2022-09-02T05:53:17.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fa35145b7cdb8ee-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.349 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:53:17.350 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:53:17.351 DEBUG:boto:Method: GET 2022-09-02T05:53:17.351 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:53:17.352 DEBUG:boto:Data: 2022-09-02T05:53:17.352 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.353 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.354 DEBUG:boto:Params: {} 2022-09-02T05:53:17.354 DEBUG:boto:Token: None 2022-09-02T05:53:17.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj11 2022-09-02T05:53:17.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mxd2g3s50iIje5rR2RKAUJ8VQkE= 2022-09-02T05:53:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mxd2g3s50iIje5rR2RKAUJ8VQkE=', 'Content-Length': '0'} 2022-09-02T05:53:17.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a41e7dcf0b34005e-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.406 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.407 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.407 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.408 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.408 DEBUG:boto:Method: GET 2022-09-02T05:53:17.409 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.409 DEBUG:boto:Data: 2022-09-02T05:53:17.410 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.411 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.412 DEBUG:boto:Params: {} 2022-09-02T05:53:17.412 DEBUG:boto:Token: None 2022-09-02T05:53:17.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001007e44fac1cff92-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.418 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.418 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.419 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.420 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.420 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.421 DEBUG:boto:Method: GET 2022-09-02T05:53:17.421 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:53:17.422 DEBUG:boto:Data: 2022-09-02T05:53:17.422 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.423 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.424 DEBUG:boto:Params: {} 2022-09-02T05:53:17.424 DEBUG:boto:Token: None 2022-09-02T05:53:17.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f004f961ef2bcc-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.430 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:17.431 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.432 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.432 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.433 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.433 DEBUG:boto:Method: GET 2022-09-02T05:53:17.434 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.434 DEBUG:boto:Data: 2022-09-02T05:53:17.435 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.436 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.436 DEBUG:boto:Params: {} 2022-09-02T05:53:17.437 DEBUG:boto:Token: None 2022-09-02T05:53:17.438 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM= 2022-09-02T05:53:17.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM=', 'Content-Length': '0'} 2022-09-02T05:53:17.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070dc45505725f6c7-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.442 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.442 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.443 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.443 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.444 DEBUG:boto:Method: GET 2022-09-02T05:53:17.444 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.445 DEBUG:boto:Data: 2022-09-02T05:53:17.445 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.446 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.446 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.447 DEBUG:boto:Params: {} 2022-09-02T05:53:17.448 DEBUG:boto:Token: None 2022-09-02T05:53:17.448 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM= 2022-09-02T05:53:17.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM=', 'Content-Length': '0'} 2022-09-02T05:53:17.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045de19b6595cdede-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.452 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.453 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.453 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.454 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.454 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.455 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.455 DEBUG:boto:Data: 2022-09-02T05:53:17.456 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.457 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.457 DEBUG:boto:Params: {} 2022-09-02T05:53:17.458 DEBUG:boto:Token: None 2022-09-02T05:53:17.459 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GGmtRg3PgMEmLXTxPnUShpknKcI= 2022-09-02T05:53:17.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GGmtRg3PgMEmLXTxPnUShpknKcI=', 'Content-Length': '0'} 2022-09-02T05:53:17.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003310622d24fa1fae-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.463 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.463 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.464 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.464 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.465 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.465 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.466 DEBUG:boto:Data: 2022-09-02T05:53:17.466 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.467 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.467 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.468 DEBUG:boto:Params: {} 2022-09-02T05:53:17.468 DEBUG:boto:Token: None 2022-09-02T05:53:17.469 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GGmtRg3PgMEmLXTxPnUShpknKcI= 2022-09-02T05:53:17.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GGmtRg3PgMEmLXTxPnUShpknKcI=', 'Content-Length': '0'} 2022-09-02T05:53:17.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e96b79db79e009-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:17.473 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.474 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.475 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.475 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.476 DEBUG:boto:Method: GET 2022-09-02T05:53:17.476 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.477 DEBUG:boto:Data: 2022-09-02T05:53:17.477 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.478 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.479 DEBUG:boto:Params: {} 2022-09-02T05:53:17.479 DEBUG:boto:Token: None 2022-09-02T05:53:17.480 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM= 2022-09-02T05:53:17.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM=', 'Content-Length': '0'} 2022-09-02T05:53:17.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ee33e9eb6bcdee3-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.484 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.485 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.485 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.486 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.487 DEBUG:boto:Method: GET 2022-09-02T05:53:17.487 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.488 DEBUG:boto:Data: 2022-09-02T05:53:17.488 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.489 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.490 DEBUG:boto:Params: {} 2022-09-02T05:53:17.491 DEBUG:boto:Token: None 2022-09-02T05:53:17.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:53:17.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM= 2022-09-02T05:53:17.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tZg/ohhneFP2SYC1YcFjsbBEYjM=', 'Content-Length': '0'} 2022-09-02T05:53:17.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac7d36ca6f3e3427-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.495 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.496 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.496 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.497 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.497 DEBUG:boto:Method: GET 2022-09-02T05:53:17.498 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.498 DEBUG:boto:Data: 2022-09-02T05:53:17.499 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.500 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.500 DEBUG:boto:Params: {} 2022-09-02T05:53:17.501 DEBUG:boto:Token: None 2022-09-02T05:53:17.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003075e3fa55bdcaea-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.506 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.507 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.508 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.508 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.509 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.509 DEBUG:boto:Method: GET 2022-09-02T05:53:17.510 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.511 DEBUG:boto:Data: 2022-09-02T05:53:17.511 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.512 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.513 DEBUG:boto:Params: {} 2022-09-02T05:53:17.513 DEBUG:boto:Token: None 2022-09-02T05:53:17.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc7c7eb130d44256-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.519 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.519 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.520 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.520 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.521 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:17.521 DEBUG:boto:Data: 2022-09-02T05:53:17.522 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.523 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.523 DEBUG:boto:Params: {} 2022-09-02T05:53:17.524 DEBUG:boto:Token: None 2022-09-02T05:53:17.525 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115 2022-09-02T05:53:17.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TYZ0y3h8XjCkciw2fUxGJcdULZI= 2022-09-02T05:53:17.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TYZ0y3h8XjCkciw2fUxGJcdULZI=', 'Content-Length': '0'} 2022-09-02T05:53:17.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001307c06277b05924-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.529 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.529 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.530 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.530 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:17.531 DEBUG:boto:Data: 2022-09-02T05:53:17.531 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.532 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.533 DEBUG:boto:Params: {} 2022-09-02T05:53:17.533 DEBUG:boto:Token: None 2022-09-02T05:53:17.534 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115 2022-09-02T05:53:17.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TYZ0y3h8XjCkciw2fUxGJcdULZI= 2022-09-02T05:53:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TYZ0y3h8XjCkciw2fUxGJcdULZI=', 'Content-Length': '0'} 2022-09-02T05:53:17.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a936b7b969d7c2e1-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:53:17.539 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.540 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.541 DEBUG:boto:Method: GET 2022-09-02T05:53:17.542 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:17.543 DEBUG:boto:Data: 2022-09-02T05:53:17.544 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.546 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.547 DEBUG:boto:Params: {} 2022-09-02T05:53:17.548 DEBUG:boto:Token: None 2022-09-02T05:53:17.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115 2022-09-02T05:53:17.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+eHGNRyFZrijMMd8aFoK4sH5gA= 2022-09-02T05:53:17.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+eHGNRyFZrijMMd8aFoK4sH5gA=', 'Content-Length': '0'} 2022-09-02T05:53:17.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6eb22a192b4662e-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.598 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:53:17.599 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:53:17.599 DEBUG:boto:Method: GET 2022-09-02T05:53:17.600 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:53:17.600 DEBUG:boto:Data: 2022-09-02T05:53:17.601 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.602 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.603 DEBUG:boto:Params: {} 2022-09-02T05:53:17.603 DEBUG:boto:Token: None 2022-09-02T05:53:17.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj115 2022-09-02T05:53:17.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+eHGNRyFZrijMMd8aFoK4sH5gA= 2022-09-02T05:53:17.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+eHGNRyFZrijMMd8aFoK4sH5gA=', 'Content-Length': '0'} 2022-09-02T05:53:17.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3d1b680eb2e1ec-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.608 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.609 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.609 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.610 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.611 DEBUG:boto:Method: GET 2022-09-02T05:53:17.611 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.612 DEBUG:boto:Data: 2022-09-02T05:53:17.612 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.613 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.614 DEBUG:boto:Params: {} 2022-09-02T05:53:17.614 DEBUG:boto:Token: None 2022-09-02T05:53:17.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed40f17a505b4268-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.620 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.620 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.621 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.621 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.622 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.623 DEBUG:boto:Method: GET 2022-09-02T05:53:17.623 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:53:17.624 DEBUG:boto:Data: 2022-09-02T05:53:17.624 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.625 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.626 DEBUG:boto:Params: {} 2022-09-02T05:53:17.626 DEBUG:boto:Token: None 2022-09-02T05:53:17.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61499b08c9d7ca8-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.631 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:17.633 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.633 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.634 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.634 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.635 DEBUG:boto:Method: GET 2022-09-02T05:53:17.636 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.636 DEBUG:boto:Data: 2022-09-02T05:53:17.637 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.638 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.639 DEBUG:boto:Params: {} 2022-09-02T05:53:17.639 DEBUG:boto:Token: None 2022-09-02T05:53:17.640 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA= 2022-09-02T05:53:17.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA=', 'Content-Length': '0'} 2022-09-02T05:53:17.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca2db73ad5fd4647-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.644 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.645 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.645 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.646 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.646 DEBUG:boto:Method: GET 2022-09-02T05:53:17.647 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.648 DEBUG:boto:Data: 2022-09-02T05:53:17.648 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.649 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.650 DEBUG:boto:Params: {} 2022-09-02T05:53:17.650 DEBUG:boto:Token: None 2022-09-02T05:53:17.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA= 2022-09-02T05:53:17.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA=', 'Content-Length': '0'} 2022-09-02T05:53:17.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8246400b9b3e34d-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.655 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.655 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.656 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.656 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.657 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.657 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.658 DEBUG:boto:Data: 2022-09-02T05:53:17.659 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.660 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.660 DEBUG:boto:Params: {} 2022-09-02T05:53:17.661 DEBUG:boto:Token: None 2022-09-02T05:53:17.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KKckV+LFA4Yqxf39TgWHq/zt7/E= 2022-09-02T05:53:17.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KKckV+LFA4Yqxf39TgWHq/zt7/E=', 'Content-Length': '0'} 2022-09-02T05:53:17.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebec2fe08ec97c86-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.665 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.666 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.667 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.667 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.668 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.668 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.669 DEBUG:boto:Data: 2022-09-02T05:53:17.669 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.671 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.671 DEBUG:boto:Params: {} 2022-09-02T05:53:17.672 DEBUG:boto:Token: None 2022-09-02T05:53:17.673 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KKckV+LFA4Yqxf39TgWHq/zt7/E= 2022-09-02T05:53:17.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KKckV+LFA4Yqxf39TgWHq/zt7/E=', 'Content-Length': '0'} 2022-09-02T05:53:17.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a08a57bb36d593-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:17.678 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.679 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.680 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.681 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.682 DEBUG:boto:Method: GET 2022-09-02T05:53:17.682 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.683 DEBUG:boto:Data: 2022-09-02T05:53:17.684 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.685 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.686 DEBUG:boto:Params: {} 2022-09-02T05:53:17.686 DEBUG:boto:Token: None 2022-09-02T05:53:17.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA= 2022-09-02T05:53:17.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA=', 'Content-Length': '0'} 2022-09-02T05:53:17.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004017bb91abd8f4e5-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.692 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.693 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.693 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.694 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.694 DEBUG:boto:Method: GET 2022-09-02T05:53:17.695 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.695 DEBUG:boto:Data: 2022-09-02T05:53:17.696 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.696 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.697 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.697 DEBUG:boto:Params: {} 2022-09-02T05:53:17.698 DEBUG:boto:Token: None 2022-09-02T05:53:17.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:53:17.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA= 2022-09-02T05:53:17.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dHtKy6i7dpOhSs2QYCccouXyfsA=', 'Content-Length': '0'} 2022-09-02T05:53:17.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a1de1235701e49-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.703 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.703 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.704 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.704 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.705 DEBUG:boto:Method: GET 2022-09-02T05:53:17.706 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.706 DEBUG:boto:Data: 2022-09-02T05:53:17.707 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.708 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.708 DEBUG:boto:Params: {} 2022-09-02T05:53:17.709 DEBUG:boto:Token: None 2022-09-02T05:53:17.709 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecd413027b0b8b48-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.754 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.755 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.756 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.756 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.757 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.757 DEBUG:boto:Method: GET 2022-09-02T05:53:17.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.758 DEBUG:boto:Data: 2022-09-02T05:53:17.759 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.760 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.760 DEBUG:boto:Params: {} 2022-09-02T05:53:17.761 DEBUG:boto:Token: None 2022-09-02T05:53:17.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c60f07e4f17a56b-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.766 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.767 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.768 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.768 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.769 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:17.769 DEBUG:boto:Data: 2022-09-02T05:53:17.770 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.771 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.772 DEBUG:boto:Params: {} 2022-09-02T05:53:17.773 DEBUG:boto:Token: None 2022-09-02T05:53:17.773 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17 2022-09-02T05:53:17.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RBtLVeVj2s+lvuFvWihu0R263ng= 2022-09-02T05:53:17.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RBtLVeVj2s+lvuFvWihu0R263ng=', 'Content-Length': '0'} 2022-09-02T05:53:17.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000414f27e8e24ecc4a-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.777 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.778 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.779 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.779 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:17.780 DEBUG:boto:Data: 2022-09-02T05:53:17.780 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.781 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.781 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.782 DEBUG:boto:Params: {} 2022-09-02T05:53:17.782 DEBUG:boto:Token: None 2022-09-02T05:53:17.783 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17 2022-09-02T05:53:17.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RBtLVeVj2s+lvuFvWihu0R263ng= 2022-09-02T05:53:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RBtLVeVj2s+lvuFvWihu0R263ng=', 'Content-Length': '0'} 2022-09-02T05:53:17.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331631769fbaa773-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:53:17.788 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.788 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.789 DEBUG:boto:Method: GET 2022-09-02T05:53:17.790 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:17.790 DEBUG:boto:Data: 2022-09-02T05:53:17.791 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.792 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.793 DEBUG:boto:Params: {} 2022-09-02T05:53:17.793 DEBUG:boto:Token: None 2022-09-02T05:53:17.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17 2022-09-02T05:53:17.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e5Hr73rx2PUzvR3nEuCtsiPNAS8= 2022-09-02T05:53:17.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e5Hr73rx2PUzvR3nEuCtsiPNAS8=', 'Content-Length': '0'} 2022-09-02T05:53:17.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748c9bbea459405b-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.799 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:53:17.800 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:53:17.800 DEBUG:boto:Method: GET 2022-09-02T05:53:17.801 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:53:17.802 DEBUG:boto:Data: 2022-09-02T05:53:17.802 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.803 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.804 DEBUG:boto:Params: {} 2022-09-02T05:53:17.805 DEBUG:boto:Token: None 2022-09-02T05:53:17.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj17 2022-09-02T05:53:17.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e5Hr73rx2PUzvR3nEuCtsiPNAS8= 2022-09-02T05:53:17.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e5Hr73rx2PUzvR3nEuCtsiPNAS8=', 'Content-Length': '0'} 2022-09-02T05:53:17.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39ea14dd868f41a-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.850 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.851 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.851 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.852 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.852 DEBUG:boto:Method: GET 2022-09-02T05:53:17.853 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.853 DEBUG:boto:Data: 2022-09-02T05:53:17.854 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.855 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.856 DEBUG:boto:Params: {} 2022-09-02T05:53:17.856 DEBUG:boto:Token: None 2022-09-02T05:53:17.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008209488ed6d2359d-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.862 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.862 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.863 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.864 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.864 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.865 DEBUG:boto:Method: GET 2022-09-02T05:53:17.865 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:53:17.866 DEBUG:boto:Data: 2022-09-02T05:53:17.867 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.868 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.868 DEBUG:boto:Params: {} 2022-09-02T05:53:17.869 DEBUG:boto:Token: None 2022-09-02T05:53:17.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196c4ce8c4a9d214-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.875 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:17.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:17.876 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:17.877 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:17.877 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.878 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.879 DEBUG:boto:Method: GET 2022-09-02T05:53:17.879 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.880 DEBUG:boto:Data: 2022-09-02T05:53:17.880 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.881 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.882 DEBUG:boto:Params: {} 2022-09-02T05:53:17.883 DEBUG:boto:Token: None 2022-09-02T05:53:17.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao= 2022-09-02T05:53:17.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao=', 'Content-Length': '0'} 2022-09-02T05:53:17.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6c8d97b7d8d884-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.888 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:17.888 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:17.889 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.889 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.890 DEBUG:boto:Method: GET 2022-09-02T05:53:17.890 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.891 DEBUG:boto:Data: 2022-09-02T05:53:17.892 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.892 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.893 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.893 DEBUG:boto:Params: {} 2022-09-02T05:53:17.894 DEBUG:boto:Token: None 2022-09-02T05:53:17.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao= 2022-09-02T05:53:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao=', 'Content-Length': '0'} 2022-09-02T05:53:17.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000694a68960308b19a-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.899 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:17.899 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:17.900 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.900 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.901 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.901 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.902 DEBUG:boto:Data: 2022-09-02T05:53:17.903 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.904 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.904 DEBUG:boto:Params: {} 2022-09-02T05:53:17.905 DEBUG:boto:Token: None 2022-09-02T05:53:17.905 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MchMDm3jvXm9J/UNJtpkugGNqhQ= 2022-09-02T05:53:17.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MchMDm3jvXm9J/UNJtpkugGNqhQ=', 'Content-Length': '0'} 2022-09-02T05:53:17.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd7d1c146bc1973-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.909 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:17.909 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:17.910 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.910 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.911 DEBUG:boto:Method: HEAD 2022-09-02T05:53:17.911 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.912 DEBUG:boto:Data: 2022-09-02T05:53:17.913 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.913 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.914 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.914 DEBUG:boto:Params: {} 2022-09-02T05:53:17.915 DEBUG:boto:Token: None 2022-09-02T05:53:17.915 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MchMDm3jvXm9J/UNJtpkugGNqhQ= 2022-09-02T05:53:17.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MchMDm3jvXm9J/UNJtpkugGNqhQ=', 'Content-Length': '0'} 2022-09-02T05:53:17.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000085eac33dd7c6b9e-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:17.920 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:17.920 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:17.921 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.921 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.922 DEBUG:boto:Method: GET 2022-09-02T05:53:17.922 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.923 DEBUG:boto:Data: 2022-09-02T05:53:17.923 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.924 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.925 DEBUG:boto:Params: {} 2022-09-02T05:53:17.926 DEBUG:boto:Token: None 2022-09-02T05:53:17.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao= 2022-09-02T05:53:17.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao=', 'Content-Length': '0'} 2022-09-02T05:53:17.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b671512eb9590b8-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.931 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:17.932 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:17.932 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.933 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.933 DEBUG:boto:Method: GET 2022-09-02T05:53:17.934 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.934 DEBUG:boto:Data: 2022-09-02T05:53:17.935 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.936 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:17.936 DEBUG:boto:Port: 8000 2022-09-02T05:53:17.937 DEBUG:boto:Params: {} 2022-09-02T05:53:17.937 DEBUG:boto:Token: None 2022-09-02T05:53:17.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:53:17.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao= 2022-09-02T05:53:17.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeVzjq4+ucW6ZRaGL5Ny+upogao=', 'Content-Length': '0'} 2022-09-02T05:53:17.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad1baa51324cd0a-0063119a4d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:17.987 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:17.988 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:17.988 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:17.989 DEBUG:boto:Method: GET 2022-09-02T05:53:17.989 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:17.990 DEBUG:boto:Data: 2022-09-02T05:53:17.991 DEBUG:boto:Headers: {} 2022-09-02T05:53:17.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:17.992 DEBUG:boto:Port: 8001 2022-09-02T05:53:17.993 DEBUG:boto:Params: {} 2022-09-02T05:53:17.994 DEBUG:boto:Token: None 2022-09-02T05:53:17.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:17 GMT /xljbkf-74/ 2022-09-02T05:53:17.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8= 2022-09-02T05:53:17.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:17 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvOLMeDvkd943eO93ljN4biRff8=', 'Content-Length': '0'} 2022-09-02T05:53:17.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187cae11e11de1f8-0063119a4d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:17 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:17.999 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.000 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.000 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.001 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.002 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.002 DEBUG:boto:Method: GET 2022-09-02T05:53:18.003 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.004 DEBUG:boto:Data: 2022-09-02T05:53:18.004 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.006 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.006 DEBUG:boto:Params: {} 2022-09-02T05:53:18.007 DEBUG:boto:Token: None 2022-09-02T05:53:18.008 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ca43fedb437795-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.013 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.014 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:18.014 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:18.015 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.016 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:18.016 DEBUG:boto:Data: 2022-09-02T05:53:18.017 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.018 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.018 DEBUG:boto:Params: {} 2022-09-02T05:53:18.019 DEBUG:boto:Token: None 2022-09-02T05:53:18.019 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj19 2022-09-02T05:53:18.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AwUWvz+a3nFEVAFc23fStlmoRU4= 2022-09-02T05:53:18.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AwUWvz+a3nFEVAFc23fStlmoRU4=', 'Content-Length': '0'} 2022-09-02T05:53:18.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b412427de5507851-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.024 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:18.024 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:18.025 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.026 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:18.027 DEBUG:boto:Data: 2022-09-02T05:53:18.027 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.028 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.029 DEBUG:boto:Params: {} 2022-09-02T05:53:18.030 DEBUG:boto:Token: None 2022-09-02T05:53:18.031 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj19 2022-09-02T05:53:18.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AwUWvz+a3nFEVAFc23fStlmoRU4= 2022-09-02T05:53:18.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AwUWvz+a3nFEVAFc23fStlmoRU4=', 'Content-Length': '0'} 2022-09-02T05:53:18.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000199f802c86b22e4b-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:53:18.036 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:18.036 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:18.037 DEBUG:boto:Method: GET 2022-09-02T05:53:18.038 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:18.038 DEBUG:boto:Data: 2022-09-02T05:53:18.039 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.040 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.040 DEBUG:boto:Params: {} 2022-09-02T05:53:18.041 DEBUG:boto:Token: None 2022-09-02T05:53:18.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj19 2022-09-02T05:53:18.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W++mCZAAsDq+bQhGJECXSdX34xo= 2022-09-02T05:53:18.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W++mCZAAsDq+bQhGJECXSdX34xo=', 'Content-Length': '0'} 2022-09-02T05:53:18.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86e98a24815bec8-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.086 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:53:18.087 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:53:18.087 DEBUG:boto:Method: GET 2022-09-02T05:53:18.088 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:53:18.088 DEBUG:boto:Data: 2022-09-02T05:53:18.089 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.089 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.090 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.091 DEBUG:boto:Params: {} 2022-09-02T05:53:18.091 DEBUG:boto:Token: None 2022-09-02T05:53:18.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj19 2022-09-02T05:53:18.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W++mCZAAsDq+bQhGJECXSdX34xo= 2022-09-02T05:53:18.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W++mCZAAsDq+bQhGJECXSdX34xo=', 'Content-Length': '0'} 2022-09-02T05:53:18.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010d4a4f1addf9538-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.096 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.097 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.097 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.098 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.099 DEBUG:boto:Method: GET 2022-09-02T05:53:18.100 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.100 DEBUG:boto:Data: 2022-09-02T05:53:18.101 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.102 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.102 DEBUG:boto:Params: {} 2022-09-02T05:53:18.103 DEBUG:boto:Token: None 2022-09-02T05:53:18.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b1219a95667e7a-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.108 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.109 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.110 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.110 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.111 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.111 DEBUG:boto:Method: GET 2022-09-02T05:53:18.112 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:53:18.113 DEBUG:boto:Data: 2022-09-02T05:53:18.113 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.114 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.115 DEBUG:boto:Params: {} 2022-09-02T05:53:18.116 DEBUG:boto:Token: None 2022-09-02T05:53:18.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000742ff987228ff571-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.122 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:18.123 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.124 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.125 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.125 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.126 DEBUG:boto:Method: GET 2022-09-02T05:53:18.126 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.127 DEBUG:boto:Data: 2022-09-02T05:53:18.127 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.128 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.129 DEBUG:boto:Params: {} 2022-09-02T05:53:18.130 DEBUG:boto:Token: None 2022-09-02T05:53:18.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4= 2022-09-02T05:53:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4=', 'Content-Length': '0'} 2022-09-02T05:53:18.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005849b032ba7868e6-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.135 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.136 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.137 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.138 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.139 DEBUG:boto:Method: GET 2022-09-02T05:53:18.140 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.141 DEBUG:boto:Data: 2022-09-02T05:53:18.142 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.143 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.145 DEBUG:boto:Params: {} 2022-09-02T05:53:18.146 DEBUG:boto:Token: None 2022-09-02T05:53:18.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4= 2022-09-02T05:53:18.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4=', 'Content-Length': '0'} 2022-09-02T05:53:18.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee14e8e58566143-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.153 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.154 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.155 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.155 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.156 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.156 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.157 DEBUG:boto:Data: 2022-09-02T05:53:18.157 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.159 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.159 DEBUG:boto:Params: {} 2022-09-02T05:53:18.160 DEBUG:boto:Token: None 2022-09-02T05:53:18.160 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xiec2yEDs6n1YFQvwMPIAYfR8lc= 2022-09-02T05:53:18.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xiec2yEDs6n1YFQvwMPIAYfR8lc=', 'Content-Length': '0'} 2022-09-02T05:53:18.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4acb5fb9b101a4e-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.165 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.165 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.166 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.167 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.168 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.169 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.170 DEBUG:boto:Data: 2022-09-02T05:53:18.171 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.173 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.174 DEBUG:boto:Params: {} 2022-09-02T05:53:18.175 DEBUG:boto:Token: None 2022-09-02T05:53:18.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xiec2yEDs6n1YFQvwMPIAYfR8lc= 2022-09-02T05:53:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xiec2yEDs6n1YFQvwMPIAYfR8lc=', 'Content-Length': '0'} 2022-09-02T05:53:18.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b2b0d8ce29f3c8-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:18.183 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.184 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.185 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.186 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.187 DEBUG:boto:Method: GET 2022-09-02T05:53:18.188 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.189 DEBUG:boto:Data: 2022-09-02T05:53:18.190 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.192 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.193 DEBUG:boto:Params: {} 2022-09-02T05:53:18.194 DEBUG:boto:Token: None 2022-09-02T05:53:18.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4= 2022-09-02T05:53:18.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4=', 'Content-Length': '0'} 2022-09-02T05:53:18.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c6a2a9deaa1120-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.201 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.202 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.203 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.204 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.205 DEBUG:boto:Method: GET 2022-09-02T05:53:18.206 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.207 DEBUG:boto:Data: 2022-09-02T05:53:18.208 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.210 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.211 DEBUG:boto:Params: {} 2022-09-02T05:53:18.212 DEBUG:boto:Token: None 2022-09-02T05:53:18.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:53:18.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4= 2022-09-02T05:53:18.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wD+x5naji+JSXKf0WKD+ePud8W4=', 'Content-Length': '0'} 2022-09-02T05:53:18.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e44876731351cb-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.218 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.219 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.219 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.220 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.220 DEBUG:boto:Method: GET 2022-09-02T05:53:18.221 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.221 DEBUG:boto:Data: 2022-09-02T05:53:18.222 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.223 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.223 DEBUG:boto:Params: {} 2022-09-02T05:53:18.224 DEBUG:boto:Token: None 2022-09-02T05:53:18.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fc6c64d66ef5d25-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.229 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.230 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.231 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.231 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.232 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.232 DEBUG:boto:Method: GET 2022-09-02T05:53:18.233 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.233 DEBUG:boto:Data: 2022-09-02T05:53:18.234 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.235 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.235 DEBUG:boto:Params: {} 2022-09-02T05:53:18.236 DEBUG:boto:Token: None 2022-09-02T05:53:18.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d21824f1179a61-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.241 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.242 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.242 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.243 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.243 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:18.244 DEBUG:boto:Data: 2022-09-02T05:53:18.244 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.246 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.246 DEBUG:boto:Params: {} 2022-09-02T05:53:18.247 DEBUG:boto:Token: None 2022-09-02T05:53:18.247 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22 2022-09-02T05:53:18.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ntPQAbgVTwDBT2IqpTS7DUyZ4QM= 2022-09-02T05:53:18.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ntPQAbgVTwDBT2IqpTS7DUyZ4QM=', 'Content-Length': '0'} 2022-09-02T05:53:18.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb96e274efbe65c-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.251 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.252 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.252 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.253 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:18.253 DEBUG:boto:Data: 2022-09-02T05:53:18.254 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.254 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.255 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.255 DEBUG:boto:Params: {} 2022-09-02T05:53:18.256 DEBUG:boto:Token: None 2022-09-02T05:53:18.256 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22 2022-09-02T05:53:18.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ntPQAbgVTwDBT2IqpTS7DUyZ4QM= 2022-09-02T05:53:18.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ntPQAbgVTwDBT2IqpTS7DUyZ4QM=', 'Content-Length': '0'} 2022-09-02T05:53:18.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632d212343022b16-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:53:18.261 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.261 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.262 DEBUG:boto:Method: GET 2022-09-02T05:53:18.262 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:18.263 DEBUG:boto:Data: 2022-09-02T05:53:18.263 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.264 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.265 DEBUG:boto:Params: {} 2022-09-02T05:53:18.265 DEBUG:boto:Token: None 2022-09-02T05:53:18.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22 2022-09-02T05:53:18.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RIxlHV/+bvBo9sKP5FGPy9ueFT4= 2022-09-02T05:53:18.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RIxlHV/+bvBo9sKP5FGPy9ueFT4=', 'Content-Length': '0'} 2022-09-02T05:53:18.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3b69b65a4865ab-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.310 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:53:18.311 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:53:18.311 DEBUG:boto:Method: GET 2022-09-02T05:53:18.312 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:53:18.312 DEBUG:boto:Data: 2022-09-02T05:53:18.313 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.313 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.314 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.314 DEBUG:boto:Params: {} 2022-09-02T05:53:18.315 DEBUG:boto:Token: None 2022-09-02T05:53:18.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj22 2022-09-02T05:53:18.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RIxlHV/+bvBo9sKP5FGPy9ueFT4= 2022-09-02T05:53:18.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RIxlHV/+bvBo9sKP5FGPy9ueFT4=', 'Content-Length': '0'} 2022-09-02T05:53:18.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001899a145b9efcb1e-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.320 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.321 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.321 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.322 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.322 DEBUG:boto:Method: GET 2022-09-02T05:53:18.323 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.323 DEBUG:boto:Data: 2022-09-02T05:53:18.324 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.325 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.325 DEBUG:boto:Params: {} 2022-09-02T05:53:18.326 DEBUG:boto:Token: None 2022-09-02T05:53:18.326 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000366fe276fd3b1688-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.331 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.332 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.333 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.333 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.334 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.334 DEBUG:boto:Method: GET 2022-09-02T05:53:18.335 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:53:18.335 DEBUG:boto:Data: 2022-09-02T05:53:18.336 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.336 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.337 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.337 DEBUG:boto:Params: {} 2022-09-02T05:53:18.338 DEBUG:boto:Token: None 2022-09-02T05:53:18.339 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e749ceefa03e1f1b-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.343 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:18.344 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.345 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.346 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.346 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.347 DEBUG:boto:Method: GET 2022-09-02T05:53:18.347 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.348 DEBUG:boto:Data: 2022-09-02T05:53:18.348 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.349 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.350 DEBUG:boto:Params: {} 2022-09-02T05:53:18.350 DEBUG:boto:Token: None 2022-09-02T05:53:18.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40= 2022-09-02T05:53:18.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40=', 'Content-Length': '0'} 2022-09-02T05:53:18.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d51cbe934fe8241-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.356 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.356 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.357 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.357 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.358 DEBUG:boto:Method: GET 2022-09-02T05:53:18.358 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.359 DEBUG:boto:Data: 2022-09-02T05:53:18.359 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.360 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.361 DEBUG:boto:Params: {} 2022-09-02T05:53:18.362 DEBUG:boto:Token: None 2022-09-02T05:53:18.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40= 2022-09-02T05:53:18.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40=', 'Content-Length': '0'} 2022-09-02T05:53:18.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba8340511410a3e-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.366 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.367 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.367 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.368 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.368 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.369 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.369 DEBUG:boto:Data: 2022-09-02T05:53:18.370 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.371 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.371 DEBUG:boto:Params: {} 2022-09-02T05:53:18.372 DEBUG:boto:Token: None 2022-09-02T05:53:18.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sxeMPXZI+6869qv9uGzNpfom+oc= 2022-09-02T05:53:18.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sxeMPXZI+6869qv9uGzNpfom+oc=', 'Content-Length': '0'} 2022-09-02T05:53:18.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004295074801cf3db4-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.378 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.378 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.379 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.379 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.380 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.380 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.381 DEBUG:boto:Data: 2022-09-02T05:53:18.381 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.382 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.383 DEBUG:boto:Params: {} 2022-09-02T05:53:18.383 DEBUG:boto:Token: None 2022-09-02T05:53:18.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sxeMPXZI+6869qv9uGzNpfom+oc= 2022-09-02T05:53:18.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sxeMPXZI+6869qv9uGzNpfom+oc=', 'Content-Length': '0'} 2022-09-02T05:53:18.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c645b984fe244d3-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:18.389 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.389 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.390 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.390 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.391 DEBUG:boto:Method: GET 2022-09-02T05:53:18.392 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.392 DEBUG:boto:Data: 2022-09-02T05:53:18.393 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.395 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.396 DEBUG:boto:Params: {} 2022-09-02T05:53:18.397 DEBUG:boto:Token: None 2022-09-02T05:53:18.397 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40= 2022-09-02T05:53:18.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40=', 'Content-Length': '0'} 2022-09-02T05:53:18.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea35adb4ab6b560c-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.446 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.447 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.448 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.449 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.450 DEBUG:boto:Method: GET 2022-09-02T05:53:18.451 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.452 DEBUG:boto:Data: 2022-09-02T05:53:18.453 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.455 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.456 DEBUG:boto:Params: {} 2022-09-02T05:53:18.457 DEBUG:boto:Token: None 2022-09-02T05:53:18.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:53:18.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40= 2022-09-02T05:53:18.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:P31SZoZVJ6qGNJCphWiiTC5Xv40=', 'Content-Length': '0'} 2022-09-02T05:53:18.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20c504164a3cd51-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.465 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.466 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.467 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.468 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.469 DEBUG:boto:Method: GET 2022-09-02T05:53:18.470 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.471 DEBUG:boto:Data: 2022-09-02T05:53:18.472 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.474 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.475 DEBUG:boto:Params: {} 2022-09-02T05:53:18.477 DEBUG:boto:Token: None 2022-09-02T05:53:18.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215a1bc140406e4b-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.484 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.486 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.487 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.488 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.489 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.490 DEBUG:boto:Method: GET 2022-09-02T05:53:18.491 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.492 DEBUG:boto:Data: 2022-09-02T05:53:18.493 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.495 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.496 DEBUG:boto:Params: {} 2022-09-02T05:53:18.497 DEBUG:boto:Token: None 2022-09-02T05:53:18.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a91f6a007655a04f-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.505 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.506 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.506 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.507 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.507 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:18.508 DEBUG:boto:Data: 2022-09-02T05:53:18.508 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.510 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.510 DEBUG:boto:Params: {} 2022-09-02T05:53:18.511 DEBUG:boto:Token: None 2022-09-02T05:53:18.512 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23 2022-09-02T05:53:18.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pH1F0J+tcD0K2IM+0YCI2fC7y6I= 2022-09-02T05:53:18.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pH1F0J+tcD0K2IM+0YCI2fC7y6I=', 'Content-Length': '0'} 2022-09-02T05:53:18.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3488a04519bac6c-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.518 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.518 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.519 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.520 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:18.521 DEBUG:boto:Data: 2022-09-02T05:53:18.522 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.523 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.524 DEBUG:boto:Params: {} 2022-09-02T05:53:18.525 DEBUG:boto:Token: None 2022-09-02T05:53:18.525 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23 2022-09-02T05:53:18.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pH1F0J+tcD0K2IM+0YCI2fC7y6I= 2022-09-02T05:53:18.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pH1F0J+tcD0K2IM+0YCI2fC7y6I=', 'Content-Length': '0'} 2022-09-02T05:53:18.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b46503b5d7ae13-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:53:18.530 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.531 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.531 DEBUG:boto:Method: GET 2022-09-02T05:53:18.532 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:18.533 DEBUG:boto:Data: 2022-09-02T05:53:18.533 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.534 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.535 DEBUG:boto:Params: {} 2022-09-02T05:53:18.535 DEBUG:boto:Token: None 2022-09-02T05:53:18.536 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23 2022-09-02T05:53:18.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kp+gDuhsd/J9OA1DPf0DkpAZbFo= 2022-09-02T05:53:18.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kp+gDuhsd/J9OA1DPf0DkpAZbFo=', 'Content-Length': '0'} 2022-09-02T05:53:18.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ad91f8bddfff8f-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.540 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:53:18.541 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:53:18.541 DEBUG:boto:Method: GET 2022-09-02T05:53:18.542 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:53:18.543 DEBUG:boto:Data: 2022-09-02T05:53:18.544 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.545 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.546 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.547 DEBUG:boto:Params: {} 2022-09-02T05:53:18.547 DEBUG:boto:Token: None 2022-09-02T05:53:18.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj23 2022-09-02T05:53:18.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kp+gDuhsd/J9OA1DPf0DkpAZbFo= 2022-09-02T05:53:18.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kp+gDuhsd/J9OA1DPf0DkpAZbFo=', 'Content-Length': '0'} 2022-09-02T05:53:18.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb89ccfb99de06a-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.594 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.595 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.596 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.596 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.597 DEBUG:boto:Method: GET 2022-09-02T05:53:18.598 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.598 DEBUG:boto:Data: 2022-09-02T05:53:18.599 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.600 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.600 DEBUG:boto:Params: {} 2022-09-02T05:53:18.601 DEBUG:boto:Token: None 2022-09-02T05:53:18.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dafeaf55a164e09-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.607 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.607 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.608 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.609 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.609 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.610 DEBUG:boto:Method: GET 2022-09-02T05:53:18.610 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:53:18.611 DEBUG:boto:Data: 2022-09-02T05:53:18.611 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.613 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.613 DEBUG:boto:Params: {} 2022-09-02T05:53:18.614 DEBUG:boto:Token: None 2022-09-02T05:53:18.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdbc215c1d77d0a4-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.619 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:18.621 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.622 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.623 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.623 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.624 DEBUG:boto:Method: GET 2022-09-02T05:53:18.624 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.625 DEBUG:boto:Data: 2022-09-02T05:53:18.625 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.626 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.627 DEBUG:boto:Params: {} 2022-09-02T05:53:18.628 DEBUG:boto:Token: None 2022-09-02T05:53:18.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo= 2022-09-02T05:53:18.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo=', 'Content-Length': '0'} 2022-09-02T05:53:18.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecad1214fa4bbdf9-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.674 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.675 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.676 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.677 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.678 DEBUG:boto:Method: GET 2022-09-02T05:53:18.679 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.680 DEBUG:boto:Data: 2022-09-02T05:53:18.681 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.683 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.684 DEBUG:boto:Params: {} 2022-09-02T05:53:18.685 DEBUG:boto:Token: None 2022-09-02T05:53:18.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo= 2022-09-02T05:53:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo=', 'Content-Length': '0'} 2022-09-02T05:53:18.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cd89635ab8adcf-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.691 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.692 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.692 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.693 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.693 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.694 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.694 DEBUG:boto:Data: 2022-09-02T05:53:18.695 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.696 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.696 DEBUG:boto:Params: {} 2022-09-02T05:53:18.697 DEBUG:boto:Token: None 2022-09-02T05:53:18.698 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7N0oQvu2wjdvcHowXCpnDtQT5gA= 2022-09-02T05:53:18.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7N0oQvu2wjdvcHowXCpnDtQT5gA=', 'Content-Length': '0'} 2022-09-02T05:53:18.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b0a923559e05369-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.702 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.702 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.703 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.703 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.704 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.704 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.705 DEBUG:boto:Data: 2022-09-02T05:53:18.706 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.707 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.707 DEBUG:boto:Params: {} 2022-09-02T05:53:18.708 DEBUG:boto:Token: None 2022-09-02T05:53:18.709 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7N0oQvu2wjdvcHowXCpnDtQT5gA= 2022-09-02T05:53:18.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7N0oQvu2wjdvcHowXCpnDtQT5gA=', 'Content-Length': '0'} 2022-09-02T05:53:18.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c50ac2df76eb495-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:18.713 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.714 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.714 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.715 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.715 DEBUG:boto:Method: GET 2022-09-02T05:53:18.716 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.717 DEBUG:boto:Data: 2022-09-02T05:53:18.717 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.718 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.719 DEBUG:boto:Params: {} 2022-09-02T05:53:18.719 DEBUG:boto:Token: None 2022-09-02T05:53:18.720 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo= 2022-09-02T05:53:18.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo=', 'Content-Length': '0'} 2022-09-02T05:53:18.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c0e61f7784b5d10-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.725 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.726 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.726 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.727 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.728 DEBUG:boto:Method: GET 2022-09-02T05:53:18.728 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.729 DEBUG:boto:Data: 2022-09-02T05:53:18.730 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.731 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.732 DEBUG:boto:Params: {} 2022-09-02T05:53:18.733 DEBUG:boto:Token: None 2022-09-02T05:53:18.734 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:53:18.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo= 2022-09-02T05:53:18.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QJPSFDAjoRFEB6/Mzn2TVCUgzCo=', 'Content-Length': '0'} 2022-09-02T05:53:18.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d5dbd78152806a-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.739 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.739 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.740 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.742 DEBUG:boto:Method: GET 2022-09-02T05:53:18.743 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.744 DEBUG:boto:Data: 2022-09-02T05:53:18.744 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.746 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.746 DEBUG:boto:Params: {} 2022-09-02T05:53:18.747 DEBUG:boto:Token: None 2022-09-02T05:53:18.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082478f524708b4e0-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.753 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.754 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.755 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.756 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.756 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.757 DEBUG:boto:Method: GET 2022-09-02T05:53:18.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.758 DEBUG:boto:Data: 2022-09-02T05:53:18.759 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.760 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.761 DEBUG:boto:Params: {} 2022-09-02T05:53:18.762 DEBUG:boto:Token: None 2022-09-02T05:53:18.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d89a48b78fd52c5-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.768 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.769 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.769 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.770 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.770 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:18.771 DEBUG:boto:Data: 2022-09-02T05:53:18.772 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.773 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.774 DEBUG:boto:Params: {} 2022-09-02T05:53:18.775 DEBUG:boto:Token: None 2022-09-02T05:53:18.776 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33 2022-09-02T05:53:18.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/AhXj/ISAm0x6XY+j3ga2Ie2lgI= 2022-09-02T05:53:18.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/AhXj/ISAm0x6XY+j3ga2Ie2lgI=', 'Content-Length': '0'} 2022-09-02T05:53:18.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cbd326b9f804389-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.780 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.781 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.782 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.782 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:18.783 DEBUG:boto:Data: 2022-09-02T05:53:18.784 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.785 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.786 DEBUG:boto:Params: {} 2022-09-02T05:53:18.787 DEBUG:boto:Token: None 2022-09-02T05:53:18.788 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33 2022-09-02T05:53:18.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/AhXj/ISAm0x6XY+j3ga2Ie2lgI= 2022-09-02T05:53:18.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/AhXj/ISAm0x6XY+j3ga2Ie2lgI=', 'Content-Length': '0'} 2022-09-02T05:53:18.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f2db0d124e5657-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:53:18.794 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.796 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.797 DEBUG:boto:Method: GET 2022-09-02T05:53:18.798 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:18.799 DEBUG:boto:Data: 2022-09-02T05:53:18.801 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.803 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.804 DEBUG:boto:Params: {} 2022-09-02T05:53:18.806 DEBUG:boto:Token: None 2022-09-02T05:53:18.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33 2022-09-02T05:53:18.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYDMRXu76ss2v5309Fp5Iqvf8pg= 2022-09-02T05:53:18.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYDMRXu76ss2v5309Fp5Iqvf8pg=', 'Content-Length': '0'} 2022-09-02T05:53:18.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007163670824769cdc-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.815 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:53:18.817 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:53:18.818 DEBUG:boto:Method: GET 2022-09-02T05:53:18.820 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:53:18.822 DEBUG:boto:Data: 2022-09-02T05:53:18.823 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.826 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.828 DEBUG:boto:Params: {} 2022-09-02T05:53:18.829 DEBUG:boto:Token: None 2022-09-02T05:53:18.830 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj33 2022-09-02T05:53:18.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYDMRXu76ss2v5309Fp5Iqvf8pg= 2022-09-02T05:53:18.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYDMRXu76ss2v5309Fp5Iqvf8pg=', 'Content-Length': '0'} 2022-09-02T05:53:18.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749ac1fbe601c6b1-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.835 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.836 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.837 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.837 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.838 DEBUG:boto:Method: GET 2022-09-02T05:53:18.840 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.841 DEBUG:boto:Data: 2022-09-02T05:53:18.843 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.846 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.848 DEBUG:boto:Params: {} 2022-09-02T05:53:18.849 DEBUG:boto:Token: None 2022-09-02T05:53:18.851 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b422b1ffd6d927-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.860 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.861 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.863 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.864 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.865 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.866 DEBUG:boto:Method: GET 2022-09-02T05:53:18.867 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:53:18.868 DEBUG:boto:Data: 2022-09-02T05:53:18.871 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.872 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.873 DEBUG:boto:Params: {} 2022-09-02T05:53:18.874 DEBUG:boto:Token: None 2022-09-02T05:53:18.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/ 2022-09-02T05:53:18.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU= 2022-09-02T05:53:18.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxWqlXMn0vYOGgDzjHcdBdtgFU=', 'Content-Length': '0'} 2022-09-02T05:53:18.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004578281811412cca-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.891 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:18.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:18.894 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:18.896 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:18.898 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.899 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.900 DEBUG:boto:Method: GET 2022-09-02T05:53:18.902 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.903 DEBUG:boto:Data: 2022-09-02T05:53:18.904 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.907 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.908 DEBUG:boto:Params: {} 2022-09-02T05:53:18.910 DEBUG:boto:Token: None 2022-09-02T05:53:18.911 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8= 2022-09-02T05:53:18.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8=', 'Content-Length': '0'} 2022-09-02T05:53:18.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002732d1b255ff5ef4-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.919 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:18.921 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:18.922 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.923 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.925 DEBUG:boto:Method: GET 2022-09-02T05:53:18.926 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.928 DEBUG:boto:Data: 2022-09-02T05:53:18.930 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.933 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.935 DEBUG:boto:Params: {} 2022-09-02T05:53:18.936 DEBUG:boto:Token: None 2022-09-02T05:53:18.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8= 2022-09-02T05:53:18.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8=', 'Content-Length': '0'} 2022-09-02T05:53:18.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ebdd80868775ea-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.946 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:18.947 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:18.948 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.949 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.951 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.952 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.953 DEBUG:boto:Data: 2022-09-02T05:53:18.954 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.956 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.957 DEBUG:boto:Params: {} 2022-09-02T05:53:18.958 DEBUG:boto:Token: None 2022-09-02T05:53:18.959 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9DywPgI1cAaNzP92FUOzV6JF/+c= 2022-09-02T05:53:18.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9DywPgI1cAaNzP92FUOzV6JF/+c=', 'Content-Length': '0'} 2022-09-02T05:53:18.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001848b9e13c247a3e-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.963 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:18.963 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:18.964 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.964 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.965 DEBUG:boto:Method: HEAD 2022-09-02T05:53:18.965 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.966 DEBUG:boto:Data: 2022-09-02T05:53:18.966 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.967 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.968 DEBUG:boto:Params: {} 2022-09-02T05:53:18.968 DEBUG:boto:Token: None 2022-09-02T05:53:18.969 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9DywPgI1cAaNzP92FUOzV6JF/+c= 2022-09-02T05:53:18.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9DywPgI1cAaNzP92FUOzV6JF/+c=', 'Content-Length': '0'} 2022-09-02T05:53:18.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb70dfd3912d4ecd-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:18.974 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:18.974 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:18.975 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.975 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.976 DEBUG:boto:Method: GET 2022-09-02T05:53:18.976 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.977 DEBUG:boto:Data: 2022-09-02T05:53:18.977 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:18.978 DEBUG:boto:Port: 8001 2022-09-02T05:53:18.979 DEBUG:boto:Params: {} 2022-09-02T05:53:18.980 DEBUG:boto:Token: None 2022-09-02T05:53:18.980 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8= 2022-09-02T05:53:18.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8=', 'Content-Length': '0'} 2022-09-02T05:53:18.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a15936074a84dfb-0063119a4e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.985 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:18.986 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:18.986 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.987 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.987 DEBUG:boto:Method: GET 2022-09-02T05:53:18.988 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.988 DEBUG:boto:Data: 2022-09-02T05:53:18.989 DEBUG:boto:Headers: {} 2022-09-02T05:53:18.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:18.990 DEBUG:boto:Port: 8000 2022-09-02T05:53:18.991 DEBUG:boto:Params: {} 2022-09-02T05:53:18.992 DEBUG:boto:Token: None 2022-09-02T05:53:18.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:18 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:53:18.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8= 2022-09-02T05:53:18.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:18 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyZhTeyXG07snb4ci/RXsedNJL8=', 'Content-Length': '0'} 2022-09-02T05:53:18.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001098a77b69d4eedf-0063119a4e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:18 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:18.997 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:18.998 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:18.998 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:18.999 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:18.999 DEBUG:boto:Method: GET 2022-09-02T05:53:19.000 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.000 DEBUG:boto:Data: 2022-09-02T05:53:19.001 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.002 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.002 DEBUG:boto:Params: {} 2022-09-02T05:53:19.003 DEBUG:boto:Token: None 2022-09-02T05:53:19.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe1c50537184688-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.008 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.009 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.010 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.010 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.011 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.011 DEBUG:boto:Method: GET 2022-09-02T05:53:19.012 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.013 DEBUG:boto:Data: 2022-09-02T05:53:19.013 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.014 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.015 DEBUG:boto:Params: {} 2022-09-02T05:53:19.015 DEBUG:boto:Token: None 2022-09-02T05:53:19.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfca915cd8f411ec-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.021 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.021 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:19.022 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:19.023 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.023 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:19.024 DEBUG:boto:Data: 2022-09-02T05:53:19.024 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.025 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.026 DEBUG:boto:Params: {} 2022-09-02T05:53:19.026 DEBUG:boto:Token: None 2022-09-02T05:53:19.027 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj34 2022-09-02T05:53:19.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Omyh++O8rOy+qAcIKDLHQcAAek0= 2022-09-02T05:53:19.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Omyh++O8rOy+qAcIKDLHQcAAek0=', 'Content-Length': '0'} 2022-09-02T05:53:19.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d49aee3abcb9999-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.031 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:19.032 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:19.033 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.033 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:19.034 DEBUG:boto:Data: 2022-09-02T05:53:19.035 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.037 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.038 DEBUG:boto:Params: {} 2022-09-02T05:53:19.038 DEBUG:boto:Token: None 2022-09-02T05:53:19.039 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj34 2022-09-02T05:53:19.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Omyh++O8rOy+qAcIKDLHQcAAek0= 2022-09-02T05:53:19.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Omyh++O8rOy+qAcIKDLHQcAAek0=', 'Content-Length': '0'} 2022-09-02T05:53:19.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80707f8ab1f30df-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:53:19.045 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:19.045 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:19.046 DEBUG:boto:Method: GET 2022-09-02T05:53:19.047 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:19.047 DEBUG:boto:Data: 2022-09-02T05:53:19.048 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.049 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.049 DEBUG:boto:Params: {} 2022-09-02T05:53:19.051 DEBUG:boto:Token: None 2022-09-02T05:53:19.052 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj34 2022-09-02T05:53:19.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5gr91GNS5v/mVbYI7pkHAj1xtVk= 2022-09-02T05:53:19.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5gr91GNS5v/mVbYI7pkHAj1xtVk=', 'Content-Length': '0'} 2022-09-02T05:53:19.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8db59d78e63f92f-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.098 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:53:19.099 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:53:19.099 DEBUG:boto:Method: GET 2022-09-02T05:53:19.100 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:53:19.101 DEBUG:boto:Data: 2022-09-02T05:53:19.101 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.102 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.103 DEBUG:boto:Params: {} 2022-09-02T05:53:19.104 DEBUG:boto:Token: None 2022-09-02T05:53:19.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj34 2022-09-02T05:53:19.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5gr91GNS5v/mVbYI7pkHAj1xtVk= 2022-09-02T05:53:19.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5gr91GNS5v/mVbYI7pkHAj1xtVk=', 'Content-Length': '0'} 2022-09-02T05:53:19.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006812bda52c77f3b0-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.109 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.110 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.110 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.111 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.111 DEBUG:boto:Method: GET 2022-09-02T05:53:19.112 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.112 DEBUG:boto:Data: 2022-09-02T05:53:19.113 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.114 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.115 DEBUG:boto:Params: {} 2022-09-02T05:53:19.115 DEBUG:boto:Token: None 2022-09-02T05:53:19.116 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002428e40d60853b24-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.121 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.121 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.122 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.123 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.123 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.124 DEBUG:boto:Method: GET 2022-09-02T05:53:19.124 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:53:19.125 DEBUG:boto:Data: 2022-09-02T05:53:19.125 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.126 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.127 DEBUG:boto:Params: {} 2022-09-02T05:53:19.128 DEBUG:boto:Token: None 2022-09-02T05:53:19.128 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b018289943cb73e-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.133 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:19.134 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.135 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.135 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.136 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.137 DEBUG:boto:Method: GET 2022-09-02T05:53:19.138 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.138 DEBUG:boto:Data: 2022-09-02T05:53:19.139 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.140 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.140 DEBUG:boto:Params: {} 2022-09-02T05:53:19.141 DEBUG:boto:Token: None 2022-09-02T05:53:19.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w= 2022-09-02T05:53:19.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w=', 'Content-Length': '0'} 2022-09-02T05:53:19.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9b5649360e9cbb-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.146 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.146 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.147 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.147 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.148 DEBUG:boto:Method: GET 2022-09-02T05:53:19.148 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.149 DEBUG:boto:Data: 2022-09-02T05:53:19.150 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.150 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.151 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.152 DEBUG:boto:Params: {} 2022-09-02T05:53:19.153 DEBUG:boto:Token: None 2022-09-02T05:53:19.153 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w= 2022-09-02T05:53:19.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w=', 'Content-Length': '0'} 2022-09-02T05:53:19.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a583c93c4bb4e5-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.158 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.159 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.160 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.161 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.162 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.163 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.164 DEBUG:boto:Data: 2022-09-02T05:53:19.165 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.167 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.168 DEBUG:boto:Params: {} 2022-09-02T05:53:19.169 DEBUG:boto:Token: None 2022-09-02T05:53:19.170 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ctRMq0YKmMihUFxmaZmdT+okiao= 2022-09-02T05:53:19.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ctRMq0YKmMihUFxmaZmdT+okiao=', 'Content-Length': '0'} 2022-09-02T05:53:19.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000495f689e081e6987-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.176 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.177 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.178 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.179 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.180 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.181 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.181 DEBUG:boto:Data: 2022-09-02T05:53:19.182 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.183 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.183 DEBUG:boto:Params: {} 2022-09-02T05:53:19.184 DEBUG:boto:Token: None 2022-09-02T05:53:19.185 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ctRMq0YKmMihUFxmaZmdT+okiao= 2022-09-02T05:53:19.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ctRMq0YKmMihUFxmaZmdT+okiao=', 'Content-Length': '0'} 2022-09-02T05:53:19.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3296268778e21e-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:19.190 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.190 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.191 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.191 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.192 DEBUG:boto:Method: GET 2022-09-02T05:53:19.193 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.193 DEBUG:boto:Data: 2022-09-02T05:53:19.194 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.195 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.195 DEBUG:boto:Params: {} 2022-09-02T05:53:19.196 DEBUG:boto:Token: None 2022-09-02T05:53:19.196 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w= 2022-09-02T05:53:19.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w=', 'Content-Length': '0'} 2022-09-02T05:53:19.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008514d550212fed2c-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.201 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.202 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.202 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.203 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.203 DEBUG:boto:Method: GET 2022-09-02T05:53:19.204 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.205 DEBUG:boto:Data: 2022-09-02T05:53:19.205 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.206 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.207 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.207 DEBUG:boto:Params: {} 2022-09-02T05:53:19.208 DEBUG:boto:Token: None 2022-09-02T05:53:19.208 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:53:19.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w= 2022-09-02T05:53:19.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hftb+ONCUdpg834c1ee8hsWfP9w=', 'Content-Length': '0'} 2022-09-02T05:53:19.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063b16ebb9cf5a333-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.212 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.213 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.213 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.214 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.215 DEBUG:boto:Method: GET 2022-09-02T05:53:19.215 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.216 DEBUG:boto:Data: 2022-09-02T05:53:19.216 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.217 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.218 DEBUG:boto:Params: {} 2022-09-02T05:53:19.219 DEBUG:boto:Token: None 2022-09-02T05:53:19.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb96dc926c7e8e0c-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.224 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.225 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.225 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.226 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.227 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.227 DEBUG:boto:Method: GET 2022-09-02T05:53:19.228 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.228 DEBUG:boto:Data: 2022-09-02T05:53:19.229 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.229 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.230 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.231 DEBUG:boto:Params: {} 2022-09-02T05:53:19.231 DEBUG:boto:Token: None 2022-09-02T05:53:19.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036ed10862ffc5604-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.237 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.237 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.238 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.238 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.239 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:19.239 DEBUG:boto:Data: 2022-09-02T05:53:19.240 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.241 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.242 DEBUG:boto:Params: {} 2022-09-02T05:53:19.242 DEBUG:boto:Token: None 2022-09-02T05:53:19.243 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38 2022-09-02T05:53:19.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GCLXC6+GhIrvQCtxeIBO1OTeIrg= 2022-09-02T05:53:19.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GCLXC6+GhIrvQCtxeIBO1OTeIrg=', 'Content-Length': '0'} 2022-09-02T05:53:19.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b06fe8a4459fc2-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.247 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.247 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.248 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.248 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:19.249 DEBUG:boto:Data: 2022-09-02T05:53:19.249 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.250 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.251 DEBUG:boto:Params: {} 2022-09-02T05:53:19.252 DEBUG:boto:Token: None 2022-09-02T05:53:19.252 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38 2022-09-02T05:53:19.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GCLXC6+GhIrvQCtxeIBO1OTeIrg= 2022-09-02T05:53:19.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GCLXC6+GhIrvQCtxeIBO1OTeIrg=', 'Content-Length': '0'} 2022-09-02T05:53:19.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d02503b11bc51f-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:53:19.256 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.257 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.257 DEBUG:boto:Method: GET 2022-09-02T05:53:19.258 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:19.258 DEBUG:boto:Data: 2022-09-02T05:53:19.259 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.260 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.260 DEBUG:boto:Params: {} 2022-09-02T05:53:19.261 DEBUG:boto:Token: None 2022-09-02T05:53:19.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38 2022-09-02T05:53:19.262 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rb8Q6n+3mnv8j6og26Jqf4dOmNA= 2022-09-02T05:53:19.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rb8Q6n+3mnv8j6og26Jqf4dOmNA=', 'Content-Length': '0'} 2022-09-02T05:53:19.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bc67df499d525f6-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.306 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:53:19.307 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:53:19.307 DEBUG:boto:Method: GET 2022-09-02T05:53:19.308 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:53:19.309 DEBUG:boto:Data: 2022-09-02T05:53:19.310 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.311 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.312 DEBUG:boto:Params: {} 2022-09-02T05:53:19.313 DEBUG:boto:Token: None 2022-09-02T05:53:19.313 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj38 2022-09-02T05:53:19.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rb8Q6n+3mnv8j6og26Jqf4dOmNA= 2022-09-02T05:53:19.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rb8Q6n+3mnv8j6og26Jqf4dOmNA=', 'Content-Length': '0'} 2022-09-02T05:53:19.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4aee67f1aa8bde-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.318 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.318 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.319 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.319 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.320 DEBUG:boto:Method: GET 2022-09-02T05:53:19.321 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.321 DEBUG:boto:Data: 2022-09-02T05:53:19.322 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.323 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.323 DEBUG:boto:Params: {} 2022-09-02T05:53:19.324 DEBUG:boto:Token: None 2022-09-02T05:53:19.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f81e94f55f1e3eb-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.330 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.331 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.331 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.332 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.332 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.333 DEBUG:boto:Method: GET 2022-09-02T05:53:19.334 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:53:19.334 DEBUG:boto:Data: 2022-09-02T05:53:19.335 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.336 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.336 DEBUG:boto:Params: {} 2022-09-02T05:53:19.337 DEBUG:boto:Token: None 2022-09-02T05:53:19.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087a9f40b2f0547d-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.343 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:19.344 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.345 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.345 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.346 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.346 DEBUG:boto:Method: GET 2022-09-02T05:53:19.347 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.347 DEBUG:boto:Data: 2022-09-02T05:53:19.348 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.349 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.349 DEBUG:boto:Params: {} 2022-09-02T05:53:19.350 DEBUG:boto:Token: None 2022-09-02T05:53:19.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo= 2022-09-02T05:53:19.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo=', 'Content-Length': '0'} 2022-09-02T05:53:19.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf083eb6107ca775-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.355 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.356 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.356 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.357 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.357 DEBUG:boto:Method: GET 2022-09-02T05:53:19.358 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.358 DEBUG:boto:Data: 2022-09-02T05:53:19.359 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.361 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.361 DEBUG:boto:Params: {} 2022-09-02T05:53:19.362 DEBUG:boto:Token: None 2022-09-02T05:53:19.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo= 2022-09-02T05:53:19.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo=', 'Content-Length': '0'} 2022-09-02T05:53:19.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a08471f82f0606-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.367 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.367 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.368 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.368 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.369 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.369 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.370 DEBUG:boto:Data: 2022-09-02T05:53:19.371 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.372 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.372 DEBUG:boto:Params: {} 2022-09-02T05:53:19.373 DEBUG:boto:Token: None 2022-09-02T05:53:19.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2pfE5+YUQJWhCJ1P9Ap8LcbcOt0= 2022-09-02T05:53:19.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2pfE5+YUQJWhCJ1P9Ap8LcbcOt0=', 'Content-Length': '0'} 2022-09-02T05:53:19.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f872f635a53c9e97-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.378 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.378 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.379 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.379 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.380 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.380 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.381 DEBUG:boto:Data: 2022-09-02T05:53:19.381 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.382 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.383 DEBUG:boto:Params: {} 2022-09-02T05:53:19.383 DEBUG:boto:Token: None 2022-09-02T05:53:19.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2pfE5+YUQJWhCJ1P9Ap8LcbcOt0= 2022-09-02T05:53:19.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2pfE5+YUQJWhCJ1P9Ap8LcbcOt0=', 'Content-Length': '0'} 2022-09-02T05:53:19.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7fe58ecd709c56d-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:19.389 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.389 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.390 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.391 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.391 DEBUG:boto:Method: GET 2022-09-02T05:53:19.392 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.392 DEBUG:boto:Data: 2022-09-02T05:53:19.393 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.394 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.395 DEBUG:boto:Params: {} 2022-09-02T05:53:19.395 DEBUG:boto:Token: None 2022-09-02T05:53:19.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo= 2022-09-02T05:53:19.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo=', 'Content-Length': '0'} 2022-09-02T05:53:19.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af1fdeaeee2cac68-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.401 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.401 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.402 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.402 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.403 DEBUG:boto:Method: GET 2022-09-02T05:53:19.404 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.404 DEBUG:boto:Data: 2022-09-02T05:53:19.405 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.406 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.406 DEBUG:boto:Params: {} 2022-09-02T05:53:19.407 DEBUG:boto:Token: None 2022-09-02T05:53:19.408 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:53:19.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo= 2022-09-02T05:53:19.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SATsGc6H3oN7fsMt515CTdofiWo=', 'Content-Length': '0'} 2022-09-02T05:53:19.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a6e0d4714b1a67-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.412 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.413 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.414 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.414 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.415 DEBUG:boto:Method: GET 2022-09-02T05:53:19.415 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.416 DEBUG:boto:Data: 2022-09-02T05:53:19.416 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.417 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.418 DEBUG:boto:Params: {} 2022-09-02T05:53:19.419 DEBUG:boto:Token: None 2022-09-02T05:53:19.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007624b56c6f74b84a-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.424 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.425 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.425 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.426 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.426 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.427 DEBUG:boto:Method: GET 2022-09-02T05:53:19.427 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.428 DEBUG:boto:Data: 2022-09-02T05:53:19.428 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.429 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.429 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.430 DEBUG:boto:Params: {} 2022-09-02T05:53:19.431 DEBUG:boto:Token: None 2022-09-02T05:53:19.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ad535673c214b6-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.436 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.437 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.438 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.438 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.439 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:19.440 DEBUG:boto:Data: 2022-09-02T05:53:19.440 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.441 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.442 DEBUG:boto:Params: {} 2022-09-02T05:53:19.442 DEBUG:boto:Token: None 2022-09-02T05:53:19.443 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44 2022-09-02T05:53:19.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zbnhrsw+2aaoP3+u0QA8Wgk0R3A= 2022-09-02T05:53:19.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zbnhrsw+2aaoP3+u0QA8Wgk0R3A=', 'Content-Length': '0'} 2022-09-02T05:53:19.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9402b0962951874-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.447 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.447 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.448 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.448 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:19.449 DEBUG:boto:Data: 2022-09-02T05:53:19.450 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.450 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.451 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.451 DEBUG:boto:Params: {} 2022-09-02T05:53:19.452 DEBUG:boto:Token: None 2022-09-02T05:53:19.452 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44 2022-09-02T05:53:19.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zbnhrsw+2aaoP3+u0QA8Wgk0R3A= 2022-09-02T05:53:19.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zbnhrsw+2aaoP3+u0QA8Wgk0R3A=', 'Content-Length': '0'} 2022-09-02T05:53:19.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000356df3dab5924868-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:53:19.457 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.458 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.458 DEBUG:boto:Method: GET 2022-09-02T05:53:19.459 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:19.459 DEBUG:boto:Data: 2022-09-02T05:53:19.460 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.461 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.461 DEBUG:boto:Params: {} 2022-09-02T05:53:19.462 DEBUG:boto:Token: None 2022-09-02T05:53:19.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44 2022-09-02T05:53:19.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:13sQTyN5EZdTs7v2Zx73rjm1fHE= 2022-09-02T05:53:19.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:13sQTyN5EZdTs7v2Zx73rjm1fHE=', 'Content-Length': '0'} 2022-09-02T05:53:19.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0024fca0f8336b6-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.514 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:53:19.515 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:53:19.515 DEBUG:boto:Method: GET 2022-09-02T05:53:19.516 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:53:19.516 DEBUG:boto:Data: 2022-09-02T05:53:19.517 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.517 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.518 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.519 DEBUG:boto:Params: {} 2022-09-02T05:53:19.519 DEBUG:boto:Token: None 2022-09-02T05:53:19.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj44 2022-09-02T05:53:19.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:13sQTyN5EZdTs7v2Zx73rjm1fHE= 2022-09-02T05:53:19.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:13sQTyN5EZdTs7v2Zx73rjm1fHE=', 'Content-Length': '0'} 2022-09-02T05:53:19.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d28447a50ae98d1e-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.524 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.525 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.525 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.526 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.526 DEBUG:boto:Method: GET 2022-09-02T05:53:19.527 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.527 DEBUG:boto:Data: 2022-09-02T05:53:19.528 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.529 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.529 DEBUG:boto:Params: {} 2022-09-02T05:53:19.530 DEBUG:boto:Token: None 2022-09-02T05:53:19.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9a72d54b6c3a1f-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.535 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.536 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.537 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.537 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.538 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.538 DEBUG:boto:Method: GET 2022-09-02T05:53:19.539 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:53:19.539 DEBUG:boto:Data: 2022-09-02T05:53:19.540 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.541 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.541 DEBUG:boto:Params: {} 2022-09-02T05:53:19.542 DEBUG:boto:Token: None 2022-09-02T05:53:19.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d34b5862a3615a-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.548 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:19.549 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.549 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.550 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.551 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.551 DEBUG:boto:Method: GET 2022-09-02T05:53:19.552 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.553 DEBUG:boto:Data: 2022-09-02T05:53:19.553 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.554 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.555 DEBUG:boto:Params: {} 2022-09-02T05:53:19.555 DEBUG:boto:Token: None 2022-09-02T05:53:19.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ= 2022-09-02T05:53:19.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ=', 'Content-Length': '0'} 2022-09-02T05:53:19.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb2614c675c0fd0-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.560 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.561 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.561 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.562 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.562 DEBUG:boto:Method: GET 2022-09-02T05:53:19.563 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.563 DEBUG:boto:Data: 2022-09-02T05:53:19.564 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.565 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.566 DEBUG:boto:Params: {} 2022-09-02T05:53:19.566 DEBUG:boto:Token: None 2022-09-02T05:53:19.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ= 2022-09-02T05:53:19.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ=', 'Content-Length': '0'} 2022-09-02T05:53:19.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d24b649f5e55167-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.614 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.615 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.616 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.616 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.617 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.617 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.618 DEBUG:boto:Data: 2022-09-02T05:53:19.618 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.619 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.620 DEBUG:boto:Params: {} 2022-09-02T05:53:19.621 DEBUG:boto:Token: None 2022-09-02T05:53:19.621 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3/OAmPhPxgUqmcjOrHYqfz0tJoY= 2022-09-02T05:53:19.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3/OAmPhPxgUqmcjOrHYqfz0tJoY=', 'Content-Length': '0'} 2022-09-02T05:53:19.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d566fc7b63a07d-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.626 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.627 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.627 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.628 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.628 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.629 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.629 DEBUG:boto:Data: 2022-09-02T05:53:19.630 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.631 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.631 DEBUG:boto:Params: {} 2022-09-02T05:53:19.632 DEBUG:boto:Token: None 2022-09-02T05:53:19.633 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3/OAmPhPxgUqmcjOrHYqfz0tJoY= 2022-09-02T05:53:19.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3/OAmPhPxgUqmcjOrHYqfz0tJoY=', 'Content-Length': '0'} 2022-09-02T05:53:19.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007818ad1c57b9632b-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:19.638 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.638 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.639 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.640 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.640 DEBUG:boto:Method: GET 2022-09-02T05:53:19.641 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.641 DEBUG:boto:Data: 2022-09-02T05:53:19.642 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.643 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.644 DEBUG:boto:Params: {} 2022-09-02T05:53:19.644 DEBUG:boto:Token: None 2022-09-02T05:53:19.645 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ= 2022-09-02T05:53:19.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ=', 'Content-Length': '0'} 2022-09-02T05:53:19.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1fcc0420974c0f5-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.649 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.650 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.651 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.651 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.652 DEBUG:boto:Method: GET 2022-09-02T05:53:19.652 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.653 DEBUG:boto:Data: 2022-09-02T05:53:19.654 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.655 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.656 DEBUG:boto:Params: {} 2022-09-02T05:53:19.657 DEBUG:boto:Token: None 2022-09-02T05:53:19.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:53:19.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ= 2022-09-02T05:53:19.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mHp9oj/jttuMKJ5B1uEqdSm+iMQ=', 'Content-Length': '0'} 2022-09-02T05:53:19.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3460162a52bb57-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.663 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.664 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.664 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.665 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.665 DEBUG:boto:Method: GET 2022-09-02T05:53:19.666 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.667 DEBUG:boto:Data: 2022-09-02T05:53:19.667 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.669 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.669 DEBUG:boto:Params: {} 2022-09-02T05:53:19.670 DEBUG:boto:Token: None 2022-09-02T05:53:19.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871f0f0b73af91f7-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.676 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.677 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.677 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.678 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.679 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.679 DEBUG:boto:Method: GET 2022-09-02T05:53:19.680 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.680 DEBUG:boto:Data: 2022-09-02T05:53:19.681 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.682 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.683 DEBUG:boto:Params: {} 2022-09-02T05:53:19.684 DEBUG:boto:Token: None 2022-09-02T05:53:19.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ba5fa776d26f1a5-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.690 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.691 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.692 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.692 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.693 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:19.693 DEBUG:boto:Data: 2022-09-02T05:53:19.694 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.695 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.696 DEBUG:boto:Params: {} 2022-09-02T05:53:19.698 DEBUG:boto:Token: None 2022-09-02T05:53:19.698 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46 2022-09-02T05:53:19.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8IHfujvLwxoWEIBZ2X5T7sKE4oY= 2022-09-02T05:53:19.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8IHfujvLwxoWEIBZ2X5T7sKE4oY=', 'Content-Length': '0'} 2022-09-02T05:53:19.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc33524749fcbac-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.703 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.703 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.704 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.705 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:19.705 DEBUG:boto:Data: 2022-09-02T05:53:19.706 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.707 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.707 DEBUG:boto:Params: {} 2022-09-02T05:53:19.708 DEBUG:boto:Token: None 2022-09-02T05:53:19.708 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46 2022-09-02T05:53:19.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8IHfujvLwxoWEIBZ2X5T7sKE4oY= 2022-09-02T05:53:19.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8IHfujvLwxoWEIBZ2X5T7sKE4oY=', 'Content-Length': '0'} 2022-09-02T05:53:19.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f475f29d854286ea-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:53:19.714 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.715 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.715 DEBUG:boto:Method: GET 2022-09-02T05:53:19.716 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:19.716 DEBUG:boto:Data: 2022-09-02T05:53:19.717 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.718 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.719 DEBUG:boto:Params: {} 2022-09-02T05:53:19.720 DEBUG:boto:Token: None 2022-09-02T05:53:19.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46 2022-09-02T05:53:19.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IgJBo84LznqpCNWKtzkdMLqvsmY= 2022-09-02T05:53:19.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IgJBo84LznqpCNWKtzkdMLqvsmY=', 'Content-Length': '0'} 2022-09-02T05:53:19.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecda6d173cfa12bb-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.770 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:53:19.771 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:53:19.772 DEBUG:boto:Method: GET 2022-09-02T05:53:19.773 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:53:19.774 DEBUG:boto:Data: 2022-09-02T05:53:19.775 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.777 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.778 DEBUG:boto:Params: {} 2022-09-02T05:53:19.780 DEBUG:boto:Token: None 2022-09-02T05:53:19.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj46 2022-09-02T05:53:19.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IgJBo84LznqpCNWKtzkdMLqvsmY= 2022-09-02T05:53:19.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IgJBo84LznqpCNWKtzkdMLqvsmY=', 'Content-Length': '0'} 2022-09-02T05:53:19.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066771e5292f4acb1-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.789 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.790 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.791 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.791 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.792 DEBUG:boto:Method: GET 2022-09-02T05:53:19.793 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.794 DEBUG:boto:Data: 2022-09-02T05:53:19.795 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.797 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.798 DEBUG:boto:Params: {} 2022-09-02T05:53:19.800 DEBUG:boto:Token: None 2022-09-02T05:53:19.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52db4b9a2b4e755-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.808 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.809 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.810 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.811 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.811 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.812 DEBUG:boto:Method: GET 2022-09-02T05:53:19.812 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:53:19.813 DEBUG:boto:Data: 2022-09-02T05:53:19.814 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.815 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.816 DEBUG:boto:Params: {} 2022-09-02T05:53:19.816 DEBUG:boto:Token: None 2022-09-02T05:53:19.817 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c7c2eede7ea0bb9-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.822 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:19.824 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.824 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.825 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.825 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.826 DEBUG:boto:Method: GET 2022-09-02T05:53:19.827 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.827 DEBUG:boto:Data: 2022-09-02T05:53:19.828 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.829 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.829 DEBUG:boto:Params: {} 2022-09-02T05:53:19.830 DEBUG:boto:Token: None 2022-09-02T05:53:19.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw= 2022-09-02T05:53:19.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw=', 'Content-Length': '0'} 2022-09-02T05:53:19.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be70a26cafb428b-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.837 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.838 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.838 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.839 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.840 DEBUG:boto:Method: GET 2022-09-02T05:53:19.841 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.843 DEBUG:boto:Data: 2022-09-02T05:53:19.844 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.845 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.846 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.847 DEBUG:boto:Params: {} 2022-09-02T05:53:19.848 DEBUG:boto:Token: None 2022-09-02T05:53:19.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw= 2022-09-02T05:53:19.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw=', 'Content-Length': '0'} 2022-09-02T05:53:19.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b513e7b78756a878-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.856 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.857 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.857 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.858 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.858 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.859 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.860 DEBUG:boto:Data: 2022-09-02T05:53:19.860 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.861 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.862 DEBUG:boto:Params: {} 2022-09-02T05:53:19.862 DEBUG:boto:Token: None 2022-09-02T05:53:19.863 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CKCj0mFZ6breIji1QRiyt1ABQAQ= 2022-09-02T05:53:19.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CKCj0mFZ6breIji1QRiyt1ABQAQ=', 'Content-Length': '0'} 2022-09-02T05:53:19.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607ecfc948fd9652-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.867 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.868 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.868 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.869 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.869 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.870 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.870 DEBUG:boto:Data: 2022-09-02T05:53:19.871 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.872 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.873 DEBUG:boto:Params: {} 2022-09-02T05:53:19.873 DEBUG:boto:Token: None 2022-09-02T05:53:19.874 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CKCj0mFZ6breIji1QRiyt1ABQAQ= 2022-09-02T05:53:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CKCj0mFZ6breIji1QRiyt1ABQAQ=', 'Content-Length': '0'} 2022-09-02T05:53:19.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003714a2917b282e5a-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:19.879 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.880 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.880 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.881 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.881 DEBUG:boto:Method: GET 2022-09-02T05:53:19.882 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.883 DEBUG:boto:Data: 2022-09-02T05:53:19.883 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.884 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.885 DEBUG:boto:Params: {} 2022-09-02T05:53:19.886 DEBUG:boto:Token: None 2022-09-02T05:53:19.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw= 2022-09-02T05:53:19.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw=', 'Content-Length': '0'} 2022-09-02T05:53:19.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967f2dec9545b1c6-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.934 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.935 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.936 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.936 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.937 DEBUG:boto:Method: GET 2022-09-02T05:53:19.937 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.938 DEBUG:boto:Data: 2022-09-02T05:53:19.938 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.939 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.940 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.940 DEBUG:boto:Params: {} 2022-09-02T05:53:19.941 DEBUG:boto:Token: None 2022-09-02T05:53:19.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:53:19.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw= 2022-09-02T05:53:19.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mRzhooA+yagwTRPTbpT49iqUvnw=', 'Content-Length': '0'} 2022-09-02T05:53:19.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8832790532e2f1-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.947 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.948 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.949 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:19.949 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:19.950 DEBUG:boto:Method: GET 2022-09-02T05:53:19.951 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:19.951 DEBUG:boto:Data: 2022-09-02T05:53:19.952 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.953 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.953 DEBUG:boto:Params: {} 2022-09-02T05:53:19.954 DEBUG:boto:Token: None 2022-09-02T05:53:19.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d99a734eefb36cd-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.960 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.961 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:19.962 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:19.962 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:19.964 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:19.964 DEBUG:boto:Method: GET 2022-09-02T05:53:19.965 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:19.966 DEBUG:boto:Data: 2022-09-02T05:53:19.966 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.967 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.968 DEBUG:boto:Params: {} 2022-09-02T05:53:19.969 DEBUG:boto:Token: None 2022-09-02T05:53:19.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/ 2022-09-02T05:53:19.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw= 2022-09-02T05:53:19.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j/9l5GWxlMabJbGR2LoC5wA7dZw=', 'Content-Length': '0'} 2022-09-02T05:53:19.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863d76e8419f6c4c-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.976 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:19.977 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.977 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.978 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.978 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:19.979 DEBUG:boto:Data: 2022-09-02T05:53:19.979 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:19.981 DEBUG:boto:Port: 8001 2022-09-02T05:53:19.982 DEBUG:boto:Params: {} 2022-09-02T05:53:19.982 DEBUG:boto:Token: None 2022-09-02T05:53:19.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51 2022-09-02T05:53:19.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XL2VomK06dasSAsVdCtHF+R0ZVU= 2022-09-02T05:53:19.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XL2VomK06dasSAsVdCtHF+R0ZVU=', 'Content-Length': '0'} 2022-09-02T05:53:19.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe991ee5f31677c5-0063119a4f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.987 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.988 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.988 DEBUG:boto:Method: HEAD 2022-09-02T05:53:19.989 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:19.989 DEBUG:boto:Data: 2022-09-02T05:53:19.990 DEBUG:boto:Headers: {} 2022-09-02T05:53:19.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:19.991 DEBUG:boto:Port: 8000 2022-09-02T05:53:19.992 DEBUG:boto:Params: {} 2022-09-02T05:53:19.993 DEBUG:boto:Token: None 2022-09-02T05:53:19.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:19 GMT /xljbkf-74/obj51 2022-09-02T05:53:19.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XL2VomK06dasSAsVdCtHF+R0ZVU= 2022-09-02T05:53:19.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:19 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XL2VomK06dasSAsVdCtHF+R0ZVU=', 'Content-Length': '0'} 2022-09-02T05:53:19.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d598bf5eea986c-0063119a4f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:19 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:19.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:53:19.998 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:19.999 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:19.999 DEBUG:boto:Method: GET 2022-09-02T05:53:20.000 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:20.000 DEBUG:boto:Data: 2022-09-02T05:53:20.001 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.002 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.003 DEBUG:boto:Params: {} 2022-09-02T05:53:20.003 DEBUG:boto:Token: None 2022-09-02T05:53:20.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj51 2022-09-02T05:53:20.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VFXPkGwOMaXK+uauNOpTHT+1PWo= 2022-09-02T05:53:20.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VFXPkGwOMaXK+uauNOpTHT+1PWo=', 'Content-Length': '0'} 2022-09-02T05:53:20.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db6073f6453c1a44-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.050 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:53:20.051 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:53:20.052 DEBUG:boto:Method: GET 2022-09-02T05:53:20.052 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:53:20.053 DEBUG:boto:Data: 2022-09-02T05:53:20.054 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.055 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.056 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.057 DEBUG:boto:Params: {} 2022-09-02T05:53:20.058 DEBUG:boto:Token: None 2022-09-02T05:53:20.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj51 2022-09-02T05:53:20.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VFXPkGwOMaXK+uauNOpTHT+1PWo= 2022-09-02T05:53:20.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VFXPkGwOMaXK+uauNOpTHT+1PWo=', 'Content-Length': '0'} 2022-09-02T05:53:20.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a634df471b11dd-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.063 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.064 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.065 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:20.065 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:20.066 DEBUG:boto:Method: GET 2022-09-02T05:53:20.067 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:20.068 DEBUG:boto:Data: 2022-09-02T05:53:20.068 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.069 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.070 DEBUG:boto:Params: {} 2022-09-02T05:53:20.071 DEBUG:boto:Token: None 2022-09-02T05:53:20.072 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d19d55bf9a5f2f5a-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.077 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.078 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.079 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.080 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:20.081 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:53:20.082 DEBUG:boto:Method: GET 2022-09-02T05:53:20.083 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:53:20.084 DEBUG:boto:Data: 2022-09-02T05:53:20.085 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.086 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.087 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.088 DEBUG:boto:Params: {} 2022-09-02T05:53:20.089 DEBUG:boto:Token: None 2022-09-02T05:53:20.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544a5ab2e9e92dcc-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.096 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:20.098 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.098 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.099 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.100 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.100 DEBUG:boto:Method: GET 2022-09-02T05:53:20.101 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.101 DEBUG:boto:Data: 2022-09-02T05:53:20.102 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.103 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.103 DEBUG:boto:Params: {} 2022-09-02T05:53:20.104 DEBUG:boto:Token: None 2022-09-02T05:53:20.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg= 2022-09-02T05:53:20.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg=', 'Content-Length': '0'} 2022-09-02T05:53:20.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5ac5aaf96467af-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.110 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.110 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.111 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.111 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.112 DEBUG:boto:Method: GET 2022-09-02T05:53:20.113 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.114 DEBUG:boto:Data: 2022-09-02T05:53:20.114 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.115 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.116 DEBUG:boto:Params: {} 2022-09-02T05:53:20.117 DEBUG:boto:Token: None 2022-09-02T05:53:20.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg= 2022-09-02T05:53:20.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg=', 'Content-Length': '0'} 2022-09-02T05:53:20.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee805d16a0dad79-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.162 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.163 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.164 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.165 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.166 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.167 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.168 DEBUG:boto:Data: 2022-09-02T05:53:20.169 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.171 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.172 DEBUG:boto:Params: {} 2022-09-02T05:53:20.173 DEBUG:boto:Token: None 2022-09-02T05:53:20.174 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hKlEr+pPggYf3ZedvDGCrGyWyCk= 2022-09-02T05:53:20.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hKlEr+pPggYf3ZedvDGCrGyWyCk=', 'Content-Length': '0'} 2022-09-02T05:53:20.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9ae4ed9dec02b4-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.179 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.180 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.180 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.181 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.181 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.182 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.182 DEBUG:boto:Data: 2022-09-02T05:53:20.183 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.184 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.184 DEBUG:boto:Params: {} 2022-09-02T05:53:20.185 DEBUG:boto:Token: None 2022-09-02T05:53:20.186 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hKlEr+pPggYf3ZedvDGCrGyWyCk= 2022-09-02T05:53:20.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hKlEr+pPggYf3ZedvDGCrGyWyCk=', 'Content-Length': '0'} 2022-09-02T05:53:20.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073459d36719413bc-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:20.190 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.191 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.191 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.192 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.192 DEBUG:boto:Method: GET 2022-09-02T05:53:20.193 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.194 DEBUG:boto:Data: 2022-09-02T05:53:20.194 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.195 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.196 DEBUG:boto:Params: {} 2022-09-02T05:53:20.197 DEBUG:boto:Token: None 2022-09-02T05:53:20.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg= 2022-09-02T05:53:20.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg=', 'Content-Length': '0'} 2022-09-02T05:53:20.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a206eff662d9346-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.203 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.204 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.204 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.205 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.205 DEBUG:boto:Method: GET 2022-09-02T05:53:20.206 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.206 DEBUG:boto:Data: 2022-09-02T05:53:20.207 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.208 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.208 DEBUG:boto:Params: {} 2022-09-02T05:53:20.209 DEBUG:boto:Token: None 2022-09-02T05:53:20.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:53:20.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg= 2022-09-02T05:53:20.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dzRrfLG84UJULNitSNQeBCPyNAg=', 'Content-Length': '0'} 2022-09-02T05:53:20.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e738f774fd2f61c-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.217 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.218 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.218 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.219 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.219 DEBUG:boto:Method: GET 2022-09-02T05:53:20.220 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.220 DEBUG:boto:Data: 2022-09-02T05:53:20.221 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.222 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.222 DEBUG:boto:Params: {} 2022-09-02T05:53:20.223 DEBUG:boto:Token: None 2022-09-02T05:53:20.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adce5c13d6c79940-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.229 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.230 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.230 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.231 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.231 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.232 DEBUG:boto:Method: GET 2022-09-02T05:53:20.232 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.233 DEBUG:boto:Data: 2022-09-02T05:53:20.233 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.234 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.235 DEBUG:boto:Params: {} 2022-09-02T05:53:20.236 DEBUG:boto:Token: None 2022-09-02T05:53:20.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.237 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b6824aa16e8266-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.241 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.242 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.242 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.243 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.243 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:20.244 DEBUG:boto:Data: 2022-09-02T05:53:20.244 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.246 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.246 DEBUG:boto:Params: {} 2022-09-02T05:53:20.247 DEBUG:boto:Token: None 2022-09-02T05:53:20.248 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55 2022-09-02T05:53:20.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nOYtyg1gdCEhklgiN9JAD0PV9yc= 2022-09-02T05:53:20.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nOYtyg1gdCEhklgiN9JAD0PV9yc=', 'Content-Length': '0'} 2022-09-02T05:53:20.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009116bd1393560280-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.252 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.253 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.253 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.254 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:20.255 DEBUG:boto:Data: 2022-09-02T05:53:20.256 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.257 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.258 DEBUG:boto:Params: {} 2022-09-02T05:53:20.259 DEBUG:boto:Token: None 2022-09-02T05:53:20.259 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55 2022-09-02T05:53:20.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nOYtyg1gdCEhklgiN9JAD0PV9yc= 2022-09-02T05:53:20.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nOYtyg1gdCEhklgiN9JAD0PV9yc=', 'Content-Length': '0'} 2022-09-02T05:53:20.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062a4f8916ee3ba56-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:53:20.264 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.265 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.265 DEBUG:boto:Method: GET 2022-09-02T05:53:20.266 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:20.267 DEBUG:boto:Data: 2022-09-02T05:53:20.267 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.268 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.269 DEBUG:boto:Params: {} 2022-09-02T05:53:20.270 DEBUG:boto:Token: None 2022-09-02T05:53:20.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55 2022-09-02T05:53:20.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VLuquyBQMl+w1PdRNH2ibQ/xvJ0= 2022-09-02T05:53:20.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VLuquyBQMl+w1PdRNH2ibQ/xvJ0=', 'Content-Length': '0'} 2022-09-02T05:53:20.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72bbd615491725a-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.275 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:53:20.276 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:53:20.276 DEBUG:boto:Method: GET 2022-09-02T05:53:20.277 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:53:20.277 DEBUG:boto:Data: 2022-09-02T05:53:20.278 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.279 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.280 DEBUG:boto:Params: {} 2022-09-02T05:53:20.280 DEBUG:boto:Token: None 2022-09-02T05:53:20.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj55 2022-09-02T05:53:20.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VLuquyBQMl+w1PdRNH2ibQ/xvJ0= 2022-09-02T05:53:20.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VLuquyBQMl+w1PdRNH2ibQ/xvJ0=', 'Content-Length': '0'} 2022-09-02T05:53:20.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087a839084bd1eb00-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.286 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.287 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.287 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.288 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.289 DEBUG:boto:Method: GET 2022-09-02T05:53:20.290 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.291 DEBUG:boto:Data: 2022-09-02T05:53:20.291 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.293 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.293 DEBUG:boto:Params: {} 2022-09-02T05:53:20.294 DEBUG:boto:Token: None 2022-09-02T05:53:20.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbfe6ed38a461840-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.347 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.349 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.350 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.351 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.352 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.353 DEBUG:boto:Method: GET 2022-09-02T05:53:20.354 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:53:20.355 DEBUG:boto:Data: 2022-09-02T05:53:20.356 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.358 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.359 DEBUG:boto:Params: {} 2022-09-02T05:53:20.360 DEBUG:boto:Token: None 2022-09-02T05:53:20.360 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096b36f6061def984-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.366 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:20.367 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.368 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.368 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.369 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.369 DEBUG:boto:Method: GET 2022-09-02T05:53:20.370 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.371 DEBUG:boto:Data: 2022-09-02T05:53:20.371 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.372 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.373 DEBUG:boto:Params: {} 2022-09-02T05:53:20.373 DEBUG:boto:Token: None 2022-09-02T05:53:20.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI= 2022-09-02T05:53:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI=', 'Content-Length': '0'} 2022-09-02T05:53:20.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c4301a7d542c7a-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.379 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.380 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.380 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.381 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.381 DEBUG:boto:Method: GET 2022-09-02T05:53:20.382 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.382 DEBUG:boto:Data: 2022-09-02T05:53:20.384 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.385 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.386 DEBUG:boto:Params: {} 2022-09-02T05:53:20.387 DEBUG:boto:Token: None 2022-09-02T05:53:20.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI= 2022-09-02T05:53:20.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI=', 'Content-Length': '0'} 2022-09-02T05:53:20.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e382071fcd3ee91a-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.393 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.393 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.394 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.394 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.395 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.396 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.397 DEBUG:boto:Data: 2022-09-02T05:53:20.397 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.399 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.399 DEBUG:boto:Params: {} 2022-09-02T05:53:20.400 DEBUG:boto:Token: None 2022-09-02T05:53:20.401 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MjAFfnJSxpGsmTcW8ksDWj+qn7U= 2022-09-02T05:53:20.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MjAFfnJSxpGsmTcW8ksDWj+qn7U=', 'Content-Length': '0'} 2022-09-02T05:53:20.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca3200ba88d7054c-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.406 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.407 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.408 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.408 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.409 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.410 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.410 DEBUG:boto:Data: 2022-09-02T05:53:20.411 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.412 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.412 DEBUG:boto:Params: {} 2022-09-02T05:53:20.413 DEBUG:boto:Token: None 2022-09-02T05:53:20.413 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MjAFfnJSxpGsmTcW8ksDWj+qn7U= 2022-09-02T05:53:20.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MjAFfnJSxpGsmTcW8ksDWj+qn7U=', 'Content-Length': '0'} 2022-09-02T05:53:20.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000804dac2bd5905638-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:20.420 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.421 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.421 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.422 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.423 DEBUG:boto:Method: GET 2022-09-02T05:53:20.423 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.424 DEBUG:boto:Data: 2022-09-02T05:53:20.425 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.426 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.427 DEBUG:boto:Params: {} 2022-09-02T05:53:20.428 DEBUG:boto:Token: None 2022-09-02T05:53:20.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI= 2022-09-02T05:53:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI=', 'Content-Length': '0'} 2022-09-02T05:53:20.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b850f332b3f3445-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.433 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.434 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.435 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.435 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.436 DEBUG:boto:Method: GET 2022-09-02T05:53:20.437 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.438 DEBUG:boto:Data: 2022-09-02T05:53:20.439 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.440 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.441 DEBUG:boto:Params: {} 2022-09-02T05:53:20.441 DEBUG:boto:Token: None 2022-09-02T05:53:20.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:53:20.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI= 2022-09-02T05:53:20.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G3FlaFSviNl+3IrPmk46b/+irFI=', 'Content-Length': '0'} 2022-09-02T05:53:20.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4185f8f0818376-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.490 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.491 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.492 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.492 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.493 DEBUG:boto:Method: GET 2022-09-02T05:53:20.494 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.494 DEBUG:boto:Data: 2022-09-02T05:53:20.495 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.496 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.497 DEBUG:boto:Params: {} 2022-09-02T05:53:20.497 DEBUG:boto:Token: None 2022-09-02T05:53:20.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7125dab31404cfd-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.504 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.505 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.505 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.506 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.506 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.507 DEBUG:boto:Method: GET 2022-09-02T05:53:20.507 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.508 DEBUG:boto:Data: 2022-09-02T05:53:20.508 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.510 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.510 DEBUG:boto:Params: {} 2022-09-02T05:53:20.511 DEBUG:boto:Token: None 2022-09-02T05:53:20.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cffca01ee4f98bca-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.516 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.517 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.518 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.519 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.519 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:20.520 DEBUG:boto:Data: 2022-09-02T05:53:20.521 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.522 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.522 DEBUG:boto:Params: {} 2022-09-02T05:53:20.523 DEBUG:boto:Token: None 2022-09-02T05:53:20.524 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57 2022-09-02T05:53:20.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DnqDTruQ5cmYDJsYIgrQ5sYU7CM= 2022-09-02T05:53:20.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DnqDTruQ5cmYDJsYIgrQ5sYU7CM=', 'Content-Length': '0'} 2022-09-02T05:53:20.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009809efc288923d37-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.527 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.528 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.529 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.529 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:20.530 DEBUG:boto:Data: 2022-09-02T05:53:20.531 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.532 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.532 DEBUG:boto:Params: {} 2022-09-02T05:53:20.533 DEBUG:boto:Token: None 2022-09-02T05:53:20.534 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57 2022-09-02T05:53:20.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DnqDTruQ5cmYDJsYIgrQ5sYU7CM= 2022-09-02T05:53:20.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DnqDTruQ5cmYDJsYIgrQ5sYU7CM=', 'Content-Length': '0'} 2022-09-02T05:53:20.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f719a0dd8ad6c9a-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:53:20.539 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.540 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.541 DEBUG:boto:Method: GET 2022-09-02T05:53:20.541 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:20.542 DEBUG:boto:Data: 2022-09-02T05:53:20.543 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.544 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.545 DEBUG:boto:Params: {} 2022-09-02T05:53:20.546 DEBUG:boto:Token: None 2022-09-02T05:53:20.547 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57 2022-09-02T05:53:20.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eLTbJN4+T2rXMRA6iBJmilMCWWM= 2022-09-02T05:53:20.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eLTbJN4+T2rXMRA6iBJmilMCWWM=', 'Content-Length': '0'} 2022-09-02T05:53:20.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d0f237ee5a5eeb-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.594 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:53:20.595 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:53:20.596 DEBUG:boto:Method: GET 2022-09-02T05:53:20.596 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:53:20.597 DEBUG:boto:Data: 2022-09-02T05:53:20.597 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.599 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.599 DEBUG:boto:Params: {} 2022-09-02T05:53:20.600 DEBUG:boto:Token: None 2022-09-02T05:53:20.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj57 2022-09-02T05:53:20.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eLTbJN4+T2rXMRA6iBJmilMCWWM= 2022-09-02T05:53:20.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eLTbJN4+T2rXMRA6iBJmilMCWWM=', 'Content-Length': '0'} 2022-09-02T05:53:20.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a13a88cb8f05329-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.607 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.608 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.608 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.609 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.609 DEBUG:boto:Method: GET 2022-09-02T05:53:20.610 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.611 DEBUG:boto:Data: 2022-09-02T05:53:20.611 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.613 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.613 DEBUG:boto:Params: {} 2022-09-02T05:53:20.614 DEBUG:boto:Token: None 2022-09-02T05:53:20.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd02f753a7947ef-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.620 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.621 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.622 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.623 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.623 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.624 DEBUG:boto:Method: GET 2022-09-02T05:53:20.624 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:53:20.625 DEBUG:boto:Data: 2022-09-02T05:53:20.625 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.626 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.627 DEBUG:boto:Params: {} 2022-09-02T05:53:20.628 DEBUG:boto:Token: None 2022-09-02T05:53:20.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000389a243e3d1b32ec-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.634 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:20.636 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.636 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.637 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.637 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.638 DEBUG:boto:Method: GET 2022-09-02T05:53:20.638 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.639 DEBUG:boto:Data: 2022-09-02T05:53:20.640 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.641 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.642 DEBUG:boto:Params: {} 2022-09-02T05:53:20.643 DEBUG:boto:Token: None 2022-09-02T05:53:20.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU= 2022-09-02T05:53:20.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU=', 'Content-Length': '0'} 2022-09-02T05:53:20.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded31c18706a1da1-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.648 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.648 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.649 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.650 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.651 DEBUG:boto:Method: GET 2022-09-02T05:53:20.651 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.652 DEBUG:boto:Data: 2022-09-02T05:53:20.652 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.654 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.655 DEBUG:boto:Params: {} 2022-09-02T05:53:20.655 DEBUG:boto:Token: None 2022-09-02T05:53:20.656 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU= 2022-09-02T05:53:20.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU=', 'Content-Length': '0'} 2022-09-02T05:53:20.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffba0f9021d9dac-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.661 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.661 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.662 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.663 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.663 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.664 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.665 DEBUG:boto:Data: 2022-09-02T05:53:20.666 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.667 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.667 DEBUG:boto:Params: {} 2022-09-02T05:53:20.668 DEBUG:boto:Token: None 2022-09-02T05:53:20.669 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cGtzoOYhuj75l0ua3vI9nHl2pWc= 2022-09-02T05:53:20.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cGtzoOYhuj75l0ua3vI9nHl2pWc=', 'Content-Length': '0'} 2022-09-02T05:53:20.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044118be927abf8c4-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.673 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.674 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.674 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.675 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.675 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.676 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.676 DEBUG:boto:Data: 2022-09-02T05:53:20.677 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.678 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.678 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.679 DEBUG:boto:Params: {} 2022-09-02T05:53:20.680 DEBUG:boto:Token: None 2022-09-02T05:53:20.681 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cGtzoOYhuj75l0ua3vI9nHl2pWc= 2022-09-02T05:53:20.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cGtzoOYhuj75l0ua3vI9nHl2pWc=', 'Content-Length': '0'} 2022-09-02T05:53:20.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d810f96126f9dd-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:20.686 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.686 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.687 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.688 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.688 DEBUG:boto:Method: GET 2022-09-02T05:53:20.689 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.690 DEBUG:boto:Data: 2022-09-02T05:53:20.690 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.692 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.692 DEBUG:boto:Params: {} 2022-09-02T05:53:20.693 DEBUG:boto:Token: None 2022-09-02T05:53:20.693 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU= 2022-09-02T05:53:20.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU=', 'Content-Length': '0'} 2022-09-02T05:53:20.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ccad3bf1cfea03a-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.742 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.743 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.744 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.745 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.745 DEBUG:boto:Method: GET 2022-09-02T05:53:20.746 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.747 DEBUG:boto:Data: 2022-09-02T05:53:20.747 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.748 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.749 DEBUG:boto:Params: {} 2022-09-02T05:53:20.750 DEBUG:boto:Token: None 2022-09-02T05:53:20.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:53:20.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU= 2022-09-02T05:53:20.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:27Wo0RLMa77RYHrJGpITmzL5PcU=', 'Content-Length': '0'} 2022-09-02T05:53:20.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b5d91baaaa7d66-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.755 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.756 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.756 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.757 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.757 DEBUG:boto:Method: GET 2022-09-02T05:53:20.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.758 DEBUG:boto:Data: 2022-09-02T05:53:20.759 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.760 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.760 DEBUG:boto:Params: {} 2022-09-02T05:53:20.761 DEBUG:boto:Token: None 2022-09-02T05:53:20.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f569bae20b516cd-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.767 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.768 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.768 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.769 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.770 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.770 DEBUG:boto:Method: GET 2022-09-02T05:53:20.771 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.771 DEBUG:boto:Data: 2022-09-02T05:53:20.772 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.772 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.773 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.773 DEBUG:boto:Params: {} 2022-09-02T05:53:20.774 DEBUG:boto:Token: None 2022-09-02T05:53:20.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632298e55c3f0f1f-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.822 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.823 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.824 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.824 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.825 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:20.825 DEBUG:boto:Data: 2022-09-02T05:53:20.826 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.827 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.827 DEBUG:boto:Params: {} 2022-09-02T05:53:20.828 DEBUG:boto:Token: None 2022-09-02T05:53:20.829 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68 2022-09-02T05:53:20.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c0bq6iow2Rc1RNGuUcfW7COJVDY= 2022-09-02T05:53:20.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c0bq6iow2Rc1RNGuUcfW7COJVDY=', 'Content-Length': '0'} 2022-09-02T05:53:20.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095009936609591a2-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.833 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.833 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.834 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.834 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:20.835 DEBUG:boto:Data: 2022-09-02T05:53:20.835 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.836 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.837 DEBUG:boto:Params: {} 2022-09-02T05:53:20.837 DEBUG:boto:Token: None 2022-09-02T05:53:20.838 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68 2022-09-02T05:53:20.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c0bq6iow2Rc1RNGuUcfW7COJVDY= 2022-09-02T05:53:20.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c0bq6iow2Rc1RNGuUcfW7COJVDY=', 'Content-Length': '0'} 2022-09-02T05:53:20.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592af396c11210b5-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:53:20.843 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.843 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.844 DEBUG:boto:Method: GET 2022-09-02T05:53:20.844 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:20.845 DEBUG:boto:Data: 2022-09-02T05:53:20.846 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.847 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.847 DEBUG:boto:Params: {} 2022-09-02T05:53:20.848 DEBUG:boto:Token: None 2022-09-02T05:53:20.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68 2022-09-02T05:53:20.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D6MaLoW9VY4ZOgG0hAv0BzlJ6mk= 2022-09-02T05:53:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D6MaLoW9VY4ZOgG0hAv0BzlJ6mk=', 'Content-Length': '0'} 2022-09-02T05:53:20.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4097ef626edcc1-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.853 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:53:20.853 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:53:20.854 DEBUG:boto:Method: GET 2022-09-02T05:53:20.854 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:53:20.855 DEBUG:boto:Data: 2022-09-02T05:53:20.856 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.856 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.857 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.857 DEBUG:boto:Params: {} 2022-09-02T05:53:20.858 DEBUG:boto:Token: None 2022-09-02T05:53:20.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj68 2022-09-02T05:53:20.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D6MaLoW9VY4ZOgG0hAv0BzlJ6mk= 2022-09-02T05:53:20.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D6MaLoW9VY4ZOgG0hAv0BzlJ6mk=', 'Content-Length': '0'} 2022-09-02T05:53:20.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f20ac8db7f6925-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.863 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.863 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.864 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.864 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.865 DEBUG:boto:Method: GET 2022-09-02T05:53:20.865 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.866 DEBUG:boto:Data: 2022-09-02T05:53:20.866 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.867 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.868 DEBUG:boto:Params: {} 2022-09-02T05:53:20.869 DEBUG:boto:Token: None 2022-09-02T05:53:20.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfea2aa48d8638dd-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.874 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.875 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.875 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.876 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.876 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.877 DEBUG:boto:Method: GET 2022-09-02T05:53:20.877 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:53:20.878 DEBUG:boto:Data: 2022-09-02T05:53:20.878 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.879 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.880 DEBUG:boto:Params: {} 2022-09-02T05:53:20.880 DEBUG:boto:Token: None 2022-09-02T05:53:20.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028016cc2cb11a40d-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.886 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:20.887 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.888 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.888 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.889 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.889 DEBUG:boto:Method: GET 2022-09-02T05:53:20.890 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.890 DEBUG:boto:Data: 2022-09-02T05:53:20.891 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.892 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.892 DEBUG:boto:Params: {} 2022-09-02T05:53:20.893 DEBUG:boto:Token: None 2022-09-02T05:53:20.894 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0= 2022-09-02T05:53:20.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0=', 'Content-Length': '0'} 2022-09-02T05:53:20.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108fb172d15776e1-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.898 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.899 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.899 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.900 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.900 DEBUG:boto:Method: GET 2022-09-02T05:53:20.901 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.901 DEBUG:boto:Data: 2022-09-02T05:53:20.902 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.903 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.903 DEBUG:boto:Params: {} 2022-09-02T05:53:20.904 DEBUG:boto:Token: None 2022-09-02T05:53:20.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0= 2022-09-02T05:53:20.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0=', 'Content-Length': '0'} 2022-09-02T05:53:20.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000527c1268554a26e9-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.909 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.909 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.910 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.910 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.911 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.911 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.912 DEBUG:boto:Data: 2022-09-02T05:53:20.912 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.913 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.914 DEBUG:boto:Params: {} 2022-09-02T05:53:20.914 DEBUG:boto:Token: None 2022-09-02T05:53:20.915 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KEGOLZRlZdQrh4ZzyvYfKENArTc= 2022-09-02T05:53:20.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KEGOLZRlZdQrh4ZzyvYfKENArTc=', 'Content-Length': '0'} 2022-09-02T05:53:20.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb055c3fade748f-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.919 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.920 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.920 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.921 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.921 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.922 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.922 DEBUG:boto:Data: 2022-09-02T05:53:20.923 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.923 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.924 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.924 DEBUG:boto:Params: {} 2022-09-02T05:53:20.925 DEBUG:boto:Token: None 2022-09-02T05:53:20.925 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KEGOLZRlZdQrh4ZzyvYfKENArTc= 2022-09-02T05:53:20.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KEGOLZRlZdQrh4ZzyvYfKENArTc=', 'Content-Length': '0'} 2022-09-02T05:53:20.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d23019fb5f46c52-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:20.930 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.930 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.931 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.932 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.932 DEBUG:boto:Method: GET 2022-09-02T05:53:20.933 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.933 DEBUG:boto:Data: 2022-09-02T05:53:20.934 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.935 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.935 DEBUG:boto:Params: {} 2022-09-02T05:53:20.936 DEBUG:boto:Token: None 2022-09-02T05:53:20.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0= 2022-09-02T05:53:20.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0=', 'Content-Length': '0'} 2022-09-02T05:53:20.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe453bdeedac0df-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.941 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.941 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.942 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.943 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.943 DEBUG:boto:Method: GET 2022-09-02T05:53:20.944 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.944 DEBUG:boto:Data: 2022-09-02T05:53:20.945 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.945 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.946 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.946 DEBUG:boto:Params: {} 2022-09-02T05:53:20.947 DEBUG:boto:Token: None 2022-09-02T05:53:20.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:53:20.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0= 2022-09-02T05:53:20.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayRHsWC4rhmKOI/A/BAiG8gpRe0=', 'Content-Length': '0'} 2022-09-02T05:53:20.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028e7590a35708824-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.952 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.952 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.953 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:20.953 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:20.954 DEBUG:boto:Method: GET 2022-09-02T05:53:20.955 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:20.955 DEBUG:boto:Data: 2022-09-02T05:53:20.956 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.957 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.957 DEBUG:boto:Params: {} 2022-09-02T05:53:20.958 DEBUG:boto:Token: None 2022-09-02T05:53:20.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e170a385d0f013-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.963 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.964 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:20.964 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:20.965 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:20.965 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:20.966 DEBUG:boto:Method: GET 2022-09-02T05:53:20.966 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:20.967 DEBUG:boto:Data: 2022-09-02T05:53:20.968 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.969 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.969 DEBUG:boto:Params: {} 2022-09-02T05:53:20.970 DEBUG:boto:Token: None 2022-09-02T05:53:20.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/ 2022-09-02T05:53:20.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s= 2022-09-02T05:53:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2rSGtIwPYGAu1d8t72IY4VpLV9s=', 'Content-Length': '0'} 2022-09-02T05:53:20.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070826d20438ce88b-0063119a50-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.975 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:20.976 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.977 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.977 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.978 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:20.979 DEBUG:boto:Data: 2022-09-02T05:53:20.979 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:20.981 DEBUG:boto:Port: 8001 2022-09-02T05:53:20.982 DEBUG:boto:Params: {} 2022-09-02T05:53:20.982 DEBUG:boto:Token: None 2022-09-02T05:53:20.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69 2022-09-02T05:53:20.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TSJjr7ic/bm+lP9rXKqbiJX/gWk= 2022-09-02T05:53:20.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TSJjr7ic/bm+lP9rXKqbiJX/gWk=', 'Content-Length': '0'} 2022-09-02T05:53:20.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a948d0b331daa792-0063119a50-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:20.988 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:20.989 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:20.989 DEBUG:boto:Method: HEAD 2022-09-02T05:53:20.990 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:20.991 DEBUG:boto:Data: 2022-09-02T05:53:20.991 DEBUG:boto:Headers: {} 2022-09-02T05:53:20.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:20.993 DEBUG:boto:Port: 8000 2022-09-02T05:53:20.993 DEBUG:boto:Params: {} 2022-09-02T05:53:20.994 DEBUG:boto:Token: None 2022-09-02T05:53:20.995 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:20 GMT /xljbkf-74/obj69 2022-09-02T05:53:20.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TSJjr7ic/bm+lP9rXKqbiJX/gWk= 2022-09-02T05:53:20.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:20 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TSJjr7ic/bm+lP9rXKqbiJX/gWk=', 'Content-Length': '0'} 2022-09-02T05:53:20.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13a92dc1418c0c1-0063119a50-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:20 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:53:21.000 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:21.001 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:21.002 DEBUG:boto:Method: GET 2022-09-02T05:53:21.002 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:21.003 DEBUG:boto:Data: 2022-09-02T05:53:21.004 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.005 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.006 DEBUG:boto:Params: {} 2022-09-02T05:53:21.006 DEBUG:boto:Token: None 2022-09-02T05:53:21.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj69 2022-09-02T05:53:21.008 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1u/lqv4yYL+jHkp5h3gUZIIrv4E= 2022-09-02T05:53:21.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1u/lqv4yYL+jHkp5h3gUZIIrv4E=', 'Content-Length': '0'} 2022-09-02T05:53:21.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003871f22c1fdd838f-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.054 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:53:21.055 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:53:21.056 DEBUG:boto:Method: GET 2022-09-02T05:53:21.056 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:53:21.057 DEBUG:boto:Data: 2022-09-02T05:53:21.058 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.059 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.060 DEBUG:boto:Params: {} 2022-09-02T05:53:21.061 DEBUG:boto:Token: None 2022-09-02T05:53:21.061 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj69 2022-09-02T05:53:21.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1u/lqv4yYL+jHkp5h3gUZIIrv4E= 2022-09-02T05:53:21.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1u/lqv4yYL+jHkp5h3gUZIIrv4E=', 'Content-Length': '0'} 2022-09-02T05:53:21.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e752c298378c3db-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.066 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.067 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.068 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:21.068 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:21.069 DEBUG:boto:Method: GET 2022-09-02T05:53:21.070 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:21.071 DEBUG:boto:Data: 2022-09-02T05:53:21.071 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.072 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.073 DEBUG:boto:Params: {} 2022-09-02T05:53:21.074 DEBUG:boto:Token: None 2022-09-02T05:53:21.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ec42c5dc7694e61-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.080 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.081 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.082 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.083 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:21.083 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:53:21.084 DEBUG:boto:Method: GET 2022-09-02T05:53:21.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:53:21.086 DEBUG:boto:Data: 2022-09-02T05:53:21.087 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.088 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.089 DEBUG:boto:Params: {} 2022-09-02T05:53:21.090 DEBUG:boto:Token: None 2022-09-02T05:53:21.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d56a1a22fc1f4c0-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.096 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:21.098 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.098 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.099 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.100 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.101 DEBUG:boto:Method: GET 2022-09-02T05:53:21.101 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.102 DEBUG:boto:Data: 2022-09-02T05:53:21.102 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.103 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.104 DEBUG:boto:Params: {} 2022-09-02T05:53:21.105 DEBUG:boto:Token: None 2022-09-02T05:53:21.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc= 2022-09-02T05:53:21.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc=', 'Content-Length': '0'} 2022-09-02T05:53:21.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9cb10387e19ecd-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.110 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.110 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.111 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.111 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.112 DEBUG:boto:Method: GET 2022-09-02T05:53:21.113 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.113 DEBUG:boto:Data: 2022-09-02T05:53:21.114 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.115 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.116 DEBUG:boto:Params: {} 2022-09-02T05:53:21.117 DEBUG:boto:Token: None 2022-09-02T05:53:21.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc= 2022-09-02T05:53:21.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc=', 'Content-Length': '0'} 2022-09-02T05:53:21.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f580b32652e44a96-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.123 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.123 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.124 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.125 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.126 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.127 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.127 DEBUG:boto:Data: 2022-09-02T05:53:21.128 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.130 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.131 DEBUG:boto:Params: {} 2022-09-02T05:53:21.131 DEBUG:boto:Token: None 2022-09-02T05:53:21.132 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/VFEzdHXo1JTKCT4olq85816fM4= 2022-09-02T05:53:21.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/VFEzdHXo1JTKCT4olq85816fM4=', 'Content-Length': '0'} 2022-09-02T05:53:21.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092dd65cebf1abd06-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.136 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.137 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.137 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.138 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.138 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.139 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.139 DEBUG:boto:Data: 2022-09-02T05:53:21.140 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.141 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.142 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.142 DEBUG:boto:Params: {} 2022-09-02T05:53:21.143 DEBUG:boto:Token: None 2022-09-02T05:53:21.144 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/VFEzdHXo1JTKCT4olq85816fM4= 2022-09-02T05:53:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/VFEzdHXo1JTKCT4olq85816fM4=', 'Content-Length': '0'} 2022-09-02T05:53:21.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f765fb0132f022d6-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:21.148 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.149 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.149 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.150 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.151 DEBUG:boto:Method: GET 2022-09-02T05:53:21.152 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.152 DEBUG:boto:Data: 2022-09-02T05:53:21.153 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.154 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.155 DEBUG:boto:Params: {} 2022-09-02T05:53:21.155 DEBUG:boto:Token: None 2022-09-02T05:53:21.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc= 2022-09-02T05:53:21.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc=', 'Content-Length': '0'} 2022-09-02T05:53:21.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53ce992ef257ac4-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.160 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.161 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.162 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.162 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.163 DEBUG:boto:Method: GET 2022-09-02T05:53:21.163 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.164 DEBUG:boto:Data: 2022-09-02T05:53:21.165 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.167 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.167 DEBUG:boto:Params: {} 2022-09-02T05:53:21.168 DEBUG:boto:Token: None 2022-09-02T05:53:21.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:53:21.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc= 2022-09-02T05:53:21.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VRjYz4DyJCXVTx4bzJ+IlND00xc=', 'Content-Length': '0'} 2022-09-02T05:53:21.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000250df73b5b8d515b-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.173 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.174 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.174 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.175 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.175 DEBUG:boto:Method: GET 2022-09-02T05:53:21.176 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.176 DEBUG:boto:Data: 2022-09-02T05:53:21.177 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.178 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.178 DEBUG:boto:Params: {} 2022-09-02T05:53:21.179 DEBUG:boto:Token: None 2022-09-02T05:53:21.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071d03565f1845e78-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.185 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.186 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.186 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.187 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.187 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.188 DEBUG:boto:Method: GET 2022-09-02T05:53:21.188 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.189 DEBUG:boto:Data: 2022-09-02T05:53:21.190 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.190 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.191 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.191 DEBUG:boto:Params: {} 2022-09-02T05:53:21.192 DEBUG:boto:Token: None 2022-09-02T05:53:21.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.193 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfd45a5dbdc7025-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.198 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.198 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.199 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.199 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.200 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:21.201 DEBUG:boto:Data: 2022-09-02T05:53:21.202 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.203 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.204 DEBUG:boto:Params: {} 2022-09-02T05:53:21.205 DEBUG:boto:Token: None 2022-09-02T05:53:21.205 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76 2022-09-02T05:53:21.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sAjnLsPwQV051hEEAnaLxZ6X9fM= 2022-09-02T05:53:21.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sAjnLsPwQV051hEEAnaLxZ6X9fM=', 'Content-Length': '0'} 2022-09-02T05:53:21.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f2049e5a5485a1-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.210 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.210 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.211 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.211 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:21.212 DEBUG:boto:Data: 2022-09-02T05:53:21.212 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.213 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.213 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.214 DEBUG:boto:Params: {} 2022-09-02T05:53:21.215 DEBUG:boto:Token: None 2022-09-02T05:53:21.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76 2022-09-02T05:53:21.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sAjnLsPwQV051hEEAnaLxZ6X9fM= 2022-09-02T05:53:21.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sAjnLsPwQV051hEEAnaLxZ6X9fM=', 'Content-Length': '0'} 2022-09-02T05:53:21.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ccbb1312e8dfc68-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:53:21.220 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.221 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.221 DEBUG:boto:Method: GET 2022-09-02T05:53:21.222 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:21.223 DEBUG:boto:Data: 2022-09-02T05:53:21.223 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.224 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.225 DEBUG:boto:Params: {} 2022-09-02T05:53:21.226 DEBUG:boto:Token: None 2022-09-02T05:53:21.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76 2022-09-02T05:53:21.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEleVBpW8Dgd3kPAfJKnRTlQS4I= 2022-09-02T05:53:21.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEleVBpW8Dgd3kPAfJKnRTlQS4I=', 'Content-Length': '0'} 2022-09-02T05:53:21.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000806c868fa83ca340-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.231 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:53:21.232 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:53:21.232 DEBUG:boto:Method: GET 2022-09-02T05:53:21.233 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:53:21.233 DEBUG:boto:Data: 2022-09-02T05:53:21.234 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.236 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.237 DEBUG:boto:Params: {} 2022-09-02T05:53:21.238 DEBUG:boto:Token: None 2022-09-02T05:53:21.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj76 2022-09-02T05:53:21.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEleVBpW8Dgd3kPAfJKnRTlQS4I= 2022-09-02T05:53:21.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEleVBpW8Dgd3kPAfJKnRTlQS4I=', 'Content-Length': '0'} 2022-09-02T05:53:21.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460af20a59066354-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.286 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.287 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.287 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.288 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.288 DEBUG:boto:Method: GET 2022-09-02T05:53:21.289 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.289 DEBUG:boto:Data: 2022-09-02T05:53:21.290 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.291 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.292 DEBUG:boto:Params: {} 2022-09-02T05:53:21.292 DEBUG:boto:Token: None 2022-09-02T05:53:21.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588afe8cb8711d2d-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.297 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.298 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.299 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.299 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.300 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.300 DEBUG:boto:Method: GET 2022-09-02T05:53:21.301 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:53:21.301 DEBUG:boto:Data: 2022-09-02T05:53:21.302 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.303 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.303 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.304 DEBUG:boto:Params: {} 2022-09-02T05:53:21.304 DEBUG:boto:Token: None 2022-09-02T05:53:21.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095226734b6c458d6-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.310 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:21.311 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.311 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.312 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.312 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.313 DEBUG:boto:Method: GET 2022-09-02T05:53:21.314 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.314 DEBUG:boto:Data: 2022-09-02T05:53:21.315 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.316 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.316 DEBUG:boto:Params: {} 2022-09-02T05:53:21.317 DEBUG:boto:Token: None 2022-09-02T05:53:21.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE= 2022-09-02T05:53:21.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE=', 'Content-Length': '0'} 2022-09-02T05:53:21.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c36523f33cc0c82-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.322 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.322 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.323 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.323 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.324 DEBUG:boto:Method: GET 2022-09-02T05:53:21.324 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.325 DEBUG:boto:Data: 2022-09-02T05:53:21.326 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.327 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.327 DEBUG:boto:Params: {} 2022-09-02T05:53:21.328 DEBUG:boto:Token: None 2022-09-02T05:53:21.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE= 2022-09-02T05:53:21.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE=', 'Content-Length': '0'} 2022-09-02T05:53:21.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000790156042968ef2a-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.333 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.333 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.334 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.334 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.335 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.335 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.336 DEBUG:boto:Data: 2022-09-02T05:53:21.336 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.337 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.338 DEBUG:boto:Params: {} 2022-09-02T05:53:21.338 DEBUG:boto:Token: None 2022-09-02T05:53:21.339 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.340 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7+P5TREAMDDAfEHvwaWThxhnoo= 2022-09-02T05:53:21.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7+P5TREAMDDAfEHvwaWThxhnoo=', 'Content-Length': '0'} 2022-09-02T05:53:21.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002cd8901fc85e51a-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.343 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.344 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.344 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.345 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.345 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.346 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.346 DEBUG:boto:Data: 2022-09-02T05:53:21.347 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.347 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.348 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.348 DEBUG:boto:Params: {} 2022-09-02T05:53:21.349 DEBUG:boto:Token: None 2022-09-02T05:53:21.349 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h7+P5TREAMDDAfEHvwaWThxhnoo= 2022-09-02T05:53:21.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h7+P5TREAMDDAfEHvwaWThxhnoo=', 'Content-Length': '0'} 2022-09-02T05:53:21.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee28ce60396a8e2c-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:21.354 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.354 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.355 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.355 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.356 DEBUG:boto:Method: GET 2022-09-02T05:53:21.356 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.357 DEBUG:boto:Data: 2022-09-02T05:53:21.358 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.359 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.359 DEBUG:boto:Params: {} 2022-09-02T05:53:21.360 DEBUG:boto:Token: None 2022-09-02T05:53:21.360 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE= 2022-09-02T05:53:21.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE=', 'Content-Length': '0'} 2022-09-02T05:53:21.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715dc99143437516-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.410 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.412 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.413 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.413 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.414 DEBUG:boto:Method: GET 2022-09-02T05:53:21.414 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.415 DEBUG:boto:Data: 2022-09-02T05:53:21.415 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.416 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.417 DEBUG:boto:Params: {} 2022-09-02T05:53:21.418 DEBUG:boto:Token: None 2022-09-02T05:53:21.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:53:21.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE= 2022-09-02T05:53:21.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o2RgCKEcJq5TuDFTiTJBkjyHWSE=', 'Content-Length': '0'} 2022-09-02T05:53:21.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a2ce265a8fb0af-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.423 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.423 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.424 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.424 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.425 DEBUG:boto:Method: GET 2022-09-02T05:53:21.426 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.426 DEBUG:boto:Data: 2022-09-02T05:53:21.427 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.428 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.428 DEBUG:boto:Params: {} 2022-09-02T05:53:21.429 DEBUG:boto:Token: None 2022-09-02T05:53:21.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000714407de0db29c98-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.434 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.435 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.436 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.437 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.438 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.438 DEBUG:boto:Method: GET 2022-09-02T05:53:21.439 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.439 DEBUG:boto:Data: 2022-09-02T05:53:21.440 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.441 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.441 DEBUG:boto:Params: {} 2022-09-02T05:53:21.442 DEBUG:boto:Token: None 2022-09-02T05:53:21.443 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c80b9507c0f323-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.447 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.448 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.448 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.449 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.449 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:21.450 DEBUG:boto:Data: 2022-09-02T05:53:21.451 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.452 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.452 DEBUG:boto:Params: {} 2022-09-02T05:53:21.453 DEBUG:boto:Token: None 2022-09-02T05:53:21.453 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85 2022-09-02T05:53:21.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZOoOWv2aPL+mRfAqMLlCMixKiJA= 2022-09-02T05:53:21.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZOoOWv2aPL+mRfAqMLlCMixKiJA=', 'Content-Length': '0'} 2022-09-02T05:53:21.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80114cb3df7f5d9-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.457 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.458 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.459 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.459 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:21.460 DEBUG:boto:Data: 2022-09-02T05:53:21.461 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.463 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.464 DEBUG:boto:Params: {} 2022-09-02T05:53:21.465 DEBUG:boto:Token: None 2022-09-02T05:53:21.466 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85 2022-09-02T05:53:21.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZOoOWv2aPL+mRfAqMLlCMixKiJA= 2022-09-02T05:53:21.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZOoOWv2aPL+mRfAqMLlCMixKiJA=', 'Content-Length': '0'} 2022-09-02T05:53:21.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005270f2caa0e52931-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:53:21.471 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.471 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.472 DEBUG:boto:Method: GET 2022-09-02T05:53:21.472 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:21.473 DEBUG:boto:Data: 2022-09-02T05:53:21.473 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.474 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.475 DEBUG:boto:Params: {} 2022-09-02T05:53:21.475 DEBUG:boto:Token: None 2022-09-02T05:53:21.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85 2022-09-02T05:53:21.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ax1ytgcZgq+H55YH1XmYf/7LFAA= 2022-09-02T05:53:21.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ax1ytgcZgq+H55YH1XmYf/7LFAA=', 'Content-Length': '0'} 2022-09-02T05:53:21.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000161c6a13ee7f3e1c-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.481 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:53:21.481 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:53:21.482 DEBUG:boto:Method: GET 2022-09-02T05:53:21.483 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:53:21.483 DEBUG:boto:Data: 2022-09-02T05:53:21.484 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.485 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.485 DEBUG:boto:Params: {} 2022-09-02T05:53:21.486 DEBUG:boto:Token: None 2022-09-02T05:53:21.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj85 2022-09-02T05:53:21.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ax1ytgcZgq+H55YH1XmYf/7LFAA= 2022-09-02T05:53:21.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ax1ytgcZgq+H55YH1XmYf/7LFAA=', 'Content-Length': '0'} 2022-09-02T05:53:21.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ab1ed1b63f2329-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.534 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.535 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.536 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.536 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.537 DEBUG:boto:Method: GET 2022-09-02T05:53:21.537 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.538 DEBUG:boto:Data: 2022-09-02T05:53:21.539 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.540 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.540 DEBUG:boto:Params: {} 2022-09-02T05:53:21.541 DEBUG:boto:Token: None 2022-09-02T05:53:21.542 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525c58388fd7c4e8-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.547 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.547 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.548 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.549 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.549 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.550 DEBUG:boto:Method: GET 2022-09-02T05:53:21.551 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:53:21.551 DEBUG:boto:Data: 2022-09-02T05:53:21.552 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.553 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.553 DEBUG:boto:Params: {} 2022-09-02T05:53:21.554 DEBUG:boto:Token: None 2022-09-02T05:53:21.554 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c0eee9c606bbf3a-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.559 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:21.561 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.561 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.562 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.563 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.563 DEBUG:boto:Method: GET 2022-09-02T05:53:21.564 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.564 DEBUG:boto:Data: 2022-09-02T05:53:21.565 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.566 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.567 DEBUG:boto:Params: {} 2022-09-02T05:53:21.567 DEBUG:boto:Token: None 2022-09-02T05:53:21.568 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04= 2022-09-02T05:53:21.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04=', 'Content-Length': '0'} 2022-09-02T05:53:21.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008409435c6827df91-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.572 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.573 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.574 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.574 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.575 DEBUG:boto:Method: GET 2022-09-02T05:53:21.575 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.576 DEBUG:boto:Data: 2022-09-02T05:53:21.577 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.577 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.578 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.578 DEBUG:boto:Params: {} 2022-09-02T05:53:21.579 DEBUG:boto:Token: None 2022-09-02T05:53:21.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04= 2022-09-02T05:53:21.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04=', 'Content-Length': '0'} 2022-09-02T05:53:21.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056a9c32cebf3d17f-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.584 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.585 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.585 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.586 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.586 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.587 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.587 DEBUG:boto:Data: 2022-09-02T05:53:21.588 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.589 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.590 DEBUG:boto:Params: {} 2022-09-02T05:53:21.590 DEBUG:boto:Token: None 2022-09-02T05:53:21.591 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uznDAtBg9aCV+bII0vIK28vzRSc= 2022-09-02T05:53:21.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uznDAtBg9aCV+bII0vIK28vzRSc=', 'Content-Length': '0'} 2022-09-02T05:53:21.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe27374b88dc55e-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.596 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.596 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.597 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.597 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.598 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.598 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.599 DEBUG:boto:Data: 2022-09-02T05:53:21.600 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.601 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.602 DEBUG:boto:Params: {} 2022-09-02T05:53:21.603 DEBUG:boto:Token: None 2022-09-02T05:53:21.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uznDAtBg9aCV+bII0vIK28vzRSc= 2022-09-02T05:53:21.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uznDAtBg9aCV+bII0vIK28vzRSc=', 'Content-Length': '0'} 2022-09-02T05:53:21.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9909ed9051ab770-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:21.608 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.609 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.610 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.610 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.611 DEBUG:boto:Method: GET 2022-09-02T05:53:21.612 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.613 DEBUG:boto:Data: 2022-09-02T05:53:21.614 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.616 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.617 DEBUG:boto:Params: {} 2022-09-02T05:53:21.618 DEBUG:boto:Token: None 2022-09-02T05:53:21.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04= 2022-09-02T05:53:21.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04=', 'Content-Length': '0'} 2022-09-02T05:53:21.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001259a53cad780d75-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.666 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.667 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.668 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.669 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.670 DEBUG:boto:Method: GET 2022-09-02T05:53:21.670 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.671 DEBUG:boto:Data: 2022-09-02T05:53:21.672 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.673 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.674 DEBUG:boto:Params: {} 2022-09-02T05:53:21.675 DEBUG:boto:Token: None 2022-09-02T05:53:21.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:53:21.676 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04= 2022-09-02T05:53:21.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCYUZzrltqaXcRIt4RCIYJ/Ys04=', 'Content-Length': '0'} 2022-09-02T05:53:21.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d25fa067bec4afc-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.680 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.681 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.681 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.682 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.682 DEBUG:boto:Method: GET 2022-09-02T05:53:21.683 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.683 DEBUG:boto:Data: 2022-09-02T05:53:21.684 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.685 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.685 DEBUG:boto:Params: {} 2022-09-02T05:53:21.686 DEBUG:boto:Token: None 2022-09-02T05:53:21.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac94343f3996156b-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.692 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.692 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.693 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.694 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.694 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.695 DEBUG:boto:Method: GET 2022-09-02T05:53:21.695 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.696 DEBUG:boto:Data: 2022-09-02T05:53:21.696 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.697 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.698 DEBUG:boto:Params: {} 2022-09-02T05:53:21.699 DEBUG:boto:Token: None 2022-09-02T05:53:21.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a469e7f3b86728d-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.746 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.747 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.748 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.748 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.749 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:21.749 DEBUG:boto:Data: 2022-09-02T05:53:21.750 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.751 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.751 DEBUG:boto:Params: {} 2022-09-02T05:53:21.752 DEBUG:boto:Token: None 2022-09-02T05:53:21.752 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92 2022-09-02T05:53:21.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LCCJZVFPTYiIHO1IWH43Q4xqDT8= 2022-09-02T05:53:21.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LCCJZVFPTYiIHO1IWH43Q4xqDT8=', 'Content-Length': '0'} 2022-09-02T05:53:21.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9366dee44299b8-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.757 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.758 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.759 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.760 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:21.761 DEBUG:boto:Data: 2022-09-02T05:53:21.762 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.764 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.765 DEBUG:boto:Params: {} 2022-09-02T05:53:21.766 DEBUG:boto:Token: None 2022-09-02T05:53:21.767 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92 2022-09-02T05:53:21.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LCCJZVFPTYiIHO1IWH43Q4xqDT8= 2022-09-02T05:53:21.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LCCJZVFPTYiIHO1IWH43Q4xqDT8=', 'Content-Length': '0'} 2022-09-02T05:53:21.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f10142394f44ddc-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:53:21.774 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.775 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.776 DEBUG:boto:Method: GET 2022-09-02T05:53:21.777 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:21.778 DEBUG:boto:Data: 2022-09-02T05:53:21.779 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.781 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.782 DEBUG:boto:Params: {} 2022-09-02T05:53:21.783 DEBUG:boto:Token: None 2022-09-02T05:53:21.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92 2022-09-02T05:53:21.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QLH+uRmfm09Sx77jB9Gs9YdLYAs= 2022-09-02T05:53:21.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QLH+uRmfm09Sx77jB9Gs9YdLYAs=', 'Content-Length': '0'} 2022-09-02T05:53:21.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996c2783317e1f6d-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.788 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:53:21.789 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:53:21.790 DEBUG:boto:Method: GET 2022-09-02T05:53:21.791 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:53:21.792 DEBUG:boto:Data: 2022-09-02T05:53:21.792 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.793 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.794 DEBUG:boto:Params: {} 2022-09-02T05:53:21.794 DEBUG:boto:Token: None 2022-09-02T05:53:21.795 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj92 2022-09-02T05:53:21.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QLH+uRmfm09Sx77jB9Gs9YdLYAs= 2022-09-02T05:53:21.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QLH+uRmfm09Sx77jB9Gs9YdLYAs=', 'Content-Length': '0'} 2022-09-02T05:53:21.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d342b29fb2b2a9ca-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.800 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.800 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.801 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.801 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.802 DEBUG:boto:Method: GET 2022-09-02T05:53:21.803 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.803 DEBUG:boto:Data: 2022-09-02T05:53:21.804 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.805 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.805 DEBUG:boto:Params: {} 2022-09-02T05:53:21.806 DEBUG:boto:Token: None 2022-09-02T05:53:21.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dd3b894b6a85cc9-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.811 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.812 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.813 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.813 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.814 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.814 DEBUG:boto:Method: GET 2022-09-02T05:53:21.815 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:53:21.815 DEBUG:boto:Data: 2022-09-02T05:53:21.816 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.816 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.817 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.817 DEBUG:boto:Params: {} 2022-09-02T05:53:21.818 DEBUG:boto:Token: None 2022-09-02T05:53:21.819 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0c3790b9f9b199-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.823 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:21.824 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:21.825 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:21.826 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.826 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.827 DEBUG:boto:Method: GET 2022-09-02T05:53:21.827 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.828 DEBUG:boto:Data: 2022-09-02T05:53:21.828 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.829 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.830 DEBUG:boto:Params: {} 2022-09-02T05:53:21.831 DEBUG:boto:Token: None 2022-09-02T05:53:21.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs= 2022-09-02T05:53:21.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs=', 'Content-Length': '0'} 2022-09-02T05:53:21.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007825732464c0b988-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.836 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:21.836 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:21.837 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.837 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.838 DEBUG:boto:Method: GET 2022-09-02T05:53:21.839 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.839 DEBUG:boto:Data: 2022-09-02T05:53:21.840 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.841 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.841 DEBUG:boto:Params: {} 2022-09-02T05:53:21.842 DEBUG:boto:Token: None 2022-09-02T05:53:21.843 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs= 2022-09-02T05:53:21.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs=', 'Content-Length': '0'} 2022-09-02T05:53:21.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098174dd86394b1a1-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.886 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:21.887 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:21.887 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.888 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.888 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.889 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.889 DEBUG:boto:Data: 2022-09-02T05:53:21.890 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.891 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.892 DEBUG:boto:Params: {} 2022-09-02T05:53:21.892 DEBUG:boto:Token: None 2022-09-02T05:53:21.893 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4pSLlrIYTIkM1b2g7jBqWacYPtY= 2022-09-02T05:53:21.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4pSLlrIYTIkM1b2g7jBqWacYPtY=', 'Content-Length': '0'} 2022-09-02T05:53:21.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc0cd5182cdd74a-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.897 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:21.897 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:21.898 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.898 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.899 DEBUG:boto:Method: HEAD 2022-09-02T05:53:21.899 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.900 DEBUG:boto:Data: 2022-09-02T05:53:21.900 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.901 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.901 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.902 DEBUG:boto:Params: {} 2022-09-02T05:53:21.903 DEBUG:boto:Token: None 2022-09-02T05:53:21.903 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4pSLlrIYTIkM1b2g7jBqWacYPtY= 2022-09-02T05:53:21.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4pSLlrIYTIkM1b2g7jBqWacYPtY=', 'Content-Length': '0'} 2022-09-02T05:53:21.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003708c32e6469f2b0-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:21.908 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:21.908 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:21.909 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.909 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.910 DEBUG:boto:Method: GET 2022-09-02T05:53:21.911 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.911 DEBUG:boto:Data: 2022-09-02T05:53:21.912 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.913 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.914 DEBUG:boto:Params: {} 2022-09-02T05:53:21.914 DEBUG:boto:Token: None 2022-09-02T05:53:21.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs= 2022-09-02T05:53:21.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs=', 'Content-Length': '0'} 2022-09-02T05:53:21.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf439782e03c241-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.919 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:21.920 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:21.920 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.921 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.922 DEBUG:boto:Method: GET 2022-09-02T05:53:21.922 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.923 DEBUG:boto:Data: 2022-09-02T05:53:21.923 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.924 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.925 DEBUG:boto:Params: {} 2022-09-02T05:53:21.926 DEBUG:boto:Token: None 2022-09-02T05:53:21.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:53:21.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs= 2022-09-02T05:53:21.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N8gTeeYBeTPJR+fJTffCbOXQHVs=', 'Content-Length': '0'} 2022-09-02T05:53:21.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812ed403ce09638f-0063119a51-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.930 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.931 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.932 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:21.932 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:21.933 DEBUG:boto:Method: GET 2022-09-02T05:53:21.933 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:21.934 DEBUG:boto:Data: 2022-09-02T05:53:21.935 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:21.936 DEBUG:boto:Port: 8001 2022-09-02T05:53:21.936 DEBUG:boto:Params: {} 2022-09-02T05:53:21.937 DEBUG:boto:Token: None 2022-09-02T05:53:21.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:21.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fdbaee8e00395be-0063119a51-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:21.982 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:21.984 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:21.985 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:21.986 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:21.986 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:21.987 DEBUG:boto:Method: GET 2022-09-02T05:53:21.988 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:21.989 DEBUG:boto:Data: 2022-09-02T05:53:21.990 DEBUG:boto:Headers: {} 2022-09-02T05:53:21.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:21.991 DEBUG:boto:Port: 8000 2022-09-02T05:53:21.992 DEBUG:boto:Params: {} 2022-09-02T05:53:21.993 DEBUG:boto:Token: None 2022-09-02T05:53:21.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:21 GMT /xljbkf-74/ 2022-09-02T05:53:21.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw= 2022-09-02T05:53:21.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:21 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RYyjjtcEv4/yuIE7bIZab0S/ZCw=', 'Content-Length': '0'} 2022-09-02T05:53:22.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbdedb6c78b75478-0063119a51-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:21 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.001 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:22.003 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:22.004 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:22.005 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.006 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:22.007 DEBUG:boto:Data: 2022-09-02T05:53:22.008 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:22.010 DEBUG:boto:Port: 8001 2022-09-02T05:53:22.011 DEBUG:boto:Params: {} 2022-09-02T05:53:22.012 DEBUG:boto:Token: None 2022-09-02T05:53:22.013 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj95 2022-09-02T05:53:22.014 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EleRvVDPfd0cWYoV8PhRe4YbukY= 2022-09-02T05:53:22.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EleRvVDPfd0cWYoV8PhRe4YbukY=', 'Content-Length': '0'} 2022-09-02T05:53:22.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c63fcbc1f3d4bd3-0063119a52-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.020 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:22.020 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:22.021 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.022 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:22.023 DEBUG:boto:Data: 2022-09-02T05:53:22.024 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.025 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:22.026 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.027 DEBUG:boto:Params: {} 2022-09-02T05:53:22.028 DEBUG:boto:Token: None 2022-09-02T05:53:22.029 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj95 2022-09-02T05:53:22.030 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EleRvVDPfd0cWYoV8PhRe4YbukY= 2022-09-02T05:53:22.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EleRvVDPfd0cWYoV8PhRe4YbukY=', 'Content-Length': '0'} 2022-09-02T05:53:22.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b77b3bc6193862a0-0063119a52-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:53:22.034 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:22.035 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:22.036 DEBUG:boto:Method: GET 2022-09-02T05:53:22.036 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:22.037 DEBUG:boto:Data: 2022-09-02T05:53:22.037 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:22.038 DEBUG:boto:Port: 8001 2022-09-02T05:53:22.039 DEBUG:boto:Params: {} 2022-09-02T05:53:22.039 DEBUG:boto:Token: None 2022-09-02T05:53:22.040 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj95 2022-09-02T05:53:22.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IZCU4gJtXtcOMNz129qWC/C9YOM= 2022-09-02T05:53:22.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IZCU4gJtXtcOMNz129qWC/C9YOM=', 'Content-Length': '0'} 2022-09-02T05:53:22.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005788f580e3395aac-0063119a52-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.044 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:53:22.045 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:53:22.045 DEBUG:boto:Method: GET 2022-09-02T05:53:22.046 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:53:22.047 DEBUG:boto:Data: 2022-09-02T05:53:22.047 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.048 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:22.048 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.049 DEBUG:boto:Params: {} 2022-09-02T05:53:22.050 DEBUG:boto:Token: None 2022-09-02T05:53:22.050 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj95 2022-09-02T05:53:22.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IZCU4gJtXtcOMNz129qWC/C9YOM= 2022-09-02T05:53:22.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IZCU4gJtXtcOMNz129qWC/C9YOM=', 'Content-Length': '0'} 2022-09-02T05:53:22.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0a63da1419a89a2-0063119a52-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.099 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.100 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.100 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:22.101 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:22.101 DEBUG:boto:Method: GET 2022-09-02T05:53:22.102 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:22.102 DEBUG:boto:Data: 2022-09-02T05:53:22.103 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:53:22.104 DEBUG:boto:Port: 8001 2022-09-02T05:53:22.104 DEBUG:boto:Params: {} 2022-09-02T05:53:22.105 DEBUG:boto:Token: None 2022-09-02T05:53:22.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AUkyNQ1TJbHjXDZZNz2Ln/m0yzA= 2022-09-02T05:53:22.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AUkyNQ1TJbHjXDZZNz2Ln/m0yzA=', 'Content-Length': '0'} 2022-09-02T05:53:22.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c51abba7a804ad-0063119a52-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.115 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:22.116 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.116 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.117 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:22.118 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:53:22.118 DEBUG:boto:Method: GET 2022-09-02T05:53:22.119 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:53:22.120 DEBUG:boto:Data: 2022-09-02T05:53:22.121 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:53:22.122 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.122 DEBUG:boto:Params: {} 2022-09-02T05:53:22.123 DEBUG:boto:Token: None 2022-09-02T05:53:22.124 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AUkyNQ1TJbHjXDZZNz2Ln/m0yzA= 2022-09-02T05:53:22.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AUkyNQ1TJbHjXDZZNz2Ln/m0yzA=', 'Content-Length': '0'} 2022-09-02T05:53:22.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e2ce8b0740f5e2f-0063119a52-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.131 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:53:22.132 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:53:22.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:53:22.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:53:22.134 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:53:22.195 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:53:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:53:22.280 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:53:22.281 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-74 2022-09-02T05:53:22.290 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2022-09-02T05:53:22.351 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.354 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.355 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.356 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:22.357 DEBUG:boto:Data: 2022-09-02T05:53:22.357 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.359 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.360 DEBUG:boto:Params: {} 2022-09-02T05:53:22.361 DEBUG:boto:Token: None 2022-09-02T05:53:22.362 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0= 2022-09-02T05:53:22.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0=', 'Content-Length': '0'} 2022-09-02T05:53:22.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4e7c83646773e8a-0063119a52-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.370 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:53:22.371 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:53:22.372 DEBUG:boto:Method: PUT 2022-09-02T05:53:22.373 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:53:22.374 DEBUG:boto:Data: 2022-09-02T05:53:22.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:53:22.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.376 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.377 DEBUG:boto:Params: {} 2022-09-02T05:53:22.378 DEBUG:boto:Token: None 2022-09-02T05:53:22.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj29 2022-09-02T05:53:22.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lUg8RC1RkBrPNuochv0NE6byXlk= 2022-09-02T05:53:22.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lUg8RC1RkBrPNuochv0NE6byXlk='} 2022-09-02T05:53:22.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f35ee32467ad21d2-0063119a52-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.391 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.392 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.393 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.394 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:22.395 DEBUG:boto:Data: 2022-09-02T05:53:22.395 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.396 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.397 DEBUG:boto:Params: {} 2022-09-02T05:53:22.398 DEBUG:boto:Token: None 2022-09-02T05:53:22.399 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0= 2022-09-02T05:53:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0=', 'Content-Length': '0'} 2022-09-02T05:53:22.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e88d4fe388bdbfa-0063119a52-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.405 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:53:22.405 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:53:22.406 DEBUG:boto:Method: PUT 2022-09-02T05:53:22.407 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:53:22.407 DEBUG:boto:Data: 2022-09-02T05:53:22.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:53:22.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.409 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.410 DEBUG:boto:Params: {} 2022-09-02T05:53:22.410 DEBUG:boto:Token: None 2022-09-02T05:53:22.411 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj58 2022-09-02T05:53:22.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zSCi2po9kWjpH4KQoR73pkz6lQo= 2022-09-02T05:53:22.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zSCi2po9kWjpH4KQoR73pkz6lQo='} 2022-09-02T05:53:22.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042a4e1a9a3244273-0063119a52-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.459 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.460 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.460 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.461 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:22.461 DEBUG:boto:Data: 2022-09-02T05:53:22.462 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.463 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.463 DEBUG:boto:Params: {} 2022-09-02T05:53:22.464 DEBUG:boto:Token: None 2022-09-02T05:53:22.464 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0= 2022-09-02T05:53:22.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0=', 'Content-Length': '0'} 2022-09-02T05:53:22.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ab3e154044ee462-0063119a52-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.470 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:53:22.471 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:53:22.471 DEBUG:boto:Method: PUT 2022-09-02T05:53:22.472 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:53:22.472 DEBUG:boto:Data: 2022-09-02T05:53:22.473 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:53:22.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.474 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.474 DEBUG:boto:Params: {} 2022-09-02T05:53:22.475 DEBUG:boto:Token: None 2022-09-02T05:53:22.476 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj87 2022-09-02T05:53:22.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XSuUouCedt5+Ewsql+WX5SgrHaw= 2022-09-02T05:53:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XSuUouCedt5+Ewsql+WX5SgrHaw='} 2022-09-02T05:53:22.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffffb3eed90128cf-0063119a52-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.484 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.485 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.485 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.486 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:22.486 DEBUG:boto:Data: 2022-09-02T05:53:22.487 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.488 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.488 DEBUG:boto:Params: {} 2022-09-02T05:53:22.489 DEBUG:boto:Token: None 2022-09-02T05:53:22.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0= 2022-09-02T05:53:22.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0=', 'Content-Length': '0'} 2022-09-02T05:53:22.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd70c294762e1286-0063119a52-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.495 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:53:22.495 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:53:22.496 DEBUG:boto:Method: PUT 2022-09-02T05:53:22.496 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:53:22.497 DEBUG:boto:Data: 2022-09-02T05:53:22.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:53:22.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.499 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.499 DEBUG:boto:Params: {} 2022-09-02T05:53:22.500 DEBUG:boto:Token: None 2022-09-02T05:53:22.500 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj116 2022-09-02T05:53:22.501 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dkgEGeBHY3AM8cbsQXZwt+UwxT4= 2022-09-02T05:53:22.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dkgEGeBHY3AM8cbsQXZwt+UwxT4='} 2022-09-02T05:53:22.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005def0edb7bb604e5-0063119a52-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.515 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:53:22.516 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:53:22.516 DEBUG:boto:Method: HEAD 2022-09-02T05:53:22.517 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:53:22.517 DEBUG:boto:Data: 2022-09-02T05:53:22.518 DEBUG:boto:Headers: {} 2022-09-02T05:53:22.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.519 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.520 DEBUG:boto:Params: {} 2022-09-02T05:53:22.520 DEBUG:boto:Token: None 2022-09-02T05:53:22.521 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/ 2022-09-02T05:53:22.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0= 2022-09-02T05:53:22.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dDk0qM+3x7H7L+OxpGhKbbb2yj0=', 'Content-Length': '0'} 2022-09-02T05:53:22.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c493778d48d10fe1-0063119a52-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.526 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:53:22.527 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:53:22.528 DEBUG:boto:Method: PUT 2022-09-02T05:53:22.528 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:53:22.529 DEBUG:boto:Data: 2022-09-02T05:53:22.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:53:22.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:53:22.530 DEBUG:boto:Port: 8000 2022-09-02T05:53:22.531 DEBUG:boto:Params: {} 2022-09-02T05:53:22.531 DEBUG:boto:Token: None 2022-09-02T05:53:22.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:53:22 GMT /xljbkf-74/obj145 2022-09-02T05:53:22.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ESd87GHbSucF+qJYXjwGwr3m8M= 2022-09-02T05:53:22.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:53:22 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ESd87GHbSucF+qJYXjwGwr3m8M='} 2022-09-02T05:53:22.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043e2d081b043d629-0063119a52-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:53:22.579 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:53:22.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:22.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:53:22.581 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:53:22.647 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:53:22.648 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:53:22.709 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:22.708+0000 7f6e6c050600 1 waiting to reach latest gen 4, on 3.. 2022-09-02T05:53:27.712 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:27.711+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:27.712 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:27.713 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:53:32.714 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:32.713+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:32.715 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:32.716 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:53:37.716 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:37.716+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:37.717 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:37.717 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:53:42.719 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:42.718+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:42.719 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:42.720 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:53:47.721 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:47.719+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:47.721 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:47.722 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:53:52.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:52.721+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:52.723 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:52.723 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:53:57.725 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:53:57.723+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:53:57.725 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:53:57.726 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:02.727 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:02.726+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:02.728 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:54:02.728 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:07.728+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:54:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:12.732 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:12.730+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:12.732 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:54:12.732 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:17.735 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:17.733+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:17.735 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:54:17.736 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:22.737 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:22.736+0000 7f6e6c050600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:22.738 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:54:22.738 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:27.741 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:27.738+0000 7f6e6c050600 1 bucket sync caught up with source: 2022-09-02T05:54:27.741 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:27.742 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:27.742 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:27.738+0000 7f6e6c050600 0 bucket checkpoint complete 2022-09-02T05:54:27.763 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:54:27.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:54:27.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:54:27.767 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:54:27.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:54:27.833 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:54:27.878 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:54:27.877+0000 7fa901a84600 1 bucket sync caught up with source: 2022-09-02T05:54:27.879 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:27.880 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:27.880 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:54:27.877+0000 7fa901a84600 0 bucket checkpoint complete 2022-09-02T05:54:27.901 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:54:27.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:54:27.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:54:27.903 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:54:27.966 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:54:27.967 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:54:28.005 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:28.003+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:28.005 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:28.006 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:33.007 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:33.005+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:33.008 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:33.008 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:38.009 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:38.008+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:38.010 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:38.011 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:43.011 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:43.011+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:43.012 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:43.012 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:48.014 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:48.013+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:48.014 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:48.015 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:53.015 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:53.015+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:53.015 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:53.016 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:58.017 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:54:58.016+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:54:58.018 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , , , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:54:58.018 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:03.020 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:03.018+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:03.020 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:03.021 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:08.022 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:08.020+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:08.022 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:08.023 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:13.023 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:13.022+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:13.024 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:13.024 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:18.026 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:18.024+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:18.026 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:18.026 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:23.028 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:23.027+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:23.028 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:23.029 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:28.030 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:28.029+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:28.031 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:28.032 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:33.033 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:33.031+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:33.033 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:33.034 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:38.035 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:38.033+0000 7f7ea9992600 1 waiting for incremental sync to catch up: 2022-09-02T05:55:38.036 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , , 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:38.036 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:43.037 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:43.036+0000 7f7ea9992600 1 bucket sync caught up with source: 2022-09-02T05:55:43.038 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:43.038 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [00000000002.167.5, 00000000001.68.5, , , , , 00000000001.48.5, , , , , , , , , , , 00000000002.70.5, 00000000001.67.5, , , , , , , , , , ] 2022-09-02T05:55:43.038 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:43.036+0000 7f7ea9992600 0 bucket checkpoint complete 2022-09-02T05:55:43.055 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:55:43.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.058 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:55:43.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:55:43.123 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:55:43.175 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:55:43.174+0000 7fd8c508a600 1 bucket sync caught up with source: 2022-09-02T05:55:43.176 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000003.595.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:55:43.176 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000003.595.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:55:43.177 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:55:43.174+0000 7fd8c508a600 0 bucket checkpoint complete 2022-09-02T05:55:43.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:55:43.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.195 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:55:43.252 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:55:43.253 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:55:43.314 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:55:43.312+0000 7fd704742600 1 bucket sync caught up with source: 2022-09-02T05:55:43.314 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000003.595.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:55:43.314 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.470.5, 00000000002.413.5, 00000000001.1063.5, 00000000003.595.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000002.544.5, 00000000005.1204.5, 00000000004.839.5] 2022-09-02T05:55:43.315 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:55:43.312+0000 7fd704742600 0 bucket checkpoint complete 2022-09-02T05:55:43.334 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:55:43.334 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:55:43.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.336 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:55:43.392 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:55:43.392 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:55:43.451 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:55:43.449+0000 7ff9477c6600 1 bucket sync caught up with source: 2022-09-02T05:55:43.451 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000003.355.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:55:43.451 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000003.355.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:55:43.452 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:55:43.449+0000 7ff9477c6600 0 bucket checkpoint complete 2022-09-02T05:55:43.468 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:55:43.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:55:43.470 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:55:43.535 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:55:43.535 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:55:43.575 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:43.574+0000 7fa14e777600 1 bucket sync caught up with source: 2022-09-02T05:55:43.576 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000003.355.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:55:43.576 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000002.92.5, 00000000002.392.5, 00000000001.524.5, 00000000003.355.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000002.200.5, 00000000005.334.5, 00000000004.528.5] 2022-09-02T05:55:43.577 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:55:43.574+0000 7fa14e777600 0 bucket checkpoint complete 2022-09-02T05:55:43.597 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:55:43.598 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:55:43.600 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.601 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.602 DEBUG:boto:Method: HEAD 2022-09-02T05:55:43.603 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:55:43.604 DEBUG:boto:Data: 2022-09-02T05:55:43.605 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.606 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.607 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.608 DEBUG:boto:Params: {} 2022-09-02T05:55:43.610 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:55:43.612 DEBUG:boto:Token: None 2022-09-02T05:55:43.613 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/ 2022-09-02T05:55:43.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l+SBwkIjfhjH3/QKGUTE66t8ms0= 2022-09-02T05:55:43.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l+SBwkIjfhjH3/QKGUTE66t8ms0=', 'Content-Length': '0'} 2022-09-02T05:55:43.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e6be755a2925220-0063119adf-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.623 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.624 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.625 DEBUG:boto:Method: HEAD 2022-09-02T05:55:43.626 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:55:43.627 DEBUG:boto:Data: 2022-09-02T05:55:43.628 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.631 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.631 DEBUG:boto:Params: {} 2022-09-02T05:55:43.633 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:55:43.634 DEBUG:boto:Token: None 2022-09-02T05:55:43.635 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/ 2022-09-02T05:55:43.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l+SBwkIjfhjH3/QKGUTE66t8ms0= 2022-09-02T05:55:43.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l+SBwkIjfhjH3/QKGUTE66t8ms0=', 'Content-Length': '0'} 2022-09-02T05:55:43.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d3dbb34eaa48b8d-0063119adf-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.643 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:55:43.644 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.645 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.645 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.646 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.646 DEBUG:boto:Method: GET 2022-09-02T05:55:43.647 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.647 DEBUG:boto:Data: 2022-09-02T05:55:43.648 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.649 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.650 DEBUG:boto:Params: {} 2022-09-02T05:55:43.650 DEBUG:boto:Token: None 2022-09-02T05:55:43.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/?versions 2022-09-02T05:55:43.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4= 2022-09-02T05:55:43.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4=', 'Content-Length': '0'} 2022-09-02T05:55:43.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d53b40a8f546e62-0063119adf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.656 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:55:43.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:55:43.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:55:43.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:55:43.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:55:43.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:55:43.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:55:43.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:55:43.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:55:43.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:55:43.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:55:43.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:55:43.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:55:43.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:55:43.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:55:43.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:55:43.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:55:43.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:55:43.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:55:43.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:55:43.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:55:43.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:55:43.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:55:43.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:55:43.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:55:43.674 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:55:43.674 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.675 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.676 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.677 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.677 DEBUG:boto:Method: GET 2022-09-02T05:55:43.678 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.679 DEBUG:boto:Data: 2022-09-02T05:55:43.679 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.680 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.681 DEBUG:boto:Params: {} 2022-09-02T05:55:43.681 DEBUG:boto:Token: None 2022-09-02T05:55:43.682 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/?versions 2022-09-02T05:55:43.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4= 2022-09-02T05:55:43.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4=', 'Content-Length': '0'} 2022-09-02T05:55:43.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edab3284ce5a6f5e-0063119adf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.688 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:55:43.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:55:43.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:55:43.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:55:43.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:55:43.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:55:43.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:55:43.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:55:43.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:55:43.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:55:43.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:55:43.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:55:43.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:55:43.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:55:43.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:55:43.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:55:43.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:55:43.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:55:43.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:55:43.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:55:43.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:55:43.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:55:43.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:55:43.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:55:43.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:55:43.703 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.703 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.704 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.704 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.705 DEBUG:boto:Method: GET 2022-09-02T05:55:43.705 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.706 DEBUG:boto:Data: 2022-09-02T05:55:43.706 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.708 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.708 DEBUG:boto:Params: {} 2022-09-02T05:55:43.709 DEBUG:boto:Token: None 2022-09-02T05:55:43.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/?versions 2022-09-02T05:55:43.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4= 2022-09-02T05:55:43.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4=', 'Content-Length': '0'} 2022-09-02T05:55:43.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e145b1e7c854f98-0063119adf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.714 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.717 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.718 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.718 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.719 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.719 DEBUG:boto:Method: GET 2022-09-02T05:55:43.720 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:43.720 DEBUG:boto:Data: 2022-09-02T05:55:43.721 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.722 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.722 DEBUG:boto:Params: {} 2022-09-02T05:55:43.723 DEBUG:boto:Token: None 2022-09-02T05:55:43.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/?versions 2022-09-02T05:55:43.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4= 2022-09-02T05:55:43.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Os0NXEQ52s16hVO5cgEgCdbJFU4=', 'Content-Length': '0'} 2022-09-02T05:55:43.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059fa02e4f1c1eb6-0063119adf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.729 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:43.732 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.732 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.733 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.733 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.734 DEBUG:boto:Method: GET 2022-09-02T05:55:43.735 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.735 DEBUG:boto:Data: 2022-09-02T05:55:43.736 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.737 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.738 DEBUG:boto:Params: {} 2022-09-02T05:55:43.738 DEBUG:boto:Token: None 2022-09-02T05:55:43.739 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg= 2022-09-02T05:55:43.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg=', 'Content-Length': '0'} 2022-09-02T05:55:43.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000604dd7f9ac06c5a5-0063119adf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.744 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.744 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.745 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.745 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.746 DEBUG:boto:Method: GET 2022-09-02T05:55:43.746 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.747 DEBUG:boto:Data: 2022-09-02T05:55:43.748 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.749 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.750 DEBUG:boto:Params: {} 2022-09-02T05:55:43.750 DEBUG:boto:Token: None 2022-09-02T05:55:43.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg= 2022-09-02T05:55:43.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg=', 'Content-Length': '0'} 2022-09-02T05:55:43.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f303d9513b3add-0063119adf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.756 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.756 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.757 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.757 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.758 DEBUG:boto:Method: HEAD 2022-09-02T05:55:43.758 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.759 DEBUG:boto:Data: 2022-09-02T05:55:43.760 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.761 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.761 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.762 DEBUG:boto:Params: {} 2022-09-02T05:55:43.762 DEBUG:boto:Token: None 2022-09-02T05:55:43.763 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/WTc5gqorEgDaPgjQkaLrtTunB0= 2022-09-02T05:55:43.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/WTc5gqorEgDaPgjQkaLrtTunB0=', 'Content-Length': '0'} 2022-09-02T05:55:43.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003430b36901e3502e-0063119adf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.767 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.767 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.768 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.769 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.770 DEBUG:boto:Method: HEAD 2022-09-02T05:55:43.770 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.771 DEBUG:boto:Data: 2022-09-02T05:55:43.771 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.772 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.773 DEBUG:boto:Params: {} 2022-09-02T05:55:43.774 DEBUG:boto:Token: None 2022-09-02T05:55:43.774 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/WTc5gqorEgDaPgjQkaLrtTunB0= 2022-09-02T05:55:43.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/WTc5gqorEgDaPgjQkaLrtTunB0=', 'Content-Length': '0'} 2022-09-02T05:55:43.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c80dd7e1e97c6b05-0063119adf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:43.779 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.780 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.780 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.781 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.782 DEBUG:boto:Method: GET 2022-09-02T05:55:43.782 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.783 DEBUG:boto:Data: 2022-09-02T05:55:43.783 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.784 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.785 DEBUG:boto:Params: {} 2022-09-02T05:55:43.785 DEBUG:boto:Token: None 2022-09-02T05:55:43.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg= 2022-09-02T05:55:43.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg=', 'Content-Length': '0'} 2022-09-02T05:55:43.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb2b7473f760409-0063119adf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.834 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.835 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.835 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.836 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.836 DEBUG:boto:Method: GET 2022-09-02T05:55:43.837 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.837 DEBUG:boto:Data: 2022-09-02T05:55:43.838 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.839 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.840 DEBUG:boto:Params: {} 2022-09-02T05:55:43.840 DEBUG:boto:Token: None 2022-09-02T05:55:43.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:43.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg= 2022-09-02T05:55:43.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2L4pf+THWnbiCe/Lc17mddRaHUg=', 'Content-Length': '0'} 2022-09-02T05:55:43.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfaff2f792344790-0063119adf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.845 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.845 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.846 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.846 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.847 DEBUG:boto:Method: GET 2022-09-02T05:55:43.848 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.849 DEBUG:boto:Data: 2022-09-02T05:55:43.849 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.850 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.851 DEBUG:boto:Params: {} 2022-09-02T05:55:43.852 DEBUG:boto:Token: None 2022-09-02T05:55:43.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/ 2022-09-02T05:55:43.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0= 2022-09-02T05:55:43.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0=', 'Content-Length': '0'} 2022-09-02T05:55:43.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6fe7b4cfeabca55-0063119adf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.857 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.858 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.859 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.860 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.860 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.861 DEBUG:boto:Method: GET 2022-09-02T05:55:43.861 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.862 DEBUG:boto:Data: 2022-09-02T05:55:43.862 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.863 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.864 DEBUG:boto:Params: {} 2022-09-02T05:55:43.865 DEBUG:boto:Token: None 2022-09-02T05:55:43.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/ 2022-09-02T05:55:43.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0= 2022-09-02T05:55:43.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0=', 'Content-Length': '0'} 2022-09-02T05:55:43.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e01a44b238d97c2-0063119adf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.871 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.872 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.873 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.873 DEBUG:boto:Method: HEAD 2022-09-02T05:55:43.874 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:43.875 DEBUG:boto:Data: 2022-09-02T05:55:43.875 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.876 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.876 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.877 DEBUG:boto:Params: {} 2022-09-02T05:55:43.877 DEBUG:boto:Token: None 2022-09-02T05:55:43.878 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11 2022-09-02T05:55:43.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lahsZOz6MQwsXy27Z/II7dHuITY= 2022-09-02T05:55:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lahsZOz6MQwsXy27Z/II7dHuITY=', 'Content-Length': '0'} 2022-09-02T05:55:43.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be7adeabef8fc35-0063119adf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.882 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.882 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.883 DEBUG:boto:Method: HEAD 2022-09-02T05:55:43.884 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:43.884 DEBUG:boto:Data: 2022-09-02T05:55:43.885 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.886 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.886 DEBUG:boto:Params: {} 2022-09-02T05:55:43.887 DEBUG:boto:Token: None 2022-09-02T05:55:43.887 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11 2022-09-02T05:55:43.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lahsZOz6MQwsXy27Z/II7dHuITY= 2022-09-02T05:55:43.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lahsZOz6MQwsXy27Z/II7dHuITY=', 'Content-Length': '0'} 2022-09-02T05:55:43.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000123e037d81b68f23-0063119adf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:43.894 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.895 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.896 DEBUG:boto:Method: GET 2022-09-02T05:55:43.897 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:43.898 DEBUG:boto:Data: 2022-09-02T05:55:43.899 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.901 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.902 DEBUG:boto:Params: {} 2022-09-02T05:55:43.903 DEBUG:boto:Token: None 2022-09-02T05:55:43.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11 2022-09-02T05:55:43.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djrqXf2F+BAiiewZc6C8jlHyAtw= 2022-09-02T05:55:43.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djrqXf2F+BAiiewZc6C8jlHyAtw=', 'Content-Length': '0'} 2022-09-02T05:55:43.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e2f1bc4d4da9fc-0063119adf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.909 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:43.910 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:43.911 DEBUG:boto:Method: GET 2022-09-02T05:55:43.911 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:43.912 DEBUG:boto:Data: 2022-09-02T05:55:43.912 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.914 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.915 DEBUG:boto:Params: {} 2022-09-02T05:55:43.915 DEBUG:boto:Token: None 2022-09-02T05:55:43.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj11 2022-09-02T05:55:43.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djrqXf2F+BAiiewZc6C8jlHyAtw= 2022-09-02T05:55:43.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djrqXf2F+BAiiewZc6C8jlHyAtw=', 'Content-Length': '0'} 2022-09-02T05:55:43.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097758cb4eefdb238-0063119adf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.962 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.963 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.963 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.964 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.964 DEBUG:boto:Method: GET 2022-09-02T05:55:43.965 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.965 DEBUG:boto:Data: 2022-09-02T05:55:43.966 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.966 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.967 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.967 DEBUG:boto:Params: {} 2022-09-02T05:55:43.968 DEBUG:boto:Token: None 2022-09-02T05:55:43.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/ 2022-09-02T05:55:43.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0= 2022-09-02T05:55:43.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0=', 'Content-Length': '0'} 2022-09-02T05:55:43.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ae8ae24189d910-0063119adf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.973 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.974 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:43.975 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:43.976 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.976 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.977 DEBUG:boto:Method: GET 2022-09-02T05:55:43.977 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:43.978 DEBUG:boto:Data: 2022-09-02T05:55:43.978 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:43.979 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.980 DEBUG:boto:Params: {} 2022-09-02T05:55:43.981 DEBUG:boto:Token: None 2022-09-02T05:55:43.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/ 2022-09-02T05:55:43.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0= 2022-09-02T05:55:43.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8rl0/obcp3+pojS5OOpkv1KOK0=', 'Content-Length': '0'} 2022-09-02T05:55:43.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f441c55cc3057090-0063119adf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.987 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:43.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:43.988 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:43.989 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:43.989 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:43.990 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:43.991 DEBUG:boto:Method: GET 2022-09-02T05:55:43.991 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:43.992 DEBUG:boto:Data: 2022-09-02T05:55:43.992 DEBUG:boto:Headers: {} 2022-09-02T05:55:43.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:43.994 DEBUG:boto:Port: 8000 2022-09-02T05:55:43.994 DEBUG:boto:Params: {} 2022-09-02T05:55:43.995 DEBUG:boto:Token: None 2022-09-02T05:55:43.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:43 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:43.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sJ1taCR0JygXlfEfgvyHj0HTDLo= 2022-09-02T05:55:43.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sJ1taCR0JygXlfEfgvyHj0HTDLo=', 'Content-Length': '0'} 2022-09-02T05:55:43.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000767981426c6469-0063119adf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:43.999 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.000 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.000 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.001 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.001 DEBUG:boto:Method: GET 2022-09-02T05:55:44.002 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.002 DEBUG:boto:Data: 2022-09-02T05:55:44.003 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.004 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.004 DEBUG:boto:Params: {} 2022-09-02T05:55:44.005 DEBUG:boto:Token: None 2022-09-02T05:55:44.006 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QhTyBhUahek6SGO1OshC0LQea4= 2022-09-02T05:55:44.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QhTyBhUahek6SGO1OshC0LQea4=', 'Content-Length': '0'} 2022-09-02T05:55:44.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2973416bf603c26-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.010 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.010 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.011 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.011 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.012 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.012 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.013 DEBUG:boto:Data: 2022-09-02T05:55:44.013 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.014 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.014 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.015 DEBUG:boto:Params: {} 2022-09-02T05:55:44.015 DEBUG:boto:Token: None 2022-09-02T05:55:44.016 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zux6kceIFmOx17iXb1+O2rB25xE= 2022-09-02T05:55:44.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zux6kceIFmOx17iXb1+O2rB25xE=', 'Content-Length': '0'} 2022-09-02T05:55:44.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec16f0aefbcee3e3-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.020 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.020 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.021 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.021 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.022 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.022 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.023 DEBUG:boto:Data: 2022-09-02T05:55:44.023 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.024 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.025 DEBUG:boto:Params: {} 2022-09-02T05:55:44.025 DEBUG:boto:Token: None 2022-09-02T05:55:44.026 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zux6kceIFmOx17iXb1+O2rB25xE= 2022-09-02T05:55:44.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zux6kceIFmOx17iXb1+O2rB25xE=', 'Content-Length': '0'} 2022-09-02T05:55:44.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91d55f9e57d8984-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:44.030 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.031 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.031 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.032 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.032 DEBUG:boto:Method: GET 2022-09-02T05:55:44.033 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.033 DEBUG:boto:Data: 2022-09-02T05:55:44.034 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.034 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.035 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.035 DEBUG:boto:Params: {} 2022-09-02T05:55:44.036 DEBUG:boto:Token: None 2022-09-02T05:55:44.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QhTyBhUahek6SGO1OshC0LQea4= 2022-09-02T05:55:44.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QhTyBhUahek6SGO1OshC0LQea4=', 'Content-Length': '0'} 2022-09-02T05:55:44.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4df07db0980b0bb-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.040 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.041 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.042 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.042 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.043 DEBUG:boto:Method: GET 2022-09-02T05:55:44.043 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.044 DEBUG:boto:Data: 2022-09-02T05:55:44.044 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.045 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.046 DEBUG:boto:Params: {} 2022-09-02T05:55:44.047 DEBUG:boto:Token: None 2022-09-02T05:55:44.047 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:44.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QhTyBhUahek6SGO1OshC0LQea4= 2022-09-02T05:55:44.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QhTyBhUahek6SGO1OshC0LQea4=', 'Content-Length': '0'} 2022-09-02T05:55:44.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d1ade7909b7b7c-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.051 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.052 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.052 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.053 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.053 DEBUG:boto:Method: GET 2022-09-02T05:55:44.054 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.055 DEBUG:boto:Data: 2022-09-02T05:55:44.055 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.056 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.057 DEBUG:boto:Params: {} 2022-09-02T05:55:44.057 DEBUG:boto:Token: None 2022-09-02T05:55:44.058 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c15ab365516ebf-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.062 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.063 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.064 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.064 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.065 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.065 DEBUG:boto:Method: GET 2022-09-02T05:55:44.066 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.066 DEBUG:boto:Data: 2022-09-02T05:55:44.067 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.068 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.068 DEBUG:boto:Params: {} 2022-09-02T05:55:44.069 DEBUG:boto:Token: None 2022-09-02T05:55:44.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000657b2d975df122fb-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.118 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.119 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.120 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.120 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.121 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:44.121 DEBUG:boto:Data: 2022-09-02T05:55:44.122 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.122 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.123 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.123 DEBUG:boto:Params: {} 2022-09-02T05:55:44.124 DEBUG:boto:Token: None 2022-09-02T05:55:44.125 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115 2022-09-02T05:55:44.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EgSL0Wr51aF3/5NIk39PZYV/O6k= 2022-09-02T05:55:44.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EgSL0Wr51aF3/5NIk39PZYV/O6k=', 'Content-Length': '0'} 2022-09-02T05:55:44.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e82e1efab753b73-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.130 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.131 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.132 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.133 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:44.134 DEBUG:boto:Data: 2022-09-02T05:55:44.135 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.136 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.137 DEBUG:boto:Params: {} 2022-09-02T05:55:44.138 DEBUG:boto:Token: None 2022-09-02T05:55:44.139 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115 2022-09-02T05:55:44.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EgSL0Wr51aF3/5NIk39PZYV/O6k= 2022-09-02T05:55:44.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EgSL0Wr51aF3/5NIk39PZYV/O6k=', 'Content-Length': '0'} 2022-09-02T05:55:44.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005cf75fb5757d329-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:44.145 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.146 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.146 DEBUG:boto:Method: GET 2022-09-02T05:55:44.147 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:44.148 DEBUG:boto:Data: 2022-09-02T05:55:44.148 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.149 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.150 DEBUG:boto:Params: {} 2022-09-02T05:55:44.150 DEBUG:boto:Token: None 2022-09-02T05:55:44.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115 2022-09-02T05:55:44.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6hJQwWn8kj0BdaCu7xNcSIQC9fE= 2022-09-02T05:55:44.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6hJQwWn8kj0BdaCu7xNcSIQC9fE=', 'Content-Length': '0'} 2022-09-02T05:55:44.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd509259943d8ae-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.155 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:44.156 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:44.156 DEBUG:boto:Method: GET 2022-09-02T05:55:44.157 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:44.158 DEBUG:boto:Data: 2022-09-02T05:55:44.158 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.159 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.160 DEBUG:boto:Params: {} 2022-09-02T05:55:44.160 DEBUG:boto:Token: None 2022-09-02T05:55:44.161 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj115 2022-09-02T05:55:44.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6hJQwWn8kj0BdaCu7xNcSIQC9fE= 2022-09-02T05:55:44.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6hJQwWn8kj0BdaCu7xNcSIQC9fE=', 'Content-Length': '0'} 2022-09-02T05:55:44.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048aa7f3bc0658ad4-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.165 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.166 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.166 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.167 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.167 DEBUG:boto:Method: GET 2022-09-02T05:55:44.168 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.169 DEBUG:boto:Data: 2022-09-02T05:55:44.169 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.170 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.170 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.171 DEBUG:boto:Params: {} 2022-09-02T05:55:44.171 DEBUG:boto:Token: None 2022-09-02T05:55:44.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b713e8ac1be4783-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.218 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.219 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.220 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.221 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.221 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.222 DEBUG:boto:Method: GET 2022-09-02T05:55:44.222 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:44.223 DEBUG:boto:Data: 2022-09-02T05:55:44.224 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.225 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.226 DEBUG:boto:Params: {} 2022-09-02T05:55:44.227 DEBUG:boto:Token: None 2022-09-02T05:55:44.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026dfd6f347b32669-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.233 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:44.235 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.236 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.237 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.237 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.238 DEBUG:boto:Method: GET 2022-09-02T05:55:44.238 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.239 DEBUG:boto:Data: 2022-09-02T05:55:44.239 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.240 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.241 DEBUG:boto:Params: {} 2022-09-02T05:55:44.242 DEBUG:boto:Token: None 2022-09-02T05:55:44.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM= 2022-09-02T05:55:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM=', 'Content-Length': '0'} 2022-09-02T05:55:44.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058acbe3d8749d7d4-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.247 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.247 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.248 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.249 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.249 DEBUG:boto:Method: GET 2022-09-02T05:55:44.250 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.251 DEBUG:boto:Data: 2022-09-02T05:55:44.252 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.253 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.254 DEBUG:boto:Params: {} 2022-09-02T05:55:44.255 DEBUG:boto:Token: None 2022-09-02T05:55:44.256 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM= 2022-09-02T05:55:44.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM=', 'Content-Length': '0'} 2022-09-02T05:55:44.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde96b949789cc2f-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.264 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.264 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.265 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.265 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.266 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.267 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.267 DEBUG:boto:Data: 2022-09-02T05:55:44.268 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.269 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.270 DEBUG:boto:Params: {} 2022-09-02T05:55:44.271 DEBUG:boto:Token: None 2022-09-02T05:55:44.272 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tS+0HcxqF35EmTl3HDt/HezunvM= 2022-09-02T05:55:44.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tS+0HcxqF35EmTl3HDt/HezunvM=', 'Content-Length': '0'} 2022-09-02T05:55:44.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b678667bdb41dc33-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.277 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.278 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.279 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.280 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.281 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.281 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.282 DEBUG:boto:Data: 2022-09-02T05:55:44.283 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.285 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.286 DEBUG:boto:Params: {} 2022-09-02T05:55:44.287 DEBUG:boto:Token: None 2022-09-02T05:55:44.287 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tS+0HcxqF35EmTl3HDt/HezunvM= 2022-09-02T05:55:44.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tS+0HcxqF35EmTl3HDt/HezunvM=', 'Content-Length': '0'} 2022-09-02T05:55:44.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3897ff7623dbb74-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:44.294 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.295 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.295 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.296 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.297 DEBUG:boto:Method: GET 2022-09-02T05:55:44.298 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.299 DEBUG:boto:Data: 2022-09-02T05:55:44.300 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.302 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.303 DEBUG:boto:Params: {} 2022-09-02T05:55:44.304 DEBUG:boto:Token: None 2022-09-02T05:55:44.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM= 2022-09-02T05:55:44.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM=', 'Content-Length': '0'} 2022-09-02T05:55:44.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4687b8afd6f939a-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.310 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.310 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.311 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.311 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.312 DEBUG:boto:Method: GET 2022-09-02T05:55:44.312 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.313 DEBUG:boto:Data: 2022-09-02T05:55:44.313 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.315 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.315 DEBUG:boto:Params: {} 2022-09-02T05:55:44.316 DEBUG:boto:Token: None 2022-09-02T05:55:44.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:44.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM= 2022-09-02T05:55:44.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fja/Nsyu3X0kGbF4NtTiV1I+jiM=', 'Content-Length': '0'} 2022-09-02T05:55:44.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e871cd5cecffb472-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.321 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.321 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.322 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.322 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.323 DEBUG:boto:Method: GET 2022-09-02T05:55:44.324 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.324 DEBUG:boto:Data: 2022-09-02T05:55:44.325 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.325 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.326 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.326 DEBUG:boto:Params: {} 2022-09-02T05:55:44.327 DEBUG:boto:Token: None 2022-09-02T05:55:44.327 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e637472b556951b4-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.332 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.333 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.333 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.334 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.334 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.335 DEBUG:boto:Method: GET 2022-09-02T05:55:44.335 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.336 DEBUG:boto:Data: 2022-09-02T05:55:44.336 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.337 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.338 DEBUG:boto:Params: {} 2022-09-02T05:55:44.339 DEBUG:boto:Token: None 2022-09-02T05:55:44.339 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.340 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b558b560d34769-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.345 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.346 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.346 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.347 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.348 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:44.348 DEBUG:boto:Data: 2022-09-02T05:55:44.349 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.350 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.350 DEBUG:boto:Params: {} 2022-09-02T05:55:44.351 DEBUG:boto:Token: None 2022-09-02T05:55:44.352 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116 2022-09-02T05:55:44.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eviWNrU03cDcD4/I3OMGaOM2Pqo= 2022-09-02T05:55:44.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eviWNrU03cDcD4/I3OMGaOM2Pqo=', 'Content-Length': '0'} 2022-09-02T05:55:44.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebabe95f183411f-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.358 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.358 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.359 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.359 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:44.360 DEBUG:boto:Data: 2022-09-02T05:55:44.361 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.362 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.362 DEBUG:boto:Params: {} 2022-09-02T05:55:44.363 DEBUG:boto:Token: None 2022-09-02T05:55:44.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116 2022-09-02T05:55:44.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eviWNrU03cDcD4/I3OMGaOM2Pqo= 2022-09-02T05:55:44.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eviWNrU03cDcD4/I3OMGaOM2Pqo=', 'Content-Length': '0'} 2022-09-02T05:55:44.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be495d028bd03c3e-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:44.368 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.369 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.369 DEBUG:boto:Method: GET 2022-09-02T05:55:44.370 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:44.371 DEBUG:boto:Data: 2022-09-02T05:55:44.371 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.373 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.374 DEBUG:boto:Params: {} 2022-09-02T05:55:44.374 DEBUG:boto:Token: None 2022-09-02T05:55:44.375 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116 2022-09-02T05:55:44.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4eesPMx/4UHxHq8+ijBP+uMx+0s= 2022-09-02T05:55:44.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4eesPMx/4UHxHq8+ijBP+uMx+0s=', 'Content-Length': '0'} 2022-09-02T05:55:44.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f697cebef6b31ad-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.422 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:44.423 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:44.423 DEBUG:boto:Method: GET 2022-09-02T05:55:44.424 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:44.425 DEBUG:boto:Data: 2022-09-02T05:55:44.425 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.426 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.427 DEBUG:boto:Params: {} 2022-09-02T05:55:44.428 DEBUG:boto:Token: None 2022-09-02T05:55:44.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj116 2022-09-02T05:55:44.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4eesPMx/4UHxHq8+ijBP+uMx+0s= 2022-09-02T05:55:44.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4eesPMx/4UHxHq8+ijBP+uMx+0s=', 'Content-Length': '0'} 2022-09-02T05:55:44.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dfb0189fc585fe1-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.433 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.434 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.435 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.436 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.437 DEBUG:boto:Method: GET 2022-09-02T05:55:44.438 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.439 DEBUG:boto:Data: 2022-09-02T05:55:44.440 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.442 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.453 DEBUG:boto:Params: {} 2022-09-02T05:55:44.454 DEBUG:boto:Token: None 2022-09-02T05:55:44.455 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f1abbbccc0454f-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.463 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.464 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.465 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.466 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.467 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.468 DEBUG:boto:Method: GET 2022-09-02T05:55:44.469 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:44.470 DEBUG:boto:Data: 2022-09-02T05:55:44.471 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.473 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.474 DEBUG:boto:Params: {} 2022-09-02T05:55:44.476 DEBUG:boto:Token: None 2022-09-02T05:55:44.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2d734f43251e558-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.485 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:44.487 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.488 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.489 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.490 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.491 DEBUG:boto:Method: GET 2022-09-02T05:55:44.492 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.493 DEBUG:boto:Data: 2022-09-02T05:55:44.494 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.496 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.497 DEBUG:boto:Params: {} 2022-09-02T05:55:44.498 DEBUG:boto:Token: None 2022-09-02T05:55:44.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g= 2022-09-02T05:55:44.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g=', 'Content-Length': '0'} 2022-09-02T05:55:44.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49875eb586cb112-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.506 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.507 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.508 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.509 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.510 DEBUG:boto:Method: GET 2022-09-02T05:55:44.511 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.512 DEBUG:boto:Data: 2022-09-02T05:55:44.513 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.515 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.516 DEBUG:boto:Params: {} 2022-09-02T05:55:44.517 DEBUG:boto:Token: None 2022-09-02T05:55:44.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g= 2022-09-02T05:55:44.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g=', 'Content-Length': '0'} 2022-09-02T05:55:44.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce18dc5e4bf772b5-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.525 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.525 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.526 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.528 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.528 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.529 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.529 DEBUG:boto:Data: 2022-09-02T05:55:44.530 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.531 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.531 DEBUG:boto:Params: {} 2022-09-02T05:55:44.532 DEBUG:boto:Token: None 2022-09-02T05:55:44.532 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jom6JzXE9Hl2VAedJVieu9T6akA= 2022-09-02T05:55:44.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jom6JzXE9Hl2VAedJVieu9T6akA=', 'Content-Length': '0'} 2022-09-02T05:55:44.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eca1565af5f7dd8-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.537 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.537 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.538 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.538 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.539 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.539 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.540 DEBUG:boto:Data: 2022-09-02T05:55:44.540 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.541 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.542 DEBUG:boto:Params: {} 2022-09-02T05:55:44.542 DEBUG:boto:Token: None 2022-09-02T05:55:44.543 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jom6JzXE9Hl2VAedJVieu9T6akA= 2022-09-02T05:55:44.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jom6JzXE9Hl2VAedJVieu9T6akA=', 'Content-Length': '0'} 2022-09-02T05:55:44.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587ee44e038349ea-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:44.547 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.548 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.549 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.549 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.550 DEBUG:boto:Method: GET 2022-09-02T05:55:44.550 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.551 DEBUG:boto:Data: 2022-09-02T05:55:44.551 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.553 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.553 DEBUG:boto:Params: {} 2022-09-02T05:55:44.554 DEBUG:boto:Token: None 2022-09-02T05:55:44.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g= 2022-09-02T05:55:44.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g=', 'Content-Length': '0'} 2022-09-02T05:55:44.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038d8a3654a3ce972-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.559 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.559 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.560 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.560 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.561 DEBUG:boto:Method: GET 2022-09-02T05:55:44.561 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.562 DEBUG:boto:Data: 2022-09-02T05:55:44.562 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.564 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.564 DEBUG:boto:Params: {} 2022-09-02T05:55:44.565 DEBUG:boto:Token: None 2022-09-02T05:55:44.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:44.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g= 2022-09-02T05:55:44.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aReEPILYZcqQruIyXgkH2/MDG9g=', 'Content-Length': '0'} 2022-09-02T05:55:44.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab4bfb5365bb532-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.618 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.620 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.621 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.622 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.623 DEBUG:boto:Method: GET 2022-09-02T05:55:44.624 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.625 DEBUG:boto:Data: 2022-09-02T05:55:44.625 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.628 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.629 DEBUG:boto:Params: {} 2022-09-02T05:55:44.630 DEBUG:boto:Token: None 2022-09-02T05:55:44.631 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008982ba4f2522a06d-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.638 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.639 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.640 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.642 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.643 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.644 DEBUG:boto:Method: GET 2022-09-02T05:55:44.645 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.645 DEBUG:boto:Data: 2022-09-02T05:55:44.647 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.649 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.650 DEBUG:boto:Params: {} 2022-09-02T05:55:44.651 DEBUG:boto:Token: None 2022-09-02T05:55:44.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bac43f8ae9414bf1-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.660 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.661 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.662 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.664 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.665 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:44.666 DEBUG:boto:Data: 2022-09-02T05:55:44.667 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.668 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.669 DEBUG:boto:Params: {} 2022-09-02T05:55:44.671 DEBUG:boto:Token: None 2022-09-02T05:55:44.672 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145 2022-09-02T05:55:44.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vIVY3/acH2lr4qWXBA/jg1FLWw4= 2022-09-02T05:55:44.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vIVY3/acH2lr4qWXBA/jg1FLWw4=', 'Content-Length': '0'} 2022-09-02T05:55:44.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a47f9a0158b3bd35-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.678 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.679 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.680 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.681 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:44.682 DEBUG:boto:Data: 2022-09-02T05:55:44.683 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.685 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.686 DEBUG:boto:Params: {} 2022-09-02T05:55:44.687 DEBUG:boto:Token: None 2022-09-02T05:55:44.688 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145 2022-09-02T05:55:44.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vIVY3/acH2lr4qWXBA/jg1FLWw4= 2022-09-02T05:55:44.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vIVY3/acH2lr4qWXBA/jg1FLWw4=', 'Content-Length': '0'} 2022-09-02T05:55:44.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cafccb0c08fd362b-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:44.696 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.696 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.697 DEBUG:boto:Method: GET 2022-09-02T05:55:44.697 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:44.698 DEBUG:boto:Data: 2022-09-02T05:55:44.698 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.700 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.700 DEBUG:boto:Params: {} 2022-09-02T05:55:44.701 DEBUG:boto:Token: None 2022-09-02T05:55:44.702 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145 2022-09-02T05:55:44.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xSz5ks8uJNbmwk6CTU54ftYQySU= 2022-09-02T05:55:44.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xSz5ks8uJNbmwk6CTU54ftYQySU=', 'Content-Length': '0'} 2022-09-02T05:55:44.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000884fb642b1510e15-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.706 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:44.707 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:44.707 DEBUG:boto:Method: GET 2022-09-02T05:55:44.708 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:44.709 DEBUG:boto:Data: 2022-09-02T05:55:44.710 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.711 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.711 DEBUG:boto:Params: {} 2022-09-02T05:55:44.712 DEBUG:boto:Token: None 2022-09-02T05:55:44.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj145 2022-09-02T05:55:44.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xSz5ks8uJNbmwk6CTU54ftYQySU= 2022-09-02T05:55:44.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xSz5ks8uJNbmwk6CTU54ftYQySU=', 'Content-Length': '0'} 2022-09-02T05:55:44.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000271c035ea3d341e-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.717 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.717 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.718 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.719 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.719 DEBUG:boto:Method: GET 2022-09-02T05:55:44.720 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.720 DEBUG:boto:Data: 2022-09-02T05:55:44.721 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.722 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.722 DEBUG:boto:Params: {} 2022-09-02T05:55:44.723 DEBUG:boto:Token: None 2022-09-02T05:55:44.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea957905cc268ed8-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.728 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.729 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.730 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.730 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.731 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.731 DEBUG:boto:Method: GET 2022-09-02T05:55:44.732 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:44.732 DEBUG:boto:Data: 2022-09-02T05:55:44.733 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.734 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.735 DEBUG:boto:Params: {} 2022-09-02T05:55:44.735 DEBUG:boto:Token: None 2022-09-02T05:55:44.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e298c857aed5cdd-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.742 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:44.743 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.744 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.745 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.746 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.746 DEBUG:boto:Method: GET 2022-09-02T05:55:44.747 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.747 DEBUG:boto:Data: 2022-09-02T05:55:44.748 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.750 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.750 DEBUG:boto:Params: {} 2022-09-02T05:55:44.751 DEBUG:boto:Token: None 2022-09-02T05:55:44.752 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.753 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU= 2022-09-02T05:55:44.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU=', 'Content-Length': '0'} 2022-09-02T05:55:44.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9812e3b5a0914c-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.757 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.757 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.758 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.758 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.759 DEBUG:boto:Method: GET 2022-09-02T05:55:44.759 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.760 DEBUG:boto:Data: 2022-09-02T05:55:44.760 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.761 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.762 DEBUG:boto:Params: {} 2022-09-02T05:55:44.763 DEBUG:boto:Token: None 2022-09-02T05:55:44.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU= 2022-09-02T05:55:44.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU=', 'Content-Length': '0'} 2022-09-02T05:55:44.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87f6a0fe81e993e-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.769 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.769 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.770 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.770 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.771 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.771 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.772 DEBUG:boto:Data: 2022-09-02T05:55:44.772 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.773 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.774 DEBUG:boto:Params: {} 2022-09-02T05:55:44.775 DEBUG:boto:Token: None 2022-09-02T05:55:44.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p+5n+3sUWX9t6CnguCXXMBQBSVg= 2022-09-02T05:55:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p+5n+3sUWX9t6CnguCXXMBQBSVg=', 'Content-Length': '0'} 2022-09-02T05:55:44.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004befba371f974d62-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.779 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.780 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.781 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.781 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.782 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.782 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.783 DEBUG:boto:Data: 2022-09-02T05:55:44.783 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.784 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.785 DEBUG:boto:Params: {} 2022-09-02T05:55:44.785 DEBUG:boto:Token: None 2022-09-02T05:55:44.786 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p+5n+3sUWX9t6CnguCXXMBQBSVg= 2022-09-02T05:55:44.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p+5n+3sUWX9t6CnguCXXMBQBSVg=', 'Content-Length': '0'} 2022-09-02T05:55:44.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce33eda42db70f2-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:44.790 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.791 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.791 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.792 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.792 DEBUG:boto:Method: GET 2022-09-02T05:55:44.793 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.793 DEBUG:boto:Data: 2022-09-02T05:55:44.794 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.795 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.795 DEBUG:boto:Params: {} 2022-09-02T05:55:44.796 DEBUG:boto:Token: None 2022-09-02T05:55:44.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU= 2022-09-02T05:55:44.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU=', 'Content-Length': '0'} 2022-09-02T05:55:44.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000e352c4951246c-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.842 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.843 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.843 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.844 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.844 DEBUG:boto:Method: GET 2022-09-02T05:55:44.845 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.846 DEBUG:boto:Data: 2022-09-02T05:55:44.846 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.847 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.848 DEBUG:boto:Params: {} 2022-09-02T05:55:44.848 DEBUG:boto:Token: None 2022-09-02T05:55:44.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:44.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU= 2022-09-02T05:55:44.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JAq57pkXvEZWaj2vArd7h/Vl+PU=', 'Content-Length': '0'} 2022-09-02T05:55:44.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e97a724f296881-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.854 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.854 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.855 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.855 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.856 DEBUG:boto:Method: GET 2022-09-02T05:55:44.857 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.857 DEBUG:boto:Data: 2022-09-02T05:55:44.858 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.859 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.859 DEBUG:boto:Params: {} 2022-09-02T05:55:44.860 DEBUG:boto:Token: None 2022-09-02T05:55:44.860 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c5b36673f6096f-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.865 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.866 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.867 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.867 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.868 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.868 DEBUG:boto:Method: GET 2022-09-02T05:55:44.869 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.869 DEBUG:boto:Data: 2022-09-02T05:55:44.870 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.871 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.872 DEBUG:boto:Params: {} 2022-09-02T05:55:44.872 DEBUG:boto:Token: None 2022-09-02T05:55:44.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf94b6931a6d64e7-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.879 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.880 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.880 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.881 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.881 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:44.882 DEBUG:boto:Data: 2022-09-02T05:55:44.882 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.883 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.884 DEBUG:boto:Params: {} 2022-09-02T05:55:44.885 DEBUG:boto:Token: None 2022-09-02T05:55:44.885 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17 2022-09-02T05:55:44.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XxF7sJ2d56jHu4aMOAmyE/ACqo= 2022-09-02T05:55:44.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XxF7sJ2d56jHu4aMOAmyE/ACqo=', 'Content-Length': '0'} 2022-09-02T05:55:44.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f2cb7d104fdc919-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.889 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.890 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.890 DEBUG:boto:Method: HEAD 2022-09-02T05:55:44.891 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:44.892 DEBUG:boto:Data: 2022-09-02T05:55:44.892 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.893 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.894 DEBUG:boto:Params: {} 2022-09-02T05:55:44.894 DEBUG:boto:Token: None 2022-09-02T05:55:44.895 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17 2022-09-02T05:55:44.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0XxF7sJ2d56jHu4aMOAmyE/ACqo= 2022-09-02T05:55:44.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0XxF7sJ2d56jHu4aMOAmyE/ACqo=', 'Content-Length': '0'} 2022-09-02T05:55:44.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51224955ffa7ec2-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:44.899 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.900 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.901 DEBUG:boto:Method: GET 2022-09-02T05:55:44.901 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:44.902 DEBUG:boto:Data: 2022-09-02T05:55:44.902 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.903 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.904 DEBUG:boto:Params: {} 2022-09-02T05:55:44.905 DEBUG:boto:Token: None 2022-09-02T05:55:44.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17 2022-09-02T05:55:44.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hQ+G9Qdyw+3204i1ABapJpci8mQ= 2022-09-02T05:55:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hQ+G9Qdyw+3204i1ABapJpci8mQ=', 'Content-Length': '0'} 2022-09-02T05:55:44.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419e3ace6fd5b372-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.910 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:44.910 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:44.911 DEBUG:boto:Method: GET 2022-09-02T05:55:44.912 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:44.912 DEBUG:boto:Data: 2022-09-02T05:55:44.913 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.914 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.914 DEBUG:boto:Params: {} 2022-09-02T05:55:44.915 DEBUG:boto:Token: None 2022-09-02T05:55:44.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj17 2022-09-02T05:55:44.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hQ+G9Qdyw+3204i1ABapJpci8mQ= 2022-09-02T05:55:44.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hQ+G9Qdyw+3204i1ABapJpci8mQ=', 'Content-Length': '0'} 2022-09-02T05:55:44.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deb2f6c2f4049249-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.920 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.921 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.921 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.922 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.923 DEBUG:boto:Method: GET 2022-09-02T05:55:44.923 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.924 DEBUG:boto:Data: 2022-09-02T05:55:44.924 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.925 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.926 DEBUG:boto:Params: {} 2022-09-02T05:55:44.926 DEBUG:boto:Token: None 2022-09-02T05:55:44.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d44e96bd95e6ef3-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.932 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.933 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:44.933 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:44.934 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.934 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.935 DEBUG:boto:Method: GET 2022-09-02T05:55:44.935 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:44.936 DEBUG:boto:Data: 2022-09-02T05:55:44.936 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.937 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.938 DEBUG:boto:Params: {} 2022-09-02T05:55:44.939 DEBUG:boto:Token: None 2022-09-02T05:55:44.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/ 2022-09-02T05:55:44.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU= 2022-09-02T05:55:44.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0P9UlbV/hIbvzqFYJXs0iEAvyU=', 'Content-Length': '0'} 2022-09-02T05:55:44.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d269930d58c7d2b8-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.945 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:44.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:44.947 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:44.947 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:44.948 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.948 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.949 DEBUG:boto:Method: GET 2022-09-02T05:55:44.949 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.950 DEBUG:boto:Data: 2022-09-02T05:55:44.951 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:44.952 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.952 DEBUG:boto:Params: {} 2022-09-02T05:55:44.953 DEBUG:boto:Token: None 2022-09-02T05:55:44.954 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G5YEO3OnFOI7VdAeDpwQ235G0X0= 2022-09-02T05:55:44.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G5YEO3OnFOI7VdAeDpwQ235G0X0=', 'Content-Length': '0'} 2022-09-02T05:55:44.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f1da0cda7dc544d-0063119ae0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:44.959 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:44.959 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:44.960 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.960 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.961 DEBUG:boto:Method: GET 2022-09-02T05:55:44.962 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.962 DEBUG:boto:Data: 2022-09-02T05:55:44.963 DEBUG:boto:Headers: {} 2022-09-02T05:55:44.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:44.965 DEBUG:boto:Port: 8000 2022-09-02T05:55:44.966 DEBUG:boto:Params: {} 2022-09-02T05:55:44.966 DEBUG:boto:Token: None 2022-09-02T05:55:44.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:44 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:44.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G5YEO3OnFOI7VdAeDpwQ235G0X0= 2022-09-02T05:55:44.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G5YEO3OnFOI7VdAeDpwQ235G0X0=', 'Content-Length': '0'} 2022-09-02T05:55:44.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013dddf17474bd952-0063119ae0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.018 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.019 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.019 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.020 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.020 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.021 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.021 DEBUG:boto:Data: 2022-09-02T05:55:45.022 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.023 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.024 DEBUG:boto:Params: {} 2022-09-02T05:55:45.025 DEBUG:boto:Token: None 2022-09-02T05:55:45.026 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zgQagouOpsWBhBGVd7f2SHjpUEE= 2022-09-02T05:55:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zgQagouOpsWBhBGVd7f2SHjpUEE=', 'Content-Length': '0'} 2022-09-02T05:55:45.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e8c3c2e17569885-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.033 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.034 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.035 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.036 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.036 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.037 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.038 DEBUG:boto:Data: 2022-09-02T05:55:45.039 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.041 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.041 DEBUG:boto:Params: {} 2022-09-02T05:55:45.042 DEBUG:boto:Token: None 2022-09-02T05:55:45.043 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.044 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zgQagouOpsWBhBGVd7f2SHjpUEE= 2022-09-02T05:55:45.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zgQagouOpsWBhBGVd7f2SHjpUEE=', 'Content-Length': '0'} 2022-09-02T05:55:45.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc091fbf05f7b24-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:45.050 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.051 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.052 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.053 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.054 DEBUG:boto:Method: GET 2022-09-02T05:55:45.055 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.056 DEBUG:boto:Data: 2022-09-02T05:55:45.057 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.058 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.059 DEBUG:boto:Params: {} 2022-09-02T05:55:45.061 DEBUG:boto:Token: None 2022-09-02T05:55:45.062 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:klFIrVd/Zg422VsWzpWac8IWwwE= 2022-09-02T05:55:45.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:klFIrVd/Zg422VsWzpWac8IWwwE=', 'Content-Length': '0'} 2022-09-02T05:55:45.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7bb5a115b69f99-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.068 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.068 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.069 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.069 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.070 DEBUG:boto:Method: GET 2022-09-02T05:55:45.070 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.071 DEBUG:boto:Data: 2022-09-02T05:55:45.071 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.072 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.073 DEBUG:boto:Params: {} 2022-09-02T05:55:45.074 DEBUG:boto:Token: None 2022-09-02T05:55:45.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:45.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:klFIrVd/Zg422VsWzpWac8IWwwE= 2022-09-02T05:55:45.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:klFIrVd/Zg422VsWzpWac8IWwwE=', 'Content-Length': '0'} 2022-09-02T05:55:45.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010425c81a8855442-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.079 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.079 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.080 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.080 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.081 DEBUG:boto:Method: GET 2022-09-02T05:55:45.081 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.082 DEBUG:boto:Data: 2022-09-02T05:55:45.083 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.084 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.084 DEBUG:boto:Params: {} 2022-09-02T05:55:45.085 DEBUG:boto:Token: None 2022-09-02T05:55:45.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa55fa575bf8f6b5-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.134 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.135 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.136 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.136 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.137 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.137 DEBUG:boto:Method: GET 2022-09-02T05:55:45.138 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.138 DEBUG:boto:Data: 2022-09-02T05:55:45.139 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.140 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.140 DEBUG:boto:Params: {} 2022-09-02T05:55:45.141 DEBUG:boto:Token: None 2022-09-02T05:55:45.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034d337df5aaa175-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.148 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.149 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.149 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.150 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.150 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:45.151 DEBUG:boto:Data: 2022-09-02T05:55:45.151 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.152 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.152 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.153 DEBUG:boto:Params: {} 2022-09-02T05:55:45.154 DEBUG:boto:Token: None 2022-09-02T05:55:45.154 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19 2022-09-02T05:55:45.155 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q3/dXDiFOd6ALl+2Il6eO66YygU= 2022-09-02T05:55:45.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q3/dXDiFOd6ALl+2Il6eO66YygU=', 'Content-Length': '0'} 2022-09-02T05:55:45.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000484a3972e0dbc4fd-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.158 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.159 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.159 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.160 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:45.160 DEBUG:boto:Data: 2022-09-02T05:55:45.161 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.162 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.162 DEBUG:boto:Params: {} 2022-09-02T05:55:45.163 DEBUG:boto:Token: None 2022-09-02T05:55:45.164 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19 2022-09-02T05:55:45.164 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q3/dXDiFOd6ALl+2Il6eO66YygU= 2022-09-02T05:55:45.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q3/dXDiFOd6ALl+2Il6eO66YygU=', 'Content-Length': '0'} 2022-09-02T05:55:45.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009164db1c23490ac5-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:45.168 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.169 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.169 DEBUG:boto:Method: GET 2022-09-02T05:55:45.170 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:45.170 DEBUG:boto:Data: 2022-09-02T05:55:45.171 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.172 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.172 DEBUG:boto:Params: {} 2022-09-02T05:55:45.173 DEBUG:boto:Token: None 2022-09-02T05:55:45.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19 2022-09-02T05:55:45.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jiaVQ5XRMdrI33nJofYVY78vCS4= 2022-09-02T05:55:45.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jiaVQ5XRMdrI33nJofYVY78vCS4=', 'Content-Length': '0'} 2022-09-02T05:55:45.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408e839316878a4e-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.178 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:45.178 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:45.179 DEBUG:boto:Method: GET 2022-09-02T05:55:45.179 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:45.180 DEBUG:boto:Data: 2022-09-02T05:55:45.180 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.182 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.182 DEBUG:boto:Params: {} 2022-09-02T05:55:45.183 DEBUG:boto:Token: None 2022-09-02T05:55:45.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj19 2022-09-02T05:55:45.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jiaVQ5XRMdrI33nJofYVY78vCS4= 2022-09-02T05:55:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jiaVQ5XRMdrI33nJofYVY78vCS4=', 'Content-Length': '0'} 2022-09-02T05:55:45.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ce0e5dd0ea33a36-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.235 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.235 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.236 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.238 DEBUG:boto:Method: GET 2022-09-02T05:55:45.239 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.239 DEBUG:boto:Data: 2022-09-02T05:55:45.240 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.240 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.241 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.242 DEBUG:boto:Params: {} 2022-09-02T05:55:45.242 DEBUG:boto:Token: None 2022-09-02T05:55:45.243 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080166d7f80863448-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.247 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.248 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.249 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.249 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.250 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.250 DEBUG:boto:Method: GET 2022-09-02T05:55:45.251 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:45.252 DEBUG:boto:Data: 2022-09-02T05:55:45.252 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.253 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.254 DEBUG:boto:Params: {} 2022-09-02T05:55:45.254 DEBUG:boto:Token: None 2022-09-02T05:55:45.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35eb684f0c9e8af-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.261 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:45.262 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.263 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.263 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.264 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.265 DEBUG:boto:Method: GET 2022-09-02T05:55:45.265 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.266 DEBUG:boto:Data: 2022-09-02T05:55:45.266 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.267 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.268 DEBUG:boto:Params: {} 2022-09-02T05:55:45.268 DEBUG:boto:Token: None 2022-09-02T05:55:45.269 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c= 2022-09-02T05:55:45.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c=', 'Content-Length': '0'} 2022-09-02T05:55:45.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0d4986f4780474-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.273 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.274 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.275 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.275 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.276 DEBUG:boto:Method: GET 2022-09-02T05:55:45.276 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.277 DEBUG:boto:Data: 2022-09-02T05:55:45.277 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.279 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.279 DEBUG:boto:Params: {} 2022-09-02T05:55:45.280 DEBUG:boto:Token: None 2022-09-02T05:55:45.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c= 2022-09-02T05:55:45.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c=', 'Content-Length': '0'} 2022-09-02T05:55:45.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c2e8c6bd767c10-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.284 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.285 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.285 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.286 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.286 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.287 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.287 DEBUG:boto:Data: 2022-09-02T05:55:45.288 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.289 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.289 DEBUG:boto:Params: {} 2022-09-02T05:55:45.290 DEBUG:boto:Token: None 2022-09-02T05:55:45.291 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dsPjG87X10C8+G8HJ2iZana8o6M= 2022-09-02T05:55:45.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dsPjG87X10C8+G8HJ2iZana8o6M=', 'Content-Length': '0'} 2022-09-02T05:55:45.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9fc47f45bc1b623-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.295 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.295 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.296 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.296 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.297 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.297 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.298 DEBUG:boto:Data: 2022-09-02T05:55:45.298 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.299 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.300 DEBUG:boto:Params: {} 2022-09-02T05:55:45.300 DEBUG:boto:Token: None 2022-09-02T05:55:45.301 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dsPjG87X10C8+G8HJ2iZana8o6M= 2022-09-02T05:55:45.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dsPjG87X10C8+G8HJ2iZana8o6M=', 'Content-Length': '0'} 2022-09-02T05:55:45.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080373da05482ff4e-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:45.306 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.306 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.307 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.307 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.308 DEBUG:boto:Method: GET 2022-09-02T05:55:45.308 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.309 DEBUG:boto:Data: 2022-09-02T05:55:45.309 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.310 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.311 DEBUG:boto:Params: {} 2022-09-02T05:55:45.311 DEBUG:boto:Token: None 2022-09-02T05:55:45.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c= 2022-09-02T05:55:45.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c=', 'Content-Length': '0'} 2022-09-02T05:55:45.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566563a15b4bd066-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.316 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.317 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.318 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.318 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.319 DEBUG:boto:Method: GET 2022-09-02T05:55:45.319 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.320 DEBUG:boto:Data: 2022-09-02T05:55:45.320 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.322 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.322 DEBUG:boto:Params: {} 2022-09-02T05:55:45.323 DEBUG:boto:Token: None 2022-09-02T05:55:45.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:45.324 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c= 2022-09-02T05:55:45.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nAJnIgE5j2Y+cTHvI21ZX2kF60c=', 'Content-Length': '0'} 2022-09-02T05:55:45.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b02073a5884a08-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.328 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.328 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.329 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.329 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.330 DEBUG:boto:Method: GET 2022-09-02T05:55:45.331 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.331 DEBUG:boto:Data: 2022-09-02T05:55:45.332 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.333 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.333 DEBUG:boto:Params: {} 2022-09-02T05:55:45.334 DEBUG:boto:Token: None 2022-09-02T05:55:45.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000812e4bb1081fc436-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.339 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.340 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.341 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.341 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.342 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.342 DEBUG:boto:Method: GET 2022-09-02T05:55:45.343 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.343 DEBUG:boto:Data: 2022-09-02T05:55:45.344 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.345 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.345 DEBUG:boto:Params: {} 2022-09-02T05:55:45.346 DEBUG:boto:Token: None 2022-09-02T05:55:45.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000205b6ee3a42d1cbf-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.398 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.399 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.400 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.400 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.401 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:45.401 DEBUG:boto:Data: 2022-09-02T05:55:45.402 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.402 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.403 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.403 DEBUG:boto:Params: {} 2022-09-02T05:55:45.404 DEBUG:boto:Token: None 2022-09-02T05:55:45.405 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22 2022-09-02T05:55:45.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z0jj5YrkJoFLR6Il7XW+8LzZ8yk= 2022-09-02T05:55:45.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z0jj5YrkJoFLR6Il7XW+8LzZ8yk=', 'Content-Length': '0'} 2022-09-02T05:55:45.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983f57f36c536e98-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.409 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.409 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.410 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.410 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:45.411 DEBUG:boto:Data: 2022-09-02T05:55:45.411 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.412 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.413 DEBUG:boto:Params: {} 2022-09-02T05:55:45.413 DEBUG:boto:Token: None 2022-09-02T05:55:45.414 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22 2022-09-02T05:55:45.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z0jj5YrkJoFLR6Il7XW+8LzZ8yk= 2022-09-02T05:55:45.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z0jj5YrkJoFLR6Il7XW+8LzZ8yk=', 'Content-Length': '0'} 2022-09-02T05:55:45.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bb50b746732adb7-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:45.419 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.420 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.420 DEBUG:boto:Method: GET 2022-09-02T05:55:45.421 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:45.421 DEBUG:boto:Data: 2022-09-02T05:55:45.422 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.423 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.424 DEBUG:boto:Params: {} 2022-09-02T05:55:45.424 DEBUG:boto:Token: None 2022-09-02T05:55:45.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22 2022-09-02T05:55:45.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tJIsvlrN9ylcZGvW9BWz3VMeE4Y= 2022-09-02T05:55:45.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tJIsvlrN9ylcZGvW9BWz3VMeE4Y=', 'Content-Length': '0'} 2022-09-02T05:55:45.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002eb84475cc4d62a-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.430 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:45.431 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:45.431 DEBUG:boto:Method: GET 2022-09-02T05:55:45.432 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:45.432 DEBUG:boto:Data: 2022-09-02T05:55:45.433 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.434 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.434 DEBUG:boto:Params: {} 2022-09-02T05:55:45.435 DEBUG:boto:Token: None 2022-09-02T05:55:45.436 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj22 2022-09-02T05:55:45.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tJIsvlrN9ylcZGvW9BWz3VMeE4Y= 2022-09-02T05:55:45.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tJIsvlrN9ylcZGvW9BWz3VMeE4Y=', 'Content-Length': '0'} 2022-09-02T05:55:45.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdfaf5b6f64b0fab-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.440 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.441 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.441 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.442 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.442 DEBUG:boto:Method: GET 2022-09-02T05:55:45.443 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.443 DEBUG:boto:Data: 2022-09-02T05:55:45.444 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.445 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.445 DEBUG:boto:Params: {} 2022-09-02T05:55:45.446 DEBUG:boto:Token: None 2022-09-02T05:55:45.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087deae0bb97088bb-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.452 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.453 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.453 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.454 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.454 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.455 DEBUG:boto:Method: GET 2022-09-02T05:55:45.456 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:45.456 DEBUG:boto:Data: 2022-09-02T05:55:45.457 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.458 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.459 DEBUG:boto:Params: {} 2022-09-02T05:55:45.459 DEBUG:boto:Token: None 2022-09-02T05:55:45.460 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038198e014b0e8dc6-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.466 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:45.467 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.468 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.468 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.469 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.469 DEBUG:boto:Method: GET 2022-09-02T05:55:45.470 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.471 DEBUG:boto:Data: 2022-09-02T05:55:45.471 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.472 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.473 DEBUG:boto:Params: {} 2022-09-02T05:55:45.473 DEBUG:boto:Token: None 2022-09-02T05:55:45.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI= 2022-09-02T05:55:45.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI=', 'Content-Length': '0'} 2022-09-02T05:55:45.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f25bda4912c68d-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.522 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.523 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.523 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.524 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.524 DEBUG:boto:Method: GET 2022-09-02T05:55:45.525 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.526 DEBUG:boto:Data: 2022-09-02T05:55:45.526 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.527 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.528 DEBUG:boto:Params: {} 2022-09-02T05:55:45.528 DEBUG:boto:Token: None 2022-09-02T05:55:45.529 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI= 2022-09-02T05:55:45.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI=', 'Content-Length': '0'} 2022-09-02T05:55:45.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17d2c363bd22309-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.533 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.534 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.534 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.535 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.535 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.536 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.536 DEBUG:boto:Data: 2022-09-02T05:55:45.537 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.538 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.538 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.539 DEBUG:boto:Params: {} 2022-09-02T05:55:45.539 DEBUG:boto:Token: None 2022-09-02T05:55:45.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tFcC0/Z+EjZM2wkInuUrpXnnprQ= 2022-09-02T05:55:45.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tFcC0/Z+EjZM2wkInuUrpXnnprQ=', 'Content-Length': '0'} 2022-09-02T05:55:45.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf214beb4987cff-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.544 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.545 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.545 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.546 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.546 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.547 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.548 DEBUG:boto:Data: 2022-09-02T05:55:45.548 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.549 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.550 DEBUG:boto:Params: {} 2022-09-02T05:55:45.550 DEBUG:boto:Token: None 2022-09-02T05:55:45.551 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tFcC0/Z+EjZM2wkInuUrpXnnprQ= 2022-09-02T05:55:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tFcC0/Z+EjZM2wkInuUrpXnnprQ=', 'Content-Length': '0'} 2022-09-02T05:55:45.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018cf227baa85c6f1-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:45.556 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.556 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.557 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.558 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.558 DEBUG:boto:Method: GET 2022-09-02T05:55:45.559 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.559 DEBUG:boto:Data: 2022-09-02T05:55:45.560 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.560 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.561 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.561 DEBUG:boto:Params: {} 2022-09-02T05:55:45.562 DEBUG:boto:Token: None 2022-09-02T05:55:45.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI= 2022-09-02T05:55:45.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI=', 'Content-Length': '0'} 2022-09-02T05:55:45.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d840336adce20c3c-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.566 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.567 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.568 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.568 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.569 DEBUG:boto:Method: GET 2022-09-02T05:55:45.569 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.570 DEBUG:boto:Data: 2022-09-02T05:55:45.570 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.571 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.572 DEBUG:boto:Params: {} 2022-09-02T05:55:45.573 DEBUG:boto:Token: None 2022-09-02T05:55:45.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:45.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI= 2022-09-02T05:55:45.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/6LNZpuQnci0xhf/qhRQOkD5LhI=', 'Content-Length': '0'} 2022-09-02T05:55:45.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceacd5402f18c8fb-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.578 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.578 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.579 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.580 DEBUG:boto:Method: GET 2022-09-02T05:55:45.581 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.582 DEBUG:boto:Data: 2022-09-02T05:55:45.582 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.583 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.584 DEBUG:boto:Params: {} 2022-09-02T05:55:45.585 DEBUG:boto:Token: None 2022-09-02T05:55:45.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f099b5a036a4eb-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.590 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.591 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.591 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.592 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.592 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.593 DEBUG:boto:Method: GET 2022-09-02T05:55:45.594 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.594 DEBUG:boto:Data: 2022-09-02T05:55:45.595 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.596 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.597 DEBUG:boto:Params: {} 2022-09-02T05:55:45.597 DEBUG:boto:Token: None 2022-09-02T05:55:45.598 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081930a81b2e39e62-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.604 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.605 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.605 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.606 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.606 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:45.607 DEBUG:boto:Data: 2022-09-02T05:55:45.607 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.608 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.609 DEBUG:boto:Params: {} 2022-09-02T05:55:45.609 DEBUG:boto:Token: None 2022-09-02T05:55:45.610 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23 2022-09-02T05:55:45.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VBSiyJ2P/HodrYQcesRi7JLqN9s= 2022-09-02T05:55:45.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VBSiyJ2P/HodrYQcesRi7JLqN9s=', 'Content-Length': '0'} 2022-09-02T05:55:45.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e30e2e61e3d98b0-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.614 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.614 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.615 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.615 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:45.616 DEBUG:boto:Data: 2022-09-02T05:55:45.616 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.617 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.618 DEBUG:boto:Params: {} 2022-09-02T05:55:45.618 DEBUG:boto:Token: None 2022-09-02T05:55:45.619 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23 2022-09-02T05:55:45.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VBSiyJ2P/HodrYQcesRi7JLqN9s= 2022-09-02T05:55:45.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VBSiyJ2P/HodrYQcesRi7JLqN9s=', 'Content-Length': '0'} 2022-09-02T05:55:45.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009e5c67df0fe2639-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:45.623 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.624 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.624 DEBUG:boto:Method: GET 2022-09-02T05:55:45.625 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:45.625 DEBUG:boto:Data: 2022-09-02T05:55:45.626 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.627 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.627 DEBUG:boto:Params: {} 2022-09-02T05:55:45.628 DEBUG:boto:Token: None 2022-09-02T05:55:45.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23 2022-09-02T05:55:45.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1dV4z5BvilnnW5rqxtlRRJs9ub0= 2022-09-02T05:55:45.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1dV4z5BvilnnW5rqxtlRRJs9ub0=', 'Content-Length': '0'} 2022-09-02T05:55:45.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009748cd5a7c231261-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.633 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:45.633 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:45.634 DEBUG:boto:Method: GET 2022-09-02T05:55:45.634 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:45.635 DEBUG:boto:Data: 2022-09-02T05:55:45.635 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.636 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.637 DEBUG:boto:Params: {} 2022-09-02T05:55:45.638 DEBUG:boto:Token: None 2022-09-02T05:55:45.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj23 2022-09-02T05:55:45.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1dV4z5BvilnnW5rqxtlRRJs9ub0= 2022-09-02T05:55:45.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1dV4z5BvilnnW5rqxtlRRJs9ub0=', 'Content-Length': '0'} 2022-09-02T05:55:45.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf77b2bc1912f193-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.682 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.684 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.684 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.685 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.685 DEBUG:boto:Method: GET 2022-09-02T05:55:45.686 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.686 DEBUG:boto:Data: 2022-09-02T05:55:45.687 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.688 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.689 DEBUG:boto:Params: {} 2022-09-02T05:55:45.689 DEBUG:boto:Token: None 2022-09-02T05:55:45.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec234b9a5208818-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.695 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.696 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.696 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.697 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.698 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.698 DEBUG:boto:Method: GET 2022-09-02T05:55:45.699 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:45.700 DEBUG:boto:Data: 2022-09-02T05:55:45.700 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.701 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.702 DEBUG:boto:Params: {} 2022-09-02T05:55:45.703 DEBUG:boto:Token: None 2022-09-02T05:55:45.703 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879a7d00c0a1ba47-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.709 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:45.711 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.712 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.712 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.713 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.713 DEBUG:boto:Method: GET 2022-09-02T05:55:45.714 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.715 DEBUG:boto:Data: 2022-09-02T05:55:45.715 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.717 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.717 DEBUG:boto:Params: {} 2022-09-02T05:55:45.718 DEBUG:boto:Token: None 2022-09-02T05:55:45.719 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY= 2022-09-02T05:55:45.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY=', 'Content-Length': '0'} 2022-09-02T05:55:45.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f044f4113cd70880-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.723 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.724 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.724 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.725 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.726 DEBUG:boto:Method: GET 2022-09-02T05:55:45.727 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.727 DEBUG:boto:Data: 2022-09-02T05:55:45.728 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.729 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.729 DEBUG:boto:Params: {} 2022-09-02T05:55:45.730 DEBUG:boto:Token: None 2022-09-02T05:55:45.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY= 2022-09-02T05:55:45.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY=', 'Content-Length': '0'} 2022-09-02T05:55:45.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c509d186fb15146-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.735 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.736 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.736 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.737 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.737 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.738 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.738 DEBUG:boto:Data: 2022-09-02T05:55:45.739 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.739 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.740 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.740 DEBUG:boto:Params: {} 2022-09-02T05:55:45.741 DEBUG:boto:Token: None 2022-09-02T05:55:45.741 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o+PvAUVRR6Zc+Rv3sDEHqUzrNLw= 2022-09-02T05:55:45.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o+PvAUVRR6Zc+Rv3sDEHqUzrNLw=', 'Content-Length': '0'} 2022-09-02T05:55:45.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b36d8d7aed1f679-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.746 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.747 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.747 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.748 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.748 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.749 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.749 DEBUG:boto:Data: 2022-09-02T05:55:45.750 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.751 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.752 DEBUG:boto:Params: {} 2022-09-02T05:55:45.752 DEBUG:boto:Token: None 2022-09-02T05:55:45.753 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o+PvAUVRR6Zc+Rv3sDEHqUzrNLw= 2022-09-02T05:55:45.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o+PvAUVRR6Zc+Rv3sDEHqUzrNLw=', 'Content-Length': '0'} 2022-09-02T05:55:45.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d3438a80a4bfea-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:45.758 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.758 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.759 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.759 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.760 DEBUG:boto:Method: GET 2022-09-02T05:55:45.761 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.761 DEBUG:boto:Data: 2022-09-02T05:55:45.762 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.763 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.764 DEBUG:boto:Params: {} 2022-09-02T05:55:45.764 DEBUG:boto:Token: None 2022-09-02T05:55:45.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY= 2022-09-02T05:55:45.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY=', 'Content-Length': '0'} 2022-09-02T05:55:45.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a108bb480461c0f-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.814 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.815 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.815 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.816 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.816 DEBUG:boto:Method: GET 2022-09-02T05:55:45.817 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.818 DEBUG:boto:Data: 2022-09-02T05:55:45.818 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.819 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.820 DEBUG:boto:Params: {} 2022-09-02T05:55:45.820 DEBUG:boto:Token: None 2022-09-02T05:55:45.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:45.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY= 2022-09-02T05:55:45.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2l8WD4GvglVW5BE1ELTEhIqsxgY=', 'Content-Length': '0'} 2022-09-02T05:55:45.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7bb6921e2fc89a9-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.825 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.826 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.826 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.827 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.827 DEBUG:boto:Method: GET 2022-09-02T05:55:45.828 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.828 DEBUG:boto:Data: 2022-09-02T05:55:45.829 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.830 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.831 DEBUG:boto:Params: {} 2022-09-02T05:55:45.831 DEBUG:boto:Token: None 2022-09-02T05:55:45.832 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000757af547e1107f8f-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.837 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.838 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.839 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.841 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.842 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.843 DEBUG:boto:Method: GET 2022-09-02T05:55:45.844 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.845 DEBUG:boto:Data: 2022-09-02T05:55:45.846 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.848 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.849 DEBUG:boto:Params: {} 2022-09-02T05:55:45.850 DEBUG:boto:Token: None 2022-09-02T05:55:45.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000995a5c801176a5d7-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.859 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.861 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.862 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.863 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.864 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:45.865 DEBUG:boto:Data: 2022-09-02T05:55:45.866 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.868 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.869 DEBUG:boto:Params: {} 2022-09-02T05:55:45.870 DEBUG:boto:Token: None 2022-09-02T05:55:45.872 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29 2022-09-02T05:55:45.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yArt4524kegLnhXFt5Zq5lBDYT0= 2022-09-02T05:55:45.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yArt4524kegLnhXFt5Zq5lBDYT0=', 'Content-Length': '0'} 2022-09-02T05:55:45.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008675c9d124549ee0-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.878 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.879 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.880 DEBUG:boto:Method: HEAD 2022-09-02T05:55:45.881 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:45.882 DEBUG:boto:Data: 2022-09-02T05:55:45.883 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.885 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.886 DEBUG:boto:Params: {} 2022-09-02T05:55:45.887 DEBUG:boto:Token: None 2022-09-02T05:55:45.888 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29 2022-09-02T05:55:45.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yArt4524kegLnhXFt5Zq5lBDYT0= 2022-09-02T05:55:45.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yArt4524kegLnhXFt5Zq5lBDYT0=', 'Content-Length': '0'} 2022-09-02T05:55:45.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838e56025d2a350f-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:45.896 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.897 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.898 DEBUG:boto:Method: GET 2022-09-02T05:55:45.899 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:45.900 DEBUG:boto:Data: 2022-09-02T05:55:45.901 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.903 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.904 DEBUG:boto:Params: {} 2022-09-02T05:55:45.905 DEBUG:boto:Token: None 2022-09-02T05:55:45.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29 2022-09-02T05:55:45.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VEi03PEYHot7tEFcuAPi0qckZmY= 2022-09-02T05:55:45.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VEi03PEYHot7tEFcuAPi0qckZmY=', 'Content-Length': '0'} 2022-09-02T05:55:45.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000023d2e5ef46a06c2-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.911 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:45.911 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:45.912 DEBUG:boto:Method: GET 2022-09-02T05:55:45.912 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:45.913 DEBUG:boto:Data: 2022-09-02T05:55:45.913 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.915 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.915 DEBUG:boto:Params: {} 2022-09-02T05:55:45.916 DEBUG:boto:Token: None 2022-09-02T05:55:45.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj29 2022-09-02T05:55:45.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VEi03PEYHot7tEFcuAPi0qckZmY= 2022-09-02T05:55:45.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VEi03PEYHot7tEFcuAPi0qckZmY=', 'Content-Length': '0'} 2022-09-02T05:55:45.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032578658570cd604-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.921 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.921 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.922 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.922 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.923 DEBUG:boto:Method: GET 2022-09-02T05:55:45.923 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.924 DEBUG:boto:Data: 2022-09-02T05:55:45.924 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.926 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.926 DEBUG:boto:Params: {} 2022-09-02T05:55:45.927 DEBUG:boto:Token: None 2022-09-02T05:55:45.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b09bd5394b2ad177-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.932 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.933 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:45.934 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:45.934 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.935 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.936 DEBUG:boto:Method: GET 2022-09-02T05:55:45.936 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:45.937 DEBUG:boto:Data: 2022-09-02T05:55:45.937 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:45.938 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.939 DEBUG:boto:Params: {} 2022-09-02T05:55:45.939 DEBUG:boto:Token: None 2022-09-02T05:55:45.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/ 2022-09-02T05:55:45.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw= 2022-09-02T05:55:45.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:29zgDX59aDomvaAQOkN00J2E8uw=', 'Content-Length': '0'} 2022-09-02T05:55:45.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c92e34634c2d2c76-0063119ae1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:45.946 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:45.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:45.947 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:45.948 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:45.949 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:45.949 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:45.950 DEBUG:boto:Method: GET 2022-09-02T05:55:45.950 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:45.951 DEBUG:boto:Data: 2022-09-02T05:55:45.951 DEBUG:boto:Headers: {} 2022-09-02T05:55:45.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:45.952 DEBUG:boto:Port: 8000 2022-09-02T05:55:45.953 DEBUG:boto:Params: {} 2022-09-02T05:55:45.954 DEBUG:boto:Token: None 2022-09-02T05:55:45.954 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:45 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:45.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G7qiLMbq/4SutAlf8a0XFpTXYe0= 2022-09-02T05:55:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G7qiLMbq/4SutAlf8a0XFpTXYe0=', 'Content-Length': '0'} 2022-09-02T05:55:45.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a95acae26a34dfd-0063119ae1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.002 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.003 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.004 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.004 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.005 DEBUG:boto:Method: GET 2022-09-02T05:55:46.005 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.006 DEBUG:boto:Data: 2022-09-02T05:55:46.006 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.007 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.008 DEBUG:boto:Params: {} 2022-09-02T05:55:46.008 DEBUG:boto:Token: None 2022-09-02T05:55:46.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.010 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qesJ/xal9xDkPc973r+hmnuCHUM= 2022-09-02T05:55:46.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qesJ/xal9xDkPc973r+hmnuCHUM=', 'Content-Length': '0'} 2022-09-02T05:55:46.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b39660ebdd29181-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.015 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.016 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.017 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.018 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.019 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.020 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.021 DEBUG:boto:Data: 2022-09-02T05:55:46.022 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.024 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.025 DEBUG:boto:Params: {} 2022-09-02T05:55:46.026 DEBUG:boto:Token: None 2022-09-02T05:55:46.027 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qDPUJfxOMdwNKOaqLo9AKSKXjY0= 2022-09-02T05:55:46.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qDPUJfxOMdwNKOaqLo9AKSKXjY0=', 'Content-Length': '0'} 2022-09-02T05:55:46.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000159f42fabf502562-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.033 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.034 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.035 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.036 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.037 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.038 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.039 DEBUG:boto:Data: 2022-09-02T05:55:46.040 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.042 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.042 DEBUG:boto:Params: {} 2022-09-02T05:55:46.044 DEBUG:boto:Token: None 2022-09-02T05:55:46.045 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qDPUJfxOMdwNKOaqLo9AKSKXjY0= 2022-09-02T05:55:46.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qDPUJfxOMdwNKOaqLo9AKSKXjY0=', 'Content-Length': '0'} 2022-09-02T05:55:46.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a826a13c5faf090-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:46.052 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.053 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.054 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.056 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.057 DEBUG:boto:Method: GET 2022-09-02T05:55:46.058 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.059 DEBUG:boto:Data: 2022-09-02T05:55:46.060 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.061 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.062 DEBUG:boto:Params: {} 2022-09-02T05:55:46.063 DEBUG:boto:Token: None 2022-09-02T05:55:46.063 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qesJ/xal9xDkPc973r+hmnuCHUM= 2022-09-02T05:55:46.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qesJ/xal9xDkPc973r+hmnuCHUM=', 'Content-Length': '0'} 2022-09-02T05:55:46.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5e0316bac20658-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.067 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.068 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.068 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.069 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.070 DEBUG:boto:Method: GET 2022-09-02T05:55:46.070 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.071 DEBUG:boto:Data: 2022-09-02T05:55:46.071 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.072 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.073 DEBUG:boto:Params: {} 2022-09-02T05:55:46.073 DEBUG:boto:Token: None 2022-09-02T05:55:46.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:46.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qesJ/xal9xDkPc973r+hmnuCHUM= 2022-09-02T05:55:46.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qesJ/xal9xDkPc973r+hmnuCHUM=', 'Content-Length': '0'} 2022-09-02T05:55:46.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db49ccbd271d98b3-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.078 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.079 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.079 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.080 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.080 DEBUG:boto:Method: GET 2022-09-02T05:55:46.081 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.082 DEBUG:boto:Data: 2022-09-02T05:55:46.082 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.083 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.084 DEBUG:boto:Params: {} 2022-09-02T05:55:46.084 DEBUG:boto:Token: None 2022-09-02T05:55:46.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d84210e651dc2ae-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.089 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.090 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.097 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.098 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.098 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.099 DEBUG:boto:Method: GET 2022-09-02T05:55:46.099 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.100 DEBUG:boto:Data: 2022-09-02T05:55:46.100 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.101 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.102 DEBUG:boto:Params: {} 2022-09-02T05:55:46.103 DEBUG:boto:Token: None 2022-09-02T05:55:46.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004071a9ee4ad1317e-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.109 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.110 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.111 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.111 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.112 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:46.112 DEBUG:boto:Data: 2022-09-02T05:55:46.113 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.114 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.114 DEBUG:boto:Params: {} 2022-09-02T05:55:46.115 DEBUG:boto:Token: None 2022-09-02T05:55:46.115 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33 2022-09-02T05:55:46.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Ic+IiPu8lR15D4mbQuGWmGQOGc= 2022-09-02T05:55:46.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Ic+IiPu8lR15D4mbQuGWmGQOGc=', 'Content-Length': '0'} 2022-09-02T05:55:46.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e2f4ab54cfddd8-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.119 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.120 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.120 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.121 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:46.121 DEBUG:boto:Data: 2022-09-02T05:55:46.122 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.123 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.123 DEBUG:boto:Params: {} 2022-09-02T05:55:46.124 DEBUG:boto:Token: None 2022-09-02T05:55:46.124 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33 2022-09-02T05:55:46.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Ic+IiPu8lR15D4mbQuGWmGQOGc= 2022-09-02T05:55:46.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Ic+IiPu8lR15D4mbQuGWmGQOGc=', 'Content-Length': '0'} 2022-09-02T05:55:46.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002af74c43f7838706-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:46.129 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.130 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.130 DEBUG:boto:Method: GET 2022-09-02T05:55:46.131 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:46.131 DEBUG:boto:Data: 2022-09-02T05:55:46.132 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.133 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.134 DEBUG:boto:Params: {} 2022-09-02T05:55:46.134 DEBUG:boto:Token: None 2022-09-02T05:55:46.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33 2022-09-02T05:55:46.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hfHP9Gf6l4ydQw7+6gvO/StZrww= 2022-09-02T05:55:46.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hfHP9Gf6l4ydQw7+6gvO/StZrww=', 'Content-Length': '0'} 2022-09-02T05:55:46.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f486de44b2bd5be7-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.182 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:46.183 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:46.184 DEBUG:boto:Method: GET 2022-09-02T05:55:46.184 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:46.185 DEBUG:boto:Data: 2022-09-02T05:55:46.185 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.186 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.187 DEBUG:boto:Params: {} 2022-09-02T05:55:46.188 DEBUG:boto:Token: None 2022-09-02T05:55:46.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj33 2022-09-02T05:55:46.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hfHP9Gf6l4ydQw7+6gvO/StZrww= 2022-09-02T05:55:46.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hfHP9Gf6l4ydQw7+6gvO/StZrww=', 'Content-Length': '0'} 2022-09-02T05:55:46.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d4ea0fbd51d3535-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.193 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.193 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.194 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.195 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.195 DEBUG:boto:Method: GET 2022-09-02T05:55:46.196 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.196 DEBUG:boto:Data: 2022-09-02T05:55:46.197 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.198 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.198 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.199 DEBUG:boto:Params: {} 2022-09-02T05:55:46.200 DEBUG:boto:Token: None 2022-09-02T05:55:46.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b1df7462c5f8a2-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.206 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.207 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.207 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.208 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.208 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.209 DEBUG:boto:Method: GET 2022-09-02T05:55:46.209 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:46.210 DEBUG:boto:Data: 2022-09-02T05:55:46.211 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.212 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.212 DEBUG:boto:Params: {} 2022-09-02T05:55:46.213 DEBUG:boto:Token: None 2022-09-02T05:55:46.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007366ae59e6e832a1-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.220 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:46.221 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.222 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.223 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.223 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.224 DEBUG:boto:Method: GET 2022-09-02T05:55:46.224 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.225 DEBUG:boto:Data: 2022-09-02T05:55:46.226 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.227 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.227 DEBUG:boto:Params: {} 2022-09-02T05:55:46.228 DEBUG:boto:Token: None 2022-09-02T05:55:46.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU= 2022-09-02T05:55:46.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU=', 'Content-Length': '0'} 2022-09-02T05:55:46.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a69b1c0d8af2a2-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.233 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.234 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.234 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.235 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.236 DEBUG:boto:Method: GET 2022-09-02T05:55:46.236 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.237 DEBUG:boto:Data: 2022-09-02T05:55:46.237 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.238 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.239 DEBUG:boto:Params: {} 2022-09-02T05:55:46.239 DEBUG:boto:Token: None 2022-09-02T05:55:46.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU= 2022-09-02T05:55:46.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU=', 'Content-Length': '0'} 2022-09-02T05:55:46.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf90faf730815d8-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.286 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.287 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.288 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.289 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.290 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.291 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.292 DEBUG:boto:Data: 2022-09-02T05:55:46.293 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.295 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.296 DEBUG:boto:Params: {} 2022-09-02T05:55:46.297 DEBUG:boto:Token: None 2022-09-02T05:55:46.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5U+KFP1G0Xp9zswx2g3+ZOwdHE= 2022-09-02T05:55:46.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5U+KFP1G0Xp9zswx2g3+ZOwdHE=', 'Content-Length': '0'} 2022-09-02T05:55:46.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f5e7c95cb121d5e-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.304 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.305 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.306 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.307 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.308 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.310 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.311 DEBUG:boto:Data: 2022-09-02T05:55:46.312 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.314 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.315 DEBUG:boto:Params: {} 2022-09-02T05:55:46.316 DEBUG:boto:Token: None 2022-09-02T05:55:46.317 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M5U+KFP1G0Xp9zswx2g3+ZOwdHE= 2022-09-02T05:55:46.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M5U+KFP1G0Xp9zswx2g3+ZOwdHE=', 'Content-Length': '0'} 2022-09-02T05:55:46.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc03dc58f2e29e7b-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:46.324 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.324 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.325 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.326 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.326 DEBUG:boto:Method: GET 2022-09-02T05:55:46.327 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.327 DEBUG:boto:Data: 2022-09-02T05:55:46.328 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.329 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.329 DEBUG:boto:Params: {} 2022-09-02T05:55:46.330 DEBUG:boto:Token: None 2022-09-02T05:55:46.330 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU= 2022-09-02T05:55:46.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU=', 'Content-Length': '0'} 2022-09-02T05:55:46.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de24b78f808ef254-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.335 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.336 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.336 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.337 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.338 DEBUG:boto:Method: GET 2022-09-02T05:55:46.338 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.339 DEBUG:boto:Data: 2022-09-02T05:55:46.339 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.340 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.341 DEBUG:boto:Params: {} 2022-09-02T05:55:46.341 DEBUG:boto:Token: None 2022-09-02T05:55:46.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:46.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU= 2022-09-02T05:55:46.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KISGthMAi49P4MxIHCnPYZOttYU=', 'Content-Length': '0'} 2022-09-02T05:55:46.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6c5a6d359aa681-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.346 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.347 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.348 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.348 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.349 DEBUG:boto:Method: GET 2022-09-02T05:55:46.359 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.359 DEBUG:boto:Data: 2022-09-02T05:55:46.360 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.361 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.362 DEBUG:boto:Params: {} 2022-09-02T05:55:46.362 DEBUG:boto:Token: None 2022-09-02T05:55:46.363 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f550724293bcdc7c-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.410 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.412 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.413 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.413 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.414 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.414 DEBUG:boto:Method: GET 2022-09-02T05:55:46.415 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.415 DEBUG:boto:Data: 2022-09-02T05:55:46.416 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.417 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.418 DEBUG:boto:Params: {} 2022-09-02T05:55:46.419 DEBUG:boto:Token: None 2022-09-02T05:55:46.420 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9c83d966b72bdd-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.426 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.427 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.428 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.428 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.429 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:46.429 DEBUG:boto:Data: 2022-09-02T05:55:46.430 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.431 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.432 DEBUG:boto:Params: {} 2022-09-02T05:55:46.432 DEBUG:boto:Token: None 2022-09-02T05:55:46.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34 2022-09-02T05:55:46.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ujGa1qX+v+L68VBYgxSaoYKFsbA= 2022-09-02T05:55:46.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ujGa1qX+v+L68VBYgxSaoYKFsbA=', 'Content-Length': '0'} 2022-09-02T05:55:46.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68ac689a7b87843-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.437 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.438 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.438 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.439 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:46.439 DEBUG:boto:Data: 2022-09-02T05:55:46.440 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.441 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.442 DEBUG:boto:Params: {} 2022-09-02T05:55:46.442 DEBUG:boto:Token: None 2022-09-02T05:55:46.443 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34 2022-09-02T05:55:46.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ujGa1qX+v+L68VBYgxSaoYKFsbA= 2022-09-02T05:55:46.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ujGa1qX+v+L68VBYgxSaoYKFsbA=', 'Content-Length': '0'} 2022-09-02T05:55:46.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d87db50ee812e89-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:46.448 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.448 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.449 DEBUG:boto:Method: GET 2022-09-02T05:55:46.450 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:46.450 DEBUG:boto:Data: 2022-09-02T05:55:46.451 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.451 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.452 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.452 DEBUG:boto:Params: {} 2022-09-02T05:55:46.453 DEBUG:boto:Token: None 2022-09-02T05:55:46.454 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34 2022-09-02T05:55:46.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RQL56huHdxMu4istTOq/YukyKDk= 2022-09-02T05:55:46.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RQL56huHdxMu4istTOq/YukyKDk=', 'Content-Length': '0'} 2022-09-02T05:55:46.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d0e16019b56c8a0-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.458 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:46.458 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:46.459 DEBUG:boto:Method: GET 2022-09-02T05:55:46.459 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:46.460 DEBUG:boto:Data: 2022-09-02T05:55:46.460 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.462 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.462 DEBUG:boto:Params: {} 2022-09-02T05:55:46.472 DEBUG:boto:Token: None 2022-09-02T05:55:46.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj34 2022-09-02T05:55:46.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RQL56huHdxMu4istTOq/YukyKDk= 2022-09-02T05:55:46.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RQL56huHdxMu4istTOq/YukyKDk=', 'Content-Length': '0'} 2022-09-02T05:55:46.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b88cd7176ac9fdc8-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.518 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.519 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.519 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.520 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.520 DEBUG:boto:Method: GET 2022-09-02T05:55:46.521 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.521 DEBUG:boto:Data: 2022-09-02T05:55:46.522 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.523 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.524 DEBUG:boto:Params: {} 2022-09-02T05:55:46.524 DEBUG:boto:Token: None 2022-09-02T05:55:46.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f73da3d802b423f-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.530 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.531 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.531 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.532 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.532 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.533 DEBUG:boto:Method: GET 2022-09-02T05:55:46.533 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:46.534 DEBUG:boto:Data: 2022-09-02T05:55:46.534 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.535 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.536 DEBUG:boto:Params: {} 2022-09-02T05:55:46.536 DEBUG:boto:Token: None 2022-09-02T05:55:46.537 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e28d5f0fd4a8806-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.543 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:55:46.544 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.545 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.545 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.546 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.547 DEBUG:boto:Method: GET 2022-09-02T05:55:46.547 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.548 DEBUG:boto:Data: 2022-09-02T05:55:46.548 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.549 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.550 DEBUG:boto:Params: {} 2022-09-02T05:55:46.551 DEBUG:boto:Token: None 2022-09-02T05:55:46.552 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU= 2022-09-02T05:55:46.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU=', 'Content-Length': '0'} 2022-09-02T05:55:46.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007330d5a1c971653a-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.557 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.557 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.558 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.559 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.559 DEBUG:boto:Method: GET 2022-09-02T05:55:46.560 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.561 DEBUG:boto:Data: 2022-09-02T05:55:46.562 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.564 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.565 DEBUG:boto:Params: {} 2022-09-02T05:55:46.566 DEBUG:boto:Token: None 2022-09-02T05:55:46.568 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU= 2022-09-02T05:55:46.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU=', 'Content-Length': '0'} 2022-09-02T05:55:46.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000966872b16ce461a1-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.575 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.575 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.576 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.576 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.577 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.577 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.578 DEBUG:boto:Data: 2022-09-02T05:55:46.578 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.579 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.579 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.580 DEBUG:boto:Params: {} 2022-09-02T05:55:46.580 DEBUG:boto:Token: None 2022-09-02T05:55:46.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3HPzqbG1wOqPnEDXcIkL/irgXR4= 2022-09-02T05:55:46.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3HPzqbG1wOqPnEDXcIkL/irgXR4=', 'Content-Length': '0'} 2022-09-02T05:55:46.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e36861e2265c78d-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.585 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.586 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.586 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.587 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.587 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.588 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.588 DEBUG:boto:Data: 2022-09-02T05:55:46.589 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.590 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.590 DEBUG:boto:Params: {} 2022-09-02T05:55:46.591 DEBUG:boto:Token: None 2022-09-02T05:55:46.592 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3HPzqbG1wOqPnEDXcIkL/irgXR4= 2022-09-02T05:55:46.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3HPzqbG1wOqPnEDXcIkL/irgXR4=', 'Content-Length': '0'} 2022-09-02T05:55:46.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b9edc6626b53ba5-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:55:46.597 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.597 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.598 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.599 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.599 DEBUG:boto:Method: GET 2022-09-02T05:55:46.600 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.600 DEBUG:boto:Data: 2022-09-02T05:55:46.601 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.603 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.603 DEBUG:boto:Params: {} 2022-09-02T05:55:46.604 DEBUG:boto:Token: None 2022-09-02T05:55:46.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU= 2022-09-02T05:55:46.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU=', 'Content-Length': '0'} 2022-09-02T05:55:46.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d80b3f8634e9c6e-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.609 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.610 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.610 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.611 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.611 DEBUG:boto:Method: GET 2022-09-02T05:55:46.612 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.612 DEBUG:boto:Data: 2022-09-02T05:55:46.613 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.614 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.614 DEBUG:boto:Params: {} 2022-09-02T05:55:46.615 DEBUG:boto:Token: None 2022-09-02T05:55:46.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:46.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU= 2022-09-02T05:55:46.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QAxTmd3axTsFzHAlxx07l3CsvVU=', 'Content-Length': '0'} 2022-09-02T05:55:46.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f58fd8019981cc-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.620 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.621 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.622 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.622 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.623 DEBUG:boto:Method: GET 2022-09-02T05:55:46.623 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.624 DEBUG:boto:Data: 2022-09-02T05:55:46.624 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.625 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.626 DEBUG:boto:Params: {} 2022-09-02T05:55:46.626 DEBUG:boto:Token: None 2022-09-02T05:55:46.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.628 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086271f68cc63e2da-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.632 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.633 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.633 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.634 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.634 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.635 DEBUG:boto:Method: GET 2022-09-02T05:55:46.635 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.636 DEBUG:boto:Data: 2022-09-02T05:55:46.636 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.637 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.638 DEBUG:boto:Params: {} 2022-09-02T05:55:46.639 DEBUG:boto:Token: None 2022-09-02T05:55:46.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081867b4841f7cb36-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.645 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.646 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.647 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.647 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.648 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:46.648 DEBUG:boto:Data: 2022-09-02T05:55:46.649 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.649 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.650 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.651 DEBUG:boto:Params: {} 2022-09-02T05:55:46.651 DEBUG:boto:Token: None 2022-09-02T05:55:46.652 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38 2022-09-02T05:55:46.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LrMxB69i+Nu2X33GsiaIHWgvMJY= 2022-09-02T05:55:46.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LrMxB69i+Nu2X33GsiaIHWgvMJY=', 'Content-Length': '0'} 2022-09-02T05:55:46.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004408865d8739fbe8-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.656 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.656 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.657 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.657 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:46.658 DEBUG:boto:Data: 2022-09-02T05:55:46.658 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.659 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.660 DEBUG:boto:Params: {} 2022-09-02T05:55:46.661 DEBUG:boto:Token: None 2022-09-02T05:55:46.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38 2022-09-02T05:55:46.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LrMxB69i+Nu2X33GsiaIHWgvMJY= 2022-09-02T05:55:46.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LrMxB69i+Nu2X33GsiaIHWgvMJY=', 'Content-Length': '0'} 2022-09-02T05:55:46.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f7d745e2ce69fc-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:55:46.668 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.668 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.669 DEBUG:boto:Method: GET 2022-09-02T05:55:46.670 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:46.671 DEBUG:boto:Data: 2022-09-02T05:55:46.671 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.672 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.673 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.673 DEBUG:boto:Params: {} 2022-09-02T05:55:46.674 DEBUG:boto:Token: None 2022-09-02T05:55:46.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38 2022-09-02T05:55:46.675 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SU/dktYshn1npsrYesi3klhb7Zs= 2022-09-02T05:55:46.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SU/dktYshn1npsrYesi3klhb7Zs=', 'Content-Length': '0'} 2022-09-02T05:55:46.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e56902470c9af00-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.679 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:46.679 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:46.680 DEBUG:boto:Method: GET 2022-09-02T05:55:46.680 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:46.681 DEBUG:boto:Data: 2022-09-02T05:55:46.681 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.683 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.684 DEBUG:boto:Params: {} 2022-09-02T05:55:46.685 DEBUG:boto:Token: None 2022-09-02T05:55:46.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj38 2022-09-02T05:55:46.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SU/dktYshn1npsrYesi3klhb7Zs= 2022-09-02T05:55:46.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SU/dktYshn1npsrYesi3klhb7Zs=', 'Content-Length': '0'} 2022-09-02T05:55:46.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea7a48335a4efc09-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.738 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.741 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.742 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.743 DEBUG:boto:Method: GET 2022-09-02T05:55:46.744 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.745 DEBUG:boto:Data: 2022-09-02T05:55:46.746 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.748 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.749 DEBUG:boto:Params: {} 2022-09-02T05:55:46.751 DEBUG:boto:Token: None 2022-09-02T05:55:46.752 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004236de73b7b2b266-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.757 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.759 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.760 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.761 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.762 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.763 DEBUG:boto:Method: GET 2022-09-02T05:55:46.764 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:46.765 DEBUG:boto:Data: 2022-09-02T05:55:46.765 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.767 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.768 DEBUG:boto:Params: {} 2022-09-02T05:55:46.769 DEBUG:boto:Token: None 2022-09-02T05:55:46.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000bded5be75cb15b-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.779 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:55:46.782 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.783 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.784 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.785 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.786 DEBUG:boto:Method: GET 2022-09-02T05:55:46.787 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.788 DEBUG:boto:Data: 2022-09-02T05:55:46.789 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.791 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.792 DEBUG:boto:Params: {} 2022-09-02T05:55:46.793 DEBUG:boto:Token: None 2022-09-02T05:55:46.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE= 2022-09-02T05:55:46.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE=', 'Content-Length': '0'} 2022-09-02T05:55:46.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4698b29a8bc4355-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.801 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.802 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.803 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.804 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.805 DEBUG:boto:Method: GET 2022-09-02T05:55:46.806 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.807 DEBUG:boto:Data: 2022-09-02T05:55:46.808 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.809 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.810 DEBUG:boto:Params: {} 2022-09-02T05:55:46.811 DEBUG:boto:Token: None 2022-09-02T05:55:46.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE= 2022-09-02T05:55:46.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE=', 'Content-Length': '0'} 2022-09-02T05:55:46.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f90b4bd4c51eb5d-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.819 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.820 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.821 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.823 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.823 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.824 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.824 DEBUG:boto:Data: 2022-09-02T05:55:46.825 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.826 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.827 DEBUG:boto:Params: {} 2022-09-02T05:55:46.827 DEBUG:boto:Token: None 2022-09-02T05:55:46.828 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VSxvX46lw6Bm90EuC+VAYICxVdg= 2022-09-02T05:55:46.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VSxvX46lw6Bm90EuC+VAYICxVdg=', 'Content-Length': '0'} 2022-09-02T05:55:46.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae42a2a078a188f-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.833 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.833 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.834 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.835 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.836 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.837 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.838 DEBUG:boto:Data: 2022-09-02T05:55:46.838 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.840 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.841 DEBUG:boto:Params: {} 2022-09-02T05:55:46.842 DEBUG:boto:Token: None 2022-09-02T05:55:46.844 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VSxvX46lw6Bm90EuC+VAYICxVdg= 2022-09-02T05:55:46.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VSxvX46lw6Bm90EuC+VAYICxVdg=', 'Content-Length': '0'} 2022-09-02T05:55:46.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067f9e252ed0ee02-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:55:46.850 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.851 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.851 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.852 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.852 DEBUG:boto:Method: GET 2022-09-02T05:55:46.853 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.854 DEBUG:boto:Data: 2022-09-02T05:55:46.854 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.856 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.856 DEBUG:boto:Params: {} 2022-09-02T05:55:46.857 DEBUG:boto:Token: None 2022-09-02T05:55:46.858 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE= 2022-09-02T05:55:46.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE=', 'Content-Length': '0'} 2022-09-02T05:55:46.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1b07b634c56c64-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.862 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.863 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.864 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.864 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.865 DEBUG:boto:Method: GET 2022-09-02T05:55:46.866 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.866 DEBUG:boto:Data: 2022-09-02T05:55:46.867 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.868 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.868 DEBUG:boto:Params: {} 2022-09-02T05:55:46.869 DEBUG:boto:Token: None 2022-09-02T05:55:46.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:46.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE= 2022-09-02T05:55:46.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I8FYaRt6ytyRpjmn8pxw4J5HsEE=', 'Content-Length': '0'} 2022-09-02T05:55:46.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9cd073efffd172a-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.875 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.876 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.877 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.878 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.879 DEBUG:boto:Method: GET 2022-09-02T05:55:46.879 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.880 DEBUG:boto:Data: 2022-09-02T05:55:46.881 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.882 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.883 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.883 DEBUG:boto:Params: {} 2022-09-02T05:55:46.884 DEBUG:boto:Token: None 2022-09-02T05:55:46.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf742a28692a4e7d-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.890 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.891 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.892 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.892 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.893 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.893 DEBUG:boto:Method: GET 2022-09-02T05:55:46.894 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.894 DEBUG:boto:Data: 2022-09-02T05:55:46.895 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.896 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.897 DEBUG:boto:Params: {} 2022-09-02T05:55:46.897 DEBUG:boto:Token: None 2022-09-02T05:55:46.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:46.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000443d76dbad97fa8a-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.904 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:46.905 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.906 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.906 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.907 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:46.908 DEBUG:boto:Data: 2022-09-02T05:55:46.909 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.910 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.910 DEBUG:boto:Params: {} 2022-09-02T05:55:46.911 DEBUG:boto:Token: None 2022-09-02T05:55:46.911 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44 2022-09-02T05:55:46.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IFx9zEN5ojXJNIU4ofyDOZODSVI= 2022-09-02T05:55:46.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IFx9zEN5ojXJNIU4ofyDOZODSVI=', 'Content-Length': '0'} 2022-09-02T05:55:46.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0069c4863e55a44-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.915 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.916 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.916 DEBUG:boto:Method: HEAD 2022-09-02T05:55:46.917 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:46.917 DEBUG:boto:Data: 2022-09-02T05:55:46.918 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.919 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.920 DEBUG:boto:Params: {} 2022-09-02T05:55:46.920 DEBUG:boto:Token: None 2022-09-02T05:55:46.921 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44 2022-09-02T05:55:46.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IFx9zEN5ojXJNIU4ofyDOZODSVI= 2022-09-02T05:55:46.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IFx9zEN5ojXJNIU4ofyDOZODSVI=', 'Content-Length': '0'} 2022-09-02T05:55:46.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a4780af6fb0939-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:55:46.926 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.926 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.927 DEBUG:boto:Method: GET 2022-09-02T05:55:46.927 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:46.928 DEBUG:boto:Data: 2022-09-02T05:55:46.928 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.929 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.929 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.930 DEBUG:boto:Params: {} 2022-09-02T05:55:46.931 DEBUG:boto:Token: None 2022-09-02T05:55:46.931 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44 2022-09-02T05:55:46.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e7inrm3yolrMM4mPCDqu5oVvHtY= 2022-09-02T05:55:46.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e7inrm3yolrMM4mPCDqu5oVvHtY=', 'Content-Length': '0'} 2022-09-02T05:55:46.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46223ed5cc3b15e-0063119ae2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.978 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:46.979 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:46.980 DEBUG:boto:Method: GET 2022-09-02T05:55:46.980 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:46.981 DEBUG:boto:Data: 2022-09-02T05:55:46.982 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:46.983 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.984 DEBUG:boto:Params: {} 2022-09-02T05:55:46.985 DEBUG:boto:Token: None 2022-09-02T05:55:46.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/obj44 2022-09-02T05:55:46.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e7inrm3yolrMM4mPCDqu5oVvHtY= 2022-09-02T05:55:46.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e7inrm3yolrMM4mPCDqu5oVvHtY=', 'Content-Length': '0'} 2022-09-02T05:55:46.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7bc3b93aee9628f-0063119ae2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:46.991 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:46.992 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:46.992 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.993 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.993 DEBUG:boto:Method: GET 2022-09-02T05:55:46.994 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:46.995 DEBUG:boto:Data: 2022-09-02T05:55:46.995 DEBUG:boto:Headers: {} 2022-09-02T05:55:46.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:46.996 DEBUG:boto:Port: 8000 2022-09-02T05:55:46.997 DEBUG:boto:Params: {} 2022-09-02T05:55:46.997 DEBUG:boto:Token: None 2022-09-02T05:55:46.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:46 GMT /xljbkf-74/ 2022-09-02T05:55:46.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg= 2022-09-02T05:55:46.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vSFE+926Lcep0OTxdAPEvWY0hgg=', 'Content-Length': '0'} 2022-09-02T05:55:47.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4444b6b0e6940d-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.003 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.004 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.004 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.005 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:47.005 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:47.006 DEBUG:boto:Method: GET 2022-09-02T05:55:47.007 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:47.007 DEBUG:boto:Data: 2022-09-02T05:55:47.008 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.009 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.010 DEBUG:boto:Params: {} 2022-09-02T05:55:47.010 DEBUG:boto:Token: None 2022-09-02T05:55:47.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018abff6a256baf3e-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.058 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:55:47.060 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.061 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.061 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.062 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.063 DEBUG:boto:Method: GET 2022-09-02T05:55:47.063 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.064 DEBUG:boto:Data: 2022-09-02T05:55:47.064 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.065 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.066 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.066 DEBUG:boto:Params: {} 2022-09-02T05:55:47.067 DEBUG:boto:Token: None 2022-09-02T05:55:47.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs= 2022-09-02T05:55:47.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs=', 'Content-Length': '0'} 2022-09-02T05:55:47.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cbcea2d7a7158c7-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.072 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.072 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.073 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.073 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.074 DEBUG:boto:Method: GET 2022-09-02T05:55:47.075 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.075 DEBUG:boto:Data: 2022-09-02T05:55:47.076 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.077 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.078 DEBUG:boto:Params: {} 2022-09-02T05:55:47.079 DEBUG:boto:Token: None 2022-09-02T05:55:47.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs= 2022-09-02T05:55:47.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs=', 'Content-Length': '0'} 2022-09-02T05:55:47.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1ebdfbfa1db012-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.084 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.085 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.086 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.087 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.088 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.089 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.090 DEBUG:boto:Data: 2022-09-02T05:55:47.091 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.092 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.093 DEBUG:boto:Params: {} 2022-09-02T05:55:47.093 DEBUG:boto:Token: None 2022-09-02T05:55:47.094 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R2YWFGcf+tcQ7fMCwYYpbbLpg5w= 2022-09-02T05:55:47.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R2YWFGcf+tcQ7fMCwYYpbbLpg5w=', 'Content-Length': '0'} 2022-09-02T05:55:47.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d47e732024d41d-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.098 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.098 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.099 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.099 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.100 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.100 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.101 DEBUG:boto:Data: 2022-09-02T05:55:47.101 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.102 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.103 DEBUG:boto:Params: {} 2022-09-02T05:55:47.103 DEBUG:boto:Token: None 2022-09-02T05:55:47.104 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R2YWFGcf+tcQ7fMCwYYpbbLpg5w= 2022-09-02T05:55:47.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R2YWFGcf+tcQ7fMCwYYpbbLpg5w=', 'Content-Length': '0'} 2022-09-02T05:55:47.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001acfa79502c80d38-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:55:47.109 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.109 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.110 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.111 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.111 DEBUG:boto:Method: GET 2022-09-02T05:55:47.112 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.112 DEBUG:boto:Data: 2022-09-02T05:55:47.113 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.114 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.114 DEBUG:boto:Params: {} 2022-09-02T05:55:47.115 DEBUG:boto:Token: None 2022-09-02T05:55:47.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs= 2022-09-02T05:55:47.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs=', 'Content-Length': '0'} 2022-09-02T05:55:47.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7005a9c14a0c1f-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.120 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.120 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.121 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.121 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.122 DEBUG:boto:Method: GET 2022-09-02T05:55:47.122 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.123 DEBUG:boto:Data: 2022-09-02T05:55:47.123 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.124 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.125 DEBUG:boto:Params: {} 2022-09-02T05:55:47.126 DEBUG:boto:Token: None 2022-09-02T05:55:47.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:47.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs= 2022-09-02T05:55:47.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZOuS99rmGVdumbWzEkkUjuqTTvs=', 'Content-Length': '0'} 2022-09-02T05:55:47.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb5aec218f2cf58-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.130 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.131 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.131 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.132 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.132 DEBUG:boto:Method: GET 2022-09-02T05:55:47.133 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.134 DEBUG:boto:Data: 2022-09-02T05:55:47.134 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.135 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.135 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.136 DEBUG:boto:Params: {} 2022-09-02T05:55:47.136 DEBUG:boto:Token: None 2022-09-02T05:55:47.137 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b25e43bf4bed5801-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.182 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.184 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.185 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.186 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.188 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.189 DEBUG:boto:Method: GET 2022-09-02T05:55:47.189 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.191 DEBUG:boto:Data: 2022-09-02T05:55:47.192 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.194 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.195 DEBUG:boto:Params: {} 2022-09-02T05:55:47.196 DEBUG:boto:Token: None 2022-09-02T05:55:47.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d58166c96f043e3c-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.205 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.206 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.207 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.207 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.208 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:47.208 DEBUG:boto:Data: 2022-09-02T05:55:47.209 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.210 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.211 DEBUG:boto:Params: {} 2022-09-02T05:55:47.211 DEBUG:boto:Token: None 2022-09-02T05:55:47.212 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46 2022-09-02T05:55:47.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xlM0mJfe/eBfgmJTgSAZDP3bRrc= 2022-09-02T05:55:47.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xlM0mJfe/eBfgmJTgSAZDP3bRrc=', 'Content-Length': '0'} 2022-09-02T05:55:47.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e57b71f08122982-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.216 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.217 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.217 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.218 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:47.218 DEBUG:boto:Data: 2022-09-02T05:55:47.219 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.220 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.220 DEBUG:boto:Params: {} 2022-09-02T05:55:47.221 DEBUG:boto:Token: None 2022-09-02T05:55:47.221 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46 2022-09-02T05:55:47.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xlM0mJfe/eBfgmJTgSAZDP3bRrc= 2022-09-02T05:55:47.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xlM0mJfe/eBfgmJTgSAZDP3bRrc=', 'Content-Length': '0'} 2022-09-02T05:55:47.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089652d937d93ff9c-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:55:47.226 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.227 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.227 DEBUG:boto:Method: GET 2022-09-02T05:55:47.228 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:47.228 DEBUG:boto:Data: 2022-09-02T05:55:47.229 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.230 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.231 DEBUG:boto:Params: {} 2022-09-02T05:55:47.231 DEBUG:boto:Token: None 2022-09-02T05:55:47.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46 2022-09-02T05:55:47.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZTtfd0eteMHV7RfYGrPfOZ/OtKk= 2022-09-02T05:55:47.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZTtfd0eteMHV7RfYGrPfOZ/OtKk=', 'Content-Length': '0'} 2022-09-02T05:55:47.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000265225d91d1da654-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.236 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:47.237 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:47.237 DEBUG:boto:Method: GET 2022-09-02T05:55:47.238 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:47.238 DEBUG:boto:Data: 2022-09-02T05:55:47.239 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.240 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.241 DEBUG:boto:Params: {} 2022-09-02T05:55:47.241 DEBUG:boto:Token: None 2022-09-02T05:55:47.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj46 2022-09-02T05:55:47.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZTtfd0eteMHV7RfYGrPfOZ/OtKk= 2022-09-02T05:55:47.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZTtfd0eteMHV7RfYGrPfOZ/OtKk=', 'Content-Length': '0'} 2022-09-02T05:55:47.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001120875018a56057-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.290 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.291 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.292 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.293 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.294 DEBUG:boto:Method: GET 2022-09-02T05:55:47.295 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.295 DEBUG:boto:Data: 2022-09-02T05:55:47.296 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.297 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.297 DEBUG:boto:Params: {} 2022-09-02T05:55:47.298 DEBUG:boto:Token: None 2022-09-02T05:55:47.299 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ef610277d9bb13-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.304 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.305 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.305 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.306 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.307 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.307 DEBUG:boto:Method: GET 2022-09-02T05:55:47.308 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:47.308 DEBUG:boto:Data: 2022-09-02T05:55:47.309 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.310 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.310 DEBUG:boto:Params: {} 2022-09-02T05:55:47.311 DEBUG:boto:Token: None 2022-09-02T05:55:47.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007496276f0996f2ea-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.318 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:55:47.319 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.320 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.321 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.321 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.322 DEBUG:boto:Method: GET 2022-09-02T05:55:47.322 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.323 DEBUG:boto:Data: 2022-09-02T05:55:47.323 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.325 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.326 DEBUG:boto:Params: {} 2022-09-02T05:55:47.326 DEBUG:boto:Token: None 2022-09-02T05:55:47.327 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc= 2022-09-02T05:55:47.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc=', 'Content-Length': '0'} 2022-09-02T05:55:47.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b631eff4385291a-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.331 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.332 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.333 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.333 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.334 DEBUG:boto:Method: GET 2022-09-02T05:55:47.334 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.335 DEBUG:boto:Data: 2022-09-02T05:55:47.336 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.337 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.338 DEBUG:boto:Params: {} 2022-09-02T05:55:47.339 DEBUG:boto:Token: None 2022-09-02T05:55:47.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.340 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc= 2022-09-02T05:55:47.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc=', 'Content-Length': '0'} 2022-09-02T05:55:47.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de8124e86752f2b-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.345 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.345 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.346 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.346 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.347 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.347 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.348 DEBUG:boto:Data: 2022-09-02T05:55:47.348 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.350 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.350 DEBUG:boto:Params: {} 2022-09-02T05:55:47.352 DEBUG:boto:Token: None 2022-09-02T05:55:47.353 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b/d6iWgEmiUa6mBljKdDWKyUB5g= 2022-09-02T05:55:47.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b/d6iWgEmiUa6mBljKdDWKyUB5g=', 'Content-Length': '0'} 2022-09-02T05:55:47.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030ad71ae534fdd2b-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.359 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.360 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.361 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.362 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.363 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.364 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.365 DEBUG:boto:Data: 2022-09-02T05:55:47.366 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.368 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.368 DEBUG:boto:Params: {} 2022-09-02T05:55:47.369 DEBUG:boto:Token: None 2022-09-02T05:55:47.370 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b/d6iWgEmiUa6mBljKdDWKyUB5g= 2022-09-02T05:55:47.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b/d6iWgEmiUa6mBljKdDWKyUB5g=', 'Content-Length': '0'} 2022-09-02T05:55:47.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004491170110b77637-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:55:47.375 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.376 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.376 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.377 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.377 DEBUG:boto:Method: GET 2022-09-02T05:55:47.378 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.378 DEBUG:boto:Data: 2022-09-02T05:55:47.379 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.380 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.381 DEBUG:boto:Params: {} 2022-09-02T05:55:47.381 DEBUG:boto:Token: None 2022-09-02T05:55:47.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc= 2022-09-02T05:55:47.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc=', 'Content-Length': '0'} 2022-09-02T05:55:47.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c1d3a13b543d14-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.426 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.427 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.427 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.428 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.429 DEBUG:boto:Method: GET 2022-09-02T05:55:47.430 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.430 DEBUG:boto:Data: 2022-09-02T05:55:47.431 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.432 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.432 DEBUG:boto:Params: {} 2022-09-02T05:55:47.433 DEBUG:boto:Token: None 2022-09-02T05:55:47.434 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:47.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc= 2022-09-02T05:55:47.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wv5RkM5IdJBF5tgG9Er3RXfKXDc=', 'Content-Length': '0'} 2022-09-02T05:55:47.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c8a44cc216520f-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.438 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.438 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.439 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.439 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.440 DEBUG:boto:Method: GET 2022-09-02T05:55:47.440 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.441 DEBUG:boto:Data: 2022-09-02T05:55:47.442 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.443 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.443 DEBUG:boto:Params: {} 2022-09-02T05:55:47.444 DEBUG:boto:Token: None 2022-09-02T05:55:47.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0e3a1b23dbac89-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.449 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.450 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.450 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.451 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.451 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.452 DEBUG:boto:Method: GET 2022-09-02T05:55:47.452 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.453 DEBUG:boto:Data: 2022-09-02T05:55:47.453 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.454 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.455 DEBUG:boto:Params: {} 2022-09-02T05:55:47.456 DEBUG:boto:Token: None 2022-09-02T05:55:47.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5bf7782a383ae8c-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.462 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.463 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.463 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.464 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.464 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:47.465 DEBUG:boto:Data: 2022-09-02T05:55:47.465 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.466 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.467 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.467 DEBUG:boto:Params: {} 2022-09-02T05:55:47.468 DEBUG:boto:Token: None 2022-09-02T05:55:47.468 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51 2022-09-02T05:55:47.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VPf5n+oeolD7rD6VumKry/h0kT4= 2022-09-02T05:55:47.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VPf5n+oeolD7rD6VumKry/h0kT4=', 'Content-Length': '0'} 2022-09-02T05:55:47.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080fc7ea87dcf260e-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.472 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.473 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.473 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.474 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:47.474 DEBUG:boto:Data: 2022-09-02T05:55:47.475 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.476 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.476 DEBUG:boto:Params: {} 2022-09-02T05:55:47.477 DEBUG:boto:Token: None 2022-09-02T05:55:47.478 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51 2022-09-02T05:55:47.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VPf5n+oeolD7rD6VumKry/h0kT4= 2022-09-02T05:55:47.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VPf5n+oeolD7rD6VumKry/h0kT4=', 'Content-Length': '0'} 2022-09-02T05:55:47.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7968ed69c9fd701-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:55:47.483 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.483 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.484 DEBUG:boto:Method: GET 2022-09-02T05:55:47.484 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:47.485 DEBUG:boto:Data: 2022-09-02T05:55:47.486 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.486 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.487 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.487 DEBUG:boto:Params: {} 2022-09-02T05:55:47.488 DEBUG:boto:Token: None 2022-09-02T05:55:47.489 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51 2022-09-02T05:55:47.489 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TDE/Wd4I8mcvNRyJSkEAZ1yoODE= 2022-09-02T05:55:47.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TDE/Wd4I8mcvNRyJSkEAZ1yoODE=', 'Content-Length': '0'} 2022-09-02T05:55:47.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66ef29be23b71f2-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.493 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:47.494 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:47.494 DEBUG:boto:Method: GET 2022-09-02T05:55:47.495 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:47.495 DEBUG:boto:Data: 2022-09-02T05:55:47.496 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.497 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.497 DEBUG:boto:Params: {} 2022-09-02T05:55:47.498 DEBUG:boto:Token: None 2022-09-02T05:55:47.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj51 2022-09-02T05:55:47.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TDE/Wd4I8mcvNRyJSkEAZ1yoODE= 2022-09-02T05:55:47.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TDE/Wd4I8mcvNRyJSkEAZ1yoODE=', 'Content-Length': '0'} 2022-09-02T05:55:47.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8838a0885ac0097-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.504 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.504 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.505 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.506 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.506 DEBUG:boto:Method: GET 2022-09-02T05:55:47.507 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.507 DEBUG:boto:Data: 2022-09-02T05:55:47.508 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.509 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.510 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.510 DEBUG:boto:Params: {} 2022-09-02T05:55:47.511 DEBUG:boto:Token: None 2022-09-02T05:55:47.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cbc72245ce25d3c-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.516 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.517 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.518 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.518 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.519 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.520 DEBUG:boto:Method: GET 2022-09-02T05:55:47.520 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:47.521 DEBUG:boto:Data: 2022-09-02T05:55:47.522 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.523 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.524 DEBUG:boto:Params: {} 2022-09-02T05:55:47.524 DEBUG:boto:Token: None 2022-09-02T05:55:47.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022efa2feb9ae0032-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.532 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:55:47.534 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.534 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.535 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.536 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.537 DEBUG:boto:Method: GET 2022-09-02T05:55:47.538 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.538 DEBUG:boto:Data: 2022-09-02T05:55:47.539 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.539 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.540 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.540 DEBUG:boto:Params: {} 2022-09-02T05:55:47.541 DEBUG:boto:Token: None 2022-09-02T05:55:47.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk= 2022-09-02T05:55:47.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk=', 'Content-Length': '0'} 2022-09-02T05:55:47.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2a322071da8300-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.546 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.546 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.547 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.548 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.548 DEBUG:boto:Method: GET 2022-09-02T05:55:47.549 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.550 DEBUG:boto:Data: 2022-09-02T05:55:47.550 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.551 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.552 DEBUG:boto:Params: {} 2022-09-02T05:55:47.552 DEBUG:boto:Token: None 2022-09-02T05:55:47.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk= 2022-09-02T05:55:47.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk=', 'Content-Length': '0'} 2022-09-02T05:55:47.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bdcc41026269381-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.602 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.603 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.604 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.605 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.607 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.607 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.608 DEBUG:boto:Data: 2022-09-02T05:55:47.609 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.611 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.612 DEBUG:boto:Params: {} 2022-09-02T05:55:47.613 DEBUG:boto:Token: None 2022-09-02T05:55:47.614 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N13YM1AN/vqlMudplIF/5K0Wpqs= 2022-09-02T05:55:47.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N13YM1AN/vqlMudplIF/5K0Wpqs=', 'Content-Length': '0'} 2022-09-02T05:55:47.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a24521d249724e-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.620 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.620 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.621 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.622 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.623 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.623 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.624 DEBUG:boto:Data: 2022-09-02T05:55:47.625 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.627 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.628 DEBUG:boto:Params: {} 2022-09-02T05:55:47.629 DEBUG:boto:Token: None 2022-09-02T05:55:47.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N13YM1AN/vqlMudplIF/5K0Wpqs= 2022-09-02T05:55:47.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N13YM1AN/vqlMudplIF/5K0Wpqs=', 'Content-Length': '0'} 2022-09-02T05:55:47.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81ae9571d3255e4-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:55:47.636 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.637 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.637 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.638 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.638 DEBUG:boto:Method: GET 2022-09-02T05:55:47.639 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.640 DEBUG:boto:Data: 2022-09-02T05:55:47.640 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.641 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.642 DEBUG:boto:Params: {} 2022-09-02T05:55:47.642 DEBUG:boto:Token: None 2022-09-02T05:55:47.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk= 2022-09-02T05:55:47.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk=', 'Content-Length': '0'} 2022-09-02T05:55:47.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7bfebacd3424a8-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.649 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.649 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.650 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.650 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.651 DEBUG:boto:Method: GET 2022-09-02T05:55:47.651 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.652 DEBUG:boto:Data: 2022-09-02T05:55:47.652 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.653 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.654 DEBUG:boto:Params: {} 2022-09-02T05:55:47.654 DEBUG:boto:Token: None 2022-09-02T05:55:47.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:47.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk= 2022-09-02T05:55:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR4/tADB23p9R10ff9y6OcSvRgk=', 'Content-Length': '0'} 2022-09-02T05:55:47.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093e032ac78bf4e96-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.659 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.660 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.660 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.661 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.661 DEBUG:boto:Method: GET 2022-09-02T05:55:47.662 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.663 DEBUG:boto:Data: 2022-09-02T05:55:47.663 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.664 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.665 DEBUG:boto:Params: {} 2022-09-02T05:55:47.665 DEBUG:boto:Token: None 2022-09-02T05:55:47.666 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f3a475ed97b171-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.670 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.671 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.672 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.672 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.673 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.673 DEBUG:boto:Method: GET 2022-09-02T05:55:47.674 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.674 DEBUG:boto:Data: 2022-09-02T05:55:47.675 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.676 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.676 DEBUG:boto:Params: {} 2022-09-02T05:55:47.677 DEBUG:boto:Token: None 2022-09-02T05:55:47.677 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.678 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000235969bd78f530d5-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.726 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.727 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.728 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.729 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.730 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:47.731 DEBUG:boto:Data: 2022-09-02T05:55:47.732 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.734 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.735 DEBUG:boto:Params: {} 2022-09-02T05:55:47.735 DEBUG:boto:Token: None 2022-09-02T05:55:47.736 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55 2022-09-02T05:55:47.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znccfmaD/zfbXi5aUmp1gXgmP98= 2022-09-02T05:55:47.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znccfmaD/zfbXi5aUmp1gXgmP98=', 'Content-Length': '0'} 2022-09-02T05:55:47.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89e3bc5f656f781-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.740 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.741 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.741 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.742 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:47.743 DEBUG:boto:Data: 2022-09-02T05:55:47.743 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.744 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.745 DEBUG:boto:Params: {} 2022-09-02T05:55:47.746 DEBUG:boto:Token: None 2022-09-02T05:55:47.746 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55 2022-09-02T05:55:47.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:znccfmaD/zfbXi5aUmp1gXgmP98= 2022-09-02T05:55:47.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:znccfmaD/zfbXi5aUmp1gXgmP98=', 'Content-Length': '0'} 2022-09-02T05:55:47.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7479378db2af949-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:55:47.751 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.752 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.752 DEBUG:boto:Method: GET 2022-09-02T05:55:47.753 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:47.754 DEBUG:boto:Data: 2022-09-02T05:55:47.754 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.755 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.756 DEBUG:boto:Params: {} 2022-09-02T05:55:47.756 DEBUG:boto:Token: None 2022-09-02T05:55:47.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55 2022-09-02T05:55:47.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5X6wLdw6YT1AbfCKkaBsoLZ2V4= 2022-09-02T05:55:47.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5X6wLdw6YT1AbfCKkaBsoLZ2V4=', 'Content-Length': '0'} 2022-09-02T05:55:47.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a7c81a3be62492-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.761 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:47.762 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:47.763 DEBUG:boto:Method: GET 2022-09-02T05:55:47.763 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:47.764 DEBUG:boto:Data: 2022-09-02T05:55:47.764 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.765 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.766 DEBUG:boto:Params: {} 2022-09-02T05:55:47.766 DEBUG:boto:Token: None 2022-09-02T05:55:47.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj55 2022-09-02T05:55:47.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J5X6wLdw6YT1AbfCKkaBsoLZ2V4= 2022-09-02T05:55:47.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J5X6wLdw6YT1AbfCKkaBsoLZ2V4=', 'Content-Length': '0'} 2022-09-02T05:55:47.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0630f610d84cd56-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.771 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.772 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.772 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.773 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.773 DEBUG:boto:Method: GET 2022-09-02T05:55:47.774 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.775 DEBUG:boto:Data: 2022-09-02T05:55:47.775 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.776 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.776 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.777 DEBUG:boto:Params: {} 2022-09-02T05:55:47.777 DEBUG:boto:Token: None 2022-09-02T05:55:47.778 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bab1d040f50d3c7-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.783 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.784 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.785 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.785 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.786 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.786 DEBUG:boto:Method: GET 2022-09-02T05:55:47.787 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:47.787 DEBUG:boto:Data: 2022-09-02T05:55:47.788 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.789 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.790 DEBUG:boto:Params: {} 2022-09-02T05:55:47.791 DEBUG:boto:Token: None 2022-09-02T05:55:47.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c228f3431475c50-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.797 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:55:47.798 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.799 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.799 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.800 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.800 DEBUG:boto:Method: GET 2022-09-02T05:55:47.801 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.802 DEBUG:boto:Data: 2022-09-02T05:55:47.802 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.803 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.804 DEBUG:boto:Params: {} 2022-09-02T05:55:47.805 DEBUG:boto:Token: None 2022-09-02T05:55:47.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI= 2022-09-02T05:55:47.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI=', 'Content-Length': '0'} 2022-09-02T05:55:47.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ba0f32ad8f7b69b-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.850 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.851 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.851 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.852 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.853 DEBUG:boto:Method: GET 2022-09-02T05:55:47.853 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.854 DEBUG:boto:Data: 2022-09-02T05:55:47.854 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.856 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.856 DEBUG:boto:Params: {} 2022-09-02T05:55:47.857 DEBUG:boto:Token: None 2022-09-02T05:55:47.858 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI= 2022-09-02T05:55:47.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI=', 'Content-Length': '0'} 2022-09-02T05:55:47.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066989669bd152879-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.862 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.863 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.863 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.864 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.864 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.865 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.866 DEBUG:boto:Data: 2022-09-02T05:55:47.866 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.868 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.868 DEBUG:boto:Params: {} 2022-09-02T05:55:47.869 DEBUG:boto:Token: None 2022-09-02T05:55:47.869 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A6iBGiumT4pKXa8EFwwSWO9S6UQ= 2022-09-02T05:55:47.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A6iBGiumT4pKXa8EFwwSWO9S6UQ=', 'Content-Length': '0'} 2022-09-02T05:55:47.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014711a074903b349-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.874 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.875 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.875 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.876 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.877 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.877 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.878 DEBUG:boto:Data: 2022-09-02T05:55:47.878 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.879 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.880 DEBUG:boto:Params: {} 2022-09-02T05:55:47.881 DEBUG:boto:Token: None 2022-09-02T05:55:47.882 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A6iBGiumT4pKXa8EFwwSWO9S6UQ= 2022-09-02T05:55:47.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A6iBGiumT4pKXa8EFwwSWO9S6UQ=', 'Content-Length': '0'} 2022-09-02T05:55:47.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff889d40c631b90-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:55:47.887 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.888 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.888 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.889 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.890 DEBUG:boto:Method: GET 2022-09-02T05:55:47.891 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.891 DEBUG:boto:Data: 2022-09-02T05:55:47.892 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.893 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.893 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.894 DEBUG:boto:Params: {} 2022-09-02T05:55:47.894 DEBUG:boto:Token: None 2022-09-02T05:55:47.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI= 2022-09-02T05:55:47.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI=', 'Content-Length': '0'} 2022-09-02T05:55:47.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfebefba35e28d76-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.900 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.900 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.901 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.901 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.902 DEBUG:boto:Method: GET 2022-09-02T05:55:47.903 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.903 DEBUG:boto:Data: 2022-09-02T05:55:47.904 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.905 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.905 DEBUG:boto:Params: {} 2022-09-02T05:55:47.906 DEBUG:boto:Token: None 2022-09-02T05:55:47.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:47.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI= 2022-09-02T05:55:47.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q4DkrhLeTVWrXUylO7OKLrBVZXI=', 'Content-Length': '0'} 2022-09-02T05:55:47.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dff072a2c65c2b-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.912 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.913 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.913 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:47.914 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:47.915 DEBUG:boto:Method: GET 2022-09-02T05:55:47.916 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:47.916 DEBUG:boto:Data: 2022-09-02T05:55:47.917 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.918 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.918 DEBUG:boto:Params: {} 2022-09-02T05:55:47.919 DEBUG:boto:Token: None 2022-09-02T05:55:47.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de03c4101a866bc-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.924 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.925 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:47.926 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:47.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:47.927 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:47.928 DEBUG:boto:Method: GET 2022-09-02T05:55:47.928 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:47.929 DEBUG:boto:Data: 2022-09-02T05:55:47.929 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.930 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.931 DEBUG:boto:Params: {} 2022-09-02T05:55:47.932 DEBUG:boto:Token: None 2022-09-02T05:55:47.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/ 2022-09-02T05:55:47.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8= 2022-09-02T05:55:47.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6oLOaJ4ip0A1NeHkD8D/DsQZWj8=', 'Content-Length': '0'} 2022-09-02T05:55:47.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ace9a5628b2df486-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.978 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:47.979 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.980 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.980 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.981 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:47.981 DEBUG:boto:Data: 2022-09-02T05:55:47.982 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.983 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:47.984 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.985 DEBUG:boto:Params: {} 2022-09-02T05:55:47.985 DEBUG:boto:Token: None 2022-09-02T05:55:47.986 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57 2022-09-02T05:55:47.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xHqFAYvgN4wUmYL0tf6sGNneKa4= 2022-09-02T05:55:47.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xHqFAYvgN4wUmYL0tf6sGNneKa4=', 'Content-Length': '0'} 2022-09-02T05:55:47.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a331a27993dde2-0063119ae3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:47.991 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:47.991 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:47.992 DEBUG:boto:Method: HEAD 2022-09-02T05:55:47.992 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:47.993 DEBUG:boto:Data: 2022-09-02T05:55:47.993 DEBUG:boto:Headers: {} 2022-09-02T05:55:47.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:47.994 DEBUG:boto:Port: 8000 2022-09-02T05:55:47.995 DEBUG:boto:Params: {} 2022-09-02T05:55:47.995 DEBUG:boto:Token: None 2022-09-02T05:55:47.996 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:47 GMT /xljbkf-74/obj57 2022-09-02T05:55:47.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xHqFAYvgN4wUmYL0tf6sGNneKa4= 2022-09-02T05:55:47.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xHqFAYvgN4wUmYL0tf6sGNneKa4=', 'Content-Length': '0'} 2022-09-02T05:55:48.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464da58a0e28c699-0063119ae3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:55:48.002 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:48.002 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:48.003 DEBUG:boto:Method: GET 2022-09-02T05:55:48.003 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:48.004 DEBUG:boto:Data: 2022-09-02T05:55:48.004 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.005 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.006 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.006 DEBUG:boto:Params: {} 2022-09-02T05:55:48.007 DEBUG:boto:Token: None 2022-09-02T05:55:48.008 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj57 2022-09-02T05:55:48.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gU7vINuXMI0PKNtCDixikN6wWS0= 2022-09-02T05:55:48.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gU7vINuXMI0PKNtCDixikN6wWS0=', 'Content-Length': '0'} 2022-09-02T05:55:48.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3c01085de211c1-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.012 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:48.013 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:48.013 DEBUG:boto:Method: GET 2022-09-02T05:55:48.014 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:48.015 DEBUG:boto:Data: 2022-09-02T05:55:48.015 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.017 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.017 DEBUG:boto:Params: {} 2022-09-02T05:55:48.018 DEBUG:boto:Token: None 2022-09-02T05:55:48.019 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj57 2022-09-02T05:55:48.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gU7vINuXMI0PKNtCDixikN6wWS0= 2022-09-02T05:55:48.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gU7vINuXMI0PKNtCDixikN6wWS0=', 'Content-Length': '0'} 2022-09-02T05:55:48.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4734eb38a9e56da-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.023 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.024 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.025 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:48.025 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:48.026 DEBUG:boto:Method: GET 2022-09-02T05:55:48.026 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:48.027 DEBUG:boto:Data: 2022-09-02T05:55:48.028 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.029 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.029 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.030 DEBUG:boto:Params: {} 2022-09-02T05:55:48.030 DEBUG:boto:Token: None 2022-09-02T05:55:48.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.032 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017534700db6862dd-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.036 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.037 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.037 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.038 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:48.039 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:48.039 DEBUG:boto:Method: GET 2022-09-02T05:55:48.040 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:48.040 DEBUG:boto:Data: 2022-09-02T05:55:48.041 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.042 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.042 DEBUG:boto:Params: {} 2022-09-02T05:55:48.043 DEBUG:boto:Token: None 2022-09-02T05:55:48.044 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e460b0177e12a9af-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.050 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:55:48.051 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.052 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.052 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.053 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.053 DEBUG:boto:Method: GET 2022-09-02T05:55:48.054 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.054 DEBUG:boto:Data: 2022-09-02T05:55:48.055 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.057 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.058 DEBUG:boto:Params: {} 2022-09-02T05:55:48.059 DEBUG:boto:Token: None 2022-09-02T05:55:48.060 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w= 2022-09-02T05:55:48.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w=', 'Content-Length': '0'} 2022-09-02T05:55:48.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca3157e9b2440ad-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.106 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.107 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.107 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.108 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.109 DEBUG:boto:Method: GET 2022-09-02T05:55:48.109 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.110 DEBUG:boto:Data: 2022-09-02T05:55:48.111 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.112 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.112 DEBUG:boto:Params: {} 2022-09-02T05:55:48.113 DEBUG:boto:Token: None 2022-09-02T05:55:48.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w= 2022-09-02T05:55:48.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w=', 'Content-Length': '0'} 2022-09-02T05:55:48.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa484b1157537a6b-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.118 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.119 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.119 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.120 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.120 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.121 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.121 DEBUG:boto:Data: 2022-09-02T05:55:48.122 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.122 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.123 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.123 DEBUG:boto:Params: {} 2022-09-02T05:55:48.124 DEBUG:boto:Token: None 2022-09-02T05:55:48.125 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSau8yp8nyXtoz0PqI36VtRIOtI= 2022-09-02T05:55:48.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSau8yp8nyXtoz0PqI36VtRIOtI=', 'Content-Length': '0'} 2022-09-02T05:55:48.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56827b77c71ccee-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.130 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.131 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.132 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.133 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.134 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.135 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.136 DEBUG:boto:Data: 2022-09-02T05:55:48.137 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.139 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.140 DEBUG:boto:Params: {} 2022-09-02T05:55:48.142 DEBUG:boto:Token: None 2022-09-02T05:55:48.143 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DSau8yp8nyXtoz0PqI36VtRIOtI= 2022-09-02T05:55:48.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DSau8yp8nyXtoz0PqI36VtRIOtI=', 'Content-Length': '0'} 2022-09-02T05:55:48.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c829d184425079f5-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:55:48.152 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.153 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.154 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.155 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.156 DEBUG:boto:Method: GET 2022-09-02T05:55:48.157 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.158 DEBUG:boto:Data: 2022-09-02T05:55:48.159 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.161 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.162 DEBUG:boto:Params: {} 2022-09-02T05:55:48.164 DEBUG:boto:Token: None 2022-09-02T05:55:48.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w= 2022-09-02T05:55:48.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w=', 'Content-Length': '0'} 2022-09-02T05:55:48.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef79f251a04387d5-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.171 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.171 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.172 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.172 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.173 DEBUG:boto:Method: GET 2022-09-02T05:55:48.173 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.174 DEBUG:boto:Data: 2022-09-02T05:55:48.174 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.175 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.176 DEBUG:boto:Params: {} 2022-09-02T05:55:48.177 DEBUG:boto:Token: None 2022-09-02T05:55:48.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:48.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w= 2022-09-02T05:55:48.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mMNnitUFh2DEgDCo5JGNQHoYq1w=', 'Content-Length': '0'} 2022-09-02T05:55:48.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007524fa0448a25334-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.182 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.183 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.183 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.184 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.184 DEBUG:boto:Method: GET 2022-09-02T05:55:48.185 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.185 DEBUG:boto:Data: 2022-09-02T05:55:48.186 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.186 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.187 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.187 DEBUG:boto:Params: {} 2022-09-02T05:55:48.188 DEBUG:boto:Token: None 2022-09-02T05:55:48.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e4ce4c69b83380-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.194 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.195 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.195 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.196 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.196 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.197 DEBUG:boto:Method: GET 2022-09-02T05:55:48.197 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.198 DEBUG:boto:Data: 2022-09-02T05:55:48.198 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.199 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.200 DEBUG:boto:Params: {} 2022-09-02T05:55:48.201 DEBUG:boto:Token: None 2022-09-02T05:55:48.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afca53dcd9691663-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.207 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.208 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.209 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.209 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.210 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:48.211 DEBUG:boto:Data: 2022-09-02T05:55:48.211 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.212 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.212 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.213 DEBUG:boto:Params: {} 2022-09-02T05:55:48.213 DEBUG:boto:Token: None 2022-09-02T05:55:48.214 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58 2022-09-02T05:55:48.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JxVPtVWUBRNxWNTROhqJRbpFCLE= 2022-09-02T05:55:48.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JxVPtVWUBRNxWNTROhqJRbpFCLE=', 'Content-Length': '0'} 2022-09-02T05:55:48.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671a4affc1fc1e5a-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.218 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.218 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.219 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.219 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:48.220 DEBUG:boto:Data: 2022-09-02T05:55:48.221 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.222 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.222 DEBUG:boto:Params: {} 2022-09-02T05:55:48.223 DEBUG:boto:Token: None 2022-09-02T05:55:48.224 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58 2022-09-02T05:55:48.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JxVPtVWUBRNxWNTROhqJRbpFCLE= 2022-09-02T05:55:48.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JxVPtVWUBRNxWNTROhqJRbpFCLE=', 'Content-Length': '0'} 2022-09-02T05:55:48.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df9bc622596e6353-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:55:48.228 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.228 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.229 DEBUG:boto:Method: GET 2022-09-02T05:55:48.230 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:48.230 DEBUG:boto:Data: 2022-09-02T05:55:48.231 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.232 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.233 DEBUG:boto:Params: {} 2022-09-02T05:55:48.233 DEBUG:boto:Token: None 2022-09-02T05:55:48.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58 2022-09-02T05:55:48.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mBnVqb8Ff09Z/n7930bJPMZyWyA= 2022-09-02T05:55:48.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mBnVqb8Ff09Z/n7930bJPMZyWyA=', 'Content-Length': '0'} 2022-09-02T05:55:48.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db6bf4652b480766-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.238 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:48.239 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:48.239 DEBUG:boto:Method: GET 2022-09-02T05:55:48.240 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:48.240 DEBUG:boto:Data: 2022-09-02T05:55:48.241 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.242 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.243 DEBUG:boto:Params: {} 2022-09-02T05:55:48.244 DEBUG:boto:Token: None 2022-09-02T05:55:48.244 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj58 2022-09-02T05:55:48.245 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mBnVqb8Ff09Z/n7930bJPMZyWyA= 2022-09-02T05:55:48.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mBnVqb8Ff09Z/n7930bJPMZyWyA=', 'Content-Length': '0'} 2022-09-02T05:55:48.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007419950aa6ab33dd-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.290 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.291 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.292 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.292 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.293 DEBUG:boto:Method: GET 2022-09-02T05:55:48.293 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.294 DEBUG:boto:Data: 2022-09-02T05:55:48.295 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.295 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.296 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.296 DEBUG:boto:Params: {} 2022-09-02T05:55:48.297 DEBUG:boto:Token: None 2022-09-02T05:55:48.298 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002889a44f26de98b6-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.303 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.304 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.304 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.305 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.306 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.306 DEBUG:boto:Method: GET 2022-09-02T05:55:48.307 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:48.307 DEBUG:boto:Data: 2022-09-02T05:55:48.308 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.309 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.309 DEBUG:boto:Params: {} 2022-09-02T05:55:48.310 DEBUG:boto:Token: None 2022-09-02T05:55:48.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0a6f4bfdfbd22b-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.317 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:55:48.319 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.319 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.320 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.320 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.321 DEBUG:boto:Method: GET 2022-09-02T05:55:48.321 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.322 DEBUG:boto:Data: 2022-09-02T05:55:48.322 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.323 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.324 DEBUG:boto:Params: {} 2022-09-02T05:55:48.324 DEBUG:boto:Token: None 2022-09-02T05:55:48.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM= 2022-09-02T05:55:48.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM=', 'Content-Length': '0'} 2022-09-02T05:55:48.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a91cb0fe90e9196-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.330 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.331 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.331 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.332 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.332 DEBUG:boto:Method: GET 2022-09-02T05:55:48.333 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.333 DEBUG:boto:Data: 2022-09-02T05:55:48.334 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.335 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.336 DEBUG:boto:Params: {} 2022-09-02T05:55:48.336 DEBUG:boto:Token: None 2022-09-02T05:55:48.337 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM= 2022-09-02T05:55:48.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM=', 'Content-Length': '0'} 2022-09-02T05:55:48.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c042c047bbcce40-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.341 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.342 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.342 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.343 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.343 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.344 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.344 DEBUG:boto:Data: 2022-09-02T05:55:48.345 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.346 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.346 DEBUG:boto:Params: {} 2022-09-02T05:55:48.347 DEBUG:boto:Token: None 2022-09-02T05:55:48.348 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5E79kuffd+HFO8ZCuMqpoxROri8= 2022-09-02T05:55:48.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5E79kuffd+HFO8ZCuMqpoxROri8=', 'Content-Length': '0'} 2022-09-02T05:55:48.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e778a8ada758adb-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.352 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.352 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.353 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.353 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.354 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.354 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.355 DEBUG:boto:Data: 2022-09-02T05:55:48.355 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.356 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.357 DEBUG:boto:Params: {} 2022-09-02T05:55:48.357 DEBUG:boto:Token: None 2022-09-02T05:55:48.358 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5E79kuffd+HFO8ZCuMqpoxROri8= 2022-09-02T05:55:48.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5E79kuffd+HFO8ZCuMqpoxROri8=', 'Content-Length': '0'} 2022-09-02T05:55:48.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f04dedca573924a9-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:55:48.364 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.364 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.365 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.365 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.366 DEBUG:boto:Method: GET 2022-09-02T05:55:48.366 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.367 DEBUG:boto:Data: 2022-09-02T05:55:48.367 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.368 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.368 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.369 DEBUG:boto:Params: {} 2022-09-02T05:55:48.370 DEBUG:boto:Token: None 2022-09-02T05:55:48.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.371 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM= 2022-09-02T05:55:48.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM=', 'Content-Length': '0'} 2022-09-02T05:55:48.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000284fdfe886ab5f5a-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.414 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.415 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.415 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.416 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.416 DEBUG:boto:Method: GET 2022-09-02T05:55:48.417 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.418 DEBUG:boto:Data: 2022-09-02T05:55:48.418 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.419 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.420 DEBUG:boto:Params: {} 2022-09-02T05:55:48.420 DEBUG:boto:Token: None 2022-09-02T05:55:48.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:48.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM= 2022-09-02T05:55:48.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c4U12Tdc2Rwog/NI1t/UQsNAWKM=', 'Content-Length': '0'} 2022-09-02T05:55:48.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007120c1c5f6a21184-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.425 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.426 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.426 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.427 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.427 DEBUG:boto:Method: GET 2022-09-02T05:55:48.428 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.429 DEBUG:boto:Data: 2022-09-02T05:55:48.429 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.430 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.430 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.431 DEBUG:boto:Params: {} 2022-09-02T05:55:48.431 DEBUG:boto:Token: None 2022-09-02T05:55:48.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002878ea6192e4517e-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.437 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.437 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.438 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.438 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.439 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.439 DEBUG:boto:Method: GET 2022-09-02T05:55:48.440 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.441 DEBUG:boto:Data: 2022-09-02T05:55:48.441 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.442 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.443 DEBUG:boto:Params: {} 2022-09-02T05:55:48.443 DEBUG:boto:Token: None 2022-09-02T05:55:48.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c982b0ae204da39-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.450 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.451 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.451 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.452 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.453 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:48.453 DEBUG:boto:Data: 2022-09-02T05:55:48.454 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.454 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.455 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.455 DEBUG:boto:Params: {} 2022-09-02T05:55:48.456 DEBUG:boto:Token: None 2022-09-02T05:55:48.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68 2022-09-02T05:55:48.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Md91w7Fzr+F2xFJpGpRMPyqnZPI= 2022-09-02T05:55:48.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Md91w7Fzr+F2xFJpGpRMPyqnZPI=', 'Content-Length': '0'} 2022-09-02T05:55:48.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e7061bf4ebea90-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.460 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.461 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.461 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.462 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:48.462 DEBUG:boto:Data: 2022-09-02T05:55:48.463 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.464 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.464 DEBUG:boto:Params: {} 2022-09-02T05:55:48.465 DEBUG:boto:Token: None 2022-09-02T05:55:48.465 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68 2022-09-02T05:55:48.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Md91w7Fzr+F2xFJpGpRMPyqnZPI= 2022-09-02T05:55:48.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Md91w7Fzr+F2xFJpGpRMPyqnZPI=', 'Content-Length': '0'} 2022-09-02T05:55:48.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b205c87c6b56952-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:55:48.470 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.471 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.472 DEBUG:boto:Method: GET 2022-09-02T05:55:48.472 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:48.473 DEBUG:boto:Data: 2022-09-02T05:55:48.473 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.474 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.485 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.486 DEBUG:boto:Params: {} 2022-09-02T05:55:48.487 DEBUG:boto:Token: None 2022-09-02T05:55:48.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68 2022-09-02T05:55:48.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4SmTnfLAc9H5ZToSdiMspL04HI8= 2022-09-02T05:55:48.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4SmTnfLAc9H5ZToSdiMspL04HI8=', 'Content-Length': '0'} 2022-09-02T05:55:48.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002932208c8f25fbaa-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.492 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:48.492 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:48.493 DEBUG:boto:Method: GET 2022-09-02T05:55:48.493 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:48.494 DEBUG:boto:Data: 2022-09-02T05:55:48.495 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.496 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.496 DEBUG:boto:Params: {} 2022-09-02T05:55:48.497 DEBUG:boto:Token: None 2022-09-02T05:55:48.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj68 2022-09-02T05:55:48.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4SmTnfLAc9H5ZToSdiMspL04HI8= 2022-09-02T05:55:48.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4SmTnfLAc9H5ZToSdiMspL04HI8=', 'Content-Length': '0'} 2022-09-02T05:55:48.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268f3d3771bbc364-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.502 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.503 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.503 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.504 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.505 DEBUG:boto:Method: GET 2022-09-02T05:55:48.505 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.506 DEBUG:boto:Data: 2022-09-02T05:55:48.506 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.507 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.508 DEBUG:boto:Params: {} 2022-09-02T05:55:48.508 DEBUG:boto:Token: None 2022-09-02T05:55:48.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b74376c47f59d35-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.514 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.515 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.516 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.517 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.517 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.518 DEBUG:boto:Method: GET 2022-09-02T05:55:48.519 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:48.519 DEBUG:boto:Data: 2022-09-02T05:55:48.520 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.521 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.521 DEBUG:boto:Params: {} 2022-09-02T05:55:48.522 DEBUG:boto:Token: None 2022-09-02T05:55:48.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab6c3525eca69a5-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.529 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:55:48.531 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.531 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.532 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.532 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.533 DEBUG:boto:Method: GET 2022-09-02T05:55:48.534 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.534 DEBUG:boto:Data: 2022-09-02T05:55:48.535 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.536 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.537 DEBUG:boto:Params: {} 2022-09-02T05:55:48.537 DEBUG:boto:Token: None 2022-09-02T05:55:48.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A= 2022-09-02T05:55:48.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A=', 'Content-Length': '0'} 2022-09-02T05:55:48.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ba4a7689f572be-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.543 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.543 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.544 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.544 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.545 DEBUG:boto:Method: GET 2022-09-02T05:55:48.545 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.546 DEBUG:boto:Data: 2022-09-02T05:55:48.547 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.548 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.548 DEBUG:boto:Params: {} 2022-09-02T05:55:48.549 DEBUG:boto:Token: None 2022-09-02T05:55:48.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A= 2022-09-02T05:55:48.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A=', 'Content-Length': '0'} 2022-09-02T05:55:48.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000496e92cdcf16e9d1-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.594 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.594 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.595 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.595 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.596 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.596 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.597 DEBUG:boto:Data: 2022-09-02T05:55:48.597 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.599 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.600 DEBUG:boto:Params: {} 2022-09-02T05:55:48.601 DEBUG:boto:Token: None 2022-09-02T05:55:48.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:17ce+59E6nH4QsQasURto7e5cpI= 2022-09-02T05:55:48.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:17ce+59E6nH4QsQasURto7e5cpI=', 'Content-Length': '0'} 2022-09-02T05:55:48.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf48f1eb9f1ae11-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.609 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.610 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.611 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.612 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.612 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.613 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.614 DEBUG:boto:Data: 2022-09-02T05:55:48.615 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.616 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.617 DEBUG:boto:Params: {} 2022-09-02T05:55:48.618 DEBUG:boto:Token: None 2022-09-02T05:55:48.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:17ce+59E6nH4QsQasURto7e5cpI= 2022-09-02T05:55:48.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:17ce+59E6nH4QsQasURto7e5cpI=', 'Content-Length': '0'} 2022-09-02T05:55:48.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c40d27ab672dbb8-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:55:48.623 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.624 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.624 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.625 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.625 DEBUG:boto:Method: GET 2022-09-02T05:55:48.626 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.626 DEBUG:boto:Data: 2022-09-02T05:55:48.627 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.628 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.629 DEBUG:boto:Params: {} 2022-09-02T05:55:48.629 DEBUG:boto:Token: None 2022-09-02T05:55:48.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A= 2022-09-02T05:55:48.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A=', 'Content-Length': '0'} 2022-09-02T05:55:48.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001371ebe1eaa32aec-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.636 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.636 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.637 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.637 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.638 DEBUG:boto:Method: GET 2022-09-02T05:55:48.639 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.639 DEBUG:boto:Data: 2022-09-02T05:55:48.640 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.641 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.641 DEBUG:boto:Params: {} 2022-09-02T05:55:48.642 DEBUG:boto:Token: None 2022-09-02T05:55:48.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:48.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A= 2022-09-02T05:55:48.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rZPzzuHGREFQPQH++NSpDjQoZ1A=', 'Content-Length': '0'} 2022-09-02T05:55:48.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4c6b1bbee810cf-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.647 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.647 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.648 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.648 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.649 DEBUG:boto:Method: GET 2022-09-02T05:55:48.650 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.650 DEBUG:boto:Data: 2022-09-02T05:55:48.651 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.651 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.652 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.652 DEBUG:boto:Params: {} 2022-09-02T05:55:48.653 DEBUG:boto:Token: None 2022-09-02T05:55:48.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e08b783246db6f8-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.702 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.703 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.704 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.704 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.705 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.705 DEBUG:boto:Method: GET 2022-09-02T05:55:48.706 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.707 DEBUG:boto:Data: 2022-09-02T05:55:48.707 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.709 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.710 DEBUG:boto:Params: {} 2022-09-02T05:55:48.710 DEBUG:boto:Token: None 2022-09-02T05:55:48.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0021ae0b4600c3e-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.717 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.718 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.718 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.719 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.720 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:48.720 DEBUG:boto:Data: 2022-09-02T05:55:48.721 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.722 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.722 DEBUG:boto:Params: {} 2022-09-02T05:55:48.723 DEBUG:boto:Token: None 2022-09-02T05:55:48.723 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69 2022-09-02T05:55:48.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tO5rTXo4694sXWPA94N5gqaHlE0= 2022-09-02T05:55:48.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tO5rTXo4694sXWPA94N5gqaHlE0=', 'Content-Length': '0'} 2022-09-02T05:55:48.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a556f86c4c45ca00-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.728 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.728 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.729 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.729 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:48.730 DEBUG:boto:Data: 2022-09-02T05:55:48.730 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.731 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.732 DEBUG:boto:Params: {} 2022-09-02T05:55:48.732 DEBUG:boto:Token: None 2022-09-02T05:55:48.733 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69 2022-09-02T05:55:48.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tO5rTXo4694sXWPA94N5gqaHlE0= 2022-09-02T05:55:48.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tO5rTXo4694sXWPA94N5gqaHlE0=', 'Content-Length': '0'} 2022-09-02T05:55:48.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e1d27159f74db6-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:55:48.737 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.738 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.739 DEBUG:boto:Method: GET 2022-09-02T05:55:48.740 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:48.741 DEBUG:boto:Data: 2022-09-02T05:55:48.741 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.743 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.743 DEBUG:boto:Params: {} 2022-09-02T05:55:48.744 DEBUG:boto:Token: None 2022-09-02T05:55:48.745 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69 2022-09-02T05:55:48.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nzJEIB378RfQnITUnCR6+vCYn0o= 2022-09-02T05:55:48.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nzJEIB378RfQnITUnCR6+vCYn0o=', 'Content-Length': '0'} 2022-09-02T05:55:48.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006084aba5ca30f995-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.750 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:48.751 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:48.752 DEBUG:boto:Method: GET 2022-09-02T05:55:48.752 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:48.753 DEBUG:boto:Data: 2022-09-02T05:55:48.753 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.755 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.755 DEBUG:boto:Params: {} 2022-09-02T05:55:48.756 DEBUG:boto:Token: None 2022-09-02T05:55:48.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj69 2022-09-02T05:55:48.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nzJEIB378RfQnITUnCR6+vCYn0o= 2022-09-02T05:55:48.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nzJEIB378RfQnITUnCR6+vCYn0o=', 'Content-Length': '0'} 2022-09-02T05:55:48.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e8820497be0594e-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.761 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.762 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.763 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.763 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.764 DEBUG:boto:Method: GET 2022-09-02T05:55:48.764 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.765 DEBUG:boto:Data: 2022-09-02T05:55:48.765 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.767 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.767 DEBUG:boto:Params: {} 2022-09-02T05:55:48.768 DEBUG:boto:Token: None 2022-09-02T05:55:48.768 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000393ed0f7b1180db4-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.773 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.774 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.774 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.775 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.776 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.776 DEBUG:boto:Method: GET 2022-09-02T05:55:48.777 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:48.777 DEBUG:boto:Data: 2022-09-02T05:55:48.778 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.779 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.780 DEBUG:boto:Params: {} 2022-09-02T05:55:48.780 DEBUG:boto:Token: None 2022-09-02T05:55:48.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0cab1895374775f-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.787 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:55:48.788 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.789 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.789 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.790 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.791 DEBUG:boto:Method: GET 2022-09-02T05:55:48.791 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.792 DEBUG:boto:Data: 2022-09-02T05:55:48.792 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.793 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.794 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.794 DEBUG:boto:Params: {} 2022-09-02T05:55:48.795 DEBUG:boto:Token: None 2022-09-02T05:55:48.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms= 2022-09-02T05:55:48.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms=', 'Content-Length': '0'} 2022-09-02T05:55:48.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c67015f6b7aa2e3-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.800 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.801 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.801 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.802 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.803 DEBUG:boto:Method: GET 2022-09-02T05:55:48.803 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.804 DEBUG:boto:Data: 2022-09-02T05:55:48.804 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.806 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.806 DEBUG:boto:Params: {} 2022-09-02T05:55:48.807 DEBUG:boto:Token: None 2022-09-02T05:55:48.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms= 2022-09-02T05:55:48.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms=', 'Content-Length': '0'} 2022-09-02T05:55:48.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f52f66fe220de0-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.858 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.859 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.859 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.860 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.860 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.861 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.862 DEBUG:boto:Data: 2022-09-02T05:55:48.862 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.863 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.863 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.864 DEBUG:boto:Params: {} 2022-09-02T05:55:48.865 DEBUG:boto:Token: None 2022-09-02T05:55:48.865 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sJJ0HurvGC4/MP87MRBoy4axojs= 2022-09-02T05:55:48.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sJJ0HurvGC4/MP87MRBoy4axojs=', 'Content-Length': '0'} 2022-09-02T05:55:48.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857b2fb09f056efa-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.870 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.871 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.871 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.872 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.872 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.873 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.874 DEBUG:boto:Data: 2022-09-02T05:55:48.874 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.875 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.876 DEBUG:boto:Params: {} 2022-09-02T05:55:48.877 DEBUG:boto:Token: None 2022-09-02T05:55:48.878 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sJJ0HurvGC4/MP87MRBoy4axojs= 2022-09-02T05:55:48.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sJJ0HurvGC4/MP87MRBoy4axojs=', 'Content-Length': '0'} 2022-09-02T05:55:48.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b87f93ee1d96573b-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:55:48.882 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.883 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.883 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.884 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.885 DEBUG:boto:Method: GET 2022-09-02T05:55:48.885 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.886 DEBUG:boto:Data: 2022-09-02T05:55:48.887 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.889 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.890 DEBUG:boto:Params: {} 2022-09-02T05:55:48.891 DEBUG:boto:Token: None 2022-09-02T05:55:48.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms= 2022-09-02T05:55:48.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms=', 'Content-Length': '0'} 2022-09-02T05:55:48.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ea952da8282321-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.897 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.898 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.899 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.900 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.901 DEBUG:boto:Method: GET 2022-09-02T05:55:48.902 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.903 DEBUG:boto:Data: 2022-09-02T05:55:48.903 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.905 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.906 DEBUG:boto:Params: {} 2022-09-02T05:55:48.907 DEBUG:boto:Token: None 2022-09-02T05:55:48.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:48.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms= 2022-09-02T05:55:48.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lzfb4ylErqZl5P2H0Gib5r7fvms=', 'Content-Length': '0'} 2022-09-02T05:55:48.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004679a28b1d576c10-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.913 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.913 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.914 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:48.915 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:48.915 DEBUG:boto:Method: GET 2022-09-02T05:55:48.916 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:48.916 DEBUG:boto:Data: 2022-09-02T05:55:48.917 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.918 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.918 DEBUG:boto:Params: {} 2022-09-02T05:55:48.919 DEBUG:boto:Token: None 2022-09-02T05:55:48.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007039f37c985ad2d8-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.924 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.925 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:48.925 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:48.926 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:48.926 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:48.927 DEBUG:boto:Method: GET 2022-09-02T05:55:48.928 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:48.928 DEBUG:boto:Data: 2022-09-02T05:55:48.929 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.930 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.931 DEBUG:boto:Params: {} 2022-09-02T05:55:48.931 DEBUG:boto:Token: None 2022-09-02T05:55:48.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/ 2022-09-02T05:55:48.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA= 2022-09-02T05:55:48.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1+VU7sdt8dd8/ug64WC/CDVlhMA=', 'Content-Length': '0'} 2022-09-02T05:55:48.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e505da8e328b9419-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.937 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:48.938 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.939 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.939 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.940 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:48.940 DEBUG:boto:Data: 2022-09-02T05:55:48.941 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.941 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.942 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.942 DEBUG:boto:Params: {} 2022-09-02T05:55:48.943 DEBUG:boto:Token: None 2022-09-02T05:55:48.944 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76 2022-09-02T05:55:48.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hZNks6rSmJhYeP64gRzutPhz2wA= 2022-09-02T05:55:48.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hZNks6rSmJhYeP64gRzutPhz2wA=', 'Content-Length': '0'} 2022-09-02T05:55:48.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a7bf1f234ce481-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.949 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.950 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.951 DEBUG:boto:Method: HEAD 2022-09-02T05:55:48.951 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:48.952 DEBUG:boto:Data: 2022-09-02T05:55:48.953 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:48.955 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.956 DEBUG:boto:Params: {} 2022-09-02T05:55:48.957 DEBUG:boto:Token: None 2022-09-02T05:55:48.958 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76 2022-09-02T05:55:48.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hZNks6rSmJhYeP64gRzutPhz2wA= 2022-09-02T05:55:48.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hZNks6rSmJhYeP64gRzutPhz2wA=', 'Content-Length': '0'} 2022-09-02T05:55:48.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093703a1c8c2fc483-0063119ae4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:48.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:55:48.965 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:48.966 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:48.967 DEBUG:boto:Method: GET 2022-09-02T05:55:48.968 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:48.969 DEBUG:boto:Data: 2022-09-02T05:55:48.970 DEBUG:boto:Headers: {} 2022-09-02T05:55:48.971 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:48.971 DEBUG:boto:Port: 8000 2022-09-02T05:55:48.972 DEBUG:boto:Params: {} 2022-09-02T05:55:48.973 DEBUG:boto:Token: None 2022-09-02T05:55:48.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:48 GMT /xljbkf-74/obj76 2022-09-02T05:55:48.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M73rsmUbs9Fn+miXUJ4CFniAUE0= 2022-09-02T05:55:48.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M73rsmUbs9Fn+miXUJ4CFniAUE0=', 'Content-Length': '0'} 2022-09-02T05:55:48.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26faf48bafed0ea-0063119ae4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.022 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:49.023 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:49.024 DEBUG:boto:Method: GET 2022-09-02T05:55:49.025 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:49.026 DEBUG:boto:Data: 2022-09-02T05:55:49.026 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.028 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.029 DEBUG:boto:Params: {} 2022-09-02T05:55:49.029 DEBUG:boto:Token: None 2022-09-02T05:55:49.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj76 2022-09-02T05:55:49.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwbbzN2T4YRtrOqbyRV7J+Nw/L8= 2022-09-02T05:55:49.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwbbzN2T4YRtrOqbyRV7J+Nw/L8=', 'Content-Length': '0'} 2022-09-02T05:55:49.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40e3f22b66f9cf0-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.034 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.035 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.035 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:49.036 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:49.036 DEBUG:boto:Method: GET 2022-09-02T05:55:49.037 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:49.037 DEBUG:boto:Data: 2022-09-02T05:55:49.038 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.038 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.039 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.039 DEBUG:boto:Params: {} 2022-09-02T05:55:49.040 DEBUG:boto:Token: None 2022-09-02T05:55:49.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054fdd4c87543793b-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.045 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.046 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.047 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.048 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:49.049 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:49.050 DEBUG:boto:Method: GET 2022-09-02T05:55:49.051 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:49.052 DEBUG:boto:Data: 2022-09-02T05:55:49.053 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.055 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.056 DEBUG:boto:Params: {} 2022-09-02T05:55:49.057 DEBUG:boto:Token: None 2022-09-02T05:55:49.058 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e68984b502b092a-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.065 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:55:49.068 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.069 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.070 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.071 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.072 DEBUG:boto:Method: GET 2022-09-02T05:55:49.073 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.074 DEBUG:boto:Data: 2022-09-02T05:55:49.075 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.077 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.078 DEBUG:boto:Params: {} 2022-09-02T05:55:49.079 DEBUG:boto:Token: None 2022-09-02T05:55:49.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg= 2022-09-02T05:55:49.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg=', 'Content-Length': '0'} 2022-09-02T05:55:49.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027803001039f0f1a-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.086 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.087 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.088 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.089 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.089 DEBUG:boto:Method: GET 2022-09-02T05:55:49.090 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.090 DEBUG:boto:Data: 2022-09-02T05:55:49.091 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.092 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.093 DEBUG:boto:Params: {} 2022-09-02T05:55:49.094 DEBUG:boto:Token: None 2022-09-02T05:55:49.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg= 2022-09-02T05:55:49.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg=', 'Content-Length': '0'} 2022-09-02T05:55:49.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000894a5650b9ac8a5-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.100 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.100 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.101 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.101 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.102 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.102 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.103 DEBUG:boto:Data: 2022-09-02T05:55:49.104 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.104 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.105 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.105 DEBUG:boto:Params: {} 2022-09-02T05:55:49.106 DEBUG:boto:Token: None 2022-09-02T05:55:49.106 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.107 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GW0LpEgzvWKEafntmLNnaP8nwhs= 2022-09-02T05:55:49.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GW0LpEgzvWKEafntmLNnaP8nwhs=', 'Content-Length': '0'} 2022-09-02T05:55:49.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a07c6cb20fb0cae-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.111 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.111 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.112 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.112 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.113 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.114 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.114 DEBUG:boto:Data: 2022-09-02T05:55:49.115 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.116 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.116 DEBUG:boto:Params: {} 2022-09-02T05:55:49.117 DEBUG:boto:Token: None 2022-09-02T05:55:49.117 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GW0LpEgzvWKEafntmLNnaP8nwhs= 2022-09-02T05:55:49.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GW0LpEgzvWKEafntmLNnaP8nwhs=', 'Content-Length': '0'} 2022-09-02T05:55:49.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a29e5368867549b3-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:55:49.122 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.123 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.123 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.124 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.124 DEBUG:boto:Method: GET 2022-09-02T05:55:49.125 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.125 DEBUG:boto:Data: 2022-09-02T05:55:49.126 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.127 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.127 DEBUG:boto:Params: {} 2022-09-02T05:55:49.128 DEBUG:boto:Token: None 2022-09-02T05:55:49.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg= 2022-09-02T05:55:49.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg=', 'Content-Length': '0'} 2022-09-02T05:55:49.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f8d80aa47e298c-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.133 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.133 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.134 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.135 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.135 DEBUG:boto:Method: GET 2022-09-02T05:55:49.136 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.136 DEBUG:boto:Data: 2022-09-02T05:55:49.137 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.138 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.138 DEBUG:boto:Params: {} 2022-09-02T05:55:49.139 DEBUG:boto:Token: None 2022-09-02T05:55:49.140 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:49.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg= 2022-09-02T05:55:49.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtLy0tOSnBBTXXjezYr2/bHNKxg=', 'Content-Length': '0'} 2022-09-02T05:55:49.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dfa5ce2e7e7b513-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.190 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.191 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.191 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.192 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.193 DEBUG:boto:Method: GET 2022-09-02T05:55:49.194 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.195 DEBUG:boto:Data: 2022-09-02T05:55:49.196 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.197 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.198 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.199 DEBUG:boto:Params: {} 2022-09-02T05:55:49.200 DEBUG:boto:Token: None 2022-09-02T05:55:49.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d78d0b1d7cc4d3-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.208 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.210 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.210 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.211 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.211 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.212 DEBUG:boto:Method: GET 2022-09-02T05:55:49.212 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.213 DEBUG:boto:Data: 2022-09-02T05:55:49.213 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.214 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.215 DEBUG:boto:Params: {} 2022-09-02T05:55:49.216 DEBUG:boto:Token: None 2022-09-02T05:55:49.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c1533903798592-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.222 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.223 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.224 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.224 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.225 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:49.225 DEBUG:boto:Data: 2022-09-02T05:55:49.226 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.227 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.227 DEBUG:boto:Params: {} 2022-09-02T05:55:49.228 DEBUG:boto:Token: None 2022-09-02T05:55:49.228 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85 2022-09-02T05:55:49.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U1SOAhZLsq6BbAZrfLUMnDn1Gx8= 2022-09-02T05:55:49.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U1SOAhZLsq6BbAZrfLUMnDn1Gx8=', 'Content-Length': '0'} 2022-09-02T05:55:49.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f934d82a14694c-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.232 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.233 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.233 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.234 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:49.234 DEBUG:boto:Data: 2022-09-02T05:55:49.235 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.236 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.236 DEBUG:boto:Params: {} 2022-09-02T05:55:49.237 DEBUG:boto:Token: None 2022-09-02T05:55:49.237 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85 2022-09-02T05:55:49.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U1SOAhZLsq6BbAZrfLUMnDn1Gx8= 2022-09-02T05:55:49.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U1SOAhZLsq6BbAZrfLUMnDn1Gx8=', 'Content-Length': '0'} 2022-09-02T05:55:49.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be420168c5e9153-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:55:49.242 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.242 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.243 DEBUG:boto:Method: GET 2022-09-02T05:55:49.243 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:49.244 DEBUG:boto:Data: 2022-09-02T05:55:49.244 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.246 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.246 DEBUG:boto:Params: {} 2022-09-02T05:55:49.247 DEBUG:boto:Token: None 2022-09-02T05:55:49.247 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85 2022-09-02T05:55:49.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QiBHySU5xygI505w73OvDWRZU7Q= 2022-09-02T05:55:49.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QiBHySU5xygI505w73OvDWRZU7Q=', 'Content-Length': '0'} 2022-09-02T05:55:49.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca4f3d6407554ae9-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.251 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:49.252 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:49.252 DEBUG:boto:Method: GET 2022-09-02T05:55:49.253 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:49.253 DEBUG:boto:Data: 2022-09-02T05:55:49.254 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.255 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.256 DEBUG:boto:Params: {} 2022-09-02T05:55:49.256 DEBUG:boto:Token: None 2022-09-02T05:55:49.257 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj85 2022-09-02T05:55:49.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QiBHySU5xygI505w73OvDWRZU7Q= 2022-09-02T05:55:49.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QiBHySU5xygI505w73OvDWRZU7Q=', 'Content-Length': '0'} 2022-09-02T05:55:49.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047537044d092d10e-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.261 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.261 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.262 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.263 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.263 DEBUG:boto:Method: GET 2022-09-02T05:55:49.264 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.264 DEBUG:boto:Data: 2022-09-02T05:55:49.265 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.265 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.266 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.266 DEBUG:boto:Params: {} 2022-09-02T05:55:49.267 DEBUG:boto:Token: None 2022-09-02T05:55:49.267 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa0c31cfe50691bd-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.314 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.315 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.316 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.316 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.317 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.317 DEBUG:boto:Method: GET 2022-09-02T05:55:49.318 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:49.318 DEBUG:boto:Data: 2022-09-02T05:55:49.319 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.320 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.320 DEBUG:boto:Params: {} 2022-09-02T05:55:49.321 DEBUG:boto:Token: None 2022-09-02T05:55:49.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000577e871d89b8875e-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.328 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:55:49.329 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.330 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.330 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.331 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.331 DEBUG:boto:Method: GET 2022-09-02T05:55:49.332 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.332 DEBUG:boto:Data: 2022-09-02T05:55:49.333 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.334 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.335 DEBUG:boto:Params: {} 2022-09-02T05:55:49.335 DEBUG:boto:Token: None 2022-09-02T05:55:49.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8= 2022-09-02T05:55:49.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8=', 'Content-Length': '0'} 2022-09-02T05:55:49.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4804314cf6af000-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.340 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.341 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.341 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.342 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.342 DEBUG:boto:Method: GET 2022-09-02T05:55:49.343 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.343 DEBUG:boto:Data: 2022-09-02T05:55:49.344 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.345 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.345 DEBUG:boto:Params: {} 2022-09-02T05:55:49.346 DEBUG:boto:Token: None 2022-09-02T05:55:49.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8= 2022-09-02T05:55:49.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8=', 'Content-Length': '0'} 2022-09-02T05:55:49.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd46abb1e06e606-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.351 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.351 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.352 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.352 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.353 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.353 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.354 DEBUG:boto:Data: 2022-09-02T05:55:49.354 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.355 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.356 DEBUG:boto:Params: {} 2022-09-02T05:55:49.357 DEBUG:boto:Token: None 2022-09-02T05:55:49.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nd0jHQ5Q6EnuZKrr8G5RTAeyFrY= 2022-09-02T05:55:49.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nd0jHQ5Q6EnuZKrr8G5RTAeyFrY=', 'Content-Length': '0'} 2022-09-02T05:55:49.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153e902427d86dd8-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.361 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.362 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.362 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.363 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.363 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.364 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.364 DEBUG:boto:Data: 2022-09-02T05:55:49.365 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.366 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.366 DEBUG:boto:Params: {} 2022-09-02T05:55:49.367 DEBUG:boto:Token: None 2022-09-02T05:55:49.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nd0jHQ5Q6EnuZKrr8G5RTAeyFrY= 2022-09-02T05:55:49.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nd0jHQ5Q6EnuZKrr8G5RTAeyFrY=', 'Content-Length': '0'} 2022-09-02T05:55:49.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ee695ca0d62a00-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:55:49.372 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.373 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.373 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.374 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.374 DEBUG:boto:Method: GET 2022-09-02T05:55:49.375 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.375 DEBUG:boto:Data: 2022-09-02T05:55:49.376 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.377 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.377 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.378 DEBUG:boto:Params: {} 2022-09-02T05:55:49.378 DEBUG:boto:Token: None 2022-09-02T05:55:49.379 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8= 2022-09-02T05:55:49.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8=', 'Content-Length': '0'} 2022-09-02T05:55:49.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7d41399e98fc72-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.383 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.384 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.385 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.385 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.386 DEBUG:boto:Method: GET 2022-09-02T05:55:49.386 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.387 DEBUG:boto:Data: 2022-09-02T05:55:49.387 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.389 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.390 DEBUG:boto:Params: {} 2022-09-02T05:55:49.391 DEBUG:boto:Token: None 2022-09-02T05:55:49.391 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:49.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8= 2022-09-02T05:55:49.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DLhxUXWPRhzbzfUhy8y7Q5QYjm8=', 'Content-Length': '0'} 2022-09-02T05:55:49.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004869b01303a7ef5d-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.438 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.439 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.440 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.440 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.441 DEBUG:boto:Method: GET 2022-09-02T05:55:49.441 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.442 DEBUG:boto:Data: 2022-09-02T05:55:49.443 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.444 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.445 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.446 DEBUG:boto:Params: {} 2022-09-02T05:55:49.448 DEBUG:boto:Token: None 2022-09-02T05:55:49.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e39d90216f4715-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.454 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.455 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.456 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.456 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.457 DEBUG:boto:Method: GET 2022-09-02T05:55:49.457 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.458 DEBUG:boto:Data: 2022-09-02T05:55:49.458 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.460 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.460 DEBUG:boto:Params: {} 2022-09-02T05:55:49.461 DEBUG:boto:Token: None 2022-09-02T05:55:49.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c8516db7933d3a-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.467 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.468 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.468 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.469 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.469 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:49.470 DEBUG:boto:Data: 2022-09-02T05:55:49.471 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.472 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.472 DEBUG:boto:Params: {} 2022-09-02T05:55:49.473 DEBUG:boto:Token: None 2022-09-02T05:55:49.473 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87 2022-09-02T05:55:49.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QHFnv0SMd2LzTFYhn1i0DbzrJbc= 2022-09-02T05:55:49.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QHFnv0SMd2LzTFYhn1i0DbzrJbc=', 'Content-Length': '0'} 2022-09-02T05:55:49.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf5fdabffed26bb-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.478 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.478 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.479 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.479 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:49.480 DEBUG:boto:Data: 2022-09-02T05:55:49.480 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.481 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.482 DEBUG:boto:Params: {} 2022-09-02T05:55:49.482 DEBUG:boto:Token: None 2022-09-02T05:55:49.483 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87 2022-09-02T05:55:49.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QHFnv0SMd2LzTFYhn1i0DbzrJbc= 2022-09-02T05:55:49.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QHFnv0SMd2LzTFYhn1i0DbzrJbc=', 'Content-Length': '0'} 2022-09-02T05:55:49.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de3735427c50b3d-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:55:49.488 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.488 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.489 DEBUG:boto:Method: GET 2022-09-02T05:55:49.489 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:49.490 DEBUG:boto:Data: 2022-09-02T05:55:49.490 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.491 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.492 DEBUG:boto:Params: {} 2022-09-02T05:55:49.493 DEBUG:boto:Token: None 2022-09-02T05:55:49.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87 2022-09-02T05:55:49.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mMyyh1Gj+uJPG3S+c0XdUX5QT4E= 2022-09-02T05:55:49.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mMyyh1Gj+uJPG3S+c0XdUX5QT4E=', 'Content-Length': '0'} 2022-09-02T05:55:49.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164410c4c442006f-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.497 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:49.498 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:49.498 DEBUG:boto:Method: GET 2022-09-02T05:55:49.499 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:49.500 DEBUG:boto:Data: 2022-09-02T05:55:49.500 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.501 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.502 DEBUG:boto:Params: {} 2022-09-02T05:55:49.502 DEBUG:boto:Token: None 2022-09-02T05:55:49.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj87 2022-09-02T05:55:49.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mMyyh1Gj+uJPG3S+c0XdUX5QT4E= 2022-09-02T05:55:49.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mMyyh1Gj+uJPG3S+c0XdUX5QT4E=', 'Content-Length': '0'} 2022-09-02T05:55:49.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22595ea9a9708cb-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.554 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.555 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.555 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.556 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.556 DEBUG:boto:Method: GET 2022-09-02T05:55:49.557 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.557 DEBUG:boto:Data: 2022-09-02T05:55:49.558 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.559 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.559 DEBUG:boto:Params: {} 2022-09-02T05:55:49.560 DEBUG:boto:Token: None 2022-09-02T05:55:49.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6b997a52a68742-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.565 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.566 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.567 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.567 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.568 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.568 DEBUG:boto:Method: GET 2022-09-02T05:55:49.569 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:49.569 DEBUG:boto:Data: 2022-09-02T05:55:49.570 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.571 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.571 DEBUG:boto:Params: {} 2022-09-02T05:55:49.572 DEBUG:boto:Token: None 2022-09-02T05:55:49.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e664f8c1447e693f-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.578 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:55:49.580 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.580 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.581 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.581 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.582 DEBUG:boto:Method: GET 2022-09-02T05:55:49.582 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.583 DEBUG:boto:Data: 2022-09-02T05:55:49.583 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.584 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.584 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.585 DEBUG:boto:Params: {} 2022-09-02T05:55:49.586 DEBUG:boto:Token: None 2022-09-02T05:55:49.586 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.587 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds= 2022-09-02T05:55:49.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds=', 'Content-Length': '0'} 2022-09-02T05:55:49.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000defdcbb7d49f8068-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.591 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.591 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.592 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.592 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.593 DEBUG:boto:Method: GET 2022-09-02T05:55:49.593 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.594 DEBUG:boto:Data: 2022-09-02T05:55:49.595 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.596 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.597 DEBUG:boto:Params: {} 2022-09-02T05:55:49.597 DEBUG:boto:Token: None 2022-09-02T05:55:49.598 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds= 2022-09-02T05:55:49.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds=', 'Content-Length': '0'} 2022-09-02T05:55:49.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a54aa82f7ba7a2f7-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.603 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.603 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.604 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.605 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.605 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.606 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.606 DEBUG:boto:Data: 2022-09-02T05:55:49.607 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.608 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.608 DEBUG:boto:Params: {} 2022-09-02T05:55:49.609 DEBUG:boto:Token: None 2022-09-02T05:55:49.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Tb3vPBz30t5p5uxqCENfpquA8s= 2022-09-02T05:55:49.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Tb3vPBz30t5p5uxqCENfpquA8s=', 'Content-Length': '0'} 2022-09-02T05:55:49.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000234fec551b0a7716-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.614 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.615 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.615 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.616 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.616 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.617 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.617 DEBUG:boto:Data: 2022-09-02T05:55:49.618 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.619 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.619 DEBUG:boto:Params: {} 2022-09-02T05:55:49.620 DEBUG:boto:Token: None 2022-09-02T05:55:49.620 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2Tb3vPBz30t5p5uxqCENfpquA8s= 2022-09-02T05:55:49.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2Tb3vPBz30t5p5uxqCENfpquA8s=', 'Content-Length': '0'} 2022-09-02T05:55:49.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf5c7ab4439f06f-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:55:49.626 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.627 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.628 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.628 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.629 DEBUG:boto:Method: GET 2022-09-02T05:55:49.630 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.630 DEBUG:boto:Data: 2022-09-02T05:55:49.631 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.632 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.632 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.633 DEBUG:boto:Params: {} 2022-09-02T05:55:49.634 DEBUG:boto:Token: None 2022-09-02T05:55:49.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds= 2022-09-02T05:55:49.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds=', 'Content-Length': '0'} 2022-09-02T05:55:49.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013a83fe4c756454-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.639 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.640 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.640 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.641 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.642 DEBUG:boto:Method: GET 2022-09-02T05:55:49.642 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.643 DEBUG:boto:Data: 2022-09-02T05:55:49.643 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.644 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.645 DEBUG:boto:Params: {} 2022-09-02T05:55:49.645 DEBUG:boto:Token: None 2022-09-02T05:55:49.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:49.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds= 2022-09-02T05:55:49.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EAfz5atV/lx5fsuvBHxqWBFBgds=', 'Content-Length': '0'} 2022-09-02T05:55:49.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075777ffd9b0b7882-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.652 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.652 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.653 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.653 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.654 DEBUG:boto:Method: GET 2022-09-02T05:55:49.655 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.655 DEBUG:boto:Data: 2022-09-02T05:55:49.656 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.657 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.657 DEBUG:boto:Params: {} 2022-09-02T05:55:49.658 DEBUG:boto:Token: None 2022-09-02T05:55:49.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d23c269990816a-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.665 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.666 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.667 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.668 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.668 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.669 DEBUG:boto:Method: GET 2022-09-02T05:55:49.669 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.670 DEBUG:boto:Data: 2022-09-02T05:55:49.670 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.671 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.672 DEBUG:boto:Params: {} 2022-09-02T05:55:49.673 DEBUG:boto:Token: None 2022-09-02T05:55:49.674 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7fbd07e4aa2968-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.680 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.681 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.681 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.682 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.683 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:49.683 DEBUG:boto:Data: 2022-09-02T05:55:49.684 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.685 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.685 DEBUG:boto:Params: {} 2022-09-02T05:55:49.686 DEBUG:boto:Token: None 2022-09-02T05:55:49.686 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92 2022-09-02T05:55:49.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tLKH6GkPnGqU8mrgBreNwFSYmpI= 2022-09-02T05:55:49.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tLKH6GkPnGqU8mrgBreNwFSYmpI=', 'Content-Length': '0'} 2022-09-02T05:55:49.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3228c7155dd202f-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.690 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.691 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.691 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.692 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:49.692 DEBUG:boto:Data: 2022-09-02T05:55:49.693 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.694 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.694 DEBUG:boto:Params: {} 2022-09-02T05:55:49.695 DEBUG:boto:Token: None 2022-09-02T05:55:49.696 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92 2022-09-02T05:55:49.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tLKH6GkPnGqU8mrgBreNwFSYmpI= 2022-09-02T05:55:49.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tLKH6GkPnGqU8mrgBreNwFSYmpI=', 'Content-Length': '0'} 2022-09-02T05:55:49.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc2f4e951afa068-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:55:49.702 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.703 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.704 DEBUG:boto:Method: GET 2022-09-02T05:55:49.705 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:49.706 DEBUG:boto:Data: 2022-09-02T05:55:49.706 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.707 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.708 DEBUG:boto:Params: {} 2022-09-02T05:55:49.709 DEBUG:boto:Token: None 2022-09-02T05:55:49.709 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92 2022-09-02T05:55:49.710 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0av7I+aViJr+cFjeJlpZAbjorBA= 2022-09-02T05:55:49.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0av7I+aViJr+cFjeJlpZAbjorBA=', 'Content-Length': '0'} 2022-09-02T05:55:49.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383edf46ee6e9c83-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.715 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:49.716 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:49.717 DEBUG:boto:Method: GET 2022-09-02T05:55:49.718 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:49.719 DEBUG:boto:Data: 2022-09-02T05:55:49.720 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.721 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.722 DEBUG:boto:Params: {} 2022-09-02T05:55:49.722 DEBUG:boto:Token: None 2022-09-02T05:55:49.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj92 2022-09-02T05:55:49.724 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0av7I+aViJr+cFjeJlpZAbjorBA= 2022-09-02T05:55:49.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0av7I+aViJr+cFjeJlpZAbjorBA=', 'Content-Length': '0'} 2022-09-02T05:55:49.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018af13b6b9b2b2de-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.770 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.772 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.773 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.774 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.775 DEBUG:boto:Method: GET 2022-09-02T05:55:49.776 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.777 DEBUG:boto:Data: 2022-09-02T05:55:49.778 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.781 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.782 DEBUG:boto:Params: {} 2022-09-02T05:55:49.783 DEBUG:boto:Token: None 2022-09-02T05:55:49.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098fc2da8c4451651-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.790 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.791 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.792 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.793 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.794 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.795 DEBUG:boto:Method: GET 2022-09-02T05:55:49.796 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:49.797 DEBUG:boto:Data: 2022-09-02T05:55:49.798 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.800 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.801 DEBUG:boto:Params: {} 2022-09-02T05:55:49.802 DEBUG:boto:Token: None 2022-09-02T05:55:49.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036bb050b05a4d19d-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.812 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:55:49.814 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.815 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.816 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.817 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.818 DEBUG:boto:Method: GET 2022-09-02T05:55:49.819 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.820 DEBUG:boto:Data: 2022-09-02T05:55:49.821 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.823 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.824 DEBUG:boto:Params: {} 2022-09-02T05:55:49.825 DEBUG:boto:Token: None 2022-09-02T05:55:49.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y= 2022-09-02T05:55:49.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y=', 'Content-Length': '0'} 2022-09-02T05:55:49.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae86f3cd81e1489-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.833 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.834 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.835 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.836 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.836 DEBUG:boto:Method: GET 2022-09-02T05:55:49.837 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.837 DEBUG:boto:Data: 2022-09-02T05:55:49.838 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.839 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.840 DEBUG:boto:Params: {} 2022-09-02T05:55:49.841 DEBUG:boto:Token: None 2022-09-02T05:55:49.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y= 2022-09-02T05:55:49.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y=', 'Content-Length': '0'} 2022-09-02T05:55:49.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9cb8585ba2aa471-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.845 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.846 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.847 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.847 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.848 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.848 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.849 DEBUG:boto:Data: 2022-09-02T05:55:49.849 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.850 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.851 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.851 DEBUG:boto:Params: {} 2022-09-02T05:55:49.852 DEBUG:boto:Token: None 2022-09-02T05:55:49.852 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CShqrkZ2n/r4YpEUk0C7D93609Y= 2022-09-02T05:55:49.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CShqrkZ2n/r4YpEUk0C7D93609Y=', 'Content-Length': '0'} 2022-09-02T05:55:49.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafda3a3181168e1-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.857 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.857 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.858 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.858 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.859 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.859 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.860 DEBUG:boto:Data: 2022-09-02T05:55:49.860 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.861 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.862 DEBUG:boto:Params: {} 2022-09-02T05:55:49.862 DEBUG:boto:Token: None 2022-09-02T05:55:49.863 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CShqrkZ2n/r4YpEUk0C7D93609Y= 2022-09-02T05:55:49.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CShqrkZ2n/r4YpEUk0C7D93609Y=', 'Content-Length': '0'} 2022-09-02T05:55:49.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075b4068e25665005-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:55:49.868 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.869 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.869 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.870 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.871 DEBUG:boto:Method: GET 2022-09-02T05:55:49.872 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.873 DEBUG:boto:Data: 2022-09-02T05:55:49.874 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.876 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.877 DEBUG:boto:Params: {} 2022-09-02T05:55:49.878 DEBUG:boto:Token: None 2022-09-02T05:55:49.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y= 2022-09-02T05:55:49.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y=', 'Content-Length': '0'} 2022-09-02T05:55:49.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf809402c6fc20b-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.885 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.886 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.886 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.887 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.888 DEBUG:boto:Method: GET 2022-09-02T05:55:49.888 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.889 DEBUG:boto:Data: 2022-09-02T05:55:49.889 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.891 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.892 DEBUG:boto:Params: {} 2022-09-02T05:55:49.892 DEBUG:boto:Token: None 2022-09-02T05:55:49.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:49.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y= 2022-09-02T05:55:49.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4+4nNWzBKiBAZ7R8Q7shcQH48Y=', 'Content-Length': '0'} 2022-09-02T05:55:49.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d36fd4d55b1c51-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.938 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.939 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.940 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:49.941 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:49.941 DEBUG:boto:Method: GET 2022-09-02T05:55:49.942 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:49.942 DEBUG:boto:Data: 2022-09-02T05:55:49.943 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.944 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.944 DEBUG:boto:Params: {} 2022-09-02T05:55:49.945 DEBUG:boto:Token: None 2022-09-02T05:55:49.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5bfa9da9d54b046-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.951 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.952 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:49.953 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:49.953 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:49.954 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:49.955 DEBUG:boto:Method: GET 2022-09-02T05:55:49.955 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:49.956 DEBUG:boto:Data: 2022-09-02T05:55:49.956 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.957 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.958 DEBUG:boto:Params: {} 2022-09-02T05:55:49.958 DEBUG:boto:Token: None 2022-09-02T05:55:49.959 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/ 2022-09-02T05:55:49.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw= 2022-09-02T05:55:49.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RcB+dGpk/70e40nNklLceG7Dzzw=', 'Content-Length': '0'} 2022-09-02T05:55:49.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693de0c895eb47a6-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.965 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:49.966 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.967 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.967 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.968 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:49.969 DEBUG:boto:Data: 2022-09-02T05:55:49.969 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.971 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.971 DEBUG:boto:Params: {} 2022-09-02T05:55:49.972 DEBUG:boto:Token: None 2022-09-02T05:55:49.972 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95 2022-09-02T05:55:49.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8DFRLDoRWlRCcsdsoGi6CUJfYgQ= 2022-09-02T05:55:49.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8DFRLDoRWlRCcsdsoGi6CUJfYgQ=', 'Content-Length': '0'} 2022-09-02T05:55:49.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002db02eb3dcd44e31-0063119ae5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.977 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.978 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.979 DEBUG:boto:Method: HEAD 2022-09-02T05:55:49.979 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:49.980 DEBUG:boto:Data: 2022-09-02T05:55:49.981 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:49.983 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.984 DEBUG:boto:Params: {} 2022-09-02T05:55:49.985 DEBUG:boto:Token: None 2022-09-02T05:55:49.986 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95 2022-09-02T05:55:49.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8DFRLDoRWlRCcsdsoGi6CUJfYgQ= 2022-09-02T05:55:49.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8DFRLDoRWlRCcsdsoGi6CUJfYgQ=', 'Content-Length': '0'} 2022-09-02T05:55:49.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c71ceec35e0ab8-0063119ae5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:49.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:55:49.991 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:49.992 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:49.993 DEBUG:boto:Method: GET 2022-09-02T05:55:49.993 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:49.994 DEBUG:boto:Data: 2022-09-02T05:55:49.994 DEBUG:boto:Headers: {} 2022-09-02T05:55:49.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:49.995 DEBUG:boto:Port: 8000 2022-09-02T05:55:49.996 DEBUG:boto:Params: {} 2022-09-02T05:55:49.997 DEBUG:boto:Token: None 2022-09-02T05:55:49.997 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:49 GMT /xljbkf-74/obj95 2022-09-02T05:55:49.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Buv+HqLuPczmL5N2s2h+ISvUptc= 2022-09-02T05:55:49.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Buv+HqLuPczmL5N2s2h+ISvUptc=', 'Content-Length': '0'} 2022-09-02T05:55:50.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772e95c6ca1de304-0063119ae6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.042 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:50.043 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:50.043 DEBUG:boto:Method: GET 2022-09-02T05:55:50.044 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:50.044 DEBUG:boto:Data: 2022-09-02T05:55:50.045 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.046 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.046 DEBUG:boto:Params: {} 2022-09-02T05:55:50.047 DEBUG:boto:Token: None 2022-09-02T05:55:50.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj95 2022-09-02T05:55:50.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8sjGvV3+FM53G6isnBpr1FJcZvg= 2022-09-02T05:55:50.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8sjGvV3+FM53G6isnBpr1FJcZvg=', 'Content-Length': '0'} 2022-09-02T05:55:50.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c0bf44a198b4d4-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.052 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.052 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.053 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:50.053 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:50.054 DEBUG:boto:Method: GET 2022-09-02T05:55:50.055 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:50.055 DEBUG:boto:Data: 2022-09-02T05:55:50.056 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:50.057 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.057 DEBUG:boto:Params: {} 2022-09-02T05:55:50.058 DEBUG:boto:Token: None 2022-09-02T05:55:50.058 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a3938d91ca380fc-0063119ae6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.063 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.064 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.064 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.065 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:50.065 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:50.066 DEBUG:boto:Method: GET 2022-09-02T05:55:50.067 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:50.067 DEBUG:boto:Data: 2022-09-02T05:55:50.068 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.069 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.069 DEBUG:boto:Params: {} 2022-09-02T05:55:50.070 DEBUG:boto:Token: None 2022-09-02T05:55:50.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082797fe46392f547-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.122 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.124 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:55:50.124 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:55:50.125 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.125 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.126 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.126 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:55:50.127 DEBUG:boto:Data: 2022-09-02T05:55:50.127 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.128 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.129 DEBUG:boto:Params: {} 2022-09-02T05:55:50.130 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:55:50.130 DEBUG:boto:Token: None 2022-09-02T05:55:50.131 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q1vSiqEBPnFvqYDpOlUXj49O8zU= 2022-09-02T05:55:50.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q1vSiqEBPnFvqYDpOlUXj49O8zU=', 'Content-Length': '0'} 2022-09-02T05:55:50.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011daf7b4b60d4ba3-0063119ae6-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.137 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.137 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.138 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.139 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:55:50.139 DEBUG:boto:Data: 2022-09-02T05:55:50.140 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.141 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.141 DEBUG:boto:Params: {} 2022-09-02T05:55:50.142 DEBUG:boto:Token: None 2022-09-02T05:55:50.142 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q1vSiqEBPnFvqYDpOlUXj49O8zU= 2022-09-02T05:55:50.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q1vSiqEBPnFvqYDpOlUXj49O8zU=', 'Content-Length': '0'} 2022-09-02T05:55:50.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef121fe6e1943522-0063119ae6-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:55:50.149 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.150 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.150 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.151 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.151 DEBUG:boto:Method: GET 2022-09-02T05:55:50.152 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.152 DEBUG:boto:Data: 2022-09-02T05:55:50.153 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.154 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.155 DEBUG:boto:Params: {} 2022-09-02T05:55:50.155 DEBUG:boto:Token: None 2022-09-02T05:55:50.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/?versions 2022-09-02T05:55:50.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM= 2022-09-02T05:55:50.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM=', 'Content-Length': '0'} 2022-09-02T05:55:50.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a2c2b124ad05e59-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.161 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:55:50.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:55:50.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:55:50.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:55:50.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:55:50.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:55:50.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:55:50.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:55:50.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:55:50.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:55:50.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:55:50.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:55:50.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:55:50.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:55:50.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:55:50.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:55:50.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:55:50.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:55:50.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:55:50.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:55:50.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:55:50.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:55:50.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:55:50.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:55:50.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:55:50.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:55:50.176 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.177 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.177 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.178 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.178 DEBUG:boto:Method: GET 2022-09-02T05:55:50.179 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.179 DEBUG:boto:Data: 2022-09-02T05:55:50.180 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.181 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.181 DEBUG:boto:Params: {} 2022-09-02T05:55:50.182 DEBUG:boto:Token: None 2022-09-02T05:55:50.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/?versions 2022-09-02T05:55:50.183 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM= 2022-09-02T05:55:50.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM=', 'Content-Length': '0'} 2022-09-02T05:55:50.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026cc0d9f21f4f80d-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.189 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:55:50.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:55:50.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:55:50.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:55:50.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:55:50.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:55:50.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:55:50.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:55:50.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:55:50.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:55:50.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:55:50.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:55:50.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:55:50.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:55:50.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:55:50.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:55:50.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:55:50.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:55:50.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:55:50.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:55:50.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:55:50.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:55:50.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:55:50.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:55:50.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:55:50.203 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.204 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.204 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.205 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.205 DEBUG:boto:Method: GET 2022-09-02T05:55:50.206 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.206 DEBUG:boto:Data: 2022-09-02T05:55:50.207 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.208 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.208 DEBUG:boto:Params: {} 2022-09-02T05:55:50.209 DEBUG:boto:Token: None 2022-09-02T05:55:50.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/?versions 2022-09-02T05:55:50.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM= 2022-09-02T05:55:50.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM=', 'Content-Length': '0'} 2022-09-02T05:55:50.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb08f52c86106795-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.215 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.217 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.217 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.218 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.218 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.219 DEBUG:boto:Method: GET 2022-09-02T05:55:50.219 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:50.220 DEBUG:boto:Data: 2022-09-02T05:55:50.220 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.221 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.222 DEBUG:boto:Params: {} 2022-09-02T05:55:50.222 DEBUG:boto:Token: None 2022-09-02T05:55:50.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/?versions 2022-09-02T05:55:50.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM= 2022-09-02T05:55:50.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Kq78IQoE8LlC0Qy9gSX1AebyGM=', 'Content-Length': '0'} 2022-09-02T05:55:50.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb248aedcbe537c-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.229 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:50.232 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.232 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.233 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.234 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.234 DEBUG:boto:Method: GET 2022-09-02T05:55:50.235 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.235 DEBUG:boto:Data: 2022-09-02T05:55:50.236 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.237 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.237 DEBUG:boto:Params: {} 2022-09-02T05:55:50.238 DEBUG:boto:Token: None 2022-09-02T05:55:50.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs= 2022-09-02T05:55:50.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs=', 'Content-Length': '0'} 2022-09-02T05:55:50.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d95ba080ba6496-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.243 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.244 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.244 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.245 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.245 DEBUG:boto:Method: GET 2022-09-02T05:55:50.246 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.247 DEBUG:boto:Data: 2022-09-02T05:55:50.247 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.248 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.249 DEBUG:boto:Params: {} 2022-09-02T05:55:50.249 DEBUG:boto:Token: None 2022-09-02T05:55:50.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs= 2022-09-02T05:55:50.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs=', 'Content-Length': '0'} 2022-09-02T05:55:50.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059119ab34141f458-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.298 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.299 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.300 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.301 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.302 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.303 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.303 DEBUG:boto:Data: 2022-09-02T05:55:50.304 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.306 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.307 DEBUG:boto:Params: {} 2022-09-02T05:55:50.308 DEBUG:boto:Token: None 2022-09-02T05:55:50.309 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.310 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2QOb1aYIz4DI/iwP3xhFWTOHSLg= 2022-09-02T05:55:50.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2QOb1aYIz4DI/iwP3xhFWTOHSLg=', 'Content-Length': '0'} 2022-09-02T05:55:50.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a615e1c4b4dd87d9-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.315 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.316 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.317 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.317 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.318 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.318 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.319 DEBUG:boto:Data: 2022-09-02T05:55:50.320 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.320 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.321 DEBUG:boto:Params: {} 2022-09-02T05:55:50.322 DEBUG:boto:Token: None 2022-09-02T05:55:50.322 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2QOb1aYIz4DI/iwP3xhFWTOHSLg= 2022-09-02T05:55:50.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2QOb1aYIz4DI/iwP3xhFWTOHSLg=', 'Content-Length': '0'} 2022-09-02T05:55:50.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f851935785fe2ea3-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:50.328 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.328 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.329 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.329 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.330 DEBUG:boto:Method: GET 2022-09-02T05:55:50.330 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.331 DEBUG:boto:Data: 2022-09-02T05:55:50.332 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.333 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.333 DEBUG:boto:Params: {} 2022-09-02T05:55:50.334 DEBUG:boto:Token: None 2022-09-02T05:55:50.335 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs= 2022-09-02T05:55:50.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs=', 'Content-Length': '0'} 2022-09-02T05:55:50.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346261ddfc859d33-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.341 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.342 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.343 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.343 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.345 DEBUG:boto:Method: GET 2022-09-02T05:55:50.345 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.346 DEBUG:boto:Data: 2022-09-02T05:55:50.347 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.348 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.349 DEBUG:boto:Params: {} 2022-09-02T05:55:50.350 DEBUG:boto:Token: None 2022-09-02T05:55:50.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:50.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs= 2022-09-02T05:55:50.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3A6uBwM/7sM9LNCjBpn7Ze91PHs=', 'Content-Length': '0'} 2022-09-02T05:55:50.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5f3c09a540d69e-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.356 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.357 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.357 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.358 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.358 DEBUG:boto:Method: GET 2022-09-02T05:55:50.359 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.360 DEBUG:boto:Data: 2022-09-02T05:55:50.360 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.361 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.362 DEBUG:boto:Params: {} 2022-09-02T05:55:50.362 DEBUG:boto:Token: None 2022-09-02T05:55:50.363 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d6657339e6ec2e-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.368 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.369 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.370 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.370 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.371 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.371 DEBUG:boto:Method: GET 2022-09-02T05:55:50.372 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.372 DEBUG:boto:Data: 2022-09-02T05:55:50.373 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.374 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.375 DEBUG:boto:Params: {} 2022-09-02T05:55:50.375 DEBUG:boto:Token: None 2022-09-02T05:55:50.376 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c983d4d0b627b03-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.382 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.383 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.384 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.384 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.385 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:50.385 DEBUG:boto:Data: 2022-09-02T05:55:50.386 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.387 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.387 DEBUG:boto:Params: {} 2022-09-02T05:55:50.388 DEBUG:boto:Token: None 2022-09-02T05:55:50.388 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11 2022-09-02T05:55:50.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sQedGYNxaRo6o82jHBSRKy9blxU= 2022-09-02T05:55:50.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sQedGYNxaRo6o82jHBSRKy9blxU=', 'Content-Length': '0'} 2022-09-02T05:55:50.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6452cf57606aca-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.393 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.393 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.394 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.394 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:50.395 DEBUG:boto:Data: 2022-09-02T05:55:50.395 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.396 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.397 DEBUG:boto:Params: {} 2022-09-02T05:55:50.397 DEBUG:boto:Token: None 2022-09-02T05:55:50.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11 2022-09-02T05:55:50.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sQedGYNxaRo6o82jHBSRKy9blxU= 2022-09-02T05:55:50.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sQedGYNxaRo6o82jHBSRKy9blxU=', 'Content-Length': '0'} 2022-09-02T05:55:50.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a166baceb94c5def-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:50.403 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.403 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.404 DEBUG:boto:Method: GET 2022-09-02T05:55:50.404 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:50.405 DEBUG:boto:Data: 2022-09-02T05:55:50.406 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.407 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.408 DEBUG:boto:Params: {} 2022-09-02T05:55:50.408 DEBUG:boto:Token: None 2022-09-02T05:55:50.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11 2022-09-02T05:55:50.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0xppQALMX5JnS0QPY8/IdLxd9nU= 2022-09-02T05:55:50.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0xppQALMX5JnS0QPY8/IdLxd9nU=', 'Content-Length': '0'} 2022-09-02T05:55:50.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae2ae4884b63a27-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.454 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:50.455 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:50.456 DEBUG:boto:Method: GET 2022-09-02T05:55:50.457 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:50.458 DEBUG:boto:Data: 2022-09-02T05:55:50.459 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.461 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.462 DEBUG:boto:Params: {} 2022-09-02T05:55:50.464 DEBUG:boto:Token: None 2022-09-02T05:55:50.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj11 2022-09-02T05:55:50.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0xppQALMX5JnS0QPY8/IdLxd9nU= 2022-09-02T05:55:50.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0xppQALMX5JnS0QPY8/IdLxd9nU=', 'Content-Length': '0'} 2022-09-02T05:55:50.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174a8145dbb089c6-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.471 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.472 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.472 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.473 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.473 DEBUG:boto:Method: GET 2022-09-02T05:55:50.474 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.475 DEBUG:boto:Data: 2022-09-02T05:55:50.475 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.476 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.477 DEBUG:boto:Params: {} 2022-09-02T05:55:50.478 DEBUG:boto:Token: None 2022-09-02T05:55:50.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585f545653b805bf-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.484 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.485 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.485 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.486 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.486 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.487 DEBUG:boto:Method: GET 2022-09-02T05:55:50.488 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:50.488 DEBUG:boto:Data: 2022-09-02T05:55:50.489 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.502 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.503 DEBUG:boto:Params: {} 2022-09-02T05:55:50.503 DEBUG:boto:Token: None 2022-09-02T05:55:50.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7965b886279d4b-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.554 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:50.556 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.557 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.558 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.558 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.559 DEBUG:boto:Method: GET 2022-09-02T05:55:50.559 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.560 DEBUG:boto:Data: 2022-09-02T05:55:50.560 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.561 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.562 DEBUG:boto:Params: {} 2022-09-02T05:55:50.563 DEBUG:boto:Token: None 2022-09-02T05:55:50.563 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI= 2022-09-02T05:55:50.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI=', 'Content-Length': '0'} 2022-09-02T05:55:50.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000912533ecfe55a9ee-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.568 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.569 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.569 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.570 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.570 DEBUG:boto:Method: GET 2022-09-02T05:55:50.571 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.572 DEBUG:boto:Data: 2022-09-02T05:55:50.572 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.573 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.574 DEBUG:boto:Params: {} 2022-09-02T05:55:50.575 DEBUG:boto:Token: None 2022-09-02T05:55:50.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI= 2022-09-02T05:55:50.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI=', 'Content-Length': '0'} 2022-09-02T05:55:50.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271835a73074a2ab-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.580 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.581 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.581 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.582 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.582 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.583 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.583 DEBUG:boto:Data: 2022-09-02T05:55:50.584 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.585 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.586 DEBUG:boto:Params: {} 2022-09-02T05:55:50.586 DEBUG:boto:Token: None 2022-09-02T05:55:50.587 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2mOuqMa+IioyIg+oTJWjFnwsACg= 2022-09-02T05:55:50.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2mOuqMa+IioyIg+oTJWjFnwsACg=', 'Content-Length': '0'} 2022-09-02T05:55:50.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77795599a00b156-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.591 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.592 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.593 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.593 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.594 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.594 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.595 DEBUG:boto:Data: 2022-09-02T05:55:50.595 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.596 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.597 DEBUG:boto:Params: {} 2022-09-02T05:55:50.598 DEBUG:boto:Token: None 2022-09-02T05:55:50.598 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2mOuqMa+IioyIg+oTJWjFnwsACg= 2022-09-02T05:55:50.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2mOuqMa+IioyIg+oTJWjFnwsACg=', 'Content-Length': '0'} 2022-09-02T05:55:50.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4722538721fd86-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:50.604 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.604 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.605 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.605 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.606 DEBUG:boto:Method: GET 2022-09-02T05:55:50.606 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.607 DEBUG:boto:Data: 2022-09-02T05:55:50.607 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.609 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.609 DEBUG:boto:Params: {} 2022-09-02T05:55:50.610 DEBUG:boto:Token: None 2022-09-02T05:55:50.610 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI= 2022-09-02T05:55:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI=', 'Content-Length': '0'} 2022-09-02T05:55:50.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8b190da20aed38-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.615 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.616 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.616 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.617 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.618 DEBUG:boto:Method: GET 2022-09-02T05:55:50.618 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.619 DEBUG:boto:Data: 2022-09-02T05:55:50.619 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.620 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.621 DEBUG:boto:Params: {} 2022-09-02T05:55:50.621 DEBUG:boto:Token: None 2022-09-02T05:55:50.622 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:50.623 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI= 2022-09-02T05:55:50.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XD/kREXjDCQmqdMY3p9ylsFvqRI=', 'Content-Length': '0'} 2022-09-02T05:55:50.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004651b882a50b6bf2-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.626 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.627 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.628 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.628 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.629 DEBUG:boto:Method: GET 2022-09-02T05:55:50.630 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.631 DEBUG:boto:Data: 2022-09-02T05:55:50.632 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.634 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.635 DEBUG:boto:Params: {} 2022-09-02T05:55:50.636 DEBUG:boto:Token: None 2022-09-02T05:55:50.637 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0b4a34783eaad7-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.643 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.644 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.644 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.645 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.646 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.646 DEBUG:boto:Method: GET 2022-09-02T05:55:50.647 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.647 DEBUG:boto:Data: 2022-09-02T05:55:50.648 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.649 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.649 DEBUG:boto:Params: {} 2022-09-02T05:55:50.650 DEBUG:boto:Token: None 2022-09-02T05:55:50.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648610c52957574c-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.658 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.659 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.660 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.661 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.662 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:50.663 DEBUG:boto:Data: 2022-09-02T05:55:50.664 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.665 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.666 DEBUG:boto:Params: {} 2022-09-02T05:55:50.666 DEBUG:boto:Token: None 2022-09-02T05:55:50.667 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115 2022-09-02T05:55:50.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khy99d149BWjl0YwxVYMAvoh+oE= 2022-09-02T05:55:50.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khy99d149BWjl0YwxVYMAvoh+oE=', 'Content-Length': '0'} 2022-09-02T05:55:50.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90ee2899c3ce4e7-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.671 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.672 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.672 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.673 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:50.673 DEBUG:boto:Data: 2022-09-02T05:55:50.674 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.675 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.676 DEBUG:boto:Params: {} 2022-09-02T05:55:50.676 DEBUG:boto:Token: None 2022-09-02T05:55:50.677 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115 2022-09-02T05:55:50.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:khy99d149BWjl0YwxVYMAvoh+oE= 2022-09-02T05:55:50.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:khy99d149BWjl0YwxVYMAvoh+oE=', 'Content-Length': '0'} 2022-09-02T05:55:50.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e130b2d6039aecfb-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:50.682 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.682 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.683 DEBUG:boto:Method: GET 2022-09-02T05:55:50.684 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:50.684 DEBUG:boto:Data: 2022-09-02T05:55:50.685 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.686 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.686 DEBUG:boto:Params: {} 2022-09-02T05:55:50.687 DEBUG:boto:Token: None 2022-09-02T05:55:50.688 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115 2022-09-02T05:55:50.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZrZzANoYogOz7OVjJi1MI++K1bg= 2022-09-02T05:55:50.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZrZzANoYogOz7OVjJi1MI++K1bg=', 'Content-Length': '0'} 2022-09-02T05:55:50.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abfafc66a5b826b9-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.738 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:50.739 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:50.739 DEBUG:boto:Method: GET 2022-09-02T05:55:50.740 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:50.740 DEBUG:boto:Data: 2022-09-02T05:55:50.741 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.742 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.743 DEBUG:boto:Params: {} 2022-09-02T05:55:50.743 DEBUG:boto:Token: None 2022-09-02T05:55:50.744 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj115 2022-09-02T05:55:50.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZrZzANoYogOz7OVjJi1MI++K1bg= 2022-09-02T05:55:50.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZrZzANoYogOz7OVjJi1MI++K1bg=', 'Content-Length': '0'} 2022-09-02T05:55:50.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d143d86e3579627-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.748 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.749 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.749 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.750 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.750 DEBUG:boto:Method: GET 2022-09-02T05:55:50.751 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.751 DEBUG:boto:Data: 2022-09-02T05:55:50.752 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.753 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.753 DEBUG:boto:Params: {} 2022-09-02T05:55:50.754 DEBUG:boto:Token: None 2022-09-02T05:55:50.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b92791f4c5235aee-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.759 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.760 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.761 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.761 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.762 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.762 DEBUG:boto:Method: GET 2022-09-02T05:55:50.763 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:50.763 DEBUG:boto:Data: 2022-09-02T05:55:50.764 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.765 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.765 DEBUG:boto:Params: {} 2022-09-02T05:55:50.766 DEBUG:boto:Token: None 2022-09-02T05:55:50.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001852f23bf9ef13f1-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.773 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:50.774 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.774 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.775 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.776 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.776 DEBUG:boto:Method: GET 2022-09-02T05:55:50.777 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.777 DEBUG:boto:Data: 2022-09-02T05:55:50.778 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.779 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.779 DEBUG:boto:Params: {} 2022-09-02T05:55:50.780 DEBUG:boto:Token: None 2022-09-02T05:55:50.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI= 2022-09-02T05:55:50.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI=', 'Content-Length': '0'} 2022-09-02T05:55:50.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c19fe832ccef32-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.785 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.786 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.786 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.787 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.787 DEBUG:boto:Method: GET 2022-09-02T05:55:50.788 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.788 DEBUG:boto:Data: 2022-09-02T05:55:50.789 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.790 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.790 DEBUG:boto:Params: {} 2022-09-02T05:55:50.791 DEBUG:boto:Token: None 2022-09-02T05:55:50.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI= 2022-09-02T05:55:50.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI=', 'Content-Length': '0'} 2022-09-02T05:55:50.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d0086e42745a0f-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.796 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.796 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.797 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.797 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.798 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.798 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.799 DEBUG:boto:Data: 2022-09-02T05:55:50.799 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.800 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.801 DEBUG:boto:Params: {} 2022-09-02T05:55:50.801 DEBUG:boto:Token: None 2022-09-02T05:55:50.802 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oL9MIM5q+1eTo4izsbT1d550s2Y= 2022-09-02T05:55:50.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oL9MIM5q+1eTo4izsbT1d550s2Y=', 'Content-Length': '0'} 2022-09-02T05:55:50.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e842f513e6c7d428-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.806 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.806 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.807 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.807 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.808 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.808 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.809 DEBUG:boto:Data: 2022-09-02T05:55:50.809 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.810 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.811 DEBUG:boto:Params: {} 2022-09-02T05:55:50.811 DEBUG:boto:Token: None 2022-09-02T05:55:50.812 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oL9MIM5q+1eTo4izsbT1d550s2Y= 2022-09-02T05:55:50.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oL9MIM5q+1eTo4izsbT1d550s2Y=', 'Content-Length': '0'} 2022-09-02T05:55:50.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de6d017993474fe8-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:50.816 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.817 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.818 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.818 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.819 DEBUG:boto:Method: GET 2022-09-02T05:55:50.819 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.820 DEBUG:boto:Data: 2022-09-02T05:55:50.820 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.821 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.822 DEBUG:boto:Params: {} 2022-09-02T05:55:50.822 DEBUG:boto:Token: None 2022-09-02T05:55:50.823 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI= 2022-09-02T05:55:50.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI=', 'Content-Length': '0'} 2022-09-02T05:55:50.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087389a910d7902b-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.828 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.828 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.829 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.829 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.830 DEBUG:boto:Method: GET 2022-09-02T05:55:50.830 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.831 DEBUG:boto:Data: 2022-09-02T05:55:50.831 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.833 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.833 DEBUG:boto:Params: {} 2022-09-02T05:55:50.834 DEBUG:boto:Token: None 2022-09-02T05:55:50.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:50.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI= 2022-09-02T05:55:50.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gg8/07rFLoPIwh5RNHCeMGKh8oI=', 'Content-Length': '0'} 2022-09-02T05:55:50.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b72ab12acb030080-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.839 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.839 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.840 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.840 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.841 DEBUG:boto:Method: GET 2022-09-02T05:55:50.841 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.842 DEBUG:boto:Data: 2022-09-02T05:55:50.842 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.843 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.844 DEBUG:boto:Params: {} 2022-09-02T05:55:50.844 DEBUG:boto:Token: None 2022-09-02T05:55:50.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bea50ffc92e7e6e0-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.890 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.891 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.892 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.892 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.893 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.893 DEBUG:boto:Method: GET 2022-09-02T05:55:50.894 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.894 DEBUG:boto:Data: 2022-09-02T05:55:50.895 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.896 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.896 DEBUG:boto:Params: {} 2022-09-02T05:55:50.897 DEBUG:boto:Token: None 2022-09-02T05:55:50.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a59c9312fe8e9e6-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.903 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.904 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.904 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.905 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.905 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:50.906 DEBUG:boto:Data: 2022-09-02T05:55:50.906 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.907 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.908 DEBUG:boto:Params: {} 2022-09-02T05:55:50.908 DEBUG:boto:Token: None 2022-09-02T05:55:50.909 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116 2022-09-02T05:55:50.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cD2OcZfAhmthXncx9dpXuaKgaEk= 2022-09-02T05:55:50.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cD2OcZfAhmthXncx9dpXuaKgaEk=', 'Content-Length': '0'} 2022-09-02T05:55:50.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc64e312b58091a-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.913 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.913 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.914 DEBUG:boto:Method: HEAD 2022-09-02T05:55:50.914 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:50.915 DEBUG:boto:Data: 2022-09-02T05:55:50.915 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.916 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.917 DEBUG:boto:Params: {} 2022-09-02T05:55:50.917 DEBUG:boto:Token: None 2022-09-02T05:55:50.918 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116 2022-09-02T05:55:50.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cD2OcZfAhmthXncx9dpXuaKgaEk= 2022-09-02T05:55:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cD2OcZfAhmthXncx9dpXuaKgaEk=', 'Content-Length': '0'} 2022-09-02T05:55:50.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e22b40f782d6222-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:50.922 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.923 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.923 DEBUG:boto:Method: GET 2022-09-02T05:55:50.924 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:50.924 DEBUG:boto:Data: 2022-09-02T05:55:50.925 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.926 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.926 DEBUG:boto:Params: {} 2022-09-02T05:55:50.927 DEBUG:boto:Token: None 2022-09-02T05:55:50.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116 2022-09-02T05:55:50.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ArA6HnitD1yXWR+oynyhsBOEeKs= 2022-09-02T05:55:50.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ArA6HnitD1yXWR+oynyhsBOEeKs=', 'Content-Length': '0'} 2022-09-02T05:55:50.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20feaa7fdb07ffc-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.931 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:50.932 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:50.933 DEBUG:boto:Method: GET 2022-09-02T05:55:50.933 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:50.934 DEBUG:boto:Data: 2022-09-02T05:55:50.934 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:50.935 DEBUG:boto:Port: 8000 2022-09-02T05:55:50.936 DEBUG:boto:Params: {} 2022-09-02T05:55:50.936 DEBUG:boto:Token: None 2022-09-02T05:55:50.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/obj116 2022-09-02T05:55:50.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ArA6HnitD1yXWR+oynyhsBOEeKs= 2022-09-02T05:55:50.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ArA6HnitD1yXWR+oynyhsBOEeKs=', 'Content-Length': '0'} 2022-09-02T05:55:50.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9154a7d0548dde1-0063119ae6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.982 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.983 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.984 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.985 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.985 DEBUG:boto:Method: GET 2022-09-02T05:55:50.986 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.986 DEBUG:boto:Data: 2022-09-02T05:55:50.987 DEBUG:boto:Headers: {} 2022-09-02T05:55:50.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:50.988 DEBUG:boto:Port: 8001 2022-09-02T05:55:50.988 DEBUG:boto:Params: {} 2022-09-02T05:55:50.989 DEBUG:boto:Token: None 2022-09-02T05:55:50.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:50 GMT /xljbkf-74/ 2022-09-02T05:55:50.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc= 2022-09-02T05:55:50.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WqoQdCvjwsdHQjN+TOw36M/+ngc=', 'Content-Length': '0'} 2022-09-02T05:55:50.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad87826480e7ddbc-0063119ae6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:50.995 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:50.996 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:50.997 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:50.998 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.998 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:50.999 DEBUG:boto:Method: GET 2022-09-02T05:55:50.999 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:51.000 DEBUG:boto:Data: 2022-09-02T05:55:51.000 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.001 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.002 DEBUG:boto:Params: {} 2022-09-02T05:55:51.003 DEBUG:boto:Token: None 2022-09-02T05:55:51.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000900dff584db87863-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.012 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:51.015 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.016 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.016 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.017 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.018 DEBUG:boto:Method: GET 2022-09-02T05:55:51.019 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.019 DEBUG:boto:Data: 2022-09-02T05:55:51.020 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.022 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.023 DEBUG:boto:Params: {} 2022-09-02T05:55:51.023 DEBUG:boto:Token: None 2022-09-02T05:55:51.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY= 2022-09-02T05:55:51.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY=', 'Content-Length': '0'} 2022-09-02T05:55:51.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf123e493128c0fd-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.028 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.029 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.030 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.030 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.031 DEBUG:boto:Method: GET 2022-09-02T05:55:51.031 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.032 DEBUG:boto:Data: 2022-09-02T05:55:51.032 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.034 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.034 DEBUG:boto:Params: {} 2022-09-02T05:55:51.035 DEBUG:boto:Token: None 2022-09-02T05:55:51.035 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY= 2022-09-02T05:55:51.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY=', 'Content-Length': '0'} 2022-09-02T05:55:51.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c105475d81dfa9-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.040 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.040 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.041 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.041 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.042 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.042 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.043 DEBUG:boto:Data: 2022-09-02T05:55:51.043 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.044 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.045 DEBUG:boto:Params: {} 2022-09-02T05:55:51.046 DEBUG:boto:Token: None 2022-09-02T05:55:51.046 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.047 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iM3itWB7oeMIIOBMCzX0p+orNE0= 2022-09-02T05:55:51.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iM3itWB7oeMIIOBMCzX0p+orNE0=', 'Content-Length': '0'} 2022-09-02T05:55:51.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7bdbded950d2efe-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.050 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.051 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.052 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.053 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.053 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.054 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.055 DEBUG:boto:Data: 2022-09-02T05:55:51.056 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.057 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.058 DEBUG:boto:Params: {} 2022-09-02T05:55:51.059 DEBUG:boto:Token: None 2022-09-02T05:55:51.060 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iM3itWB7oeMIIOBMCzX0p+orNE0= 2022-09-02T05:55:51.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iM3itWB7oeMIIOBMCzX0p+orNE0=', 'Content-Length': '0'} 2022-09-02T05:55:51.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e21655ef4871de20-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:51.067 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.068 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.069 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.070 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.071 DEBUG:boto:Method: GET 2022-09-02T05:55:51.071 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.072 DEBUG:boto:Data: 2022-09-02T05:55:51.073 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.075 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.076 DEBUG:boto:Params: {} 2022-09-02T05:55:51.077 DEBUG:boto:Token: None 2022-09-02T05:55:51.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY= 2022-09-02T05:55:51.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY=', 'Content-Length': '0'} 2022-09-02T05:55:51.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c57971363a301a2-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.085 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.086 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.087 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.087 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.088 DEBUG:boto:Method: GET 2022-09-02T05:55:51.089 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.090 DEBUG:boto:Data: 2022-09-02T05:55:51.091 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.093 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.094 DEBUG:boto:Params: {} 2022-09-02T05:55:51.095 DEBUG:boto:Token: None 2022-09-02T05:55:51.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:51.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY= 2022-09-02T05:55:51.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d34abOUSntmkOMcMzN9uzIZHGlY=', 'Content-Length': '0'} 2022-09-02T05:55:51.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc29df70719cccb-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.111 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.112 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.112 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.113 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.114 DEBUG:boto:Method: GET 2022-09-02T05:55:51.114 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.115 DEBUG:boto:Data: 2022-09-02T05:55:51.115 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.116 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.117 DEBUG:boto:Params: {} 2022-09-02T05:55:51.117 DEBUG:boto:Token: None 2022-09-02T05:55:51.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b45fa1c51374fc6-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.123 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.124 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.124 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.125 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.125 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.126 DEBUG:boto:Method: GET 2022-09-02T05:55:51.126 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.127 DEBUG:boto:Data: 2022-09-02T05:55:51.127 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.128 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.129 DEBUG:boto:Params: {} 2022-09-02T05:55:51.129 DEBUG:boto:Token: None 2022-09-02T05:55:51.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000833e4eb3d24aec8c-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.137 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.137 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.138 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.139 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.139 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:51.140 DEBUG:boto:Data: 2022-09-02T05:55:51.141 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.142 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.142 DEBUG:boto:Params: {} 2022-09-02T05:55:51.143 DEBUG:boto:Token: None 2022-09-02T05:55:51.144 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145 2022-09-02T05:55:51.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p8HGiKaXalMwzMoOQwE1UU+LmCM= 2022-09-02T05:55:51.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p8HGiKaXalMwzMoOQwE1UU+LmCM=', 'Content-Length': '0'} 2022-09-02T05:55:51.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f5a8fb707f568c-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.148 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.148 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.149 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.149 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:51.150 DEBUG:boto:Data: 2022-09-02T05:55:51.150 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.152 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.152 DEBUG:boto:Params: {} 2022-09-02T05:55:51.153 DEBUG:boto:Token: None 2022-09-02T05:55:51.153 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145 2022-09-02T05:55:51.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p8HGiKaXalMwzMoOQwE1UU+LmCM= 2022-09-02T05:55:51.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p8HGiKaXalMwzMoOQwE1UU+LmCM=', 'Content-Length': '0'} 2022-09-02T05:55:51.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31fdc5af4faf7d1-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:51.158 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.159 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.159 DEBUG:boto:Method: GET 2022-09-02T05:55:51.160 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:51.161 DEBUG:boto:Data: 2022-09-02T05:55:51.161 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.162 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.163 DEBUG:boto:Params: {} 2022-09-02T05:55:51.163 DEBUG:boto:Token: None 2022-09-02T05:55:51.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145 2022-09-02T05:55:51.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AJ4tCKohIyMHha3mwXxfQrJxpek= 2022-09-02T05:55:51.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AJ4tCKohIyMHha3mwXxfQrJxpek=', 'Content-Length': '0'} 2022-09-02T05:55:51.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec7ffa0dd9ffad2-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.214 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:51.215 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:51.216 DEBUG:boto:Method: GET 2022-09-02T05:55:51.217 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:51.218 DEBUG:boto:Data: 2022-09-02T05:55:51.219 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.220 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.221 DEBUG:boto:Params: {} 2022-09-02T05:55:51.222 DEBUG:boto:Token: None 2022-09-02T05:55:51.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj145 2022-09-02T05:55:51.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AJ4tCKohIyMHha3mwXxfQrJxpek= 2022-09-02T05:55:51.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AJ4tCKohIyMHha3mwXxfQrJxpek=', 'Content-Length': '0'} 2022-09-02T05:55:51.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc036f62bb16c7d-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.228 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.229 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.230 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.230 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.231 DEBUG:boto:Method: GET 2022-09-02T05:55:51.231 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.232 DEBUG:boto:Data: 2022-09-02T05:55:51.232 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.234 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.234 DEBUG:boto:Params: {} 2022-09-02T05:55:51.235 DEBUG:boto:Token: None 2022-09-02T05:55:51.235 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92b743c230e0ea6-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.241 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.241 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.242 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.243 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.243 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.244 DEBUG:boto:Method: GET 2022-09-02T05:55:51.244 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:51.245 DEBUG:boto:Data: 2022-09-02T05:55:51.245 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.247 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.247 DEBUG:boto:Params: {} 2022-09-02T05:55:51.248 DEBUG:boto:Token: None 2022-09-02T05:55:51.248 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.249 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8fc110db5a445b-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.294 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:51.296 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.296 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.297 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.298 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.298 DEBUG:boto:Method: GET 2022-09-02T05:55:51.299 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.299 DEBUG:boto:Data: 2022-09-02T05:55:51.300 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.303 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.304 DEBUG:boto:Params: {} 2022-09-02T05:55:51.305 DEBUG:boto:Token: None 2022-09-02T05:55:51.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24= 2022-09-02T05:55:51.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24=', 'Content-Length': '0'} 2022-09-02T05:55:51.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000494b3ace2ed7bb52-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.312 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.313 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.314 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.314 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.315 DEBUG:boto:Method: GET 2022-09-02T05:55:51.315 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.316 DEBUG:boto:Data: 2022-09-02T05:55:51.316 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.318 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.319 DEBUG:boto:Params: {} 2022-09-02T05:55:51.319 DEBUG:boto:Token: None 2022-09-02T05:55:51.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24= 2022-09-02T05:55:51.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24=', 'Content-Length': '0'} 2022-09-02T05:55:51.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a948accf21ec03-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.325 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.325 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.326 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.326 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.327 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.327 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.328 DEBUG:boto:Data: 2022-09-02T05:55:51.329 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.330 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.331 DEBUG:boto:Params: {} 2022-09-02T05:55:51.332 DEBUG:boto:Token: None 2022-09-02T05:55:51.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nlQmdlTl+DXW0/arzyn/KhJFG5E= 2022-09-02T05:55:51.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nlQmdlTl+DXW0/arzyn/KhJFG5E=', 'Content-Length': '0'} 2022-09-02T05:55:51.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056bcf252429fa2ea-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.339 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.340 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.341 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.341 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.342 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.342 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.343 DEBUG:boto:Data: 2022-09-02T05:55:51.343 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.344 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.345 DEBUG:boto:Params: {} 2022-09-02T05:55:51.346 DEBUG:boto:Token: None 2022-09-02T05:55:51.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nlQmdlTl+DXW0/arzyn/KhJFG5E= 2022-09-02T05:55:51.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nlQmdlTl+DXW0/arzyn/KhJFG5E=', 'Content-Length': '0'} 2022-09-02T05:55:51.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92dffbd712da698-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:51.352 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.353 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.353 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.354 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.355 DEBUG:boto:Method: GET 2022-09-02T05:55:51.356 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.357 DEBUG:boto:Data: 2022-09-02T05:55:51.357 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.359 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.360 DEBUG:boto:Params: {} 2022-09-02T05:55:51.361 DEBUG:boto:Token: None 2022-09-02T05:55:51.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24= 2022-09-02T05:55:51.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24=', 'Content-Length': '0'} 2022-09-02T05:55:51.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206f8fbff6cf306b-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.367 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.368 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.368 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.369 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.370 DEBUG:boto:Method: GET 2022-09-02T05:55:51.371 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.372 DEBUG:boto:Data: 2022-09-02T05:55:51.372 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.374 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.375 DEBUG:boto:Params: {} 2022-09-02T05:55:51.376 DEBUG:boto:Token: None 2022-09-02T05:55:51.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:51.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24= 2022-09-02T05:55:51.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BkuF/ulfFqAOsm8Vb8JUTlm1z24=', 'Content-Length': '0'} 2022-09-02T05:55:51.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000988b60016a6cecf2-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.386 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.387 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.388 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.389 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.390 DEBUG:boto:Method: GET 2022-09-02T05:55:51.392 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.392 DEBUG:boto:Data: 2022-09-02T05:55:51.393 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.395 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.395 DEBUG:boto:Params: {} 2022-09-02T05:55:51.397 DEBUG:boto:Token: None 2022-09-02T05:55:51.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da462005faabc41d-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.404 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.405 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.406 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.406 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.407 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.407 DEBUG:boto:Method: GET 2022-09-02T05:55:51.408 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.409 DEBUG:boto:Data: 2022-09-02T05:55:51.409 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.410 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.411 DEBUG:boto:Params: {} 2022-09-02T05:55:51.412 DEBUG:boto:Token: None 2022-09-02T05:55:51.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b106468361e86f48-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.418 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.419 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.420 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.420 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.421 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:51.421 DEBUG:boto:Data: 2022-09-02T05:55:51.422 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.423 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.423 DEBUG:boto:Params: {} 2022-09-02T05:55:51.424 DEBUG:boto:Token: None 2022-09-02T05:55:51.425 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17 2022-09-02T05:55:51.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pQKZtDnOesyibsQtbkkCVRtXsAc= 2022-09-02T05:55:51.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pQKZtDnOesyibsQtbkkCVRtXsAc=', 'Content-Length': '0'} 2022-09-02T05:55:51.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed811d1fb93d4f5b-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.429 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.429 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.430 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.430 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:51.431 DEBUG:boto:Data: 2022-09-02T05:55:51.431 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.432 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.433 DEBUG:boto:Params: {} 2022-09-02T05:55:51.433 DEBUG:boto:Token: None 2022-09-02T05:55:51.434 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17 2022-09-02T05:55:51.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pQKZtDnOesyibsQtbkkCVRtXsAc= 2022-09-02T05:55:51.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pQKZtDnOesyibsQtbkkCVRtXsAc=', 'Content-Length': '0'} 2022-09-02T05:55:51.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ca0141590944b7-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:51.439 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.440 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.440 DEBUG:boto:Method: GET 2022-09-02T05:55:51.441 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:51.441 DEBUG:boto:Data: 2022-09-02T05:55:51.442 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.443 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.444 DEBUG:boto:Params: {} 2022-09-02T05:55:51.445 DEBUG:boto:Token: None 2022-09-02T05:55:51.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17 2022-09-02T05:55:51.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0swMqlNsjVgApuD/ISjIURQpVSs= 2022-09-02T05:55:51.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0swMqlNsjVgApuD/ISjIURQpVSs=', 'Content-Length': '0'} 2022-09-02T05:55:51.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d5fa4a16f0f2c9-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.494 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:51.495 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:51.496 DEBUG:boto:Method: GET 2022-09-02T05:55:51.496 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:51.497 DEBUG:boto:Data: 2022-09-02T05:55:51.498 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.499 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.500 DEBUG:boto:Params: {} 2022-09-02T05:55:51.501 DEBUG:boto:Token: None 2022-09-02T05:55:51.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj17 2022-09-02T05:55:51.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0swMqlNsjVgApuD/ISjIURQpVSs= 2022-09-02T05:55:51.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0swMqlNsjVgApuD/ISjIURQpVSs=', 'Content-Length': '0'} 2022-09-02T05:55:51.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c674de852bc468e-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.506 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.507 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.508 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.508 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.509 DEBUG:boto:Method: GET 2022-09-02T05:55:51.509 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.510 DEBUG:boto:Data: 2022-09-02T05:55:51.511 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.512 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.513 DEBUG:boto:Params: {} 2022-09-02T05:55:51.513 DEBUG:boto:Token: None 2022-09-02T05:55:51.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e261b8d3f1d761fa-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.519 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.520 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.520 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.521 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.521 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.522 DEBUG:boto:Method: GET 2022-09-02T05:55:51.523 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:51.523 DEBUG:boto:Data: 2022-09-02T05:55:51.524 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.525 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.526 DEBUG:boto:Params: {} 2022-09-02T05:55:51.527 DEBUG:boto:Token: None 2022-09-02T05:55:51.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d481f026eb76cabd-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.534 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:51.536 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.536 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.537 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.537 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.538 DEBUG:boto:Method: GET 2022-09-02T05:55:51.538 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.539 DEBUG:boto:Data: 2022-09-02T05:55:51.539 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.541 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.541 DEBUG:boto:Params: {} 2022-09-02T05:55:51.542 DEBUG:boto:Token: None 2022-09-02T05:55:51.543 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA= 2022-09-02T05:55:51.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA=', 'Content-Length': '0'} 2022-09-02T05:55:51.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e22c639017ee539-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.547 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.547 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.548 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.549 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.549 DEBUG:boto:Method: GET 2022-09-02T05:55:51.550 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.551 DEBUG:boto:Data: 2022-09-02T05:55:51.552 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.554 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.555 DEBUG:boto:Params: {} 2022-09-02T05:55:51.556 DEBUG:boto:Token: None 2022-09-02T05:55:51.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA= 2022-09-02T05:55:51.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA=', 'Content-Length': '0'} 2022-09-02T05:55:51.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014fdc7b4ee8db080-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.561 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.561 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.562 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.562 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.563 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.563 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.564 DEBUG:boto:Data: 2022-09-02T05:55:51.564 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.565 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.566 DEBUG:boto:Params: {} 2022-09-02T05:55:51.567 DEBUG:boto:Token: None 2022-09-02T05:55:51.567 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zHIq+zTEPIHTW1Om9cpsOGOGVcI= 2022-09-02T05:55:51.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zHIq+zTEPIHTW1Om9cpsOGOGVcI=', 'Content-Length': '0'} 2022-09-02T05:55:51.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fdc4608ed844286-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.572 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.572 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.573 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.574 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.574 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.575 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.575 DEBUG:boto:Data: 2022-09-02T05:55:51.576 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.577 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.577 DEBUG:boto:Params: {} 2022-09-02T05:55:51.578 DEBUG:boto:Token: None 2022-09-02T05:55:51.579 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zHIq+zTEPIHTW1Om9cpsOGOGVcI= 2022-09-02T05:55:51.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zHIq+zTEPIHTW1Om9cpsOGOGVcI=', 'Content-Length': '0'} 2022-09-02T05:55:51.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f876207044c2c303-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:51.585 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.585 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.586 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.586 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.587 DEBUG:boto:Method: GET 2022-09-02T05:55:51.588 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.588 DEBUG:boto:Data: 2022-09-02T05:55:51.589 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.590 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.591 DEBUG:boto:Params: {} 2022-09-02T05:55:51.592 DEBUG:boto:Token: None 2022-09-02T05:55:51.594 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.595 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA= 2022-09-02T05:55:51.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA=', 'Content-Length': '0'} 2022-09-02T05:55:51.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020cdfcfce400fb6e-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.647 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.648 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.649 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.650 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.651 DEBUG:boto:Method: GET 2022-09-02T05:55:51.651 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.652 DEBUG:boto:Data: 2022-09-02T05:55:51.653 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.654 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.655 DEBUG:boto:Params: {} 2022-09-02T05:55:51.656 DEBUG:boto:Token: None 2022-09-02T05:55:51.656 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:51.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA= 2022-09-02T05:55:51.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1t4U3U4rY9gpd3+HtdLiulnNxhA=', 'Content-Length': '0'} 2022-09-02T05:55:51.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b716d9419e4052-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.661 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.661 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.662 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.662 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.663 DEBUG:boto:Method: GET 2022-09-02T05:55:51.663 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.664 DEBUG:boto:Data: 2022-09-02T05:55:51.665 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.666 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.667 DEBUG:boto:Params: {} 2022-09-02T05:55:51.668 DEBUG:boto:Token: None 2022-09-02T05:55:51.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a275b04f2fac4f-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.673 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.675 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.676 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.677 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.678 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.678 DEBUG:boto:Method: GET 2022-09-02T05:55:51.679 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.680 DEBUG:boto:Data: 2022-09-02T05:55:51.681 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.682 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.683 DEBUG:boto:Params: {} 2022-09-02T05:55:51.683 DEBUG:boto:Token: None 2022-09-02T05:55:51.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fdf4970d9205f98-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.690 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.691 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.692 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.692 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.693 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:51.693 DEBUG:boto:Data: 2022-09-02T05:55:51.694 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.695 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.695 DEBUG:boto:Params: {} 2022-09-02T05:55:51.696 DEBUG:boto:Token: None 2022-09-02T05:55:51.697 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19 2022-09-02T05:55:51.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PiIjYq58ot/ctndmBklwUPBOfLQ= 2022-09-02T05:55:51.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PiIjYq58ot/ctndmBklwUPBOfLQ=', 'Content-Length': '0'} 2022-09-02T05:55:51.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae5b90ab4192ff4-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.701 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.701 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.702 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.703 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:51.703 DEBUG:boto:Data: 2022-09-02T05:55:51.704 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.705 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.705 DEBUG:boto:Params: {} 2022-09-02T05:55:51.706 DEBUG:boto:Token: None 2022-09-02T05:55:51.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19 2022-09-02T05:55:51.707 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PiIjYq58ot/ctndmBklwUPBOfLQ= 2022-09-02T05:55:51.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PiIjYq58ot/ctndmBklwUPBOfLQ=', 'Content-Length': '0'} 2022-09-02T05:55:51.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668f0eff5a088426-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:51.711 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.712 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.712 DEBUG:boto:Method: GET 2022-09-02T05:55:51.713 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:51.713 DEBUG:boto:Data: 2022-09-02T05:55:51.714 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.715 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.715 DEBUG:boto:Params: {} 2022-09-02T05:55:51.716 DEBUG:boto:Token: None 2022-09-02T05:55:51.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19 2022-09-02T05:55:51.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jMx+0+U7DmIwCRqbOAszbcq8O7M= 2022-09-02T05:55:51.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jMx+0+U7DmIwCRqbOAszbcq8O7M=', 'Content-Length': '0'} 2022-09-02T05:55:51.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000264d237c0b75606d-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.721 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:51.722 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:51.722 DEBUG:boto:Method: GET 2022-09-02T05:55:51.723 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:51.723 DEBUG:boto:Data: 2022-09-02T05:55:51.724 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.725 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.725 DEBUG:boto:Params: {} 2022-09-02T05:55:51.726 DEBUG:boto:Token: None 2022-09-02T05:55:51.727 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj19 2022-09-02T05:55:51.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jMx+0+U7DmIwCRqbOAszbcq8O7M= 2022-09-02T05:55:51.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jMx+0+U7DmIwCRqbOAszbcq8O7M=', 'Content-Length': '0'} 2022-09-02T05:55:51.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c8bc7026773112-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.731 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.732 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.732 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.733 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.734 DEBUG:boto:Method: GET 2022-09-02T05:55:51.734 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.735 DEBUG:boto:Data: 2022-09-02T05:55:51.736 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.737 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.738 DEBUG:boto:Params: {} 2022-09-02T05:55:51.739 DEBUG:boto:Token: None 2022-09-02T05:55:51.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.740 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0dc7862717ff7b-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.745 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.746 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.747 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.747 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.748 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.749 DEBUG:boto:Method: GET 2022-09-02T05:55:51.749 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:51.750 DEBUG:boto:Data: 2022-09-02T05:55:51.750 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.751 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.752 DEBUG:boto:Params: {} 2022-09-02T05:55:51.753 DEBUG:boto:Token: None 2022-09-02T05:55:51.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b85cf934e1a53f-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.802 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:51.804 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:51.805 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:51.806 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.807 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.808 DEBUG:boto:Method: GET 2022-09-02T05:55:51.809 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.810 DEBUG:boto:Data: 2022-09-02T05:55:51.811 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.813 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.814 DEBUG:boto:Params: {} 2022-09-02T05:55:51.815 DEBUG:boto:Token: None 2022-09-02T05:55:51.816 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU= 2022-09-02T05:55:51.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU=', 'Content-Length': '0'} 2022-09-02T05:55:51.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeef7ccb1e19bae6-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.824 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:51.825 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:51.825 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.826 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.827 DEBUG:boto:Method: GET 2022-09-02T05:55:51.827 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.828 DEBUG:boto:Data: 2022-09-02T05:55:51.828 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.829 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.830 DEBUG:boto:Params: {} 2022-09-02T05:55:51.830 DEBUG:boto:Token: None 2022-09-02T05:55:51.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU= 2022-09-02T05:55:51.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU=', 'Content-Length': '0'} 2022-09-02T05:55:51.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f01aed2d2fb59d-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.835 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:51.836 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:51.836 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.836 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.837 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.837 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.842 DEBUG:boto:Data: 2022-09-02T05:55:51.843 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.846 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.847 DEBUG:boto:Params: {} 2022-09-02T05:55:51.849 DEBUG:boto:Token: None 2022-09-02T05:55:51.850 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KGW1tEWNdcw47kEaQ+YChHE42do= 2022-09-02T05:55:51.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KGW1tEWNdcw47kEaQ+YChHE42do=', 'Content-Length': '0'} 2022-09-02T05:55:51.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079aa7a2fe7cfa6ea-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.856 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:51.856 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:51.857 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.857 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.858 DEBUG:boto:Method: HEAD 2022-09-02T05:55:51.859 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.859 DEBUG:boto:Data: 2022-09-02T05:55:51.860 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.861 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.861 DEBUG:boto:Params: {} 2022-09-02T05:55:51.862 DEBUG:boto:Token: None 2022-09-02T05:55:51.862 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KGW1tEWNdcw47kEaQ+YChHE42do= 2022-09-02T05:55:51.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KGW1tEWNdcw47kEaQ+YChHE42do=', 'Content-Length': '0'} 2022-09-02T05:55:51.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df12227b42e69f80-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:51.869 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:51.870 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:51.871 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.871 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.872 DEBUG:boto:Method: GET 2022-09-02T05:55:51.872 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.873 DEBUG:boto:Data: 2022-09-02T05:55:51.874 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.875 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.876 DEBUG:boto:Params: {} 2022-09-02T05:55:51.876 DEBUG:boto:Token: None 2022-09-02T05:55:51.878 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.879 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU= 2022-09-02T05:55:51.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU=', 'Content-Length': '0'} 2022-09-02T05:55:51.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9d56ac66d82519-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.929 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:51.931 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:51.932 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.933 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.934 DEBUG:boto:Method: GET 2022-09-02T05:55:51.935 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.936 DEBUG:boto:Data: 2022-09-02T05:55:51.937 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.939 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.940 DEBUG:boto:Params: {} 2022-09-02T05:55:51.941 DEBUG:boto:Token: None 2022-09-02T05:55:51.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:51.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU= 2022-09-02T05:55:51.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hRnscQqJBA48yJpivEaoeWyzAqU=', 'Content-Length': '0'} 2022-09-02T05:55:51.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d42a17dede77e44-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.947 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.948 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.949 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:51.949 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:51.950 DEBUG:boto:Method: GET 2022-09-02T05:55:51.950 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:51.951 DEBUG:boto:Data: 2022-09-02T05:55:51.952 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:51.954 DEBUG:boto:Port: 8001 2022-09-02T05:55:51.955 DEBUG:boto:Params: {} 2022-09-02T05:55:51.956 DEBUG:boto:Token: None 2022-09-02T05:55:51.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b2c3f1d7f43d6a-0063119ae7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:51.962 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:51.964 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:51.965 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:51.966 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:51.967 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:51.968 DEBUG:boto:Method: GET 2022-09-02T05:55:51.969 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:51.969 DEBUG:boto:Data: 2022-09-02T05:55:51.970 DEBUG:boto:Headers: {} 2022-09-02T05:55:51.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:51.971 DEBUG:boto:Port: 8000 2022-09-02T05:55:51.972 DEBUG:boto:Params: {} 2022-09-02T05:55:51.972 DEBUG:boto:Token: None 2022-09-02T05:55:51.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:51 GMT /xljbkf-74/ 2022-09-02T05:55:51.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s= 2022-09-02T05:55:51.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C5gADS4bXiM/lOtOzKHpzXRSP1s=', 'Content-Length': '0'} 2022-09-02T05:55:51.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a456ae5fdc4e4ab-0063119ae7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.026 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.029 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:52.031 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:52.032 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.033 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:52.034 DEBUG:boto:Data: 2022-09-02T05:55:52.034 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.036 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.037 DEBUG:boto:Params: {} 2022-09-02T05:55:52.038 DEBUG:boto:Token: None 2022-09-02T05:55:52.039 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj22 2022-09-02T05:55:52.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+o7sxyy1rAI39WiTK424bedGCNk= 2022-09-02T05:55:52.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+o7sxyy1rAI39WiTK424bedGCNk=', 'Content-Length': '0'} 2022-09-02T05:55:52.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4183b925580aa4d-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.046 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:52.047 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:52.048 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.049 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:52.050 DEBUG:boto:Data: 2022-09-02T05:55:52.051 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.053 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.054 DEBUG:boto:Params: {} 2022-09-02T05:55:52.055 DEBUG:boto:Token: None 2022-09-02T05:55:52.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj22 2022-09-02T05:55:52.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+o7sxyy1rAI39WiTK424bedGCNk= 2022-09-02T05:55:52.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+o7sxyy1rAI39WiTK424bedGCNk=', 'Content-Length': '0'} 2022-09-02T05:55:52.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ca7923b4878814-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:52.065 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:52.066 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:52.067 DEBUG:boto:Method: GET 2022-09-02T05:55:52.068 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:52.069 DEBUG:boto:Data: 2022-09-02T05:55:52.070 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.072 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.073 DEBUG:boto:Params: {} 2022-09-02T05:55:52.074 DEBUG:boto:Token: None 2022-09-02T05:55:52.076 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj22 2022-09-02T05:55:52.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qLWPjNTZwGFS2NADmuTgfGbC9R0= 2022-09-02T05:55:52.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qLWPjNTZwGFS2NADmuTgfGbC9R0=', 'Content-Length': '0'} 2022-09-02T05:55:52.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009286ddee5e8f9448-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.083 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:52.084 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:52.085 DEBUG:boto:Method: GET 2022-09-02T05:55:52.086 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:52.087 DEBUG:boto:Data: 2022-09-02T05:55:52.088 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.091 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.092 DEBUG:boto:Params: {} 2022-09-02T05:55:52.093 DEBUG:boto:Token: None 2022-09-02T05:55:52.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj22 2022-09-02T05:55:52.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qLWPjNTZwGFS2NADmuTgfGbC9R0= 2022-09-02T05:55:52.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qLWPjNTZwGFS2NADmuTgfGbC9R0=', 'Content-Length': '0'} 2022-09-02T05:55:52.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba0f903d3f51f651-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.102 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.103 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.104 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:52.105 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:52.106 DEBUG:boto:Method: GET 2022-09-02T05:55:52.107 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:52.108 DEBUG:boto:Data: 2022-09-02T05:55:52.109 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.112 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.113 DEBUG:boto:Params: {} 2022-09-02T05:55:52.115 DEBUG:boto:Token: None 2022-09-02T05:55:52.116 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cabf9ab4a794d22-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.123 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.123 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.124 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.125 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:52.125 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:52.126 DEBUG:boto:Method: GET 2022-09-02T05:55:52.126 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:52.127 DEBUG:boto:Data: 2022-09-02T05:55:52.128 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.129 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.129 DEBUG:boto:Params: {} 2022-09-02T05:55:52.130 DEBUG:boto:Token: None 2022-09-02T05:55:52.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004071f74a727da216-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.138 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:52.140 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.141 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.142 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.144 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.145 DEBUG:boto:Method: GET 2022-09-02T05:55:52.145 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.146 DEBUG:boto:Data: 2022-09-02T05:55:52.146 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.148 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.148 DEBUG:boto:Params: {} 2022-09-02T05:55:52.149 DEBUG:boto:Token: None 2022-09-02T05:55:52.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA= 2022-09-02T05:55:52.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA=', 'Content-Length': '0'} 2022-09-02T05:55:52.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003338ea01ee33626a-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.154 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.155 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.155 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.156 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.156 DEBUG:boto:Method: GET 2022-09-02T05:55:52.157 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.158 DEBUG:boto:Data: 2022-09-02T05:55:52.158 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.160 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.160 DEBUG:boto:Params: {} 2022-09-02T05:55:52.161 DEBUG:boto:Token: None 2022-09-02T05:55:52.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA= 2022-09-02T05:55:52.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA=', 'Content-Length': '0'} 2022-09-02T05:55:52.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c1d392a34d759d-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.166 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.167 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.168 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.169 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.170 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.170 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.171 DEBUG:boto:Data: 2022-09-02T05:55:52.171 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.172 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.173 DEBUG:boto:Params: {} 2022-09-02T05:55:52.173 DEBUG:boto:Token: None 2022-09-02T05:55:52.174 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yhiaW83csql5Uwt/kCdv7i3hg2I= 2022-09-02T05:55:52.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yhiaW83csql5Uwt/kCdv7i3hg2I=', 'Content-Length': '0'} 2022-09-02T05:55:52.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a482099e18a93e6-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.178 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.179 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.179 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.180 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.180 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.181 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.181 DEBUG:boto:Data: 2022-09-02T05:55:52.182 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.183 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.183 DEBUG:boto:Params: {} 2022-09-02T05:55:52.184 DEBUG:boto:Token: None 2022-09-02T05:55:52.185 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yhiaW83csql5Uwt/kCdv7i3hg2I= 2022-09-02T05:55:52.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yhiaW83csql5Uwt/kCdv7i3hg2I=', 'Content-Length': '0'} 2022-09-02T05:55:52.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d84232fc658d7f-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:52.190 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.190 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.191 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.191 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.192 DEBUG:boto:Method: GET 2022-09-02T05:55:52.192 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.193 DEBUG:boto:Data: 2022-09-02T05:55:52.194 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.195 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.195 DEBUG:boto:Params: {} 2022-09-02T05:55:52.196 DEBUG:boto:Token: None 2022-09-02T05:55:52.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA= 2022-09-02T05:55:52.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA=', 'Content-Length': '0'} 2022-09-02T05:55:52.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f88a5ab6a4f2c6-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.204 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.205 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.206 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.207 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.208 DEBUG:boto:Method: GET 2022-09-02T05:55:52.209 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.209 DEBUG:boto:Data: 2022-09-02T05:55:52.210 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.211 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.211 DEBUG:boto:Params: {} 2022-09-02T05:55:52.212 DEBUG:boto:Token: None 2022-09-02T05:55:52.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:52.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA= 2022-09-02T05:55:52.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yOpKjgKHD4b5ZcFDS6G5bEo/yxA=', 'Content-Length': '0'} 2022-09-02T05:55:52.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000116d103fd8db9ec-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.217 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.218 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.218 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.219 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.220 DEBUG:boto:Method: GET 2022-09-02T05:55:52.220 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.221 DEBUG:boto:Data: 2022-09-02T05:55:52.222 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.223 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.224 DEBUG:boto:Params: {} 2022-09-02T05:55:52.224 DEBUG:boto:Token: None 2022-09-02T05:55:52.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048fbf71508467c87-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.230 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.231 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.231 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.232 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.232 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.233 DEBUG:boto:Method: GET 2022-09-02T05:55:52.234 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.234 DEBUG:boto:Data: 2022-09-02T05:55:52.235 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.236 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.236 DEBUG:boto:Params: {} 2022-09-02T05:55:52.237 DEBUG:boto:Token: None 2022-09-02T05:55:52.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227e3d8611fa0169-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.244 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.246 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.247 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.248 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.249 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:52.249 DEBUG:boto:Data: 2022-09-02T05:55:52.250 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.252 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.253 DEBUG:boto:Params: {} 2022-09-02T05:55:52.253 DEBUG:boto:Token: None 2022-09-02T05:55:52.254 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23 2022-09-02T05:55:52.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3mXr6kc9QP5cplaBhL+QxH5Cv9k= 2022-09-02T05:55:52.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3mXr6kc9QP5cplaBhL+QxH5Cv9k=', 'Content-Length': '0'} 2022-09-02T05:55:52.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157a9d93f414a076-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.258 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.258 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.259 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.260 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:52.260 DEBUG:boto:Data: 2022-09-02T05:55:52.261 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.262 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.262 DEBUG:boto:Params: {} 2022-09-02T05:55:52.263 DEBUG:boto:Token: None 2022-09-02T05:55:52.264 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23 2022-09-02T05:55:52.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3mXr6kc9QP5cplaBhL+QxH5Cv9k= 2022-09-02T05:55:52.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3mXr6kc9QP5cplaBhL+QxH5Cv9k=', 'Content-Length': '0'} 2022-09-02T05:55:52.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3b5f778b041da6-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:52.268 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.269 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.269 DEBUG:boto:Method: GET 2022-09-02T05:55:52.270 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:52.271 DEBUG:boto:Data: 2022-09-02T05:55:52.271 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.273 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.273 DEBUG:boto:Params: {} 2022-09-02T05:55:52.274 DEBUG:boto:Token: None 2022-09-02T05:55:52.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23 2022-09-02T05:55:52.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z98wevO6z7Aib+SGM6qqjjzOcVU= 2022-09-02T05:55:52.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z98wevO6z7Aib+SGM6qqjjzOcVU=', 'Content-Length': '0'} 2022-09-02T05:55:52.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b4a1d865089c09d-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.279 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:52.279 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:52.280 DEBUG:boto:Method: GET 2022-09-02T05:55:52.280 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:52.281 DEBUG:boto:Data: 2022-09-02T05:55:52.282 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.283 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.283 DEBUG:boto:Params: {} 2022-09-02T05:55:52.284 DEBUG:boto:Token: None 2022-09-02T05:55:52.285 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj23 2022-09-02T05:55:52.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z98wevO6z7Aib+SGM6qqjjzOcVU= 2022-09-02T05:55:52.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z98wevO6z7Aib+SGM6qqjjzOcVU=', 'Content-Length': '0'} 2022-09-02T05:55:52.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2fe1a7324ff329e-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.334 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.336 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.337 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.338 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.339 DEBUG:boto:Method: GET 2022-09-02T05:55:52.340 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.340 DEBUG:boto:Data: 2022-09-02T05:55:52.341 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.343 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.344 DEBUG:boto:Params: {} 2022-09-02T05:55:52.344 DEBUG:boto:Token: None 2022-09-02T05:55:52.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a2ec4b5c02fc42e-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.350 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.351 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.352 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.352 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.353 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.353 DEBUG:boto:Method: GET 2022-09-02T05:55:52.354 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:52.354 DEBUG:boto:Data: 2022-09-02T05:55:52.355 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.356 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.357 DEBUG:boto:Params: {} 2022-09-02T05:55:52.357 DEBUG:boto:Token: None 2022-09-02T05:55:52.358 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ecfbbeaf5375d3f-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.365 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:52.367 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.368 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.369 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.370 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.371 DEBUG:boto:Method: GET 2022-09-02T05:55:52.371 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.372 DEBUG:boto:Data: 2022-09-02T05:55:52.373 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.374 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.375 DEBUG:boto:Params: {} 2022-09-02T05:55:52.376 DEBUG:boto:Token: None 2022-09-02T05:55:52.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU= 2022-09-02T05:55:52.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU=', 'Content-Length': '0'} 2022-09-02T05:55:52.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40bdfa5fee45409-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.382 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.383 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.384 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.384 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.385 DEBUG:boto:Method: GET 2022-09-02T05:55:52.385 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.386 DEBUG:boto:Data: 2022-09-02T05:55:52.386 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.388 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.388 DEBUG:boto:Params: {} 2022-09-02T05:55:52.389 DEBUG:boto:Token: None 2022-09-02T05:55:52.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU= 2022-09-02T05:55:52.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU=', 'Content-Length': '0'} 2022-09-02T05:55:52.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b52867380bb3f2-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.394 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.394 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.395 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.395 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.396 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.396 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.397 DEBUG:boto:Data: 2022-09-02T05:55:52.397 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.398 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.399 DEBUG:boto:Params: {} 2022-09-02T05:55:52.399 DEBUG:boto:Token: None 2022-09-02T05:55:52.400 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8/f26/pM/Ma6zdm7lXqZ6SjMFo4= 2022-09-02T05:55:52.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8/f26/pM/Ma6zdm7lXqZ6SjMFo4=', 'Content-Length': '0'} 2022-09-02T05:55:52.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891dc8dbe066fbc8-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.404 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.405 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.405 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.406 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.406 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.407 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.408 DEBUG:boto:Data: 2022-09-02T05:55:52.408 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.409 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.410 DEBUG:boto:Params: {} 2022-09-02T05:55:52.410 DEBUG:boto:Token: None 2022-09-02T05:55:52.411 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8/f26/pM/Ma6zdm7lXqZ6SjMFo4= 2022-09-02T05:55:52.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8/f26/pM/Ma6zdm7lXqZ6SjMFo4=', 'Content-Length': '0'} 2022-09-02T05:55:52.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825dc5f2ab68a0dc-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:52.416 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.417 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.417 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.418 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.418 DEBUG:boto:Method: GET 2022-09-02T05:55:52.419 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.419 DEBUG:boto:Data: 2022-09-02T05:55:52.420 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.421 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.422 DEBUG:boto:Params: {} 2022-09-02T05:55:52.422 DEBUG:boto:Token: None 2022-09-02T05:55:52.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU= 2022-09-02T05:55:52.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU=', 'Content-Length': '0'} 2022-09-02T05:55:52.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001702c88f136f7f96-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.428 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.428 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.429 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.429 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.430 DEBUG:boto:Method: GET 2022-09-02T05:55:52.430 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.431 DEBUG:boto:Data: 2022-09-02T05:55:52.431 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.432 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.433 DEBUG:boto:Params: {} 2022-09-02T05:55:52.434 DEBUG:boto:Token: None 2022-09-02T05:55:52.434 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:52.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU= 2022-09-02T05:55:52.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EPf27GfaVOOr7kldD0/C0NvbYfU=', 'Content-Length': '0'} 2022-09-02T05:55:52.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff33990eac40c33e-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.482 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.483 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.483 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.484 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.484 DEBUG:boto:Method: GET 2022-09-02T05:55:52.485 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.486 DEBUG:boto:Data: 2022-09-02T05:55:52.487 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.490 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.491 DEBUG:boto:Params: {} 2022-09-02T05:55:52.491 DEBUG:boto:Token: None 2022-09-02T05:55:52.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e9501aa7720d86-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.497 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.498 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.498 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.499 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.499 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.500 DEBUG:boto:Method: GET 2022-09-02T05:55:52.501 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.501 DEBUG:boto:Data: 2022-09-02T05:55:52.502 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.503 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.503 DEBUG:boto:Params: {} 2022-09-02T05:55:52.504 DEBUG:boto:Token: None 2022-09-02T05:55:52.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f3bb712560a41f-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.510 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.511 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.512 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.512 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.513 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:52.522 DEBUG:boto:Data: 2022-09-02T05:55:52.522 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.524 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.524 DEBUG:boto:Params: {} 2022-09-02T05:55:52.525 DEBUG:boto:Token: None 2022-09-02T05:55:52.525 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29 2022-09-02T05:55:52.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vv2yE01a5j4+Mq5MR/N6h69PVh8= 2022-09-02T05:55:52.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vv2yE01a5j4+Mq5MR/N6h69PVh8=', 'Content-Length': '0'} 2022-09-02T05:55:52.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000244a6f6b14889e73-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.530 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.530 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.531 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.531 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:52.532 DEBUG:boto:Data: 2022-09-02T05:55:52.532 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.533 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.534 DEBUG:boto:Params: {} 2022-09-02T05:55:52.535 DEBUG:boto:Token: None 2022-09-02T05:55:52.535 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29 2022-09-02T05:55:52.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vv2yE01a5j4+Mq5MR/N6h69PVh8= 2022-09-02T05:55:52.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vv2yE01a5j4+Mq5MR/N6h69PVh8=', 'Content-Length': '0'} 2022-09-02T05:55:52.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e00aa48036936c-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:52.540 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.541 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.541 DEBUG:boto:Method: GET 2022-09-02T05:55:52.542 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:52.542 DEBUG:boto:Data: 2022-09-02T05:55:52.543 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.544 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.544 DEBUG:boto:Params: {} 2022-09-02T05:55:52.545 DEBUG:boto:Token: None 2022-09-02T05:55:52.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29 2022-09-02T05:55:52.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v6c20hLcF3xJS6EdGxW6ZAYtnR0= 2022-09-02T05:55:52.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v6c20hLcF3xJS6EdGxW6ZAYtnR0=', 'Content-Length': '0'} 2022-09-02T05:55:52.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472b78fdd17c015f-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.550 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:52.550 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:52.551 DEBUG:boto:Method: GET 2022-09-02T05:55:52.552 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:52.552 DEBUG:boto:Data: 2022-09-02T05:55:52.553 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.554 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.554 DEBUG:boto:Params: {} 2022-09-02T05:55:52.555 DEBUG:boto:Token: None 2022-09-02T05:55:52.555 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj29 2022-09-02T05:55:52.556 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v6c20hLcF3xJS6EdGxW6ZAYtnR0= 2022-09-02T05:55:52.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v6c20hLcF3xJS6EdGxW6ZAYtnR0=', 'Content-Length': '0'} 2022-09-02T05:55:52.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b29aa6e2c73ecc-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.559 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.560 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.560 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.561 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.562 DEBUG:boto:Method: GET 2022-09-02T05:55:52.562 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.563 DEBUG:boto:Data: 2022-09-02T05:55:52.563 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.564 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.565 DEBUG:boto:Params: {} 2022-09-02T05:55:52.565 DEBUG:boto:Token: None 2022-09-02T05:55:52.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b7f3a534dd35f0-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.571 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.572 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.572 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.573 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.573 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.574 DEBUG:boto:Method: GET 2022-09-02T05:55:52.575 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:52.575 DEBUG:boto:Data: 2022-09-02T05:55:52.576 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.577 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.578 DEBUG:boto:Params: {} 2022-09-02T05:55:52.578 DEBUG:boto:Token: None 2022-09-02T05:55:52.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147f2b067fc71a0d-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.626 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:52.628 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.628 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.629 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.629 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.630 DEBUG:boto:Method: GET 2022-09-02T05:55:52.630 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.631 DEBUG:boto:Data: 2022-09-02T05:55:52.631 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.632 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.633 DEBUG:boto:Params: {} 2022-09-02T05:55:52.633 DEBUG:boto:Token: None 2022-09-02T05:55:52.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ= 2022-09-02T05:55:52.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ=', 'Content-Length': '0'} 2022-09-02T05:55:52.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008271c3bf6c0fa38d-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.639 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.639 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.640 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.640 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.641 DEBUG:boto:Method: GET 2022-09-02T05:55:52.641 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.642 DEBUG:boto:Data: 2022-09-02T05:55:52.642 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.643 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.644 DEBUG:boto:Params: {} 2022-09-02T05:55:52.645 DEBUG:boto:Token: None 2022-09-02T05:55:52.645 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ= 2022-09-02T05:55:52.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ=', 'Content-Length': '0'} 2022-09-02T05:55:52.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df31a438eb7ca9f-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.650 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.651 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.652 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.652 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.653 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.653 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.654 DEBUG:boto:Data: 2022-09-02T05:55:52.654 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.655 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.656 DEBUG:boto:Params: {} 2022-09-02T05:55:52.656 DEBUG:boto:Token: None 2022-09-02T05:55:52.657 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4hhviW5EOhLWtMPh55XcnBFRUuE= 2022-09-02T05:55:52.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4hhviW5EOhLWtMPh55XcnBFRUuE=', 'Content-Length': '0'} 2022-09-02T05:55:52.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db58991cbdf98add-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.661 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.662 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.662 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.663 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.663 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.664 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.664 DEBUG:boto:Data: 2022-09-02T05:55:52.665 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.666 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.667 DEBUG:boto:Params: {} 2022-09-02T05:55:52.667 DEBUG:boto:Token: None 2022-09-02T05:55:52.668 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4hhviW5EOhLWtMPh55XcnBFRUuE= 2022-09-02T05:55:52.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4hhviW5EOhLWtMPh55XcnBFRUuE=', 'Content-Length': '0'} 2022-09-02T05:55:52.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6c9c2c6952bbe7-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:52.673 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.674 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.675 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.676 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.677 DEBUG:boto:Method: GET 2022-09-02T05:55:52.677 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.678 DEBUG:boto:Data: 2022-09-02T05:55:52.679 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.680 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.680 DEBUG:boto:Params: {} 2022-09-02T05:55:52.681 DEBUG:boto:Token: None 2022-09-02T05:55:52.682 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ= 2022-09-02T05:55:52.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ=', 'Content-Length': '0'} 2022-09-02T05:55:52.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6eb51a6aae9e93-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.686 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.687 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.687 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.688 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.688 DEBUG:boto:Method: GET 2022-09-02T05:55:52.689 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.689 DEBUG:boto:Data: 2022-09-02T05:55:52.690 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.691 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.692 DEBUG:boto:Params: {} 2022-09-02T05:55:52.692 DEBUG:boto:Token: None 2022-09-02T05:55:52.693 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:52.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ= 2022-09-02T05:55:52.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7j7sWVMefo78BYUAEFwXV9aSRhQ=', 'Content-Length': '0'} 2022-09-02T05:55:52.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001836f6faa046300b-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.697 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.698 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.699 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.699 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.700 DEBUG:boto:Method: GET 2022-09-02T05:55:52.700 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.701 DEBUG:boto:Data: 2022-09-02T05:55:52.701 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.702 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.703 DEBUG:boto:Params: {} 2022-09-02T05:55:52.703 DEBUG:boto:Token: None 2022-09-02T05:55:52.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6fd9ecbad9513dc-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.709 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.710 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.710 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.711 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.711 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.712 DEBUG:boto:Method: GET 2022-09-02T05:55:52.712 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.713 DEBUG:boto:Data: 2022-09-02T05:55:52.713 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.714 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.715 DEBUG:boto:Params: {} 2022-09-02T05:55:52.716 DEBUG:boto:Token: None 2022-09-02T05:55:52.716 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048091534787b4f29-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.722 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.723 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.723 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.724 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.724 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:52.725 DEBUG:boto:Data: 2022-09-02T05:55:52.725 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.727 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.727 DEBUG:boto:Params: {} 2022-09-02T05:55:52.728 DEBUG:boto:Token: None 2022-09-02T05:55:52.728 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33 2022-09-02T05:55:52.729 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1JDcSuCOUjMbbWkboxuN/14csxc= 2022-09-02T05:55:52.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1JDcSuCOUjMbbWkboxuN/14csxc=', 'Content-Length': '0'} 2022-09-02T05:55:52.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae8c0a891de1e6d4-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.732 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.733 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.733 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.734 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:52.735 DEBUG:boto:Data: 2022-09-02T05:55:52.735 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.736 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.737 DEBUG:boto:Params: {} 2022-09-02T05:55:52.737 DEBUG:boto:Token: None 2022-09-02T05:55:52.738 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33 2022-09-02T05:55:52.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1JDcSuCOUjMbbWkboxuN/14csxc= 2022-09-02T05:55:52.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1JDcSuCOUjMbbWkboxuN/14csxc=', 'Content-Length': '0'} 2022-09-02T05:55:52.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be2bb39755ae626-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:52.742 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.743 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.744 DEBUG:boto:Method: GET 2022-09-02T05:55:52.744 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:52.745 DEBUG:boto:Data: 2022-09-02T05:55:52.745 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.746 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.747 DEBUG:boto:Params: {} 2022-09-02T05:55:52.747 DEBUG:boto:Token: None 2022-09-02T05:55:52.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33 2022-09-02T05:55:52.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ikuojqvBF4VU+fzjO/6vbZ4XbM= 2022-09-02T05:55:52.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ikuojqvBF4VU+fzjO/6vbZ4XbM=', 'Content-Length': '0'} 2022-09-02T05:55:52.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002165c7b0bc9f058-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.752 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:52.752 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:52.753 DEBUG:boto:Method: GET 2022-09-02T05:55:52.754 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:52.754 DEBUG:boto:Data: 2022-09-02T05:55:52.755 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.756 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.756 DEBUG:boto:Params: {} 2022-09-02T05:55:52.757 DEBUG:boto:Token: None 2022-09-02T05:55:52.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj33 2022-09-02T05:55:52.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ikuojqvBF4VU+fzjO/6vbZ4XbM= 2022-09-02T05:55:52.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ikuojqvBF4VU+fzjO/6vbZ4XbM=', 'Content-Length': '0'} 2022-09-02T05:55:52.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb78947f84db8c8d-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.762 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.763 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.763 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.764 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.764 DEBUG:boto:Method: GET 2022-09-02T05:55:52.765 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.765 DEBUG:boto:Data: 2022-09-02T05:55:52.766 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.767 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.767 DEBUG:boto:Params: {} 2022-09-02T05:55:52.768 DEBUG:boto:Token: None 2022-09-02T05:55:52.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dca12a443f453ba-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.773 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.774 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.775 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.775 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.776 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.777 DEBUG:boto:Method: GET 2022-09-02T05:55:52.777 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:52.778 DEBUG:boto:Data: 2022-09-02T05:55:52.778 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.779 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.780 DEBUG:boto:Params: {} 2022-09-02T05:55:52.780 DEBUG:boto:Token: None 2022-09-02T05:55:52.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016aaa6166051dee7-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.789 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:52.791 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.792 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.792 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.793 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.794 DEBUG:boto:Method: GET 2022-09-02T05:55:52.795 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.796 DEBUG:boto:Data: 2022-09-02T05:55:52.797 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.799 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.800 DEBUG:boto:Params: {} 2022-09-02T05:55:52.801 DEBUG:boto:Token: None 2022-09-02T05:55:52.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s= 2022-09-02T05:55:52.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s=', 'Content-Length': '0'} 2022-09-02T05:55:52.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ad781d929c8c68-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.808 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.809 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.810 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.811 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.812 DEBUG:boto:Method: GET 2022-09-02T05:55:52.813 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.814 DEBUG:boto:Data: 2022-09-02T05:55:52.815 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.817 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.818 DEBUG:boto:Params: {} 2022-09-02T05:55:52.819 DEBUG:boto:Token: None 2022-09-02T05:55:52.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s= 2022-09-02T05:55:52.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s=', 'Content-Length': '0'} 2022-09-02T05:55:52.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ade532388d19d28-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.828 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.829 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.830 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.830 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.831 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.832 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.833 DEBUG:boto:Data: 2022-09-02T05:55:52.834 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.836 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.837 DEBUG:boto:Params: {} 2022-09-02T05:55:52.838 DEBUG:boto:Token: None 2022-09-02T05:55:52.839 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nh/2kLOjBAbfhGtQ1jIGne5ElRU= 2022-09-02T05:55:52.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nh/2kLOjBAbfhGtQ1jIGne5ElRU=', 'Content-Length': '0'} 2022-09-02T05:55:52.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a2f1902cba9464-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.845 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.846 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.846 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.847 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.847 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.848 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.848 DEBUG:boto:Data: 2022-09-02T05:55:52.849 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.850 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.850 DEBUG:boto:Params: {} 2022-09-02T05:55:52.851 DEBUG:boto:Token: None 2022-09-02T05:55:52.852 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nh/2kLOjBAbfhGtQ1jIGne5ElRU= 2022-09-02T05:55:52.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nh/2kLOjBAbfhGtQ1jIGne5ElRU=', 'Content-Length': '0'} 2022-09-02T05:55:52.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99e8ab908d4342f-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:52.857 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.858 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.859 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.860 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.861 DEBUG:boto:Method: GET 2022-09-02T05:55:52.862 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.863 DEBUG:boto:Data: 2022-09-02T05:55:52.863 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.864 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.865 DEBUG:boto:Params: {} 2022-09-02T05:55:52.865 DEBUG:boto:Token: None 2022-09-02T05:55:52.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s= 2022-09-02T05:55:52.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s=', 'Content-Length': '0'} 2022-09-02T05:55:52.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea80339ce37c050e-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.910 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.911 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.912 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.913 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.914 DEBUG:boto:Method: GET 2022-09-02T05:55:52.915 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.916 DEBUG:boto:Data: 2022-09-02T05:55:52.917 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.919 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.920 DEBUG:boto:Params: {} 2022-09-02T05:55:52.921 DEBUG:boto:Token: None 2022-09-02T05:55:52.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:52.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s= 2022-09-02T05:55:52.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:olhe15FO3L2L09PWTCizgsbGa2s=', 'Content-Length': '0'} 2022-09-02T05:55:52.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cea87b75f6b1fa-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.929 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.930 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.931 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:52.932 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:52.933 DEBUG:boto:Method: GET 2022-09-02T05:55:52.934 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:52.935 DEBUG:boto:Data: 2022-09-02T05:55:52.936 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.938 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.939 DEBUG:boto:Params: {} 2022-09-02T05:55:52.940 DEBUG:boto:Token: None 2022-09-02T05:55:52.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ec865a4d202d34-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.948 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.949 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:52.949 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:52.950 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:52.950 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:52.951 DEBUG:boto:Method: GET 2022-09-02T05:55:52.952 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:52.952 DEBUG:boto:Data: 2022-09-02T05:55:52.953 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.954 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.954 DEBUG:boto:Params: {} 2022-09-02T05:55:52.955 DEBUG:boto:Token: None 2022-09-02T05:55:52.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/ 2022-09-02T05:55:52.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo= 2022-09-02T05:55:52.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZUK3SKP7imtkZwL/5AdTpjzGlWo=', 'Content-Length': '0'} 2022-09-02T05:55:52.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085964fe90859a8eb-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.961 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:52.962 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.963 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.963 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.964 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:52.964 DEBUG:boto:Data: 2022-09-02T05:55:52.965 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.966 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.966 DEBUG:boto:Params: {} 2022-09-02T05:55:52.967 DEBUG:boto:Token: None 2022-09-02T05:55:52.968 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34 2022-09-02T05:55:52.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UkRzwEOfYjwg2E9G+Rmo07ejDCY= 2022-09-02T05:55:52.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UkRzwEOfYjwg2E9G+Rmo07ejDCY=', 'Content-Length': '0'} 2022-09-02T05:55:52.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74088815e8537a0-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.972 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.973 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.973 DEBUG:boto:Method: HEAD 2022-09-02T05:55:52.974 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:52.974 DEBUG:boto:Data: 2022-09-02T05:55:52.975 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.976 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.976 DEBUG:boto:Params: {} 2022-09-02T05:55:52.977 DEBUG:boto:Token: None 2022-09-02T05:55:52.978 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34 2022-09-02T05:55:52.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UkRzwEOfYjwg2E9G+Rmo07ejDCY= 2022-09-02T05:55:52.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UkRzwEOfYjwg2E9G+Rmo07ejDCY=', 'Content-Length': '0'} 2022-09-02T05:55:52.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d32a4df1b1853a-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:52.982 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.983 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.983 DEBUG:boto:Method: GET 2022-09-02T05:55:52.984 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:52.984 DEBUG:boto:Data: 2022-09-02T05:55:52.985 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:52.986 DEBUG:boto:Port: 8001 2022-09-02T05:55:52.986 DEBUG:boto:Params: {} 2022-09-02T05:55:52.987 DEBUG:boto:Token: None 2022-09-02T05:55:52.988 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34 2022-09-02T05:55:52.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/r1xZ9gT1QYB/Uvo98Xh2GI5jsU= 2022-09-02T05:55:52.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/r1xZ9gT1QYB/Uvo98Xh2GI5jsU=', 'Content-Length': '0'} 2022-09-02T05:55:52.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31efa1a40f2f225-0063119ae8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:52.992 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:52.992 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:52.993 DEBUG:boto:Method: GET 2022-09-02T05:55:52.993 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:55:52.994 DEBUG:boto:Data: 2022-09-02T05:55:52.994 DEBUG:boto:Headers: {} 2022-09-02T05:55:52.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:52.995 DEBUG:boto:Port: 8000 2022-09-02T05:55:52.996 DEBUG:boto:Params: {} 2022-09-02T05:55:52.997 DEBUG:boto:Token: None 2022-09-02T05:55:52.997 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:52 GMT /xljbkf-74/obj34 2022-09-02T05:55:52.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/r1xZ9gT1QYB/Uvo98Xh2GI5jsU= 2022-09-02T05:55:52.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/r1xZ9gT1QYB/Uvo98Xh2GI5jsU=', 'Content-Length': '0'} 2022-09-02T05:55:53.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae748ca211aa4cf6-0063119ae8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.046 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.047 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.048 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:53.049 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:53.050 DEBUG:boto:Method: GET 2022-09-02T05:55:53.051 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:53.052 DEBUG:boto:Data: 2022-09-02T05:55:53.053 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.055 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.056 DEBUG:boto:Params: {} 2022-09-02T05:55:53.057 DEBUG:boto:Token: None 2022-09-02T05:55:53.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e43e747befb75fa-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.063 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.064 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.064 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.065 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:53.065 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:53.066 DEBUG:boto:Method: GET 2022-09-02T05:55:53.066 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:53.067 DEBUG:boto:Data: 2022-09-02T05:55:53.067 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.068 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.069 DEBUG:boto:Params: {} 2022-09-02T05:55:53.070 DEBUG:boto:Token: None 2022-09-02T05:55:53.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2d0c731e64ba398-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.076 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:55:53.078 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.079 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.079 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.080 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.080 DEBUG:boto:Method: GET 2022-09-02T05:55:53.081 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.081 DEBUG:boto:Data: 2022-09-02T05:55:53.082 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.083 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.084 DEBUG:boto:Params: {} 2022-09-02T05:55:53.084 DEBUG:boto:Token: None 2022-09-02T05:55:53.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M= 2022-09-02T05:55:53.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M=', 'Content-Length': '0'} 2022-09-02T05:55:53.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008018eb8a47be02d3-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.134 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.135 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.136 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.136 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.137 DEBUG:boto:Method: GET 2022-09-02T05:55:53.138 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.139 DEBUG:boto:Data: 2022-09-02T05:55:53.139 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.140 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.141 DEBUG:boto:Params: {} 2022-09-02T05:55:53.142 DEBUG:boto:Token: None 2022-09-02T05:55:53.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M= 2022-09-02T05:55:53.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M=', 'Content-Length': '0'} 2022-09-02T05:55:53.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5566cf904fb964-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.148 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.149 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.149 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.150 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.151 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.152 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.153 DEBUG:boto:Data: 2022-09-02T05:55:53.154 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.156 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.157 DEBUG:boto:Params: {} 2022-09-02T05:55:53.158 DEBUG:boto:Token: None 2022-09-02T05:55:53.159 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ybMwri9H84THyVBBWNSTD7dyoY= 2022-09-02T05:55:53.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ybMwri9H84THyVBBWNSTD7dyoY=', 'Content-Length': '0'} 2022-09-02T05:55:53.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000621fbd219af67d13-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.165 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.166 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.167 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.168 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.169 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.170 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.171 DEBUG:boto:Data: 2022-09-02T05:55:53.172 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.174 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.175 DEBUG:boto:Params: {} 2022-09-02T05:55:53.176 DEBUG:boto:Token: None 2022-09-02T05:55:53.177 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/ybMwri9H84THyVBBWNSTD7dyoY= 2022-09-02T05:55:53.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/ybMwri9H84THyVBBWNSTD7dyoY=', 'Content-Length': '0'} 2022-09-02T05:55:53.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dda0457b13fb04d-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:55:53.184 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.185 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.186 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.187 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.188 DEBUG:boto:Method: GET 2022-09-02T05:55:53.189 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.190 DEBUG:boto:Data: 2022-09-02T05:55:53.191 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.192 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.192 DEBUG:boto:Params: {} 2022-09-02T05:55:53.193 DEBUG:boto:Token: None 2022-09-02T05:55:53.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M= 2022-09-02T05:55:53.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M=', 'Content-Length': '0'} 2022-09-02T05:55:53.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073dfbe327d029132-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.198 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.198 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.199 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.199 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.200 DEBUG:boto:Method: GET 2022-09-02T05:55:53.200 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.201 DEBUG:boto:Data: 2022-09-02T05:55:53.201 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.203 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.203 DEBUG:boto:Params: {} 2022-09-02T05:55:53.204 DEBUG:boto:Token: None 2022-09-02T05:55:53.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:55:53.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M= 2022-09-02T05:55:53.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zzkuVhzWxsnSxds5yIZ1WSfzG6M=', 'Content-Length': '0'} 2022-09-02T05:55:53.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aac76a3b690f6e3-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.209 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.209 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.210 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.210 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.211 DEBUG:boto:Method: GET 2022-09-02T05:55:53.211 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.212 DEBUG:boto:Data: 2022-09-02T05:55:53.213 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.214 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.214 DEBUG:boto:Params: {} 2022-09-02T05:55:53.215 DEBUG:boto:Token: None 2022-09-02T05:55:53.215 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a884139a4c4481a-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.221 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.221 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.222 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.223 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.223 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.224 DEBUG:boto:Method: GET 2022-09-02T05:55:53.224 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.225 DEBUG:boto:Data: 2022-09-02T05:55:53.226 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.227 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.228 DEBUG:boto:Params: {} 2022-09-02T05:55:53.229 DEBUG:boto:Token: None 2022-09-02T05:55:53.229 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a96e45a2679110e-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.236 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.237 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.238 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.239 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.240 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:53.241 DEBUG:boto:Data: 2022-09-02T05:55:53.242 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.243 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.244 DEBUG:boto:Params: {} 2022-09-02T05:55:53.245 DEBUG:boto:Token: None 2022-09-02T05:55:53.246 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38 2022-09-02T05:55:53.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t63/OEkj3fQCsLElxlV7cYlfwWo= 2022-09-02T05:55:53.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t63/OEkj3fQCsLElxlV7cYlfwWo=', 'Content-Length': '0'} 2022-09-02T05:55:53.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c4d13941f6a82a-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.250 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.251 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.252 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.252 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:53.253 DEBUG:boto:Data: 2022-09-02T05:55:53.253 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.254 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.255 DEBUG:boto:Params: {} 2022-09-02T05:55:53.255 DEBUG:boto:Token: None 2022-09-02T05:55:53.256 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38 2022-09-02T05:55:53.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t63/OEkj3fQCsLElxlV7cYlfwWo= 2022-09-02T05:55:53.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t63/OEkj3fQCsLElxlV7cYlfwWo=', 'Content-Length': '0'} 2022-09-02T05:55:53.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf888a24bc2baf59-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:55:53.261 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.262 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.263 DEBUG:boto:Method: GET 2022-09-02T05:55:53.264 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:53.264 DEBUG:boto:Data: 2022-09-02T05:55:53.265 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.266 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.267 DEBUG:boto:Params: {} 2022-09-02T05:55:53.267 DEBUG:boto:Token: None 2022-09-02T05:55:53.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38 2022-09-02T05:55:53.269 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZQBcSEOJ/WUS9Srj2umdqI8w89I= 2022-09-02T05:55:53.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZQBcSEOJ/WUS9Srj2umdqI8w89I=', 'Content-Length': '0'} 2022-09-02T05:55:53.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b695d5667385eb9a-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.273 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:55:53.273 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:55:53.274 DEBUG:boto:Method: GET 2022-09-02T05:55:53.275 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:55:53.275 DEBUG:boto:Data: 2022-09-02T05:55:53.276 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.277 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.278 DEBUG:boto:Params: {} 2022-09-02T05:55:53.278 DEBUG:boto:Token: None 2022-09-02T05:55:53.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj38 2022-09-02T05:55:53.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZQBcSEOJ/WUS9Srj2umdqI8w89I= 2022-09-02T05:55:53.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZQBcSEOJ/WUS9Srj2umdqI8w89I=', 'Content-Length': '0'} 2022-09-02T05:55:53.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067351d8d837ba582-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.330 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.331 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.331 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.332 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.332 DEBUG:boto:Method: GET 2022-09-02T05:55:53.333 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.334 DEBUG:boto:Data: 2022-09-02T05:55:53.334 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.335 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.336 DEBUG:boto:Params: {} 2022-09-02T05:55:53.336 DEBUG:boto:Token: None 2022-09-02T05:55:53.337 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94869023cadbf87-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.343 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.344 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.345 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.346 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.346 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.347 DEBUG:boto:Method: GET 2022-09-02T05:55:53.348 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:55:53.348 DEBUG:boto:Data: 2022-09-02T05:55:53.349 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.351 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.351 DEBUG:boto:Params: {} 2022-09-02T05:55:53.352 DEBUG:boto:Token: None 2022-09-02T05:55:53.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0ca1a2a860413c-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.359 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:55:53.360 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.361 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.362 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.363 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.364 DEBUG:boto:Method: GET 2022-09-02T05:55:53.365 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.366 DEBUG:boto:Data: 2022-09-02T05:55:53.367 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.369 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.370 DEBUG:boto:Params: {} 2022-09-02T05:55:53.371 DEBUG:boto:Token: None 2022-09-02T05:55:53.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg= 2022-09-02T05:55:53.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg=', 'Content-Length': '0'} 2022-09-02T05:55:53.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628b1ca6abe128f3-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.378 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.379 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.380 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.381 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.382 DEBUG:boto:Method: GET 2022-09-02T05:55:53.383 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.384 DEBUG:boto:Data: 2022-09-02T05:55:53.385 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.387 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.388 DEBUG:boto:Params: {} 2022-09-02T05:55:53.389 DEBUG:boto:Token: None 2022-09-02T05:55:53.391 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg= 2022-09-02T05:55:53.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg=', 'Content-Length': '0'} 2022-09-02T05:55:53.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000462ef67251b0e189-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.397 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.398 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.398 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.399 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.399 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.400 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.400 DEBUG:boto:Data: 2022-09-02T05:55:53.401 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.402 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.403 DEBUG:boto:Params: {} 2022-09-02T05:55:53.403 DEBUG:boto:Token: None 2022-09-02T05:55:53.404 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QWtjADk29/rHndILQ79P73QQJ9s= 2022-09-02T05:55:53.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QWtjADk29/rHndILQ79P73QQJ9s=', 'Content-Length': '0'} 2022-09-02T05:55:53.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bb3176306bdbfdc-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.408 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.409 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.409 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.410 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.410 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.411 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.411 DEBUG:boto:Data: 2022-09-02T05:55:53.412 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.413 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.414 DEBUG:boto:Params: {} 2022-09-02T05:55:53.414 DEBUG:boto:Token: None 2022-09-02T05:55:53.415 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QWtjADk29/rHndILQ79P73QQJ9s= 2022-09-02T05:55:53.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QWtjADk29/rHndILQ79P73QQJ9s=', 'Content-Length': '0'} 2022-09-02T05:55:53.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd8dd2699dc4922-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:55:53.420 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.420 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.421 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.421 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.422 DEBUG:boto:Method: GET 2022-09-02T05:55:53.423 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.424 DEBUG:boto:Data: 2022-09-02T05:55:53.424 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.425 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.426 DEBUG:boto:Params: {} 2022-09-02T05:55:53.427 DEBUG:boto:Token: None 2022-09-02T05:55:53.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg= 2022-09-02T05:55:53.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg=', 'Content-Length': '0'} 2022-09-02T05:55:53.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dfeb24f40defc5e-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.432 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.432 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.433 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.434 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.435 DEBUG:boto:Method: GET 2022-09-02T05:55:53.435 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.436 DEBUG:boto:Data: 2022-09-02T05:55:53.436 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.437 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.438 DEBUG:boto:Params: {} 2022-09-02T05:55:53.439 DEBUG:boto:Token: None 2022-09-02T05:55:53.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:55:53.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg= 2022-09-02T05:55:53.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n7jzpU1xqno+reJeGq+9HnnPmrg=', 'Content-Length': '0'} 2022-09-02T05:55:53.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0210e7bd3fb6c4a-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.490 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.491 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.491 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.492 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.492 DEBUG:boto:Method: GET 2022-09-02T05:55:53.493 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.494 DEBUG:boto:Data: 2022-09-02T05:55:53.495 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.496 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.496 DEBUG:boto:Params: {} 2022-09-02T05:55:53.497 DEBUG:boto:Token: None 2022-09-02T05:55:53.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16e346f0eb3ae74-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.502 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.503 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.504 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.505 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.505 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.506 DEBUG:boto:Method: GET 2022-09-02T05:55:53.506 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.507 DEBUG:boto:Data: 2022-09-02T05:55:53.508 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.509 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.509 DEBUG:boto:Params: {} 2022-09-02T05:55:53.510 DEBUG:boto:Token: None 2022-09-02T05:55:53.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4a280c35ef0b4a-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.516 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.517 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.518 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.518 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.519 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:53.519 DEBUG:boto:Data: 2022-09-02T05:55:53.520 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.521 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.521 DEBUG:boto:Params: {} 2022-09-02T05:55:53.522 DEBUG:boto:Token: None 2022-09-02T05:55:53.522 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44 2022-09-02T05:55:53.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Uzx6vHf/8YydJiAfNi/GsFCrtrA= 2022-09-02T05:55:53.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Uzx6vHf/8YydJiAfNi/GsFCrtrA=', 'Content-Length': '0'} 2022-09-02T05:55:53.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a43519b2a399d0b-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.527 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.528 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.528 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.530 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:53.530 DEBUG:boto:Data: 2022-09-02T05:55:53.531 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.532 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.533 DEBUG:boto:Params: {} 2022-09-02T05:55:53.534 DEBUG:boto:Token: None 2022-09-02T05:55:53.535 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44 2022-09-02T05:55:53.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Uzx6vHf/8YydJiAfNi/GsFCrtrA= 2022-09-02T05:55:53.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Uzx6vHf/8YydJiAfNi/GsFCrtrA=', 'Content-Length': '0'} 2022-09-02T05:55:53.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b418d41f43bc3587-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:55:53.540 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.540 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.541 DEBUG:boto:Method: GET 2022-09-02T05:55:53.542 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:53.542 DEBUG:boto:Data: 2022-09-02T05:55:53.543 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.545 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.545 DEBUG:boto:Params: {} 2022-09-02T05:55:53.546 DEBUG:boto:Token: None 2022-09-02T05:55:53.547 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44 2022-09-02T05:55:53.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jOzNMyTQ50VR0CjEWkOKwG6dloc= 2022-09-02T05:55:53.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jOzNMyTQ50VR0CjEWkOKwG6dloc=', 'Content-Length': '0'} 2022-09-02T05:55:53.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009187c6c80bfa2ea1-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.552 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:55:53.553 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:55:53.553 DEBUG:boto:Method: GET 2022-09-02T05:55:53.554 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:55:53.555 DEBUG:boto:Data: 2022-09-02T05:55:53.555 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.557 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.558 DEBUG:boto:Params: {} 2022-09-02T05:55:53.559 DEBUG:boto:Token: None 2022-09-02T05:55:53.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj44 2022-09-02T05:55:53.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jOzNMyTQ50VR0CjEWkOKwG6dloc= 2022-09-02T05:55:53.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jOzNMyTQ50VR0CjEWkOKwG6dloc=', 'Content-Length': '0'} 2022-09-02T05:55:53.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d41c62e5d74404-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.610 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.612 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.613 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.614 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.614 DEBUG:boto:Method: GET 2022-09-02T05:55:53.615 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.616 DEBUG:boto:Data: 2022-09-02T05:55:53.617 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.619 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.619 DEBUG:boto:Params: {} 2022-09-02T05:55:53.620 DEBUG:boto:Token: None 2022-09-02T05:55:53.621 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d855de2df98731b0-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.627 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.628 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.629 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.630 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.630 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.631 DEBUG:boto:Method: GET 2022-09-02T05:55:53.631 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:55:53.632 DEBUG:boto:Data: 2022-09-02T05:55:53.632 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.634 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.634 DEBUG:boto:Params: {} 2022-09-02T05:55:53.635 DEBUG:boto:Token: None 2022-09-02T05:55:53.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000526df7fd19b18147-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.643 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:55:53.644 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.645 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.646 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.647 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.648 DEBUG:boto:Method: GET 2022-09-02T05:55:53.648 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.649 DEBUG:boto:Data: 2022-09-02T05:55:53.650 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.652 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.653 DEBUG:boto:Params: {} 2022-09-02T05:55:53.654 DEBUG:boto:Token: None 2022-09-02T05:55:53.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s= 2022-09-02T05:55:53.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s=', 'Content-Length': '0'} 2022-09-02T05:55:53.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc048222a8255d7-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.706 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.707 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.708 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.709 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.710 DEBUG:boto:Method: GET 2022-09-02T05:55:53.711 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.712 DEBUG:boto:Data: 2022-09-02T05:55:53.712 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.714 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.715 DEBUG:boto:Params: {} 2022-09-02T05:55:53.716 DEBUG:boto:Token: None 2022-09-02T05:55:53.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s= 2022-09-02T05:55:53.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s=', 'Content-Length': '0'} 2022-09-02T05:55:53.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009110d0c909b59571-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.721 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.722 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.723 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.724 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.725 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.726 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.726 DEBUG:boto:Data: 2022-09-02T05:55:53.727 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.728 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.729 DEBUG:boto:Params: {} 2022-09-02T05:55:53.729 DEBUG:boto:Token: None 2022-09-02T05:55:53.730 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POMNroryPtNiE/EQKGvktoIOjJE= 2022-09-02T05:55:53.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POMNroryPtNiE/EQKGvktoIOjJE=', 'Content-Length': '0'} 2022-09-02T05:55:53.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f17d0b78cd680a-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.734 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.735 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.735 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.736 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.737 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.737 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.738 DEBUG:boto:Data: 2022-09-02T05:55:53.738 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.739 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.740 DEBUG:boto:Params: {} 2022-09-02T05:55:53.741 DEBUG:boto:Token: None 2022-09-02T05:55:53.741 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:POMNroryPtNiE/EQKGvktoIOjJE= 2022-09-02T05:55:53.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:POMNroryPtNiE/EQKGvktoIOjJE=', 'Content-Length': '0'} 2022-09-02T05:55:53.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82e566c73a14af9-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:55:53.747 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.747 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.748 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.749 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.750 DEBUG:boto:Method: GET 2022-09-02T05:55:53.751 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.751 DEBUG:boto:Data: 2022-09-02T05:55:53.752 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.754 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.754 DEBUG:boto:Params: {} 2022-09-02T05:55:53.755 DEBUG:boto:Token: None 2022-09-02T05:55:53.756 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s= 2022-09-02T05:55:53.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s=', 'Content-Length': '0'} 2022-09-02T05:55:53.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8822495ee5b2666-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.761 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.762 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.763 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.764 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.765 DEBUG:boto:Method: GET 2022-09-02T05:55:53.766 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.766 DEBUG:boto:Data: 2022-09-02T05:55:53.767 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.768 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.768 DEBUG:boto:Params: {} 2022-09-02T05:55:53.769 DEBUG:boto:Token: None 2022-09-02T05:55:53.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:55:53.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s= 2022-09-02T05:55:53.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q1jMIFHgfYVuH1xf6NFw9AIu8/s=', 'Content-Length': '0'} 2022-09-02T05:55:53.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007847c5d17efb26ad-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.774 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.774 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.775 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.775 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.776 DEBUG:boto:Method: GET 2022-09-02T05:55:53.776 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.777 DEBUG:boto:Data: 2022-09-02T05:55:53.777 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.778 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.779 DEBUG:boto:Params: {} 2022-09-02T05:55:53.780 DEBUG:boto:Token: None 2022-09-02T05:55:53.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab9047aaec63f599-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.785 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.786 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.787 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.788 DEBUG:boto:Method: GET 2022-09-02T05:55:53.788 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.789 DEBUG:boto:Data: 2022-09-02T05:55:53.789 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.790 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.791 DEBUG:boto:Params: {} 2022-09-02T05:55:53.792 DEBUG:boto:Token: None 2022-09-02T05:55:53.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d415feae8feb00-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.798 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.799 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.799 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.800 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.800 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:53.801 DEBUG:boto:Data: 2022-09-02T05:55:53.801 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.802 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.803 DEBUG:boto:Params: {} 2022-09-02T05:55:53.803 DEBUG:boto:Token: None 2022-09-02T05:55:53.804 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46 2022-09-02T05:55:53.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X7Tv8ovo8ubTX1OlFjGqc+IZA5g= 2022-09-02T05:55:53.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X7Tv8ovo8ubTX1OlFjGqc+IZA5g=', 'Content-Length': '0'} 2022-09-02T05:55:53.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5c4c446bb6d3c5-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.808 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.809 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.809 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.810 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:53.810 DEBUG:boto:Data: 2022-09-02T05:55:53.811 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.812 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.812 DEBUG:boto:Params: {} 2022-09-02T05:55:53.813 DEBUG:boto:Token: None 2022-09-02T05:55:53.813 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46 2022-09-02T05:55:53.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X7Tv8ovo8ubTX1OlFjGqc+IZA5g= 2022-09-02T05:55:53.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X7Tv8ovo8ubTX1OlFjGqc+IZA5g=', 'Content-Length': '0'} 2022-09-02T05:55:53.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae3b9264d9295dc-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:55:53.818 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.818 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.819 DEBUG:boto:Method: GET 2022-09-02T05:55:53.819 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:53.820 DEBUG:boto:Data: 2022-09-02T05:55:53.820 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.821 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.822 DEBUG:boto:Params: {} 2022-09-02T05:55:53.823 DEBUG:boto:Token: None 2022-09-02T05:55:53.823 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46 2022-09-02T05:55:53.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4l8TrSNXBQl48oJ4qlDa+atfe0A= 2022-09-02T05:55:53.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4l8TrSNXBQl48oJ4qlDa+atfe0A=', 'Content-Length': '0'} 2022-09-02T05:55:53.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8c887fd479448c-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.827 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:55:53.828 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:55:53.828 DEBUG:boto:Method: GET 2022-09-02T05:55:53.829 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:55:53.829 DEBUG:boto:Data: 2022-09-02T05:55:53.830 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.831 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.831 DEBUG:boto:Params: {} 2022-09-02T05:55:53.832 DEBUG:boto:Token: None 2022-09-02T05:55:53.832 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj46 2022-09-02T05:55:53.833 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4l8TrSNXBQl48oJ4qlDa+atfe0A= 2022-09-02T05:55:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4l8TrSNXBQl48oJ4qlDa+atfe0A=', 'Content-Length': '0'} 2022-09-02T05:55:53.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32ea9384c17eae1-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.837 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.837 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.838 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.838 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.839 DEBUG:boto:Method: GET 2022-09-02T05:55:53.839 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.840 DEBUG:boto:Data: 2022-09-02T05:55:53.840 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.841 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.842 DEBUG:boto:Params: {} 2022-09-02T05:55:53.842 DEBUG:boto:Token: None 2022-09-02T05:55:53.843 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.844 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027d94f38385b3a5b-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.848 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.849 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.850 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.850 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.851 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.851 DEBUG:boto:Method: GET 2022-09-02T05:55:53.852 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:55:53.853 DEBUG:boto:Data: 2022-09-02T05:55:53.853 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.854 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.855 DEBUG:boto:Params: {} 2022-09-02T05:55:53.855 DEBUG:boto:Token: None 2022-09-02T05:55:53.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:53.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:53.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58e6026d65e87b1-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.862 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:53.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:55:53.863 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:53.864 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:53.864 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.865 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.865 DEBUG:boto:Method: GET 2022-09-02T05:55:53.866 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.867 DEBUG:boto:Data: 2022-09-02T05:55:53.867 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.868 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.869 DEBUG:boto:Params: {} 2022-09-02T05:55:53.870 DEBUG:boto:Token: None 2022-09-02T05:55:53.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw= 2022-09-02T05:55:53.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw=', 'Content-Length': '0'} 2022-09-02T05:55:53.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b383c3e251564c4c-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.914 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:53.915 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:53.915 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.916 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.916 DEBUG:boto:Method: GET 2022-09-02T05:55:53.917 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.918 DEBUG:boto:Data: 2022-09-02T05:55:53.918 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.919 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.920 DEBUG:boto:Params: {} 2022-09-02T05:55:53.921 DEBUG:boto:Token: None 2022-09-02T05:55:53.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw= 2022-09-02T05:55:53.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw=', 'Content-Length': '0'} 2022-09-02T05:55:53.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002366df285feea12b-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.926 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:53.926 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:53.927 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.927 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.928 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.929 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.929 DEBUG:boto:Data: 2022-09-02T05:55:53.930 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.931 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.931 DEBUG:boto:Params: {} 2022-09-02T05:55:53.932 DEBUG:boto:Token: None 2022-09-02T05:55:53.933 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t1PMIfIgXtYm72X68QmCKEUOkQI= 2022-09-02T05:55:53.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t1PMIfIgXtYm72X68QmCKEUOkQI=', 'Content-Length': '0'} 2022-09-02T05:55:53.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32ef7ed74758bd9-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.944 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:53.944 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:53.945 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.946 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.947 DEBUG:boto:Method: HEAD 2022-09-02T05:55:53.948 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.948 DEBUG:boto:Data: 2022-09-02T05:55:53.949 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.951 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.951 DEBUG:boto:Params: {} 2022-09-02T05:55:53.952 DEBUG:boto:Token: None 2022-09-02T05:55:53.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t1PMIfIgXtYm72X68QmCKEUOkQI= 2022-09-02T05:55:53.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t1PMIfIgXtYm72X68QmCKEUOkQI=', 'Content-Length': '0'} 2022-09-02T05:55:53.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0f9ed3c4cd8676-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:55:53.958 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:53.959 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:53.960 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.960 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.961 DEBUG:boto:Method: GET 2022-09-02T05:55:53.961 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.962 DEBUG:boto:Data: 2022-09-02T05:55:53.963 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.964 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.964 DEBUG:boto:Params: {} 2022-09-02T05:55:53.965 DEBUG:boto:Token: None 2022-09-02T05:55:53.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw= 2022-09-02T05:55:53.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw=', 'Content-Length': '0'} 2022-09-02T05:55:53.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f61f2bcc186fdb8f-0063119ae9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.971 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:53.971 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:53.972 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.973 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.973 DEBUG:boto:Method: GET 2022-09-02T05:55:53.974 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.974 DEBUG:boto:Data: 2022-09-02T05:55:53.975 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:53.976 DEBUG:boto:Port: 8000 2022-09-02T05:55:53.976 DEBUG:boto:Params: {} 2022-09-02T05:55:53.978 DEBUG:boto:Token: None 2022-09-02T05:55:53.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:55:53.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw= 2022-09-02T05:55:53.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XElBJpOeH2l9AYOdq84TtsgpDKw=', 'Content-Length': '0'} 2022-09-02T05:55:53.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005bb7eb99a0bd5a6-0063119ae9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:53.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:53.987 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:53.988 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:53.989 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:53.990 DEBUG:boto:Method: GET 2022-09-02T05:55:53.991 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:53.992 DEBUG:boto:Data: 2022-09-02T05:55:53.993 DEBUG:boto:Headers: {} 2022-09-02T05:55:53.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:53.995 DEBUG:boto:Port: 8001 2022-09-02T05:55:53.996 DEBUG:boto:Params: {} 2022-09-02T05:55:53.997 DEBUG:boto:Token: None 2022-09-02T05:55:53.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:53 GMT /xljbkf-74/ 2022-09-02T05:55:53.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk= 2022-09-02T05:55:54.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jvi7vt3vDETGfnPpu4XIuhtRSZk=', 'Content-Length': '0'} 2022-09-02T05:55:54.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b6115e4f86c900-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.005 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.006 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.007 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.008 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.009 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.010 DEBUG:boto:Method: GET 2022-09-02T05:55:54.011 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.012 DEBUG:boto:Data: 2022-09-02T05:55:54.013 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.014 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.015 DEBUG:boto:Params: {} 2022-09-02T05:55:54.015 DEBUG:boto:Token: None 2022-09-02T05:55:54.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006695975133ade1e1-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.023 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.024 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:54.025 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:54.026 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.027 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:54.028 DEBUG:boto:Data: 2022-09-02T05:55:54.028 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.030 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.031 DEBUG:boto:Params: {} 2022-09-02T05:55:54.032 DEBUG:boto:Token: None 2022-09-02T05:55:54.032 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj51 2022-09-02T05:55:54.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oMA+Oyt/7GxgTK6ofyMyoOqazXA= 2022-09-02T05:55:54.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oMA+Oyt/7GxgTK6ofyMyoOqazXA=', 'Content-Length': '0'} 2022-09-02T05:55:54.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13aed49c278ece8-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.038 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:54.039 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:54.040 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.041 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:54.042 DEBUG:boto:Data: 2022-09-02T05:55:54.042 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.043 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.044 DEBUG:boto:Params: {} 2022-09-02T05:55:54.045 DEBUG:boto:Token: None 2022-09-02T05:55:54.045 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj51 2022-09-02T05:55:54.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oMA+Oyt/7GxgTK6ofyMyoOqazXA= 2022-09-02T05:55:54.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oMA+Oyt/7GxgTK6ofyMyoOqazXA=', 'Content-Length': '0'} 2022-09-02T05:55:54.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366a628e2b3023ce-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:55:54.051 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:54.052 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:54.052 DEBUG:boto:Method: GET 2022-09-02T05:55:54.053 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:54.054 DEBUG:boto:Data: 2022-09-02T05:55:54.054 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.055 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.056 DEBUG:boto:Params: {} 2022-09-02T05:55:54.057 DEBUG:boto:Token: None 2022-09-02T05:55:54.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj51 2022-09-02T05:55:54.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iwVctbHDBTP0ogzFtUTPfmFDCnQ= 2022-09-02T05:55:54.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iwVctbHDBTP0ogzFtUTPfmFDCnQ=', 'Content-Length': '0'} 2022-09-02T05:55:54.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001caf7f3298d23cd1-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.106 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:55:54.107 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:55:54.107 DEBUG:boto:Method: GET 2022-09-02T05:55:54.108 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:55:54.109 DEBUG:boto:Data: 2022-09-02T05:55:54.109 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.110 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.111 DEBUG:boto:Params: {} 2022-09-02T05:55:54.112 DEBUG:boto:Token: None 2022-09-02T05:55:54.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj51 2022-09-02T05:55:54.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iwVctbHDBTP0ogzFtUTPfmFDCnQ= 2022-09-02T05:55:54.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iwVctbHDBTP0ogzFtUTPfmFDCnQ=', 'Content-Length': '0'} 2022-09-02T05:55:54.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae338470b7af6c0-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.118 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.119 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.119 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.120 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.120 DEBUG:boto:Method: GET 2022-09-02T05:55:54.121 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.122 DEBUG:boto:Data: 2022-09-02T05:55:54.122 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.123 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.124 DEBUG:boto:Params: {} 2022-09-02T05:55:54.125 DEBUG:boto:Token: None 2022-09-02T05:55:54.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033303996f969a273-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.130 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.131 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.131 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.132 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.133 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.134 DEBUG:boto:Method: GET 2022-09-02T05:55:54.135 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:55:54.135 DEBUG:boto:Data: 2022-09-02T05:55:54.136 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.138 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.139 DEBUG:boto:Params: {} 2022-09-02T05:55:54.140 DEBUG:boto:Token: None 2022-09-02T05:55:54.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f4fe4322edf98ee-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.147 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:55:54.148 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.149 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.150 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.150 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.151 DEBUG:boto:Method: GET 2022-09-02T05:55:54.151 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.152 DEBUG:boto:Data: 2022-09-02T05:55:54.152 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.153 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.154 DEBUG:boto:Params: {} 2022-09-02T05:55:54.155 DEBUG:boto:Token: None 2022-09-02T05:55:54.155 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc= 2022-09-02T05:55:54.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc=', 'Content-Length': '0'} 2022-09-02T05:55:54.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e8bbe9d36f0c78b-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.160 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.160 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.161 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.161 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.162 DEBUG:boto:Method: GET 2022-09-02T05:55:54.162 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.163 DEBUG:boto:Data: 2022-09-02T05:55:54.163 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.164 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.165 DEBUG:boto:Params: {} 2022-09-02T05:55:54.166 DEBUG:boto:Token: None 2022-09-02T05:55:54.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc= 2022-09-02T05:55:54.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc=', 'Content-Length': '0'} 2022-09-02T05:55:54.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc490c010e77b0d-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.171 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.172 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.172 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.173 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.173 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.174 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.174 DEBUG:boto:Data: 2022-09-02T05:55:54.175 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.176 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.176 DEBUG:boto:Params: {} 2022-09-02T05:55:54.177 DEBUG:boto:Token: None 2022-09-02T05:55:54.178 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.179 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sXXZ8ehWFC8TKSkrz9inrzpXVxs= 2022-09-02T05:55:54.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sXXZ8ehWFC8TKSkrz9inrzpXVxs=', 'Content-Length': '0'} 2022-09-02T05:55:54.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84077069f8e7bbe-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.182 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.183 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.183 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.184 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.185 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.186 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.187 DEBUG:boto:Data: 2022-09-02T05:55:54.188 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.190 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.191 DEBUG:boto:Params: {} 2022-09-02T05:55:54.192 DEBUG:boto:Token: None 2022-09-02T05:55:54.193 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sXXZ8ehWFC8TKSkrz9inrzpXVxs= 2022-09-02T05:55:54.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sXXZ8ehWFC8TKSkrz9inrzpXVxs=', 'Content-Length': '0'} 2022-09-02T05:55:54.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a0fdaf565b941f-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:55:54.200 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.201 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.202 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.203 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.204 DEBUG:boto:Method: GET 2022-09-02T05:55:54.205 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.206 DEBUG:boto:Data: 2022-09-02T05:55:54.206 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.207 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.208 DEBUG:boto:Params: {} 2022-09-02T05:55:54.209 DEBUG:boto:Token: None 2022-09-02T05:55:54.211 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc= 2022-09-02T05:55:54.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc=', 'Content-Length': '0'} 2022-09-02T05:55:54.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096981604db3d5275-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.217 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.218 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.219 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.220 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.220 DEBUG:boto:Method: GET 2022-09-02T05:55:54.221 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.221 DEBUG:boto:Data: 2022-09-02T05:55:54.222 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.223 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.224 DEBUG:boto:Params: {} 2022-09-02T05:55:54.224 DEBUG:boto:Token: None 2022-09-02T05:55:54.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:55:54.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc= 2022-09-02T05:55:54.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZnnHo/LaSECtUQqbjgISixsA4fc=', 'Content-Length': '0'} 2022-09-02T05:55:54.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af36bfec2a0bff21-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.270 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.271 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.271 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.272 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.272 DEBUG:boto:Method: GET 2022-09-02T05:55:54.273 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.273 DEBUG:boto:Data: 2022-09-02T05:55:54.274 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.275 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.276 DEBUG:boto:Params: {} 2022-09-02T05:55:54.276 DEBUG:boto:Token: None 2022-09-02T05:55:54.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a73f702bcedb5b7-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.282 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.283 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.283 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.284 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.284 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.285 DEBUG:boto:Method: GET 2022-09-02T05:55:54.286 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.287 DEBUG:boto:Data: 2022-09-02T05:55:54.287 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.289 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.290 DEBUG:boto:Params: {} 2022-09-02T05:55:54.290 DEBUG:boto:Token: None 2022-09-02T05:55:54.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e418f5d6b5723d-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.297 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.298 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.298 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.299 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.300 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:54.300 DEBUG:boto:Data: 2022-09-02T05:55:54.301 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.302 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.302 DEBUG:boto:Params: {} 2022-09-02T05:55:54.303 DEBUG:boto:Token: None 2022-09-02T05:55:54.304 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55 2022-09-02T05:55:54.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+URuHiEkTn43HAy6oxtJXNRtfHc= 2022-09-02T05:55:54.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+URuHiEkTn43HAy6oxtJXNRtfHc=', 'Content-Length': '0'} 2022-09-02T05:55:54.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353d2f52b2e7e0e6-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.308 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.308 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.309 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.310 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:54.310 DEBUG:boto:Data: 2022-09-02T05:55:54.311 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.312 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.313 DEBUG:boto:Params: {} 2022-09-02T05:55:54.313 DEBUG:boto:Token: None 2022-09-02T05:55:54.314 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55 2022-09-02T05:55:54.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+URuHiEkTn43HAy6oxtJXNRtfHc= 2022-09-02T05:55:54.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+URuHiEkTn43HAy6oxtJXNRtfHc=', 'Content-Length': '0'} 2022-09-02T05:55:54.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005426113ccdd48f8b-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:55:54.319 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.320 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.320 DEBUG:boto:Method: GET 2022-09-02T05:55:54.321 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:54.321 DEBUG:boto:Data: 2022-09-02T05:55:54.322 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.323 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.323 DEBUG:boto:Params: {} 2022-09-02T05:55:54.324 DEBUG:boto:Token: None 2022-09-02T05:55:54.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55 2022-09-02T05:55:54.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C/hK8h/bbC8VEiyhpUF9Tq2Y0v0= 2022-09-02T05:55:54.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C/hK8h/bbC8VEiyhpUF9Tq2Y0v0=', 'Content-Length': '0'} 2022-09-02T05:55:54.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088430ffbb492a882-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.329 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:55:54.330 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:55:54.330 DEBUG:boto:Method: GET 2022-09-02T05:55:54.331 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:55:54.332 DEBUG:boto:Data: 2022-09-02T05:55:54.332 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.333 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.334 DEBUG:boto:Params: {} 2022-09-02T05:55:54.334 DEBUG:boto:Token: None 2022-09-02T05:55:54.335 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj55 2022-09-02T05:55:54.336 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C/hK8h/bbC8VEiyhpUF9Tq2Y0v0= 2022-09-02T05:55:54.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C/hK8h/bbC8VEiyhpUF9Tq2Y0v0=', 'Content-Length': '0'} 2022-09-02T05:55:54.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b3f0c27a3add79-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.339 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.341 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.342 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.342 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.343 DEBUG:boto:Method: GET 2022-09-02T05:55:54.344 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.344 DEBUG:boto:Data: 2022-09-02T05:55:54.345 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.346 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.348 DEBUG:boto:Params: {} 2022-09-02T05:55:54.349 DEBUG:boto:Token: None 2022-09-02T05:55:54.350 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009180e62f0511c98-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.354 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.355 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.356 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.357 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.357 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.360 DEBUG:boto:Method: GET 2022-09-02T05:55:54.361 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:55:54.361 DEBUG:boto:Data: 2022-09-02T05:55:54.362 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.364 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.364 DEBUG:boto:Params: {} 2022-09-02T05:55:54.365 DEBUG:boto:Token: None 2022-09-02T05:55:54.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007055c5745cd76ca5-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.414 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:55:54.416 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.416 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.417 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.418 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.419 DEBUG:boto:Method: GET 2022-09-02T05:55:54.419 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.420 DEBUG:boto:Data: 2022-09-02T05:55:54.420 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.421 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.422 DEBUG:boto:Params: {} 2022-09-02T05:55:54.423 DEBUG:boto:Token: None 2022-09-02T05:55:54.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY= 2022-09-02T05:55:54.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY=', 'Content-Length': '0'} 2022-09-02T05:55:54.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056422132c61c10d1-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.428 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.429 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.429 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.430 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.430 DEBUG:boto:Method: GET 2022-09-02T05:55:54.431 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.432 DEBUG:boto:Data: 2022-09-02T05:55:54.432 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.433 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.434 DEBUG:boto:Params: {} 2022-09-02T05:55:54.435 DEBUG:boto:Token: None 2022-09-02T05:55:54.436 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY= 2022-09-02T05:55:54.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY=', 'Content-Length': '0'} 2022-09-02T05:55:54.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072eaabeecdd2be34-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.441 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.442 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.442 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.443 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.444 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.444 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.445 DEBUG:boto:Data: 2022-09-02T05:55:54.445 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.447 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.448 DEBUG:boto:Params: {} 2022-09-02T05:55:54.448 DEBUG:boto:Token: None 2022-09-02T05:55:54.449 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bi6TksIk8dxHVt0co2tga0omAr0= 2022-09-02T05:55:54.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bi6TksIk8dxHVt0co2tga0omAr0=', 'Content-Length': '0'} 2022-09-02T05:55:54.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a57df8b74cc53ad-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.454 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.455 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.455 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.456 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.457 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.457 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.458 DEBUG:boto:Data: 2022-09-02T05:55:54.458 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.460 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.460 DEBUG:boto:Params: {} 2022-09-02T05:55:54.461 DEBUG:boto:Token: None 2022-09-02T05:55:54.462 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bi6TksIk8dxHVt0co2tga0omAr0= 2022-09-02T05:55:54.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bi6TksIk8dxHVt0co2tga0omAr0=', 'Content-Length': '0'} 2022-09-02T05:55:54.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49ccdad5296b7a3-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:55:54.467 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.467 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.468 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.469 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.470 DEBUG:boto:Method: GET 2022-09-02T05:55:54.470 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.471 DEBUG:boto:Data: 2022-09-02T05:55:54.471 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.473 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.473 DEBUG:boto:Params: {} 2022-09-02T05:55:54.474 DEBUG:boto:Token: None 2022-09-02T05:55:54.474 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY= 2022-09-02T05:55:54.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY=', 'Content-Length': '0'} 2022-09-02T05:55:54.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183f109b47bab9f8-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.481 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.482 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.483 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.484 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.485 DEBUG:boto:Method: GET 2022-09-02T05:55:54.486 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.487 DEBUG:boto:Data: 2022-09-02T05:55:54.487 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.488 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.489 DEBUG:boto:Params: {} 2022-09-02T05:55:54.490 DEBUG:boto:Token: None 2022-09-02T05:55:54.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:55:54.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY= 2022-09-02T05:55:54.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d+nVD/pflMt0mK5ysogtz7ZLcPY=', 'Content-Length': '0'} 2022-09-02T05:55:54.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025cb3629e975d034-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.495 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.495 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.496 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.497 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.497 DEBUG:boto:Method: GET 2022-09-02T05:55:54.499 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.499 DEBUG:boto:Data: 2022-09-02T05:55:54.500 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.501 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.502 DEBUG:boto:Params: {} 2022-09-02T05:55:54.503 DEBUG:boto:Token: None 2022-09-02T05:55:54.504 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fed432983f4ff13-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.509 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.511 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.512 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.513 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.514 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.515 DEBUG:boto:Method: GET 2022-09-02T05:55:54.515 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.516 DEBUG:boto:Data: 2022-09-02T05:55:54.516 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.518 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.518 DEBUG:boto:Params: {} 2022-09-02T05:55:54.519 DEBUG:boto:Token: None 2022-09-02T05:55:54.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000311e6e9fc5688fee-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.527 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.529 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.530 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.531 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.532 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:54.533 DEBUG:boto:Data: 2022-09-02T05:55:54.534 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.536 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.537 DEBUG:boto:Params: {} 2022-09-02T05:55:54.538 DEBUG:boto:Token: None 2022-09-02T05:55:54.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57 2022-09-02T05:55:54.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JAwjibQDVamk1bumxz0IQHkATtU= 2022-09-02T05:55:54.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JAwjibQDVamk1bumxz0IQHkATtU=', 'Content-Length': '0'} 2022-09-02T05:55:54.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d96b137ca409ad7-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.547 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.548 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.549 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.550 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:54.551 DEBUG:boto:Data: 2022-09-02T05:55:54.552 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.553 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.553 DEBUG:boto:Params: {} 2022-09-02T05:55:54.554 DEBUG:boto:Token: None 2022-09-02T05:55:54.554 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57 2022-09-02T05:55:54.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JAwjibQDVamk1bumxz0IQHkATtU= 2022-09-02T05:55:54.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JAwjibQDVamk1bumxz0IQHkATtU=', 'Content-Length': '0'} 2022-09-02T05:55:54.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9157f8a607d7829-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:55:54.566 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.567 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.567 DEBUG:boto:Method: GET 2022-09-02T05:55:54.568 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:54.569 DEBUG:boto:Data: 2022-09-02T05:55:54.569 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.571 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.572 DEBUG:boto:Params: {} 2022-09-02T05:55:54.572 DEBUG:boto:Token: None 2022-09-02T05:55:54.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57 2022-09-02T05:55:54.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:02qKa0MtNsI/qWb3hBu7WgBT5R0= 2022-09-02T05:55:54.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:02qKa0MtNsI/qWb3hBu7WgBT5R0=', 'Content-Length': '0'} 2022-09-02T05:55:54.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec1d615d495e807-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.578 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:55:54.579 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:55:54.579 DEBUG:boto:Method: GET 2022-09-02T05:55:54.580 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:55:54.580 DEBUG:boto:Data: 2022-09-02T05:55:54.581 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.582 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.582 DEBUG:boto:Params: {} 2022-09-02T05:55:54.583 DEBUG:boto:Token: None 2022-09-02T05:55:54.584 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj57 2022-09-02T05:55:54.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:02qKa0MtNsI/qWb3hBu7WgBT5R0= 2022-09-02T05:55:54.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:02qKa0MtNsI/qWb3hBu7WgBT5R0=', 'Content-Length': '0'} 2022-09-02T05:55:54.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d16caecf286598-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.588 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.589 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.589 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.590 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.590 DEBUG:boto:Method: GET 2022-09-02T05:55:54.591 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.591 DEBUG:boto:Data: 2022-09-02T05:55:54.592 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.593 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.594 DEBUG:boto:Params: {} 2022-09-02T05:55:54.594 DEBUG:boto:Token: None 2022-09-02T05:55:54.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee639d43e24d645-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.600 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.601 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.602 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.602 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.603 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.604 DEBUG:boto:Method: GET 2022-09-02T05:55:54.604 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:55:54.605 DEBUG:boto:Data: 2022-09-02T05:55:54.605 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.607 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.607 DEBUG:boto:Params: {} 2022-09-02T05:55:54.608 DEBUG:boto:Token: None 2022-09-02T05:55:54.608 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c0aaa77f05c86d-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.616 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:55:54.617 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.618 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.619 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.619 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.620 DEBUG:boto:Method: GET 2022-09-02T05:55:54.621 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.621 DEBUG:boto:Data: 2022-09-02T05:55:54.622 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.623 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.624 DEBUG:boto:Params: {} 2022-09-02T05:55:54.625 DEBUG:boto:Token: None 2022-09-02T05:55:54.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho= 2022-09-02T05:55:54.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho=', 'Content-Length': '0'} 2022-09-02T05:55:54.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b78cb8081191a2-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.674 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.675 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.675 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.676 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.677 DEBUG:boto:Method: GET 2022-09-02T05:55:54.677 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.678 DEBUG:boto:Data: 2022-09-02T05:55:54.679 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.680 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.681 DEBUG:boto:Params: {} 2022-09-02T05:55:54.682 DEBUG:boto:Token: None 2022-09-02T05:55:54.682 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho= 2022-09-02T05:55:54.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho=', 'Content-Length': '0'} 2022-09-02T05:55:54.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9639caeeffe230-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.687 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.688 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.688 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.689 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.690 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.691 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.691 DEBUG:boto:Data: 2022-09-02T05:55:54.692 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.693 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.694 DEBUG:boto:Params: {} 2022-09-02T05:55:54.695 DEBUG:boto:Token: None 2022-09-02T05:55:54.696 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lXtHrNNwZ/JY7PCz1ctbZc2pWZU= 2022-09-02T05:55:54.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lXtHrNNwZ/JY7PCz1ctbZc2pWZU=', 'Content-Length': '0'} 2022-09-02T05:55:54.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa3a7ccf22fd903-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.701 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.702 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.703 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.703 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.704 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.704 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.705 DEBUG:boto:Data: 2022-09-02T05:55:54.705 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.706 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.707 DEBUG:boto:Params: {} 2022-09-02T05:55:54.708 DEBUG:boto:Token: None 2022-09-02T05:55:54.708 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lXtHrNNwZ/JY7PCz1ctbZc2pWZU= 2022-09-02T05:55:54.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lXtHrNNwZ/JY7PCz1ctbZc2pWZU=', 'Content-Length': '0'} 2022-09-02T05:55:54.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b2b0bda6184a75d-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:55:54.714 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.715 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.715 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.716 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.716 DEBUG:boto:Method: GET 2022-09-02T05:55:54.717 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.717 DEBUG:boto:Data: 2022-09-02T05:55:54.718 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.719 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.720 DEBUG:boto:Params: {} 2022-09-02T05:55:54.720 DEBUG:boto:Token: None 2022-09-02T05:55:54.721 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho= 2022-09-02T05:55:54.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho=', 'Content-Length': '0'} 2022-09-02T05:55:54.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a750148dd7ad1235-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.726 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.726 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.727 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.727 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.728 DEBUG:boto:Method: GET 2022-09-02T05:55:54.728 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.729 DEBUG:boto:Data: 2022-09-02T05:55:54.729 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.731 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.732 DEBUG:boto:Params: {} 2022-09-02T05:55:54.733 DEBUG:boto:Token: None 2022-09-02T05:55:54.734 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:55:54.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho= 2022-09-02T05:55:54.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ak/T8HEoEi8NNtudeHziE7zoRho=', 'Content-Length': '0'} 2022-09-02T05:55:54.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbdc90f42c188af-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.789 DEBUG:boto:Method: GET 2022-09-02T05:55:54.789 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.790 DEBUG:boto:Data: 2022-09-02T05:55:54.790 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.791 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.792 DEBUG:boto:Params: {} 2022-09-02T05:55:54.793 DEBUG:boto:Token: None 2022-09-02T05:55:54.793 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f8611f53803e53-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.798 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.799 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.800 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.800 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.801 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.802 DEBUG:boto:Method: GET 2022-09-02T05:55:54.802 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.803 DEBUG:boto:Data: 2022-09-02T05:55:54.803 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.804 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.805 DEBUG:boto:Params: {} 2022-09-02T05:55:54.806 DEBUG:boto:Token: None 2022-09-02T05:55:54.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1af67949edda5d-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.812 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.813 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.814 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.814 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.815 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:54.815 DEBUG:boto:Data: 2022-09-02T05:55:54.816 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.817 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.817 DEBUG:boto:Params: {} 2022-09-02T05:55:54.818 DEBUG:boto:Token: None 2022-09-02T05:55:54.819 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58 2022-09-02T05:55:54.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dD3sKBkHvsF1Wfvxisz3muyws7k= 2022-09-02T05:55:54.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dD3sKBkHvsF1Wfvxisz3muyws7k=', 'Content-Length': '0'} 2022-09-02T05:55:54.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a384310ce9524806-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.823 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.824 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.825 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.825 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:54.826 DEBUG:boto:Data: 2022-09-02T05:55:54.826 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.828 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.828 DEBUG:boto:Params: {} 2022-09-02T05:55:54.829 DEBUG:boto:Token: None 2022-09-02T05:55:54.830 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58 2022-09-02T05:55:54.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dD3sKBkHvsF1Wfvxisz3muyws7k= 2022-09-02T05:55:54.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dD3sKBkHvsF1Wfvxisz3muyws7k=', 'Content-Length': '0'} 2022-09-02T05:55:54.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000265f3009cd1969e6-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:55:54.835 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.835 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.836 DEBUG:boto:Method: GET 2022-09-02T05:55:54.837 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:54.838 DEBUG:boto:Data: 2022-09-02T05:55:54.839 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.841 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.842 DEBUG:boto:Params: {} 2022-09-02T05:55:54.843 DEBUG:boto:Token: None 2022-09-02T05:55:54.844 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58 2022-09-02T05:55:54.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lpuxo6zkPvsPt9DRz6OjfXQn7z0= 2022-09-02T05:55:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lpuxo6zkPvsPt9DRz6OjfXQn7z0=', 'Content-Length': '0'} 2022-09-02T05:55:54.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001428389ce8990ab5-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.848 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:55:54.849 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:55:54.849 DEBUG:boto:Method: GET 2022-09-02T05:55:54.850 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:55:54.850 DEBUG:boto:Data: 2022-09-02T05:55:54.851 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.852 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.853 DEBUG:boto:Params: {} 2022-09-02T05:55:54.853 DEBUG:boto:Token: None 2022-09-02T05:55:54.854 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj58 2022-09-02T05:55:54.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lpuxo6zkPvsPt9DRz6OjfXQn7z0= 2022-09-02T05:55:54.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lpuxo6zkPvsPt9DRz6OjfXQn7z0=', 'Content-Length': '0'} 2022-09-02T05:55:54.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dae2cf8355c896d-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.858 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.859 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.859 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.860 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.861 DEBUG:boto:Method: GET 2022-09-02T05:55:54.861 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.862 DEBUG:boto:Data: 2022-09-02T05:55:54.862 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.864 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.864 DEBUG:boto:Params: {} 2022-09-02T05:55:54.865 DEBUG:boto:Token: None 2022-09-02T05:55:54.867 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb9beb35a4966b0-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.873 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.874 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:54.875 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:54.876 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.876 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.877 DEBUG:boto:Method: GET 2022-09-02T05:55:54.877 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:55:54.878 DEBUG:boto:Data: 2022-09-02T05:55:54.878 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.879 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.880 DEBUG:boto:Params: {} 2022-09-02T05:55:54.881 DEBUG:boto:Token: None 2022-09-02T05:55:54.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/ 2022-09-02T05:55:54.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE= 2022-09-02T05:55:54.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:waRtnI/mxmblolFSaK2wn/TBkEE=', 'Content-Length': '0'} 2022-09-02T05:55:54.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8fde2d1e477d1a5-0063119aea-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.892 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:54.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:55:54.895 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:54.896 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:54.897 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.898 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.899 DEBUG:boto:Method: GET 2022-09-02T05:55:54.900 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.900 DEBUG:boto:Data: 2022-09-02T05:55:54.901 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.903 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.904 DEBUG:boto:Params: {} 2022-09-02T05:55:54.904 DEBUG:boto:Token: None 2022-09-02T05:55:54.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M024O0CjID2RRdcGDOqd9gTFsYM= 2022-09-02T05:55:54.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M024O0CjID2RRdcGDOqd9gTFsYM=', 'Content-Length': '0'} 2022-09-02T05:55:54.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24e1306fe50d91b-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.911 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:54.911 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:54.912 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.913 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.914 DEBUG:boto:Method: GET 2022-09-02T05:55:54.914 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.915 DEBUG:boto:Data: 2022-09-02T05:55:54.916 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.917 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.917 DEBUG:boto:Params: {} 2022-09-02T05:55:54.918 DEBUG:boto:Token: None 2022-09-02T05:55:54.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M024O0CjID2RRdcGDOqd9gTFsYM= 2022-09-02T05:55:54.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M024O0CjID2RRdcGDOqd9gTFsYM=', 'Content-Length': '0'} 2022-09-02T05:55:54.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d63999d76665fd2-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.966 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:54.967 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:54.967 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.968 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.968 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.969 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.969 DEBUG:boto:Data: 2022-09-02T05:55:54.970 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.971 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.971 DEBUG:boto:Params: {} 2022-09-02T05:55:54.972 DEBUG:boto:Token: None 2022-09-02T05:55:54.973 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr42MJ2zwsNavyrjN+qrwFvw0xU= 2022-09-02T05:55:54.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr42MJ2zwsNavyrjN+qrwFvw0xU=', 'Content-Length': '0'} 2022-09-02T05:55:54.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdecc1120d3f16da-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.977 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:54.977 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:54.978 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.978 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.979 DEBUG:boto:Method: HEAD 2022-09-02T05:55:54.979 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.980 DEBUG:boto:Data: 2022-09-02T05:55:54.980 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:54.981 DEBUG:boto:Port: 8000 2022-09-02T05:55:54.982 DEBUG:boto:Params: {} 2022-09-02T05:55:54.982 DEBUG:boto:Token: None 2022-09-02T05:55:54.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pr42MJ2zwsNavyrjN+qrwFvw0xU= 2022-09-02T05:55:54.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pr42MJ2zwsNavyrjN+qrwFvw0xU=', 'Content-Length': '0'} 2022-09-02T05:55:54.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee24a43b96a5b21b-0063119aea-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:55:54.987 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:54.988 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:54.988 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.989 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.989 DEBUG:boto:Method: GET 2022-09-02T05:55:54.990 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.990 DEBUG:boto:Data: 2022-09-02T05:55:54.991 DEBUG:boto:Headers: {} 2022-09-02T05:55:54.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:54.992 DEBUG:boto:Port: 8001 2022-09-02T05:55:54.993 DEBUG:boto:Params: {} 2022-09-02T05:55:54.994 DEBUG:boto:Token: None 2022-09-02T05:55:54.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:54 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:54.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M024O0CjID2RRdcGDOqd9gTFsYM= 2022-09-02T05:55:54.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M024O0CjID2RRdcGDOqd9gTFsYM=', 'Content-Length': '0'} 2022-09-02T05:55:54.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba168969530f26c-0063119aea-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:54.999 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:54.999 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:55.000 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:55.000 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:55:55.001 DEBUG:boto:Method: GET 2022-09-02T05:55:55.001 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:55.002 DEBUG:boto:Data: 2022-09-02T05:55:55.002 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.003 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.004 DEBUG:boto:Params: {} 2022-09-02T05:55:55.004 DEBUG:boto:Token: None 2022-09-02T05:55:55.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:55:55.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PjeLB14tEcC41mENbEMrbDV9dXQ= 2022-09-02T05:55:55.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PjeLB14tEcC41mENbEMrbDV9dXQ=', 'Content-Length': '0'} 2022-09-02T05:55:55.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090da2439ff9a86e3-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.010 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.010 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.011 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.011 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.012 DEBUG:boto:Method: GET 2022-09-02T05:55:55.012 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.013 DEBUG:boto:Data: 2022-09-02T05:55:55.013 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.015 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.015 DEBUG:boto:Params: {} 2022-09-02T05:55:55.016 DEBUG:boto:Token: None 2022-09-02T05:55:55.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004668ebb48af2ce7e-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.066 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.067 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.068 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.068 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.069 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.069 DEBUG:boto:Method: GET 2022-09-02T05:55:55.070 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.070 DEBUG:boto:Data: 2022-09-02T05:55:55.071 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.072 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.072 DEBUG:boto:Params: {} 2022-09-02T05:55:55.073 DEBUG:boto:Token: None 2022-09-02T05:55:55.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000396d01be6167bac4-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.080 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.081 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:55.081 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:55.082 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.082 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:55.083 DEBUG:boto:Data: 2022-09-02T05:55:55.083 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.084 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.085 DEBUG:boto:Params: {} 2022-09-02T05:55:55.085 DEBUG:boto:Token: None 2022-09-02T05:55:55.086 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj68 2022-09-02T05:55:55.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dtnRl5lGtb2Rv5o5WfHV+loi2y0= 2022-09-02T05:55:55.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dtnRl5lGtb2Rv5o5WfHV+loi2y0=', 'Content-Length': '0'} 2022-09-02T05:55:55.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000393f87b38eced267-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.090 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:55.091 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:55.091 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.092 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:55.093 DEBUG:boto:Data: 2022-09-02T05:55:55.093 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.094 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.095 DEBUG:boto:Params: {} 2022-09-02T05:55:55.095 DEBUG:boto:Token: None 2022-09-02T05:55:55.096 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj68 2022-09-02T05:55:55.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dtnRl5lGtb2Rv5o5WfHV+loi2y0= 2022-09-02T05:55:55.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dtnRl5lGtb2Rv5o5WfHV+loi2y0=', 'Content-Length': '0'} 2022-09-02T05:55:55.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000014738f150ba0b9f-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:55:55.101 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:55.102 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:55.103 DEBUG:boto:Method: GET 2022-09-02T05:55:55.103 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:55.104 DEBUG:boto:Data: 2022-09-02T05:55:55.104 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.105 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.106 DEBUG:boto:Params: {} 2022-09-02T05:55:55.107 DEBUG:boto:Token: None 2022-09-02T05:55:55.108 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj68 2022-09-02T05:55:55.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MsCzIfZSVukFWTgIcw5CMX78yns= 2022-09-02T05:55:55.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MsCzIfZSVukFWTgIcw5CMX78yns=', 'Content-Length': '0'} 2022-09-02T05:55:55.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ac4b5b5c5fa8d4-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.112 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:55:55.112 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:55:55.113 DEBUG:boto:Method: GET 2022-09-02T05:55:55.114 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:55:55.114 DEBUG:boto:Data: 2022-09-02T05:55:55.115 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.116 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.116 DEBUG:boto:Params: {} 2022-09-02T05:55:55.117 DEBUG:boto:Token: None 2022-09-02T05:55:55.118 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj68 2022-09-02T05:55:55.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MsCzIfZSVukFWTgIcw5CMX78yns= 2022-09-02T05:55:55.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MsCzIfZSVukFWTgIcw5CMX78yns=', 'Content-Length': '0'} 2022-09-02T05:55:55.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab4d12c3f51ad30-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.162 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.163 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.164 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.164 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.165 DEBUG:boto:Method: GET 2022-09-02T05:55:55.166 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.167 DEBUG:boto:Data: 2022-09-02T05:55:55.168 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.170 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.171 DEBUG:boto:Params: {} 2022-09-02T05:55:55.172 DEBUG:boto:Token: None 2022-09-02T05:55:55.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019cb0669d8f874bf-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.179 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.180 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.180 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.181 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.182 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.183 DEBUG:boto:Method: GET 2022-09-02T05:55:55.184 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:55:55.185 DEBUG:boto:Data: 2022-09-02T05:55:55.186 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.188 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.188 DEBUG:boto:Params: {} 2022-09-02T05:55:55.190 DEBUG:boto:Token: None 2022-09-02T05:55:55.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a7411cc6acb73e-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.197 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:55:55.198 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.199 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.200 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.200 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.201 DEBUG:boto:Method: GET 2022-09-02T05:55:55.202 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.202 DEBUG:boto:Data: 2022-09-02T05:55:55.203 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.205 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.205 DEBUG:boto:Params: {} 2022-09-02T05:55:55.206 DEBUG:boto:Token: None 2022-09-02T05:55:55.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk= 2022-09-02T05:55:55.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk=', 'Content-Length': '0'} 2022-09-02T05:55:55.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f7506ea1a5e719d-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.213 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.213 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.214 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.214 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.215 DEBUG:boto:Method: GET 2022-09-02T05:55:55.215 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.216 DEBUG:boto:Data: 2022-09-02T05:55:55.216 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.218 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.218 DEBUG:boto:Params: {} 2022-09-02T05:55:55.219 DEBUG:boto:Token: None 2022-09-02T05:55:55.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk= 2022-09-02T05:55:55.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk=', 'Content-Length': '0'} 2022-09-02T05:55:55.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991146db9643367f-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.224 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.224 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.225 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.225 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.226 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.227 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.227 DEBUG:boto:Data: 2022-09-02T05:55:55.228 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.229 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.229 DEBUG:boto:Params: {} 2022-09-02T05:55:55.230 DEBUG:boto:Token: None 2022-09-02T05:55:55.230 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oWVtrXZJDP2imepixWRVSejKFi4= 2022-09-02T05:55:55.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oWVtrXZJDP2imepixWRVSejKFi4=', 'Content-Length': '0'} 2022-09-02T05:55:55.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c316b7cd04a260db-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.234 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.235 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.235 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.236 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.236 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.237 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.237 DEBUG:boto:Data: 2022-09-02T05:55:55.238 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.239 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.240 DEBUG:boto:Params: {} 2022-09-02T05:55:55.240 DEBUG:boto:Token: None 2022-09-02T05:55:55.241 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oWVtrXZJDP2imepixWRVSejKFi4= 2022-09-02T05:55:55.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oWVtrXZJDP2imepixWRVSejKFi4=', 'Content-Length': '0'} 2022-09-02T05:55:55.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088930a5338cbcc9f-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:55:55.245 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.246 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.246 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.247 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.247 DEBUG:boto:Method: GET 2022-09-02T05:55:55.248 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.248 DEBUG:boto:Data: 2022-09-02T05:55:55.249 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.250 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.250 DEBUG:boto:Params: {} 2022-09-02T05:55:55.251 DEBUG:boto:Token: None 2022-09-02T05:55:55.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk= 2022-09-02T05:55:55.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk=', 'Content-Length': '0'} 2022-09-02T05:55:55.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740464e148e64cee-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.257 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.258 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.258 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.259 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.260 DEBUG:boto:Method: GET 2022-09-02T05:55:55.261 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.262 DEBUG:boto:Data: 2022-09-02T05:55:55.263 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.264 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.264 DEBUG:boto:Params: {} 2022-09-02T05:55:55.265 DEBUG:boto:Token: None 2022-09-02T05:55:55.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:55:55.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk= 2022-09-02T05:55:55.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X73uxUI/NKxNqsPFrarzQeBDzKk=', 'Content-Length': '0'} 2022-09-02T05:55:55.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecf80a406cf5bd59-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.317 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.318 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.318 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.320 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.321 DEBUG:boto:Method: GET 2022-09-02T05:55:55.322 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.323 DEBUG:boto:Data: 2022-09-02T05:55:55.324 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.326 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.328 DEBUG:boto:Params: {} 2022-09-02T05:55:55.329 DEBUG:boto:Token: None 2022-09-02T05:55:55.330 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004803fdb4dbfc0423-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.337 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.339 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.340 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.341 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.342 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.343 DEBUG:boto:Method: GET 2022-09-02T05:55:55.344 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.345 DEBUG:boto:Data: 2022-09-02T05:55:55.346 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.348 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.349 DEBUG:boto:Params: {} 2022-09-02T05:55:55.349 DEBUG:boto:Token: None 2022-09-02T05:55:55.350 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e63cbca9a239391-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.357 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.358 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.359 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.360 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.361 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:55.362 DEBUG:boto:Data: 2022-09-02T05:55:55.363 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.365 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.366 DEBUG:boto:Params: {} 2022-09-02T05:55:55.367 DEBUG:boto:Token: None 2022-09-02T05:55:55.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69 2022-09-02T05:55:55.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:btTKd4L9Lk73Bdx51n4xZlSpqNY= 2022-09-02T05:55:55.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:btTKd4L9Lk73Bdx51n4xZlSpqNY=', 'Content-Length': '0'} 2022-09-02T05:55:55.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfa2866f8febde5-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.372 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.373 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.374 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.374 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:55.375 DEBUG:boto:Data: 2022-09-02T05:55:55.375 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.376 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.377 DEBUG:boto:Params: {} 2022-09-02T05:55:55.377 DEBUG:boto:Token: None 2022-09-02T05:55:55.378 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69 2022-09-02T05:55:55.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:btTKd4L9Lk73Bdx51n4xZlSpqNY= 2022-09-02T05:55:55.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:btTKd4L9Lk73Bdx51n4xZlSpqNY=', 'Content-Length': '0'} 2022-09-02T05:55:55.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c048559b18ba6586-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:55:55.384 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.385 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.385 DEBUG:boto:Method: GET 2022-09-02T05:55:55.386 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:55.387 DEBUG:boto:Data: 2022-09-02T05:55:55.387 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.388 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.389 DEBUG:boto:Params: {} 2022-09-02T05:55:55.390 DEBUG:boto:Token: None 2022-09-02T05:55:55.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69 2022-09-02T05:55:55.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEQAOVhhj+Xrvpr11NdLmHWPrSU= 2022-09-02T05:55:55.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEQAOVhhj+Xrvpr11NdLmHWPrSU=', 'Content-Length': '0'} 2022-09-02T05:55:55.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c72f4804c8dc1e-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.395 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:55:55.395 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:55:55.396 DEBUG:boto:Method: GET 2022-09-02T05:55:55.397 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:55:55.397 DEBUG:boto:Data: 2022-09-02T05:55:55.398 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.399 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.400 DEBUG:boto:Params: {} 2022-09-02T05:55:55.401 DEBUG:boto:Token: None 2022-09-02T05:55:55.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj69 2022-09-02T05:55:55.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HEQAOVhhj+Xrvpr11NdLmHWPrSU= 2022-09-02T05:55:55.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HEQAOVhhj+Xrvpr11NdLmHWPrSU=', 'Content-Length': '0'} 2022-09-02T05:55:55.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0e70bf2bdea446c-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.446 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.447 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.448 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.448 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.449 DEBUG:boto:Method: GET 2022-09-02T05:55:55.449 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.450 DEBUG:boto:Data: 2022-09-02T05:55:55.450 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.452 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.452 DEBUG:boto:Params: {} 2022-09-02T05:55:55.453 DEBUG:boto:Token: None 2022-09-02T05:55:55.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb86c9eaae63c27-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.459 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.459 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.460 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.461 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.461 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.462 DEBUG:boto:Method: GET 2022-09-02T05:55:55.462 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:55:55.463 DEBUG:boto:Data: 2022-09-02T05:55:55.464 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.465 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.465 DEBUG:boto:Params: {} 2022-09-02T05:55:55.466 DEBUG:boto:Token: None 2022-09-02T05:55:55.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb32c8a5195e8424-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.473 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:55:55.474 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.475 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.476 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.476 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.477 DEBUG:boto:Method: GET 2022-09-02T05:55:55.477 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.478 DEBUG:boto:Data: 2022-09-02T05:55:55.478 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.479 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.480 DEBUG:boto:Params: {} 2022-09-02T05:55:55.481 DEBUG:boto:Token: None 2022-09-02T05:55:55.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M= 2022-09-02T05:55:55.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M=', 'Content-Length': '0'} 2022-09-02T05:55:55.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb296cefacbbcf14-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.526 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.527 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.527 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.528 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.529 DEBUG:boto:Method: GET 2022-09-02T05:55:55.530 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.531 DEBUG:boto:Data: 2022-09-02T05:55:55.532 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.533 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.533 DEBUG:boto:Params: {} 2022-09-02T05:55:55.534 DEBUG:boto:Token: None 2022-09-02T05:55:55.535 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M= 2022-09-02T05:55:55.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M=', 'Content-Length': '0'} 2022-09-02T05:55:55.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006908e43ce95ee49d-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.541 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.542 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.542 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.543 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.543 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.544 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.544 DEBUG:boto:Data: 2022-09-02T05:55:55.545 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.546 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.547 DEBUG:boto:Params: {} 2022-09-02T05:55:55.547 DEBUG:boto:Token: None 2022-09-02T05:55:55.548 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AoBSSE5qhlpWQ95CdS1YwITlAos= 2022-09-02T05:55:55.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AoBSSE5qhlpWQ95CdS1YwITlAos=', 'Content-Length': '0'} 2022-09-02T05:55:55.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7c3b17f827b303-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.553 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.554 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.554 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.555 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.555 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.556 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.556 DEBUG:boto:Data: 2022-09-02T05:55:55.557 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.558 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.558 DEBUG:boto:Params: {} 2022-09-02T05:55:55.560 DEBUG:boto:Token: None 2022-09-02T05:55:55.561 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AoBSSE5qhlpWQ95CdS1YwITlAos= 2022-09-02T05:55:55.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AoBSSE5qhlpWQ95CdS1YwITlAos=', 'Content-Length': '0'} 2022-09-02T05:55:55.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869c086cc16c4a7a-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:55:55.565 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.566 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.566 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.567 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.567 DEBUG:boto:Method: GET 2022-09-02T05:55:55.568 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.568 DEBUG:boto:Data: 2022-09-02T05:55:55.569 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.571 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.571 DEBUG:boto:Params: {} 2022-09-02T05:55:55.572 DEBUG:boto:Token: None 2022-09-02T05:55:55.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M= 2022-09-02T05:55:55.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M=', 'Content-Length': '0'} 2022-09-02T05:55:55.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008413c7af4ce8989a-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.577 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.578 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.578 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.579 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.580 DEBUG:boto:Method: GET 2022-09-02T05:55:55.581 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.582 DEBUG:boto:Data: 2022-09-02T05:55:55.582 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.583 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.584 DEBUG:boto:Params: {} 2022-09-02T05:55:55.585 DEBUG:boto:Token: None 2022-09-02T05:55:55.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:55:55.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M= 2022-09-02T05:55:55.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OdyICVihL8qE+XUo2bAKDlMFr1M=', 'Content-Length': '0'} 2022-09-02T05:55:55.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096f942563101d6e8-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.592 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.593 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.594 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.595 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.596 DEBUG:boto:Method: GET 2022-09-02T05:55:55.597 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.598 DEBUG:boto:Data: 2022-09-02T05:55:55.599 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.601 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.602 DEBUG:boto:Params: {} 2022-09-02T05:55:55.603 DEBUG:boto:Token: None 2022-09-02T05:55:55.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4b17894ec3f0cf-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.612 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.614 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.615 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.616 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.617 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.618 DEBUG:boto:Method: GET 2022-09-02T05:55:55.619 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.621 DEBUG:boto:Data: 2022-09-02T05:55:55.622 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.624 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.624 DEBUG:boto:Params: {} 2022-09-02T05:55:55.625 DEBUG:boto:Token: None 2022-09-02T05:55:55.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa086c9d762be8a-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.632 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.633 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.634 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.634 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.635 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:55.635 DEBUG:boto:Data: 2022-09-02T05:55:55.636 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.637 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.637 DEBUG:boto:Params: {} 2022-09-02T05:55:55.638 DEBUG:boto:Token: None 2022-09-02T05:55:55.640 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76 2022-09-02T05:55:55.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gb63/9Oziwf/ObzR5MdYeEv96JU= 2022-09-02T05:55:55.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gb63/9Oziwf/ObzR5MdYeEv96JU=', 'Content-Length': '0'} 2022-09-02T05:55:55.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054aaf196730b3672-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.644 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.645 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.645 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.646 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:55.646 DEBUG:boto:Data: 2022-09-02T05:55:55.647 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.648 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.649 DEBUG:boto:Params: {} 2022-09-02T05:55:55.650 DEBUG:boto:Token: None 2022-09-02T05:55:55.651 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76 2022-09-02T05:55:55.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gb63/9Oziwf/ObzR5MdYeEv96JU= 2022-09-02T05:55:55.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gb63/9Oziwf/ObzR5MdYeEv96JU=', 'Content-Length': '0'} 2022-09-02T05:55:55.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a81dfe99f20db13-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:55:55.655 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.656 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.656 DEBUG:boto:Method: GET 2022-09-02T05:55:55.657 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:55.658 DEBUG:boto:Data: 2022-09-02T05:55:55.658 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.660 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.660 DEBUG:boto:Params: {} 2022-09-02T05:55:55.661 DEBUG:boto:Token: None 2022-09-02T05:55:55.662 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76 2022-09-02T05:55:55.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F5gQGBmsqZwQcDpyNN2pX/xTmck= 2022-09-02T05:55:55.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F5gQGBmsqZwQcDpyNN2pX/xTmck=', 'Content-Length': '0'} 2022-09-02T05:55:55.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd8fa58f4d53d1f-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.667 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:55:55.667 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:55:55.668 DEBUG:boto:Method: GET 2022-09-02T05:55:55.668 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:55:55.669 DEBUG:boto:Data: 2022-09-02T05:55:55.670 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.671 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.672 DEBUG:boto:Params: {} 2022-09-02T05:55:55.672 DEBUG:boto:Token: None 2022-09-02T05:55:55.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj76 2022-09-02T05:55:55.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:F5gQGBmsqZwQcDpyNN2pX/xTmck= 2022-09-02T05:55:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:F5gQGBmsqZwQcDpyNN2pX/xTmck=', 'Content-Length': '0'} 2022-09-02T05:55:55.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002131aa000259c958-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.722 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.723 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.724 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.725 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.726 DEBUG:boto:Method: GET 2022-09-02T05:55:55.727 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.728 DEBUG:boto:Data: 2022-09-02T05:55:55.729 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.731 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.732 DEBUG:boto:Params: {} 2022-09-02T05:55:55.734 DEBUG:boto:Token: None 2022-09-02T05:55:55.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a994a4e2955a4c1b-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.743 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.745 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.746 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.748 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.749 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.750 DEBUG:boto:Method: GET 2022-09-02T05:55:55.751 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:55:55.753 DEBUG:boto:Data: 2022-09-02T05:55:55.754 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.756 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.757 DEBUG:boto:Params: {} 2022-09-02T05:55:55.758 DEBUG:boto:Token: None 2022-09-02T05:55:55.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027ccd514d23fc18b-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.766 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:55:55.768 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.768 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.769 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.770 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.771 DEBUG:boto:Method: GET 2022-09-02T05:55:55.771 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.772 DEBUG:boto:Data: 2022-09-02T05:55:55.773 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.774 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.775 DEBUG:boto:Params: {} 2022-09-02T05:55:55.776 DEBUG:boto:Token: None 2022-09-02T05:55:55.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc= 2022-09-02T05:55:55.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc=', 'Content-Length': '0'} 2022-09-02T05:55:55.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04f11f3ed0dce91-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.781 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.782 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.782 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.783 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.784 DEBUG:boto:Method: GET 2022-09-02T05:55:55.784 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.785 DEBUG:boto:Data: 2022-09-02T05:55:55.785 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.786 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.787 DEBUG:boto:Params: {} 2022-09-02T05:55:55.788 DEBUG:boto:Token: None 2022-09-02T05:55:55.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc= 2022-09-02T05:55:55.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc=', 'Content-Length': '0'} 2022-09-02T05:55:55.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e57b23d65a6529-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.793 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.794 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.795 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.795 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.796 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.796 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.797 DEBUG:boto:Data: 2022-09-02T05:55:55.798 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.799 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.799 DEBUG:boto:Params: {} 2022-09-02T05:55:55.800 DEBUG:boto:Token: None 2022-09-02T05:55:55.800 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xIn/zxfzAZ20Le+IW+zKFbOseEQ= 2022-09-02T05:55:55.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xIn/zxfzAZ20Le+IW+zKFbOseEQ=', 'Content-Length': '0'} 2022-09-02T05:55:55.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8fba401fdaddda2-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.804 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.805 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.805 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.806 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.806 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.807 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.807 DEBUG:boto:Data: 2022-09-02T05:55:55.808 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.809 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.809 DEBUG:boto:Params: {} 2022-09-02T05:55:55.810 DEBUG:boto:Token: None 2022-09-02T05:55:55.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xIn/zxfzAZ20Le+IW+zKFbOseEQ= 2022-09-02T05:55:55.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xIn/zxfzAZ20Le+IW+zKFbOseEQ=', 'Content-Length': '0'} 2022-09-02T05:55:55.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376311a3b4aac184-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:55:55.816 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.816 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.817 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.817 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.818 DEBUG:boto:Method: GET 2022-09-02T05:55:55.818 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.819 DEBUG:boto:Data: 2022-09-02T05:55:55.819 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.820 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.821 DEBUG:boto:Params: {} 2022-09-02T05:55:55.822 DEBUG:boto:Token: None 2022-09-02T05:55:55.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc= 2022-09-02T05:55:55.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc=', 'Content-Length': '0'} 2022-09-02T05:55:55.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f082e7ba826f343f-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.827 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.827 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.828 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.828 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.830 DEBUG:boto:Method: GET 2022-09-02T05:55:55.831 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.831 DEBUG:boto:Data: 2022-09-02T05:55:55.832 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.834 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.835 DEBUG:boto:Params: {} 2022-09-02T05:55:55.836 DEBUG:boto:Token: None 2022-09-02T05:55:55.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:55:55.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc= 2022-09-02T05:55:55.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dbaymPbkws8Y+czspm6tb1RZdyc=', 'Content-Length': '0'} 2022-09-02T05:55:55.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c664bdec66520634-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.843 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.843 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.844 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.844 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.845 DEBUG:boto:Method: GET 2022-09-02T05:55:55.845 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.846 DEBUG:boto:Data: 2022-09-02T05:55:55.847 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.848 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.849 DEBUG:boto:Params: {} 2022-09-02T05:55:55.850 DEBUG:boto:Token: None 2022-09-02T05:55:55.851 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000700ca05459b3511a-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.858 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.860 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.861 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.862 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.862 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.863 DEBUG:boto:Method: GET 2022-09-02T05:55:55.863 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.864 DEBUG:boto:Data: 2022-09-02T05:55:55.865 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.866 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.866 DEBUG:boto:Params: {} 2022-09-02T05:55:55.867 DEBUG:boto:Token: None 2022-09-02T05:55:55.868 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f288ed4c2d0aa2e-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.874 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.875 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.875 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.876 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.877 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:55.877 DEBUG:boto:Data: 2022-09-02T05:55:55.878 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.879 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.879 DEBUG:boto:Params: {} 2022-09-02T05:55:55.880 DEBUG:boto:Token: None 2022-09-02T05:55:55.881 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85 2022-09-02T05:55:55.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EI91v6QEXuoEww/Nj1OihalOBnU= 2022-09-02T05:55:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EI91v6QEXuoEww/Nj1OihalOBnU=', 'Content-Length': '0'} 2022-09-02T05:55:55.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa8b0cee7406933-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.885 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.886 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.886 DEBUG:boto:Method: HEAD 2022-09-02T05:55:55.887 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:55.888 DEBUG:boto:Data: 2022-09-02T05:55:55.888 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.889 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.890 DEBUG:boto:Params: {} 2022-09-02T05:55:55.891 DEBUG:boto:Token: None 2022-09-02T05:55:55.891 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85 2022-09-02T05:55:55.892 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EI91v6QEXuoEww/Nj1OihalOBnU= 2022-09-02T05:55:55.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EI91v6QEXuoEww/Nj1OihalOBnU=', 'Content-Length': '0'} 2022-09-02T05:55:55.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be5ce07a8dbaa502-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:55:55.897 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.898 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.898 DEBUG:boto:Method: GET 2022-09-02T05:55:55.899 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:55.899 DEBUG:boto:Data: 2022-09-02T05:55:55.900 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.901 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.901 DEBUG:boto:Params: {} 2022-09-02T05:55:55.902 DEBUG:boto:Token: None 2022-09-02T05:55:55.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85 2022-09-02T05:55:55.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j9LoMZET6/5kTmAOQ45ybS58Jjo= 2022-09-02T05:55:55.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j9LoMZET6/5kTmAOQ45ybS58Jjo=', 'Content-Length': '0'} 2022-09-02T05:55:55.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c9507d8e6189e48-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.908 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:55:55.908 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:55:55.909 DEBUG:boto:Method: GET 2022-09-02T05:55:55.910 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:55:55.910 DEBUG:boto:Data: 2022-09-02T05:55:55.911 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.912 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.913 DEBUG:boto:Params: {} 2022-09-02T05:55:55.914 DEBUG:boto:Token: None 2022-09-02T05:55:55.914 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/obj85 2022-09-02T05:55:55.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j9LoMZET6/5kTmAOQ45ybS58Jjo= 2022-09-02T05:55:55.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j9LoMZET6/5kTmAOQ45ybS58Jjo=', 'Content-Length': '0'} 2022-09-02T05:55:55.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb83bf09fedb1a9a-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.919 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.920 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.921 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.921 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.922 DEBUG:boto:Method: GET 2022-09-02T05:55:55.923 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.923 DEBUG:boto:Data: 2022-09-02T05:55:55.924 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:55.926 DEBUG:boto:Port: 8001 2022-09-02T05:55:55.926 DEBUG:boto:Params: {} 2022-09-02T05:55:55.927 DEBUG:boto:Token: None 2022-09-02T05:55:55.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d789ed93627494b6-0063119aeb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:55.978 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:55.980 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:55.981 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:55.983 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.984 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.985 DEBUG:boto:Method: GET 2022-09-02T05:55:55.987 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:55:55.987 DEBUG:boto:Data: 2022-09-02T05:55:55.988 DEBUG:boto:Headers: {} 2022-09-02T05:55:55.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:55.991 DEBUG:boto:Port: 8000 2022-09-02T05:55:55.991 DEBUG:boto:Params: {} 2022-09-02T05:55:55.992 DEBUG:boto:Token: None 2022-09-02T05:55:55.993 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:55 GMT /xljbkf-74/ 2022-09-02T05:55:55.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc= 2022-09-02T05:55:55.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NrJMdmlM8xX5kJez+RYMtMe/Yqc=', 'Content-Length': '0'} 2022-09-02T05:55:55.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e2e116e4c3f033c-0063119aeb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.000 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:55:56.002 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.003 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.003 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.004 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.005 DEBUG:boto:Method: GET 2022-09-02T05:55:56.006 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.007 DEBUG:boto:Data: 2022-09-02T05:55:56.008 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.009 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.009 DEBUG:boto:Params: {} 2022-09-02T05:55:56.010 DEBUG:boto:Token: None 2022-09-02T05:55:56.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM= 2022-09-02T05:55:56.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM=', 'Content-Length': '0'} 2022-09-02T05:55:56.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410c9321bcedf580-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.017 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.018 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.019 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.019 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.020 DEBUG:boto:Method: GET 2022-09-02T05:55:56.021 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.021 DEBUG:boto:Data: 2022-09-02T05:55:56.022 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.023 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.024 DEBUG:boto:Params: {} 2022-09-02T05:55:56.025 DEBUG:boto:Token: None 2022-09-02T05:55:56.026 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM= 2022-09-02T05:55:56.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM=', 'Content-Length': '0'} 2022-09-02T05:55:56.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4566fcefc3fd40-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.031 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.032 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.032 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.033 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.034 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.035 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.036 DEBUG:boto:Data: 2022-09-02T05:55:56.037 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.039 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.040 DEBUG:boto:Params: {} 2022-09-02T05:55:56.041 DEBUG:boto:Token: None 2022-09-02T05:55:56.041 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANiHRm+oduqfkS83GoO6V1a2yCQ= 2022-09-02T05:55:56.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANiHRm+oduqfkS83GoO6V1a2yCQ=', 'Content-Length': '0'} 2022-09-02T05:55:56.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c6339a77b3b49d-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.047 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.048 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.049 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.049 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.050 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.051 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.052 DEBUG:boto:Data: 2022-09-02T05:55:56.053 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.055 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.055 DEBUG:boto:Params: {} 2022-09-02T05:55:56.056 DEBUG:boto:Token: None 2022-09-02T05:55:56.057 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANiHRm+oduqfkS83GoO6V1a2yCQ= 2022-09-02T05:55:56.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANiHRm+oduqfkS83GoO6V1a2yCQ=', 'Content-Length': '0'} 2022-09-02T05:55:56.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a1a3fd43a40efd2-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:55:56.068 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.069 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.071 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.071 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.072 DEBUG:boto:Method: GET 2022-09-02T05:55:56.072 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.073 DEBUG:boto:Data: 2022-09-02T05:55:56.074 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.075 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.076 DEBUG:boto:Params: {} 2022-09-02T05:55:56.077 DEBUG:boto:Token: None 2022-09-02T05:55:56.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM= 2022-09-02T05:55:56.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM=', 'Content-Length': '0'} 2022-09-02T05:55:56.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0bbf32323e89c1-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.082 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.083 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.084 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.084 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.085 DEBUG:boto:Method: GET 2022-09-02T05:55:56.085 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.086 DEBUG:boto:Data: 2022-09-02T05:55:56.087 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.088 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.089 DEBUG:boto:Params: {} 2022-09-02T05:55:56.089 DEBUG:boto:Token: None 2022-09-02T05:55:56.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:55:56.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM= 2022-09-02T05:55:56.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LBVgriEWbLtEZOvz3Yl6TtZf2iM=', 'Content-Length': '0'} 2022-09-02T05:55:56.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3f6d3cb53bd62c-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.134 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.135 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.136 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.137 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.138 DEBUG:boto:Method: GET 2022-09-02T05:55:56.139 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.140 DEBUG:boto:Data: 2022-09-02T05:55:56.141 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.143 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.144 DEBUG:boto:Params: {} 2022-09-02T05:55:56.144 DEBUG:boto:Token: None 2022-09-02T05:55:56.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007867433ba542e717-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.152 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.154 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.155 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.155 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.156 DEBUG:boto:Method: GET 2022-09-02T05:55:56.156 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.157 DEBUG:boto:Data: 2022-09-02T05:55:56.157 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.159 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.159 DEBUG:boto:Params: {} 2022-09-02T05:55:56.160 DEBUG:boto:Token: None 2022-09-02T05:55:56.161 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.162 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d695e448bb405c5-0063119aec-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.168 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.169 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.170 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.171 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.172 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:56.173 DEBUG:boto:Data: 2022-09-02T05:55:56.174 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.176 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.177 DEBUG:boto:Params: {} 2022-09-02T05:55:56.178 DEBUG:boto:Token: None 2022-09-02T05:55:56.179 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87 2022-09-02T05:55:56.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6RZfiytG0sxUb68j7Fqafy/ngxU= 2022-09-02T05:55:56.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6RZfiytG0sxUb68j7Fqafy/ngxU=', 'Content-Length': '0'} 2022-09-02T05:55:56.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef1740b402f3847-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.184 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.184 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.185 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.185 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:56.186 DEBUG:boto:Data: 2022-09-02T05:55:56.187 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.189 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.190 DEBUG:boto:Params: {} 2022-09-02T05:55:56.191 DEBUG:boto:Token: None 2022-09-02T05:55:56.192 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87 2022-09-02T05:55:56.193 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6RZfiytG0sxUb68j7Fqafy/ngxU= 2022-09-02T05:55:56.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6RZfiytG0sxUb68j7Fqafy/ngxU=', 'Content-Length': '0'} 2022-09-02T05:55:56.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c80569ee977ccb-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:55:56.197 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.198 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.199 DEBUG:boto:Method: GET 2022-09-02T05:55:56.199 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:56.200 DEBUG:boto:Data: 2022-09-02T05:55:56.200 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.201 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.202 DEBUG:boto:Params: {} 2022-09-02T05:55:56.202 DEBUG:boto:Token: None 2022-09-02T05:55:56.203 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87 2022-09-02T05:55:56.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h85ES16n/HnMR0h6JPiQFBe5qHA= 2022-09-02T05:55:56.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h85ES16n/HnMR0h6JPiQFBe5qHA=', 'Content-Length': '0'} 2022-09-02T05:55:56.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad74de5cf1949cfe-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.254 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:55:56.255 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:55:56.256 DEBUG:boto:Method: GET 2022-09-02T05:55:56.256 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:55:56.257 DEBUG:boto:Data: 2022-09-02T05:55:56.257 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.259 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.259 DEBUG:boto:Params: {} 2022-09-02T05:55:56.260 DEBUG:boto:Token: None 2022-09-02T05:55:56.261 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj87 2022-09-02T05:55:56.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h85ES16n/HnMR0h6JPiQFBe5qHA= 2022-09-02T05:55:56.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h85ES16n/HnMR0h6JPiQFBe5qHA=', 'Content-Length': '0'} 2022-09-02T05:55:56.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0a00975935768b-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.265 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.266 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.266 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.267 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.267 DEBUG:boto:Method: GET 2022-09-02T05:55:56.268 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.268 DEBUG:boto:Data: 2022-09-02T05:55:56.269 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.270 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.271 DEBUG:boto:Params: {} 2022-09-02T05:55:56.271 DEBUG:boto:Token: None 2022-09-02T05:55:56.272 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33d716700cde5f0-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.277 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.278 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.279 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.280 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.281 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.282 DEBUG:boto:Method: GET 2022-09-02T05:55:56.283 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:55:56.283 DEBUG:boto:Data: 2022-09-02T05:55:56.284 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.286 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.287 DEBUG:boto:Params: {} 2022-09-02T05:55:56.289 DEBUG:boto:Token: None 2022-09-02T05:55:56.290 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e70a739722323fe-0063119aec-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.297 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:55:56.298 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.299 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.300 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.300 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.301 DEBUG:boto:Method: GET 2022-09-02T05:55:56.301 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.302 DEBUG:boto:Data: 2022-09-02T05:55:56.303 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.304 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.304 DEBUG:boto:Params: {} 2022-09-02T05:55:56.305 DEBUG:boto:Token: None 2022-09-02T05:55:56.306 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.307 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0= 2022-09-02T05:55:56.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0=', 'Content-Length': '0'} 2022-09-02T05:55:56.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f899b1ee827e4b93-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.310 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.311 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.311 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.312 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.313 DEBUG:boto:Method: GET 2022-09-02T05:55:56.313 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.314 DEBUG:boto:Data: 2022-09-02T05:55:56.315 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.316 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.316 DEBUG:boto:Params: {} 2022-09-02T05:55:56.317 DEBUG:boto:Token: None 2022-09-02T05:55:56.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0= 2022-09-02T05:55:56.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0=', 'Content-Length': '0'} 2022-09-02T05:55:56.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1740c898e30966-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.362 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.363 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.363 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.364 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.364 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.365 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.365 DEBUG:boto:Data: 2022-09-02T05:55:56.366 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.367 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.367 DEBUG:boto:Params: {} 2022-09-02T05:55:56.368 DEBUG:boto:Token: None 2022-09-02T05:55:56.369 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nVAhxA4WOcwUOnJyzz+ejvxD55U= 2022-09-02T05:55:56.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nVAhxA4WOcwUOnJyzz+ejvxD55U=', 'Content-Length': '0'} 2022-09-02T05:55:56.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6deb6c3d3e9e96c-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.377 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.377 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.378 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.379 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.380 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.381 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.381 DEBUG:boto:Data: 2022-09-02T05:55:56.381 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.383 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.383 DEBUG:boto:Params: {} 2022-09-02T05:55:56.384 DEBUG:boto:Token: None 2022-09-02T05:55:56.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nVAhxA4WOcwUOnJyzz+ejvxD55U= 2022-09-02T05:55:56.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nVAhxA4WOcwUOnJyzz+ejvxD55U=', 'Content-Length': '0'} 2022-09-02T05:55:56.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d8583979aef9457-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:55:56.389 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.389 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.390 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.390 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.391 DEBUG:boto:Method: GET 2022-09-02T05:55:56.391 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.392 DEBUG:boto:Data: 2022-09-02T05:55:56.393 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.394 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.395 DEBUG:boto:Params: {} 2022-09-02T05:55:56.396 DEBUG:boto:Token: None 2022-09-02T05:55:56.396 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0= 2022-09-02T05:55:56.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0=', 'Content-Length': '0'} 2022-09-02T05:55:56.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055046538d312cb47-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.402 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.403 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.403 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.404 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.404 DEBUG:boto:Method: GET 2022-09-02T05:55:56.405 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.405 DEBUG:boto:Data: 2022-09-02T05:55:56.406 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.407 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.408 DEBUG:boto:Params: {} 2022-09-02T05:55:56.408 DEBUG:boto:Token: None 2022-09-02T05:55:56.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:55:56.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0= 2022-09-02T05:55:56.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1trXZfe7alpw0RFqKaMMYTYIOH0=', 'Content-Length': '0'} 2022-09-02T05:55:56.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d220ab9f754f95b-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.415 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.416 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.417 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.417 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.418 DEBUG:boto:Method: GET 2022-09-02T05:55:56.419 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.420 DEBUG:boto:Data: 2022-09-02T05:55:56.421 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.423 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.424 DEBUG:boto:Params: {} 2022-09-02T05:55:56.425 DEBUG:boto:Token: None 2022-09-02T05:55:56.426 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097dda77a52c986d-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.433 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.436 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.436 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.437 DEBUG:boto:Method: GET 2022-09-02T05:55:56.438 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.438 DEBUG:boto:Data: 2022-09-02T05:55:56.439 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.440 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.441 DEBUG:boto:Params: {} 2022-09-02T05:55:56.442 DEBUG:boto:Token: None 2022-09-02T05:55:56.443 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec38a9624711e0c3-0063119aec-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.451 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.452 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.453 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.453 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.454 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:56.454 DEBUG:boto:Data: 2022-09-02T05:55:56.455 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.456 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.456 DEBUG:boto:Params: {} 2022-09-02T05:55:56.457 DEBUG:boto:Token: None 2022-09-02T05:55:56.458 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92 2022-09-02T05:55:56.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GarTglVHNytsKU6hJpFGaUuSzY0= 2022-09-02T05:55:56.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GarTglVHNytsKU6hJpFGaUuSzY0=', 'Content-Length': '0'} 2022-09-02T05:55:56.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003198ee1b980d9c9e-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.465 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.466 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.467 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.468 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:56.469 DEBUG:boto:Data: 2022-09-02T05:55:56.469 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.470 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.471 DEBUG:boto:Params: {} 2022-09-02T05:55:56.471 DEBUG:boto:Token: None 2022-09-02T05:55:56.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92 2022-09-02T05:55:56.472 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GarTglVHNytsKU6hJpFGaUuSzY0= 2022-09-02T05:55:56.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GarTglVHNytsKU6hJpFGaUuSzY0=', 'Content-Length': '0'} 2022-09-02T05:55:56.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000626ed2c429e4e5dd-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:55:56.477 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.477 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.478 DEBUG:boto:Method: GET 2022-09-02T05:55:56.479 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:56.479 DEBUG:boto:Data: 2022-09-02T05:55:56.480 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.481 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.482 DEBUG:boto:Params: {} 2022-09-02T05:55:56.482 DEBUG:boto:Token: None 2022-09-02T05:55:56.483 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92 2022-09-02T05:55:56.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nrrcIaPJUxcFf9a06xMeqGWCo04= 2022-09-02T05:55:56.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nrrcIaPJUxcFf9a06xMeqGWCo04=', 'Content-Length': '0'} 2022-09-02T05:55:56.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026713b098df197f9-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.534 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:55:56.535 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:55:56.536 DEBUG:boto:Method: GET 2022-09-02T05:55:56.537 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:55:56.538 DEBUG:boto:Data: 2022-09-02T05:55:56.540 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.541 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.541 DEBUG:boto:Params: {} 2022-09-02T05:55:56.542 DEBUG:boto:Token: None 2022-09-02T05:55:56.542 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj92 2022-09-02T05:55:56.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nrrcIaPJUxcFf9a06xMeqGWCo04= 2022-09-02T05:55:56.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nrrcIaPJUxcFf9a06xMeqGWCo04=', 'Content-Length': '0'} 2022-09-02T05:55:56.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee6e31d34f630c9-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.551 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.552 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.552 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.553 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.554 DEBUG:boto:Method: GET 2022-09-02T05:55:56.554 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.555 DEBUG:boto:Data: 2022-09-02T05:55:56.555 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.556 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.557 DEBUG:boto:Params: {} 2022-09-02T05:55:56.558 DEBUG:boto:Token: None 2022-09-02T05:55:56.558 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006653be346e8b89e1-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.563 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.564 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.565 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.566 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.566 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.567 DEBUG:boto:Method: GET 2022-09-02T05:55:56.567 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:55:56.568 DEBUG:boto:Data: 2022-09-02T05:55:56.568 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.569 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.570 DEBUG:boto:Params: {} 2022-09-02T05:55:56.571 DEBUG:boto:Token: None 2022-09-02T05:55:56.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84d1317770ff9a6-0063119aec-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.622 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:55:56.624 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.624 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.625 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.625 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.626 DEBUG:boto:Method: GET 2022-09-02T05:55:56.627 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.627 DEBUG:boto:Data: 2022-09-02T05:55:56.628 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.630 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.631 DEBUG:boto:Params: {} 2022-09-02T05:55:56.633 DEBUG:boto:Token: None 2022-09-02T05:55:56.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI= 2022-09-02T05:55:56.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI=', 'Content-Length': '0'} 2022-09-02T05:55:56.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a3c5198d34b972-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.641 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.642 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.643 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.643 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.644 DEBUG:boto:Method: GET 2022-09-02T05:55:56.645 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.646 DEBUG:boto:Data: 2022-09-02T05:55:56.647 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.649 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.650 DEBUG:boto:Params: {} 2022-09-02T05:55:56.651 DEBUG:boto:Token: None 2022-09-02T05:55:56.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI= 2022-09-02T05:55:56.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI=', 'Content-Length': '0'} 2022-09-02T05:55:56.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f4e7b3b55dd84b-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.656 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.657 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.658 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.659 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.660 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.661 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.662 DEBUG:boto:Data: 2022-09-02T05:55:56.663 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.665 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.666 DEBUG:boto:Params: {} 2022-09-02T05:55:56.667 DEBUG:boto:Token: None 2022-09-02T05:55:56.668 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XmnF5ChtT7r1DdZ0WXGrGtTeLQ4= 2022-09-02T05:55:56.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XmnF5ChtT7r1DdZ0WXGrGtTeLQ4=', 'Content-Length': '0'} 2022-09-02T05:55:56.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe4ebeaf4e840d9-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.672 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.673 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.673 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.674 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.674 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.675 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.676 DEBUG:boto:Data: 2022-09-02T05:55:56.677 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.678 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.678 DEBUG:boto:Params: {} 2022-09-02T05:55:56.679 DEBUG:boto:Token: None 2022-09-02T05:55:56.679 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XmnF5ChtT7r1DdZ0WXGrGtTeLQ4= 2022-09-02T05:55:56.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XmnF5ChtT7r1DdZ0WXGrGtTeLQ4=', 'Content-Length': '0'} 2022-09-02T05:55:56.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005960de3c67ba9543-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:55:56.684 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.685 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.685 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.686 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.687 DEBUG:boto:Method: GET 2022-09-02T05:55:56.688 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.688 DEBUG:boto:Data: 2022-09-02T05:55:56.689 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.690 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.690 DEBUG:boto:Params: {} 2022-09-02T05:55:56.691 DEBUG:boto:Token: None 2022-09-02T05:55:56.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI= 2022-09-02T05:55:56.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI=', 'Content-Length': '0'} 2022-09-02T05:55:56.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9fe872982c01a77-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.696 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.696 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.697 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.697 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.698 DEBUG:boto:Method: GET 2022-09-02T05:55:56.699 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.699 DEBUG:boto:Data: 2022-09-02T05:55:56.700 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.701 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.701 DEBUG:boto:Params: {} 2022-09-02T05:55:56.702 DEBUG:boto:Token: None 2022-09-02T05:55:56.703 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:55:56.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI= 2022-09-02T05:55:56.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QYKFCGYqzoNPXaNYRMRBZ+lJZZI=', 'Content-Length': '0'} 2022-09-02T05:55:56.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309458d874592d35-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.707 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.708 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.708 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.709 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.709 DEBUG:boto:Method: GET 2022-09-02T05:55:56.710 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.711 DEBUG:boto:Data: 2022-09-02T05:55:56.711 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.712 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.713 DEBUG:boto:Params: {} 2022-09-02T05:55:56.713 DEBUG:boto:Token: None 2022-09-02T05:55:56.714 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dae97384bdbfcb4-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.719 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.720 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.720 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.721 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.721 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.722 DEBUG:boto:Method: GET 2022-09-02T05:55:56.722 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.723 DEBUG:boto:Data: 2022-09-02T05:55:56.723 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.724 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.725 DEBUG:boto:Params: {} 2022-09-02T05:55:56.726 DEBUG:boto:Token: None 2022-09-02T05:55:56.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e317ae8c432dd38a-0063119aec-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.778 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.780 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.781 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.782 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.782 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:56.783 DEBUG:boto:Data: 2022-09-02T05:55:56.784 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.786 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.787 DEBUG:boto:Params: {} 2022-09-02T05:55:56.788 DEBUG:boto:Token: None 2022-09-02T05:55:56.788 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95 2022-09-02T05:55:56.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rk/iIY0JT40nZ7rn+Dd2QhlGo7s= 2022-09-02T05:55:56.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rk/iIY0JT40nZ7rn+Dd2QhlGo7s=', 'Content-Length': '0'} 2022-09-02T05:55:56.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674c6bad77329ada-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.792 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.793 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.794 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.795 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:56.796 DEBUG:boto:Data: 2022-09-02T05:55:56.796 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.797 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.798 DEBUG:boto:Params: {} 2022-09-02T05:55:56.799 DEBUG:boto:Token: None 2022-09-02T05:55:56.799 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95 2022-09-02T05:55:56.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rk/iIY0JT40nZ7rn+Dd2QhlGo7s= 2022-09-02T05:55:56.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rk/iIY0JT40nZ7rn+Dd2QhlGo7s=', 'Content-Length': '0'} 2022-09-02T05:55:56.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b4f3277ebe3714-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:55:56.804 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.804 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.805 DEBUG:boto:Method: GET 2022-09-02T05:55:56.805 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:56.806 DEBUG:boto:Data: 2022-09-02T05:55:56.807 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.808 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.808 DEBUG:boto:Params: {} 2022-09-02T05:55:56.809 DEBUG:boto:Token: None 2022-09-02T05:55:56.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95 2022-09-02T05:55:56.810 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RQmdEvjg3keRc7VV1bfJb4kspLg= 2022-09-02T05:55:56.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RQmdEvjg3keRc7VV1bfJb4kspLg=', 'Content-Length': '0'} 2022-09-02T05:55:56.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000079cf54f84e6e0f-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.816 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:55:56.817 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:55:56.818 DEBUG:boto:Method: GET 2022-09-02T05:55:56.819 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:55:56.820 DEBUG:boto:Data: 2022-09-02T05:55:56.821 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.823 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.824 DEBUG:boto:Params: {} 2022-09-02T05:55:56.825 DEBUG:boto:Token: None 2022-09-02T05:55:56.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/obj95 2022-09-02T05:55:56.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RQmdEvjg3keRc7VV1bfJb4kspLg= 2022-09-02T05:55:56.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RQmdEvjg3keRc7VV1bfJb4kspLg=', 'Content-Length': '0'} 2022-09-02T05:55:56.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d874544218e2e5e-0063119aec-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.835 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.836 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.836 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.837 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.837 DEBUG:boto:Method: GET 2022-09-02T05:55:56.838 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.839 DEBUG:boto:Data: 2022-09-02T05:55:56.839 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.841 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.841 DEBUG:boto:Params: {} 2022-09-02T05:55:56.842 DEBUG:boto:Token: None 2022-09-02T05:55:56.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002247b8b59c40eb6c-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.894 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.895 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.897 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.898 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.900 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.901 DEBUG:boto:Method: GET 2022-09-02T05:55:56.902 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:55:56.903 DEBUG:boto:Data: 2022-09-02T05:55:56.903 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:55:56.905 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.906 DEBUG:boto:Params: {} 2022-09-02T05:55:56.907 DEBUG:boto:Token: None 2022-09-02T05:55:56.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U= 2022-09-02T05:55:56.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI9yv2s0wKiU+jKtR35m+YIcV3U=', 'Content-Length': '0'} 2022-09-02T05:55:56.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e2d58172d30405e-0063119aec-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.915 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.916 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:55:56.917 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:55:56.918 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.919 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.919 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.920 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:55:56.920 DEBUG:boto:Data: 2022-09-02T05:55:56.921 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.922 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.922 DEBUG:boto:Params: {} 2022-09-02T05:55:56.923 DEBUG:boto:Token: None 2022-09-02T05:55:56.923 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3bfFl7wvklX6Vm5S46KpELuNnbM= 2022-09-02T05:55:56.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3bfFl7wvklX6Vm5S46KpELuNnbM=', 'Content-Length': '0'} 2022-09-02T05:55:56.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007506082369ea450a-0063119aec-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.928 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.929 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.930 DEBUG:boto:Method: HEAD 2022-09-02T05:55:56.930 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:55:56.931 DEBUG:boto:Data: 2022-09-02T05:55:56.932 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:56.933 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.934 DEBUG:boto:Params: {} 2022-09-02T05:55:56.934 DEBUG:boto:Token: None 2022-09-02T05:55:56.935 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/ 2022-09-02T05:55:56.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3bfFl7wvklX6Vm5S46KpELuNnbM= 2022-09-02T05:55:56.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3bfFl7wvklX6Vm5S46KpELuNnbM=', 'Content-Length': '0'} 2022-09-02T05:55:56.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fc3bf72798ea97b-0063119aec-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:55:56.942 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.943 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.944 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:56.945 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:56.946 DEBUG:boto:Method: GET 2022-09-02T05:55:56.947 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:56.948 DEBUG:boto:Data: 2022-09-02T05:55:56.949 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:56.951 DEBUG:boto:Port: 8001 2022-09-02T05:55:56.952 DEBUG:boto:Params: {} 2022-09-02T05:55:56.953 DEBUG:boto:Token: None 2022-09-02T05:55:56.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:56 GMT /xljbkf-74/?versions 2022-09-02T05:55:56.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YmASJ8LyBAwXhXrRKbWObksdsXg= 2022-09-02T05:55:56.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YmASJ8LyBAwXhXrRKbWObksdsXg=', 'Content-Length': '0'} 2022-09-02T05:55:56.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007656816cf93df222-0063119aec-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:56.962 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:56.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:55:56.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:55:56.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:55:56.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:55:56.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:55:56.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:55:56.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:55:56.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:55:56.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:55:56.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:55:56.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:55:56.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:55:56.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:55:56.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:55:56.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:55:56.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:55:56.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:55:56.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:55:56.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:55:56.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:55:56.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:55:56.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:55:56.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:55:56.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:55:56.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:55:56.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:55:56.988 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:56.989 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:56.990 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:56.991 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:56.992 DEBUG:boto:Method: GET 2022-09-02T05:55:56.993 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:56.995 DEBUG:boto:Data: 2022-09-02T05:55:56.996 DEBUG:boto:Headers: {} 2022-09-02T05:55:56.996 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:56.997 DEBUG:boto:Port: 8000 2022-09-02T05:55:56.998 DEBUG:boto:Params: {} 2022-09-02T05:55:56.999 DEBUG:boto:Token: None 2022-09-02T05:55:57.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/?versions 2022-09-02T05:55:57.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D5OsQp/LjpNzdg4OdFom5Rfmvsc= 2022-09-02T05:55:57.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D5OsQp/LjpNzdg4OdFom5Rfmvsc=', 'Content-Length': '0'} 2022-09-02T05:55:57.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dab47d459453311-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.009 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:55:57.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:55:57.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:55:57.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:55:57.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:55:57.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:55:57.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:55:57.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:55:57.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:55:57.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:55:57.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:55:57.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:55:57.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:55:57.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:55:57.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:55:57.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:55:57.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:55:57.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:55:57.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:55:57.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:55:57.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:55:57.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:55:57.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:55:57.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:55:57.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:55:57.038 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.039 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.040 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:57.041 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:57.042 DEBUG:boto:Method: GET 2022-09-02T05:55:57.043 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:57.044 DEBUG:boto:Data: 2022-09-02T05:55:57.045 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.047 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.048 DEBUG:boto:Params: {} 2022-09-02T05:55:57.049 DEBUG:boto:Token: None 2022-09-02T05:55:57.050 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/?versions 2022-09-02T05:55:57.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D5OsQp/LjpNzdg4OdFom5Rfmvsc= 2022-09-02T05:55:57.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D5OsQp/LjpNzdg4OdFom5Rfmvsc=', 'Content-Length': '0'} 2022-09-02T05:55:57.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a82ba296f0d58ed-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.057 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.061 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.062 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.063 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:57.064 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:57.065 DEBUG:boto:Method: GET 2022-09-02T05:55:57.066 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:55:57.067 DEBUG:boto:Data: 2022-09-02T05:55:57.068 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.069 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.070 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.071 DEBUG:boto:Params: {} 2022-09-02T05:55:57.072 DEBUG:boto:Token: None 2022-09-02T05:55:57.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/?versions 2022-09-02T05:55:57.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D5OsQp/LjpNzdg4OdFom5Rfmvsc= 2022-09-02T05:55:57.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D5OsQp/LjpNzdg4OdFom5Rfmvsc=', 'Content-Length': '0'} 2022-09-02T05:55:57.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbb912b978182a75-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.080 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:57.085 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.087 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.088 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.089 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.090 DEBUG:boto:Method: GET 2022-09-02T05:55:57.091 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.092 DEBUG:boto:Data: 2022-09-02T05:55:57.093 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.095 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.096 DEBUG:boto:Params: {} 2022-09-02T05:55:57.098 DEBUG:boto:Token: None 2022-09-02T05:55:57.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw= 2022-09-02T05:55:57.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw=', 'Content-Length': '0'} 2022-09-02T05:55:57.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94b0e73d81b0951-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.107 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.108 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.109 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.110 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.111 DEBUG:boto:Method: GET 2022-09-02T05:55:57.112 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.113 DEBUG:boto:Data: 2022-09-02T05:55:57.114 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.116 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.117 DEBUG:boto:Params: {} 2022-09-02T05:55:57.118 DEBUG:boto:Token: None 2022-09-02T05:55:57.119 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw= 2022-09-02T05:55:57.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw=', 'Content-Length': '0'} 2022-09-02T05:55:57.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a1bb6f39aca0bb-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.126 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.127 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.128 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.129 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.130 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.131 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.131 DEBUG:boto:Data: 2022-09-02T05:55:57.132 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.133 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.133 DEBUG:boto:Params: {} 2022-09-02T05:55:57.134 DEBUG:boto:Token: None 2022-09-02T05:55:57.135 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O1wxLGFu36Y175q1atiNhN1ogQw= 2022-09-02T05:55:57.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O1wxLGFu36Y175q1atiNhN1ogQw=', 'Content-Length': '0'} 2022-09-02T05:55:57.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c7796f3ed04271-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.139 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.140 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.141 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.141 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.142 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.143 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.143 DEBUG:boto:Data: 2022-09-02T05:55:57.144 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.145 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.145 DEBUG:boto:Params: {} 2022-09-02T05:55:57.146 DEBUG:boto:Token: None 2022-09-02T05:55:57.147 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O1wxLGFu36Y175q1atiNhN1ogQw= 2022-09-02T05:55:57.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O1wxLGFu36Y175q1atiNhN1ogQw=', 'Content-Length': '0'} 2022-09-02T05:55:57.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd21818150f889b7-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:57.162 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.163 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.163 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.164 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.165 DEBUG:boto:Method: GET 2022-09-02T05:55:57.165 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.166 DEBUG:boto:Data: 2022-09-02T05:55:57.166 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.168 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.169 DEBUG:boto:Params: {} 2022-09-02T05:55:57.169 DEBUG:boto:Token: None 2022-09-02T05:55:57.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw= 2022-09-02T05:55:57.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw=', 'Content-Length': '0'} 2022-09-02T05:55:57.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c98a8ad029f1c2-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.222 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.223 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.224 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.224 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.225 DEBUG:boto:Method: GET 2022-09-02T05:55:57.225 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.227 DEBUG:boto:Data: 2022-09-02T05:55:57.228 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.230 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.231 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.232 DEBUG:boto:Params: {} 2022-09-02T05:55:57.234 DEBUG:boto:Token: None 2022-09-02T05:55:57.235 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:55:57.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw= 2022-09-02T05:55:57.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ykpxawY+AypBBrtwe2TH2krspRw=', 'Content-Length': '0'} 2022-09-02T05:55:57.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a592f2658e86b8dd-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.242 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.243 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.244 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.246 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.247 DEBUG:boto:Method: GET 2022-09-02T05:55:57.248 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.249 DEBUG:boto:Data: 2022-09-02T05:55:57.250 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.252 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.253 DEBUG:boto:Params: {} 2022-09-02T05:55:57.255 DEBUG:boto:Token: None 2022-09-02T05:55:57.256 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000887d201efb9d7d81-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.264 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.265 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.266 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.266 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.267 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.267 DEBUG:boto:Method: GET 2022-09-02T05:55:57.268 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.268 DEBUG:boto:Data: 2022-09-02T05:55:57.269 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.270 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.271 DEBUG:boto:Params: {} 2022-09-02T05:55:57.272 DEBUG:boto:Token: None 2022-09-02T05:55:57.272 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8444e4a17d1392d-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.277 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.278 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.279 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.279 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.280 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:57.281 DEBUG:boto:Data: 2022-09-02T05:55:57.281 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.282 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.283 DEBUG:boto:Params: {} 2022-09-02T05:55:57.283 DEBUG:boto:Token: None 2022-09-02T05:55:57.284 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11 2022-09-02T05:55:57.284 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Cm7UYFpiKJxXPhj+26mME1t3Uw= 2022-09-02T05:55:57.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Cm7UYFpiKJxXPhj+26mME1t3Uw=', 'Content-Length': '0'} 2022-09-02T05:55:57.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4590d01b369955a-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.294 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.295 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.296 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.296 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:57.297 DEBUG:boto:Data: 2022-09-02T05:55:57.297 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.299 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.300 DEBUG:boto:Params: {} 2022-09-02T05:55:57.300 DEBUG:boto:Token: None 2022-09-02T05:55:57.301 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11 2022-09-02T05:55:57.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Cm7UYFpiKJxXPhj+26mME1t3Uw= 2022-09-02T05:55:57.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Cm7UYFpiKJxXPhj+26mME1t3Uw=', 'Content-Length': '0'} 2022-09-02T05:55:57.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d236a8e0bc6256b-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:55:57.306 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.307 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.308 DEBUG:boto:Method: GET 2022-09-02T05:55:57.308 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:57.309 DEBUG:boto:Data: 2022-09-02T05:55:57.309 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.310 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.311 DEBUG:boto:Params: {} 2022-09-02T05:55:57.312 DEBUG:boto:Token: None 2022-09-02T05:55:57.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11 2022-09-02T05:55:57.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oguAwqZ/VpLvtFcH6qtajcZaxiE= 2022-09-02T05:55:57.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oguAwqZ/VpLvtFcH6qtajcZaxiE=', 'Content-Length': '0'} 2022-09-02T05:55:57.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b9826e0b5e2791f-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.358 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:55:57.359 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:55:57.359 DEBUG:boto:Method: GET 2022-09-02T05:55:57.360 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:55:57.360 DEBUG:boto:Data: 2022-09-02T05:55:57.361 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.362 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.363 DEBUG:boto:Params: {} 2022-09-02T05:55:57.363 DEBUG:boto:Token: None 2022-09-02T05:55:57.364 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj11 2022-09-02T05:55:57.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oguAwqZ/VpLvtFcH6qtajcZaxiE= 2022-09-02T05:55:57.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oguAwqZ/VpLvtFcH6qtajcZaxiE=', 'Content-Length': '0'} 2022-09-02T05:55:57.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006125585a31050f7-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.369 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.369 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.370 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.370 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.371 DEBUG:boto:Method: GET 2022-09-02T05:55:57.371 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.372 DEBUG:boto:Data: 2022-09-02T05:55:57.372 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.373 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.374 DEBUG:boto:Params: {} 2022-09-02T05:55:57.375 DEBUG:boto:Token: None 2022-09-02T05:55:57.375 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.376 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004052c76bf3faf025-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.380 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.381 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.382 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.382 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.383 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.383 DEBUG:boto:Method: GET 2022-09-02T05:55:57.384 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:55:57.384 DEBUG:boto:Data: 2022-09-02T05:55:57.385 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.386 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.387 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.387 DEBUG:boto:Params: {} 2022-09-02T05:55:57.388 DEBUG:boto:Token: None 2022-09-02T05:55:57.389 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6bde1853d4dedd9-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.395 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:57.397 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.397 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.398 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.398 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.399 DEBUG:boto:Method: GET 2022-09-02T05:55:57.400 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.400 DEBUG:boto:Data: 2022-09-02T05:55:57.401 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.402 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.403 DEBUG:boto:Params: {} 2022-09-02T05:55:57.403 DEBUG:boto:Token: None 2022-09-02T05:55:57.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8= 2022-09-02T05:55:57.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8=', 'Content-Length': '0'} 2022-09-02T05:55:57.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe6b80b68e2de09-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.408 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.409 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.410 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.410 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.411 DEBUG:boto:Method: GET 2022-09-02T05:55:57.411 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.412 DEBUG:boto:Data: 2022-09-02T05:55:57.413 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.414 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.414 DEBUG:boto:Params: {} 2022-09-02T05:55:57.415 DEBUG:boto:Token: None 2022-09-02T05:55:57.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8= 2022-09-02T05:55:57.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8=', 'Content-Length': '0'} 2022-09-02T05:55:57.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357728536e151cc6-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.420 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.420 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.421 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.421 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.422 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.422 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.423 DEBUG:boto:Data: 2022-09-02T05:55:57.423 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.424 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.425 DEBUG:boto:Params: {} 2022-09-02T05:55:57.426 DEBUG:boto:Token: None 2022-09-02T05:55:57.426 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ChvPe/Q3wJiqkM+wf6imsiEVO3Q= 2022-09-02T05:55:57.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ChvPe/Q3wJiqkM+wf6imsiEVO3Q=', 'Content-Length': '0'} 2022-09-02T05:55:57.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c288159fde6a36a1-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.430 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.431 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.431 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.432 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.433 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.433 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.434 DEBUG:boto:Data: 2022-09-02T05:55:57.434 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.435 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.436 DEBUG:boto:Params: {} 2022-09-02T05:55:57.436 DEBUG:boto:Token: None 2022-09-02T05:55:57.437 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ChvPe/Q3wJiqkM+wf6imsiEVO3Q= 2022-09-02T05:55:57.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ChvPe/Q3wJiqkM+wf6imsiEVO3Q=', 'Content-Length': '0'} 2022-09-02T05:55:57.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce919c963945285-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:57.441 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.442 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.442 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.443 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.444 DEBUG:boto:Method: GET 2022-09-02T05:55:57.444 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.445 DEBUG:boto:Data: 2022-09-02T05:55:57.445 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.446 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.447 DEBUG:boto:Params: {} 2022-09-02T05:55:57.447 DEBUG:boto:Token: None 2022-09-02T05:55:57.448 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8= 2022-09-02T05:55:57.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8=', 'Content-Length': '0'} 2022-09-02T05:55:57.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248481fedcd377d1-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.454 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.455 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.456 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.457 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.458 DEBUG:boto:Method: GET 2022-09-02T05:55:57.459 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.460 DEBUG:boto:Data: 2022-09-02T05:55:57.461 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.463 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.464 DEBUG:boto:Params: {} 2022-09-02T05:55:57.465 DEBUG:boto:Token: None 2022-09-02T05:55:57.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:55:57.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8= 2022-09-02T05:55:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bv+dpljwR3CbFzuaI8nCTQBgnS8=', 'Content-Length': '0'} 2022-09-02T05:55:57.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b02b55891d1870-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.510 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.511 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.512 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.512 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.513 DEBUG:boto:Method: GET 2022-09-02T05:55:57.514 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.515 DEBUG:boto:Data: 2022-09-02T05:55:57.516 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.517 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.517 DEBUG:boto:Params: {} 2022-09-02T05:55:57.518 DEBUG:boto:Token: None 2022-09-02T05:55:57.519 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b455a385f3719ce6-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.524 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.525 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.525 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.526 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.527 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.528 DEBUG:boto:Method: GET 2022-09-02T05:55:57.528 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.529 DEBUG:boto:Data: 2022-09-02T05:55:57.530 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.531 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.531 DEBUG:boto:Params: {} 2022-09-02T05:55:57.532 DEBUG:boto:Token: None 2022-09-02T05:55:57.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c44040759b94ee6-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.537 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.538 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.539 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.539 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.540 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:57.541 DEBUG:boto:Data: 2022-09-02T05:55:57.541 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.542 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.543 DEBUG:boto:Params: {} 2022-09-02T05:55:57.544 DEBUG:boto:Token: None 2022-09-02T05:55:57.544 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115 2022-09-02T05:55:57.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/H80F5o+7FMv/QHfsYLsQs5bgvU= 2022-09-02T05:55:57.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/H80F5o+7FMv/QHfsYLsQs5bgvU=', 'Content-Length': '0'} 2022-09-02T05:55:57.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6207dcac5c70c47-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.548 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.549 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.550 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.550 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:57.551 DEBUG:boto:Data: 2022-09-02T05:55:57.551 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.552 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.553 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.553 DEBUG:boto:Params: {} 2022-09-02T05:55:57.554 DEBUG:boto:Token: None 2022-09-02T05:55:57.554 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115 2022-09-02T05:55:57.555 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/H80F5o+7FMv/QHfsYLsQs5bgvU= 2022-09-02T05:55:57.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/H80F5o+7FMv/QHfsYLsQs5bgvU=', 'Content-Length': '0'} 2022-09-02T05:55:57.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f2407b8d082e36-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:55:57.559 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.559 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.560 DEBUG:boto:Method: GET 2022-09-02T05:55:57.561 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:57.561 DEBUG:boto:Data: 2022-09-02T05:55:57.562 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.563 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.564 DEBUG:boto:Params: {} 2022-09-02T05:55:57.564 DEBUG:boto:Token: None 2022-09-02T05:55:57.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115 2022-09-02T05:55:57.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xoqZz2WWcCKVwF/aEOr8DvKJTV0= 2022-09-02T05:55:57.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xoqZz2WWcCKVwF/aEOr8DvKJTV0=', 'Content-Length': '0'} 2022-09-02T05:55:57.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1dc360ff9558a6f-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.610 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:55:57.611 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:55:57.613 DEBUG:boto:Method: GET 2022-09-02T05:55:57.613 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:55:57.615 DEBUG:boto:Data: 2022-09-02T05:55:57.615 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.617 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.618 DEBUG:boto:Params: {} 2022-09-02T05:55:57.620 DEBUG:boto:Token: None 2022-09-02T05:55:57.621 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj115 2022-09-02T05:55:57.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xoqZz2WWcCKVwF/aEOr8DvKJTV0= 2022-09-02T05:55:57.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xoqZz2WWcCKVwF/aEOr8DvKJTV0=', 'Content-Length': '0'} 2022-09-02T05:55:57.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d855aab05f1f16e9-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.627 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.628 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.629 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.630 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.631 DEBUG:boto:Method: GET 2022-09-02T05:55:57.632 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.633 DEBUG:boto:Data: 2022-09-02T05:55:57.634 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.636 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.637 DEBUG:boto:Params: {} 2022-09-02T05:55:57.638 DEBUG:boto:Token: None 2022-09-02T05:55:57.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cc5f36ddcca696e-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.646 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.647 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.648 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.648 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.649 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.650 DEBUG:boto:Method: GET 2022-09-02T05:55:57.650 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:55:57.651 DEBUG:boto:Data: 2022-09-02T05:55:57.651 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.652 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.653 DEBUG:boto:Params: {} 2022-09-02T05:55:57.654 DEBUG:boto:Token: None 2022-09-02T05:55:57.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a883371fde7b77-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.660 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:57.662 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.662 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.663 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.664 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.664 DEBUG:boto:Method: GET 2022-09-02T05:55:57.665 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.665 DEBUG:boto:Data: 2022-09-02T05:55:57.666 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.668 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.668 DEBUG:boto:Params: {} 2022-09-02T05:55:57.669 DEBUG:boto:Token: None 2022-09-02T05:55:57.670 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw= 2022-09-02T05:55:57.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw=', 'Content-Length': '0'} 2022-09-02T05:55:57.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b091cebfa6f7d99-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.674 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.675 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.676 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.676 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.677 DEBUG:boto:Method: GET 2022-09-02T05:55:57.677 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.678 DEBUG:boto:Data: 2022-09-02T05:55:57.679 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.680 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.681 DEBUG:boto:Params: {} 2022-09-02T05:55:57.682 DEBUG:boto:Token: None 2022-09-02T05:55:57.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw= 2022-09-02T05:55:57.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw=', 'Content-Length': '0'} 2022-09-02T05:55:57.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c07c71955225abe-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.688 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.688 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.690 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.691 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.692 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.693 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.694 DEBUG:boto:Data: 2022-09-02T05:55:57.695 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.696 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.697 DEBUG:boto:Params: {} 2022-09-02T05:55:57.698 DEBUG:boto:Token: None 2022-09-02T05:55:57.698 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.699 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ay+QRK4x1T1+eF51ZnmsWMWjimM= 2022-09-02T05:55:57.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ay+QRK4x1T1+eF51ZnmsWMWjimM=', 'Content-Length': '0'} 2022-09-02T05:55:57.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf21274336afb540-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.703 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.704 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.704 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.705 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.705 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.706 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.706 DEBUG:boto:Data: 2022-09-02T05:55:57.707 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.707 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.708 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.709 DEBUG:boto:Params: {} 2022-09-02T05:55:57.709 DEBUG:boto:Token: None 2022-09-02T05:55:57.710 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ay+QRK4x1T1+eF51ZnmsWMWjimM= 2022-09-02T05:55:57.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ay+QRK4x1T1+eF51ZnmsWMWjimM=', 'Content-Length': '0'} 2022-09-02T05:55:57.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082c17e104f72f249-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:57.715 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.716 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.716 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.717 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.717 DEBUG:boto:Method: GET 2022-09-02T05:55:57.718 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.718 DEBUG:boto:Data: 2022-09-02T05:55:57.719 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.720 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.721 DEBUG:boto:Params: {} 2022-09-02T05:55:57.722 DEBUG:boto:Token: None 2022-09-02T05:55:57.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw= 2022-09-02T05:55:57.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw=', 'Content-Length': '0'} 2022-09-02T05:55:57.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b1d0b80b373870-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.727 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.728 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.728 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.729 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.730 DEBUG:boto:Method: GET 2022-09-02T05:55:57.730 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.731 DEBUG:boto:Data: 2022-09-02T05:55:57.732 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.733 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.734 DEBUG:boto:Params: {} 2022-09-02T05:55:57.734 DEBUG:boto:Token: None 2022-09-02T05:55:57.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:55:57.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw= 2022-09-02T05:55:57.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r/Lt4GTMW6hs3FAwf0VLwIAmCJw=', 'Content-Length': '0'} 2022-09-02T05:55:57.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a014032883695498-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.788 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.789 DEBUG:boto:Method: GET 2022-09-02T05:55:57.789 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.790 DEBUG:boto:Data: 2022-09-02T05:55:57.790 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.792 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.793 DEBUG:boto:Params: {} 2022-09-02T05:55:57.793 DEBUG:boto:Token: None 2022-09-02T05:55:57.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.795 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf2d752aa61c02c8-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.800 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.802 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.803 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.803 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.804 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.805 DEBUG:boto:Method: GET 2022-09-02T05:55:57.805 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.806 DEBUG:boto:Data: 2022-09-02T05:55:57.806 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.807 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.808 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.808 DEBUG:boto:Params: {} 2022-09-02T05:55:57.809 DEBUG:boto:Token: None 2022-09-02T05:55:57.810 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9af5c1748d31ee3-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.815 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.816 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.817 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.817 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.818 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:57.818 DEBUG:boto:Data: 2022-09-02T05:55:57.819 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.820 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.820 DEBUG:boto:Params: {} 2022-09-02T05:55:57.821 DEBUG:boto:Token: None 2022-09-02T05:55:57.821 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116 2022-09-02T05:55:57.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i05ADA4LVuWSMiG29bkikBQyFkA= 2022-09-02T05:55:57.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i05ADA4LVuWSMiG29bkikBQyFkA=', 'Content-Length': '0'} 2022-09-02T05:55:57.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052cc4197f49d1976-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.827 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.827 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.828 DEBUG:boto:Method: HEAD 2022-09-02T05:55:57.829 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:57.829 DEBUG:boto:Data: 2022-09-02T05:55:57.830 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.831 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.832 DEBUG:boto:Params: {} 2022-09-02T05:55:57.833 DEBUG:boto:Token: None 2022-09-02T05:55:57.833 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116 2022-09-02T05:55:57.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i05ADA4LVuWSMiG29bkikBQyFkA= 2022-09-02T05:55:57.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i05ADA4LVuWSMiG29bkikBQyFkA=', 'Content-Length': '0'} 2022-09-02T05:55:57.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f808b02224a86207-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:55:57.840 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.842 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.843 DEBUG:boto:Method: GET 2022-09-02T05:55:57.844 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:57.845 DEBUG:boto:Data: 2022-09-02T05:55:57.846 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.848 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.849 DEBUG:boto:Params: {} 2022-09-02T05:55:57.850 DEBUG:boto:Token: None 2022-09-02T05:55:57.851 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116 2022-09-02T05:55:57.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UMjAiv1D3qdMiQD+OiwQ2zglCWY= 2022-09-02T05:55:57.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UMjAiv1D3qdMiQD+OiwQ2zglCWY=', 'Content-Length': '0'} 2022-09-02T05:55:57.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fada82765542ad-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.855 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:55:57.856 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:55:57.857 DEBUG:boto:Method: GET 2022-09-02T05:55:57.857 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:55:57.858 DEBUG:boto:Data: 2022-09-02T05:55:57.859 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.860 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.861 DEBUG:boto:Params: {} 2022-09-02T05:55:57.862 DEBUG:boto:Token: None 2022-09-02T05:55:57.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj116 2022-09-02T05:55:57.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UMjAiv1D3qdMiQD+OiwQ2zglCWY= 2022-09-02T05:55:57.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UMjAiv1D3qdMiQD+OiwQ2zglCWY=', 'Content-Length': '0'} 2022-09-02T05:55:57.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a870727cbb198773-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.867 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.868 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.868 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.869 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.869 DEBUG:boto:Method: GET 2022-09-02T05:55:57.870 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.871 DEBUG:boto:Data: 2022-09-02T05:55:57.871 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.873 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.873 DEBUG:boto:Params: {} 2022-09-02T05:55:57.874 DEBUG:boto:Token: None 2022-09-02T05:55:57.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c794e983648d27b8-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.922 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.924 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:57.925 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:57.925 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.926 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.926 DEBUG:boto:Method: GET 2022-09-02T05:55:57.927 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:55:57.927 DEBUG:boto:Data: 2022-09-02T05:55:57.928 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.929 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.930 DEBUG:boto:Params: {} 2022-09-02T05:55:57.930 DEBUG:boto:Token: None 2022-09-02T05:55:57.931 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/ 2022-09-02T05:55:57.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw= 2022-09-02T05:55:57.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WGqg65TJB7ObcPV002ra7gW9+Gw=', 'Content-Length': '0'} 2022-09-02T05:55:57.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000604d05e7c04ad670-0063119aed-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.936 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:57.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:57.938 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:57.939 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:57.939 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.940 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.940 DEBUG:boto:Method: GET 2022-09-02T05:55:57.941 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.941 DEBUG:boto:Data: 2022-09-02T05:55:57.942 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:57.943 DEBUG:boto:Port: 8001 2022-09-02T05:55:57.944 DEBUG:boto:Params: {} 2022-09-02T05:55:57.944 DEBUG:boto:Token: None 2022-09-02T05:55:57.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lNOf/BCpmBPV3L/QEAbU0yMLmCY= 2022-09-02T05:55:57.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lNOf/BCpmBPV3L/QEAbU0yMLmCY=', 'Content-Length': '0'} 2022-09-02T05:55:57.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f20dfb791fd9dd5-0063119aed-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:57.949 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:57.950 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:57.950 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.951 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.951 DEBUG:boto:Method: GET 2022-09-02T05:55:57.952 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.953 DEBUG:boto:Data: 2022-09-02T05:55:57.953 DEBUG:boto:Headers: {} 2022-09-02T05:55:57.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:57.954 DEBUG:boto:Port: 8000 2022-09-02T05:55:57.955 DEBUG:boto:Params: {} 2022-09-02T05:55:57.955 DEBUG:boto:Token: None 2022-09-02T05:55:57.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:57 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:57.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lNOf/BCpmBPV3L/QEAbU0yMLmCY= 2022-09-02T05:55:57.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lNOf/BCpmBPV3L/QEAbU0yMLmCY=', 'Content-Length': '0'} 2022-09-02T05:55:57.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f845fed396c04d-0063119aed-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.002 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.003 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.003 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.004 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.004 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.005 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.005 DEBUG:boto:Data: 2022-09-02T05:55:58.006 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.007 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.007 DEBUG:boto:Params: {} 2022-09-02T05:55:58.008 DEBUG:boto:Token: None 2022-09-02T05:55:58.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPyxjXJvfdRtQESQKZhnM3SN0ec= 2022-09-02T05:55:58.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPyxjXJvfdRtQESQKZhnM3SN0ec=', 'Content-Length': '0'} 2022-09-02T05:55:58.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc91f960d4d21b5d-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.013 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.014 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.015 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.015 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.016 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.016 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.017 DEBUG:boto:Data: 2022-09-02T05:55:58.017 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.018 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.019 DEBUG:boto:Params: {} 2022-09-02T05:55:58.019 DEBUG:boto:Token: None 2022-09-02T05:55:58.020 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XPyxjXJvfdRtQESQKZhnM3SN0ec= 2022-09-02T05:55:58.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XPyxjXJvfdRtQESQKZhnM3SN0ec=', 'Content-Length': '0'} 2022-09-02T05:55:58.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95a56858e66c286-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:58.025 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.025 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.026 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.027 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.027 DEBUG:boto:Method: GET 2022-09-02T05:55:58.028 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.028 DEBUG:boto:Data: 2022-09-02T05:55:58.029 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.030 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.030 DEBUG:boto:Params: {} 2022-09-02T05:55:58.031 DEBUG:boto:Token: None 2022-09-02T05:55:58.032 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.032 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gmyRTixKVV2+VTjaP461Er6gDFo= 2022-09-02T05:55:58.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gmyRTixKVV2+VTjaP461Er6gDFo=', 'Content-Length': '0'} 2022-09-02T05:55:58.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005207d987ba96d40d-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.036 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.037 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.038 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.038 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.039 DEBUG:boto:Method: GET 2022-09-02T05:55:58.039 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.040 DEBUG:boto:Data: 2022-09-02T05:55:58.040 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.041 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.041 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.042 DEBUG:boto:Params: {} 2022-09-02T05:55:58.043 DEBUG:boto:Token: None 2022-09-02T05:55:58.043 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:55:58.044 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gmyRTixKVV2+VTjaP461Er6gDFo= 2022-09-02T05:55:58.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gmyRTixKVV2+VTjaP461Er6gDFo=', 'Content-Length': '0'} 2022-09-02T05:55:58.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000657841dfde7a196d-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.048 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.048 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.049 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.050 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.050 DEBUG:boto:Method: GET 2022-09-02T05:55:58.051 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.051 DEBUG:boto:Data: 2022-09-02T05:55:58.052 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.053 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.053 DEBUG:boto:Params: {} 2022-09-02T05:55:58.054 DEBUG:boto:Token: None 2022-09-02T05:55:58.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b2abc6998c6cd49-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.059 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.060 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.061 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.061 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.062 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.062 DEBUG:boto:Method: GET 2022-09-02T05:55:58.063 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.064 DEBUG:boto:Data: 2022-09-02T05:55:58.064 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.065 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.065 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.066 DEBUG:boto:Params: {} 2022-09-02T05:55:58.066 DEBUG:boto:Token: None 2022-09-02T05:55:58.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000076e039671a35646-0063119aee-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.072 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.074 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.075 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.075 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.076 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:58.077 DEBUG:boto:Data: 2022-09-02T05:55:58.077 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.078 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.079 DEBUG:boto:Params: {} 2022-09-02T05:55:58.079 DEBUG:boto:Token: None 2022-09-02T05:55:58.080 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145 2022-09-02T05:55:58.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5zpG19B+r6Jat7+dCxM/nQAhEs= 2022-09-02T05:55:58.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5zpG19B+r6Jat7+dCxM/nQAhEs=', 'Content-Length': '0'} 2022-09-02T05:55:58.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ac2865927201e1-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.084 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.085 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.085 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.086 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:58.086 DEBUG:boto:Data: 2022-09-02T05:55:58.087 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.088 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.088 DEBUG:boto:Params: {} 2022-09-02T05:55:58.089 DEBUG:boto:Token: None 2022-09-02T05:55:58.089 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145 2022-09-02T05:55:58.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j5zpG19B+r6Jat7+dCxM/nQAhEs= 2022-09-02T05:55:58.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j5zpG19B+r6Jat7+dCxM/nQAhEs=', 'Content-Length': '0'} 2022-09-02T05:55:58.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11645df5634f810-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:55:58.094 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.094 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.095 DEBUG:boto:Method: GET 2022-09-02T05:55:58.095 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:58.096 DEBUG:boto:Data: 2022-09-02T05:55:58.097 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.098 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.098 DEBUG:boto:Params: {} 2022-09-02T05:55:58.099 DEBUG:boto:Token: None 2022-09-02T05:55:58.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145 2022-09-02T05:55:58.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s043kG1tuL9sz/buvMuFMWFW6YA= 2022-09-02T05:55:58.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s043kG1tuL9sz/buvMuFMWFW6YA=', 'Content-Length': '0'} 2022-09-02T05:55:58.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a42b1e79866fbb-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.103 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:55:58.104 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:55:58.104 DEBUG:boto:Method: GET 2022-09-02T05:55:58.105 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:55:58.106 DEBUG:boto:Data: 2022-09-02T05:55:58.106 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.107 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.108 DEBUG:boto:Params: {} 2022-09-02T05:55:58.108 DEBUG:boto:Token: None 2022-09-02T05:55:58.109 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj145 2022-09-02T05:55:58.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s043kG1tuL9sz/buvMuFMWFW6YA= 2022-09-02T05:55:58.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s043kG1tuL9sz/buvMuFMWFW6YA=', 'Content-Length': '0'} 2022-09-02T05:55:58.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006562cdcfdde25cb7-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.154 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.155 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.156 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.156 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.157 DEBUG:boto:Method: GET 2022-09-02T05:55:58.158 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.159 DEBUG:boto:Data: 2022-09-02T05:55:58.159 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.160 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.161 DEBUG:boto:Params: {} 2022-09-02T05:55:58.162 DEBUG:boto:Token: None 2022-09-02T05:55:58.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b037fd47974f0f-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.167 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.168 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.169 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.169 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.170 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.170 DEBUG:boto:Method: GET 2022-09-02T05:55:58.171 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:55:58.172 DEBUG:boto:Data: 2022-09-02T05:55:58.172 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.173 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.173 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.174 DEBUG:boto:Params: {} 2022-09-02T05:55:58.174 DEBUG:boto:Token: None 2022-09-02T05:55:58.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc8699965f833792-0063119aee-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.180 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:58.181 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.182 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.183 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.183 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.184 DEBUG:boto:Method: GET 2022-09-02T05:55:58.184 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.185 DEBUG:boto:Data: 2022-09-02T05:55:58.186 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.187 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.187 DEBUG:boto:Params: {} 2022-09-02T05:55:58.188 DEBUG:boto:Token: None 2022-09-02T05:55:58.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck= 2022-09-02T05:55:58.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck=', 'Content-Length': '0'} 2022-09-02T05:55:58.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024a735a428c9164c-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.234 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.235 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.235 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.236 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.237 DEBUG:boto:Method: GET 2022-09-02T05:55:58.238 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.238 DEBUG:boto:Data: 2022-09-02T05:55:58.239 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.239 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.240 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.240 DEBUG:boto:Params: {} 2022-09-02T05:55:58.241 DEBUG:boto:Token: None 2022-09-02T05:55:58.242 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck= 2022-09-02T05:55:58.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck=', 'Content-Length': '0'} 2022-09-02T05:55:58.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f73a8baa6c686e8-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.247 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.247 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.248 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.249 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.249 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.251 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.252 DEBUG:boto:Data: 2022-09-02T05:55:58.252 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.254 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.256 DEBUG:boto:Params: {} 2022-09-02T05:55:58.257 DEBUG:boto:Token: None 2022-09-02T05:55:58.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xIfMIOA+YpOeTZ16M0yOW5yL7I0= 2022-09-02T05:55:58.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xIfMIOA+YpOeTZ16M0yOW5yL7I0=', 'Content-Length': '0'} 2022-09-02T05:55:58.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061eb9897dd314ef9-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.265 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.267 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.268 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.269 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.270 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.271 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.272 DEBUG:boto:Data: 2022-09-02T05:55:58.273 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.275 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.276 DEBUG:boto:Params: {} 2022-09-02T05:55:58.277 DEBUG:boto:Token: None 2022-09-02T05:55:58.278 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xIfMIOA+YpOeTZ16M0yOW5yL7I0= 2022-09-02T05:55:58.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xIfMIOA+YpOeTZ16M0yOW5yL7I0=', 'Content-Length': '0'} 2022-09-02T05:55:58.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1e2a79507741c5-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:58.286 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.287 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.288 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.289 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.290 DEBUG:boto:Method: GET 2022-09-02T05:55:58.291 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.292 DEBUG:boto:Data: 2022-09-02T05:55:58.293 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.295 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.296 DEBUG:boto:Params: {} 2022-09-02T05:55:58.297 DEBUG:boto:Token: None 2022-09-02T05:55:58.298 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck= 2022-09-02T05:55:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck=', 'Content-Length': '0'} 2022-09-02T05:55:58.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cdd5fc42001a8bf-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.303 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.303 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.304 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.304 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.305 DEBUG:boto:Method: GET 2022-09-02T05:55:58.306 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.306 DEBUG:boto:Data: 2022-09-02T05:55:58.307 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.307 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.308 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.308 DEBUG:boto:Params: {} 2022-09-02T05:55:58.309 DEBUG:boto:Token: None 2022-09-02T05:55:58.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:55:58.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck= 2022-09-02T05:55:58.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t3I+zUJqL4ncBpK22k6zqmROEck=', 'Content-Length': '0'} 2022-09-02T05:55:58.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2d02e0d97dbd9a-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.315 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.315 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.316 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.316 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.317 DEBUG:boto:Method: GET 2022-09-02T05:55:58.317 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.318 DEBUG:boto:Data: 2022-09-02T05:55:58.319 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.320 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.320 DEBUG:boto:Params: {} 2022-09-02T05:55:58.321 DEBUG:boto:Token: None 2022-09-02T05:55:58.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ce6aea90610afcc-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.374 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.376 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.377 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.377 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.378 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.379 DEBUG:boto:Method: GET 2022-09-02T05:55:58.380 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.381 DEBUG:boto:Data: 2022-09-02T05:55:58.381 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.382 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.383 DEBUG:boto:Params: {} 2022-09-02T05:55:58.384 DEBUG:boto:Token: None 2022-09-02T05:55:58.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b4c0e55af537ab-0063119aee-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.390 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.391 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.391 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.392 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.393 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:58.393 DEBUG:boto:Data: 2022-09-02T05:55:58.394 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.395 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.395 DEBUG:boto:Params: {} 2022-09-02T05:55:58.396 DEBUG:boto:Token: None 2022-09-02T05:55:58.397 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17 2022-09-02T05:55:58.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KxhIRxMwXX3UcaL6aMK+ABsE8zc= 2022-09-02T05:55:58.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KxhIRxMwXX3UcaL6aMK+ABsE8zc=', 'Content-Length': '0'} 2022-09-02T05:55:58.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000340b84e0f057588-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.402 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.403 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.403 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.404 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:58.404 DEBUG:boto:Data: 2022-09-02T05:55:58.405 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.405 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.406 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.406 DEBUG:boto:Params: {} 2022-09-02T05:55:58.407 DEBUG:boto:Token: None 2022-09-02T05:55:58.408 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17 2022-09-02T05:55:58.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KxhIRxMwXX3UcaL6aMK+ABsE8zc= 2022-09-02T05:55:58.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KxhIRxMwXX3UcaL6aMK+ABsE8zc=', 'Content-Length': '0'} 2022-09-02T05:55:58.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4ed3f083401607-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:55:58.413 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.413 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.414 DEBUG:boto:Method: GET 2022-09-02T05:55:58.415 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:58.415 DEBUG:boto:Data: 2022-09-02T05:55:58.416 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.417 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.417 DEBUG:boto:Params: {} 2022-09-02T05:55:58.418 DEBUG:boto:Token: None 2022-09-02T05:55:58.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17 2022-09-02T05:55:58.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+478ukrdopYV9aw58pfSV9oWKwU= 2022-09-02T05:55:58.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+478ukrdopYV9aw58pfSV9oWKwU=', 'Content-Length': '0'} 2022-09-02T05:55:58.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2a3fdd2ee4dd0f4-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.423 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:55:58.423 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:55:58.424 DEBUG:boto:Method: GET 2022-09-02T05:55:58.424 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:55:58.425 DEBUG:boto:Data: 2022-09-02T05:55:58.425 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.426 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.426 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.427 DEBUG:boto:Params: {} 2022-09-02T05:55:58.427 DEBUG:boto:Token: None 2022-09-02T05:55:58.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj17 2022-09-02T05:55:58.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+478ukrdopYV9aw58pfSV9oWKwU= 2022-09-02T05:55:58.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+478ukrdopYV9aw58pfSV9oWKwU=', 'Content-Length': '0'} 2022-09-02T05:55:58.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fe25e6f4dedf2d-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.432 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.433 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.433 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.434 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.434 DEBUG:boto:Method: GET 2022-09-02T05:55:58.435 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.435 DEBUG:boto:Data: 2022-09-02T05:55:58.436 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.437 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.437 DEBUG:boto:Params: {} 2022-09-02T05:55:58.438 DEBUG:boto:Token: None 2022-09-02T05:55:58.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007624f3a28f5c55e6-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.443 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.444 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.445 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.445 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.446 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.446 DEBUG:boto:Method: GET 2022-09-02T05:55:58.447 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:55:58.447 DEBUG:boto:Data: 2022-09-02T05:55:58.448 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.449 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.450 DEBUG:boto:Params: {} 2022-09-02T05:55:58.451 DEBUG:boto:Token: None 2022-09-02T05:55:58.451 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.452 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0569747c8755b79-0063119aee-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.456 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:58.458 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.458 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.459 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.460 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.460 DEBUG:boto:Method: GET 2022-09-02T05:55:58.461 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.462 DEBUG:boto:Data: 2022-09-02T05:55:58.462 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.463 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.464 DEBUG:boto:Params: {} 2022-09-02T05:55:58.465 DEBUG:boto:Token: None 2022-09-02T05:55:58.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo= 2022-09-02T05:55:58.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo=', 'Content-Length': '0'} 2022-09-02T05:55:58.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4423161fa6b3902-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.470 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.471 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.471 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.472 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.472 DEBUG:boto:Method: GET 2022-09-02T05:55:58.473 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.473 DEBUG:boto:Data: 2022-09-02T05:55:58.474 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.475 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.475 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.476 DEBUG:boto:Params: {} 2022-09-02T05:55:58.477 DEBUG:boto:Token: None 2022-09-02T05:55:58.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo= 2022-09-02T05:55:58.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo=', 'Content-Length': '0'} 2022-09-02T05:55:58.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b33c25839698293-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.526 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.527 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.528 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.528 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.529 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.529 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.530 DEBUG:boto:Data: 2022-09-02T05:55:58.530 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.531 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.532 DEBUG:boto:Params: {} 2022-09-02T05:55:58.533 DEBUG:boto:Token: None 2022-09-02T05:55:58.533 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zUjVgfUe0NAZaDg7s46pWTFMDho= 2022-09-02T05:55:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zUjVgfUe0NAZaDg7s46pWTFMDho=', 'Content-Length': '0'} 2022-09-02T05:55:58.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096890b17244c5f99-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.538 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.538 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.539 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.539 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.540 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.540 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.541 DEBUG:boto:Data: 2022-09-02T05:55:58.542 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.542 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.543 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.543 DEBUG:boto:Params: {} 2022-09-02T05:55:58.544 DEBUG:boto:Token: None 2022-09-02T05:55:58.544 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.545 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zUjVgfUe0NAZaDg7s46pWTFMDho= 2022-09-02T05:55:58.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zUjVgfUe0NAZaDg7s46pWTFMDho=', 'Content-Length': '0'} 2022-09-02T05:55:58.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37170fb4b9ed4fc-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:58.549 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.550 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.550 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.551 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.551 DEBUG:boto:Method: GET 2022-09-02T05:55:58.552 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.552 DEBUG:boto:Data: 2022-09-02T05:55:58.553 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.554 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.554 DEBUG:boto:Params: {} 2022-09-02T05:55:58.555 DEBUG:boto:Token: None 2022-09-02T05:55:58.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo= 2022-09-02T05:55:58.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo=', 'Content-Length': '0'} 2022-09-02T05:55:58.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe32007890de694-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.560 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.561 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.561 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.562 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.562 DEBUG:boto:Method: GET 2022-09-02T05:55:58.563 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.563 DEBUG:boto:Data: 2022-09-02T05:55:58.564 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.565 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.565 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.566 DEBUG:boto:Params: {} 2022-09-02T05:55:58.567 DEBUG:boto:Token: None 2022-09-02T05:55:58.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:55:58.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo= 2022-09-02T05:55:58.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cbbpOZ4QfiaoNbZXZPyhk2AP/zo=', 'Content-Length': '0'} 2022-09-02T05:55:58.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65ac69309de5de9-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.642 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.643 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.644 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.644 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.645 DEBUG:boto:Method: GET 2022-09-02T05:55:58.646 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.647 DEBUG:boto:Data: 2022-09-02T05:55:58.648 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.649 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.650 DEBUG:boto:Params: {} 2022-09-02T05:55:58.651 DEBUG:boto:Token: None 2022-09-02T05:55:58.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005413ddd97845ca35-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.657 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.659 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.659 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.660 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.660 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.661 DEBUG:boto:Method: GET 2022-09-02T05:55:58.661 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.662 DEBUG:boto:Data: 2022-09-02T05:55:58.662 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.663 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.663 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.664 DEBUG:boto:Params: {} 2022-09-02T05:55:58.665 DEBUG:boto:Token: None 2022-09-02T05:55:58.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c45f650595936e-0063119aee-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.670 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.671 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.672 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.672 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.673 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:58.673 DEBUG:boto:Data: 2022-09-02T05:55:58.674 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.675 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.675 DEBUG:boto:Params: {} 2022-09-02T05:55:58.676 DEBUG:boto:Token: None 2022-09-02T05:55:58.677 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19 2022-09-02T05:55:58.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yhmm1WoOHiecBoFqP+M74FJiWP4= 2022-09-02T05:55:58.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yhmm1WoOHiecBoFqP+M74FJiWP4=', 'Content-Length': '0'} 2022-09-02T05:55:58.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2216e86d93c700d-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.681 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.682 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.682 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.683 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:58.684 DEBUG:boto:Data: 2022-09-02T05:55:58.685 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.686 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.687 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.687 DEBUG:boto:Params: {} 2022-09-02T05:55:58.688 DEBUG:boto:Token: None 2022-09-02T05:55:58.689 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19 2022-09-02T05:55:58.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yhmm1WoOHiecBoFqP+M74FJiWP4= 2022-09-02T05:55:58.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yhmm1WoOHiecBoFqP+M74FJiWP4=', 'Content-Length': '0'} 2022-09-02T05:55:58.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f132ed12c9bb99-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:55:58.694 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.695 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.696 DEBUG:boto:Method: GET 2022-09-02T05:55:58.697 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:58.698 DEBUG:boto:Data: 2022-09-02T05:55:58.699 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.702 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.703 DEBUG:boto:Params: {} 2022-09-02T05:55:58.704 DEBUG:boto:Token: None 2022-09-02T05:55:58.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19 2022-09-02T05:55:58.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MptkuIl8HYvhHFleeNYnELFFOWc= 2022-09-02T05:55:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MptkuIl8HYvhHFleeNYnELFFOWc=', 'Content-Length': '0'} 2022-09-02T05:55:58.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f74f6abf209b3a2-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.710 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:55:58.711 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:55:58.711 DEBUG:boto:Method: GET 2022-09-02T05:55:58.712 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:55:58.712 DEBUG:boto:Data: 2022-09-02T05:55:58.713 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.715 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.715 DEBUG:boto:Params: {} 2022-09-02T05:55:58.716 DEBUG:boto:Token: None 2022-09-02T05:55:58.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj19 2022-09-02T05:55:58.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MptkuIl8HYvhHFleeNYnELFFOWc= 2022-09-02T05:55:58.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MptkuIl8HYvhHFleeNYnELFFOWc=', 'Content-Length': '0'} 2022-09-02T05:55:58.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cceaa6d54f5146a-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.723 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.724 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.725 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.726 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.726 DEBUG:boto:Method: GET 2022-09-02T05:55:58.727 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.728 DEBUG:boto:Data: 2022-09-02T05:55:58.728 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.729 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.730 DEBUG:boto:Params: {} 2022-09-02T05:55:58.731 DEBUG:boto:Token: None 2022-09-02T05:55:58.732 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce93e50b652e980-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.738 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.739 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.739 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.740 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.742 DEBUG:boto:Method: GET 2022-09-02T05:55:58.743 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:55:58.744 DEBUG:boto:Data: 2022-09-02T05:55:58.745 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.747 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.748 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.750 DEBUG:boto:Params: {} 2022-09-02T05:55:58.751 DEBUG:boto:Token: None 2022-09-02T05:55:58.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000313a82daabe05873-0063119aee-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.802 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:58.806 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:58.807 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:58.809 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.810 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.812 DEBUG:boto:Method: GET 2022-09-02T05:55:58.814 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.815 DEBUG:boto:Data: 2022-09-02T05:55:58.816 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.819 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.821 DEBUG:boto:Params: {} 2022-09-02T05:55:58.823 DEBUG:boto:Token: None 2022-09-02T05:55:58.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ= 2022-09-02T05:55:58.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ=', 'Content-Length': '0'} 2022-09-02T05:55:58.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b92b4066a89b816-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.831 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:58.832 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:58.833 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.833 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.834 DEBUG:boto:Method: GET 2022-09-02T05:55:58.834 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.835 DEBUG:boto:Data: 2022-09-02T05:55:58.835 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.836 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.837 DEBUG:boto:Params: {} 2022-09-02T05:55:58.838 DEBUG:boto:Token: None 2022-09-02T05:55:58.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ= 2022-09-02T05:55:58.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ=', 'Content-Length': '0'} 2022-09-02T05:55:58.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f44bf0797ac2bc-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.844 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:58.845 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:58.845 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.846 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.847 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.847 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.848 DEBUG:boto:Data: 2022-09-02T05:55:58.848 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.850 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.850 DEBUG:boto:Params: {} 2022-09-02T05:55:58.851 DEBUG:boto:Token: None 2022-09-02T05:55:58.852 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LAF6ixGo58dyCn4Wb1oHFl64DmM= 2022-09-02T05:55:58.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LAF6ixGo58dyCn4Wb1oHFl64DmM=', 'Content-Length': '0'} 2022-09-02T05:55:58.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ace550f356270d-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.858 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:58.860 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:58.861 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.862 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.863 DEBUG:boto:Method: HEAD 2022-09-02T05:55:58.864 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.865 DEBUG:boto:Data: 2022-09-02T05:55:58.866 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.867 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.868 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.869 DEBUG:boto:Params: {} 2022-09-02T05:55:58.871 DEBUG:boto:Token: None 2022-09-02T05:55:58.872 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LAF6ixGo58dyCn4Wb1oHFl64DmM= 2022-09-02T05:55:58.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LAF6ixGo58dyCn4Wb1oHFl64DmM=', 'Content-Length': '0'} 2022-09-02T05:55:58.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcdc2663bf4c1a69-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:58.884 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:58.885 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:58.887 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.888 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.889 DEBUG:boto:Method: GET 2022-09-02T05:55:58.890 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.891 DEBUG:boto:Data: 2022-09-02T05:55:58.892 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.895 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.896 DEBUG:boto:Params: {} 2022-09-02T05:55:58.897 DEBUG:boto:Token: None 2022-09-02T05:55:58.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ= 2022-09-02T05:55:58.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ=', 'Content-Length': '0'} 2022-09-02T05:55:58.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268708a63c761ccc-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.905 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:58.907 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:58.908 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.909 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.910 DEBUG:boto:Method: GET 2022-09-02T05:55:58.911 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.912 DEBUG:boto:Data: 2022-09-02T05:55:58.913 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.916 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.917 DEBUG:boto:Params: {} 2022-09-02T05:55:58.919 DEBUG:boto:Token: None 2022-09-02T05:55:58.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:55:58.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ= 2022-09-02T05:55:58.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Iysmh6jtzORbjJDLUjB4jIbXFgQ=', 'Content-Length': '0'} 2022-09-02T05:55:58.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034093a3fc37a4f2c-0063119aee-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.970 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.971 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.972 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:58.973 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:58.974 DEBUG:boto:Method: GET 2022-09-02T05:55:58.974 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:58.975 DEBUG:boto:Data: 2022-09-02T05:55:58.975 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:58.977 DEBUG:boto:Port: 8001 2022-09-02T05:55:58.977 DEBUG:boto:Params: {} 2022-09-02T05:55:58.978 DEBUG:boto:Token: None 2022-09-02T05:55:58.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:58.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:58.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e552b38d1bcccb1e-0063119aee-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:58.984 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:58.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:58.986 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:58.987 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:58.987 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:58.988 DEBUG:boto:Method: GET 2022-09-02T05:55:58.990 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:58.991 DEBUG:boto:Data: 2022-09-02T05:55:58.992 DEBUG:boto:Headers: {} 2022-09-02T05:55:58.993 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:58.995 DEBUG:boto:Port: 8000 2022-09-02T05:55:58.995 DEBUG:boto:Params: {} 2022-09-02T05:55:58.997 DEBUG:boto:Token: None 2022-09-02T05:55:58.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:58 GMT /xljbkf-74/ 2022-09-02T05:55:58.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0= 2022-09-02T05:55:59.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Oucc7Gycuma4K2RSeNsAY93pbX0=', 'Content-Length': '0'} 2022-09-02T05:55:59.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005485ef7bf8450b23-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.004 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.005 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:59.006 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:59.006 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.007 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:59.007 DEBUG:boto:Data: 2022-09-02T05:55:59.008 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.009 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.010 DEBUG:boto:Params: {} 2022-09-02T05:55:59.011 DEBUG:boto:Token: None 2022-09-02T05:55:59.011 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj22 2022-09-02T05:55:59.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FM/4aXNftWwPTlgQe0v6dMqei4M= 2022-09-02T05:55:59.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FM/4aXNftWwPTlgQe0v6dMqei4M=', 'Content-Length': '0'} 2022-09-02T05:55:59.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa673571326f629-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.015 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:59.017 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:59.018 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.019 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:59.020 DEBUG:boto:Data: 2022-09-02T05:55:59.021 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.022 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.023 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.024 DEBUG:boto:Params: {} 2022-09-02T05:55:59.026 DEBUG:boto:Token: None 2022-09-02T05:55:59.027 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj22 2022-09-02T05:55:59.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FM/4aXNftWwPTlgQe0v6dMqei4M= 2022-09-02T05:55:59.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FM/4aXNftWwPTlgQe0v6dMqei4M=', 'Content-Length': '0'} 2022-09-02T05:55:59.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007915a269925eb370-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:55:59.033 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:59.034 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:59.035 DEBUG:boto:Method: GET 2022-09-02T05:55:59.036 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:59.037 DEBUG:boto:Data: 2022-09-02T05:55:59.038 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.040 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.041 DEBUG:boto:Params: {} 2022-09-02T05:55:59.043 DEBUG:boto:Token: None 2022-09-02T05:55:59.043 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj22 2022-09-02T05:55:59.044 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tFBNF7FlQVg030P1HKIRLLCgDEs= 2022-09-02T05:55:59.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tFBNF7FlQVg030P1HKIRLLCgDEs=', 'Content-Length': '0'} 2022-09-02T05:55:59.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002aa5c33726d041f-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.049 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:55:59.049 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:55:59.050 DEBUG:boto:Method: GET 2022-09-02T05:55:59.050 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:55:59.051 DEBUG:boto:Data: 2022-09-02T05:55:59.051 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.053 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.054 DEBUG:boto:Params: {} 2022-09-02T05:55:59.055 DEBUG:boto:Token: None 2022-09-02T05:55:59.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj22 2022-09-02T05:55:59.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tFBNF7FlQVg030P1HKIRLLCgDEs= 2022-09-02T05:55:59.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tFBNF7FlQVg030P1HKIRLLCgDEs=', 'Content-Length': '0'} 2022-09-02T05:55:59.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d29b268045904b29-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.061 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.062 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.063 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:59.064 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:59.065 DEBUG:boto:Method: GET 2022-09-02T05:55:59.066 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:59.067 DEBUG:boto:Data: 2022-09-02T05:55:59.067 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.069 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.070 DEBUG:boto:Params: {} 2022-09-02T05:55:59.071 DEBUG:boto:Token: None 2022-09-02T05:55:59.072 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.072 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a03f360aaaefb018-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.076 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.077 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.078 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.079 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:59.080 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:55:59.081 DEBUG:boto:Method: GET 2022-09-02T05:55:59.082 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:55:59.083 DEBUG:boto:Data: 2022-09-02T05:55:59.084 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.086 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.086 DEBUG:boto:Params: {} 2022-09-02T05:55:59.087 DEBUG:boto:Token: None 2022-09-02T05:55:59.088 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e31b2408896e58a-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.134 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:59.136 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.136 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.137 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.137 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.138 DEBUG:boto:Method: GET 2022-09-02T05:55:59.138 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.139 DEBUG:boto:Data: 2022-09-02T05:55:59.140 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.141 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.142 DEBUG:boto:Params: {} 2022-09-02T05:55:59.142 DEBUG:boto:Token: None 2022-09-02T05:55:59.143 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.144 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE= 2022-09-02T05:55:59.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE=', 'Content-Length': '0'} 2022-09-02T05:55:59.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc49e4c5fd70e5ae-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.149 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.150 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.151 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.152 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.153 DEBUG:boto:Method: GET 2022-09-02T05:55:59.154 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.155 DEBUG:boto:Data: 2022-09-02T05:55:59.156 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.157 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.158 DEBUG:boto:Params: {} 2022-09-02T05:55:59.158 DEBUG:boto:Token: None 2022-09-02T05:55:59.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE= 2022-09-02T05:55:59.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE=', 'Content-Length': '0'} 2022-09-02T05:55:59.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8af95233c1617fc-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.164 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.164 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.165 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.165 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.166 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.167 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.167 DEBUG:boto:Data: 2022-09-02T05:55:59.168 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.169 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.169 DEBUG:boto:Params: {} 2022-09-02T05:55:59.170 DEBUG:boto:Token: None 2022-09-02T05:55:59.171 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCAaC6PErOnZ9ENJEdeBtDtwnaI= 2022-09-02T05:55:59.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCAaC6PErOnZ9ENJEdeBtDtwnaI=', 'Content-Length': '0'} 2022-09-02T05:55:59.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7ae13e1ccb8a36-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.175 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.175 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.176 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.176 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.177 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.177 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.178 DEBUG:boto:Data: 2022-09-02T05:55:59.178 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.179 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.179 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.180 DEBUG:boto:Params: {} 2022-09-02T05:55:59.181 DEBUG:boto:Token: None 2022-09-02T05:55:59.181 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCAaC6PErOnZ9ENJEdeBtDtwnaI= 2022-09-02T05:55:59.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCAaC6PErOnZ9ENJEdeBtDtwnaI=', 'Content-Length': '0'} 2022-09-02T05:55:59.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002145ac8fc571f567-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:59.186 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.187 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.187 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.188 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.188 DEBUG:boto:Method: GET 2022-09-02T05:55:59.189 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.189 DEBUG:boto:Data: 2022-09-02T05:55:59.190 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.191 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.192 DEBUG:boto:Params: {} 2022-09-02T05:55:59.193 DEBUG:boto:Token: None 2022-09-02T05:55:59.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE= 2022-09-02T05:55:59.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE=', 'Content-Length': '0'} 2022-09-02T05:55:59.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1fd9f12bbeb79b-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.238 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.239 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.240 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.241 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.241 DEBUG:boto:Method: GET 2022-09-02T05:55:59.242 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.243 DEBUG:boto:Data: 2022-09-02T05:55:59.244 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.246 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.247 DEBUG:boto:Params: {} 2022-09-02T05:55:59.248 DEBUG:boto:Token: None 2022-09-02T05:55:59.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:55:59.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE= 2022-09-02T05:55:59.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AIWSENFJmxPbR16z5lSGEV+ftSE=', 'Content-Length': '0'} 2022-09-02T05:55:59.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f96c74669906895-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.256 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.257 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.258 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.259 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.260 DEBUG:boto:Method: GET 2022-09-02T05:55:59.261 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.262 DEBUG:boto:Data: 2022-09-02T05:55:59.263 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.265 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.266 DEBUG:boto:Params: {} 2022-09-02T05:55:59.267 DEBUG:boto:Token: None 2022-09-02T05:55:59.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.268 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026ab770ceb7cbbb8-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.272 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.273 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.274 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.275 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.275 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.276 DEBUG:boto:Method: GET 2022-09-02T05:55:59.276 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.277 DEBUG:boto:Data: 2022-09-02T05:55:59.277 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.278 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.279 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.279 DEBUG:boto:Params: {} 2022-09-02T05:55:59.280 DEBUG:boto:Token: None 2022-09-02T05:55:59.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047565565cca8ef9b-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.330 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.332 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.332 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.333 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.333 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:59.334 DEBUG:boto:Data: 2022-09-02T05:55:59.335 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.337 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.337 DEBUG:boto:Params: {} 2022-09-02T05:55:59.338 DEBUG:boto:Token: None 2022-09-02T05:55:59.338 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23 2022-09-02T05:55:59.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dE+bN3tOfnSV1/5NWVUuVQq5RVQ= 2022-09-02T05:55:59.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dE+bN3tOfnSV1/5NWVUuVQq5RVQ=', 'Content-Length': '0'} 2022-09-02T05:55:59.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd31bd08a41f0edb-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.343 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.344 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.345 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.346 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:59.347 DEBUG:boto:Data: 2022-09-02T05:55:59.348 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.349 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.350 DEBUG:boto:Params: {} 2022-09-02T05:55:59.351 DEBUG:boto:Token: None 2022-09-02T05:55:59.352 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23 2022-09-02T05:55:59.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dE+bN3tOfnSV1/5NWVUuVQq5RVQ= 2022-09-02T05:55:59.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dE+bN3tOfnSV1/5NWVUuVQq5RVQ=', 'Content-Length': '0'} 2022-09-02T05:55:59.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa252dbe35543cf-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:55:59.357 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.359 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.359 DEBUG:boto:Method: GET 2022-09-02T05:55:59.360 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:59.361 DEBUG:boto:Data: 2022-09-02T05:55:59.361 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.363 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.364 DEBUG:boto:Params: {} 2022-09-02T05:55:59.365 DEBUG:boto:Token: None 2022-09-02T05:55:59.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23 2022-09-02T05:55:59.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ec1KWJX73LhQ0WFuA2QP/BhAURo= 2022-09-02T05:55:59.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ec1KWJX73LhQ0WFuA2QP/BhAURo=', 'Content-Length': '0'} 2022-09-02T05:55:59.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e025ed95b38de5c4-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.371 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:55:59.372 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:55:59.372 DEBUG:boto:Method: GET 2022-09-02T05:55:59.373 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:55:59.374 DEBUG:boto:Data: 2022-09-02T05:55:59.375 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.377 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.378 DEBUG:boto:Params: {} 2022-09-02T05:55:59.379 DEBUG:boto:Token: None 2022-09-02T05:55:59.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj23 2022-09-02T05:55:59.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ec1KWJX73LhQ0WFuA2QP/BhAURo= 2022-09-02T05:55:59.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ec1KWJX73LhQ0WFuA2QP/BhAURo=', 'Content-Length': '0'} 2022-09-02T05:55:59.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009540a2ecb810c425-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.385 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.386 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.386 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.387 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.388 DEBUG:boto:Method: GET 2022-09-02T05:55:59.388 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.389 DEBUG:boto:Data: 2022-09-02T05:55:59.390 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.391 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.392 DEBUG:boto:Params: {} 2022-09-02T05:55:59.393 DEBUG:boto:Token: None 2022-09-02T05:55:59.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9b08f9d06bf803-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.402 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.403 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.404 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.405 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.405 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.406 DEBUG:boto:Method: GET 2022-09-02T05:55:59.406 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:55:59.407 DEBUG:boto:Data: 2022-09-02T05:55:59.408 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.409 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.409 DEBUG:boto:Params: {} 2022-09-02T05:55:59.410 DEBUG:boto:Token: None 2022-09-02T05:55:59.411 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277ee47a59992512-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.416 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:59.417 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.418 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.419 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.419 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.420 DEBUG:boto:Method: GET 2022-09-02T05:55:59.421 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.421 DEBUG:boto:Data: 2022-09-02T05:55:59.422 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.423 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.423 DEBUG:boto:Params: {} 2022-09-02T05:55:59.424 DEBUG:boto:Token: None 2022-09-02T05:55:59.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0= 2022-09-02T05:55:59.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0=', 'Content-Length': '0'} 2022-09-02T05:55:59.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c09af079f5db05ab-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.431 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.431 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.432 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.432 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.433 DEBUG:boto:Method: GET 2022-09-02T05:55:59.434 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.435 DEBUG:boto:Data: 2022-09-02T05:55:59.436 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.437 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.437 DEBUG:boto:Params: {} 2022-09-02T05:55:59.438 DEBUG:boto:Token: None 2022-09-02T05:55:59.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0= 2022-09-02T05:55:59.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0=', 'Content-Length': '0'} 2022-09-02T05:55:59.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f61fcfed6f0469-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.444 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.445 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.445 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.446 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.446 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.447 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.447 DEBUG:boto:Data: 2022-09-02T05:55:59.448 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.449 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.449 DEBUG:boto:Params: {} 2022-09-02T05:55:59.450 DEBUG:boto:Token: None 2022-09-02T05:55:59.450 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oSUrVXkLHOiCmWo51EEUOGSg91Q= 2022-09-02T05:55:59.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oSUrVXkLHOiCmWo51EEUOGSg91Q=', 'Content-Length': '0'} 2022-09-02T05:55:59.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b9d8168d4ebf8b-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.455 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.456 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.457 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.457 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.458 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.458 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.459 DEBUG:boto:Data: 2022-09-02T05:55:59.459 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.460 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.461 DEBUG:boto:Params: {} 2022-09-02T05:55:59.462 DEBUG:boto:Token: None 2022-09-02T05:55:59.463 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oSUrVXkLHOiCmWo51EEUOGSg91Q= 2022-09-02T05:55:59.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oSUrVXkLHOiCmWo51EEUOGSg91Q=', 'Content-Length': '0'} 2022-09-02T05:55:59.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000822d98d27663f3c5-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:59.468 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.469 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.469 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.470 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.471 DEBUG:boto:Method: GET 2022-09-02T05:55:59.471 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.472 DEBUG:boto:Data: 2022-09-02T05:55:59.473 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.475 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.476 DEBUG:boto:Params: {} 2022-09-02T05:55:59.477 DEBUG:boto:Token: None 2022-09-02T05:55:59.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0= 2022-09-02T05:55:59.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0=', 'Content-Length': '0'} 2022-09-02T05:55:59.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002118ed63d2af2e81-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.526 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.527 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.528 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.528 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.529 DEBUG:boto:Method: GET 2022-09-02T05:55:59.530 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.530 DEBUG:boto:Data: 2022-09-02T05:55:59.531 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.532 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.532 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.533 DEBUG:boto:Params: {} 2022-09-02T05:55:59.533 DEBUG:boto:Token: None 2022-09-02T05:55:59.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:55:59.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0= 2022-09-02T05:55:59.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qC6g3RRtvJ1w95tF9UolcYKPss0=', 'Content-Length': '0'} 2022-09-02T05:55:59.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d7aade0cc745d0-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.539 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.540 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.541 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.541 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.542 DEBUG:boto:Method: GET 2022-09-02T05:55:59.543 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.544 DEBUG:boto:Data: 2022-09-02T05:55:59.544 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.546 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.547 DEBUG:boto:Params: {} 2022-09-02T05:55:59.548 DEBUG:boto:Token: None 2022-09-02T05:55:59.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879f3062a9c8baa0-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.553 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.554 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.555 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.555 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.556 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.557 DEBUG:boto:Method: GET 2022-09-02T05:55:59.557 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.558 DEBUG:boto:Data: 2022-09-02T05:55:59.558 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.560 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.561 DEBUG:boto:Params: {} 2022-09-02T05:55:59.562 DEBUG:boto:Token: None 2022-09-02T05:55:59.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb3986d36ea2f70d-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.567 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.568 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.569 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.569 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.570 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:59.570 DEBUG:boto:Data: 2022-09-02T05:55:59.571 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.572 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.573 DEBUG:boto:Params: {} 2022-09-02T05:55:59.573 DEBUG:boto:Token: None 2022-09-02T05:55:59.574 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29 2022-09-02T05:55:59.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HyQz5Tr8sp2AkMzthfHmHkkLam8= 2022-09-02T05:55:59.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HyQz5Tr8sp2AkMzthfHmHkkLam8=', 'Content-Length': '0'} 2022-09-02T05:55:59.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dece70c60b65a1e9-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.578 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.579 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.580 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.580 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:59.581 DEBUG:boto:Data: 2022-09-02T05:55:59.581 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.582 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.582 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.583 DEBUG:boto:Params: {} 2022-09-02T05:55:59.583 DEBUG:boto:Token: None 2022-09-02T05:55:59.584 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29 2022-09-02T05:55:59.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HyQz5Tr8sp2AkMzthfHmHkkLam8= 2022-09-02T05:55:59.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HyQz5Tr8sp2AkMzthfHmHkkLam8=', 'Content-Length': '0'} 2022-09-02T05:55:59.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017bf48113f2c2391-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:55:59.588 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.589 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.589 DEBUG:boto:Method: GET 2022-09-02T05:55:59.590 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:59.590 DEBUG:boto:Data: 2022-09-02T05:55:59.591 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.592 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.593 DEBUG:boto:Params: {} 2022-09-02T05:55:59.593 DEBUG:boto:Token: None 2022-09-02T05:55:59.594 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29 2022-09-02T05:55:59.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VKwPan6fFSauNLkkTAlEZYo7g14= 2022-09-02T05:55:59.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VKwPan6fFSauNLkkTAlEZYo7g14=', 'Content-Length': '0'} 2022-09-02T05:55:59.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003440ca0fc2215185-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.599 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:55:59.600 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:55:59.601 DEBUG:boto:Method: GET 2022-09-02T05:55:59.601 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:55:59.602 DEBUG:boto:Data: 2022-09-02T05:55:59.603 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.604 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.604 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.605 DEBUG:boto:Params: {} 2022-09-02T05:55:59.605 DEBUG:boto:Token: None 2022-09-02T05:55:59.606 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj29 2022-09-02T05:55:59.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VKwPan6fFSauNLkkTAlEZYo7g14= 2022-09-02T05:55:59.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VKwPan6fFSauNLkkTAlEZYo7g14=', 'Content-Length': '0'} 2022-09-02T05:55:59.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e6f70d45300cfd-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.610 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.611 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.611 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.612 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.612 DEBUG:boto:Method: GET 2022-09-02T05:55:59.613 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.613 DEBUG:boto:Data: 2022-09-02T05:55:59.614 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.615 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.616 DEBUG:boto:Params: {} 2022-09-02T05:55:59.616 DEBUG:boto:Token: None 2022-09-02T05:55:59.617 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006045e60198e8ebed-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.622 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.623 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.623 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.624 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.624 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.625 DEBUG:boto:Method: GET 2022-09-02T05:55:59.625 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:55:59.626 DEBUG:boto:Data: 2022-09-02T05:55:59.626 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.628 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.628 DEBUG:boto:Params: {} 2022-09-02T05:55:59.629 DEBUG:boto:Token: None 2022-09-02T05:55:59.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8659c450c59e966-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.635 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:59.637 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.638 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.638 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.639 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.640 DEBUG:boto:Method: GET 2022-09-02T05:55:59.641 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.641 DEBUG:boto:Data: 2022-09-02T05:55:59.642 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.643 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.644 DEBUG:boto:Params: {} 2022-09-02T05:55:59.644 DEBUG:boto:Token: None 2022-09-02T05:55:59.645 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA= 2022-09-02T05:55:59.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA=', 'Content-Length': '0'} 2022-09-02T05:55:59.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044c90884c17990f3-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.649 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.650 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.650 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.651 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.651 DEBUG:boto:Method: GET 2022-09-02T05:55:59.652 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.652 DEBUG:boto:Data: 2022-09-02T05:55:59.653 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.654 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.655 DEBUG:boto:Params: {} 2022-09-02T05:55:59.655 DEBUG:boto:Token: None 2022-09-02T05:55:59.656 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA= 2022-09-02T05:55:59.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA=', 'Content-Length': '0'} 2022-09-02T05:55:59.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c346eeecbec76ea7-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.660 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.661 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.661 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.662 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.662 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.663 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.663 DEBUG:boto:Data: 2022-09-02T05:55:59.664 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.666 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.666 DEBUG:boto:Params: {} 2022-09-02T05:55:59.667 DEBUG:boto:Token: None 2022-09-02T05:55:59.667 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5HXN+uopBzJzx7zUgZ23llLmhcc= 2022-09-02T05:55:59.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5HXN+uopBzJzx7zUgZ23llLmhcc=', 'Content-Length': '0'} 2022-09-02T05:55:59.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043023dbd843f1c9b-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.671 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.672 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.672 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.673 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.673 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.674 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.675 DEBUG:boto:Data: 2022-09-02T05:55:59.675 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.676 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.676 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.677 DEBUG:boto:Params: {} 2022-09-02T05:55:59.677 DEBUG:boto:Token: None 2022-09-02T05:55:59.678 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5HXN+uopBzJzx7zUgZ23llLmhcc= 2022-09-02T05:55:59.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5HXN+uopBzJzx7zUgZ23llLmhcc=', 'Content-Length': '0'} 2022-09-02T05:55:59.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a97638457afa8f9a-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:59.684 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.684 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.685 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.685 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.686 DEBUG:boto:Method: GET 2022-09-02T05:55:59.686 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.687 DEBUG:boto:Data: 2022-09-02T05:55:59.688 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.689 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.689 DEBUG:boto:Params: {} 2022-09-02T05:55:59.690 DEBUG:boto:Token: None 2022-09-02T05:55:59.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA= 2022-09-02T05:55:59.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA=', 'Content-Length': '0'} 2022-09-02T05:55:59.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010cdbdfa23b02b26-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.695 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.695 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.696 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.696 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.697 DEBUG:boto:Method: GET 2022-09-02T05:55:59.697 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.698 DEBUG:boto:Data: 2022-09-02T05:55:59.698 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.700 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.700 DEBUG:boto:Params: {} 2022-09-02T05:55:59.701 DEBUG:boto:Token: None 2022-09-02T05:55:59.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:55:59.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA= 2022-09-02T05:55:59.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jz+s3EDnhL2ZXVD290hy3PARPJA=', 'Content-Length': '0'} 2022-09-02T05:55:59.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3bbaeeefedba43-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.754 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.755 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.755 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.756 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.757 DEBUG:boto:Method: GET 2022-09-02T05:55:59.757 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.758 DEBUG:boto:Data: 2022-09-02T05:55:59.758 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.760 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.760 DEBUG:boto:Params: {} 2022-09-02T05:55:59.761 DEBUG:boto:Token: None 2022-09-02T05:55:59.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fe262dc3007dc01-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.768 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.769 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.770 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.771 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.772 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.773 DEBUG:boto:Method: GET 2022-09-02T05:55:59.774 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.775 DEBUG:boto:Data: 2022-09-02T05:55:59.776 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.777 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.778 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.779 DEBUG:boto:Params: {} 2022-09-02T05:55:59.779 DEBUG:boto:Token: None 2022-09-02T05:55:59.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c92644a0ba43087-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.785 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.785 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.786 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.787 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.787 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:59.788 DEBUG:boto:Data: 2022-09-02T05:55:59.789 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.790 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.791 DEBUG:boto:Params: {} 2022-09-02T05:55:59.791 DEBUG:boto:Token: None 2022-09-02T05:55:59.792 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33 2022-09-02T05:55:59.792 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VW9nUjrlFWy/P4mJNsIgxRCjlds= 2022-09-02T05:55:59.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VW9nUjrlFWy/P4mJNsIgxRCjlds=', 'Content-Length': '0'} 2022-09-02T05:55:59.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006badbf148902701e-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.797 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.798 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.799 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.800 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:59.801 DEBUG:boto:Data: 2022-09-02T05:55:59.802 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.804 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.805 DEBUG:boto:Params: {} 2022-09-02T05:55:59.806 DEBUG:boto:Token: None 2022-09-02T05:55:59.806 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33 2022-09-02T05:55:59.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VW9nUjrlFWy/P4mJNsIgxRCjlds= 2022-09-02T05:55:59.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VW9nUjrlFWy/P4mJNsIgxRCjlds=', 'Content-Length': '0'} 2022-09-02T05:55:59.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006686b96b93d54cfb-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:55:59.812 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.813 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.814 DEBUG:boto:Method: GET 2022-09-02T05:55:59.815 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:59.816 DEBUG:boto:Data: 2022-09-02T05:55:59.817 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.819 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.820 DEBUG:boto:Params: {} 2022-09-02T05:55:59.821 DEBUG:boto:Token: None 2022-09-02T05:55:59.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33 2022-09-02T05:55:59.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zdpTauCbF5LHqI0929ChCB0M4Uc= 2022-09-02T05:55:59.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zdpTauCbF5LHqI0929ChCB0M4Uc=', 'Content-Length': '0'} 2022-09-02T05:55:59.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037960c8559c8ab67-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.870 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:55:59.871 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:55:59.872 DEBUG:boto:Method: GET 2022-09-02T05:55:59.872 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:55:59.873 DEBUG:boto:Data: 2022-09-02T05:55:59.873 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.874 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.874 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.875 DEBUG:boto:Params: {} 2022-09-02T05:55:59.875 DEBUG:boto:Token: None 2022-09-02T05:55:59.877 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj33 2022-09-02T05:55:59.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zdpTauCbF5LHqI0929ChCB0M4Uc= 2022-09-02T05:55:59.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zdpTauCbF5LHqI0929ChCB0M4Uc=', 'Content-Length': '0'} 2022-09-02T05:55:59.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a86f522da747409-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.881 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.882 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.882 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.883 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.883 DEBUG:boto:Method: GET 2022-09-02T05:55:59.884 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.884 DEBUG:boto:Data: 2022-09-02T05:55:59.885 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.886 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.887 DEBUG:boto:Params: {} 2022-09-02T05:55:59.888 DEBUG:boto:Token: None 2022-09-02T05:55:59.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48f9f76d02115dc-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.894 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.895 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.896 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.896 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.897 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.898 DEBUG:boto:Method: GET 2022-09-02T05:55:59.898 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:55:59.899 DEBUG:boto:Data: 2022-09-02T05:55:59.900 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.900 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.901 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.901 DEBUG:boto:Params: {} 2022-09-02T05:55:59.902 DEBUG:boto:Token: None 2022-09-02T05:55:59.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336f998379d16291-0063119aef-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.909 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:55:59.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:59.910 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:59.911 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:59.912 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.912 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.913 DEBUG:boto:Method: GET 2022-09-02T05:55:59.913 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.914 DEBUG:boto:Data: 2022-09-02T05:55:59.915 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.916 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.916 DEBUG:boto:Params: {} 2022-09-02T05:55:59.917 DEBUG:boto:Token: None 2022-09-02T05:55:59.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE= 2022-09-02T05:55:59.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE=', 'Content-Length': '0'} 2022-09-02T05:55:59.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f9ca1cb8a12ed84-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.922 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:59.923 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:59.924 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.924 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.925 DEBUG:boto:Method: GET 2022-09-02T05:55:59.925 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.926 DEBUG:boto:Data: 2022-09-02T05:55:59.926 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.927 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.928 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.928 DEBUG:boto:Params: {} 2022-09-02T05:55:59.929 DEBUG:boto:Token: None 2022-09-02T05:55:59.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE= 2022-09-02T05:55:59.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE=', 'Content-Length': '0'} 2022-09-02T05:55:59.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002328354bbce5b1ae-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.934 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:59.935 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:59.935 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.936 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.936 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.937 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.937 DEBUG:boto:Data: 2022-09-02T05:55:59.938 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.939 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.940 DEBUG:boto:Params: {} 2022-09-02T05:55:59.940 DEBUG:boto:Token: None 2022-09-02T05:55:59.941 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:adxGw5EKF6xS5TP4gG+ZFHX4lik= 2022-09-02T05:55:59.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:adxGw5EKF6xS5TP4gG+ZFHX4lik=', 'Content-Length': '0'} 2022-09-02T05:55:59.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0af77687cd61231-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.945 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:59.946 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:59.946 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.947 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.948 DEBUG:boto:Method: HEAD 2022-09-02T05:55:59.948 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.949 DEBUG:boto:Data: 2022-09-02T05:55:59.949 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.950 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.951 DEBUG:boto:Params: {} 2022-09-02T05:55:59.951 DEBUG:boto:Token: None 2022-09-02T05:55:59.952 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:adxGw5EKF6xS5TP4gG+ZFHX4lik= 2022-09-02T05:55:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:adxGw5EKF6xS5TP4gG+ZFHX4lik=', 'Content-Length': '0'} 2022-09-02T05:55:59.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c863be74fb21e3-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:55:59.957 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:59.958 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:59.959 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.960 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.960 DEBUG:boto:Method: GET 2022-09-02T05:55:59.961 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.962 DEBUG:boto:Data: 2022-09-02T05:55:59.962 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.963 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.964 DEBUG:boto:Params: {} 2022-09-02T05:55:59.965 DEBUG:boto:Token: None 2022-09-02T05:55:59.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE= 2022-09-02T05:55:59.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE=', 'Content-Length': '0'} 2022-09-02T05:55:59.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080df8ab0a498b840-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.970 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:55:59.971 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:55:59.971 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.972 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.972 DEBUG:boto:Method: GET 2022-09-02T05:55:59.973 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.973 DEBUG:boto:Data: 2022-09-02T05:55:59.974 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:55:59.975 DEBUG:boto:Port: 8000 2022-09-02T05:55:59.976 DEBUG:boto:Params: {} 2022-09-02T05:55:59.976 DEBUG:boto:Token: None 2022-09-02T05:55:59.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:55:59.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE= 2022-09-02T05:55:59.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2gG8gvaJcYA8QQwqPb6tFdHQ8NE=', 'Content-Length': '0'} 2022-09-02T05:55:59.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad38aafc91e3615b-0063119aef-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:55:59.983 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:55:59.984 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:55:59.985 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:59.986 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:55:59.987 DEBUG:boto:Method: GET 2022-09-02T05:55:59.988 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:55:59.989 DEBUG:boto:Data: 2022-09-02T05:55:59.990 DEBUG:boto:Headers: {} 2022-09-02T05:55:59.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:55:59.993 DEBUG:boto:Port: 8001 2022-09-02T05:55:59.994 DEBUG:boto:Params: {} 2022-09-02T05:55:59.995 DEBUG:boto:Token: None 2022-09-02T05:55:59.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:55:59 GMT /xljbkf-74/ 2022-09-02T05:55:59.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo= 2022-09-02T05:55:59.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:55:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VbHp6Ug2DSNvdu3wpIhf3DQWbFo=', 'Content-Length': '0'} 2022-09-02T05:55:59.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9ff96c4ee0c261a-0063119aef-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:55:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.000 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.001 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.002 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.002 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.003 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.003 DEBUG:boto:Method: GET 2022-09-02T05:56:00.004 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.004 DEBUG:boto:Data: 2022-09-02T05:56:00.005 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.006 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.006 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.007 DEBUG:boto:Params: {} 2022-09-02T05:56:00.008 DEBUG:boto:Token: None 2022-09-02T05:56:00.008 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71fba49eb83248a-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.013 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.013 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:56:00.014 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:56:00.015 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.015 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:56:00.016 DEBUG:boto:Data: 2022-09-02T05:56:00.016 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.017 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.018 DEBUG:boto:Params: {} 2022-09-02T05:56:00.018 DEBUG:boto:Token: None 2022-09-02T05:56:00.019 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj34 2022-09-02T05:56:00.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FVs5//JhOBFbFLSXMV9u9XrajkY= 2022-09-02T05:56:00.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FVs5//JhOBFbFLSXMV9u9XrajkY=', 'Content-Length': '0'} 2022-09-02T05:56:00.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b771957a271229-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.023 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:56:00.024 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:56:00.025 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.025 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:56:00.026 DEBUG:boto:Data: 2022-09-02T05:56:00.026 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.027 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.027 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.028 DEBUG:boto:Params: {} 2022-09-02T05:56:00.028 DEBUG:boto:Token: None 2022-09-02T05:56:00.029 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj34 2022-09-02T05:56:00.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FVs5//JhOBFbFLSXMV9u9XrajkY= 2022-09-02T05:56:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FVs5//JhOBFbFLSXMV9u9XrajkY=', 'Content-Length': '0'} 2022-09-02T05:56:00.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f04cbcddbad7b925-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:56:00.033 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:56:00.034 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:56:00.034 DEBUG:boto:Method: GET 2022-09-02T05:56:00.035 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:56:00.036 DEBUG:boto:Data: 2022-09-02T05:56:00.036 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.037 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.038 DEBUG:boto:Params: {} 2022-09-02T05:56:00.038 DEBUG:boto:Token: None 2022-09-02T05:56:00.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj34 2022-09-02T05:56:00.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Qb8MU5S+QsNx2LBqfwzJYUfO8w= 2022-09-02T05:56:00.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Qb8MU5S+QsNx2LBqfwzJYUfO8w=', 'Content-Length': '0'} 2022-09-02T05:56:00.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b225a96cc4a86aa-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.086 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:56:00.087 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:56:00.088 DEBUG:boto:Method: GET 2022-09-02T05:56:00.088 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:56:00.089 DEBUG:boto:Data: 2022-09-02T05:56:00.089 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.090 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.091 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.092 DEBUG:boto:Params: {} 2022-09-02T05:56:00.093 DEBUG:boto:Token: None 2022-09-02T05:56:00.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj34 2022-09-02T05:56:00.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Qb8MU5S+QsNx2LBqfwzJYUfO8w= 2022-09-02T05:56:00.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Qb8MU5S+QsNx2LBqfwzJYUfO8w=', 'Content-Length': '0'} 2022-09-02T05:56:00.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c32588949efbfb-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.101 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.101 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.102 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.102 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.103 DEBUG:boto:Method: GET 2022-09-02T05:56:00.103 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.104 DEBUG:boto:Data: 2022-09-02T05:56:00.105 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.106 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.106 DEBUG:boto:Params: {} 2022-09-02T05:56:00.108 DEBUG:boto:Token: None 2022-09-02T05:56:00.109 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9e23c5e6087021-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.115 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.116 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.117 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.117 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.118 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.119 DEBUG:boto:Method: GET 2022-09-02T05:56:00.119 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:56:00.120 DEBUG:boto:Data: 2022-09-02T05:56:00.120 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.121 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.121 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.122 DEBUG:boto:Params: {} 2022-09-02T05:56:00.122 DEBUG:boto:Token: None 2022-09-02T05:56:00.123 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008930fc2b39353db1-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.128 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:56:00.129 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.130 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.130 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.131 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.131 DEBUG:boto:Method: GET 2022-09-02T05:56:00.132 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.133 DEBUG:boto:Data: 2022-09-02T05:56:00.133 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.134 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.135 DEBUG:boto:Params: {} 2022-09-02T05:56:00.135 DEBUG:boto:Token: None 2022-09-02T05:56:00.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q= 2022-09-02T05:56:00.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q=', 'Content-Length': '0'} 2022-09-02T05:56:00.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bcc1f8bdd2d800-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.141 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.141 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.142 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.142 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.143 DEBUG:boto:Method: GET 2022-09-02T05:56:00.143 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.144 DEBUG:boto:Data: 2022-09-02T05:56:00.144 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.145 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.145 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.146 DEBUG:boto:Params: {} 2022-09-02T05:56:00.146 DEBUG:boto:Token: None 2022-09-02T05:56:00.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q= 2022-09-02T05:56:00.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q=', 'Content-Length': '0'} 2022-09-02T05:56:00.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000530617221d9a7195-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.198 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.200 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.201 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.202 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.203 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.204 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.205 DEBUG:boto:Data: 2022-09-02T05:56:00.206 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.207 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.208 DEBUG:boto:Params: {} 2022-09-02T05:56:00.209 DEBUG:boto:Token: None 2022-09-02T05:56:00.210 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:psPeRT6S94c4CCzEYSaJGI/IzYc= 2022-09-02T05:56:00.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:psPeRT6S94c4CCzEYSaJGI/IzYc=', 'Content-Length': '0'} 2022-09-02T05:56:00.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc7113ac55d3ebb-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.216 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.217 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.218 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.219 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.219 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.220 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.220 DEBUG:boto:Data: 2022-09-02T05:56:00.221 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.222 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.222 DEBUG:boto:Params: {} 2022-09-02T05:56:00.223 DEBUG:boto:Token: None 2022-09-02T05:56:00.223 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:psPeRT6S94c4CCzEYSaJGI/IzYc= 2022-09-02T05:56:00.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:psPeRT6S94c4CCzEYSaJGI/IzYc=', 'Content-Length': '0'} 2022-09-02T05:56:00.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f967634e029729-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:56:00.228 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.229 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.229 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.230 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.230 DEBUG:boto:Method: GET 2022-09-02T05:56:00.231 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.231 DEBUG:boto:Data: 2022-09-02T05:56:00.232 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.233 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.234 DEBUG:boto:Params: {} 2022-09-02T05:56:00.234 DEBUG:boto:Token: None 2022-09-02T05:56:00.235 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.235 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q= 2022-09-02T05:56:00.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q=', 'Content-Length': '0'} 2022-09-02T05:56:00.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f59110b237a3ed1-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.282 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.284 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.284 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.285 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.286 DEBUG:boto:Method: GET 2022-09-02T05:56:00.287 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.287 DEBUG:boto:Data: 2022-09-02T05:56:00.288 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.289 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.290 DEBUG:boto:Params: {} 2022-09-02T05:56:00.290 DEBUG:boto:Token: None 2022-09-02T05:56:00.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:56:00.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q= 2022-09-02T05:56:00.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HzHOt1um3GKq/GCeP1KeR7PDV9Q=', 'Content-Length': '0'} 2022-09-02T05:56:00.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db1ba16a79b06c3-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.297 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.298 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.299 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.300 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.301 DEBUG:boto:Method: GET 2022-09-02T05:56:00.302 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.303 DEBUG:boto:Data: 2022-09-02T05:56:00.304 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.305 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.306 DEBUG:boto:Params: {} 2022-09-02T05:56:00.307 DEBUG:boto:Token: None 2022-09-02T05:56:00.307 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae92b5d589a2e70-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.314 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.316 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.317 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.318 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.319 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.320 DEBUG:boto:Method: GET 2022-09-02T05:56:00.321 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.322 DEBUG:boto:Data: 2022-09-02T05:56:00.323 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.325 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.325 DEBUG:boto:Params: {} 2022-09-02T05:56:00.326 DEBUG:boto:Token: None 2022-09-02T05:56:00.326 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd6852e1a29bda9-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.331 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.332 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.333 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.333 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.334 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:56:00.334 DEBUG:boto:Data: 2022-09-02T05:56:00.335 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.336 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.337 DEBUG:boto:Params: {} 2022-09-02T05:56:00.338 DEBUG:boto:Token: None 2022-09-02T05:56:00.338 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38 2022-09-02T05:56:00.340 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OvBF6Be+47iKrtvb2VkDvS7EAjg= 2022-09-02T05:56:00.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OvBF6Be+47iKrtvb2VkDvS7EAjg=', 'Content-Length': '0'} 2022-09-02T05:56:00.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c3782f9b0133fdd-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.344 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.345 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.345 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.346 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:56:00.348 DEBUG:boto:Data: 2022-09-02T05:56:00.348 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.349 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.350 DEBUG:boto:Params: {} 2022-09-02T05:56:00.350 DEBUG:boto:Token: None 2022-09-02T05:56:00.351 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38 2022-09-02T05:56:00.352 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OvBF6Be+47iKrtvb2VkDvS7EAjg= 2022-09-02T05:56:00.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OvBF6Be+47iKrtvb2VkDvS7EAjg=', 'Content-Length': '0'} 2022-09-02T05:56:00.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001003feff2ca484c9-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:56:00.356 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.356 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.357 DEBUG:boto:Method: GET 2022-09-02T05:56:00.358 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:56:00.358 DEBUG:boto:Data: 2022-09-02T05:56:00.359 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.360 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.360 DEBUG:boto:Params: {} 2022-09-02T05:56:00.361 DEBUG:boto:Token: None 2022-09-02T05:56:00.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38 2022-09-02T05:56:00.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:74vo7qgM5Oms/7cuaB8und2pX9U= 2022-09-02T05:56:00.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:74vo7qgM5Oms/7cuaB8und2pX9U=', 'Content-Length': '0'} 2022-09-02T05:56:00.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baef3584718b12bc-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.366 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:56:00.366 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:56:00.367 DEBUG:boto:Method: GET 2022-09-02T05:56:00.368 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:56:00.368 DEBUG:boto:Data: 2022-09-02T05:56:00.369 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.369 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.370 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.370 DEBUG:boto:Params: {} 2022-09-02T05:56:00.371 DEBUG:boto:Token: None 2022-09-02T05:56:00.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj38 2022-09-02T05:56:00.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:74vo7qgM5Oms/7cuaB8und2pX9U= 2022-09-02T05:56:00.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:74vo7qgM5Oms/7cuaB8und2pX9U=', 'Content-Length': '0'} 2022-09-02T05:56:00.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a606f0240ae68de5-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.418 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.419 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.419 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.420 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.420 DEBUG:boto:Method: GET 2022-09-02T05:56:00.421 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.421 DEBUG:boto:Data: 2022-09-02T05:56:00.422 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.423 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.423 DEBUG:boto:Params: {} 2022-09-02T05:56:00.424 DEBUG:boto:Token: None 2022-09-02T05:56:00.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6765a558e7349af-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.430 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.432 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.432 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.433 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.433 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.434 DEBUG:boto:Method: GET 2022-09-02T05:56:00.435 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:56:00.435 DEBUG:boto:Data: 2022-09-02T05:56:00.436 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.437 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.438 DEBUG:boto:Params: {} 2022-09-02T05:56:00.438 DEBUG:boto:Token: None 2022-09-02T05:56:00.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57c88332fb480bc-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.444 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:56:00.445 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.446 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.447 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.447 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.448 DEBUG:boto:Method: GET 2022-09-02T05:56:00.448 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.449 DEBUG:boto:Data: 2022-09-02T05:56:00.449 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.451 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.451 DEBUG:boto:Params: {} 2022-09-02T05:56:00.452 DEBUG:boto:Token: None 2022-09-02T05:56:00.452 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk= 2022-09-02T05:56:00.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk=', 'Content-Length': '0'} 2022-09-02T05:56:00.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ebff227fc68d587-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.456 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.457 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.458 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.458 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.459 DEBUG:boto:Method: GET 2022-09-02T05:56:00.459 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.460 DEBUG:boto:Data: 2022-09-02T05:56:00.460 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.461 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.462 DEBUG:boto:Params: {} 2022-09-02T05:56:00.462 DEBUG:boto:Token: None 2022-09-02T05:56:00.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk= 2022-09-02T05:56:00.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk=', 'Content-Length': '0'} 2022-09-02T05:56:00.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004447d8c480b282ef-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.467 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.468 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.468 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.469 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.470 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.470 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.471 DEBUG:boto:Data: 2022-09-02T05:56:00.471 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.472 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.473 DEBUG:boto:Params: {} 2022-09-02T05:56:00.473 DEBUG:boto:Token: None 2022-09-02T05:56:00.474 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PcYAfoXThjxAiWMvhga8ixZ+Vps= 2022-09-02T05:56:00.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PcYAfoXThjxAiWMvhga8ixZ+Vps=', 'Content-Length': '0'} 2022-09-02T05:56:00.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c400e1e11bdac793-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.479 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.479 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.480 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.480 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.481 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.481 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.482 DEBUG:boto:Data: 2022-09-02T05:56:00.482 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.483 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.484 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.485 DEBUG:boto:Params: {} 2022-09-02T05:56:00.485 DEBUG:boto:Token: None 2022-09-02T05:56:00.486 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PcYAfoXThjxAiWMvhga8ixZ+Vps= 2022-09-02T05:56:00.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PcYAfoXThjxAiWMvhga8ixZ+Vps=', 'Content-Length': '0'} 2022-09-02T05:56:00.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3910a03a0107b7b-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:56:00.491 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.492 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.493 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.494 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.495 DEBUG:boto:Method: GET 2022-09-02T05:56:00.496 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.497 DEBUG:boto:Data: 2022-09-02T05:56:00.498 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.500 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.500 DEBUG:boto:Params: {} 2022-09-02T05:56:00.502 DEBUG:boto:Token: None 2022-09-02T05:56:00.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk= 2022-09-02T05:56:00.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk=', 'Content-Length': '0'} 2022-09-02T05:56:00.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c06fb4ae49c29a2c-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.508 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.508 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.509 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.509 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.510 DEBUG:boto:Method: GET 2022-09-02T05:56:00.510 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.511 DEBUG:boto:Data: 2022-09-02T05:56:00.511 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.512 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.513 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.513 DEBUG:boto:Params: {} 2022-09-02T05:56:00.514 DEBUG:boto:Token: None 2022-09-02T05:56:00.515 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:56:00.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk= 2022-09-02T05:56:00.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HLdse6aZmnMrAY+7IfXxUH/kGAk=', 'Content-Length': '0'} 2022-09-02T05:56:00.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ffc76bbefbce8cd-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.562 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.564 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.565 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.566 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.567 DEBUG:boto:Method: GET 2022-09-02T05:56:00.568 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.569 DEBUG:boto:Data: 2022-09-02T05:56:00.570 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.572 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.573 DEBUG:boto:Params: {} 2022-09-02T05:56:00.574 DEBUG:boto:Token: None 2022-09-02T05:56:00.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c821c0cd1f6e0007-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.581 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.583 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.583 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.584 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.584 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.585 DEBUG:boto:Method: GET 2022-09-02T05:56:00.585 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.586 DEBUG:boto:Data: 2022-09-02T05:56:00.586 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.587 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.588 DEBUG:boto:Params: {} 2022-09-02T05:56:00.588 DEBUG:boto:Token: None 2022-09-02T05:56:00.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012790cd748905164-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.594 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.595 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.595 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.596 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.596 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:56:00.597 DEBUG:boto:Data: 2022-09-02T05:56:00.597 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.599 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.599 DEBUG:boto:Params: {} 2022-09-02T05:56:00.600 DEBUG:boto:Token: None 2022-09-02T05:56:00.600 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44 2022-09-02T05:56:00.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1bRjenJRZlh5NU2VzRT4GX6/5k4= 2022-09-02T05:56:00.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1bRjenJRZlh5NU2VzRT4GX6/5k4=', 'Content-Length': '0'} 2022-09-02T05:56:00.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf55ae86c73ad3f-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.612 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.613 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.614 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.615 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:56:00.615 DEBUG:boto:Data: 2022-09-02T05:56:00.616 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.617 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.617 DEBUG:boto:Params: {} 2022-09-02T05:56:00.618 DEBUG:boto:Token: None 2022-09-02T05:56:00.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44 2022-09-02T05:56:00.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1bRjenJRZlh5NU2VzRT4GX6/5k4= 2022-09-02T05:56:00.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1bRjenJRZlh5NU2VzRT4GX6/5k4=', 'Content-Length': '0'} 2022-09-02T05:56:00.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9350deeefaf13a-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:56:00.623 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.624 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.624 DEBUG:boto:Method: GET 2022-09-02T05:56:00.625 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:56:00.625 DEBUG:boto:Data: 2022-09-02T05:56:00.626 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.627 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.627 DEBUG:boto:Params: {} 2022-09-02T05:56:00.628 DEBUG:boto:Token: None 2022-09-02T05:56:00.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44 2022-09-02T05:56:00.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gNrv8sb5NMe/yKOkWBEjCpKRUxk= 2022-09-02T05:56:00.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gNrv8sb5NMe/yKOkWBEjCpKRUxk=', 'Content-Length': '0'} 2022-09-02T05:56:00.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135099e9d40f911a-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.633 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:56:00.633 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:56:00.634 DEBUG:boto:Method: GET 2022-09-02T05:56:00.634 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:56:00.635 DEBUG:boto:Data: 2022-09-02T05:56:00.635 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.636 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.637 DEBUG:boto:Params: {} 2022-09-02T05:56:00.638 DEBUG:boto:Token: None 2022-09-02T05:56:00.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj44 2022-09-02T05:56:00.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gNrv8sb5NMe/yKOkWBEjCpKRUxk= 2022-09-02T05:56:00.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gNrv8sb5NMe/yKOkWBEjCpKRUxk=', 'Content-Length': '0'} 2022-09-02T05:56:00.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfb9449fd9c10f3-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.644 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.644 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.645 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.646 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.646 DEBUG:boto:Method: GET 2022-09-02T05:56:00.647 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.647 DEBUG:boto:Data: 2022-09-02T05:56:00.648 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.649 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.649 DEBUG:boto:Params: {} 2022-09-02T05:56:00.650 DEBUG:boto:Token: None 2022-09-02T05:56:00.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041d10845fda98057-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.656 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.657 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.657 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.658 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.658 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.659 DEBUG:boto:Method: GET 2022-09-02T05:56:00.659 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:56:00.660 DEBUG:boto:Data: 2022-09-02T05:56:00.660 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.661 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.661 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.662 DEBUG:boto:Params: {} 2022-09-02T05:56:00.663 DEBUG:boto:Token: None 2022-09-02T05:56:00.664 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.664 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fa1e6063d839c8a-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.668 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:56:00.670 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.671 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.671 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.672 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.672 DEBUG:boto:Method: GET 2022-09-02T05:56:00.673 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.674 DEBUG:boto:Data: 2022-09-02T05:56:00.674 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.675 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.676 DEBUG:boto:Params: {} 2022-09-02T05:56:00.676 DEBUG:boto:Token: None 2022-09-02T05:56:00.677 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.678 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM= 2022-09-02T05:56:00.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM=', 'Content-Length': '0'} 2022-09-02T05:56:00.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033b1853e488037e3-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.726 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.727 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.728 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.728 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.729 DEBUG:boto:Method: GET 2022-09-02T05:56:00.730 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.730 DEBUG:boto:Data: 2022-09-02T05:56:00.731 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.732 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.733 DEBUG:boto:Params: {} 2022-09-02T05:56:00.733 DEBUG:boto:Token: None 2022-09-02T05:56:00.734 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM= 2022-09-02T05:56:00.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM=', 'Content-Length': '0'} 2022-09-02T05:56:00.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046f0e448c67465d1-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.738 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.739 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.740 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.740 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.741 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.741 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.742 DEBUG:boto:Data: 2022-09-02T05:56:00.742 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.743 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.744 DEBUG:boto:Params: {} 2022-09-02T05:56:00.744 DEBUG:boto:Token: None 2022-09-02T05:56:00.745 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aS2PcOsMYB5hJlr2AkCvKM0sXpU= 2022-09-02T05:56:00.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aS2PcOsMYB5hJlr2AkCvKM0sXpU=', 'Content-Length': '0'} 2022-09-02T05:56:00.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc381d73bf7efab-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.750 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.751 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.751 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.752 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.752 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.753 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.753 DEBUG:boto:Data: 2022-09-02T05:56:00.754 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.755 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.756 DEBUG:boto:Params: {} 2022-09-02T05:56:00.756 DEBUG:boto:Token: None 2022-09-02T05:56:00.757 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.757 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aS2PcOsMYB5hJlr2AkCvKM0sXpU= 2022-09-02T05:56:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aS2PcOsMYB5hJlr2AkCvKM0sXpU=', 'Content-Length': '0'} 2022-09-02T05:56:00.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e5e6710f60b656-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:56:00.762 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.762 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.763 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.764 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.764 DEBUG:boto:Method: GET 2022-09-02T05:56:00.765 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.766 DEBUG:boto:Data: 2022-09-02T05:56:00.766 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.768 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.768 DEBUG:boto:Params: {} 2022-09-02T05:56:00.769 DEBUG:boto:Token: None 2022-09-02T05:56:00.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM= 2022-09-02T05:56:00.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM=', 'Content-Length': '0'} 2022-09-02T05:56:00.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511274fde135252d-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.774 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.775 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.775 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.776 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.776 DEBUG:boto:Method: GET 2022-09-02T05:56:00.777 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.777 DEBUG:boto:Data: 2022-09-02T05:56:00.778 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.779 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.780 DEBUG:boto:Params: {} 2022-09-02T05:56:00.780 DEBUG:boto:Token: None 2022-09-02T05:56:00.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:56:00.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM= 2022-09-02T05:56:00.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0bD0v/UGyphJ1+GwceL46rUzEQM=', 'Content-Length': '0'} 2022-09-02T05:56:00.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f71ec9b832bb32-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.785 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.786 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.789 DEBUG:boto:Method: GET 2022-09-02T05:56:00.790 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.791 DEBUG:boto:Data: 2022-09-02T05:56:00.792 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.793 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.794 DEBUG:boto:Params: {} 2022-09-02T05:56:00.796 DEBUG:boto:Token: None 2022-09-02T05:56:00.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011a3747a3ddaaea7-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.803 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.804 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.804 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.805 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.806 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.806 DEBUG:boto:Method: GET 2022-09-02T05:56:00.807 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.808 DEBUG:boto:Data: 2022-09-02T05:56:00.808 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.809 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.810 DEBUG:boto:Params: {} 2022-09-02T05:56:00.810 DEBUG:boto:Token: None 2022-09-02T05:56:00.811 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027c1cfe52b68f345-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.816 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.817 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.818 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.818 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.819 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:56:00.819 DEBUG:boto:Data: 2022-09-02T05:56:00.820 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.821 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.821 DEBUG:boto:Params: {} 2022-09-02T05:56:00.822 DEBUG:boto:Token: None 2022-09-02T05:56:00.823 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46 2022-09-02T05:56:00.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:leJ6mRgY531iXwjTfLszKZTMFP0= 2022-09-02T05:56:00.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:leJ6mRgY531iXwjTfLszKZTMFP0=', 'Content-Length': '0'} 2022-09-02T05:56:00.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082287bde5718b938-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.827 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.828 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.829 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.830 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:56:00.831 DEBUG:boto:Data: 2022-09-02T05:56:00.832 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.834 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.835 DEBUG:boto:Params: {} 2022-09-02T05:56:00.836 DEBUG:boto:Token: None 2022-09-02T05:56:00.837 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46 2022-09-02T05:56:00.838 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:leJ6mRgY531iXwjTfLszKZTMFP0= 2022-09-02T05:56:00.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:leJ6mRgY531iXwjTfLszKZTMFP0=', 'Content-Length': '0'} 2022-09-02T05:56:00.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000769b72df54b95502-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:56:00.842 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.843 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.844 DEBUG:boto:Method: GET 2022-09-02T05:56:00.844 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:56:00.845 DEBUG:boto:Data: 2022-09-02T05:56:00.845 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.846 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.847 DEBUG:boto:Params: {} 2022-09-02T05:56:00.848 DEBUG:boto:Token: None 2022-09-02T05:56:00.848 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46 2022-09-02T05:56:00.849 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sOEeeRqV61JiOXTbK3kJJY7NwY= 2022-09-02T05:56:00.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sOEeeRqV61JiOXTbK3kJJY7NwY=', 'Content-Length': '0'} 2022-09-02T05:56:00.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebff72c0ef93b2ff-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.898 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:56:00.899 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:56:00.899 DEBUG:boto:Method: GET 2022-09-02T05:56:00.900 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:56:00.900 DEBUG:boto:Data: 2022-09-02T05:56:00.901 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.902 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.903 DEBUG:boto:Params: {} 2022-09-02T05:56:00.903 DEBUG:boto:Token: None 2022-09-02T05:56:00.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj46 2022-09-02T05:56:00.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5sOEeeRqV61JiOXTbK3kJJY7NwY= 2022-09-02T05:56:00.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5sOEeeRqV61JiOXTbK3kJJY7NwY=', 'Content-Length': '0'} 2022-09-02T05:56:00.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f125de7715ccfce3-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.908 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.909 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.909 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.910 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.910 DEBUG:boto:Method: GET 2022-09-02T05:56:00.911 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.911 DEBUG:boto:Data: 2022-09-02T05:56:00.912 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.913 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.913 DEBUG:boto:Params: {} 2022-09-02T05:56:00.914 DEBUG:boto:Token: None 2022-09-02T05:56:00.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e26b79ce754a19-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.919 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.920 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:00.921 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:00.921 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.922 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.922 DEBUG:boto:Method: GET 2022-09-02T05:56:00.923 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:56:00.924 DEBUG:boto:Data: 2022-09-02T05:56:00.924 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.925 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.926 DEBUG:boto:Params: {} 2022-09-02T05:56:00.927 DEBUG:boto:Token: None 2022-09-02T05:56:00.927 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/ 2022-09-02T05:56:00.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30= 2022-09-02T05:56:00.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Pa2QC6cfbD4lvuEP5hzhjdrJd30=', 'Content-Length': '0'} 2022-09-02T05:56:00.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368cbcffc9665d40-0063119af0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.932 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:00.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:56:00.933 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:00.934 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:00.934 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.935 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.935 DEBUG:boto:Method: GET 2022-09-02T05:56:00.936 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.936 DEBUG:boto:Data: 2022-09-02T05:56:00.937 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.938 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.938 DEBUG:boto:Params: {} 2022-09-02T05:56:00.939 DEBUG:boto:Token: None 2022-09-02T05:56:00.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8Ek+KdoEDlmaWr1OExv0sLvf3Cg= 2022-09-02T05:56:00.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8Ek+KdoEDlmaWr1OExv0sLvf3Cg=', 'Content-Length': '0'} 2022-09-02T05:56:00.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd836e5a499abe0-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.944 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:00.944 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:00.945 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.945 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.946 DEBUG:boto:Method: GET 2022-09-02T05:56:00.947 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.947 DEBUG:boto:Data: 2022-09-02T05:56:00.948 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.948 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.949 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.949 DEBUG:boto:Params: {} 2022-09-02T05:56:00.950 DEBUG:boto:Token: None 2022-09-02T05:56:00.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8Ek+KdoEDlmaWr1OExv0sLvf3Cg= 2022-09-02T05:56:00.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8Ek+KdoEDlmaWr1OExv0sLvf3Cg=', 'Content-Length': '0'} 2022-09-02T05:56:00.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa5d444cf927e2e-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.955 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:00.955 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:00.956 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.956 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.957 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.957 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.958 DEBUG:boto:Data: 2022-09-02T05:56:00.958 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.959 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.960 DEBUG:boto:Params: {} 2022-09-02T05:56:00.961 DEBUG:boto:Token: None 2022-09-02T05:56:00.961 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lVP3Cz21NqoRn3WdCWjycDsme+8= 2022-09-02T05:56:00.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lVP3Cz21NqoRn3WdCWjycDsme+8=', 'Content-Length': '0'} 2022-09-02T05:56:00.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f313c8ca637b7581-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.965 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:00.966 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:00.966 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.967 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.967 DEBUG:boto:Method: HEAD 2022-09-02T05:56:00.968 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.969 DEBUG:boto:Data: 2022-09-02T05:56:00.969 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:00.970 DEBUG:boto:Port: 8000 2022-09-02T05:56:00.971 DEBUG:boto:Params: {} 2022-09-02T05:56:00.971 DEBUG:boto:Token: None 2022-09-02T05:56:00.972 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lVP3Cz21NqoRn3WdCWjycDsme+8= 2022-09-02T05:56:00.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lVP3Cz21NqoRn3WdCWjycDsme+8=', 'Content-Length': '0'} 2022-09-02T05:56:00.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd30ffdc031afcf7-0063119af0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:00.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:56:00.977 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:00.977 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:00.978 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.978 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.979 DEBUG:boto:Method: GET 2022-09-02T05:56:00.979 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.980 DEBUG:boto:Data: 2022-09-02T05:56:00.980 DEBUG:boto:Headers: {} 2022-09-02T05:56:00.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:00.982 DEBUG:boto:Port: 8001 2022-09-02T05:56:00.982 DEBUG:boto:Params: {} 2022-09-02T05:56:00.983 DEBUG:boto:Token: None 2022-09-02T05:56:00.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:00 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:00.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8Ek+KdoEDlmaWr1OExv0sLvf3Cg= 2022-09-02T05:56:00.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8Ek+KdoEDlmaWr1OExv0sLvf3Cg=', 'Content-Length': '0'} 2022-09-02T05:56:00.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020257c2d79619046-0063119af0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.034 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:01.035 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:01.036 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:01.036 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:56:01.037 DEBUG:boto:Method: GET 2022-09-02T05:56:01.037 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:01.039 DEBUG:boto:Data: 2022-09-02T05:56:01.039 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.040 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.040 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.041 DEBUG:boto:Params: {} 2022-09-02T05:56:01.042 DEBUG:boto:Token: None 2022-09-02T05:56:01.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:56:01.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6XOH2X7002q+y0CRcW29P/wnbkE= 2022-09-02T05:56:01.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6XOH2X7002q+y0CRcW29P/wnbkE=', 'Content-Length': '0'} 2022-09-02T05:56:01.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6c54f4c4cbb8c4-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.047 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.048 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.048 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.049 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.049 DEBUG:boto:Method: GET 2022-09-02T05:56:01.050 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.051 DEBUG:boto:Data: 2022-09-02T05:56:01.051 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.052 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.053 DEBUG:boto:Params: {} 2022-09-02T05:56:01.054 DEBUG:boto:Token: None 2022-09-02T05:56:01.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d85faf0b023ea89c-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.059 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.060 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.061 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.061 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.062 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.062 DEBUG:boto:Method: GET 2022-09-02T05:56:01.063 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.064 DEBUG:boto:Data: 2022-09-02T05:56:01.064 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.065 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.065 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.066 DEBUG:boto:Params: {} 2022-09-02T05:56:01.066 DEBUG:boto:Token: None 2022-09-02T05:56:01.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000433031eb957b3c16-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.121 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.122 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:01.122 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:01.123 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.123 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:56:01.124 DEBUG:boto:Data: 2022-09-02T05:56:01.124 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.126 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.126 DEBUG:boto:Params: {} 2022-09-02T05:56:01.127 DEBUG:boto:Token: None 2022-09-02T05:56:01.127 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj51 2022-09-02T05:56:01.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5XC1RWAoLlXLZ+YgHHswJoMfWds= 2022-09-02T05:56:01.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5XC1RWAoLlXLZ+YgHHswJoMfWds=', 'Content-Length': '0'} 2022-09-02T05:56:01.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001070206da101baa9-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.134 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:01.135 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:01.136 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.136 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:56:01.137 DEBUG:boto:Data: 2022-09-02T05:56:01.137 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.138 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.138 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.139 DEBUG:boto:Params: {} 2022-09-02T05:56:01.140 DEBUG:boto:Token: None 2022-09-02T05:56:01.140 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj51 2022-09-02T05:56:01.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5XC1RWAoLlXLZ+YgHHswJoMfWds= 2022-09-02T05:56:01.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5XC1RWAoLlXLZ+YgHHswJoMfWds=', 'Content-Length': '0'} 2022-09-02T05:56:01.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e6b0e11536040b-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:56:01.147 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:01.147 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:01.148 DEBUG:boto:Method: GET 2022-09-02T05:56:01.149 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:56:01.149 DEBUG:boto:Data: 2022-09-02T05:56:01.150 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.151 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.152 DEBUG:boto:Params: {} 2022-09-02T05:56:01.152 DEBUG:boto:Token: None 2022-09-02T05:56:01.153 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj51 2022-09-02T05:56:01.154 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSesjcMB0VQ0QJxuHmaVlA5O85c= 2022-09-02T05:56:01.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSesjcMB0VQ0QJxuHmaVlA5O85c=', 'Content-Length': '0'} 2022-09-02T05:56:01.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068499d37d0bc8bc1-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.158 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:56:01.159 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:56:01.160 DEBUG:boto:Method: GET 2022-09-02T05:56:01.160 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:56:01.161 DEBUG:boto:Data: 2022-09-02T05:56:01.162 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.163 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.164 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.164 DEBUG:boto:Params: {} 2022-09-02T05:56:01.165 DEBUG:boto:Token: None 2022-09-02T05:56:01.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj51 2022-09-02T05:56:01.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eSesjcMB0VQ0QJxuHmaVlA5O85c= 2022-09-02T05:56:01.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eSesjcMB0VQ0QJxuHmaVlA5O85c=', 'Content-Length': '0'} 2022-09-02T05:56:01.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df68c518b12839e0-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.171 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.172 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.172 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.173 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.174 DEBUG:boto:Method: GET 2022-09-02T05:56:01.175 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.175 DEBUG:boto:Data: 2022-09-02T05:56:01.176 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.178 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.179 DEBUG:boto:Params: {} 2022-09-02T05:56:01.179 DEBUG:boto:Token: None 2022-09-02T05:56:01.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a1e0260e7ee2c7-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.186 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.187 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.188 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.189 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.190 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.190 DEBUG:boto:Method: GET 2022-09-02T05:56:01.191 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:56:01.192 DEBUG:boto:Data: 2022-09-02T05:56:01.193 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.193 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.194 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.195 DEBUG:boto:Params: {} 2022-09-02T05:56:01.196 DEBUG:boto:Token: None 2022-09-02T05:56:01.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9cf53116daa797-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.202 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:56:01.204 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.205 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.205 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.206 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.207 DEBUG:boto:Method: GET 2022-09-02T05:56:01.208 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.209 DEBUG:boto:Data: 2022-09-02T05:56:01.209 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.211 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.212 DEBUG:boto:Params: {} 2022-09-02T05:56:01.213 DEBUG:boto:Token: None 2022-09-02T05:56:01.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas= 2022-09-02T05:56:01.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas=', 'Content-Length': '0'} 2022-09-02T05:56:01.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aba8f6f71554adc-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.219 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.220 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.221 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.222 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.223 DEBUG:boto:Method: GET 2022-09-02T05:56:01.224 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.224 DEBUG:boto:Data: 2022-09-02T05:56:01.225 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.227 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.228 DEBUG:boto:Params: {} 2022-09-02T05:56:01.230 DEBUG:boto:Token: None 2022-09-02T05:56:01.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas= 2022-09-02T05:56:01.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas=', 'Content-Length': '0'} 2022-09-02T05:56:01.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bca3755d2c519436-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.240 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.241 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.242 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.243 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.244 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.244 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.245 DEBUG:boto:Data: 2022-09-02T05:56:01.246 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.248 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.249 DEBUG:boto:Params: {} 2022-09-02T05:56:01.250 DEBUG:boto:Token: None 2022-09-02T05:56:01.251 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UHVLZMsSkyxPHxGZBYitXfE1F3Q= 2022-09-02T05:56:01.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UHVLZMsSkyxPHxGZBYitXfE1F3Q=', 'Content-Length': '0'} 2022-09-02T05:56:01.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea069dc388a86732-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.258 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.258 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.259 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.260 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.261 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.261 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.262 DEBUG:boto:Data: 2022-09-02T05:56:01.263 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.265 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.266 DEBUG:boto:Params: {} 2022-09-02T05:56:01.267 DEBUG:boto:Token: None 2022-09-02T05:56:01.268 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.269 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UHVLZMsSkyxPHxGZBYitXfE1F3Q= 2022-09-02T05:56:01.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UHVLZMsSkyxPHxGZBYitXfE1F3Q=', 'Content-Length': '0'} 2022-09-02T05:56:01.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b32b3868ebc529-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:56:01.273 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.274 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.274 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.275 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.275 DEBUG:boto:Method: GET 2022-09-02T05:56:01.276 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.276 DEBUG:boto:Data: 2022-09-02T05:56:01.277 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.278 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.279 DEBUG:boto:Params: {} 2022-09-02T05:56:01.279 DEBUG:boto:Token: None 2022-09-02T05:56:01.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas= 2022-09-02T05:56:01.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas=', 'Content-Length': '0'} 2022-09-02T05:56:01.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90ed47795055c0d-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.330 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.331 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.331 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.332 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.332 DEBUG:boto:Method: GET 2022-09-02T05:56:01.333 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.333 DEBUG:boto:Data: 2022-09-02T05:56:01.335 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.336 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.336 DEBUG:boto:Params: {} 2022-09-02T05:56:01.337 DEBUG:boto:Token: None 2022-09-02T05:56:01.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:56:01.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas= 2022-09-02T05:56:01.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NtqlaH0z1YC6LRVs514D5HftLas=', 'Content-Length': '0'} 2022-09-02T05:56:01.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082eb9540657b3654-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.342 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.343 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.344 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.344 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.345 DEBUG:boto:Method: GET 2022-09-02T05:56:01.345 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.346 DEBUG:boto:Data: 2022-09-02T05:56:01.346 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.347 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.348 DEBUG:boto:Params: {} 2022-09-02T05:56:01.348 DEBUG:boto:Token: None 2022-09-02T05:56:01.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087ff6668013d291a-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.354 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.355 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.355 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.356 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.356 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.357 DEBUG:boto:Method: GET 2022-09-02T05:56:01.358 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.358 DEBUG:boto:Data: 2022-09-02T05:56:01.359 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.359 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.360 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.360 DEBUG:boto:Params: {} 2022-09-02T05:56:01.361 DEBUG:boto:Token: None 2022-09-02T05:56:01.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0014c92ca9e09d-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.367 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.367 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.368 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.368 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.369 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:56:01.369 DEBUG:boto:Data: 2022-09-02T05:56:01.370 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.371 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.371 DEBUG:boto:Params: {} 2022-09-02T05:56:01.372 DEBUG:boto:Token: None 2022-09-02T05:56:01.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55 2022-09-02T05:56:01.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JPlDwcm5YWiSdMYHJogTRrMJlj8= 2022-09-02T05:56:01.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JPlDwcm5YWiSdMYHJogTRrMJlj8=', 'Content-Length': '0'} 2022-09-02T05:56:01.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6c0aa896af9b92-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.377 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.377 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.378 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.378 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:56:01.379 DEBUG:boto:Data: 2022-09-02T05:56:01.379 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.380 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.381 DEBUG:boto:Params: {} 2022-09-02T05:56:01.381 DEBUG:boto:Token: None 2022-09-02T05:56:01.382 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55 2022-09-02T05:56:01.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JPlDwcm5YWiSdMYHJogTRrMJlj8= 2022-09-02T05:56:01.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JPlDwcm5YWiSdMYHJogTRrMJlj8=', 'Content-Length': '0'} 2022-09-02T05:56:01.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000781439aa0f5c6c8d-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:56:01.387 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.387 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.388 DEBUG:boto:Method: GET 2022-09-02T05:56:01.389 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:56:01.390 DEBUG:boto:Data: 2022-09-02T05:56:01.391 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.392 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.392 DEBUG:boto:Params: {} 2022-09-02T05:56:01.393 DEBUG:boto:Token: None 2022-09-02T05:56:01.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55 2022-09-02T05:56:01.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tIfLgRvmLEede5ayPoN69GQbMak= 2022-09-02T05:56:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tIfLgRvmLEede5ayPoN69GQbMak=', 'Content-Length': '0'} 2022-09-02T05:56:01.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c4b5d447ec1d35-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.398 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:56:01.399 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:56:01.399 DEBUG:boto:Method: GET 2022-09-02T05:56:01.400 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:56:01.400 DEBUG:boto:Data: 2022-09-02T05:56:01.401 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.401 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.402 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.402 DEBUG:boto:Params: {} 2022-09-02T05:56:01.403 DEBUG:boto:Token: None 2022-09-02T05:56:01.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj55 2022-09-02T05:56:01.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tIfLgRvmLEede5ayPoN69GQbMak= 2022-09-02T05:56:01.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tIfLgRvmLEede5ayPoN69GQbMak=', 'Content-Length': '0'} 2022-09-02T05:56:01.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8bed6b18f2c0b6-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.450 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.451 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.452 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.452 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.453 DEBUG:boto:Method: GET 2022-09-02T05:56:01.453 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.454 DEBUG:boto:Data: 2022-09-02T05:56:01.455 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.457 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.457 DEBUG:boto:Params: {} 2022-09-02T05:56:01.458 DEBUG:boto:Token: None 2022-09-02T05:56:01.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a699934a3ec62977-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.467 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.468 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.469 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.470 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.470 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.471 DEBUG:boto:Method: GET 2022-09-02T05:56:01.471 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:56:01.472 DEBUG:boto:Data: 2022-09-02T05:56:01.473 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.474 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.474 DEBUG:boto:Params: {} 2022-09-02T05:56:01.475 DEBUG:boto:Token: None 2022-09-02T05:56:01.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a31b42e5191f39b0-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.481 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:56:01.482 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.483 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.483 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.484 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.484 DEBUG:boto:Method: GET 2022-09-02T05:56:01.485 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.486 DEBUG:boto:Data: 2022-09-02T05:56:01.486 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.487 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.488 DEBUG:boto:Params: {} 2022-09-02T05:56:01.489 DEBUG:boto:Token: None 2022-09-02T05:56:01.489 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps= 2022-09-02T05:56:01.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps=', 'Content-Length': '0'} 2022-09-02T05:56:01.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4192afb906e2b2-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.494 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.495 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.496 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.497 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.498 DEBUG:boto:Method: GET 2022-09-02T05:56:01.499 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.500 DEBUG:boto:Data: 2022-09-02T05:56:01.500 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.502 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.503 DEBUG:boto:Params: {} 2022-09-02T05:56:01.504 DEBUG:boto:Token: None 2022-09-02T05:56:01.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps= 2022-09-02T05:56:01.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps=', 'Content-Length': '0'} 2022-09-02T05:56:01.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061dab704b4e1e16a-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.511 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.512 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.513 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.514 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.514 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.515 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.515 DEBUG:boto:Data: 2022-09-02T05:56:01.516 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.517 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.517 DEBUG:boto:Params: {} 2022-09-02T05:56:01.518 DEBUG:boto:Token: None 2022-09-02T05:56:01.518 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9X77HLSpHXlFqUdgUUkdsFK+cd8= 2022-09-02T05:56:01.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9X77HLSpHXlFqUdgUUkdsFK+cd8=', 'Content-Length': '0'} 2022-09-02T05:56:01.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6d18b1d72f9994-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.522 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.523 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.523 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.524 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.525 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.525 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.525 DEBUG:boto:Data: 2022-09-02T05:56:01.526 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.527 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.528 DEBUG:boto:Params: {} 2022-09-02T05:56:01.528 DEBUG:boto:Token: None 2022-09-02T05:56:01.529 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9X77HLSpHXlFqUdgUUkdsFK+cd8= 2022-09-02T05:56:01.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9X77HLSpHXlFqUdgUUkdsFK+cd8=', 'Content-Length': '0'} 2022-09-02T05:56:01.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000effa30da6555b6f-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:56:01.534 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.535 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.535 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.536 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.536 DEBUG:boto:Method: GET 2022-09-02T05:56:01.537 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.537 DEBUG:boto:Data: 2022-09-02T05:56:01.538 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.539 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.540 DEBUG:boto:Params: {} 2022-09-02T05:56:01.540 DEBUG:boto:Token: None 2022-09-02T05:56:01.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps= 2022-09-02T05:56:01.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps=', 'Content-Length': '0'} 2022-09-02T05:56:01.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d907594d7ed8ba2-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.547 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.548 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.549 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.550 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.551 DEBUG:boto:Method: GET 2022-09-02T05:56:01.552 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.552 DEBUG:boto:Data: 2022-09-02T05:56:01.553 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.555 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.556 DEBUG:boto:Params: {} 2022-09-02T05:56:01.557 DEBUG:boto:Token: None 2022-09-02T05:56:01.558 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:56:01.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps= 2022-09-02T05:56:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YFrogxT1U3V0WHWS8L9DG1ALJps=', 'Content-Length': '0'} 2022-09-02T05:56:01.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a504525b696715c1-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.563 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.563 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.564 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.564 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.565 DEBUG:boto:Method: GET 2022-09-02T05:56:01.566 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.566 DEBUG:boto:Data: 2022-09-02T05:56:01.567 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.569 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.570 DEBUG:boto:Params: {} 2022-09-02T05:56:01.571 DEBUG:boto:Token: None 2022-09-02T05:56:01.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d32e9c296565c99a-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.578 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.579 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.580 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.580 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.581 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.581 DEBUG:boto:Method: GET 2022-09-02T05:56:01.582 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.582 DEBUG:boto:Data: 2022-09-02T05:56:01.583 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.584 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.584 DEBUG:boto:Params: {} 2022-09-02T05:56:01.585 DEBUG:boto:Token: None 2022-09-02T05:56:01.586 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d60ef01185547e8-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.630 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.631 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.632 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.633 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.633 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:56:01.634 DEBUG:boto:Data: 2022-09-02T05:56:01.634 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.635 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.636 DEBUG:boto:Params: {} 2022-09-02T05:56:01.636 DEBUG:boto:Token: None 2022-09-02T05:56:01.637 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57 2022-09-02T05:56:01.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqK9Y3fiRcgI2iq8iHFIxL+yJe0= 2022-09-02T05:56:01.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqK9Y3fiRcgI2iq8iHFIxL+yJe0=', 'Content-Length': '0'} 2022-09-02T05:56:01.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eaf97fe998a825f-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.641 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.642 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.643 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.643 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:56:01.644 DEBUG:boto:Data: 2022-09-02T05:56:01.644 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.646 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.646 DEBUG:boto:Params: {} 2022-09-02T05:56:01.647 DEBUG:boto:Token: None 2022-09-02T05:56:01.647 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57 2022-09-02T05:56:01.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqK9Y3fiRcgI2iq8iHFIxL+yJe0= 2022-09-02T05:56:01.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqK9Y3fiRcgI2iq8iHFIxL+yJe0=', 'Content-Length': '0'} 2022-09-02T05:56:01.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be27ed8fe61e2bdd-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:56:01.652 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.653 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.653 DEBUG:boto:Method: GET 2022-09-02T05:56:01.654 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:56:01.655 DEBUG:boto:Data: 2022-09-02T05:56:01.655 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.656 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.657 DEBUG:boto:Params: {} 2022-09-02T05:56:01.658 DEBUG:boto:Token: None 2022-09-02T05:56:01.658 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57 2022-09-02T05:56:01.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nli3x9UGqxmQfJiBvS4yyslDkN0= 2022-09-02T05:56:01.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nli3x9UGqxmQfJiBvS4yyslDkN0=', 'Content-Length': '0'} 2022-09-02T05:56:01.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae8f7165dc80b9ba-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.663 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:56:01.663 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:56:01.664 DEBUG:boto:Method: GET 2022-09-02T05:56:01.664 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:56:01.665 DEBUG:boto:Data: 2022-09-02T05:56:01.665 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.666 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.666 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.667 DEBUG:boto:Params: {} 2022-09-02T05:56:01.668 DEBUG:boto:Token: None 2022-09-02T05:56:01.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj57 2022-09-02T05:56:01.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nli3x9UGqxmQfJiBvS4yyslDkN0= 2022-09-02T05:56:01.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nli3x9UGqxmQfJiBvS4yyslDkN0=', 'Content-Length': '0'} 2022-09-02T05:56:01.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022d389205051aa44-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.672 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.673 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.673 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.674 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.675 DEBUG:boto:Method: GET 2022-09-02T05:56:01.675 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.676 DEBUG:boto:Data: 2022-09-02T05:56:01.676 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.677 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.678 DEBUG:boto:Params: {} 2022-09-02T05:56:01.679 DEBUG:boto:Token: None 2022-09-02T05:56:01.679 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.680 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed30540ecaff7d1e-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.684 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.685 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.686 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.686 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.687 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.688 DEBUG:boto:Method: GET 2022-09-02T05:56:01.688 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:56:01.689 DEBUG:boto:Data: 2022-09-02T05:56:01.689 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.690 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.691 DEBUG:boto:Params: {} 2022-09-02T05:56:01.692 DEBUG:boto:Token: None 2022-09-02T05:56:01.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007524aec4a20ad2b3-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.697 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:56:01.698 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.699 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.699 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.700 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.701 DEBUG:boto:Method: GET 2022-09-02T05:56:01.701 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.702 DEBUG:boto:Data: 2022-09-02T05:56:01.702 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.703 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.704 DEBUG:boto:Params: {} 2022-09-02T05:56:01.705 DEBUG:boto:Token: None 2022-09-02T05:56:01.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4= 2022-09-02T05:56:01.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4=', 'Content-Length': '0'} 2022-09-02T05:56:01.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50b7fb537af91dc-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.750 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.751 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.751 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.752 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.752 DEBUG:boto:Method: GET 2022-09-02T05:56:01.753 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.754 DEBUG:boto:Data: 2022-09-02T05:56:01.755 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.756 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.756 DEBUG:boto:Params: {} 2022-09-02T05:56:01.757 DEBUG:boto:Token: None 2022-09-02T05:56:01.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4= 2022-09-02T05:56:01.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4=', 'Content-Length': '0'} 2022-09-02T05:56:01.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a0895f26b619f3-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.762 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.763 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.763 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.764 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.765 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.765 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.766 DEBUG:boto:Data: 2022-09-02T05:56:01.767 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.768 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.769 DEBUG:boto:Params: {} 2022-09-02T05:56:01.770 DEBUG:boto:Token: None 2022-09-02T05:56:01.771 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.772 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:In45Cv3LkAeGSFZg9Lkt/T6G1g4= 2022-09-02T05:56:01.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:In45Cv3LkAeGSFZg9Lkt/T6G1g4=', 'Content-Length': '0'} 2022-09-02T05:56:01.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a334a7fc1f313770-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.777 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.779 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.779 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.780 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.780 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.781 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.781 DEBUG:boto:Data: 2022-09-02T05:56:01.782 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.783 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.784 DEBUG:boto:Params: {} 2022-09-02T05:56:01.784 DEBUG:boto:Token: None 2022-09-02T05:56:01.785 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:In45Cv3LkAeGSFZg9Lkt/T6G1g4= 2022-09-02T05:56:01.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:In45Cv3LkAeGSFZg9Lkt/T6G1g4=', 'Content-Length': '0'} 2022-09-02T05:56:01.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324ebd23d58ccf4e-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:56:01.790 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.791 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.791 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.792 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.792 DEBUG:boto:Method: GET 2022-09-02T05:56:01.793 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.793 DEBUG:boto:Data: 2022-09-02T05:56:01.794 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.795 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.796 DEBUG:boto:Params: {} 2022-09-02T05:56:01.796 DEBUG:boto:Token: None 2022-09-02T05:56:01.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4= 2022-09-02T05:56:01.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4=', 'Content-Length': '0'} 2022-09-02T05:56:01.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f3a2d222c08ca8-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.801 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.802 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.803 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.803 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.804 DEBUG:boto:Method: GET 2022-09-02T05:56:01.804 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.805 DEBUG:boto:Data: 2022-09-02T05:56:01.806 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.807 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.807 DEBUG:boto:Params: {} 2022-09-02T05:56:01.808 DEBUG:boto:Token: None 2022-09-02T05:56:01.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:56:01.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4= 2022-09-02T05:56:01.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AjX/zsphBz+3gNHDOf6AWLeE0u4=', 'Content-Length': '0'} 2022-09-02T05:56:01.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098fed9c2231be233-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.813 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.814 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.815 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.815 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.816 DEBUG:boto:Method: GET 2022-09-02T05:56:01.816 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.817 DEBUG:boto:Data: 2022-09-02T05:56:01.817 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.819 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.819 DEBUG:boto:Params: {} 2022-09-02T05:56:01.820 DEBUG:boto:Token: None 2022-09-02T05:56:01.821 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554c191898d828ab-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.825 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.827 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.827 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.828 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.829 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.829 DEBUG:boto:Method: GET 2022-09-02T05:56:01.830 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.830 DEBUG:boto:Data: 2022-09-02T05:56:01.831 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.831 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.832 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.832 DEBUG:boto:Params: {} 2022-09-02T05:56:01.833 DEBUG:boto:Token: None 2022-09-02T05:56:01.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038f8695d95ba0b42-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.839 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.840 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.840 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.841 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.842 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:56:01.842 DEBUG:boto:Data: 2022-09-02T05:56:01.843 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.844 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.845 DEBUG:boto:Params: {} 2022-09-02T05:56:01.845 DEBUG:boto:Token: None 2022-09-02T05:56:01.846 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58 2022-09-02T05:56:01.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7KBbNkWEpMOGM9lAqT+Pjj+lLWs= 2022-09-02T05:56:01.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7KBbNkWEpMOGM9lAqT+Pjj+lLWs=', 'Content-Length': '0'} 2022-09-02T05:56:01.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2ea017d7694bb7-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.850 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.850 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.851 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.851 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:56:01.852 DEBUG:boto:Data: 2022-09-02T05:56:01.853 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.854 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.854 DEBUG:boto:Params: {} 2022-09-02T05:56:01.855 DEBUG:boto:Token: None 2022-09-02T05:56:01.856 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58 2022-09-02T05:56:01.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7KBbNkWEpMOGM9lAqT+Pjj+lLWs= 2022-09-02T05:56:01.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7KBbNkWEpMOGM9lAqT+Pjj+lLWs=', 'Content-Length': '0'} 2022-09-02T05:56:01.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7b3888bf8d32e9-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:56:01.860 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.861 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.862 DEBUG:boto:Method: GET 2022-09-02T05:56:01.862 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:56:01.863 DEBUG:boto:Data: 2022-09-02T05:56:01.863 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.865 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.865 DEBUG:boto:Params: {} 2022-09-02T05:56:01.866 DEBUG:boto:Token: None 2022-09-02T05:56:01.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58 2022-09-02T05:56:01.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VyAqzKoWIf4fccJcEB/mtZCW+Ug= 2022-09-02T05:56:01.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VyAqzKoWIf4fccJcEB/mtZCW+Ug=', 'Content-Length': '0'} 2022-09-02T05:56:01.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a57b1209cfd839f-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.871 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:56:01.871 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:56:01.872 DEBUG:boto:Method: GET 2022-09-02T05:56:01.872 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:56:01.873 DEBUG:boto:Data: 2022-09-02T05:56:01.874 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.874 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.875 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.876 DEBUG:boto:Params: {} 2022-09-02T05:56:01.876 DEBUG:boto:Token: None 2022-09-02T05:56:01.877 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj58 2022-09-02T05:56:01.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VyAqzKoWIf4fccJcEB/mtZCW+Ug= 2022-09-02T05:56:01.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VyAqzKoWIf4fccJcEB/mtZCW+Ug=', 'Content-Length': '0'} 2022-09-02T05:56:01.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143b20861abc4f22-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.927 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.928 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.929 DEBUG:boto:Method: GET 2022-09-02T05:56:01.929 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.930 DEBUG:boto:Data: 2022-09-02T05:56:01.930 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.931 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.932 DEBUG:boto:Params: {} 2022-09-02T05:56:01.932 DEBUG:boto:Token: None 2022-09-02T05:56:01.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f7b3202c434b1de-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.938 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.939 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:01.939 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:01.940 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.940 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.941 DEBUG:boto:Method: GET 2022-09-02T05:56:01.941 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:56:01.942 DEBUG:boto:Data: 2022-09-02T05:56:01.943 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.944 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.944 DEBUG:boto:Params: {} 2022-09-02T05:56:01.945 DEBUG:boto:Token: None 2022-09-02T05:56:01.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/ 2022-09-02T05:56:01.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs= 2022-09-02T05:56:01.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S3m+mJc9Tq4vP4zsnCVqhzXRwOs=', 'Content-Length': '0'} 2022-09-02T05:56:01.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054cfacb149681d66-0063119af1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.950 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:01.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:56:01.952 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:01.952 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:01.953 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.953 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.954 DEBUG:boto:Method: GET 2022-09-02T05:56:01.955 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.955 DEBUG:boto:Data: 2022-09-02T05:56:01.956 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.957 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.957 DEBUG:boto:Params: {} 2022-09-02T05:56:01.958 DEBUG:boto:Token: None 2022-09-02T05:56:01.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ISYL0vs6MgrVz32SJqalwz7wggE= 2022-09-02T05:56:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ISYL0vs6MgrVz32SJqalwz7wggE=', 'Content-Length': '0'} 2022-09-02T05:56:01.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad0ee686a673085-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.963 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:01.964 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:01.965 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.965 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.966 DEBUG:boto:Method: GET 2022-09-02T05:56:01.966 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.967 DEBUG:boto:Data: 2022-09-02T05:56:01.967 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.968 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.968 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.969 DEBUG:boto:Params: {} 2022-09-02T05:56:01.970 DEBUG:boto:Token: None 2022-09-02T05:56:01.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ISYL0vs6MgrVz32SJqalwz7wggE= 2022-09-02T05:56:01.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ISYL0vs6MgrVz32SJqalwz7wggE=', 'Content-Length': '0'} 2022-09-02T05:56:01.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f69e03f4a5b752ec-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.975 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:01.975 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:01.976 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.976 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.977 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.978 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.978 DEBUG:boto:Data: 2022-09-02T05:56:01.979 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:01.980 DEBUG:boto:Port: 8001 2022-09-02T05:56:01.981 DEBUG:boto:Params: {} 2022-09-02T05:56:01.981 DEBUG:boto:Token: None 2022-09-02T05:56:01.982 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mw04yvR+OlwbKP+dhlQnAGh3/8o= 2022-09-02T05:56:01.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mw04yvR+OlwbKP+dhlQnAGh3/8o=', 'Content-Length': '0'} 2022-09-02T05:56:01.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa8b24219ff150a-0063119af1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.986 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:01.986 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:01.987 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.987 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.988 DEBUG:boto:Method: HEAD 2022-09-02T05:56:01.988 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.989 DEBUG:boto:Data: 2022-09-02T05:56:01.990 DEBUG:boto:Headers: {} 2022-09-02T05:56:01.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:01.991 DEBUG:boto:Port: 8000 2022-09-02T05:56:01.991 DEBUG:boto:Params: {} 2022-09-02T05:56:01.992 DEBUG:boto:Token: None 2022-09-02T05:56:01.992 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:01 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mw04yvR+OlwbKP+dhlQnAGh3/8o= 2022-09-02T05:56:01.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mw04yvR+OlwbKP+dhlQnAGh3/8o=', 'Content-Length': '0'} 2022-09-02T05:56:01.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8781ea06a685a03-0063119af1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:01.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:56:01.997 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:01.998 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:01.999 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:01.999 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.000 DEBUG:boto:Method: GET 2022-09-02T05:56:02.000 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.001 DEBUG:boto:Data: 2022-09-02T05:56:02.001 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.003 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.003 DEBUG:boto:Params: {} 2022-09-02T05:56:02.004 DEBUG:boto:Token: None 2022-09-02T05:56:02.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6Yi1uQVxAEt65n+4owA13jHcEhs= 2022-09-02T05:56:02.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6Yi1uQVxAEt65n+4owA13jHcEhs=', 'Content-Length': '0'} 2022-09-02T05:56:02.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036456dc5373f4b8-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.011 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:02.012 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:02.012 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.013 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.014 DEBUG:boto:Method: GET 2022-09-02T05:56:02.014 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.015 DEBUG:boto:Data: 2022-09-02T05:56:02.015 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.016 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.016 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.017 DEBUG:boto:Params: {} 2022-09-02T05:56:02.017 DEBUG:boto:Token: None 2022-09-02T05:56:02.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:56:02.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6Yi1uQVxAEt65n+4owA13jHcEhs= 2022-09-02T05:56:02.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6Yi1uQVxAEt65n+4owA13jHcEhs=', 'Content-Length': '0'} 2022-09-02T05:56:02.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053051cb21d2abd34-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.023 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.023 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.024 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.025 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.026 DEBUG:boto:Method: GET 2022-09-02T05:56:02.026 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.027 DEBUG:boto:Data: 2022-09-02T05:56:02.027 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.028 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.029 DEBUG:boto:Params: {} 2022-09-02T05:56:02.030 DEBUG:boto:Token: None 2022-09-02T05:56:02.032 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc7549861e8a34f1-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.078 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.080 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.081 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.082 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.083 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.084 DEBUG:boto:Method: GET 2022-09-02T05:56:02.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.086 DEBUG:boto:Data: 2022-09-02T05:56:02.086 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.088 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.088 DEBUG:boto:Params: {} 2022-09-02T05:56:02.089 DEBUG:boto:Token: None 2022-09-02T05:56:02.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079bf088a72991ca8-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.094 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.095 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:02.095 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:02.096 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.096 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:56:02.097 DEBUG:boto:Data: 2022-09-02T05:56:02.098 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.099 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.099 DEBUG:boto:Params: {} 2022-09-02T05:56:02.100 DEBUG:boto:Token: None 2022-09-02T05:56:02.100 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj68 2022-09-02T05:56:02.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kDVTHUd/q7GBmfF0wQfpkF4K2I0= 2022-09-02T05:56:02.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kDVTHUd/q7GBmfF0wQfpkF4K2I0=', 'Content-Length': '0'} 2022-09-02T05:56:02.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9397ccd157e5926-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.104 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:02.105 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:02.105 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.106 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:56:02.106 DEBUG:boto:Data: 2022-09-02T05:56:02.107 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.108 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.108 DEBUG:boto:Params: {} 2022-09-02T05:56:02.109 DEBUG:boto:Token: None 2022-09-02T05:56:02.110 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj68 2022-09-02T05:56:02.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kDVTHUd/q7GBmfF0wQfpkF4K2I0= 2022-09-02T05:56:02.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kDVTHUd/q7GBmfF0wQfpkF4K2I0=', 'Content-Length': '0'} 2022-09-02T05:56:02.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c43855518c164fc-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:56:02.114 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:02.114 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:02.115 DEBUG:boto:Method: GET 2022-09-02T05:56:02.116 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:56:02.116 DEBUG:boto:Data: 2022-09-02T05:56:02.117 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.119 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.119 DEBUG:boto:Params: {} 2022-09-02T05:56:02.120 DEBUG:boto:Token: None 2022-09-02T05:56:02.121 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj68 2022-09-02T05:56:02.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u3WHkuJ5h5Ii+RmltiwEdRe4g+o= 2022-09-02T05:56:02.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u3WHkuJ5h5Ii+RmltiwEdRe4g+o=', 'Content-Length': '0'} 2022-09-02T05:56:02.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d2d1583e122695-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.125 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:56:02.125 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:56:02.126 DEBUG:boto:Method: GET 2022-09-02T05:56:02.127 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:56:02.127 DEBUG:boto:Data: 2022-09-02T05:56:02.128 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.129 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.129 DEBUG:boto:Params: {} 2022-09-02T05:56:02.130 DEBUG:boto:Token: None 2022-09-02T05:56:02.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj68 2022-09-02T05:56:02.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u3WHkuJ5h5Ii+RmltiwEdRe4g+o= 2022-09-02T05:56:02.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u3WHkuJ5h5Ii+RmltiwEdRe4g+o=', 'Content-Length': '0'} 2022-09-02T05:56:02.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8369c15b1ab2a81-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.135 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.135 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.136 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.136 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.137 DEBUG:boto:Method: GET 2022-09-02T05:56:02.137 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.138 DEBUG:boto:Data: 2022-09-02T05:56:02.138 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.140 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.140 DEBUG:boto:Params: {} 2022-09-02T05:56:02.141 DEBUG:boto:Token: None 2022-09-02T05:56:02.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33e5831972a8e09-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.147 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.148 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.149 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.150 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.151 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.152 DEBUG:boto:Method: GET 2022-09-02T05:56:02.153 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:56:02.154 DEBUG:boto:Data: 2022-09-02T05:56:02.155 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.157 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.158 DEBUG:boto:Params: {} 2022-09-02T05:56:02.159 DEBUG:boto:Token: None 2022-09-02T05:56:02.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63d9187af47716f-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.165 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:56:02.167 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.167 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.168 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.168 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.169 DEBUG:boto:Method: GET 2022-09-02T05:56:02.169 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.170 DEBUG:boto:Data: 2022-09-02T05:56:02.170 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.171 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.172 DEBUG:boto:Params: {} 2022-09-02T05:56:02.173 DEBUG:boto:Token: None 2022-09-02T05:56:02.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.174 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54= 2022-09-02T05:56:02.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54=', 'Content-Length': '0'} 2022-09-02T05:56:02.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9eebd4da9e51294-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.178 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.179 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.179 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.180 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.180 DEBUG:boto:Method: GET 2022-09-02T05:56:02.181 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.182 DEBUG:boto:Data: 2022-09-02T05:56:02.182 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.183 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.184 DEBUG:boto:Params: {} 2022-09-02T05:56:02.184 DEBUG:boto:Token: None 2022-09-02T05:56:02.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54= 2022-09-02T05:56:02.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54=', 'Content-Length': '0'} 2022-09-02T05:56:02.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005966a97960678beb-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.189 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.190 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.190 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.191 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.191 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.192 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.192 DEBUG:boto:Data: 2022-09-02T05:56:02.193 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.194 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.194 DEBUG:boto:Params: {} 2022-09-02T05:56:02.195 DEBUG:boto:Token: None 2022-09-02T05:56:02.195 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A8Qi5XpeBYjjv4jahVGCG5FiyLw= 2022-09-02T05:56:02.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A8Qi5XpeBYjjv4jahVGCG5FiyLw=', 'Content-Length': '0'} 2022-09-02T05:56:02.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584e8612b54fdbab-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.200 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.200 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.201 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.201 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.202 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.202 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.203 DEBUG:boto:Data: 2022-09-02T05:56:02.203 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.204 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.204 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.205 DEBUG:boto:Params: {} 2022-09-02T05:56:02.205 DEBUG:boto:Token: None 2022-09-02T05:56:02.206 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A8Qi5XpeBYjjv4jahVGCG5FiyLw= 2022-09-02T05:56:02.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A8Qi5XpeBYjjv4jahVGCG5FiyLw=', 'Content-Length': '0'} 2022-09-02T05:56:02.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b20af40114bdae-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:56:02.211 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.211 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.212 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.212 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.213 DEBUG:boto:Method: GET 2022-09-02T05:56:02.214 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.214 DEBUG:boto:Data: 2022-09-02T05:56:02.215 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.216 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.217 DEBUG:boto:Params: {} 2022-09-02T05:56:02.217 DEBUG:boto:Token: None 2022-09-02T05:56:02.218 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54= 2022-09-02T05:56:02.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54=', 'Content-Length': '0'} 2022-09-02T05:56:02.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb4a472d25e8af8c-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.222 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.223 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.223 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.224 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.224 DEBUG:boto:Method: GET 2022-09-02T05:56:02.225 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.226 DEBUG:boto:Data: 2022-09-02T05:56:02.226 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.228 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.228 DEBUG:boto:Params: {} 2022-09-02T05:56:02.229 DEBUG:boto:Token: None 2022-09-02T05:56:02.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:56:02.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54= 2022-09-02T05:56:02.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:49PdWYfCSoLiWH/wbfav/gg4E54=', 'Content-Length': '0'} 2022-09-02T05:56:02.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bed36428555e7e22-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.235 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.235 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.236 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.236 DEBUG:boto:Method: GET 2022-09-02T05:56:02.237 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.237 DEBUG:boto:Data: 2022-09-02T05:56:02.238 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.239 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.239 DEBUG:boto:Params: {} 2022-09-02T05:56:02.240 DEBUG:boto:Token: None 2022-09-02T05:56:02.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351fbc639894190c-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.246 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.247 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.247 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.248 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.248 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.249 DEBUG:boto:Method: GET 2022-09-02T05:56:02.250 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.250 DEBUG:boto:Data: 2022-09-02T05:56:02.251 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.252 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.253 DEBUG:boto:Params: {} 2022-09-02T05:56:02.254 DEBUG:boto:Token: None 2022-09-02T05:56:02.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000155262cfd26157a5-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.261 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.262 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.263 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.264 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.265 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:56:02.266 DEBUG:boto:Data: 2022-09-02T05:56:02.267 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.269 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.270 DEBUG:boto:Params: {} 2022-09-02T05:56:02.271 DEBUG:boto:Token: None 2022-09-02T05:56:02.272 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69 2022-09-02T05:56:02.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9nezR7UISI1mv5zq+3g+5faSiIc= 2022-09-02T05:56:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9nezR7UISI1mv5zq+3g+5faSiIc=', 'Content-Length': '0'} 2022-09-02T05:56:02.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017daa7d64917a19f-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.276 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.277 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.277 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.278 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:56:02.279 DEBUG:boto:Data: 2022-09-02T05:56:02.279 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.280 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.280 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.281 DEBUG:boto:Params: {} 2022-09-02T05:56:02.281 DEBUG:boto:Token: None 2022-09-02T05:56:02.282 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69 2022-09-02T05:56:02.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9nezR7UISI1mv5zq+3g+5faSiIc= 2022-09-02T05:56:02.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9nezR7UISI1mv5zq+3g+5faSiIc=', 'Content-Length': '0'} 2022-09-02T05:56:02.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097f9742f94924ba8-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:56:02.286 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.287 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.287 DEBUG:boto:Method: GET 2022-09-02T05:56:02.288 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:56:02.289 DEBUG:boto:Data: 2022-09-02T05:56:02.289 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.290 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.291 DEBUG:boto:Params: {} 2022-09-02T05:56:02.291 DEBUG:boto:Token: None 2022-09-02T05:56:02.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69 2022-09-02T05:56:02.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JaleqfTXODXmhIPFikgCWlmlwvM= 2022-09-02T05:56:02.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JaleqfTXODXmhIPFikgCWlmlwvM=', 'Content-Length': '0'} 2022-09-02T05:56:02.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f06c62b6f5ef324-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.296 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:56:02.297 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:56:02.298 DEBUG:boto:Method: GET 2022-09-02T05:56:02.298 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:56:02.299 DEBUG:boto:Data: 2022-09-02T05:56:02.299 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.300 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.301 DEBUG:boto:Params: {} 2022-09-02T05:56:02.301 DEBUG:boto:Token: None 2022-09-02T05:56:02.302 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj69 2022-09-02T05:56:02.303 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JaleqfTXODXmhIPFikgCWlmlwvM= 2022-09-02T05:56:02.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JaleqfTXODXmhIPFikgCWlmlwvM=', 'Content-Length': '0'} 2022-09-02T05:56:02.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000926f888c671fbfdc-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.346 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.347 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.347 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.348 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.349 DEBUG:boto:Method: GET 2022-09-02T05:56:02.349 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.350 DEBUG:boto:Data: 2022-09-02T05:56:02.351 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.352 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.352 DEBUG:boto:Params: {} 2022-09-02T05:56:02.353 DEBUG:boto:Token: None 2022-09-02T05:56:02.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0b19fc2b81c1c46-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.359 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.360 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.361 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.362 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.362 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.363 DEBUG:boto:Method: GET 2022-09-02T05:56:02.363 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:56:02.364 DEBUG:boto:Data: 2022-09-02T05:56:02.364 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.366 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.366 DEBUG:boto:Params: {} 2022-09-02T05:56:02.367 DEBUG:boto:Token: None 2022-09-02T05:56:02.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003baa12224671a5f8-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.374 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:56:02.376 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.377 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.378 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.379 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.379 DEBUG:boto:Method: GET 2022-09-02T05:56:02.381 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.381 DEBUG:boto:Data: 2022-09-02T05:56:02.382 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.383 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.384 DEBUG:boto:Params: {} 2022-09-02T05:56:02.384 DEBUG:boto:Token: None 2022-09-02T05:56:02.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s= 2022-09-02T05:56:02.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s=', 'Content-Length': '0'} 2022-09-02T05:56:02.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dddf4cab66e0a8f-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.434 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.436 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.436 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.437 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.439 DEBUG:boto:Method: GET 2022-09-02T05:56:02.439 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.440 DEBUG:boto:Data: 2022-09-02T05:56:02.440 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.441 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.442 DEBUG:boto:Params: {} 2022-09-02T05:56:02.443 DEBUG:boto:Token: None 2022-09-02T05:56:02.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s= 2022-09-02T05:56:02.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s=', 'Content-Length': '0'} 2022-09-02T05:56:02.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd363c7672a0ed06-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.448 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.449 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.449 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.450 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.450 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.451 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.451 DEBUG:boto:Data: 2022-09-02T05:56:02.452 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.453 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.454 DEBUG:boto:Params: {} 2022-09-02T05:56:02.454 DEBUG:boto:Token: None 2022-09-02T05:56:02.455 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XJn9eIEYH74Othj/mL+RZ+jj+fI= 2022-09-02T05:56:02.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XJn9eIEYH74Othj/mL+RZ+jj+fI=', 'Content-Length': '0'} 2022-09-02T05:56:02.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f835a9a13570b33-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.459 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.459 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.460 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.460 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.461 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.461 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.462 DEBUG:boto:Data: 2022-09-02T05:56:02.462 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.463 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.463 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.464 DEBUG:boto:Params: {} 2022-09-02T05:56:02.464 DEBUG:boto:Token: None 2022-09-02T05:56:02.465 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XJn9eIEYH74Othj/mL+RZ+jj+fI= 2022-09-02T05:56:02.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XJn9eIEYH74Othj/mL+RZ+jj+fI=', 'Content-Length': '0'} 2022-09-02T05:56:02.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481626e6ac5082aa-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:56:02.470 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.470 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.471 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.471 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.472 DEBUG:boto:Method: GET 2022-09-02T05:56:02.472 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.473 DEBUG:boto:Data: 2022-09-02T05:56:02.473 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.474 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.475 DEBUG:boto:Params: {} 2022-09-02T05:56:02.476 DEBUG:boto:Token: None 2022-09-02T05:56:02.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s= 2022-09-02T05:56:02.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s=', 'Content-Length': '0'} 2022-09-02T05:56:02.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ab6b784f8c76bb-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.482 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.483 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.484 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.484 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.485 DEBUG:boto:Method: GET 2022-09-02T05:56:02.486 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.487 DEBUG:boto:Data: 2022-09-02T05:56:02.488 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.489 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.489 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.490 DEBUG:boto:Params: {} 2022-09-02T05:56:02.491 DEBUG:boto:Token: None 2022-09-02T05:56:02.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:56:02.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s= 2022-09-02T05:56:02.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y93Gag2vNfVw459C9mpetscz5/s=', 'Content-Length': '0'} 2022-09-02T05:56:02.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e232afda52a71d2b-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.496 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.496 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.497 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.497 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.498 DEBUG:boto:Method: GET 2022-09-02T05:56:02.498 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.499 DEBUG:boto:Data: 2022-09-02T05:56:02.499 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.500 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.501 DEBUG:boto:Params: {} 2022-09-02T05:56:02.502 DEBUG:boto:Token: None 2022-09-02T05:56:02.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81826ad32c912bd-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.507 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.508 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.509 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.509 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.510 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.511 DEBUG:boto:Method: GET 2022-09-02T05:56:02.512 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.512 DEBUG:boto:Data: 2022-09-02T05:56:02.513 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.514 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.514 DEBUG:boto:Params: {} 2022-09-02T05:56:02.515 DEBUG:boto:Token: None 2022-09-02T05:56:02.516 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062c1bb3eaed2c91b-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.520 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.521 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.521 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.522 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.523 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:56:02.523 DEBUG:boto:Data: 2022-09-02T05:56:02.524 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.525 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.526 DEBUG:boto:Params: {} 2022-09-02T05:56:02.527 DEBUG:boto:Token: None 2022-09-02T05:56:02.527 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76 2022-09-02T05:56:02.528 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DgDZg486cK88eV9udtkmevDYTsw= 2022-09-02T05:56:02.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DgDZg486cK88eV9udtkmevDYTsw=', 'Content-Length': '0'} 2022-09-02T05:56:02.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3729e85c643ce1d-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.531 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.532 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.533 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.533 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:56:02.534 DEBUG:boto:Data: 2022-09-02T05:56:02.535 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.535 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.536 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.536 DEBUG:boto:Params: {} 2022-09-02T05:56:02.537 DEBUG:boto:Token: None 2022-09-02T05:56:02.537 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76 2022-09-02T05:56:02.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DgDZg486cK88eV9udtkmevDYTsw= 2022-09-02T05:56:02.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DgDZg486cK88eV9udtkmevDYTsw=', 'Content-Length': '0'} 2022-09-02T05:56:02.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bde423e6a25d1dc-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:56:02.542 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.542 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.543 DEBUG:boto:Method: GET 2022-09-02T05:56:02.544 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:56:02.544 DEBUG:boto:Data: 2022-09-02T05:56:02.545 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.546 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.547 DEBUG:boto:Params: {} 2022-09-02T05:56:02.547 DEBUG:boto:Token: None 2022-09-02T05:56:02.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76 2022-09-02T05:56:02.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DilpJwUDMcB78GCwEyRhr7W9WZI= 2022-09-02T05:56:02.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DilpJwUDMcB78GCwEyRhr7W9WZI=', 'Content-Length': '0'} 2022-09-02T05:56:02.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a6d0ada90424d7-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.594 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:56:02.595 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:56:02.596 DEBUG:boto:Method: GET 2022-09-02T05:56:02.597 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:56:02.598 DEBUG:boto:Data: 2022-09-02T05:56:02.598 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.599 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.600 DEBUG:boto:Params: {} 2022-09-02T05:56:02.601 DEBUG:boto:Token: None 2022-09-02T05:56:02.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj76 2022-09-02T05:56:02.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DilpJwUDMcB78GCwEyRhr7W9WZI= 2022-09-02T05:56:02.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DilpJwUDMcB78GCwEyRhr7W9WZI=', 'Content-Length': '0'} 2022-09-02T05:56:02.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6977b6ccc10ad5-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.606 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.607 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.607 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.608 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.608 DEBUG:boto:Method: GET 2022-09-02T05:56:02.609 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.609 DEBUG:boto:Data: 2022-09-02T05:56:02.610 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.611 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.611 DEBUG:boto:Params: {} 2022-09-02T05:56:02.612 DEBUG:boto:Token: None 2022-09-02T05:56:02.613 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e2a2f294e7ed0b-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.617 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.618 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.626 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.626 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.627 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.627 DEBUG:boto:Method: GET 2022-09-02T05:56:02.628 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:56:02.628 DEBUG:boto:Data: 2022-09-02T05:56:02.629 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.630 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.630 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.631 DEBUG:boto:Params: {} 2022-09-02T05:56:02.631 DEBUG:boto:Token: None 2022-09-02T05:56:02.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd88bec6432c9c4-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.637 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:56:02.639 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.639 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.640 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.640 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.641 DEBUG:boto:Method: GET 2022-09-02T05:56:02.642 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.642 DEBUG:boto:Data: 2022-09-02T05:56:02.643 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.644 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.645 DEBUG:boto:Params: {} 2022-09-02T05:56:02.645 DEBUG:boto:Token: None 2022-09-02T05:56:02.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA= 2022-09-02T05:56:02.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA=', 'Content-Length': '0'} 2022-09-02T05:56:02.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce7cb57580fe346-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.651 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.652 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.652 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.653 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.653 DEBUG:boto:Method: GET 2022-09-02T05:56:02.654 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.655 DEBUG:boto:Data: 2022-09-02T05:56:02.655 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.656 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.656 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.657 DEBUG:boto:Params: {} 2022-09-02T05:56:02.657 DEBUG:boto:Token: None 2022-09-02T05:56:02.658 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.659 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA= 2022-09-02T05:56:02.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA=', 'Content-Length': '0'} 2022-09-02T05:56:02.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea54512ce1ed4081-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.665 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.666 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.667 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.667 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.668 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.669 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.670 DEBUG:boto:Data: 2022-09-02T05:56:02.671 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.673 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.674 DEBUG:boto:Params: {} 2022-09-02T05:56:02.675 DEBUG:boto:Token: None 2022-09-02T05:56:02.676 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2sBAxS31ujW0QFPziru9o6RaqDU= 2022-09-02T05:56:02.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2sBAxS31ujW0QFPziru9o6RaqDU=', 'Content-Length': '0'} 2022-09-02T05:56:02.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004528ab20cd54666d-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.682 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.683 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.684 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.685 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.687 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.688 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.689 DEBUG:boto:Data: 2022-09-02T05:56:02.690 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.691 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.692 DEBUG:boto:Params: {} 2022-09-02T05:56:02.694 DEBUG:boto:Token: None 2022-09-02T05:56:02.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2sBAxS31ujW0QFPziru9o6RaqDU= 2022-09-02T05:56:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2sBAxS31ujW0QFPziru9o6RaqDU=', 'Content-Length': '0'} 2022-09-02T05:56:02.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a2e4a01416a93c-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:56:02.700 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.701 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.702 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.702 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.703 DEBUG:boto:Method: GET 2022-09-02T05:56:02.703 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.704 DEBUG:boto:Data: 2022-09-02T05:56:02.704 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.705 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.706 DEBUG:boto:Params: {} 2022-09-02T05:56:02.707 DEBUG:boto:Token: None 2022-09-02T05:56:02.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA= 2022-09-02T05:56:02.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA=', 'Content-Length': '0'} 2022-09-02T05:56:02.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059acf30bec5ec80c-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.713 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.714 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.715 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.716 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.717 DEBUG:boto:Method: GET 2022-09-02T05:56:02.717 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.718 DEBUG:boto:Data: 2022-09-02T05:56:02.719 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.719 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.720 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.720 DEBUG:boto:Params: {} 2022-09-02T05:56:02.721 DEBUG:boto:Token: None 2022-09-02T05:56:02.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:56:02.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA= 2022-09-02T05:56:02.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ILwV3Auu6MPhRXPbKbuyXSRCkA=', 'Content-Length': '0'} 2022-09-02T05:56:02.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff89317091f8f5ea-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.726 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.727 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.727 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.728 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.728 DEBUG:boto:Method: GET 2022-09-02T05:56:02.729 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.730 DEBUG:boto:Data: 2022-09-02T05:56:02.730 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.731 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.732 DEBUG:boto:Params: {} 2022-09-02T05:56:02.732 DEBUG:boto:Token: None 2022-09-02T05:56:02.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa864d520482c46-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.739 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.741 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.742 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.743 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.744 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.745 DEBUG:boto:Method: GET 2022-09-02T05:56:02.746 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.747 DEBUG:boto:Data: 2022-09-02T05:56:02.748 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.750 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.751 DEBUG:boto:Params: {} 2022-09-02T05:56:02.752 DEBUG:boto:Token: None 2022-09-02T05:56:02.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000446be0567afc6948-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.759 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.760 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.760 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.761 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.762 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:56:02.762 DEBUG:boto:Data: 2022-09-02T05:56:02.763 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.764 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.764 DEBUG:boto:Params: {} 2022-09-02T05:56:02.765 DEBUG:boto:Token: None 2022-09-02T05:56:02.765 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85 2022-09-02T05:56:02.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7VhJlNDiTog8nfnWURKqxhPt40E= 2022-09-02T05:56:02.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7VhJlNDiTog8nfnWURKqxhPt40E=', 'Content-Length': '0'} 2022-09-02T05:56:02.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009017c8e652b06579-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.770 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.770 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.771 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.771 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:56:02.772 DEBUG:boto:Data: 2022-09-02T05:56:02.772 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.773 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.774 DEBUG:boto:Params: {} 2022-09-02T05:56:02.774 DEBUG:boto:Token: None 2022-09-02T05:56:02.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85 2022-09-02T05:56:02.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7VhJlNDiTog8nfnWURKqxhPt40E= 2022-09-02T05:56:02.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7VhJlNDiTog8nfnWURKqxhPt40E=', 'Content-Length': '0'} 2022-09-02T05:56:02.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd7285b19da9856-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:56:02.779 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.780 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.780 DEBUG:boto:Method: GET 2022-09-02T05:56:02.781 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:56:02.781 DEBUG:boto:Data: 2022-09-02T05:56:02.782 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.783 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.784 DEBUG:boto:Params: {} 2022-09-02T05:56:02.784 DEBUG:boto:Token: None 2022-09-02T05:56:02.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85 2022-09-02T05:56:02.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bd6dGiNFksqGgjM7D2MqMesA8EE= 2022-09-02T05:56:02.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bd6dGiNFksqGgjM7D2MqMesA8EE=', 'Content-Length': '0'} 2022-09-02T05:56:02.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637ded86b2f42aac-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.830 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:56:02.831 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:56:02.832 DEBUG:boto:Method: GET 2022-09-02T05:56:02.833 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:56:02.834 DEBUG:boto:Data: 2022-09-02T05:56:02.834 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.835 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.836 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.837 DEBUG:boto:Params: {} 2022-09-02T05:56:02.838 DEBUG:boto:Token: None 2022-09-02T05:56:02.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj85 2022-09-02T05:56:02.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bd6dGiNFksqGgjM7D2MqMesA8EE= 2022-09-02T05:56:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bd6dGiNFksqGgjM7D2MqMesA8EE=', 'Content-Length': '0'} 2022-09-02T05:56:02.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001adbfbaf5561ec6e-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.845 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.846 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.846 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.847 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.847 DEBUG:boto:Method: GET 2022-09-02T05:56:02.848 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.849 DEBUG:boto:Data: 2022-09-02T05:56:02.849 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.851 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.851 DEBUG:boto:Params: {} 2022-09-02T05:56:02.852 DEBUG:boto:Token: None 2022-09-02T05:56:02.852 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c31a15ccbebfcec-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.857 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.858 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:02.859 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:02.860 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.861 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.861 DEBUG:boto:Method: GET 2022-09-02T05:56:02.862 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:56:02.863 DEBUG:boto:Data: 2022-09-02T05:56:02.863 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.865 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.865 DEBUG:boto:Params: {} 2022-09-02T05:56:02.866 DEBUG:boto:Token: None 2022-09-02T05:56:02.867 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/ 2022-09-02T05:56:02.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU= 2022-09-02T05:56:02.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vntUyDDIFY7hDbxAwtOOAIPdYpU=', 'Content-Length': '0'} 2022-09-02T05:56:02.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab113a3bc3ca045b-0063119af2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.873 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:02.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:56:02.875 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:02.876 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:02.877 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.878 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.879 DEBUG:boto:Method: GET 2022-09-02T05:56:02.880 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.881 DEBUG:boto:Data: 2022-09-02T05:56:02.882 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.884 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.884 DEBUG:boto:Params: {} 2022-09-02T05:56:02.885 DEBUG:boto:Token: None 2022-09-02T05:56:02.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VwicvCJqp0s28SWbh7g6n+LqecI= 2022-09-02T05:56:02.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VwicvCJqp0s28SWbh7g6n+LqecI=', 'Content-Length': '0'} 2022-09-02T05:56:02.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015de1a26f995dc5a-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.890 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:02.891 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:02.892 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.893 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.894 DEBUG:boto:Method: GET 2022-09-02T05:56:02.895 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.896 DEBUG:boto:Data: 2022-09-02T05:56:02.896 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.897 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.898 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.898 DEBUG:boto:Params: {} 2022-09-02T05:56:02.899 DEBUG:boto:Token: None 2022-09-02T05:56:02.899 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VwicvCJqp0s28SWbh7g6n+LqecI= 2022-09-02T05:56:02.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VwicvCJqp0s28SWbh7g6n+LqecI=', 'Content-Length': '0'} 2022-09-02T05:56:02.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a2d3e1df7de0da4-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.946 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:02.948 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:02.948 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.949 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.950 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.951 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.952 DEBUG:boto:Data: 2022-09-02T05:56:02.952 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.954 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.954 DEBUG:boto:Params: {} 2022-09-02T05:56:02.955 DEBUG:boto:Token: None 2022-09-02T05:56:02.956 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DUXd0383NcnuvHoxOehzAi5R0Os= 2022-09-02T05:56:02.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DUXd0383NcnuvHoxOehzAi5R0Os=', 'Content-Length': '0'} 2022-09-02T05:56:02.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa88a383572f61a1-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.961 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:02.961 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:02.962 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.962 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.963 DEBUG:boto:Method: HEAD 2022-09-02T05:56:02.963 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.964 DEBUG:boto:Data: 2022-09-02T05:56:02.964 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:02.965 DEBUG:boto:Port: 8000 2022-09-02T05:56:02.966 DEBUG:boto:Params: {} 2022-09-02T05:56:02.967 DEBUG:boto:Token: None 2022-09-02T05:56:02.967 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DUXd0383NcnuvHoxOehzAi5R0Os= 2022-09-02T05:56:02.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DUXd0383NcnuvHoxOehzAi5R0Os=', 'Content-Length': '0'} 2022-09-02T05:56:02.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7cb32295b7737c-0063119af2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:56:02.977 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:02.977 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:02.979 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.980 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.981 DEBUG:boto:Method: GET 2022-09-02T05:56:02.982 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.982 DEBUG:boto:Data: 2022-09-02T05:56:02.983 DEBUG:boto:Headers: {} 2022-09-02T05:56:02.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:02.984 DEBUG:boto:Port: 8001 2022-09-02T05:56:02.985 DEBUG:boto:Params: {} 2022-09-02T05:56:02.985 DEBUG:boto:Token: None 2022-09-02T05:56:02.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:02 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VwicvCJqp0s28SWbh7g6n+LqecI= 2022-09-02T05:56:02.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VwicvCJqp0s28SWbh7g6n+LqecI=', 'Content-Length': '0'} 2022-09-02T05:56:02.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e05f68d7ae576e9-0063119af2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:02.992 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:02.993 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:02.994 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.995 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.996 DEBUG:boto:Method: GET 2022-09-02T05:56:02.997 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:02.998 DEBUG:boto:Data: 2022-09-02T05:56:02.999 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.000 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.001 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.002 DEBUG:boto:Params: {} 2022-09-02T05:56:03.003 DEBUG:boto:Token: None 2022-09-02T05:56:03.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:56:03.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:70PAS1ky91PA7+7W8mMg6Cnhj48= 2022-09-02T05:56:03.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:70PAS1ky91PA7+7W8mMg6Cnhj48=', 'Content-Length': '0'} 2022-09-02T05:56:03.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a74b342bd782bb0-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.018 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.019 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.020 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.021 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.022 DEBUG:boto:Method: GET 2022-09-02T05:56:03.023 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.024 DEBUG:boto:Data: 2022-09-02T05:56:03.025 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.027 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.028 DEBUG:boto:Params: {} 2022-09-02T05:56:03.029 DEBUG:boto:Token: None 2022-09-02T05:56:03.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a6dc1c1ade25ff7-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.045 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.046 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.047 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.048 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.049 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.050 DEBUG:boto:Method: GET 2022-09-02T05:56:03.051 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.052 DEBUG:boto:Data: 2022-09-02T05:56:03.053 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.055 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.056 DEBUG:boto:Params: {} 2022-09-02T05:56:03.057 DEBUG:boto:Token: None 2022-09-02T05:56:03.058 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e44b63ae91e4558-0063119af3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.065 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.066 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:03.067 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:03.069 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.070 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:56:03.071 DEBUG:boto:Data: 2022-09-02T05:56:03.073 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.077 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.078 DEBUG:boto:Params: {} 2022-09-02T05:56:03.079 DEBUG:boto:Token: None 2022-09-02T05:56:03.080 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj87 2022-09-02T05:56:03.083 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4OVUxRIpEPztPZ6XtCR6tEkYkGk= 2022-09-02T05:56:03.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4OVUxRIpEPztPZ6XtCR6tEkYkGk=', 'Content-Length': '0'} 2022-09-02T05:56:03.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e079a72423b574cb-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.088 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:03.089 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:03.090 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.091 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:56:03.092 DEBUG:boto:Data: 2022-09-02T05:56:03.093 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.095 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.096 DEBUG:boto:Params: {} 2022-09-02T05:56:03.098 DEBUG:boto:Token: None 2022-09-02T05:56:03.099 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj87 2022-09-02T05:56:03.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4OVUxRIpEPztPZ6XtCR6tEkYkGk= 2022-09-02T05:56:03.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4OVUxRIpEPztPZ6XtCR6tEkYkGk=', 'Content-Length': '0'} 2022-09-02T05:56:03.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2e0f8055410757-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:56:03.108 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:03.109 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:03.109 DEBUG:boto:Method: GET 2022-09-02T05:56:03.110 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:56:03.111 DEBUG:boto:Data: 2022-09-02T05:56:03.111 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.112 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.113 DEBUG:boto:Params: {} 2022-09-02T05:56:03.114 DEBUG:boto:Token: None 2022-09-02T05:56:03.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj87 2022-09-02T05:56:03.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rJ9a0iqWApp3DSKMqpRqQc3sCqM= 2022-09-02T05:56:03.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rJ9a0iqWApp3DSKMqpRqQc3sCqM=', 'Content-Length': '0'} 2022-09-02T05:56:03.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435eb5417021f37c-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.165 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:56:03.167 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:56:03.168 DEBUG:boto:Method: GET 2022-09-02T05:56:03.169 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:56:03.169 DEBUG:boto:Data: 2022-09-02T05:56:03.171 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.172 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.173 DEBUG:boto:Params: {} 2022-09-02T05:56:03.174 DEBUG:boto:Token: None 2022-09-02T05:56:03.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj87 2022-09-02T05:56:03.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rJ9a0iqWApp3DSKMqpRqQc3sCqM= 2022-09-02T05:56:03.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rJ9a0iqWApp3DSKMqpRqQc3sCqM=', 'Content-Length': '0'} 2022-09-02T05:56:03.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f9d6e5f32fd282f-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.186 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.187 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.188 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.188 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.190 DEBUG:boto:Method: GET 2022-09-02T05:56:03.190 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.191 DEBUG:boto:Data: 2022-09-02T05:56:03.191 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.193 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.193 DEBUG:boto:Params: {} 2022-09-02T05:56:03.194 DEBUG:boto:Token: None 2022-09-02T05:56:03.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000708bad4d02103af1-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.209 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.211 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.211 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.212 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.212 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.213 DEBUG:boto:Method: GET 2022-09-02T05:56:03.213 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:56:03.214 DEBUG:boto:Data: 2022-09-02T05:56:03.214 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.215 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.216 DEBUG:boto:Params: {} 2022-09-02T05:56:03.217 DEBUG:boto:Token: None 2022-09-02T05:56:03.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54f4d035d588936-0063119af3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.266 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:56:03.268 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.269 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.269 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.270 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.270 DEBUG:boto:Method: GET 2022-09-02T05:56:03.271 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.271 DEBUG:boto:Data: 2022-09-02T05:56:03.272 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.273 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.274 DEBUG:boto:Params: {} 2022-09-02T05:56:03.274 DEBUG:boto:Token: None 2022-09-02T05:56:03.275 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc= 2022-09-02T05:56:03.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc=', 'Content-Length': '0'} 2022-09-02T05:56:03.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617bc9ca860ca045-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.284 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.285 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.285 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.286 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.286 DEBUG:boto:Method: GET 2022-09-02T05:56:03.287 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.287 DEBUG:boto:Data: 2022-09-02T05:56:03.288 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.289 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.290 DEBUG:boto:Params: {} 2022-09-02T05:56:03.290 DEBUG:boto:Token: None 2022-09-02T05:56:03.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc= 2022-09-02T05:56:03.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc=', 'Content-Length': '0'} 2022-09-02T05:56:03.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdbaa15dd8888ef9-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.300 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.301 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.302 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.302 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.303 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.304 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.304 DEBUG:boto:Data: 2022-09-02T05:56:03.305 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.306 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.307 DEBUG:boto:Params: {} 2022-09-02T05:56:03.308 DEBUG:boto:Token: None 2022-09-02T05:56:03.308 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6jI7V9XQvD078NIGanrXL3UY2P8= 2022-09-02T05:56:03.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6jI7V9XQvD078NIGanrXL3UY2P8=', 'Content-Length': '0'} 2022-09-02T05:56:03.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090f5c003a84dabbe-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.317 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.317 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.318 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.319 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.319 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.320 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.321 DEBUG:boto:Data: 2022-09-02T05:56:03.321 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.322 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.322 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.323 DEBUG:boto:Params: {} 2022-09-02T05:56:03.324 DEBUG:boto:Token: None 2022-09-02T05:56:03.324 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6jI7V9XQvD078NIGanrXL3UY2P8= 2022-09-02T05:56:03.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6jI7V9XQvD078NIGanrXL3UY2P8=', 'Content-Length': '0'} 2022-09-02T05:56:03.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86f18c569cda815-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:56:03.333 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.334 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.334 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.335 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.335 DEBUG:boto:Method: GET 2022-09-02T05:56:03.336 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.336 DEBUG:boto:Data: 2022-09-02T05:56:03.337 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.338 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.339 DEBUG:boto:Params: {} 2022-09-02T05:56:03.339 DEBUG:boto:Token: None 2022-09-02T05:56:03.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc= 2022-09-02T05:56:03.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc=', 'Content-Length': '0'} 2022-09-02T05:56:03.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce6ed1b46186b67-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.390 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.391 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.392 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.393 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.393 DEBUG:boto:Method: GET 2022-09-02T05:56:03.394 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.395 DEBUG:boto:Data: 2022-09-02T05:56:03.396 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.397 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.397 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.398 DEBUG:boto:Params: {} 2022-09-02T05:56:03.400 DEBUG:boto:Token: None 2022-09-02T05:56:03.401 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:56:03.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc= 2022-09-02T05:56:03.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VLAWW6hM1N+LqQYkbdI1LXLRvoc=', 'Content-Length': '0'} 2022-09-02T05:56:03.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e09e019dc89cea1-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.412 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.413 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.414 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.415 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.416 DEBUG:boto:Method: GET 2022-09-02T05:56:03.416 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.417 DEBUG:boto:Data: 2022-09-02T05:56:03.417 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.419 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.419 DEBUG:boto:Params: {} 2022-09-02T05:56:03.420 DEBUG:boto:Token: None 2022-09-02T05:56:03.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.422 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460bd941cf9a9fe0-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.427 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.428 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.429 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.430 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.431 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.432 DEBUG:boto:Method: GET 2022-09-02T05:56:03.433 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.434 DEBUG:boto:Data: 2022-09-02T05:56:03.435 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.436 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.437 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.437 DEBUG:boto:Params: {} 2022-09-02T05:56:03.438 DEBUG:boto:Token: None 2022-09-02T05:56:03.438 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b56bbd8293221b7-0063119af3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.486 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.488 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.489 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.490 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.491 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:56:03.492 DEBUG:boto:Data: 2022-09-02T05:56:03.493 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.495 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.496 DEBUG:boto:Params: {} 2022-09-02T05:56:03.497 DEBUG:boto:Token: None 2022-09-02T05:56:03.499 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92 2022-09-02T05:56:03.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fhbUXt33ntGkXb1CB7bpRk3FpuM= 2022-09-02T05:56:03.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fhbUXt33ntGkXb1CB7bpRk3FpuM=', 'Content-Length': '0'} 2022-09-02T05:56:03.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae766cb207dd4198-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.506 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.507 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.508 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.508 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:56:03.509 DEBUG:boto:Data: 2022-09-02T05:56:03.510 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.511 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.512 DEBUG:boto:Params: {} 2022-09-02T05:56:03.513 DEBUG:boto:Token: None 2022-09-02T05:56:03.513 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92 2022-09-02T05:56:03.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fhbUXt33ntGkXb1CB7bpRk3FpuM= 2022-09-02T05:56:03.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fhbUXt33ntGkXb1CB7bpRk3FpuM=', 'Content-Length': '0'} 2022-09-02T05:56:03.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6590940257c2ae9-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:56:03.522 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.523 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.524 DEBUG:boto:Method: GET 2022-09-02T05:56:03.525 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:56:03.526 DEBUG:boto:Data: 2022-09-02T05:56:03.527 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.529 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.530 DEBUG:boto:Params: {} 2022-09-02T05:56:03.531 DEBUG:boto:Token: None 2022-09-02T05:56:03.532 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92 2022-09-02T05:56:03.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e0nuywp9a4/DQKT4hXZNFLPIJrw= 2022-09-02T05:56:03.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e0nuywp9a4/DQKT4hXZNFLPIJrw=', 'Content-Length': '0'} 2022-09-02T05:56:03.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b883069e6245cfc4-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.538 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:56:03.539 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:56:03.539 DEBUG:boto:Method: GET 2022-09-02T05:56:03.540 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:56:03.541 DEBUG:boto:Data: 2022-09-02T05:56:03.542 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.543 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.544 DEBUG:boto:Params: {} 2022-09-02T05:56:03.545 DEBUG:boto:Token: None 2022-09-02T05:56:03.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj92 2022-09-02T05:56:03.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e0nuywp9a4/DQKT4hXZNFLPIJrw= 2022-09-02T05:56:03.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e0nuywp9a4/DQKT4hXZNFLPIJrw=', 'Content-Length': '0'} 2022-09-02T05:56:03.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e622f06e6738405-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.551 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.552 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.553 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.554 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.555 DEBUG:boto:Method: GET 2022-09-02T05:56:03.556 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.557 DEBUG:boto:Data: 2022-09-02T05:56:03.557 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.560 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.562 DEBUG:boto:Params: {} 2022-09-02T05:56:03.563 DEBUG:boto:Token: None 2022-09-02T05:56:03.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e148edc0f0c1f45c-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.575 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.577 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.578 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.579 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.581 DEBUG:boto:Method: GET 2022-09-02T05:56:03.582 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:56:03.583 DEBUG:boto:Data: 2022-09-02T05:56:03.584 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.586 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.587 DEBUG:boto:Params: {} 2022-09-02T05:56:03.588 DEBUG:boto:Token: None 2022-09-02T05:56:03.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa80674dfdfd82a0-0063119af3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.634 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:56:03.636 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.637 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.638 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.638 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.639 DEBUG:boto:Method: GET 2022-09-02T05:56:03.640 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.640 DEBUG:boto:Data: 2022-09-02T05:56:03.641 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.642 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.643 DEBUG:boto:Params: {} 2022-09-02T05:56:03.643 DEBUG:boto:Token: None 2022-09-02T05:56:03.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs= 2022-09-02T05:56:03.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs=', 'Content-Length': '0'} 2022-09-02T05:56:03.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056460d80c92a4961-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.648 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.649 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.650 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.650 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.651 DEBUG:boto:Method: GET 2022-09-02T05:56:03.651 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.652 DEBUG:boto:Data: 2022-09-02T05:56:03.653 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.654 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.654 DEBUG:boto:Params: {} 2022-09-02T05:56:03.655 DEBUG:boto:Token: None 2022-09-02T05:56:03.656 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs= 2022-09-02T05:56:03.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs=', 'Content-Length': '0'} 2022-09-02T05:56:03.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e183aa4b10375422-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.661 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.662 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.663 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.663 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.664 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.664 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.665 DEBUG:boto:Data: 2022-09-02T05:56:03.665 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.666 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.667 DEBUG:boto:Params: {} 2022-09-02T05:56:03.668 DEBUG:boto:Token: None 2022-09-02T05:56:03.669 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OApZM+SawbxKdiQHSj+mSYFqx20= 2022-09-02T05:56:03.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OApZM+SawbxKdiQHSj+mSYFqx20=', 'Content-Length': '0'} 2022-09-02T05:56:03.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e76fa92009d602c7-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.676 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.677 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.677 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.678 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.679 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.680 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.680 DEBUG:boto:Data: 2022-09-02T05:56:03.681 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.683 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.683 DEBUG:boto:Params: {} 2022-09-02T05:56:03.684 DEBUG:boto:Token: None 2022-09-02T05:56:03.684 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OApZM+SawbxKdiQHSj+mSYFqx20= 2022-09-02T05:56:03.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OApZM+SawbxKdiQHSj+mSYFqx20=', 'Content-Length': '0'} 2022-09-02T05:56:03.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f9ee0937112387-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:56:03.695 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.696 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.696 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.697 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.697 DEBUG:boto:Method: GET 2022-09-02T05:56:03.698 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.698 DEBUG:boto:Data: 2022-09-02T05:56:03.699 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.701 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.702 DEBUG:boto:Params: {} 2022-09-02T05:56:03.703 DEBUG:boto:Token: None 2022-09-02T05:56:03.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs= 2022-09-02T05:56:03.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs=', 'Content-Length': '0'} 2022-09-02T05:56:03.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def8d7774a33e43e-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.726 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.727 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.728 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.729 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.730 DEBUG:boto:Method: GET 2022-09-02T05:56:03.731 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.732 DEBUG:boto:Data: 2022-09-02T05:56:03.733 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.735 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.736 DEBUG:boto:Params: {} 2022-09-02T05:56:03.737 DEBUG:boto:Token: None 2022-09-02T05:56:03.738 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:56:03.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs= 2022-09-02T05:56:03.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u0/t7Y5SuoMTnrrvR07PB/nFfgs=', 'Content-Length': '0'} 2022-09-02T05:56:03.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d6c75c3a1c0657-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.745 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.746 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.747 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.749 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.750 DEBUG:boto:Method: GET 2022-09-02T05:56:03.751 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.752 DEBUG:boto:Data: 2022-09-02T05:56:03.753 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.755 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.756 DEBUG:boto:Params: {} 2022-09-02T05:56:03.757 DEBUG:boto:Token: None 2022-09-02T05:56:03.759 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a729b0dca790ade4-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.765 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.766 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.767 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.767 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.768 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.768 DEBUG:boto:Method: GET 2022-09-02T05:56:03.769 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.769 DEBUG:boto:Data: 2022-09-02T05:56:03.770 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.771 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.772 DEBUG:boto:Params: {} 2022-09-02T05:56:03.772 DEBUG:boto:Token: None 2022-09-02T05:56:03.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a81b8b50b37683-0063119af3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.822 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.824 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.825 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.826 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.827 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:56:03.828 DEBUG:boto:Data: 2022-09-02T05:56:03.829 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.831 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.832 DEBUG:boto:Params: {} 2022-09-02T05:56:03.834 DEBUG:boto:Token: None 2022-09-02T05:56:03.835 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95 2022-09-02T05:56:03.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XokaowU5Hj2f1z5607ZiqDYPNxw= 2022-09-02T05:56:03.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XokaowU5Hj2f1z5607ZiqDYPNxw=', 'Content-Length': '0'} 2022-09-02T05:56:03.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008185ef4d7a2e254d-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.841 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.842 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.843 DEBUG:boto:Method: HEAD 2022-09-02T05:56:03.844 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:56:03.845 DEBUG:boto:Data: 2022-09-02T05:56:03.846 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.848 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.849 DEBUG:boto:Params: {} 2022-09-02T05:56:03.850 DEBUG:boto:Token: None 2022-09-02T05:56:03.851 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95 2022-09-02T05:56:03.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XokaowU5Hj2f1z5607ZiqDYPNxw= 2022-09-02T05:56:03.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XokaowU5Hj2f1z5607ZiqDYPNxw=', 'Content-Length': '0'} 2022-09-02T05:56:03.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6e7e8b80b34ee5-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:56:03.856 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.857 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.857 DEBUG:boto:Method: GET 2022-09-02T05:56:03.858 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:56:03.858 DEBUG:boto:Data: 2022-09-02T05:56:03.859 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.860 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.861 DEBUG:boto:Params: {} 2022-09-02T05:56:03.861 DEBUG:boto:Token: None 2022-09-02T05:56:03.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95 2022-09-02T05:56:03.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gepfUB8p52p7vZB/LLfuvc5RP0= 2022-09-02T05:56:03.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gepfUB8p52p7vZB/LLfuvc5RP0=', 'Content-Length': '0'} 2022-09-02T05:56:03.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b309836d9f1db17-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.867 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:56:03.868 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:56:03.868 DEBUG:boto:Method: GET 2022-09-02T05:56:03.869 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:56:03.869 DEBUG:boto:Data: 2022-09-02T05:56:03.870 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.871 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.872 DEBUG:boto:Params: {} 2022-09-02T05:56:03.872 DEBUG:boto:Token: None 2022-09-02T05:56:03.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/obj95 2022-09-02T05:56:03.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gepfUB8p52p7vZB/LLfuvc5RP0= 2022-09-02T05:56:03.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gepfUB8p52p7vZB/LLfuvc5RP0=', 'Content-Length': '0'} 2022-09-02T05:56:03.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000782779e2eeca281b-0063119af3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.879 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.879 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.880 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.881 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.882 DEBUG:boto:Method: GET 2022-09-02T05:56:03.883 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.884 DEBUG:boto:Data: 2022-09-02T05:56:03.884 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:56:03.886 DEBUG:boto:Port: 8001 2022-09-02T05:56:03.886 DEBUG:boto:Params: {} 2022-09-02T05:56:03.887 DEBUG:boto:Token: None 2022-09-02T05:56:03.888 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000adf51416b224e3c-0063119af3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.934 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.936 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:03.936 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:03.937 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.937 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.938 DEBUG:boto:Method: GET 2022-09-02T05:56:03.939 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:56:03.939 DEBUG:boto:Data: 2022-09-02T05:56:03.940 DEBUG:boto:Headers: {} 2022-09-02T05:56:03.940 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:56:03.941 DEBUG:boto:Port: 8000 2022-09-02T05:56:03.941 DEBUG:boto:Params: {} 2022-09-02T05:56:03.942 DEBUG:boto:Token: None 2022-09-02T05:56:03.943 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:56:03 GMT /xljbkf-74/ 2022-09-02T05:56:03.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U= 2022-09-02T05:56:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KA9sJKf0tIlKq/ibt+JLPJokk+U=', 'Content-Length': '0'} 2022-09-02T05:56:03.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f31504a4059c3de-0063119af3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:56:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:03.948 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:56:03.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:56:03.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:56:03.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:56:03.953 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:56:04.016 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:56:04.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:56:04.089 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:56:04.090 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-74 2022-09-02T05:56:04.101 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2022-09-02T05:56:04.209 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:04.210 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:04.210 DEBUG:boto:Method: HEAD 2022-09-02T05:56:04.211 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:56:04.212 DEBUG:boto:Data: 2022-09-02T05:56:04.212 DEBUG:boto:Headers: {} 2022-09-02T05:56:04.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.214 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.215 DEBUG:boto:Params: {} 2022-09-02T05:56:04.215 DEBUG:boto:Token: None 2022-09-02T05:56:04.216 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/ 2022-09-02T05:56:04.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk= 2022-09-02T05:56:04.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk=', 'Content-Length': '0'} 2022-09-02T05:56:04.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddb49617ff051ed6-0063119af4-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.222 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:56:04.223 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:56:04.223 DEBUG:boto:Method: PUT 2022-09-02T05:56:04.224 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:56:04.224 DEBUG:boto:Data: 2022-09-02T05:56:04.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:56:04.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.226 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.227 DEBUG:boto:Params: {} 2022-09-02T05:56:04.227 DEBUG:boto:Token: None 2022-09-02T05:56:04.228 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/obj31 2022-09-02T05:56:04.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JDnrl1kVbRnvu8SWi4QGENOhTVs= 2022-09-02T05:56:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JDnrl1kVbRnvu8SWi4QGENOhTVs='} 2022-09-02T05:56:04.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001dea0bace9a14757-0063119af4-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.239 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:04.240 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:04.241 DEBUG:boto:Method: HEAD 2022-09-02T05:56:04.242 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:56:04.242 DEBUG:boto:Data: 2022-09-02T05:56:04.243 DEBUG:boto:Headers: {} 2022-09-02T05:56:04.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.244 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.244 DEBUG:boto:Params: {} 2022-09-02T05:56:04.245 DEBUG:boto:Token: None 2022-09-02T05:56:04.246 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/ 2022-09-02T05:56:04.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk= 2022-09-02T05:56:04.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk=', 'Content-Length': '0'} 2022-09-02T05:56:04.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eef0fde8639b25ef-0063119af4-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.253 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:56:04.253 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:56:04.254 DEBUG:boto:Method: PUT 2022-09-02T05:56:04.255 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:56:04.256 DEBUG:boto:Data: 2022-09-02T05:56:04.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:56:04.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.258 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.259 DEBUG:boto:Params: {} 2022-09-02T05:56:04.259 DEBUG:boto:Token: None 2022-09-02T05:56:04.260 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/obj62 2022-09-02T05:56:04.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zUqamdIaOwnk0wNVrmhxAlIjt0A= 2022-09-02T05:56:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zUqamdIaOwnk0wNVrmhxAlIjt0A='} 2022-09-02T05:56:04.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001df4ce8e6e770111-0063119af4-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.313 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:04.314 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:04.315 DEBUG:boto:Method: HEAD 2022-09-02T05:56:04.315 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:56:04.316 DEBUG:boto:Data: 2022-09-02T05:56:04.316 DEBUG:boto:Headers: {} 2022-09-02T05:56:04.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.317 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.318 DEBUG:boto:Params: {} 2022-09-02T05:56:04.319 DEBUG:boto:Token: None 2022-09-02T05:56:04.319 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/ 2022-09-02T05:56:04.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk= 2022-09-02T05:56:04.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk=', 'Content-Length': '0'} 2022-09-02T05:56:04.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090a0a6df03d6b95a-0063119af4-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.327 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:56:04.328 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:56:04.329 DEBUG:boto:Method: PUT 2022-09-02T05:56:04.330 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:56:04.331 DEBUG:boto:Data: 2022-09-02T05:56:04.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:56:04.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.334 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.335 DEBUG:boto:Params: {} 2022-09-02T05:56:04.337 DEBUG:boto:Token: None 2022-09-02T05:56:04.338 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/obj93 2022-09-02T05:56:04.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8TKMETLOPyukyhzZRw8cPiXdw20= 2022-09-02T05:56:04.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8TKMETLOPyukyhzZRw8cPiXdw20='} 2022-09-02T05:56:04.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000daf56d79e1fa8459-0063119af4-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.348 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:04.349 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:04.350 DEBUG:boto:Method: HEAD 2022-09-02T05:56:04.350 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:56:04.351 DEBUG:boto:Data: 2022-09-02T05:56:04.352 DEBUG:boto:Headers: {} 2022-09-02T05:56:04.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.354 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.354 DEBUG:boto:Params: {} 2022-09-02T05:56:04.355 DEBUG:boto:Token: None 2022-09-02T05:56:04.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/ 2022-09-02T05:56:04.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk= 2022-09-02T05:56:04.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk=', 'Content-Length': '0'} 2022-09-02T05:56:04.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c3c4d9c8527f089-0063119af4-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.365 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:56:04.365 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:56:04.366 DEBUG:boto:Method: PUT 2022-09-02T05:56:04.366 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:56:04.367 DEBUG:boto:Data: 2022-09-02T05:56:04.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:56:04.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.369 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.370 DEBUG:boto:Params: {} 2022-09-02T05:56:04.371 DEBUG:boto:Token: None 2022-09-02T05:56:04.371 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/obj124 2022-09-02T05:56:04.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jLaoM/VgQGorWFlxPX1zmPvdxuo= 2022-09-02T05:56:04.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jLaoM/VgQGorWFlxPX1zmPvdxuo='} 2022-09-02T05:56:04.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0349d06bb936f7d-0063119af4-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.421 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:56:04.423 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:56:04.425 DEBUG:boto:Method: HEAD 2022-09-02T05:56:04.426 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:56:04.427 DEBUG:boto:Data: 2022-09-02T05:56:04.429 DEBUG:boto:Headers: {} 2022-09-02T05:56:04.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.432 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.432 DEBUG:boto:Params: {} 2022-09-02T05:56:04.433 DEBUG:boto:Token: None 2022-09-02T05:56:04.434 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/ 2022-09-02T05:56:04.434 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk= 2022-09-02T05:56:04.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2+l1OL7OY3PhSI7SKqYJ8OaNbbk=', 'Content-Length': '0'} 2022-09-02T05:56:04.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f18207eb59a7132-0063119af4-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.440 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:56:04.440 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:56:04.441 DEBUG:boto:Method: PUT 2022-09-02T05:56:04.442 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:56:04.442 DEBUG:boto:Data: 2022-09-02T05:56:04.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:56:04.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:56:04.444 DEBUG:boto:Port: 8000 2022-09-02T05:56:04.444 DEBUG:boto:Params: {} 2022-09-02T05:56:04.445 DEBUG:boto:Token: None 2022-09-02T05:56:04.446 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:56:04 GMT /xljbkf-74/obj155 2022-09-02T05:56:04.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o3LoWZE0rTWNmW6wyCSub7KasyA= 2022-09-02T05:56:04.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:56:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o3LoWZE0rTWNmW6wyCSub7KasyA='} 2022-09-02T05:56:04.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053d83df7e4392e49-0063119af4-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:56:04.456 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:56:04.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:56:04.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:56:04.458 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:56:04.522 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:56:04.522 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:56:04.584 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:04.583+0000 7f753f6b2600 1 waiting to reach latest gen 5, on 4.. 2022-09-02T05:56:09.587 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:09.586+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:09.588 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:56:09.588 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:14.589 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:14.588+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:14.590 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:56:14.590 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:19.592 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:19.591+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:19.593 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:56:19.593 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:24.595 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:24.593+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:24.596 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:56:24.597 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:29.597 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:29.596+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:29.598 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.81.5, , , , 00000000001.60.5, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:29.599 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:34.600 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:34.598+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:34.600 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.81.5, , , , 00000000001.60.5, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:34.600 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:39.603 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:39.601+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:39.603 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.81.5, , , , 00000000001.60.5, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:39.604 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:44.605 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:44.603+0000 7f753f6b2600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:44.605 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.81.5, , , , 00000000001.60.5, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:44.605 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:49.607 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:49.606+0000 7f753f6b2600 1 bucket sync caught up with source: 2022-09-02T05:56:49.608 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:49.608 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:49.608 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:56:49.606+0000 7f753f6b2600 0 bucket checkpoint complete 2022-09-02T05:56:49.631 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:56:49.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:56:49.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:56:49.634 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:56:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:56:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:56:49.760 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:56:49.758+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:49.761 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:49.761 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:54.762 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:56:54.761+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:54.763 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:54.763 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:59.766 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:56:59.764+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:56:59.767 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:56:59.767 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:04.768 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:04.766+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:04.768 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:04.769 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:09.770 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:09.769+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:09.771 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:09.771 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:14.773 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:14.772+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:14.774 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:14.774 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:19.775 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:19.773+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:19.775 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:19.776 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:24.778 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:24.776+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:24.778 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:24.779 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:29.780 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:29.779+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:29.781 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:29.781 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:34.783 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:34.782+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:34.784 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:34.784 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:39.786 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:39.784+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:39.786 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:39.786 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:44.790 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:44.787+0000 7fd52e55c600 1 waiting for incremental sync to catch up: 2022-09-02T05:57:44.791 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , , , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:44.791 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:49.792 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:49.789+0000 7fd52e55c600 1 bucket sync caught up with source: 2022-09-02T05:57:49.793 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:49.793 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:49.794 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:49.790+0000 7fd52e55c600 0 bucket checkpoint complete 2022-09-02T05:57:49.810 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:57:49.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:49.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:49.813 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:57:49.878 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:57:49.878 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:57:49.927 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:57:49.925+0000 7f9dd5b3e600 1 bucket sync caught up with source: 2022-09-02T05:57:49.927 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:49.927 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , 00000000002.81.5, , 00000000001.76.5, , 00000000001.60.5, , , , , , , , , , 00000000001.83.5, , , , , , , , , , , , , , 00000000001.62.5] 2022-09-02T05:57:49.928 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:57:49.925+0000 7f9dd5b3e600 0 bucket checkpoint complete 2022-09-02T05:57:49.942 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:57:49.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:49.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:49.944 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:57:50.005 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:57:50.005 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:57:50.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:50.046+0000 7fca9ee5e600 1 bucket sync caught up with source: 2022-09-02T05:57:50.048 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000004.599.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000003.552.5, 00000000006.1210.5, 00000000004.839.5] 2022-09-02T05:57:50.048 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000004.599.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000003.552.5, 00000000006.1210.5, 00000000004.839.5] 2022-09-02T05:57:50.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:50.046+0000 7fca9ee5e600 0 bucket checkpoint complete 2022-09-02T05:57:50.072 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:57:50.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:50.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:50.075 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:57:50.133 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:57:50.134 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:57:50.181 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:50.179+0000 7f9fd85b1600 1 bucket sync caught up with source: 2022-09-02T05:57:50.182 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000004.599.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000003.552.5, 00000000006.1210.5, 00000000004.839.5] 2022-09-02T05:57:50.182 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000004.599.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000003.552.5, 00000000006.1210.5, 00000000004.839.5] 2022-09-02T05:57:50.183 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:50.179+0000 7f9fd85b1600 0 bucket checkpoint complete 2022-09-02T05:57:50.195 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:57:50.197 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:57:50.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:50.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:50.199 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:57:50.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:57:50.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:57:50.301 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:50.299+0000 7f2f80fe4600 1 bucket sync caught up with source: 2022-09-02T05:57:50.301 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000004.359.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000003.208.5, 00000000006.340.5, 00000000004.528.5] 2022-09-02T05:57:50.301 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000004.359.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000003.208.5, 00000000006.340.5, 00000000004.528.5] 2022-09-02T05:57:50.302 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:57:50.299+0000 7f2f80fe4600 0 bucket checkpoint complete 2022-09-02T05:57:50.318 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:57:50.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:50.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:57:50.320 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:57:50.378 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:57:50.378 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:57:50.426 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:57:50.425+0000 7f8f319ed600 1 bucket sync caught up with source: 2022-09-02T05:57:50.427 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000004.359.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000003.208.5, 00000000006.340.5, 00000000004.528.5] 2022-09-02T05:57:50.427 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000004.359.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000003.208.5, 00000000006.340.5, 00000000004.528.5] 2022-09-02T05:57:50.427 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:57:50.425+0000 7f8f319ed600 0 bucket checkpoint complete 2022-09-02T05:57:50.444 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:57:50.445 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:57:50.446 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.447 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.449 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.450 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:57:50.451 DEBUG:boto:Data: 2022-09-02T05:57:50.451 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.453 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.454 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.455 DEBUG:boto:Params: {} 2022-09-02T05:57:50.456 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:57:50.458 DEBUG:boto:Token: None 2022-09-02T05:57:50.459 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j1b7AhoQvbM2y6BljBCwGYpjJdQ= 2022-09-02T05:57:50.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j1b7AhoQvbM2y6BljBCwGYpjJdQ=', 'Content-Length': '0'} 2022-09-02T05:57:50.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2f4e7cd2d6c40b8-0063119b5e-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.469 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.470 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.471 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.472 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:57:50.473 DEBUG:boto:Data: 2022-09-02T05:57:50.474 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.476 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.477 DEBUG:boto:Params: {} 2022-09-02T05:57:50.479 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:57:50.481 DEBUG:boto:Token: None 2022-09-02T05:57:50.482 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j1b7AhoQvbM2y6BljBCwGYpjJdQ= 2022-09-02T05:57:50.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j1b7AhoQvbM2y6BljBCwGYpjJdQ=', 'Content-Length': '0'} 2022-09-02T05:57:50.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a9da3b9bc598ca8-0063119b5e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.489 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:57:50.490 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.491 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.492 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.493 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.494 DEBUG:boto:Method: GET 2022-09-02T05:57:50.494 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.495 DEBUG:boto:Data: 2022-09-02T05:57:50.496 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.497 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.497 DEBUG:boto:Params: {} 2022-09-02T05:57:50.498 DEBUG:boto:Token: None 2022-09-02T05:57:50.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/?versions 2022-09-02T05:57:50.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI= 2022-09-02T05:57:50.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI=', 'Content-Length': '0'} 2022-09-02T05:57:50.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f9b2bf24a589023-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.506 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:57:50.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:57:50.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:57:50.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:57:50.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:57:50.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:57:50.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:57:50.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:57:50.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:57:50.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:57:50.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:57:50.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:57:50.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:57:50.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:57:50.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:57:50.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:57:50.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:57:50.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:57:50.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:57:50.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:57:50.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:57:50.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:57:50.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:57:50.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:57:50.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:57:50.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:57:50.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:57:50.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:57:50.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:57:50.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:57:50.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:57:50.524 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.525 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.525 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.526 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.526 DEBUG:boto:Method: GET 2022-09-02T05:57:50.527 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.527 DEBUG:boto:Data: 2022-09-02T05:57:50.528 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.529 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.529 DEBUG:boto:Params: {} 2022-09-02T05:57:50.530 DEBUG:boto:Token: None 2022-09-02T05:57:50.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/?versions 2022-09-02T05:57:50.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI= 2022-09-02T05:57:50.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI=', 'Content-Length': '0'} 2022-09-02T05:57:50.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5bc6b0a3340bc43-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.536 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:57:50.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:57:50.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:57:50.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:57:50.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:57:50.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:57:50.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:57:50.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:57:50.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:57:50.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:57:50.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:57:50.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:57:50.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:57:50.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:57:50.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:57:50.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:57:50.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:57:50.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:57:50.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:57:50.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:57:50.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:57:50.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:57:50.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:57:50.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:57:50.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:57:50.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:57:50.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:57:50.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:57:50.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:57:50.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:57:50.553 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.554 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.554 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.555 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.555 DEBUG:boto:Method: GET 2022-09-02T05:57:50.556 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.556 DEBUG:boto:Data: 2022-09-02T05:57:50.557 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.557 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.558 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.558 DEBUG:boto:Params: {} 2022-09-02T05:57:50.559 DEBUG:boto:Token: None 2022-09-02T05:57:50.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/?versions 2022-09-02T05:57:50.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI= 2022-09-02T05:57:50.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI=', 'Content-Length': '0'} 2022-09-02T05:57:50.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3ef25dcc3a92ee-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.565 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.567 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.568 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.568 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.569 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.569 DEBUG:boto:Method: GET 2022-09-02T05:57:50.570 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:50.570 DEBUG:boto:Data: 2022-09-02T05:57:50.571 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.572 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.573 DEBUG:boto:Params: {} 2022-09-02T05:57:50.573 DEBUG:boto:Token: None 2022-09-02T05:57:50.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/?versions 2022-09-02T05:57:50.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI= 2022-09-02T05:57:50.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lxOIT6r4MyhvLGjgld2mMK2DmQI=', 'Content-Length': '0'} 2022-09-02T05:57:50.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cd5ecdd4e0d3659-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.580 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:57:50.583 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.584 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.584 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.585 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.585 DEBUG:boto:Method: GET 2022-09-02T05:57:50.586 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.586 DEBUG:boto:Data: 2022-09-02T05:57:50.587 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.588 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.589 DEBUG:boto:Params: {} 2022-09-02T05:57:50.589 DEBUG:boto:Token: None 2022-09-02T05:57:50.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g= 2022-09-02T05:57:50.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g=', 'Content-Length': '0'} 2022-09-02T05:57:50.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42fbfab04e4ae71-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.594 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.595 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.596 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.596 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.597 DEBUG:boto:Method: GET 2022-09-02T05:57:50.597 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.598 DEBUG:boto:Data: 2022-09-02T05:57:50.599 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.600 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.601 DEBUG:boto:Params: {} 2022-09-02T05:57:50.601 DEBUG:boto:Token: None 2022-09-02T05:57:50.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g= 2022-09-02T05:57:50.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g=', 'Content-Length': '0'} 2022-09-02T05:57:50.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d823fe7ae5b9793-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.606 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.607 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.607 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.608 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.608 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.609 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.609 DEBUG:boto:Data: 2022-09-02T05:57:50.610 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.611 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.612 DEBUG:boto:Params: {} 2022-09-02T05:57:50.613 DEBUG:boto:Token: None 2022-09-02T05:57:50.613 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zTrvqT0rIC7arYx/shzwt5iI0iw= 2022-09-02T05:57:50.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zTrvqT0rIC7arYx/shzwt5iI0iw=', 'Content-Length': '0'} 2022-09-02T05:57:50.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de78c6a01cb31bd-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.617 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.618 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.618 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.619 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.619 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.620 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.620 DEBUG:boto:Data: 2022-09-02T05:57:50.621 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.622 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.622 DEBUG:boto:Params: {} 2022-09-02T05:57:50.623 DEBUG:boto:Token: None 2022-09-02T05:57:50.623 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.624 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zTrvqT0rIC7arYx/shzwt5iI0iw= 2022-09-02T05:57:50.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zTrvqT0rIC7arYx/shzwt5iI0iw=', 'Content-Length': '0'} 2022-09-02T05:57:50.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f90ff36f288bff-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:57:50.628 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.629 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.629 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.630 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.630 DEBUG:boto:Method: GET 2022-09-02T05:57:50.631 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.632 DEBUG:boto:Data: 2022-09-02T05:57:50.632 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.633 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.633 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.634 DEBUG:boto:Params: {} 2022-09-02T05:57:50.635 DEBUG:boto:Token: None 2022-09-02T05:57:50.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g= 2022-09-02T05:57:50.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g=', 'Content-Length': '0'} 2022-09-02T05:57:50.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8bbfd921953729-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.639 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.640 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.640 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.641 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.641 DEBUG:boto:Method: GET 2022-09-02T05:57:50.642 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.643 DEBUG:boto:Data: 2022-09-02T05:57:50.643 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.644 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.645 DEBUG:boto:Params: {} 2022-09-02T05:57:50.645 DEBUG:boto:Token: None 2022-09-02T05:57:50.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:50.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g= 2022-09-02T05:57:50.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVkej94e2V3kcDLMEd2SnkP5n+g=', 'Content-Length': '0'} 2022-09-02T05:57:50.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db5dcb06681e5b5-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.650 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.651 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.651 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.652 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.652 DEBUG:boto:Method: GET 2022-09-02T05:57:50.653 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.653 DEBUG:boto:Data: 2022-09-02T05:57:50.654 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.655 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.656 DEBUG:boto:Params: {} 2022-09-02T05:57:50.656 DEBUG:boto:Token: None 2022-09-02T05:57:50.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fc05e119df982e-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.661 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.663 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.663 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.664 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.664 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.665 DEBUG:boto:Method: GET 2022-09-02T05:57:50.666 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.666 DEBUG:boto:Data: 2022-09-02T05:57:50.667 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.668 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.668 DEBUG:boto:Params: {} 2022-09-02T05:57:50.669 DEBUG:boto:Token: None 2022-09-02T05:57:50.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009027235cb26aeda4-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.676 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.677 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.677 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.678 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.678 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:50.679 DEBUG:boto:Data: 2022-09-02T05:57:50.679 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.680 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.681 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.681 DEBUG:boto:Params: {} 2022-09-02T05:57:50.682 DEBUG:boto:Token: None 2022-09-02T05:57:50.682 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11 2022-09-02T05:57:50.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:81YTkB0uP3yGO6DFmedYwEWKJp0= 2022-09-02T05:57:50.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:81YTkB0uP3yGO6DFmedYwEWKJp0=', 'Content-Length': '0'} 2022-09-02T05:57:50.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000738b930052937e68-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.686 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.687 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.687 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.688 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:50.688 DEBUG:boto:Data: 2022-09-02T05:57:50.689 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.690 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.690 DEBUG:boto:Params: {} 2022-09-02T05:57:50.691 DEBUG:boto:Token: None 2022-09-02T05:57:50.692 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11 2022-09-02T05:57:50.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:81YTkB0uP3yGO6DFmedYwEWKJp0= 2022-09-02T05:57:50.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:81YTkB0uP3yGO6DFmedYwEWKJp0=', 'Content-Length': '0'} 2022-09-02T05:57:50.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b01d149bcd3c96-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:57:50.697 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.697 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.698 DEBUG:boto:Method: GET 2022-09-02T05:57:50.698 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:50.699 DEBUG:boto:Data: 2022-09-02T05:57:50.699 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.701 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.701 DEBUG:boto:Params: {} 2022-09-02T05:57:50.702 DEBUG:boto:Token: None 2022-09-02T05:57:50.703 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11 2022-09-02T05:57:50.703 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rv3z0qtyl9UwuTZ1F4nETsw6K1Q= 2022-09-02T05:57:50.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rv3z0qtyl9UwuTZ1F4nETsw6K1Q=', 'Content-Length': '0'} 2022-09-02T05:57:50.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e827edbd8a36718b-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.707 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:50.707 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:50.708 DEBUG:boto:Method: GET 2022-09-02T05:57:50.708 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:50.709 DEBUG:boto:Data: 2022-09-02T05:57:50.709 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.710 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.711 DEBUG:boto:Params: {} 2022-09-02T05:57:50.712 DEBUG:boto:Token: None 2022-09-02T05:57:50.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj11 2022-09-02T05:57:50.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rv3z0qtyl9UwuTZ1F4nETsw6K1Q= 2022-09-02T05:57:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rv3z0qtyl9UwuTZ1F4nETsw6K1Q=', 'Content-Length': '0'} 2022-09-02T05:57:50.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829ffe54a1671374-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.762 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.763 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.763 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.764 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.765 DEBUG:boto:Method: GET 2022-09-02T05:57:50.765 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.766 DEBUG:boto:Data: 2022-09-02T05:57:50.766 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.767 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.767 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.768 DEBUG:boto:Params: {} 2022-09-02T05:57:50.769 DEBUG:boto:Token: None 2022-09-02T05:57:50.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047b83d6182abf2d8-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.775 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.776 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.776 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.777 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.778 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.779 DEBUG:boto:Method: GET 2022-09-02T05:57:50.779 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:50.780 DEBUG:boto:Data: 2022-09-02T05:57:50.781 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.783 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.784 DEBUG:boto:Params: {} 2022-09-02T05:57:50.784 DEBUG:boto:Token: None 2022-09-02T05:57:50.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa1fa91960d8e5c-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.791 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:57:50.792 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.793 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.793 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.794 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.795 DEBUG:boto:Method: GET 2022-09-02T05:57:50.795 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.796 DEBUG:boto:Data: 2022-09-02T05:57:50.797 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.799 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.800 DEBUG:boto:Params: {} 2022-09-02T05:57:50.801 DEBUG:boto:Token: None 2022-09-02T05:57:50.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s= 2022-09-02T05:57:50.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s=', 'Content-Length': '0'} 2022-09-02T05:57:50.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2705d38743461c-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.846 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.847 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.847 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.848 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.848 DEBUG:boto:Method: GET 2022-09-02T05:57:50.849 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.849 DEBUG:boto:Data: 2022-09-02T05:57:50.850 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.851 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.852 DEBUG:boto:Params: {} 2022-09-02T05:57:50.852 DEBUG:boto:Token: None 2022-09-02T05:57:50.853 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s= 2022-09-02T05:57:50.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s=', 'Content-Length': '0'} 2022-09-02T05:57:50.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfbc9e32f198dae-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.857 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.858 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.858 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.859 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.859 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.860 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.860 DEBUG:boto:Data: 2022-09-02T05:57:50.861 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.862 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.863 DEBUG:boto:Params: {} 2022-09-02T05:57:50.863 DEBUG:boto:Token: None 2022-09-02T05:57:50.864 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aXUIuSHVyJrPCrXBa3S+j5BQ0i4= 2022-09-02T05:57:50.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aXUIuSHVyJrPCrXBa3S+j5BQ0i4=', 'Content-Length': '0'} 2022-09-02T05:57:50.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03f4e3997576f37-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.867 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.868 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.869 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.869 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.870 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.870 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.871 DEBUG:boto:Data: 2022-09-02T05:57:50.871 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.872 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.873 DEBUG:boto:Params: {} 2022-09-02T05:57:50.873 DEBUG:boto:Token: None 2022-09-02T05:57:50.874 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aXUIuSHVyJrPCrXBa3S+j5BQ0i4= 2022-09-02T05:57:50.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aXUIuSHVyJrPCrXBa3S+j5BQ0i4=', 'Content-Length': '0'} 2022-09-02T05:57:50.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c11da8277fa2287e-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:57:50.878 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.879 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.879 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.880 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.880 DEBUG:boto:Method: GET 2022-09-02T05:57:50.881 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.882 DEBUG:boto:Data: 2022-09-02T05:57:50.882 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.883 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.883 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.884 DEBUG:boto:Params: {} 2022-09-02T05:57:50.885 DEBUG:boto:Token: None 2022-09-02T05:57:50.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s= 2022-09-02T05:57:50.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s=', 'Content-Length': '0'} 2022-09-02T05:57:50.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158f5ebf18ed77f9-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.890 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.890 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.891 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.891 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.892 DEBUG:boto:Method: GET 2022-09-02T05:57:50.892 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.893 DEBUG:boto:Data: 2022-09-02T05:57:50.893 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.895 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.895 DEBUG:boto:Params: {} 2022-09-02T05:57:50.896 DEBUG:boto:Token: None 2022-09-02T05:57:50.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:50.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s= 2022-09-02T05:57:50.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kCeKEoz2ScZPVmHTMYOCSfE2e0s=', 'Content-Length': '0'} 2022-09-02T05:57:50.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064a82db328017e3b-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.901 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.902 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.903 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.903 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.904 DEBUG:boto:Method: GET 2022-09-02T05:57:50.904 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.905 DEBUG:boto:Data: 2022-09-02T05:57:50.905 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.906 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.907 DEBUG:boto:Params: {} 2022-09-02T05:57:50.908 DEBUG:boto:Token: None 2022-09-02T05:57:50.909 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698d65f55e0dcdfa-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.915 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.916 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.917 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.918 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.918 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.919 DEBUG:boto:Method: GET 2022-09-02T05:57:50.920 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.920 DEBUG:boto:Data: 2022-09-02T05:57:50.921 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.922 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.922 DEBUG:boto:Params: {} 2022-09-02T05:57:50.923 DEBUG:boto:Token: None 2022-09-02T05:57:50.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8cb9b8d1b77ef17-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.928 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.929 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.930 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.930 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.931 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:50.931 DEBUG:boto:Data: 2022-09-02T05:57:50.932 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.932 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.933 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.933 DEBUG:boto:Params: {} 2022-09-02T05:57:50.934 DEBUG:boto:Token: None 2022-09-02T05:57:50.934 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115 2022-09-02T05:57:50.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1GkToMx3/JjZesyyNYhkb/CCcL8= 2022-09-02T05:57:50.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1GkToMx3/JjZesyyNYhkb/CCcL8=', 'Content-Length': '0'} 2022-09-02T05:57:50.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6363c446c8847fc-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.938 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.939 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.939 DEBUG:boto:Method: HEAD 2022-09-02T05:57:50.940 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:50.941 DEBUG:boto:Data: 2022-09-02T05:57:50.941 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.942 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.943 DEBUG:boto:Params: {} 2022-09-02T05:57:50.943 DEBUG:boto:Token: None 2022-09-02T05:57:50.944 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115 2022-09-02T05:57:50.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1GkToMx3/JjZesyyNYhkb/CCcL8= 2022-09-02T05:57:50.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1GkToMx3/JjZesyyNYhkb/CCcL8=', 'Content-Length': '0'} 2022-09-02T05:57:50.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddeab63541198015-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:57:50.950 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.951 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.952 DEBUG:boto:Method: GET 2022-09-02T05:57:50.953 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:50.953 DEBUG:boto:Data: 2022-09-02T05:57:50.954 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.955 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.956 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.957 DEBUG:boto:Params: {} 2022-09-02T05:57:50.957 DEBUG:boto:Token: None 2022-09-02T05:57:50.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115 2022-09-02T05:57:50.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x7eAPnqVqV8Rk8DSxnijomJeP/g= 2022-09-02T05:57:50.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x7eAPnqVqV8Rk8DSxnijomJeP/g=', 'Content-Length': '0'} 2022-09-02T05:57:50.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba530ebe5ca3c68-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.962 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:50.963 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:50.963 DEBUG:boto:Method: GET 2022-09-02T05:57:50.964 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:50.964 DEBUG:boto:Data: 2022-09-02T05:57:50.965 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.966 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.966 DEBUG:boto:Params: {} 2022-09-02T05:57:50.967 DEBUG:boto:Token: None 2022-09-02T05:57:50.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/obj115 2022-09-02T05:57:50.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x7eAPnqVqV8Rk8DSxnijomJeP/g= 2022-09-02T05:57:50.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x7eAPnqVqV8Rk8DSxnijomJeP/g=', 'Content-Length': '0'} 2022-09-02T05:57:50.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002779a0e5a67e9ea0-0063119b5e-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.971 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.972 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.972 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.973 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.974 DEBUG:boto:Method: GET 2022-09-02T05:57:50.974 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.975 DEBUG:boto:Data: 2022-09-02T05:57:50.975 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:50.977 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.978 DEBUG:boto:Params: {} 2022-09-02T05:57:50.979 DEBUG:boto:Token: None 2022-09-02T05:57:50.980 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:50.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:50.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:50.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055f839bc5f4d84c4-0063119b5e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:50.986 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:50.988 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:50.989 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:50.990 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.991 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.992 DEBUG:boto:Method: GET 2022-09-02T05:57:50.992 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:50.993 DEBUG:boto:Data: 2022-09-02T05:57:50.994 DEBUG:boto:Headers: {} 2022-09-02T05:57:50.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:50.996 DEBUG:boto:Port: 8000 2022-09-02T05:57:50.997 DEBUG:boto:Params: {} 2022-09-02T05:57:50.998 DEBUG:boto:Token: None 2022-09-02T05:57:50.999 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:50 GMT /xljbkf-74/ 2022-09-02T05:57:51.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4= 2022-09-02T05:57:51.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqPbOtzZecgQXiAYLWtv8TVPuQ4=', 'Content-Length': '0'} 2022-09-02T05:57:51.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c12ef914e121999-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.046 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:57:51.048 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.048 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.049 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.049 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.050 DEBUG:boto:Method: GET 2022-09-02T05:57:51.050 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.051 DEBUG:boto:Data: 2022-09-02T05:57:51.051 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.052 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.052 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.053 DEBUG:boto:Params: {} 2022-09-02T05:57:51.054 DEBUG:boto:Token: None 2022-09-02T05:57:51.054 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M= 2022-09-02T05:57:51.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M=', 'Content-Length': '0'} 2022-09-02T05:57:51.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001a5693a1ea95c1a-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.059 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.059 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.060 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.060 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.061 DEBUG:boto:Method: GET 2022-09-02T05:57:51.061 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.062 DEBUG:boto:Data: 2022-09-02T05:57:51.062 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.063 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.064 DEBUG:boto:Params: {} 2022-09-02T05:57:51.065 DEBUG:boto:Token: None 2022-09-02T05:57:51.065 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M= 2022-09-02T05:57:51.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M=', 'Content-Length': '0'} 2022-09-02T05:57:51.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e275e14b48603b1a-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.069 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.070 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.070 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.071 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.071 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.072 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.072 DEBUG:boto:Data: 2022-09-02T05:57:51.073 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.073 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.074 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.074 DEBUG:boto:Params: {} 2022-09-02T05:57:51.075 DEBUG:boto:Token: None 2022-09-02T05:57:51.076 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5fz27h3xM6/rXH9tM+qCaY1n2g0= 2022-09-02T05:57:51.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5fz27h3xM6/rXH9tM+qCaY1n2g0=', 'Content-Length': '0'} 2022-09-02T05:57:51.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971d46ffa9cdec8c-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.079 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.080 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.080 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.081 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.081 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.082 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.083 DEBUG:boto:Data: 2022-09-02T05:57:51.083 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.084 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.085 DEBUG:boto:Params: {} 2022-09-02T05:57:51.085 DEBUG:boto:Token: None 2022-09-02T05:57:51.086 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5fz27h3xM6/rXH9tM+qCaY1n2g0= 2022-09-02T05:57:51.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5fz27h3xM6/rXH9tM+qCaY1n2g0=', 'Content-Length': '0'} 2022-09-02T05:57:51.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c07d371fb78d021-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:57:51.090 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.091 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.091 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.092 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.093 DEBUG:boto:Method: GET 2022-09-02T05:57:51.093 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.094 DEBUG:boto:Data: 2022-09-02T05:57:51.094 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.095 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.096 DEBUG:boto:Params: {} 2022-09-02T05:57:51.096 DEBUG:boto:Token: None 2022-09-02T05:57:51.097 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M= 2022-09-02T05:57:51.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M=', 'Content-Length': '0'} 2022-09-02T05:57:51.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30f82b2125f91e6-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.142 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.143 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.143 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.144 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.145 DEBUG:boto:Method: GET 2022-09-02T05:57:51.145 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.146 DEBUG:boto:Data: 2022-09-02T05:57:51.146 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.147 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.148 DEBUG:boto:Params: {} 2022-09-02T05:57:51.148 DEBUG:boto:Token: None 2022-09-02T05:57:51.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:51.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M= 2022-09-02T05:57:51.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TpQ4hkILRCdkEVmtFHJOnlQJR/M=', 'Content-Length': '0'} 2022-09-02T05:57:51.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39023f42a140c7f-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.154 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.154 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.155 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.156 DEBUG:boto:Method: GET 2022-09-02T05:57:51.156 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.157 DEBUG:boto:Data: 2022-09-02T05:57:51.157 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.158 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.158 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.159 DEBUG:boto:Params: {} 2022-09-02T05:57:51.160 DEBUG:boto:Token: None 2022-09-02T05:57:51.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a93583744bbf1648-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.165 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.166 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.166 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.167 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.168 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.168 DEBUG:boto:Method: GET 2022-09-02T05:57:51.169 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.169 DEBUG:boto:Data: 2022-09-02T05:57:51.170 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.171 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.171 DEBUG:boto:Params: {} 2022-09-02T05:57:51.172 DEBUG:boto:Token: None 2022-09-02T05:57:51.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ddf69171577cba-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.178 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.179 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.179 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.180 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.180 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:51.181 DEBUG:boto:Data: 2022-09-02T05:57:51.181 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.182 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.182 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.183 DEBUG:boto:Params: {} 2022-09-02T05:57:51.184 DEBUG:boto:Token: None 2022-09-02T05:57:51.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116 2022-09-02T05:57:51.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5alyLQophd/aABKRRFQxz/GFODo= 2022-09-02T05:57:51.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5alyLQophd/aABKRRFQxz/GFODo=', 'Content-Length': '0'} 2022-09-02T05:57:51.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eacdc01b28cb1bc9-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.188 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.189 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.189 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.190 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:51.190 DEBUG:boto:Data: 2022-09-02T05:57:51.191 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.192 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.192 DEBUG:boto:Params: {} 2022-09-02T05:57:51.193 DEBUG:boto:Token: None 2022-09-02T05:57:51.194 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116 2022-09-02T05:57:51.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5alyLQophd/aABKRRFQxz/GFODo= 2022-09-02T05:57:51.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5alyLQophd/aABKRRFQxz/GFODo=', 'Content-Length': '0'} 2022-09-02T05:57:51.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ad524049328b25-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:57:51.198 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.199 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.199 DEBUG:boto:Method: GET 2022-09-02T05:57:51.200 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:51.200 DEBUG:boto:Data: 2022-09-02T05:57:51.201 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.201 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.202 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.202 DEBUG:boto:Params: {} 2022-09-02T05:57:51.203 DEBUG:boto:Token: None 2022-09-02T05:57:51.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116 2022-09-02T05:57:51.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ux9RbGv0bN8OcVxZBhM8ckg/qz8= 2022-09-02T05:57:51.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ux9RbGv0bN8OcVxZBhM8ckg/qz8=', 'Content-Length': '0'} 2022-09-02T05:57:51.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0fa176ea24b1429-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.208 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:51.208 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:51.209 DEBUG:boto:Method: GET 2022-09-02T05:57:51.209 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:51.210 DEBUG:boto:Data: 2022-09-02T05:57:51.210 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.212 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.212 DEBUG:boto:Params: {} 2022-09-02T05:57:51.213 DEBUG:boto:Token: None 2022-09-02T05:57:51.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj116 2022-09-02T05:57:51.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ux9RbGv0bN8OcVxZBhM8ckg/qz8= 2022-09-02T05:57:51.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ux9RbGv0bN8OcVxZBhM8ckg/qz8=', 'Content-Length': '0'} 2022-09-02T05:57:51.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039812b0238597481-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.217 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.218 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.218 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.219 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.219 DEBUG:boto:Method: GET 2022-09-02T05:57:51.220 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.220 DEBUG:boto:Data: 2022-09-02T05:57:51.221 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.221 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.222 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.222 DEBUG:boto:Params: {} 2022-09-02T05:57:51.223 DEBUG:boto:Token: None 2022-09-02T05:57:51.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003994ab00b87bafa6-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.228 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.229 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.230 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.230 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.231 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.231 DEBUG:boto:Method: GET 2022-09-02T05:57:51.232 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:51.232 DEBUG:boto:Data: 2022-09-02T05:57:51.233 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.234 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.235 DEBUG:boto:Params: {} 2022-09-02T05:57:51.235 DEBUG:boto:Token: None 2022-09-02T05:57:51.236 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004482586700eb8fa4-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.241 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:57:51.243 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.243 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.244 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.244 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.245 DEBUG:boto:Method: GET 2022-09-02T05:57:51.245 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.246 DEBUG:boto:Data: 2022-09-02T05:57:51.247 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.247 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.248 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.248 DEBUG:boto:Params: {} 2022-09-02T05:57:51.249 DEBUG:boto:Token: None 2022-09-02T05:57:51.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q= 2022-09-02T05:57:51.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q=', 'Content-Length': '0'} 2022-09-02T05:57:51.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cdf43b2d82ff312-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.254 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.254 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.255 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.255 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.256 DEBUG:boto:Method: GET 2022-09-02T05:57:51.256 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.257 DEBUG:boto:Data: 2022-09-02T05:57:51.257 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.258 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.259 DEBUG:boto:Params: {} 2022-09-02T05:57:51.260 DEBUG:boto:Token: None 2022-09-02T05:57:51.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q= 2022-09-02T05:57:51.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q=', 'Content-Length': '0'} 2022-09-02T05:57:51.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010c06ece50540606-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.265 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.265 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.266 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.266 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.267 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.268 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.268 DEBUG:boto:Data: 2022-09-02T05:57:51.269 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.269 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.270 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.270 DEBUG:boto:Params: {} 2022-09-02T05:57:51.271 DEBUG:boto:Token: None 2022-09-02T05:57:51.271 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z0YzQLuONAAXp8Rv0owVCEa+2OA= 2022-09-02T05:57:51.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z0YzQLuONAAXp8Rv0owVCEa+2OA=', 'Content-Length': '0'} 2022-09-02T05:57:51.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f67d040b393d7a7-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.275 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.276 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.276 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.277 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.277 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.278 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.279 DEBUG:boto:Data: 2022-09-02T05:57:51.279 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.280 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.281 DEBUG:boto:Params: {} 2022-09-02T05:57:51.281 DEBUG:boto:Token: None 2022-09-02T05:57:51.282 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z0YzQLuONAAXp8Rv0owVCEa+2OA= 2022-09-02T05:57:51.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z0YzQLuONAAXp8Rv0owVCEa+2OA=', 'Content-Length': '0'} 2022-09-02T05:57:51.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5deae3a0588d281-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:57:51.286 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.287 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.288 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.288 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.289 DEBUG:boto:Method: GET 2022-09-02T05:57:51.289 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.290 DEBUG:boto:Data: 2022-09-02T05:57:51.290 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.291 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.291 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.292 DEBUG:boto:Params: {} 2022-09-02T05:57:51.293 DEBUG:boto:Token: None 2022-09-02T05:57:51.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q= 2022-09-02T05:57:51.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q=', 'Content-Length': '0'} 2022-09-02T05:57:51.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286bd41f5cb7dab7-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.338 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.339 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.340 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.340 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.341 DEBUG:boto:Method: GET 2022-09-02T05:57:51.341 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.342 DEBUG:boto:Data: 2022-09-02T05:57:51.342 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.344 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.344 DEBUG:boto:Params: {} 2022-09-02T05:57:51.345 DEBUG:boto:Token: None 2022-09-02T05:57:51.346 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:51.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q= 2022-09-02T05:57:51.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K65Yel1EcwXzWWaoixkSByleD5Q=', 'Content-Length': '0'} 2022-09-02T05:57:51.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c3ce329b1d93b4a-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.350 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.351 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.352 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.353 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.353 DEBUG:boto:Method: GET 2022-09-02T05:57:51.354 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.354 DEBUG:boto:Data: 2022-09-02T05:57:51.355 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.355 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.356 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.357 DEBUG:boto:Params: {} 2022-09-02T05:57:51.357 DEBUG:boto:Token: None 2022-09-02T05:57:51.358 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb63758df015b89-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.363 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.364 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.364 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.365 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.366 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.366 DEBUG:boto:Method: GET 2022-09-02T05:57:51.367 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.367 DEBUG:boto:Data: 2022-09-02T05:57:51.368 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.369 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.370 DEBUG:boto:Params: {} 2022-09-02T05:57:51.371 DEBUG:boto:Token: None 2022-09-02T05:57:51.371 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6183c1c89c7c44d-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.378 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.379 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.379 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.380 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.380 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:51.381 DEBUG:boto:Data: 2022-09-02T05:57:51.381 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.382 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.383 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.383 DEBUG:boto:Params: {} 2022-09-02T05:57:51.384 DEBUG:boto:Token: None 2022-09-02T05:57:51.384 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124 2022-09-02T05:57:51.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GyReD6RCXWtmzNnOh0tcyHW07uA= 2022-09-02T05:57:51.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GyReD6RCXWtmzNnOh0tcyHW07uA=', 'Content-Length': '0'} 2022-09-02T05:57:51.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628a22e92bccffab-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.388 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.389 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.389 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.390 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:51.390 DEBUG:boto:Data: 2022-09-02T05:57:51.391 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.392 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.393 DEBUG:boto:Params: {} 2022-09-02T05:57:51.394 DEBUG:boto:Token: None 2022-09-02T05:57:51.396 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124 2022-09-02T05:57:51.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GyReD6RCXWtmzNnOh0tcyHW07uA= 2022-09-02T05:57:51.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GyReD6RCXWtmzNnOh0tcyHW07uA=', 'Content-Length': '0'} 2022-09-02T05:57:51.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113c99a2fe232572-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:57:51.400 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.401 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.401 DEBUG:boto:Method: GET 2022-09-02T05:57:51.402 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:51.403 DEBUG:boto:Data: 2022-09-02T05:57:51.403 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.404 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.405 DEBUG:boto:Params: {} 2022-09-02T05:57:51.405 DEBUG:boto:Token: None 2022-09-02T05:57:51.406 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124 2022-09-02T05:57:51.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v53A32kTRhlJi+AOKufarkinf8E= 2022-09-02T05:57:51.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v53A32kTRhlJi+AOKufarkinf8E=', 'Content-Length': '0'} 2022-09-02T05:57:51.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a89cbf1af343403-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.410 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:51.411 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:51.412 DEBUG:boto:Method: GET 2022-09-02T05:57:51.412 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:51.413 DEBUG:boto:Data: 2022-09-02T05:57:51.413 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.414 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.415 DEBUG:boto:Params: {} 2022-09-02T05:57:51.415 DEBUG:boto:Token: None 2022-09-02T05:57:51.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj124 2022-09-02T05:57:51.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v53A32kTRhlJi+AOKufarkinf8E= 2022-09-02T05:57:51.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v53A32kTRhlJi+AOKufarkinf8E=', 'Content-Length': '0'} 2022-09-02T05:57:51.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95319c1f837798b-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.420 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.421 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.422 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.423 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.424 DEBUG:boto:Method: GET 2022-09-02T05:57:51.425 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.426 DEBUG:boto:Data: 2022-09-02T05:57:51.427 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.429 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.430 DEBUG:boto:Params: {} 2022-09-02T05:57:51.431 DEBUG:boto:Token: None 2022-09-02T05:57:51.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094c8986127f81331-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.438 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.439 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.440 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.440 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.441 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.441 DEBUG:boto:Method: GET 2022-09-02T05:57:51.442 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:51.442 DEBUG:boto:Data: 2022-09-02T05:57:51.443 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.444 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.444 DEBUG:boto:Params: {} 2022-09-02T05:57:51.445 DEBUG:boto:Token: None 2022-09-02T05:57:51.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2cd6ba82d7dde8-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.451 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:57:51.454 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.454 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.455 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.455 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.456 DEBUG:boto:Method: GET 2022-09-02T05:57:51.456 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.457 DEBUG:boto:Data: 2022-09-02T05:57:51.458 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.458 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.459 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.459 DEBUG:boto:Params: {} 2022-09-02T05:57:51.460 DEBUG:boto:Token: None 2022-09-02T05:57:51.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY= 2022-09-02T05:57:51.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY=', 'Content-Length': '0'} 2022-09-02T05:57:51.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3eb6f94dc50aca-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.465 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.465 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.466 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.467 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.468 DEBUG:boto:Method: GET 2022-09-02T05:57:51.469 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.470 DEBUG:boto:Data: 2022-09-02T05:57:51.471 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.473 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.474 DEBUG:boto:Params: {} 2022-09-02T05:57:51.475 DEBUG:boto:Token: None 2022-09-02T05:57:51.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY= 2022-09-02T05:57:51.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY=', 'Content-Length': '0'} 2022-09-02T05:57:51.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f20592a8cfd3632-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.480 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.481 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.481 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.482 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.482 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.483 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.483 DEBUG:boto:Data: 2022-09-02T05:57:51.484 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.485 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.485 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.486 DEBUG:boto:Params: {} 2022-09-02T05:57:51.486 DEBUG:boto:Token: None 2022-09-02T05:57:51.487 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U2iV7y4Jmb8vfAI9e8u8ztNEeHM= 2022-09-02T05:57:51.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U2iV7y4Jmb8vfAI9e8u8ztNEeHM=', 'Content-Length': '0'} 2022-09-02T05:57:51.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5eb9fcda8e56357-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.491 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.492 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.492 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.493 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.493 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.494 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.494 DEBUG:boto:Data: 2022-09-02T05:57:51.495 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.496 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.496 DEBUG:boto:Params: {} 2022-09-02T05:57:51.497 DEBUG:boto:Token: None 2022-09-02T05:57:51.497 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.498 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U2iV7y4Jmb8vfAI9e8u8ztNEeHM= 2022-09-02T05:57:51.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U2iV7y4Jmb8vfAI9e8u8ztNEeHM=', 'Content-Length': '0'} 2022-09-02T05:57:51.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075cd1a2881ae0a48-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:57:51.502 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.503 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.503 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.504 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.504 DEBUG:boto:Method: GET 2022-09-02T05:57:51.505 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.505 DEBUG:boto:Data: 2022-09-02T05:57:51.506 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.506 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.507 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.507 DEBUG:boto:Params: {} 2022-09-02T05:57:51.508 DEBUG:boto:Token: None 2022-09-02T05:57:51.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY= 2022-09-02T05:57:51.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY=', 'Content-Length': '0'} 2022-09-02T05:57:51.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de0e8e93a3436e3-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.554 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.555 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.555 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.556 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.556 DEBUG:boto:Method: GET 2022-09-02T05:57:51.557 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.557 DEBUG:boto:Data: 2022-09-02T05:57:51.558 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.559 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.560 DEBUG:boto:Params: {} 2022-09-02T05:57:51.560 DEBUG:boto:Token: None 2022-09-02T05:57:51.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:51.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY= 2022-09-02T05:57:51.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikty294Ja6fhvZH9hjnSSZBO6HY=', 'Content-Length': '0'} 2022-09-02T05:57:51.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030efd030dcdbf6a8-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.565 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.566 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.566 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.567 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.567 DEBUG:boto:Method: GET 2022-09-02T05:57:51.568 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.568 DEBUG:boto:Data: 2022-09-02T05:57:51.569 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.569 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.570 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.570 DEBUG:boto:Params: {} 2022-09-02T05:57:51.571 DEBUG:boto:Token: None 2022-09-02T05:57:51.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cc21ef2c5f3bacd-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.576 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.577 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.577 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.578 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.579 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.579 DEBUG:boto:Method: GET 2022-09-02T05:57:51.580 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.580 DEBUG:boto:Data: 2022-09-02T05:57:51.581 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.582 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.582 DEBUG:boto:Params: {} 2022-09-02T05:57:51.583 DEBUG:boto:Token: None 2022-09-02T05:57:51.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cfca1f3c7e822eb-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.589 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.590 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.590 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.591 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.591 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:51.592 DEBUG:boto:Data: 2022-09-02T05:57:51.592 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.593 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.593 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.594 DEBUG:boto:Params: {} 2022-09-02T05:57:51.594 DEBUG:boto:Token: None 2022-09-02T05:57:51.595 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145 2022-09-02T05:57:51.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxJxEp1TaSjcziEzBZShbMdnHas= 2022-09-02T05:57:51.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxJxEp1TaSjcziEzBZShbMdnHas=', 'Content-Length': '0'} 2022-09-02T05:57:51.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059388e8f3b4e0618-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.599 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.600 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.600 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.601 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:51.601 DEBUG:boto:Data: 2022-09-02T05:57:51.602 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.603 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.603 DEBUG:boto:Params: {} 2022-09-02T05:57:51.604 DEBUG:boto:Token: None 2022-09-02T05:57:51.604 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145 2022-09-02T05:57:51.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxJxEp1TaSjcziEzBZShbMdnHas= 2022-09-02T05:57:51.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxJxEp1TaSjcziEzBZShbMdnHas=', 'Content-Length': '0'} 2022-09-02T05:57:51.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5592e0fce5cebf4-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:57:51.608 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.609 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.610 DEBUG:boto:Method: GET 2022-09-02T05:57:51.610 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:51.611 DEBUG:boto:Data: 2022-09-02T05:57:51.611 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.612 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.613 DEBUG:boto:Params: {} 2022-09-02T05:57:51.613 DEBUG:boto:Token: None 2022-09-02T05:57:51.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145 2022-09-02T05:57:51.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tnlbLieXauOMfUuF9aR4mJW7PA0= 2022-09-02T05:57:51.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tnlbLieXauOMfUuF9aR4mJW7PA0=', 'Content-Length': '0'} 2022-09-02T05:57:51.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f47c55c14bd69b59-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.658 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:51.659 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:51.659 DEBUG:boto:Method: GET 2022-09-02T05:57:51.660 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:51.660 DEBUG:boto:Data: 2022-09-02T05:57:51.661 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.662 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.663 DEBUG:boto:Params: {} 2022-09-02T05:57:51.663 DEBUG:boto:Token: None 2022-09-02T05:57:51.664 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj145 2022-09-02T05:57:51.664 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tnlbLieXauOMfUuF9aR4mJW7PA0= 2022-09-02T05:57:51.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tnlbLieXauOMfUuF9aR4mJW7PA0=', 'Content-Length': '0'} 2022-09-02T05:57:51.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a588dc826268fa8f-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.668 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.669 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.669 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.670 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.671 DEBUG:boto:Method: GET 2022-09-02T05:57:51.671 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.672 DEBUG:boto:Data: 2022-09-02T05:57:51.672 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.673 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.673 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.674 DEBUG:boto:Params: {} 2022-09-02T05:57:51.674 DEBUG:boto:Token: None 2022-09-02T05:57:51.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.676 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5330084da2b635-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.680 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.680 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.681 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.681 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.682 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.683 DEBUG:boto:Method: GET 2022-09-02T05:57:51.683 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:51.684 DEBUG:boto:Data: 2022-09-02T05:57:51.684 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.685 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.686 DEBUG:boto:Params: {} 2022-09-02T05:57:51.686 DEBUG:boto:Token: None 2022-09-02T05:57:51.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4b818b85219b63-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.693 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:57:51.694 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.695 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.695 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.696 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.696 DEBUG:boto:Method: GET 2022-09-02T05:57:51.697 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.697 DEBUG:boto:Data: 2022-09-02T05:57:51.698 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.698 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.699 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.699 DEBUG:boto:Params: {} 2022-09-02T05:57:51.700 DEBUG:boto:Token: None 2022-09-02T05:57:51.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00= 2022-09-02T05:57:51.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00=', 'Content-Length': '0'} 2022-09-02T05:57:51.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec614d4e1ecbe70-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.705 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.705 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.706 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.706 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.707 DEBUG:boto:Method: GET 2022-09-02T05:57:51.707 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.708 DEBUG:boto:Data: 2022-09-02T05:57:51.708 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.709 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.710 DEBUG:boto:Params: {} 2022-09-02T05:57:51.710 DEBUG:boto:Token: None 2022-09-02T05:57:51.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00= 2022-09-02T05:57:51.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00=', 'Content-Length': '0'} 2022-09-02T05:57:51.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8e7546a532b48c-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.716 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.716 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.717 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.717 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.718 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.718 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.719 DEBUG:boto:Data: 2022-09-02T05:57:51.719 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.720 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.721 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.721 DEBUG:boto:Params: {} 2022-09-02T05:57:51.722 DEBUG:boto:Token: None 2022-09-02T05:57:51.723 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b5sMU+PysvR/x9M9TgbpapwhrPo= 2022-09-02T05:57:51.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b5sMU+PysvR/x9M9TgbpapwhrPo=', 'Content-Length': '0'} 2022-09-02T05:57:51.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9f9249a3fe9290-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.727 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.727 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.728 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.728 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.729 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.729 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.730 DEBUG:boto:Data: 2022-09-02T05:57:51.731 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.732 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.732 DEBUG:boto:Params: {} 2022-09-02T05:57:51.733 DEBUG:boto:Token: None 2022-09-02T05:57:51.734 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b5sMU+PysvR/x9M9TgbpapwhrPo= 2022-09-02T05:57:51.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b5sMU+PysvR/x9M9TgbpapwhrPo=', 'Content-Length': '0'} 2022-09-02T05:57:51.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1f4bd7dadbd1e8-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:57:51.749 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.750 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.751 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.751 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.752 DEBUG:boto:Method: GET 2022-09-02T05:57:51.752 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.753 DEBUG:boto:Data: 2022-09-02T05:57:51.754 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.754 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.755 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.755 DEBUG:boto:Params: {} 2022-09-02T05:57:51.756 DEBUG:boto:Token: None 2022-09-02T05:57:51.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00= 2022-09-02T05:57:51.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00=', 'Content-Length': '0'} 2022-09-02T05:57:51.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aebbfdda3540af6f-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.802 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.803 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.803 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.804 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.805 DEBUG:boto:Method: GET 2022-09-02T05:57:51.806 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.807 DEBUG:boto:Data: 2022-09-02T05:57:51.808 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.810 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.811 DEBUG:boto:Params: {} 2022-09-02T05:57:51.812 DEBUG:boto:Token: None 2022-09-02T05:57:51.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:51.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00= 2022-09-02T05:57:51.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8GCzCXZQYDwV3/H/17jCUEdRB00=', 'Content-Length': '0'} 2022-09-02T05:57:51.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a07ecc33473ee5e-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.818 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.819 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.819 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.820 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.820 DEBUG:boto:Method: GET 2022-09-02T05:57:51.821 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.821 DEBUG:boto:Data: 2022-09-02T05:57:51.822 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.822 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.823 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.823 DEBUG:boto:Params: {} 2022-09-02T05:57:51.824 DEBUG:boto:Token: None 2022-09-02T05:57:51.824 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c55dabebfd7337-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.829 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.830 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.830 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.831 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.831 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.832 DEBUG:boto:Method: GET 2022-09-02T05:57:51.832 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.833 DEBUG:boto:Data: 2022-09-02T05:57:51.834 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.835 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.835 DEBUG:boto:Params: {} 2022-09-02T05:57:51.836 DEBUG:boto:Token: None 2022-09-02T05:57:51.836 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c3d4c90280aec6-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.842 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.843 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.844 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.844 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.845 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:51.845 DEBUG:boto:Data: 2022-09-02T05:57:51.846 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.847 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.848 DEBUG:boto:Params: {} 2022-09-02T05:57:51.848 DEBUG:boto:Token: None 2022-09-02T05:57:51.849 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155 2022-09-02T05:57:51.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FCuIq8IE1mNbw/gZpL2F1vjIG0s= 2022-09-02T05:57:51.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FCuIq8IE1mNbw/gZpL2F1vjIG0s=', 'Content-Length': '0'} 2022-09-02T05:57:51.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000792db9feed670540-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.853 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.854 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.855 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.855 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:51.856 DEBUG:boto:Data: 2022-09-02T05:57:51.856 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.857 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.858 DEBUG:boto:Params: {} 2022-09-02T05:57:51.859 DEBUG:boto:Token: None 2022-09-02T05:57:51.860 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155 2022-09-02T05:57:51.860 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FCuIq8IE1mNbw/gZpL2F1vjIG0s= 2022-09-02T05:57:51.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FCuIq8IE1mNbw/gZpL2F1vjIG0s=', 'Content-Length': '0'} 2022-09-02T05:57:51.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379f42ad1328895b-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:57:51.864 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.865 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.865 DEBUG:boto:Method: GET 2022-09-02T05:57:51.866 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:51.867 DEBUG:boto:Data: 2022-09-02T05:57:51.867 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.868 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.869 DEBUG:boto:Params: {} 2022-09-02T05:57:51.869 DEBUG:boto:Token: None 2022-09-02T05:57:51.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155 2022-09-02T05:57:51.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xVghqW1B2smKEgSlCjYxqYMYe/c= 2022-09-02T05:57:51.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xVghqW1B2smKEgSlCjYxqYMYe/c=', 'Content-Length': '0'} 2022-09-02T05:57:51.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5d0ba88fe0aed9-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.874 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:51.875 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:51.875 DEBUG:boto:Method: GET 2022-09-02T05:57:51.876 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:51.876 DEBUG:boto:Data: 2022-09-02T05:57:51.877 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.878 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.879 DEBUG:boto:Params: {} 2022-09-02T05:57:51.879 DEBUG:boto:Token: None 2022-09-02T05:57:51.880 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj155 2022-09-02T05:57:51.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xVghqW1B2smKEgSlCjYxqYMYe/c= 2022-09-02T05:57:51.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xVghqW1B2smKEgSlCjYxqYMYe/c=', 'Content-Length': '0'} 2022-09-02T05:57:51.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d06cfd620539966-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.884 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.885 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.885 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.886 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.886 DEBUG:boto:Method: GET 2022-09-02T05:57:51.887 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.887 DEBUG:boto:Data: 2022-09-02T05:57:51.888 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.888 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.889 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.889 DEBUG:boto:Params: {} 2022-09-02T05:57:51.890 DEBUG:boto:Token: None 2022-09-02T05:57:51.891 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7854d0ca2f96aba-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.896 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.896 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:51.897 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:51.897 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.898 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.899 DEBUG:boto:Method: GET 2022-09-02T05:57:51.899 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:51.900 DEBUG:boto:Data: 2022-09-02T05:57:51.900 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.902 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.902 DEBUG:boto:Params: {} 2022-09-02T05:57:51.903 DEBUG:boto:Token: None 2022-09-02T05:57:51.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/ 2022-09-02T05:57:51.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw= 2022-09-02T05:57:51.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2nF6YgouXdAnn6krU/duaWyAwZw=', 'Content-Length': '0'} 2022-09-02T05:57:51.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000550939e24d4b00cb-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.910 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:51.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:57:51.911 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:51.912 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:51.912 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.913 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.913 DEBUG:boto:Method: GET 2022-09-02T05:57:51.914 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.914 DEBUG:boto:Data: 2022-09-02T05:57:51.915 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.916 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.916 DEBUG:boto:Params: {} 2022-09-02T05:57:51.917 DEBUG:boto:Token: None 2022-09-02T05:57:51.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SEdbtqNxGk3UoAbv77UJq6Qsa40= 2022-09-02T05:57:51.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SEdbtqNxGk3UoAbv77UJq6Qsa40=', 'Content-Length': '0'} 2022-09-02T05:57:51.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001072f68295605bbe-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.922 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:51.922 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:51.923 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.923 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.924 DEBUG:boto:Method: GET 2022-09-02T05:57:51.924 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.925 DEBUG:boto:Data: 2022-09-02T05:57:51.925 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.926 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.927 DEBUG:boto:Params: {} 2022-09-02T05:57:51.927 DEBUG:boto:Token: None 2022-09-02T05:57:51.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SEdbtqNxGk3UoAbv77UJq6Qsa40= 2022-09-02T05:57:51.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SEdbtqNxGk3UoAbv77UJq6Qsa40=', 'Content-Length': '0'} 2022-09-02T05:57:51.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002341bb2653c53271-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.974 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:51.975 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:51.976 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.976 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.977 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.978 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.978 DEBUG:boto:Data: 2022-09-02T05:57:51.979 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:51.980 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.980 DEBUG:boto:Params: {} 2022-09-02T05:57:51.981 DEBUG:boto:Token: None 2022-09-02T05:57:51.981 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HNbGJQ0112ohEXjxL/2xpJBnKMw= 2022-09-02T05:57:51.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HNbGJQ0112ohEXjxL/2xpJBnKMw=', 'Content-Length': '0'} 2022-09-02T05:57:51.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce342e924eedd1d9-0063119b5f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.985 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:51.986 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:51.986 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.987 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.987 DEBUG:boto:Method: HEAD 2022-09-02T05:57:51.988 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.988 DEBUG:boto:Data: 2022-09-02T05:57:51.989 DEBUG:boto:Headers: {} 2022-09-02T05:57:51.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:51.990 DEBUG:boto:Port: 8000 2022-09-02T05:57:51.990 DEBUG:boto:Params: {} 2022-09-02T05:57:51.991 DEBUG:boto:Token: None 2022-09-02T05:57:51.992 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:51 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HNbGJQ0112ohEXjxL/2xpJBnKMw= 2022-09-02T05:57:51.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HNbGJQ0112ohEXjxL/2xpJBnKMw=', 'Content-Length': '0'} 2022-09-02T05:57:51.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c188cc75722e4fd2-0063119b5f-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:51.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:57:51.996 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:51.997 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:51.997 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.998 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:51.999 DEBUG:boto:Method: GET 2022-09-02T05:57:51.999 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:52.000 DEBUG:boto:Data: 2022-09-02T05:57:52.001 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.002 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.002 DEBUG:boto:Params: {} 2022-09-02T05:57:52.003 DEBUG:boto:Token: None 2022-09-02T05:57:52.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:52.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JEQxFzBXnrFgqbbzSPs435R9t9g= 2022-09-02T05:57:52.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JEQxFzBXnrFgqbbzSPs435R9t9g=', 'Content-Length': '0'} 2022-09-02T05:57:52.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000098127a7d6a0785-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.008 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:52.009 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:52.010 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:52.010 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:52.011 DEBUG:boto:Method: GET 2022-09-02T05:57:52.011 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:52.012 DEBUG:boto:Data: 2022-09-02T05:57:52.012 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.013 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.014 DEBUG:boto:Params: {} 2022-09-02T05:57:52.015 DEBUG:boto:Token: None 2022-09-02T05:57:52.015 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:52.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JEQxFzBXnrFgqbbzSPs435R9t9g= 2022-09-02T05:57:52.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JEQxFzBXnrFgqbbzSPs435R9t9g=', 'Content-Length': '0'} 2022-09-02T05:57:52.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065dae58cf4bf327a-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.020 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.020 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.021 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.022 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.022 DEBUG:boto:Method: GET 2022-09-02T05:57:52.023 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.023 DEBUG:boto:Data: 2022-09-02T05:57:52.024 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.025 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.025 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.026 DEBUG:boto:Params: {} 2022-09-02T05:57:52.026 DEBUG:boto:Token: None 2022-09-02T05:57:52.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003255588538f89e04-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.032 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.033 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.034 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.034 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.035 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.036 DEBUG:boto:Method: GET 2022-09-02T05:57:52.036 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.037 DEBUG:boto:Data: 2022-09-02T05:57:52.038 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.039 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.040 DEBUG:boto:Params: {} 2022-09-02T05:57:52.041 DEBUG:boto:Token: None 2022-09-02T05:57:52.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000766baef9d5ac43fa-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.090 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.091 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:52.092 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:52.092 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.093 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:52.094 DEBUG:boto:Data: 2022-09-02T05:57:52.094 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.095 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.095 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.096 DEBUG:boto:Params: {} 2022-09-02T05:57:52.096 DEBUG:boto:Token: None 2022-09-02T05:57:52.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj17 2022-09-02T05:57:52.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6HXB252JJxf3m/WYFu0rYxJwdaQ= 2022-09-02T05:57:52.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6HXB252JJxf3m/WYFu0rYxJwdaQ=', 'Content-Length': '0'} 2022-09-02T05:57:52.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad1cf8a9ddc8987-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.101 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:52.102 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:52.102 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.103 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:52.104 DEBUG:boto:Data: 2022-09-02T05:57:52.105 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.106 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.107 DEBUG:boto:Params: {} 2022-09-02T05:57:52.107 DEBUG:boto:Token: None 2022-09-02T05:57:52.108 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj17 2022-09-02T05:57:52.109 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6HXB252JJxf3m/WYFu0rYxJwdaQ= 2022-09-02T05:57:52.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6HXB252JJxf3m/WYFu0rYxJwdaQ=', 'Content-Length': '0'} 2022-09-02T05:57:52.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1987b79bd459090-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:57:52.113 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:52.114 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:52.114 DEBUG:boto:Method: GET 2022-09-02T05:57:52.115 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:52.115 DEBUG:boto:Data: 2022-09-02T05:57:52.116 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.116 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.117 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.118 DEBUG:boto:Params: {} 2022-09-02T05:57:52.119 DEBUG:boto:Token: None 2022-09-02T05:57:52.119 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj17 2022-09-02T05:57:52.120 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ob9H1LdGaO7ew4QOeoVC5eAzqD0= 2022-09-02T05:57:52.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ob9H1LdGaO7ew4QOeoVC5eAzqD0=', 'Content-Length': '0'} 2022-09-02T05:57:52.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f88a529aa05b524-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.124 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:52.125 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:52.125 DEBUG:boto:Method: GET 2022-09-02T05:57:52.126 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:52.126 DEBUG:boto:Data: 2022-09-02T05:57:52.127 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.128 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.129 DEBUG:boto:Params: {} 2022-09-02T05:57:52.129 DEBUG:boto:Token: None 2022-09-02T05:57:52.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj17 2022-09-02T05:57:52.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ob9H1LdGaO7ew4QOeoVC5eAzqD0= 2022-09-02T05:57:52.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ob9H1LdGaO7ew4QOeoVC5eAzqD0=', 'Content-Length': '0'} 2022-09-02T05:57:52.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e98a7e0163def1-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.134 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.135 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.135 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.136 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.136 DEBUG:boto:Method: GET 2022-09-02T05:57:52.137 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.137 DEBUG:boto:Data: 2022-09-02T05:57:52.138 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.139 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.139 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.140 DEBUG:boto:Params: {} 2022-09-02T05:57:52.140 DEBUG:boto:Token: None 2022-09-02T05:57:52.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020cc13a0e90972d4-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.145 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.146 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.147 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.147 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.148 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.148 DEBUG:boto:Method: GET 2022-09-02T05:57:52.149 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:52.149 DEBUG:boto:Data: 2022-09-02T05:57:52.150 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.151 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.151 DEBUG:boto:Params: {} 2022-09-02T05:57:52.152 DEBUG:boto:Token: None 2022-09-02T05:57:52.152 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc30fd6a1810c15-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.158 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:57:52.159 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.160 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.161 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.161 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.162 DEBUG:boto:Method: GET 2022-09-02T05:57:52.162 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.163 DEBUG:boto:Data: 2022-09-02T05:57:52.163 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.164 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.165 DEBUG:boto:Params: {} 2022-09-02T05:57:52.165 DEBUG:boto:Token: None 2022-09-02T05:57:52.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw= 2022-09-02T05:57:52.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw=', 'Content-Length': '0'} 2022-09-02T05:57:52.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043096c2d3315975d-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.170 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.171 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.171 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.172 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.172 DEBUG:boto:Method: GET 2022-09-02T05:57:52.173 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.174 DEBUG:boto:Data: 2022-09-02T05:57:52.174 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.175 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.176 DEBUG:boto:Params: {} 2022-09-02T05:57:52.176 DEBUG:boto:Token: None 2022-09-02T05:57:52.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw= 2022-09-02T05:57:52.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw=', 'Content-Length': '0'} 2022-09-02T05:57:52.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24f9eeae9594715-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.181 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.182 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.182 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.183 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.183 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.184 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.184 DEBUG:boto:Data: 2022-09-02T05:57:52.185 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.186 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.187 DEBUG:boto:Params: {} 2022-09-02T05:57:52.187 DEBUG:boto:Token: None 2022-09-02T05:57:52.188 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hsKZ3z0nZJUtltf2qzod6c9ibfM= 2022-09-02T05:57:52.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hsKZ3z0nZJUtltf2qzod6c9ibfM=', 'Content-Length': '0'} 2022-09-02T05:57:52.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003248b01fbb240020-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.192 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.192 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.193 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.193 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.194 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.194 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.195 DEBUG:boto:Data: 2022-09-02T05:57:52.195 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.196 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.197 DEBUG:boto:Params: {} 2022-09-02T05:57:52.197 DEBUG:boto:Token: None 2022-09-02T05:57:52.198 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hsKZ3z0nZJUtltf2qzod6c9ibfM= 2022-09-02T05:57:52.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hsKZ3z0nZJUtltf2qzod6c9ibfM=', 'Content-Length': '0'} 2022-09-02T05:57:52.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000576a1da491d91f4f-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:57:52.203 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.203 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.204 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.204 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.205 DEBUG:boto:Method: GET 2022-09-02T05:57:52.205 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.206 DEBUG:boto:Data: 2022-09-02T05:57:52.206 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.207 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.208 DEBUG:boto:Params: {} 2022-09-02T05:57:52.208 DEBUG:boto:Token: None 2022-09-02T05:57:52.209 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.210 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw= 2022-09-02T05:57:52.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw=', 'Content-Length': '0'} 2022-09-02T05:57:52.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb0912b2f5608d6-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.254 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.255 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.256 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.256 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.257 DEBUG:boto:Method: GET 2022-09-02T05:57:52.257 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.258 DEBUG:boto:Data: 2022-09-02T05:57:52.259 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.260 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.260 DEBUG:boto:Params: {} 2022-09-02T05:57:52.261 DEBUG:boto:Token: None 2022-09-02T05:57:52.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:52.262 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw= 2022-09-02T05:57:52.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ud2iUy+8ducmlUckGOqS3ohwIyw=', 'Content-Length': '0'} 2022-09-02T05:57:52.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1d2875ac2596ca-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.266 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.267 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.267 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.268 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.268 DEBUG:boto:Method: GET 2022-09-02T05:57:52.269 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.269 DEBUG:boto:Data: 2022-09-02T05:57:52.270 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.270 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.271 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.271 DEBUG:boto:Params: {} 2022-09-02T05:57:52.272 DEBUG:boto:Token: None 2022-09-02T05:57:52.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.273 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a57280216aefd3-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.278 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.279 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.279 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.280 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.280 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.281 DEBUG:boto:Method: GET 2022-09-02T05:57:52.282 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.282 DEBUG:boto:Data: 2022-09-02T05:57:52.283 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.284 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.284 DEBUG:boto:Params: {} 2022-09-02T05:57:52.285 DEBUG:boto:Token: None 2022-09-02T05:57:52.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad0b205536c59641-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.292 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.293 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.294 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.294 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.295 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:52.295 DEBUG:boto:Data: 2022-09-02T05:57:52.296 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.297 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.297 DEBUG:boto:Params: {} 2022-09-02T05:57:52.298 DEBUG:boto:Token: None 2022-09-02T05:57:52.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19 2022-09-02T05:57:52.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EusIfCX6y7ipGy+qKGeh7i5lA2E= 2022-09-02T05:57:52.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EusIfCX6y7ipGy+qKGeh7i5lA2E=', 'Content-Length': '0'} 2022-09-02T05:57:52.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c0d0d5149b0d2f-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.302 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.303 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.303 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.304 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:52.304 DEBUG:boto:Data: 2022-09-02T05:57:52.305 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.306 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.307 DEBUG:boto:Params: {} 2022-09-02T05:57:52.307 DEBUG:boto:Token: None 2022-09-02T05:57:52.308 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19 2022-09-02T05:57:52.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EusIfCX6y7ipGy+qKGeh7i5lA2E= 2022-09-02T05:57:52.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EusIfCX6y7ipGy+qKGeh7i5lA2E=', 'Content-Length': '0'} 2022-09-02T05:57:52.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5002e3e69bb2702-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:57:52.312 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.313 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.313 DEBUG:boto:Method: GET 2022-09-02T05:57:52.314 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:52.315 DEBUG:boto:Data: 2022-09-02T05:57:52.315 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.316 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.316 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.317 DEBUG:boto:Params: {} 2022-09-02T05:57:52.317 DEBUG:boto:Token: None 2022-09-02T05:57:52.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19 2022-09-02T05:57:52.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PuhsSQEtXaoP4N1GZqFLgRH/fYo= 2022-09-02T05:57:52.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PuhsSQEtXaoP4N1GZqFLgRH/fYo=', 'Content-Length': '0'} 2022-09-02T05:57:52.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028751bc89f124e40-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.322 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:52.323 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:52.323 DEBUG:boto:Method: GET 2022-09-02T05:57:52.324 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:52.324 DEBUG:boto:Data: 2022-09-02T05:57:52.325 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.326 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.326 DEBUG:boto:Params: {} 2022-09-02T05:57:52.327 DEBUG:boto:Token: None 2022-09-02T05:57:52.328 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj19 2022-09-02T05:57:52.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PuhsSQEtXaoP4N1GZqFLgRH/fYo= 2022-09-02T05:57:52.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PuhsSQEtXaoP4N1GZqFLgRH/fYo=', 'Content-Length': '0'} 2022-09-02T05:57:52.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033b8b396f74ae6ff-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.374 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.375 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.375 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.376 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.377 DEBUG:boto:Method: GET 2022-09-02T05:57:52.377 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.378 DEBUG:boto:Data: 2022-09-02T05:57:52.378 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.379 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.379 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.380 DEBUG:boto:Params: {} 2022-09-02T05:57:52.381 DEBUG:boto:Token: None 2022-09-02T05:57:52.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca53b33b511d9c9-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.386 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.387 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.387 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.388 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.388 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.389 DEBUG:boto:Method: GET 2022-09-02T05:57:52.389 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:52.390 DEBUG:boto:Data: 2022-09-02T05:57:52.391 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.392 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.392 DEBUG:boto:Params: {} 2022-09-02T05:57:52.393 DEBUG:boto:Token: None 2022-09-02T05:57:52.393 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f45faa1397b26da-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.400 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:57:52.401 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.402 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.402 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.403 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.403 DEBUG:boto:Method: GET 2022-09-02T05:57:52.404 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.405 DEBUG:boto:Data: 2022-09-02T05:57:52.405 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.406 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.407 DEBUG:boto:Params: {} 2022-09-02T05:57:52.407 DEBUG:boto:Token: None 2022-09-02T05:57:52.408 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A= 2022-09-02T05:57:52.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A=', 'Content-Length': '0'} 2022-09-02T05:57:52.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022bc68e974263546-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.413 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.413 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.414 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.414 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.415 DEBUG:boto:Method: GET 2022-09-02T05:57:52.415 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.416 DEBUG:boto:Data: 2022-09-02T05:57:52.416 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.418 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.418 DEBUG:boto:Params: {} 2022-09-02T05:57:52.419 DEBUG:boto:Token: None 2022-09-02T05:57:52.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.420 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A= 2022-09-02T05:57:52.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A=', 'Content-Length': '0'} 2022-09-02T05:57:52.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005dc791b03b41a77-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.423 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.424 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.424 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.425 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.425 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.426 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.427 DEBUG:boto:Data: 2022-09-02T05:57:52.427 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.428 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.428 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.429 DEBUG:boto:Params: {} 2022-09-02T05:57:52.429 DEBUG:boto:Token: None 2022-09-02T05:57:52.430 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t8IRghOWERDKwK/qTixgWOrsHwM= 2022-09-02T05:57:52.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t8IRghOWERDKwK/qTixgWOrsHwM=', 'Content-Length': '0'} 2022-09-02T05:57:52.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af40888f2e8b6749-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.434 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.435 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.435 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.436 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.436 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.437 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.437 DEBUG:boto:Data: 2022-09-02T05:57:52.438 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.439 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.440 DEBUG:boto:Params: {} 2022-09-02T05:57:52.440 DEBUG:boto:Token: None 2022-09-02T05:57:52.441 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.441 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t8IRghOWERDKwK/qTixgWOrsHwM= 2022-09-02T05:57:52.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t8IRghOWERDKwK/qTixgWOrsHwM=', 'Content-Length': '0'} 2022-09-02T05:57:52.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a1988039c356c7-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:57:52.445 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.446 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.446 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.447 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.447 DEBUG:boto:Method: GET 2022-09-02T05:57:52.448 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.448 DEBUG:boto:Data: 2022-09-02T05:57:52.449 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.450 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.451 DEBUG:boto:Params: {} 2022-09-02T05:57:52.451 DEBUG:boto:Token: None 2022-09-02T05:57:52.452 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A= 2022-09-02T05:57:52.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A=', 'Content-Length': '0'} 2022-09-02T05:57:52.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4efc384de5f0ec3-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.456 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.457 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.457 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.458 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.459 DEBUG:boto:Method: GET 2022-09-02T05:57:52.459 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.460 DEBUG:boto:Data: 2022-09-02T05:57:52.460 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.462 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.462 DEBUG:boto:Params: {} 2022-09-02T05:57:52.463 DEBUG:boto:Token: None 2022-09-02T05:57:52.463 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:57:52.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A= 2022-09-02T05:57:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:83nusHkEQlcUL/NyvsOyZ5v1R+A=', 'Content-Length': '0'} 2022-09-02T05:57:52.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c391e2342f7a55-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.510 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.511 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.511 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.512 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.513 DEBUG:boto:Method: GET 2022-09-02T05:57:52.513 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.514 DEBUG:boto:Data: 2022-09-02T05:57:52.514 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.516 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.516 DEBUG:boto:Params: {} 2022-09-02T05:57:52.517 DEBUG:boto:Token: None 2022-09-02T05:57:52.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f136554b875e13e1-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.522 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.523 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.524 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.524 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.525 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.525 DEBUG:boto:Method: GET 2022-09-02T05:57:52.526 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.527 DEBUG:boto:Data: 2022-09-02T05:57:52.527 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.528 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.529 DEBUG:boto:Params: {} 2022-09-02T05:57:52.529 DEBUG:boto:Token: None 2022-09-02T05:57:52.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cd9f3df3eecfef-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.537 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.537 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.538 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.538 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.539 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:57:52.539 DEBUG:boto:Data: 2022-09-02T05:57:52.540 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.540 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.541 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.542 DEBUG:boto:Params: {} 2022-09-02T05:57:52.542 DEBUG:boto:Token: None 2022-09-02T05:57:52.543 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22 2022-09-02T05:57:52.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+YLtsEbU7FbdFC9WiCvneNGuHiw= 2022-09-02T05:57:52.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+YLtsEbU7FbdFC9WiCvneNGuHiw=', 'Content-Length': '0'} 2022-09-02T05:57:52.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006601e6107548a325-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.547 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.548 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.548 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.549 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:57:52.549 DEBUG:boto:Data: 2022-09-02T05:57:52.550 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.551 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.552 DEBUG:boto:Params: {} 2022-09-02T05:57:52.552 DEBUG:boto:Token: None 2022-09-02T05:57:52.553 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22 2022-09-02T05:57:52.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+YLtsEbU7FbdFC9WiCvneNGuHiw= 2022-09-02T05:57:52.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+YLtsEbU7FbdFC9WiCvneNGuHiw=', 'Content-Length': '0'} 2022-09-02T05:57:52.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e351af9a8287d5f-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:57:52.558 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.558 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.559 DEBUG:boto:Method: GET 2022-09-02T05:57:52.560 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:57:52.560 DEBUG:boto:Data: 2022-09-02T05:57:52.561 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.561 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.562 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.562 DEBUG:boto:Params: {} 2022-09-02T05:57:52.563 DEBUG:boto:Token: None 2022-09-02T05:57:52.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22 2022-09-02T05:57:52.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IpSTXMM9Myl1CG1wSS/3Ewr02rA= 2022-09-02T05:57:52.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IpSTXMM9Myl1CG1wSS/3Ewr02rA=', 'Content-Length': '0'} 2022-09-02T05:57:52.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca6ded3ec44ad62-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.568 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:57:52.568 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:57:52.569 DEBUG:boto:Method: GET 2022-09-02T05:57:52.569 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:57:52.570 DEBUG:boto:Data: 2022-09-02T05:57:52.571 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.573 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.573 DEBUG:boto:Params: {} 2022-09-02T05:57:52.574 DEBUG:boto:Token: None 2022-09-02T05:57:52.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj22 2022-09-02T05:57:52.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IpSTXMM9Myl1CG1wSS/3Ewr02rA= 2022-09-02T05:57:52.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IpSTXMM9Myl1CG1wSS/3Ewr02rA=', 'Content-Length': '0'} 2022-09-02T05:57:52.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ee71a94919ac3c-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.579 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.580 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.581 DEBUG:boto:Method: GET 2022-09-02T05:57:52.582 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.582 DEBUG:boto:Data: 2022-09-02T05:57:52.583 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.584 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.584 DEBUG:boto:Params: {} 2022-09-02T05:57:52.585 DEBUG:boto:Token: None 2022-09-02T05:57:52.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004791aa09ba1f5340-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.630 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.631 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.632 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.633 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.634 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.635 DEBUG:boto:Method: GET 2022-09-02T05:57:52.636 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:57:52.636 DEBUG:boto:Data: 2022-09-02T05:57:52.638 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.639 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.640 DEBUG:boto:Params: {} 2022-09-02T05:57:52.641 DEBUG:boto:Token: None 2022-09-02T05:57:52.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087526f41ccc503c7-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.649 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:57:52.650 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.651 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.651 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.652 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.653 DEBUG:boto:Method: GET 2022-09-02T05:57:52.653 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.654 DEBUG:boto:Data: 2022-09-02T05:57:52.654 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.655 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.655 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.656 DEBUG:boto:Params: {} 2022-09-02T05:57:52.656 DEBUG:boto:Token: None 2022-09-02T05:57:52.657 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc= 2022-09-02T05:57:52.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc=', 'Content-Length': '0'} 2022-09-02T05:57:52.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c6ed839f0e51f8-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.662 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.662 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.663 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.663 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.664 DEBUG:boto:Method: GET 2022-09-02T05:57:52.664 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.665 DEBUG:boto:Data: 2022-09-02T05:57:52.665 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.667 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.667 DEBUG:boto:Params: {} 2022-09-02T05:57:52.668 DEBUG:boto:Token: None 2022-09-02T05:57:52.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc= 2022-09-02T05:57:52.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc=', 'Content-Length': '0'} 2022-09-02T05:57:52.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008194996e7f7a35e6-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.714 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.715 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.716 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.717 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.718 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.719 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.720 DEBUG:boto:Data: 2022-09-02T05:57:52.721 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.723 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.724 DEBUG:boto:Params: {} 2022-09-02T05:57:52.725 DEBUG:boto:Token: None 2022-09-02T05:57:52.726 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hl+9yq/EeyPxMTJ9rg/YVsmkjjQ= 2022-09-02T05:57:52.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hl+9yq/EeyPxMTJ9rg/YVsmkjjQ=', 'Content-Length': '0'} 2022-09-02T05:57:52.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007795ac542874d0d5-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.732 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.733 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.734 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.735 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.736 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.737 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.738 DEBUG:boto:Data: 2022-09-02T05:57:52.739 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.741 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.742 DEBUG:boto:Params: {} 2022-09-02T05:57:52.743 DEBUG:boto:Token: None 2022-09-02T05:57:52.744 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.745 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hl+9yq/EeyPxMTJ9rg/YVsmkjjQ= 2022-09-02T05:57:52.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hl+9yq/EeyPxMTJ9rg/YVsmkjjQ=', 'Content-Length': '0'} 2022-09-02T05:57:52.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004290ae0e46e4c4eb-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:57:52.751 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.753 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.753 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.754 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.754 DEBUG:boto:Method: GET 2022-09-02T05:57:52.755 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.756 DEBUG:boto:Data: 2022-09-02T05:57:52.756 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.757 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.757 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.759 DEBUG:boto:Params: {} 2022-09-02T05:57:52.760 DEBUG:boto:Token: None 2022-09-02T05:57:52.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc= 2022-09-02T05:57:52.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc=', 'Content-Length': '0'} 2022-09-02T05:57:52.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa028bde726c6f2-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.767 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.767 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.768 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.768 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.769 DEBUG:boto:Method: GET 2022-09-02T05:57:52.769 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.770 DEBUG:boto:Data: 2022-09-02T05:57:52.771 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.772 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.772 DEBUG:boto:Params: {} 2022-09-02T05:57:52.773 DEBUG:boto:Token: None 2022-09-02T05:57:52.774 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:57:52.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc= 2022-09-02T05:57:52.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xT+8IguhHQOLRsxOz/JcH/EeXLc=', 'Content-Length': '0'} 2022-09-02T05:57:52.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005375da8deee93340-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.779 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.779 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.780 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.780 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.781 DEBUG:boto:Method: GET 2022-09-02T05:57:52.781 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.782 DEBUG:boto:Data: 2022-09-02T05:57:52.782 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.783 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.784 DEBUG:boto:Params: {} 2022-09-02T05:57:52.785 DEBUG:boto:Token: None 2022-09-02T05:57:52.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f266149ace111094-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.830 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.831 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.832 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.832 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.833 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.834 DEBUG:boto:Method: GET 2022-09-02T05:57:52.835 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.835 DEBUG:boto:Data: 2022-09-02T05:57:52.836 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.837 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.838 DEBUG:boto:Params: {} 2022-09-02T05:57:52.840 DEBUG:boto:Token: None 2022-09-02T05:57:52.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018271e4b7fcfbebf-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.848 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.848 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.849 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.850 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.850 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:57:52.851 DEBUG:boto:Data: 2022-09-02T05:57:52.851 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.852 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.853 DEBUG:boto:Params: {} 2022-09-02T05:57:52.854 DEBUG:boto:Token: None 2022-09-02T05:57:52.854 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23 2022-09-02T05:57:52.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NsjRajVpnYhXK4gY2M3FKUlzt5c= 2022-09-02T05:57:52.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NsjRajVpnYhXK4gY2M3FKUlzt5c=', 'Content-Length': '0'} 2022-09-02T05:57:52.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c172a48f955f290-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.859 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.859 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.860 DEBUG:boto:Method: HEAD 2022-09-02T05:57:52.860 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:57:52.861 DEBUG:boto:Data: 2022-09-02T05:57:52.861 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.862 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.863 DEBUG:boto:Params: {} 2022-09-02T05:57:52.864 DEBUG:boto:Token: None 2022-09-02T05:57:52.864 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23 2022-09-02T05:57:52.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NsjRajVpnYhXK4gY2M3FKUlzt5c= 2022-09-02T05:57:52.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NsjRajVpnYhXK4gY2M3FKUlzt5c=', 'Content-Length': '0'} 2022-09-02T05:57:52.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3064d02593e0394-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:57:52.872 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.873 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.874 DEBUG:boto:Method: GET 2022-09-02T05:57:52.875 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:57:52.876 DEBUG:boto:Data: 2022-09-02T05:57:52.877 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.879 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.879 DEBUG:boto:Params: {} 2022-09-02T05:57:52.880 DEBUG:boto:Token: None 2022-09-02T05:57:52.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23 2022-09-02T05:57:52.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qwc1d88O2GxJH0qBlr8GNmwbqfU= 2022-09-02T05:57:52.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qwc1d88O2GxJH0qBlr8GNmwbqfU=', 'Content-Length': '0'} 2022-09-02T05:57:52.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d119722ff6ccf8d8-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.886 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:57:52.887 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:57:52.887 DEBUG:boto:Method: GET 2022-09-02T05:57:52.888 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:57:52.889 DEBUG:boto:Data: 2022-09-02T05:57:52.889 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.891 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.891 DEBUG:boto:Params: {} 2022-09-02T05:57:52.892 DEBUG:boto:Token: None 2022-09-02T05:57:52.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj23 2022-09-02T05:57:52.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qwc1d88O2GxJH0qBlr8GNmwbqfU= 2022-09-02T05:57:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qwc1d88O2GxJH0qBlr8GNmwbqfU=', 'Content-Length': '0'} 2022-09-02T05:57:52.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b012545bc67a3863-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.897 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.898 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.898 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.899 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.899 DEBUG:boto:Method: GET 2022-09-02T05:57:52.900 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.900 DEBUG:boto:Data: 2022-09-02T05:57:52.901 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.902 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.903 DEBUG:boto:Params: {} 2022-09-02T05:57:52.903 DEBUG:boto:Token: None 2022-09-02T05:57:52.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198acd3c6925a4d6-0063119b60-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.909 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.910 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:52.911 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:52.912 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.912 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.913 DEBUG:boto:Method: GET 2022-09-02T05:57:52.913 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:57:52.914 DEBUG:boto:Data: 2022-09-02T05:57:52.915 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.916 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.916 DEBUG:boto:Params: {} 2022-09-02T05:57:52.917 DEBUG:boto:Token: None 2022-09-02T05:57:52.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/ 2022-09-02T05:57:52.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U= 2022-09-02T05:57:52.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BWJusiS12fMyOwoow2ItnYiEV2U=', 'Content-Length': '0'} 2022-09-02T05:57:52.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b72c3586923815b-0063119b60-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.926 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:52.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:57:52.929 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:52.930 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:52.930 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.931 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.932 DEBUG:boto:Method: GET 2022-09-02T05:57:52.933 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.935 DEBUG:boto:Data: 2022-09-02T05:57:52.936 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:52.938 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.939 DEBUG:boto:Params: {} 2022-09-02T05:57:52.940 DEBUG:boto:Token: None 2022-09-02T05:57:52.941 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gwz/3yHRAeAyT9QCch4UfwBcB5I= 2022-09-02T05:57:52.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gwz/3yHRAeAyT9QCch4UfwBcB5I=', 'Content-Length': '0'} 2022-09-02T05:57:52.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9f34fc8a22d308-0063119b60-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:52.947 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:52.948 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:52.949 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.950 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.951 DEBUG:boto:Method: GET 2022-09-02T05:57:52.951 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.952 DEBUG:boto:Data: 2022-09-02T05:57:52.952 DEBUG:boto:Headers: {} 2022-09-02T05:57:52.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:52.954 DEBUG:boto:Port: 8000 2022-09-02T05:57:52.954 DEBUG:boto:Params: {} 2022-09-02T05:57:52.955 DEBUG:boto:Token: None 2022-09-02T05:57:52.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:52 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:52.957 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gwz/3yHRAeAyT9QCch4UfwBcB5I= 2022-09-02T05:57:52.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gwz/3yHRAeAyT9QCch4UfwBcB5I=', 'Content-Length': '0'} 2022-09-02T05:57:52.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051d177cc56dba122-0063119b60-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.002 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.003 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.003 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.004 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.004 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.005 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.005 DEBUG:boto:Data: 2022-09-02T05:57:53.006 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.007 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.007 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.008 DEBUG:boto:Params: {} 2022-09-02T05:57:53.008 DEBUG:boto:Token: None 2022-09-02T05:57:53.009 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wd0XTeQ10PdRmlDh37W2o7PDwQs= 2022-09-02T05:57:53.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wd0XTeQ10PdRmlDh37W2o7PDwQs=', 'Content-Length': '0'} 2022-09-02T05:57:53.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002578f68a3b208b9f-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.013 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.014 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.014 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.015 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.015 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.016 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.016 DEBUG:boto:Data: 2022-09-02T05:57:53.017 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.018 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.018 DEBUG:boto:Params: {} 2022-09-02T05:57:53.019 DEBUG:boto:Token: None 2022-09-02T05:57:53.019 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.020 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wd0XTeQ10PdRmlDh37W2o7PDwQs= 2022-09-02T05:57:53.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wd0XTeQ10PdRmlDh37W2o7PDwQs=', 'Content-Length': '0'} 2022-09-02T05:57:53.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e26d5a3b68d1e65-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:57:53.024 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.025 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.025 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.026 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.026 DEBUG:boto:Method: GET 2022-09-02T05:57:53.027 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.027 DEBUG:boto:Data: 2022-09-02T05:57:53.028 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.028 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.029 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.029 DEBUG:boto:Params: {} 2022-09-02T05:57:53.030 DEBUG:boto:Token: None 2022-09-02T05:57:53.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6Ip4S39B7y1d6lyeq79uAMGVKNA= 2022-09-02T05:57:53.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6Ip4S39B7y1d6lyeq79uAMGVKNA=', 'Content-Length': '0'} 2022-09-02T05:57:53.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c766588a02e792e9-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.035 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.035 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.036 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.036 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.037 DEBUG:boto:Method: GET 2022-09-02T05:57:53.037 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.038 DEBUG:boto:Data: 2022-09-02T05:57:53.038 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.040 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.040 DEBUG:boto:Params: {} 2022-09-02T05:57:53.041 DEBUG:boto:Token: None 2022-09-02T05:57:53.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:57:53.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6Ip4S39B7y1d6lyeq79uAMGVKNA= 2022-09-02T05:57:53.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6Ip4S39B7y1d6lyeq79uAMGVKNA=', 'Content-Length': '0'} 2022-09-02T05:57:53.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ea1e438f29a016-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.046 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.046 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.047 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.047 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.048 DEBUG:boto:Method: GET 2022-09-02T05:57:53.048 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.049 DEBUG:boto:Data: 2022-09-02T05:57:53.049 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.050 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.050 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.051 DEBUG:boto:Params: {} 2022-09-02T05:57:53.052 DEBUG:boto:Token: None 2022-09-02T05:57:53.052 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2cce5cd08d518d9-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.057 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.058 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.058 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.059 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.059 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.060 DEBUG:boto:Method: GET 2022-09-02T05:57:53.060 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.061 DEBUG:boto:Data: 2022-09-02T05:57:53.062 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.063 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.063 DEBUG:boto:Params: {} 2022-09-02T05:57:53.064 DEBUG:boto:Token: None 2022-09-02T05:57:53.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0a25d2984af8e3-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.071 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.071 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.072 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.073 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.073 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:57:53.074 DEBUG:boto:Data: 2022-09-02T05:57:53.074 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.075 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.076 DEBUG:boto:Params: {} 2022-09-02T05:57:53.076 DEBUG:boto:Token: None 2022-09-02T05:57:53.077 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29 2022-09-02T05:57:53.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zkyXkDiuI0jqhETrPhUF2y+2/pE= 2022-09-02T05:57:53.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zkyXkDiuI0jqhETrPhUF2y+2/pE=', 'Content-Length': '0'} 2022-09-02T05:57:53.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c242f13868ccb9ef-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.082 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.082 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.083 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.083 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:57:53.084 DEBUG:boto:Data: 2022-09-02T05:57:53.084 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.085 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.086 DEBUG:boto:Params: {} 2022-09-02T05:57:53.086 DEBUG:boto:Token: None 2022-09-02T05:57:53.087 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29 2022-09-02T05:57:53.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zkyXkDiuI0jqhETrPhUF2y+2/pE= 2022-09-02T05:57:53.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zkyXkDiuI0jqhETrPhUF2y+2/pE=', 'Content-Length': '0'} 2022-09-02T05:57:53.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca95214adf2860f7-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:57:53.092 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.092 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.093 DEBUG:boto:Method: GET 2022-09-02T05:57:53.094 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:57:53.095 DEBUG:boto:Data: 2022-09-02T05:57:53.095 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.096 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.097 DEBUG:boto:Params: {} 2022-09-02T05:57:53.098 DEBUG:boto:Token: None 2022-09-02T05:57:53.099 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29 2022-09-02T05:57:53.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1XSvlF3VvtBr/tc6ofGRIf4ck1o= 2022-09-02T05:57:53.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1XSvlF3VvtBr/tc6ofGRIf4ck1o=', 'Content-Length': '0'} 2022-09-02T05:57:53.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21ac5374eab7e7c-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.105 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:57:53.105 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:57:53.106 DEBUG:boto:Method: GET 2022-09-02T05:57:53.106 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:57:53.107 DEBUG:boto:Data: 2022-09-02T05:57:53.107 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.108 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.109 DEBUG:boto:Params: {} 2022-09-02T05:57:53.109 DEBUG:boto:Token: None 2022-09-02T05:57:53.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj29 2022-09-02T05:57:53.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1XSvlF3VvtBr/tc6ofGRIf4ck1o= 2022-09-02T05:57:53.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1XSvlF3VvtBr/tc6ofGRIf4ck1o=', 'Content-Length': '0'} 2022-09-02T05:57:53.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130d275d3cf28eb8-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.154 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.155 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.155 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.156 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.156 DEBUG:boto:Method: GET 2022-09-02T05:57:53.157 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.157 DEBUG:boto:Data: 2022-09-02T05:57:53.158 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.159 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.159 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.160 DEBUG:boto:Params: {} 2022-09-02T05:57:53.160 DEBUG:boto:Token: None 2022-09-02T05:57:53.161 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b949f039e54fea03-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.165 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.166 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.167 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.167 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.168 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.169 DEBUG:boto:Method: GET 2022-09-02T05:57:53.169 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:57:53.170 DEBUG:boto:Data: 2022-09-02T05:57:53.170 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.172 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.173 DEBUG:boto:Params: {} 2022-09-02T05:57:53.173 DEBUG:boto:Token: None 2022-09-02T05:57:53.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bce9c2993a299bd6-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.180 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:57:53.182 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.182 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.183 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.184 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.184 DEBUG:boto:Method: GET 2022-09-02T05:57:53.185 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.185 DEBUG:boto:Data: 2022-09-02T05:57:53.186 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.187 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.188 DEBUG:boto:Params: {} 2022-09-02T05:57:53.188 DEBUG:boto:Token: None 2022-09-02T05:57:53.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8= 2022-09-02T05:57:53.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8=', 'Content-Length': '0'} 2022-09-02T05:57:53.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0c358d3af0fcf3-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.194 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.194 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.195 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.196 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.196 DEBUG:boto:Method: GET 2022-09-02T05:57:53.197 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.197 DEBUG:boto:Data: 2022-09-02T05:57:53.198 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.199 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.199 DEBUG:boto:Params: {} 2022-09-02T05:57:53.200 DEBUG:boto:Token: None 2022-09-02T05:57:53.201 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8= 2022-09-02T05:57:53.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8=', 'Content-Length': '0'} 2022-09-02T05:57:53.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6d0f068e36e5420-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.205 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.206 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.206 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.207 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.207 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.208 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.208 DEBUG:boto:Data: 2022-09-02T05:57:53.209 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.209 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.210 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.210 DEBUG:boto:Params: {} 2022-09-02T05:57:53.211 DEBUG:boto:Token: None 2022-09-02T05:57:53.211 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FS7+wepjn0K78fuG4FsJB+FmQoE= 2022-09-02T05:57:53.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FS7+wepjn0K78fuG4FsJB+FmQoE=', 'Content-Length': '0'} 2022-09-02T05:57:53.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aac402ddaa2744e-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.216 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.216 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.217 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.218 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.219 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.219 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.220 DEBUG:boto:Data: 2022-09-02T05:57:53.220 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.221 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.222 DEBUG:boto:Params: {} 2022-09-02T05:57:53.222 DEBUG:boto:Token: None 2022-09-02T05:57:53.223 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FS7+wepjn0K78fuG4FsJB+FmQoE= 2022-09-02T05:57:53.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FS7+wepjn0K78fuG4FsJB+FmQoE=', 'Content-Length': '0'} 2022-09-02T05:57:53.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016dff402b69a6afb-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:57:53.227 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.228 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.228 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.229 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.230 DEBUG:boto:Method: GET 2022-09-02T05:57:53.230 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.231 DEBUG:boto:Data: 2022-09-02T05:57:53.231 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.232 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.232 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.233 DEBUG:boto:Params: {} 2022-09-02T05:57:53.233 DEBUG:boto:Token: None 2022-09-02T05:57:53.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.235 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8= 2022-09-02T05:57:53.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8=', 'Content-Length': '0'} 2022-09-02T05:57:53.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332a16a7d3c33c57-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.278 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.279 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.280 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.281 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.282 DEBUG:boto:Method: GET 2022-09-02T05:57:53.283 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.284 DEBUG:boto:Data: 2022-09-02T05:57:53.285 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.287 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.287 DEBUG:boto:Params: {} 2022-09-02T05:57:53.288 DEBUG:boto:Token: None 2022-09-02T05:57:53.288 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:57:53.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8= 2022-09-02T05:57:53.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZiCgXzxomuHAhC+0AErhy58kGN8=', 'Content-Length': '0'} 2022-09-02T05:57:53.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165eee500b642bb8-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.293 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.293 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.294 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.295 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.296 DEBUG:boto:Method: GET 2022-09-02T05:57:53.296 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.297 DEBUG:boto:Data: 2022-09-02T05:57:53.297 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.298 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.299 DEBUG:boto:Params: {} 2022-09-02T05:57:53.299 DEBUG:boto:Token: None 2022-09-02T05:57:53.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009038cfab42b3298d-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.305 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.306 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.306 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.307 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.308 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.308 DEBUG:boto:Method: GET 2022-09-02T05:57:53.309 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.310 DEBUG:boto:Data: 2022-09-02T05:57:53.310 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.311 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.312 DEBUG:boto:Params: {} 2022-09-02T05:57:53.313 DEBUG:boto:Token: None 2022-09-02T05:57:53.314 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d9787bad87b5bd-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.320 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.321 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.321 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.322 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.323 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:57:53.323 DEBUG:boto:Data: 2022-09-02T05:57:53.324 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.325 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.325 DEBUG:boto:Params: {} 2022-09-02T05:57:53.326 DEBUG:boto:Token: None 2022-09-02T05:57:53.327 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31 2022-09-02T05:57:53.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CRDHSwbSqvt+dXW6ybm+lNmVEl8= 2022-09-02T05:57:53.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CRDHSwbSqvt+dXW6ybm+lNmVEl8=', 'Content-Length': '0'} 2022-09-02T05:57:53.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce2f597d547e7ac2-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.331 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.332 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.332 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.333 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:57:53.333 DEBUG:boto:Data: 2022-09-02T05:57:53.334 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.335 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.335 DEBUG:boto:Params: {} 2022-09-02T05:57:53.336 DEBUG:boto:Token: None 2022-09-02T05:57:53.336 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31 2022-09-02T05:57:53.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CRDHSwbSqvt+dXW6ybm+lNmVEl8= 2022-09-02T05:57:53.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CRDHSwbSqvt+dXW6ybm+lNmVEl8=', 'Content-Length': '0'} 2022-09-02T05:57:53.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca99339ef5c91ff-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:57:53.342 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.342 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.343 DEBUG:boto:Method: GET 2022-09-02T05:57:53.343 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:57:53.344 DEBUG:boto:Data: 2022-09-02T05:57:53.344 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.345 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.346 DEBUG:boto:Params: {} 2022-09-02T05:57:53.347 DEBUG:boto:Token: None 2022-09-02T05:57:53.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31 2022-09-02T05:57:53.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAVzEJ5FCOeKE62Q1qbuyi4oGwc= 2022-09-02T05:57:53.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAVzEJ5FCOeKE62Q1qbuyi4oGwc=', 'Content-Length': '0'} 2022-09-02T05:57:53.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027a99bf841d7a9aa-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.351 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:57:53.352 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:57:53.352 DEBUG:boto:Method: GET 2022-09-02T05:57:53.353 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:57:53.354 DEBUG:boto:Data: 2022-09-02T05:57:53.354 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.355 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.356 DEBUG:boto:Params: {} 2022-09-02T05:57:53.356 DEBUG:boto:Token: None 2022-09-02T05:57:53.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj31 2022-09-02T05:57:53.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bAVzEJ5FCOeKE62Q1qbuyi4oGwc= 2022-09-02T05:57:53.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bAVzEJ5FCOeKE62Q1qbuyi4oGwc=', 'Content-Length': '0'} 2022-09-02T05:57:53.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135fb4c3932aa088-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.402 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.403 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.404 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.405 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.406 DEBUG:boto:Method: GET 2022-09-02T05:57:53.407 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.407 DEBUG:boto:Data: 2022-09-02T05:57:53.408 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.409 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.409 DEBUG:boto:Params: {} 2022-09-02T05:57:53.410 DEBUG:boto:Token: None 2022-09-02T05:57:53.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9fcebe56fd610ed-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.415 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.416 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.417 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.417 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.418 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.418 DEBUG:boto:Method: GET 2022-09-02T05:57:53.419 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:57:53.419 DEBUG:boto:Data: 2022-09-02T05:57:53.420 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.421 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.422 DEBUG:boto:Params: {} 2022-09-02T05:57:53.422 DEBUG:boto:Token: None 2022-09-02T05:57:53.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bce4018f46dad2c-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.429 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:57:53.431 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.431 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.432 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.432 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.433 DEBUG:boto:Method: GET 2022-09-02T05:57:53.433 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.434 DEBUG:boto:Data: 2022-09-02T05:57:53.434 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.435 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.436 DEBUG:boto:Params: {} 2022-09-02T05:57:53.436 DEBUG:boto:Token: None 2022-09-02T05:57:53.437 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk= 2022-09-02T05:57:53.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk=', 'Content-Length': '0'} 2022-09-02T05:57:53.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050218b3253ee6ce3-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.441 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.442 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.442 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.443 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.443 DEBUG:boto:Method: GET 2022-09-02T05:57:53.444 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.444 DEBUG:boto:Data: 2022-09-02T05:57:53.445 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.446 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.446 DEBUG:boto:Params: {} 2022-09-02T05:57:53.447 DEBUG:boto:Token: None 2022-09-02T05:57:53.448 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk= 2022-09-02T05:57:53.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk=', 'Content-Length': '0'} 2022-09-02T05:57:53.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f710bdd27bfd65e8-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.452 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.452 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.453 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.453 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.454 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.454 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.455 DEBUG:boto:Data: 2022-09-02T05:57:53.455 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.456 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.457 DEBUG:boto:Params: {} 2022-09-02T05:57:53.457 DEBUG:boto:Token: None 2022-09-02T05:57:53.458 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.458 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WXDebYKoWfu0DlPZDBP9MPCus1M= 2022-09-02T05:57:53.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WXDebYKoWfu0DlPZDBP9MPCus1M=', 'Content-Length': '0'} 2022-09-02T05:57:53.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000633a9c0eacce35b7-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.462 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.462 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.463 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.463 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.464 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.464 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.465 DEBUG:boto:Data: 2022-09-02T05:57:53.466 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.467 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.467 DEBUG:boto:Params: {} 2022-09-02T05:57:53.468 DEBUG:boto:Token: None 2022-09-02T05:57:53.468 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WXDebYKoWfu0DlPZDBP9MPCus1M= 2022-09-02T05:57:53.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WXDebYKoWfu0DlPZDBP9MPCus1M=', 'Content-Length': '0'} 2022-09-02T05:57:53.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeb5a0f8f1e3b9f7-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:57:53.473 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.474 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.474 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.475 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.476 DEBUG:boto:Method: GET 2022-09-02T05:57:53.476 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.477 DEBUG:boto:Data: 2022-09-02T05:57:53.477 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.478 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.479 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.479 DEBUG:boto:Params: {} 2022-09-02T05:57:53.480 DEBUG:boto:Token: None 2022-09-02T05:57:53.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk= 2022-09-02T05:57:53.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk=', 'Content-Length': '0'} 2022-09-02T05:57:53.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde8bf323f219035-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.486 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.488 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.488 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.489 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.490 DEBUG:boto:Method: GET 2022-09-02T05:57:53.491 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.492 DEBUG:boto:Data: 2022-09-02T05:57:53.493 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.495 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.496 DEBUG:boto:Params: {} 2022-09-02T05:57:53.497 DEBUG:boto:Token: None 2022-09-02T05:57:53.498 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:57:53.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk= 2022-09-02T05:57:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:85+WyrzStgss/CI/+TbsE3Hw2uk=', 'Content-Length': '0'} 2022-09-02T05:57:53.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edc4441529963317-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.546 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.547 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.548 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.548 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.549 DEBUG:boto:Method: GET 2022-09-02T05:57:53.549 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.550 DEBUG:boto:Data: 2022-09-02T05:57:53.550 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.551 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.552 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.552 DEBUG:boto:Params: {} 2022-09-02T05:57:53.553 DEBUG:boto:Token: None 2022-09-02T05:57:53.554 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074210220f5a33fe6-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.559 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.560 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.560 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.561 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.561 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.562 DEBUG:boto:Method: GET 2022-09-02T05:57:53.563 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.564 DEBUG:boto:Data: 2022-09-02T05:57:53.564 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.565 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.566 DEBUG:boto:Params: {} 2022-09-02T05:57:53.566 DEBUG:boto:Token: None 2022-09-02T05:57:53.567 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a21a445bcc954869-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.574 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.575 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.576 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.576 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.577 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:57:53.577 DEBUG:boto:Data: 2022-09-02T05:57:53.578 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.579 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.579 DEBUG:boto:Params: {} 2022-09-02T05:57:53.580 DEBUG:boto:Token: None 2022-09-02T05:57:53.581 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33 2022-09-02T05:57:53.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:plOkIcYhlNhikHgQEXlRKIzQuX8= 2022-09-02T05:57:53.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:plOkIcYhlNhikHgQEXlRKIzQuX8=', 'Content-Length': '0'} 2022-09-02T05:57:53.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9443231f370537-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.585 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.585 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.586 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.587 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:57:53.587 DEBUG:boto:Data: 2022-09-02T05:57:53.587 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.588 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.589 DEBUG:boto:Params: {} 2022-09-02T05:57:53.590 DEBUG:boto:Token: None 2022-09-02T05:57:53.590 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33 2022-09-02T05:57:53.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:plOkIcYhlNhikHgQEXlRKIzQuX8= 2022-09-02T05:57:53.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:plOkIcYhlNhikHgQEXlRKIzQuX8=', 'Content-Length': '0'} 2022-09-02T05:57:53.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d4d0ab844c39daa-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:57:53.595 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.595 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.596 DEBUG:boto:Method: GET 2022-09-02T05:57:53.597 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:57:53.597 DEBUG:boto:Data: 2022-09-02T05:57:53.598 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.599 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.599 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.600 DEBUG:boto:Params: {} 2022-09-02T05:57:53.600 DEBUG:boto:Token: None 2022-09-02T05:57:53.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33 2022-09-02T05:57:53.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4GAvqWXTAPDhfJb0LvQLKaw72yY= 2022-09-02T05:57:53.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4GAvqWXTAPDhfJb0LvQLKaw72yY=', 'Content-Length': '0'} 2022-09-02T05:57:53.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056949f4614783ba3-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.646 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:57:53.647 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:57:53.647 DEBUG:boto:Method: GET 2022-09-02T05:57:53.648 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:57:53.649 DEBUG:boto:Data: 2022-09-02T05:57:53.649 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.650 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.651 DEBUG:boto:Params: {} 2022-09-02T05:57:53.652 DEBUG:boto:Token: None 2022-09-02T05:57:53.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj33 2022-09-02T05:57:53.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4GAvqWXTAPDhfJb0LvQLKaw72yY= 2022-09-02T05:57:53.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4GAvqWXTAPDhfJb0LvQLKaw72yY=', 'Content-Length': '0'} 2022-09-02T05:57:53.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005858f8fd1122d159-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.658 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.658 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.659 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.659 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.660 DEBUG:boto:Method: GET 2022-09-02T05:57:53.660 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.661 DEBUG:boto:Data: 2022-09-02T05:57:53.661 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.662 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.662 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.663 DEBUG:boto:Params: {} 2022-09-02T05:57:53.664 DEBUG:boto:Token: None 2022-09-02T05:57:53.664 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2c771271c99167-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.669 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.670 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.671 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.672 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.673 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.674 DEBUG:boto:Method: GET 2022-09-02T05:57:53.675 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:57:53.675 DEBUG:boto:Data: 2022-09-02T05:57:53.676 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.678 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.679 DEBUG:boto:Params: {} 2022-09-02T05:57:53.680 DEBUG:boto:Token: None 2022-09-02T05:57:53.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d5833d67e060ed7-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.689 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:57:53.691 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.691 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.692 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.692 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.693 DEBUG:boto:Method: GET 2022-09-02T05:57:53.694 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.694 DEBUG:boto:Data: 2022-09-02T05:57:53.695 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.695 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.696 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.696 DEBUG:boto:Params: {} 2022-09-02T05:57:53.697 DEBUG:boto:Token: None 2022-09-02T05:57:53.698 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs= 2022-09-02T05:57:53.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs=', 'Content-Length': '0'} 2022-09-02T05:57:53.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791ee87553e516f2-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.702 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.703 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.703 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.704 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.704 DEBUG:boto:Method: GET 2022-09-02T05:57:53.705 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.705 DEBUG:boto:Data: 2022-09-02T05:57:53.706 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.707 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.708 DEBUG:boto:Params: {} 2022-09-02T05:57:53.708 DEBUG:boto:Token: None 2022-09-02T05:57:53.709 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs= 2022-09-02T05:57:53.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs=', 'Content-Length': '0'} 2022-09-02T05:57:53.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40e959182c47d08-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.754 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.768 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.768 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.769 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.769 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.770 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.770 DEBUG:boto:Data: 2022-09-02T05:57:53.771 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.771 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.772 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.773 DEBUG:boto:Params: {} 2022-09-02T05:57:53.773 DEBUG:boto:Token: None 2022-09-02T05:57:53.774 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YqKOVw+C2nKndySqdJ82OkHGfaM= 2022-09-02T05:57:53.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YqKOVw+C2nKndySqdJ82OkHGfaM=', 'Content-Length': '0'} 2022-09-02T05:57:53.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663143453fab7202-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.778 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.779 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.779 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.780 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.780 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.781 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.781 DEBUG:boto:Data: 2022-09-02T05:57:53.782 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.783 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.783 DEBUG:boto:Params: {} 2022-09-02T05:57:53.784 DEBUG:boto:Token: None 2022-09-02T05:57:53.784 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YqKOVw+C2nKndySqdJ82OkHGfaM= 2022-09-02T05:57:53.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YqKOVw+C2nKndySqdJ82OkHGfaM=', 'Content-Length': '0'} 2022-09-02T05:57:53.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e50fa34fb71aa092-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:57:53.789 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.790 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.790 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.791 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.791 DEBUG:boto:Method: GET 2022-09-02T05:57:53.792 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.792 DEBUG:boto:Data: 2022-09-02T05:57:53.793 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.794 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.795 DEBUG:boto:Params: {} 2022-09-02T05:57:53.796 DEBUG:boto:Token: None 2022-09-02T05:57:53.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs= 2022-09-02T05:57:53.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs=', 'Content-Length': '0'} 2022-09-02T05:57:53.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001730021ac4f4c63d-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.801 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.801 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.802 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.802 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.803 DEBUG:boto:Method: GET 2022-09-02T05:57:53.803 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.804 DEBUG:boto:Data: 2022-09-02T05:57:53.805 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.806 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.806 DEBUG:boto:Params: {} 2022-09-02T05:57:53.807 DEBUG:boto:Token: None 2022-09-02T05:57:53.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:57:53.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs= 2022-09-02T05:57:53.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EraswQNmyawTv0VXmQo7ypJLUDs=', 'Content-Length': '0'} 2022-09-02T05:57:53.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686b92503af8ee79-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.814 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.814 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.816 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.817 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.817 DEBUG:boto:Method: GET 2022-09-02T05:57:53.818 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.818 DEBUG:boto:Data: 2022-09-02T05:57:53.819 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.820 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.820 DEBUG:boto:Params: {} 2022-09-02T05:57:53.821 DEBUG:boto:Token: None 2022-09-02T05:57:53.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc169913eee049e6-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.827 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.828 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.828 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.829 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.829 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.830 DEBUG:boto:Method: GET 2022-09-02T05:57:53.830 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.831 DEBUG:boto:Data: 2022-09-02T05:57:53.831 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.832 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.833 DEBUG:boto:Params: {} 2022-09-02T05:57:53.834 DEBUG:boto:Token: None 2022-09-02T05:57:53.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd4743c12e4869c6-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.842 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.843 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.844 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.844 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.845 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:57:53.846 DEBUG:boto:Data: 2022-09-02T05:57:53.847 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.849 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.850 DEBUG:boto:Params: {} 2022-09-02T05:57:53.851 DEBUG:boto:Token: None 2022-09-02T05:57:53.852 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34 2022-09-02T05:57:53.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H7P5NOQu+D6nBJCZTooipuYgCpw= 2022-09-02T05:57:53.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H7P5NOQu+D6nBJCZTooipuYgCpw=', 'Content-Length': '0'} 2022-09-02T05:57:53.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f60dca6a320c18-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.857 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.858 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.859 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.859 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:57:53.860 DEBUG:boto:Data: 2022-09-02T05:57:53.860 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.861 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.862 DEBUG:boto:Params: {} 2022-09-02T05:57:53.863 DEBUG:boto:Token: None 2022-09-02T05:57:53.863 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34 2022-09-02T05:57:53.864 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H7P5NOQu+D6nBJCZTooipuYgCpw= 2022-09-02T05:57:53.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H7P5NOQu+D6nBJCZTooipuYgCpw=', 'Content-Length': '0'} 2022-09-02T05:57:53.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e09be346b27a233-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:57:53.868 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.869 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.869 DEBUG:boto:Method: GET 2022-09-02T05:57:53.870 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:57:53.871 DEBUG:boto:Data: 2022-09-02T05:57:53.871 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.873 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.873 DEBUG:boto:Params: {} 2022-09-02T05:57:53.874 DEBUG:boto:Token: None 2022-09-02T05:57:53.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34 2022-09-02T05:57:53.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hzj2+/fNicWw65DoLXQ4rH0VnGw= 2022-09-02T05:57:53.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hzj2+/fNicWw65DoLXQ4rH0VnGw=', 'Content-Length': '0'} 2022-09-02T05:57:53.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b67c4a805a33b82-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.879 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:57:53.879 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:57:53.880 DEBUG:boto:Method: GET 2022-09-02T05:57:53.881 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:57:53.881 DEBUG:boto:Data: 2022-09-02T05:57:53.882 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.883 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.884 DEBUG:boto:Params: {} 2022-09-02T05:57:53.884 DEBUG:boto:Token: None 2022-09-02T05:57:53.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj34 2022-09-02T05:57:53.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hzj2+/fNicWw65DoLXQ4rH0VnGw= 2022-09-02T05:57:53.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hzj2+/fNicWw65DoLXQ4rH0VnGw=', 'Content-Length': '0'} 2022-09-02T05:57:53.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e163c841366b2e8-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.930 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.931 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.931 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.932 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.932 DEBUG:boto:Method: GET 2022-09-02T05:57:53.933 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.934 DEBUG:boto:Data: 2022-09-02T05:57:53.934 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.935 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.936 DEBUG:boto:Params: {} 2022-09-02T05:57:53.937 DEBUG:boto:Token: None 2022-09-02T05:57:53.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009691b51595cabeb3-0063119b61-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.942 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.943 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:53.944 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:53.944 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.945 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.946 DEBUG:boto:Method: GET 2022-09-02T05:57:53.946 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:57:53.947 DEBUG:boto:Data: 2022-09-02T05:57:53.947 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.948 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.949 DEBUG:boto:Params: {} 2022-09-02T05:57:53.949 DEBUG:boto:Token: None 2022-09-02T05:57:53.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/ 2022-09-02T05:57:53.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw= 2022-09-02T05:57:53.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kunA56ODbcTSQrtloMeFnJ+CGbw=', 'Content-Length': '0'} 2022-09-02T05:57:53.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9f7477c6b12bf24-0063119b61-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.958 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:53.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:57:53.960 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:53.961 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:53.962 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.963 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.964 DEBUG:boto:Method: GET 2022-09-02T05:57:53.965 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.966 DEBUG:boto:Data: 2022-09-02T05:57:53.967 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.967 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.968 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.969 DEBUG:boto:Params: {} 2022-09-02T05:57:53.970 DEBUG:boto:Token: None 2022-09-02T05:57:53.971 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R3jqPPyp43cAKVUbEpAeA8KldRc= 2022-09-02T05:57:53.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R3jqPPyp43cAKVUbEpAeA8KldRc=', 'Content-Length': '0'} 2022-09-02T05:57:53.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d61ba9ce0be97c2-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.975 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:53.976 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:53.977 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.977 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.978 DEBUG:boto:Method: GET 2022-09-02T05:57:53.978 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.979 DEBUG:boto:Data: 2022-09-02T05:57:53.979 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:53.980 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.981 DEBUG:boto:Params: {} 2022-09-02T05:57:53.981 DEBUG:boto:Token: None 2022-09-02T05:57:53.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R3jqPPyp43cAKVUbEpAeA8KldRc= 2022-09-02T05:57:53.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R3jqPPyp43cAKVUbEpAeA8KldRc=', 'Content-Length': '0'} 2022-09-02T05:57:53.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc9d629d0f1d17b-0063119b61-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.986 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:53.987 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:53.987 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.988 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.988 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.989 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.989 DEBUG:boto:Data: 2022-09-02T05:57:53.990 DEBUG:boto:Headers: {} 2022-09-02T05:57:53.990 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:53.991 DEBUG:boto:Port: 8000 2022-09-02T05:57:53.991 DEBUG:boto:Params: {} 2022-09-02T05:57:53.992 DEBUG:boto:Token: None 2022-09-02T05:57:53.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:53 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qPIVlXa3fqWrkz9GqpMh1OVLMhc= 2022-09-02T05:57:53.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qPIVlXa3fqWrkz9GqpMh1OVLMhc=', 'Content-Length': '0'} 2022-09-02T05:57:53.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72544b982f499c3-0063119b61-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:53.997 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:53.997 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:53.998 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.998 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:53.999 DEBUG:boto:Method: HEAD 2022-09-02T05:57:53.999 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.000 DEBUG:boto:Data: 2022-09-02T05:57:54.000 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.001 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.002 DEBUG:boto:Params: {} 2022-09-02T05:57:54.003 DEBUG:boto:Token: None 2022-09-02T05:57:54.003 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ljhWI8yqEREIU3MXNZXW0jMCc3A= 2022-09-02T05:57:54.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ljhWI8yqEREIU3MXNZXW0jMCc3A=', 'Content-Length': '0'} 2022-09-02T05:57:54.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b60c376ce3af61-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:57:54.007 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:54.008 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:54.009 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.009 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.010 DEBUG:boto:Method: GET 2022-09-02T05:57:54.011 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.011 DEBUG:boto:Data: 2022-09-02T05:57:54.012 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.013 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.014 DEBUG:boto:Params: {} 2022-09-02T05:57:54.015 DEBUG:boto:Token: None 2022-09-02T05:57:54.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:miBIcVKsZM9CMwq1kBrQ1u4/hw8= 2022-09-02T05:57:54.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:miBIcVKsZM9CMwq1kBrQ1u4/hw8=', 'Content-Length': '0'} 2022-09-02T05:57:54.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d837e6671b12bd0b-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.062 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:54.063 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:54.064 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.065 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.065 DEBUG:boto:Method: GET 2022-09-02T05:57:54.066 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.067 DEBUG:boto:Data: 2022-09-02T05:57:54.069 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.071 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.073 DEBUG:boto:Params: {} 2022-09-02T05:57:54.074 DEBUG:boto:Token: None 2022-09-02T05:57:54.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:57:54.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:miBIcVKsZM9CMwq1kBrQ1u4/hw8= 2022-09-02T05:57:54.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:miBIcVKsZM9CMwq1kBrQ1u4/hw8=', 'Content-Length': '0'} 2022-09-02T05:57:54.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f7fca521f934dd-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.081 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.082 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.082 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.083 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.084 DEBUG:boto:Method: GET 2022-09-02T05:57:54.084 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.085 DEBUG:boto:Data: 2022-09-02T05:57:54.086 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.089 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.090 DEBUG:boto:Params: {} 2022-09-02T05:57:54.091 DEBUG:boto:Token: None 2022-09-02T05:57:54.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d1073cf1220dac-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.098 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.099 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.100 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.101 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.101 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.102 DEBUG:boto:Method: GET 2022-09-02T05:57:54.103 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.104 DEBUG:boto:Data: 2022-09-02T05:57:54.105 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.106 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.107 DEBUG:boto:Params: {} 2022-09-02T05:57:54.108 DEBUG:boto:Token: None 2022-09-02T05:57:54.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1876d9afe3eb2ae-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.119 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.120 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:54.121 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:54.121 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.122 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:57:54.123 DEBUG:boto:Data: 2022-09-02T05:57:54.124 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.125 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.126 DEBUG:boto:Params: {} 2022-09-02T05:57:54.127 DEBUG:boto:Token: None 2022-09-02T05:57:54.127 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38 2022-09-02T05:57:54.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WImGu+4otEp9axgHouNnBHRZGK0= 2022-09-02T05:57:54.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WImGu+4otEp9axgHouNnBHRZGK0=', 'Content-Length': '0'} 2022-09-02T05:57:54.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f11f5348bc2242-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.132 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:54.133 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:54.133 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.134 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:57:54.135 DEBUG:boto:Data: 2022-09-02T05:57:54.135 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.136 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.137 DEBUG:boto:Params: {} 2022-09-02T05:57:54.138 DEBUG:boto:Token: None 2022-09-02T05:57:54.138 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38 2022-09-02T05:57:54.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WImGu+4otEp9axgHouNnBHRZGK0= 2022-09-02T05:57:54.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WImGu+4otEp9axgHouNnBHRZGK0=', 'Content-Length': '0'} 2022-09-02T05:57:54.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f34753abd50f05-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:57:54.144 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:54.144 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:54.145 DEBUG:boto:Method: GET 2022-09-02T05:57:54.145 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:57:54.146 DEBUG:boto:Data: 2022-09-02T05:57:54.146 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.147 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.147 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.148 DEBUG:boto:Params: {} 2022-09-02T05:57:54.149 DEBUG:boto:Token: None 2022-09-02T05:57:54.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38 2022-09-02T05:57:54.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WFMoRKfilYn/DU2NROtqzRJjuQA= 2022-09-02T05:57:54.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WFMoRKfilYn/DU2NROtqzRJjuQA=', 'Content-Length': '0'} 2022-09-02T05:57:54.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a448d971c456a502-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.153 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:57:54.154 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:57:54.155 DEBUG:boto:Method: GET 2022-09-02T05:57:54.155 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:57:54.156 DEBUG:boto:Data: 2022-09-02T05:57:54.156 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.157 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.158 DEBUG:boto:Params: {} 2022-09-02T05:57:54.158 DEBUG:boto:Token: None 2022-09-02T05:57:54.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj38 2022-09-02T05:57:54.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WFMoRKfilYn/DU2NROtqzRJjuQA= 2022-09-02T05:57:54.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WFMoRKfilYn/DU2NROtqzRJjuQA=', 'Content-Length': '0'} 2022-09-02T05:57:54.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007250a9efd248ace0-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.206 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.207 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.207 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.208 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.208 DEBUG:boto:Method: GET 2022-09-02T05:57:54.209 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.209 DEBUG:boto:Data: 2022-09-02T05:57:54.210 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.211 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.212 DEBUG:boto:Params: {} 2022-09-02T05:57:54.212 DEBUG:boto:Token: None 2022-09-02T05:57:54.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e56d4ee5c0fbfc4e-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.218 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.219 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.219 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.220 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.221 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.221 DEBUG:boto:Method: GET 2022-09-02T05:57:54.222 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:57:54.222 DEBUG:boto:Data: 2022-09-02T05:57:54.223 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.224 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.225 DEBUG:boto:Params: {} 2022-09-02T05:57:54.226 DEBUG:boto:Token: None 2022-09-02T05:57:54.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7615d4354e24fe5-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.232 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:57:54.234 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.234 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.235 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.236 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.237 DEBUG:boto:Method: GET 2022-09-02T05:57:54.238 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.239 DEBUG:boto:Data: 2022-09-02T05:57:54.240 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.242 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.243 DEBUG:boto:Params: {} 2022-09-02T05:57:54.244 DEBUG:boto:Token: None 2022-09-02T05:57:54.245 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug= 2022-09-02T05:57:54.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug=', 'Content-Length': '0'} 2022-09-02T05:57:54.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070a0791f2c5f9bba-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.251 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.252 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.253 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.254 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.255 DEBUG:boto:Method: GET 2022-09-02T05:57:54.256 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.257 DEBUG:boto:Data: 2022-09-02T05:57:54.258 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.260 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.261 DEBUG:boto:Params: {} 2022-09-02T05:57:54.262 DEBUG:boto:Token: None 2022-09-02T05:57:54.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug= 2022-09-02T05:57:54.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug=', 'Content-Length': '0'} 2022-09-02T05:57:54.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a292e8792b6fd1ec-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.269 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.269 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.270 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.270 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.271 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.271 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.272 DEBUG:boto:Data: 2022-09-02T05:57:54.272 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.273 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.274 DEBUG:boto:Params: {} 2022-09-02T05:57:54.274 DEBUG:boto:Token: None 2022-09-02T05:57:54.275 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DxsyJ2+Ic511af+xQXUO0DK+o+Y= 2022-09-02T05:57:54.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DxsyJ2+Ic511af+xQXUO0DK+o+Y=', 'Content-Length': '0'} 2022-09-02T05:57:54.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d639a8ed6e33050-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.279 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.280 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.280 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.281 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.281 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.282 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.282 DEBUG:boto:Data: 2022-09-02T05:57:54.283 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.284 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.284 DEBUG:boto:Params: {} 2022-09-02T05:57:54.285 DEBUG:boto:Token: None 2022-09-02T05:57:54.286 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DxsyJ2+Ic511af+xQXUO0DK+o+Y= 2022-09-02T05:57:54.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DxsyJ2+Ic511af+xQXUO0DK+o+Y=', 'Content-Length': '0'} 2022-09-02T05:57:54.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdaf30951a83ad89-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:57:54.291 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.293 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.294 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.295 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.296 DEBUG:boto:Method: GET 2022-09-02T05:57:54.297 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.298 DEBUG:boto:Data: 2022-09-02T05:57:54.299 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.300 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.301 DEBUG:boto:Params: {} 2022-09-02T05:57:54.301 DEBUG:boto:Token: None 2022-09-02T05:57:54.302 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.303 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug= 2022-09-02T05:57:54.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug=', 'Content-Length': '0'} 2022-09-02T05:57:54.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3f7f989ea7f3e4-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.350 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.351 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.351 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.352 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.352 DEBUG:boto:Method: GET 2022-09-02T05:57:54.353 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.353 DEBUG:boto:Data: 2022-09-02T05:57:54.354 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.355 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.356 DEBUG:boto:Params: {} 2022-09-02T05:57:54.356 DEBUG:boto:Token: None 2022-09-02T05:57:54.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:57:54.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug= 2022-09-02T05:57:54.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fl+RfvR6L9dMFXYxzgArEEka2ug=', 'Content-Length': '0'} 2022-09-02T05:57:54.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a976fc98dfcc672-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.361 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.362 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.362 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.363 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.363 DEBUG:boto:Method: GET 2022-09-02T05:57:54.364 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.364 DEBUG:boto:Data: 2022-09-02T05:57:54.365 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.366 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.366 DEBUG:boto:Params: {} 2022-09-02T05:57:54.367 DEBUG:boto:Token: None 2022-09-02T05:57:54.368 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027f2a280b54b66ea-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.373 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.374 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.374 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.375 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.375 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.376 DEBUG:boto:Method: GET 2022-09-02T05:57:54.376 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.377 DEBUG:boto:Data: 2022-09-02T05:57:54.377 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.378 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.379 DEBUG:boto:Params: {} 2022-09-02T05:57:54.379 DEBUG:boto:Token: None 2022-09-02T05:57:54.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2b29635ea7f8009-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.386 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.387 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.388 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.388 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.389 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:57:54.389 DEBUG:boto:Data: 2022-09-02T05:57:54.390 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.391 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.391 DEBUG:boto:Params: {} 2022-09-02T05:57:54.392 DEBUG:boto:Token: None 2022-09-02T05:57:54.392 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44 2022-09-02T05:57:54.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9baPKmy0FvxJB2wWnSSLvncmVBw= 2022-09-02T05:57:54.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9baPKmy0FvxJB2wWnSSLvncmVBw=', 'Content-Length': '0'} 2022-09-02T05:57:54.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046f204155a0d3c6f-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.396 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.397 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.397 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.398 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:57:54.398 DEBUG:boto:Data: 2022-09-02T05:57:54.399 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.400 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.400 DEBUG:boto:Params: {} 2022-09-02T05:57:54.401 DEBUG:boto:Token: None 2022-09-02T05:57:54.401 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44 2022-09-02T05:57:54.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9baPKmy0FvxJB2wWnSSLvncmVBw= 2022-09-02T05:57:54.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9baPKmy0FvxJB2wWnSSLvncmVBw=', 'Content-Length': '0'} 2022-09-02T05:57:54.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db42eb2a9fd33fd-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:57:54.406 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.407 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.407 DEBUG:boto:Method: GET 2022-09-02T05:57:54.408 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:57:54.408 DEBUG:boto:Data: 2022-09-02T05:57:54.409 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.409 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.410 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.410 DEBUG:boto:Params: {} 2022-09-02T05:57:54.411 DEBUG:boto:Token: None 2022-09-02T05:57:54.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44 2022-09-02T05:57:54.412 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2NFj2fr4xTY02XOwA7i9OxH8JVU= 2022-09-02T05:57:54.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2NFj2fr4xTY02XOwA7i9OxH8JVU=', 'Content-Length': '0'} 2022-09-02T05:57:54.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004936d923baf94074-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.416 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:57:54.417 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:57:54.417 DEBUG:boto:Method: GET 2022-09-02T05:57:54.418 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:57:54.418 DEBUG:boto:Data: 2022-09-02T05:57:54.419 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.420 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.420 DEBUG:boto:Params: {} 2022-09-02T05:57:54.421 DEBUG:boto:Token: None 2022-09-02T05:57:54.422 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj44 2022-09-02T05:57:54.422 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2NFj2fr4xTY02XOwA7i9OxH8JVU= 2022-09-02T05:57:54.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2NFj2fr4xTY02XOwA7i9OxH8JVU=', 'Content-Length': '0'} 2022-09-02T05:57:54.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb9e955c626f08c-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.426 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.427 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.427 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.428 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.428 DEBUG:boto:Method: GET 2022-09-02T05:57:54.429 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.429 DEBUG:boto:Data: 2022-09-02T05:57:54.430 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.431 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.432 DEBUG:boto:Params: {} 2022-09-02T05:57:54.432 DEBUG:boto:Token: None 2022-09-02T05:57:54.433 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d5fe6d790d9a238-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.438 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.438 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.439 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.440 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.440 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.441 DEBUG:boto:Method: GET 2022-09-02T05:57:54.441 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:57:54.442 DEBUG:boto:Data: 2022-09-02T05:57:54.442 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.443 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.444 DEBUG:boto:Params: {} 2022-09-02T05:57:54.444 DEBUG:boto:Token: None 2022-09-02T05:57:54.445 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.446 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c3a81cb798f6d5-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.451 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:57:54.453 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.454 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.454 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.455 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.455 DEBUG:boto:Method: GET 2022-09-02T05:57:54.456 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.456 DEBUG:boto:Data: 2022-09-02T05:57:54.457 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.457 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.458 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.458 DEBUG:boto:Params: {} 2022-09-02T05:57:54.459 DEBUG:boto:Token: None 2022-09-02T05:57:54.460 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc= 2022-09-02T05:57:54.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc=', 'Content-Length': '0'} 2022-09-02T05:57:54.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8060d8c8ab23e2f-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.464 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.465 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.465 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.466 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.466 DEBUG:boto:Method: GET 2022-09-02T05:57:54.467 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.467 DEBUG:boto:Data: 2022-09-02T05:57:54.468 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.469 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.469 DEBUG:boto:Params: {} 2022-09-02T05:57:54.470 DEBUG:boto:Token: None 2022-09-02T05:57:54.471 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc= 2022-09-02T05:57:54.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc=', 'Content-Length': '0'} 2022-09-02T05:57:54.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053ba01c062ae31c-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.518 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.519 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.519 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.520 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.520 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.521 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.521 DEBUG:boto:Data: 2022-09-02T05:57:54.522 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.522 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.523 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.523 DEBUG:boto:Params: {} 2022-09-02T05:57:54.524 DEBUG:boto:Token: None 2022-09-02T05:57:54.524 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ljElbnOV8V42YbmvRhtc7QAiO0= 2022-09-02T05:57:54.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ljElbnOV8V42YbmvRhtc7QAiO0=', 'Content-Length': '0'} 2022-09-02T05:57:54.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca71f682561927b8-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.529 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.529 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.530 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.531 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.531 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.532 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.532 DEBUG:boto:Data: 2022-09-02T05:57:54.533 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.534 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.534 DEBUG:boto:Params: {} 2022-09-02T05:57:54.535 DEBUG:boto:Token: None 2022-09-02T05:57:54.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8ljElbnOV8V42YbmvRhtc7QAiO0= 2022-09-02T05:57:54.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8ljElbnOV8V42YbmvRhtc7QAiO0=', 'Content-Length': '0'} 2022-09-02T05:57:54.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93681dcce392096-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:57:54.541 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.541 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.542 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.542 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.543 DEBUG:boto:Method: GET 2022-09-02T05:57:54.544 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.545 DEBUG:boto:Data: 2022-09-02T05:57:54.546 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.548 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.549 DEBUG:boto:Params: {} 2022-09-02T05:57:54.551 DEBUG:boto:Token: None 2022-09-02T05:57:54.552 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.553 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc= 2022-09-02T05:57:54.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc=', 'Content-Length': '0'} 2022-09-02T05:57:54.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ec21c4d989f55d-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.558 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.559 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.559 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.560 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.560 DEBUG:boto:Method: GET 2022-09-02T05:57:54.561 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.561 DEBUG:boto:Data: 2022-09-02T05:57:54.562 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.563 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.563 DEBUG:boto:Params: {} 2022-09-02T05:57:54.564 DEBUG:boto:Token: None 2022-09-02T05:57:54.565 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:57:54.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc= 2022-09-02T05:57:54.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M33zCq6WRoNpWfvLzWRfjIM3rxc=', 'Content-Length': '0'} 2022-09-02T05:57:54.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf79368ffff8daed-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.569 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.570 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.570 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.571 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.571 DEBUG:boto:Method: GET 2022-09-02T05:57:54.572 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.573 DEBUG:boto:Data: 2022-09-02T05:57:54.573 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.574 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.574 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.575 DEBUG:boto:Params: {} 2022-09-02T05:57:54.576 DEBUG:boto:Token: None 2022-09-02T05:57:54.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8fb4eb791f07911-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.581 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.583 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.583 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.584 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.585 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.585 DEBUG:boto:Method: GET 2022-09-02T05:57:54.586 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.587 DEBUG:boto:Data: 2022-09-02T05:57:54.587 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.588 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.589 DEBUG:boto:Params: {} 2022-09-02T05:57:54.590 DEBUG:boto:Token: None 2022-09-02T05:57:54.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ab45245be4b883-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.597 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.598 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.599 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.599 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.600 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:57:54.600 DEBUG:boto:Data: 2022-09-02T05:57:54.601 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.601 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.602 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.603 DEBUG:boto:Params: {} 2022-09-02T05:57:54.603 DEBUG:boto:Token: None 2022-09-02T05:57:54.604 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46 2022-09-02T05:57:54.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pddARVHnxQ+KL4a+GmAdCohkT74= 2022-09-02T05:57:54.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pddARVHnxQ+KL4a+GmAdCohkT74=', 'Content-Length': '0'} 2022-09-02T05:57:54.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffd0690a18029512-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.608 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.608 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.609 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.610 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:57:54.610 DEBUG:boto:Data: 2022-09-02T05:57:54.611 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.612 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.612 DEBUG:boto:Params: {} 2022-09-02T05:57:54.613 DEBUG:boto:Token: None 2022-09-02T05:57:54.614 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46 2022-09-02T05:57:54.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pddARVHnxQ+KL4a+GmAdCohkT74= 2022-09-02T05:57:54.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pddARVHnxQ+KL4a+GmAdCohkT74=', 'Content-Length': '0'} 2022-09-02T05:57:54.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872a891fd67917ca-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:57:54.619 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.619 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.620 DEBUG:boto:Method: GET 2022-09-02T05:57:54.621 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:57:54.621 DEBUG:boto:Data: 2022-09-02T05:57:54.622 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.623 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.624 DEBUG:boto:Params: {} 2022-09-02T05:57:54.624 DEBUG:boto:Token: None 2022-09-02T05:57:54.625 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46 2022-09-02T05:57:54.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KVMXtVGHRc4wM+xguLcyF/3s4zU= 2022-09-02T05:57:54.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KVMXtVGHRc4wM+xguLcyF/3s4zU=', 'Content-Length': '0'} 2022-09-02T05:57:54.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd7796f976fea49-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.629 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:57:54.630 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:57:54.631 DEBUG:boto:Method: GET 2022-09-02T05:57:54.631 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:57:54.632 DEBUG:boto:Data: 2022-09-02T05:57:54.632 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.633 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.634 DEBUG:boto:Params: {} 2022-09-02T05:57:54.635 DEBUG:boto:Token: None 2022-09-02T05:57:54.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj46 2022-09-02T05:57:54.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KVMXtVGHRc4wM+xguLcyF/3s4zU= 2022-09-02T05:57:54.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KVMXtVGHRc4wM+xguLcyF/3s4zU=', 'Content-Length': '0'} 2022-09-02T05:57:54.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad63a1521a8978d-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.641 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.642 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.643 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.644 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.645 DEBUG:boto:Method: GET 2022-09-02T05:57:54.646 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.646 DEBUG:boto:Data: 2022-09-02T05:57:54.647 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.648 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.649 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.650 DEBUG:boto:Params: {} 2022-09-02T05:57:54.651 DEBUG:boto:Token: None 2022-09-02T05:57:54.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16f8c21fef73ebb-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.658 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.659 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.660 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.660 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.661 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.661 DEBUG:boto:Method: GET 2022-09-02T05:57:54.662 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:57:54.662 DEBUG:boto:Data: 2022-09-02T05:57:54.663 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.664 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.665 DEBUG:boto:Params: {} 2022-09-02T05:57:54.665 DEBUG:boto:Token: None 2022-09-02T05:57:54.666 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f34bcfee1f54b07d-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.714 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:57:54.716 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.716 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.717 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.718 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.719 DEBUG:boto:Method: GET 2022-09-02T05:57:54.720 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.721 DEBUG:boto:Data: 2022-09-02T05:57:54.723 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.725 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.726 DEBUG:boto:Params: {} 2022-09-02T05:57:54.727 DEBUG:boto:Token: None 2022-09-02T05:57:54.728 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.729 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c= 2022-09-02T05:57:54.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c=', 'Content-Length': '0'} 2022-09-02T05:57:54.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f22bce1c6c6e32-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.733 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.734 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.734 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.735 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.735 DEBUG:boto:Method: GET 2022-09-02T05:57:54.736 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.737 DEBUG:boto:Data: 2022-09-02T05:57:54.737 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.738 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.739 DEBUG:boto:Params: {} 2022-09-02T05:57:54.739 DEBUG:boto:Token: None 2022-09-02T05:57:54.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c= 2022-09-02T05:57:54.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c=', 'Content-Length': '0'} 2022-09-02T05:57:54.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c316ae6d029c71-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.745 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.745 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.746 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.746 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.747 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.748 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.748 DEBUG:boto:Data: 2022-09-02T05:57:54.749 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.750 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.751 DEBUG:boto:Params: {} 2022-09-02T05:57:54.752 DEBUG:boto:Token: None 2022-09-02T05:57:54.754 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wj6uXn2KuHpqUxWdHh4WLlrnpfs= 2022-09-02T05:57:54.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wj6uXn2KuHpqUxWdHh4WLlrnpfs=', 'Content-Length': '0'} 2022-09-02T05:57:54.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b20bf8a797c9131-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.760 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.761 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.762 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.763 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.764 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.765 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.766 DEBUG:boto:Data: 2022-09-02T05:57:54.767 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.769 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.770 DEBUG:boto:Params: {} 2022-09-02T05:57:54.772 DEBUG:boto:Token: None 2022-09-02T05:57:54.773 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wj6uXn2KuHpqUxWdHh4WLlrnpfs= 2022-09-02T05:57:54.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wj6uXn2KuHpqUxWdHh4WLlrnpfs=', 'Content-Length': '0'} 2022-09-02T05:57:54.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6de31f72fcf32df-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:57:54.779 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.779 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.780 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.781 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.781 DEBUG:boto:Method: GET 2022-09-02T05:57:54.782 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.782 DEBUG:boto:Data: 2022-09-02T05:57:54.783 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.783 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.784 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.785 DEBUG:boto:Params: {} 2022-09-02T05:57:54.785 DEBUG:boto:Token: None 2022-09-02T05:57:54.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c= 2022-09-02T05:57:54.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c=', 'Content-Length': '0'} 2022-09-02T05:57:54.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b3baa854782ad35-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.790 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.791 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.791 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.792 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.792 DEBUG:boto:Method: GET 2022-09-02T05:57:54.793 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.794 DEBUG:boto:Data: 2022-09-02T05:57:54.794 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.795 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.796 DEBUG:boto:Params: {} 2022-09-02T05:57:54.796 DEBUG:boto:Token: None 2022-09-02T05:57:54.797 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:57:54.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c= 2022-09-02T05:57:54.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8i9AY8tRisMKkazwdVuclhrrp/c=', 'Content-Length': '0'} 2022-09-02T05:57:54.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cb19dc1af1345c1-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.802 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.802 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.803 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.803 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.804 DEBUG:boto:Method: GET 2022-09-02T05:57:54.804 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.805 DEBUG:boto:Data: 2022-09-02T05:57:54.806 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.806 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.807 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.808 DEBUG:boto:Params: {} 2022-09-02T05:57:54.808 DEBUG:boto:Token: None 2022-09-02T05:57:54.809 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ceb03ec9ef2cf6a-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.814 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.815 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.815 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.816 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.817 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.817 DEBUG:boto:Method: GET 2022-09-02T05:57:54.818 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.818 DEBUG:boto:Data: 2022-09-02T05:57:54.819 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.820 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.821 DEBUG:boto:Params: {} 2022-09-02T05:57:54.821 DEBUG:boto:Token: None 2022-09-02T05:57:54.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000346aeac1dd09db24-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.828 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.829 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.830 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.830 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.831 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:57:54.832 DEBUG:boto:Data: 2022-09-02T05:57:54.832 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.833 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.833 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.834 DEBUG:boto:Params: {} 2022-09-02T05:57:54.835 DEBUG:boto:Token: None 2022-09-02T05:57:54.835 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51 2022-09-02T05:57:54.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vvWIgEGc6QO8bwyZzKx234HAWY8= 2022-09-02T05:57:54.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vvWIgEGc6QO8bwyZzKx234HAWY8=', 'Content-Length': '0'} 2022-09-02T05:57:54.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef12e7f1482b67ac-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.839 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.840 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.841 DEBUG:boto:Method: HEAD 2022-09-02T05:57:54.841 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:57:54.842 DEBUG:boto:Data: 2022-09-02T05:57:54.843 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.844 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.844 DEBUG:boto:Params: {} 2022-09-02T05:57:54.845 DEBUG:boto:Token: None 2022-09-02T05:57:54.846 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51 2022-09-02T05:57:54.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vvWIgEGc6QO8bwyZzKx234HAWY8= 2022-09-02T05:57:54.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vvWIgEGc6QO8bwyZzKx234HAWY8=', 'Content-Length': '0'} 2022-09-02T05:57:54.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d878261f8e409222-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:57:54.851 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.851 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.852 DEBUG:boto:Method: GET 2022-09-02T05:57:54.853 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:57:54.853 DEBUG:boto:Data: 2022-09-02T05:57:54.854 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.855 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.856 DEBUG:boto:Params: {} 2022-09-02T05:57:54.857 DEBUG:boto:Token: None 2022-09-02T05:57:54.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51 2022-09-02T05:57:54.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FW1LeE/OlvI97kj01byjK9IkoDg= 2022-09-02T05:57:54.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FW1LeE/OlvI97kj01byjK9IkoDg=', 'Content-Length': '0'} 2022-09-02T05:57:54.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a376fa7d22b563b-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.902 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:57:54.903 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:57:54.904 DEBUG:boto:Method: GET 2022-09-02T05:57:54.904 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:57:54.905 DEBUG:boto:Data: 2022-09-02T05:57:54.905 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.906 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.907 DEBUG:boto:Params: {} 2022-09-02T05:57:54.908 DEBUG:boto:Token: None 2022-09-02T05:57:54.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj51 2022-09-02T05:57:54.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FW1LeE/OlvI97kj01byjK9IkoDg= 2022-09-02T05:57:54.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FW1LeE/OlvI97kj01byjK9IkoDg=', 'Content-Length': '0'} 2022-09-02T05:57:54.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b6a2f4102e1b29f-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.913 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.913 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.914 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.915 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.915 DEBUG:boto:Method: GET 2022-09-02T05:57:54.916 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.916 DEBUG:boto:Data: 2022-09-02T05:57:54.917 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.918 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.919 DEBUG:boto:Params: {} 2022-09-02T05:57:54.919 DEBUG:boto:Token: None 2022-09-02T05:57:54.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dcbe2dd3208b5ce-0063119b62-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.925 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:54.927 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:54.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.928 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.928 DEBUG:boto:Method: GET 2022-09-02T05:57:54.929 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:57:54.930 DEBUG:boto:Data: 2022-09-02T05:57:54.931 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.932 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.932 DEBUG:boto:Params: {} 2022-09-02T05:57:54.933 DEBUG:boto:Token: None 2022-09-02T05:57:54.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/ 2022-09-02T05:57:54.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg= 2022-09-02T05:57:54.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tDC1zJ/pmd1bJlLZIpUMiF4P4wg=', 'Content-Length': '0'} 2022-09-02T05:57:54.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d063290cfaaeeaff-0063119b62-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.940 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:54.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:57:54.942 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:54.942 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:54.943 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.943 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.944 DEBUG:boto:Method: GET 2022-09-02T05:57:54.945 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.945 DEBUG:boto:Data: 2022-09-02T05:57:54.946 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:54.947 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.948 DEBUG:boto:Params: {} 2022-09-02T05:57:54.949 DEBUG:boto:Token: None 2022-09-02T05:57:54.949 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RUP03I13Zcj8XrmBJ480mjdPyC0= 2022-09-02T05:57:54.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RUP03I13Zcj8XrmBJ480mjdPyC0=', 'Content-Length': '0'} 2022-09-02T05:57:54.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef7c75fb6a5d94fc-0063119b62-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:54.954 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:54.955 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:54.956 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.956 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.957 DEBUG:boto:Method: GET 2022-09-02T05:57:54.958 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.959 DEBUG:boto:Data: 2022-09-02T05:57:54.959 DEBUG:boto:Headers: {} 2022-09-02T05:57:54.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:54.960 DEBUG:boto:Port: 8000 2022-09-02T05:57:54.961 DEBUG:boto:Params: {} 2022-09-02T05:57:54.961 DEBUG:boto:Token: None 2022-09-02T05:57:54.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:54 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:54.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RUP03I13Zcj8XrmBJ480mjdPyC0= 2022-09-02T05:57:54.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:54 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RUP03I13Zcj8XrmBJ480mjdPyC0=', 'Content-Length': '0'} 2022-09-02T05:57:54.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37a7b854ea06e80-0063119b62-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:54 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.006 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.007 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.008 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.009 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.010 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.011 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.012 DEBUG:boto:Data: 2022-09-02T05:57:55.013 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.014 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.015 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.016 DEBUG:boto:Params: {} 2022-09-02T05:57:55.017 DEBUG:boto:Token: None 2022-09-02T05:57:55.018 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ME2h8HnSqbFw8unF6juHigf640o= 2022-09-02T05:57:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ME2h8HnSqbFw8unF6juHigf640o=', 'Content-Length': '0'} 2022-09-02T05:57:55.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e0a0e76d45559c-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.024 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.025 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.026 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.027 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.028 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.029 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.030 DEBUG:boto:Data: 2022-09-02T05:57:55.031 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.032 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.033 DEBUG:boto:Params: {} 2022-09-02T05:57:55.034 DEBUG:boto:Token: None 2022-09-02T05:57:55.035 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ME2h8HnSqbFw8unF6juHigf640o= 2022-09-02T05:57:55.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ME2h8HnSqbFw8unF6juHigf640o=', 'Content-Length': '0'} 2022-09-02T05:57:55.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4afa2149aed74d-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:57:55.043 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.044 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.045 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.046 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.047 DEBUG:boto:Method: GET 2022-09-02T05:57:55.048 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.049 DEBUG:boto:Data: 2022-09-02T05:57:55.050 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.052 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.053 DEBUG:boto:Params: {} 2022-09-02T05:57:55.054 DEBUG:boto:Token: None 2022-09-02T05:57:55.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GQKW+vkQaJkbn3hZqPDhglJLm8U= 2022-09-02T05:57:55.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GQKW+vkQaJkbn3hZqPDhglJLm8U=', 'Content-Length': '0'} 2022-09-02T05:57:55.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055226d6939f1faeb-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.061 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.062 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.064 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.065 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.066 DEBUG:boto:Method: GET 2022-09-02T05:57:55.067 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.068 DEBUG:boto:Data: 2022-09-02T05:57:55.069 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.071 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.072 DEBUG:boto:Params: {} 2022-09-02T05:57:55.073 DEBUG:boto:Token: None 2022-09-02T05:57:55.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:57:55.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GQKW+vkQaJkbn3hZqPDhglJLm8U= 2022-09-02T05:57:55.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GQKW+vkQaJkbn3hZqPDhglJLm8U=', 'Content-Length': '0'} 2022-09-02T05:57:55.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60b242a32bca9a5-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.079 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.079 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.080 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.080 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.081 DEBUG:boto:Method: GET 2022-09-02T05:57:55.082 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.082 DEBUG:boto:Data: 2022-09-02T05:57:55.083 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.083 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.084 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.084 DEBUG:boto:Params: {} 2022-09-02T05:57:55.085 DEBUG:boto:Token: None 2022-09-02T05:57:55.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000370ad3e5f390ae3d-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.090 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.091 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.091 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.092 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.093 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.093 DEBUG:boto:Method: GET 2022-09-02T05:57:55.094 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.094 DEBUG:boto:Data: 2022-09-02T05:57:55.095 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.096 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.096 DEBUG:boto:Params: {} 2022-09-02T05:57:55.097 DEBUG:boto:Token: None 2022-09-02T05:57:55.097 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.098 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9da819e89ab326-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.104 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.106 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.107 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.107 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.108 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:57:55.109 DEBUG:boto:Data: 2022-09-02T05:57:55.109 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.110 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.111 DEBUG:boto:Params: {} 2022-09-02T05:57:55.112 DEBUG:boto:Token: None 2022-09-02T05:57:55.112 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55 2022-09-02T05:57:55.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZmXIRiCYhrtjuz2Og74WZps+IrQ= 2022-09-02T05:57:55.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZmXIRiCYhrtjuz2Og74WZps+IrQ=', 'Content-Length': '0'} 2022-09-02T05:57:55.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de5b6ed9902b0d96-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.116 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.117 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.117 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.118 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:57:55.118 DEBUG:boto:Data: 2022-09-02T05:57:55.119 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.120 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.121 DEBUG:boto:Params: {} 2022-09-02T05:57:55.121 DEBUG:boto:Token: None 2022-09-02T05:57:55.122 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55 2022-09-02T05:57:55.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZmXIRiCYhrtjuz2Og74WZps+IrQ= 2022-09-02T05:57:55.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZmXIRiCYhrtjuz2Og74WZps+IrQ=', 'Content-Length': '0'} 2022-09-02T05:57:55.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2a2b48b6595d51-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:57:55.127 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.127 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.128 DEBUG:boto:Method: GET 2022-09-02T05:57:55.128 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:57:55.129 DEBUG:boto:Data: 2022-09-02T05:57:55.129 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.131 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.131 DEBUG:boto:Params: {} 2022-09-02T05:57:55.132 DEBUG:boto:Token: None 2022-09-02T05:57:55.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55 2022-09-02T05:57:55.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YZtmOadxmQYDZfJ5eDT5nqirtNg= 2022-09-02T05:57:55.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YZtmOadxmQYDZfJ5eDT5nqirtNg=', 'Content-Length': '0'} 2022-09-02T05:57:55.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33baef1f9c9c0b3-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.137 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:57:55.137 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:57:55.138 DEBUG:boto:Method: GET 2022-09-02T05:57:55.138 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:57:55.139 DEBUG:boto:Data: 2022-09-02T05:57:55.139 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.140 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.141 DEBUG:boto:Params: {} 2022-09-02T05:57:55.142 DEBUG:boto:Token: None 2022-09-02T05:57:55.142 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj55 2022-09-02T05:57:55.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YZtmOadxmQYDZfJ5eDT5nqirtNg= 2022-09-02T05:57:55.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YZtmOadxmQYDZfJ5eDT5nqirtNg=', 'Content-Length': '0'} 2022-09-02T05:57:55.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6c7cc8f1374ae1-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.186 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.187 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.188 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.189 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.190 DEBUG:boto:Method: GET 2022-09-02T05:57:55.190 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.191 DEBUG:boto:Data: 2022-09-02T05:57:55.191 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.192 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.192 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.193 DEBUG:boto:Params: {} 2022-09-02T05:57:55.194 DEBUG:boto:Token: None 2022-09-02T05:57:55.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000266c2aa8751938ea-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.199 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.200 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.201 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.201 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.202 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.202 DEBUG:boto:Method: GET 2022-09-02T05:57:55.203 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:57:55.204 DEBUG:boto:Data: 2022-09-02T05:57:55.204 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.205 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.206 DEBUG:boto:Params: {} 2022-09-02T05:57:55.206 DEBUG:boto:Token: None 2022-09-02T05:57:55.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006072965d0412b90e-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.213 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:57:55.215 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.215 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.216 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.217 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.217 DEBUG:boto:Method: GET 2022-09-02T05:57:55.218 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.219 DEBUG:boto:Data: 2022-09-02T05:57:55.219 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.220 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.221 DEBUG:boto:Params: {} 2022-09-02T05:57:55.221 DEBUG:boto:Token: None 2022-09-02T05:57:55.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs= 2022-09-02T05:57:55.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs=', 'Content-Length': '0'} 2022-09-02T05:57:55.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb8c93b02638d016-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.266 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.267 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.267 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.268 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.268 DEBUG:boto:Method: GET 2022-09-02T05:57:55.269 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.269 DEBUG:boto:Data: 2022-09-02T05:57:55.270 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.272 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.272 DEBUG:boto:Params: {} 2022-09-02T05:57:55.273 DEBUG:boto:Token: None 2022-09-02T05:57:55.273 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs= 2022-09-02T05:57:55.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs=', 'Content-Length': '0'} 2022-09-02T05:57:55.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a55aafbd35a866-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.278 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.279 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.279 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.280 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.280 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.281 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.281 DEBUG:boto:Data: 2022-09-02T05:57:55.282 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.283 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.283 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.284 DEBUG:boto:Params: {} 2022-09-02T05:57:55.285 DEBUG:boto:Token: None 2022-09-02T05:57:55.285 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zuX6uLSe6iYcRMems37GTJmUKaE= 2022-09-02T05:57:55.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zuX6uLSe6iYcRMems37GTJmUKaE=', 'Content-Length': '0'} 2022-09-02T05:57:55.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165f8f745e086dc6-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.289 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.290 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.290 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.291 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.291 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.292 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.292 DEBUG:boto:Data: 2022-09-02T05:57:55.293 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.294 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.295 DEBUG:boto:Params: {} 2022-09-02T05:57:55.295 DEBUG:boto:Token: None 2022-09-02T05:57:55.296 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zuX6uLSe6iYcRMems37GTJmUKaE= 2022-09-02T05:57:55.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zuX6uLSe6iYcRMems37GTJmUKaE=', 'Content-Length': '0'} 2022-09-02T05:57:55.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005629d31a4f1ef423-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:57:55.300 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.301 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.302 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.302 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.303 DEBUG:boto:Method: GET 2022-09-02T05:57:55.303 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.304 DEBUG:boto:Data: 2022-09-02T05:57:55.305 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.305 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.306 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.306 DEBUG:boto:Params: {} 2022-09-02T05:57:55.307 DEBUG:boto:Token: None 2022-09-02T05:57:55.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs= 2022-09-02T05:57:55.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs=', 'Content-Length': '0'} 2022-09-02T05:57:55.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d01f7eb6471992a-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.312 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.312 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.313 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.314 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.315 DEBUG:boto:Method: GET 2022-09-02T05:57:55.316 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.317 DEBUG:boto:Data: 2022-09-02T05:57:55.318 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.320 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.321 DEBUG:boto:Params: {} 2022-09-02T05:57:55.323 DEBUG:boto:Token: None 2022-09-02T05:57:55.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:57:55.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs= 2022-09-02T05:57:55.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UzYgEaxkM9P39wPSDDqfwN6zFBs=', 'Content-Length': '0'} 2022-09-02T05:57:55.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000795f958dff2aa9c9-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.331 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.333 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.334 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.335 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.336 DEBUG:boto:Method: GET 2022-09-02T05:57:55.337 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.338 DEBUG:boto:Data: 2022-09-02T05:57:55.339 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.340 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.341 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.343 DEBUG:boto:Params: {} 2022-09-02T05:57:55.344 DEBUG:boto:Token: None 2022-09-02T05:57:55.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f89f63b56e1c841f-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.352 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.353 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.354 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.354 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.355 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.355 DEBUG:boto:Method: GET 2022-09-02T05:57:55.356 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.356 DEBUG:boto:Data: 2022-09-02T05:57:55.357 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.358 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.358 DEBUG:boto:Params: {} 2022-09-02T05:57:55.359 DEBUG:boto:Token: None 2022-09-02T05:57:55.360 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000583305a31aa36b2e-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.367 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.367 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.368 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.369 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.369 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:57:55.370 DEBUG:boto:Data: 2022-09-02T05:57:55.370 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.371 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.372 DEBUG:boto:Params: {} 2022-09-02T05:57:55.372 DEBUG:boto:Token: None 2022-09-02T05:57:55.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57 2022-09-02T05:57:55.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZkKPI0FDOocEYX3PwfpgZRxl63Q= 2022-09-02T05:57:55.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZkKPI0FDOocEYX3PwfpgZRxl63Q=', 'Content-Length': '0'} 2022-09-02T05:57:55.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001325cc7f6092b0a5-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.377 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.378 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.378 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.379 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:57:55.379 DEBUG:boto:Data: 2022-09-02T05:57:55.380 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.381 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.381 DEBUG:boto:Params: {} 2022-09-02T05:57:55.382 DEBUG:boto:Token: None 2022-09-02T05:57:55.383 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57 2022-09-02T05:57:55.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZkKPI0FDOocEYX3PwfpgZRxl63Q= 2022-09-02T05:57:55.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZkKPI0FDOocEYX3PwfpgZRxl63Q=', 'Content-Length': '0'} 2022-09-02T05:57:55.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa973e215486ed5c-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:57:55.388 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.388 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.389 DEBUG:boto:Method: GET 2022-09-02T05:57:55.389 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:57:55.390 DEBUG:boto:Data: 2022-09-02T05:57:55.390 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.392 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.392 DEBUG:boto:Params: {} 2022-09-02T05:57:55.393 DEBUG:boto:Token: None 2022-09-02T05:57:55.393 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57 2022-09-02T05:57:55.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vv9KsSEPUxo2yu1V5O3PRkTvGzY= 2022-09-02T05:57:55.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vv9KsSEPUxo2yu1V5O3PRkTvGzY=', 'Content-Length': '0'} 2022-09-02T05:57:55.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c70846b8c51390-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.398 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:57:55.399 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:57:55.399 DEBUG:boto:Method: GET 2022-09-02T05:57:55.400 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:57:55.401 DEBUG:boto:Data: 2022-09-02T05:57:55.401 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.402 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.403 DEBUG:boto:Params: {} 2022-09-02T05:57:55.403 DEBUG:boto:Token: None 2022-09-02T05:57:55.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj57 2022-09-02T05:57:55.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vv9KsSEPUxo2yu1V5O3PRkTvGzY= 2022-09-02T05:57:55.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vv9KsSEPUxo2yu1V5O3PRkTvGzY=', 'Content-Length': '0'} 2022-09-02T05:57:55.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f6c7f395a8b74c-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.454 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.456 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.457 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.457 DEBUG:boto:Method: GET 2022-09-02T05:57:55.458 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.459 DEBUG:boto:Data: 2022-09-02T05:57:55.459 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.460 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.461 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.461 DEBUG:boto:Params: {} 2022-09-02T05:57:55.462 DEBUG:boto:Token: None 2022-09-02T05:57:55.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ef5fdd3d3ff3cb0-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.468 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.469 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.469 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.470 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.471 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.471 DEBUG:boto:Method: GET 2022-09-02T05:57:55.472 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:57:55.473 DEBUG:boto:Data: 2022-09-02T05:57:55.473 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.474 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.475 DEBUG:boto:Params: {} 2022-09-02T05:57:55.476 DEBUG:boto:Token: None 2022-09-02T05:57:55.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3938c5fe97f7fbb-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.483 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:57:55.485 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.486 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.486 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.487 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.487 DEBUG:boto:Method: GET 2022-09-02T05:57:55.488 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.488 DEBUG:boto:Data: 2022-09-02T05:57:55.489 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.490 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.490 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.491 DEBUG:boto:Params: {} 2022-09-02T05:57:55.491 DEBUG:boto:Token: None 2022-09-02T05:57:55.492 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU= 2022-09-02T05:57:55.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU=', 'Content-Length': '0'} 2022-09-02T05:57:55.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d956ffc859e720-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.538 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.539 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.540 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.540 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.541 DEBUG:boto:Method: GET 2022-09-02T05:57:55.541 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.542 DEBUG:boto:Data: 2022-09-02T05:57:55.543 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.544 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.544 DEBUG:boto:Params: {} 2022-09-02T05:57:55.547 DEBUG:boto:Token: None 2022-09-02T05:57:55.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU= 2022-09-02T05:57:55.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU=', 'Content-Length': '0'} 2022-09-02T05:57:55.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f37fe27a53dff2d0-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.555 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.556 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.557 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.558 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.559 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.559 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.560 DEBUG:boto:Data: 2022-09-02T05:57:55.561 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.562 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.563 DEBUG:boto:Params: {} 2022-09-02T05:57:55.564 DEBUG:boto:Token: None 2022-09-02T05:57:55.564 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YyobugbzQALnm9ZF3zWsUzFqQRo= 2022-09-02T05:57:55.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YyobugbzQALnm9ZF3zWsUzFqQRo=', 'Content-Length': '0'} 2022-09-02T05:57:55.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000edeae7d3fa8b808-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.569 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.570 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.570 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.571 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.571 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.572 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.572 DEBUG:boto:Data: 2022-09-02T05:57:55.573 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.574 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.575 DEBUG:boto:Params: {} 2022-09-02T05:57:55.575 DEBUG:boto:Token: None 2022-09-02T05:57:55.576 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YyobugbzQALnm9ZF3zWsUzFqQRo= 2022-09-02T05:57:55.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YyobugbzQALnm9ZF3zWsUzFqQRo=', 'Content-Length': '0'} 2022-09-02T05:57:55.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e424b2b33a10fce4-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:57:55.581 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.581 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.582 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.583 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.584 DEBUG:boto:Method: GET 2022-09-02T05:57:55.585 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.586 DEBUG:boto:Data: 2022-09-02T05:57:55.587 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.589 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.590 DEBUG:boto:Params: {} 2022-09-02T05:57:55.591 DEBUG:boto:Token: None 2022-09-02T05:57:55.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU= 2022-09-02T05:57:55.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU=', 'Content-Length': '0'} 2022-09-02T05:57:55.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb2a57b96556aa7-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.598 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.599 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.599 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.600 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.601 DEBUG:boto:Method: GET 2022-09-02T05:57:55.601 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.602 DEBUG:boto:Data: 2022-09-02T05:57:55.602 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.604 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.604 DEBUG:boto:Params: {} 2022-09-02T05:57:55.605 DEBUG:boto:Token: None 2022-09-02T05:57:55.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:57:55.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU= 2022-09-02T05:57:55.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nGdGPj4csJSWuF6bpX+9sDPpdhU=', 'Content-Length': '0'} 2022-09-02T05:57:55.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf44e8498ccbd4c6-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.610 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.611 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.611 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.612 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.612 DEBUG:boto:Method: GET 2022-09-02T05:57:55.613 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.614 DEBUG:boto:Data: 2022-09-02T05:57:55.614 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.615 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.616 DEBUG:boto:Params: {} 2022-09-02T05:57:55.617 DEBUG:boto:Token: None 2022-09-02T05:57:55.617 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000443f3bf015f783c-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.622 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.623 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.623 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.624 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.625 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.625 DEBUG:boto:Method: GET 2022-09-02T05:57:55.626 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.626 DEBUG:boto:Data: 2022-09-02T05:57:55.627 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.628 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.629 DEBUG:boto:Params: {} 2022-09-02T05:57:55.629 DEBUG:boto:Token: None 2022-09-02T05:57:55.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a38d077a71b0dd4-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.636 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.637 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.638 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.638 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.639 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:57:55.639 DEBUG:boto:Data: 2022-09-02T05:57:55.640 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.641 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.642 DEBUG:boto:Params: {} 2022-09-02T05:57:55.643 DEBUG:boto:Token: None 2022-09-02T05:57:55.644 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58 2022-09-02T05:57:55.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1KfB0CO9YzfEkLHQC5N2hB4xZbc= 2022-09-02T05:57:55.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1KfB0CO9YzfEkLHQC5N2hB4xZbc=', 'Content-Length': '0'} 2022-09-02T05:57:55.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fbc317459e8430-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.650 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.651 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.652 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.653 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:57:55.654 DEBUG:boto:Data: 2022-09-02T05:57:55.654 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.655 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.656 DEBUG:boto:Params: {} 2022-09-02T05:57:55.656 DEBUG:boto:Token: None 2022-09-02T05:57:55.657 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58 2022-09-02T05:57:55.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1KfB0CO9YzfEkLHQC5N2hB4xZbc= 2022-09-02T05:57:55.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1KfB0CO9YzfEkLHQC5N2hB4xZbc=', 'Content-Length': '0'} 2022-09-02T05:57:55.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd464a604e87664-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:57:55.662 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.662 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.663 DEBUG:boto:Method: GET 2022-09-02T05:57:55.664 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:57:55.664 DEBUG:boto:Data: 2022-09-02T05:57:55.665 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.665 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.666 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.666 DEBUG:boto:Params: {} 2022-09-02T05:57:55.667 DEBUG:boto:Token: None 2022-09-02T05:57:55.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58 2022-09-02T05:57:55.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6nQIE9SFqApS4HEdjba+7BcWylw= 2022-09-02T05:57:55.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6nQIE9SFqApS4HEdjba+7BcWylw=', 'Content-Length': '0'} 2022-09-02T05:57:55.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028004db8b7dc8d28-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.672 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:57:55.672 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:57:55.673 DEBUG:boto:Method: GET 2022-09-02T05:57:55.674 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:57:55.674 DEBUG:boto:Data: 2022-09-02T05:57:55.675 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.676 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.677 DEBUG:boto:Params: {} 2022-09-02T05:57:55.677 DEBUG:boto:Token: None 2022-09-02T05:57:55.678 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj58 2022-09-02T05:57:55.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6nQIE9SFqApS4HEdjba+7BcWylw= 2022-09-02T05:57:55.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6nQIE9SFqApS4HEdjba+7BcWylw=', 'Content-Length': '0'} 2022-09-02T05:57:55.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad135659ecc9c2bc-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.682 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.683 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.684 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.684 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.685 DEBUG:boto:Method: GET 2022-09-02T05:57:55.685 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.686 DEBUG:boto:Data: 2022-09-02T05:57:55.687 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.687 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.688 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.689 DEBUG:boto:Params: {} 2022-09-02T05:57:55.689 DEBUG:boto:Token: None 2022-09-02T05:57:55.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38923df7302720a-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.695 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.696 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.697 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.698 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.698 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.699 DEBUG:boto:Method: GET 2022-09-02T05:57:55.699 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:57:55.700 DEBUG:boto:Data: 2022-09-02T05:57:55.701 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.702 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.702 DEBUG:boto:Params: {} 2022-09-02T05:57:55.703 DEBUG:boto:Token: None 2022-09-02T05:57:55.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.704 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f1861ead73b1bb-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.710 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:57:55.712 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.712 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.713 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.713 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.714 DEBUG:boto:Method: GET 2022-09-02T05:57:55.715 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.715 DEBUG:boto:Data: 2022-09-02T05:57:55.716 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.716 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.717 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.717 DEBUG:boto:Params: {} 2022-09-02T05:57:55.718 DEBUG:boto:Token: None 2022-09-02T05:57:55.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM= 2022-09-02T05:57:55.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM=', 'Content-Length': '0'} 2022-09-02T05:57:55.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4646ccd48b80640-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.723 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.723 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.724 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.724 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.725 DEBUG:boto:Method: GET 2022-09-02T05:57:55.726 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.726 DEBUG:boto:Data: 2022-09-02T05:57:55.727 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.728 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.728 DEBUG:boto:Params: {} 2022-09-02T05:57:55.729 DEBUG:boto:Token: None 2022-09-02T05:57:55.729 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM= 2022-09-02T05:57:55.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM=', 'Content-Length': '0'} 2022-09-02T05:57:55.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c07f15b8acff220-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.734 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.734 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.735 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.735 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.736 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.736 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.737 DEBUG:boto:Data: 2022-09-02T05:57:55.737 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.738 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.738 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.739 DEBUG:boto:Params: {} 2022-09-02T05:57:55.739 DEBUG:boto:Token: None 2022-09-02T05:57:55.740 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uol7CiD1oa2Sk7e8VLeRBCktS30= 2022-09-02T05:57:55.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uol7CiD1oa2Sk7e8VLeRBCktS30=', 'Content-Length': '0'} 2022-09-02T05:57:55.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000880f32789b3a976f-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.744 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.745 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.745 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.746 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.746 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.747 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.747 DEBUG:boto:Data: 2022-09-02T05:57:55.748 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.749 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.749 DEBUG:boto:Params: {} 2022-09-02T05:57:55.750 DEBUG:boto:Token: None 2022-09-02T05:57:55.750 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uol7CiD1oa2Sk7e8VLeRBCktS30= 2022-09-02T05:57:55.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uol7CiD1oa2Sk7e8VLeRBCktS30=', 'Content-Length': '0'} 2022-09-02T05:57:55.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25cf51c004fe86b-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:57:55.755 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.755 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.756 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.757 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.757 DEBUG:boto:Method: GET 2022-09-02T05:57:55.758 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.758 DEBUG:boto:Data: 2022-09-02T05:57:55.759 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.760 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.760 DEBUG:boto:Params: {} 2022-09-02T05:57:55.761 DEBUG:boto:Token: None 2022-09-02T05:57:55.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM= 2022-09-02T05:57:55.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM=', 'Content-Length': '0'} 2022-09-02T05:57:55.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daa2e97090b1980-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.810 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.811 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.811 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.812 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.812 DEBUG:boto:Method: GET 2022-09-02T05:57:55.813 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.813 DEBUG:boto:Data: 2022-09-02T05:57:55.814 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.815 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.816 DEBUG:boto:Params: {} 2022-09-02T05:57:55.817 DEBUG:boto:Token: None 2022-09-02T05:57:55.817 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:57:55.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM= 2022-09-02T05:57:55.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rdR+6imP0d+88JpxEsaiBBJYKqM=', 'Content-Length': '0'} 2022-09-02T05:57:55.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424ec93d7bc1eee3-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.822 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.823 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.823 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.824 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.824 DEBUG:boto:Method: GET 2022-09-02T05:57:55.825 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.825 DEBUG:boto:Data: 2022-09-02T05:57:55.826 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.827 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.827 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.828 DEBUG:boto:Params: {} 2022-09-02T05:57:55.828 DEBUG:boto:Token: None 2022-09-02T05:57:55.829 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a431f6050768484-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.834 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.835 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.835 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.836 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.836 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.837 DEBUG:boto:Method: GET 2022-09-02T05:57:55.837 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.838 DEBUG:boto:Data: 2022-09-02T05:57:55.838 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.839 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.840 DEBUG:boto:Params: {} 2022-09-02T05:57:55.841 DEBUG:boto:Token: None 2022-09-02T05:57:55.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd0cea0ec186f69-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.848 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.849 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.849 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.850 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.850 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:57:55.851 DEBUG:boto:Data: 2022-09-02T05:57:55.852 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.853 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.853 DEBUG:boto:Params: {} 2022-09-02T05:57:55.854 DEBUG:boto:Token: None 2022-09-02T05:57:55.855 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62 2022-09-02T05:57:55.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yi22+QOXAHXuOtoVgtXhP81yF44= 2022-09-02T05:57:55.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yi22+QOXAHXuOtoVgtXhP81yF44=', 'Content-Length': '0'} 2022-09-02T05:57:55.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000272ed8174c1ee722-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.859 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.859 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.860 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.860 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:57:55.861 DEBUG:boto:Data: 2022-09-02T05:57:55.861 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.862 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.863 DEBUG:boto:Params: {} 2022-09-02T05:57:55.864 DEBUG:boto:Token: None 2022-09-02T05:57:55.864 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62 2022-09-02T05:57:55.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yi22+QOXAHXuOtoVgtXhP81yF44= 2022-09-02T05:57:55.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yi22+QOXAHXuOtoVgtXhP81yF44=', 'Content-Length': '0'} 2022-09-02T05:57:55.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c057663d52981a69-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:57:55.869 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.869 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.870 DEBUG:boto:Method: GET 2022-09-02T05:57:55.871 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:57:55.871 DEBUG:boto:Data: 2022-09-02T05:57:55.872 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.872 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.873 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.873 DEBUG:boto:Params: {} 2022-09-02T05:57:55.874 DEBUG:boto:Token: None 2022-09-02T05:57:55.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62 2022-09-02T05:57:55.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1E1Bi9wEROkBY7NeRxi49bMNZHQ= 2022-09-02T05:57:55.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1E1Bi9wEROkBY7NeRxi49bMNZHQ=', 'Content-Length': '0'} 2022-09-02T05:57:55.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f8227c4b6011fb-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.879 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:57:55.879 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:57:55.880 DEBUG:boto:Method: GET 2022-09-02T05:57:55.880 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:57:55.881 DEBUG:boto:Data: 2022-09-02T05:57:55.881 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.882 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.883 DEBUG:boto:Params: {} 2022-09-02T05:57:55.884 DEBUG:boto:Token: None 2022-09-02T05:57:55.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj62 2022-09-02T05:57:55.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1E1Bi9wEROkBY7NeRxi49bMNZHQ= 2022-09-02T05:57:55.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1E1Bi9wEROkBY7NeRxi49bMNZHQ=', 'Content-Length': '0'} 2022-09-02T05:57:55.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970de83dc7df4268-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.930 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.931 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.931 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.932 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.933 DEBUG:boto:Method: GET 2022-09-02T05:57:55.933 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.934 DEBUG:boto:Data: 2022-09-02T05:57:55.934 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.936 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.936 DEBUG:boto:Params: {} 2022-09-02T05:57:55.937 DEBUG:boto:Token: None 2022-09-02T05:57:55.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa9c052247e4f13-0063119b63-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.942 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.943 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:55.944 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:55.944 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.945 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.946 DEBUG:boto:Method: GET 2022-09-02T05:57:55.946 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:57:55.947 DEBUG:boto:Data: 2022-09-02T05:57:55.947 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.948 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.949 DEBUG:boto:Params: {} 2022-09-02T05:57:55.949 DEBUG:boto:Token: None 2022-09-02T05:57:55.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/ 2022-09-02T05:57:55.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc= 2022-09-02T05:57:55.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+AAXrMZHZjlDemIUXVRjGeKFkDc=', 'Content-Length': '0'} 2022-09-02T05:57:55.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e15b56f6e2039e4-0063119b63-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.957 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:55.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:57:55.958 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:55.959 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:55.959 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.960 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.960 DEBUG:boto:Method: GET 2022-09-02T05:57:55.961 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.961 DEBUG:boto:Data: 2022-09-02T05:57:55.962 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.962 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.963 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.964 DEBUG:boto:Params: {} 2022-09-02T05:57:55.964 DEBUG:boto:Token: None 2022-09-02T05:57:55.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DsVfQaCWHkiBqPHC+/h5SZtGtoE= 2022-09-02T05:57:55.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DsVfQaCWHkiBqPHC+/h5SZtGtoE=', 'Content-Length': '0'} 2022-09-02T05:57:55.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001142e25a372ca18-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.969 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:55.970 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:55.970 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.971 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.971 DEBUG:boto:Method: GET 2022-09-02T05:57:55.972 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.973 DEBUG:boto:Data: 2022-09-02T05:57:55.973 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.974 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.975 DEBUG:boto:Params: {} 2022-09-02T05:57:55.975 DEBUG:boto:Token: None 2022-09-02T05:57:55.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DsVfQaCWHkiBqPHC+/h5SZtGtoE= 2022-09-02T05:57:55.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DsVfQaCWHkiBqPHC+/h5SZtGtoE=', 'Content-Length': '0'} 2022-09-02T05:57:55.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc1a40df0ee5f48-0063119b63-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.980 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:55.981 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:55.981 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.982 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.983 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.983 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.984 DEBUG:boto:Data: 2022-09-02T05:57:55.984 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:55.985 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.986 DEBUG:boto:Params: {} 2022-09-02T05:57:55.986 DEBUG:boto:Token: None 2022-09-02T05:57:55.987 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fKwwK+XHQVYe3xQyyP+16j09IRQ= 2022-09-02T05:57:55.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fKwwK+XHQVYe3xQyyP+16j09IRQ=', 'Content-Length': '0'} 2022-09-02T05:57:55.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d89fb71e5b26c8-0063119b63-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:55 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:55.991 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:55.992 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:55.992 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.993 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.993 DEBUG:boto:Method: HEAD 2022-09-02T05:57:55.994 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.995 DEBUG:boto:Data: 2022-09-02T05:57:55.996 DEBUG:boto:Headers: {} 2022-09-02T05:57:55.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:55.997 DEBUG:boto:Port: 8000 2022-09-02T05:57:55.997 DEBUG:boto:Params: {} 2022-09-02T05:57:55.998 DEBUG:boto:Token: None 2022-09-02T05:57:55.999 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:55 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:55.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fKwwK+XHQVYe3xQyyP+16j09IRQ= 2022-09-02T05:57:56.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:55 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fKwwK+XHQVYe3xQyyP+16j09IRQ=', 'Content-Length': '0'} 2022-09-02T05:57:56.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094f6f28bed8224cf-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:57:56.004 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:56.004 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:56.005 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.005 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.006 DEBUG:boto:Method: GET 2022-09-02T05:57:56.006 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.007 DEBUG:boto:Data: 2022-09-02T05:57:56.007 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.008 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.009 DEBUG:boto:Params: {} 2022-09-02T05:57:56.009 DEBUG:boto:Token: None 2022-09-02T05:57:56.010 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+06W5wv9XOb43inPUCFZkQIM5kY= 2022-09-02T05:57:56.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+06W5wv9XOb43inPUCFZkQIM5kY=', 'Content-Length': '0'} 2022-09-02T05:57:56.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002190e9c4db83c606-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.015 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:56.015 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:56.016 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.017 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.017 DEBUG:boto:Method: GET 2022-09-02T05:57:56.018 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.018 DEBUG:boto:Data: 2022-09-02T05:57:56.019 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.021 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.021 DEBUG:boto:Params: {} 2022-09-02T05:57:56.022 DEBUG:boto:Token: None 2022-09-02T05:57:56.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:57:56.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+06W5wv9XOb43inPUCFZkQIM5kY= 2022-09-02T05:57:56.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+06W5wv9XOb43inPUCFZkQIM5kY=', 'Content-Length': '0'} 2022-09-02T05:57:56.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776cd5a9b6cd83e2-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.070 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.071 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.071 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.072 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.073 DEBUG:boto:Method: GET 2022-09-02T05:57:56.073 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.074 DEBUG:boto:Data: 2022-09-02T05:57:56.074 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.075 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.076 DEBUG:boto:Params: {} 2022-09-02T05:57:56.077 DEBUG:boto:Token: None 2022-09-02T05:57:56.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091fb8a656a8a531f-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.082 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.083 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.083 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.084 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.084 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.085 DEBUG:boto:Method: GET 2022-09-02T05:57:56.086 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.086 DEBUG:boto:Data: 2022-09-02T05:57:56.087 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.088 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.088 DEBUG:boto:Params: {} 2022-09-02T05:57:56.089 DEBUG:boto:Token: None 2022-09-02T05:57:56.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4dd05c5c1db787a-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.096 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.097 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:56.098 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:56.099 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.099 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:57:56.100 DEBUG:boto:Data: 2022-09-02T05:57:56.100 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.101 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.101 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.102 DEBUG:boto:Params: {} 2022-09-02T05:57:56.103 DEBUG:boto:Token: None 2022-09-02T05:57:56.104 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj68 2022-09-02T05:57:56.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cam5Why+tTM+cVEdzGY9UPNDaMo= 2022-09-02T05:57:56.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cam5Why+tTM+cVEdzGY9UPNDaMo=', 'Content-Length': '0'} 2022-09-02T05:57:56.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538675b3f31a7e4d-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.109 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:56.109 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:56.110 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.110 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:57:56.111 DEBUG:boto:Data: 2022-09-02T05:57:56.111 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.113 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.113 DEBUG:boto:Params: {} 2022-09-02T05:57:56.114 DEBUG:boto:Token: None 2022-09-02T05:57:56.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj68 2022-09-02T05:57:56.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cam5Why+tTM+cVEdzGY9UPNDaMo= 2022-09-02T05:57:56.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cam5Why+tTM+cVEdzGY9UPNDaMo=', 'Content-Length': '0'} 2022-09-02T05:57:56.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869826a8c3c5bbce-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:57:56.119 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:56.120 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:56.120 DEBUG:boto:Method: GET 2022-09-02T05:57:56.121 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:57:56.122 DEBUG:boto:Data: 2022-09-02T05:57:56.122 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.123 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.124 DEBUG:boto:Params: {} 2022-09-02T05:57:56.125 DEBUG:boto:Token: None 2022-09-02T05:57:56.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj68 2022-09-02T05:57:56.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vwyD+XfuuByKoDe8IDQRvB3XwEk= 2022-09-02T05:57:56.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vwyD+XfuuByKoDe8IDQRvB3XwEk=', 'Content-Length': '0'} 2022-09-02T05:57:56.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94ad8875c73bf83-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.129 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:57:56.130 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:57:56.131 DEBUG:boto:Method: GET 2022-09-02T05:57:56.131 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:57:56.132 DEBUG:boto:Data: 2022-09-02T05:57:56.132 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.133 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.134 DEBUG:boto:Params: {} 2022-09-02T05:57:56.135 DEBUG:boto:Token: None 2022-09-02T05:57:56.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj68 2022-09-02T05:57:56.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vwyD+XfuuByKoDe8IDQRvB3XwEk= 2022-09-02T05:57:56.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vwyD+XfuuByKoDe8IDQRvB3XwEk=', 'Content-Length': '0'} 2022-09-02T05:57:56.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d338bae51610b8b-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.140 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.140 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.141 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.141 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.142 DEBUG:boto:Method: GET 2022-09-02T05:57:56.142 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.143 DEBUG:boto:Data: 2022-09-02T05:57:56.144 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.145 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.145 DEBUG:boto:Params: {} 2022-09-02T05:57:56.146 DEBUG:boto:Token: None 2022-09-02T05:57:56.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a366476692310e2-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.151 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.152 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.153 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.153 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.154 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.154 DEBUG:boto:Method: GET 2022-09-02T05:57:56.155 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:57:56.155 DEBUG:boto:Data: 2022-09-02T05:57:56.156 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.157 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.158 DEBUG:boto:Params: {} 2022-09-02T05:57:56.158 DEBUG:boto:Token: None 2022-09-02T05:57:56.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee76977d9df5339-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.206 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:57:56.209 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.209 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.210 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.211 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.212 DEBUG:boto:Method: GET 2022-09-02T05:57:56.213 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.213 DEBUG:boto:Data: 2022-09-02T05:57:56.214 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.215 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.215 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.216 DEBUG:boto:Params: {} 2022-09-02T05:57:56.216 DEBUG:boto:Token: None 2022-09-02T05:57:56.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M= 2022-09-02T05:57:56.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M=', 'Content-Length': '0'} 2022-09-02T05:57:56.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6776491d729b2a-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.221 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.222 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.223 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.223 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.224 DEBUG:boto:Method: GET 2022-09-02T05:57:56.224 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.225 DEBUG:boto:Data: 2022-09-02T05:57:56.225 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.226 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.227 DEBUG:boto:Params: {} 2022-09-02T05:57:56.228 DEBUG:boto:Token: None 2022-09-02T05:57:56.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M= 2022-09-02T05:57:56.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M=', 'Content-Length': '0'} 2022-09-02T05:57:56.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc1f2782523d07f8-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.233 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.233 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.234 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.235 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.236 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.236 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.237 DEBUG:boto:Data: 2022-09-02T05:57:56.238 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.239 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.239 DEBUG:boto:Params: {} 2022-09-02T05:57:56.240 DEBUG:boto:Token: None 2022-09-02T05:57:56.241 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E3cyatMXT34c5fHIuyqLtXAmcGI= 2022-09-02T05:57:56.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E3cyatMXT34c5fHIuyqLtXAmcGI=', 'Content-Length': '0'} 2022-09-02T05:57:56.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe9bf4a16773d14-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.245 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.246 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.246 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.247 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.248 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.248 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.249 DEBUG:boto:Data: 2022-09-02T05:57:56.249 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.250 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.251 DEBUG:boto:Params: {} 2022-09-02T05:57:56.252 DEBUG:boto:Token: None 2022-09-02T05:57:56.252 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E3cyatMXT34c5fHIuyqLtXAmcGI= 2022-09-02T05:57:56.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E3cyatMXT34c5fHIuyqLtXAmcGI=', 'Content-Length': '0'} 2022-09-02T05:57:56.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022169e9c6db4535a-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:57:56.257 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.258 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.259 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.259 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.260 DEBUG:boto:Method: GET 2022-09-02T05:57:56.260 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.261 DEBUG:boto:Data: 2022-09-02T05:57:56.261 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.263 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.263 DEBUG:boto:Params: {} 2022-09-02T05:57:56.264 DEBUG:boto:Token: None 2022-09-02T05:57:56.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M= 2022-09-02T05:57:56.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M=', 'Content-Length': '0'} 2022-09-02T05:57:56.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000392f8d525d10cf4d-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.269 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.270 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.270 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.271 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.271 DEBUG:boto:Method: GET 2022-09-02T05:57:56.272 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.272 DEBUG:boto:Data: 2022-09-02T05:57:56.273 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.274 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.275 DEBUG:boto:Params: {} 2022-09-02T05:57:56.275 DEBUG:boto:Token: None 2022-09-02T05:57:56.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:57:56.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M= 2022-09-02T05:57:56.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q2s52M/PC9lpsS7d1KVyG7KN12M=', 'Content-Length': '0'} 2022-09-02T05:57:56.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e46c9fd79daec4-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.281 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.282 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.283 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.283 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.284 DEBUG:boto:Method: GET 2022-09-02T05:57:56.285 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.285 DEBUG:boto:Data: 2022-09-02T05:57:56.286 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.287 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.287 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.288 DEBUG:boto:Params: {} 2022-09-02T05:57:56.289 DEBUG:boto:Token: None 2022-09-02T05:57:56.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bee1a4b8a48ecae-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.294 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.295 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.296 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.297 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.297 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.298 DEBUG:boto:Method: GET 2022-09-02T05:57:56.299 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.299 DEBUG:boto:Data: 2022-09-02T05:57:56.300 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.301 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.302 DEBUG:boto:Params: {} 2022-09-02T05:57:56.302 DEBUG:boto:Token: None 2022-09-02T05:57:56.303 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a16d99b63f594361-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.310 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.311 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.311 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.312 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.313 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:57:56.313 DEBUG:boto:Data: 2022-09-02T05:57:56.314 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.315 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.315 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.316 DEBUG:boto:Params: {} 2022-09-02T05:57:56.317 DEBUG:boto:Token: None 2022-09-02T05:57:56.317 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69 2022-09-02T05:57:56.318 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:izx47Szg8V4fHMN7xe9D1Kmpg40= 2022-09-02T05:57:56.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:izx47Szg8V4fHMN7xe9D1Kmpg40=', 'Content-Length': '0'} 2022-09-02T05:57:56.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fb32c3cd97b76d-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.322 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.323 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.324 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.324 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:57:56.325 DEBUG:boto:Data: 2022-09-02T05:57:56.326 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.327 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.328 DEBUG:boto:Params: {} 2022-09-02T05:57:56.341 DEBUG:boto:Token: None 2022-09-02T05:57:56.342 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69 2022-09-02T05:57:56.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:izx47Szg8V4fHMN7xe9D1Kmpg40= 2022-09-02T05:57:56.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:izx47Szg8V4fHMN7xe9D1Kmpg40=', 'Content-Length': '0'} 2022-09-02T05:57:56.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f441e303523a3631-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:57:56.347 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.348 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.348 DEBUG:boto:Method: GET 2022-09-02T05:57:56.349 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:57:56.349 DEBUG:boto:Data: 2022-09-02T05:57:56.350 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.350 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.351 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.351 DEBUG:boto:Params: {} 2022-09-02T05:57:56.352 DEBUG:boto:Token: None 2022-09-02T05:57:56.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69 2022-09-02T05:57:56.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9DQs+VGlqabaKIc0KzWukXhKvAs= 2022-09-02T05:57:56.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9DQs+VGlqabaKIc0KzWukXhKvAs=', 'Content-Length': '0'} 2022-09-02T05:57:56.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cfea014ad856476-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.357 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:57:56.358 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:57:56.358 DEBUG:boto:Method: GET 2022-09-02T05:57:56.359 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:57:56.359 DEBUG:boto:Data: 2022-09-02T05:57:56.360 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.361 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.361 DEBUG:boto:Params: {} 2022-09-02T05:57:56.362 DEBUG:boto:Token: None 2022-09-02T05:57:56.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj69 2022-09-02T05:57:56.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9DQs+VGlqabaKIc0KzWukXhKvAs= 2022-09-02T05:57:56.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9DQs+VGlqabaKIc0KzWukXhKvAs=', 'Content-Length': '0'} 2022-09-02T05:57:56.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ec9cca68a4b317-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.366 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.367 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.368 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.368 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.369 DEBUG:boto:Method: GET 2022-09-02T05:57:56.369 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.370 DEBUG:boto:Data: 2022-09-02T05:57:56.370 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.371 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.372 DEBUG:boto:Params: {} 2022-09-02T05:57:56.372 DEBUG:boto:Token: None 2022-09-02T05:57:56.373 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000275b545e1c9328d6-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.377 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.378 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.379 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.379 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.380 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.380 DEBUG:boto:Method: GET 2022-09-02T05:57:56.381 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:57:56.381 DEBUG:boto:Data: 2022-09-02T05:57:56.382 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.383 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.383 DEBUG:boto:Params: {} 2022-09-02T05:57:56.384 DEBUG:boto:Token: None 2022-09-02T05:57:56.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ae4a2bbc586e4f-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.390 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:57:56.392 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.392 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.393 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.393 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.394 DEBUG:boto:Method: GET 2022-09-02T05:57:56.394 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.395 DEBUG:boto:Data: 2022-09-02T05:57:56.395 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.397 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.397 DEBUG:boto:Params: {} 2022-09-02T05:57:56.398 DEBUG:boto:Token: None 2022-09-02T05:57:56.398 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.399 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc= 2022-09-02T05:57:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc=', 'Content-Length': '0'} 2022-09-02T05:57:56.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000932c247e217168ec-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.402 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.403 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.403 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.404 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.404 DEBUG:boto:Method: GET 2022-09-02T05:57:56.405 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.405 DEBUG:boto:Data: 2022-09-02T05:57:56.406 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.407 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.407 DEBUG:boto:Params: {} 2022-09-02T05:57:56.408 DEBUG:boto:Token: None 2022-09-02T05:57:56.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc= 2022-09-02T05:57:56.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc=', 'Content-Length': '0'} 2022-09-02T05:57:56.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000524b7fbd6c324049-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.413 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.413 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.414 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.414 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.415 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.415 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.416 DEBUG:boto:Data: 2022-09-02T05:57:56.416 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.417 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.417 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.418 DEBUG:boto:Params: {} 2022-09-02T05:57:56.418 DEBUG:boto:Token: None 2022-09-02T05:57:56.419 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w4Dc9LRR7UD2sEeJ+pLGFsTD1dU= 2022-09-02T05:57:56.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w4Dc9LRR7UD2sEeJ+pLGFsTD1dU=', 'Content-Length': '0'} 2022-09-02T05:57:56.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5ebe17c4ac5a62-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.423 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.423 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.424 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.424 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.425 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.425 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.426 DEBUG:boto:Data: 2022-09-02T05:57:56.426 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.427 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.428 DEBUG:boto:Params: {} 2022-09-02T05:57:56.428 DEBUG:boto:Token: None 2022-09-02T05:57:56.429 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w4Dc9LRR7UD2sEeJ+pLGFsTD1dU= 2022-09-02T05:57:56.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w4Dc9LRR7UD2sEeJ+pLGFsTD1dU=', 'Content-Length': '0'} 2022-09-02T05:57:56.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c0bc07a7f0887b-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:57:56.433 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.434 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.434 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.435 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.435 DEBUG:boto:Method: GET 2022-09-02T05:57:56.436 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.436 DEBUG:boto:Data: 2022-09-02T05:57:56.437 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.438 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.438 DEBUG:boto:Params: {} 2022-09-02T05:57:56.439 DEBUG:boto:Token: None 2022-09-02T05:57:56.440 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc= 2022-09-02T05:57:56.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc=', 'Content-Length': '0'} 2022-09-02T05:57:56.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b2d97e64fe94a11-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.486 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.487 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.488 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.488 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.489 DEBUG:boto:Method: GET 2022-09-02T05:57:56.490 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.490 DEBUG:boto:Data: 2022-09-02T05:57:56.491 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.492 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.493 DEBUG:boto:Params: {} 2022-09-02T05:57:56.493 DEBUG:boto:Token: None 2022-09-02T05:57:56.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:57:56.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc= 2022-09-02T05:57:56.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zGcAFZBH1aVwZ2MbjrJn42kuPjc=', 'Content-Length': '0'} 2022-09-02T05:57:56.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ca69ee5ebb6b5c-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.499 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.499 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.500 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.501 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.501 DEBUG:boto:Method: GET 2022-09-02T05:57:56.502 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.503 DEBUG:boto:Data: 2022-09-02T05:57:56.503 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.504 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.504 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.505 DEBUG:boto:Params: {} 2022-09-02T05:57:56.505 DEBUG:boto:Token: None 2022-09-02T05:57:56.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c41c2ca51b4256-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.511 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.512 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.512 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.513 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.513 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.514 DEBUG:boto:Method: GET 2022-09-02T05:57:56.514 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.515 DEBUG:boto:Data: 2022-09-02T05:57:56.515 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.516 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.517 DEBUG:boto:Params: {} 2022-09-02T05:57:56.517 DEBUG:boto:Token: None 2022-09-02T05:57:56.518 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.519 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd9be1f433d8009-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.524 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.525 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.526 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.526 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.527 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:57:56.527 DEBUG:boto:Data: 2022-09-02T05:57:56.528 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.529 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.529 DEBUG:boto:Params: {} 2022-09-02T05:57:56.530 DEBUG:boto:Token: None 2022-09-02T05:57:56.530 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76 2022-09-02T05:57:56.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0qtNmzlHT8DfwT9hGBJbzUzEe5E= 2022-09-02T05:57:56.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0qtNmzlHT8DfwT9hGBJbzUzEe5E=', 'Content-Length': '0'} 2022-09-02T05:57:56.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001489e7e18185c266-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.534 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.535 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.536 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.536 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:57:56.537 DEBUG:boto:Data: 2022-09-02T05:57:56.537 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.538 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.539 DEBUG:boto:Params: {} 2022-09-02T05:57:56.539 DEBUG:boto:Token: None 2022-09-02T05:57:56.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76 2022-09-02T05:57:56.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0qtNmzlHT8DfwT9hGBJbzUzEe5E= 2022-09-02T05:57:56.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0qtNmzlHT8DfwT9hGBJbzUzEe5E=', 'Content-Length': '0'} 2022-09-02T05:57:56.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1aadfbd0be5cb2f-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:57:56.544 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.545 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.545 DEBUG:boto:Method: GET 2022-09-02T05:57:56.546 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:57:56.546 DEBUG:boto:Data: 2022-09-02T05:57:56.547 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.547 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.548 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.548 DEBUG:boto:Params: {} 2022-09-02T05:57:56.549 DEBUG:boto:Token: None 2022-09-02T05:57:56.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76 2022-09-02T05:57:56.550 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SQzEIRzPGmq4bJ0qSV2RDS2+3cg= 2022-09-02T05:57:56.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SQzEIRzPGmq4bJ0qSV2RDS2+3cg=', 'Content-Length': '0'} 2022-09-02T05:57:56.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6f3d62fa931e0f-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.554 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:57:56.554 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:57:56.555 DEBUG:boto:Method: GET 2022-09-02T05:57:56.555 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:57:56.556 DEBUG:boto:Data: 2022-09-02T05:57:56.556 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.557 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.558 DEBUG:boto:Params: {} 2022-09-02T05:57:56.558 DEBUG:boto:Token: None 2022-09-02T05:57:56.559 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj76 2022-09-02T05:57:56.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SQzEIRzPGmq4bJ0qSV2RDS2+3cg= 2022-09-02T05:57:56.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SQzEIRzPGmq4bJ0qSV2RDS2+3cg=', 'Content-Length': '0'} 2022-09-02T05:57:56.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d026487dc236930-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.606 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.607 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.607 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.608 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.609 DEBUG:boto:Method: GET 2022-09-02T05:57:56.609 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.610 DEBUG:boto:Data: 2022-09-02T05:57:56.610 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.611 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.612 DEBUG:boto:Params: {} 2022-09-02T05:57:56.612 DEBUG:boto:Token: None 2022-09-02T05:57:56.613 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de76335cca76e48f-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.618 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.619 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.619 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.620 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.620 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.621 DEBUG:boto:Method: GET 2022-09-02T05:57:56.621 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:57:56.622 DEBUG:boto:Data: 2022-09-02T05:57:56.622 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.623 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.624 DEBUG:boto:Params: {} 2022-09-02T05:57:56.625 DEBUG:boto:Token: None 2022-09-02T05:57:56.625 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aca3e387c4da56e-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.631 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:57:56.632 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.633 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.633 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.634 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.635 DEBUG:boto:Method: GET 2022-09-02T05:57:56.635 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.636 DEBUG:boto:Data: 2022-09-02T05:57:56.636 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.637 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.638 DEBUG:boto:Params: {} 2022-09-02T05:57:56.638 DEBUG:boto:Token: None 2022-09-02T05:57:56.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg= 2022-09-02T05:57:56.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg=', 'Content-Length': '0'} 2022-09-02T05:57:56.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3df7659182b2a8a-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.643 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.644 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.645 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.645 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.646 DEBUG:boto:Method: GET 2022-09-02T05:57:56.646 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.647 DEBUG:boto:Data: 2022-09-02T05:57:56.647 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.648 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.649 DEBUG:boto:Params: {} 2022-09-02T05:57:56.650 DEBUG:boto:Token: None 2022-09-02T05:57:56.650 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg= 2022-09-02T05:57:56.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg=', 'Content-Length': '0'} 2022-09-02T05:57:56.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072262135020575a6-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.655 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.655 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.656 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.656 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.657 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.657 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.658 DEBUG:boto:Data: 2022-09-02T05:57:56.658 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.659 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.659 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.660 DEBUG:boto:Params: {} 2022-09-02T05:57:56.660 DEBUG:boto:Token: None 2022-09-02T05:57:56.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qZ+g2JB8eLoIlC2pr58HM6DowLA= 2022-09-02T05:57:56.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qZ+g2JB8eLoIlC2pr58HM6DowLA=', 'Content-Length': '0'} 2022-09-02T05:57:56.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b244ed85cd7afec8-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.665 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.666 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.666 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.667 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.667 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.668 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.668 DEBUG:boto:Data: 2022-09-02T05:57:56.669 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.670 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.670 DEBUG:boto:Params: {} 2022-09-02T05:57:56.671 DEBUG:boto:Token: None 2022-09-02T05:57:56.672 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qZ+g2JB8eLoIlC2pr58HM6DowLA= 2022-09-02T05:57:56.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qZ+g2JB8eLoIlC2pr58HM6DowLA=', 'Content-Length': '0'} 2022-09-02T05:57:56.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009faefa2eb29ccdec-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:57:56.676 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.677 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.677 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.678 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.679 DEBUG:boto:Method: GET 2022-09-02T05:57:56.679 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.680 DEBUG:boto:Data: 2022-09-02T05:57:56.680 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.681 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.682 DEBUG:boto:Params: {} 2022-09-02T05:57:56.682 DEBUG:boto:Token: None 2022-09-02T05:57:56.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg= 2022-09-02T05:57:56.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg=', 'Content-Length': '0'} 2022-09-02T05:57:56.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064344348358af9d6-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.730 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.731 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.731 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.732 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.732 DEBUG:boto:Method: GET 2022-09-02T05:57:56.733 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.734 DEBUG:boto:Data: 2022-09-02T05:57:56.736 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.737 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.738 DEBUG:boto:Params: {} 2022-09-02T05:57:56.738 DEBUG:boto:Token: None 2022-09-02T05:57:56.739 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:57:56.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg= 2022-09-02T05:57:56.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sV/eZjo/xb2KEyGNIFTk4DbRbtg=', 'Content-Length': '0'} 2022-09-02T05:57:56.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ea4be83c91edb3-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.744 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.744 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.745 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.745 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.746 DEBUG:boto:Method: GET 2022-09-02T05:57:56.746 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.747 DEBUG:boto:Data: 2022-09-02T05:57:56.747 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.749 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.749 DEBUG:boto:Params: {} 2022-09-02T05:57:56.750 DEBUG:boto:Token: None 2022-09-02T05:57:56.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336b752a5a0555d3-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.756 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.757 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.759 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.762 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.763 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.763 DEBUG:boto:Method: GET 2022-09-02T05:57:56.764 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.765 DEBUG:boto:Data: 2022-09-02T05:57:56.765 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.766 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.767 DEBUG:boto:Params: {} 2022-09-02T05:57:56.767 DEBUG:boto:Token: None 2022-09-02T05:57:56.768 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b38fa02e66a2837-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.774 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.775 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.776 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.777 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.777 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:57:56.778 DEBUG:boto:Data: 2022-09-02T05:57:56.778 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.779 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.780 DEBUG:boto:Params: {} 2022-09-02T05:57:56.780 DEBUG:boto:Token: None 2022-09-02T05:57:56.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85 2022-09-02T05:57:56.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oPKTiZQ5Oh6melhjoYuz26QKTlY= 2022-09-02T05:57:56.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oPKTiZQ5Oh6melhjoYuz26QKTlY=', 'Content-Length': '0'} 2022-09-02T05:57:56.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed48a9cec5e986b7-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.785 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.785 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.786 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.786 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:57:56.787 DEBUG:boto:Data: 2022-09-02T05:57:56.787 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.789 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.789 DEBUG:boto:Params: {} 2022-09-02T05:57:56.790 DEBUG:boto:Token: None 2022-09-02T05:57:56.791 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85 2022-09-02T05:57:56.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oPKTiZQ5Oh6melhjoYuz26QKTlY= 2022-09-02T05:57:56.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oPKTiZQ5Oh6melhjoYuz26QKTlY=', 'Content-Length': '0'} 2022-09-02T05:57:56.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035e3fef9bf625cd7-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:57:56.797 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.798 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.799 DEBUG:boto:Method: GET 2022-09-02T05:57:56.800 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:57:56.801 DEBUG:boto:Data: 2022-09-02T05:57:56.802 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.803 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.804 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.804 DEBUG:boto:Params: {} 2022-09-02T05:57:56.805 DEBUG:boto:Token: None 2022-09-02T05:57:56.806 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85 2022-09-02T05:57:56.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dYfIY2H6TuIdWdAwwqHguPw73RE= 2022-09-02T05:57:56.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dYfIY2H6TuIdWdAwwqHguPw73RE=', 'Content-Length': '0'} 2022-09-02T05:57:56.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c34e358b7a0535a5-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.811 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:57:56.811 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:57:56.812 DEBUG:boto:Method: GET 2022-09-02T05:57:56.812 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:57:56.813 DEBUG:boto:Data: 2022-09-02T05:57:56.813 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.814 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.815 DEBUG:boto:Params: {} 2022-09-02T05:57:56.816 DEBUG:boto:Token: None 2022-09-02T05:57:56.816 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj85 2022-09-02T05:57:56.817 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dYfIY2H6TuIdWdAwwqHguPw73RE= 2022-09-02T05:57:56.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dYfIY2H6TuIdWdAwwqHguPw73RE=', 'Content-Length': '0'} 2022-09-02T05:57:56.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000871e075fec7697e3-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.821 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.821 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.822 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.823 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.823 DEBUG:boto:Method: GET 2022-09-02T05:57:56.824 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.824 DEBUG:boto:Data: 2022-09-02T05:57:56.825 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.825 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.826 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.827 DEBUG:boto:Params: {} 2022-09-02T05:57:56.827 DEBUG:boto:Token: None 2022-09-02T05:57:56.828 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.828 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f3ddc102f7e6ff7-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.833 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.833 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.834 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.835 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.835 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.836 DEBUG:boto:Method: GET 2022-09-02T05:57:56.836 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:57:56.837 DEBUG:boto:Data: 2022-09-02T05:57:56.837 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.838 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.839 DEBUG:boto:Params: {} 2022-09-02T05:57:56.839 DEBUG:boto:Token: None 2022-09-02T05:57:56.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b182a1b72886272a-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.846 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:57:56.848 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.848 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.849 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.850 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.850 DEBUG:boto:Method: GET 2022-09-02T05:57:56.851 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.851 DEBUG:boto:Data: 2022-09-02T05:57:56.852 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.853 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.853 DEBUG:boto:Params: {} 2022-09-02T05:57:56.854 DEBUG:boto:Token: None 2022-09-02T05:57:56.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw= 2022-09-02T05:57:56.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw=', 'Content-Length': '0'} 2022-09-02T05:57:56.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef629056a25dc42f-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.859 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.860 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.860 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.861 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.861 DEBUG:boto:Method: GET 2022-09-02T05:57:56.862 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.862 DEBUG:boto:Data: 2022-09-02T05:57:56.863 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.864 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.864 DEBUG:boto:Params: {} 2022-09-02T05:57:56.865 DEBUG:boto:Token: None 2022-09-02T05:57:56.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw= 2022-09-02T05:57:56.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw=', 'Content-Length': '0'} 2022-09-02T05:57:56.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bffcf0c7ee6a413-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.870 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.871 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.871 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.872 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.872 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.873 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.873 DEBUG:boto:Data: 2022-09-02T05:57:56.874 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.874 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.875 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.876 DEBUG:boto:Params: {} 2022-09-02T05:57:56.876 DEBUG:boto:Token: None 2022-09-02T05:57:56.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KAtC1/iqHrB8Kh5KDfenYHVkLqM= 2022-09-02T05:57:56.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KAtC1/iqHrB8Kh5KDfenYHVkLqM=', 'Content-Length': '0'} 2022-09-02T05:57:56.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86fdc6c6e24fd6b-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.882 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.883 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.883 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.884 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.884 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.885 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.885 DEBUG:boto:Data: 2022-09-02T05:57:56.886 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.887 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.888 DEBUG:boto:Params: {} 2022-09-02T05:57:56.888 DEBUG:boto:Token: None 2022-09-02T05:57:56.889 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KAtC1/iqHrB8Kh5KDfenYHVkLqM= 2022-09-02T05:57:56.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KAtC1/iqHrB8Kh5KDfenYHVkLqM=', 'Content-Length': '0'} 2022-09-02T05:57:56.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35f30da4f8757f6-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:57:56.894 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.895 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.896 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.896 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.897 DEBUG:boto:Method: GET 2022-09-02T05:57:56.897 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.898 DEBUG:boto:Data: 2022-09-02T05:57:56.899 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.900 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.900 DEBUG:boto:Params: {} 2022-09-02T05:57:56.901 DEBUG:boto:Token: None 2022-09-02T05:57:56.902 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw= 2022-09-02T05:57:56.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw=', 'Content-Length': '0'} 2022-09-02T05:57:56.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0f3ce38a903eca6-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.946 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.947 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.948 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.948 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.949 DEBUG:boto:Method: GET 2022-09-02T05:57:56.949 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.950 DEBUG:boto:Data: 2022-09-02T05:57:56.950 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.951 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.952 DEBUG:boto:Params: {} 2022-09-02T05:57:56.952 DEBUG:boto:Token: None 2022-09-02T05:57:56.953 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:57:56.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw= 2022-09-02T05:57:56.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+9lsZEslrU2Q61TzlyjUGB2l7Vw=', 'Content-Length': '0'} 2022-09-02T05:57:56.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0d4f7981867fcb-0063119b64-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.958 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.958 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.959 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:56.960 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:56.960 DEBUG:boto:Method: GET 2022-09-02T05:57:56.961 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:57:56.961 DEBUG:boto:Data: 2022-09-02T05:57:56.962 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.962 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.963 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.964 DEBUG:boto:Params: {} 2022-09-02T05:57:56.965 DEBUG:boto:Token: None 2022-09-02T05:57:56.965 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbb88d3b73a1b790-0063119b64-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.970 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.971 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:56.972 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:56.972 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:56.973 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:56.974 DEBUG:boto:Method: GET 2022-09-02T05:57:56.975 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:57:56.976 DEBUG:boto:Data: 2022-09-02T05:57:56.977 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:56.979 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.979 DEBUG:boto:Params: {} 2022-09-02T05:57:56.980 DEBUG:boto:Token: None 2022-09-02T05:57:56.981 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/ 2022-09-02T05:57:56.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I= 2022-09-02T05:57:56.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:39+snAUIWRSpmlVycGzGFuajf2I=', 'Content-Length': '0'} 2022-09-02T05:57:56.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5674b0aeb45ad27-0063119b64-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.987 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:56.988 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.988 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:56.989 DEBUG:boto:Method: HEAD 2022-09-02T05:57:56.990 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:57:56.990 DEBUG:boto:Data: 2022-09-02T05:57:56.991 DEBUG:boto:Headers: {} 2022-09-02T05:57:56.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:56.992 DEBUG:boto:Port: 8000 2022-09-02T05:57:56.993 DEBUG:boto:Params: {} 2022-09-02T05:57:56.994 DEBUG:boto:Token: None 2022-09-02T05:57:56.994 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:56 GMT /xljbkf-74/obj87 2022-09-02T05:57:56.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qr3oYEJ179BBLmrB/OHrQ0OFcoA= 2022-09-02T05:57:56.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:56 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qr3oYEJ179BBLmrB/OHrQ0OFcoA=', 'Content-Length': '0'} 2022-09-02T05:57:56.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020579da0aefd565d-0063119b64-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:56 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:56.999 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:56.999 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:57.000 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.000 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:57:57.001 DEBUG:boto:Data: 2022-09-02T05:57:57.001 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.002 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.003 DEBUG:boto:Params: {} 2022-09-02T05:57:57.003 DEBUG:boto:Token: None 2022-09-02T05:57:57.004 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj87 2022-09-02T05:57:57.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wca8pFYEETYml38Vxi+Iy1pj41k= 2022-09-02T05:57:57.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wca8pFYEETYml38Vxi+Iy1pj41k=', 'Content-Length': '0'} 2022-09-02T05:57:57.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eec89615aaef8d4-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:57:57.009 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:57.009 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:57.010 DEBUG:boto:Method: GET 2022-09-02T05:57:57.011 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:57:57.011 DEBUG:boto:Data: 2022-09-02T05:57:57.012 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.013 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.014 DEBUG:boto:Params: {} 2022-09-02T05:57:57.014 DEBUG:boto:Token: None 2022-09-02T05:57:57.015 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj87 2022-09-02T05:57:57.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uHcO+x9gVQlVJNKCeTG4E9iD7Dw= 2022-09-02T05:57:57.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uHcO+x9gVQlVJNKCeTG4E9iD7Dw=', 'Content-Length': '0'} 2022-09-02T05:57:57.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2caed8ae6393db4-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.019 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:57:57.020 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:57:57.020 DEBUG:boto:Method: GET 2022-09-02T05:57:57.021 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:57:57.022 DEBUG:boto:Data: 2022-09-02T05:57:57.022 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.024 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.024 DEBUG:boto:Params: {} 2022-09-02T05:57:57.025 DEBUG:boto:Token: None 2022-09-02T05:57:57.026 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj87 2022-09-02T05:57:57.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uHcO+x9gVQlVJNKCeTG4E9iD7Dw= 2022-09-02T05:57:57.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uHcO+x9gVQlVJNKCeTG4E9iD7Dw=', 'Content-Length': '0'} 2022-09-02T05:57:57.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ffc0a36f815ddd3-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.071 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.071 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.072 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:57.072 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:57.073 DEBUG:boto:Method: GET 2022-09-02T05:57:57.073 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:57:57.074 DEBUG:boto:Data: 2022-09-02T05:57:57.074 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.075 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.075 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.076 DEBUG:boto:Params: {} 2022-09-02T05:57:57.077 DEBUG:boto:Token: None 2022-09-02T05:57:57.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4abe766d12106c-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.082 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.083 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.083 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.084 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:57.084 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:57:57.085 DEBUG:boto:Method: GET 2022-09-02T05:57:57.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:57:57.086 DEBUG:boto:Data: 2022-09-02T05:57:57.086 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.091 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.091 DEBUG:boto:Params: {} 2022-09-02T05:57:57.092 DEBUG:boto:Token: None 2022-09-02T05:57:57.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e15e7a727dfbb5e7-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.099 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:57:57.100 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.101 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.102 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.102 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.103 DEBUG:boto:Method: GET 2022-09-02T05:57:57.103 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.104 DEBUG:boto:Data: 2022-09-02T05:57:57.104 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.106 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.106 DEBUG:boto:Params: {} 2022-09-02T05:57:57.107 DEBUG:boto:Token: None 2022-09-02T05:57:57.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU= 2022-09-02T05:57:57.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU=', 'Content-Length': '0'} 2022-09-02T05:57:57.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c50b044b9a3bcb-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.112 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.113 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.113 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.114 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.115 DEBUG:boto:Method: GET 2022-09-02T05:57:57.115 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.116 DEBUG:boto:Data: 2022-09-02T05:57:57.116 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.118 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.118 DEBUG:boto:Params: {} 2022-09-02T05:57:57.119 DEBUG:boto:Token: None 2022-09-02T05:57:57.120 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.120 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU= 2022-09-02T05:57:57.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU=', 'Content-Length': '0'} 2022-09-02T05:57:57.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359b8ad016ff4a68-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.124 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.125 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.125 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.126 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.126 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.127 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.127 DEBUG:boto:Data: 2022-09-02T05:57:57.128 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.128 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.129 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.129 DEBUG:boto:Params: {} 2022-09-02T05:57:57.130 DEBUG:boto:Token: None 2022-09-02T05:57:57.130 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jsfz9IOJsYHotnbrtljkRBTqN9o= 2022-09-02T05:57:57.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jsfz9IOJsYHotnbrtljkRBTqN9o=', 'Content-Length': '0'} 2022-09-02T05:57:57.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91f64c993a909d2-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.135 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.135 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.136 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.136 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.136 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.137 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.137 DEBUG:boto:Data: 2022-09-02T05:57:57.138 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.139 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.140 DEBUG:boto:Params: {} 2022-09-02T05:57:57.140 DEBUG:boto:Token: None 2022-09-02T05:57:57.141 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jsfz9IOJsYHotnbrtljkRBTqN9o= 2022-09-02T05:57:57.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jsfz9IOJsYHotnbrtljkRBTqN9o=', 'Content-Length': '0'} 2022-09-02T05:57:57.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5fbb07da648b76-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:57:57.148 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.148 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.149 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.150 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.150 DEBUG:boto:Method: GET 2022-09-02T05:57:57.151 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.152 DEBUG:boto:Data: 2022-09-02T05:57:57.152 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.153 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.153 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.154 DEBUG:boto:Params: {} 2022-09-02T05:57:57.154 DEBUG:boto:Token: None 2022-09-02T05:57:57.155 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU= 2022-09-02T05:57:57.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU=', 'Content-Length': '0'} 2022-09-02T05:57:57.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a51f699ecfb42a-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.161 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.162 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.162 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.163 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.163 DEBUG:boto:Method: GET 2022-09-02T05:57:57.164 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.164 DEBUG:boto:Data: 2022-09-02T05:57:57.165 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.166 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.166 DEBUG:boto:Params: {} 2022-09-02T05:57:57.167 DEBUG:boto:Token: None 2022-09-02T05:57:57.168 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:57:57.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU= 2022-09-02T05:57:57.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MG+pJNEpYZbmUgnb+dck9IWejcU=', 'Content-Length': '0'} 2022-09-02T05:57:57.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008a250660e0fb9fd-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.214 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.215 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.216 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.216 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.217 DEBUG:boto:Method: GET 2022-09-02T05:57:57.218 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.219 DEBUG:boto:Data: 2022-09-02T05:57:57.220 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.221 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.222 DEBUG:boto:Params: {} 2022-09-02T05:57:57.223 DEBUG:boto:Token: None 2022-09-02T05:57:57.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.225 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd07322c52c41572-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.230 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.231 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.232 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.232 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.233 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.233 DEBUG:boto:Method: GET 2022-09-02T05:57:57.234 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.235 DEBUG:boto:Data: 2022-09-02T05:57:57.235 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.236 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.237 DEBUG:boto:Params: {} 2022-09-02T05:57:57.237 DEBUG:boto:Token: None 2022-09-02T05:57:57.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9ddafc80b2e051-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.244 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.244 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.245 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.245 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.246 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:57:57.247 DEBUG:boto:Data: 2022-09-02T05:57:57.247 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.248 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.248 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.249 DEBUG:boto:Params: {} 2022-09-02T05:57:57.249 DEBUG:boto:Token: None 2022-09-02T05:57:57.250 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92 2022-09-02T05:57:57.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uORwo0c16ef1K7tcAWo2wJn/j04= 2022-09-02T05:57:57.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uORwo0c16ef1K7tcAWo2wJn/j04=', 'Content-Length': '0'} 2022-09-02T05:57:57.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df48e0c4d6193d77-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.254 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.255 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.255 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.256 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:57:57.256 DEBUG:boto:Data: 2022-09-02T05:57:57.257 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.258 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.258 DEBUG:boto:Params: {} 2022-09-02T05:57:57.259 DEBUG:boto:Token: None 2022-09-02T05:57:57.259 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92 2022-09-02T05:57:57.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uORwo0c16ef1K7tcAWo2wJn/j04= 2022-09-02T05:57:57.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uORwo0c16ef1K7tcAWo2wJn/j04=', 'Content-Length': '0'} 2022-09-02T05:57:57.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010866b488fc16f0e-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:57:57.264 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.264 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.265 DEBUG:boto:Method: GET 2022-09-02T05:57:57.266 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:57:57.266 DEBUG:boto:Data: 2022-09-02T05:57:57.267 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.267 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.268 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.268 DEBUG:boto:Params: {} 2022-09-02T05:57:57.269 DEBUG:boto:Token: None 2022-09-02T05:57:57.270 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92 2022-09-02T05:57:57.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vVy3qn/+K/KgxnJ/P7cU0B2fxns= 2022-09-02T05:57:57.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vVy3qn/+K/KgxnJ/P7cU0B2fxns=', 'Content-Length': '0'} 2022-09-02T05:57:57.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d065c1ca4f9d6ce-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.276 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:57:57.276 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:57:57.277 DEBUG:boto:Method: GET 2022-09-02T05:57:57.278 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:57:57.278 DEBUG:boto:Data: 2022-09-02T05:57:57.279 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.280 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.281 DEBUG:boto:Params: {} 2022-09-02T05:57:57.282 DEBUG:boto:Token: None 2022-09-02T05:57:57.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj92 2022-09-02T05:57:57.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vVy3qn/+K/KgxnJ/P7cU0B2fxns= 2022-09-02T05:57:57.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vVy3qn/+K/KgxnJ/P7cU0B2fxns=', 'Content-Length': '0'} 2022-09-02T05:57:57.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f721ddedc72f16-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.326 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.327 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.327 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.328 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.328 DEBUG:boto:Method: GET 2022-09-02T05:57:57.329 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.330 DEBUG:boto:Data: 2022-09-02T05:57:57.330 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.331 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.332 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.332 DEBUG:boto:Params: {} 2022-09-02T05:57:57.333 DEBUG:boto:Token: None 2022-09-02T05:57:57.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdab0cf2fb157ab1-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.338 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.339 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.340 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.340 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.341 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.342 DEBUG:boto:Method: GET 2022-09-02T05:57:57.343 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:57:57.344 DEBUG:boto:Data: 2022-09-02T05:57:57.345 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.346 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.346 DEBUG:boto:Params: {} 2022-09-02T05:57:57.347 DEBUG:boto:Token: None 2022-09-02T05:57:57.348 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0291bb6d5141e51-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.355 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:57:57.357 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.357 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.358 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.358 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.359 DEBUG:boto:Method: GET 2022-09-02T05:57:57.359 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.360 DEBUG:boto:Data: 2022-09-02T05:57:57.360 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.361 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.361 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.362 DEBUG:boto:Params: {} 2022-09-02T05:57:57.363 DEBUG:boto:Token: None 2022-09-02T05:57:57.363 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.364 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M= 2022-09-02T05:57:57.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M=', 'Content-Length': '0'} 2022-09-02T05:57:57.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b30dc4f3cdc68f6-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.367 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.368 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.369 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.369 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.370 DEBUG:boto:Method: GET 2022-09-02T05:57:57.370 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.371 DEBUG:boto:Data: 2022-09-02T05:57:57.371 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.372 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.373 DEBUG:boto:Params: {} 2022-09-02T05:57:57.374 DEBUG:boto:Token: None 2022-09-02T05:57:57.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M= 2022-09-02T05:57:57.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M=', 'Content-Length': '0'} 2022-09-02T05:57:57.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4accdf20765632-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.379 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.379 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.380 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.380 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.381 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.381 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.382 DEBUG:boto:Data: 2022-09-02T05:57:57.383 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.384 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.385 DEBUG:boto:Params: {} 2022-09-02T05:57:57.385 DEBUG:boto:Token: None 2022-09-02T05:57:57.386 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dFjG6foeHa6KeRrwYiBi3kK3owU= 2022-09-02T05:57:57.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dFjG6foeHa6KeRrwYiBi3kK3owU=', 'Content-Length': '0'} 2022-09-02T05:57:57.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000980dba81fa6ad716-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.391 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.392 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.392 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.393 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.393 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.394 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.394 DEBUG:boto:Data: 2022-09-02T05:57:57.395 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.396 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.396 DEBUG:boto:Params: {} 2022-09-02T05:57:57.397 DEBUG:boto:Token: None 2022-09-02T05:57:57.398 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dFjG6foeHa6KeRrwYiBi3kK3owU= 2022-09-02T05:57:57.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dFjG6foeHa6KeRrwYiBi3kK3owU=', 'Content-Length': '0'} 2022-09-02T05:57:57.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092b5007a0792f34a-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:57:57.403 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.403 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.404 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.404 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.405 DEBUG:boto:Method: GET 2022-09-02T05:57:57.405 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.406 DEBUG:boto:Data: 2022-09-02T05:57:57.406 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.407 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.407 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.408 DEBUG:boto:Params: {} 2022-09-02T05:57:57.409 DEBUG:boto:Token: None 2022-09-02T05:57:57.409 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.410 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M= 2022-09-02T05:57:57.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M=', 'Content-Length': '0'} 2022-09-02T05:57:57.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd84b449b0d30e1-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.454 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.455 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.455 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.456 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.456 DEBUG:boto:Method: GET 2022-09-02T05:57:57.457 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.458 DEBUG:boto:Data: 2022-09-02T05:57:57.458 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.459 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.460 DEBUG:boto:Params: {} 2022-09-02T05:57:57.460 DEBUG:boto:Token: None 2022-09-02T05:57:57.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:57:57.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M= 2022-09-02T05:57:57.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:agbNSFZLgKrXPGADxShlhVzlp6M=', 'Content-Length': '0'} 2022-09-02T05:57:57.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d4d9350cad0c0c6-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.466 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.466 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.467 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.467 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.468 DEBUG:boto:Method: GET 2022-09-02T05:57:57.468 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.469 DEBUG:boto:Data: 2022-09-02T05:57:57.470 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.471 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.471 DEBUG:boto:Params: {} 2022-09-02T05:57:57.472 DEBUG:boto:Token: None 2022-09-02T05:57:57.472 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7202e9147246187-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.477 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.478 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.479 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.479 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.480 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.480 DEBUG:boto:Method: GET 2022-09-02T05:57:57.481 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.482 DEBUG:boto:Data: 2022-09-02T05:57:57.482 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.483 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.484 DEBUG:boto:Params: {} 2022-09-02T05:57:57.485 DEBUG:boto:Token: None 2022-09-02T05:57:57.485 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de33060256e202c-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.492 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.493 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.494 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.494 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.495 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:57:57.495 DEBUG:boto:Data: 2022-09-02T05:57:57.496 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.497 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.497 DEBUG:boto:Params: {} 2022-09-02T05:57:57.498 DEBUG:boto:Token: None 2022-09-02T05:57:57.499 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93 2022-09-02T05:57:57.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TO7rmIw4evrvrnKpcpA2XFslkiA= 2022-09-02T05:57:57.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TO7rmIw4evrvrnKpcpA2XFslkiA=', 'Content-Length': '0'} 2022-09-02T05:57:57.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3fce920726c9cd-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.505 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.506 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.506 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.507 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:57:57.508 DEBUG:boto:Data: 2022-09-02T05:57:57.509 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.510 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.510 DEBUG:boto:Params: {} 2022-09-02T05:57:57.511 DEBUG:boto:Token: None 2022-09-02T05:57:57.512 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93 2022-09-02T05:57:57.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TO7rmIw4evrvrnKpcpA2XFslkiA= 2022-09-02T05:57:57.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TO7rmIw4evrvrnKpcpA2XFslkiA=', 'Content-Length': '0'} 2022-09-02T05:57:57.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8c8ea29b9e9f8df-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:57:57.516 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.517 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.518 DEBUG:boto:Method: GET 2022-09-02T05:57:57.519 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:57:57.519 DEBUG:boto:Data: 2022-09-02T05:57:57.520 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.521 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.521 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.522 DEBUG:boto:Params: {} 2022-09-02T05:57:57.522 DEBUG:boto:Token: None 2022-09-02T05:57:57.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93 2022-09-02T05:57:57.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5Fwc3Dvf/SLfGsF5CJiJ83YVQIo= 2022-09-02T05:57:57.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5Fwc3Dvf/SLfGsF5CJiJ83YVQIo=', 'Content-Length': '0'} 2022-09-02T05:57:57.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002205fc7a65ee0f75-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.529 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:57:57.529 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:57:57.530 DEBUG:boto:Method: GET 2022-09-02T05:57:57.531 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:57:57.532 DEBUG:boto:Data: 2022-09-02T05:57:57.533 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.534 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.535 DEBUG:boto:Params: {} 2022-09-02T05:57:57.536 DEBUG:boto:Token: None 2022-09-02T05:57:57.537 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj93 2022-09-02T05:57:57.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5Fwc3Dvf/SLfGsF5CJiJ83YVQIo= 2022-09-02T05:57:57.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5Fwc3Dvf/SLfGsF5CJiJ83YVQIo=', 'Content-Length': '0'} 2022-09-02T05:57:57.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb093d0d2aed0ffa-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.586 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.587 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.587 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.588 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.589 DEBUG:boto:Method: GET 2022-09-02T05:57:57.589 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.590 DEBUG:boto:Data: 2022-09-02T05:57:57.590 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.591 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.592 DEBUG:boto:Params: {} 2022-09-02T05:57:57.593 DEBUG:boto:Token: None 2022-09-02T05:57:57.594 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7cfd2055a8dfdc3-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.603 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.604 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.604 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.605 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.605 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.606 DEBUG:boto:Method: GET 2022-09-02T05:57:57.607 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:57:57.607 DEBUG:boto:Data: 2022-09-02T05:57:57.608 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.610 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.610 DEBUG:boto:Params: {} 2022-09-02T05:57:57.611 DEBUG:boto:Token: None 2022-09-02T05:57:57.612 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.613 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61bf1e350ded2c8-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.619 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:57:57.620 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.621 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.622 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.622 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.623 DEBUG:boto:Method: GET 2022-09-02T05:57:57.624 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.625 DEBUG:boto:Data: 2022-09-02T05:57:57.625 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.627 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.627 DEBUG:boto:Params: {} 2022-09-02T05:57:57.628 DEBUG:boto:Token: None 2022-09-02T05:57:57.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go= 2022-09-02T05:57:57.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go=', 'Content-Length': '0'} 2022-09-02T05:57:57.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9c8e7b8e2b219d-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.634 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.635 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.635 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.636 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.636 DEBUG:boto:Method: GET 2022-09-02T05:57:57.637 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.637 DEBUG:boto:Data: 2022-09-02T05:57:57.638 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.639 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.640 DEBUG:boto:Params: {} 2022-09-02T05:57:57.640 DEBUG:boto:Token: None 2022-09-02T05:57:57.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go= 2022-09-02T05:57:57.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go=', 'Content-Length': '0'} 2022-09-02T05:57:57.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0497a708ee8bea0-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.645 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.646 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.647 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.647 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.648 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.648 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.649 DEBUG:boto:Data: 2022-09-02T05:57:57.650 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.651 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.651 DEBUG:boto:Params: {} 2022-09-02T05:57:57.652 DEBUG:boto:Token: None 2022-09-02T05:57:57.653 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRFrlncv2AEaMp8ObGYXKyVeD/k= 2022-09-02T05:57:57.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRFrlncv2AEaMp8ObGYXKyVeD/k=', 'Content-Length': '0'} 2022-09-02T05:57:57.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d05502d9b9116b7-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.659 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.659 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.660 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.660 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.661 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.661 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.662 DEBUG:boto:Data: 2022-09-02T05:57:57.663 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.664 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.664 DEBUG:boto:Params: {} 2022-09-02T05:57:57.665 DEBUG:boto:Token: None 2022-09-02T05:57:57.665 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lRFrlncv2AEaMp8ObGYXKyVeD/k= 2022-09-02T05:57:57.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lRFrlncv2AEaMp8ObGYXKyVeD/k=', 'Content-Length': '0'} 2022-09-02T05:57:57.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090aabed8b1065403-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:57:57.670 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.671 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.671 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.672 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.672 DEBUG:boto:Method: GET 2022-09-02T05:57:57.673 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.673 DEBUG:boto:Data: 2022-09-02T05:57:57.674 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.674 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.675 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.676 DEBUG:boto:Params: {} 2022-09-02T05:57:57.676 DEBUG:boto:Token: None 2022-09-02T05:57:57.677 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go= 2022-09-02T05:57:57.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go=', 'Content-Length': '0'} 2022-09-02T05:57:57.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72b0145f418ffe6-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.681 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.682 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.682 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.683 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.683 DEBUG:boto:Method: GET 2022-09-02T05:57:57.684 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.684 DEBUG:boto:Data: 2022-09-02T05:57:57.685 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.686 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.686 DEBUG:boto:Params: {} 2022-09-02T05:57:57.687 DEBUG:boto:Token: None 2022-09-02T05:57:57.688 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:57:57.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go= 2022-09-02T05:57:57.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:viQjfdk+/uo9LR334wXEwat54Go=', 'Content-Length': '0'} 2022-09-02T05:57:57.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310a97c518ba638b-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.692 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.693 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.693 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.694 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.695 DEBUG:boto:Method: GET 2022-09-02T05:57:57.695 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.696 DEBUG:boto:Data: 2022-09-02T05:57:57.696 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.697 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.697 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.698 DEBUG:boto:Params: {} 2022-09-02T05:57:57.698 DEBUG:boto:Token: None 2022-09-02T05:57:57.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf71c06a2c5b41e9-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.704 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.705 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.705 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.706 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.706 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.707 DEBUG:boto:Method: GET 2022-09-02T05:57:57.707 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.708 DEBUG:boto:Data: 2022-09-02T05:57:57.708 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.709 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.710 DEBUG:boto:Params: {} 2022-09-02T05:57:57.711 DEBUG:boto:Token: None 2022-09-02T05:57:57.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ad7da0f56fcacd1-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.717 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.718 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.719 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.720 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.720 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:57:57.721 DEBUG:boto:Data: 2022-09-02T05:57:57.722 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.724 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.725 DEBUG:boto:Params: {} 2022-09-02T05:57:57.725 DEBUG:boto:Token: None 2022-09-02T05:57:57.726 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95 2022-09-02T05:57:57.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XLuIHS4cDxC+ZY7F1cFW+kSRd8A= 2022-09-02T05:57:57.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XLuIHS4cDxC+ZY7F1cFW+kSRd8A=', 'Content-Length': '0'} 2022-09-02T05:57:57.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158dc41b46551a35-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.731 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.732 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.733 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.733 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:57:57.734 DEBUG:boto:Data: 2022-09-02T05:57:57.734 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.735 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.736 DEBUG:boto:Params: {} 2022-09-02T05:57:57.737 DEBUG:boto:Token: None 2022-09-02T05:57:57.737 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95 2022-09-02T05:57:57.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XLuIHS4cDxC+ZY7F1cFW+kSRd8A= 2022-09-02T05:57:57.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XLuIHS4cDxC+ZY7F1cFW+kSRd8A=', 'Content-Length': '0'} 2022-09-02T05:57:57.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc17bda884ede0ec-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:57:57.743 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.744 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.745 DEBUG:boto:Method: GET 2022-09-02T05:57:57.746 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:57:57.747 DEBUG:boto:Data: 2022-09-02T05:57:57.748 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.748 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.749 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.749 DEBUG:boto:Params: {} 2022-09-02T05:57:57.750 DEBUG:boto:Token: None 2022-09-02T05:57:57.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95 2022-09-02T05:57:57.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+kj89Z4ynZOVlwAwH1fGYqoEgWo= 2022-09-02T05:57:57.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+kj89Z4ynZOVlwAwH1fGYqoEgWo=', 'Content-Length': '0'} 2022-09-02T05:57:57.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042efdf369890b482-0063119b65-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.802 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:57:57.803 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:57:57.803 DEBUG:boto:Method: GET 2022-09-02T05:57:57.804 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:57:57.805 DEBUG:boto:Data: 2022-09-02T05:57:57.805 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.806 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.807 DEBUG:boto:Params: {} 2022-09-02T05:57:57.807 DEBUG:boto:Token: None 2022-09-02T05:57:57.808 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj95 2022-09-02T05:57:57.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+kj89Z4ynZOVlwAwH1fGYqoEgWo= 2022-09-02T05:57:57.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+kj89Z4ynZOVlwAwH1fGYqoEgWo=', 'Content-Length': '0'} 2022-09-02T05:57:57.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c52785f637008ce-0063119b65-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.813 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.813 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.814 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.814 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.815 DEBUG:boto:Method: GET 2022-09-02T05:57:57.815 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.816 DEBUG:boto:Data: 2022-09-02T05:57:57.816 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.817 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:57:57.817 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.818 DEBUG:boto:Params: {} 2022-09-02T05:57:57.819 DEBUG:boto:Token: None 2022-09-02T05:57:57.819 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6890446e0cf608a-0063119b65-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.837 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.838 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.839 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.840 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.840 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.841 DEBUG:boto:Method: GET 2022-09-02T05:57:57.841 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:57:57.842 DEBUG:boto:Data: 2022-09-02T05:57:57.842 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.843 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.844 DEBUG:boto:Params: {} 2022-09-02T05:57:57.845 DEBUG:boto:Token: None 2022-09-02T05:57:57.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw= 2022-09-02T05:57:57.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bekVEa9bLf60D2O3Ez21ACJFbw=', 'Content-Length': '0'} 2022-09-02T05:57:57.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542b698e8b91cb9f-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.851 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.852 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:57:57.853 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:57:57.854 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.854 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.855 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.855 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:57:57.856 DEBUG:boto:Data: 2022-09-02T05:57:57.856 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:57.858 DEBUG:boto:Port: 8001 2022-09-02T05:57:57.859 DEBUG:boto:Params: {} 2022-09-02T05:57:57.860 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:57:57.860 DEBUG:boto:Token: None 2022-09-02T05:57:57.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kR9yg1aQjmieLD/oUipxB/2EeTk= 2022-09-02T05:57:57.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kR9yg1aQjmieLD/oUipxB/2EeTk=', 'Content-Length': '0'} 2022-09-02T05:57:57.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b596b9529e9049b-0063119b65-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.867 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.867 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.868 DEBUG:boto:Method: HEAD 2022-09-02T05:57:57.868 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:57:57.869 DEBUG:boto:Data: 2022-09-02T05:57:57.869 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.872 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.873 DEBUG:boto:Params: {} 2022-09-02T05:57:57.874 DEBUG:boto:Token: None 2022-09-02T05:57:57.875 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/ 2022-09-02T05:57:57.876 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kR9yg1aQjmieLD/oUipxB/2EeTk= 2022-09-02T05:57:57.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kR9yg1aQjmieLD/oUipxB/2EeTk=', 'Content-Length': '0'} 2022-09-02T05:57:57.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017201ea49ef7626b-0063119b65-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:57:57.884 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.885 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.886 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.887 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.888 DEBUG:boto:Method: GET 2022-09-02T05:57:57.889 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.890 DEBUG:boto:Data: 2022-09-02T05:57:57.890 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:57.892 DEBUG:boto:Port: 8001 2022-09-02T05:57:57.893 DEBUG:boto:Params: {} 2022-09-02T05:57:57.894 DEBUG:boto:Token: None 2022-09-02T05:57:57.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/?versions 2022-09-02T05:57:57.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds= 2022-09-02T05:57:57.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds=', 'Content-Length': '0'} 2022-09-02T05:57:57.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005245a903eda14224-0063119b65-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.901 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:57:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:57:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:57:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:57:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:57:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:57:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:57:57.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:57:57.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:57:57.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:57:57.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:57:57.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:57:57.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:57:57.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:57:57.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:57:57.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:57:57.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:57:57.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:57:57.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:57:57.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:57:57.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:57:57.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:57:57.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:57:57.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:57:57.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:57:57.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:57:57.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:57:57.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:57:57.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:57:57.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:57:57.917 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:57:57.918 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.919 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.919 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.920 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.920 DEBUG:boto:Method: GET 2022-09-02T05:57:57.921 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.921 DEBUG:boto:Data: 2022-09-02T05:57:57.922 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.923 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.924 DEBUG:boto:Params: {} 2022-09-02T05:57:57.924 DEBUG:boto:Token: None 2022-09-02T05:57:57.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/?versions 2022-09-02T05:57:57.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds= 2022-09-02T05:57:57.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds=', 'Content-Length': '0'} 2022-09-02T05:57:57.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3bff62890b0cc8-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.931 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:57:57.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:57:57.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:57:57.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:57:57.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:57:57.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:57:57.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:57:57.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:57:57.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:57:57.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:57:57.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:57:57.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:57:57.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:57:57.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:57:57.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:57:57.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:57:57.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:57:57.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:57:57.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:57:57.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:57:57.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:57:57.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:57:57.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:57:57.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:57:57.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:57:57.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:57:57.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:57:57.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:57:57.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:57:57.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:57:57.949 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.950 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.951 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.952 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.953 DEBUG:boto:Method: GET 2022-09-02T05:57:57.954 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.955 DEBUG:boto:Data: 2022-09-02T05:57:57.956 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:57.958 DEBUG:boto:Port: 8001 2022-09-02T05:57:57.959 DEBUG:boto:Params: {} 2022-09-02T05:57:57.960 DEBUG:boto:Token: None 2022-09-02T05:57:57.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/?versions 2022-09-02T05:57:57.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds= 2022-09-02T05:57:57.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds=', 'Content-Length': '0'} 2022-09-02T05:57:57.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ab999b11366b69-0063119b65-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.967 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.969 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:57.970 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:57.970 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.971 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.971 DEBUG:boto:Method: GET 2022-09-02T05:57:57.972 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:57:57.972 DEBUG:boto:Data: 2022-09-02T05:57:57.973 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:57.974 DEBUG:boto:Port: 8000 2022-09-02T05:57:57.975 DEBUG:boto:Params: {} 2022-09-02T05:57:57.975 DEBUG:boto:Token: None 2022-09-02T05:57:57.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/?versions 2022-09-02T05:57:57.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds= 2022-09-02T05:57:57.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s9kDhl8YOH8PxQdWBxrj66zp9Ds=', 'Content-Length': '0'} 2022-09-02T05:57:57.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad28a8615bed4f7d-0063119b65-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.982 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:57.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:57:57.985 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:57.985 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:57.986 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.986 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.987 DEBUG:boto:Method: GET 2022-09-02T05:57:57.988 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.988 DEBUG:boto:Data: 2022-09-02T05:57:57.989 DEBUG:boto:Headers: {} 2022-09-02T05:57:57.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:57.990 DEBUG:boto:Port: 8001 2022-09-02T05:57:57.990 DEBUG:boto:Params: {} 2022-09-02T05:57:57.991 DEBUG:boto:Token: None 2022-09-02T05:57:57.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:57 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hbTnnvW5xvKFP6v25I4mUOxtT8A= 2022-09-02T05:57:57.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:57 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hbTnnvW5xvKFP6v25I4mUOxtT8A=', 'Content-Length': '0'} 2022-09-02T05:57:57.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cbc57c0cf5ad891-0063119b65-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:57 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:57.996 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:57.996 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:57.997 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.998 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.998 DEBUG:boto:Method: GET 2022-09-02T05:57:57.999 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:57.999 DEBUG:boto:Data: 2022-09-02T05:57:58.000 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.001 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.002 DEBUG:boto:Params: {} 2022-09-02T05:57:58.002 DEBUG:boto:Token: None 2022-09-02T05:57:58.003 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.003 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D90i58hGtpPG6Jv6J17qgR1Xj60= 2022-09-02T05:57:58.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D90i58hGtpPG6Jv6J17qgR1Xj60=', 'Content-Length': '0'} 2022-09-02T05:57:58.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93320ddf29c8973-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.007 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.008 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.008 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.009 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.009 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.010 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.010 DEBUG:boto:Data: 2022-09-02T05:57:58.011 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.012 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.012 DEBUG:boto:Params: {} 2022-09-02T05:57:58.013 DEBUG:boto:Token: None 2022-09-02T05:57:58.013 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.014 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qbk5HjeO66MmxGMEGKWwldZNH6E= 2022-09-02T05:57:58.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qbk5HjeO66MmxGMEGKWwldZNH6E=', 'Content-Length': '0'} 2022-09-02T05:57:58.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e4c6ea8febc801-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.017 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.018 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.018 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.019 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.020 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.020 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.021 DEBUG:boto:Data: 2022-09-02T05:57:58.021 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.022 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.023 DEBUG:boto:Params: {} 2022-09-02T05:57:58.023 DEBUG:boto:Token: None 2022-09-02T05:57:58.024 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qbk5HjeO66MmxGMEGKWwldZNH6E= 2022-09-02T05:57:58.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qbk5HjeO66MmxGMEGKWwldZNH6E=', 'Content-Length': '0'} 2022-09-02T05:57:58.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004449782455b8730b-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:57:58.029 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.029 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.030 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.030 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.031 DEBUG:boto:Method: GET 2022-09-02T05:57:58.031 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.032 DEBUG:boto:Data: 2022-09-02T05:57:58.032 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.033 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.034 DEBUG:boto:Params: {} 2022-09-02T05:57:58.034 DEBUG:boto:Token: None 2022-09-02T05:57:58.035 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D90i58hGtpPG6Jv6J17qgR1Xj60= 2022-09-02T05:57:58.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D90i58hGtpPG6Jv6J17qgR1Xj60=', 'Content-Length': '0'} 2022-09-02T05:57:58.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63a48c18016d48a-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.039 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.040 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.040 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.041 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.041 DEBUG:boto:Method: GET 2022-09-02T05:57:58.042 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.043 DEBUG:boto:Data: 2022-09-02T05:57:58.043 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.045 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.046 DEBUG:boto:Params: {} 2022-09-02T05:57:58.047 DEBUG:boto:Token: None 2022-09-02T05:57:58.048 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:57:58.049 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D90i58hGtpPG6Jv6J17qgR1Xj60= 2022-09-02T05:57:58.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D90i58hGtpPG6Jv6J17qgR1Xj60=', 'Content-Length': '0'} 2022-09-02T05:57:58.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6a035d0c5ddd3d-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.054 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.055 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.056 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.057 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.058 DEBUG:boto:Method: GET 2022-09-02T05:57:58.060 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.061 DEBUG:boto:Data: 2022-09-02T05:57:58.062 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.064 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.065 DEBUG:boto:Params: {} 2022-09-02T05:57:58.066 DEBUG:boto:Token: None 2022-09-02T05:57:58.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96c2ceb3444ee15-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.114 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.116 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.116 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.117 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.117 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.118 DEBUG:boto:Method: GET 2022-09-02T05:57:58.119 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.119 DEBUG:boto:Data: 2022-09-02T05:57:58.120 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.121 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.121 DEBUG:boto:Params: {} 2022-09-02T05:57:58.122 DEBUG:boto:Token: None 2022-09-02T05:57:58.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3cb0bfec011e01-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.129 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.130 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.131 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.131 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.132 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:58.132 DEBUG:boto:Data: 2022-09-02T05:57:58.133 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.134 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.135 DEBUG:boto:Params: {} 2022-09-02T05:57:58.135 DEBUG:boto:Token: None 2022-09-02T05:57:58.136 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11 2022-09-02T05:57:58.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VZ+axptcF4PkprjD+qvuOXsFFIk= 2022-09-02T05:57:58.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VZ+axptcF4PkprjD+qvuOXsFFIk=', 'Content-Length': '0'} 2022-09-02T05:57:58.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe01eaaf67a97be9-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.140 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.141 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.141 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.142 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:58.142 DEBUG:boto:Data: 2022-09-02T05:57:58.143 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.144 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.145 DEBUG:boto:Params: {} 2022-09-02T05:57:58.147 DEBUG:boto:Token: None 2022-09-02T05:57:58.148 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11 2022-09-02T05:57:58.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VZ+axptcF4PkprjD+qvuOXsFFIk= 2022-09-02T05:57:58.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VZ+axptcF4PkprjD+qvuOXsFFIk=', 'Content-Length': '0'} 2022-09-02T05:57:58.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e12775f64826be-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:57:58.154 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.155 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.155 DEBUG:boto:Method: GET 2022-09-02T05:57:58.156 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:58.157 DEBUG:boto:Data: 2022-09-02T05:57:58.157 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.159 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.160 DEBUG:boto:Params: {} 2022-09-02T05:57:58.161 DEBUG:boto:Token: None 2022-09-02T05:57:58.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11 2022-09-02T05:57:58.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KBe1VUojvJu4oBoVuk5bPMSw1wc= 2022-09-02T05:57:58.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KBe1VUojvJu4oBoVuk5bPMSw1wc=', 'Content-Length': '0'} 2022-09-02T05:57:58.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fffeb9db58e6555-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.168 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:57:58.168 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:57:58.169 DEBUG:boto:Method: GET 2022-09-02T05:57:58.169 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:57:58.170 DEBUG:boto:Data: 2022-09-02T05:57:58.171 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.172 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.172 DEBUG:boto:Params: {} 2022-09-02T05:57:58.173 DEBUG:boto:Token: None 2022-09-02T05:57:58.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj11 2022-09-02T05:57:58.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KBe1VUojvJu4oBoVuk5bPMSw1wc= 2022-09-02T05:57:58.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KBe1VUojvJu4oBoVuk5bPMSw1wc=', 'Content-Length': '0'} 2022-09-02T05:57:58.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0744e9e6611ddf-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.179 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.179 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.180 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.180 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.181 DEBUG:boto:Method: GET 2022-09-02T05:57:58.181 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.182 DEBUG:boto:Data: 2022-09-02T05:57:58.182 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.183 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.184 DEBUG:boto:Params: {} 2022-09-02T05:57:58.185 DEBUG:boto:Token: None 2022-09-02T05:57:58.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f2fdb3f8143423-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.190 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.191 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.192 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.192 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.193 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.194 DEBUG:boto:Method: GET 2022-09-02T05:57:58.194 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:57:58.195 DEBUG:boto:Data: 2022-09-02T05:57:58.196 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.197 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.197 DEBUG:boto:Params: {} 2022-09-02T05:57:58.198 DEBUG:boto:Token: None 2022-09-02T05:57:58.199 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af80603fec5f53b5-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.246 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:57:58.248 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.248 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.249 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.249 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.250 DEBUG:boto:Method: GET 2022-09-02T05:57:58.251 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.251 DEBUG:boto:Data: 2022-09-02T05:57:58.252 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.254 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.255 DEBUG:boto:Params: {} 2022-09-02T05:57:58.256 DEBUG:boto:Token: None 2022-09-02T05:57:58.257 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI= 2022-09-02T05:57:58.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI=', 'Content-Length': '0'} 2022-09-02T05:57:58.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f9972f468680aa-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.264 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.265 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.266 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.267 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.268 DEBUG:boto:Method: GET 2022-09-02T05:57:58.269 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.270 DEBUG:boto:Data: 2022-09-02T05:57:58.271 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.272 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.273 DEBUG:boto:Params: {} 2022-09-02T05:57:58.274 DEBUG:boto:Token: None 2022-09-02T05:57:58.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI= 2022-09-02T05:57:58.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI=', 'Content-Length': '0'} 2022-09-02T05:57:58.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eac00c922201a54-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.279 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.279 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.280 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.280 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.281 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.281 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.282 DEBUG:boto:Data: 2022-09-02T05:57:58.282 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.284 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.284 DEBUG:boto:Params: {} 2022-09-02T05:57:58.285 DEBUG:boto:Token: None 2022-09-02T05:57:58.285 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WJk2RqrdQwvoKZ8wTbJHmpyhkaQ= 2022-09-02T05:57:58.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WJk2RqrdQwvoKZ8wTbJHmpyhkaQ=', 'Content-Length': '0'} 2022-09-02T05:57:58.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a66c8d01a7ce2d-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.290 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.290 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.291 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.291 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.292 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.292 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.293 DEBUG:boto:Data: 2022-09-02T05:57:58.293 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.295 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.295 DEBUG:boto:Params: {} 2022-09-02T05:57:58.296 DEBUG:boto:Token: None 2022-09-02T05:57:58.296 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.297 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WJk2RqrdQwvoKZ8wTbJHmpyhkaQ= 2022-09-02T05:57:58.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WJk2RqrdQwvoKZ8wTbJHmpyhkaQ=', 'Content-Length': '0'} 2022-09-02T05:57:58.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433120820fd73718-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:57:58.301 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.302 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.302 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.303 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.304 DEBUG:boto:Method: GET 2022-09-02T05:57:58.304 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.305 DEBUG:boto:Data: 2022-09-02T05:57:58.305 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.307 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.307 DEBUG:boto:Params: {} 2022-09-02T05:57:58.308 DEBUG:boto:Token: None 2022-09-02T05:57:58.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI= 2022-09-02T05:57:58.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI=', 'Content-Length': '0'} 2022-09-02T05:57:58.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d05935866030b294-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.354 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.355 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.355 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.356 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.356 DEBUG:boto:Method: GET 2022-09-02T05:57:58.357 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.357 DEBUG:boto:Data: 2022-09-02T05:57:58.358 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.359 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.360 DEBUG:boto:Params: {} 2022-09-02T05:57:58.360 DEBUG:boto:Token: None 2022-09-02T05:57:58.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:57:58.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI= 2022-09-02T05:57:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zZgYbOO25iuoanSG9BiYcZpRGYI=', 'Content-Length': '0'} 2022-09-02T05:57:58.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070cd204292cf16ab-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.365 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.366 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.367 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.367 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.368 DEBUG:boto:Method: GET 2022-09-02T05:57:58.368 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.369 DEBUG:boto:Data: 2022-09-02T05:57:58.369 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.370 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.371 DEBUG:boto:Params: {} 2022-09-02T05:57:58.371 DEBUG:boto:Token: None 2022-09-02T05:57:58.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e49d28f30165ebe0-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.377 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.378 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.378 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.379 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.379 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.380 DEBUG:boto:Method: GET 2022-09-02T05:57:58.380 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.381 DEBUG:boto:Data: 2022-09-02T05:57:58.381 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.382 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.383 DEBUG:boto:Params: {} 2022-09-02T05:57:58.383 DEBUG:boto:Token: None 2022-09-02T05:57:58.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d9e62c9bc7b4a3-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.390 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.391 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.392 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.392 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.393 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:58.393 DEBUG:boto:Data: 2022-09-02T05:57:58.394 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.395 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.395 DEBUG:boto:Params: {} 2022-09-02T05:57:58.396 DEBUG:boto:Token: None 2022-09-02T05:57:58.397 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115 2022-09-02T05:57:58.397 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0OnH9mAAB4rr9yA6uQUzlPVFIbI= 2022-09-02T05:57:58.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0OnH9mAAB4rr9yA6uQUzlPVFIbI=', 'Content-Length': '0'} 2022-09-02T05:57:58.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2485d1fd095b36f-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.401 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.401 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.402 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.402 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:58.403 DEBUG:boto:Data: 2022-09-02T05:57:58.403 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.405 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.405 DEBUG:boto:Params: {} 2022-09-02T05:57:58.406 DEBUG:boto:Token: None 2022-09-02T05:57:58.406 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115 2022-09-02T05:57:58.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0OnH9mAAB4rr9yA6uQUzlPVFIbI= 2022-09-02T05:57:58.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0OnH9mAAB4rr9yA6uQUzlPVFIbI=', 'Content-Length': '0'} 2022-09-02T05:57:58.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a949c981b185f08-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:57:58.411 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.412 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.412 DEBUG:boto:Method: GET 2022-09-02T05:57:58.413 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:58.413 DEBUG:boto:Data: 2022-09-02T05:57:58.414 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.415 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.416 DEBUG:boto:Params: {} 2022-09-02T05:57:58.416 DEBUG:boto:Token: None 2022-09-02T05:57:58.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115 2022-09-02T05:57:58.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J21hjEq99QUIdb2fsFvJXLYT6Jg= 2022-09-02T05:57:58.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J21hjEq99QUIdb2fsFvJXLYT6Jg=', 'Content-Length': '0'} 2022-09-02T05:57:58.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec1a95f1daa597f-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.421 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:57:58.422 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:57:58.423 DEBUG:boto:Method: GET 2022-09-02T05:57:58.423 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:57:58.424 DEBUG:boto:Data: 2022-09-02T05:57:58.424 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.425 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.426 DEBUG:boto:Params: {} 2022-09-02T05:57:58.426 DEBUG:boto:Token: None 2022-09-02T05:57:58.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj115 2022-09-02T05:57:58.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J21hjEq99QUIdb2fsFvJXLYT6Jg= 2022-09-02T05:57:58.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J21hjEq99QUIdb2fsFvJXLYT6Jg=', 'Content-Length': '0'} 2022-09-02T05:57:58.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d587702007a3564-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.431 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.432 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.432 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.433 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.433 DEBUG:boto:Method: GET 2022-09-02T05:57:58.434 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.434 DEBUG:boto:Data: 2022-09-02T05:57:58.435 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.436 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.437 DEBUG:boto:Params: {} 2022-09-02T05:57:58.437 DEBUG:boto:Token: None 2022-09-02T05:57:58.438 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003509d6535d8733e-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.442 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.443 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.444 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.444 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.445 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.445 DEBUG:boto:Method: GET 2022-09-02T05:57:58.446 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:57:58.447 DEBUG:boto:Data: 2022-09-02T05:57:58.447 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.448 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.449 DEBUG:boto:Params: {} 2022-09-02T05:57:58.449 DEBUG:boto:Token: None 2022-09-02T05:57:58.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.450 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb96b362576a1d0-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.498 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:57:58.500 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.500 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.501 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.501 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.502 DEBUG:boto:Method: GET 2022-09-02T05:57:58.502 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.503 DEBUG:boto:Data: 2022-09-02T05:57:58.504 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.506 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.506 DEBUG:boto:Params: {} 2022-09-02T05:57:58.507 DEBUG:boto:Token: None 2022-09-02T05:57:58.508 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.508 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI= 2022-09-02T05:57:58.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI=', 'Content-Length': '0'} 2022-09-02T05:57:58.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8b784b7cdf94fd9-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.515 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.516 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.516 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.517 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.517 DEBUG:boto:Method: GET 2022-09-02T05:57:58.518 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.518 DEBUG:boto:Data: 2022-09-02T05:57:58.519 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.520 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.521 DEBUG:boto:Params: {} 2022-09-02T05:57:58.521 DEBUG:boto:Token: None 2022-09-02T05:57:58.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI= 2022-09-02T05:57:58.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI=', 'Content-Length': '0'} 2022-09-02T05:57:58.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008d92d743109ca41-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.527 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.527 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.528 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.528 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.529 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.529 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.530 DEBUG:boto:Data: 2022-09-02T05:57:58.530 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.531 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.532 DEBUG:boto:Params: {} 2022-09-02T05:57:58.532 DEBUG:boto:Token: None 2022-09-02T05:57:58.533 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s3QO3vk1FJ+UpQxnRJTkKtDDccs= 2022-09-02T05:57:58.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s3QO3vk1FJ+UpQxnRJTkKtDDccs=', 'Content-Length': '0'} 2022-09-02T05:57:58.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84120583663bf55-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.537 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.538 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.539 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.539 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.540 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.541 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.541 DEBUG:boto:Data: 2022-09-02T05:57:58.542 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.544 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.544 DEBUG:boto:Params: {} 2022-09-02T05:57:58.545 DEBUG:boto:Token: None 2022-09-02T05:57:58.547 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s3QO3vk1FJ+UpQxnRJTkKtDDccs= 2022-09-02T05:57:58.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s3QO3vk1FJ+UpQxnRJTkKtDDccs=', 'Content-Length': '0'} 2022-09-02T05:57:58.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010843836eefcb77d-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:57:58.554 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.556 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.557 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.557 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.558 DEBUG:boto:Method: GET 2022-09-02T05:57:58.559 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.560 DEBUG:boto:Data: 2022-09-02T05:57:58.560 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.562 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.562 DEBUG:boto:Params: {} 2022-09-02T05:57:58.563 DEBUG:boto:Token: None 2022-09-02T05:57:58.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI= 2022-09-02T05:57:58.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI=', 'Content-Length': '0'} 2022-09-02T05:57:58.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0ea3ccb80e043c-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.568 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.569 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.569 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.570 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.571 DEBUG:boto:Method: GET 2022-09-02T05:57:58.572 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.572 DEBUG:boto:Data: 2022-09-02T05:57:58.573 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.574 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.575 DEBUG:boto:Params: {} 2022-09-02T05:57:58.576 DEBUG:boto:Token: None 2022-09-02T05:57:58.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:57:58.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI= 2022-09-02T05:57:58.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rbCuIVYOxL7Tm2KB1f8d7vIxjEI=', 'Content-Length': '0'} 2022-09-02T05:57:58.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2711d36f6a961b7-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.582 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.584 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.585 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.586 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.587 DEBUG:boto:Method: GET 2022-09-02T05:57:58.588 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.589 DEBUG:boto:Data: 2022-09-02T05:57:58.591 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.592 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.593 DEBUG:boto:Params: {} 2022-09-02T05:57:58.594 DEBUG:boto:Token: None 2022-09-02T05:57:58.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e750ecc2c68fb720-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.601 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.602 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.602 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.603 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.604 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.604 DEBUG:boto:Method: GET 2022-09-02T05:57:58.605 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.606 DEBUG:boto:Data: 2022-09-02T05:57:58.607 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.609 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.610 DEBUG:boto:Params: {} 2022-09-02T05:57:58.610 DEBUG:boto:Token: None 2022-09-02T05:57:58.611 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dc44db6ae838c7e-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.619 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.620 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.620 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.621 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.621 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:58.622 DEBUG:boto:Data: 2022-09-02T05:57:58.623 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.624 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.625 DEBUG:boto:Params: {} 2022-09-02T05:57:58.625 DEBUG:boto:Token: None 2022-09-02T05:57:58.626 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116 2022-09-02T05:57:58.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hm6CRpICKLlZQqbboQpi2fN6jFY= 2022-09-02T05:57:58.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hm6CRpICKLlZQqbboQpi2fN6jFY=', 'Content-Length': '0'} 2022-09-02T05:57:58.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595cce1c524d8bff-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.630 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.631 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.631 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.632 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:58.632 DEBUG:boto:Data: 2022-09-02T05:57:58.633 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.634 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.634 DEBUG:boto:Params: {} 2022-09-02T05:57:58.635 DEBUG:boto:Token: None 2022-09-02T05:57:58.636 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116 2022-09-02T05:57:58.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hm6CRpICKLlZQqbboQpi2fN6jFY= 2022-09-02T05:57:58.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hm6CRpICKLlZQqbboQpi2fN6jFY=', 'Content-Length': '0'} 2022-09-02T05:57:58.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac310fa96135014f-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:57:58.641 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.642 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.643 DEBUG:boto:Method: GET 2022-09-02T05:57:58.644 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:58.645 DEBUG:boto:Data: 2022-09-02T05:57:58.646 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.647 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.648 DEBUG:boto:Params: {} 2022-09-02T05:57:58.649 DEBUG:boto:Token: None 2022-09-02T05:57:58.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116 2022-09-02T05:57:58.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ErBxfzwIAgnaDUmBbNyQU6wA0= 2022-09-02T05:57:58.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ErBxfzwIAgnaDUmBbNyQU6wA0=', 'Content-Length': '0'} 2022-09-02T05:57:58.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000811a647f531059ad-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.698 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:57:58.699 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:57:58.699 DEBUG:boto:Method: GET 2022-09-02T05:57:58.700 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:57:58.700 DEBUG:boto:Data: 2022-09-02T05:57:58.701 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.702 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.703 DEBUG:boto:Params: {} 2022-09-02T05:57:58.703 DEBUG:boto:Token: None 2022-09-02T05:57:58.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj116 2022-09-02T05:57:58.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E/ErBxfzwIAgnaDUmBbNyQU6wA0= 2022-09-02T05:57:58.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E/ErBxfzwIAgnaDUmBbNyQU6wA0=', 'Content-Length': '0'} 2022-09-02T05:57:58.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad5fb358169cacc-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.709 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.709 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.710 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.711 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.712 DEBUG:boto:Method: GET 2022-09-02T05:57:58.712 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.713 DEBUG:boto:Data: 2022-09-02T05:57:58.714 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.716 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.717 DEBUG:boto:Params: {} 2022-09-02T05:57:58.718 DEBUG:boto:Token: None 2022-09-02T05:57:58.719 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5b9b3358e64f58e-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.726 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.728 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.729 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.730 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.731 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.732 DEBUG:boto:Method: GET 2022-09-02T05:57:58.733 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:57:58.734 DEBUG:boto:Data: 2022-09-02T05:57:58.735 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.737 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.738 DEBUG:boto:Params: {} 2022-09-02T05:57:58.739 DEBUG:boto:Token: None 2022-09-02T05:57:58.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba965ab2dfe8419-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.748 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:57:58.749 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.750 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.750 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.751 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.751 DEBUG:boto:Method: GET 2022-09-02T05:57:58.752 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.752 DEBUG:boto:Data: 2022-09-02T05:57:58.753 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.754 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.755 DEBUG:boto:Params: {} 2022-09-02T05:57:58.755 DEBUG:boto:Token: None 2022-09-02T05:57:58.756 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao= 2022-09-02T05:57:58.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao=', 'Content-Length': '0'} 2022-09-02T05:57:58.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38b6191d93b019f-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.760 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.761 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.761 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.762 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.762 DEBUG:boto:Method: GET 2022-09-02T05:57:58.763 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.763 DEBUG:boto:Data: 2022-09-02T05:57:58.764 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.765 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.766 DEBUG:boto:Params: {} 2022-09-02T05:57:58.766 DEBUG:boto:Token: None 2022-09-02T05:57:58.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao= 2022-09-02T05:57:58.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao=', 'Content-Length': '0'} 2022-09-02T05:57:58.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fac2db3232a2da7-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.772 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.773 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.774 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.775 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.776 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.777 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.778 DEBUG:boto:Data: 2022-09-02T05:57:58.779 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.781 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.781 DEBUG:boto:Params: {} 2022-09-02T05:57:58.782 DEBUG:boto:Token: None 2022-09-02T05:57:58.782 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.783 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CYOKfSTZ4t/GBXrKUeWwzNk/pZg= 2022-09-02T05:57:58.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CYOKfSTZ4t/GBXrKUeWwzNk/pZg=', 'Content-Length': '0'} 2022-09-02T05:57:58.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c50340639da049-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.787 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.787 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.788 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.788 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.789 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.789 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.790 DEBUG:boto:Data: 2022-09-02T05:57:58.790 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.791 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.792 DEBUG:boto:Params: {} 2022-09-02T05:57:58.792 DEBUG:boto:Token: None 2022-09-02T05:57:58.793 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CYOKfSTZ4t/GBXrKUeWwzNk/pZg= 2022-09-02T05:57:58.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CYOKfSTZ4t/GBXrKUeWwzNk/pZg=', 'Content-Length': '0'} 2022-09-02T05:57:58.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319cc2d40f038377-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:57:58.797 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.798 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.799 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.799 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.800 DEBUG:boto:Method: GET 2022-09-02T05:57:58.800 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.801 DEBUG:boto:Data: 2022-09-02T05:57:58.801 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.803 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.803 DEBUG:boto:Params: {} 2022-09-02T05:57:58.804 DEBUG:boto:Token: None 2022-09-02T05:57:58.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao= 2022-09-02T05:57:58.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao=', 'Content-Length': '0'} 2022-09-02T05:57:58.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002bc9f354ffe68c6-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.809 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.809 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.810 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.810 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.811 DEBUG:boto:Method: GET 2022-09-02T05:57:58.811 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.812 DEBUG:boto:Data: 2022-09-02T05:57:58.813 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.814 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.814 DEBUG:boto:Params: {} 2022-09-02T05:57:58.815 DEBUG:boto:Token: None 2022-09-02T05:57:58.816 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:57:58.816 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao= 2022-09-02T05:57:58.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ANo0FmpxipHMxrYS38M3gx689Ao=', 'Content-Length': '0'} 2022-09-02T05:57:58.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecc0dc38a07456d0-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.862 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.863 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.864 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.865 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.865 DEBUG:boto:Method: GET 2022-09-02T05:57:58.866 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.866 DEBUG:boto:Data: 2022-09-02T05:57:58.867 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.868 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.869 DEBUG:boto:Params: {} 2022-09-02T05:57:58.869 DEBUG:boto:Token: None 2022-09-02T05:57:58.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007990df9dc650429c-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.877 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.878 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.879 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.880 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.881 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.881 DEBUG:boto:Method: GET 2022-09-02T05:57:58.882 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.883 DEBUG:boto:Data: 2022-09-02T05:57:58.883 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.884 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.885 DEBUG:boto:Params: {} 2022-09-02T05:57:58.885 DEBUG:boto:Token: None 2022-09-02T05:57:58.886 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:58.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007605ac0704ad1053-0063119b66-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.892 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:58.893 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.894 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.894 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.895 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:58.895 DEBUG:boto:Data: 2022-09-02T05:57:58.896 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.897 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.897 DEBUG:boto:Params: {} 2022-09-02T05:57:58.898 DEBUG:boto:Token: None 2022-09-02T05:57:58.899 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124 2022-09-02T05:57:58.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nWDkhzOhtxHdSBYkIm7E93USf/c= 2022-09-02T05:57:58.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nWDkhzOhtxHdSBYkIm7E93USf/c=', 'Content-Length': '0'} 2022-09-02T05:57:58.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edc0b863d8143eea-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.903 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.903 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.904 DEBUG:boto:Method: HEAD 2022-09-02T05:57:58.905 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:58.905 DEBUG:boto:Data: 2022-09-02T05:57:58.906 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.907 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.908 DEBUG:boto:Params: {} 2022-09-02T05:57:58.908 DEBUG:boto:Token: None 2022-09-02T05:57:58.909 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124 2022-09-02T05:57:58.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nWDkhzOhtxHdSBYkIm7E93USf/c= 2022-09-02T05:57:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nWDkhzOhtxHdSBYkIm7E93USf/c=', 'Content-Length': '0'} 2022-09-02T05:57:58.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002273aacb25d5e650-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:57:58.914 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.914 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.915 DEBUG:boto:Method: GET 2022-09-02T05:57:58.916 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:58.916 DEBUG:boto:Data: 2022-09-02T05:57:58.917 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.919 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.919 DEBUG:boto:Params: {} 2022-09-02T05:57:58.920 DEBUG:boto:Token: None 2022-09-02T05:57:58.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124 2022-09-02T05:57:58.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v3sZxthpcGi4Dr+lW5u3t+K1Abs= 2022-09-02T05:57:58.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v3sZxthpcGi4Dr+lW5u3t+K1Abs=', 'Content-Length': '0'} 2022-09-02T05:57:58.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002321d72ef8e5e7eb-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.970 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:57:58.971 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:57:58.971 DEBUG:boto:Method: GET 2022-09-02T05:57:58.972 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:57:58.972 DEBUG:boto:Data: 2022-09-02T05:57:58.973 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:58.974 DEBUG:boto:Port: 8000 2022-09-02T05:57:58.975 DEBUG:boto:Params: {} 2022-09-02T05:57:58.975 DEBUG:boto:Token: None 2022-09-02T05:57:58.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/obj124 2022-09-02T05:57:58.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v3sZxthpcGi4Dr+lW5u3t+K1Abs= 2022-09-02T05:57:58.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v3sZxthpcGi4Dr+lW5u3t+K1Abs=', 'Content-Length': '0'} 2022-09-02T05:57:58.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e90c4628a58e5a0-0063119b66-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:58.982 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:58.983 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:58.984 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.985 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.986 DEBUG:boto:Method: GET 2022-09-02T05:57:58.987 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:58.988 DEBUG:boto:Data: 2022-09-02T05:57:58.989 DEBUG:boto:Headers: {} 2022-09-02T05:57:58.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:58.991 DEBUG:boto:Port: 8001 2022-09-02T05:57:58.992 DEBUG:boto:Params: {} 2022-09-02T05:57:58.993 DEBUG:boto:Token: None 2022-09-02T05:57:58.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:58 GMT /xljbkf-74/ 2022-09-02T05:57:58.995 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4= 2022-09-02T05:57:58.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:58 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4e34GpliF7ecApCvvCNJZz3g+T4=', 'Content-Length': '0'} 2022-09-02T05:57:59.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d6ad24f0a808dd7-0063119b66-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:58 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.001 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.002 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.003 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.004 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:59.005 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:57:59.006 DEBUG:boto:Method: GET 2022-09-02T05:57:59.007 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:57:59.009 DEBUG:boto:Data: 2022-09-02T05:57:59.010 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.012 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.013 DEBUG:boto:Params: {} 2022-09-02T05:57:59.014 DEBUG:boto:Token: None 2022-09-02T05:57:59.015 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad728549bd7f469-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.023 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:57:59.024 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.025 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.025 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.026 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.027 DEBUG:boto:Method: GET 2022-09-02T05:57:59.027 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.028 DEBUG:boto:Data: 2022-09-02T05:57:59.028 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.029 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.030 DEBUG:boto:Params: {} 2022-09-02T05:57:59.030 DEBUG:boto:Token: None 2022-09-02T05:57:59.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.032 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo= 2022-09-02T05:57:59.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo=', 'Content-Length': '0'} 2022-09-02T05:57:59.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2f5f7af990eb75-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.035 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.036 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.036 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.037 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.037 DEBUG:boto:Method: GET 2022-09-02T05:57:59.038 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.039 DEBUG:boto:Data: 2022-09-02T05:57:59.039 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.040 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.041 DEBUG:boto:Params: {} 2022-09-02T05:57:59.041 DEBUG:boto:Token: None 2022-09-02T05:57:59.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo= 2022-09-02T05:57:59.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo=', 'Content-Length': '0'} 2022-09-02T05:57:59.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008223ae8960a2dfa1-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.086 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.087 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.088 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.089 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.090 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.090 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.091 DEBUG:boto:Data: 2022-09-02T05:57:59.091 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.092 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.093 DEBUG:boto:Params: {} 2022-09-02T05:57:59.093 DEBUG:boto:Token: None 2022-09-02T05:57:59.094 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cDvZeg4hMddxNbYc4WL3/vSIlpM= 2022-09-02T05:57:59.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cDvZeg4hMddxNbYc4WL3/vSIlpM=', 'Content-Length': '0'} 2022-09-02T05:57:59.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d48c1153203942-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.099 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.099 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.100 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.100 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.101 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.101 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.102 DEBUG:boto:Data: 2022-09-02T05:57:59.102 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.103 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.104 DEBUG:boto:Params: {} 2022-09-02T05:57:59.104 DEBUG:boto:Token: None 2022-09-02T05:57:59.105 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cDvZeg4hMddxNbYc4WL3/vSIlpM= 2022-09-02T05:57:59.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cDvZeg4hMddxNbYc4WL3/vSIlpM=', 'Content-Length': '0'} 2022-09-02T05:57:59.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff4281d6d1e89b4-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:57:59.109 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.110 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.111 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.112 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.113 DEBUG:boto:Method: GET 2022-09-02T05:57:59.114 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.115 DEBUG:boto:Data: 2022-09-02T05:57:59.116 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.118 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.119 DEBUG:boto:Params: {} 2022-09-02T05:57:59.120 DEBUG:boto:Token: None 2022-09-02T05:57:59.121 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo= 2022-09-02T05:57:59.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo=', 'Content-Length': '0'} 2022-09-02T05:57:59.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c1c38b3d12ecae-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.125 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.126 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.127 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.128 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.129 DEBUG:boto:Method: GET 2022-09-02T05:57:59.130 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.131 DEBUG:boto:Data: 2022-09-02T05:57:59.132 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.134 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.134 DEBUG:boto:Params: {} 2022-09-02T05:57:59.135 DEBUG:boto:Token: None 2022-09-02T05:57:59.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:57:59.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo= 2022-09-02T05:57:59.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9jzsSKEnEY6/Ojn6RCiVG90Kyvo=', 'Content-Length': '0'} 2022-09-02T05:57:59.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d5f57e3a7ac01d-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.140 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.141 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.141 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.142 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.142 DEBUG:boto:Method: GET 2022-09-02T05:57:59.143 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.143 DEBUG:boto:Data: 2022-09-02T05:57:59.144 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.145 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.146 DEBUG:boto:Params: {} 2022-09-02T05:57:59.146 DEBUG:boto:Token: None 2022-09-02T05:57:59.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085569560d979e3b4-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.152 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.152 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.153 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.154 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.154 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.155 DEBUG:boto:Method: GET 2022-09-02T05:57:59.155 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.156 DEBUG:boto:Data: 2022-09-02T05:57:59.157 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.158 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.158 DEBUG:boto:Params: {} 2022-09-02T05:57:59.159 DEBUG:boto:Token: None 2022-09-02T05:57:59.160 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ca72dca34c1e904-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.166 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.167 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.167 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.168 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.169 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:59.170 DEBUG:boto:Data: 2022-09-02T05:57:59.170 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.172 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.173 DEBUG:boto:Params: {} 2022-09-02T05:57:59.174 DEBUG:boto:Token: None 2022-09-02T05:57:59.175 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145 2022-09-02T05:57:59.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fxau3yZkpm8EgK09n6SuDcjoP5Y= 2022-09-02T05:57:59.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fxau3yZkpm8EgK09n6SuDcjoP5Y=', 'Content-Length': '0'} 2022-09-02T05:57:59.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdae665692cae9e7-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.180 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.181 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.181 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.182 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:59.182 DEBUG:boto:Data: 2022-09-02T05:57:59.183 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.184 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.184 DEBUG:boto:Params: {} 2022-09-02T05:57:59.185 DEBUG:boto:Token: None 2022-09-02T05:57:59.186 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145 2022-09-02T05:57:59.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fxau3yZkpm8EgK09n6SuDcjoP5Y= 2022-09-02T05:57:59.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fxau3yZkpm8EgK09n6SuDcjoP5Y=', 'Content-Length': '0'} 2022-09-02T05:57:59.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d79c125b31f278-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:57:59.190 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.191 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.191 DEBUG:boto:Method: GET 2022-09-02T05:57:59.192 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:59.192 DEBUG:boto:Data: 2022-09-02T05:57:59.193 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.194 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.194 DEBUG:boto:Params: {} 2022-09-02T05:57:59.195 DEBUG:boto:Token: None 2022-09-02T05:57:59.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145 2022-09-02T05:57:59.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HbSjo+qtnA2JPh/y15rpA/Jnt34= 2022-09-02T05:57:59.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HbSjo+qtnA2JPh/y15rpA/Jnt34=', 'Content-Length': '0'} 2022-09-02T05:57:59.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e2b5f1c4a37711c-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.246 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:57:59.247 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:57:59.248 DEBUG:boto:Method: GET 2022-09-02T05:57:59.248 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:57:59.249 DEBUG:boto:Data: 2022-09-02T05:57:59.250 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.251 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.251 DEBUG:boto:Params: {} 2022-09-02T05:57:59.252 DEBUG:boto:Token: None 2022-09-02T05:57:59.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj145 2022-09-02T05:57:59.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HbSjo+qtnA2JPh/y15rpA/Jnt34= 2022-09-02T05:57:59.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HbSjo+qtnA2JPh/y15rpA/Jnt34=', 'Content-Length': '0'} 2022-09-02T05:57:59.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000871ca3e3b6272064-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.257 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.258 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.259 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.259 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.260 DEBUG:boto:Method: GET 2022-09-02T05:57:59.260 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.261 DEBUG:boto:Data: 2022-09-02T05:57:59.261 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.262 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.263 DEBUG:boto:Params: {} 2022-09-02T05:57:59.264 DEBUG:boto:Token: None 2022-09-02T05:57:59.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a386710d10b8866-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.269 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.270 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.271 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.271 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.272 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.272 DEBUG:boto:Method: GET 2022-09-02T05:57:59.273 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:57:59.274 DEBUG:boto:Data: 2022-09-02T05:57:59.274 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.275 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.276 DEBUG:boto:Params: {} 2022-09-02T05:57:59.276 DEBUG:boto:Token: None 2022-09-02T05:57:59.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081929ae8f9a81127-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.283 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:57:59.285 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.285 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.286 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.286 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.287 DEBUG:boto:Method: GET 2022-09-02T05:57:59.287 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.288 DEBUG:boto:Data: 2022-09-02T05:57:59.288 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.289 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.290 DEBUG:boto:Params: {} 2022-09-02T05:57:59.290 DEBUG:boto:Token: None 2022-09-02T05:57:59.291 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E= 2022-09-02T05:57:59.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E=', 'Content-Length': '0'} 2022-09-02T05:57:59.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bea10beb83ba707-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.295 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.296 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.296 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.297 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.298 DEBUG:boto:Method: GET 2022-09-02T05:57:59.298 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.299 DEBUG:boto:Data: 2022-09-02T05:57:59.299 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.300 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.301 DEBUG:boto:Params: {} 2022-09-02T05:57:59.301 DEBUG:boto:Token: None 2022-09-02T05:57:59.302 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.302 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E= 2022-09-02T05:57:59.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E=', 'Content-Length': '0'} 2022-09-02T05:57:59.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000090ea03a9a388b59-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.346 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.347 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.347 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.348 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.348 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.349 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.349 DEBUG:boto:Data: 2022-09-02T05:57:59.350 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.351 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.351 DEBUG:boto:Params: {} 2022-09-02T05:57:59.352 DEBUG:boto:Token: None 2022-09-02T05:57:59.352 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.353 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MOS/ZeLoYT7OXQWmrlqZH2htSsM= 2022-09-02T05:57:59.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MOS/ZeLoYT7OXQWmrlqZH2htSsM=', 'Content-Length': '0'} 2022-09-02T05:57:59.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaee55ed5ee12244-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.356 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.357 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.358 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.358 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.359 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.359 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.360 DEBUG:boto:Data: 2022-09-02T05:57:59.360 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.361 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.361 DEBUG:boto:Params: {} 2022-09-02T05:57:59.362 DEBUG:boto:Token: None 2022-09-02T05:57:59.363 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MOS/ZeLoYT7OXQWmrlqZH2htSsM= 2022-09-02T05:57:59.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MOS/ZeLoYT7OXQWmrlqZH2htSsM=', 'Content-Length': '0'} 2022-09-02T05:57:59.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080760e228b5e0cc8-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:57:59.367 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.368 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.368 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.369 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.369 DEBUG:boto:Method: GET 2022-09-02T05:57:59.370 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.370 DEBUG:boto:Data: 2022-09-02T05:57:59.371 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.372 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.372 DEBUG:boto:Params: {} 2022-09-02T05:57:59.373 DEBUG:boto:Token: None 2022-09-02T05:57:59.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E= 2022-09-02T05:57:59.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E=', 'Content-Length': '0'} 2022-09-02T05:57:59.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ebf4e1d2846c2c0-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.378 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.378 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.379 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.379 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.380 DEBUG:boto:Method: GET 2022-09-02T05:57:59.380 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.381 DEBUG:boto:Data: 2022-09-02T05:57:59.382 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.383 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.383 DEBUG:boto:Params: {} 2022-09-02T05:57:59.384 DEBUG:boto:Token: None 2022-09-02T05:57:59.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:57:59.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E= 2022-09-02T05:57:59.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AkIGGfpgqTu0UYuLRmb3zZw205E=', 'Content-Length': '0'} 2022-09-02T05:57:59.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009474e794bc4e93e4-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.389 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.389 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.390 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.390 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.391 DEBUG:boto:Method: GET 2022-09-02T05:57:59.391 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.392 DEBUG:boto:Data: 2022-09-02T05:57:59.392 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.393 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.394 DEBUG:boto:Params: {} 2022-09-02T05:57:59.394 DEBUG:boto:Token: None 2022-09-02T05:57:59.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4a7d6355b2eb6d-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.403 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.404 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.405 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.406 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.407 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.408 DEBUG:boto:Method: GET 2022-09-02T05:57:59.409 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.410 DEBUG:boto:Data: 2022-09-02T05:57:59.411 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.413 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.414 DEBUG:boto:Params: {} 2022-09-02T05:57:59.416 DEBUG:boto:Token: None 2022-09-02T05:57:59.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a52a694df2281127-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.426 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.427 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.428 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.429 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.430 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:59.431 DEBUG:boto:Data: 2022-09-02T05:57:59.431 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.432 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.433 DEBUG:boto:Params: {} 2022-09-02T05:57:59.434 DEBUG:boto:Token: None 2022-09-02T05:57:59.435 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155 2022-09-02T05:57:59.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O3Z28hzCIYp8F7QMhPaCw11YzRk= 2022-09-02T05:57:59.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O3Z28hzCIYp8F7QMhPaCw11YzRk=', 'Content-Length': '0'} 2022-09-02T05:57:59.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c387f32916d42ef-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.442 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.443 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.444 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.445 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:59.446 DEBUG:boto:Data: 2022-09-02T05:57:59.447 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.449 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.450 DEBUG:boto:Params: {} 2022-09-02T05:57:59.451 DEBUG:boto:Token: None 2022-09-02T05:57:59.452 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155 2022-09-02T05:57:59.453 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O3Z28hzCIYp8F7QMhPaCw11YzRk= 2022-09-02T05:57:59.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O3Z28hzCIYp8F7QMhPaCw11YzRk=', 'Content-Length': '0'} 2022-09-02T05:57:59.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313701a65fbf6368-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:57:59.460 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.461 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.462 DEBUG:boto:Method: GET 2022-09-02T05:57:59.463 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:59.464 DEBUG:boto:Data: 2022-09-02T05:57:59.465 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.467 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.468 DEBUG:boto:Params: {} 2022-09-02T05:57:59.469 DEBUG:boto:Token: None 2022-09-02T05:57:59.470 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155 2022-09-02T05:57:59.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyffw9gyv/GInCZp/f8Rm7KccAU= 2022-09-02T05:57:59.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyffw9gyv/GInCZp/f8Rm7KccAU=', 'Content-Length': '0'} 2022-09-02T05:57:59.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078434531afdc5a0e-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.476 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:57:59.477 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:57:59.477 DEBUG:boto:Method: GET 2022-09-02T05:57:59.478 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:57:59.478 DEBUG:boto:Data: 2022-09-02T05:57:59.479 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.480 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.481 DEBUG:boto:Params: {} 2022-09-02T05:57:59.481 DEBUG:boto:Token: None 2022-09-02T05:57:59.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj155 2022-09-02T05:57:59.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wyffw9gyv/GInCZp/f8Rm7KccAU= 2022-09-02T05:57:59.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wyffw9gyv/GInCZp/f8Rm7KccAU=', 'Content-Length': '0'} 2022-09-02T05:57:59.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7ff54b058279671-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.487 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.487 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.488 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.488 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.489 DEBUG:boto:Method: GET 2022-09-02T05:57:59.489 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.490 DEBUG:boto:Data: 2022-09-02T05:57:59.491 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.492 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.492 DEBUG:boto:Params: {} 2022-09-02T05:57:59.493 DEBUG:boto:Token: None 2022-09-02T05:57:59.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002240f58d246ec830-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.501 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.502 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.503 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.503 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.504 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.504 DEBUG:boto:Method: GET 2022-09-02T05:57:59.505 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:57:59.505 DEBUG:boto:Data: 2022-09-02T05:57:59.506 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.507 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.508 DEBUG:boto:Params: {} 2022-09-02T05:57:59.508 DEBUG:boto:Token: None 2022-09-02T05:57:59.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73eb4bb8e9ef9d9-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.517 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:57:59.519 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.520 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.521 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.521 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.522 DEBUG:boto:Method: GET 2022-09-02T05:57:59.523 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.523 DEBUG:boto:Data: 2022-09-02T05:57:59.524 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.525 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.525 DEBUG:boto:Params: {} 2022-09-02T05:57:59.526 DEBUG:boto:Token: None 2022-09-02T05:57:59.527 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM= 2022-09-02T05:57:59.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM=', 'Content-Length': '0'} 2022-09-02T05:57:59.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71e9fbd5a9b926c-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.531 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.532 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.533 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.533 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.534 DEBUG:boto:Method: GET 2022-09-02T05:57:59.534 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.535 DEBUG:boto:Data: 2022-09-02T05:57:59.536 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.537 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.538 DEBUG:boto:Params: {} 2022-09-02T05:57:59.538 DEBUG:boto:Token: None 2022-09-02T05:57:59.539 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM= 2022-09-02T05:57:59.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM=', 'Content-Length': '0'} 2022-09-02T05:57:59.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7d3a7ec723ac9f-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.586 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.588 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.589 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.590 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.590 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.591 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.592 DEBUG:boto:Data: 2022-09-02T05:57:59.592 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.594 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.595 DEBUG:boto:Params: {} 2022-09-02T05:57:59.596 DEBUG:boto:Token: None 2022-09-02T05:57:59.597 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hloYkvEZW/Ys/NjmqWhmZWKMh5k= 2022-09-02T05:57:59.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hloYkvEZW/Ys/NjmqWhmZWKMh5k=', 'Content-Length': '0'} 2022-09-02T05:57:59.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0262ea294e9f80e-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.602 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.603 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.604 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.604 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.605 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.605 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.606 DEBUG:boto:Data: 2022-09-02T05:57:59.606 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.608 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.608 DEBUG:boto:Params: {} 2022-09-02T05:57:59.609 DEBUG:boto:Token: None 2022-09-02T05:57:59.610 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hloYkvEZW/Ys/NjmqWhmZWKMh5k= 2022-09-02T05:57:59.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hloYkvEZW/Ys/NjmqWhmZWKMh5k=', 'Content-Length': '0'} 2022-09-02T05:57:59.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd47c966f2d694d-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:57:59.614 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.615 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.615 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.616 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.616 DEBUG:boto:Method: GET 2022-09-02T05:57:59.617 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.617 DEBUG:boto:Data: 2022-09-02T05:57:59.618 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.619 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.620 DEBUG:boto:Params: {} 2022-09-02T05:57:59.620 DEBUG:boto:Token: None 2022-09-02T05:57:59.621 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM= 2022-09-02T05:57:59.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM=', 'Content-Length': '0'} 2022-09-02T05:57:59.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee6afc59aa192fe-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.625 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.626 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.626 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.627 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.627 DEBUG:boto:Method: GET 2022-09-02T05:57:59.628 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.628 DEBUG:boto:Data: 2022-09-02T05:57:59.629 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.630 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.630 DEBUG:boto:Params: {} 2022-09-02T05:57:59.631 DEBUG:boto:Token: None 2022-09-02T05:57:59.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:57:59.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM= 2022-09-02T05:57:59.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0IEFyULTB0mR8KRGjjjQc7kiLiM=', 'Content-Length': '0'} 2022-09-02T05:57:59.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e15a484c599bd7-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.636 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.637 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.637 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.638 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.639 DEBUG:boto:Method: GET 2022-09-02T05:57:59.639 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.640 DEBUG:boto:Data: 2022-09-02T05:57:59.640 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.641 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.642 DEBUG:boto:Params: {} 2022-09-02T05:57:59.642 DEBUG:boto:Token: None 2022-09-02T05:57:59.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ce98123d8972ce6-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.647 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.648 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.649 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.650 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.650 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.651 DEBUG:boto:Method: GET 2022-09-02T05:57:59.651 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.652 DEBUG:boto:Data: 2022-09-02T05:57:59.652 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.653 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.654 DEBUG:boto:Params: {} 2022-09-02T05:57:59.655 DEBUG:boto:Token: None 2022-09-02T05:57:59.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000847ec4b82b5158a2-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.661 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.662 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.663 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.663 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.664 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:59.664 DEBUG:boto:Data: 2022-09-02T05:57:59.665 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.666 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.666 DEBUG:boto:Params: {} 2022-09-02T05:57:59.667 DEBUG:boto:Token: None 2022-09-02T05:57:59.668 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17 2022-09-02T05:57:59.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X/nMqKpCOi1+2RE2Ax4utgUbwRc= 2022-09-02T05:57:59.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X/nMqKpCOi1+2RE2Ax4utgUbwRc=', 'Content-Length': '0'} 2022-09-02T05:57:59.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068ccd51b1c441e8b-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.672 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.672 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.673 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.673 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:59.674 DEBUG:boto:Data: 2022-09-02T05:57:59.674 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.675 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.676 DEBUG:boto:Params: {} 2022-09-02T05:57:59.676 DEBUG:boto:Token: None 2022-09-02T05:57:59.677 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17 2022-09-02T05:57:59.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X/nMqKpCOi1+2RE2Ax4utgUbwRc= 2022-09-02T05:57:59.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X/nMqKpCOi1+2RE2Ax4utgUbwRc=', 'Content-Length': '0'} 2022-09-02T05:57:59.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002367db48613f798-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:57:59.681 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.682 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.682 DEBUG:boto:Method: GET 2022-09-02T05:57:59.683 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:59.683 DEBUG:boto:Data: 2022-09-02T05:57:59.684 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.685 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.686 DEBUG:boto:Params: {} 2022-09-02T05:57:59.686 DEBUG:boto:Token: None 2022-09-02T05:57:59.687 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17 2022-09-02T05:57:59.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7mCCk4mMM5h0mbBD2XRgzH4gjd0= 2022-09-02T05:57:59.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7mCCk4mMM5h0mbBD2XRgzH4gjd0=', 'Content-Length': '0'} 2022-09-02T05:57:59.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15a619de4178950-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.734 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:57:59.735 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:57:59.735 DEBUG:boto:Method: GET 2022-09-02T05:57:59.736 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:57:59.737 DEBUG:boto:Data: 2022-09-02T05:57:59.737 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.739 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.739 DEBUG:boto:Params: {} 2022-09-02T05:57:59.740 DEBUG:boto:Token: None 2022-09-02T05:57:59.741 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj17 2022-09-02T05:57:59.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7mCCk4mMM5h0mbBD2XRgzH4gjd0= 2022-09-02T05:57:59.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7mCCk4mMM5h0mbBD2XRgzH4gjd0=', 'Content-Length': '0'} 2022-09-02T05:57:59.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ad0c47d150c9e2-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.745 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.746 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.747 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.747 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.748 DEBUG:boto:Method: GET 2022-09-02T05:57:59.749 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.749 DEBUG:boto:Data: 2022-09-02T05:57:59.750 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.751 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.752 DEBUG:boto:Params: {} 2022-09-02T05:57:59.752 DEBUG:boto:Token: None 2022-09-02T05:57:59.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4ae5b1493052ba8-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.759 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.759 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.760 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.761 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.762 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.763 DEBUG:boto:Method: GET 2022-09-02T05:57:59.763 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:57:59.764 DEBUG:boto:Data: 2022-09-02T05:57:59.765 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.767 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.767 DEBUG:boto:Params: {} 2022-09-02T05:57:59.768 DEBUG:boto:Token: None 2022-09-02T05:57:59.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046540dc5575918ad-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.775 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:57:59.777 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.778 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.778 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.779 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.779 DEBUG:boto:Method: GET 2022-09-02T05:57:59.780 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.780 DEBUG:boto:Data: 2022-09-02T05:57:59.781 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.782 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.782 DEBUG:boto:Params: {} 2022-09-02T05:57:59.783 DEBUG:boto:Token: None 2022-09-02T05:57:59.784 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.785 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M= 2022-09-02T05:57:59.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M=', 'Content-Length': '0'} 2022-09-02T05:57:59.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008306f71ff07adf8b-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.789 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.789 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.790 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.791 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.791 DEBUG:boto:Method: GET 2022-09-02T05:57:59.792 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.792 DEBUG:boto:Data: 2022-09-02T05:57:59.793 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.794 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.794 DEBUG:boto:Params: {} 2022-09-02T05:57:59.795 DEBUG:boto:Token: None 2022-09-02T05:57:59.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M= 2022-09-02T05:57:59.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M=', 'Content-Length': '0'} 2022-09-02T05:57:59.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29966eea007976b-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.842 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.843 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.844 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.845 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.846 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.847 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.848 DEBUG:boto:Data: 2022-09-02T05:57:59.849 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.851 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.852 DEBUG:boto:Params: {} 2022-09-02T05:57:59.853 DEBUG:boto:Token: None 2022-09-02T05:57:59.854 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VE49QprvyTqY7ynLYtE+1gRNs8Y= 2022-09-02T05:57:59.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VE49QprvyTqY7ynLYtE+1gRNs8Y=', 'Content-Length': '0'} 2022-09-02T05:57:59.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074021e6c9e301019-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.860 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.861 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.862 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.863 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.864 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.865 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.866 DEBUG:boto:Data: 2022-09-02T05:57:59.880 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.881 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.882 DEBUG:boto:Params: {} 2022-09-02T05:57:59.883 DEBUG:boto:Token: None 2022-09-02T05:57:59.884 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VE49QprvyTqY7ynLYtE+1gRNs8Y= 2022-09-02T05:57:59.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VE49QprvyTqY7ynLYtE+1gRNs8Y=', 'Content-Length': '0'} 2022-09-02T05:57:59.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000461929e223b60fd3-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:57:59.891 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.892 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.893 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.894 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.894 DEBUG:boto:Method: GET 2022-09-02T05:57:59.895 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.896 DEBUG:boto:Data: 2022-09-02T05:57:59.896 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.898 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.899 DEBUG:boto:Params: {} 2022-09-02T05:57:59.899 DEBUG:boto:Token: None 2022-09-02T05:57:59.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.901 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M= 2022-09-02T05:57:59.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M=', 'Content-Length': '0'} 2022-09-02T05:57:59.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2565096384dbe8-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.905 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.906 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.906 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.907 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.907 DEBUG:boto:Method: GET 2022-09-02T05:57:59.908 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.908 DEBUG:boto:Data: 2022-09-02T05:57:59.909 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.910 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.910 DEBUG:boto:Params: {} 2022-09-02T05:57:59.911 DEBUG:boto:Token: None 2022-09-02T05:57:59.912 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:57:59.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M= 2022-09-02T05:57:59.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OUU79p2Gq4mTCyufiJk8luXBr0M=', 'Content-Length': '0'} 2022-09-02T05:57:59.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000482e57ab14432910-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.916 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.917 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.917 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.918 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.918 DEBUG:boto:Method: GET 2022-09-02T05:57:59.919 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.919 DEBUG:boto:Data: 2022-09-02T05:57:59.920 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.921 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.922 DEBUG:boto:Params: {} 2022-09-02T05:57:59.922 DEBUG:boto:Token: None 2022-09-02T05:57:59.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc080c9898c90f5a-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.928 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.929 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.930 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.931 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.932 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.933 DEBUG:boto:Method: GET 2022-09-02T05:57:59.934 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.934 DEBUG:boto:Data: 2022-09-02T05:57:59.935 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.937 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.938 DEBUG:boto:Params: {} 2022-09-02T05:57:59.939 DEBUG:boto:Token: None 2022-09-02T05:57:59.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:57:59.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a4908acf88f72ce-0063119b67-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.947 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:57:59.948 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.949 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.950 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.950 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:59.951 DEBUG:boto:Data: 2022-09-02T05:57:59.951 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.952 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.953 DEBUG:boto:Params: {} 2022-09-02T05:57:59.953 DEBUG:boto:Token: None 2022-09-02T05:57:59.954 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19 2022-09-02T05:57:59.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Embk8jaQ7/vGj+pdEM2GKULxtq0= 2022-09-02T05:57:59.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Embk8jaQ7/vGj+pdEM2GKULxtq0=', 'Content-Length': '0'} 2022-09-02T05:57:59.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9729ba6a0c9eca0-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.959 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.959 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.960 DEBUG:boto:Method: HEAD 2022-09-02T05:57:59.960 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:59.961 DEBUG:boto:Data: 2022-09-02T05:57:59.961 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.962 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.963 DEBUG:boto:Params: {} 2022-09-02T05:57:59.964 DEBUG:boto:Token: None 2022-09-02T05:57:59.964 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19 2022-09-02T05:57:59.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Embk8jaQ7/vGj+pdEM2GKULxtq0= 2022-09-02T05:57:59.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Embk8jaQ7/vGj+pdEM2GKULxtq0=', 'Content-Length': '0'} 2022-09-02T05:57:59.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc3d2cc5b7d5895-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:57:59.969 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.969 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.970 DEBUG:boto:Method: GET 2022-09-02T05:57:59.970 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:59.971 DEBUG:boto:Data: 2022-09-02T05:57:59.971 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.972 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.973 DEBUG:boto:Params: {} 2022-09-02T05:57:59.974 DEBUG:boto:Token: None 2022-09-02T05:57:59.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19 2022-09-02T05:57:59.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2HjWH3LC/YGkApS9bpumUpfi+iM= 2022-09-02T05:57:59.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2HjWH3LC/YGkApS9bpumUpfi+iM=', 'Content-Length': '0'} 2022-09-02T05:57:59.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e063eb6afe5d4495-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.978 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:57:59.979 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:57:59.980 DEBUG:boto:Method: GET 2022-09-02T05:57:59.980 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:57:59.981 DEBUG:boto:Data: 2022-09-02T05:57:59.981 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:57:59.983 DEBUG:boto:Port: 8000 2022-09-02T05:57:59.983 DEBUG:boto:Params: {} 2022-09-02T05:57:59.984 DEBUG:boto:Token: None 2022-09-02T05:57:59.985 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/obj19 2022-09-02T05:57:59.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2HjWH3LC/YGkApS9bpumUpfi+iM= 2022-09-02T05:57:59.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2HjWH3LC/YGkApS9bpumUpfi+iM=', 'Content-Length': '0'} 2022-09-02T05:57:59.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c7195d2388a550-0063119b67-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:57:59.989 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:57:59.990 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:57:59.991 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.991 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.992 DEBUG:boto:Method: GET 2022-09-02T05:57:59.992 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:57:59.993 DEBUG:boto:Data: 2022-09-02T05:57:59.993 DEBUG:boto:Headers: {} 2022-09-02T05:57:59.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:57:59.994 DEBUG:boto:Port: 8001 2022-09-02T05:57:59.995 DEBUG:boto:Params: {} 2022-09-02T05:57:59.996 DEBUG:boto:Token: None 2022-09-02T05:57:59.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:57:59 GMT /xljbkf-74/ 2022-09-02T05:57:59.997 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80= 2022-09-02T05:57:59.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:57:59 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K5OCkJmlcTdCJqZjuQjVOiThr80=', 'Content-Length': '0'} 2022-09-02T05:58:00.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c51472226e7514c-0063119b67-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:57:59 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.002 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.003 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.004 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.005 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:00.006 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:00.007 DEBUG:boto:Method: GET 2022-09-02T05:58:00.007 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:58:00.008 DEBUG:boto:Data: 2022-09-02T05:58:00.009 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.011 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.012 DEBUG:boto:Params: {} 2022-09-02T05:58:00.013 DEBUG:boto:Token: None 2022-09-02T05:58:00.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016193246c80335d7-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.020 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:58:00.022 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.023 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.024 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.025 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.026 DEBUG:boto:Method: GET 2022-09-02T05:58:00.027 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.028 DEBUG:boto:Data: 2022-09-02T05:58:00.029 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.031 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.031 DEBUG:boto:Params: {} 2022-09-02T05:58:00.032 DEBUG:boto:Token: None 2022-09-02T05:58:00.033 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA= 2022-09-02T05:58:00.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA=', 'Content-Length': '0'} 2022-09-02T05:58:00.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007653bd6f0e29a62-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.037 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.037 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.038 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.039 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.039 DEBUG:boto:Method: GET 2022-09-02T05:58:00.040 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.040 DEBUG:boto:Data: 2022-09-02T05:58:00.041 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.042 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.043 DEBUG:boto:Params: {} 2022-09-02T05:58:00.044 DEBUG:boto:Token: None 2022-09-02T05:58:00.044 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA= 2022-09-02T05:58:00.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA=', 'Content-Length': '0'} 2022-09-02T05:58:00.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076db86f5b186a745-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.049 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.049 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.050 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.050 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.051 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.052 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.052 DEBUG:boto:Data: 2022-09-02T05:58:00.053 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.054 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.054 DEBUG:boto:Params: {} 2022-09-02T05:58:00.055 DEBUG:boto:Token: None 2022-09-02T05:58:00.055 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NbFyTt2+KvQrJNof+e/o+uQ7chM= 2022-09-02T05:58:00.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NbFyTt2+KvQrJNof+e/o+uQ7chM=', 'Content-Length': '0'} 2022-09-02T05:58:00.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106cc51168493e0e-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.060 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.061 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.062 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.062 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.063 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.064 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.065 DEBUG:boto:Data: 2022-09-02T05:58:00.066 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.067 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.068 DEBUG:boto:Params: {} 2022-09-02T05:58:00.069 DEBUG:boto:Token: None 2022-09-02T05:58:00.070 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NbFyTt2+KvQrJNof+e/o+uQ7chM= 2022-09-02T05:58:00.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NbFyTt2+KvQrJNof+e/o+uQ7chM=', 'Content-Length': '0'} 2022-09-02T05:58:00.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ab71cba0cd6fe7-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:58:00.076 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.077 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.077 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.078 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.078 DEBUG:boto:Method: GET 2022-09-02T05:58:00.079 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.079 DEBUG:boto:Data: 2022-09-02T05:58:00.080 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.081 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.082 DEBUG:boto:Params: {} 2022-09-02T05:58:00.082 DEBUG:boto:Token: None 2022-09-02T05:58:00.083 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.084 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA= 2022-09-02T05:58:00.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA=', 'Content-Length': '0'} 2022-09-02T05:58:00.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b2983b7d893d50-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.130 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.131 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.131 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.132 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.133 DEBUG:boto:Method: GET 2022-09-02T05:58:00.133 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.134 DEBUG:boto:Data: 2022-09-02T05:58:00.134 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.135 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.136 DEBUG:boto:Params: {} 2022-09-02T05:58:00.137 DEBUG:boto:Token: None 2022-09-02T05:58:00.137 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:00.138 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA= 2022-09-02T05:58:00.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qUK3GK/0hEd0XPFGO57Whr3eGnA=', 'Content-Length': '0'} 2022-09-02T05:58:00.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009404f3d0d234d0c9-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.143 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.143 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.144 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.144 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.145 DEBUG:boto:Method: GET 2022-09-02T05:58:00.145 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.146 DEBUG:boto:Data: 2022-09-02T05:58:00.146 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.148 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.148 DEBUG:boto:Params: {} 2022-09-02T05:58:00.149 DEBUG:boto:Token: None 2022-09-02T05:58:00.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086951d44b516d8e3-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.155 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.156 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.156 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.157 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.157 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.158 DEBUG:boto:Method: GET 2022-09-02T05:58:00.159 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.159 DEBUG:boto:Data: 2022-09-02T05:58:00.160 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.161 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.161 DEBUG:boto:Params: {} 2022-09-02T05:58:00.162 DEBUG:boto:Token: None 2022-09-02T05:58:00.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0d400717984a5d1-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.169 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.170 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.171 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.171 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.172 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:00.172 DEBUG:boto:Data: 2022-09-02T05:58:00.173 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.174 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.174 DEBUG:boto:Params: {} 2022-09-02T05:58:00.175 DEBUG:boto:Token: None 2022-09-02T05:58:00.176 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22 2022-09-02T05:58:00.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kk9V46v8W4085ui4WEpN/55tCN8= 2022-09-02T05:58:00.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kk9V46v8W4085ui4WEpN/55tCN8=', 'Content-Length': '0'} 2022-09-02T05:58:00.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b242d59f3aa522a5-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.180 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.180 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.181 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.181 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:00.182 DEBUG:boto:Data: 2022-09-02T05:58:00.183 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.184 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.184 DEBUG:boto:Params: {} 2022-09-02T05:58:00.185 DEBUG:boto:Token: None 2022-09-02T05:58:00.185 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22 2022-09-02T05:58:00.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kk9V46v8W4085ui4WEpN/55tCN8= 2022-09-02T05:58:00.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kk9V46v8W4085ui4WEpN/55tCN8=', 'Content-Length': '0'} 2022-09-02T05:58:00.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001811e9441f8b93fe-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:58:00.190 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.191 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.191 DEBUG:boto:Method: GET 2022-09-02T05:58:00.192 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:00.192 DEBUG:boto:Data: 2022-09-02T05:58:00.193 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.195 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.196 DEBUG:boto:Params: {} 2022-09-02T05:58:00.197 DEBUG:boto:Token: None 2022-09-02T05:58:00.198 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22 2022-09-02T05:58:00.199 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zwMjp37sK83iPR45r7fbHdJeIWg= 2022-09-02T05:58:00.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zwMjp37sK83iPR45r7fbHdJeIWg=', 'Content-Length': '0'} 2022-09-02T05:58:00.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2554cdfb314553-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.246 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:00.247 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:00.247 DEBUG:boto:Method: GET 2022-09-02T05:58:00.248 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:00.249 DEBUG:boto:Data: 2022-09-02T05:58:00.249 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.250 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.251 DEBUG:boto:Params: {} 2022-09-02T05:58:00.252 DEBUG:boto:Token: None 2022-09-02T05:58:00.252 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj22 2022-09-02T05:58:00.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zwMjp37sK83iPR45r7fbHdJeIWg= 2022-09-02T05:58:00.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zwMjp37sK83iPR45r7fbHdJeIWg=', 'Content-Length': '0'} 2022-09-02T05:58:00.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f98e246ddfd262d-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.257 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.257 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.258 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.258 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.259 DEBUG:boto:Method: GET 2022-09-02T05:58:00.260 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.260 DEBUG:boto:Data: 2022-09-02T05:58:00.261 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.262 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.263 DEBUG:boto:Params: {} 2022-09-02T05:58:00.263 DEBUG:boto:Token: None 2022-09-02T05:58:00.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.265 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac044bffec24b5a2-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.269 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.270 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.271 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.272 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.272 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.273 DEBUG:boto:Method: GET 2022-09-02T05:58:00.274 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:00.275 DEBUG:boto:Data: 2022-09-02T05:58:00.276 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.277 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.277 DEBUG:boto:Params: {} 2022-09-02T05:58:00.278 DEBUG:boto:Token: None 2022-09-02T05:58:00.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.279 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5783066d5883bd-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.285 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:58:00.286 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.287 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.287 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.288 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.288 DEBUG:boto:Method: GET 2022-09-02T05:58:00.289 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.289 DEBUG:boto:Data: 2022-09-02T05:58:00.290 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.291 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.292 DEBUG:boto:Params: {} 2022-09-02T05:58:00.292 DEBUG:boto:Token: None 2022-09-02T05:58:00.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY= 2022-09-02T05:58:00.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY=', 'Content-Length': '0'} 2022-09-02T05:58:00.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003023a400a73a5e51-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.298 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.298 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.299 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.300 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.300 DEBUG:boto:Method: GET 2022-09-02T05:58:00.301 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.301 DEBUG:boto:Data: 2022-09-02T05:58:00.302 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.303 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.304 DEBUG:boto:Params: {} 2022-09-02T05:58:00.304 DEBUG:boto:Token: None 2022-09-02T05:58:00.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY= 2022-09-02T05:58:00.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY=', 'Content-Length': '0'} 2022-09-02T05:58:00.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002625335f900b6287-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.309 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.309 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.310 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.310 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.311 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.311 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.312 DEBUG:boto:Data: 2022-09-02T05:58:00.312 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.313 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.314 DEBUG:boto:Params: {} 2022-09-02T05:58:00.314 DEBUG:boto:Token: None 2022-09-02T05:58:00.315 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:URxJUhkcdnNMRAYweFvWGDoigwo= 2022-09-02T05:58:00.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:URxJUhkcdnNMRAYweFvWGDoigwo=', 'Content-Length': '0'} 2022-09-02T05:58:00.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a733c73fd61c07-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.319 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.320 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.320 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.321 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.321 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.322 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.322 DEBUG:boto:Data: 2022-09-02T05:58:00.323 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.324 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.324 DEBUG:boto:Params: {} 2022-09-02T05:58:00.325 DEBUG:boto:Token: None 2022-09-02T05:58:00.326 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:URxJUhkcdnNMRAYweFvWGDoigwo= 2022-09-02T05:58:00.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:URxJUhkcdnNMRAYweFvWGDoigwo=', 'Content-Length': '0'} 2022-09-02T05:58:00.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f00e3312a85cb53-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:58:00.331 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.331 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.332 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.332 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.333 DEBUG:boto:Method: GET 2022-09-02T05:58:00.333 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.334 DEBUG:boto:Data: 2022-09-02T05:58:00.335 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.336 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.336 DEBUG:boto:Params: {} 2022-09-02T05:58:00.337 DEBUG:boto:Token: None 2022-09-02T05:58:00.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY= 2022-09-02T05:58:00.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY=', 'Content-Length': '0'} 2022-09-02T05:58:00.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41cfa27e4669418-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.342 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.343 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.343 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.344 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.345 DEBUG:boto:Method: GET 2022-09-02T05:58:00.346 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.347 DEBUG:boto:Data: 2022-09-02T05:58:00.348 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.350 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.351 DEBUG:boto:Params: {} 2022-09-02T05:58:00.352 DEBUG:boto:Token: None 2022-09-02T05:58:00.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:00.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY= 2022-09-02T05:58:00.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dCL7YOi42ilc/TF66BGx+HYpSCY=', 'Content-Length': '0'} 2022-09-02T05:58:00.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb4fa82dc84d762-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.359 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.359 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.360 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.360 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.361 DEBUG:boto:Method: GET 2022-09-02T05:58:00.361 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.362 DEBUG:boto:Data: 2022-09-02T05:58:00.363 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.364 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.364 DEBUG:boto:Params: {} 2022-09-02T05:58:00.365 DEBUG:boto:Token: None 2022-09-02T05:58:00.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ded43e7b05aa92-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.370 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.371 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.371 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.372 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.372 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.373 DEBUG:boto:Method: GET 2022-09-02T05:58:00.373 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.374 DEBUG:boto:Data: 2022-09-02T05:58:00.374 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.376 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.377 DEBUG:boto:Params: {} 2022-09-02T05:58:00.377 DEBUG:boto:Token: None 2022-09-02T05:58:00.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7dc0cce602c7b09-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.384 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.385 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.386 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.387 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.388 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:00.389 DEBUG:boto:Data: 2022-09-02T05:58:00.390 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.392 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.392 DEBUG:boto:Params: {} 2022-09-02T05:58:00.394 DEBUG:boto:Token: None 2022-09-02T05:58:00.395 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23 2022-09-02T05:58:00.396 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqICyj+AOOihiBd5kK6mqh1mvHM= 2022-09-02T05:58:00.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqICyj+AOOihiBd5kK6mqh1mvHM=', 'Content-Length': '0'} 2022-09-02T05:58:00.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a7a2d596c86a7a6-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.401 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.402 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.403 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.403 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:00.404 DEBUG:boto:Data: 2022-09-02T05:58:00.405 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.407 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.408 DEBUG:boto:Params: {} 2022-09-02T05:58:00.409 DEBUG:boto:Token: None 2022-09-02T05:58:00.410 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23 2022-09-02T05:58:00.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZqICyj+AOOihiBd5kK6mqh1mvHM= 2022-09-02T05:58:00.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZqICyj+AOOihiBd5kK6mqh1mvHM=', 'Content-Length': '0'} 2022-09-02T05:58:00.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007970b0f7c523d47f-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:58:00.416 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.417 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.417 DEBUG:boto:Method: GET 2022-09-02T05:58:00.418 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:00.418 DEBUG:boto:Data: 2022-09-02T05:58:00.419 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.420 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.421 DEBUG:boto:Params: {} 2022-09-02T05:58:00.421 DEBUG:boto:Token: None 2022-09-02T05:58:00.422 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23 2022-09-02T05:58:00.423 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oq3IU81oMiMTezn3c7GE+y8Qj4s= 2022-09-02T05:58:00.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oq3IU81oMiMTezn3c7GE+y8Qj4s=', 'Content-Length': '0'} 2022-09-02T05:58:00.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099568da29d81c2f2-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.426 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:00.427 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:00.428 DEBUG:boto:Method: GET 2022-09-02T05:58:00.429 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:00.429 DEBUG:boto:Data: 2022-09-02T05:58:00.430 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.431 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.431 DEBUG:boto:Params: {} 2022-09-02T05:58:00.432 DEBUG:boto:Token: None 2022-09-02T05:58:00.432 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj23 2022-09-02T05:58:00.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oq3IU81oMiMTezn3c7GE+y8Qj4s= 2022-09-02T05:58:00.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oq3IU81oMiMTezn3c7GE+y8Qj4s=', 'Content-Length': '0'} 2022-09-02T05:58:00.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012474c4944a5fe25-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.478 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.480 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.480 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.481 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.481 DEBUG:boto:Method: GET 2022-09-02T05:58:00.482 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.483 DEBUG:boto:Data: 2022-09-02T05:58:00.483 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.484 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.485 DEBUG:boto:Params: {} 2022-09-02T05:58:00.485 DEBUG:boto:Token: None 2022-09-02T05:58:00.486 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e6c55a2fb24f409-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.491 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.492 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.492 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.493 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.494 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.494 DEBUG:boto:Method: GET 2022-09-02T05:58:00.495 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:00.497 DEBUG:boto:Data: 2022-09-02T05:58:00.498 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.500 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.501 DEBUG:boto:Params: {} 2022-09-02T05:58:00.502 DEBUG:boto:Token: None 2022-09-02T05:58:00.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b44dae6c5eb8b6ff-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.511 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:58:00.513 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.513 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.514 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.515 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.516 DEBUG:boto:Method: GET 2022-09-02T05:58:00.517 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.518 DEBUG:boto:Data: 2022-09-02T05:58:00.519 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.522 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.523 DEBUG:boto:Params: {} 2022-09-02T05:58:00.524 DEBUG:boto:Token: None 2022-09-02T05:58:00.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ= 2022-09-02T05:58:00.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ=', 'Content-Length': '0'} 2022-09-02T05:58:00.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a754de2ecc46be0-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.570 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.571 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.571 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.572 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.572 DEBUG:boto:Method: GET 2022-09-02T05:58:00.573 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.573 DEBUG:boto:Data: 2022-09-02T05:58:00.574 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.575 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.576 DEBUG:boto:Params: {} 2022-09-02T05:58:00.577 DEBUG:boto:Token: None 2022-09-02T05:58:00.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ= 2022-09-02T05:58:00.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ=', 'Content-Length': '0'} 2022-09-02T05:58:00.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6aab9260a1bb5b9-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.582 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.582 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.583 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.583 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.584 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.584 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.585 DEBUG:boto:Data: 2022-09-02T05:58:00.586 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.587 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.588 DEBUG:boto:Params: {} 2022-09-02T05:58:00.588 DEBUG:boto:Token: None 2022-09-02T05:58:00.589 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8ZQPVxbXOmv9lYLA73x0p1TsF8= 2022-09-02T05:58:00.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8ZQPVxbXOmv9lYLA73x0p1TsF8=', 'Content-Length': '0'} 2022-09-02T05:58:00.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4acecfb5770674-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.593 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.594 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.595 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.596 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.597 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.598 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.599 DEBUG:boto:Data: 2022-09-02T05:58:00.600 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.602 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.602 DEBUG:boto:Params: {} 2022-09-02T05:58:00.604 DEBUG:boto:Token: None 2022-09-02T05:58:00.605 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s8ZQPVxbXOmv9lYLA73x0p1TsF8= 2022-09-02T05:58:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s8ZQPVxbXOmv9lYLA73x0p1TsF8=', 'Content-Length': '0'} 2022-09-02T05:58:00.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05e3c7b3434353e-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:58:00.609 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.610 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.611 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.611 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.612 DEBUG:boto:Method: GET 2022-09-02T05:58:00.612 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.613 DEBUG:boto:Data: 2022-09-02T05:58:00.613 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.615 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.615 DEBUG:boto:Params: {} 2022-09-02T05:58:00.616 DEBUG:boto:Token: None 2022-09-02T05:58:00.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ= 2022-09-02T05:58:00.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ=', 'Content-Length': '0'} 2022-09-02T05:58:00.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca278044a62125b-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.621 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.621 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.622 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.622 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.623 DEBUG:boto:Method: GET 2022-09-02T05:58:00.623 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.624 DEBUG:boto:Data: 2022-09-02T05:58:00.624 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.625 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.626 DEBUG:boto:Params: {} 2022-09-02T05:58:00.627 DEBUG:boto:Token: None 2022-09-02T05:58:00.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:00.628 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ= 2022-09-02T05:58:00.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QrKwDCQ12f3w5nsqB0epjBKaEyQ=', 'Content-Length': '0'} 2022-09-02T05:58:00.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9f52fb17882304-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.632 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.632 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.633 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.633 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.634 DEBUG:boto:Method: GET 2022-09-02T05:58:00.634 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.635 DEBUG:boto:Data: 2022-09-02T05:58:00.636 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.637 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.637 DEBUG:boto:Params: {} 2022-09-02T05:58:00.638 DEBUG:boto:Token: None 2022-09-02T05:58:00.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a3fe0db468a3f0-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.643 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.644 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.645 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.645 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.646 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.646 DEBUG:boto:Method: GET 2022-09-02T05:58:00.647 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.647 DEBUG:boto:Data: 2022-09-02T05:58:00.648 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.649 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.650 DEBUG:boto:Params: {} 2022-09-02T05:58:00.651 DEBUG:boto:Token: None 2022-09-02T05:58:00.651 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da99252a8e4cee7f-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.657 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.658 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.659 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.659 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.660 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:00.660 DEBUG:boto:Data: 2022-09-02T05:58:00.661 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.662 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.662 DEBUG:boto:Params: {} 2022-09-02T05:58:00.663 DEBUG:boto:Token: None 2022-09-02T05:58:00.663 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29 2022-09-02T05:58:00.664 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7LAxD/krInwc9AZa8mXT7fz/pTM= 2022-09-02T05:58:00.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7LAxD/krInwc9AZa8mXT7fz/pTM=', 'Content-Length': '0'} 2022-09-02T05:58:00.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f681af219e1e932b-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.667 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.668 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.668 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.669 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:00.669 DEBUG:boto:Data: 2022-09-02T05:58:00.670 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.671 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.671 DEBUG:boto:Params: {} 2022-09-02T05:58:00.672 DEBUG:boto:Token: None 2022-09-02T05:58:00.673 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29 2022-09-02T05:58:00.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7LAxD/krInwc9AZa8mXT7fz/pTM= 2022-09-02T05:58:00.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7LAxD/krInwc9AZa8mXT7fz/pTM=', 'Content-Length': '0'} 2022-09-02T05:58:00.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006498db8fcc8b9568-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:58:00.677 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.678 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.678 DEBUG:boto:Method: GET 2022-09-02T05:58:00.679 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:00.679 DEBUG:boto:Data: 2022-09-02T05:58:00.680 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.681 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.681 DEBUG:boto:Params: {} 2022-09-02T05:58:00.682 DEBUG:boto:Token: None 2022-09-02T05:58:00.682 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29 2022-09-02T05:58:00.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QBe/EzYgm37yFBKrXd0lKDFaDFU= 2022-09-02T05:58:00.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QBe/EzYgm37yFBKrXd0lKDFaDFU=', 'Content-Length': '0'} 2022-09-02T05:58:00.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdfc078e02bb3cd2-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.730 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:00.731 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:00.731 DEBUG:boto:Method: GET 2022-09-02T05:58:00.732 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:00.732 DEBUG:boto:Data: 2022-09-02T05:58:00.733 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.734 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.735 DEBUG:boto:Params: {} 2022-09-02T05:58:00.735 DEBUG:boto:Token: None 2022-09-02T05:58:00.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj29 2022-09-02T05:58:00.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QBe/EzYgm37yFBKrXd0lKDFaDFU= 2022-09-02T05:58:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QBe/EzYgm37yFBKrXd0lKDFaDFU=', 'Content-Length': '0'} 2022-09-02T05:58:00.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a65020159396fe7f-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.740 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.741 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.742 DEBUG:boto:Method: GET 2022-09-02T05:58:00.743 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.743 DEBUG:boto:Data: 2022-09-02T05:58:00.744 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.745 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.745 DEBUG:boto:Params: {} 2022-09-02T05:58:00.746 DEBUG:boto:Token: None 2022-09-02T05:58:00.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000831a574fe04d3d7f-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.751 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.751 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.752 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.752 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.753 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.753 DEBUG:boto:Method: GET 2022-09-02T05:58:00.754 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:00.754 DEBUG:boto:Data: 2022-09-02T05:58:00.755 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.756 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.757 DEBUG:boto:Params: {} 2022-09-02T05:58:00.757 DEBUG:boto:Token: None 2022-09-02T05:58:00.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000043975c11fde12c5-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.764 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:58:00.767 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.768 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.769 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.770 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.771 DEBUG:boto:Method: GET 2022-09-02T05:58:00.772 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.772 DEBUG:boto:Data: 2022-09-02T05:58:00.773 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.775 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.775 DEBUG:boto:Params: {} 2022-09-02T05:58:00.776 DEBUG:boto:Token: None 2022-09-02T05:58:00.776 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo= 2022-09-02T05:58:00.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo=', 'Content-Length': '0'} 2022-09-02T05:58:00.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813c0541120c17e4-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.781 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.782 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.782 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.783 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.783 DEBUG:boto:Method: GET 2022-09-02T05:58:00.784 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.785 DEBUG:boto:Data: 2022-09-02T05:58:00.785 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.786 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.787 DEBUG:boto:Params: {} 2022-09-02T05:58:00.787 DEBUG:boto:Token: None 2022-09-02T05:58:00.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo= 2022-09-02T05:58:00.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo=', 'Content-Length': '0'} 2022-09-02T05:58:00.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3213cc5e9b7fa3a-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.792 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.793 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.793 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.794 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.795 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.795 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.796 DEBUG:boto:Data: 2022-09-02T05:58:00.797 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.798 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.799 DEBUG:boto:Params: {} 2022-09-02T05:58:00.800 DEBUG:boto:Token: None 2022-09-02T05:58:00.802 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yy5JLhKH2G6JDTR3iGqBTO8d6FM= 2022-09-02T05:58:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yy5JLhKH2G6JDTR3iGqBTO8d6FM=', 'Content-Length': '0'} 2022-09-02T05:58:00.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051193d64dbb7c7fa-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.807 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.808 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.809 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.809 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.810 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.810 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.811 DEBUG:boto:Data: 2022-09-02T05:58:00.812 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.813 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.813 DEBUG:boto:Params: {} 2022-09-02T05:58:00.814 DEBUG:boto:Token: None 2022-09-02T05:58:00.815 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.815 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yy5JLhKH2G6JDTR3iGqBTO8d6FM= 2022-09-02T05:58:00.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yy5JLhKH2G6JDTR3iGqBTO8d6FM=', 'Content-Length': '0'} 2022-09-02T05:58:00.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95f11331b4bd341-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:58:00.820 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.820 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.821 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.821 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.822 DEBUG:boto:Method: GET 2022-09-02T05:58:00.823 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.823 DEBUG:boto:Data: 2022-09-02T05:58:00.824 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.825 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.826 DEBUG:boto:Params: {} 2022-09-02T05:58:00.827 DEBUG:boto:Token: None 2022-09-02T05:58:00.828 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo= 2022-09-02T05:58:00.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo=', 'Content-Length': '0'} 2022-09-02T05:58:00.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b817963748b29c90-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.874 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.875 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.876 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.876 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.877 DEBUG:boto:Method: GET 2022-09-02T05:58:00.878 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.879 DEBUG:boto:Data: 2022-09-02T05:58:00.879 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.881 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.881 DEBUG:boto:Params: {} 2022-09-02T05:58:00.882 DEBUG:boto:Token: None 2022-09-02T05:58:00.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:00.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo= 2022-09-02T05:58:00.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VmhGmXeg6m9atFD2AO5J6gDXrbo=', 'Content-Length': '0'} 2022-09-02T05:58:00.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073dd0df41c0ba39e-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.890 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.892 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.893 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.894 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.895 DEBUG:boto:Method: GET 2022-09-02T05:58:00.895 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.896 DEBUG:boto:Data: 2022-09-02T05:58:00.897 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.898 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.898 DEBUG:boto:Params: {} 2022-09-02T05:58:00.899 DEBUG:boto:Token: None 2022-09-02T05:58:00.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f155dffbb4743ae1-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.905 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.906 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.907 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.907 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.908 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.909 DEBUG:boto:Method: GET 2022-09-02T05:58:00.909 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.910 DEBUG:boto:Data: 2022-09-02T05:58:00.911 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.912 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.913 DEBUG:boto:Params: {} 2022-09-02T05:58:00.914 DEBUG:boto:Token: None 2022-09-02T05:58:00.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.916 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f95baceefe021bff-0063119b68-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.922 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.924 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.925 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.926 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.927 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:00.927 DEBUG:boto:Data: 2022-09-02T05:58:00.928 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.929 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.930 DEBUG:boto:Params: {} 2022-09-02T05:58:00.931 DEBUG:boto:Token: None 2022-09-02T05:58:00.933 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31 2022-09-02T05:58:00.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ItJc97dK6MjzEESTR4iutFGcSJ4= 2022-09-02T05:58:00.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ItJc97dK6MjzEESTR4iutFGcSJ4=', 'Content-Length': '0'} 2022-09-02T05:58:00.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004776f31f06cc73dc-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.939 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.940 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.941 DEBUG:boto:Method: HEAD 2022-09-02T05:58:00.942 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:00.942 DEBUG:boto:Data: 2022-09-02T05:58:00.943 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.944 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.945 DEBUG:boto:Params: {} 2022-09-02T05:58:00.946 DEBUG:boto:Token: None 2022-09-02T05:58:00.946 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31 2022-09-02T05:58:00.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ItJc97dK6MjzEESTR4iutFGcSJ4= 2022-09-02T05:58:00.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ItJc97dK6MjzEESTR4iutFGcSJ4=', 'Content-Length': '0'} 2022-09-02T05:58:00.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80307d71de602ac-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:58:00.951 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.952 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.953 DEBUG:boto:Method: GET 2022-09-02T05:58:00.953 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:00.954 DEBUG:boto:Data: 2022-09-02T05:58:00.954 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.956 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.956 DEBUG:boto:Params: {} 2022-09-02T05:58:00.957 DEBUG:boto:Token: None 2022-09-02T05:58:00.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31 2022-09-02T05:58:00.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LzaWnlC3eCxCcLgcgB3gdpnHh1Y= 2022-09-02T05:58:00.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LzaWnlC3eCxCcLgcgB3gdpnHh1Y=', 'Content-Length': '0'} 2022-09-02T05:58:00.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002686e193286c184e-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.962 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:00.963 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:00.964 DEBUG:boto:Method: GET 2022-09-02T05:58:00.965 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:00.965 DEBUG:boto:Data: 2022-09-02T05:58:00.966 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.968 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.968 DEBUG:boto:Params: {} 2022-09-02T05:58:00.969 DEBUG:boto:Token: None 2022-09-02T05:58:00.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/obj31 2022-09-02T05:58:00.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LzaWnlC3eCxCcLgcgB3gdpnHh1Y= 2022-09-02T05:58:00.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LzaWnlC3eCxCcLgcgB3gdpnHh1Y=', 'Content-Length': '0'} 2022-09-02T05:58:00.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000682845fab776a6e-0063119b68-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.975 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.975 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.976 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.977 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.977 DEBUG:boto:Method: GET 2022-09-02T05:58:00.978 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.979 DEBUG:boto:Data: 2022-09-02T05:58:00.980 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:00.981 DEBUG:boto:Port: 8001 2022-09-02T05:58:00.982 DEBUG:boto:Params: {} 2022-09-02T05:58:00.983 DEBUG:boto:Token: None 2022-09-02T05:58:00.984 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:00.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009068d1b3e19e19b3-0063119b68-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:00 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:00.989 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:00.991 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:00.991 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:00.992 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.992 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.993 DEBUG:boto:Method: GET 2022-09-02T05:58:00.993 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:00.994 DEBUG:boto:Data: 2022-09-02T05:58:00.995 DEBUG:boto:Headers: {} 2022-09-02T05:58:00.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:00.996 DEBUG:boto:Port: 8000 2022-09-02T05:58:00.997 DEBUG:boto:Params: {} 2022-09-02T05:58:00.997 DEBUG:boto:Token: None 2022-09-02T05:58:00.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:00 GMT /xljbkf-74/ 2022-09-02T05:58:00.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM= 2022-09-02T05:58:00.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:00 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:buow6Tfm49DTGaW1WlkghwS1BoM=', 'Content-Length': '0'} 2022-09-02T05:58:01.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046212f29c0bb2d06-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.007 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:58:01.008 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.009 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.009 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.010 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.011 DEBUG:boto:Method: GET 2022-09-02T05:58:01.011 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.012 DEBUG:boto:Data: 2022-09-02T05:58:01.013 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.014 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.015 DEBUG:boto:Params: {} 2022-09-02T05:58:01.016 DEBUG:boto:Token: None 2022-09-02T05:58:01.017 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM= 2022-09-02T05:58:01.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM=', 'Content-Length': '0'} 2022-09-02T05:58:01.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e426412c066c16e0-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.022 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.023 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.023 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.024 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.025 DEBUG:boto:Method: GET 2022-09-02T05:58:01.025 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.026 DEBUG:boto:Data: 2022-09-02T05:58:01.027 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.028 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.029 DEBUG:boto:Params: {} 2022-09-02T05:58:01.029 DEBUG:boto:Token: None 2022-09-02T05:58:01.031 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM= 2022-09-02T05:58:01.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM=', 'Content-Length': '0'} 2022-09-02T05:58:01.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3bad0648ca67018-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.078 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.079 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.079 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.080 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.080 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.081 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.081 DEBUG:boto:Data: 2022-09-02T05:58:01.082 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.083 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.084 DEBUG:boto:Params: {} 2022-09-02T05:58:01.084 DEBUG:boto:Token: None 2022-09-02T05:58:01.085 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qr4V3zrbej5UIG6REzv2Bz8POe4= 2022-09-02T05:58:01.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qr4V3zrbej5UIG6REzv2Bz8POe4=', 'Content-Length': '0'} 2022-09-02T05:58:01.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad549fc8683c8870-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.089 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.089 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.090 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.090 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.091 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.091 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.092 DEBUG:boto:Data: 2022-09-02T05:58:01.092 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.093 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.094 DEBUG:boto:Params: {} 2022-09-02T05:58:01.094 DEBUG:boto:Token: None 2022-09-02T05:58:01.095 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qr4V3zrbej5UIG6REzv2Bz8POe4= 2022-09-02T05:58:01.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qr4V3zrbej5UIG6REzv2Bz8POe4=', 'Content-Length': '0'} 2022-09-02T05:58:01.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f17e1d4358e7ad-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:58:01.099 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.100 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.100 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.101 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.101 DEBUG:boto:Method: GET 2022-09-02T05:58:01.102 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.102 DEBUG:boto:Data: 2022-09-02T05:58:01.103 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.104 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.104 DEBUG:boto:Params: {} 2022-09-02T05:58:01.105 DEBUG:boto:Token: None 2022-09-02T05:58:01.106 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM= 2022-09-02T05:58:01.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM=', 'Content-Length': '0'} 2022-09-02T05:58:01.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd72a7b26efd868-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.109 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.110 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.110 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.111 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.111 DEBUG:boto:Method: GET 2022-09-02T05:58:01.112 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.112 DEBUG:boto:Data: 2022-09-02T05:58:01.113 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.114 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.115 DEBUG:boto:Params: {} 2022-09-02T05:58:01.115 DEBUG:boto:Token: None 2022-09-02T05:58:01.116 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:01.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM= 2022-09-02T05:58:01.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVV6LYueLGGnJfUYJa8sHmM1xlM=', 'Content-Length': '0'} 2022-09-02T05:58:01.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081ec787f9006642a-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.120 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.120 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.121 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.121 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.122 DEBUG:boto:Method: GET 2022-09-02T05:58:01.122 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.123 DEBUG:boto:Data: 2022-09-02T05:58:01.123 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.124 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.125 DEBUG:boto:Params: {} 2022-09-02T05:58:01.126 DEBUG:boto:Token: None 2022-09-02T05:58:01.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ea6d5d39bb7d10-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.130 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.131 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.132 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.132 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.133 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.134 DEBUG:boto:Method: GET 2022-09-02T05:58:01.134 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.135 DEBUG:boto:Data: 2022-09-02T05:58:01.136 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.138 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.139 DEBUG:boto:Params: {} 2022-09-02T05:58:01.140 DEBUG:boto:Token: None 2022-09-02T05:58:01.141 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.142 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e38d1e7efd698941-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.148 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.149 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.150 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.151 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.151 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:01.152 DEBUG:boto:Data: 2022-09-02T05:58:01.152 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.153 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.154 DEBUG:boto:Params: {} 2022-09-02T05:58:01.155 DEBUG:boto:Token: None 2022-09-02T05:58:01.155 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33 2022-09-02T05:58:01.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aro5cF+vjXYlZ82LZcwdtjOhQ+4= 2022-09-02T05:58:01.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aro5cF+vjXYlZ82LZcwdtjOhQ+4=', 'Content-Length': '0'} 2022-09-02T05:58:01.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dec5c9cb9b893bb-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.159 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.160 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.161 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.161 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:01.162 DEBUG:boto:Data: 2022-09-02T05:58:01.163 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.163 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.164 DEBUG:boto:Params: {} 2022-09-02T05:58:01.165 DEBUG:boto:Token: None 2022-09-02T05:58:01.165 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33 2022-09-02T05:58:01.166 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Aro5cF+vjXYlZ82LZcwdtjOhQ+4= 2022-09-02T05:58:01.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Aro5cF+vjXYlZ82LZcwdtjOhQ+4=', 'Content-Length': '0'} 2022-09-02T05:58:01.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a2a97813bd71a5d-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:58:01.170 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.170 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.171 DEBUG:boto:Method: GET 2022-09-02T05:58:01.171 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:01.172 DEBUG:boto:Data: 2022-09-02T05:58:01.173 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.174 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.174 DEBUG:boto:Params: {} 2022-09-02T05:58:01.175 DEBUG:boto:Token: None 2022-09-02T05:58:01.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33 2022-09-02T05:58:01.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NF5cznHPqp8+yXWDa/Xcmbde1Mg= 2022-09-02T05:58:01.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NF5cznHPqp8+yXWDa/Xcmbde1Mg=', 'Content-Length': '0'} 2022-09-02T05:58:01.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000685a502b6123225b-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.222 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:01.223 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:01.223 DEBUG:boto:Method: GET 2022-09-02T05:58:01.224 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:01.224 DEBUG:boto:Data: 2022-09-02T05:58:01.225 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.226 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.227 DEBUG:boto:Params: {} 2022-09-02T05:58:01.227 DEBUG:boto:Token: None 2022-09-02T05:58:01.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj33 2022-09-02T05:58:01.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NF5cznHPqp8+yXWDa/Xcmbde1Mg= 2022-09-02T05:58:01.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NF5cznHPqp8+yXWDa/Xcmbde1Mg=', 'Content-Length': '0'} 2022-09-02T05:58:01.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31d0abe82e851a9-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.232 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.233 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.233 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.234 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.234 DEBUG:boto:Method: GET 2022-09-02T05:58:01.235 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.235 DEBUG:boto:Data: 2022-09-02T05:58:01.236 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.237 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.238 DEBUG:boto:Params: {} 2022-09-02T05:58:01.239 DEBUG:boto:Token: None 2022-09-02T05:58:01.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c04a4aeb99de872-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.247 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.249 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.250 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.251 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.252 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.253 DEBUG:boto:Method: GET 2022-09-02T05:58:01.255 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:01.256 DEBUG:boto:Data: 2022-09-02T05:58:01.257 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.259 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.260 DEBUG:boto:Params: {} 2022-09-02T05:58:01.261 DEBUG:boto:Token: None 2022-09-02T05:58:01.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000edc71a599e0db3b-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.272 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:58:01.274 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.275 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.276 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.277 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.278 DEBUG:boto:Method: GET 2022-09-02T05:58:01.279 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.280 DEBUG:boto:Data: 2022-09-02T05:58:01.281 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.283 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.284 DEBUG:boto:Params: {} 2022-09-02T05:58:01.285 DEBUG:boto:Token: None 2022-09-02T05:58:01.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8= 2022-09-02T05:58:01.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8=', 'Content-Length': '0'} 2022-09-02T05:58:01.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ecd465172536fe5-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.294 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.295 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.296 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.297 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.299 DEBUG:boto:Method: GET 2022-09-02T05:58:01.300 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.301 DEBUG:boto:Data: 2022-09-02T05:58:01.302 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.304 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.305 DEBUG:boto:Params: {} 2022-09-02T05:58:01.306 DEBUG:boto:Token: None 2022-09-02T05:58:01.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8= 2022-09-02T05:58:01.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8=', 'Content-Length': '0'} 2022-09-02T05:58:01.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef0336b7bbf17ed7-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.312 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.313 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.313 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.314 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.314 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.315 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.315 DEBUG:boto:Data: 2022-09-02T05:58:01.316 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.318 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.318 DEBUG:boto:Params: {} 2022-09-02T05:58:01.319 DEBUG:boto:Token: None 2022-09-02T05:58:01.319 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DdvsX+YAmNercx/x6D9HQ4ZoCGo= 2022-09-02T05:58:01.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DdvsX+YAmNercx/x6D9HQ4ZoCGo=', 'Content-Length': '0'} 2022-09-02T05:58:01.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df23ec45624d51f6-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.323 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.324 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.324 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.325 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.325 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.326 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.326 DEBUG:boto:Data: 2022-09-02T05:58:01.327 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.328 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.329 DEBUG:boto:Params: {} 2022-09-02T05:58:01.329 DEBUG:boto:Token: None 2022-09-02T05:58:01.330 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DdvsX+YAmNercx/x6D9HQ4ZoCGo= 2022-09-02T05:58:01.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DdvsX+YAmNercx/x6D9HQ4ZoCGo=', 'Content-Length': '0'} 2022-09-02T05:58:01.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ca401e37460b7c2-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:58:01.335 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.335 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.336 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.337 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.337 DEBUG:boto:Method: GET 2022-09-02T05:58:01.338 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.338 DEBUG:boto:Data: 2022-09-02T05:58:01.339 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.340 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.340 DEBUG:boto:Params: {} 2022-09-02T05:58:01.341 DEBUG:boto:Token: None 2022-09-02T05:58:01.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8= 2022-09-02T05:58:01.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8=', 'Content-Length': '0'} 2022-09-02T05:58:01.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c629abfa1b100004-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.358 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.359 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.360 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.360 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.361 DEBUG:boto:Method: GET 2022-09-02T05:58:01.361 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.362 DEBUG:boto:Data: 2022-09-02T05:58:01.362 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.363 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.364 DEBUG:boto:Params: {} 2022-09-02T05:58:01.365 DEBUG:boto:Token: None 2022-09-02T05:58:01.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:01.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8= 2022-09-02T05:58:01.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZPQ5ajYG+WtbhabAs2MvJhuQSp8=', 'Content-Length': '0'} 2022-09-02T05:58:01.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006945a64f923e75b4-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.370 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.370 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.371 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.371 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.372 DEBUG:boto:Method: GET 2022-09-02T05:58:01.373 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.373 DEBUG:boto:Data: 2022-09-02T05:58:01.374 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.375 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.376 DEBUG:boto:Params: {} 2022-09-02T05:58:01.376 DEBUG:boto:Token: None 2022-09-02T05:58:01.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc109b849c6575aa-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.383 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.384 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.385 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.386 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.387 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.388 DEBUG:boto:Method: GET 2022-09-02T05:58:01.389 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.390 DEBUG:boto:Data: 2022-09-02T05:58:01.391 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.392 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.393 DEBUG:boto:Params: {} 2022-09-02T05:58:01.393 DEBUG:boto:Token: None 2022-09-02T05:58:01.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b08e4e45db47b86-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.400 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.401 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.402 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.403 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.403 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:01.404 DEBUG:boto:Data: 2022-09-02T05:58:01.404 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.405 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.406 DEBUG:boto:Params: {} 2022-09-02T05:58:01.406 DEBUG:boto:Token: None 2022-09-02T05:58:01.407 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34 2022-09-02T05:58:01.408 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:obUVsR8fBUEg2o8EuGrGNk1vLrM= 2022-09-02T05:58:01.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:obUVsR8fBUEg2o8EuGrGNk1vLrM=', 'Content-Length': '0'} 2022-09-02T05:58:01.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d6dc664ae426ea0-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.411 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.412 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.412 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.413 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:01.413 DEBUG:boto:Data: 2022-09-02T05:58:01.414 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.415 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.415 DEBUG:boto:Params: {} 2022-09-02T05:58:01.416 DEBUG:boto:Token: None 2022-09-02T05:58:01.417 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34 2022-09-02T05:58:01.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:obUVsR8fBUEg2o8EuGrGNk1vLrM= 2022-09-02T05:58:01.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:obUVsR8fBUEg2o8EuGrGNk1vLrM=', 'Content-Length': '0'} 2022-09-02T05:58:01.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de0d45a4fa96a9c-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:58:01.422 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.423 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.423 DEBUG:boto:Method: GET 2022-09-02T05:58:01.424 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:01.424 DEBUG:boto:Data: 2022-09-02T05:58:01.425 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.426 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.427 DEBUG:boto:Params: {} 2022-09-02T05:58:01.428 DEBUG:boto:Token: None 2022-09-02T05:58:01.430 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34 2022-09-02T05:58:01.431 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XuC61KmOpDY0Bfd5Hgm3O0XcOd4= 2022-09-02T05:58:01.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XuC61KmOpDY0Bfd5Hgm3O0XcOd4=', 'Content-Length': '0'} 2022-09-02T05:58:01.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d587889f141a4679-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.474 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:01.475 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:01.475 DEBUG:boto:Method: GET 2022-09-02T05:58:01.476 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:01.477 DEBUG:boto:Data: 2022-09-02T05:58:01.477 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.478 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.479 DEBUG:boto:Params: {} 2022-09-02T05:58:01.479 DEBUG:boto:Token: None 2022-09-02T05:58:01.480 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj34 2022-09-02T05:58:01.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XuC61KmOpDY0Bfd5Hgm3O0XcOd4= 2022-09-02T05:58:01.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XuC61KmOpDY0Bfd5Hgm3O0XcOd4=', 'Content-Length': '0'} 2022-09-02T05:58:01.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607128c64894c666-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.484 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.485 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.485 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.486 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.487 DEBUG:boto:Method: GET 2022-09-02T05:58:01.487 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.488 DEBUG:boto:Data: 2022-09-02T05:58:01.488 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.489 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.490 DEBUG:boto:Params: {} 2022-09-02T05:58:01.490 DEBUG:boto:Token: None 2022-09-02T05:58:01.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047322d1ec198f5f0-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.496 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.496 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.497 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.497 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.498 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.498 DEBUG:boto:Method: GET 2022-09-02T05:58:01.499 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:01.499 DEBUG:boto:Data: 2022-09-02T05:58:01.500 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.501 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.502 DEBUG:boto:Params: {} 2022-09-02T05:58:01.502 DEBUG:boto:Token: None 2022-09-02T05:58:01.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa01da7bb75a6e6c-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.509 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:58:01.510 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.511 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.512 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.512 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.513 DEBUG:boto:Method: GET 2022-09-02T05:58:01.513 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.514 DEBUG:boto:Data: 2022-09-02T05:58:01.514 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.515 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.516 DEBUG:boto:Params: {} 2022-09-02T05:58:01.516 DEBUG:boto:Token: None 2022-09-02T05:58:01.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs= 2022-09-02T05:58:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs=', 'Content-Length': '0'} 2022-09-02T05:58:01.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c4fac56d44369a-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.521 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.522 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.522 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.523 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.523 DEBUG:boto:Method: GET 2022-09-02T05:58:01.524 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.525 DEBUG:boto:Data: 2022-09-02T05:58:01.525 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.526 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.527 DEBUG:boto:Params: {} 2022-09-02T05:58:01.527 DEBUG:boto:Token: None 2022-09-02T05:58:01.528 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs= 2022-09-02T05:58:01.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs=', 'Content-Length': '0'} 2022-09-02T05:58:01.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c85259b60bcd6f2-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.533 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.533 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.534 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.534 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.535 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.536 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.536 DEBUG:boto:Data: 2022-09-02T05:58:01.537 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.538 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.538 DEBUG:boto:Params: {} 2022-09-02T05:58:01.539 DEBUG:boto:Token: None 2022-09-02T05:58:01.540 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhKANFbxSQhZhx7GkLuY8LZFrPk= 2022-09-02T05:58:01.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhKANFbxSQhZhx7GkLuY8LZFrPk=', 'Content-Length': '0'} 2022-09-02T05:58:01.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb3f6e9415eec1af-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.544 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.545 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.545 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.546 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.546 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.547 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.547 DEBUG:boto:Data: 2022-09-02T05:58:01.548 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.549 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.550 DEBUG:boto:Params: {} 2022-09-02T05:58:01.550 DEBUG:boto:Token: None 2022-09-02T05:58:01.551 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LhKANFbxSQhZhx7GkLuY8LZFrPk= 2022-09-02T05:58:01.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LhKANFbxSQhZhx7GkLuY8LZFrPk=', 'Content-Length': '0'} 2022-09-02T05:58:01.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67071a466ab6316-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:58:01.556 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.557 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.557 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.558 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.558 DEBUG:boto:Method: GET 2022-09-02T05:58:01.559 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.559 DEBUG:boto:Data: 2022-09-02T05:58:01.560 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.561 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.562 DEBUG:boto:Params: {} 2022-09-02T05:58:01.562 DEBUG:boto:Token: None 2022-09-02T05:58:01.563 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.564 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs= 2022-09-02T05:58:01.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs=', 'Content-Length': '0'} 2022-09-02T05:58:01.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000159e98bce8013575-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.567 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.568 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.568 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.569 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.569 DEBUG:boto:Method: GET 2022-09-02T05:58:01.570 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.570 DEBUG:boto:Data: 2022-09-02T05:58:01.571 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.572 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.572 DEBUG:boto:Params: {} 2022-09-02T05:58:01.573 DEBUG:boto:Token: None 2022-09-02T05:58:01.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:01.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs= 2022-09-02T05:58:01.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ikZ47LbuOIMxqLA40UYuefMFtvs=', 'Content-Length': '0'} 2022-09-02T05:58:01.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d01aab41cf3823-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.618 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.619 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.619 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.620 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.620 DEBUG:boto:Method: GET 2022-09-02T05:58:01.621 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.621 DEBUG:boto:Data: 2022-09-02T05:58:01.622 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.623 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.624 DEBUG:boto:Params: {} 2022-09-02T05:58:01.624 DEBUG:boto:Token: None 2022-09-02T05:58:01.625 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602e9a9726959f8c-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.629 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.630 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.631 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.632 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.632 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.633 DEBUG:boto:Method: GET 2022-09-02T05:58:01.633 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.634 DEBUG:boto:Data: 2022-09-02T05:58:01.634 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.636 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.636 DEBUG:boto:Params: {} 2022-09-02T05:58:01.637 DEBUG:boto:Token: None 2022-09-02T05:58:01.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f3b9b6c87c51cca-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.645 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.646 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.646 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.647 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.647 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:01.648 DEBUG:boto:Data: 2022-09-02T05:58:01.648 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.649 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.650 DEBUG:boto:Params: {} 2022-09-02T05:58:01.650 DEBUG:boto:Token: None 2022-09-02T05:58:01.651 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38 2022-09-02T05:58:01.652 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PjAI4wK22AKkleDwnPYMO4Y37fY= 2022-09-02T05:58:01.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PjAI4wK22AKkleDwnPYMO4Y37fY=', 'Content-Length': '0'} 2022-09-02T05:58:01.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7ec060431611d5-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.656 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.657 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.657 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.658 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:01.659 DEBUG:boto:Data: 2022-09-02T05:58:01.659 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.661 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.661 DEBUG:boto:Params: {} 2022-09-02T05:58:01.662 DEBUG:boto:Token: None 2022-09-02T05:58:01.662 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38 2022-09-02T05:58:01.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PjAI4wK22AKkleDwnPYMO4Y37fY= 2022-09-02T05:58:01.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PjAI4wK22AKkleDwnPYMO4Y37fY=', 'Content-Length': '0'} 2022-09-02T05:58:01.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3ae7caeda69f88-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:58:01.667 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.668 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.668 DEBUG:boto:Method: GET 2022-09-02T05:58:01.669 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:01.670 DEBUG:boto:Data: 2022-09-02T05:58:01.670 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.671 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.672 DEBUG:boto:Params: {} 2022-09-02T05:58:01.672 DEBUG:boto:Token: None 2022-09-02T05:58:01.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38 2022-09-02T05:58:01.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v1PjSNaqhnA+92BUQ/yapvi2muY= 2022-09-02T05:58:01.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v1PjSNaqhnA+92BUQ/yapvi2muY=', 'Content-Length': '0'} 2022-09-02T05:58:01.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa996318ce0eeac2-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.718 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:01.720 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:01.721 DEBUG:boto:Method: GET 2022-09-02T05:58:01.721 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:01.722 DEBUG:boto:Data: 2022-09-02T05:58:01.722 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.723 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.724 DEBUG:boto:Params: {} 2022-09-02T05:58:01.724 DEBUG:boto:Token: None 2022-09-02T05:58:01.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj38 2022-09-02T05:58:01.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v1PjSNaqhnA+92BUQ/yapvi2muY= 2022-09-02T05:58:01.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v1PjSNaqhnA+92BUQ/yapvi2muY=', 'Content-Length': '0'} 2022-09-02T05:58:01.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32fe8460fb85f58-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.729 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.730 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.731 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.731 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.732 DEBUG:boto:Method: GET 2022-09-02T05:58:01.732 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.733 DEBUG:boto:Data: 2022-09-02T05:58:01.733 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.734 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.735 DEBUG:boto:Params: {} 2022-09-02T05:58:01.735 DEBUG:boto:Token: None 2022-09-02T05:58:01.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ceb0e366438c109-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.741 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.742 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.742 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.743 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.743 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.744 DEBUG:boto:Method: GET 2022-09-02T05:58:01.744 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:01.745 DEBUG:boto:Data: 2022-09-02T05:58:01.745 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.747 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.747 DEBUG:boto:Params: {} 2022-09-02T05:58:01.748 DEBUG:boto:Token: None 2022-09-02T05:58:01.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006be408dbe074da81-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.755 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:58:01.756 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.757 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.757 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.758 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.759 DEBUG:boto:Method: GET 2022-09-02T05:58:01.759 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.760 DEBUG:boto:Data: 2022-09-02T05:58:01.760 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.761 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.762 DEBUG:boto:Params: {} 2022-09-02T05:58:01.763 DEBUG:boto:Token: None 2022-09-02T05:58:01.763 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.764 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk= 2022-09-02T05:58:01.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk=', 'Content-Length': '0'} 2022-09-02T05:58:01.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffbe7e4567b1e90d-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.768 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.768 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.769 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.769 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.770 DEBUG:boto:Method: GET 2022-09-02T05:58:01.771 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.771 DEBUG:boto:Data: 2022-09-02T05:58:01.772 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.773 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.773 DEBUG:boto:Params: {} 2022-09-02T05:58:01.774 DEBUG:boto:Token: None 2022-09-02T05:58:01.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk= 2022-09-02T05:58:01.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk=', 'Content-Length': '0'} 2022-09-02T05:58:01.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002622104e7431ce4-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.822 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.823 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.823 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.824 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.824 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.825 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.825 DEBUG:boto:Data: 2022-09-02T05:58:01.826 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.827 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.828 DEBUG:boto:Params: {} 2022-09-02T05:58:01.828 DEBUG:boto:Token: None 2022-09-02T05:58:01.829 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.829 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rr/TqeL92JoTtMKcJe/lSoNkYcw= 2022-09-02T05:58:01.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rr/TqeL92JoTtMKcJe/lSoNkYcw=', 'Content-Length': '0'} 2022-09-02T05:58:01.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a687daeaa84dff4a-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.833 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.834 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.834 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.835 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.835 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.836 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.836 DEBUG:boto:Data: 2022-09-02T05:58:01.837 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.838 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.839 DEBUG:boto:Params: {} 2022-09-02T05:58:01.839 DEBUG:boto:Token: None 2022-09-02T05:58:01.840 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rr/TqeL92JoTtMKcJe/lSoNkYcw= 2022-09-02T05:58:01.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rr/TqeL92JoTtMKcJe/lSoNkYcw=', 'Content-Length': '0'} 2022-09-02T05:58:01.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cdfd46d42677d5c-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:58:01.845 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.845 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.846 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.846 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.847 DEBUG:boto:Method: GET 2022-09-02T05:58:01.848 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.849 DEBUG:boto:Data: 2022-09-02T05:58:01.850 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.852 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.853 DEBUG:boto:Params: {} 2022-09-02T05:58:01.854 DEBUG:boto:Token: None 2022-09-02T05:58:01.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk= 2022-09-02T05:58:01.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk=', 'Content-Length': '0'} 2022-09-02T05:58:01.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f0c3dda1d54a6e-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.859 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.860 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.860 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.861 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.861 DEBUG:boto:Method: GET 2022-09-02T05:58:01.862 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.863 DEBUG:boto:Data: 2022-09-02T05:58:01.863 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.864 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.865 DEBUG:boto:Params: {} 2022-09-02T05:58:01.865 DEBUG:boto:Token: None 2022-09-02T05:58:01.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:01.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk= 2022-09-02T05:58:01.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mYj2XU+AySSqNW7O+vV5ZRwkJdk=', 'Content-Length': '0'} 2022-09-02T05:58:01.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e3fa3260ee29fc7-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.870 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.871 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.871 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.872 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.873 DEBUG:boto:Method: GET 2022-09-02T05:58:01.873 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.874 DEBUG:boto:Data: 2022-09-02T05:58:01.874 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.875 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.876 DEBUG:boto:Params: {} 2022-09-02T05:58:01.877 DEBUG:boto:Token: None 2022-09-02T05:58:01.877 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093c30a7484bd0648-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.882 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.883 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.884 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.884 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.885 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.885 DEBUG:boto:Method: GET 2022-09-02T05:58:01.886 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.887 DEBUG:boto:Data: 2022-09-02T05:58:01.900 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.901 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.902 DEBUG:boto:Params: {} 2022-09-02T05:58:01.902 DEBUG:boto:Token: None 2022-09-02T05:58:01.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002747eb03280d2374-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.910 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.911 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.911 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.912 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.912 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:01.913 DEBUG:boto:Data: 2022-09-02T05:58:01.914 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.915 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.915 DEBUG:boto:Params: {} 2022-09-02T05:58:01.916 DEBUG:boto:Token: None 2022-09-02T05:58:01.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44 2022-09-02T05:58:01.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CNicn2Rbq+187YDntOYnaMWg9Wc= 2022-09-02T05:58:01.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CNicn2Rbq+187YDntOYnaMWg9Wc=', 'Content-Length': '0'} 2022-09-02T05:58:01.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a999b132ac2df9b-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.921 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.921 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.922 DEBUG:boto:Method: HEAD 2022-09-02T05:58:01.923 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:01.923 DEBUG:boto:Data: 2022-09-02T05:58:01.924 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.925 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.925 DEBUG:boto:Params: {} 2022-09-02T05:58:01.926 DEBUG:boto:Token: None 2022-09-02T05:58:01.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44 2022-09-02T05:58:01.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CNicn2Rbq+187YDntOYnaMWg9Wc= 2022-09-02T05:58:01.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CNicn2Rbq+187YDntOYnaMWg9Wc=', 'Content-Length': '0'} 2022-09-02T05:58:01.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f81ad1d17139c00-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:58:01.931 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.931 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.932 DEBUG:boto:Method: GET 2022-09-02T05:58:01.933 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:01.933 DEBUG:boto:Data: 2022-09-02T05:58:01.934 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.935 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.935 DEBUG:boto:Params: {} 2022-09-02T05:58:01.936 DEBUG:boto:Token: None 2022-09-02T05:58:01.936 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44 2022-09-02T05:58:01.937 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+Tug76d3HfX5yYpS6hFfLA3ntw= 2022-09-02T05:58:01.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+Tug76d3HfX5yYpS6hFfLA3ntw=', 'Content-Length': '0'} 2022-09-02T05:58:01.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eb486b9211fed35-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.941 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:01.941 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:01.942 DEBUG:boto:Method: GET 2022-09-02T05:58:01.942 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:01.943 DEBUG:boto:Data: 2022-09-02T05:58:01.944 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.945 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.945 DEBUG:boto:Params: {} 2022-09-02T05:58:01.946 DEBUG:boto:Token: None 2022-09-02T05:58:01.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/obj44 2022-09-02T05:58:01.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:T+Tug76d3HfX5yYpS6hFfLA3ntw= 2022-09-02T05:58:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:T+Tug76d3HfX5yYpS6hFfLA3ntw=', 'Content-Length': '0'} 2022-09-02T05:58:01.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d890550df5422eac-0063119b69-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.951 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.951 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.952 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.952 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.953 DEBUG:boto:Method: GET 2022-09-02T05:58:01.953 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.954 DEBUG:boto:Data: 2022-09-02T05:58:01.954 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:01.956 DEBUG:boto:Port: 8001 2022-09-02T05:58:01.956 DEBUG:boto:Params: {} 2022-09-02T05:58:01.957 DEBUG:boto:Token: None 2022-09-02T05:58:01.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed7e483e45a08a1-0063119b69-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:01.962 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:01.963 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:01.963 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:01.964 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.964 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.965 DEBUG:boto:Method: GET 2022-09-02T05:58:01.965 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:01.966 DEBUG:boto:Data: 2022-09-02T05:58:01.966 DEBUG:boto:Headers: {} 2022-09-02T05:58:01.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:01.967 DEBUG:boto:Port: 8000 2022-09-02T05:58:01.968 DEBUG:boto:Params: {} 2022-09-02T05:58:01.968 DEBUG:boto:Token: None 2022-09-02T05:58:01.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:01 GMT /xljbkf-74/ 2022-09-02T05:58:01.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo= 2022-09-02T05:58:01.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:01 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:18WjkPQi++nj6Z26xhsvSH/HUVo=', 'Content-Length': '0'} 2022-09-02T05:58:01.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f71a6b69f67715-0063119b69-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:01 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.022 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:58:02.024 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.024 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.025 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.025 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.026 DEBUG:boto:Method: GET 2022-09-02T05:58:02.026 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.027 DEBUG:boto:Data: 2022-09-02T05:58:02.028 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.029 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.029 DEBUG:boto:Params: {} 2022-09-02T05:58:02.030 DEBUG:boto:Token: None 2022-09-02T05:58:02.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE= 2022-09-02T05:58:02.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE=', 'Content-Length': '0'} 2022-09-02T05:58:02.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e65fa7958a8f13a-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.035 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.035 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.036 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.036 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.037 DEBUG:boto:Method: GET 2022-09-02T05:58:02.037 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.038 DEBUG:boto:Data: 2022-09-02T05:58:02.038 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.039 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.040 DEBUG:boto:Params: {} 2022-09-02T05:58:02.040 DEBUG:boto:Token: None 2022-09-02T05:58:02.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE= 2022-09-02T05:58:02.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE=', 'Content-Length': '0'} 2022-09-02T05:58:02.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18905ab2f858bd8-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.045 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.046 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.046 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.047 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.047 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.048 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.048 DEBUG:boto:Data: 2022-09-02T05:58:02.049 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.050 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.050 DEBUG:boto:Params: {} 2022-09-02T05:58:02.051 DEBUG:boto:Token: None 2022-09-02T05:58:02.052 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y3Y05v3JdA4f2hRvGHjHLp9nJiA= 2022-09-02T05:58:02.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y3Y05v3JdA4f2hRvGHjHLp9nJiA=', 'Content-Length': '0'} 2022-09-02T05:58:02.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e73fb97a745e2d3-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.055 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.056 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.057 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.057 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.058 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.058 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.059 DEBUG:boto:Data: 2022-09-02T05:58:02.059 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.060 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.061 DEBUG:boto:Params: {} 2022-09-02T05:58:02.061 DEBUG:boto:Token: None 2022-09-02T05:58:02.062 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y3Y05v3JdA4f2hRvGHjHLp9nJiA= 2022-09-02T05:58:02.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y3Y05v3JdA4f2hRvGHjHLp9nJiA=', 'Content-Length': '0'} 2022-09-02T05:58:02.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000084ed34a3ce83b8f-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:58:02.066 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.067 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.067 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.068 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.068 DEBUG:boto:Method: GET 2022-09-02T05:58:02.069 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.069 DEBUG:boto:Data: 2022-09-02T05:58:02.070 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.071 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.072 DEBUG:boto:Params: {} 2022-09-02T05:58:02.072 DEBUG:boto:Token: None 2022-09-02T05:58:02.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.073 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE= 2022-09-02T05:58:02.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE=', 'Content-Length': '0'} 2022-09-02T05:58:02.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca59eb0b814833df-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.118 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.119 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.119 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.120 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.121 DEBUG:boto:Method: GET 2022-09-02T05:58:02.121 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.122 DEBUG:boto:Data: 2022-09-02T05:58:02.123 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.124 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.124 DEBUG:boto:Params: {} 2022-09-02T05:58:02.125 DEBUG:boto:Token: None 2022-09-02T05:58:02.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:02.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE= 2022-09-02T05:58:02.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CsMyehG36HxSWFx5EcAI+JOkOaE=', 'Content-Length': '0'} 2022-09-02T05:58:02.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db90f2ac2417dd12-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.130 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.130 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.131 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.131 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.132 DEBUG:boto:Method: GET 2022-09-02T05:58:02.132 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.133 DEBUG:boto:Data: 2022-09-02T05:58:02.133 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.134 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.135 DEBUG:boto:Params: {} 2022-09-02T05:58:02.135 DEBUG:boto:Token: None 2022-09-02T05:58:02.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007b758268d1b7831-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.141 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.142 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.142 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.143 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.143 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.144 DEBUG:boto:Method: GET 2022-09-02T05:58:02.144 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.145 DEBUG:boto:Data: 2022-09-02T05:58:02.145 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.146 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.147 DEBUG:boto:Params: {} 2022-09-02T05:58:02.147 DEBUG:boto:Token: None 2022-09-02T05:58:02.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cace5cc539230075-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.154 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.155 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.156 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.156 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.157 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:02.157 DEBUG:boto:Data: 2022-09-02T05:58:02.158 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.159 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.159 DEBUG:boto:Params: {} 2022-09-02T05:58:02.160 DEBUG:boto:Token: None 2022-09-02T05:58:02.160 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46 2022-09-02T05:58:02.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BHaDsYAprqUWFHE6gZph1wR08zY= 2022-09-02T05:58:02.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BHaDsYAprqUWFHE6gZph1wR08zY=', 'Content-Length': '0'} 2022-09-02T05:58:02.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e13a50c7f3bf90-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.164 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.165 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.165 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.166 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:02.166 DEBUG:boto:Data: 2022-09-02T05:58:02.167 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.168 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.168 DEBUG:boto:Params: {} 2022-09-02T05:58:02.169 DEBUG:boto:Token: None 2022-09-02T05:58:02.170 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46 2022-09-02T05:58:02.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BHaDsYAprqUWFHE6gZph1wR08zY= 2022-09-02T05:58:02.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BHaDsYAprqUWFHE6gZph1wR08zY=', 'Content-Length': '0'} 2022-09-02T05:58:02.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f7c4ce635549d8-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:58:02.175 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.175 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.176 DEBUG:boto:Method: GET 2022-09-02T05:58:02.176 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:02.177 DEBUG:boto:Data: 2022-09-02T05:58:02.177 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.178 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.179 DEBUG:boto:Params: {} 2022-09-02T05:58:02.179 DEBUG:boto:Token: None 2022-09-02T05:58:02.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46 2022-09-02T05:58:02.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9a7zngMQyGh0IKoZCo12hj/m1RA= 2022-09-02T05:58:02.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9a7zngMQyGh0IKoZCo12hj/m1RA=', 'Content-Length': '0'} 2022-09-02T05:58:02.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8025be905ba29ff-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.184 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:02.185 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:02.185 DEBUG:boto:Method: GET 2022-09-02T05:58:02.186 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:02.186 DEBUG:boto:Data: 2022-09-02T05:58:02.187 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.188 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.189 DEBUG:boto:Params: {} 2022-09-02T05:58:02.189 DEBUG:boto:Token: None 2022-09-02T05:58:02.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj46 2022-09-02T05:58:02.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9a7zngMQyGh0IKoZCo12hj/m1RA= 2022-09-02T05:58:02.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9a7zngMQyGh0IKoZCo12hj/m1RA=', 'Content-Length': '0'} 2022-09-02T05:58:02.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a91dd2bb27a5dc-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.194 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.195 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.195 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.196 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.196 DEBUG:boto:Method: GET 2022-09-02T05:58:02.197 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.197 DEBUG:boto:Data: 2022-09-02T05:58:02.198 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.199 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.199 DEBUG:boto:Params: {} 2022-09-02T05:58:02.200 DEBUG:boto:Token: None 2022-09-02T05:58:02.200 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005430650119ae123d-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.205 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.206 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.206 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.207 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.207 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.208 DEBUG:boto:Method: GET 2022-09-02T05:58:02.208 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:02.209 DEBUG:boto:Data: 2022-09-02T05:58:02.209 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.211 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.211 DEBUG:boto:Params: {} 2022-09-02T05:58:02.212 DEBUG:boto:Token: None 2022-09-02T05:58:02.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000235d845793c1af5d-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.219 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:58:02.221 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.222 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.222 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.223 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.223 DEBUG:boto:Method: GET 2022-09-02T05:58:02.224 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.224 DEBUG:boto:Data: 2022-09-02T05:58:02.225 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.226 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.227 DEBUG:boto:Params: {} 2022-09-02T05:58:02.227 DEBUG:boto:Token: None 2022-09-02T05:58:02.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg= 2022-09-02T05:58:02.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg=', 'Content-Length': '0'} 2022-09-02T05:58:02.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c34f51dc0be99d9-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.232 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.233 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.233 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.234 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.234 DEBUG:boto:Method: GET 2022-09-02T05:58:02.235 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.235 DEBUG:boto:Data: 2022-09-02T05:58:02.236 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.237 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.237 DEBUG:boto:Params: {} 2022-09-02T05:58:02.238 DEBUG:boto:Token: None 2022-09-02T05:58:02.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg= 2022-09-02T05:58:02.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg=', 'Content-Length': '0'} 2022-09-02T05:58:02.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067534cd57e67287-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.243 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.244 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.245 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.245 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.246 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.246 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.247 DEBUG:boto:Data: 2022-09-02T05:58:02.247 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.248 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.249 DEBUG:boto:Params: {} 2022-09-02T05:58:02.249 DEBUG:boto:Token: None 2022-09-02T05:58:02.250 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yDWDNVGFFwL3e/VFXF0xmqZT6NM= 2022-09-02T05:58:02.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yDWDNVGFFwL3e/VFXF0xmqZT6NM=', 'Content-Length': '0'} 2022-09-02T05:58:02.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ccd4123dbe410d-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.254 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.254 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.255 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.255 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.256 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.256 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.257 DEBUG:boto:Data: 2022-09-02T05:58:02.257 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.259 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.259 DEBUG:boto:Params: {} 2022-09-02T05:58:02.260 DEBUG:boto:Token: None 2022-09-02T05:58:02.260 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yDWDNVGFFwL3e/VFXF0xmqZT6NM= 2022-09-02T05:58:02.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yDWDNVGFFwL3e/VFXF0xmqZT6NM=', 'Content-Length': '0'} 2022-09-02T05:58:02.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201acbe77e16c9ff-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:58:02.265 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.265 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.266 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.267 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.267 DEBUG:boto:Method: GET 2022-09-02T05:58:02.268 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.268 DEBUG:boto:Data: 2022-09-02T05:58:02.269 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.270 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.270 DEBUG:boto:Params: {} 2022-09-02T05:58:02.271 DEBUG:boto:Token: None 2022-09-02T05:58:02.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg= 2022-09-02T05:58:02.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg=', 'Content-Length': '0'} 2022-09-02T05:58:02.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051eae60b619fbc72-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.275 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.276 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.277 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.277 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.278 DEBUG:boto:Method: GET 2022-09-02T05:58:02.278 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.279 DEBUG:boto:Data: 2022-09-02T05:58:02.279 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.280 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.281 DEBUG:boto:Params: {} 2022-09-02T05:58:02.281 DEBUG:boto:Token: None 2022-09-02T05:58:02.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:02.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg= 2022-09-02T05:58:02.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8y07xQ+W576NLt/86g3TSN/gcxg=', 'Content-Length': '0'} 2022-09-02T05:58:02.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e52d3714624a479-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.326 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.327 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.327 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.328 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.328 DEBUG:boto:Method: GET 2022-09-02T05:58:02.329 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.330 DEBUG:boto:Data: 2022-09-02T05:58:02.330 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.331 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.332 DEBUG:boto:Params: {} 2022-09-02T05:58:02.332 DEBUG:boto:Token: None 2022-09-02T05:58:02.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a80345c03da62306-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.338 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.339 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.339 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.340 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.340 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.341 DEBUG:boto:Method: GET 2022-09-02T05:58:02.341 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.342 DEBUG:boto:Data: 2022-09-02T05:58:02.342 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.343 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.344 DEBUG:boto:Params: {} 2022-09-02T05:58:02.345 DEBUG:boto:Token: None 2022-09-02T05:58:02.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f48e58a58889b46-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.351 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.352 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.353 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.353 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.354 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:02.354 DEBUG:boto:Data: 2022-09-02T05:58:02.355 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.356 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.356 DEBUG:boto:Params: {} 2022-09-02T05:58:02.357 DEBUG:boto:Token: None 2022-09-02T05:58:02.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51 2022-09-02T05:58:02.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/86vuHe8+cv4MX7fW3j8pm4SqdE= 2022-09-02T05:58:02.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/86vuHe8+cv4MX7fW3j8pm4SqdE=', 'Content-Length': '0'} 2022-09-02T05:58:02.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a5df595d3d52c77-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.362 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.362 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.363 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.363 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:02.364 DEBUG:boto:Data: 2022-09-02T05:58:02.364 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.365 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.366 DEBUG:boto:Params: {} 2022-09-02T05:58:02.366 DEBUG:boto:Token: None 2022-09-02T05:58:02.367 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51 2022-09-02T05:58:02.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/86vuHe8+cv4MX7fW3j8pm4SqdE= 2022-09-02T05:58:02.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/86vuHe8+cv4MX7fW3j8pm4SqdE=', 'Content-Length': '0'} 2022-09-02T05:58:02.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f9bb600b1273c7d-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:58:02.372 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.372 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.373 DEBUG:boto:Method: GET 2022-09-02T05:58:02.373 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:02.374 DEBUG:boto:Data: 2022-09-02T05:58:02.374 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.375 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.376 DEBUG:boto:Params: {} 2022-09-02T05:58:02.377 DEBUG:boto:Token: None 2022-09-02T05:58:02.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51 2022-09-02T05:58:02.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:amv5uywYQR0OM5farXnlJH7HVnM= 2022-09-02T05:58:02.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:amv5uywYQR0OM5farXnlJH7HVnM=', 'Content-Length': '0'} 2022-09-02T05:58:02.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f02a7ad729274fa1-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.381 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:02.382 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:02.382 DEBUG:boto:Method: GET 2022-09-02T05:58:02.383 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:02.383 DEBUG:boto:Data: 2022-09-02T05:58:02.384 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.385 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.386 DEBUG:boto:Params: {} 2022-09-02T05:58:02.386 DEBUG:boto:Token: None 2022-09-02T05:58:02.387 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj51 2022-09-02T05:58:02.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:amv5uywYQR0OM5farXnlJH7HVnM= 2022-09-02T05:58:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:amv5uywYQR0OM5farXnlJH7HVnM=', 'Content-Length': '0'} 2022-09-02T05:58:02.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cddc1a52231af559-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.436 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.436 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.437 DEBUG:boto:Method: GET 2022-09-02T05:58:02.437 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.438 DEBUG:boto:Data: 2022-09-02T05:58:02.438 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.440 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.440 DEBUG:boto:Params: {} 2022-09-02T05:58:02.441 DEBUG:boto:Token: None 2022-09-02T05:58:02.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f5082ff004f1ef3-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.447 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.448 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.448 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.449 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.449 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.450 DEBUG:boto:Method: GET 2022-09-02T05:58:02.450 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:02.451 DEBUG:boto:Data: 2022-09-02T05:58:02.452 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.453 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.453 DEBUG:boto:Params: {} 2022-09-02T05:58:02.454 DEBUG:boto:Token: None 2022-09-02T05:58:02.454 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004622de209dc0256e-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.461 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:58:02.462 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.463 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.463 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.464 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.464 DEBUG:boto:Method: GET 2022-09-02T05:58:02.465 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.465 DEBUG:boto:Data: 2022-09-02T05:58:02.466 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.467 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.468 DEBUG:boto:Params: {} 2022-09-02T05:58:02.468 DEBUG:boto:Token: None 2022-09-02T05:58:02.469 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s= 2022-09-02T05:58:02.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s=', 'Content-Length': '0'} 2022-09-02T05:58:02.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f1b356eca35903b-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.473 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.474 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.474 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.475 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.475 DEBUG:boto:Method: GET 2022-09-02T05:58:02.476 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.477 DEBUG:boto:Data: 2022-09-02T05:58:02.478 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.479 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.479 DEBUG:boto:Params: {} 2022-09-02T05:58:02.480 DEBUG:boto:Token: None 2022-09-02T05:58:02.480 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s= 2022-09-02T05:58:02.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s=', 'Content-Length': '0'} 2022-09-02T05:58:02.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000835d3c46aed0e5f-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.485 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.486 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.486 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.487 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.487 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.488 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.488 DEBUG:boto:Data: 2022-09-02T05:58:02.489 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.490 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.490 DEBUG:boto:Params: {} 2022-09-02T05:58:02.491 DEBUG:boto:Token: None 2022-09-02T05:58:02.492 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gOzLBjXQ/COIyCra2fnqg/TqoI8= 2022-09-02T05:58:02.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gOzLBjXQ/COIyCra2fnqg/TqoI8=', 'Content-Length': '0'} 2022-09-02T05:58:02.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d035f301cafb4ee-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.496 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.496 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.497 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.497 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.498 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.498 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.499 DEBUG:boto:Data: 2022-09-02T05:58:02.499 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.500 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.501 DEBUG:boto:Params: {} 2022-09-02T05:58:02.501 DEBUG:boto:Token: None 2022-09-02T05:58:02.502 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gOzLBjXQ/COIyCra2fnqg/TqoI8= 2022-09-02T05:58:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gOzLBjXQ/COIyCra2fnqg/TqoI8=', 'Content-Length': '0'} 2022-09-02T05:58:02.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d374d739ea63ce-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:58:02.506 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.507 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.507 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.508 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.508 DEBUG:boto:Method: GET 2022-09-02T05:58:02.509 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.509 DEBUG:boto:Data: 2022-09-02T05:58:02.510 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.511 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.512 DEBUG:boto:Params: {} 2022-09-02T05:58:02.512 DEBUG:boto:Token: None 2022-09-02T05:58:02.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s= 2022-09-02T05:58:02.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s=', 'Content-Length': '0'} 2022-09-02T05:58:02.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c57bfb418a54f1-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.517 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.517 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.518 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.518 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.519 DEBUG:boto:Method: GET 2022-09-02T05:58:02.519 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.520 DEBUG:boto:Data: 2022-09-02T05:58:02.520 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.522 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.522 DEBUG:boto:Params: {} 2022-09-02T05:58:02.523 DEBUG:boto:Token: None 2022-09-02T05:58:02.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:02.524 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s= 2022-09-02T05:58:02.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wg2gE1jgeuXTKeOVeMY9VeWVJ7s=', 'Content-Length': '0'} 2022-09-02T05:58:02.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae8884764f7be42-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.528 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.528 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.529 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.529 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.530 DEBUG:boto:Method: GET 2022-09-02T05:58:02.531 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.531 DEBUG:boto:Data: 2022-09-02T05:58:02.532 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.533 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.533 DEBUG:boto:Params: {} 2022-09-02T05:58:02.534 DEBUG:boto:Token: None 2022-09-02T05:58:02.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.535 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6de2e37cde37113-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.539 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.539 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.540 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.541 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.541 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.542 DEBUG:boto:Method: GET 2022-09-02T05:58:02.542 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.543 DEBUG:boto:Data: 2022-09-02T05:58:02.543 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.544 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.545 DEBUG:boto:Params: {} 2022-09-02T05:58:02.545 DEBUG:boto:Token: None 2022-09-02T05:58:02.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fca4d7fd624a1e9-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.598 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.600 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.600 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.601 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.602 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:02.603 DEBUG:boto:Data: 2022-09-02T05:58:02.603 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.604 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.605 DEBUG:boto:Params: {} 2022-09-02T05:58:02.606 DEBUG:boto:Token: None 2022-09-02T05:58:02.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55 2022-09-02T05:58:02.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fXCtZgBaErl1Ks4KiQHDXsMjjO0= 2022-09-02T05:58:02.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fXCtZgBaErl1Ks4KiQHDXsMjjO0=', 'Content-Length': '0'} 2022-09-02T05:58:02.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075380edd91d1650-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.610 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.611 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.611 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.612 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:02.612 DEBUG:boto:Data: 2022-09-02T05:58:02.613 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.614 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.614 DEBUG:boto:Params: {} 2022-09-02T05:58:02.615 DEBUG:boto:Token: None 2022-09-02T05:58:02.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55 2022-09-02T05:58:02.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fXCtZgBaErl1Ks4KiQHDXsMjjO0= 2022-09-02T05:58:02.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fXCtZgBaErl1Ks4KiQHDXsMjjO0=', 'Content-Length': '0'} 2022-09-02T05:58:02.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c903d8d3132eb0-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:58:02.620 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.621 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.621 DEBUG:boto:Method: GET 2022-09-02T05:58:02.622 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:02.623 DEBUG:boto:Data: 2022-09-02T05:58:02.623 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.624 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.625 DEBUG:boto:Params: {} 2022-09-02T05:58:02.625 DEBUG:boto:Token: None 2022-09-02T05:58:02.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55 2022-09-02T05:58:02.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PNNk2OfA27b3hZyVkDYVUCv8sAY= 2022-09-02T05:58:02.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PNNk2OfA27b3hZyVkDYVUCv8sAY=', 'Content-Length': '0'} 2022-09-02T05:58:02.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fa184e08bb15733-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.631 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:02.632 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:02.633 DEBUG:boto:Method: GET 2022-09-02T05:58:02.634 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:02.635 DEBUG:boto:Data: 2022-09-02T05:58:02.636 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.638 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.639 DEBUG:boto:Params: {} 2022-09-02T05:58:02.639 DEBUG:boto:Token: None 2022-09-02T05:58:02.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj55 2022-09-02T05:58:02.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PNNk2OfA27b3hZyVkDYVUCv8sAY= 2022-09-02T05:58:02.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PNNk2OfA27b3hZyVkDYVUCv8sAY=', 'Content-Length': '0'} 2022-09-02T05:58:02.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31ec47ee5743c92-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.646 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.646 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.647 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.647 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.648 DEBUG:boto:Method: GET 2022-09-02T05:58:02.648 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.649 DEBUG:boto:Data: 2022-09-02T05:58:02.649 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.650 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.651 DEBUG:boto:Params: {} 2022-09-02T05:58:02.651 DEBUG:boto:Token: None 2022-09-02T05:58:02.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b8bc9bb9edd63e-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.657 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.658 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.658 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.659 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.659 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.660 DEBUG:boto:Method: GET 2022-09-02T05:58:02.660 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:02.661 DEBUG:boto:Data: 2022-09-02T05:58:02.661 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.662 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.663 DEBUG:boto:Params: {} 2022-09-02T05:58:02.663 DEBUG:boto:Token: None 2022-09-02T05:58:02.664 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.665 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039ebf40b7c172c88-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.670 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:58:02.672 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.672 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.673 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.673 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.674 DEBUG:boto:Method: GET 2022-09-02T05:58:02.674 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.675 DEBUG:boto:Data: 2022-09-02T05:58:02.675 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.676 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.677 DEBUG:boto:Params: {} 2022-09-02T05:58:02.677 DEBUG:boto:Token: None 2022-09-02T05:58:02.678 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0= 2022-09-02T05:58:02.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0=', 'Content-Length': '0'} 2022-09-02T05:58:02.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff2f68e086634bee-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.682 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.683 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.683 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.684 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.684 DEBUG:boto:Method: GET 2022-09-02T05:58:02.685 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.686 DEBUG:boto:Data: 2022-09-02T05:58:02.686 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.687 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.688 DEBUG:boto:Params: {} 2022-09-02T05:58:02.688 DEBUG:boto:Token: None 2022-09-02T05:58:02.689 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0= 2022-09-02T05:58:02.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0=', 'Content-Length': '0'} 2022-09-02T05:58:02.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb5da281934dfeb-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.693 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.694 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.694 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.695 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.695 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.696 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.696 DEBUG:boto:Data: 2022-09-02T05:58:02.697 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.697 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.698 DEBUG:boto:Params: {} 2022-09-02T05:58:02.699 DEBUG:boto:Token: None 2022-09-02T05:58:02.699 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tlZAE9sNCrug8v6V30NZEl8LEZ8= 2022-09-02T05:58:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tlZAE9sNCrug8v6V30NZEl8LEZ8=', 'Content-Length': '0'} 2022-09-02T05:58:02.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbabd086eb62cdd6-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.703 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.704 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.704 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.705 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.705 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.706 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.707 DEBUG:boto:Data: 2022-09-02T05:58:02.708 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.709 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.709 DEBUG:boto:Params: {} 2022-09-02T05:58:02.710 DEBUG:boto:Token: None 2022-09-02T05:58:02.710 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tlZAE9sNCrug8v6V30NZEl8LEZ8= 2022-09-02T05:58:02.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tlZAE9sNCrug8v6V30NZEl8LEZ8=', 'Content-Length': '0'} 2022-09-02T05:58:02.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c12bfffb5c1274-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:58:02.715 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.716 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.716 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.717 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.717 DEBUG:boto:Method: GET 2022-09-02T05:58:02.718 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.719 DEBUG:boto:Data: 2022-09-02T05:58:02.719 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.720 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.721 DEBUG:boto:Params: {} 2022-09-02T05:58:02.722 DEBUG:boto:Token: None 2022-09-02T05:58:02.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0= 2022-09-02T05:58:02.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0=', 'Content-Length': '0'} 2022-09-02T05:58:02.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e35c3311e6dd5fb-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.766 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.767 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.767 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.768 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.769 DEBUG:boto:Method: GET 2022-09-02T05:58:02.769 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.770 DEBUG:boto:Data: 2022-09-02T05:58:02.770 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.771 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.772 DEBUG:boto:Params: {} 2022-09-02T05:58:02.773 DEBUG:boto:Token: None 2022-09-02T05:58:02.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:02.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0= 2022-09-02T05:58:02.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SF6bE9HL2eUyRHAsTmRsRnUkWK0=', 'Content-Length': '0'} 2022-09-02T05:58:02.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf25ff85b2daeb6-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.778 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.778 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.779 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.779 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.780 DEBUG:boto:Method: GET 2022-09-02T05:58:02.781 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.781 DEBUG:boto:Data: 2022-09-02T05:58:02.782 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.784 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.784 DEBUG:boto:Params: {} 2022-09-02T05:58:02.785 DEBUG:boto:Token: None 2022-09-02T05:58:02.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4fcca6a8ca9b8b5-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.791 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.792 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.793 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.794 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.795 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.796 DEBUG:boto:Method: GET 2022-09-02T05:58:02.797 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.798 DEBUG:boto:Data: 2022-09-02T05:58:02.799 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.801 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.801 DEBUG:boto:Params: {} 2022-09-02T05:58:02.803 DEBUG:boto:Token: None 2022-09-02T05:58:02.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0dbca4f7812a09d-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.812 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.813 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.814 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.815 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.816 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:02.817 DEBUG:boto:Data: 2022-09-02T05:58:02.818 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.820 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.821 DEBUG:boto:Params: {} 2022-09-02T05:58:02.821 DEBUG:boto:Token: None 2022-09-02T05:58:02.822 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57 2022-09-02T05:58:02.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kGZbXMpsKKjBgrA2yENSfX1zlnA= 2022-09-02T05:58:02.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kGZbXMpsKKjBgrA2yENSfX1zlnA=', 'Content-Length': '0'} 2022-09-02T05:58:02.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693f3cb2485855a6-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.826 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.827 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.827 DEBUG:boto:Method: HEAD 2022-09-02T05:58:02.828 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:02.828 DEBUG:boto:Data: 2022-09-02T05:58:02.829 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.830 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.831 DEBUG:boto:Params: {} 2022-09-02T05:58:02.831 DEBUG:boto:Token: None 2022-09-02T05:58:02.832 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57 2022-09-02T05:58:02.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kGZbXMpsKKjBgrA2yENSfX1zlnA= 2022-09-02T05:58:02.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kGZbXMpsKKjBgrA2yENSfX1zlnA=', 'Content-Length': '0'} 2022-09-02T05:58:02.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e96c0d45c42d8ba-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:58:02.836 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.837 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.837 DEBUG:boto:Method: GET 2022-09-02T05:58:02.838 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:02.838 DEBUG:boto:Data: 2022-09-02T05:58:02.839 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.840 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.841 DEBUG:boto:Params: {} 2022-09-02T05:58:02.841 DEBUG:boto:Token: None 2022-09-02T05:58:02.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57 2022-09-02T05:58:02.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TSnP0inJo6w5/G/hAhtIDPNzVyw= 2022-09-02T05:58:02.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TSnP0inJo6w5/G/hAhtIDPNzVyw=', 'Content-Length': '0'} 2022-09-02T05:58:02.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025bf7462e1e6b5fc-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.846 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:02.847 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:02.847 DEBUG:boto:Method: GET 2022-09-02T05:58:02.848 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:02.848 DEBUG:boto:Data: 2022-09-02T05:58:02.849 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.850 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.850 DEBUG:boto:Params: {} 2022-09-02T05:58:02.851 DEBUG:boto:Token: None 2022-09-02T05:58:02.851 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj57 2022-09-02T05:58:02.852 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TSnP0inJo6w5/G/hAhtIDPNzVyw= 2022-09-02T05:58:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TSnP0inJo6w5/G/hAhtIDPNzVyw=', 'Content-Length': '0'} 2022-09-02T05:58:02.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79e833a90fb9788-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.898 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.899 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.899 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.900 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.901 DEBUG:boto:Method: GET 2022-09-02T05:58:02.901 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.902 DEBUG:boto:Data: 2022-09-02T05:58:02.902 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.903 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.904 DEBUG:boto:Params: {} 2022-09-02T05:58:02.904 DEBUG:boto:Token: None 2022-09-02T05:58:02.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064933502e6af49a0-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.912 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.914 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:02.915 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:02.916 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.916 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.917 DEBUG:boto:Method: GET 2022-09-02T05:58:02.918 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:02.918 DEBUG:boto:Data: 2022-09-02T05:58:02.919 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.920 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.921 DEBUG:boto:Params: {} 2022-09-02T05:58:02.921 DEBUG:boto:Token: None 2022-09-02T05:58:02.922 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/ 2022-09-02T05:58:02.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w= 2022-09-02T05:58:02.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:te74qJ8qmztfKW7EHuHwpmgrP9w=', 'Content-Length': '0'} 2022-09-02T05:58:02.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae2de23b540a978d-0063119b6a-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.929 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:02.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:58:02.932 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:02.932 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:02.933 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.934 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.934 DEBUG:boto:Method: GET 2022-09-02T05:58:02.935 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.936 DEBUG:boto:Data: 2022-09-02T05:58:02.936 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:02.937 DEBUG:boto:Port: 8001 2022-09-02T05:58:02.938 DEBUG:boto:Params: {} 2022-09-02T05:58:02.939 DEBUG:boto:Token: None 2022-09-02T05:58:02.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kb13d7z90zq/bJ4wrq30prezv+E= 2022-09-02T05:58:02.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kb13d7z90zq/bJ4wrq30prezv+E=', 'Content-Length': '0'} 2022-09-02T05:58:02.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fa2e3ac7fa4cd2-0063119b6a-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:02 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:02.990 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:02.991 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:02.991 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.992 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.993 DEBUG:boto:Method: GET 2022-09-02T05:58:02.994 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.994 DEBUG:boto:Data: 2022-09-02T05:58:02.995 DEBUG:boto:Headers: {} 2022-09-02T05:58:02.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:02.996 DEBUG:boto:Port: 8000 2022-09-02T05:58:02.996 DEBUG:boto:Params: {} 2022-09-02T05:58:02.997 DEBUG:boto:Token: None 2022-09-02T05:58:02.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:02 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:02.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kb13d7z90zq/bJ4wrq30prezv+E= 2022-09-02T05:58:02.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:02 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kb13d7z90zq/bJ4wrq30prezv+E=', 'Content-Length': '0'} 2022-09-02T05:58:03.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9dbdde19c1c49b2-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.003 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.003 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.004 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.004 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.005 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.006 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.006 DEBUG:boto:Data: 2022-09-02T05:58:03.007 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.008 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.009 DEBUG:boto:Params: {} 2022-09-02T05:58:03.010 DEBUG:boto:Token: None 2022-09-02T05:58:03.010 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shzhyPMQss/o2llNXxIu50nNLZ0= 2022-09-02T05:58:03.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shzhyPMQss/o2llNXxIu50nNLZ0=', 'Content-Length': '0'} 2022-09-02T05:58:03.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0c9d7246387e96-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.015 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.015 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.016 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.016 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.017 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.018 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.018 DEBUG:boto:Data: 2022-09-02T05:58:03.019 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.020 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.021 DEBUG:boto:Params: {} 2022-09-02T05:58:03.021 DEBUG:boto:Token: None 2022-09-02T05:58:03.022 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:shzhyPMQss/o2llNXxIu50nNLZ0= 2022-09-02T05:58:03.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:shzhyPMQss/o2llNXxIu50nNLZ0=', 'Content-Length': '0'} 2022-09-02T05:58:03.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006225c99d35194b4c-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:58:03.027 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.027 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.028 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.028 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.029 DEBUG:boto:Method: GET 2022-09-02T05:58:03.030 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.030 DEBUG:boto:Data: 2022-09-02T05:58:03.031 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.032 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.032 DEBUG:boto:Params: {} 2022-09-02T05:58:03.033 DEBUG:boto:Token: None 2022-09-02T05:58:03.033 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.034 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QaGe7ifgNlNrWkWRGmrMIAZl+/I= 2022-09-02T05:58:03.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QaGe7ifgNlNrWkWRGmrMIAZl+/I=', 'Content-Length': '0'} 2022-09-02T05:58:03.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5f7c1e301ef213-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.038 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.038 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.039 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.039 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.040 DEBUG:boto:Method: GET 2022-09-02T05:58:03.040 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.041 DEBUG:boto:Data: 2022-09-02T05:58:03.041 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.043 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.043 DEBUG:boto:Params: {} 2022-09-02T05:58:03.044 DEBUG:boto:Token: None 2022-09-02T05:58:03.045 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:03.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QaGe7ifgNlNrWkWRGmrMIAZl+/I= 2022-09-02T05:58:03.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QaGe7ifgNlNrWkWRGmrMIAZl+/I=', 'Content-Length': '0'} 2022-09-02T05:58:03.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb91aba5c2fd98e-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.049 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.049 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.050 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.051 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.051 DEBUG:boto:Method: GET 2022-09-02T05:58:03.052 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.052 DEBUG:boto:Data: 2022-09-02T05:58:03.053 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.054 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.054 DEBUG:boto:Params: {} 2022-09-02T05:58:03.055 DEBUG:boto:Token: None 2022-09-02T05:58:03.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2cfe12d7053d232-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.060 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.061 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.062 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.063 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.063 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.064 DEBUG:boto:Method: GET 2022-09-02T05:58:03.064 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.065 DEBUG:boto:Data: 2022-09-02T05:58:03.065 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.066 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.067 DEBUG:boto:Params: {} 2022-09-02T05:58:03.067 DEBUG:boto:Token: None 2022-09-02T05:58:03.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000482a559d860da8a5-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.074 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.075 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.076 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.076 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.077 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:03.078 DEBUG:boto:Data: 2022-09-02T05:58:03.078 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.079 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.080 DEBUG:boto:Params: {} 2022-09-02T05:58:03.080 DEBUG:boto:Token: None 2022-09-02T05:58:03.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58 2022-09-02T05:58:03.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:otC3Q5yxrt0rDpqvZ3bRdRUaInE= 2022-09-02T05:58:03.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:otC3Q5yxrt0rDpqvZ3bRdRUaInE=', 'Content-Length': '0'} 2022-09-02T05:58:03.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012fd0fb27c2f3639-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.085 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.085 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.086 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.087 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:03.087 DEBUG:boto:Data: 2022-09-02T05:58:03.088 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.089 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.089 DEBUG:boto:Params: {} 2022-09-02T05:58:03.090 DEBUG:boto:Token: None 2022-09-02T05:58:03.090 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58 2022-09-02T05:58:03.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:otC3Q5yxrt0rDpqvZ3bRdRUaInE= 2022-09-02T05:58:03.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:otC3Q5yxrt0rDpqvZ3bRdRUaInE=', 'Content-Length': '0'} 2022-09-02T05:58:03.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49edc9140cda124-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:58:03.095 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.096 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.096 DEBUG:boto:Method: GET 2022-09-02T05:58:03.097 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:03.098 DEBUG:boto:Data: 2022-09-02T05:58:03.098 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.099 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.100 DEBUG:boto:Params: {} 2022-09-02T05:58:03.100 DEBUG:boto:Token: None 2022-09-02T05:58:03.101 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58 2022-09-02T05:58:03.101 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UmOvhNHu2DBqMOQ4dVd7rKmwwo8= 2022-09-02T05:58:03.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UmOvhNHu2DBqMOQ4dVd7rKmwwo8=', 'Content-Length': '0'} 2022-09-02T05:58:03.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003189e8b8e3f4a696-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.146 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:03.147 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:03.148 DEBUG:boto:Method: GET 2022-09-02T05:58:03.149 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:03.150 DEBUG:boto:Data: 2022-09-02T05:58:03.151 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.153 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.153 DEBUG:boto:Params: {} 2022-09-02T05:58:03.154 DEBUG:boto:Token: None 2022-09-02T05:58:03.154 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj58 2022-09-02T05:58:03.155 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UmOvhNHu2DBqMOQ4dVd7rKmwwo8= 2022-09-02T05:58:03.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UmOvhNHu2DBqMOQ4dVd7rKmwwo8=', 'Content-Length': '0'} 2022-09-02T05:58:03.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74c9c3db01cb3b9-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.160 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.161 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.162 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.162 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.163 DEBUG:boto:Method: GET 2022-09-02T05:58:03.164 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.166 DEBUG:boto:Data: 2022-09-02T05:58:03.167 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.169 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.170 DEBUG:boto:Params: {} 2022-09-02T05:58:03.171 DEBUG:boto:Token: None 2022-09-02T05:58:03.172 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e878c4e97fb02896-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.179 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.180 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.181 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.183 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.184 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.185 DEBUG:boto:Method: GET 2022-09-02T05:58:03.186 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:03.187 DEBUG:boto:Data: 2022-09-02T05:58:03.188 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.190 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.191 DEBUG:boto:Params: {} 2022-09-02T05:58:03.192 DEBUG:boto:Token: None 2022-09-02T05:58:03.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81bc8332f7da78c-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.201 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:58:03.204 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.205 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.206 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.207 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.208 DEBUG:boto:Method: GET 2022-09-02T05:58:03.209 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.210 DEBUG:boto:Data: 2022-09-02T05:58:03.211 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.213 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.214 DEBUG:boto:Params: {} 2022-09-02T05:58:03.215 DEBUG:boto:Token: None 2022-09-02T05:58:03.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU= 2022-09-02T05:58:03.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU=', 'Content-Length': '0'} 2022-09-02T05:58:03.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b940789f8e080d-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.221 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.221 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.222 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.222 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.223 DEBUG:boto:Method: GET 2022-09-02T05:58:03.224 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.224 DEBUG:boto:Data: 2022-09-02T05:58:03.225 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.226 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.226 DEBUG:boto:Params: {} 2022-09-02T05:58:03.227 DEBUG:boto:Token: None 2022-09-02T05:58:03.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU= 2022-09-02T05:58:03.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU=', 'Content-Length': '0'} 2022-09-02T05:58:03.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a8a70406375c09b-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.278 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.279 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.279 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.280 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.280 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.281 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.281 DEBUG:boto:Data: 2022-09-02T05:58:03.282 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.283 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.284 DEBUG:boto:Params: {} 2022-09-02T05:58:03.285 DEBUG:boto:Token: None 2022-09-02T05:58:03.286 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bILJihyDZpt2PwnzJcTq9YBpMY= 2022-09-02T05:58:03.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bILJihyDZpt2PwnzJcTq9YBpMY=', 'Content-Length': '0'} 2022-09-02T05:58:03.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639972e850299d77-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.290 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.291 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.292 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.292 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.293 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.293 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.294 DEBUG:boto:Data: 2022-09-02T05:58:03.294 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.295 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.296 DEBUG:boto:Params: {} 2022-09-02T05:58:03.296 DEBUG:boto:Token: None 2022-09-02T05:58:03.297 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9bILJihyDZpt2PwnzJcTq9YBpMY= 2022-09-02T05:58:03.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9bILJihyDZpt2PwnzJcTq9YBpMY=', 'Content-Length': '0'} 2022-09-02T05:58:03.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005773ac6078a695b1-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:58:03.302 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.302 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.303 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.303 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.304 DEBUG:boto:Method: GET 2022-09-02T05:58:03.304 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.305 DEBUG:boto:Data: 2022-09-02T05:58:03.305 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.307 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.307 DEBUG:boto:Params: {} 2022-09-02T05:58:03.308 DEBUG:boto:Token: None 2022-09-02T05:58:03.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU= 2022-09-02T05:58:03.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU=', 'Content-Length': '0'} 2022-09-02T05:58:03.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a6bf78f8320f02-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.313 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.313 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.314 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.314 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.315 DEBUG:boto:Method: GET 2022-09-02T05:58:03.315 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.316 DEBUG:boto:Data: 2022-09-02T05:58:03.316 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.318 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.318 DEBUG:boto:Params: {} 2022-09-02T05:58:03.319 DEBUG:boto:Token: None 2022-09-02T05:58:03.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:03.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU= 2022-09-02T05:58:03.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8uoEvsbHeAlacHbWotPBOX0CxU=', 'Content-Length': '0'} 2022-09-02T05:58:03.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32f8e15a177de4b-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.324 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.325 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.325 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.326 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.326 DEBUG:boto:Method: GET 2022-09-02T05:58:03.327 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.328 DEBUG:boto:Data: 2022-09-02T05:58:03.328 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.329 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.330 DEBUG:boto:Params: {} 2022-09-02T05:58:03.330 DEBUG:boto:Token: None 2022-09-02T05:58:03.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee8bad336e053235-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.336 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.337 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.338 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.338 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.339 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.339 DEBUG:boto:Method: GET 2022-09-02T05:58:03.340 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.340 DEBUG:boto:Data: 2022-09-02T05:58:03.341 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.342 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.342 DEBUG:boto:Params: {} 2022-09-02T05:58:03.343 DEBUG:boto:Token: None 2022-09-02T05:58:03.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c53274365210945e-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.351 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.351 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.352 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.353 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.353 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:03.354 DEBUG:boto:Data: 2022-09-02T05:58:03.355 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.356 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.356 DEBUG:boto:Params: {} 2022-09-02T05:58:03.357 DEBUG:boto:Token: None 2022-09-02T05:58:03.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62 2022-09-02T05:58:03.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwUJ8B6Ki3lbC4AhcFRb68YOMfk= 2022-09-02T05:58:03.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwUJ8B6Ki3lbC4AhcFRb68YOMfk=', 'Content-Length': '0'} 2022-09-02T05:58:03.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d8b0a6b309e3ed-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.361 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.362 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.363 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.364 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:03.364 DEBUG:boto:Data: 2022-09-02T05:58:03.365 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.366 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.366 DEBUG:boto:Params: {} 2022-09-02T05:58:03.367 DEBUG:boto:Token: None 2022-09-02T05:58:03.367 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62 2022-09-02T05:58:03.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwUJ8B6Ki3lbC4AhcFRb68YOMfk= 2022-09-02T05:58:03.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwUJ8B6Ki3lbC4AhcFRb68YOMfk=', 'Content-Length': '0'} 2022-09-02T05:58:03.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000708863a6d26cfef2-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:58:03.373 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.373 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.374 DEBUG:boto:Method: GET 2022-09-02T05:58:03.374 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:03.375 DEBUG:boto:Data: 2022-09-02T05:58:03.375 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.376 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.377 DEBUG:boto:Params: {} 2022-09-02T05:58:03.377 DEBUG:boto:Token: None 2022-09-02T05:58:03.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62 2022-09-02T05:58:03.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YydXhswShmEOFAjUdx13MIsuJbk= 2022-09-02T05:58:03.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YydXhswShmEOFAjUdx13MIsuJbk=', 'Content-Length': '0'} 2022-09-02T05:58:03.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029010d0663f8c4bc-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.382 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:03.383 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:03.383 DEBUG:boto:Method: GET 2022-09-02T05:58:03.384 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:03.384 DEBUG:boto:Data: 2022-09-02T05:58:03.385 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.386 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.387 DEBUG:boto:Params: {} 2022-09-02T05:58:03.387 DEBUG:boto:Token: None 2022-09-02T05:58:03.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj62 2022-09-02T05:58:03.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YydXhswShmEOFAjUdx13MIsuJbk= 2022-09-02T05:58:03.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YydXhswShmEOFAjUdx13MIsuJbk=', 'Content-Length': '0'} 2022-09-02T05:58:03.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006754768817138d8b-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.435 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.436 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.436 DEBUG:boto:Method: GET 2022-09-02T05:58:03.437 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.438 DEBUG:boto:Data: 2022-09-02T05:58:03.438 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.439 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.440 DEBUG:boto:Params: {} 2022-09-02T05:58:03.440 DEBUG:boto:Token: None 2022-09-02T05:58:03.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a1f7789874eb9d-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.446 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.447 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.447 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.448 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.448 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.449 DEBUG:boto:Method: GET 2022-09-02T05:58:03.449 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:03.450 DEBUG:boto:Data: 2022-09-02T05:58:03.450 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.452 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.452 DEBUG:boto:Params: {} 2022-09-02T05:58:03.453 DEBUG:boto:Token: None 2022-09-02T05:58:03.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c67168db657f04d-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.460 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:58:03.461 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.462 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.462 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.463 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.463 DEBUG:boto:Method: GET 2022-09-02T05:58:03.464 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.464 DEBUG:boto:Data: 2022-09-02T05:58:03.465 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.466 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.467 DEBUG:boto:Params: {} 2022-09-02T05:58:03.467 DEBUG:boto:Token: None 2022-09-02T05:58:03.468 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.468 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU= 2022-09-02T05:58:03.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU=', 'Content-Length': '0'} 2022-09-02T05:58:03.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183b26ea460399ca-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.472 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.473 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.473 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.474 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.475 DEBUG:boto:Method: GET 2022-09-02T05:58:03.475 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.476 DEBUG:boto:Data: 2022-09-02T05:58:03.476 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.477 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.478 DEBUG:boto:Params: {} 2022-09-02T05:58:03.478 DEBUG:boto:Token: None 2022-09-02T05:58:03.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU= 2022-09-02T05:58:03.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU=', 'Content-Length': '0'} 2022-09-02T05:58:03.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbdf2bd841f5e81a-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.484 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.485 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.485 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.486 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.486 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.487 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.487 DEBUG:boto:Data: 2022-09-02T05:58:03.488 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.489 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.489 DEBUG:boto:Params: {} 2022-09-02T05:58:03.490 DEBUG:boto:Token: None 2022-09-02T05:58:03.491 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J1fOFMvyZbsOQbacsBKg19KjZBU= 2022-09-02T05:58:03.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J1fOFMvyZbsOQbacsBKg19KjZBU=', 'Content-Length': '0'} 2022-09-02T05:58:03.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000778eeb27b1a73471-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.495 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.495 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.496 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.496 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.497 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.497 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.498 DEBUG:boto:Data: 2022-09-02T05:58:03.498 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.499 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.500 DEBUG:boto:Params: {} 2022-09-02T05:58:03.500 DEBUG:boto:Token: None 2022-09-02T05:58:03.501 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J1fOFMvyZbsOQbacsBKg19KjZBU= 2022-09-02T05:58:03.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J1fOFMvyZbsOQbacsBKg19KjZBU=', 'Content-Length': '0'} 2022-09-02T05:58:03.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000171e5a3b36f4ee4-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:58:03.506 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.506 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.507 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.507 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.508 DEBUG:boto:Method: GET 2022-09-02T05:58:03.508 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.509 DEBUG:boto:Data: 2022-09-02T05:58:03.510 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.511 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.511 DEBUG:boto:Params: {} 2022-09-02T05:58:03.512 DEBUG:boto:Token: None 2022-09-02T05:58:03.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU= 2022-09-02T05:58:03.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU=', 'Content-Length': '0'} 2022-09-02T05:58:03.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a372d709035799f-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.517 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.518 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.518 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.519 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.519 DEBUG:boto:Method: GET 2022-09-02T05:58:03.520 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.521 DEBUG:boto:Data: 2022-09-02T05:58:03.521 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.522 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.523 DEBUG:boto:Params: {} 2022-09-02T05:58:03.523 DEBUG:boto:Token: None 2022-09-02T05:58:03.524 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:03.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU= 2022-09-02T05:58:03.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8NroB3FNDIjHvnz1wEy1ABRMmMU=', 'Content-Length': '0'} 2022-09-02T05:58:03.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003449799cb6793750-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.570 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.571 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.571 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.572 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.572 DEBUG:boto:Method: GET 2022-09-02T05:58:03.573 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.574 DEBUG:boto:Data: 2022-09-02T05:58:03.574 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.575 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.576 DEBUG:boto:Params: {} 2022-09-02T05:58:03.576 DEBUG:boto:Token: None 2022-09-02T05:58:03.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aca50087d6fa7d97-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.582 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.583 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.583 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.584 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.585 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.585 DEBUG:boto:Method: GET 2022-09-02T05:58:03.586 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.587 DEBUG:boto:Data: 2022-09-02T05:58:03.587 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.588 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.589 DEBUG:boto:Params: {} 2022-09-02T05:58:03.589 DEBUG:boto:Token: None 2022-09-02T05:58:03.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9fdd93aa99db4d-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.596 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.597 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.598 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.599 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.599 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:03.600 DEBUG:boto:Data: 2022-09-02T05:58:03.600 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.601 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.602 DEBUG:boto:Params: {} 2022-09-02T05:58:03.603 DEBUG:boto:Token: None 2022-09-02T05:58:03.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68 2022-09-02T05:58:03.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1P6tMn9h5bH3dBQR8O4WIf274/Y= 2022-09-02T05:58:03.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1P6tMn9h5bH3dBQR8O4WIf274/Y=', 'Content-Length': '0'} 2022-09-02T05:58:03.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd98178ba206422-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.611 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.612 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.612 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.613 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:03.613 DEBUG:boto:Data: 2022-09-02T05:58:03.614 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.615 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.615 DEBUG:boto:Params: {} 2022-09-02T05:58:03.616 DEBUG:boto:Token: None 2022-09-02T05:58:03.617 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68 2022-09-02T05:58:03.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1P6tMn9h5bH3dBQR8O4WIf274/Y= 2022-09-02T05:58:03.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1P6tMn9h5bH3dBQR8O4WIf274/Y=', 'Content-Length': '0'} 2022-09-02T05:58:03.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e3bf78a6d3c460-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:58:03.625 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.627 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.628 DEBUG:boto:Method: GET 2022-09-02T05:58:03.629 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:03.630 DEBUG:boto:Data: 2022-09-02T05:58:03.631 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.634 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.635 DEBUG:boto:Params: {} 2022-09-02T05:58:03.636 DEBUG:boto:Token: None 2022-09-02T05:58:03.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68 2022-09-02T05:58:03.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:epmdxK9fZS1bqRVSdg8HQRUBfiI= 2022-09-02T05:58:03.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:epmdxK9fZS1bqRVSdg8HQRUBfiI=', 'Content-Length': '0'} 2022-09-02T05:58:03.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7e78446a1a919f-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.690 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:03.691 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:03.692 DEBUG:boto:Method: GET 2022-09-02T05:58:03.693 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:03.693 DEBUG:boto:Data: 2022-09-02T05:58:03.694 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.695 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.696 DEBUG:boto:Params: {} 2022-09-02T05:58:03.696 DEBUG:boto:Token: None 2022-09-02T05:58:03.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj68 2022-09-02T05:58:03.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:epmdxK9fZS1bqRVSdg8HQRUBfiI= 2022-09-02T05:58:03.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:epmdxK9fZS1bqRVSdg8HQRUBfiI=', 'Content-Length': '0'} 2022-09-02T05:58:03.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faed79448db6eed2-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.702 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.703 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.703 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.704 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.705 DEBUG:boto:Method: GET 2022-09-02T05:58:03.705 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.706 DEBUG:boto:Data: 2022-09-02T05:58:03.706 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.708 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.708 DEBUG:boto:Params: {} 2022-09-02T05:58:03.709 DEBUG:boto:Token: None 2022-09-02T05:58:03.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da12292dd455e178-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.716 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.718 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.719 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.720 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.720 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.721 DEBUG:boto:Method: GET 2022-09-02T05:58:03.722 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:03.723 DEBUG:boto:Data: 2022-09-02T05:58:03.724 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.725 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.725 DEBUG:boto:Params: {} 2022-09-02T05:58:03.727 DEBUG:boto:Token: None 2022-09-02T05:58:03.728 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af21763abf452ca-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.735 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:58:03.736 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.737 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.737 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.738 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.738 DEBUG:boto:Method: GET 2022-09-02T05:58:03.739 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.739 DEBUG:boto:Data: 2022-09-02T05:58:03.740 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.741 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.742 DEBUG:boto:Params: {} 2022-09-02T05:58:03.742 DEBUG:boto:Token: None 2022-09-02T05:58:03.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM= 2022-09-02T05:58:03.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM=', 'Content-Length': '0'} 2022-09-02T05:58:03.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b121e3558e9fe4-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.747 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.748 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.749 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.749 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.750 DEBUG:boto:Method: GET 2022-09-02T05:58:03.750 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.751 DEBUG:boto:Data: 2022-09-02T05:58:03.751 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.752 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.753 DEBUG:boto:Params: {} 2022-09-02T05:58:03.754 DEBUG:boto:Token: None 2022-09-02T05:58:03.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM= 2022-09-02T05:58:03.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM=', 'Content-Length': '0'} 2022-09-02T05:58:03.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d052819c702ff2e9-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.759 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.760 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.760 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.761 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.761 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.762 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.762 DEBUG:boto:Data: 2022-09-02T05:58:03.763 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.764 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.764 DEBUG:boto:Params: {} 2022-09-02T05:58:03.765 DEBUG:boto:Token: None 2022-09-02T05:58:03.766 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j+VaIQ97fyd4owzmz5EhZtbnHjA= 2022-09-02T05:58:03.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j+VaIQ97fyd4owzmz5EhZtbnHjA=', 'Content-Length': '0'} 2022-09-02T05:58:03.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63e190714c52cdb-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.770 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.770 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.771 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.772 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.772 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.773 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.773 DEBUG:boto:Data: 2022-09-02T05:58:03.774 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.775 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.775 DEBUG:boto:Params: {} 2022-09-02T05:58:03.776 DEBUG:boto:Token: None 2022-09-02T05:58:03.777 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j+VaIQ97fyd4owzmz5EhZtbnHjA= 2022-09-02T05:58:03.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j+VaIQ97fyd4owzmz5EhZtbnHjA=', 'Content-Length': '0'} 2022-09-02T05:58:03.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001144a4d49e30b649-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:58:03.781 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.782 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.783 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.783 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.784 DEBUG:boto:Method: GET 2022-09-02T05:58:03.784 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.785 DEBUG:boto:Data: 2022-09-02T05:58:03.785 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.786 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.787 DEBUG:boto:Params: {} 2022-09-02T05:58:03.788 DEBUG:boto:Token: None 2022-09-02T05:58:03.788 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.789 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM= 2022-09-02T05:58:03.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM=', 'Content-Length': '0'} 2022-09-02T05:58:03.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea9a6b20d579624-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.834 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.835 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.836 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.836 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.837 DEBUG:boto:Method: GET 2022-09-02T05:58:03.837 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.838 DEBUG:boto:Data: 2022-09-02T05:58:03.838 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.839 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.840 DEBUG:boto:Params: {} 2022-09-02T05:58:03.841 DEBUG:boto:Token: None 2022-09-02T05:58:03.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:03.842 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM= 2022-09-02T05:58:03.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:psR2G5axYtFTbBGPomusGbzfXOM=', 'Content-Length': '0'} 2022-09-02T05:58:03.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017827f7a12f4a107-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.846 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.846 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.847 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.847 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.848 DEBUG:boto:Method: GET 2022-09-02T05:58:03.848 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.849 DEBUG:boto:Data: 2022-09-02T05:58:03.849 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.851 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.851 DEBUG:boto:Params: {} 2022-09-02T05:58:03.852 DEBUG:boto:Token: None 2022-09-02T05:58:03.853 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.853 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003066159e3a92ce9c-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.857 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.858 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.859 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.859 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.860 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.861 DEBUG:boto:Method: GET 2022-09-02T05:58:03.861 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.862 DEBUG:boto:Data: 2022-09-02T05:58:03.862 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.863 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.864 DEBUG:boto:Params: {} 2022-09-02T05:58:03.864 DEBUG:boto:Token: None 2022-09-02T05:58:03.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001a032b0723efa30-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.871 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.872 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.873 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.873 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.874 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:03.874 DEBUG:boto:Data: 2022-09-02T05:58:03.875 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.876 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.876 DEBUG:boto:Params: {} 2022-09-02T05:58:03.877 DEBUG:boto:Token: None 2022-09-02T05:58:03.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69 2022-09-02T05:58:03.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v2xXiPR9Hd6nCobbmICXeIBuIDk= 2022-09-02T05:58:03.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v2xXiPR9Hd6nCobbmICXeIBuIDk=', 'Content-Length': '0'} 2022-09-02T05:58:03.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cb5dc3cfc4591ca-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.882 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.882 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.883 DEBUG:boto:Method: HEAD 2022-09-02T05:58:03.883 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:03.884 DEBUG:boto:Data: 2022-09-02T05:58:03.884 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.885 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.886 DEBUG:boto:Params: {} 2022-09-02T05:58:03.886 DEBUG:boto:Token: None 2022-09-02T05:58:03.887 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69 2022-09-02T05:58:03.887 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v2xXiPR9Hd6nCobbmICXeIBuIDk= 2022-09-02T05:58:03.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v2xXiPR9Hd6nCobbmICXeIBuIDk=', 'Content-Length': '0'} 2022-09-02T05:58:03.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a823422aa69a42-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:58:03.891 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.892 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.892 DEBUG:boto:Method: GET 2022-09-02T05:58:03.893 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:03.893 DEBUG:boto:Data: 2022-09-02T05:58:03.894 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.895 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.895 DEBUG:boto:Params: {} 2022-09-02T05:58:03.896 DEBUG:boto:Token: None 2022-09-02T05:58:03.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69 2022-09-02T05:58:03.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O1oFcB21dsi/XkpGxtXK9yqEK1I= 2022-09-02T05:58:03.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O1oFcB21dsi/XkpGxtXK9yqEK1I=', 'Content-Length': '0'} 2022-09-02T05:58:03.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4e132cf6635bce-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.901 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:03.901 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:03.902 DEBUG:boto:Method: GET 2022-09-02T05:58:03.903 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:03.903 DEBUG:boto:Data: 2022-09-02T05:58:03.904 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.905 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.906 DEBUG:boto:Params: {} 2022-09-02T05:58:03.906 DEBUG:boto:Token: None 2022-09-02T05:58:03.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj69 2022-09-02T05:58:03.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O1oFcB21dsi/XkpGxtXK9yqEK1I= 2022-09-02T05:58:03.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O1oFcB21dsi/XkpGxtXK9yqEK1I=', 'Content-Length': '0'} 2022-09-02T05:58:03.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d1bcd398428fb4-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.912 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.913 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.913 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.914 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.915 DEBUG:boto:Method: GET 2022-09-02T05:58:03.915 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.916 DEBUG:boto:Data: 2022-09-02T05:58:03.917 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.918 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.919 DEBUG:boto:Params: {} 2022-09-02T05:58:03.920 DEBUG:boto:Token: None 2022-09-02T05:58:03.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8c5ecd4116ab22e-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.925 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:03.927 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:03.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.928 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.928 DEBUG:boto:Method: GET 2022-09-02T05:58:03.929 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:03.930 DEBUG:boto:Data: 2022-09-02T05:58:03.939 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.940 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.941 DEBUG:boto:Params: {} 2022-09-02T05:58:03.941 DEBUG:boto:Token: None 2022-09-02T05:58:03.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/ 2022-09-02T05:58:03.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY= 2022-09-02T05:58:03.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z7Ybxk9TMCRCnjQmrebh3L3aRiY=', 'Content-Length': '0'} 2022-09-02T05:58:03.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a5315e10036c247-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.948 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:03.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:58:03.950 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:03.951 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:03.951 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.952 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.952 DEBUG:boto:Method: GET 2022-09-02T05:58:03.953 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.953 DEBUG:boto:Data: 2022-09-02T05:58:03.954 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:03.955 DEBUG:boto:Port: 8001 2022-09-02T05:58:03.956 DEBUG:boto:Params: {} 2022-09-02T05:58:03.956 DEBUG:boto:Token: None 2022-09-02T05:58:03.957 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OAFIRWKbRpeJbZm9/lYWHKTIRQw= 2022-09-02T05:58:03.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OAFIRWKbRpeJbZm9/lYWHKTIRQw=', 'Content-Length': '0'} 2022-09-02T05:58:03.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d2fd1e28864949-0063119b6b-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:03.962 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:03.962 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:03.963 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.963 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.964 DEBUG:boto:Method: GET 2022-09-02T05:58:03.965 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.965 DEBUG:boto:Data: 2022-09-02T05:58:03.966 DEBUG:boto:Headers: {} 2022-09-02T05:58:03.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:03.967 DEBUG:boto:Port: 8000 2022-09-02T05:58:03.968 DEBUG:boto:Params: {} 2022-09-02T05:58:03.968 DEBUG:boto:Token: None 2022-09-02T05:58:03.969 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:03 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:03.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OAFIRWKbRpeJbZm9/lYWHKTIRQw= 2022-09-02T05:58:03.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:03 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OAFIRWKbRpeJbZm9/lYWHKTIRQw=', 'Content-Length': '0'} 2022-09-02T05:58:03.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cddc6ca34d629d4-0063119b6b-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:03 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.016 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.017 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.018 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.019 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.020 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.021 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.022 DEBUG:boto:Data: 2022-09-02T05:58:04.023 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.024 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.025 DEBUG:boto:Params: {} 2022-09-02T05:58:04.026 DEBUG:boto:Token: None 2022-09-02T05:58:04.026 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91RoAmE5rB9eIgGyIQdgK6LwULI= 2022-09-02T05:58:04.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91RoAmE5rB9eIgGyIQdgK6LwULI=', 'Content-Length': '0'} 2022-09-02T05:58:04.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b21f72e194e35d94-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.030 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.031 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.032 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.032 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.033 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.033 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.034 DEBUG:boto:Data: 2022-09-02T05:58:04.034 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.036 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.036 DEBUG:boto:Params: {} 2022-09-02T05:58:04.037 DEBUG:boto:Token: None 2022-09-02T05:58:04.037 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91RoAmE5rB9eIgGyIQdgK6LwULI= 2022-09-02T05:58:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91RoAmE5rB9eIgGyIQdgK6LwULI=', 'Content-Length': '0'} 2022-09-02T05:58:04.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d048770149ecbf8-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:58:04.042 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.043 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.044 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.044 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.045 DEBUG:boto:Method: GET 2022-09-02T05:58:04.045 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.046 DEBUG:boto:Data: 2022-09-02T05:58:04.046 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.047 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.048 DEBUG:boto:Params: {} 2022-09-02T05:58:04.049 DEBUG:boto:Token: None 2022-09-02T05:58:04.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lrtZ/QSx1Cvs4DOyyhk0NXN/2yo= 2022-09-02T05:58:04.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lrtZ/QSx1Cvs4DOyyhk0NXN/2yo=', 'Content-Length': '0'} 2022-09-02T05:58:04.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5685ba2d73824a-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.057 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.058 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.059 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.060 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.060 DEBUG:boto:Method: GET 2022-09-02T05:58:04.061 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.061 DEBUG:boto:Data: 2022-09-02T05:58:04.062 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.063 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.063 DEBUG:boto:Params: {} 2022-09-02T05:58:04.064 DEBUG:boto:Token: None 2022-09-02T05:58:04.065 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:04.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lrtZ/QSx1Cvs4DOyyhk0NXN/2yo= 2022-09-02T05:58:04.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lrtZ/QSx1Cvs4DOyyhk0NXN/2yo=', 'Content-Length': '0'} 2022-09-02T05:58:04.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d868f71a466b0e-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.070 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.070 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.071 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.071 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.072 DEBUG:boto:Method: GET 2022-09-02T05:58:04.072 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.073 DEBUG:boto:Data: 2022-09-02T05:58:04.074 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.075 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.075 DEBUG:boto:Params: {} 2022-09-02T05:58:04.076 DEBUG:boto:Token: None 2022-09-02T05:58:04.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e8f3d45c1c4003c-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.082 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.083 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.084 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.085 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.086 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.087 DEBUG:boto:Method: GET 2022-09-02T05:58:04.088 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.089 DEBUG:boto:Data: 2022-09-02T05:58:04.090 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.092 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.093 DEBUG:boto:Params: {} 2022-09-02T05:58:04.094 DEBUG:boto:Token: None 2022-09-02T05:58:04.095 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1f3fc8b4ae63b3-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.104 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.105 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.106 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.107 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.108 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:04.109 DEBUG:boto:Data: 2022-09-02T05:58:04.110 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.112 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.113 DEBUG:boto:Params: {} 2022-09-02T05:58:04.114 DEBUG:boto:Token: None 2022-09-02T05:58:04.115 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76 2022-09-02T05:58:04.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5uzM5Va5l0knBC/SgL6E7RYAQTA= 2022-09-02T05:58:04.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5uzM5Va5l0knBC/SgL6E7RYAQTA=', 'Content-Length': '0'} 2022-09-02T05:58:04.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069af15c0a4bf7fca-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.120 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.120 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.121 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.122 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:04.122 DEBUG:boto:Data: 2022-09-02T05:58:04.123 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.124 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.124 DEBUG:boto:Params: {} 2022-09-02T05:58:04.125 DEBUG:boto:Token: None 2022-09-02T05:58:04.125 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76 2022-09-02T05:58:04.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5uzM5Va5l0knBC/SgL6E7RYAQTA= 2022-09-02T05:58:04.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5uzM5Va5l0knBC/SgL6E7RYAQTA=', 'Content-Length': '0'} 2022-09-02T05:58:04.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2a6d46be2f02431-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:58:04.130 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.130 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.131 DEBUG:boto:Method: GET 2022-09-02T05:58:04.131 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:04.132 DEBUG:boto:Data: 2022-09-02T05:58:04.133 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.134 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.134 DEBUG:boto:Params: {} 2022-09-02T05:58:04.135 DEBUG:boto:Token: None 2022-09-02T05:58:04.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76 2022-09-02T05:58:04.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:utgjSAtJMW7S5cF6z0NWuXgZUW0= 2022-09-02T05:58:04.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:utgjSAtJMW7S5cF6z0NWuXgZUW0=', 'Content-Length': '0'} 2022-09-02T05:58:04.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa45d544148add0-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.140 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:04.141 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:04.141 DEBUG:boto:Method: GET 2022-09-02T05:58:04.142 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:04.142 DEBUG:boto:Data: 2022-09-02T05:58:04.143 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.145 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.146 DEBUG:boto:Params: {} 2022-09-02T05:58:04.147 DEBUG:boto:Token: None 2022-09-02T05:58:04.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj76 2022-09-02T05:58:04.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:utgjSAtJMW7S5cF6z0NWuXgZUW0= 2022-09-02T05:58:04.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:utgjSAtJMW7S5cF6z0NWuXgZUW0=', 'Content-Length': '0'} 2022-09-02T05:58:04.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d057396bdb9c4369-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.154 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.156 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.156 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.157 DEBUG:boto:Method: GET 2022-09-02T05:58:04.158 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.158 DEBUG:boto:Data: 2022-09-02T05:58:04.159 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.161 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.161 DEBUG:boto:Params: {} 2022-09-02T05:58:04.162 DEBUG:boto:Token: None 2022-09-02T05:58:04.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035087356a7775432-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.168 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.169 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.169 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.170 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.171 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.171 DEBUG:boto:Method: GET 2022-09-02T05:58:04.172 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:04.172 DEBUG:boto:Data: 2022-09-02T05:58:04.173 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.175 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.175 DEBUG:boto:Params: {} 2022-09-02T05:58:04.176 DEBUG:boto:Token: None 2022-09-02T05:58:04.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.177 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000746268ef002661b5-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.183 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:58:04.184 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.185 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.185 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.186 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.186 DEBUG:boto:Method: GET 2022-09-02T05:58:04.187 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.188 DEBUG:boto:Data: 2022-09-02T05:58:04.188 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.190 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.190 DEBUG:boto:Params: {} 2022-09-02T05:58:04.192 DEBUG:boto:Token: None 2022-09-02T05:58:04.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.193 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc= 2022-09-02T05:58:04.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc=', 'Content-Length': '0'} 2022-09-02T05:58:04.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5d438262831201-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.238 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.239 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.239 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.240 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.241 DEBUG:boto:Method: GET 2022-09-02T05:58:04.242 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.243 DEBUG:boto:Data: 2022-09-02T05:58:04.243 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.244 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.245 DEBUG:boto:Params: {} 2022-09-02T05:58:04.245 DEBUG:boto:Token: None 2022-09-02T05:58:04.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.247 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc= 2022-09-02T05:58:04.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc=', 'Content-Length': '0'} 2022-09-02T05:58:04.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027f004dadddde2eb-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.251 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.251 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.252 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.253 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.253 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.254 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.254 DEBUG:boto:Data: 2022-09-02T05:58:04.255 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.256 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.256 DEBUG:boto:Params: {} 2022-09-02T05:58:04.257 DEBUG:boto:Token: None 2022-09-02T05:58:04.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HLQlSllQIZ0c1c86qabKeclJ0a8= 2022-09-02T05:58:04.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HLQlSllQIZ0c1c86qabKeclJ0a8=', 'Content-Length': '0'} 2022-09-02T05:58:04.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007418379d73776b5a-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.262 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.263 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.264 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.264 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.265 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.265 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.266 DEBUG:boto:Data: 2022-09-02T05:58:04.266 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.267 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.268 DEBUG:boto:Params: {} 2022-09-02T05:58:04.269 DEBUG:boto:Token: None 2022-09-02T05:58:04.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HLQlSllQIZ0c1c86qabKeclJ0a8= 2022-09-02T05:58:04.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HLQlSllQIZ0c1c86qabKeclJ0a8=', 'Content-Length': '0'} 2022-09-02T05:58:04.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab53595b2cf0578-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:58:04.275 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.276 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.276 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.277 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.277 DEBUG:boto:Method: GET 2022-09-02T05:58:04.278 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.279 DEBUG:boto:Data: 2022-09-02T05:58:04.279 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.280 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.281 DEBUG:boto:Params: {} 2022-09-02T05:58:04.281 DEBUG:boto:Token: None 2022-09-02T05:58:04.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.283 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc= 2022-09-02T05:58:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc=', 'Content-Length': '0'} 2022-09-02T05:58:04.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098625fdb637f6fcc-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.287 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.287 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.288 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.288 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.289 DEBUG:boto:Method: GET 2022-09-02T05:58:04.289 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.290 DEBUG:boto:Data: 2022-09-02T05:58:04.290 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.292 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.292 DEBUG:boto:Params: {} 2022-09-02T05:58:04.293 DEBUG:boto:Token: None 2022-09-02T05:58:04.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:04.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc= 2022-09-02T05:58:04.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GY+uTzv6ZB6xIQdsT8V9eB5hdHc=', 'Content-Length': '0'} 2022-09-02T05:58:04.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8862555bbe53b72-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.342 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.343 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.343 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.344 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.344 DEBUG:boto:Method: GET 2022-09-02T05:58:04.345 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.346 DEBUG:boto:Data: 2022-09-02T05:58:04.346 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.347 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.348 DEBUG:boto:Params: {} 2022-09-02T05:58:04.349 DEBUG:boto:Token: None 2022-09-02T05:58:04.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008676bdb32167c435-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.355 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.356 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.357 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.358 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.359 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.360 DEBUG:boto:Method: GET 2022-09-02T05:58:04.360 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.361 DEBUG:boto:Data: 2022-09-02T05:58:04.362 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.364 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.365 DEBUG:boto:Params: {} 2022-09-02T05:58:04.365 DEBUG:boto:Token: None 2022-09-02T05:58:04.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c3779250b0dfd2-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.372 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.373 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.374 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.374 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.375 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:04.375 DEBUG:boto:Data: 2022-09-02T05:58:04.376 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.377 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.377 DEBUG:boto:Params: {} 2022-09-02T05:58:04.378 DEBUG:boto:Token: None 2022-09-02T05:58:04.379 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85 2022-09-02T05:58:04.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9OjyYc7ZdW+V+lewQY0yjxLhDx4= 2022-09-02T05:58:04.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9OjyYc7ZdW+V+lewQY0yjxLhDx4=', 'Content-Length': '0'} 2022-09-02T05:58:04.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ce9a1bd0953923-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.383 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.384 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.384 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.385 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:04.385 DEBUG:boto:Data: 2022-09-02T05:58:04.386 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.387 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.387 DEBUG:boto:Params: {} 2022-09-02T05:58:04.388 DEBUG:boto:Token: None 2022-09-02T05:58:04.388 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85 2022-09-02T05:58:04.389 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9OjyYc7ZdW+V+lewQY0yjxLhDx4= 2022-09-02T05:58:04.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9OjyYc7ZdW+V+lewQY0yjxLhDx4=', 'Content-Length': '0'} 2022-09-02T05:58:04.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed25de4eedb87539-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:58:04.394 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.394 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.395 DEBUG:boto:Method: GET 2022-09-02T05:58:04.395 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:04.396 DEBUG:boto:Data: 2022-09-02T05:58:04.397 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.398 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.399 DEBUG:boto:Params: {} 2022-09-02T05:58:04.399 DEBUG:boto:Token: None 2022-09-02T05:58:04.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85 2022-09-02T05:58:04.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:do4+88FM0RW6189oqr7ZdvGQ5Fs= 2022-09-02T05:58:04.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:do4+88FM0RW6189oqr7ZdvGQ5Fs=', 'Content-Length': '0'} 2022-09-02T05:58:04.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17803f9b6bc95d8-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.406 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:04.407 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:04.408 DEBUG:boto:Method: GET 2022-09-02T05:58:04.409 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:04.410 DEBUG:boto:Data: 2022-09-02T05:58:04.411 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.413 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.414 DEBUG:boto:Params: {} 2022-09-02T05:58:04.415 DEBUG:boto:Token: None 2022-09-02T05:58:04.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj85 2022-09-02T05:58:04.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:do4+88FM0RW6189oqr7ZdvGQ5Fs= 2022-09-02T05:58:04.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:do4+88FM0RW6189oqr7ZdvGQ5Fs=', 'Content-Length': '0'} 2022-09-02T05:58:04.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071add92cd2648adb-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.423 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.424 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.425 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.426 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.427 DEBUG:boto:Method: GET 2022-09-02T05:58:04.428 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.429 DEBUG:boto:Data: 2022-09-02T05:58:04.430 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.432 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.433 DEBUG:boto:Params: {} 2022-09-02T05:58:04.434 DEBUG:boto:Token: None 2022-09-02T05:58:04.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.436 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e746e5dc2221000-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.442 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.444 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.445 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.446 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.446 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.447 DEBUG:boto:Method: GET 2022-09-02T05:58:04.448 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:04.449 DEBUG:boto:Data: 2022-09-02T05:58:04.450 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.452 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.453 DEBUG:boto:Params: {} 2022-09-02T05:58:04.455 DEBUG:boto:Token: None 2022-09-02T05:58:04.456 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a52d71a601da7a-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.464 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:58:04.466 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.467 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.468 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.469 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.470 DEBUG:boto:Method: GET 2022-09-02T05:58:04.471 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.472 DEBUG:boto:Data: 2022-09-02T05:58:04.473 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.475 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.476 DEBUG:boto:Params: {} 2022-09-02T05:58:04.477 DEBUG:boto:Token: None 2022-09-02T05:58:04.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4= 2022-09-02T05:58:04.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4=', 'Content-Length': '0'} 2022-09-02T05:58:04.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0d7a968da22807-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.482 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.483 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.484 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.484 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.485 DEBUG:boto:Method: GET 2022-09-02T05:58:04.485 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.486 DEBUG:boto:Data: 2022-09-02T05:58:04.486 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.487 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.488 DEBUG:boto:Params: {} 2022-09-02T05:58:04.489 DEBUG:boto:Token: None 2022-09-02T05:58:04.489 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4= 2022-09-02T05:58:04.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4=', 'Content-Length': '0'} 2022-09-02T05:58:04.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ffaead9c4b62587-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.538 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.539 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.540 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.541 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.543 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.543 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.544 DEBUG:boto:Data: 2022-09-02T05:58:04.545 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.547 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.548 DEBUG:boto:Params: {} 2022-09-02T05:58:04.549 DEBUG:boto:Token: None 2022-09-02T05:58:04.551 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A01C5ZZf3watn8Nj3GId/jLQntM= 2022-09-02T05:58:04.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A01C5ZZf3watn8Nj3GId/jLQntM=', 'Content-Length': '0'} 2022-09-02T05:58:04.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f31c60bfcf42fc-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.557 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.558 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.559 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.560 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.561 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.562 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.563 DEBUG:boto:Data: 2022-09-02T05:58:04.564 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.566 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.566 DEBUG:boto:Params: {} 2022-09-02T05:58:04.567 DEBUG:boto:Token: None 2022-09-02T05:58:04.568 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.568 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A01C5ZZf3watn8Nj3GId/jLQntM= 2022-09-02T05:58:04.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A01C5ZZf3watn8Nj3GId/jLQntM=', 'Content-Length': '0'} 2022-09-02T05:58:04.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004304a3a2bf14bfcf-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:58:04.573 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.573 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.574 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.574 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.575 DEBUG:boto:Method: GET 2022-09-02T05:58:04.576 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.576 DEBUG:boto:Data: 2022-09-02T05:58:04.577 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.578 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.578 DEBUG:boto:Params: {} 2022-09-02T05:58:04.579 DEBUG:boto:Token: None 2022-09-02T05:58:04.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4= 2022-09-02T05:58:04.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4=', 'Content-Length': '0'} 2022-09-02T05:58:04.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ca9eba11cb5654-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.585 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.585 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.586 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.586 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.587 DEBUG:boto:Method: GET 2022-09-02T05:58:04.587 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.588 DEBUG:boto:Data: 2022-09-02T05:58:04.589 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.590 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.591 DEBUG:boto:Params: {} 2022-09-02T05:58:04.592 DEBUG:boto:Token: None 2022-09-02T05:58:04.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:04.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4= 2022-09-02T05:58:04.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lz+N2My3ohw7Jyi3jTcr4GIK9O4=', 'Content-Length': '0'} 2022-09-02T05:58:04.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd6feaa247a50d3-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.599 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.599 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.600 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.600 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.601 DEBUG:boto:Method: GET 2022-09-02T05:58:04.601 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.602 DEBUG:boto:Data: 2022-09-02T05:58:04.603 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.604 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.604 DEBUG:boto:Params: {} 2022-09-02T05:58:04.605 DEBUG:boto:Token: None 2022-09-02T05:58:04.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d262eb69ef12b2ee-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.611 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.612 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.612 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.613 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.613 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.614 DEBUG:boto:Method: GET 2022-09-02T05:58:04.614 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.615 DEBUG:boto:Data: 2022-09-02T05:58:04.615 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.617 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.617 DEBUG:boto:Params: {} 2022-09-02T05:58:04.618 DEBUG:boto:Token: None 2022-09-02T05:58:04.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e79276eee33e60d-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.625 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.626 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.627 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.627 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.628 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:04.628 DEBUG:boto:Data: 2022-09-02T05:58:04.629 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.630 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.630 DEBUG:boto:Params: {} 2022-09-02T05:58:04.631 DEBUG:boto:Token: None 2022-09-02T05:58:04.632 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87 2022-09-02T05:58:04.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:09opy0d3gSH89o4Ul234HedhOEA= 2022-09-02T05:58:04.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:09opy0d3gSH89o4Ul234HedhOEA=', 'Content-Length': '0'} 2022-09-02T05:58:04.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a4e486209c5450-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.636 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.636 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.637 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.637 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:04.638 DEBUG:boto:Data: 2022-09-02T05:58:04.638 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.639 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.640 DEBUG:boto:Params: {} 2022-09-02T05:58:04.641 DEBUG:boto:Token: None 2022-09-02T05:58:04.641 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87 2022-09-02T05:58:04.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:09opy0d3gSH89o4Ul234HedhOEA= 2022-09-02T05:58:04.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:09opy0d3gSH89o4Ul234HedhOEA=', 'Content-Length': '0'} 2022-09-02T05:58:04.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e09525133dc84e-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:58:04.646 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.647 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.647 DEBUG:boto:Method: GET 2022-09-02T05:58:04.648 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:04.649 DEBUG:boto:Data: 2022-09-02T05:58:04.649 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.650 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.651 DEBUG:boto:Params: {} 2022-09-02T05:58:04.651 DEBUG:boto:Token: None 2022-09-02T05:58:04.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87 2022-09-02T05:58:04.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCGDwInSJi3dxJX0IbaJEgIo4W4= 2022-09-02T05:58:04.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCGDwInSJi3dxJX0IbaJEgIo4W4=', 'Content-Length': '0'} 2022-09-02T05:58:04.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4098093e78daca4-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.698 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:04.699 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:04.699 DEBUG:boto:Method: GET 2022-09-02T05:58:04.700 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:04.700 DEBUG:boto:Data: 2022-09-02T05:58:04.701 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.702 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.703 DEBUG:boto:Params: {} 2022-09-02T05:58:04.703 DEBUG:boto:Token: None 2022-09-02T05:58:04.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj87 2022-09-02T05:58:04.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RCGDwInSJi3dxJX0IbaJEgIo4W4= 2022-09-02T05:58:04.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RCGDwInSJi3dxJX0IbaJEgIo4W4=', 'Content-Length': '0'} 2022-09-02T05:58:04.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015be40f4243bfa7b-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.709 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.709 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.710 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.710 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.711 DEBUG:boto:Method: GET 2022-09-02T05:58:04.711 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.712 DEBUG:boto:Data: 2022-09-02T05:58:04.712 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.713 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.714 DEBUG:boto:Params: {} 2022-09-02T05:58:04.715 DEBUG:boto:Token: None 2022-09-02T05:58:04.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d45d98d14866635-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.720 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.721 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.721 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.722 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.723 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.723 DEBUG:boto:Method: GET 2022-09-02T05:58:04.724 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:04.724 DEBUG:boto:Data: 2022-09-02T05:58:04.725 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.726 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.726 DEBUG:boto:Params: {} 2022-09-02T05:58:04.727 DEBUG:boto:Token: None 2022-09-02T05:58:04.727 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f75f92cee9c42cb6-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.734 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:58:04.736 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.736 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.737 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.737 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.738 DEBUG:boto:Method: GET 2022-09-02T05:58:04.738 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.739 DEBUG:boto:Data: 2022-09-02T05:58:04.739 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.741 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.741 DEBUG:boto:Params: {} 2022-09-02T05:58:04.742 DEBUG:boto:Token: None 2022-09-02T05:58:04.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE= 2022-09-02T05:58:04.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE=', 'Content-Length': '0'} 2022-09-02T05:58:04.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a287427ab2209acf-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.747 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.748 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.748 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.749 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.750 DEBUG:boto:Method: GET 2022-09-02T05:58:04.750 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.751 DEBUG:boto:Data: 2022-09-02T05:58:04.752 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.753 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.753 DEBUG:boto:Params: {} 2022-09-02T05:58:04.754 DEBUG:boto:Token: None 2022-09-02T05:58:04.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE= 2022-09-02T05:58:04.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE=', 'Content-Length': '0'} 2022-09-02T05:58:04.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58341d362a3ad22-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.802 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.803 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.803 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.804 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.804 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.805 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.805 DEBUG:boto:Data: 2022-09-02T05:58:04.806 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.807 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.807 DEBUG:boto:Params: {} 2022-09-02T05:58:04.808 DEBUG:boto:Token: None 2022-09-02T05:58:04.809 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwSgsKPcndPDz/+ExTIOtIs7N1s= 2022-09-02T05:58:04.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwSgsKPcndPDz/+ExTIOtIs7N1s=', 'Content-Length': '0'} 2022-09-02T05:58:04.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028f587d97b8110d-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.813 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.813 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.814 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.814 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.815 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.816 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.817 DEBUG:boto:Data: 2022-09-02T05:58:04.817 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.818 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.819 DEBUG:boto:Params: {} 2022-09-02T05:58:04.819 DEBUG:boto:Token: None 2022-09-02T05:58:04.820 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fwSgsKPcndPDz/+ExTIOtIs7N1s= 2022-09-02T05:58:04.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fwSgsKPcndPDz/+ExTIOtIs7N1s=', 'Content-Length': '0'} 2022-09-02T05:58:04.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d08c972c3a12835-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:58:04.825 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.825 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.826 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.826 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.827 DEBUG:boto:Method: GET 2022-09-02T05:58:04.827 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.828 DEBUG:boto:Data: 2022-09-02T05:58:04.828 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.830 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.830 DEBUG:boto:Params: {} 2022-09-02T05:58:04.831 DEBUG:boto:Token: None 2022-09-02T05:58:04.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE= 2022-09-02T05:58:04.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE=', 'Content-Length': '0'} 2022-09-02T05:58:04.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898a5c0e0f1c1eab-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.836 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.836 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.837 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.837 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.838 DEBUG:boto:Method: GET 2022-09-02T05:58:04.838 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.839 DEBUG:boto:Data: 2022-09-02T05:58:04.839 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.840 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.841 DEBUG:boto:Params: {} 2022-09-02T05:58:04.842 DEBUG:boto:Token: None 2022-09-02T05:58:04.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:04.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE= 2022-09-02T05:58:04.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WC+WBosnGSe2a3x5zSH0AJxzMsE=', 'Content-Length': '0'} 2022-09-02T05:58:04.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c060d6892a76ce-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.847 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.847 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.848 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:04.848 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:04.849 DEBUG:boto:Method: GET 2022-09-02T05:58:04.850 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:04.850 DEBUG:boto:Data: 2022-09-02T05:58:04.851 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.852 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.853 DEBUG:boto:Params: {} 2022-09-02T05:58:04.853 DEBUG:boto:Token: None 2022-09-02T05:58:04.854 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966a9fa50a85fa86-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.902 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.904 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:04.905 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:04.905 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:04.906 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:04.907 DEBUG:boto:Method: GET 2022-09-02T05:58:04.908 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:04.909 DEBUG:boto:Data: 2022-09-02T05:58:04.910 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.912 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.912 DEBUG:boto:Params: {} 2022-09-02T05:58:04.914 DEBUG:boto:Token: None 2022-09-02T05:58:04.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/ 2022-09-02T05:58:04.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM= 2022-09-02T05:58:04.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9fdAF5Y8vEoRhjEImeBCwVM8woM=', 'Content-Length': '0'} 2022-09-02T05:58:04.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f58c605dd25dd5-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.924 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:04.926 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.927 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.927 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.928 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:04.929 DEBUG:boto:Data: 2022-09-02T05:58:04.930 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.932 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.933 DEBUG:boto:Params: {} 2022-09-02T05:58:04.934 DEBUG:boto:Token: None 2022-09-02T05:58:04.935 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92 2022-09-02T05:58:04.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7P7xsFfYoi/7bkmvGWHXNAi1Et8= 2022-09-02T05:58:04.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7P7xsFfYoi/7bkmvGWHXNAi1Et8=', 'Content-Length': '0'} 2022-09-02T05:58:04.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1bdffe0fb18c1a-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.941 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.942 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.942 DEBUG:boto:Method: HEAD 2022-09-02T05:58:04.943 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:04.943 DEBUG:boto:Data: 2022-09-02T05:58:04.944 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.945 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.945 DEBUG:boto:Params: {} 2022-09-02T05:58:04.946 DEBUG:boto:Token: None 2022-09-02T05:58:04.947 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92 2022-09-02T05:58:04.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7P7xsFfYoi/7bkmvGWHXNAi1Et8= 2022-09-02T05:58:04.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7P7xsFfYoi/7bkmvGWHXNAi1Et8=', 'Content-Length': '0'} 2022-09-02T05:58:04.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a500a1d10522d35-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:58:04.952 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.953 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.954 DEBUG:boto:Method: GET 2022-09-02T05:58:04.955 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:04.955 DEBUG:boto:Data: 2022-09-02T05:58:04.956 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:04.957 DEBUG:boto:Port: 8001 2022-09-02T05:58:04.958 DEBUG:boto:Params: {} 2022-09-02T05:58:04.959 DEBUG:boto:Token: None 2022-09-02T05:58:04.960 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92 2022-09-02T05:58:04.961 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ODcjKvfwx83pqGQenbjsuNkI1o= 2022-09-02T05:58:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ODcjKvfwx83pqGQenbjsuNkI1o=', 'Content-Length': '0'} 2022-09-02T05:58:04.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec10a7d2c2ba1f9-0063119b6c-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:04.966 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:04.966 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:04.967 DEBUG:boto:Method: GET 2022-09-02T05:58:04.968 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:04.968 DEBUG:boto:Data: 2022-09-02T05:58:04.969 DEBUG:boto:Headers: {} 2022-09-02T05:58:04.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:04.970 DEBUG:boto:Port: 8000 2022-09-02T05:58:04.971 DEBUG:boto:Params: {} 2022-09-02T05:58:04.973 DEBUG:boto:Token: None 2022-09-02T05:58:04.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:04 GMT /xljbkf-74/obj92 2022-09-02T05:58:04.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7ODcjKvfwx83pqGQenbjsuNkI1o= 2022-09-02T05:58:04.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:04 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7ODcjKvfwx83pqGQenbjsuNkI1o=', 'Content-Length': '0'} 2022-09-02T05:58:04.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c9e51c9a9a7cdf-0063119b6c-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:04 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.022 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.023 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.024 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:05.026 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:05.027 DEBUG:boto:Method: GET 2022-09-02T05:58:05.029 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:05.030 DEBUG:boto:Data: 2022-09-02T05:58:05.031 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.033 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.034 DEBUG:boto:Params: {} 2022-09-02T05:58:05.035 DEBUG:boto:Token: None 2022-09-02T05:58:05.036 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa0d8a9615daa0c-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.044 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.045 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.046 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.047 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:05.048 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:05.049 DEBUG:boto:Method: GET 2022-09-02T05:58:05.050 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:05.051 DEBUG:boto:Data: 2022-09-02T05:58:05.052 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.053 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.054 DEBUG:boto:Params: {} 2022-09-02T05:58:05.055 DEBUG:boto:Token: None 2022-09-02T05:58:05.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008324170ae03b5a97-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.064 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:58:05.067 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.067 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.068 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.069 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.070 DEBUG:boto:Method: GET 2022-09-02T05:58:05.071 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.072 DEBUG:boto:Data: 2022-09-02T05:58:05.074 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.076 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.076 DEBUG:boto:Params: {} 2022-09-02T05:58:05.077 DEBUG:boto:Token: None 2022-09-02T05:58:05.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg= 2022-09-02T05:58:05.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg=', 'Content-Length': '0'} 2022-09-02T05:58:05.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69de2bbb4acd8ea-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.085 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.085 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.086 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.086 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.087 DEBUG:boto:Method: GET 2022-09-02T05:58:05.088 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.088 DEBUG:boto:Data: 2022-09-02T05:58:05.089 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.090 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.091 DEBUG:boto:Params: {} 2022-09-02T05:58:05.091 DEBUG:boto:Token: None 2022-09-02T05:58:05.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg= 2022-09-02T05:58:05.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg=', 'Content-Length': '0'} 2022-09-02T05:58:05.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eecf74c9b0bea5d6-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.096 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.097 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.098 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.098 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.099 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.099 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.100 DEBUG:boto:Data: 2022-09-02T05:58:05.100 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.102 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.103 DEBUG:boto:Params: {} 2022-09-02T05:58:05.104 DEBUG:boto:Token: None 2022-09-02T05:58:05.105 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jb16OTLzTDb81kSFPflJzcDnNGA= 2022-09-02T05:58:05.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jb16OTLzTDb81kSFPflJzcDnNGA=', 'Content-Length': '0'} 2022-09-02T05:58:05.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cbf938717e0c3df-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.110 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.111 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.112 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.112 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.113 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.114 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.115 DEBUG:boto:Data: 2022-09-02T05:58:05.116 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.118 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.118 DEBUG:boto:Params: {} 2022-09-02T05:58:05.119 DEBUG:boto:Token: None 2022-09-02T05:58:05.120 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jb16OTLzTDb81kSFPflJzcDnNGA= 2022-09-02T05:58:05.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jb16OTLzTDb81kSFPflJzcDnNGA=', 'Content-Length': '0'} 2022-09-02T05:58:05.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c260d7dbbe8d5f3-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:58:05.128 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.128 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.130 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.130 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.131 DEBUG:boto:Method: GET 2022-09-02T05:58:05.132 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.132 DEBUG:boto:Data: 2022-09-02T05:58:05.133 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.134 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.135 DEBUG:boto:Params: {} 2022-09-02T05:58:05.135 DEBUG:boto:Token: None 2022-09-02T05:58:05.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg= 2022-09-02T05:58:05.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg=', 'Content-Length': '0'} 2022-09-02T05:58:05.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014724325e164319b-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.141 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.142 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.143 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.144 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.145 DEBUG:boto:Method: GET 2022-09-02T05:58:05.146 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.147 DEBUG:boto:Data: 2022-09-02T05:58:05.148 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.150 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.151 DEBUG:boto:Params: {} 2022-09-02T05:58:05.152 DEBUG:boto:Token: None 2022-09-02T05:58:05.153 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:05.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg= 2022-09-02T05:58:05.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MR2J1ShaL+BLOnCe0er9NKzkuhg=', 'Content-Length': '0'} 2022-09-02T05:58:05.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001909f99bd79d566d-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.157 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.158 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.158 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.159 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.160 DEBUG:boto:Method: GET 2022-09-02T05:58:05.160 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.161 DEBUG:boto:Data: 2022-09-02T05:58:05.161 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.162 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.163 DEBUG:boto:Params: {} 2022-09-02T05:58:05.163 DEBUG:boto:Token: None 2022-09-02T05:58:05.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074e3812634dbc8c5-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.169 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.170 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.171 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.171 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.172 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.172 DEBUG:boto:Method: GET 2022-09-02T05:58:05.173 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.173 DEBUG:boto:Data: 2022-09-02T05:58:05.174 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.175 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.176 DEBUG:boto:Params: {} 2022-09-02T05:58:05.177 DEBUG:boto:Token: None 2022-09-02T05:58:05.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36d87e58a3711c9-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.183 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.184 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.185 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.186 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.186 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:05.187 DEBUG:boto:Data: 2022-09-02T05:58:05.187 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.188 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.189 DEBUG:boto:Params: {} 2022-09-02T05:58:05.189 DEBUG:boto:Token: None 2022-09-02T05:58:05.190 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93 2022-09-02T05:58:05.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pq6S4KB79rYrQ1SnSJfjYYlfZvA= 2022-09-02T05:58:05.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pq6S4KB79rYrQ1SnSJfjYYlfZvA=', 'Content-Length': '0'} 2022-09-02T05:58:05.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f9db7a0b8c9c45-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.194 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.194 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.195 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.195 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:05.196 DEBUG:boto:Data: 2022-09-02T05:58:05.197 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.198 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.198 DEBUG:boto:Params: {} 2022-09-02T05:58:05.199 DEBUG:boto:Token: None 2022-09-02T05:58:05.199 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93 2022-09-02T05:58:05.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pq6S4KB79rYrQ1SnSJfjYYlfZvA= 2022-09-02T05:58:05.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pq6S4KB79rYrQ1SnSJfjYYlfZvA=', 'Content-Length': '0'} 2022-09-02T05:58:05.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c31db81a9c651d-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:58:05.205 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.206 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.207 DEBUG:boto:Method: GET 2022-09-02T05:58:05.208 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:05.209 DEBUG:boto:Data: 2022-09-02T05:58:05.210 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.212 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.213 DEBUG:boto:Params: {} 2022-09-02T05:58:05.215 DEBUG:boto:Token: None 2022-09-02T05:58:05.216 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93 2022-09-02T05:58:05.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r40Z239JClTu8L46KKZjCy5bR/A= 2022-09-02T05:58:05.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r40Z239JClTu8L46KKZjCy5bR/A=', 'Content-Length': '0'} 2022-09-02T05:58:05.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978c6f56ab941b24-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.222 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:05.223 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:05.224 DEBUG:boto:Method: GET 2022-09-02T05:58:05.225 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:05.226 DEBUG:boto:Data: 2022-09-02T05:58:05.227 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.229 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.230 DEBUG:boto:Params: {} 2022-09-02T05:58:05.231 DEBUG:boto:Token: None 2022-09-02T05:58:05.232 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj93 2022-09-02T05:58:05.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r40Z239JClTu8L46KKZjCy5bR/A= 2022-09-02T05:58:05.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r40Z239JClTu8L46KKZjCy5bR/A=', 'Content-Length': '0'} 2022-09-02T05:58:05.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028617d2a0f3b58db-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.278 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.279 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.279 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.280 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.280 DEBUG:boto:Method: GET 2022-09-02T05:58:05.281 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.281 DEBUG:boto:Data: 2022-09-02T05:58:05.282 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.283 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.284 DEBUG:boto:Params: {} 2022-09-02T05:58:05.284 DEBUG:boto:Token: None 2022-09-02T05:58:05.285 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e932d7726d0b2a6-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.289 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.290 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.291 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.292 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.292 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.293 DEBUG:boto:Method: GET 2022-09-02T05:58:05.293 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:05.294 DEBUG:boto:Data: 2022-09-02T05:58:05.294 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.295 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.296 DEBUG:boto:Params: {} 2022-09-02T05:58:05.296 DEBUG:boto:Token: None 2022-09-02T05:58:05.297 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b673482fbb4f718-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.303 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:58:05.305 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.305 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.306 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.306 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.307 DEBUG:boto:Method: GET 2022-09-02T05:58:05.307 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.308 DEBUG:boto:Data: 2022-09-02T05:58:05.308 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.309 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.310 DEBUG:boto:Params: {} 2022-09-02T05:58:05.311 DEBUG:boto:Token: None 2022-09-02T05:58:05.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM= 2022-09-02T05:58:05.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM=', 'Content-Length': '0'} 2022-09-02T05:58:05.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d289ccbedf9e54a8-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.315 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.316 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.316 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.317 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.317 DEBUG:boto:Method: GET 2022-09-02T05:58:05.318 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.318 DEBUG:boto:Data: 2022-09-02T05:58:05.319 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.320 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.321 DEBUG:boto:Params: {} 2022-09-02T05:58:05.321 DEBUG:boto:Token: None 2022-09-02T05:58:05.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM= 2022-09-02T05:58:05.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM=', 'Content-Length': '0'} 2022-09-02T05:58:05.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d484e731740884-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.326 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.327 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.327 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.328 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.329 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.329 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.330 DEBUG:boto:Data: 2022-09-02T05:58:05.330 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.332 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.332 DEBUG:boto:Params: {} 2022-09-02T05:58:05.333 DEBUG:boto:Token: None 2022-09-02T05:58:05.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVMhc73gUq/7k1BCqcXS2leVP5Q= 2022-09-02T05:58:05.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVMhc73gUq/7k1BCqcXS2leVP5Q=', 'Content-Length': '0'} 2022-09-02T05:58:05.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd70b1576d58282-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.338 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.338 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.339 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.339 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.340 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.341 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.341 DEBUG:boto:Data: 2022-09-02T05:58:05.342 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.343 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.343 DEBUG:boto:Params: {} 2022-09-02T05:58:05.344 DEBUG:boto:Token: None 2022-09-02T05:58:05.345 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVMhc73gUq/7k1BCqcXS2leVP5Q= 2022-09-02T05:58:05.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVMhc73gUq/7k1BCqcXS2leVP5Q=', 'Content-Length': '0'} 2022-09-02T05:58:05.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061507666cdda5930-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:58:05.350 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.351 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.351 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.352 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.352 DEBUG:boto:Method: GET 2022-09-02T05:58:05.353 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.353 DEBUG:boto:Data: 2022-09-02T05:58:05.354 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.355 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.356 DEBUG:boto:Params: {} 2022-09-02T05:58:05.356 DEBUG:boto:Token: None 2022-09-02T05:58:05.357 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.358 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM= 2022-09-02T05:58:05.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM=', 'Content-Length': '0'} 2022-09-02T05:58:05.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b38075469f27bf30-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.402 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.403 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.403 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.404 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.405 DEBUG:boto:Method: GET 2022-09-02T05:58:05.406 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.406 DEBUG:boto:Data: 2022-09-02T05:58:05.407 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.408 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.409 DEBUG:boto:Params: {} 2022-09-02T05:58:05.409 DEBUG:boto:Token: None 2022-09-02T05:58:05.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:05.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM= 2022-09-02T05:58:05.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4sTMXuaY7EGtrc5x5EbXn5tautM=', 'Content-Length': '0'} 2022-09-02T05:58:05.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326630965c67c5ff-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.415 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.415 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.416 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.417 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.418 DEBUG:boto:Method: GET 2022-09-02T05:58:05.418 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.419 DEBUG:boto:Data: 2022-09-02T05:58:05.420 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.421 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.422 DEBUG:boto:Params: {} 2022-09-02T05:58:05.423 DEBUG:boto:Token: None 2022-09-02T05:58:05.423 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.424 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cacdb334e22d4cd5-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.429 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.430 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.431 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.432 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.433 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.434 DEBUG:boto:Method: GET 2022-09-02T05:58:05.435 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.436 DEBUG:boto:Data: 2022-09-02T05:58:05.437 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.439 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.440 DEBUG:boto:Params: {} 2022-09-02T05:58:05.442 DEBUG:boto:Token: None 2022-09-02T05:58:05.443 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000832ac983cf7edc98-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.451 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.453 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.453 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.454 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.455 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:05.456 DEBUG:boto:Data: 2022-09-02T05:58:05.457 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.458 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.459 DEBUG:boto:Params: {} 2022-09-02T05:58:05.459 DEBUG:boto:Token: None 2022-09-02T05:58:05.460 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95 2022-09-02T05:58:05.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rbogbqjkzi8pti/IOe/Kcqdc+F0= 2022-09-02T05:58:05.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rbogbqjkzi8pti/IOe/Kcqdc+F0=', 'Content-Length': '0'} 2022-09-02T05:58:05.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a47a42309b38a96b-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.465 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.465 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.466 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.466 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:05.467 DEBUG:boto:Data: 2022-09-02T05:58:05.467 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.469 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.469 DEBUG:boto:Params: {} 2022-09-02T05:58:05.470 DEBUG:boto:Token: None 2022-09-02T05:58:05.470 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95 2022-09-02T05:58:05.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rbogbqjkzi8pti/IOe/Kcqdc+F0= 2022-09-02T05:58:05.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rbogbqjkzi8pti/IOe/Kcqdc+F0=', 'Content-Length': '0'} 2022-09-02T05:58:05.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e3b12ca37e55c8-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:58:05.475 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.476 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.476 DEBUG:boto:Method: GET 2022-09-02T05:58:05.477 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:05.478 DEBUG:boto:Data: 2022-09-02T05:58:05.478 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.479 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.480 DEBUG:boto:Params: {} 2022-09-02T05:58:05.480 DEBUG:boto:Token: None 2022-09-02T05:58:05.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95 2022-09-02T05:58:05.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5iBLtoWFIbupajyZKOxAQc6K7d0= 2022-09-02T05:58:05.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5iBLtoWFIbupajyZKOxAQc6K7d0=', 'Content-Length': '0'} 2022-09-02T05:58:05.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000becf1a8ca276a4ab-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.485 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:05.486 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:05.486 DEBUG:boto:Method: GET 2022-09-02T05:58:05.487 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:05.488 DEBUG:boto:Data: 2022-09-02T05:58:05.488 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.489 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.490 DEBUG:boto:Params: {} 2022-09-02T05:58:05.490 DEBUG:boto:Token: None 2022-09-02T05:58:05.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj95 2022-09-02T05:58:05.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5iBLtoWFIbupajyZKOxAQc6K7d0= 2022-09-02T05:58:05.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5iBLtoWFIbupajyZKOxAQc6K7d0=', 'Content-Length': '0'} 2022-09-02T05:58:05.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8ce9dae66979ec-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.538 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.539 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.540 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.541 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.542 DEBUG:boto:Method: GET 2022-09-02T05:58:05.543 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.545 DEBUG:boto:Data: 2022-09-02T05:58:05.546 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.547 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.547 DEBUG:boto:Params: {} 2022-09-02T05:58:05.548 DEBUG:boto:Token: None 2022-09-02T05:58:05.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006714cfe572683ad1-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.554 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.556 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.557 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.558 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.559 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.560 DEBUG:boto:Method: GET 2022-09-02T05:58:05.561 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:05.562 DEBUG:boto:Data: 2022-09-02T05:58:05.563 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:05.565 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.566 DEBUG:boto:Params: {} 2022-09-02T05:58:05.568 DEBUG:boto:Token: None 2022-09-02T05:58:05.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637f77d77d2d9918-0063119b6d-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.577 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:58:05.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:58:05.579 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.580 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.580 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:05.581 DEBUG:boto:Data: 2022-09-02T05:58:05.581 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.582 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.583 DEBUG:boto:Params: {} 2022-09-02T05:58:05.584 DEBUG:boto:Token: None 2022-09-02T05:58:05.584 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AZxGeRD9Ofk6wS59h4G789ZU040= 2022-09-02T05:58:05.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AZxGeRD9Ofk6wS59h4G789ZU040=', 'Content-Length': '0'} 2022-09-02T05:58:05.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000876c632a9c15c03c-0063119b6d-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.589 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.589 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.590 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.590 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:05.591 DEBUG:boto:Data: 2022-09-02T05:58:05.591 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.592 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.592 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.593 DEBUG:boto:Params: {} 2022-09-02T05:58:05.594 DEBUG:boto:Token: None 2022-09-02T05:58:05.594 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.595 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AZxGeRD9Ofk6wS59h4G789ZU040= 2022-09-02T05:58:05.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AZxGeRD9Ofk6wS59h4G789ZU040=', 'Content-Length': '0'} 2022-09-02T05:58:05.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032316db64d6ed109-0063119b6d-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:58:05.600 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.600 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.601 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.601 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.602 DEBUG:boto:Method: GET 2022-09-02T05:58:05.603 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.604 DEBUG:boto:Data: 2022-09-02T05:58:05.604 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.605 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.606 DEBUG:boto:Params: {} 2022-09-02T05:58:05.606 DEBUG:boto:Token: None 2022-09-02T05:58:05.607 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/?versions 2022-09-02T05:58:05.608 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI= 2022-09-02T05:58:05.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI=', 'Content-Length': '0'} 2022-09-02T05:58:05.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e7c02ad25379d83-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.612 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:58:05.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:58:05.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:58:05.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:58:05.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:58:05.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:58:05.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:58:05.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:58:05.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:58:05.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:58:05.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:58:05.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:58:05.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:58:05.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:58:05.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:58:05.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:58:05.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:58:05.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:58:05.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:58:05.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:58:05.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:58:05.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:58:05.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:58:05.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:58:05.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:58:05.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:58:05.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:58:05.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:58:05.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:58:05.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:58:05.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:58:05.636 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.637 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.638 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.638 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.639 DEBUG:boto:Method: GET 2022-09-02T05:58:05.639 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.640 DEBUG:boto:Data: 2022-09-02T05:58:05.640 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.641 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.642 DEBUG:boto:Params: {} 2022-09-02T05:58:05.642 DEBUG:boto:Token: None 2022-09-02T05:58:05.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/?versions 2022-09-02T05:58:05.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI= 2022-09-02T05:58:05.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI=', 'Content-Length': '0'} 2022-09-02T05:58:05.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb521b07a994bc3d-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.648 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:58:05.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:58:05.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:58:05.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:58:05.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:58:05.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:58:05.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:58:05.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:58:05.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:58:05.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:58:05.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:58:05.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:58:05.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:58:05.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:58:05.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:58:05.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:58:05.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:58:05.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:58:05.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:58:05.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:58:05.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:58:05.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:58:05.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:58:05.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:58:05.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:58:05.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:58:05.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:58:05.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:58:05.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:58:05.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:58:05.665 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.665 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.666 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.666 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.667 DEBUG:boto:Method: GET 2022-09-02T05:58:05.668 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.668 DEBUG:boto:Data: 2022-09-02T05:58:05.669 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.670 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.670 DEBUG:boto:Params: {} 2022-09-02T05:58:05.671 DEBUG:boto:Token: None 2022-09-02T05:58:05.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/?versions 2022-09-02T05:58:05.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI= 2022-09-02T05:58:05.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI=', 'Content-Length': '0'} 2022-09-02T05:58:05.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e87d82396dfa9d8-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.676 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.678 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.679 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.679 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.680 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.680 DEBUG:boto:Method: GET 2022-09-02T05:58:05.681 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:58:05.681 DEBUG:boto:Data: 2022-09-02T05:58:05.682 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.683 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.683 DEBUG:boto:Params: {} 2022-09-02T05:58:05.684 DEBUG:boto:Token: None 2022-09-02T05:58:05.685 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/?versions 2022-09-02T05:58:05.685 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI= 2022-09-02T05:58:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wqnZ96+48Z0s5vocFTnKlkyb4hI=', 'Content-Length': '0'} 2022-09-02T05:58:05.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a4b84d71ed0c48b-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.690 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:58:05.693 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.693 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.694 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.694 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.695 DEBUG:boto:Method: GET 2022-09-02T05:58:05.695 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.696 DEBUG:boto:Data: 2022-09-02T05:58:05.696 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.697 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.698 DEBUG:boto:Params: {} 2022-09-02T05:58:05.699 DEBUG:boto:Token: None 2022-09-02T05:58:05.699 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.700 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ= 2022-09-02T05:58:05.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ=', 'Content-Length': '0'} 2022-09-02T05:58:05.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d3e6153c010206-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.746 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.747 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.748 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.748 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.749 DEBUG:boto:Method: GET 2022-09-02T05:58:05.749 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.750 DEBUG:boto:Data: 2022-09-02T05:58:05.750 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.752 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.752 DEBUG:boto:Params: {} 2022-09-02T05:58:05.753 DEBUG:boto:Token: None 2022-09-02T05:58:05.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ= 2022-09-02T05:58:05.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ=', 'Content-Length': '0'} 2022-09-02T05:58:05.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a85a8e7d01729b-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.758 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.758 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.759 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.759 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.760 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.760 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.761 DEBUG:boto:Data: 2022-09-02T05:58:05.761 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.763 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.763 DEBUG:boto:Params: {} 2022-09-02T05:58:05.764 DEBUG:boto:Token: None 2022-09-02T05:58:05.764 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ywb7WD6rfExA6/sH7cM8wtcPJ+4= 2022-09-02T05:58:05.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ywb7WD6rfExA6/sH7cM8wtcPJ+4=', 'Content-Length': '0'} 2022-09-02T05:58:05.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838f72226ee0b650-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.768 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.769 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.770 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.770 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.771 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.771 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.772 DEBUG:boto:Data: 2022-09-02T05:58:05.772 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.773 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.774 DEBUG:boto:Params: {} 2022-09-02T05:58:05.774 DEBUG:boto:Token: None 2022-09-02T05:58:05.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ywb7WD6rfExA6/sH7cM8wtcPJ+4= 2022-09-02T05:58:05.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ywb7WD6rfExA6/sH7cM8wtcPJ+4=', 'Content-Length': '0'} 2022-09-02T05:58:05.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097341c395f83fbc2-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:58:05.780 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.781 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.781 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.782 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.782 DEBUG:boto:Method: GET 2022-09-02T05:58:05.783 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.784 DEBUG:boto:Data: 2022-09-02T05:58:05.784 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.785 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.786 DEBUG:boto:Params: {} 2022-09-02T05:58:05.786 DEBUG:boto:Token: None 2022-09-02T05:58:05.787 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ= 2022-09-02T05:58:05.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ=', 'Content-Length': '0'} 2022-09-02T05:58:05.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da44dd48552780d5-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.792 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.792 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.793 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.793 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.794 DEBUG:boto:Method: GET 2022-09-02T05:58:05.794 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.795 DEBUG:boto:Data: 2022-09-02T05:58:05.795 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.796 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.796 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.797 DEBUG:boto:Params: {} 2022-09-02T05:58:05.798 DEBUG:boto:Token: None 2022-09-02T05:58:05.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:58:05.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ= 2022-09-02T05:58:05.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d0uao6O42v2ISUSh6UU4ydFxNHQ=', 'Content-Length': '0'} 2022-09-02T05:58:05.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0e5aeb88ae734b-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.803 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.804 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.804 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.805 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.805 DEBUG:boto:Method: GET 2022-09-02T05:58:05.806 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.807 DEBUG:boto:Data: 2022-09-02T05:58:05.807 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.808 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.809 DEBUG:boto:Params: {} 2022-09-02T05:58:05.809 DEBUG:boto:Token: None 2022-09-02T05:58:05.810 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f08218541811cd1d-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.816 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.817 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.818 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.818 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.819 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.819 DEBUG:boto:Method: GET 2022-09-02T05:58:05.821 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.822 DEBUG:boto:Data: 2022-09-02T05:58:05.823 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.824 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.824 DEBUG:boto:Params: {} 2022-09-02T05:58:05.825 DEBUG:boto:Token: None 2022-09-02T05:58:05.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cfa8873543f4dbb-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.874 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:05.875 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.876 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.876 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.877 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:58:05.878 DEBUG:boto:Data: 2022-09-02T05:58:05.878 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.879 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.880 DEBUG:boto:Params: {} 2022-09-02T05:58:05.881 DEBUG:boto:Token: None 2022-09-02T05:58:05.882 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11 2022-09-02T05:58:05.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7D0KKPSiiAy7/3zkHTxCCaUgftM= 2022-09-02T05:58:05.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7D0KKPSiiAy7/3zkHTxCCaUgftM=', 'Content-Length': '0'} 2022-09-02T05:58:05.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000373e3f6fd2756256-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.887 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.888 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.889 DEBUG:boto:Method: HEAD 2022-09-02T05:58:05.891 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:58:05.892 DEBUG:boto:Data: 2022-09-02T05:58:05.893 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.894 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.895 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.896 DEBUG:boto:Params: {} 2022-09-02T05:58:05.897 DEBUG:boto:Token: None 2022-09-02T05:58:05.898 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11 2022-09-02T05:58:05.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7D0KKPSiiAy7/3zkHTxCCaUgftM= 2022-09-02T05:58:05.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7D0KKPSiiAy7/3zkHTxCCaUgftM=', 'Content-Length': '0'} 2022-09-02T05:58:05.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c44d5e005724b9-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:58:05.905 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.906 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.908 DEBUG:boto:Method: GET 2022-09-02T05:58:05.909 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:58:05.910 DEBUG:boto:Data: 2022-09-02T05:58:05.911 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.913 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.914 DEBUG:boto:Params: {} 2022-09-02T05:58:05.916 DEBUG:boto:Token: None 2022-09-02T05:58:05.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11 2022-09-02T05:58:05.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sJ3ezP9GW57t0YRZApx49nSahCc= 2022-09-02T05:58:05.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sJ3ezP9GW57t0YRZApx49nSahCc=', 'Content-Length': '0'} 2022-09-02T05:58:05.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b964dc48345218-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.924 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:58:05.925 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:58:05.926 DEBUG:boto:Method: GET 2022-09-02T05:58:05.927 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:58:05.928 DEBUG:boto:Data: 2022-09-02T05:58:05.929 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:05.931 DEBUG:boto:Port: 8000 2022-09-02T05:58:05.932 DEBUG:boto:Params: {} 2022-09-02T05:58:05.933 DEBUG:boto:Token: None 2022-09-02T05:58:05.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/obj11 2022-09-02T05:58:05.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sJ3ezP9GW57t0YRZApx49nSahCc= 2022-09-02T05:58:05.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sJ3ezP9GW57t0YRZApx49nSahCc=', 'Content-Length': '0'} 2022-09-02T05:58:05.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b31316a08951fd-0063119b6d-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:05.939 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:05.940 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:05.940 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.941 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.941 DEBUG:boto:Method: GET 2022-09-02T05:58:05.942 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:58:05.942 DEBUG:boto:Data: 2022-09-02T05:58:05.943 DEBUG:boto:Headers: {} 2022-09-02T05:58:05.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:05.944 DEBUG:boto:Port: 8001 2022-09-02T05:58:05.944 DEBUG:boto:Params: {} 2022-09-02T05:58:05.945 DEBUG:boto:Token: None 2022-09-02T05:58:05.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:05 GMT /xljbkf-74/ 2022-09-02T05:58:05.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU= 2022-09-02T05:58:05.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:05 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e3jK5vlS4PMzWl7NwzCVw6r59oU=', 'Content-Length': '0'} 2022-09-02T05:58:05.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103e8b11b33e79de-0063119b6d-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:05 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.006 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.007 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.008 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.008 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:06.009 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:58:06.010 DEBUG:boto:Method: GET 2022-09-02T05:58:06.010 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:58:06.011 DEBUG:boto:Data: 2022-09-02T05:58:06.011 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.012 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.012 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.013 DEBUG:boto:Params: {} 2022-09-02T05:58:06.014 DEBUG:boto:Token: None 2022-09-02T05:58:06.014 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.015 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071037dea74732173-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.021 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:58:06.023 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.023 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.024 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.025 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.025 DEBUG:boto:Method: GET 2022-09-02T05:58:06.026 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.026 DEBUG:boto:Data: 2022-09-02T05:58:06.027 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.028 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.029 DEBUG:boto:Params: {} 2022-09-02T05:58:06.029 DEBUG:boto:Token: None 2022-09-02T05:58:06.030 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw= 2022-09-02T05:58:06.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw=', 'Content-Length': '0'} 2022-09-02T05:58:06.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003501f3ddd25bd1d7-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.034 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.035 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.036 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.036 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.037 DEBUG:boto:Method: GET 2022-09-02T05:58:06.038 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.038 DEBUG:boto:Data: 2022-09-02T05:58:06.039 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.039 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.040 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.040 DEBUG:boto:Params: {} 2022-09-02T05:58:06.041 DEBUG:boto:Token: None 2022-09-02T05:58:06.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.042 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw= 2022-09-02T05:58:06.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw=', 'Content-Length': '0'} 2022-09-02T05:58:06.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000393c1e0acec10e2f-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.046 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.047 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.047 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.048 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.048 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.049 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.049 DEBUG:boto:Data: 2022-09-02T05:58:06.050 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.051 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.052 DEBUG:boto:Params: {} 2022-09-02T05:58:06.052 DEBUG:boto:Token: None 2022-09-02T05:58:06.053 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EkzwTec6MJwMnZHYcbKWOf4RiVA= 2022-09-02T05:58:06.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EkzwTec6MJwMnZHYcbKWOf4RiVA=', 'Content-Length': '0'} 2022-09-02T05:58:06.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5b9476af744bbc-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.057 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.058 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.058 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.059 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.059 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.060 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.060 DEBUG:boto:Data: 2022-09-02T05:58:06.061 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.062 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.062 DEBUG:boto:Params: {} 2022-09-02T05:58:06.063 DEBUG:boto:Token: None 2022-09-02T05:58:06.064 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EkzwTec6MJwMnZHYcbKWOf4RiVA= 2022-09-02T05:58:06.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EkzwTec6MJwMnZHYcbKWOf4RiVA=', 'Content-Length': '0'} 2022-09-02T05:58:06.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004468d492eb15b69c-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:58:06.068 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.069 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.069 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.070 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.070 DEBUG:boto:Method: GET 2022-09-02T05:58:06.071 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.072 DEBUG:boto:Data: 2022-09-02T05:58:06.072 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.073 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.074 DEBUG:boto:Params: {} 2022-09-02T05:58:06.074 DEBUG:boto:Token: None 2022-09-02T05:58:06.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw= 2022-09-02T05:58:06.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw=', 'Content-Length': '0'} 2022-09-02T05:58:06.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dccbb2e48f70582a-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.079 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.080 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.080 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.081 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.081 DEBUG:boto:Method: GET 2022-09-02T05:58:06.082 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.083 DEBUG:boto:Data: 2022-09-02T05:58:06.083 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.084 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.084 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.085 DEBUG:boto:Params: {} 2022-09-02T05:58:06.085 DEBUG:boto:Token: None 2022-09-02T05:58:06.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:58:06.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw= 2022-09-02T05:58:06.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m9hgzyZjHeYcfau7ZUK8OqySxAw=', 'Content-Length': '0'} 2022-09-02T05:58:06.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de149ab6bd47b8f-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.090 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.091 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.091 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.092 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.093 DEBUG:boto:Method: GET 2022-09-02T05:58:06.093 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.094 DEBUG:boto:Data: 2022-09-02T05:58:06.094 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.096 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.096 DEBUG:boto:Params: {} 2022-09-02T05:58:06.097 DEBUG:boto:Token: None 2022-09-02T05:58:06.098 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049755ad0c96643b6-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.103 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.104 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.104 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.105 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.105 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.106 DEBUG:boto:Method: GET 2022-09-02T05:58:06.107 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.107 DEBUG:boto:Data: 2022-09-02T05:58:06.108 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.109 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.110 DEBUG:boto:Params: {} 2022-09-02T05:58:06.110 DEBUG:boto:Token: None 2022-09-02T05:58:06.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.112 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a5144e031c768f-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.116 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.117 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.118 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.118 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.119 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:58:06.119 DEBUG:boto:Data: 2022-09-02T05:58:06.120 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.121 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.121 DEBUG:boto:Params: {} 2022-09-02T05:58:06.122 DEBUG:boto:Token: None 2022-09-02T05:58:06.123 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115 2022-09-02T05:58:06.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r7MKGJJddW+bWzCUmFJgeW5a7D8= 2022-09-02T05:58:06.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r7MKGJJddW+bWzCUmFJgeW5a7D8=', 'Content-Length': '0'} 2022-09-02T05:58:06.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108ee2a06745f687-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.127 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.127 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.128 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.128 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:58:06.129 DEBUG:boto:Data: 2022-09-02T05:58:06.130 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.130 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.131 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.132 DEBUG:boto:Params: {} 2022-09-02T05:58:06.132 DEBUG:boto:Token: None 2022-09-02T05:58:06.133 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115 2022-09-02T05:58:06.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r7MKGJJddW+bWzCUmFJgeW5a7D8= 2022-09-02T05:58:06.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r7MKGJJddW+bWzCUmFJgeW5a7D8=', 'Content-Length': '0'} 2022-09-02T05:58:06.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ab0f3eea4062a2-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:58:06.138 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.139 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.140 DEBUG:boto:Method: GET 2022-09-02T05:58:06.140 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:58:06.141 DEBUG:boto:Data: 2022-09-02T05:58:06.141 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.142 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.143 DEBUG:boto:Params: {} 2022-09-02T05:58:06.143 DEBUG:boto:Token: None 2022-09-02T05:58:06.144 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115 2022-09-02T05:58:06.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DixNJ4igxRRWhu0bYUGFKW3XkX0= 2022-09-02T05:58:06.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DixNJ4igxRRWhu0bYUGFKW3XkX0=', 'Content-Length': '0'} 2022-09-02T05:58:06.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d85820dba3ca93-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.190 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:58:06.191 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:58:06.191 DEBUG:boto:Method: GET 2022-09-02T05:58:06.192 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:58:06.192 DEBUG:boto:Data: 2022-09-02T05:58:06.193 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.194 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.194 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.195 DEBUG:boto:Params: {} 2022-09-02T05:58:06.195 DEBUG:boto:Token: None 2022-09-02T05:58:06.196 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj115 2022-09-02T05:58:06.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DixNJ4igxRRWhu0bYUGFKW3XkX0= 2022-09-02T05:58:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DixNJ4igxRRWhu0bYUGFKW3XkX0=', 'Content-Length': '0'} 2022-09-02T05:58:06.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000124890686ae3233-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.200 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.201 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.202 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.202 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.203 DEBUG:boto:Method: GET 2022-09-02T05:58:06.204 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.204 DEBUG:boto:Data: 2022-09-02T05:58:06.205 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.206 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.207 DEBUG:boto:Params: {} 2022-09-02T05:58:06.207 DEBUG:boto:Token: None 2022-09-02T05:58:06.208 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0955d0eb9d8351-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.213 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.214 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.215 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.216 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.216 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.217 DEBUG:boto:Method: GET 2022-09-02T05:58:06.217 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:58:06.218 DEBUG:boto:Data: 2022-09-02T05:58:06.218 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.219 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.220 DEBUG:boto:Params: {} 2022-09-02T05:58:06.221 DEBUG:boto:Token: None 2022-09-02T05:58:06.221 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eaa41825cbf151a-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.226 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:58:06.228 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.228 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.229 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.229 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.230 DEBUG:boto:Method: GET 2022-09-02T05:58:06.231 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.231 DEBUG:boto:Data: 2022-09-02T05:58:06.232 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.233 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.234 DEBUG:boto:Params: {} 2022-09-02T05:58:06.234 DEBUG:boto:Token: None 2022-09-02T05:58:06.235 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk= 2022-09-02T05:58:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk=', 'Content-Length': '0'} 2022-09-02T05:58:06.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062326964ae829184-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.241 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.242 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.242 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.243 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.243 DEBUG:boto:Method: GET 2022-09-02T05:58:06.244 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.244 DEBUG:boto:Data: 2022-09-02T05:58:06.245 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.245 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.246 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.246 DEBUG:boto:Params: {} 2022-09-02T05:58:06.247 DEBUG:boto:Token: None 2022-09-02T05:58:06.248 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.248 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk= 2022-09-02T05:58:06.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk=', 'Content-Length': '0'} 2022-09-02T05:58:06.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db6d385ad16c4cde-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.252 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.253 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.253 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.254 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.254 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.255 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.255 DEBUG:boto:Data: 2022-09-02T05:58:06.256 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.257 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.257 DEBUG:boto:Params: {} 2022-09-02T05:58:06.258 DEBUG:boto:Token: None 2022-09-02T05:58:06.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ryO1Bg29ziLzNVa4QOiNNvxLHjA= 2022-09-02T05:58:06.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ryO1Bg29ziLzNVa4QOiNNvxLHjA=', 'Content-Length': '0'} 2022-09-02T05:58:06.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf6c66a32e05531-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.262 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.263 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.263 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.264 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.264 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.265 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.265 DEBUG:boto:Data: 2022-09-02T05:58:06.266 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.267 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.267 DEBUG:boto:Params: {} 2022-09-02T05:58:06.268 DEBUG:boto:Token: None 2022-09-02T05:58:06.269 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.269 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ryO1Bg29ziLzNVa4QOiNNvxLHjA= 2022-09-02T05:58:06.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ryO1Bg29ziLzNVa4QOiNNvxLHjA=', 'Content-Length': '0'} 2022-09-02T05:58:06.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40de3bf27925eae-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:58:06.274 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.274 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.275 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.275 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.276 DEBUG:boto:Method: GET 2022-09-02T05:58:06.276 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.277 DEBUG:boto:Data: 2022-09-02T05:58:06.277 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.278 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.279 DEBUG:boto:Params: {} 2022-09-02T05:58:06.279 DEBUG:boto:Token: None 2022-09-02T05:58:06.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk= 2022-09-02T05:58:06.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk=', 'Content-Length': '0'} 2022-09-02T05:58:06.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793bf50239b6ded2-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.285 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.286 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.287 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.287 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.288 DEBUG:boto:Method: GET 2022-09-02T05:58:06.288 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.289 DEBUG:boto:Data: 2022-09-02T05:58:06.290 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.290 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.291 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.291 DEBUG:boto:Params: {} 2022-09-02T05:58:06.292 DEBUG:boto:Token: None 2022-09-02T05:58:06.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:58:06.293 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk= 2022-09-02T05:58:06.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GHlph14QhfBuYxqf42vRnJbE7dk=', 'Content-Length': '0'} 2022-09-02T05:58:06.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b78cdcf6257c62-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.297 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.297 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.298 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.298 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.299 DEBUG:boto:Method: GET 2022-09-02T05:58:06.299 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.300 DEBUG:boto:Data: 2022-09-02T05:58:06.301 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.302 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.302 DEBUG:boto:Params: {} 2022-09-02T05:58:06.303 DEBUG:boto:Token: None 2022-09-02T05:58:06.303 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005144fb04ca657867-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.308 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.309 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.310 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.310 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.311 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.311 DEBUG:boto:Method: GET 2022-09-02T05:58:06.312 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.312 DEBUG:boto:Data: 2022-09-02T05:58:06.313 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.314 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.314 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.315 DEBUG:boto:Params: {} 2022-09-02T05:58:06.316 DEBUG:boto:Token: None 2022-09-02T05:58:06.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae4a2879d428e13b-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.321 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.322 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.323 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.323 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.324 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:58:06.324 DEBUG:boto:Data: 2022-09-02T05:58:06.325 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.326 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.326 DEBUG:boto:Params: {} 2022-09-02T05:58:06.327 DEBUG:boto:Token: None 2022-09-02T05:58:06.327 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116 2022-09-02T05:58:06.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0fCs3ODBiAgWuweyc6VPxPDMi60= 2022-09-02T05:58:06.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0fCs3ODBiAgWuweyc6VPxPDMi60=', 'Content-Length': '0'} 2022-09-02T05:58:06.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005750740668a23d89-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.332 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.332 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.333 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.333 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:58:06.334 DEBUG:boto:Data: 2022-09-02T05:58:06.334 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.335 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.335 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.336 DEBUG:boto:Params: {} 2022-09-02T05:58:06.337 DEBUG:boto:Token: None 2022-09-02T05:58:06.337 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116 2022-09-02T05:58:06.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0fCs3ODBiAgWuweyc6VPxPDMi60= 2022-09-02T05:58:06.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0fCs3ODBiAgWuweyc6VPxPDMi60=', 'Content-Length': '0'} 2022-09-02T05:58:06.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7860c3a0503efca-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:58:06.342 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.342 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.343 DEBUG:boto:Method: GET 2022-09-02T05:58:06.343 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:58:06.344 DEBUG:boto:Data: 2022-09-02T05:58:06.345 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.346 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.346 DEBUG:boto:Params: {} 2022-09-02T05:58:06.347 DEBUG:boto:Token: None 2022-09-02T05:58:06.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116 2022-09-02T05:58:06.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jaAhKyyAoHSLslHRV14azsMd3iw= 2022-09-02T05:58:06.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jaAhKyyAoHSLslHRV14azsMd3iw=', 'Content-Length': '0'} 2022-09-02T05:58:06.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d02a1ebfbc6d1f-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.352 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:58:06.352 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:58:06.353 DEBUG:boto:Method: GET 2022-09-02T05:58:06.353 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:58:06.354 DEBUG:boto:Data: 2022-09-02T05:58:06.363 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.364 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.365 DEBUG:boto:Params: {} 2022-09-02T05:58:06.365 DEBUG:boto:Token: None 2022-09-02T05:58:06.366 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj116 2022-09-02T05:58:06.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jaAhKyyAoHSLslHRV14azsMd3iw= 2022-09-02T05:58:06.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jaAhKyyAoHSLslHRV14azsMd3iw=', 'Content-Length': '0'} 2022-09-02T05:58:06.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517e67e7bdb71b40-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.371 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.372 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.372 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.373 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.373 DEBUG:boto:Method: GET 2022-09-02T05:58:06.374 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.374 DEBUG:boto:Data: 2022-09-02T05:58:06.375 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.376 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.376 DEBUG:boto:Params: {} 2022-09-02T05:58:06.377 DEBUG:boto:Token: None 2022-09-02T05:58:06.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d7fd01d8e83217-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.430 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.432 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.433 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.433 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.434 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.435 DEBUG:boto:Method: GET 2022-09-02T05:58:06.435 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:58:06.436 DEBUG:boto:Data: 2022-09-02T05:58:06.436 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.437 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.438 DEBUG:boto:Params: {} 2022-09-02T05:58:06.439 DEBUG:boto:Token: None 2022-09-02T05:58:06.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.440 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e427d7dad6fcf0d2-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.445 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:58:06.446 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.447 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.448 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.448 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.449 DEBUG:boto:Method: GET 2022-09-02T05:58:06.449 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.450 DEBUG:boto:Data: 2022-09-02T05:58:06.450 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.451 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.452 DEBUG:boto:Params: {} 2022-09-02T05:58:06.452 DEBUG:boto:Token: None 2022-09-02T05:58:06.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8= 2022-09-02T05:58:06.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8=', 'Content-Length': '0'} 2022-09-02T05:58:06.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8fce4776240f71-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.457 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.458 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.458 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.459 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.460 DEBUG:boto:Method: GET 2022-09-02T05:58:06.460 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.461 DEBUG:boto:Data: 2022-09-02T05:58:06.461 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.462 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.463 DEBUG:boto:Params: {} 2022-09-02T05:58:06.463 DEBUG:boto:Token: None 2022-09-02T05:58:06.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8= 2022-09-02T05:58:06.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8=', 'Content-Length': '0'} 2022-09-02T05:58:06.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf57bc6534e11dc-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.468 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.469 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.469 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.470 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.470 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.471 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.471 DEBUG:boto:Data: 2022-09-02T05:58:06.472 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.473 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.473 DEBUG:boto:Params: {} 2022-09-02T05:58:06.474 DEBUG:boto:Token: None 2022-09-02T05:58:06.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAZ2Z620zlP89svKaLsRdsZlq4= 2022-09-02T05:58:06.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAZ2Z620zlP89svKaLsRdsZlq4=', 'Content-Length': '0'} 2022-09-02T05:58:06.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4bb057451dca5b0-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.479 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.480 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.481 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.481 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.482 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.482 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.483 DEBUG:boto:Data: 2022-09-02T05:58:06.483 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.484 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.485 DEBUG:boto:Params: {} 2022-09-02T05:58:06.486 DEBUG:boto:Token: None 2022-09-02T05:58:06.487 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OHAZ2Z620zlP89svKaLsRdsZlq4= 2022-09-02T05:58:06.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OHAZ2Z620zlP89svKaLsRdsZlq4=', 'Content-Length': '0'} 2022-09-02T05:58:06.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be118879ea0c8b0-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:58:06.492 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.492 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.493 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.494 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.495 DEBUG:boto:Method: GET 2022-09-02T05:58:06.495 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.496 DEBUG:boto:Data: 2022-09-02T05:58:06.496 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.497 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.498 DEBUG:boto:Params: {} 2022-09-02T05:58:06.499 DEBUG:boto:Token: None 2022-09-02T05:58:06.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.500 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8= 2022-09-02T05:58:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8=', 'Content-Length': '0'} 2022-09-02T05:58:06.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f6b4d317daa35b-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.504 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.504 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.505 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.505 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.506 DEBUG:boto:Method: GET 2022-09-02T05:58:06.507 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.507 DEBUG:boto:Data: 2022-09-02T05:58:06.508 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.508 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.509 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.509 DEBUG:boto:Params: {} 2022-09-02T05:58:06.510 DEBUG:boto:Token: None 2022-09-02T05:58:06.510 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:58:06.511 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8= 2022-09-02T05:58:06.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wLQttWmSuFjDabXd8k194E7YBd8=', 'Content-Length': '0'} 2022-09-02T05:58:06.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456b54d77d721a98-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.515 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.515 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.516 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.516 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.517 DEBUG:boto:Method: GET 2022-09-02T05:58:06.518 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.518 DEBUG:boto:Data: 2022-09-02T05:58:06.519 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.520 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.520 DEBUG:boto:Params: {} 2022-09-02T05:58:06.521 DEBUG:boto:Token: None 2022-09-02T05:58:06.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227cf0f2e76a6444-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.570 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.571 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.572 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.572 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.573 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.573 DEBUG:boto:Method: GET 2022-09-02T05:58:06.574 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.574 DEBUG:boto:Data: 2022-09-02T05:58:06.575 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.576 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.577 DEBUG:boto:Params: {} 2022-09-02T05:58:06.577 DEBUG:boto:Token: None 2022-09-02T05:58:06.578 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.579 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b1dce7997467abb-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.583 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.583 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.584 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.585 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.585 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:58:06.586 DEBUG:boto:Data: 2022-09-02T05:58:06.586 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.587 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.588 DEBUG:boto:Params: {} 2022-09-02T05:58:06.588 DEBUG:boto:Token: None 2022-09-02T05:58:06.589 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124 2022-09-02T05:58:06.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OKfLyt5tPe9tNzVIHKh7ntOJV0E= 2022-09-02T05:58:06.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OKfLyt5tPe9tNzVIHKh7ntOJV0E=', 'Content-Length': '0'} 2022-09-02T05:58:06.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4202b7b879090be-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.593 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.593 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.594 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.594 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:58:06.595 DEBUG:boto:Data: 2022-09-02T05:58:06.595 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.596 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.596 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.597 DEBUG:boto:Params: {} 2022-09-02T05:58:06.597 DEBUG:boto:Token: None 2022-09-02T05:58:06.598 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124 2022-09-02T05:58:06.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OKfLyt5tPe9tNzVIHKh7ntOJV0E= 2022-09-02T05:58:06.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OKfLyt5tPe9tNzVIHKh7ntOJV0E=', 'Content-Length': '0'} 2022-09-02T05:58:06.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003de2c5ba6ebc5a2f-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:58:06.603 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.603 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.604 DEBUG:boto:Method: GET 2022-09-02T05:58:06.604 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:58:06.605 DEBUG:boto:Data: 2022-09-02T05:58:06.606 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.607 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.607 DEBUG:boto:Params: {} 2022-09-02T05:58:06.608 DEBUG:boto:Token: None 2022-09-02T05:58:06.609 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124 2022-09-02T05:58:06.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ds2TzVLmXGoVlk2684SfAgz/Kqk= 2022-09-02T05:58:06.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ds2TzVLmXGoVlk2684SfAgz/Kqk=', 'Content-Length': '0'} 2022-09-02T05:58:06.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065c9df1bb98847b-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.613 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:58:06.613 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:58:06.614 DEBUG:boto:Method: GET 2022-09-02T05:58:06.614 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:58:06.615 DEBUG:boto:Data: 2022-09-02T05:58:06.615 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.616 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.617 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.617 DEBUG:boto:Params: {} 2022-09-02T05:58:06.618 DEBUG:boto:Token: None 2022-09-02T05:58:06.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj124 2022-09-02T05:58:06.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ds2TzVLmXGoVlk2684SfAgz/Kqk= 2022-09-02T05:58:06.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ds2TzVLmXGoVlk2684SfAgz/Kqk=', 'Content-Length': '0'} 2022-09-02T05:58:06.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2f9aa6f7e6e947-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.666 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.667 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.667 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.668 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.668 DEBUG:boto:Method: GET 2022-09-02T05:58:06.669 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.670 DEBUG:boto:Data: 2022-09-02T05:58:06.670 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.672 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.672 DEBUG:boto:Params: {} 2022-09-02T05:58:06.673 DEBUG:boto:Token: None 2022-09-02T05:58:06.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f7389a316d690cd-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.678 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.679 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.680 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.680 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.681 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.682 DEBUG:boto:Method: GET 2022-09-02T05:58:06.683 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:58:06.683 DEBUG:boto:Data: 2022-09-02T05:58:06.684 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.684 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.685 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.685 DEBUG:boto:Params: {} 2022-09-02T05:58:06.686 DEBUG:boto:Token: None 2022-09-02T05:58:06.686 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.687 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f8543ea0e4765bd-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.691 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:58:06.693 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.693 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.694 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.695 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.695 DEBUG:boto:Method: GET 2022-09-02T05:58:06.696 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.696 DEBUG:boto:Data: 2022-09-02T05:58:06.697 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.698 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.699 DEBUG:boto:Params: {} 2022-09-02T05:58:06.699 DEBUG:boto:Token: None 2022-09-02T05:58:06.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI= 2022-09-02T05:58:06.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI=', 'Content-Length': '0'} 2022-09-02T05:58:06.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d88f78ea071f62-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.704 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.705 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.705 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.706 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.706 DEBUG:boto:Method: GET 2022-09-02T05:58:06.707 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.707 DEBUG:boto:Data: 2022-09-02T05:58:06.708 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.709 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.710 DEBUG:boto:Params: {} 2022-09-02T05:58:06.710 DEBUG:boto:Token: None 2022-09-02T05:58:06.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI= 2022-09-02T05:58:06.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI=', 'Content-Length': '0'} 2022-09-02T05:58:06.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed67925c1fd9283e-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.715 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.716 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.716 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.717 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.718 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.718 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.719 DEBUG:boto:Data: 2022-09-02T05:58:06.720 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.721 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.721 DEBUG:boto:Params: {} 2022-09-02T05:58:06.722 DEBUG:boto:Token: None 2022-09-02T05:58:06.722 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p7EGYIevHhFH7wHZccsZ6f3FTTU= 2022-09-02T05:58:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p7EGYIevHhFH7wHZccsZ6f3FTTU=', 'Content-Length': '0'} 2022-09-02T05:58:06.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfce0eb4833f325-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.727 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.727 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.728 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.728 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.729 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.729 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.730 DEBUG:boto:Data: 2022-09-02T05:58:06.730 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.731 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.732 DEBUG:boto:Params: {} 2022-09-02T05:58:06.732 DEBUG:boto:Token: None 2022-09-02T05:58:06.733 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:p7EGYIevHhFH7wHZccsZ6f3FTTU= 2022-09-02T05:58:06.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:p7EGYIevHhFH7wHZccsZ6f3FTTU=', 'Content-Length': '0'} 2022-09-02T05:58:06.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb7a82a1c2462adb-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:58:06.738 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.739 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.739 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.740 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.740 DEBUG:boto:Method: GET 2022-09-02T05:58:06.741 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.741 DEBUG:boto:Data: 2022-09-02T05:58:06.742 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.743 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.744 DEBUG:boto:Params: {} 2022-09-02T05:58:06.745 DEBUG:boto:Token: None 2022-09-02T05:58:06.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI= 2022-09-02T05:58:06.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI=', 'Content-Length': '0'} 2022-09-02T05:58:06.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd2c297087fbff5f-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.750 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.750 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.751 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.752 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.752 DEBUG:boto:Method: GET 2022-09-02T05:58:06.753 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.753 DEBUG:boto:Data: 2022-09-02T05:58:06.754 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.755 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.755 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.756 DEBUG:boto:Params: {} 2022-09-02T05:58:06.757 DEBUG:boto:Token: None 2022-09-02T05:58:06.757 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:58:06.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI= 2022-09-02T05:58:06.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HxkoB6fbwi0v9Aw17EWJt/KV1pI=', 'Content-Length': '0'} 2022-09-02T05:58:06.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7c08cdc05d695e-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.809 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.810 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.811 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.812 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.814 DEBUG:boto:Method: GET 2022-09-02T05:58:06.815 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.816 DEBUG:boto:Data: 2022-09-02T05:58:06.817 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.818 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.818 DEBUG:boto:Params: {} 2022-09-02T05:58:06.819 DEBUG:boto:Token: None 2022-09-02T05:58:06.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009433bcb6d0889575-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.826 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.827 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.828 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.829 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.830 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.831 DEBUG:boto:Method: GET 2022-09-02T05:58:06.832 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.833 DEBUG:boto:Data: 2022-09-02T05:58:06.834 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.835 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.836 DEBUG:boto:Params: {} 2022-09-02T05:58:06.837 DEBUG:boto:Token: None 2022-09-02T05:58:06.838 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007db81e868ed9b139-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.843 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.844 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.845 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.845 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.846 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:58:06.846 DEBUG:boto:Data: 2022-09-02T05:58:06.847 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.848 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.848 DEBUG:boto:Params: {} 2022-09-02T05:58:06.849 DEBUG:boto:Token: None 2022-09-02T05:58:06.850 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145 2022-09-02T05:58:06.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z70yiDacyZM9LQxCAYZGNNgUlkQ= 2022-09-02T05:58:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z70yiDacyZM9LQxCAYZGNNgUlkQ=', 'Content-Length': '0'} 2022-09-02T05:58:06.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce8911b5d7b63c0-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.855 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.855 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.856 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.857 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:58:06.858 DEBUG:boto:Data: 2022-09-02T05:58:06.859 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.860 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.861 DEBUG:boto:Params: {} 2022-09-02T05:58:06.862 DEBUG:boto:Token: None 2022-09-02T05:58:06.863 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145 2022-09-02T05:58:06.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Z70yiDacyZM9LQxCAYZGNNgUlkQ= 2022-09-02T05:58:06.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Z70yiDacyZM9LQxCAYZGNNgUlkQ=', 'Content-Length': '0'} 2022-09-02T05:58:06.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003717551fdd4eb70e-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:58:06.868 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.868 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.869 DEBUG:boto:Method: GET 2022-09-02T05:58:06.869 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:58:06.870 DEBUG:boto:Data: 2022-09-02T05:58:06.870 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.871 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.872 DEBUG:boto:Params: {} 2022-09-02T05:58:06.873 DEBUG:boto:Token: None 2022-09-02T05:58:06.873 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145 2022-09-02T05:58:06.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wYKvqxmXbX+pOlA+CutxtHmva88= 2022-09-02T05:58:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wYKvqxmXbX+pOlA+CutxtHmva88=', 'Content-Length': '0'} 2022-09-02T05:58:06.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e16f9a41c635d4ce-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.879 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:58:06.880 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:58:06.882 DEBUG:boto:Method: GET 2022-09-02T05:58:06.883 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:58:06.884 DEBUG:boto:Data: 2022-09-02T05:58:06.884 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.886 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.887 DEBUG:boto:Params: {} 2022-09-02T05:58:06.887 DEBUG:boto:Token: None 2022-09-02T05:58:06.888 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj145 2022-09-02T05:58:06.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wYKvqxmXbX+pOlA+CutxtHmva88= 2022-09-02T05:58:06.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wYKvqxmXbX+pOlA+CutxtHmva88=', 'Content-Length': '0'} 2022-09-02T05:58:06.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4a765fb41cc826-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.934 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.935 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.936 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.936 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.937 DEBUG:boto:Method: GET 2022-09-02T05:58:06.937 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.938 DEBUG:boto:Data: 2022-09-02T05:58:06.938 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.940 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.940 DEBUG:boto:Params: {} 2022-09-02T05:58:06.941 DEBUG:boto:Token: None 2022-09-02T05:58:06.941 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043753b5dfd458ef6-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.946 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.947 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:06.948 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:06.948 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.949 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.949 DEBUG:boto:Method: GET 2022-09-02T05:58:06.950 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:58:06.951 DEBUG:boto:Data: 2022-09-02T05:58:06.951 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.952 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.952 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.953 DEBUG:boto:Params: {} 2022-09-02T05:58:06.953 DEBUG:boto:Token: None 2022-09-02T05:58:06.954 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/ 2022-09-02T05:58:06.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk= 2022-09-02T05:58:06.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxFlYG2Me78vObHnBYTOXQNjVGk=', 'Content-Length': '0'} 2022-09-02T05:58:06.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b9735cada54e69-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.959 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:06.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:58:06.960 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:06.961 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:06.961 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.962 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.962 DEBUG:boto:Method: GET 2022-09-02T05:58:06.963 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.963 DEBUG:boto:Data: 2022-09-02T05:58:06.964 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.965 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.965 DEBUG:boto:Params: {} 2022-09-02T05:58:06.966 DEBUG:boto:Token: None 2022-09-02T05:58:06.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sRZNnD5qyNr/oLgboCZoejuL6kg= 2022-09-02T05:58:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sRZNnD5qyNr/oLgboCZoejuL6kg=', 'Content-Length': '0'} 2022-09-02T05:58:06.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004273831f517e0eb7-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.971 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:06.972 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:06.972 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.973 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.973 DEBUG:boto:Method: GET 2022-09-02T05:58:06.974 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.974 DEBUG:boto:Data: 2022-09-02T05:58:06.975 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.976 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.977 DEBUG:boto:Params: {} 2022-09-02T05:58:06.977 DEBUG:boto:Token: None 2022-09-02T05:58:06.978 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sRZNnD5qyNr/oLgboCZoejuL6kg= 2022-09-02T05:58:06.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sRZNnD5qyNr/oLgboCZoejuL6kg=', 'Content-Length': '0'} 2022-09-02T05:58:06.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f312db27e99c9d07-0063119b6e-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.983 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:06.983 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:06.984 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.984 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.985 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.985 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.986 DEBUG:boto:Data: 2022-09-02T05:58:06.986 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:06.987 DEBUG:boto:Port: 8001 2022-09-02T05:58:06.988 DEBUG:boto:Params: {} 2022-09-02T05:58:06.988 DEBUG:boto:Token: None 2022-09-02T05:58:06.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1iN1QznyGQshOor7hhCqc2M243c= 2022-09-02T05:58:06.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1iN1QznyGQshOor7hhCqc2M243c=', 'Content-Length': '0'} 2022-09-02T05:58:06.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085d6b7048fba89a0-0063119b6e-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:06 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:06.993 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:06.994 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:06.994 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.995 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.995 DEBUG:boto:Method: HEAD 2022-09-02T05:58:06.996 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:06.996 DEBUG:boto:Data: 2022-09-02T05:58:06.997 DEBUG:boto:Headers: {} 2022-09-02T05:58:06.997 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:06.998 DEBUG:boto:Port: 8000 2022-09-02T05:58:06.998 DEBUG:boto:Params: {} 2022-09-02T05:58:06.999 DEBUG:boto:Token: None 2022-09-02T05:58:07.000 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:06 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1iN1QznyGQshOor7hhCqc2M243c= 2022-09-02T05:58:07.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:06 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1iN1QznyGQshOor7hhCqc2M243c=', 'Content-Length': '0'} 2022-09-02T05:58:07.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd54024cf77984c5-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:58:07.004 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:07.005 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:07.005 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.006 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.007 DEBUG:boto:Method: GET 2022-09-02T05:58:07.007 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.008 DEBUG:boto:Data: 2022-09-02T05:58:07.008 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.009 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.010 DEBUG:boto:Params: {} 2022-09-02T05:58:07.010 DEBUG:boto:Token: None 2022-09-02T05:58:07.011 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZCoKO1q5k5xlyh2CpRzmhWKpzE= 2022-09-02T05:58:07.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZCoKO1q5k5xlyh2CpRzmhWKpzE=', 'Content-Length': '0'} 2022-09-02T05:58:07.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c7d174e1f1d232-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.062 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:07.063 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:07.064 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.064 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.065 DEBUG:boto:Method: GET 2022-09-02T05:58:07.066 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.066 DEBUG:boto:Data: 2022-09-02T05:58:07.067 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.068 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.068 DEBUG:boto:Params: {} 2022-09-02T05:58:07.069 DEBUG:boto:Token: None 2022-09-02T05:58:07.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:58:07.071 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+ZCoKO1q5k5xlyh2CpRzmhWKpzE= 2022-09-02T05:58:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+ZCoKO1q5k5xlyh2CpRzmhWKpzE=', 'Content-Length': '0'} 2022-09-02T05:58:07.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2470d99c4c03d6-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.075 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.076 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.076 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.077 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.077 DEBUG:boto:Method: GET 2022-09-02T05:58:07.078 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.079 DEBUG:boto:Data: 2022-09-02T05:58:07.079 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.080 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.081 DEBUG:boto:Params: {} 2022-09-02T05:58:07.081 DEBUG:boto:Token: None 2022-09-02T05:58:07.082 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.083 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794c6ab5aab3bf46-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.087 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.088 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.088 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.089 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.089 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.090 DEBUG:boto:Method: GET 2022-09-02T05:58:07.090 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.091 DEBUG:boto:Data: 2022-09-02T05:58:07.091 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.092 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.092 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.093 DEBUG:boto:Params: {} 2022-09-02T05:58:07.093 DEBUG:boto:Token: None 2022-09-02T05:58:07.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df0f2317395b72b-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.142 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.143 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:07.144 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:07.144 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.145 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:58:07.145 DEBUG:boto:Data: 2022-09-02T05:58:07.146 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.147 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.148 DEBUG:boto:Params: {} 2022-09-02T05:58:07.149 DEBUG:boto:Token: None 2022-09-02T05:58:07.149 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj155 2022-09-02T05:58:07.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XVdAR1FHK8c2PtqpjLG3XGYJLLo= 2022-09-02T05:58:07.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XVdAR1FHK8c2PtqpjLG3XGYJLLo=', 'Content-Length': '0'} 2022-09-02T05:58:07.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc788978e9341a9c-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.153 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:07.154 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:07.155 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.155 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:58:07.156 DEBUG:boto:Data: 2022-09-02T05:58:07.156 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.157 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.157 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.158 DEBUG:boto:Params: {} 2022-09-02T05:58:07.158 DEBUG:boto:Token: None 2022-09-02T05:58:07.159 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj155 2022-09-02T05:58:07.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XVdAR1FHK8c2PtqpjLG3XGYJLLo= 2022-09-02T05:58:07.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XVdAR1FHK8c2PtqpjLG3XGYJLLo=', 'Content-Length': '0'} 2022-09-02T05:58:07.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1745802d4adbd14-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:58:07.164 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:07.164 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:07.165 DEBUG:boto:Method: GET 2022-09-02T05:58:07.165 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:58:07.166 DEBUG:boto:Data: 2022-09-02T05:58:07.167 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.168 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.169 DEBUG:boto:Params: {} 2022-09-02T05:58:07.169 DEBUG:boto:Token: None 2022-09-02T05:58:07.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj155 2022-09-02T05:58:07.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xlg8+Pyy5FZ6NpFEQVF9gZClFnk= 2022-09-02T05:58:07.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xlg8+Pyy5FZ6NpFEQVF9gZClFnk=', 'Content-Length': '0'} 2022-09-02T05:58:07.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3e9a8de92fb4a4-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.174 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:58:07.175 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:58:07.175 DEBUG:boto:Method: GET 2022-09-02T05:58:07.176 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:58:07.176 DEBUG:boto:Data: 2022-09-02T05:58:07.177 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.177 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.178 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.179 DEBUG:boto:Params: {} 2022-09-02T05:58:07.179 DEBUG:boto:Token: None 2022-09-02T05:58:07.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj155 2022-09-02T05:58:07.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xlg8+Pyy5FZ6NpFEQVF9gZClFnk= 2022-09-02T05:58:07.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xlg8+Pyy5FZ6NpFEQVF9gZClFnk=', 'Content-Length': '0'} 2022-09-02T05:58:07.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000860012dd45aedfdf-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.184 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.185 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.185 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.186 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.186 DEBUG:boto:Method: GET 2022-09-02T05:58:07.187 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.187 DEBUG:boto:Data: 2022-09-02T05:58:07.188 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.189 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.189 DEBUG:boto:Params: {} 2022-09-02T05:58:07.190 DEBUG:boto:Token: None 2022-09-02T05:58:07.191 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e60e9e55fb5743e-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.196 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.197 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.198 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.199 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.199 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.200 DEBUG:boto:Method: GET 2022-09-02T05:58:07.201 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:58:07.202 DEBUG:boto:Data: 2022-09-02T05:58:07.204 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.206 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.208 DEBUG:boto:Params: {} 2022-09-02T05:58:07.209 DEBUG:boto:Token: None 2022-09-02T05:58:07.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077572ef39c3bef8a-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.217 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:58:07.219 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.220 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.220 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.221 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.221 DEBUG:boto:Method: GET 2022-09-02T05:58:07.222 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.222 DEBUG:boto:Data: 2022-09-02T05:58:07.223 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.224 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.224 DEBUG:boto:Params: {} 2022-09-02T05:58:07.225 DEBUG:boto:Token: None 2022-09-02T05:58:07.226 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c= 2022-09-02T05:58:07.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c=', 'Content-Length': '0'} 2022-09-02T05:58:07.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc9f77076028569-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.230 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.231 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.231 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.232 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.232 DEBUG:boto:Method: GET 2022-09-02T05:58:07.233 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.234 DEBUG:boto:Data: 2022-09-02T05:58:07.234 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.235 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.235 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.236 DEBUG:boto:Params: {} 2022-09-02T05:58:07.236 DEBUG:boto:Token: None 2022-09-02T05:58:07.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c= 2022-09-02T05:58:07.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c=', 'Content-Length': '0'} 2022-09-02T05:58:07.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009374e0d9eedaefcf-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.242 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.243 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.244 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.244 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.245 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.246 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.247 DEBUG:boto:Data: 2022-09-02T05:58:07.248 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.249 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.250 DEBUG:boto:Params: {} 2022-09-02T05:58:07.251 DEBUG:boto:Token: None 2022-09-02T05:58:07.252 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iaXcvSt7rH3xv14f7DV4+x0Isbs= 2022-09-02T05:58:07.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iaXcvSt7rH3xv14f7DV4+x0Isbs=', 'Content-Length': '0'} 2022-09-02T05:58:07.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fea4b5bbe4b55fc-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.256 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.257 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.257 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.258 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.258 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.259 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.259 DEBUG:boto:Data: 2022-09-02T05:58:07.260 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.260 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.261 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.261 DEBUG:boto:Params: {} 2022-09-02T05:58:07.262 DEBUG:boto:Token: None 2022-09-02T05:58:07.263 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.263 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iaXcvSt7rH3xv14f7DV4+x0Isbs= 2022-09-02T05:58:07.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iaXcvSt7rH3xv14f7DV4+x0Isbs=', 'Content-Length': '0'} 2022-09-02T05:58:07.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc94df7eb17c605-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:58:07.267 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.268 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.268 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.269 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.269 DEBUG:boto:Method: GET 2022-09-02T05:58:07.270 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.270 DEBUG:boto:Data: 2022-09-02T05:58:07.271 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.272 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.272 DEBUG:boto:Params: {} 2022-09-02T05:58:07.273 DEBUG:boto:Token: None 2022-09-02T05:58:07.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c= 2022-09-02T05:58:07.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c=', 'Content-Length': '0'} 2022-09-02T05:58:07.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000050cce0c83243bf1-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.278 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.278 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.279 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.279 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.280 DEBUG:boto:Method: GET 2022-09-02T05:58:07.280 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.281 DEBUG:boto:Data: 2022-09-02T05:58:07.281 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.282 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.283 DEBUG:boto:Params: {} 2022-09-02T05:58:07.284 DEBUG:boto:Token: None 2022-09-02T05:58:07.284 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:58:07.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c= 2022-09-02T05:58:07.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q5NM+qiId/4cIclRbpyohm1Zj9c=', 'Content-Length': '0'} 2022-09-02T05:58:07.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193cbd66607a8eaa-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.288 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.289 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.289 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.290 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.291 DEBUG:boto:Method: GET 2022-09-02T05:58:07.291 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.292 DEBUG:boto:Data: 2022-09-02T05:58:07.292 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.293 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.294 DEBUG:boto:Params: {} 2022-09-02T05:58:07.294 DEBUG:boto:Token: None 2022-09-02T05:58:07.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000067ffb0d5e77c95-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.299 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.300 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.301 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.302 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.302 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.303 DEBUG:boto:Method: GET 2022-09-02T05:58:07.303 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.304 DEBUG:boto:Data: 2022-09-02T05:58:07.304 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.305 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.306 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.306 DEBUG:boto:Params: {} 2022-09-02T05:58:07.307 DEBUG:boto:Token: None 2022-09-02T05:58:07.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e575f56db0b3b7a7-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.313 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.314 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.314 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.315 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.315 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:58:07.316 DEBUG:boto:Data: 2022-09-02T05:58:07.316 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.317 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.318 DEBUG:boto:Params: {} 2022-09-02T05:58:07.318 DEBUG:boto:Token: None 2022-09-02T05:58:07.319 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17 2022-09-02T05:58:07.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CRFzODiUv8t7PbV1tWYz4gNSkKA= 2022-09-02T05:58:07.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CRFzODiUv8t7PbV1tWYz4gNSkKA=', 'Content-Length': '0'} 2022-09-02T05:58:07.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae069b222f4dfd85-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.323 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.324 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.325 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.326 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:58:07.326 DEBUG:boto:Data: 2022-09-02T05:58:07.327 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.329 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.330 DEBUG:boto:Params: {} 2022-09-02T05:58:07.331 DEBUG:boto:Token: None 2022-09-02T05:58:07.332 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17 2022-09-02T05:58:07.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CRFzODiUv8t7PbV1tWYz4gNSkKA= 2022-09-02T05:58:07.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CRFzODiUv8t7PbV1tWYz4gNSkKA=', 'Content-Length': '0'} 2022-09-02T05:58:07.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5dd7122e2f630a-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:58:07.337 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.337 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.338 DEBUG:boto:Method: GET 2022-09-02T05:58:07.338 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:58:07.339 DEBUG:boto:Data: 2022-09-02T05:58:07.339 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.340 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.341 DEBUG:boto:Params: {} 2022-09-02T05:58:07.342 DEBUG:boto:Token: None 2022-09-02T05:58:07.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17 2022-09-02T05:58:07.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZ6PSJ9FFPo0+0dY93gHeWlaXRI= 2022-09-02T05:58:07.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZ6PSJ9FFPo0+0dY93gHeWlaXRI=', 'Content-Length': '0'} 2022-09-02T05:58:07.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683be61ac811c651-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.386 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:58:07.387 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:58:07.388 DEBUG:boto:Method: GET 2022-09-02T05:58:07.388 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:58:07.389 DEBUG:boto:Data: 2022-09-02T05:58:07.389 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.390 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.390 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.391 DEBUG:boto:Params: {} 2022-09-02T05:58:07.392 DEBUG:boto:Token: None 2022-09-02T05:58:07.392 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj17 2022-09-02T05:58:07.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XZ6PSJ9FFPo0+0dY93gHeWlaXRI= 2022-09-02T05:58:07.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XZ6PSJ9FFPo0+0dY93gHeWlaXRI=', 'Content-Length': '0'} 2022-09-02T05:58:07.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b87731ee511da80-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.396 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.397 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.397 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.398 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.398 DEBUG:boto:Method: GET 2022-09-02T05:58:07.399 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.400 DEBUG:boto:Data: 2022-09-02T05:58:07.400 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.401 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.402 DEBUG:boto:Params: {} 2022-09-02T05:58:07.402 DEBUG:boto:Token: None 2022-09-02T05:58:07.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000310e6ec090d93ac2-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.408 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.409 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.409 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.410 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.410 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.411 DEBUG:boto:Method: GET 2022-09-02T05:58:07.411 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:58:07.412 DEBUG:boto:Data: 2022-09-02T05:58:07.412 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.413 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.413 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.414 DEBUG:boto:Params: {} 2022-09-02T05:58:07.415 DEBUG:boto:Token: None 2022-09-02T05:58:07.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004116d2357e2b6d53-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.420 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:58:07.421 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.422 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.422 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.423 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.424 DEBUG:boto:Method: GET 2022-09-02T05:58:07.424 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.425 DEBUG:boto:Data: 2022-09-02T05:58:07.425 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.426 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.427 DEBUG:boto:Params: {} 2022-09-02T05:58:07.427 DEBUG:boto:Token: None 2022-09-02T05:58:07.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA= 2022-09-02T05:58:07.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA=', 'Content-Length': '0'} 2022-09-02T05:58:07.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea6aba5e604538b-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.432 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.433 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.433 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.434 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.435 DEBUG:boto:Method: GET 2022-09-02T05:58:07.435 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.436 DEBUG:boto:Data: 2022-09-02T05:58:07.436 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.437 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.437 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.438 DEBUG:boto:Params: {} 2022-09-02T05:58:07.438 DEBUG:boto:Token: None 2022-09-02T05:58:07.439 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA= 2022-09-02T05:58:07.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA=', 'Content-Length': '0'} 2022-09-02T05:58:07.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000476b8ea0af14abbf-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.486 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.487 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.487 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.488 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.488 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.489 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.489 DEBUG:boto:Data: 2022-09-02T05:58:07.490 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.491 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.491 DEBUG:boto:Params: {} 2022-09-02T05:58:07.492 DEBUG:boto:Token: None 2022-09-02T05:58:07.493 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AQcT8GrSlY+X2wlLzWv0YDl9BDI= 2022-09-02T05:58:07.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AQcT8GrSlY+X2wlLzWv0YDl9BDI=', 'Content-Length': '0'} 2022-09-02T05:58:07.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996556d6b2bd8809-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.497 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.497 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.498 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.498 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.499 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.499 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.500 DEBUG:boto:Data: 2022-09-02T05:58:07.500 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.501 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.501 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.502 DEBUG:boto:Params: {} 2022-09-02T05:58:07.503 DEBUG:boto:Token: None 2022-09-02T05:58:07.503 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AQcT8GrSlY+X2wlLzWv0YDl9BDI= 2022-09-02T05:58:07.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AQcT8GrSlY+X2wlLzWv0YDl9BDI=', 'Content-Length': '0'} 2022-09-02T05:58:07.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fdd45fd70c9634c-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:58:07.508 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.508 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.509 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.509 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.510 DEBUG:boto:Method: GET 2022-09-02T05:58:07.510 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.511 DEBUG:boto:Data: 2022-09-02T05:58:07.512 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.513 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.513 DEBUG:boto:Params: {} 2022-09-02T05:58:07.514 DEBUG:boto:Token: None 2022-09-02T05:58:07.514 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.515 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA= 2022-09-02T05:58:07.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA=', 'Content-Length': '0'} 2022-09-02T05:58:07.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016804634a5018167-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.519 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.519 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.520 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.520 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.521 DEBUG:boto:Method: GET 2022-09-02T05:58:07.522 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.522 DEBUG:boto:Data: 2022-09-02T05:58:07.523 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.523 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.524 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.524 DEBUG:boto:Params: {} 2022-09-02T05:58:07.525 DEBUG:boto:Token: None 2022-09-02T05:58:07.526 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:58:07.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA= 2022-09-02T05:58:07.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SJqB8LV9+yjDHGv/Pf1WumANQKA=', 'Content-Length': '0'} 2022-09-02T05:58:07.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e278d9574e4db7b-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.530 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.531 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.531 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.532 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.532 DEBUG:boto:Method: GET 2022-09-02T05:58:07.533 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.533 DEBUG:boto:Data: 2022-09-02T05:58:07.534 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.535 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.535 DEBUG:boto:Params: {} 2022-09-02T05:58:07.536 DEBUG:boto:Token: None 2022-09-02T05:58:07.537 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd947b5b081f169-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.582 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.583 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.584 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.585 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.585 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.586 DEBUG:boto:Method: GET 2022-09-02T05:58:07.587 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.587 DEBUG:boto:Data: 2022-09-02T05:58:07.588 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.589 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.590 DEBUG:boto:Params: {} 2022-09-02T05:58:07.591 DEBUG:boto:Token: None 2022-09-02T05:58:07.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009defee456e4ade22-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.599 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.601 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.602 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.603 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.603 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:58:07.604 DEBUG:boto:Data: 2022-09-02T05:58:07.605 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.607 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.608 DEBUG:boto:Params: {} 2022-09-02T05:58:07.609 DEBUG:boto:Token: None 2022-09-02T05:58:07.610 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19 2022-09-02T05:58:07.611 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lhga3MLoKdfB6xMs7SNB5NForXU= 2022-09-02T05:58:07.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lhga3MLoKdfB6xMs7SNB5NForXU=', 'Content-Length': '0'} 2022-09-02T05:58:07.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1286330e0ab769-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.615 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.616 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.617 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.618 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:58:07.618 DEBUG:boto:Data: 2022-09-02T05:58:07.619 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.619 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.620 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.620 DEBUG:boto:Params: {} 2022-09-02T05:58:07.621 DEBUG:boto:Token: None 2022-09-02T05:58:07.621 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19 2022-09-02T05:58:07.622 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Lhga3MLoKdfB6xMs7SNB5NForXU= 2022-09-02T05:58:07.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Lhga3MLoKdfB6xMs7SNB5NForXU=', 'Content-Length': '0'} 2022-09-02T05:58:07.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70afb10a7a960ec-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:58:07.626 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.627 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.627 DEBUG:boto:Method: GET 2022-09-02T05:58:07.628 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:58:07.628 DEBUG:boto:Data: 2022-09-02T05:58:07.629 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.630 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.631 DEBUG:boto:Params: {} 2022-09-02T05:58:07.631 DEBUG:boto:Token: None 2022-09-02T05:58:07.632 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19 2022-09-02T05:58:07.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b5l3DA0lI0G4rTVxYT3RJ+AU4+s= 2022-09-02T05:58:07.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b5l3DA0lI0G4rTVxYT3RJ+AU4+s=', 'Content-Length': '0'} 2022-09-02T05:58:07.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e2392c5ea652664-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.636 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:58:07.637 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:58:07.637 DEBUG:boto:Method: GET 2022-09-02T05:58:07.638 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:58:07.638 DEBUG:boto:Data: 2022-09-02T05:58:07.639 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.639 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.640 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.641 DEBUG:boto:Params: {} 2022-09-02T05:58:07.641 DEBUG:boto:Token: None 2022-09-02T05:58:07.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj19 2022-09-02T05:58:07.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b5l3DA0lI0G4rTVxYT3RJ+AU4+s= 2022-09-02T05:58:07.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b5l3DA0lI0G4rTVxYT3RJ+AU4+s=', 'Content-Length': '0'} 2022-09-02T05:58:07.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049619fd1ab01b8ec-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.686 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.687 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.688 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.689 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.690 DEBUG:boto:Method: GET 2022-09-02T05:58:07.691 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.692 DEBUG:boto:Data: 2022-09-02T05:58:07.692 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.694 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.695 DEBUG:boto:Params: {} 2022-09-02T05:58:07.696 DEBUG:boto:Token: None 2022-09-02T05:58:07.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2fb1fab9dfafb9-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.702 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.704 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.705 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.705 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.706 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.707 DEBUG:boto:Method: GET 2022-09-02T05:58:07.708 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:58:07.708 DEBUG:boto:Data: 2022-09-02T05:58:07.709 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.710 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.710 DEBUG:boto:Params: {} 2022-09-02T05:58:07.711 DEBUG:boto:Token: None 2022-09-02T05:58:07.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516af6a14521494b-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.716 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:58:07.718 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.718 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.719 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.719 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.720 DEBUG:boto:Method: GET 2022-09-02T05:58:07.720 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.721 DEBUG:boto:Data: 2022-09-02T05:58:07.721 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.722 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.723 DEBUG:boto:Params: {} 2022-09-02T05:58:07.724 DEBUG:boto:Token: None 2022-09-02T05:58:07.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU= 2022-09-02T05:58:07.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU=', 'Content-Length': '0'} 2022-09-02T05:58:07.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf81346bd96589e-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.729 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.729 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.730 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.730 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.731 DEBUG:boto:Method: GET 2022-09-02T05:58:07.731 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.732 DEBUG:boto:Data: 2022-09-02T05:58:07.733 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.733 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.734 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.734 DEBUG:boto:Params: {} 2022-09-02T05:58:07.735 DEBUG:boto:Token: None 2022-09-02T05:58:07.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU= 2022-09-02T05:58:07.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU=', 'Content-Length': '0'} 2022-09-02T05:58:07.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c26ccaffe2aaea-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.741 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.741 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.742 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.742 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.743 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.744 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.744 DEBUG:boto:Data: 2022-09-02T05:58:07.745 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.746 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.746 DEBUG:boto:Params: {} 2022-09-02T05:58:07.747 DEBUG:boto:Token: None 2022-09-02T05:58:07.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0YCtiNLzk3NX/idZZUebVG94r2I= 2022-09-02T05:58:07.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0YCtiNLzk3NX/idZZUebVG94r2I=', 'Content-Length': '0'} 2022-09-02T05:58:07.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fb7bea191be94d-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.752 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.752 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.753 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.753 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.754 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.754 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.755 DEBUG:boto:Data: 2022-09-02T05:58:07.755 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.756 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.756 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.757 DEBUG:boto:Params: {} 2022-09-02T05:58:07.757 DEBUG:boto:Token: None 2022-09-02T05:58:07.758 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0YCtiNLzk3NX/idZZUebVG94r2I= 2022-09-02T05:58:07.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0YCtiNLzk3NX/idZZUebVG94r2I=', 'Content-Length': '0'} 2022-09-02T05:58:07.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd0715d7043cc85a-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:58:07.763 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.763 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.764 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.765 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.765 DEBUG:boto:Method: GET 2022-09-02T05:58:07.766 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.766 DEBUG:boto:Data: 2022-09-02T05:58:07.767 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.768 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.769 DEBUG:boto:Params: {} 2022-09-02T05:58:07.770 DEBUG:boto:Token: None 2022-09-02T05:58:07.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU= 2022-09-02T05:58:07.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU=', 'Content-Length': '0'} 2022-09-02T05:58:07.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5a8a53e4f86ddf-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.818 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.819 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.820 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.820 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.821 DEBUG:boto:Method: GET 2022-09-02T05:58:07.821 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.822 DEBUG:boto:Data: 2022-09-02T05:58:07.823 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.824 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.825 DEBUG:boto:Params: {} 2022-09-02T05:58:07.825 DEBUG:boto:Token: None 2022-09-02T05:58:07.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:58:07.827 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU= 2022-09-02T05:58:07.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8q9gvm1HRCj0DDCfZw+X9soX8AU=', 'Content-Length': '0'} 2022-09-02T05:58:07.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007739fbe522ab036-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.831 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.832 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.833 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.834 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.834 DEBUG:boto:Method: GET 2022-09-02T05:58:07.835 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.835 DEBUG:boto:Data: 2022-09-02T05:58:07.836 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.837 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.838 DEBUG:boto:Params: {} 2022-09-02T05:58:07.838 DEBUG:boto:Token: None 2022-09-02T05:58:07.840 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cb5904af553ded7-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.848 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.849 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.850 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.851 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.851 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.852 DEBUG:boto:Method: GET 2022-09-02T05:58:07.853 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.853 DEBUG:boto:Data: 2022-09-02T05:58:07.854 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.855 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.855 DEBUG:boto:Params: {} 2022-09-02T05:58:07.856 DEBUG:boto:Token: None 2022-09-02T05:58:07.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486ec97c212202bc-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.906 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.907 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.908 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.909 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.910 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:07.911 DEBUG:boto:Data: 2022-09-02T05:58:07.912 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.913 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.914 DEBUG:boto:Params: {} 2022-09-02T05:58:07.915 DEBUG:boto:Token: None 2022-09-02T05:58:07.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22 2022-09-02T05:58:07.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2UPql5qMq10YP5RxkQmYDqfFnUY= 2022-09-02T05:58:07.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2UPql5qMq10YP5RxkQmYDqfFnUY=', 'Content-Length': '0'} 2022-09-02T05:58:07.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb020f469c96593-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.920 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.921 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.922 DEBUG:boto:Method: HEAD 2022-09-02T05:58:07.923 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:07.923 DEBUG:boto:Data: 2022-09-02T05:58:07.924 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.924 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.925 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.925 DEBUG:boto:Params: {} 2022-09-02T05:58:07.926 DEBUG:boto:Token: None 2022-09-02T05:58:07.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22 2022-09-02T05:58:07.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2UPql5qMq10YP5RxkQmYDqfFnUY= 2022-09-02T05:58:07.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2UPql5qMq10YP5RxkQmYDqfFnUY=', 'Content-Length': '0'} 2022-09-02T05:58:07.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007196c2defb4e2e37-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:58:07.932 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.932 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.933 DEBUG:boto:Method: GET 2022-09-02T05:58:07.933 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:07.934 DEBUG:boto:Data: 2022-09-02T05:58:07.934 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.936 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.937 DEBUG:boto:Params: {} 2022-09-02T05:58:07.938 DEBUG:boto:Token: None 2022-09-02T05:58:07.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22 2022-09-02T05:58:07.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KBIqsu9SnkakWT2V6/qn8XHJ3aQ= 2022-09-02T05:58:07.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KBIqsu9SnkakWT2V6/qn8XHJ3aQ=', 'Content-Length': '0'} 2022-09-02T05:58:07.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec13dc4e6785d33-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.943 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:58:07.944 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:58:07.945 DEBUG:boto:Method: GET 2022-09-02T05:58:07.945 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:58:07.946 DEBUG:boto:Data: 2022-09-02T05:58:07.947 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.948 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.949 DEBUG:boto:Params: {} 2022-09-02T05:58:07.949 DEBUG:boto:Token: None 2022-09-02T05:58:07.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/obj22 2022-09-02T05:58:07.951 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KBIqsu9SnkakWT2V6/qn8XHJ3aQ= 2022-09-02T05:58:07.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KBIqsu9SnkakWT2V6/qn8XHJ3aQ=', 'Content-Length': '0'} 2022-09-02T05:58:07.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147892169d9c14a6-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.954 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.955 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.956 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.956 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.957 DEBUG:boto:Method: GET 2022-09-02T05:58:07.957 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.958 DEBUG:boto:Data: 2022-09-02T05:58:07.959 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:07.960 DEBUG:boto:Port: 8001 2022-09-02T05:58:07.961 DEBUG:boto:Params: {} 2022-09-02T05:58:07.961 DEBUG:boto:Token: None 2022-09-02T05:58:07.975 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b2ce2f7f31f3345-0063119b6f-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.981 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.982 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:07.982 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:07.983 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.984 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.984 DEBUG:boto:Method: GET 2022-09-02T05:58:07.985 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:58:07.985 DEBUG:boto:Data: 2022-09-02T05:58:07.986 DEBUG:boto:Headers: {} 2022-09-02T05:58:07.987 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:07.987 DEBUG:boto:Port: 8000 2022-09-02T05:58:07.988 DEBUG:boto:Params: {} 2022-09-02T05:58:07.989 DEBUG:boto:Token: None 2022-09-02T05:58:07.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:07 GMT /xljbkf-74/ 2022-09-02T05:58:07.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw= 2022-09-02T05:58:07.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:07 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2k1K8WnYMRWpjY2RSGz71EA6Pw=', 'Content-Length': '0'} 2022-09-02T05:58:07.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e682bc49882d6c9-0063119b6f-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:07 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:07.995 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:07.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:58:07.996 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:07.997 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:07.997 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:07.998 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:07.999 DEBUG:boto:Method: GET 2022-09-02T05:58:08.000 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.000 DEBUG:boto:Data: 2022-09-02T05:58:08.001 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.002 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.002 DEBUG:boto:Params: {} 2022-09-02T05:58:08.003 DEBUG:boto:Token: None 2022-09-02T05:58:08.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg= 2022-09-02T05:58:08.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg=', 'Content-Length': '0'} 2022-09-02T05:58:08.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023fa13605e3461f2-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.009 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.009 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.010 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.011 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.011 DEBUG:boto:Method: GET 2022-09-02T05:58:08.012 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.012 DEBUG:boto:Data: 2022-09-02T05:58:08.013 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.013 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.014 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.015 DEBUG:boto:Params: {} 2022-09-02T05:58:08.015 DEBUG:boto:Token: None 2022-09-02T05:58:08.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg= 2022-09-02T05:58:08.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg=', 'Content-Length': '0'} 2022-09-02T05:58:08.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d784627f0ad1c37-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.021 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.021 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.022 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.023 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.023 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.024 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.024 DEBUG:boto:Data: 2022-09-02T05:58:08.025 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.026 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.026 DEBUG:boto:Params: {} 2022-09-02T05:58:08.027 DEBUG:boto:Token: None 2022-09-02T05:58:08.028 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:krMt/2OSk/BWDgLdDkfPNniv3gQ= 2022-09-02T05:58:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:krMt/2OSk/BWDgLdDkfPNniv3gQ=', 'Content-Length': '0'} 2022-09-02T05:58:08.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb0e3b74fab25cf-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.032 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.033 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.033 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.034 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.034 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.035 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.036 DEBUG:boto:Data: 2022-09-02T05:58:08.036 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.037 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.038 DEBUG:boto:Params: {} 2022-09-02T05:58:08.038 DEBUG:boto:Token: None 2022-09-02T05:58:08.039 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:krMt/2OSk/BWDgLdDkfPNniv3gQ= 2022-09-02T05:58:08.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:krMt/2OSk/BWDgLdDkfPNniv3gQ=', 'Content-Length': '0'} 2022-09-02T05:58:08.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cb8e6b56b93a1d-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:58:08.044 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.045 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.045 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.046 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.047 DEBUG:boto:Method: GET 2022-09-02T05:58:08.047 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.048 DEBUG:boto:Data: 2022-09-02T05:58:08.048 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.049 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.050 DEBUG:boto:Params: {} 2022-09-02T05:58:08.050 DEBUG:boto:Token: None 2022-09-02T05:58:08.051 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg= 2022-09-02T05:58:08.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg=', 'Content-Length': '0'} 2022-09-02T05:58:08.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000f95222dd19a81e-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.056 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.057 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.057 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.058 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.059 DEBUG:boto:Method: GET 2022-09-02T05:58:08.059 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.060 DEBUG:boto:Data: 2022-09-02T05:58:08.060 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.061 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.062 DEBUG:boto:Params: {} 2022-09-02T05:58:08.063 DEBUG:boto:Token: None 2022-09-02T05:58:08.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:58:08.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg= 2022-09-02T05:58:08.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pZgj0gc+vB/6OTq7kLzkX9oVxZg=', 'Content-Length': '0'} 2022-09-02T05:58:08.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc72648c028782f4-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.114 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.115 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.116 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.116 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.117 DEBUG:boto:Method: GET 2022-09-02T05:58:08.118 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.118 DEBUG:boto:Data: 2022-09-02T05:58:08.119 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.120 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.121 DEBUG:boto:Params: {} 2022-09-02T05:58:08.121 DEBUG:boto:Token: None 2022-09-02T05:58:08.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5fbc3728fcd722a-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.127 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.128 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.128 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.129 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.130 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.130 DEBUG:boto:Method: GET 2022-09-02T05:58:08.131 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.131 DEBUG:boto:Data: 2022-09-02T05:58:08.132 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.132 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.133 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.133 DEBUG:boto:Params: {} 2022-09-02T05:58:08.134 DEBUG:boto:Token: None 2022-09-02T05:58:08.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.135 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000587da258de65da17-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.140 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.141 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.141 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.142 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.143 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:08.144 DEBUG:boto:Data: 2022-09-02T05:58:08.144 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.145 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.146 DEBUG:boto:Params: {} 2022-09-02T05:58:08.147 DEBUG:boto:Token: None 2022-09-02T05:58:08.147 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23 2022-09-02T05:58:08.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RA321JhDuT9moxWIdNyXwj6PO5M= 2022-09-02T05:58:08.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RA321JhDuT9moxWIdNyXwj6PO5M=', 'Content-Length': '0'} 2022-09-02T05:58:08.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af64e0755207c390-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.151 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.152 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.153 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.153 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:08.154 DEBUG:boto:Data: 2022-09-02T05:58:08.154 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.155 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.155 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.156 DEBUG:boto:Params: {} 2022-09-02T05:58:08.156 DEBUG:boto:Token: None 2022-09-02T05:58:08.157 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23 2022-09-02T05:58:08.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RA321JhDuT9moxWIdNyXwj6PO5M= 2022-09-02T05:58:08.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RA321JhDuT9moxWIdNyXwj6PO5M=', 'Content-Length': '0'} 2022-09-02T05:58:08.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ec014052dd66b6-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:58:08.162 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.162 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.163 DEBUG:boto:Method: GET 2022-09-02T05:58:08.163 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:08.164 DEBUG:boto:Data: 2022-09-02T05:58:08.165 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.166 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.166 DEBUG:boto:Params: {} 2022-09-02T05:58:08.167 DEBUG:boto:Token: None 2022-09-02T05:58:08.167 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23 2022-09-02T05:58:08.168 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SiReCzP1TAzgluGv1i6ZWQ0M8JA= 2022-09-02T05:58:08.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SiReCzP1TAzgluGv1i6ZWQ0M8JA=', 'Content-Length': '0'} 2022-09-02T05:58:08.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978956ad150ffbfc-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.214 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:58:08.215 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:58:08.216 DEBUG:boto:Method: GET 2022-09-02T05:58:08.216 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:58:08.217 DEBUG:boto:Data: 2022-09-02T05:58:08.218 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.219 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.220 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.221 DEBUG:boto:Params: {} 2022-09-02T05:58:08.221 DEBUG:boto:Token: None 2022-09-02T05:58:08.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj23 2022-09-02T05:58:08.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SiReCzP1TAzgluGv1i6ZWQ0M8JA= 2022-09-02T05:58:08.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SiReCzP1TAzgluGv1i6ZWQ0M8JA=', 'Content-Length': '0'} 2022-09-02T05:58:08.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a625b8a9550814-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.226 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.227 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.227 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.228 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.229 DEBUG:boto:Method: GET 2022-09-02T05:58:08.229 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.230 DEBUG:boto:Data: 2022-09-02T05:58:08.230 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.231 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.232 DEBUG:boto:Params: {} 2022-09-02T05:58:08.232 DEBUG:boto:Token: None 2022-09-02T05:58:08.233 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008434e4b6485591c2-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.238 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.239 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.240 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.240 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.241 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.241 DEBUG:boto:Method: GET 2022-09-02T05:58:08.242 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:58:08.242 DEBUG:boto:Data: 2022-09-02T05:58:08.243 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.244 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.245 DEBUG:boto:Params: {} 2022-09-02T05:58:08.245 DEBUG:boto:Token: None 2022-09-02T05:58:08.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6f2531c62966495-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.251 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:58:08.252 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.253 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.253 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.254 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.254 DEBUG:boto:Method: GET 2022-09-02T05:58:08.255 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.255 DEBUG:boto:Data: 2022-09-02T05:58:08.256 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.257 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.258 DEBUG:boto:Params: {} 2022-09-02T05:58:08.258 DEBUG:boto:Token: None 2022-09-02T05:58:08.259 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.259 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ= 2022-09-02T05:58:08.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ=', 'Content-Length': '0'} 2022-09-02T05:58:08.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6dce9d10784e5e2-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.263 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.264 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.264 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.265 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.266 DEBUG:boto:Method: GET 2022-09-02T05:58:08.266 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.267 DEBUG:boto:Data: 2022-09-02T05:58:08.267 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.268 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.269 DEBUG:boto:Params: {} 2022-09-02T05:58:08.269 DEBUG:boto:Token: None 2022-09-02T05:58:08.270 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ= 2022-09-02T05:58:08.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ=', 'Content-Length': '0'} 2022-09-02T05:58:08.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065306aed86b82130-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.314 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.315 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.315 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.316 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.316 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.317 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.317 DEBUG:boto:Data: 2022-09-02T05:58:08.318 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.319 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.320 DEBUG:boto:Params: {} 2022-09-02T05:58:08.320 DEBUG:boto:Token: None 2022-09-02T05:58:08.321 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1VDbIWCrAOU/RMbAA32Dyn9aTZ4= 2022-09-02T05:58:08.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1VDbIWCrAOU/RMbAA32Dyn9aTZ4=', 'Content-Length': '0'} 2022-09-02T05:58:08.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc24f053e9c0dd5-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.325 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.326 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.326 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.327 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.327 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.328 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.328 DEBUG:boto:Data: 2022-09-02T05:58:08.329 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.329 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.330 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.330 DEBUG:boto:Params: {} 2022-09-02T05:58:08.331 DEBUG:boto:Token: None 2022-09-02T05:58:08.331 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.332 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1VDbIWCrAOU/RMbAA32Dyn9aTZ4= 2022-09-02T05:58:08.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1VDbIWCrAOU/RMbAA32Dyn9aTZ4=', 'Content-Length': '0'} 2022-09-02T05:58:08.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a396f173873e250-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:58:08.336 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.337 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.337 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.338 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.338 DEBUG:boto:Method: GET 2022-09-02T05:58:08.339 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.340 DEBUG:boto:Data: 2022-09-02T05:58:08.340 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.341 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.342 DEBUG:boto:Params: {} 2022-09-02T05:58:08.342 DEBUG:boto:Token: None 2022-09-02T05:58:08.343 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ= 2022-09-02T05:58:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ=', 'Content-Length': '0'} 2022-09-02T05:58:08.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38b8e0d300b724b-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.347 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.348 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.348 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.349 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.350 DEBUG:boto:Method: GET 2022-09-02T05:58:08.350 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.351 DEBUG:boto:Data: 2022-09-02T05:58:08.351 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.353 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.354 DEBUG:boto:Params: {} 2022-09-02T05:58:08.355 DEBUG:boto:Token: None 2022-09-02T05:58:08.356 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:58:08.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ= 2022-09-02T05:58:08.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/wu6ZaDOhCwHDSrCthLOyLy4fbQ=', 'Content-Length': '0'} 2022-09-02T05:58:08.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9f54c9a274b11c-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.363 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.364 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.365 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.366 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.367 DEBUG:boto:Method: GET 2022-09-02T05:58:08.367 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.368 DEBUG:boto:Data: 2022-09-02T05:58:08.369 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.371 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.372 DEBUG:boto:Params: {} 2022-09-02T05:58:08.373 DEBUG:boto:Token: None 2022-09-02T05:58:08.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5eb8a4a5ddd84f3-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.380 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.380 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.381 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.382 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.382 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.383 DEBUG:boto:Method: GET 2022-09-02T05:58:08.384 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.384 DEBUG:boto:Data: 2022-09-02T05:58:08.385 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.386 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.386 DEBUG:boto:Params: {} 2022-09-02T05:58:08.388 DEBUG:boto:Token: None 2022-09-02T05:58:08.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7332d4da4fdd23f-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.396 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.397 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.397 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.398 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.398 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:08.399 DEBUG:boto:Data: 2022-09-02T05:58:08.399 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.401 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.401 DEBUG:boto:Params: {} 2022-09-02T05:58:08.402 DEBUG:boto:Token: None 2022-09-02T05:58:08.403 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29 2022-09-02T05:58:08.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N9dMEyYdSBvGtqqaGFNlajMo6RU= 2022-09-02T05:58:08.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N9dMEyYdSBvGtqqaGFNlajMo6RU=', 'Content-Length': '0'} 2022-09-02T05:58:08.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab9f8a8e40077446-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.408 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.408 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.409 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.409 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:08.410 DEBUG:boto:Data: 2022-09-02T05:58:08.411 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.412 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.412 DEBUG:boto:Params: {} 2022-09-02T05:58:08.413 DEBUG:boto:Token: None 2022-09-02T05:58:08.413 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29 2022-09-02T05:58:08.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N9dMEyYdSBvGtqqaGFNlajMo6RU= 2022-09-02T05:58:08.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N9dMEyYdSBvGtqqaGFNlajMo6RU=', 'Content-Length': '0'} 2022-09-02T05:58:08.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c81ebdaf9410642-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:58:08.418 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.419 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.420 DEBUG:boto:Method: GET 2022-09-02T05:58:08.421 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:08.422 DEBUG:boto:Data: 2022-09-02T05:58:08.423 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.424 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.425 DEBUG:boto:Params: {} 2022-09-02T05:58:08.425 DEBUG:boto:Token: None 2022-09-02T05:58:08.426 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29 2022-09-02T05:58:08.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2OJyvZ7b+GDzxA07vcgjhCLYurY= 2022-09-02T05:58:08.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2OJyvZ7b+GDzxA07vcgjhCLYurY=', 'Content-Length': '0'} 2022-09-02T05:58:08.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b19c9e507cbd52-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.470 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:58:08.472 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:58:08.473 DEBUG:boto:Method: GET 2022-09-02T05:58:08.474 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:58:08.475 DEBUG:boto:Data: 2022-09-02T05:58:08.476 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.477 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.478 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.479 DEBUG:boto:Params: {} 2022-09-02T05:58:08.480 DEBUG:boto:Token: None 2022-09-02T05:58:08.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj29 2022-09-02T05:58:08.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2OJyvZ7b+GDzxA07vcgjhCLYurY= 2022-09-02T05:58:08.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2OJyvZ7b+GDzxA07vcgjhCLYurY=', 'Content-Length': '0'} 2022-09-02T05:58:08.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f58ef22f24e0d6-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.490 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.491 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.492 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.493 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.494 DEBUG:boto:Method: GET 2022-09-02T05:58:08.495 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.496 DEBUG:boto:Data: 2022-09-02T05:58:08.497 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.500 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.501 DEBUG:boto:Params: {} 2022-09-02T05:58:08.502 DEBUG:boto:Token: None 2022-09-02T05:58:08.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066c1db1ee72be59b-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.510 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.511 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.512 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.513 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.514 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.515 DEBUG:boto:Method: GET 2022-09-02T05:58:08.516 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:58:08.518 DEBUG:boto:Data: 2022-09-02T05:58:08.519 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.521 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.522 DEBUG:boto:Params: {} 2022-09-02T05:58:08.523 DEBUG:boto:Token: None 2022-09-02T05:58:08.524 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7906e73f849199a-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.531 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:58:08.534 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.535 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.536 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.537 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.538 DEBUG:boto:Method: GET 2022-09-02T05:58:08.539 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.540 DEBUG:boto:Data: 2022-09-02T05:58:08.541 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.544 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.545 DEBUG:boto:Params: {} 2022-09-02T05:58:08.546 DEBUG:boto:Token: None 2022-09-02T05:58:08.547 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA= 2022-09-02T05:58:08.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA=', 'Content-Length': '0'} 2022-09-02T05:58:08.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df604ca1591040cc-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.554 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.555 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.556 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.556 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.558 DEBUG:boto:Method: GET 2022-09-02T05:58:08.559 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.560 DEBUG:boto:Data: 2022-09-02T05:58:08.561 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.562 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.563 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.563 DEBUG:boto:Params: {} 2022-09-02T05:58:08.565 DEBUG:boto:Token: None 2022-09-02T05:58:08.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA= 2022-09-02T05:58:08.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA=', 'Content-Length': '0'} 2022-09-02T05:58:08.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009393e421567f791a-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.571 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.571 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.572 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.572 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.573 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.573 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.574 DEBUG:boto:Data: 2022-09-02T05:58:08.574 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.575 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.576 DEBUG:boto:Params: {} 2022-09-02T05:58:08.576 DEBUG:boto:Token: None 2022-09-02T05:58:08.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D97YYTz/ye+c+yuRrKSltcJjcYA= 2022-09-02T05:58:08.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D97YYTz/ye+c+yuRrKSltcJjcYA=', 'Content-Length': '0'} 2022-09-02T05:58:08.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a78d401fb94b9b-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.581 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.582 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.582 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.583 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.583 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.584 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.584 DEBUG:boto:Data: 2022-09-02T05:58:08.585 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.586 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.586 DEBUG:boto:Params: {} 2022-09-02T05:58:08.587 DEBUG:boto:Token: None 2022-09-02T05:58:08.588 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D97YYTz/ye+c+yuRrKSltcJjcYA= 2022-09-02T05:58:08.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D97YYTz/ye+c+yuRrKSltcJjcYA=', 'Content-Length': '0'} 2022-09-02T05:58:08.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b608d59323db40b-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:58:08.592 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.593 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.593 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.594 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.594 DEBUG:boto:Method: GET 2022-09-02T05:58:08.595 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.595 DEBUG:boto:Data: 2022-09-02T05:58:08.596 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.597 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.597 DEBUG:boto:Params: {} 2022-09-02T05:58:08.598 DEBUG:boto:Token: None 2022-09-02T05:58:08.599 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA= 2022-09-02T05:58:08.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA=', 'Content-Length': '0'} 2022-09-02T05:58:08.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e00fa970cc10cf-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.603 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.604 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.605 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.607 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.608 DEBUG:boto:Method: GET 2022-09-02T05:58:08.609 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.609 DEBUG:boto:Data: 2022-09-02T05:58:08.610 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.612 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.612 DEBUG:boto:Params: {} 2022-09-02T05:58:08.613 DEBUG:boto:Token: None 2022-09-02T05:58:08.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:58:08.614 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA= 2022-09-02T05:58:08.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hHLggVfWU6T18E8pfp9O/hqkwmA=', 'Content-Length': '0'} 2022-09-02T05:58:08.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ececcc27ac8696-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.658 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.659 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.660 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.661 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.662 DEBUG:boto:Method: GET 2022-09-02T05:58:08.663 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.664 DEBUG:boto:Data: 2022-09-02T05:58:08.665 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.668 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.670 DEBUG:boto:Params: {} 2022-09-02T05:58:08.671 DEBUG:boto:Token: None 2022-09-02T05:58:08.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b235f4929fc26dd5-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.680 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.681 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.682 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.683 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.684 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.686 DEBUG:boto:Method: GET 2022-09-02T05:58:08.687 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.688 DEBUG:boto:Data: 2022-09-02T05:58:08.689 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.692 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.692 DEBUG:boto:Params: {} 2022-09-02T05:58:08.693 DEBUG:boto:Token: None 2022-09-02T05:58:08.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b6fcd26c44cb07a-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.699 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.700 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.701 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.701 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.702 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:08.702 DEBUG:boto:Data: 2022-09-02T05:58:08.703 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.704 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.705 DEBUG:boto:Params: {} 2022-09-02T05:58:08.705 DEBUG:boto:Token: None 2022-09-02T05:58:08.706 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31 2022-09-02T05:58:08.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/QGw793NiJEa3hRStEr6kMjhi4= 2022-09-02T05:58:08.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/QGw793NiJEa3hRStEr6kMjhi4=', 'Content-Length': '0'} 2022-09-02T05:58:08.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ce42a26fbbc4d5-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.710 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.711 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.711 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.712 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:08.713 DEBUG:boto:Data: 2022-09-02T05:58:08.713 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.714 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.715 DEBUG:boto:Params: {} 2022-09-02T05:58:08.715 DEBUG:boto:Token: None 2022-09-02T05:58:08.716 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31 2022-09-02T05:58:08.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V/QGw793NiJEa3hRStEr6kMjhi4= 2022-09-02T05:58:08.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V/QGw793NiJEa3hRStEr6kMjhi4=', 'Content-Length': '0'} 2022-09-02T05:58:08.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f6d1759a8b0a10-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:58:08.721 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.721 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.722 DEBUG:boto:Method: GET 2022-09-02T05:58:08.722 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:08.723 DEBUG:boto:Data: 2022-09-02T05:58:08.724 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.725 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.725 DEBUG:boto:Params: {} 2022-09-02T05:58:08.726 DEBUG:boto:Token: None 2022-09-02T05:58:08.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31 2022-09-02T05:58:08.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iuQD+s3zBvwiTJ+EqtGje4FvKHU= 2022-09-02T05:58:08.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iuQD+s3zBvwiTJ+EqtGje4FvKHU=', 'Content-Length': '0'} 2022-09-02T05:58:08.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9603a6e73ee59d-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.731 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:58:08.731 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:58:08.732 DEBUG:boto:Method: GET 2022-09-02T05:58:08.732 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:58:08.733 DEBUG:boto:Data: 2022-09-02T05:58:08.733 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.734 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.735 DEBUG:boto:Params: {} 2022-09-02T05:58:08.735 DEBUG:boto:Token: None 2022-09-02T05:58:08.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj31 2022-09-02T05:58:08.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iuQD+s3zBvwiTJ+EqtGje4FvKHU= 2022-09-02T05:58:08.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iuQD+s3zBvwiTJ+EqtGje4FvKHU=', 'Content-Length': '0'} 2022-09-02T05:58:08.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be829c85e7f1253-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.740 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.741 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.741 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.742 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.742 DEBUG:boto:Method: GET 2022-09-02T05:58:08.743 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.743 DEBUG:boto:Data: 2022-09-02T05:58:08.744 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.745 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.746 DEBUG:boto:Params: {} 2022-09-02T05:58:08.746 DEBUG:boto:Token: None 2022-09-02T05:58:08.747 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb377680ffaf1183-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.798 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.800 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.801 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.802 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.803 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.804 DEBUG:boto:Method: GET 2022-09-02T05:58:08.805 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:58:08.806 DEBUG:boto:Data: 2022-09-02T05:58:08.807 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.808 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.809 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.810 DEBUG:boto:Params: {} 2022-09-02T05:58:08.811 DEBUG:boto:Token: None 2022-09-02T05:58:08.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a071d958a323ced8-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.817 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:58:08.819 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.819 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.820 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.820 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.821 DEBUG:boto:Method: GET 2022-09-02T05:58:08.821 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.822 DEBUG:boto:Data: 2022-09-02T05:58:08.823 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.824 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.824 DEBUG:boto:Params: {} 2022-09-02T05:58:08.825 DEBUG:boto:Token: None 2022-09-02T05:58:08.826 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8= 2022-09-02T05:58:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8=', 'Content-Length': '0'} 2022-09-02T05:58:08.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f214d82ca6e56c-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.830 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.831 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.831 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.832 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.832 DEBUG:boto:Method: GET 2022-09-02T05:58:08.833 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.833 DEBUG:boto:Data: 2022-09-02T05:58:08.834 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.834 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.835 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.835 DEBUG:boto:Params: {} 2022-09-02T05:58:08.836 DEBUG:boto:Token: None 2022-09-02T05:58:08.837 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8= 2022-09-02T05:58:08.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8=', 'Content-Length': '0'} 2022-09-02T05:58:08.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002064f33885a4dc5b-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.841 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.841 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.842 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.842 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.843 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.843 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.844 DEBUG:boto:Data: 2022-09-02T05:58:08.844 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.845 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.846 DEBUG:boto:Params: {} 2022-09-02T05:58:08.846 DEBUG:boto:Token: None 2022-09-02T05:58:08.847 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.848 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GgPPVRZ6GJX6XSMrvOBEmVK6aPI= 2022-09-02T05:58:08.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GgPPVRZ6GJX6XSMrvOBEmVK6aPI=', 'Content-Length': '0'} 2022-09-02T05:58:08.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e4a3058985545b-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.851 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.851 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.852 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.852 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.853 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.853 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.854 DEBUG:boto:Data: 2022-09-02T05:58:08.854 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.855 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.856 DEBUG:boto:Params: {} 2022-09-02T05:58:08.856 DEBUG:boto:Token: None 2022-09-02T05:58:08.857 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GgPPVRZ6GJX6XSMrvOBEmVK6aPI= 2022-09-02T05:58:08.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GgPPVRZ6GJX6XSMrvOBEmVK6aPI=', 'Content-Length': '0'} 2022-09-02T05:58:08.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad350ba2364c01ae-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:58:08.862 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.862 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.863 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.863 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.864 DEBUG:boto:Method: GET 2022-09-02T05:58:08.864 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.865 DEBUG:boto:Data: 2022-09-02T05:58:08.866 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.867 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.867 DEBUG:boto:Params: {} 2022-09-02T05:58:08.868 DEBUG:boto:Token: None 2022-09-02T05:58:08.868 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8= 2022-09-02T05:58:08.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8=', 'Content-Length': '0'} 2022-09-02T05:58:08.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e646285476eea443-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.873 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.873 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.874 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.874 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.875 DEBUG:boto:Method: GET 2022-09-02T05:58:08.875 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.876 DEBUG:boto:Data: 2022-09-02T05:58:08.876 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.877 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.878 DEBUG:boto:Params: {} 2022-09-02T05:58:08.878 DEBUG:boto:Token: None 2022-09-02T05:58:08.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:58:08.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8= 2022-09-02T05:58:08.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:COxL4Lqrg2CQMrXSYrBi1djcmX8=', 'Content-Length': '0'} 2022-09-02T05:58:08.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b91544e762840c5-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.884 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.885 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.886 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:08.887 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:08.888 DEBUG:boto:Method: GET 2022-09-02T05:58:08.889 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:08.890 DEBUG:boto:Data: 2022-09-02T05:58:08.892 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.893 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.894 DEBUG:boto:Params: {} 2022-09-02T05:58:08.895 DEBUG:boto:Token: None 2022-09-02T05:58:08.896 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb41b60aa12fe98-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.900 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.901 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:08.902 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:08.902 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:08.903 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:08.903 DEBUG:boto:Method: GET 2022-09-02T05:58:08.904 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:08.904 DEBUG:boto:Data: 2022-09-02T05:58:08.905 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.906 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.907 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.907 DEBUG:boto:Params: {} 2022-09-02T05:58:08.908 DEBUG:boto:Token: None 2022-09-02T05:58:08.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/ 2022-09-02T05:58:08.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q= 2022-09-02T05:58:08.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6gAOp5rGqmfBQmF16tSiyWteN4Q=', 'Content-Length': '0'} 2022-09-02T05:58:08.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089de421f96179490-0063119b70-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.958 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:08.959 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.960 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.960 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.961 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:08.961 DEBUG:boto:Data: 2022-09-02T05:58:08.962 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.963 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.963 DEBUG:boto:Params: {} 2022-09-02T05:58:08.964 DEBUG:boto:Token: None 2022-09-02T05:58:08.965 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33 2022-09-02T05:58:08.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GweAbW3IwZHYhsM614JEnDT2j2Y= 2022-09-02T05:58:08.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GweAbW3IwZHYhsM614JEnDT2j2Y=', 'Content-Length': '0'} 2022-09-02T05:58:08.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5db69c518c41e1-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.971 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.972 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.973 DEBUG:boto:Method: HEAD 2022-09-02T05:58:08.974 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:08.975 DEBUG:boto:Data: 2022-09-02T05:58:08.976 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.976 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.977 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.977 DEBUG:boto:Params: {} 2022-09-02T05:58:08.978 DEBUG:boto:Token: None 2022-09-02T05:58:08.979 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33 2022-09-02T05:58:08.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GweAbW3IwZHYhsM614JEnDT2j2Y= 2022-09-02T05:58:08.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GweAbW3IwZHYhsM614JEnDT2j2Y=', 'Content-Length': '0'} 2022-09-02T05:58:08.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99ea3e515bcfee7-0063119b70-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:58:08.983 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.984 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.985 DEBUG:boto:Method: GET 2022-09-02T05:58:08.985 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:08.986 DEBUG:boto:Data: 2022-09-02T05:58:08.987 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:08.988 DEBUG:boto:Port: 8001 2022-09-02T05:58:08.988 DEBUG:boto:Params: {} 2022-09-02T05:58:08.989 DEBUG:boto:Token: None 2022-09-02T05:58:08.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:08 GMT /xljbkf-74/obj33 2022-09-02T05:58:08.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N6R08Bzij+f9C08DOI2GsKLs7UY= 2022-09-02T05:58:08.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:08 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N6R08Bzij+f9C08DOI2GsKLs7UY=', 'Content-Length': '0'} 2022-09-02T05:58:08.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004484cc0a42cd65c4-0063119b70-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:08 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:08.994 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:58:08.994 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:58:08.995 DEBUG:boto:Method: GET 2022-09-02T05:58:08.995 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:58:08.997 DEBUG:boto:Data: 2022-09-02T05:58:08.997 DEBUG:boto:Headers: {} 2022-09-02T05:58:08.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:08.998 DEBUG:boto:Port: 8000 2022-09-02T05:58:08.999 DEBUG:boto:Params: {} 2022-09-02T05:58:08.999 DEBUG:boto:Token: None 2022-09-02T05:58:09.000 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj33 2022-09-02T05:58:09.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oWCPmBz9/Zbt1nMaIpmHKk6XAm8= 2022-09-02T05:58:09.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oWCPmBz9/Zbt1nMaIpmHKk6XAm8=', 'Content-Length': '0'} 2022-09-02T05:58:09.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e0d253caa1321a-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.005 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.006 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.006 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:09.007 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:09.007 DEBUG:boto:Method: GET 2022-09-02T05:58:09.008 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:09.008 DEBUG:boto:Data: 2022-09-02T05:58:09.009 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.010 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.011 DEBUG:boto:Params: {} 2022-09-02T05:58:09.011 DEBUG:boto:Token: None 2022-09-02T05:58:09.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b4f080975f6e58f-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.017 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.018 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.018 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.019 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:09.019 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:58:09.020 DEBUG:boto:Method: GET 2022-09-02T05:58:09.021 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:58:09.021 DEBUG:boto:Data: 2022-09-02T05:58:09.022 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.023 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.024 DEBUG:boto:Params: {} 2022-09-02T05:58:09.025 DEBUG:boto:Token: None 2022-09-02T05:58:09.026 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002462afd904869c2-0063119b71-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.035 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:58:09.036 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.037 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.038 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.039 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.039 DEBUG:boto:Method: GET 2022-09-02T05:58:09.040 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.040 DEBUG:boto:Data: 2022-09-02T05:58:09.041 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.042 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.043 DEBUG:boto:Params: {} 2022-09-02T05:58:09.043 DEBUG:boto:Token: None 2022-09-02T05:58:09.044 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o= 2022-09-02T05:58:09.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o=', 'Content-Length': '0'} 2022-09-02T05:58:09.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b70d0a459609518a-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.049 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.050 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.051 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.052 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.053 DEBUG:boto:Method: GET 2022-09-02T05:58:09.054 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.056 DEBUG:boto:Data: 2022-09-02T05:58:09.057 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.058 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.059 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.060 DEBUG:boto:Params: {} 2022-09-02T05:58:09.061 DEBUG:boto:Token: None 2022-09-02T05:58:09.062 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o= 2022-09-02T05:58:09.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o=', 'Content-Length': '0'} 2022-09-02T05:58:09.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a60dc3990c6689-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.068 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.069 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.069 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.070 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.070 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.071 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.071 DEBUG:boto:Data: 2022-09-02T05:58:09.072 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.073 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.073 DEBUG:boto:Params: {} 2022-09-02T05:58:09.074 DEBUG:boto:Token: None 2022-09-02T05:58:09.075 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.076 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vWSmKhR5pBUkZnLufUFPYSTAR3M= 2022-09-02T05:58:09.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vWSmKhR5pBUkZnLufUFPYSTAR3M=', 'Content-Length': '0'} 2022-09-02T05:58:09.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de48aec4f274c870-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.080 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.081 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.082 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.082 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.083 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.083 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.084 DEBUG:boto:Data: 2022-09-02T05:58:09.084 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.086 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.086 DEBUG:boto:Params: {} 2022-09-02T05:58:09.087 DEBUG:boto:Token: None 2022-09-02T05:58:09.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vWSmKhR5pBUkZnLufUFPYSTAR3M= 2022-09-02T05:58:09.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vWSmKhR5pBUkZnLufUFPYSTAR3M=', 'Content-Length': '0'} 2022-09-02T05:58:09.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ea4ae458106502-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:58:09.095 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.096 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.097 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.098 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.099 DEBUG:boto:Method: GET 2022-09-02T05:58:09.100 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.101 DEBUG:boto:Data: 2022-09-02T05:58:09.102 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.104 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.105 DEBUG:boto:Params: {} 2022-09-02T05:58:09.106 DEBUG:boto:Token: None 2022-09-02T05:58:09.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o= 2022-09-02T05:58:09.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o=', 'Content-Length': '0'} 2022-09-02T05:58:09.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d3cc1d3aafe858-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.114 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.115 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.116 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.117 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.118 DEBUG:boto:Method: GET 2022-09-02T05:58:09.119 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.120 DEBUG:boto:Data: 2022-09-02T05:58:09.121 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.122 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.123 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.124 DEBUG:boto:Params: {} 2022-09-02T05:58:09.125 DEBUG:boto:Token: None 2022-09-02T05:58:09.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:58:09.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o= 2022-09-02T05:58:09.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ACzWD9hxe6aC29c1DVBugk4/60o=', 'Content-Length': '0'} 2022-09-02T05:58:09.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a21bceef16f9899-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.174 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.175 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.176 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.176 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.177 DEBUG:boto:Method: GET 2022-09-02T05:58:09.177 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.178 DEBUG:boto:Data: 2022-09-02T05:58:09.178 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.180 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.180 DEBUG:boto:Params: {} 2022-09-02T05:58:09.181 DEBUG:boto:Token: None 2022-09-02T05:58:09.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ed5a8041ebdf49d-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.188 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.189 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.189 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.190 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.191 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.192 DEBUG:boto:Method: GET 2022-09-02T05:58:09.194 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.195 DEBUG:boto:Data: 2022-09-02T05:58:09.196 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.196 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.197 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.198 DEBUG:boto:Params: {} 2022-09-02T05:58:09.199 DEBUG:boto:Token: None 2022-09-02T05:58:09.200 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111b5d9472cc5d9f-0063119b71-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.206 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.207 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.207 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.208 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.208 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:09.209 DEBUG:boto:Data: 2022-09-02T05:58:09.209 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.211 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.211 DEBUG:boto:Params: {} 2022-09-02T05:58:09.212 DEBUG:boto:Token: None 2022-09-02T05:58:09.212 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34 2022-09-02T05:58:09.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wU4pPIaXJEROTmPR0EMb5tRo5Rw= 2022-09-02T05:58:09.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wU4pPIaXJEROTmPR0EMb5tRo5Rw=', 'Content-Length': '0'} 2022-09-02T05:58:09.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db3e2a962fe0aad-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.216 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.217 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.217 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.218 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:09.219 DEBUG:boto:Data: 2022-09-02T05:58:09.220 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.220 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.221 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.221 DEBUG:boto:Params: {} 2022-09-02T05:58:09.222 DEBUG:boto:Token: None 2022-09-02T05:58:09.222 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34 2022-09-02T05:58:09.223 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wU4pPIaXJEROTmPR0EMb5tRo5Rw= 2022-09-02T05:58:09.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wU4pPIaXJEROTmPR0EMb5tRo5Rw=', 'Content-Length': '0'} 2022-09-02T05:58:09.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e9df6c78efd46f-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:58:09.227 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.228 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.229 DEBUG:boto:Method: GET 2022-09-02T05:58:09.229 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:09.230 DEBUG:boto:Data: 2022-09-02T05:58:09.230 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.232 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.232 DEBUG:boto:Params: {} 2022-09-02T05:58:09.234 DEBUG:boto:Token: None 2022-09-02T05:58:09.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34 2022-09-02T05:58:09.236 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:317rFv3+2ocKfoCom0YZesY0YxA= 2022-09-02T05:58:09.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:317rFv3+2ocKfoCom0YZesY0YxA=', 'Content-Length': '0'} 2022-09-02T05:58:09.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d215030b7ab032a-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.282 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:58:09.283 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:58:09.284 DEBUG:boto:Method: GET 2022-09-02T05:58:09.285 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:58:09.286 DEBUG:boto:Data: 2022-09-02T05:58:09.287 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.288 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.290 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.291 DEBUG:boto:Params: {} 2022-09-02T05:58:09.292 DEBUG:boto:Token: None 2022-09-02T05:58:09.293 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj34 2022-09-02T05:58:09.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:317rFv3+2ocKfoCom0YZesY0YxA= 2022-09-02T05:58:09.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:317rFv3+2ocKfoCom0YZesY0YxA=', 'Content-Length': '0'} 2022-09-02T05:58:09.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081ff9ce1612cccec-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.299 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.300 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.301 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.302 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.303 DEBUG:boto:Method: GET 2022-09-02T05:58:09.304 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.304 DEBUG:boto:Data: 2022-09-02T05:58:09.305 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.306 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.307 DEBUG:boto:Params: {} 2022-09-02T05:58:09.307 DEBUG:boto:Token: None 2022-09-02T05:58:09.308 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0494fd3590cbd97-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.313 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.314 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.314 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.315 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.315 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.316 DEBUG:boto:Method: GET 2022-09-02T05:58:09.317 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:58:09.318 DEBUG:boto:Data: 2022-09-02T05:58:09.319 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.319 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.320 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.320 DEBUG:boto:Params: {} 2022-09-02T05:58:09.321 DEBUG:boto:Token: None 2022-09-02T05:58:09.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003befc7ff06a80d7c-0063119b71-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.327 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:58:09.328 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.329 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.330 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.330 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.331 DEBUG:boto:Method: GET 2022-09-02T05:58:09.332 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.332 DEBUG:boto:Data: 2022-09-02T05:58:09.333 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.334 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.335 DEBUG:boto:Params: {} 2022-09-02T05:58:09.335 DEBUG:boto:Token: None 2022-09-02T05:58:09.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4= 2022-09-02T05:58:09.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4=', 'Content-Length': '0'} 2022-09-02T05:58:09.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab33aff4effdf89e-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.341 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.341 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.342 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.342 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.343 DEBUG:boto:Method: GET 2022-09-02T05:58:09.343 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.344 DEBUG:boto:Data: 2022-09-02T05:58:09.344 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.345 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.346 DEBUG:boto:Params: {} 2022-09-02T05:58:09.347 DEBUG:boto:Token: None 2022-09-02T05:58:09.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4= 2022-09-02T05:58:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4=', 'Content-Length': '0'} 2022-09-02T05:58:09.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e0a398e4a6aebee-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.398 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.399 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.400 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.400 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.401 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.401 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.402 DEBUG:boto:Data: 2022-09-02T05:58:09.403 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.404 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.404 DEBUG:boto:Params: {} 2022-09-02T05:58:09.405 DEBUG:boto:Token: None 2022-09-02T05:58:09.406 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ybsEVTMs0NYItFdE4aFZxT4ltc= 2022-09-02T05:58:09.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ybsEVTMs0NYItFdE4aFZxT4ltc=', 'Content-Length': '0'} 2022-09-02T05:58:09.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000905ecdac3ea918fe-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.410 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.411 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.412 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.412 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.413 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.413 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.414 DEBUG:boto:Data: 2022-09-02T05:58:09.414 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.415 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.416 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.416 DEBUG:boto:Params: {} 2022-09-02T05:58:09.417 DEBUG:boto:Token: None 2022-09-02T05:58:09.417 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ybsEVTMs0NYItFdE4aFZxT4ltc= 2022-09-02T05:58:09.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ybsEVTMs0NYItFdE4aFZxT4ltc=', 'Content-Length': '0'} 2022-09-02T05:58:09.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a56b7ba7d15e813-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:58:09.422 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.422 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.423 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.423 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.424 DEBUG:boto:Method: GET 2022-09-02T05:58:09.425 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.425 DEBUG:boto:Data: 2022-09-02T05:58:09.426 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.427 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.428 DEBUG:boto:Params: {} 2022-09-02T05:58:09.428 DEBUG:boto:Token: None 2022-09-02T05:58:09.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4= 2022-09-02T05:58:09.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4=', 'Content-Length': '0'} 2022-09-02T05:58:09.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045672f36dcfdd709-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.433 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.434 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.434 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.435 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.436 DEBUG:boto:Method: GET 2022-09-02T05:58:09.436 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.437 DEBUG:boto:Data: 2022-09-02T05:58:09.438 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.439 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.439 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.440 DEBUG:boto:Params: {} 2022-09-02T05:58:09.440 DEBUG:boto:Token: None 2022-09-02T05:58:09.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:58:09.441 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4= 2022-09-02T05:58:09.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o8LtcpZVRYnMUC16g7xa/Mqj9+4=', 'Content-Length': '0'} 2022-09-02T05:58:09.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf66c41eea2d879-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.652 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.653 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.653 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.654 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.654 DEBUG:boto:Method: GET 2022-09-02T05:58:09.655 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.655 DEBUG:boto:Data: 2022-09-02T05:58:09.656 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.657 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.658 DEBUG:boto:Params: {} 2022-09-02T05:58:09.658 DEBUG:boto:Token: None 2022-09-02T05:58:09.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c037258dc83fdb4-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.677 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.677 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.678 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.678 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.679 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.680 DEBUG:boto:Method: GET 2022-09-02T05:58:09.680 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.681 DEBUG:boto:Data: 2022-09-02T05:58:09.681 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.682 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.682 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.683 DEBUG:boto:Params: {} 2022-09-02T05:58:09.683 DEBUG:boto:Token: None 2022-09-02T05:58:09.684 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067a9bbc2d226804b-0063119b71-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.689 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.690 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.691 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.691 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.692 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:09.692 DEBUG:boto:Data: 2022-09-02T05:58:09.693 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.694 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.694 DEBUG:boto:Params: {} 2022-09-02T05:58:09.695 DEBUG:boto:Token: None 2022-09-02T05:58:09.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38 2022-09-02T05:58:09.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mJx3L9csVLJvRT9IQUKhZZAqlps= 2022-09-02T05:58:09.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mJx3L9csVLJvRT9IQUKhZZAqlps=', 'Content-Length': '0'} 2022-09-02T05:58:09.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6df8f92a31b36cc-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.699 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.700 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.700 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.701 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:09.701 DEBUG:boto:Data: 2022-09-02T05:58:09.702 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.703 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.703 DEBUG:boto:Params: {} 2022-09-02T05:58:09.704 DEBUG:boto:Token: None 2022-09-02T05:58:09.704 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38 2022-09-02T05:58:09.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mJx3L9csVLJvRT9IQUKhZZAqlps= 2022-09-02T05:58:09.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mJx3L9csVLJvRT9IQUKhZZAqlps=', 'Content-Length': '0'} 2022-09-02T05:58:09.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003567f3efa1e0cbce-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:58:09.709 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.710 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.710 DEBUG:boto:Method: GET 2022-09-02T05:58:09.711 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:09.711 DEBUG:boto:Data: 2022-09-02T05:58:09.712 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.713 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.713 DEBUG:boto:Params: {} 2022-09-02T05:58:09.714 DEBUG:boto:Token: None 2022-09-02T05:58:09.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38 2022-09-02T05:58:09.715 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6UToh7wIhLEaiKGGT650iU+KFMw= 2022-09-02T05:58:09.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6UToh7wIhLEaiKGGT650iU+KFMw=', 'Content-Length': '0'} 2022-09-02T05:58:09.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7fd263d59c1a241-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.762 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:58:09.763 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:58:09.763 DEBUG:boto:Method: GET 2022-09-02T05:58:09.764 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:58:09.764 DEBUG:boto:Data: 2022-09-02T05:58:09.765 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.766 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.766 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.767 DEBUG:boto:Params: {} 2022-09-02T05:58:09.767 DEBUG:boto:Token: None 2022-09-02T05:58:09.768 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj38 2022-09-02T05:58:09.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6UToh7wIhLEaiKGGT650iU+KFMw= 2022-09-02T05:58:09.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6UToh7wIhLEaiKGGT650iU+KFMw=', 'Content-Length': '0'} 2022-09-02T05:58:09.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064f7d27880583024-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.773 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.773 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.774 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.774 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.775 DEBUG:boto:Method: GET 2022-09-02T05:58:09.775 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.776 DEBUG:boto:Data: 2022-09-02T05:58:09.776 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.777 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.778 DEBUG:boto:Params: {} 2022-09-02T05:58:09.779 DEBUG:boto:Token: None 2022-09-02T05:58:09.779 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d9306f40910a794-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.785 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.789 DEBUG:boto:Method: GET 2022-09-02T05:58:09.789 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:58:09.790 DEBUG:boto:Data: 2022-09-02T05:58:09.791 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.793 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.794 DEBUG:boto:Params: {} 2022-09-02T05:58:09.795 DEBUG:boto:Token: None 2022-09-02T05:58:09.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa945c353a4a5ca-0063119b71-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.802 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:58:09.804 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.805 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.805 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.806 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.807 DEBUG:boto:Method: GET 2022-09-02T05:58:09.808 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.808 DEBUG:boto:Data: 2022-09-02T05:58:09.809 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.810 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.811 DEBUG:boto:Params: {} 2022-09-02T05:58:09.811 DEBUG:boto:Token: None 2022-09-02T05:58:09.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY= 2022-09-02T05:58:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY=', 'Content-Length': '0'} 2022-09-02T05:58:09.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5d45e70bbf837e-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.817 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.817 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.818 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.818 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.819 DEBUG:boto:Method: GET 2022-09-02T05:58:09.819 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.820 DEBUG:boto:Data: 2022-09-02T05:58:09.820 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.821 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.822 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.822 DEBUG:boto:Params: {} 2022-09-02T05:58:09.823 DEBUG:boto:Token: None 2022-09-02T05:58:09.824 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.824 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY= 2022-09-02T05:58:09.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY=', 'Content-Length': '0'} 2022-09-02T05:58:09.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e806758a601e2ab-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.828 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.829 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.829 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.830 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.830 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.831 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.832 DEBUG:boto:Data: 2022-09-02T05:58:09.832 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.833 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.834 DEBUG:boto:Params: {} 2022-09-02T05:58:09.834 DEBUG:boto:Token: None 2022-09-02T05:58:09.835 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H/VTXZOe/x5hH2DYrOypxax9mQY= 2022-09-02T05:58:09.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H/VTXZOe/x5hH2DYrOypxax9mQY=', 'Content-Length': '0'} 2022-09-02T05:58:09.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce72dd5ef47efeb6-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.840 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.840 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.841 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.841 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.842 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.842 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.843 DEBUG:boto:Data: 2022-09-02T05:58:09.844 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.844 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.845 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.845 DEBUG:boto:Params: {} 2022-09-02T05:58:09.846 DEBUG:boto:Token: None 2022-09-02T05:58:09.847 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.847 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H/VTXZOe/x5hH2DYrOypxax9mQY= 2022-09-02T05:58:09.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H/VTXZOe/x5hH2DYrOypxax9mQY=', 'Content-Length': '0'} 2022-09-02T05:58:09.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7711282032b3ca4-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:58:09.851 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.852 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.853 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.853 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.854 DEBUG:boto:Method: GET 2022-09-02T05:58:09.854 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.855 DEBUG:boto:Data: 2022-09-02T05:58:09.855 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.856 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.857 DEBUG:boto:Params: {} 2022-09-02T05:58:09.857 DEBUG:boto:Token: None 2022-09-02T05:58:09.858 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY= 2022-09-02T05:58:09.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY=', 'Content-Length': '0'} 2022-09-02T05:58:09.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4f86b520ec9220-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.902 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.903 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.904 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.904 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.905 DEBUG:boto:Method: GET 2022-09-02T05:58:09.905 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.906 DEBUG:boto:Data: 2022-09-02T05:58:09.906 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.907 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.907 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.908 DEBUG:boto:Params: {} 2022-09-02T05:58:09.909 DEBUG:boto:Token: None 2022-09-02T05:58:09.909 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:58:09.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY= 2022-09-02T05:58:09.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Cb/3MxClmwnKZxYOETh+t8PKmqY=', 'Content-Length': '0'} 2022-09-02T05:58:09.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b0a9ca49f78543-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.914 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.914 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.915 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.915 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.916 DEBUG:boto:Method: GET 2022-09-02T05:58:09.916 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.917 DEBUG:boto:Data: 2022-09-02T05:58:09.918 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.919 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.919 DEBUG:boto:Params: {} 2022-09-02T05:58:09.920 DEBUG:boto:Token: None 2022-09-02T05:58:09.920 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f889256405d54f8-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.925 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.927 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.928 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.928 DEBUG:boto:Method: GET 2022-09-02T05:58:09.929 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.929 DEBUG:boto:Data: 2022-09-02T05:58:09.930 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.931 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.931 DEBUG:boto:Params: {} 2022-09-02T05:58:09.932 DEBUG:boto:Token: None 2022-09-02T05:58:09.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef113ab2ec3952f-0063119b71-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.937 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:09.938 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.938 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.939 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.939 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:09.940 DEBUG:boto:Data: 2022-09-02T05:58:09.940 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.941 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.942 DEBUG:boto:Params: {} 2022-09-02T05:58:09.942 DEBUG:boto:Token: None 2022-09-02T05:58:09.943 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44 2022-09-02T05:58:09.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kcI293wMvmov3Xj6f+263wQlSuE= 2022-09-02T05:58:09.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kcI293wMvmov3Xj6f+263wQlSuE=', 'Content-Length': '0'} 2022-09-02T05:58:09.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ddb8e25bb6c6e2-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.947 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.948 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.948 DEBUG:boto:Method: HEAD 2022-09-02T05:58:09.949 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:09.949 DEBUG:boto:Data: 2022-09-02T05:58:09.950 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.950 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.951 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.951 DEBUG:boto:Params: {} 2022-09-02T05:58:09.952 DEBUG:boto:Token: None 2022-09-02T05:58:09.952 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44 2022-09-02T05:58:09.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kcI293wMvmov3Xj6f+263wQlSuE= 2022-09-02T05:58:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kcI293wMvmov3Xj6f+263wQlSuE=', 'Content-Length': '0'} 2022-09-02T05:58:09.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040881a244cb94331-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:58:09.957 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.957 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.958 DEBUG:boto:Method: GET 2022-09-02T05:58:09.958 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:09.959 DEBUG:boto:Data: 2022-09-02T05:58:09.960 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.961 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.961 DEBUG:boto:Params: {} 2022-09-02T05:58:09.962 DEBUG:boto:Token: None 2022-09-02T05:58:09.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44 2022-09-02T05:58:09.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bo2q3g0L9AJmFtjmsiUguMh51HU= 2022-09-02T05:58:09.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bo2q3g0L9AJmFtjmsiUguMh51HU=', 'Content-Length': '0'} 2022-09-02T05:58:09.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7b8c6b12936495-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.967 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:58:09.967 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:58:09.968 DEBUG:boto:Method: GET 2022-09-02T05:58:09.968 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:58:09.969 DEBUG:boto:Data: 2022-09-02T05:58:09.969 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:09.970 DEBUG:boto:Port: 8000 2022-09-02T05:58:09.971 DEBUG:boto:Params: {} 2022-09-02T05:58:09.971 DEBUG:boto:Token: None 2022-09-02T05:58:09.972 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/obj44 2022-09-02T05:58:09.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bo2q3g0L9AJmFtjmsiUguMh51HU= 2022-09-02T05:58:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bo2q3g0L9AJmFtjmsiUguMh51HU=', 'Content-Length': '0'} 2022-09-02T05:58:09.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f1e43165b143929-0063119b71-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:09.976 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:09.977 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:09.977 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.978 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.978 DEBUG:boto:Method: GET 2022-09-02T05:58:09.979 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:09.990 DEBUG:boto:Data: 2022-09-02T05:58:09.991 DEBUG:boto:Headers: {} 2022-09-02T05:58:09.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:09.992 DEBUG:boto:Port: 8001 2022-09-02T05:58:09.992 DEBUG:boto:Params: {} 2022-09-02T05:58:09.993 DEBUG:boto:Token: None 2022-09-02T05:58:09.994 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:09 GMT /xljbkf-74/ 2022-09-02T05:58:09.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA= 2022-09-02T05:58:09.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:09 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Iy0UZJHBg9+prTpb5R/mIdiihA=', 'Content-Length': '0'} 2022-09-02T05:58:09.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000352a27c8f08c14ee-0063119b71-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:09 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.042 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.044 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.044 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.045 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:10.046 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:58:10.047 DEBUG:boto:Method: GET 2022-09-02T05:58:10.047 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:58:10.048 DEBUG:boto:Data: 2022-09-02T05:58:10.048 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.049 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.050 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.050 DEBUG:boto:Params: {} 2022-09-02T05:58:10.051 DEBUG:boto:Token: None 2022-09-02T05:58:10.051 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008138c8f3e94100f9-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.056 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:58:10.058 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.058 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.059 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.060 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.060 DEBUG:boto:Method: GET 2022-09-02T05:58:10.061 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.061 DEBUG:boto:Data: 2022-09-02T05:58:10.062 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.063 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.064 DEBUG:boto:Params: {} 2022-09-02T05:58:10.064 DEBUG:boto:Token: None 2022-09-02T05:58:10.065 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk= 2022-09-02T05:58:10.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk=', 'Content-Length': '0'} 2022-09-02T05:58:10.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5631489622f9d8-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.069 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.070 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.071 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.071 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.072 DEBUG:boto:Method: GET 2022-09-02T05:58:10.073 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.073 DEBUG:boto:Data: 2022-09-02T05:58:10.074 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.074 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.075 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.075 DEBUG:boto:Params: {} 2022-09-02T05:58:10.076 DEBUG:boto:Token: None 2022-09-02T05:58:10.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk= 2022-09-02T05:58:10.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk=', 'Content-Length': '0'} 2022-09-02T05:58:10.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed1eb4efc17aa116-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.123 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.124 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.124 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.125 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.126 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.126 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.127 DEBUG:boto:Data: 2022-09-02T05:58:10.128 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.130 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.131 DEBUG:boto:Params: {} 2022-09-02T05:58:10.132 DEBUG:boto:Token: None 2022-09-02T05:58:10.133 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ayb4AGxYkKKRPCC58zBE4gFM/HI= 2022-09-02T05:58:10.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ayb4AGxYkKKRPCC58zBE4gFM/HI=', 'Content-Length': '0'} 2022-09-02T05:58:10.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb78c5cb7e242367-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.138 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.139 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.139 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.140 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.140 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.141 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.141 DEBUG:boto:Data: 2022-09-02T05:58:10.142 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.142 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.143 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.143 DEBUG:boto:Params: {} 2022-09-02T05:58:10.144 DEBUG:boto:Token: None 2022-09-02T05:58:10.144 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.145 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ayb4AGxYkKKRPCC58zBE4gFM/HI= 2022-09-02T05:58:10.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ayb4AGxYkKKRPCC58zBE4gFM/HI=', 'Content-Length': '0'} 2022-09-02T05:58:10.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be94ce993cbf158c-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:58:10.149 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.150 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.150 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.151 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.151 DEBUG:boto:Method: GET 2022-09-02T05:58:10.152 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.152 DEBUG:boto:Data: 2022-09-02T05:58:10.153 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.154 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.154 DEBUG:boto:Params: {} 2022-09-02T05:58:10.155 DEBUG:boto:Token: None 2022-09-02T05:58:10.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.156 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk= 2022-09-02T05:58:10.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk=', 'Content-Length': '0'} 2022-09-02T05:58:10.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a2f305a57861a1-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.160 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.160 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.161 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.162 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.162 DEBUG:boto:Method: GET 2022-09-02T05:58:10.163 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.163 DEBUG:boto:Data: 2022-09-02T05:58:10.164 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.164 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.165 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.165 DEBUG:boto:Params: {} 2022-09-02T05:58:10.166 DEBUG:boto:Token: None 2022-09-02T05:58:10.166 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:58:10.167 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk= 2022-09-02T05:58:10.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tWHIQCypECeAXBoNoo30EGtNZnk=', 'Content-Length': '0'} 2022-09-02T05:58:10.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c60ca065b733ce-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.171 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.172 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.172 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.173 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.173 DEBUG:boto:Method: GET 2022-09-02T05:58:10.174 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.174 DEBUG:boto:Data: 2022-09-02T05:58:10.175 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.176 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.176 DEBUG:boto:Params: {} 2022-09-02T05:58:10.177 DEBUG:boto:Token: None 2022-09-02T05:58:10.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000714cb60a8155bafe-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.183 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.184 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.184 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.185 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.185 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.186 DEBUG:boto:Method: GET 2022-09-02T05:58:10.186 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.187 DEBUG:boto:Data: 2022-09-02T05:58:10.187 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.189 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.189 DEBUG:boto:Params: {} 2022-09-02T05:58:10.190 DEBUG:boto:Token: None 2022-09-02T05:58:10.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f4cd25c5bdc932e-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.195 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.196 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.196 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.197 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.197 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:10.198 DEBUG:boto:Data: 2022-09-02T05:58:10.198 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.199 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.200 DEBUG:boto:Params: {} 2022-09-02T05:58:10.201 DEBUG:boto:Token: None 2022-09-02T05:58:10.201 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46 2022-09-02T05:58:10.202 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OscLRbr4DwBYg4njC9zgAnmR8UA= 2022-09-02T05:58:10.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OscLRbr4DwBYg4njC9zgAnmR8UA=', 'Content-Length': '0'} 2022-09-02T05:58:10.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f37e339f76e95680-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.205 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.206 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.206 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.207 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:10.207 DEBUG:boto:Data: 2022-09-02T05:58:10.208 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.208 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.209 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.210 DEBUG:boto:Params: {} 2022-09-02T05:58:10.210 DEBUG:boto:Token: None 2022-09-02T05:58:10.211 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46 2022-09-02T05:58:10.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OscLRbr4DwBYg4njC9zgAnmR8UA= 2022-09-02T05:58:10.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OscLRbr4DwBYg4njC9zgAnmR8UA=', 'Content-Length': '0'} 2022-09-02T05:58:10.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e153a47eef80e18-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:58:10.216 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.217 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.218 DEBUG:boto:Method: GET 2022-09-02T05:58:10.219 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:10.220 DEBUG:boto:Data: 2022-09-02T05:58:10.220 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.221 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.222 DEBUG:boto:Params: {} 2022-09-02T05:58:10.223 DEBUG:boto:Token: None 2022-09-02T05:58:10.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46 2022-09-02T05:58:10.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QvbI7xUrDYOspxZSHMhBh0wTegM= 2022-09-02T05:58:10.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QvbI7xUrDYOspxZSHMhBh0wTegM=', 'Content-Length': '0'} 2022-09-02T05:58:10.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6d1d4b4b886bcc-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.228 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:58:10.229 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:58:10.229 DEBUG:boto:Method: GET 2022-09-02T05:58:10.230 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:58:10.230 DEBUG:boto:Data: 2022-09-02T05:58:10.231 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.232 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.233 DEBUG:boto:Params: {} 2022-09-02T05:58:10.233 DEBUG:boto:Token: None 2022-09-02T05:58:10.234 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj46 2022-09-02T05:58:10.235 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QvbI7xUrDYOspxZSHMhBh0wTegM= 2022-09-02T05:58:10.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QvbI7xUrDYOspxZSHMhBh0wTegM=', 'Content-Length': '0'} 2022-09-02T05:58:10.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2fc27ac45338225-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.282 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.283 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.284 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.284 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.285 DEBUG:boto:Method: GET 2022-09-02T05:58:10.285 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.286 DEBUG:boto:Data: 2022-09-02T05:58:10.286 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.287 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.288 DEBUG:boto:Params: {} 2022-09-02T05:58:10.289 DEBUG:boto:Token: None 2022-09-02T05:58:10.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015026fecd1faf8b5-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.294 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.295 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.296 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.296 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.297 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.297 DEBUG:boto:Method: GET 2022-09-02T05:58:10.298 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:58:10.299 DEBUG:boto:Data: 2022-09-02T05:58:10.299 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.300 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.301 DEBUG:boto:Params: {} 2022-09-02T05:58:10.301 DEBUG:boto:Token: None 2022-09-02T05:58:10.302 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.303 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91096f41c0ddd42-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.307 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:58:10.309 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.309 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.310 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.310 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.311 DEBUG:boto:Method: GET 2022-09-02T05:58:10.312 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.312 DEBUG:boto:Data: 2022-09-02T05:58:10.313 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.314 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.315 DEBUG:boto:Params: {} 2022-09-02T05:58:10.315 DEBUG:boto:Token: None 2022-09-02T05:58:10.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c= 2022-09-02T05:58:10.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c=', 'Content-Length': '0'} 2022-09-02T05:58:10.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d7d9cef2da462c-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.321 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.322 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.322 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.323 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.324 DEBUG:boto:Method: GET 2022-09-02T05:58:10.324 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.325 DEBUG:boto:Data: 2022-09-02T05:58:10.326 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.327 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.328 DEBUG:boto:Params: {} 2022-09-02T05:58:10.329 DEBUG:boto:Token: None 2022-09-02T05:58:10.330 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c= 2022-09-02T05:58:10.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c=', 'Content-Length': '0'} 2022-09-02T05:58:10.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbad0bd84d190a5f-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.337 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.338 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.338 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.339 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.340 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.341 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.341 DEBUG:boto:Data: 2022-09-02T05:58:10.342 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.343 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.344 DEBUG:boto:Params: {} 2022-09-02T05:58:10.344 DEBUG:boto:Token: None 2022-09-02T05:58:10.345 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xkmnMMrNl8yEa3UqlT+Ev+teamw= 2022-09-02T05:58:10.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xkmnMMrNl8yEa3UqlT+Ev+teamw=', 'Content-Length': '0'} 2022-09-02T05:58:10.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a90b46b622744d31-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.350 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.351 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.351 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.352 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.352 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.353 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.354 DEBUG:boto:Data: 2022-09-02T05:58:10.355 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.356 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.356 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.357 DEBUG:boto:Params: {} 2022-09-02T05:58:10.357 DEBUG:boto:Token: None 2022-09-02T05:58:10.358 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xkmnMMrNl8yEa3UqlT+Ev+teamw= 2022-09-02T05:58:10.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xkmnMMrNl8yEa3UqlT+Ev+teamw=', 'Content-Length': '0'} 2022-09-02T05:58:10.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070509a7fb65da2cd-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:58:10.364 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.365 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.366 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.367 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.368 DEBUG:boto:Method: GET 2022-09-02T05:58:10.369 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.369 DEBUG:boto:Data: 2022-09-02T05:58:10.370 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.371 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.372 DEBUG:boto:Params: {} 2022-09-02T05:58:10.373 DEBUG:boto:Token: None 2022-09-02T05:58:10.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c= 2022-09-02T05:58:10.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c=', 'Content-Length': '0'} 2022-09-02T05:58:10.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe88b03985d58aa-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.379 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.379 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.380 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.381 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.382 DEBUG:boto:Method: GET 2022-09-02T05:58:10.383 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.383 DEBUG:boto:Data: 2022-09-02T05:58:10.384 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.385 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.386 DEBUG:boto:Params: {} 2022-09-02T05:58:10.387 DEBUG:boto:Token: None 2022-09-02T05:58:10.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:58:10.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c= 2022-09-02T05:58:10.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VxHO37cELGXndJmMGU4hljX7w+c=', 'Content-Length': '0'} 2022-09-02T05:58:10.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a09b1f2fceafe9-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.436 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.437 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.438 DEBUG:boto:Method: GET 2022-09-02T05:58:10.438 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.439 DEBUG:boto:Data: 2022-09-02T05:58:10.440 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.441 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.442 DEBUG:boto:Params: {} 2022-09-02T05:58:10.443 DEBUG:boto:Token: None 2022-09-02T05:58:10.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6d11941d882b56-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.450 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.451 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.452 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.452 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.453 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.453 DEBUG:boto:Method: GET 2022-09-02T05:58:10.454 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.455 DEBUG:boto:Data: 2022-09-02T05:58:10.455 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.457 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.457 DEBUG:boto:Params: {} 2022-09-02T05:58:10.458 DEBUG:boto:Token: None 2022-09-02T05:58:10.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e71f7751e92b32e9-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.464 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.464 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.465 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.466 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.466 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:10.467 DEBUG:boto:Data: 2022-09-02T05:58:10.468 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.469 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.470 DEBUG:boto:Params: {} 2022-09-02T05:58:10.471 DEBUG:boto:Token: None 2022-09-02T05:58:10.472 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51 2022-09-02T05:58:10.472 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t0a6hnM6FO6Qb+YY1BCUf8zhgPE= 2022-09-02T05:58:10.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t0a6hnM6FO6Qb+YY1BCUf8zhgPE=', 'Content-Length': '0'} 2022-09-02T05:58:10.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000272b78d711679e9f-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.476 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.477 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.477 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.478 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:10.479 DEBUG:boto:Data: 2022-09-02T05:58:10.479 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.481 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.482 DEBUG:boto:Params: {} 2022-09-02T05:58:10.484 DEBUG:boto:Token: None 2022-09-02T05:58:10.485 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51 2022-09-02T05:58:10.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t0a6hnM6FO6Qb+YY1BCUf8zhgPE= 2022-09-02T05:58:10.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t0a6hnM6FO6Qb+YY1BCUf8zhgPE=', 'Content-Length': '0'} 2022-09-02T05:58:10.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ca90129248e0fd-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:58:10.493 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.494 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.495 DEBUG:boto:Method: GET 2022-09-02T05:58:10.496 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:10.497 DEBUG:boto:Data: 2022-09-02T05:58:10.498 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.500 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.501 DEBUG:boto:Params: {} 2022-09-02T05:58:10.502 DEBUG:boto:Token: None 2022-09-02T05:58:10.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51 2022-09-02T05:58:10.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jlt8mEl9CsiNUpL5iCng3lAnHlI= 2022-09-02T05:58:10.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jlt8mEl9CsiNUpL5iCng3lAnHlI=', 'Content-Length': '0'} 2022-09-02T05:58:10.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd77c2e18d5bd175-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.509 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:58:10.510 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:58:10.511 DEBUG:boto:Method: GET 2022-09-02T05:58:10.511 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:58:10.512 DEBUG:boto:Data: 2022-09-02T05:58:10.513 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.513 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.514 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.514 DEBUG:boto:Params: {} 2022-09-02T05:58:10.515 DEBUG:boto:Token: None 2022-09-02T05:58:10.516 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj51 2022-09-02T05:58:10.516 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jlt8mEl9CsiNUpL5iCng3lAnHlI= 2022-09-02T05:58:10.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jlt8mEl9CsiNUpL5iCng3lAnHlI=', 'Content-Length': '0'} 2022-09-02T05:58:10.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a83f4ed507ac269-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.562 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.564 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.565 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.566 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.567 DEBUG:boto:Method: GET 2022-09-02T05:58:10.568 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.569 DEBUG:boto:Data: 2022-09-02T05:58:10.570 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.571 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.572 DEBUG:boto:Params: {} 2022-09-02T05:58:10.572 DEBUG:boto:Token: None 2022-09-02T05:58:10.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004067157e5dccf913-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.578 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.579 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.580 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.581 DEBUG:boto:Method: GET 2022-09-02T05:58:10.581 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:58:10.582 DEBUG:boto:Data: 2022-09-02T05:58:10.583 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.583 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.584 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.584 DEBUG:boto:Params: {} 2022-09-02T05:58:10.585 DEBUG:boto:Token: None 2022-09-02T05:58:10.586 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff13c5e6da093b3-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.590 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:58:10.592 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.592 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.593 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.593 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.594 DEBUG:boto:Method: GET 2022-09-02T05:58:10.595 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.595 DEBUG:boto:Data: 2022-09-02T05:58:10.596 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.597 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.597 DEBUG:boto:Params: {} 2022-09-02T05:58:10.598 DEBUG:boto:Token: None 2022-09-02T05:58:10.598 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU= 2022-09-02T05:58:10.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU=', 'Content-Length': '0'} 2022-09-02T05:58:10.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f152f0fd36bcb4-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.603 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.603 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.604 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.604 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.605 DEBUG:boto:Method: GET 2022-09-02T05:58:10.605 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.606 DEBUG:boto:Data: 2022-09-02T05:58:10.606 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.607 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.607 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.608 DEBUG:boto:Params: {} 2022-09-02T05:58:10.608 DEBUG:boto:Token: None 2022-09-02T05:58:10.609 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU= 2022-09-02T05:58:10.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU=', 'Content-Length': '0'} 2022-09-02T05:58:10.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083192790c486b348-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.613 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.614 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.615 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.615 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.616 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.616 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.617 DEBUG:boto:Data: 2022-09-02T05:58:10.617 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.618 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.619 DEBUG:boto:Params: {} 2022-09-02T05:58:10.619 DEBUG:boto:Token: None 2022-09-02T05:58:10.620 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.621 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mPajOMapm69AsM6yZfKUTRdaQEY= 2022-09-02T05:58:10.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mPajOMapm69AsM6yZfKUTRdaQEY=', 'Content-Length': '0'} 2022-09-02T05:58:10.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000386a8176cf0a0e9-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.624 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.625 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.625 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.626 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.626 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.627 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.627 DEBUG:boto:Data: 2022-09-02T05:58:10.628 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.629 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.629 DEBUG:boto:Params: {} 2022-09-02T05:58:10.630 DEBUG:boto:Token: None 2022-09-02T05:58:10.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mPajOMapm69AsM6yZfKUTRdaQEY= 2022-09-02T05:58:10.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mPajOMapm69AsM6yZfKUTRdaQEY=', 'Content-Length': '0'} 2022-09-02T05:58:10.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004abc86fa9392872a-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:58:10.635 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.635 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.636 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.637 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.637 DEBUG:boto:Method: GET 2022-09-02T05:58:10.638 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.638 DEBUG:boto:Data: 2022-09-02T05:58:10.639 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.640 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.640 DEBUG:boto:Params: {} 2022-09-02T05:58:10.641 DEBUG:boto:Token: None 2022-09-02T05:58:10.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU= 2022-09-02T05:58:10.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU=', 'Content-Length': '0'} 2022-09-02T05:58:10.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002659abdef8df7618-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.686 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.687 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.688 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.688 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.689 DEBUG:boto:Method: GET 2022-09-02T05:58:10.689 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.690 DEBUG:boto:Data: 2022-09-02T05:58:10.691 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.691 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.692 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.692 DEBUG:boto:Params: {} 2022-09-02T05:58:10.693 DEBUG:boto:Token: None 2022-09-02T05:58:10.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:58:10.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU= 2022-09-02T05:58:10.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zRZwbpRn3ZiJTDso7S02yJDKPdU=', 'Content-Length': '0'} 2022-09-02T05:58:10.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008451d42a9f3b670d-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.698 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.699 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.699 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.700 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.700 DEBUG:boto:Method: GET 2022-09-02T05:58:10.701 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.701 DEBUG:boto:Data: 2022-09-02T05:58:10.702 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.703 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.703 DEBUG:boto:Params: {} 2022-09-02T05:58:10.704 DEBUG:boto:Token: None 2022-09-02T05:58:10.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af0f3047603e6dc-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.710 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.710 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.711 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.712 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.712 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.713 DEBUG:boto:Method: GET 2022-09-02T05:58:10.713 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.714 DEBUG:boto:Data: 2022-09-02T05:58:10.715 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.715 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.716 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.716 DEBUG:boto:Params: {} 2022-09-02T05:58:10.717 DEBUG:boto:Token: None 2022-09-02T05:58:10.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4212e97cc72aa04-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.723 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.723 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.724 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.725 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.725 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:10.726 DEBUG:boto:Data: 2022-09-02T05:58:10.726 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.727 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.728 DEBUG:boto:Params: {} 2022-09-02T05:58:10.729 DEBUG:boto:Token: None 2022-09-02T05:58:10.729 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55 2022-09-02T05:58:10.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rsZ8wzOERTn9vKQIM5K5/7a+2Tc= 2022-09-02T05:58:10.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rsZ8wzOERTn9vKQIM5K5/7a+2Tc=', 'Content-Length': '0'} 2022-09-02T05:58:10.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad5742a7c28c4bd-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.733 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.734 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.734 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.735 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:10.735 DEBUG:boto:Data: 2022-09-02T05:58:10.736 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.737 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.737 DEBUG:boto:Params: {} 2022-09-02T05:58:10.738 DEBUG:boto:Token: None 2022-09-02T05:58:10.739 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55 2022-09-02T05:58:10.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rsZ8wzOERTn9vKQIM5K5/7a+2Tc= 2022-09-02T05:58:10.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rsZ8wzOERTn9vKQIM5K5/7a+2Tc=', 'Content-Length': '0'} 2022-09-02T05:58:10.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f587ebcf06f8e4d2-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:58:10.744 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.744 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.745 DEBUG:boto:Method: GET 2022-09-02T05:58:10.745 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:10.746 DEBUG:boto:Data: 2022-09-02T05:58:10.747 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.748 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.749 DEBUG:boto:Params: {} 2022-09-02T05:58:10.749 DEBUG:boto:Token: None 2022-09-02T05:58:10.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55 2022-09-02T05:58:10.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4xPPDqPNsYnCJjhyjxN2UwrtXw= 2022-09-02T05:58:10.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4xPPDqPNsYnCJjhyjxN2UwrtXw=', 'Content-Length': '0'} 2022-09-02T05:58:10.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5b23b8739fe533b-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.754 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:58:10.755 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:58:10.756 DEBUG:boto:Method: GET 2022-09-02T05:58:10.756 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:58:10.757 DEBUG:boto:Data: 2022-09-02T05:58:10.757 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.758 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.758 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.759 DEBUG:boto:Params: {} 2022-09-02T05:58:10.760 DEBUG:boto:Token: None 2022-09-02T05:58:10.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj55 2022-09-02T05:58:10.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4xPPDqPNsYnCJjhyjxN2UwrtXw= 2022-09-02T05:58:10.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4xPPDqPNsYnCJjhyjxN2UwrtXw=', 'Content-Length': '0'} 2022-09-02T05:58:10.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054c62919e8539a4-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.806 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.807 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.807 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.808 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.809 DEBUG:boto:Method: GET 2022-09-02T05:58:10.809 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.810 DEBUG:boto:Data: 2022-09-02T05:58:10.810 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.811 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.812 DEBUG:boto:Params: {} 2022-09-02T05:58:10.812 DEBUG:boto:Token: None 2022-09-02T05:58:10.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a824f929864ff66-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.818 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.819 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.819 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.820 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.820 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.821 DEBUG:boto:Method: GET 2022-09-02T05:58:10.821 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:58:10.822 DEBUG:boto:Data: 2022-09-02T05:58:10.823 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.824 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.824 DEBUG:boto:Params: {} 2022-09-02T05:58:10.825 DEBUG:boto:Token: None 2022-09-02T05:58:10.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ccaa735e6264cf9-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.830 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:58:10.831 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.832 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.832 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.833 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.834 DEBUG:boto:Method: GET 2022-09-02T05:58:10.834 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.835 DEBUG:boto:Data: 2022-09-02T05:58:10.836 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.837 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.837 DEBUG:boto:Params: {} 2022-09-02T05:58:10.838 DEBUG:boto:Token: None 2022-09-02T05:58:10.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik= 2022-09-02T05:58:10.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik=', 'Content-Length': '0'} 2022-09-02T05:58:10.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007361e0226ffe3011-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.843 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.843 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.844 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.844 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.845 DEBUG:boto:Method: GET 2022-09-02T05:58:10.846 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.846 DEBUG:boto:Data: 2022-09-02T05:58:10.847 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.847 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.848 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.849 DEBUG:boto:Params: {} 2022-09-02T05:58:10.850 DEBUG:boto:Token: None 2022-09-02T05:58:10.850 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik= 2022-09-02T05:58:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik=', 'Content-Length': '0'} 2022-09-02T05:58:10.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a839cb6c33cfcaf6-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.855 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.855 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.856 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.857 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.857 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.858 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.858 DEBUG:boto:Data: 2022-09-02T05:58:10.859 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.860 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.860 DEBUG:boto:Params: {} 2022-09-02T05:58:10.861 DEBUG:boto:Token: None 2022-09-02T05:58:10.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Wck+cbaiREOOVB1Icvcw/hxPVY= 2022-09-02T05:58:10.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Wck+cbaiREOOVB1Icvcw/hxPVY=', 'Content-Length': '0'} 2022-09-02T05:58:10.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091fd02d6ab9c0a9-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.866 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.866 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.867 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.867 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.868 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.868 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.869 DEBUG:boto:Data: 2022-09-02T05:58:10.869 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.870 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.871 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.871 DEBUG:boto:Params: {} 2022-09-02T05:58:10.872 DEBUG:boto:Token: None 2022-09-02T05:58:10.872 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.873 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Wck+cbaiREOOVB1Icvcw/hxPVY= 2022-09-02T05:58:10.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Wck+cbaiREOOVB1Icvcw/hxPVY=', 'Content-Length': '0'} 2022-09-02T05:58:10.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045a8a204eda8b27e-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:58:10.877 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.878 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.878 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.879 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.879 DEBUG:boto:Method: GET 2022-09-02T05:58:10.880 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.880 DEBUG:boto:Data: 2022-09-02T05:58:10.881 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.882 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.882 DEBUG:boto:Params: {} 2022-09-02T05:58:10.883 DEBUG:boto:Token: None 2022-09-02T05:58:10.884 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.885 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik= 2022-09-02T05:58:10.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik=', 'Content-Length': '0'} 2022-09-02T05:58:10.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fdf261526ffce1f-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.890 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.891 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.892 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.893 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.894 DEBUG:boto:Method: GET 2022-09-02T05:58:10.895 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.896 DEBUG:boto:Data: 2022-09-02T05:58:10.897 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.899 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.900 DEBUG:boto:Params: {} 2022-09-02T05:58:10.902 DEBUG:boto:Token: None 2022-09-02T05:58:10.903 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:58:10.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik= 2022-09-02T05:58:10.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BCojtMGj51uYebGyXfNarpk7Bik=', 'Content-Length': '0'} 2022-09-02T05:58:10.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a51a6fbe40d53d76-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.909 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.910 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.911 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:10.912 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:10.913 DEBUG:boto:Method: GET 2022-09-02T05:58:10.914 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:10.915 DEBUG:boto:Data: 2022-09-02T05:58:10.916 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.918 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.919 DEBUG:boto:Params: {} 2022-09-02T05:58:10.920 DEBUG:boto:Token: None 2022-09-02T05:58:10.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001d8a517e51b02d5-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.928 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.929 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:10.930 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:10.931 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:10.932 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:10.933 DEBUG:boto:Method: GET 2022-09-02T05:58:10.934 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:10.935 DEBUG:boto:Data: 2022-09-02T05:58:10.936 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.938 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.939 DEBUG:boto:Params: {} 2022-09-02T05:58:10.940 DEBUG:boto:Token: None 2022-09-02T05:58:10.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/ 2022-09-02T05:58:10.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU= 2022-09-02T05:58:10.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HkAUws+vJjyESYab5C2j6S09WjU=', 'Content-Length': '0'} 2022-09-02T05:58:10.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db0461050ced0b96-0063119b72-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.945 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:10.946 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.947 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.947 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.948 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:10.949 DEBUG:boto:Data: 2022-09-02T05:58:10.950 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.951 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.951 DEBUG:boto:Params: {} 2022-09-02T05:58:10.952 DEBUG:boto:Token: None 2022-09-02T05:58:10.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57 2022-09-02T05:58:10.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e6tafHKcoroh8LshLOBj3WPlH5Q= 2022-09-02T05:58:10.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e6tafHKcoroh8LshLOBj3WPlH5Q=', 'Content-Length': '0'} 2022-09-02T05:58:10.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc770a0604b1c40-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.957 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.958 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.960 DEBUG:boto:Method: HEAD 2022-09-02T05:58:10.961 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:10.961 DEBUG:boto:Data: 2022-09-02T05:58:10.963 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.963 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:10.964 DEBUG:boto:Port: 8000 2022-09-02T05:58:10.965 DEBUG:boto:Params: {} 2022-09-02T05:58:10.967 DEBUG:boto:Token: None 2022-09-02T05:58:10.968 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57 2022-09-02T05:58:10.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:e6tafHKcoroh8LshLOBj3WPlH5Q= 2022-09-02T05:58:10.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:e6tafHKcoroh8LshLOBj3WPlH5Q=', 'Content-Length': '0'} 2022-09-02T05:58:10.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091fb672ed87b41de-0063119b72-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:10.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:58:10.975 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:10.976 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:10.977 DEBUG:boto:Method: GET 2022-09-02T05:58:10.979 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:10.980 DEBUG:boto:Data: 2022-09-02T05:58:10.980 DEBUG:boto:Headers: {} 2022-09-02T05:58:10.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:10.981 DEBUG:boto:Port: 8001 2022-09-02T05:58:10.982 DEBUG:boto:Params: {} 2022-09-02T05:58:10.982 DEBUG:boto:Token: None 2022-09-02T05:58:10.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:10 GMT /xljbkf-74/obj57 2022-09-02T05:58:10.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/PwxIuUfxkQR2Z/OPjtOjGiDgAY= 2022-09-02T05:58:10.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:10 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/PwxIuUfxkQR2Z/OPjtOjGiDgAY=', 'Content-Length': '0'} 2022-09-02T05:58:10.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081b829e1bcacd048-0063119b72-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:10 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.030 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:58:11.031 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:58:11.032 DEBUG:boto:Method: GET 2022-09-02T05:58:11.033 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:58:11.033 DEBUG:boto:Data: 2022-09-02T05:58:11.034 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.036 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.037 DEBUG:boto:Params: {} 2022-09-02T05:58:11.038 DEBUG:boto:Token: None 2022-09-02T05:58:11.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj57 2022-09-02T05:58:11.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uHfxuP2IM/YgrYx5aUijUNkdQwg= 2022-09-02T05:58:11.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uHfxuP2IM/YgrYx5aUijUNkdQwg=', 'Content-Length': '0'} 2022-09-02T05:58:11.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003124a32674149255-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.045 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.045 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.046 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:11.047 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:11.047 DEBUG:boto:Method: GET 2022-09-02T05:58:11.048 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:11.048 DEBUG:boto:Data: 2022-09-02T05:58:11.049 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.050 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.051 DEBUG:boto:Params: {} 2022-09-02T05:58:11.051 DEBUG:boto:Token: None 2022-09-02T05:58:11.052 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e7a5e2b0d90f2fe-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.057 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.058 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.059 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.060 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:11.060 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:58:11.061 DEBUG:boto:Method: GET 2022-09-02T05:58:11.061 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:58:11.062 DEBUG:boto:Data: 2022-09-02T05:58:11.063 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.063 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.064 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.064 DEBUG:boto:Params: {} 2022-09-02T05:58:11.065 DEBUG:boto:Token: None 2022-09-02T05:58:11.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.066 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000affff98cdcb40b7a-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.070 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:58:11.072 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.072 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.073 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.073 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.074 DEBUG:boto:Method: GET 2022-09-02T05:58:11.075 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.075 DEBUG:boto:Data: 2022-09-02T05:58:11.076 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.077 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.077 DEBUG:boto:Params: {} 2022-09-02T05:58:11.078 DEBUG:boto:Token: None 2022-09-02T05:58:11.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA= 2022-09-02T05:58:11.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA=', 'Content-Length': '0'} 2022-09-02T05:58:11.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b675c64502986e7-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.083 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.084 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.084 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.085 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.085 DEBUG:boto:Method: GET 2022-09-02T05:58:11.086 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.087 DEBUG:boto:Data: 2022-09-02T05:58:11.087 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.089 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.090 DEBUG:boto:Params: {} 2022-09-02T05:58:11.091 DEBUG:boto:Token: None 2022-09-02T05:58:11.092 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA= 2022-09-02T05:58:11.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA=', 'Content-Length': '0'} 2022-09-02T05:58:11.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cf4a865b0e655d7-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.096 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.097 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.098 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.099 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.100 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.101 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.101 DEBUG:boto:Data: 2022-09-02T05:58:11.102 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.104 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.104 DEBUG:boto:Params: {} 2022-09-02T05:58:11.105 DEBUG:boto:Token: None 2022-09-02T05:58:11.105 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mdDdpYphtfpLUP7GjeJuLWB1kxQ= 2022-09-02T05:58:11.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mdDdpYphtfpLUP7GjeJuLWB1kxQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041afc71aefb36721-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.111 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.111 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.112 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.112 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.113 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.113 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.114 DEBUG:boto:Data: 2022-09-02T05:58:11.114 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.115 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.115 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.116 DEBUG:boto:Params: {} 2022-09-02T05:58:11.117 DEBUG:boto:Token: None 2022-09-02T05:58:11.118 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mdDdpYphtfpLUP7GjeJuLWB1kxQ= 2022-09-02T05:58:11.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mdDdpYphtfpLUP7GjeJuLWB1kxQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349d70cd463c7d2b-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:58:11.122 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.123 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.124 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.124 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.125 DEBUG:boto:Method: GET 2022-09-02T05:58:11.126 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.126 DEBUG:boto:Data: 2022-09-02T05:58:11.127 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.129 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.130 DEBUG:boto:Params: {} 2022-09-02T05:58:11.131 DEBUG:boto:Token: None 2022-09-02T05:58:11.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA= 2022-09-02T05:58:11.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA=', 'Content-Length': '0'} 2022-09-02T05:58:11.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6df09891e5d3914-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.138 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.139 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.140 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.140 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.141 DEBUG:boto:Method: GET 2022-09-02T05:58:11.142 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.142 DEBUG:boto:Data: 2022-09-02T05:58:11.143 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.144 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.144 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.145 DEBUG:boto:Params: {} 2022-09-02T05:58:11.146 DEBUG:boto:Token: None 2022-09-02T05:58:11.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:58:11.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA= 2022-09-02T05:58:11.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lqRQHNsHDC6XwG+Z2QMMgCXCoYA=', 'Content-Length': '0'} 2022-09-02T05:58:11.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916285c070ee8bd0-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.151 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.151 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.152 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.152 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.153 DEBUG:boto:Method: GET 2022-09-02T05:58:11.154 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.154 DEBUG:boto:Data: 2022-09-02T05:58:11.155 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.156 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.157 DEBUG:boto:Params: {} 2022-09-02T05:58:11.157 DEBUG:boto:Token: None 2022-09-02T05:58:11.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f74c893b265ced80-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.165 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.166 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.167 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.168 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.168 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.169 DEBUG:boto:Method: GET 2022-09-02T05:58:11.169 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.170 DEBUG:boto:Data: 2022-09-02T05:58:11.171 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.172 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.173 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.173 DEBUG:boto:Params: {} 2022-09-02T05:58:11.174 DEBUG:boto:Token: None 2022-09-02T05:58:11.175 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000babea43a89283526-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.222 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.223 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.224 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.225 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.225 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:11.226 DEBUG:boto:Data: 2022-09-02T05:58:11.226 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.227 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.228 DEBUG:boto:Params: {} 2022-09-02T05:58:11.228 DEBUG:boto:Token: None 2022-09-02T05:58:11.229 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58 2022-09-02T05:58:11.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WhHvPcxejylwKVNQme+dE9t85lg= 2022-09-02T05:58:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WhHvPcxejylwKVNQme+dE9t85lg=', 'Content-Length': '0'} 2022-09-02T05:58:11.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043a4c6f0df72b12-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.234 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.234 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.235 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.235 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:11.236 DEBUG:boto:Data: 2022-09-02T05:58:11.236 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.237 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.238 DEBUG:boto:Params: {} 2022-09-02T05:58:11.238 DEBUG:boto:Token: None 2022-09-02T05:58:11.239 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58 2022-09-02T05:58:11.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WhHvPcxejylwKVNQme+dE9t85lg= 2022-09-02T05:58:11.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WhHvPcxejylwKVNQme+dE9t85lg=', 'Content-Length': '0'} 2022-09-02T05:58:11.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824bfe460d17a36d-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:58:11.244 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.245 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.245 DEBUG:boto:Method: GET 2022-09-02T05:58:11.246 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:11.246 DEBUG:boto:Data: 2022-09-02T05:58:11.247 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.248 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.248 DEBUG:boto:Params: {} 2022-09-02T05:58:11.249 DEBUG:boto:Token: None 2022-09-02T05:58:11.250 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58 2022-09-02T05:58:11.251 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xJHcrQc2ELwkkbkZxmHHYoObXAU= 2022-09-02T05:58:11.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xJHcrQc2ELwkkbkZxmHHYoObXAU=', 'Content-Length': '0'} 2022-09-02T05:58:11.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e64d1d16084427a-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.254 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:58:11.256 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:58:11.256 DEBUG:boto:Method: GET 2022-09-02T05:58:11.257 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:58:11.259 DEBUG:boto:Data: 2022-09-02T05:58:11.260 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.261 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.262 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.263 DEBUG:boto:Params: {} 2022-09-02T05:58:11.264 DEBUG:boto:Token: None 2022-09-02T05:58:11.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj58 2022-09-02T05:58:11.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xJHcrQc2ELwkkbkZxmHHYoObXAU= 2022-09-02T05:58:11.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xJHcrQc2ELwkkbkZxmHHYoObXAU=', 'Content-Length': '0'} 2022-09-02T05:58:11.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248508f7a20be6cc-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.272 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.273 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.274 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.275 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.276 DEBUG:boto:Method: GET 2022-09-02T05:58:11.277 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.278 DEBUG:boto:Data: 2022-09-02T05:58:11.279 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.281 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.282 DEBUG:boto:Params: {} 2022-09-02T05:58:11.283 DEBUG:boto:Token: None 2022-09-02T05:58:11.284 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed590b34ac7ce044-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.292 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.293 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.293 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.294 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.294 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.295 DEBUG:boto:Method: GET 2022-09-02T05:58:11.296 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:58:11.296 DEBUG:boto:Data: 2022-09-02T05:58:11.297 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.298 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.298 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.299 DEBUG:boto:Params: {} 2022-09-02T05:58:11.299 DEBUG:boto:Token: None 2022-09-02T05:58:11.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.300 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8c278876be7fef7-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.346 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:58:11.348 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.348 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.349 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.350 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.351 DEBUG:boto:Method: GET 2022-09-02T05:58:11.352 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.353 DEBUG:boto:Data: 2022-09-02T05:58:11.355 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.357 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.358 DEBUG:boto:Params: {} 2022-09-02T05:58:11.359 DEBUG:boto:Token: None 2022-09-02T05:58:11.360 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M= 2022-09-02T05:58:11.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M=', 'Content-Length': '0'} 2022-09-02T05:58:11.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015d888541e5c87e-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.379 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.381 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.382 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.383 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.384 DEBUG:boto:Method: GET 2022-09-02T05:58:11.385 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.386 DEBUG:boto:Data: 2022-09-02T05:58:11.387 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.389 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.390 DEBUG:boto:Params: {} 2022-09-02T05:58:11.391 DEBUG:boto:Token: None 2022-09-02T05:58:11.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M= 2022-09-02T05:58:11.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M=', 'Content-Length': '0'} 2022-09-02T05:58:11.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000980a625843561e94-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.401 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.402 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.403 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.404 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.405 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.406 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.407 DEBUG:boto:Data: 2022-09-02T05:58:11.408 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.410 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.411 DEBUG:boto:Params: {} 2022-09-02T05:58:11.412 DEBUG:boto:Token: None 2022-09-02T05:58:11.413 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JC4N5TOBaIoTqVyBLXBgp1RMx6s= 2022-09-02T05:58:11.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JC4N5TOBaIoTqVyBLXBgp1RMx6s=', 'Content-Length': '0'} 2022-09-02T05:58:11.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003149fca09ba9ddbf-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.420 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.421 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.422 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.423 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.424 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.425 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.426 DEBUG:boto:Data: 2022-09-02T05:58:11.426 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.427 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.428 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.428 DEBUG:boto:Params: {} 2022-09-02T05:58:11.429 DEBUG:boto:Token: None 2022-09-02T05:58:11.429 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.430 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JC4N5TOBaIoTqVyBLXBgp1RMx6s= 2022-09-02T05:58:11.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JC4N5TOBaIoTqVyBLXBgp1RMx6s=', 'Content-Length': '0'} 2022-09-02T05:58:11.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000631cddddebe2286e-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:58:11.435 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.435 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.436 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.437 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.437 DEBUG:boto:Method: GET 2022-09-02T05:58:11.438 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.438 DEBUG:boto:Data: 2022-09-02T05:58:11.439 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.440 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.440 DEBUG:boto:Params: {} 2022-09-02T05:58:11.441 DEBUG:boto:Token: None 2022-09-02T05:58:11.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M= 2022-09-02T05:58:11.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M=', 'Content-Length': '0'} 2022-09-02T05:58:11.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f660cbd8c0b518-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.486 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.487 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.488 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.488 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.489 DEBUG:boto:Method: GET 2022-09-02T05:58:11.489 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.490 DEBUG:boto:Data: 2022-09-02T05:58:11.490 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.491 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.491 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.492 DEBUG:boto:Params: {} 2022-09-02T05:58:11.493 DEBUG:boto:Token: None 2022-09-02T05:58:11.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:58:11.494 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M= 2022-09-02T05:58:11.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wJlIBkRpbzaNZODfdSIRWwHnE+M=', 'Content-Length': '0'} 2022-09-02T05:58:11.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2a1cacdcdb9d27-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.498 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.499 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.499 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.500 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.500 DEBUG:boto:Method: GET 2022-09-02T05:58:11.501 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.501 DEBUG:boto:Data: 2022-09-02T05:58:11.502 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.503 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.504 DEBUG:boto:Params: {} 2022-09-02T05:58:11.504 DEBUG:boto:Token: None 2022-09-02T05:58:11.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4fc05a2a7be7ee4-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.509 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.510 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.511 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.512 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.512 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.513 DEBUG:boto:Method: GET 2022-09-02T05:58:11.513 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.514 DEBUG:boto:Data: 2022-09-02T05:58:11.514 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.515 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.516 DEBUG:boto:Params: {} 2022-09-02T05:58:11.517 DEBUG:boto:Token: None 2022-09-02T05:58:11.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000851700d73e169734-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.522 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.523 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.523 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.524 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.524 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:11.525 DEBUG:boto:Data: 2022-09-02T05:58:11.525 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.527 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.527 DEBUG:boto:Params: {} 2022-09-02T05:58:11.528 DEBUG:boto:Token: None 2022-09-02T05:58:11.528 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62 2022-09-02T05:58:11.529 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UxiWXik+uZq+8n+N1rdvLW5vguU= 2022-09-02T05:58:11.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UxiWXik+uZq+8n+N1rdvLW5vguU=', 'Content-Length': '0'} 2022-09-02T05:58:11.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a45fd8a80f954e5b-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.533 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.533 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.534 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.534 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:11.535 DEBUG:boto:Data: 2022-09-02T05:58:11.535 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.536 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.536 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.537 DEBUG:boto:Params: {} 2022-09-02T05:58:11.537 DEBUG:boto:Token: None 2022-09-02T05:58:11.538 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62 2022-09-02T05:58:11.539 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UxiWXik+uZq+8n+N1rdvLW5vguU= 2022-09-02T05:58:11.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UxiWXik+uZq+8n+N1rdvLW5vguU=', 'Content-Length': '0'} 2022-09-02T05:58:11.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ed45594112e28e3-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:58:11.543 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.543 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.544 DEBUG:boto:Method: GET 2022-09-02T05:58:11.544 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:11.545 DEBUG:boto:Data: 2022-09-02T05:58:11.545 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.546 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.547 DEBUG:boto:Params: {} 2022-09-02T05:58:11.548 DEBUG:boto:Token: None 2022-09-02T05:58:11.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62 2022-09-02T05:58:11.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O6FgsfH1s41n0Nmi0S63E3n7SeY= 2022-09-02T05:58:11.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O6FgsfH1s41n0Nmi0S63E3n7SeY=', 'Content-Length': '0'} 2022-09-02T05:58:11.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95c24293b278269-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.552 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:58:11.553 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:58:11.554 DEBUG:boto:Method: GET 2022-09-02T05:58:11.554 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:58:11.555 DEBUG:boto:Data: 2022-09-02T05:58:11.555 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.556 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.557 DEBUG:boto:Params: {} 2022-09-02T05:58:11.557 DEBUG:boto:Token: None 2022-09-02T05:58:11.558 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj62 2022-09-02T05:58:11.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O6FgsfH1s41n0Nmi0S63E3n7SeY= 2022-09-02T05:58:11.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O6FgsfH1s41n0Nmi0S63E3n7SeY=', 'Content-Length': '0'} 2022-09-02T05:58:11.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cfc17de81167341-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.562 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.563 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.563 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.564 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.564 DEBUG:boto:Method: GET 2022-09-02T05:58:11.565 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.565 DEBUG:boto:Data: 2022-09-02T05:58:11.566 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.567 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.567 DEBUG:boto:Params: {} 2022-09-02T05:58:11.568 DEBUG:boto:Token: None 2022-09-02T05:58:11.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f536ce05ed614d4-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.573 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.574 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.575 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.575 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.576 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.576 DEBUG:boto:Method: GET 2022-09-02T05:58:11.577 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:58:11.577 DEBUG:boto:Data: 2022-09-02T05:58:11.578 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.579 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.579 DEBUG:boto:Params: {} 2022-09-02T05:58:11.580 DEBUG:boto:Token: None 2022-09-02T05:58:11.581 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df0fbf078951c59b-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.585 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:58:11.587 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.587 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.588 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.589 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.589 DEBUG:boto:Method: GET 2022-09-02T05:58:11.590 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.590 DEBUG:boto:Data: 2022-09-02T05:58:11.591 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.592 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.592 DEBUG:boto:Params: {} 2022-09-02T05:58:11.593 DEBUG:boto:Token: None 2022-09-02T05:58:11.594 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg= 2022-09-02T05:58:11.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg=', 'Content-Length': '0'} 2022-09-02T05:58:11.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef01189c25ecb862-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.598 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.598 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.599 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.599 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.600 DEBUG:boto:Method: GET 2022-09-02T05:58:11.600 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.601 DEBUG:boto:Data: 2022-09-02T05:58:11.601 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.603 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.603 DEBUG:boto:Params: {} 2022-09-02T05:58:11.604 DEBUG:boto:Token: None 2022-09-02T05:58:11.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg= 2022-09-02T05:58:11.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg=', 'Content-Length': '0'} 2022-09-02T05:58:11.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7f67482eb1c997-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.608 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.609 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.609 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.610 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.610 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.611 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.612 DEBUG:boto:Data: 2022-09-02T05:58:11.612 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.613 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.614 DEBUG:boto:Params: {} 2022-09-02T05:58:11.614 DEBUG:boto:Token: None 2022-09-02T05:58:11.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VdCgGJXa4p7VVgxPmMb0Yh5LDa8= 2022-09-02T05:58:11.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VdCgGJXa4p7VVgxPmMb0Yh5LDa8=', 'Content-Length': '0'} 2022-09-02T05:58:11.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb46600aa581380f-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.619 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.620 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.620 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.621 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.621 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.622 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.622 DEBUG:boto:Data: 2022-09-02T05:58:11.623 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.623 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.624 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.624 DEBUG:boto:Params: {} 2022-09-02T05:58:11.625 DEBUG:boto:Token: None 2022-09-02T05:58:11.626 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VdCgGJXa4p7VVgxPmMb0Yh5LDa8= 2022-09-02T05:58:11.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VdCgGJXa4p7VVgxPmMb0Yh5LDa8=', 'Content-Length': '0'} 2022-09-02T05:58:11.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc0f59a4b755ff8-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:58:11.630 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.631 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.631 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.632 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.633 DEBUG:boto:Method: GET 2022-09-02T05:58:11.633 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.634 DEBUG:boto:Data: 2022-09-02T05:58:11.634 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.635 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.636 DEBUG:boto:Params: {} 2022-09-02T05:58:11.637 DEBUG:boto:Token: None 2022-09-02T05:58:11.637 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg= 2022-09-02T05:58:11.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg=', 'Content-Length': '0'} 2022-09-02T05:58:11.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000610a6c526272f17b-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.643 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.643 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.644 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.645 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.646 DEBUG:boto:Method: GET 2022-09-02T05:58:11.647 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.648 DEBUG:boto:Data: 2022-09-02T05:58:11.649 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.651 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.652 DEBUG:boto:Params: {} 2022-09-02T05:58:11.653 DEBUG:boto:Token: None 2022-09-02T05:58:11.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:58:11.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg= 2022-09-02T05:58:11.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ze8+0QJzNWkJRQ7KtlL4zxDdjkg=', 'Content-Length': '0'} 2022-09-02T05:58:11.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cbc3be5e110c414-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.660 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.661 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.662 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.662 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.663 DEBUG:boto:Method: GET 2022-09-02T05:58:11.664 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.664 DEBUG:boto:Data: 2022-09-02T05:58:11.665 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.666 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.667 DEBUG:boto:Params: {} 2022-09-02T05:58:11.667 DEBUG:boto:Token: None 2022-09-02T05:58:11.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.669 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c0f7a40080a821e-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.673 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.674 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.675 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.676 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.676 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.677 DEBUG:boto:Method: GET 2022-09-02T05:58:11.677 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.678 DEBUG:boto:Data: 2022-09-02T05:58:11.679 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.680 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.680 DEBUG:boto:Params: {} 2022-09-02T05:58:11.681 DEBUG:boto:Token: None 2022-09-02T05:58:11.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d10c0a3b98efbe-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.686 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.687 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.688 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.688 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.689 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:11.689 DEBUG:boto:Data: 2022-09-02T05:58:11.690 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.691 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.691 DEBUG:boto:Params: {} 2022-09-02T05:58:11.692 DEBUG:boto:Token: None 2022-09-02T05:58:11.692 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68 2022-09-02T05:58:11.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:seT2eGYqN6IXCVfsYmxwKuTgX64= 2022-09-02T05:58:11.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:seT2eGYqN6IXCVfsYmxwKuTgX64=', 'Content-Length': '0'} 2022-09-02T05:58:11.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e141f679deec3d-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.696 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.697 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.698 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.698 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:11.699 DEBUG:boto:Data: 2022-09-02T05:58:11.699 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.700 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.700 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.701 DEBUG:boto:Params: {} 2022-09-02T05:58:11.701 DEBUG:boto:Token: None 2022-09-02T05:58:11.702 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68 2022-09-02T05:58:11.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:seT2eGYqN6IXCVfsYmxwKuTgX64= 2022-09-02T05:58:11.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:seT2eGYqN6IXCVfsYmxwKuTgX64=', 'Content-Length': '0'} 2022-09-02T05:58:11.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b414f6f5c8720b04-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:58:11.707 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.708 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.708 DEBUG:boto:Method: GET 2022-09-02T05:58:11.709 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:11.709 DEBUG:boto:Data: 2022-09-02T05:58:11.710 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.711 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.712 DEBUG:boto:Params: {} 2022-09-02T05:58:11.712 DEBUG:boto:Token: None 2022-09-02T05:58:11.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68 2022-09-02T05:58:11.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oEvdURCbQqkq3slaeICbq6SW+2Y= 2022-09-02T05:58:11.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oEvdURCbQqkq3slaeICbq6SW+2Y=', 'Content-Length': '0'} 2022-09-02T05:58:11.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41ed1455602db21-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.717 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:58:11.718 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:58:11.718 DEBUG:boto:Method: GET 2022-09-02T05:58:11.719 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:58:11.720 DEBUG:boto:Data: 2022-09-02T05:58:11.720 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.721 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.722 DEBUG:boto:Params: {} 2022-09-02T05:58:11.722 DEBUG:boto:Token: None 2022-09-02T05:58:11.723 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj68 2022-09-02T05:58:11.723 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oEvdURCbQqkq3slaeICbq6SW+2Y= 2022-09-02T05:58:11.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oEvdURCbQqkq3slaeICbq6SW+2Y=', 'Content-Length': '0'} 2022-09-02T05:58:11.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fabd52d15d95e2-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.727 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.727 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.728 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.728 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.729 DEBUG:boto:Method: GET 2022-09-02T05:58:11.729 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.730 DEBUG:boto:Data: 2022-09-02T05:58:11.731 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.732 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.732 DEBUG:boto:Params: {} 2022-09-02T05:58:11.733 DEBUG:boto:Token: None 2022-09-02T05:58:11.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf53a96b1dfa9fb-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.738 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.739 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.740 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.741 DEBUG:boto:Method: GET 2022-09-02T05:58:11.742 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:58:11.742 DEBUG:boto:Data: 2022-09-02T05:58:11.743 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.744 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.744 DEBUG:boto:Params: {} 2022-09-02T05:58:11.745 DEBUG:boto:Token: None 2022-09-02T05:58:11.745 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000768e4ba695d9d37a-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.752 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:58:11.753 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.754 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.754 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.755 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.756 DEBUG:boto:Method: GET 2022-09-02T05:58:11.756 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.757 DEBUG:boto:Data: 2022-09-02T05:58:11.757 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.758 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.759 DEBUG:boto:Params: {} 2022-09-02T05:58:11.760 DEBUG:boto:Token: None 2022-09-02T05:58:11.760 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg= 2022-09-02T05:58:11.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg=', 'Content-Length': '0'} 2022-09-02T05:58:11.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c882cd85e4fe58e5-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.764 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.765 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.765 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.766 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.766 DEBUG:boto:Method: GET 2022-09-02T05:58:11.767 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.767 DEBUG:boto:Data: 2022-09-02T05:58:11.768 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.769 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.769 DEBUG:boto:Params: {} 2022-09-02T05:58:11.770 DEBUG:boto:Token: None 2022-09-02T05:58:11.771 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg= 2022-09-02T05:58:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg=', 'Content-Length': '0'} 2022-09-02T05:58:11.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614d2c2b1dea717a-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.818 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.818 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.819 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.820 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.820 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.820 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.821 DEBUG:boto:Data: 2022-09-02T05:58:11.821 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.822 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.823 DEBUG:boto:Params: {} 2022-09-02T05:58:11.823 DEBUG:boto:Token: None 2022-09-02T05:58:11.824 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gd8J23CuegYUQgsjBtjmJ/bWVms= 2022-09-02T05:58:11.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gd8J23CuegYUQgsjBtjmJ/bWVms=', 'Content-Length': '0'} 2022-09-02T05:58:11.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a9bc409268c96f-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.828 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.829 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.829 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.830 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.830 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.831 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.831 DEBUG:boto:Data: 2022-09-02T05:58:11.832 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.833 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.833 DEBUG:boto:Params: {} 2022-09-02T05:58:11.834 DEBUG:boto:Token: None 2022-09-02T05:58:11.835 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Gd8J23CuegYUQgsjBtjmJ/bWVms= 2022-09-02T05:58:11.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Gd8J23CuegYUQgsjBtjmJ/bWVms=', 'Content-Length': '0'} 2022-09-02T05:58:11.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2abd34ad2de46c-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:58:11.839 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.840 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.840 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.841 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.841 DEBUG:boto:Method: GET 2022-09-02T05:58:11.842 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.842 DEBUG:boto:Data: 2022-09-02T05:58:11.843 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.844 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.844 DEBUG:boto:Params: {} 2022-09-02T05:58:11.845 DEBUG:boto:Token: None 2022-09-02T05:58:11.845 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg= 2022-09-02T05:58:11.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg=', 'Content-Length': '0'} 2022-09-02T05:58:11.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000174ad80a0832bc9-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.849 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.850 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.851 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.851 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.852 DEBUG:boto:Method: GET 2022-09-02T05:58:11.852 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.853 DEBUG:boto:Data: 2022-09-02T05:58:11.853 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.854 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.854 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.855 DEBUG:boto:Params: {} 2022-09-02T05:58:11.855 DEBUG:boto:Token: None 2022-09-02T05:58:11.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:58:11.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg= 2022-09-02T05:58:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4QITwQXggILzEhlcT1iuGN9mjOg=', 'Content-Length': '0'} 2022-09-02T05:58:11.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0ff607db849d64-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.861 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.862 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.862 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.863 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.864 DEBUG:boto:Method: GET 2022-09-02T05:58:11.864 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.865 DEBUG:boto:Data: 2022-09-02T05:58:11.865 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.867 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.867 DEBUG:boto:Params: {} 2022-09-02T05:58:11.868 DEBUG:boto:Token: None 2022-09-02T05:58:11.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f0264ed1057cb25-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.874 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.875 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.876 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.876 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.877 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.877 DEBUG:boto:Method: GET 2022-09-02T05:58:11.878 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.879 DEBUG:boto:Data: 2022-09-02T05:58:11.880 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.881 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.882 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.883 DEBUG:boto:Params: {} 2022-09-02T05:58:11.884 DEBUG:boto:Token: None 2022-09-02T05:58:11.885 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.886 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2daea74394427d3-0063119b73-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.891 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.892 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.893 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.893 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.894 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:11.894 DEBUG:boto:Data: 2022-09-02T05:58:11.895 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.896 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.896 DEBUG:boto:Params: {} 2022-09-02T05:58:11.897 DEBUG:boto:Token: None 2022-09-02T05:58:11.898 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69 2022-09-02T05:58:11.898 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A587jU39pgXS8/9MjR4Qs7W9WKU= 2022-09-02T05:58:11.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A587jU39pgXS8/9MjR4Qs7W9WKU=', 'Content-Length': '0'} 2022-09-02T05:58:11.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e1623e7c7031f6-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.902 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.902 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.903 DEBUG:boto:Method: HEAD 2022-09-02T05:58:11.903 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:11.904 DEBUG:boto:Data: 2022-09-02T05:58:11.904 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.905 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.905 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.906 DEBUG:boto:Params: {} 2022-09-02T05:58:11.906 DEBUG:boto:Token: None 2022-09-02T05:58:11.907 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69 2022-09-02T05:58:11.908 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A587jU39pgXS8/9MjR4Qs7W9WKU= 2022-09-02T05:58:11.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A587jU39pgXS8/9MjR4Qs7W9WKU=', 'Content-Length': '0'} 2022-09-02T05:58:11.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90e85c4eca944f4-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:58:11.912 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.913 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.913 DEBUG:boto:Method: GET 2022-09-02T05:58:11.914 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:11.915 DEBUG:boto:Data: 2022-09-02T05:58:11.915 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.916 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.917 DEBUG:boto:Params: {} 2022-09-02T05:58:11.917 DEBUG:boto:Token: None 2022-09-02T05:58:11.918 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69 2022-09-02T05:58:11.919 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iZwnE0xnLP/WpP7LqApAPfI/uQ0= 2022-09-02T05:58:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iZwnE0xnLP/WpP7LqApAPfI/uQ0=', 'Content-Length': '0'} 2022-09-02T05:58:11.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9753b5d1972652-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.923 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:58:11.923 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:58:11.924 DEBUG:boto:Method: GET 2022-09-02T05:58:11.925 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:58:11.925 DEBUG:boto:Data: 2022-09-02T05:58:11.926 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.926 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.927 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.927 DEBUG:boto:Params: {} 2022-09-02T05:58:11.928 DEBUG:boto:Token: None 2022-09-02T05:58:11.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/obj69 2022-09-02T05:58:11.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iZwnE0xnLP/WpP7LqApAPfI/uQ0= 2022-09-02T05:58:11.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iZwnE0xnLP/WpP7LqApAPfI/uQ0=', 'Content-Length': '0'} 2022-09-02T05:58:11.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830e1dcf1b57dcce-0063119b73-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.974 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.975 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.975 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.976 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.977 DEBUG:boto:Method: GET 2022-09-02T05:58:11.977 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.978 DEBUG:boto:Data: 2022-09-02T05:58:11.978 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:11.980 DEBUG:boto:Port: 8001 2022-09-02T05:58:11.981 DEBUG:boto:Params: {} 2022-09-02T05:58:11.981 DEBUG:boto:Token: None 2022-09-02T05:58:11.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:11.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:11.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b627b888faf4f2a-0063119b73-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:11 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:11.987 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:11.988 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:11.988 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:11.989 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.989 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.990 DEBUG:boto:Method: GET 2022-09-02T05:58:11.990 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:58:11.991 DEBUG:boto:Data: 2022-09-02T05:58:11.992 DEBUG:boto:Headers: {} 2022-09-02T05:58:11.992 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:11.993 DEBUG:boto:Port: 8000 2022-09-02T05:58:11.993 DEBUG:boto:Params: {} 2022-09-02T05:58:11.994 DEBUG:boto:Token: None 2022-09-02T05:58:11.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:11 GMT /xljbkf-74/ 2022-09-02T05:58:12.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ= 2022-09-02T05:58:12.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:11 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:djU7L2LwhdZrNK+pywIvj1PSTFQ=', 'Content-Length': '0'} 2022-09-02T05:58:12.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067db75d2d2bc791e-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.006 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:58:12.007 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.008 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.010 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.011 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.012 DEBUG:boto:Method: GET 2022-09-02T05:58:12.012 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.014 DEBUG:boto:Data: 2022-09-02T05:58:12.014 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.016 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.016 DEBUG:boto:Params: {} 2022-09-02T05:58:12.017 DEBUG:boto:Token: None 2022-09-02T05:58:12.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.018 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8= 2022-09-02T05:58:12.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8=', 'Content-Length': '0'} 2022-09-02T05:58:12.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f9affbe73c87ca-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.062 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.064 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.065 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.067 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.068 DEBUG:boto:Method: GET 2022-09-02T05:58:12.069 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.070 DEBUG:boto:Data: 2022-09-02T05:58:12.071 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.072 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.073 DEBUG:boto:Params: {} 2022-09-02T05:58:12.074 DEBUG:boto:Token: None 2022-09-02T05:58:12.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8= 2022-09-02T05:58:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8=', 'Content-Length': '0'} 2022-09-02T05:58:12.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd12e607f32ff927-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.080 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.081 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.082 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.082 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.083 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.083 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.084 DEBUG:boto:Data: 2022-09-02T05:58:12.085 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.086 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.086 DEBUG:boto:Params: {} 2022-09-02T05:58:12.087 DEBUG:boto:Token: None 2022-09-02T05:58:12.088 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.088 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pKeXxEkIBCRZUyIkyKiNMYCbwZI= 2022-09-02T05:58:12.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pKeXxEkIBCRZUyIkyKiNMYCbwZI=', 'Content-Length': '0'} 2022-09-02T05:58:12.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e426b0c5bfc1074-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.091 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.092 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.093 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.094 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.094 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.095 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.095 DEBUG:boto:Data: 2022-09-02T05:58:12.096 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.096 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.097 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.097 DEBUG:boto:Params: {} 2022-09-02T05:58:12.098 DEBUG:boto:Token: None 2022-09-02T05:58:12.099 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pKeXxEkIBCRZUyIkyKiNMYCbwZI= 2022-09-02T05:58:12.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pKeXxEkIBCRZUyIkyKiNMYCbwZI=', 'Content-Length': '0'} 2022-09-02T05:58:12.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed11cd6aa351ee1-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:58:12.103 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.104 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.105 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.105 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.106 DEBUG:boto:Method: GET 2022-09-02T05:58:12.106 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.107 DEBUG:boto:Data: 2022-09-02T05:58:12.108 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.109 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.109 DEBUG:boto:Params: {} 2022-09-02T05:58:12.110 DEBUG:boto:Token: None 2022-09-02T05:58:12.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8= 2022-09-02T05:58:12.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8=', 'Content-Length': '0'} 2022-09-02T05:58:12.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020653e7eac3f7097-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.115 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.116 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.116 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.117 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.117 DEBUG:boto:Method: GET 2022-09-02T05:58:12.118 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.118 DEBUG:boto:Data: 2022-09-02T05:58:12.119 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.119 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.120 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.120 DEBUG:boto:Params: {} 2022-09-02T05:58:12.121 DEBUG:boto:Token: None 2022-09-02T05:58:12.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:58:12.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8= 2022-09-02T05:58:12.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R/tENMop4Fma/3m2Gp06WE0yin8=', 'Content-Length': '0'} 2022-09-02T05:58:12.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d35a897c6ca9ead-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.126 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.126 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.127 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.127 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.128 DEBUG:boto:Method: GET 2022-09-02T05:58:12.128 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.129 DEBUG:boto:Data: 2022-09-02T05:58:12.129 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.131 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.131 DEBUG:boto:Params: {} 2022-09-02T05:58:12.132 DEBUG:boto:Token: None 2022-09-02T05:58:12.132 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b7cdd25f75f85d-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.178 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.179 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.180 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.180 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.181 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.181 DEBUG:boto:Method: GET 2022-09-02T05:58:12.182 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.183 DEBUG:boto:Data: 2022-09-02T05:58:12.183 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.184 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.184 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.185 DEBUG:boto:Params: {} 2022-09-02T05:58:12.186 DEBUG:boto:Token: None 2022-09-02T05:58:12.186 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.187 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dea6eccbc775f01-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.191 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.192 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.193 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.193 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.194 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:12.194 DEBUG:boto:Data: 2022-09-02T05:58:12.195 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.196 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.196 DEBUG:boto:Params: {} 2022-09-02T05:58:12.197 DEBUG:boto:Token: None 2022-09-02T05:58:12.197 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76 2022-09-02T05:58:12.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RJywLv7+NmbCUujK34VSXxXGv7Y= 2022-09-02T05:58:12.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RJywLv7+NmbCUujK34VSXxXGv7Y=', 'Content-Length': '0'} 2022-09-02T05:58:12.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c92c267bbe98426-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.201 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.202 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.203 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.203 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:12.204 DEBUG:boto:Data: 2022-09-02T05:58:12.204 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.205 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.206 DEBUG:boto:Params: {} 2022-09-02T05:58:12.206 DEBUG:boto:Token: None 2022-09-02T05:58:12.207 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76 2022-09-02T05:58:12.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RJywLv7+NmbCUujK34VSXxXGv7Y= 2022-09-02T05:58:12.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RJywLv7+NmbCUujK34VSXxXGv7Y=', 'Content-Length': '0'} 2022-09-02T05:58:12.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7c4f20246c8ba7-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:58:12.211 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.212 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.212 DEBUG:boto:Method: GET 2022-09-02T05:58:12.213 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:12.213 DEBUG:boto:Data: 2022-09-02T05:58:12.214 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.215 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.216 DEBUG:boto:Params: {} 2022-09-02T05:58:12.216 DEBUG:boto:Token: None 2022-09-02T05:58:12.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76 2022-09-02T05:58:12.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FUwyBFXk4lALToN30//EVrSVmn0= 2022-09-02T05:58:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FUwyBFXk4lALToN30//EVrSVmn0=', 'Content-Length': '0'} 2022-09-02T05:58:12.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947d4e008b25317f-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.221 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:58:12.222 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:58:12.222 DEBUG:boto:Method: GET 2022-09-02T05:58:12.223 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:58:12.223 DEBUG:boto:Data: 2022-09-02T05:58:12.224 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.225 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.226 DEBUG:boto:Params: {} 2022-09-02T05:58:12.226 DEBUG:boto:Token: None 2022-09-02T05:58:12.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj76 2022-09-02T05:58:12.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FUwyBFXk4lALToN30//EVrSVmn0= 2022-09-02T05:58:12.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FUwyBFXk4lALToN30//EVrSVmn0=', 'Content-Length': '0'} 2022-09-02T05:58:12.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd11ec0d72fc873d-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.274 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.275 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.275 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.276 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.276 DEBUG:boto:Method: GET 2022-09-02T05:58:12.277 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.277 DEBUG:boto:Data: 2022-09-02T05:58:12.278 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.280 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.280 DEBUG:boto:Params: {} 2022-09-02T05:58:12.281 DEBUG:boto:Token: None 2022-09-02T05:58:12.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81e3625ce743518-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.286 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.287 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.288 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.288 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.289 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.290 DEBUG:boto:Method: GET 2022-09-02T05:58:12.290 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:58:12.291 DEBUG:boto:Data: 2022-09-02T05:58:12.291 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.292 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.293 DEBUG:boto:Params: {} 2022-09-02T05:58:12.293 DEBUG:boto:Token: None 2022-09-02T05:58:12.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004225ad45e68ff108-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.300 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:58:12.302 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.302 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.303 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.303 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.304 DEBUG:boto:Method: GET 2022-09-02T05:58:12.305 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.305 DEBUG:boto:Data: 2022-09-02T05:58:12.306 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.307 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.307 DEBUG:boto:Params: {} 2022-09-02T05:58:12.308 DEBUG:boto:Token: None 2022-09-02T05:58:12.309 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw= 2022-09-02T05:58:12.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw=', 'Content-Length': '0'} 2022-09-02T05:58:12.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfddcd07160b3eba-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.313 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.313 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.314 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.315 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.315 DEBUG:boto:Method: GET 2022-09-02T05:58:12.316 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.316 DEBUG:boto:Data: 2022-09-02T05:58:12.317 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.317 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.318 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.318 DEBUG:boto:Params: {} 2022-09-02T05:58:12.319 DEBUG:boto:Token: None 2022-09-02T05:58:12.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw= 2022-09-02T05:58:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw=', 'Content-Length': '0'} 2022-09-02T05:58:12.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009274b58d8ea6452b-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.324 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.325 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.326 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.326 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.327 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.327 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.328 DEBUG:boto:Data: 2022-09-02T05:58:12.328 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.329 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.330 DEBUG:boto:Params: {} 2022-09-02T05:58:12.330 DEBUG:boto:Token: None 2022-09-02T05:58:12.331 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ut7dZ+kc3aDxzFAsJ/Z+4tZesb8= 2022-09-02T05:58:12.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ut7dZ+kc3aDxzFAsJ/Z+4tZesb8=', 'Content-Length': '0'} 2022-09-02T05:58:12.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e191a14ab9aba01-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.335 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.336 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.336 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.337 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.337 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.338 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.338 DEBUG:boto:Data: 2022-09-02T05:58:12.339 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.339 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.340 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.340 DEBUG:boto:Params: {} 2022-09-02T05:58:12.341 DEBUG:boto:Token: None 2022-09-02T05:58:12.342 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ut7dZ+kc3aDxzFAsJ/Z+4tZesb8= 2022-09-02T05:58:12.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ut7dZ+kc3aDxzFAsJ/Z+4tZesb8=', 'Content-Length': '0'} 2022-09-02T05:58:12.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029da37eb43b21927-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:58:12.347 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.347 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.348 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.348 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.349 DEBUG:boto:Method: GET 2022-09-02T05:58:12.349 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.350 DEBUG:boto:Data: 2022-09-02T05:58:12.350 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.351 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.352 DEBUG:boto:Params: {} 2022-09-02T05:58:12.353 DEBUG:boto:Token: None 2022-09-02T05:58:12.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw= 2022-09-02T05:58:12.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw=', 'Content-Length': '0'} 2022-09-02T05:58:12.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365def2548880c67-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.358 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.358 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.359 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.359 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.360 DEBUG:boto:Method: GET 2022-09-02T05:58:12.360 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.361 DEBUG:boto:Data: 2022-09-02T05:58:12.361 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.362 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.362 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.363 DEBUG:boto:Params: {} 2022-09-02T05:58:12.364 DEBUG:boto:Token: None 2022-09-02T05:58:12.364 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:58:12.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw= 2022-09-02T05:58:12.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hA3fFTOy9Hyaz7ZueoLJw5NKdHw=', 'Content-Length': '0'} 2022-09-02T05:58:12.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000367aad828640aa66-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.368 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.369 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.369 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.370 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.370 DEBUG:boto:Method: GET 2022-09-02T05:58:12.371 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.371 DEBUG:boto:Data: 2022-09-02T05:58:12.372 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.373 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.373 DEBUG:boto:Params: {} 2022-09-02T05:58:12.374 DEBUG:boto:Token: None 2022-09-02T05:58:12.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af4860d4347b3a43-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.379 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.380 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.380 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.381 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.381 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.382 DEBUG:boto:Method: GET 2022-09-02T05:58:12.382 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.383 DEBUG:boto:Data: 2022-09-02T05:58:12.383 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.384 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.384 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.385 DEBUG:boto:Params: {} 2022-09-02T05:58:12.386 DEBUG:boto:Token: None 2022-09-02T05:58:12.386 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.387 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005460ce4b6d8da178-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.392 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.393 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.394 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.395 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.396 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:12.397 DEBUG:boto:Data: 2022-09-02T05:58:12.398 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.399 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.400 DEBUG:boto:Params: {} 2022-09-02T05:58:12.401 DEBUG:boto:Token: None 2022-09-02T05:58:12.402 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85 2022-09-02T05:58:12.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xoaSMQDQxPOW8M66YiLZciFnKcs= 2022-09-02T05:58:12.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xoaSMQDQxPOW8M66YiLZciFnKcs=', 'Content-Length': '0'} 2022-09-02T05:58:12.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a91a6ca4e73e9157-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.405 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.406 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.406 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.407 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:12.407 DEBUG:boto:Data: 2022-09-02T05:58:12.408 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.409 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.409 DEBUG:boto:Params: {} 2022-09-02T05:58:12.410 DEBUG:boto:Token: None 2022-09-02T05:58:12.411 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85 2022-09-02T05:58:12.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xoaSMQDQxPOW8M66YiLZciFnKcs= 2022-09-02T05:58:12.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xoaSMQDQxPOW8M66YiLZciFnKcs=', 'Content-Length': '0'} 2022-09-02T05:58:12.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f91803a74ad5207-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:58:12.415 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.416 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.416 DEBUG:boto:Method: GET 2022-09-02T05:58:12.417 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:12.417 DEBUG:boto:Data: 2022-09-02T05:58:12.418 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.419 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.419 DEBUG:boto:Params: {} 2022-09-02T05:58:12.420 DEBUG:boto:Token: None 2022-09-02T05:58:12.420 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85 2022-09-02T05:58:12.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SX2z7YyZPgLw7vjA4rdQwyaEfmg= 2022-09-02T05:58:12.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SX2z7YyZPgLw7vjA4rdQwyaEfmg=', 'Content-Length': '0'} 2022-09-02T05:58:12.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bef822bc2d8936f-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.470 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:58:12.471 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:58:12.471 DEBUG:boto:Method: GET 2022-09-02T05:58:12.472 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:58:12.472 DEBUG:boto:Data: 2022-09-02T05:58:12.473 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.474 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.475 DEBUG:boto:Params: {} 2022-09-02T05:58:12.475 DEBUG:boto:Token: None 2022-09-02T05:58:12.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj85 2022-09-02T05:58:12.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SX2z7YyZPgLw7vjA4rdQwyaEfmg= 2022-09-02T05:58:12.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SX2z7YyZPgLw7vjA4rdQwyaEfmg=', 'Content-Length': '0'} 2022-09-02T05:58:12.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007500cb86ba62aa48-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.480 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.481 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.481 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.482 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.482 DEBUG:boto:Method: GET 2022-09-02T05:58:12.483 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.483 DEBUG:boto:Data: 2022-09-02T05:58:12.484 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.485 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.486 DEBUG:boto:Params: {} 2022-09-02T05:58:12.486 DEBUG:boto:Token: None 2022-09-02T05:58:12.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d13b129235977b4e-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.491 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.492 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.493 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.493 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.494 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.494 DEBUG:boto:Method: GET 2022-09-02T05:58:12.495 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:58:12.495 DEBUG:boto:Data: 2022-09-02T05:58:12.496 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.497 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.497 DEBUG:boto:Params: {} 2022-09-02T05:58:12.498 DEBUG:boto:Token: None 2022-09-02T05:58:12.499 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00d56c117438ebe-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.503 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:58:12.504 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.505 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.506 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.506 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.507 DEBUG:boto:Method: GET 2022-09-02T05:58:12.507 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.508 DEBUG:boto:Data: 2022-09-02T05:58:12.508 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.510 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.510 DEBUG:boto:Params: {} 2022-09-02T05:58:12.511 DEBUG:boto:Token: None 2022-09-02T05:58:12.511 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.512 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog= 2022-09-02T05:58:12.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog=', 'Content-Length': '0'} 2022-09-02T05:58:12.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40c88153545c0b6-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.515 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.516 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.516 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.517 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.517 DEBUG:boto:Method: GET 2022-09-02T05:58:12.518 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.519 DEBUG:boto:Data: 2022-09-02T05:58:12.519 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.520 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.520 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.521 DEBUG:boto:Params: {} 2022-09-02T05:58:12.522 DEBUG:boto:Token: None 2022-09-02T05:58:12.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog= 2022-09-02T05:58:12.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog=', 'Content-Length': '0'} 2022-09-02T05:58:12.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8c8a45e1622824-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.526 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.527 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.527 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.528 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.528 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.529 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.529 DEBUG:boto:Data: 2022-09-02T05:58:12.530 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.531 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.531 DEBUG:boto:Params: {} 2022-09-02T05:58:12.532 DEBUG:boto:Token: None 2022-09-02T05:58:12.532 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mL2XLzK/x47oI1m0QbT66ziN9t4= 2022-09-02T05:58:12.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mL2XLzK/x47oI1m0QbT66ziN9t4=', 'Content-Length': '0'} 2022-09-02T05:58:12.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f86a37542922b84-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.536 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.537 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.537 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.538 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.539 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.539 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.540 DEBUG:boto:Data: 2022-09-02T05:58:12.540 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.541 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.541 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.542 DEBUG:boto:Params: {} 2022-09-02T05:58:12.542 DEBUG:boto:Token: None 2022-09-02T05:58:12.543 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.543 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mL2XLzK/x47oI1m0QbT66ziN9t4= 2022-09-02T05:58:12.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mL2XLzK/x47oI1m0QbT66ziN9t4=', 'Content-Length': '0'} 2022-09-02T05:58:12.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a7103775997a90-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:58:12.548 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.548 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.549 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.549 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.550 DEBUG:boto:Method: GET 2022-09-02T05:58:12.551 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.551 DEBUG:boto:Data: 2022-09-02T05:58:12.552 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.554 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.555 DEBUG:boto:Params: {} 2022-09-02T05:58:12.556 DEBUG:boto:Token: None 2022-09-02T05:58:12.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog= 2022-09-02T05:58:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog=', 'Content-Length': '0'} 2022-09-02T05:58:12.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9c49d1faf404b86-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.606 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.607 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.607 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.608 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.609 DEBUG:boto:Method: GET 2022-09-02T05:58:12.609 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.610 DEBUG:boto:Data: 2022-09-02T05:58:12.611 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.611 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.612 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.612 DEBUG:boto:Params: {} 2022-09-02T05:58:12.613 DEBUG:boto:Token: None 2022-09-02T05:58:12.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:58:12.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog= 2022-09-02T05:58:12.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vagiKSWQv9oirHThmjSiw/aFwog=', 'Content-Length': '0'} 2022-09-02T05:58:12.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088dd910af146e61c-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.619 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.619 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.620 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.620 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.621 DEBUG:boto:Method: GET 2022-09-02T05:58:12.621 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.622 DEBUG:boto:Data: 2022-09-02T05:58:12.622 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.624 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.624 DEBUG:boto:Params: {} 2022-09-02T05:58:12.625 DEBUG:boto:Token: None 2022-09-02T05:58:12.625 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.626 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e3596e107b416db-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.630 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.631 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.632 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.632 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.633 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.634 DEBUG:boto:Method: GET 2022-09-02T05:58:12.635 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.636 DEBUG:boto:Data: 2022-09-02T05:58:12.636 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.638 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.639 DEBUG:boto:Params: {} 2022-09-02T05:58:12.640 DEBUG:boto:Token: None 2022-09-02T05:58:12.642 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0b559068131f77-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.648 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.649 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.650 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.651 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.651 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:12.652 DEBUG:boto:Data: 2022-09-02T05:58:12.653 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.654 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.654 DEBUG:boto:Params: {} 2022-09-02T05:58:12.655 DEBUG:boto:Token: None 2022-09-02T05:58:12.656 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87 2022-09-02T05:58:12.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HxErZDhCFMaVTDqBDtJAyZ81dpc= 2022-09-02T05:58:12.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HxErZDhCFMaVTDqBDtJAyZ81dpc=', 'Content-Length': '0'} 2022-09-02T05:58:12.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3da1f2fdb29612-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.660 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.661 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.662 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.662 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:12.663 DEBUG:boto:Data: 2022-09-02T05:58:12.664 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.665 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.666 DEBUG:boto:Params: {} 2022-09-02T05:58:12.667 DEBUG:boto:Token: None 2022-09-02T05:58:12.667 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87 2022-09-02T05:58:12.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HxErZDhCFMaVTDqBDtJAyZ81dpc= 2022-09-02T05:58:12.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HxErZDhCFMaVTDqBDtJAyZ81dpc=', 'Content-Length': '0'} 2022-09-02T05:58:12.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec42f1368cd4967-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:58:12.673 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.674 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.674 DEBUG:boto:Method: GET 2022-09-02T05:58:12.675 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:12.676 DEBUG:boto:Data: 2022-09-02T05:58:12.677 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.679 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.679 DEBUG:boto:Params: {} 2022-09-02T05:58:12.680 DEBUG:boto:Token: None 2022-09-02T05:58:12.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87 2022-09-02T05:58:12.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iHYdXdaiUOq200s33pv/mJ9hae0= 2022-09-02T05:58:12.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iHYdXdaiUOq200s33pv/mJ9hae0=', 'Content-Length': '0'} 2022-09-02T05:58:12.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000526942b3af907d83-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.687 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:58:12.688 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:58:12.689 DEBUG:boto:Method: GET 2022-09-02T05:58:12.690 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:58:12.691 DEBUG:boto:Data: 2022-09-02T05:58:12.692 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.693 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.694 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.695 DEBUG:boto:Params: {} 2022-09-02T05:58:12.696 DEBUG:boto:Token: None 2022-09-02T05:58:12.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj87 2022-09-02T05:58:12.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iHYdXdaiUOq200s33pv/mJ9hae0= 2022-09-02T05:58:12.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iHYdXdaiUOq200s33pv/mJ9hae0=', 'Content-Length': '0'} 2022-09-02T05:58:12.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd08d6f0ca8e95d1-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.702 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.703 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.704 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.705 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.706 DEBUG:boto:Method: GET 2022-09-02T05:58:12.706 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.707 DEBUG:boto:Data: 2022-09-02T05:58:12.708 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.709 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.710 DEBUG:boto:Params: {} 2022-09-02T05:58:12.711 DEBUG:boto:Token: None 2022-09-02T05:58:12.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5abefa05f55e1f7-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.717 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.718 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.719 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.719 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.720 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.720 DEBUG:boto:Method: GET 2022-09-02T05:58:12.721 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:58:12.721 DEBUG:boto:Data: 2022-09-02T05:58:12.722 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.722 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.723 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.723 DEBUG:boto:Params: {} 2022-09-02T05:58:12.724 DEBUG:boto:Token: None 2022-09-02T05:58:12.725 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086002e49d3bf52bc-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.729 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:58:12.731 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.731 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.732 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.732 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.733 DEBUG:boto:Method: GET 2022-09-02T05:58:12.733 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.734 DEBUG:boto:Data: 2022-09-02T05:58:12.734 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.736 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.736 DEBUG:boto:Params: {} 2022-09-02T05:58:12.737 DEBUG:boto:Token: None 2022-09-02T05:58:12.737 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig= 2022-09-02T05:58:12.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig=', 'Content-Length': '0'} 2022-09-02T05:58:12.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a93f0e49b08c7889-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.742 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.742 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.743 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.743 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.744 DEBUG:boto:Method: GET 2022-09-02T05:58:12.744 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.745 DEBUG:boto:Data: 2022-09-02T05:58:12.746 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.746 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.747 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.747 DEBUG:boto:Params: {} 2022-09-02T05:58:12.748 DEBUG:boto:Token: None 2022-09-02T05:58:12.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig= 2022-09-02T05:58:12.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig=', 'Content-Length': '0'} 2022-09-02T05:58:12.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000711b94864efb7ea4-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.753 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.753 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.754 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.755 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.755 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.756 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.756 DEBUG:boto:Data: 2022-09-02T05:58:12.757 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.758 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.759 DEBUG:boto:Params: {} 2022-09-02T05:58:12.759 DEBUG:boto:Token: None 2022-09-02T05:58:12.760 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.760 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvmZeLdsNgtgpkl1NVd1qO8l5Bk= 2022-09-02T05:58:12.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvmZeLdsNgtgpkl1NVd1qO8l5Bk=', 'Content-Length': '0'} 2022-09-02T05:58:12.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4b001f0cc0a6f4-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.764 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.765 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.765 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.766 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.766 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.767 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.767 DEBUG:boto:Data: 2022-09-02T05:58:12.768 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.768 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.769 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.769 DEBUG:boto:Params: {} 2022-09-02T05:58:12.770 DEBUG:boto:Token: None 2022-09-02T05:58:12.770 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mvmZeLdsNgtgpkl1NVd1qO8l5Bk= 2022-09-02T05:58:12.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mvmZeLdsNgtgpkl1NVd1qO8l5Bk=', 'Content-Length': '0'} 2022-09-02T05:58:12.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080eb0249061c53e1-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:58:12.775 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.776 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.776 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.777 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.777 DEBUG:boto:Method: GET 2022-09-02T05:58:12.778 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.779 DEBUG:boto:Data: 2022-09-02T05:58:12.780 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.782 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.783 DEBUG:boto:Params: {} 2022-09-02T05:58:12.784 DEBUG:boto:Token: None 2022-09-02T05:58:12.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig= 2022-09-02T05:58:12.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig=', 'Content-Length': '0'} 2022-09-02T05:58:12.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4fb680fd6efbffd-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.792 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.793 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.794 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.795 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.795 DEBUG:boto:Method: GET 2022-09-02T05:58:12.796 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.797 DEBUG:boto:Data: 2022-09-02T05:58:12.798 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.799 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.800 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.801 DEBUG:boto:Params: {} 2022-09-02T05:58:12.802 DEBUG:boto:Token: None 2022-09-02T05:58:12.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:58:12.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig= 2022-09-02T05:58:12.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vIrulobZQeGVCmjFmz0uBCAO/ig=', 'Content-Length': '0'} 2022-09-02T05:58:12.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a37b94a1feca91b5-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.810 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.811 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.812 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.813 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.813 DEBUG:boto:Method: GET 2022-09-02T05:58:12.814 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.814 DEBUG:boto:Data: 2022-09-02T05:58:12.815 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.816 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.816 DEBUG:boto:Params: {} 2022-09-02T05:58:12.817 DEBUG:boto:Token: None 2022-09-02T05:58:12.818 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000409831e78d892c45-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.866 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.867 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.868 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.869 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.870 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.871 DEBUG:boto:Method: GET 2022-09-02T05:58:12.872 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.873 DEBUG:boto:Data: 2022-09-02T05:58:12.874 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.875 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.876 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.877 DEBUG:boto:Params: {} 2022-09-02T05:58:12.878 DEBUG:boto:Token: None 2022-09-02T05:58:12.879 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085a8791158a9068d-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.885 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.887 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.888 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.890 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.891 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:12.891 DEBUG:boto:Data: 2022-09-02T05:58:12.892 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.893 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.893 DEBUG:boto:Params: {} 2022-09-02T05:58:12.894 DEBUG:boto:Token: None 2022-09-02T05:58:12.894 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92 2022-09-02T05:58:12.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9F68psz/xPZ+5tNofeJnX5ehKjA= 2022-09-02T05:58:12.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9F68psz/xPZ+5tNofeJnX5ehKjA=', 'Content-Length': '0'} 2022-09-02T05:58:12.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0efb12d4cf64255-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.899 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.899 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.900 DEBUG:boto:Method: HEAD 2022-09-02T05:58:12.900 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:12.901 DEBUG:boto:Data: 2022-09-02T05:58:12.901 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.902 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.903 DEBUG:boto:Params: {} 2022-09-02T05:58:12.903 DEBUG:boto:Token: None 2022-09-02T05:58:12.904 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92 2022-09-02T05:58:12.904 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9F68psz/xPZ+5tNofeJnX5ehKjA= 2022-09-02T05:58:12.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9F68psz/xPZ+5tNofeJnX5ehKjA=', 'Content-Length': '0'} 2022-09-02T05:58:12.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009741fef3fe8cf539-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:58:12.909 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.909 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.910 DEBUG:boto:Method: GET 2022-09-02T05:58:12.910 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:12.911 DEBUG:boto:Data: 2022-09-02T05:58:12.911 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.912 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.913 DEBUG:boto:Params: {} 2022-09-02T05:58:12.914 DEBUG:boto:Token: None 2022-09-02T05:58:12.914 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92 2022-09-02T05:58:12.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m2Lyamwl63fXz3+EhY4rAvQ34Gc= 2022-09-02T05:58:12.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m2Lyamwl63fXz3+EhY4rAvQ34Gc=', 'Content-Length': '0'} 2022-09-02T05:58:12.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662661222ace2ad9-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.919 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:58:12.919 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:58:12.920 DEBUG:boto:Method: GET 2022-09-02T05:58:12.921 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:58:12.921 DEBUG:boto:Data: 2022-09-02T05:58:12.922 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.924 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.925 DEBUG:boto:Params: {} 2022-09-02T05:58:12.925 DEBUG:boto:Token: None 2022-09-02T05:58:12.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/obj92 2022-09-02T05:58:12.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:m2Lyamwl63fXz3+EhY4rAvQ34Gc= 2022-09-02T05:58:12.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:m2Lyamwl63fXz3+EhY4rAvQ34Gc=', 'Content-Length': '0'} 2022-09-02T05:58:12.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b324fac135044561-0063119b74-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.970 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.971 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.972 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.973 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.974 DEBUG:boto:Method: GET 2022-09-02T05:58:12.975 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.975 DEBUG:boto:Data: 2022-09-02T05:58:12.976 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:12.977 DEBUG:boto:Port: 8001 2022-09-02T05:58:12.978 DEBUG:boto:Params: {} 2022-09-02T05:58:12.978 DEBUG:boto:Token: None 2022-09-02T05:58:12.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a07b03eb56af4a0-0063119b74-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.984 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.985 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:12.986 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:12.987 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.988 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.988 DEBUG:boto:Method: GET 2022-09-02T05:58:12.989 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:58:12.990 DEBUG:boto:Data: 2022-09-02T05:58:12.990 DEBUG:boto:Headers: {} 2022-09-02T05:58:12.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:12.992 DEBUG:boto:Port: 8000 2022-09-02T05:58:12.992 DEBUG:boto:Params: {} 2022-09-02T05:58:12.993 DEBUG:boto:Token: None 2022-09-02T05:58:12.993 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:12 GMT /xljbkf-74/ 2022-09-02T05:58:12.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU= 2022-09-02T05:58:12.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:91V7OaaN79ajHi/VeFeDqjnZwEU=', 'Content-Length': '0'} 2022-09-02T05:58:12.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dfdb8a760d7741b-0063119b74-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:12.998 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:12.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:58:13.000 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.000 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.001 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.001 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.002 DEBUG:boto:Method: GET 2022-09-02T05:58:13.002 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.003 DEBUG:boto:Data: 2022-09-02T05:58:13.004 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.005 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.005 DEBUG:boto:Params: {} 2022-09-02T05:58:13.006 DEBUG:boto:Token: None 2022-09-02T05:58:13.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.007 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw= 2022-09-02T05:58:13.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw=', 'Content-Length': '0'} 2022-09-02T05:58:13.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067b8f5c4a13c4aef-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.012 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.012 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.013 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.014 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.015 DEBUG:boto:Method: GET 2022-09-02T05:58:13.016 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.017 DEBUG:boto:Data: 2022-09-02T05:58:13.018 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.020 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.021 DEBUG:boto:Params: {} 2022-09-02T05:58:13.022 DEBUG:boto:Token: None 2022-09-02T05:58:13.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw= 2022-09-02T05:58:13.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw=', 'Content-Length': '0'} 2022-09-02T05:58:13.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac4bd95e1e2aad17-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.030 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.031 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.032 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.033 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.034 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.035 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.036 DEBUG:boto:Data: 2022-09-02T05:58:13.037 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.039 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.039 DEBUG:boto:Params: {} 2022-09-02T05:58:13.041 DEBUG:boto:Token: None 2022-09-02T05:58:13.042 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/inwoUtCixjL+Tt7MPwZ9xmoTg= 2022-09-02T05:58:13.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/inwoUtCixjL+Tt7MPwZ9xmoTg=', 'Content-Length': '0'} 2022-09-02T05:58:13.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c16e2d6bcb1a7a78-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.048 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.049 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.050 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.051 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.052 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.053 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.054 DEBUG:boto:Data: 2022-09-02T05:58:13.055 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.056 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.057 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.058 DEBUG:boto:Params: {} 2022-09-02T05:58:13.059 DEBUG:boto:Token: None 2022-09-02T05:58:13.060 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.061 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D/inwoUtCixjL+Tt7MPwZ9xmoTg= 2022-09-02T05:58:13.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D/inwoUtCixjL+Tt7MPwZ9xmoTg=', 'Content-Length': '0'} 2022-09-02T05:58:13.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073cceb295593f585-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:58:13.066 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.067 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.067 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.068 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.068 DEBUG:boto:Method: GET 2022-09-02T05:58:13.069 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.069 DEBUG:boto:Data: 2022-09-02T05:58:13.070 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.071 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.072 DEBUG:boto:Params: {} 2022-09-02T05:58:13.072 DEBUG:boto:Token: None 2022-09-02T05:58:13.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.073 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw= 2022-09-02T05:58:13.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw=', 'Content-Length': '0'} 2022-09-02T05:58:13.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d2313d29f6c563-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.118 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.119 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.119 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.120 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.121 DEBUG:boto:Method: GET 2022-09-02T05:58:13.121 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.122 DEBUG:boto:Data: 2022-09-02T05:58:13.122 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.123 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.123 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.124 DEBUG:boto:Params: {} 2022-09-02T05:58:13.125 DEBUG:boto:Token: None 2022-09-02T05:58:13.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:58:13.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw= 2022-09-02T05:58:13.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wIu4lA6erZJcOGw0p9ESrBQHOZw=', 'Content-Length': '0'} 2022-09-02T05:58:13.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa44170e2f7ee889-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.129 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.130 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.131 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.131 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.132 DEBUG:boto:Method: GET 2022-09-02T05:58:13.132 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.133 DEBUG:boto:Data: 2022-09-02T05:58:13.133 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.134 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.135 DEBUG:boto:Params: {} 2022-09-02T05:58:13.136 DEBUG:boto:Token: None 2022-09-02T05:58:13.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c0b1992e984fa4-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.141 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.141 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.142 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.142 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.143 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.143 DEBUG:boto:Method: GET 2022-09-02T05:58:13.144 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.145 DEBUG:boto:Data: 2022-09-02T05:58:13.145 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.146 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.147 DEBUG:boto:Params: {} 2022-09-02T05:58:13.147 DEBUG:boto:Token: None 2022-09-02T05:58:13.148 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.149 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e231957ac5a82a-0063119b75-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.153 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.154 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.154 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.155 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.155 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:13.156 DEBUG:boto:Data: 2022-09-02T05:58:13.156 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.157 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.158 DEBUG:boto:Params: {} 2022-09-02T05:58:13.159 DEBUG:boto:Token: None 2022-09-02T05:58:13.159 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93 2022-09-02T05:58:13.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zOhPeimToD1fTHoQ0IHKSnieQzw= 2022-09-02T05:58:13.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zOhPeimToD1fTHoQ0IHKSnieQzw=', 'Content-Length': '0'} 2022-09-02T05:58:13.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036408b3d35e97339-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.163 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.164 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.164 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.165 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:13.165 DEBUG:boto:Data: 2022-09-02T05:58:13.166 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.167 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.168 DEBUG:boto:Params: {} 2022-09-02T05:58:13.168 DEBUG:boto:Token: None 2022-09-02T05:58:13.169 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93 2022-09-02T05:58:13.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zOhPeimToD1fTHoQ0IHKSnieQzw= 2022-09-02T05:58:13.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zOhPeimToD1fTHoQ0IHKSnieQzw=', 'Content-Length': '0'} 2022-09-02T05:58:13.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d003bbdd8a3a0c62-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:58:13.173 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.174 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.174 DEBUG:boto:Method: GET 2022-09-02T05:58:13.175 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:13.176 DEBUG:boto:Data: 2022-09-02T05:58:13.176 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.177 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.178 DEBUG:boto:Params: {} 2022-09-02T05:58:13.179 DEBUG:boto:Token: None 2022-09-02T05:58:13.179 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93 2022-09-02T05:58:13.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2AOzntm0uigL3OcnSSk0Y7B1948= 2022-09-02T05:58:13.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2AOzntm0uigL3OcnSSk0Y7B1948=', 'Content-Length': '0'} 2022-09-02T05:58:13.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c5d365f25daa02-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.183 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:58:13.184 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:58:13.185 DEBUG:boto:Method: GET 2022-09-02T05:58:13.185 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:58:13.186 DEBUG:boto:Data: 2022-09-02T05:58:13.186 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.187 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.187 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.188 DEBUG:boto:Params: {} 2022-09-02T05:58:13.188 DEBUG:boto:Token: None 2022-09-02T05:58:13.189 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj93 2022-09-02T05:58:13.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2AOzntm0uigL3OcnSSk0Y7B1948= 2022-09-02T05:58:13.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2AOzntm0uigL3OcnSSk0Y7B1948=', 'Content-Length': '0'} 2022-09-02T05:58:13.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3f0e7aeca3d450-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.193 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.194 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.195 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.195 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.196 DEBUG:boto:Method: GET 2022-09-02T05:58:13.196 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.197 DEBUG:boto:Data: 2022-09-02T05:58:13.197 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.198 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.199 DEBUG:boto:Params: {} 2022-09-02T05:58:13.200 DEBUG:boto:Token: None 2022-09-02T05:58:13.200 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055098957760ad79d-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.246 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.248 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.249 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.250 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.251 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.252 DEBUG:boto:Method: GET 2022-09-02T05:58:13.253 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:58:13.254 DEBUG:boto:Data: 2022-09-02T05:58:13.255 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.256 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.256 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.257 DEBUG:boto:Params: {} 2022-09-02T05:58:13.259 DEBUG:boto:Token: None 2022-09-02T05:58:13.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ec3ca4d351e9485-0063119b75-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.265 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:58:13.268 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.268 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.269 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.270 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.270 DEBUG:boto:Method: GET 2022-09-02T05:58:13.271 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.271 DEBUG:boto:Data: 2022-09-02T05:58:13.272 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.273 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.273 DEBUG:boto:Params: {} 2022-09-02T05:58:13.274 DEBUG:boto:Token: None 2022-09-02T05:58:13.275 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o= 2022-09-02T05:58:13.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o=', 'Content-Length': '0'} 2022-09-02T05:58:13.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281cb0d446c845b8-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.280 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.280 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.281 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.281 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.282 DEBUG:boto:Method: GET 2022-09-02T05:58:13.282 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.283 DEBUG:boto:Data: 2022-09-02T05:58:13.283 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.284 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.285 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.285 DEBUG:boto:Params: {} 2022-09-02T05:58:13.286 DEBUG:boto:Token: None 2022-09-02T05:58:13.287 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o= 2022-09-02T05:58:13.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o=', 'Content-Length': '0'} 2022-09-02T05:58:13.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005beff87fdf88b9e6-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.293 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.294 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.294 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.295 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.296 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.297 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.297 DEBUG:boto:Data: 2022-09-02T05:58:13.298 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.299 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.300 DEBUG:boto:Params: {} 2022-09-02T05:58:13.300 DEBUG:boto:Token: None 2022-09-02T05:58:13.301 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFqxIQ7Buv5ZV+mQmCaIVSW4m5I= 2022-09-02T05:58:13.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFqxIQ7Buv5ZV+mQmCaIVSW4m5I=', 'Content-Length': '0'} 2022-09-02T05:58:13.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000315b569301b4f56a-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.305 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.306 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.306 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.307 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.308 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.309 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.309 DEBUG:boto:Data: 2022-09-02T05:58:13.310 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.310 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.311 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.311 DEBUG:boto:Params: {} 2022-09-02T05:58:13.312 DEBUG:boto:Token: None 2022-09-02T05:58:13.312 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CFqxIQ7Buv5ZV+mQmCaIVSW4m5I= 2022-09-02T05:58:13.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CFqxIQ7Buv5ZV+mQmCaIVSW4m5I=', 'Content-Length': '0'} 2022-09-02T05:58:13.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008697b67a0a3d82dd-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:58:13.317 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.318 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.319 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.319 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.320 DEBUG:boto:Method: GET 2022-09-02T05:58:13.320 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.321 DEBUG:boto:Data: 2022-09-02T05:58:13.321 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.323 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.323 DEBUG:boto:Params: {} 2022-09-02T05:58:13.324 DEBUG:boto:Token: None 2022-09-02T05:58:13.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o= 2022-09-02T05:58:13.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o=', 'Content-Length': '0'} 2022-09-02T05:58:13.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f71c6adef9032da-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.329 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.329 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.330 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.330 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.331 DEBUG:boto:Method: GET 2022-09-02T05:58:13.332 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.332 DEBUG:boto:Data: 2022-09-02T05:58:13.333 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.333 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.334 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.334 DEBUG:boto:Params: {} 2022-09-02T05:58:13.335 DEBUG:boto:Token: None 2022-09-02T05:58:13.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:58:13.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o= 2022-09-02T05:58:13.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f/hDVh68sl6zpJxkzTSdlA0jr1o=', 'Content-Length': '0'} 2022-09-02T05:58:13.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000608300f56f5d510e-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.342 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.343 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.344 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.345 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.346 DEBUG:boto:Method: GET 2022-09-02T05:58:13.346 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.347 DEBUG:boto:Data: 2022-09-02T05:58:13.347 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.348 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.349 DEBUG:boto:Params: {} 2022-09-02T05:58:13.349 DEBUG:boto:Token: None 2022-09-02T05:58:13.350 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4b288bfd7993229-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.355 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.356 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.356 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.357 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.357 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.358 DEBUG:boto:Method: GET 2022-09-02T05:58:13.358 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.359 DEBUG:boto:Data: 2022-09-02T05:58:13.359 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.360 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.360 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.361 DEBUG:boto:Params: {} 2022-09-02T05:58:13.361 DEBUG:boto:Token: None 2022-09-02T05:58:13.362 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b9748e264414b7-0063119b75-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.367 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.368 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.368 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.369 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.369 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:13.370 DEBUG:boto:Data: 2022-09-02T05:58:13.370 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.371 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.372 DEBUG:boto:Params: {} 2022-09-02T05:58:13.372 DEBUG:boto:Token: None 2022-09-02T05:58:13.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95 2022-09-02T05:58:13.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mP8XL6Lk6TL+impuQTRyjRE/5Zs= 2022-09-02T05:58:13.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mP8XL6Lk6TL+impuQTRyjRE/5Zs=', 'Content-Length': '0'} 2022-09-02T05:58:13.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005951a588194fd386-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.377 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.377 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.378 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.378 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:13.379 DEBUG:boto:Data: 2022-09-02T05:58:13.379 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.380 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.380 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.381 DEBUG:boto:Params: {} 2022-09-02T05:58:13.382 DEBUG:boto:Token: None 2022-09-02T05:58:13.382 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95 2022-09-02T05:58:13.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mP8XL6Lk6TL+impuQTRyjRE/5Zs= 2022-09-02T05:58:13.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mP8XL6Lk6TL+impuQTRyjRE/5Zs=', 'Content-Length': '0'} 2022-09-02T05:58:13.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9def7fdb35e29cd-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:58:13.387 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.387 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.388 DEBUG:boto:Method: GET 2022-09-02T05:58:13.388 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:13.389 DEBUG:boto:Data: 2022-09-02T05:58:13.389 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.390 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.391 DEBUG:boto:Params: {} 2022-09-02T05:58:13.391 DEBUG:boto:Token: None 2022-09-02T05:58:13.392 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95 2022-09-02T05:58:13.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rEsuWMSpQM5HuDhGWXVx0V3AF3U= 2022-09-02T05:58:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rEsuWMSpQM5HuDhGWXVx0V3AF3U=', 'Content-Length': '0'} 2022-09-02T05:58:13.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000169df6435eae12dc-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.442 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:58:13.443 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:58:13.443 DEBUG:boto:Method: GET 2022-09-02T05:58:13.444 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:58:13.444 DEBUG:boto:Data: 2022-09-02T05:58:13.445 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.446 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.447 DEBUG:boto:Params: {} 2022-09-02T05:58:13.447 DEBUG:boto:Token: None 2022-09-02T05:58:13.448 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj95 2022-09-02T05:58:13.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rEsuWMSpQM5HuDhGWXVx0V3AF3U= 2022-09-02T05:58:13.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rEsuWMSpQM5HuDhGWXVx0V3AF3U=', 'Content-Length': '0'} 2022-09-02T05:58:13.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b245f80f56508f7-0063119b75-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.453 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.453 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.454 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.454 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.455 DEBUG:boto:Method: GET 2022-09-02T05:58:13.455 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.456 DEBUG:boto:Data: 2022-09-02T05:58:13.457 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:58:13.458 DEBUG:boto:Port: 8001 2022-09-02T05:58:13.458 DEBUG:boto:Params: {} 2022-09-02T05:58:13.459 DEBUG:boto:Token: None 2022-09-02T05:58:13.460 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bf93ee9c13e5c65-0063119b75-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.465 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.466 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.467 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.467 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.468 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.469 DEBUG:boto:Method: GET 2022-09-02T05:58:13.469 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:58:13.470 DEBUG:boto:Data: 2022-09-02T05:58:13.470 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:58:13.471 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.472 DEBUG:boto:Params: {} 2022-09-02T05:58:13.473 DEBUG:boto:Token: None 2022-09-02T05:58:13.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0= 2022-09-02T05:58:13.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J3dEpNBIrtww7k0tFGwaZkZRUl0=', 'Content-Length': '0'} 2022-09-02T05:58:13.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6ebf5b77f29798-0063119b75-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.478 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:58:13.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:58:13.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:58:13.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:58:13.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2022-09-02T05:58:13.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:58:13.542 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:58:13.603 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2022-09-02T05:58:13.604 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: xljbkf-74 2022-09-02T05:58:13.616 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2022-09-02T05:58:13.727 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.728 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.729 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.730 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:13.731 DEBUG:boto:Data: 2022-09-02T05:58:13.731 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.733 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.733 DEBUG:boto:Params: {} 2022-09-02T05:58:13.734 DEBUG:boto:Token: None 2022-09-02T05:58:13.735 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk= 2022-09-02T05:58:13.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk=', 'Content-Length': '0'} 2022-09-02T05:58:13.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eee9a820a0970217-0063119b75-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.741 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:58:13.741 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:58:13.742 DEBUG:boto:Method: PUT 2022-09-02T05:58:13.742 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:58:13.743 DEBUG:boto:Data: 2022-09-02T05:58:13.744 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:58:13.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.745 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.746 DEBUG:boto:Params: {} 2022-09-02T05:58:13.747 DEBUG:boto:Token: None 2022-09-02T05:58:13.747 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj37 2022-09-02T05:58:13.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xmz3zQMS7cLaeN7ZXbDnh0Cf+jo= 2022-09-02T05:58:13.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xmz3zQMS7cLaeN7ZXbDnh0Cf+jo='} 2022-09-02T05:58:13.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ec3ebedc65eb6a8-0063119b75-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.761 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.761 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.762 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.762 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:13.763 DEBUG:boto:Data: 2022-09-02T05:58:13.763 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.764 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.765 DEBUG:boto:Params: {} 2022-09-02T05:58:13.765 DEBUG:boto:Token: None 2022-09-02T05:58:13.766 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk= 2022-09-02T05:58:13.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk=', 'Content-Length': '0'} 2022-09-02T05:58:13.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0007b8eff6053ff-0063119b75-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.771 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:58:13.772 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:58:13.772 DEBUG:boto:Method: PUT 2022-09-02T05:58:13.773 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:58:13.773 DEBUG:boto:Data: 2022-09-02T05:58:13.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:58:13.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.775 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.776 DEBUG:boto:Params: {} 2022-09-02T05:58:13.776 DEBUG:boto:Token: None 2022-09-02T05:58:13.777 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj74 2022-09-02T05:58:13.777 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+spO6Yy99GNpFYKFMqC0n/LTaQE= 2022-09-02T05:58:13.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+spO6Yy99GNpFYKFMqC0n/LTaQE='} 2022-09-02T05:58:13.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060c3049843bac76e-0063119b75-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.827 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.828 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.829 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.829 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:13.830 DEBUG:boto:Data: 2022-09-02T05:58:13.831 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.832 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.832 DEBUG:boto:Params: {} 2022-09-02T05:58:13.833 DEBUG:boto:Token: None 2022-09-02T05:58:13.833 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.834 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk= 2022-09-02T05:58:13.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk=', 'Content-Length': '0'} 2022-09-02T05:58:13.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eec33dd5ed1828e7-0063119b75-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.839 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:58:13.840 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:58:13.840 DEBUG:boto:Method: PUT 2022-09-02T05:58:13.841 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:58:13.841 DEBUG:boto:Data: 2022-09-02T05:58:13.842 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:58:13.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.843 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.843 DEBUG:boto:Params: {} 2022-09-02T05:58:13.844 DEBUG:boto:Token: None 2022-09-02T05:58:13.844 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj111 2022-09-02T05:58:13.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZPcIkOoNGNJFKBkRyIz3bgSYSgQ= 2022-09-02T05:58:13.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZPcIkOoNGNJFKBkRyIz3bgSYSgQ='} 2022-09-02T05:58:13.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004a267e4f21c7d99-0063119b75-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.853 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.853 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.854 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.854 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:13.855 DEBUG:boto:Data: 2022-09-02T05:58:13.855 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.856 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.857 DEBUG:boto:Params: {} 2022-09-02T05:58:13.857 DEBUG:boto:Token: None 2022-09-02T05:58:13.858 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.858 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk= 2022-09-02T05:58:13.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk=', 'Content-Length': '0'} 2022-09-02T05:58:13.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ecb36a503b5f495-0063119b75-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.864 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:58:13.865 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:58:13.865 DEBUG:boto:Method: PUT 2022-09-02T05:58:13.866 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:58:13.866 DEBUG:boto:Data: 2022-09-02T05:58:13.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:58:13.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.868 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.868 DEBUG:boto:Params: {} 2022-09-02T05:58:13.869 DEBUG:boto:Token: None 2022-09-02T05:58:13.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj148 2022-09-02T05:58:13.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OE8JVEQVxofoUtes6g0HSgm/dhI= 2022-09-02T05:58:13.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OE8JVEQVxofoUtes6g0HSgm/dhI='} 2022-09-02T05:58:13.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b35b04a96bd1d4e-0063119b75-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.916 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:58:13.917 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:58:13.918 DEBUG:boto:Method: HEAD 2022-09-02T05:58:13.919 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:58:13.920 DEBUG:boto:Data: 2022-09-02T05:58:13.920 DEBUG:boto:Headers: {} 2022-09-02T05:58:13.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.922 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.923 DEBUG:boto:Params: {} 2022-09-02T05:58:13.924 DEBUG:boto:Token: None 2022-09-02T05:58:13.925 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/ 2022-09-02T05:58:13.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk= 2022-09-02T05:58:13.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pa4QdPSw10yXlgzZTvRF5yYmfpk=', 'Content-Length': '0'} 2022-09-02T05:58:13.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010f3fb01bff8e79e-0063119b75-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.932 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:58:13.933 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:58:13.934 DEBUG:boto:Method: PUT 2022-09-02T05:58:13.935 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:58:13.936 DEBUG:boto:Data: 2022-09-02T05:58:13.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T05:58:13.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:58:13.939 DEBUG:boto:Port: 8000 2022-09-02T05:58:13.939 DEBUG:boto:Params: {} 2022-09-02T05:58:13.940 DEBUG:boto:Token: None 2022-09-02T05:58:13.941 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 05:58:13 GMT /xljbkf-74/obj185 2022-09-02T05:58:13.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DABOqKwJ/9VhHpMsmKwLAfaEesE= 2022-09-02T05:58:13.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 05:58:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DABOqKwJ/9VhHpMsmKwLAfaEesE='} 2022-09-02T05:58:13.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010acbffab82d00a9-0063119b75-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:58:13.951 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:58:13.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:58:13.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:58:13.952 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:58:14.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:58:14.025 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:58:14.063 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:14.062+0000 7f4a7bae0600 1 waiting to reach latest gen 6, on 5.. 2022-09-02T05:58:19.064 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:19.062+0000 7f4a7bae0600 1 waiting to reach latest gen 6, on 5.. 2022-09-02T05:58:24.064 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:24.063+0000 7f4a7bae0600 1 waiting to reach latest gen 6, on 5.. 2022-09-02T05:58:29.068 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:29.066+0000 7f4a7bae0600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:29.068 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:29.068 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:34.070 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:34.069+0000 7f4a7bae0600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:34.071 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:34.071 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:39.073 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:39.072+0000 7f4a7bae0600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:39.074 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:39.074 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:44.077 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:44.074+0000 7f4a7bae0600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:44.077 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:44.077 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:49.079 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:49.077+0000 7f4a7bae0600 1 bucket sync caught up with source: 2022-09-02T05:58:49.079 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:49.080 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:49.080 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:58:49.077+0000 7f4a7bae0600 0 bucket checkpoint complete 2022-09-02T05:58:49.098 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:58:49.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:58:49.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:58:49.100 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:58:49.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:58:49.165 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:58:49.221 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:58:49.220+0000 7fd81e16b600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:49.222 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.79.5, , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:49.222 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:54.224 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:58:54.223+0000 7fd81e16b600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:54.225 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.79.5, , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:54.225 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:58:59.227 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:58:59.225+0000 7fd81e16b600 1 waiting for incremental sync to catch up: 2022-09-02T05:58:59.227 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.79.5, , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:58:59.228 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:04.230 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:04.228+0000 7fd81e16b600 1 waiting for incremental sync to catch up: 2022-09-02T05:59:04.231 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.79.5, , , , , , , , , , , , , , , , , , , ] 2022-09-02T05:59:04.231 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:09.232 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:09.231+0000 7fd81e16b600 1 bucket sync caught up with source: 2022-09-02T05:59:09.233 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:09.233 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:09.233 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:09.231+0000 7fd81e16b600 0 bucket checkpoint complete 2022-09-02T05:59:09.251 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:59:09.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:09.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:09.253 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:59:09.317 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:09.318 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:09.372 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:09.371+0000 7ff4aec73600 1 waiting for incremental sync to catch up: 2022-09-02T05:59:09.373 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:09.373 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:14.375 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:14.375+0000 7ff4aec73600 1 waiting for incremental sync to catch up: 2022-09-02T05:59:14.376 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:14.376 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:19.378 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:19.378+0000 7ff4aec73600 1 waiting for incremental sync to catch up: 2022-09-02T05:59:19.379 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , , , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:19.379 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:24.381 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:24.380+0000 7ff4aec73600 1 bucket sync caught up with source: 2022-09-02T05:59:24.382 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:24.382 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, , , , , , 00000000002.81.5, , , , , , 00000000002.64.5, , , , , 00000000002.79.5, , , , , , , , , , , , , , , 00000000001.93.5, , , , 00000000001.58.5] 2022-09-02T05:59:24.383 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:24.380+0000 7ff4aec73600 0 bucket checkpoint complete 2022-09-02T05:59:24.399 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:59:24.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.401 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:59:24.466 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:24.466 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:24.531 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:24.529+0000 7fbdc9db4600 1 bucket sync caught up with source: 2022-09-02T05:59:24.531 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000005.603.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000004.556.5, 00000000007.1214.5, 00000000006.855.5] 2022-09-02T05:59:24.531 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000005.603.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000004.556.5, 00000000007.1214.5, 00000000006.855.5] 2022-09-02T05:59:24.532 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:24.529+0000 7fbdc9db4600 0 bucket checkpoint complete 2022-09-02T05:59:24.555 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=xljbkf-74 2022-09-02T05:59:24.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.557 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:59:24.615 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:24.616 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:24.665+0000 7fd0b8d24600 1 bucket sync caught up with source: 2022-09-02T05:59:24.667 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000005.603.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000004.556.5, 00000000007.1214.5, 00000000006.855.5] 2022-09-02T05:59:24.667 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.478.5, 00000000002.413.5, 00000000002.1071.5, 00000000005.603.5, 00000000004.421.5, 00000000001.621.5, 00000000001.1097.5, 00000000004.556.5, 00000000007.1214.5, 00000000006.855.5] 2022-09-02T05:59:24.668 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:24.665+0000 7fd0b8d24600 0 bucket checkpoint complete 2022-09-02T05:59:24.685 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:59:24.685 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=xljbkf-74 2022-09-02T05:59:24.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.687 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-09-02T05:59:24.743 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:24.743 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:24.778 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:24.777+0000 7f4f18045600 1 bucket sync caught up with source: 2022-09-02T05:59:24.779 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000005.363.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000004.212.5, 00000000007.344.5, 00000000006.544.5] 2022-09-02T05:59:24.779 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000005.363.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000004.212.5, 00000000007.344.5, 00000000006.544.5] 2022-09-02T05:59:24.779 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T05:59:24.777+0000 7f4f18045600 0 bucket checkpoint complete 2022-09-02T05:59:24.791 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=xljbkf-74 2022-09-02T05:59:24.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T05:59:24.793 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-74 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T05:59:24.857 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:24.858 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:24.903 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:24.902+0000 7f0a91dd9600 1 bucket sync caught up with source: 2022-09-02T05:59:24.904 INFO:teuthology.orchestra.run.smithi160.stderr: local status: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000005.363.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000004.212.5, 00000000007.344.5, 00000000006.544.5] 2022-09-02T05:59:24.904 INFO:teuthology.orchestra.run.smithi160.stderr: remote markers: [, 00000000003.100.5, 00000000002.392.5, 00000000002.532.5, 00000000005.363.5, 00000000004.400.5, 00000000001.375.5, 00000000001.395.5, 00000000004.212.5, 00000000007.344.5, 00000000006.544.5] 2022-09-02T05:59:24.905 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T05:59:24.902+0000 7f0a91dd9600 0 bucket checkpoint complete 2022-09-02T05:59:24.923 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone4 bucket=xljbkf-74 2022-09-02T05:59:24.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:59:24.924 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:24.925 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:24.925 DEBUG:boto:Method: HEAD 2022-09-02T05:59:24.926 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:59:24.927 DEBUG:boto:Data: 2022-09-02T05:59:24.927 DEBUG:boto:Headers: {} 2022-09-02T05:59:24.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:24.928 DEBUG:boto:Port: 8000 2022-09-02T05:59:24.929 DEBUG:boto:Params: {} 2022-09-02T05:59:24.930 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:59:24.931 DEBUG:boto:Token: None 2022-09-02T05:59:24.931 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:24 GMT /xljbkf-74/ 2022-09-02T05:59:24.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hXtOYPRRXJ6RpuXaq5FLQYSNXjM= 2022-09-02T05:59:24.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hXtOYPRRXJ6RpuXaq5FLQYSNXjM=', 'Content-Length': '0'} 2022-09-02T05:59:24.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008adadf70a39d81fd-0063119bbc-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:24.938 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:24.939 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:24.939 DEBUG:boto:Method: HEAD 2022-09-02T05:59:24.940 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:59:24.940 DEBUG:boto:Data: 2022-09-02T05:59:24.941 DEBUG:boto:Headers: {} 2022-09-02T05:59:24.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:24.942 DEBUG:boto:Port: 8000 2022-09-02T05:59:24.942 DEBUG:boto:Params: {} 2022-09-02T05:59:24.943 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T05:59:24.944 DEBUG:boto:Token: None 2022-09-02T05:59:24.944 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:24 GMT /xljbkf-74/ 2022-09-02T05:59:24.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hXtOYPRRXJ6RpuXaq5FLQYSNXjM= 2022-09-02T05:59:24.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hXtOYPRRXJ6RpuXaq5FLQYSNXjM=', 'Content-Length': '0'} 2022-09-02T05:59:24.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fb183d49fc68d24-0063119bbc-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:24.951 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:59:24.951 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:24.952 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:24.953 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:24.953 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:24.954 DEBUG:boto:Method: GET 2022-09-02T05:59:24.954 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:24.955 DEBUG:boto:Data: 2022-09-02T05:59:24.955 DEBUG:boto:Headers: {} 2022-09-02T05:59:24.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:24.956 DEBUG:boto:Port: 8000 2022-09-02T05:59:24.957 DEBUG:boto:Params: {} 2022-09-02T05:59:24.958 DEBUG:boto:Token: None 2022-09-02T05:59:24.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:24 GMT /xljbkf-74/?versions 2022-09-02T05:59:24.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h+6ISo+jXwwpIf637DZRsZYHqYs= 2022-09-02T05:59:24.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h+6ISo+jXwwpIf637DZRsZYHqYs=', 'Content-Length': '0'} 2022-09-02T05:59:24.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fce32eed5acc103-0063119bbc-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:24.963 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:24.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:59:24.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-02T05:59:24.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:59:24.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:59:24.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:59:24.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:59:24.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-02T05:59:24.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:59:24.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:59:24.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-02T05:59:24.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:59:24.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:59:24.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:59:24.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:59:24.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:59:24.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:59:24.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:59:24.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-02T05:59:24.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:59:24.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:59:24.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:59:24.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:59:24.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:59:24.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:59:24.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:59:24.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:59:24.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:59:24.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:59:24.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-02T05:59:24.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:59:24.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:59:24.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:59:24.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:59:24.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:59:24.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:59:24.984 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:59:24.985 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:24.986 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:24.986 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:24.987 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:24.987 DEBUG:boto:Method: GET 2022-09-02T05:59:24.988 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:24.988 DEBUG:boto:Data: 2022-09-02T05:59:24.989 DEBUG:boto:Headers: {} 2022-09-02T05:59:24.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:24.990 DEBUG:boto:Port: 8000 2022-09-02T05:59:24.990 DEBUG:boto:Params: {} 2022-09-02T05:59:24.991 DEBUG:boto:Token: None 2022-09-02T05:59:24.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:24 GMT /xljbkf-74/?versions 2022-09-02T05:59:24.992 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h+6ISo+jXwwpIf637DZRsZYHqYs= 2022-09-02T05:59:24.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:24 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h+6ISo+jXwwpIf637DZRsZYHqYs=', 'Content-Length': '0'} 2022-09-02T05:59:24.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb5ad7e9a4fa5fa-0063119bbc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:24 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:24.998 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:59:25.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-02T05:59:25.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:59:25.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:59:25.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:59:25.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:59:25.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-02T05:59:25.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:59:25.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:59:25.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-02T05:59:25.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:59:25.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:59:25.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:59:25.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:59:25.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:59:25.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:59:25.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:59:25.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-02T05:59:25.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:59:25.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:59:25.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:59:25.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:59:25.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:59:25.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:59:25.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:59:25.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:59:25.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:59:25.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:59:25.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-02T05:59:25.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:59:25.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:59:25.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:59:25.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:59:25.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:59:25.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:59:25.018 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.019 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.019 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:25.020 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:25.020 DEBUG:boto:Method: GET 2022-09-02T05:59:25.021 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:25.022 DEBUG:boto:Data: 2022-09-02T05:59:25.022 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.023 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.024 DEBUG:boto:Params: {} 2022-09-02T05:59:25.025 DEBUG:boto:Token: None 2022-09-02T05:59:25.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/?versions 2022-09-02T05:59:25.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x0jXxPRpIcI+rhTDuEAiswxg3zU= 2022-09-02T05:59:25.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x0jXxPRpIcI+rhTDuEAiswxg3zU=', 'Content-Length': '0'} 2022-09-02T05:59:25.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198c8a924d8961b4-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.030 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.032 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.033 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.034 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:25.034 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:25.035 DEBUG:boto:Method: GET 2022-09-02T05:59:25.035 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:25.036 DEBUG:boto:Data: 2022-09-02T05:59:25.036 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.038 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.038 DEBUG:boto:Params: {} 2022-09-02T05:59:25.039 DEBUG:boto:Token: None 2022-09-02T05:59:25.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/?versions 2022-09-02T05:59:25.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x0jXxPRpIcI+rhTDuEAiswxg3zU= 2022-09-02T05:59:25.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x0jXxPRpIcI+rhTDuEAiswxg3zU=', 'Content-Length': '0'} 2022-09-02T05:59:25.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f2ecbe095f6712-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.045 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:25.049 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.049 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.050 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.050 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.051 DEBUG:boto:Method: GET 2022-09-02T05:59:25.052 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.053 DEBUG:boto:Data: 2022-09-02T05:59:25.053 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.055 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.055 DEBUG:boto:Params: {} 2022-09-02T05:59:25.056 DEBUG:boto:Token: None 2022-09-02T05:59:25.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc= 2022-09-02T05:59:25.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc=', 'Content-Length': '0'} 2022-09-02T05:59:25.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221c8f457992a0e0-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.061 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.062 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.062 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.063 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.063 DEBUG:boto:Method: GET 2022-09-02T05:59:25.064 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.064 DEBUG:boto:Data: 2022-09-02T05:59:25.065 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.066 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.067 DEBUG:boto:Params: {} 2022-09-02T05:59:25.068 DEBUG:boto:Token: None 2022-09-02T05:59:25.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.069 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc= 2022-09-02T05:59:25.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc=', 'Content-Length': '0'} 2022-09-02T05:59:25.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a7a569d21b0b35f-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.072 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.073 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.073 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.074 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.075 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.075 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.076 DEBUG:boto:Data: 2022-09-02T05:59:25.077 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.078 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.078 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.079 DEBUG:boto:Params: {} 2022-09-02T05:59:25.080 DEBUG:boto:Token: None 2022-09-02T05:59:25.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yML9yeeZqsulld3yZjaYq7TKYfk= 2022-09-02T05:59:25.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yML9yeeZqsulld3yZjaYq7TKYfk=', 'Content-Length': '0'} 2022-09-02T05:59:25.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187e6e44c6f989b8-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.086 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.087 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.087 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.088 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.088 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.089 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.089 DEBUG:boto:Data: 2022-09-02T05:59:25.090 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.091 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.092 DEBUG:boto:Params: {} 2022-09-02T05:59:25.092 DEBUG:boto:Token: None 2022-09-02T05:59:25.093 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.093 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yML9yeeZqsulld3yZjaYq7TKYfk= 2022-09-02T05:59:25.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yML9yeeZqsulld3yZjaYq7TKYfk=', 'Content-Length': '0'} 2022-09-02T05:59:25.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d07011df9990c18-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:25.098 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.098 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.099 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.099 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.100 DEBUG:boto:Method: GET 2022-09-02T05:59:25.100 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.101 DEBUG:boto:Data: 2022-09-02T05:59:25.101 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.103 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.103 DEBUG:boto:Params: {} 2022-09-02T05:59:25.104 DEBUG:boto:Token: None 2022-09-02T05:59:25.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc= 2022-09-02T05:59:25.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc=', 'Content-Length': '0'} 2022-09-02T05:59:25.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad8c535af9400cc-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.109 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.109 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.110 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.110 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.111 DEBUG:boto:Method: GET 2022-09-02T05:59:25.111 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.112 DEBUG:boto:Data: 2022-09-02T05:59:25.113 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.114 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.114 DEBUG:boto:Params: {} 2022-09-02T05:59:25.115 DEBUG:boto:Token: None 2022-09-02T05:59:25.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:25.116 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc= 2022-09-02T05:59:25.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IrPQXpH1WOlQYbfbCKi0A9z96Fc=', 'Content-Length': '0'} 2022-09-02T05:59:25.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db59e83be9d746d4-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.119 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.120 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.121 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.121 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.122 DEBUG:boto:Method: GET 2022-09-02T05:59:25.122 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.123 DEBUG:boto:Data: 2022-09-02T05:59:25.123 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.124 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.124 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.125 DEBUG:boto:Params: {} 2022-09-02T05:59:25.125 DEBUG:boto:Token: None 2022-09-02T05:59:25.126 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.127 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d133701fcb4ae85-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.130 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.131 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.132 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.133 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.133 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.134 DEBUG:boto:Method: GET 2022-09-02T05:59:25.134 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.135 DEBUG:boto:Data: 2022-09-02T05:59:25.135 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.136 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.137 DEBUG:boto:Params: {} 2022-09-02T05:59:25.137 DEBUG:boto:Token: None 2022-09-02T05:59:25.138 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f29a2f1906bb40a-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.144 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.144 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.145 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.146 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.146 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:25.147 DEBUG:boto:Data: 2022-09-02T05:59:25.147 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.148 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.149 DEBUG:boto:Params: {} 2022-09-02T05:59:25.149 DEBUG:boto:Token: None 2022-09-02T05:59:25.150 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11 2022-09-02T05:59:25.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AfvWtXgCmjCiANbgtPYsPiA5rBo= 2022-09-02T05:59:25.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AfvWtXgCmjCiANbgtPYsPiA5rBo=', 'Content-Length': '0'} 2022-09-02T05:59:25.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9998581d5b53b35-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.154 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.155 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.155 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.156 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:25.156 DEBUG:boto:Data: 2022-09-02T05:59:25.157 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.158 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.159 DEBUG:boto:Params: {} 2022-09-02T05:59:25.159 DEBUG:boto:Token: None 2022-09-02T05:59:25.160 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11 2022-09-02T05:59:25.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AfvWtXgCmjCiANbgtPYsPiA5rBo= 2022-09-02T05:59:25.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AfvWtXgCmjCiANbgtPYsPiA5rBo=', 'Content-Length': '0'} 2022-09-02T05:59:25.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c615d48a6ec44d-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:25.164 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.164 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.165 DEBUG:boto:Method: GET 2022-09-02T05:59:25.165 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:25.166 DEBUG:boto:Data: 2022-09-02T05:59:25.167 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.167 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.168 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.168 DEBUG:boto:Params: {} 2022-09-02T05:59:25.169 DEBUG:boto:Token: None 2022-09-02T05:59:25.170 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11 2022-09-02T05:59:25.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LnGq2YT5iaQVdBEhTmSKeN3QUl8= 2022-09-02T05:59:25.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LnGq2YT5iaQVdBEhTmSKeN3QUl8=', 'Content-Length': '0'} 2022-09-02T05:59:25.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2887ceed365e2f0-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.174 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:25.175 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:25.175 DEBUG:boto:Method: GET 2022-09-02T05:59:25.176 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:25.176 DEBUG:boto:Data: 2022-09-02T05:59:25.177 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.178 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.178 DEBUG:boto:Params: {} 2022-09-02T05:59:25.179 DEBUG:boto:Token: None 2022-09-02T05:59:25.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj11 2022-09-02T05:59:25.180 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LnGq2YT5iaQVdBEhTmSKeN3QUl8= 2022-09-02T05:59:25.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LnGq2YT5iaQVdBEhTmSKeN3QUl8=', 'Content-Length': '0'} 2022-09-02T05:59:25.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001026e17e90a13c48-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.226 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.227 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.227 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.228 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.229 DEBUG:boto:Method: GET 2022-09-02T05:59:25.229 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.230 DEBUG:boto:Data: 2022-09-02T05:59:25.231 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.231 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.232 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.232 DEBUG:boto:Params: {} 2022-09-02T05:59:25.233 DEBUG:boto:Token: None 2022-09-02T05:59:25.233 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.234 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c430e28c3a127a32-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.238 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.239 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.240 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.240 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.241 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.241 DEBUG:boto:Method: GET 2022-09-02T05:59:25.242 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:25.242 DEBUG:boto:Data: 2022-09-02T05:59:25.243 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.244 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.244 DEBUG:boto:Params: {} 2022-09-02T05:59:25.245 DEBUG:boto:Token: None 2022-09-02T05:59:25.246 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000651f12f721b1905d-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.251 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:25.253 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.254 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.254 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.255 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.255 DEBUG:boto:Method: GET 2022-09-02T05:59:25.256 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.256 DEBUG:boto:Data: 2022-09-02T05:59:25.257 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.257 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.258 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.258 DEBUG:boto:Params: {} 2022-09-02T05:59:25.259 DEBUG:boto:Token: None 2022-09-02T05:59:25.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg= 2022-09-02T05:59:25.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg=', 'Content-Length': '0'} 2022-09-02T05:59:25.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f274816db3f5fb9-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.264 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.264 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.265 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.265 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.266 DEBUG:boto:Method: GET 2022-09-02T05:59:25.267 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.267 DEBUG:boto:Data: 2022-09-02T05:59:25.268 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.269 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.269 DEBUG:boto:Params: {} 2022-09-02T05:59:25.270 DEBUG:boto:Token: None 2022-09-02T05:59:25.270 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg= 2022-09-02T05:59:25.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg=', 'Content-Length': '0'} 2022-09-02T05:59:25.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b61d12d0050aa81d-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.275 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.275 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.276 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.276 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.277 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.277 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.278 DEBUG:boto:Data: 2022-09-02T05:59:25.278 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.279 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.279 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.280 DEBUG:boto:Params: {} 2022-09-02T05:59:25.280 DEBUG:boto:Token: None 2022-09-02T05:59:25.281 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y/uylvOVgu21+P2DQajWAC6XvYA= 2022-09-02T05:59:25.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y/uylvOVgu21+P2DQajWAC6XvYA=', 'Content-Length': '0'} 2022-09-02T05:59:25.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899fb7d3033e74b3-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.285 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.285 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.286 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.286 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.287 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.287 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.288 DEBUG:boto:Data: 2022-09-02T05:59:25.288 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.289 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.290 DEBUG:boto:Params: {} 2022-09-02T05:59:25.291 DEBUG:boto:Token: None 2022-09-02T05:59:25.291 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y/uylvOVgu21+P2DQajWAC6XvYA= 2022-09-02T05:59:25.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y/uylvOVgu21+P2DQajWAC6XvYA=', 'Content-Length': '0'} 2022-09-02T05:59:25.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a1b4991828914fc-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:25.296 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.296 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.297 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.297 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.298 DEBUG:boto:Method: GET 2022-09-02T05:59:25.298 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.299 DEBUG:boto:Data: 2022-09-02T05:59:25.300 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.300 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.301 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.301 DEBUG:boto:Params: {} 2022-09-02T05:59:25.302 DEBUG:boto:Token: None 2022-09-02T05:59:25.302 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.303 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg= 2022-09-02T05:59:25.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg=', 'Content-Length': '0'} 2022-09-02T05:59:25.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e699d641613b676-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.306 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.307 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.307 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.308 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.308 DEBUG:boto:Method: GET 2022-09-02T05:59:25.309 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.309 DEBUG:boto:Data: 2022-09-02T05:59:25.310 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.311 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.312 DEBUG:boto:Params: {} 2022-09-02T05:59:25.312 DEBUG:boto:Token: None 2022-09-02T05:59:25.314 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:25.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg= 2022-09-02T05:59:25.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I4w6eDD7iNQJRsGYcf2y+JZIwwg=', 'Content-Length': '0'} 2022-09-02T05:59:25.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdc71e464010aa53-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.358 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.359 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.360 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.360 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.361 DEBUG:boto:Method: GET 2022-09-02T05:59:25.362 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.362 DEBUG:boto:Data: 2022-09-02T05:59:25.363 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.364 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.365 DEBUG:boto:Params: {} 2022-09-02T05:59:25.366 DEBUG:boto:Token: None 2022-09-02T05:59:25.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e247f3a03a82764-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.372 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.372 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.373 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.374 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.374 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.375 DEBUG:boto:Method: GET 2022-09-02T05:59:25.375 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.376 DEBUG:boto:Data: 2022-09-02T05:59:25.376 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.378 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.378 DEBUG:boto:Params: {} 2022-09-02T05:59:25.379 DEBUG:boto:Token: None 2022-09-02T05:59:25.379 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d5c8e2d99846c21-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.385 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.386 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.386 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.387 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.387 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:25.388 DEBUG:boto:Data: 2022-09-02T05:59:25.388 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.389 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.389 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.390 DEBUG:boto:Params: {} 2022-09-02T05:59:25.390 DEBUG:boto:Token: None 2022-09-02T05:59:25.391 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111 2022-09-02T05:59:25.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QxTcpkuhyYApD9fzZSp0p2CA5tM= 2022-09-02T05:59:25.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QxTcpkuhyYApD9fzZSp0p2CA5tM=', 'Content-Length': '0'} 2022-09-02T05:59:25.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8e683d6081bfc6b-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.395 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.395 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.396 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.396 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:25.397 DEBUG:boto:Data: 2022-09-02T05:59:25.398 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.399 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.399 DEBUG:boto:Params: {} 2022-09-02T05:59:25.400 DEBUG:boto:Token: None 2022-09-02T05:59:25.401 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111 2022-09-02T05:59:25.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QxTcpkuhyYApD9fzZSp0p2CA5tM= 2022-09-02T05:59:25.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QxTcpkuhyYApD9fzZSp0p2CA5tM=', 'Content-Length': '0'} 2022-09-02T05:59:25.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec4b26c2c2c9e44-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:25.405 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.405 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.406 DEBUG:boto:Method: GET 2022-09-02T05:59:25.407 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:25.407 DEBUG:boto:Data: 2022-09-02T05:59:25.408 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.408 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.409 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.409 DEBUG:boto:Params: {} 2022-09-02T05:59:25.410 DEBUG:boto:Token: None 2022-09-02T05:59:25.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111 2022-09-02T05:59:25.427 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4OsuBBujF2myM01RT0ltEI8jXZE= 2022-09-02T05:59:25.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4OsuBBujF2myM01RT0ltEI8jXZE=', 'Content-Length': '0'} 2022-09-02T05:59:25.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000287ab98a806c2e5a-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.430 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:25.431 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:25.432 DEBUG:boto:Method: GET 2022-09-02T05:59:25.432 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:25.433 DEBUG:boto:Data: 2022-09-02T05:59:25.433 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.434 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.435 DEBUG:boto:Params: {} 2022-09-02T05:59:25.436 DEBUG:boto:Token: None 2022-09-02T05:59:25.436 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj111 2022-09-02T05:59:25.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4OsuBBujF2myM01RT0ltEI8jXZE= 2022-09-02T05:59:25.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4OsuBBujF2myM01RT0ltEI8jXZE=', 'Content-Length': '0'} 2022-09-02T05:59:25.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bfc5ada39727e34-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.440 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.441 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.442 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.442 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.443 DEBUG:boto:Method: GET 2022-09-02T05:59:25.443 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.444 DEBUG:boto:Data: 2022-09-02T05:59:25.444 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.445 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.445 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.446 DEBUG:boto:Params: {} 2022-09-02T05:59:25.446 DEBUG:boto:Token: None 2022-09-02T05:59:25.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c7a3222a32f892-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.451 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.452 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.453 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.454 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.454 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.455 DEBUG:boto:Method: GET 2022-09-02T05:59:25.455 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:25.456 DEBUG:boto:Data: 2022-09-02T05:59:25.456 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.457 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.458 DEBUG:boto:Params: {} 2022-09-02T05:59:25.458 DEBUG:boto:Token: None 2022-09-02T05:59:25.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bf928d13623e7a-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.465 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:25.466 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.467 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.467 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.468 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.468 DEBUG:boto:Method: GET 2022-09-02T05:59:25.469 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.469 DEBUG:boto:Data: 2022-09-02T05:59:25.470 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.470 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.471 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.471 DEBUG:boto:Params: {} 2022-09-02T05:59:25.472 DEBUG:boto:Token: None 2022-09-02T05:59:25.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4= 2022-09-02T05:59:25.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4=', 'Content-Length': '0'} 2022-09-02T05:59:25.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ed54b8e8f8e675-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.518 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.519 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.519 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.520 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.520 DEBUG:boto:Method: GET 2022-09-02T05:59:25.521 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.521 DEBUG:boto:Data: 2022-09-02T05:59:25.522 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.523 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.524 DEBUG:boto:Params: {} 2022-09-02T05:59:25.525 DEBUG:boto:Token: None 2022-09-02T05:59:25.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4= 2022-09-02T05:59:25.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4=', 'Content-Length': '0'} 2022-09-02T05:59:25.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a022e528255e743-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.530 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.530 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.531 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.531 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.532 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.532 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.533 DEBUG:boto:Data: 2022-09-02T05:59:25.533 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.534 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.535 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.535 DEBUG:boto:Params: {} 2022-09-02T05:59:25.536 DEBUG:boto:Token: None 2022-09-02T05:59:25.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JuWcbpexZWGuryShRmuQoEqia5E= 2022-09-02T05:59:25.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JuWcbpexZWGuryShRmuQoEqia5E=', 'Content-Length': '0'} 2022-09-02T05:59:25.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11400664bff01e5-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.540 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.541 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.541 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.542 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.542 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.543 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.543 DEBUG:boto:Data: 2022-09-02T05:59:25.544 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.545 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.546 DEBUG:boto:Params: {} 2022-09-02T05:59:25.547 DEBUG:boto:Token: None 2022-09-02T05:59:25.547 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.548 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JuWcbpexZWGuryShRmuQoEqia5E= 2022-09-02T05:59:25.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JuWcbpexZWGuryShRmuQoEqia5E=', 'Content-Length': '0'} 2022-09-02T05:59:25.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d40efd08fa3055-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:25.552 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.552 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.553 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.553 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.554 DEBUG:boto:Method: GET 2022-09-02T05:59:25.555 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.555 DEBUG:boto:Data: 2022-09-02T05:59:25.556 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.557 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.557 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.558 DEBUG:boto:Params: {} 2022-09-02T05:59:25.558 DEBUG:boto:Token: None 2022-09-02T05:59:25.559 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4= 2022-09-02T05:59:25.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4=', 'Content-Length': '0'} 2022-09-02T05:59:25.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad87b24099a755f-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.563 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.564 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.565 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.565 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.566 DEBUG:boto:Method: GET 2022-09-02T05:59:25.566 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.567 DEBUG:boto:Data: 2022-09-02T05:59:25.568 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.569 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.569 DEBUG:boto:Params: {} 2022-09-02T05:59:25.570 DEBUG:boto:Token: None 2022-09-02T05:59:25.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:25.571 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4= 2022-09-02T05:59:25.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YLgyvg0cuYhY7t5VaUV3NzOwxt4=', 'Content-Length': '0'} 2022-09-02T05:59:25.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e263d56e8a956d-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.618 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.619 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.620 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.620 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.621 DEBUG:boto:Method: GET 2022-09-02T05:59:25.622 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.623 DEBUG:boto:Data: 2022-09-02T05:59:25.624 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.625 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.626 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.627 DEBUG:boto:Params: {} 2022-09-02T05:59:25.627 DEBUG:boto:Token: None 2022-09-02T05:59:25.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad7efbfa21ca5d0-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.634 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.635 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.635 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.636 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.636 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.637 DEBUG:boto:Method: GET 2022-09-02T05:59:25.638 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.638 DEBUG:boto:Data: 2022-09-02T05:59:25.639 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.641 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.642 DEBUG:boto:Params: {} 2022-09-02T05:59:25.643 DEBUG:boto:Token: None 2022-09-02T05:59:25.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.645 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc7c23b6824008a-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.650 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.651 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.652 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.652 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.653 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:25.653 DEBUG:boto:Data: 2022-09-02T05:59:25.654 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.655 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.656 DEBUG:boto:Params: {} 2022-09-02T05:59:25.656 DEBUG:boto:Token: None 2022-09-02T05:59:25.657 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115 2022-09-02T05:59:25.658 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b0ucJCsGIxGW1tQd+41qrWUOXh4= 2022-09-02T05:59:25.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b0ucJCsGIxGW1tQd+41qrWUOXh4=', 'Content-Length': '0'} 2022-09-02T05:59:25.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3abc9f178024d81-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.661 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.661 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.662 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.662 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:25.663 DEBUG:boto:Data: 2022-09-02T05:59:25.663 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.665 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.665 DEBUG:boto:Params: {} 2022-09-02T05:59:25.666 DEBUG:boto:Token: None 2022-09-02T05:59:25.667 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115 2022-09-02T05:59:25.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b0ucJCsGIxGW1tQd+41qrWUOXh4= 2022-09-02T05:59:25.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b0ucJCsGIxGW1tQd+41qrWUOXh4=', 'Content-Length': '0'} 2022-09-02T05:59:25.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197529014fd9fab2-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:25.671 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.672 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.673 DEBUG:boto:Method: GET 2022-09-02T05:59:25.673 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:25.674 DEBUG:boto:Data: 2022-09-02T05:59:25.674 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.676 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.676 DEBUG:boto:Params: {} 2022-09-02T05:59:25.677 DEBUG:boto:Token: None 2022-09-02T05:59:25.677 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115 2022-09-02T05:59:25.678 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0q8ikzM1XBXCgFoqcYekkTcFIM= 2022-09-02T05:59:25.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0q8ikzM1XBXCgFoqcYekkTcFIM=', 'Content-Length': '0'} 2022-09-02T05:59:25.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007127cb10126b9eef-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.682 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:25.683 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:25.683 DEBUG:boto:Method: GET 2022-09-02T05:59:25.684 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:25.685 DEBUG:boto:Data: 2022-09-02T05:59:25.685 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.686 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.687 DEBUG:boto:Params: {} 2022-09-02T05:59:25.687 DEBUG:boto:Token: None 2022-09-02T05:59:25.688 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj115 2022-09-02T05:59:25.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E0q8ikzM1XBXCgFoqcYekkTcFIM= 2022-09-02T05:59:25.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E0q8ikzM1XBXCgFoqcYekkTcFIM=', 'Content-Length': '0'} 2022-09-02T05:59:25.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d9ec9a6313d448-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.694 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.695 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.696 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.697 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.698 DEBUG:boto:Method: GET 2022-09-02T05:59:25.699 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.700 DEBUG:boto:Data: 2022-09-02T05:59:25.701 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.703 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.703 DEBUG:boto:Params: {} 2022-09-02T05:59:25.704 DEBUG:boto:Token: None 2022-09-02T05:59:25.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c77f106f3026ea34-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.710 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.711 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.712 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.713 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.713 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.714 DEBUG:boto:Method: GET 2022-09-02T05:59:25.714 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:25.715 DEBUG:boto:Data: 2022-09-02T05:59:25.716 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.717 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.718 DEBUG:boto:Params: {} 2022-09-02T05:59:25.718 DEBUG:boto:Token: None 2022-09-02T05:59:25.719 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.720 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b44a62122db9837-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.725 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:25.727 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.727 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.728 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.729 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.729 DEBUG:boto:Method: GET 2022-09-02T05:59:25.730 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.731 DEBUG:boto:Data: 2022-09-02T05:59:25.732 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.732 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.733 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.734 DEBUG:boto:Params: {} 2022-09-02T05:59:25.735 DEBUG:boto:Token: None 2022-09-02T05:59:25.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE= 2022-09-02T05:59:25.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE=', 'Content-Length': '0'} 2022-09-02T05:59:25.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a2e111104281cb-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.741 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.741 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.742 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.743 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.743 DEBUG:boto:Method: GET 2022-09-02T05:59:25.744 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.745 DEBUG:boto:Data: 2022-09-02T05:59:25.745 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.747 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.748 DEBUG:boto:Params: {} 2022-09-02T05:59:25.749 DEBUG:boto:Token: None 2022-09-02T05:59:25.750 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE= 2022-09-02T05:59:25.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE=', 'Content-Length': '0'} 2022-09-02T05:59:25.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f47e9edc77e88e29-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.756 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.757 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.758 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.758 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.759 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.760 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.761 DEBUG:boto:Data: 2022-09-02T05:59:25.761 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.763 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.764 DEBUG:boto:Params: {} 2022-09-02T05:59:25.764 DEBUG:boto:Token: None 2022-09-02T05:59:25.765 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ccSOD15uMb0VLb6DWwFNBtNfu5s= 2022-09-02T05:59:25.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ccSOD15uMb0VLb6DWwFNBtNfu5s=', 'Content-Length': '0'} 2022-09-02T05:59:25.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001152b95256c1ad4-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.770 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.771 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.771 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.772 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.773 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.773 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.774 DEBUG:boto:Data: 2022-09-02T05:59:25.775 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.776 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.777 DEBUG:boto:Params: {} 2022-09-02T05:59:25.778 DEBUG:boto:Token: None 2022-09-02T05:59:25.779 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ccSOD15uMb0VLb6DWwFNBtNfu5s= 2022-09-02T05:59:25.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ccSOD15uMb0VLb6DWwFNBtNfu5s=', 'Content-Length': '0'} 2022-09-02T05:59:25.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049de5bc2233c1e71-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:25.784 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.785 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.786 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.787 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.788 DEBUG:boto:Method: GET 2022-09-02T05:59:25.789 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.790 DEBUG:boto:Data: 2022-09-02T05:59:25.791 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.792 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.793 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.793 DEBUG:boto:Params: {} 2022-09-02T05:59:25.794 DEBUG:boto:Token: None 2022-09-02T05:59:25.795 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE= 2022-09-02T05:59:25.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE=', 'Content-Length': '0'} 2022-09-02T05:59:25.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b92f1b30a28c443-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.800 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.801 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.802 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.803 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.804 DEBUG:boto:Method: GET 2022-09-02T05:59:25.805 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.806 DEBUG:boto:Data: 2022-09-02T05:59:25.807 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.809 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.810 DEBUG:boto:Params: {} 2022-09-02T05:59:25.811 DEBUG:boto:Token: None 2022-09-02T05:59:25.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:25.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE= 2022-09-02T05:59:25.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:52SZtYkZA//5T4QnDIhRmKyT4SE=', 'Content-Length': '0'} 2022-09-02T05:59:25.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe6cd7f73e36d91-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.858 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.859 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.860 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.861 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.862 DEBUG:boto:Method: GET 2022-09-02T05:59:25.862 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.863 DEBUG:boto:Data: 2022-09-02T05:59:25.864 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.866 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.867 DEBUG:boto:Params: {} 2022-09-02T05:59:25.868 DEBUG:boto:Token: None 2022-09-02T05:59:25.868 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2fd67140928d78-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.874 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.875 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.876 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.876 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.877 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.878 DEBUG:boto:Method: GET 2022-09-02T05:59:25.879 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.879 DEBUG:boto:Data: 2022-09-02T05:59:25.880 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.881 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.882 DEBUG:boto:Params: {} 2022-09-02T05:59:25.883 DEBUG:boto:Token: None 2022-09-02T05:59:25.883 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab41ba86a3757b3b-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.891 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.893 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.894 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.895 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.895 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:25.896 DEBUG:boto:Data: 2022-09-02T05:59:25.897 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.898 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.899 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.900 DEBUG:boto:Params: {} 2022-09-02T05:59:25.901 DEBUG:boto:Token: None 2022-09-02T05:59:25.902 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116 2022-09-02T05:59:25.903 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SpVXzRX0b4uAOcEVWIvkvVk2xWM= 2022-09-02T05:59:25.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SpVXzRX0b4uAOcEVWIvkvVk2xWM=', 'Content-Length': '0'} 2022-09-02T05:59:25.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000795de894232afd29-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.907 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.908 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.909 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.909 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:25.910 DEBUG:boto:Data: 2022-09-02T05:59:25.910 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.911 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.912 DEBUG:boto:Params: {} 2022-09-02T05:59:25.913 DEBUG:boto:Token: None 2022-09-02T05:59:25.913 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116 2022-09-02T05:59:25.914 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SpVXzRX0b4uAOcEVWIvkvVk2xWM= 2022-09-02T05:59:25.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SpVXzRX0b4uAOcEVWIvkvVk2xWM=', 'Content-Length': '0'} 2022-09-02T05:59:25.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332929abda39c77e-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:25.919 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.919 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.920 DEBUG:boto:Method: GET 2022-09-02T05:59:25.920 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:25.921 DEBUG:boto:Data: 2022-09-02T05:59:25.921 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.922 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.923 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.923 DEBUG:boto:Params: {} 2022-09-02T05:59:25.924 DEBUG:boto:Token: None 2022-09-02T05:59:25.924 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116 2022-09-02T05:59:25.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bCI3FESokpl3ckHKr6NtQACDRg8= 2022-09-02T05:59:25.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bCI3FESokpl3ckHKr6NtQACDRg8=', 'Content-Length': '0'} 2022-09-02T05:59:25.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fce0e87a301342e-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.929 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:25.929 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:25.930 DEBUG:boto:Method: GET 2022-09-02T05:59:25.930 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:25.931 DEBUG:boto:Data: 2022-09-02T05:59:25.931 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.932 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.933 DEBUG:boto:Params: {} 2022-09-02T05:59:25.934 DEBUG:boto:Token: None 2022-09-02T05:59:25.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj116 2022-09-02T05:59:25.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bCI3FESokpl3ckHKr6NtQACDRg8= 2022-09-02T05:59:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bCI3FESokpl3ckHKr6NtQACDRg8=', 'Content-Length': '0'} 2022-09-02T05:59:25.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8190e50a9a76ad-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.938 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.939 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.940 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.940 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.941 DEBUG:boto:Method: GET 2022-09-02T05:59:25.941 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.942 DEBUG:boto:Data: 2022-09-02T05:59:25.942 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.943 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.943 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.944 DEBUG:boto:Params: {} 2022-09-02T05:59:25.945 DEBUG:boto:Token: None 2022-09-02T05:59:25.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.946 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b767404edd5122-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.950 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.951 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:25.951 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:25.952 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.953 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.953 DEBUG:boto:Method: GET 2022-09-02T05:59:25.954 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:25.954 DEBUG:boto:Data: 2022-09-02T05:59:25.955 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.956 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.957 DEBUG:boto:Params: {} 2022-09-02T05:59:25.957 DEBUG:boto:Token: None 2022-09-02T05:59:25.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/ 2022-09-02T05:59:25.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4= 2022-09-02T05:59:25.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZjQ8L5xRGqbLR+REqTRlsQ7xLe4=', 'Content-Length': '0'} 2022-09-02T05:59:25.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a1e0d2a4d9143e-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.964 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:25.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:25.965 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:25.966 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:25.967 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.967 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.968 DEBUG:boto:Method: GET 2022-09-02T05:59:25.968 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.969 DEBUG:boto:Data: 2022-09-02T05:59:25.970 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.971 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.972 DEBUG:boto:Params: {} 2022-09-02T05:59:25.972 DEBUG:boto:Token: None 2022-09-02T05:59:25.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qZGVvvOXm4fqdRpjCTuGhwREZnA= 2022-09-02T05:59:25.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qZGVvvOXm4fqdRpjCTuGhwREZnA=', 'Content-Length': '0'} 2022-09-02T05:59:25.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c82af721d8e4f1fd-0063119bbd-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.978 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:25.979 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:25.979 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.980 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.980 DEBUG:boto:Method: GET 2022-09-02T05:59:25.981 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.982 DEBUG:boto:Data: 2022-09-02T05:59:25.982 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:25.984 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.984 DEBUG:boto:Params: {} 2022-09-02T05:59:25.985 DEBUG:boto:Token: None 2022-09-02T05:59:25.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.986 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qZGVvvOXm4fqdRpjCTuGhwREZnA= 2022-09-02T05:59:25.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qZGVvvOXm4fqdRpjCTuGhwREZnA=', 'Content-Length': '0'} 2022-09-02T05:59:25.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef05f2ab22e96dbd-0063119bbd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:25 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:25.991 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:25.991 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:25.992 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.992 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.993 DEBUG:boto:Method: HEAD 2022-09-02T05:59:25.993 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.994 DEBUG:boto:Data: 2022-09-02T05:59:25.994 DEBUG:boto:Headers: {} 2022-09-02T05:59:25.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:25.995 DEBUG:boto:Port: 8000 2022-09-02T05:59:25.996 DEBUG:boto:Params: {} 2022-09-02T05:59:25.997 DEBUG:boto:Token: None 2022-09-02T05:59:25.998 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:25 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:25.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8v20+0dcYZkm9MBUFd4OigsAilI= 2022-09-02T05:59:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:25 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8v20+0dcYZkm9MBUFd4OigsAilI=', 'Content-Length': '0'} 2022-09-02T05:59:26.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bca67be7398a161d-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.004 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.005 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.005 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.006 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.006 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.007 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.007 DEBUG:boto:Data: 2022-09-02T05:59:26.008 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.009 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.009 DEBUG:boto:Params: {} 2022-09-02T05:59:26.010 DEBUG:boto:Token: None 2022-09-02T05:59:26.011 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.011 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kvWuuzpBdmS0AtxDtslSvharLIo= 2022-09-02T05:59:26.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kvWuuzpBdmS0AtxDtslSvharLIo=', 'Content-Length': '0'} 2022-09-02T05:59:26.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003089c8950978ef27-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:26.015 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.016 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.016 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.017 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.017 DEBUG:boto:Method: GET 2022-09-02T05:59:26.018 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.018 DEBUG:boto:Data: 2022-09-02T05:59:26.019 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.020 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.020 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.021 DEBUG:boto:Params: {} 2022-09-02T05:59:26.021 DEBUG:boto:Token: None 2022-09-02T05:59:26.022 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:huhulGM9n+42KiaRO/9ef9c1mbg= 2022-09-02T05:59:26.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:huhulGM9n+42KiaRO/9ef9c1mbg=', 'Content-Length': '0'} 2022-09-02T05:59:26.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094edfbc6d4a400d4-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.070 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.071 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.072 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.072 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.073 DEBUG:boto:Method: GET 2022-09-02T05:59:26.073 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.074 DEBUG:boto:Data: 2022-09-02T05:59:26.075 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.076 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.076 DEBUG:boto:Params: {} 2022-09-02T05:59:26.077 DEBUG:boto:Token: None 2022-09-02T05:59:26.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:26.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:huhulGM9n+42KiaRO/9ef9c1mbg= 2022-09-02T05:59:26.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:huhulGM9n+42KiaRO/9ef9c1mbg=', 'Content-Length': '0'} 2022-09-02T05:59:26.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faefb4a3b2bdf6f0-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.082 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.083 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.083 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.084 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.084 DEBUG:boto:Method: GET 2022-09-02T05:59:26.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.086 DEBUG:boto:Data: 2022-09-02T05:59:26.086 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.087 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.088 DEBUG:boto:Params: {} 2022-09-02T05:59:26.089 DEBUG:boto:Token: None 2022-09-02T05:59:26.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae59f3756d98407e-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.095 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.096 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.097 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.097 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.098 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.099 DEBUG:boto:Method: GET 2022-09-02T05:59:26.099 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.100 DEBUG:boto:Data: 2022-09-02T05:59:26.100 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.101 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.103 DEBUG:boto:Params: {} 2022-09-02T05:59:26.103 DEBUG:boto:Token: None 2022-09-02T05:59:26.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c47979c3f46fd8ad-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.110 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.111 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.111 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.112 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.112 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:26.113 DEBUG:boto:Data: 2022-09-02T05:59:26.113 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.114 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.114 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.115 DEBUG:boto:Params: {} 2022-09-02T05:59:26.116 DEBUG:boto:Token: None 2022-09-02T05:59:26.116 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124 2022-09-02T05:59:26.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2h+f8TBKnvCiYpwpBG1zaBx0y8= 2022-09-02T05:59:26.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2h+f8TBKnvCiYpwpBG1zaBx0y8=', 'Content-Length': '0'} 2022-09-02T05:59:26.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000994b04b8331c2f9d-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.120 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.121 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.121 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.122 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:26.122 DEBUG:boto:Data: 2022-09-02T05:59:26.123 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.124 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.124 DEBUG:boto:Params: {} 2022-09-02T05:59:26.125 DEBUG:boto:Token: None 2022-09-02T05:59:26.126 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124 2022-09-02T05:59:26.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s2h+f8TBKnvCiYpwpBG1zaBx0y8= 2022-09-02T05:59:26.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s2h+f8TBKnvCiYpwpBG1zaBx0y8=', 'Content-Length': '0'} 2022-09-02T05:59:26.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127783177ea13fe2-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:26.130 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.131 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.131 DEBUG:boto:Method: GET 2022-09-02T05:59:26.132 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:26.132 DEBUG:boto:Data: 2022-09-02T05:59:26.133 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.134 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.134 DEBUG:boto:Params: {} 2022-09-02T05:59:26.135 DEBUG:boto:Token: None 2022-09-02T05:59:26.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124 2022-09-02T05:59:26.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FUOMZMxS/5OP26o5w6x7yV7NHc4= 2022-09-02T05:59:26.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FUOMZMxS/5OP26o5w6x7yV7NHc4=', 'Content-Length': '0'} 2022-09-02T05:59:26.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b14720dca478611-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.140 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:26.141 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:26.141 DEBUG:boto:Method: GET 2022-09-02T05:59:26.142 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:26.142 DEBUG:boto:Data: 2022-09-02T05:59:26.143 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.144 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.144 DEBUG:boto:Params: {} 2022-09-02T05:59:26.145 DEBUG:boto:Token: None 2022-09-02T05:59:26.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj124 2022-09-02T05:59:26.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FUOMZMxS/5OP26o5w6x7yV7NHc4= 2022-09-02T05:59:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FUOMZMxS/5OP26o5w6x7yV7NHc4=', 'Content-Length': '0'} 2022-09-02T05:59:26.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e9996c8d08fc852-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.150 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.150 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.151 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.151 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.152 DEBUG:boto:Method: GET 2022-09-02T05:59:26.152 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.153 DEBUG:boto:Data: 2022-09-02T05:59:26.154 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.154 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.155 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.155 DEBUG:boto:Params: {} 2022-09-02T05:59:26.156 DEBUG:boto:Token: None 2022-09-02T05:59:26.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.157 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000558067bc6c34d049-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.161 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.162 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.163 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.163 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.164 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.164 DEBUG:boto:Method: GET 2022-09-02T05:59:26.165 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:26.165 DEBUG:boto:Data: 2022-09-02T05:59:26.166 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.167 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.168 DEBUG:boto:Params: {} 2022-09-02T05:59:26.168 DEBUG:boto:Token: None 2022-09-02T05:59:26.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde7a99010bf1751-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.175 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:26.176 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.177 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.177 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.178 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.178 DEBUG:boto:Method: GET 2022-09-02T05:59:26.179 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.180 DEBUG:boto:Data: 2022-09-02T05:59:26.180 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.181 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.182 DEBUG:boto:Params: {} 2022-09-02T05:59:26.182 DEBUG:boto:Token: None 2022-09-02T05:59:26.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc= 2022-09-02T05:59:26.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc=', 'Content-Length': '0'} 2022-09-02T05:59:26.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7498f81bfb61cd6-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.187 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.188 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.188 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.189 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.190 DEBUG:boto:Method: GET 2022-09-02T05:59:26.190 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.191 DEBUG:boto:Data: 2022-09-02T05:59:26.191 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.192 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.193 DEBUG:boto:Params: {} 2022-09-02T05:59:26.194 DEBUG:boto:Token: None 2022-09-02T05:59:26.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc= 2022-09-02T05:59:26.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc=', 'Content-Length': '0'} 2022-09-02T05:59:26.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006331d7f02de9e95d-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.238 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.239 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.239 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.240 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.240 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.241 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.241 DEBUG:boto:Data: 2022-09-02T05:59:26.242 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.243 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.243 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.244 DEBUG:boto:Params: {} 2022-09-02T05:59:26.245 DEBUG:boto:Token: None 2022-09-02T05:59:26.245 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.246 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pdC4nHPgSplMeARNjZd/MrI5YBQ= 2022-09-02T05:59:26.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pdC4nHPgSplMeARNjZd/MrI5YBQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432a2515fb2a45d9-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.250 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.250 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.251 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.252 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.253 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.253 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.254 DEBUG:boto:Data: 2022-09-02T05:59:26.255 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.256 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.256 DEBUG:boto:Params: {} 2022-09-02T05:59:26.257 DEBUG:boto:Token: None 2022-09-02T05:59:26.258 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.258 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pdC4nHPgSplMeARNjZd/MrI5YBQ= 2022-09-02T05:59:26.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pdC4nHPgSplMeARNjZd/MrI5YBQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31e639b33c4d6e7-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:26.263 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.263 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.264 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.264 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.265 DEBUG:boto:Method: GET 2022-09-02T05:59:26.266 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.266 DEBUG:boto:Data: 2022-09-02T05:59:26.267 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.268 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.268 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.269 DEBUG:boto:Params: {} 2022-09-02T05:59:26.269 DEBUG:boto:Token: None 2022-09-02T05:59:26.270 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.271 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc= 2022-09-02T05:59:26.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc=', 'Content-Length': '0'} 2022-09-02T05:59:26.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b717f4742f12170-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.275 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.276 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.276 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.277 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.277 DEBUG:boto:Method: GET 2022-09-02T05:59:26.278 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.279 DEBUG:boto:Data: 2022-09-02T05:59:26.279 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.281 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.281 DEBUG:boto:Params: {} 2022-09-02T05:59:26.282 DEBUG:boto:Token: None 2022-09-02T05:59:26.283 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:26.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc= 2022-09-02T05:59:26.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KyIum356Cw2nM2wNrXaii4qoJUc=', 'Content-Length': '0'} 2022-09-02T05:59:26.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437dc3d25d905c86-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.289 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.290 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.290 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.291 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.292 DEBUG:boto:Method: GET 2022-09-02T05:59:26.292 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.293 DEBUG:boto:Data: 2022-09-02T05:59:26.293 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.294 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.295 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.296 DEBUG:boto:Params: {} 2022-09-02T05:59:26.297 DEBUG:boto:Token: None 2022-09-02T05:59:26.298 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53f96eb4e9e979e-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.305 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.306 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.307 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.308 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.309 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.310 DEBUG:boto:Method: GET 2022-09-02T05:59:26.311 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.312 DEBUG:boto:Data: 2022-09-02T05:59:26.313 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.314 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.315 DEBUG:boto:Params: {} 2022-09-02T05:59:26.315 DEBUG:boto:Token: None 2022-09-02T05:59:26.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d9ebc81ea7e616-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.322 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.323 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.324 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.325 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.325 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:26.326 DEBUG:boto:Data: 2022-09-02T05:59:26.326 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.327 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.328 DEBUG:boto:Params: {} 2022-09-02T05:59:26.329 DEBUG:boto:Token: None 2022-09-02T05:59:26.330 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145 2022-09-02T05:59:26.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ATMo3Mxq2ZmBuuWdyY5ipSJ7FlU= 2022-09-02T05:59:26.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ATMo3Mxq2ZmBuuWdyY5ipSJ7FlU=', 'Content-Length': '0'} 2022-09-02T05:59:26.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5e278d0b74e51e-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.336 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.336 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.337 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.337 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:26.338 DEBUG:boto:Data: 2022-09-02T05:59:26.339 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.340 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.341 DEBUG:boto:Params: {} 2022-09-02T05:59:26.342 DEBUG:boto:Token: None 2022-09-02T05:59:26.343 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145 2022-09-02T05:59:26.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ATMo3Mxq2ZmBuuWdyY5ipSJ7FlU= 2022-09-02T05:59:26.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ATMo3Mxq2ZmBuuWdyY5ipSJ7FlU=', 'Content-Length': '0'} 2022-09-02T05:59:26.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4cc5198d0e59b4-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:26.349 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.350 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.351 DEBUG:boto:Method: GET 2022-09-02T05:59:26.351 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:26.352 DEBUG:boto:Data: 2022-09-02T05:59:26.353 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.354 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.355 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.356 DEBUG:boto:Params: {} 2022-09-02T05:59:26.357 DEBUG:boto:Token: None 2022-09-02T05:59:26.358 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145 2022-09-02T05:59:26.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4UwPpr7IPDr243fCGxohc4ISjzQ= 2022-09-02T05:59:26.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4UwPpr7IPDr243fCGxohc4ISjzQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483fbe6c80dfe191-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.410 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:26.411 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:26.412 DEBUG:boto:Method: GET 2022-09-02T05:59:26.413 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:26.413 DEBUG:boto:Data: 2022-09-02T05:59:26.414 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.415 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.415 DEBUG:boto:Params: {} 2022-09-02T05:59:26.416 DEBUG:boto:Token: None 2022-09-02T05:59:26.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj145 2022-09-02T05:59:26.417 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4UwPpr7IPDr243fCGxohc4ISjzQ= 2022-09-02T05:59:26.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4UwPpr7IPDr243fCGxohc4ISjzQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff18ed29fe94e3f0-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.425 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.426 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.427 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.428 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.428 DEBUG:boto:Method: GET 2022-09-02T05:59:26.429 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.430 DEBUG:boto:Data: 2022-09-02T05:59:26.431 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.432 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.433 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.433 DEBUG:boto:Params: {} 2022-09-02T05:59:26.434 DEBUG:boto:Token: None 2022-09-02T05:59:26.435 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.437 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0b87efb3e52cdc8-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.441 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.442 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.442 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.443 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.444 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.445 DEBUG:boto:Method: GET 2022-09-02T05:59:26.446 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:26.447 DEBUG:boto:Data: 2022-09-02T05:59:26.447 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.448 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.449 DEBUG:boto:Params: {} 2022-09-02T05:59:26.449 DEBUG:boto:Token: None 2022-09-02T05:59:26.450 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edfba5621062ba92-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.458 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:26.460 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.461 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.462 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.463 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.463 DEBUG:boto:Method: GET 2022-09-02T05:59:26.464 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.465 DEBUG:boto:Data: 2022-09-02T05:59:26.465 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.466 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.467 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.468 DEBUG:boto:Params: {} 2022-09-02T05:59:26.468 DEBUG:boto:Token: None 2022-09-02T05:59:26.469 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ= 2022-09-02T05:59:26.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef364747ec30892-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.484 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.485 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.486 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.486 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.487 DEBUG:boto:Method: GET 2022-09-02T05:59:26.488 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.488 DEBUG:boto:Data: 2022-09-02T05:59:26.489 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.490 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.491 DEBUG:boto:Params: {} 2022-09-02T05:59:26.492 DEBUG:boto:Token: None 2022-09-02T05:59:26.493 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.493 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ= 2022-09-02T05:59:26.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062ae4f96b91eb277-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.542 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.543 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.543 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.544 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.544 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.545 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.545 DEBUG:boto:Data: 2022-09-02T05:59:26.546 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.546 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.547 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.547 DEBUG:boto:Params: {} 2022-09-02T05:59:26.548 DEBUG:boto:Token: None 2022-09-02T05:59:26.548 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+y2fVkmnIJTPgmER9jwH16/CQOA= 2022-09-02T05:59:26.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+y2fVkmnIJTPgmER9jwH16/CQOA=', 'Content-Length': '0'} 2022-09-02T05:59:26.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d432f02f6a3146-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.552 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.553 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.553 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.554 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.554 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.555 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.555 DEBUG:boto:Data: 2022-09-02T05:59:26.556 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.557 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.557 DEBUG:boto:Params: {} 2022-09-02T05:59:26.558 DEBUG:boto:Token: None 2022-09-02T05:59:26.558 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.559 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+y2fVkmnIJTPgmER9jwH16/CQOA= 2022-09-02T05:59:26.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+y2fVkmnIJTPgmER9jwH16/CQOA=', 'Content-Length': '0'} 2022-09-02T05:59:26.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd9978bc57293240-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:26.562 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.563 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.564 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.564 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.565 DEBUG:boto:Method: GET 2022-09-02T05:59:26.565 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.566 DEBUG:boto:Data: 2022-09-02T05:59:26.566 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.567 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.567 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.568 DEBUG:boto:Params: {} 2022-09-02T05:59:26.568 DEBUG:boto:Token: None 2022-09-02T05:59:26.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ= 2022-09-02T05:59:26.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f9f55a7ace7f56-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.573 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.573 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.574 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.575 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.575 DEBUG:boto:Method: GET 2022-09-02T05:59:26.576 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.576 DEBUG:boto:Data: 2022-09-02T05:59:26.577 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.578 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.578 DEBUG:boto:Params: {} 2022-09-02T05:59:26.579 DEBUG:boto:Token: None 2022-09-02T05:59:26.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:26.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ= 2022-09-02T05:59:26.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WpW3cra7DcPg1Z7N5L2N1jXXxCQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fd09d3412bfc15f-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.583 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.584 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.585 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.585 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.586 DEBUG:boto:Method: GET 2022-09-02T05:59:26.586 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.587 DEBUG:boto:Data: 2022-09-02T05:59:26.587 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.588 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.589 DEBUG:boto:Params: {} 2022-09-02T05:59:26.589 DEBUG:boto:Token: None 2022-09-02T05:59:26.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044620c7df0bd6329-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.595 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.595 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.596 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.596 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.597 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.597 DEBUG:boto:Method: GET 2022-09-02T05:59:26.598 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.599 DEBUG:boto:Data: 2022-09-02T05:59:26.599 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.600 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.601 DEBUG:boto:Params: {} 2022-09-02T05:59:26.601 DEBUG:boto:Token: None 2022-09-02T05:59:26.602 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a26b1b3941e6bd43-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.608 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.609 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.609 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.610 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.610 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:26.611 DEBUG:boto:Data: 2022-09-02T05:59:26.611 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.612 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.613 DEBUG:boto:Params: {} 2022-09-02T05:59:26.614 DEBUG:boto:Token: None 2022-09-02T05:59:26.614 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148 2022-09-02T05:59:26.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LUYvoi9cd95eYBW1TQGIPQQ2usE= 2022-09-02T05:59:26.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LUYvoi9cd95eYBW1TQGIPQQ2usE=', 'Content-Length': '0'} 2022-09-02T05:59:26.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1545367a911799f-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.618 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.619 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.619 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.620 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:26.620 DEBUG:boto:Data: 2022-09-02T05:59:26.621 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.627 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.627 DEBUG:boto:Params: {} 2022-09-02T05:59:26.628 DEBUG:boto:Token: None 2022-09-02T05:59:26.628 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148 2022-09-02T05:59:26.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LUYvoi9cd95eYBW1TQGIPQQ2usE= 2022-09-02T05:59:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LUYvoi9cd95eYBW1TQGIPQQ2usE=', 'Content-Length': '0'} 2022-09-02T05:59:26.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a73bf4c105963b84-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:26.632 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.633 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.634 DEBUG:boto:Method: GET 2022-09-02T05:59:26.634 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:26.635 DEBUG:boto:Data: 2022-09-02T05:59:26.635 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.636 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.636 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.637 DEBUG:boto:Params: {} 2022-09-02T05:59:26.637 DEBUG:boto:Token: None 2022-09-02T05:59:26.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148 2022-09-02T05:59:26.639 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dBpwhCaoqDDqqR1ZwHoh/uu6Iwk= 2022-09-02T05:59:26.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dBpwhCaoqDDqqR1ZwHoh/uu6Iwk=', 'Content-Length': '0'} 2022-09-02T05:59:26.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f96c329479e1d7-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.642 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:26.643 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:26.643 DEBUG:boto:Method: GET 2022-09-02T05:59:26.644 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:26.644 DEBUG:boto:Data: 2022-09-02T05:59:26.645 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.646 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.647 DEBUG:boto:Params: {} 2022-09-02T05:59:26.647 DEBUG:boto:Token: None 2022-09-02T05:59:26.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj148 2022-09-02T05:59:26.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dBpwhCaoqDDqqR1ZwHoh/uu6Iwk= 2022-09-02T05:59:26.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dBpwhCaoqDDqqR1ZwHoh/uu6Iwk=', 'Content-Length': '0'} 2022-09-02T05:59:26.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ddc6cfd4d7f96d5-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.698 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.699 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.699 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.700 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.700 DEBUG:boto:Method: GET 2022-09-02T05:59:26.701 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.701 DEBUG:boto:Data: 2022-09-02T05:59:26.702 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.702 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.703 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.703 DEBUG:boto:Params: {} 2022-09-02T05:59:26.704 DEBUG:boto:Token: None 2022-09-02T05:59:26.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d1e471209493f7-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.709 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.710 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.711 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.711 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.712 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.712 DEBUG:boto:Method: GET 2022-09-02T05:59:26.713 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:26.713 DEBUG:boto:Data: 2022-09-02T05:59:26.714 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.715 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.715 DEBUG:boto:Params: {} 2022-09-02T05:59:26.716 DEBUG:boto:Token: None 2022-09-02T05:59:26.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c22538d22ad93a55-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.722 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:26.724 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.724 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.725 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.725 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.726 DEBUG:boto:Method: GET 2022-09-02T05:59:26.726 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.727 DEBUG:boto:Data: 2022-09-02T05:59:26.727 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.728 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.728 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.729 DEBUG:boto:Params: {} 2022-09-02T05:59:26.729 DEBUG:boto:Token: None 2022-09-02T05:59:26.730 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.731 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY= 2022-09-02T05:59:26.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY=', 'Content-Length': '0'} 2022-09-02T05:59:26.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4527adaf4b8ee2b-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.734 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.735 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.735 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.736 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.736 DEBUG:boto:Method: GET 2022-09-02T05:59:26.737 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.737 DEBUG:boto:Data: 2022-09-02T05:59:26.738 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.739 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.739 DEBUG:boto:Params: {} 2022-09-02T05:59:26.740 DEBUG:boto:Token: None 2022-09-02T05:59:26.740 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.741 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY= 2022-09-02T05:59:26.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY=', 'Content-Length': '0'} 2022-09-02T05:59:26.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033c46b2e79a07f56-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.745 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.745 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.746 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.746 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.747 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.747 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.748 DEBUG:boto:Data: 2022-09-02T05:59:26.748 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.749 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.749 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.750 DEBUG:boto:Params: {} 2022-09-02T05:59:26.750 DEBUG:boto:Token: None 2022-09-02T05:59:26.751 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.751 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cdkqvTPTRzT9GrE1zpv+mXpxW4I= 2022-09-02T05:59:26.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cdkqvTPTRzT9GrE1zpv+mXpxW4I=', 'Content-Length': '0'} 2022-09-02T05:59:26.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007896f2bd19daa591-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.755 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.755 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.756 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.756 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.757 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.757 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.758 DEBUG:boto:Data: 2022-09-02T05:59:26.758 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.759 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.760 DEBUG:boto:Params: {} 2022-09-02T05:59:26.760 DEBUG:boto:Token: None 2022-09-02T05:59:26.761 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cdkqvTPTRzT9GrE1zpv+mXpxW4I= 2022-09-02T05:59:26.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cdkqvTPTRzT9GrE1zpv+mXpxW4I=', 'Content-Length': '0'} 2022-09-02T05:59:26.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969d09476526ab33-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:26.765 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.766 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.767 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.767 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.768 DEBUG:boto:Method: GET 2022-09-02T05:59:26.768 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.769 DEBUG:boto:Data: 2022-09-02T05:59:26.770 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.771 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.771 DEBUG:boto:Params: {} 2022-09-02T05:59:26.772 DEBUG:boto:Token: None 2022-09-02T05:59:26.772 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY= 2022-09-02T05:59:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY=', 'Content-Length': '0'} 2022-09-02T05:59:26.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147c48e105388ab3-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.822 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.823 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.824 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.824 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.825 DEBUG:boto:Method: GET 2022-09-02T05:59:26.825 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.826 DEBUG:boto:Data: 2022-09-02T05:59:26.826 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.828 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.828 DEBUG:boto:Params: {} 2022-09-02T05:59:26.829 DEBUG:boto:Token: None 2022-09-02T05:59:26.830 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:26.830 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY= 2022-09-02T05:59:26.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bBSAk7zsbFYGJ0k/shb8YXyZHUY=', 'Content-Length': '0'} 2022-09-02T05:59:26.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b869dd7a7af27f42-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.835 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.835 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.836 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.837 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.838 DEBUG:boto:Method: GET 2022-09-02T05:59:26.838 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.839 DEBUG:boto:Data: 2022-09-02T05:59:26.839 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.840 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.840 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.841 DEBUG:boto:Params: {} 2022-09-02T05:59:26.841 DEBUG:boto:Token: None 2022-09-02T05:59:26.842 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc7f06a37d20644c-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.848 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.849 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.849 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.850 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.851 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.851 DEBUG:boto:Method: GET 2022-09-02T05:59:26.852 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.852 DEBUG:boto:Data: 2022-09-02T05:59:26.853 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.854 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.854 DEBUG:boto:Params: {} 2022-09-02T05:59:26.855 DEBUG:boto:Token: None 2022-09-02T05:59:26.855 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e661eb56b3bf08f3-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.861 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.862 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.863 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.863 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.864 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:26.864 DEBUG:boto:Data: 2022-09-02T05:59:26.865 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.866 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.867 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.867 DEBUG:boto:Params: {} 2022-09-02T05:59:26.868 DEBUG:boto:Token: None 2022-09-02T05:59:26.869 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155 2022-09-02T05:59:26.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bC62AwvJn3di8gFMAWkyQZiDXAQ= 2022-09-02T05:59:26.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bC62AwvJn3di8gFMAWkyQZiDXAQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def65e2f856d3f11-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.873 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.874 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.874 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.875 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:26.875 DEBUG:boto:Data: 2022-09-02T05:59:26.876 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.877 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.878 DEBUG:boto:Params: {} 2022-09-02T05:59:26.878 DEBUG:boto:Token: None 2022-09-02T05:59:26.879 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155 2022-09-02T05:59:26.879 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bC62AwvJn3di8gFMAWkyQZiDXAQ= 2022-09-02T05:59:26.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bC62AwvJn3di8gFMAWkyQZiDXAQ=', 'Content-Length': '0'} 2022-09-02T05:59:26.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f407a7bdf510835e-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:26.883 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.884 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.884 DEBUG:boto:Method: GET 2022-09-02T05:59:26.885 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:26.885 DEBUG:boto:Data: 2022-09-02T05:59:26.886 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.887 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.888 DEBUG:boto:Params: {} 2022-09-02T05:59:26.888 DEBUG:boto:Token: None 2022-09-02T05:59:26.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155 2022-09-02T05:59:26.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rJTl/Iy8AwWPhxRpdXDPtRGrOIE= 2022-09-02T05:59:26.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rJTl/Iy8AwWPhxRpdXDPtRGrOIE=', 'Content-Length': '0'} 2022-09-02T05:59:26.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003912fc04b6742649-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.893 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:26.894 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:26.894 DEBUG:boto:Method: GET 2022-09-02T05:59:26.895 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:26.895 DEBUG:boto:Data: 2022-09-02T05:59:26.896 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.897 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.897 DEBUG:boto:Params: {} 2022-09-02T05:59:26.898 DEBUG:boto:Token: None 2022-09-02T05:59:26.899 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj155 2022-09-02T05:59:26.899 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rJTl/Iy8AwWPhxRpdXDPtRGrOIE= 2022-09-02T05:59:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rJTl/Iy8AwWPhxRpdXDPtRGrOIE=', 'Content-Length': '0'} 2022-09-02T05:59:26.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb38abc8be4deaa-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.942 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.943 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.944 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.944 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.945 DEBUG:boto:Method: GET 2022-09-02T05:59:26.945 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.946 DEBUG:boto:Data: 2022-09-02T05:59:26.946 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.947 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.948 DEBUG:boto:Params: {} 2022-09-02T05:59:26.949 DEBUG:boto:Token: None 2022-09-02T05:59:26.949 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e539061270b1872-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.954 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.955 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:26.955 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:26.956 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.956 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.957 DEBUG:boto:Method: GET 2022-09-02T05:59:26.958 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:26.958 DEBUG:boto:Data: 2022-09-02T05:59:26.959 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.960 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.960 DEBUG:boto:Params: {} 2022-09-02T05:59:26.961 DEBUG:boto:Token: None 2022-09-02T05:59:26.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/ 2022-09-02T05:59:26.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE= 2022-09-02T05:59:26.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xn7A8jfIYo8nvP3dxoifu3uHXVE=', 'Content-Length': '0'} 2022-09-02T05:59:26.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077de07b7c4232662-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.968 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:26.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:26.969 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:26.970 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:26.970 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.971 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.971 DEBUG:boto:Method: GET 2022-09-02T05:59:26.972 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.972 DEBUG:boto:Data: 2022-09-02T05:59:26.973 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.973 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.974 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.974 DEBUG:boto:Params: {} 2022-09-02T05:59:26.975 DEBUG:boto:Token: None 2022-09-02T05:59:26.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.976 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fEW6h2RxY0h9017M1V3lhxMaTmM= 2022-09-02T05:59:26.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fEW6h2RxY0h9017M1V3lhxMaTmM=', 'Content-Length': '0'} 2022-09-02T05:59:26.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f03c608fd6e1622e-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.980 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:26.980 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:26.981 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.981 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.982 DEBUG:boto:Method: GET 2022-09-02T05:59:26.982 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.983 DEBUG:boto:Data: 2022-09-02T05:59:26.983 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:26.985 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.985 DEBUG:boto:Params: {} 2022-09-02T05:59:26.986 DEBUG:boto:Token: None 2022-09-02T05:59:26.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fEW6h2RxY0h9017M1V3lhxMaTmM= 2022-09-02T05:59:26.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fEW6h2RxY0h9017M1V3lhxMaTmM=', 'Content-Length': '0'} 2022-09-02T05:59:26.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006bf803771f7bf1d-0063119bbe-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:26 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:26.991 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:26.991 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:26.992 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.992 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.993 DEBUG:boto:Method: HEAD 2022-09-02T05:59:26.993 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.994 DEBUG:boto:Data: 2022-09-02T05:59:26.994 DEBUG:boto:Headers: {} 2022-09-02T05:59:26.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:26.995 DEBUG:boto:Port: 8000 2022-09-02T05:59:26.996 DEBUG:boto:Params: {} 2022-09-02T05:59:26.997 DEBUG:boto:Token: None 2022-09-02T05:59:26.997 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:26 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:26.998 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wNQzu6MD257jU41QNQNShcBqLQA= 2022-09-02T05:59:26.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:26 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wNQzu6MD257jU41QNQNShcBqLQA=', 'Content-Length': '0'} 2022-09-02T05:59:27.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a73ab7ad62fdf56-0063119bbe-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.001 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.002 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.002 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.003 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.004 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.004 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.005 DEBUG:boto:Data: 2022-09-02T05:59:27.005 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.006 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.007 DEBUG:boto:Params: {} 2022-09-02T05:59:27.008 DEBUG:boto:Token: None 2022-09-02T05:59:27.008 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I8xub8X6TW8eGSz6Bl1p5Sxjswk= 2022-09-02T05:59:27.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I8xub8X6TW8eGSz6Bl1p5Sxjswk=', 'Content-Length': '0'} 2022-09-02T05:59:27.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92abab7a3e9906f-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:27.014 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.015 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.015 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.016 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.016 DEBUG:boto:Method: GET 2022-09-02T05:59:27.017 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.017 DEBUG:boto:Data: 2022-09-02T05:59:27.018 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.019 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.019 DEBUG:boto:Params: {} 2022-09-02T05:59:27.020 DEBUG:boto:Token: None 2022-09-02T05:59:27.021 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7BO8zqDXuEiv82ZSh2nWc3afmMo= 2022-09-02T05:59:27.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7BO8zqDXuEiv82ZSh2nWc3afmMo=', 'Content-Length': '0'} 2022-09-02T05:59:27.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7754b24ed5e60a7-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.066 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.067 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.067 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.068 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.069 DEBUG:boto:Method: GET 2022-09-02T05:59:27.069 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.070 DEBUG:boto:Data: 2022-09-02T05:59:27.070 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.071 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.072 DEBUG:boto:Params: {} 2022-09-02T05:59:27.073 DEBUG:boto:Token: None 2022-09-02T05:59:27.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:27.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7BO8zqDXuEiv82ZSh2nWc3afmMo= 2022-09-02T05:59:27.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7BO8zqDXuEiv82ZSh2nWc3afmMo=', 'Content-Length': '0'} 2022-09-02T05:59:27.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe60baf787a6cac0-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.078 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.078 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.079 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.080 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.080 DEBUG:boto:Method: GET 2022-09-02T05:59:27.081 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.081 DEBUG:boto:Data: 2022-09-02T05:59:27.082 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.083 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.083 DEBUG:boto:Params: {} 2022-09-02T05:59:27.084 DEBUG:boto:Token: None 2022-09-02T05:59:27.084 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eb53280b9390dd9-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.089 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.090 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.090 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.091 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.091 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.092 DEBUG:boto:Method: GET 2022-09-02T05:59:27.092 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.093 DEBUG:boto:Data: 2022-09-02T05:59:27.093 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.094 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.095 DEBUG:boto:Params: {} 2022-09-02T05:59:27.096 DEBUG:boto:Token: None 2022-09-02T05:59:27.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f20c792e59a9d21-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.103 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.104 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.104 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.105 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.105 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:27.106 DEBUG:boto:Data: 2022-09-02T05:59:27.106 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.107 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.107 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.108 DEBUG:boto:Params: {} 2022-09-02T05:59:27.108 DEBUG:boto:Token: None 2022-09-02T05:59:27.109 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17 2022-09-02T05:59:27.110 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:upXpjM+202bLBSjzfmcUs9MO4oE= 2022-09-02T05:59:27.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:upXpjM+202bLBSjzfmcUs9MO4oE=', 'Content-Length': '0'} 2022-09-02T05:59:27.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe8ed088aa29f95-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.113 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.114 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.114 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.115 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:27.115 DEBUG:boto:Data: 2022-09-02T05:59:27.116 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.117 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.118 DEBUG:boto:Params: {} 2022-09-02T05:59:27.118 DEBUG:boto:Token: None 2022-09-02T05:59:27.119 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17 2022-09-02T05:59:27.119 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:upXpjM+202bLBSjzfmcUs9MO4oE= 2022-09-02T05:59:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:upXpjM+202bLBSjzfmcUs9MO4oE=', 'Content-Length': '0'} 2022-09-02T05:59:27.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef87814d9459d0bb-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:27.124 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.124 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.125 DEBUG:boto:Method: GET 2022-09-02T05:59:27.125 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:27.126 DEBUG:boto:Data: 2022-09-02T05:59:27.126 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.127 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.127 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.128 DEBUG:boto:Params: {} 2022-09-02T05:59:27.128 DEBUG:boto:Token: None 2022-09-02T05:59:27.129 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17 2022-09-02T05:59:27.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hqXdnDRzSyOtFibi4xsI8Sa6xu8= 2022-09-02T05:59:27.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hqXdnDRzSyOtFibi4xsI8Sa6xu8=', 'Content-Length': '0'} 2022-09-02T05:59:27.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24781501fda152e-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.134 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:27.134 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:27.135 DEBUG:boto:Method: GET 2022-09-02T05:59:27.135 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:27.136 DEBUG:boto:Data: 2022-09-02T05:59:27.136 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.137 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.138 DEBUG:boto:Params: {} 2022-09-02T05:59:27.138 DEBUG:boto:Token: None 2022-09-02T05:59:27.139 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj17 2022-09-02T05:59:27.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hqXdnDRzSyOtFibi4xsI8Sa6xu8= 2022-09-02T05:59:27.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hqXdnDRzSyOtFibi4xsI8Sa6xu8=', 'Content-Length': '0'} 2022-09-02T05:59:27.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d3d029b4c1ba3b-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.143 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.144 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.144 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.145 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.145 DEBUG:boto:Method: GET 2022-09-02T05:59:27.146 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.147 DEBUG:boto:Data: 2022-09-02T05:59:27.147 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.148 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.148 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.149 DEBUG:boto:Params: {} 2022-09-02T05:59:27.149 DEBUG:boto:Token: None 2022-09-02T05:59:27.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.151 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061996d3f39427d90-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.155 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.155 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.156 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.157 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.157 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.158 DEBUG:boto:Method: GET 2022-09-02T05:59:27.158 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:27.159 DEBUG:boto:Data: 2022-09-02T05:59:27.159 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.161 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.161 DEBUG:boto:Params: {} 2022-09-02T05:59:27.162 DEBUG:boto:Token: None 2022-09-02T05:59:27.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85ac2f93c04d2e9-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.210 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:27.212 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.212 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.213 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.213 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.214 DEBUG:boto:Method: GET 2022-09-02T05:59:27.214 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.215 DEBUG:boto:Data: 2022-09-02T05:59:27.216 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.217 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.217 DEBUG:boto:Params: {} 2022-09-02T05:59:27.218 DEBUG:boto:Token: None 2022-09-02T05:59:27.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA= 2022-09-02T05:59:27.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA=', 'Content-Length': '0'} 2022-09-02T05:59:27.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0dc5ac3422d626-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.223 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.224 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.224 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.225 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.225 DEBUG:boto:Method: GET 2022-09-02T05:59:27.226 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.226 DEBUG:boto:Data: 2022-09-02T05:59:27.227 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.228 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.228 DEBUG:boto:Params: {} 2022-09-02T05:59:27.229 DEBUG:boto:Token: None 2022-09-02T05:59:27.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.230 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA= 2022-09-02T05:59:27.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA=', 'Content-Length': '0'} 2022-09-02T05:59:27.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d3d5be359ea9cb-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.233 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.234 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.234 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.235 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.235 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.236 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.236 DEBUG:boto:Data: 2022-09-02T05:59:27.237 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.237 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.238 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.238 DEBUG:boto:Params: {} 2022-09-02T05:59:27.239 DEBUG:boto:Token: None 2022-09-02T05:59:27.239 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hc4UHtNhRolx9NMP3cTT8lEsyaU= 2022-09-02T05:59:27.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hc4UHtNhRolx9NMP3cTT8lEsyaU=', 'Content-Length': '0'} 2022-09-02T05:59:27.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd3b165a8f8d8b4-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.243 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.244 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.244 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.245 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.245 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.246 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.246 DEBUG:boto:Data: 2022-09-02T05:59:27.247 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.248 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.248 DEBUG:boto:Params: {} 2022-09-02T05:59:27.249 DEBUG:boto:Token: None 2022-09-02T05:59:27.249 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.250 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hc4UHtNhRolx9NMP3cTT8lEsyaU= 2022-09-02T05:59:27.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hc4UHtNhRolx9NMP3cTT8lEsyaU=', 'Content-Length': '0'} 2022-09-02T05:59:27.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003afa24198cc83ba9-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:27.254 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.254 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.255 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.255 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.256 DEBUG:boto:Method: GET 2022-09-02T05:59:27.256 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.257 DEBUG:boto:Data: 2022-09-02T05:59:27.257 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.258 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.258 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.259 DEBUG:boto:Params: {} 2022-09-02T05:59:27.260 DEBUG:boto:Token: None 2022-09-02T05:59:27.260 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.261 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA= 2022-09-02T05:59:27.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA=', 'Content-Length': '0'} 2022-09-02T05:59:27.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b509883cb57b5f85-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.306 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.307 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.308 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.309 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.309 DEBUG:boto:Method: GET 2022-09-02T05:59:27.310 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.310 DEBUG:boto:Data: 2022-09-02T05:59:27.311 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.312 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.313 DEBUG:boto:Params: {} 2022-09-02T05:59:27.314 DEBUG:boto:Token: None 2022-09-02T05:59:27.314 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:27.315 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA= 2022-09-02T05:59:27.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lhrAYXfntxXO8aMjvOnTDjM3sVA=', 'Content-Length': '0'} 2022-09-02T05:59:27.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99d21546f7c9b18-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.319 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.319 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.320 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.320 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.321 DEBUG:boto:Method: GET 2022-09-02T05:59:27.321 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.322 DEBUG:boto:Data: 2022-09-02T05:59:27.322 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.323 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.324 DEBUG:boto:Params: {} 2022-09-02T05:59:27.325 DEBUG:boto:Token: None 2022-09-02T05:59:27.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e548adf278656e-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.330 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.331 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.331 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.332 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.333 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.333 DEBUG:boto:Method: GET 2022-09-02T05:59:27.334 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.334 DEBUG:boto:Data: 2022-09-02T05:59:27.335 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.336 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.336 DEBUG:boto:Params: {} 2022-09-02T05:59:27.337 DEBUG:boto:Token: None 2022-09-02T05:59:27.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde3007a8cdfe485-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.344 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.345 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.346 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.346 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.347 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:27.347 DEBUG:boto:Data: 2022-09-02T05:59:27.348 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.349 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.349 DEBUG:boto:Params: {} 2022-09-02T05:59:27.350 DEBUG:boto:Token: None 2022-09-02T05:59:27.350 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185 2022-09-02T05:59:27.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f/bHoWETRbxUADfU3FDKTGSb1GU= 2022-09-02T05:59:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f/bHoWETRbxUADfU3FDKTGSb1GU=', 'Content-Length': '0'} 2022-09-02T05:59:27.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007410e9cc3488f9e9-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.354 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.355 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.355 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.356 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:27.356 DEBUG:boto:Data: 2022-09-02T05:59:27.357 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.358 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.358 DEBUG:boto:Params: {} 2022-09-02T05:59:27.359 DEBUG:boto:Token: None 2022-09-02T05:59:27.360 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185 2022-09-02T05:59:27.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f/bHoWETRbxUADfU3FDKTGSb1GU= 2022-09-02T05:59:27.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f/bHoWETRbxUADfU3FDKTGSb1GU=', 'Content-Length': '0'} 2022-09-02T05:59:27.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b27de23783288ed1-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:27.364 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.365 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.365 DEBUG:boto:Method: GET 2022-09-02T05:59:27.366 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:27.366 DEBUG:boto:Data: 2022-09-02T05:59:27.367 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.367 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.368 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.368 DEBUG:boto:Params: {} 2022-09-02T05:59:27.369 DEBUG:boto:Token: None 2022-09-02T05:59:27.370 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185 2022-09-02T05:59:27.370 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lLPO0YOyy8+c1LPqboXs9sy90Jg= 2022-09-02T05:59:27.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lLPO0YOyy8+c1LPqboXs9sy90Jg=', 'Content-Length': '0'} 2022-09-02T05:59:27.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138e673485fa0862-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.375 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:27.375 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:27.376 DEBUG:boto:Method: GET 2022-09-02T05:59:27.377 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:27.378 DEBUG:boto:Data: 2022-09-02T05:59:27.379 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.380 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.381 DEBUG:boto:Params: {} 2022-09-02T05:59:27.381 DEBUG:boto:Token: None 2022-09-02T05:59:27.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj185 2022-09-02T05:59:27.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lLPO0YOyy8+c1LPqboXs9sy90Jg= 2022-09-02T05:59:27.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lLPO0YOyy8+c1LPqboXs9sy90Jg=', 'Content-Length': '0'} 2022-09-02T05:59:27.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254bc75f8eee29a8-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.426 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.427 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.428 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.429 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.429 DEBUG:boto:Method: GET 2022-09-02T05:59:27.430 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.430 DEBUG:boto:Data: 2022-09-02T05:59:27.431 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.431 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.432 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.433 DEBUG:boto:Params: {} 2022-09-02T05:59:27.433 DEBUG:boto:Token: None 2022-09-02T05:59:27.434 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.435 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001275fb56b5b83f16-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.439 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.440 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.441 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.442 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.442 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.443 DEBUG:boto:Method: GET 2022-09-02T05:59:27.443 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:27.444 DEBUG:boto:Data: 2022-09-02T05:59:27.444 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.445 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.446 DEBUG:boto:Params: {} 2022-09-02T05:59:27.447 DEBUG:boto:Token: None 2022-09-02T05:59:27.447 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a4e56dfb13fcbb-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.453 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:27.455 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.455 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.456 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.457 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.458 DEBUG:boto:Method: GET 2022-09-02T05:59:27.459 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.460 DEBUG:boto:Data: 2022-09-02T05:59:27.461 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.462 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.463 DEBUG:boto:Params: {} 2022-09-02T05:59:27.464 DEBUG:boto:Token: None 2022-09-02T05:59:27.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4= 2022-09-02T05:59:27.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4=', 'Content-Length': '0'} 2022-09-02T05:59:27.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f86541df9e05eb7-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.470 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.471 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.471 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.472 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.472 DEBUG:boto:Method: GET 2022-09-02T05:59:27.473 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.473 DEBUG:boto:Data: 2022-09-02T05:59:27.474 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.475 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.476 DEBUG:boto:Params: {} 2022-09-02T05:59:27.476 DEBUG:boto:Token: None 2022-09-02T05:59:27.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4= 2022-09-02T05:59:27.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4=', 'Content-Length': '0'} 2022-09-02T05:59:27.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d7e0e6b6804c6c-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.482 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.482 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.483 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.483 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.484 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.484 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.485 DEBUG:boto:Data: 2022-09-02T05:59:27.485 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.486 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.486 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.487 DEBUG:boto:Params: {} 2022-09-02T05:59:27.487 DEBUG:boto:Token: None 2022-09-02T05:59:27.488 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.488 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6CZv5eU1+ylpqLXJ1KMgboMhOx4= 2022-09-02T05:59:27.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6CZv5eU1+ylpqLXJ1KMgboMhOx4=', 'Content-Length': '0'} 2022-09-02T05:59:27.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb4634fa8f6b53c9-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.492 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.492 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.493 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.493 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.494 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.494 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.495 DEBUG:boto:Data: 2022-09-02T05:59:27.495 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.496 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.497 DEBUG:boto:Params: {} 2022-09-02T05:59:27.498 DEBUG:boto:Token: None 2022-09-02T05:59:27.498 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6CZv5eU1+ylpqLXJ1KMgboMhOx4= 2022-09-02T05:59:27.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6CZv5eU1+ylpqLXJ1KMgboMhOx4=', 'Content-Length': '0'} 2022-09-02T05:59:27.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000056d7e92196b936b-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:27.503 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.503 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.504 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.504 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.505 DEBUG:boto:Method: GET 2022-09-02T05:59:27.505 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.506 DEBUG:boto:Data: 2022-09-02T05:59:27.507 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.507 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.508 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.508 DEBUG:boto:Params: {} 2022-09-02T05:59:27.509 DEBUG:boto:Token: None 2022-09-02T05:59:27.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.510 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4= 2022-09-02T05:59:27.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4=', 'Content-Length': '0'} 2022-09-02T05:59:27.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d089c1c2c82e4f-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.513 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.514 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.514 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.515 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.516 DEBUG:boto:Method: GET 2022-09-02T05:59:27.516 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.517 DEBUG:boto:Data: 2022-09-02T05:59:27.517 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.518 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.519 DEBUG:boto:Params: {} 2022-09-02T05:59:27.519 DEBUG:boto:Token: None 2022-09-02T05:59:27.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:27.520 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4= 2022-09-02T05:59:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ovuR3BMEg1Ww5p/92F0UWjn+IM4=', 'Content-Length': '0'} 2022-09-02T05:59:27.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e2fa85fbd14e88-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.524 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.524 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.525 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.525 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.526 DEBUG:boto:Method: GET 2022-09-02T05:59:27.526 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.527 DEBUG:boto:Data: 2022-09-02T05:59:27.528 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.529 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.529 DEBUG:boto:Params: {} 2022-09-02T05:59:27.530 DEBUG:boto:Token: None 2022-09-02T05:59:27.530 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e484152a250658ac-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.535 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.536 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.536 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.537 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.537 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.538 DEBUG:boto:Method: GET 2022-09-02T05:59:27.538 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.539 DEBUG:boto:Data: 2022-09-02T05:59:27.539 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.540 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.541 DEBUG:boto:Params: {} 2022-09-02T05:59:27.541 DEBUG:boto:Token: None 2022-09-02T05:59:27.542 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2bb55dd871af5a4-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.594 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.595 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.596 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.596 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.597 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:27.597 DEBUG:boto:Data: 2022-09-02T05:59:27.598 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.598 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.599 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.599 DEBUG:boto:Params: {} 2022-09-02T05:59:27.600 DEBUG:boto:Token: None 2022-09-02T05:59:27.600 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19 2022-09-02T05:59:27.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8MU3ThNeM02kqlS0k4Sm1aglO4= 2022-09-02T05:59:27.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8MU3ThNeM02kqlS0k4Sm1aglO4=', 'Content-Length': '0'} 2022-09-02T05:59:27.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea2f454ae2232b07-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.604 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.605 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.605 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.606 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:27.606 DEBUG:boto:Data: 2022-09-02T05:59:27.607 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.608 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.608 DEBUG:boto:Params: {} 2022-09-02T05:59:27.609 DEBUG:boto:Token: None 2022-09-02T05:59:27.609 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19 2022-09-02T05:59:27.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:f8MU3ThNeM02kqlS0k4Sm1aglO4= 2022-09-02T05:59:27.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:f8MU3ThNeM02kqlS0k4Sm1aglO4=', 'Content-Length': '0'} 2022-09-02T05:59:27.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3846620848adbcf-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:27.614 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.614 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.615 DEBUG:boto:Method: GET 2022-09-02T05:59:27.616 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:27.616 DEBUG:boto:Data: 2022-09-02T05:59:27.617 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.617 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.618 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.618 DEBUG:boto:Params: {} 2022-09-02T05:59:27.619 DEBUG:boto:Token: None 2022-09-02T05:59:27.619 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19 2022-09-02T05:59:27.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ugnGuVi6S7foILKnNnPgBPbVVLY= 2022-09-02T05:59:27.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ugnGuVi6S7foILKnNnPgBPbVVLY=', 'Content-Length': '0'} 2022-09-02T05:59:27.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003914718175fed1c7-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.623 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:27.624 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:27.624 DEBUG:boto:Method: GET 2022-09-02T05:59:27.625 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:27.625 DEBUG:boto:Data: 2022-09-02T05:59:27.626 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.627 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.627 DEBUG:boto:Params: {} 2022-09-02T05:59:27.628 DEBUG:boto:Token: None 2022-09-02T05:59:27.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj19 2022-09-02T05:59:27.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ugnGuVi6S7foILKnNnPgBPbVVLY= 2022-09-02T05:59:27.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ugnGuVi6S7foILKnNnPgBPbVVLY=', 'Content-Length': '0'} 2022-09-02T05:59:27.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005096725748012d0d-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.633 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.633 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.634 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.634 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.635 DEBUG:boto:Method: GET 2022-09-02T05:59:27.635 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.636 DEBUG:boto:Data: 2022-09-02T05:59:27.636 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.638 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.638 DEBUG:boto:Params: {} 2022-09-02T05:59:27.639 DEBUG:boto:Token: None 2022-09-02T05:59:27.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f2f5947be600873-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.686 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.687 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.688 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.689 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.689 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.690 DEBUG:boto:Method: GET 2022-09-02T05:59:27.690 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:27.691 DEBUG:boto:Data: 2022-09-02T05:59:27.692 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.693 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.693 DEBUG:boto:Params: {} 2022-09-02T05:59:27.694 DEBUG:boto:Token: None 2022-09-02T05:59:27.695 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.696 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000777e6173b6684a58-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.702 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:27.703 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.704 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.705 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.705 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.706 DEBUG:boto:Method: GET 2022-09-02T05:59:27.706 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.707 DEBUG:boto:Data: 2022-09-02T05:59:27.707 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.708 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.709 DEBUG:boto:Params: {} 2022-09-02T05:59:27.710 DEBUG:boto:Token: None 2022-09-02T05:59:27.710 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.711 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A= 2022-09-02T05:59:27.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A=', 'Content-Length': '0'} 2022-09-02T05:59:27.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009045fa64035dec6f-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.715 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.715 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.716 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.716 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.717 DEBUG:boto:Method: GET 2022-09-02T05:59:27.717 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.718 DEBUG:boto:Data: 2022-09-02T05:59:27.718 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.719 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.720 DEBUG:boto:Params: {} 2022-09-02T05:59:27.721 DEBUG:boto:Token: None 2022-09-02T05:59:27.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A= 2022-09-02T05:59:27.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A=', 'Content-Length': '0'} 2022-09-02T05:59:27.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2182c31799d7263-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.726 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.726 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.727 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.727 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.728 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.728 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.729 DEBUG:boto:Data: 2022-09-02T05:59:27.729 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.730 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.731 DEBUG:boto:Params: {} 2022-09-02T05:59:27.732 DEBUG:boto:Token: None 2022-09-02T05:59:27.732 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ti7n9LVJczITDzFiNjlsOzn9pE= 2022-09-02T05:59:27.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ti7n9LVJczITDzFiNjlsOzn9pE=', 'Content-Length': '0'} 2022-09-02T05:59:27.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093565fe54a0c8b83-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.736 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.737 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.737 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.738 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.739 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.739 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.740 DEBUG:boto:Data: 2022-09-02T05:59:27.740 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.741 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.742 DEBUG:boto:Params: {} 2022-09-02T05:59:27.743 DEBUG:boto:Token: None 2022-09-02T05:59:27.743 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9ti7n9LVJczITDzFiNjlsOzn9pE= 2022-09-02T05:59:27.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9ti7n9LVJczITDzFiNjlsOzn9pE=', 'Content-Length': '0'} 2022-09-02T05:59:27.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006125db8074459ee3-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:27.748 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.748 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.749 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.749 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.750 DEBUG:boto:Method: GET 2022-09-02T05:59:27.750 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.751 DEBUG:boto:Data: 2022-09-02T05:59:27.751 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.752 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.752 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.753 DEBUG:boto:Params: {} 2022-09-02T05:59:27.754 DEBUG:boto:Token: None 2022-09-02T05:59:27.754 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A= 2022-09-02T05:59:27.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A=', 'Content-Length': '0'} 2022-09-02T05:59:27.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e166a79aaf9a2f7-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.758 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.759 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.759 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.760 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.761 DEBUG:boto:Method: GET 2022-09-02T05:59:27.761 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.762 DEBUG:boto:Data: 2022-09-02T05:59:27.762 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.763 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.764 DEBUG:boto:Params: {} 2022-09-02T05:59:27.764 DEBUG:boto:Token: None 2022-09-02T05:59:27.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:27.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A= 2022-09-02T05:59:27.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LU4PQQq8XZdC13Op1OXBzn2ZN/A=', 'Content-Length': '0'} 2022-09-02T05:59:27.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e177e8fe88e74e79-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.810 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.811 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.811 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.812 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.812 DEBUG:boto:Method: GET 2022-09-02T05:59:27.813 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.814 DEBUG:boto:Data: 2022-09-02T05:59:27.814 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.815 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.816 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.816 DEBUG:boto:Params: {} 2022-09-02T05:59:27.817 DEBUG:boto:Token: None 2022-09-02T05:59:27.818 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000127402b7b2ed041e-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.823 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.824 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.824 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.825 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.826 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.826 DEBUG:boto:Method: GET 2022-09-02T05:59:27.827 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.827 DEBUG:boto:Data: 2022-09-02T05:59:27.828 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.830 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.830 DEBUG:boto:Params: {} 2022-09-02T05:59:27.831 DEBUG:boto:Token: None 2022-09-02T05:59:27.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be5d528b483a0f47-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.838 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.839 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.839 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.840 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.841 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:27.841 DEBUG:boto:Data: 2022-09-02T05:59:27.842 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.842 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.843 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.843 DEBUG:boto:Params: {} 2022-09-02T05:59:27.844 DEBUG:boto:Token: None 2022-09-02T05:59:27.844 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22 2022-09-02T05:59:27.845 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3//u0MtQPm7h3g7cqsF9OE6jJe4= 2022-09-02T05:59:27.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3//u0MtQPm7h3g7cqsF9OE6jJe4=', 'Content-Length': '0'} 2022-09-02T05:59:27.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000083c95a1e21570ed-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.849 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.850 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.850 DEBUG:boto:Method: HEAD 2022-09-02T05:59:27.851 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:27.851 DEBUG:boto:Data: 2022-09-02T05:59:27.852 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.853 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.853 DEBUG:boto:Params: {} 2022-09-02T05:59:27.854 DEBUG:boto:Token: None 2022-09-02T05:59:27.855 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22 2022-09-02T05:59:27.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3//u0MtQPm7h3g7cqsF9OE6jJe4= 2022-09-02T05:59:27.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3//u0MtQPm7h3g7cqsF9OE6jJe4=', 'Content-Length': '0'} 2022-09-02T05:59:27.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594bca7f9889298b-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:27.860 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.861 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.862 DEBUG:boto:Method: GET 2022-09-02T05:59:27.863 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:27.863 DEBUG:boto:Data: 2022-09-02T05:59:27.864 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.864 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.865 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.866 DEBUG:boto:Params: {} 2022-09-02T05:59:27.866 DEBUG:boto:Token: None 2022-09-02T05:59:27.867 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22 2022-09-02T05:59:27.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WFZLvSLs7Fl5gzZDqJFE1I06CTI= 2022-09-02T05:59:27.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WFZLvSLs7Fl5gzZDqJFE1I06CTI=', 'Content-Length': '0'} 2022-09-02T05:59:27.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be332a3a2968f4b2-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.914 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:27.915 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:27.916 DEBUG:boto:Method: GET 2022-09-02T05:59:27.916 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:27.917 DEBUG:boto:Data: 2022-09-02T05:59:27.918 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.919 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.919 DEBUG:boto:Params: {} 2022-09-02T05:59:27.920 DEBUG:boto:Token: None 2022-09-02T05:59:27.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj22 2022-09-02T05:59:27.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WFZLvSLs7Fl5gzZDqJFE1I06CTI= 2022-09-02T05:59:27.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WFZLvSLs7Fl5gzZDqJFE1I06CTI=', 'Content-Length': '0'} 2022-09-02T05:59:27.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028a8781af62935ce-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.927 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.928 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.929 DEBUG:boto:Method: GET 2022-09-02T05:59:27.929 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.930 DEBUG:boto:Data: 2022-09-02T05:59:27.930 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.932 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.932 DEBUG:boto:Params: {} 2022-09-02T05:59:27.933 DEBUG:boto:Token: None 2022-09-02T05:59:27.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000554e3baa3126dce2-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.939 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.941 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:27.942 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:27.943 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.944 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.946 DEBUG:boto:Method: GET 2022-09-02T05:59:27.947 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:27.948 DEBUG:boto:Data: 2022-09-02T05:59:27.949 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.952 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.953 DEBUG:boto:Params: {} 2022-09-02T05:59:27.954 DEBUG:boto:Token: None 2022-09-02T05:59:27.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/ 2022-09-02T05:59:27.955 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4= 2022-09-02T05:59:27.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A2ABNy2qaulFnZ6pv4ELKr+JFJ4=', 'Content-Length': '0'} 2022-09-02T05:59:27.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3293ebf450b9b5a-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.961 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:27.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:27.963 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:27.964 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:27.965 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.966 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.967 DEBUG:boto:Method: GET 2022-09-02T05:59:27.968 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.969 DEBUG:boto:Data: 2022-09-02T05:59:27.969 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:27.971 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.972 DEBUG:boto:Params: {} 2022-09-02T05:59:27.973 DEBUG:boto:Token: None 2022-09-02T05:59:27.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7As2ou5+fueAFuU46l9JSINtieo= 2022-09-02T05:59:27.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7As2ou5+fueAFuU46l9JSINtieo=', 'Content-Length': '0'} 2022-09-02T05:59:27.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd3a5a7ccf32a65-0063119bbf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:27.979 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:27.980 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:27.981 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.981 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.982 DEBUG:boto:Method: GET 2022-09-02T05:59:27.982 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.983 DEBUG:boto:Data: 2022-09-02T05:59:27.983 DEBUG:boto:Headers: {} 2022-09-02T05:59:27.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:27.984 DEBUG:boto:Port: 8000 2022-09-02T05:59:27.985 DEBUG:boto:Params: {} 2022-09-02T05:59:27.986 DEBUG:boto:Token: None 2022-09-02T05:59:27.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:27 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:27.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7As2ou5+fueAFuU46l9JSINtieo= 2022-09-02T05:59:27.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:27 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7As2ou5+fueAFuU46l9JSINtieo=', 'Content-Length': '0'} 2022-09-02T05:59:27.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c127a82f21c7fd-0063119bbf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:27 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.030 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.031 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.032 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.033 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.034 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.035 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.036 DEBUG:boto:Data: 2022-09-02T05:59:28.037 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.038 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.038 DEBUG:boto:Params: {} 2022-09-02T05:59:28.039 DEBUG:boto:Token: None 2022-09-02T05:59:28.040 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kFUv8fgdM8uI2tT995E4Z1J9VGU= 2022-09-02T05:59:28.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kFUv8fgdM8uI2tT995E4Z1J9VGU=', 'Content-Length': '0'} 2022-09-02T05:59:28.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0c7123bf7e3c274-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.044 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.045 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.045 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.046 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.046 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.047 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.047 DEBUG:boto:Data: 2022-09-02T05:59:28.048 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.049 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.049 DEBUG:boto:Params: {} 2022-09-02T05:59:28.050 DEBUG:boto:Token: None 2022-09-02T05:59:28.050 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kFUv8fgdM8uI2tT995E4Z1J9VGU= 2022-09-02T05:59:28.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kFUv8fgdM8uI2tT995E4Z1J9VGU=', 'Content-Length': '0'} 2022-09-02T05:59:28.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e0cd4aa408b2672-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:28.055 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.056 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.056 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.057 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.058 DEBUG:boto:Method: GET 2022-09-02T05:59:28.058 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.059 DEBUG:boto:Data: 2022-09-02T05:59:28.059 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.060 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.061 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.061 DEBUG:boto:Params: {} 2022-09-02T05:59:28.062 DEBUG:boto:Token: None 2022-09-02T05:59:28.062 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.063 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HI7CtZMZlagwayqFhm9CtlAT3Ys= 2022-09-02T05:59:28.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HI7CtZMZlagwayqFhm9CtlAT3Ys=', 'Content-Length': '0'} 2022-09-02T05:59:28.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073b0f05c7c268f26-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.066 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.067 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.068 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.068 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.069 DEBUG:boto:Method: GET 2022-09-02T05:59:28.069 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.070 DEBUG:boto:Data: 2022-09-02T05:59:28.070 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.071 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.072 DEBUG:boto:Params: {} 2022-09-02T05:59:28.073 DEBUG:boto:Token: None 2022-09-02T05:59:28.073 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:28.074 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HI7CtZMZlagwayqFhm9CtlAT3Ys= 2022-09-02T05:59:28.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HI7CtZMZlagwayqFhm9CtlAT3Ys=', 'Content-Length': '0'} 2022-09-02T05:59:28.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e5834a5e89d6b3-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.077 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.078 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.079 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.080 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.081 DEBUG:boto:Method: GET 2022-09-02T05:59:28.082 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.083 DEBUG:boto:Data: 2022-09-02T05:59:28.084 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.085 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.086 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.087 DEBUG:boto:Params: {} 2022-09-02T05:59:28.088 DEBUG:boto:Token: None 2022-09-02T05:59:28.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d72e1191c6523769-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.094 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.095 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.095 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.096 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.096 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.097 DEBUG:boto:Method: GET 2022-09-02T05:59:28.097 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.098 DEBUG:boto:Data: 2022-09-02T05:59:28.099 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.100 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.100 DEBUG:boto:Params: {} 2022-09-02T05:59:28.101 DEBUG:boto:Token: None 2022-09-02T05:59:28.101 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd14a1ce36c64054-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.108 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.108 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.109 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.110 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.110 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:28.111 DEBUG:boto:Data: 2022-09-02T05:59:28.111 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.112 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.113 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.113 DEBUG:boto:Params: {} 2022-09-02T05:59:28.114 DEBUG:boto:Token: None 2022-09-02T05:59:28.114 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23 2022-09-02T05:59:28.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pdUcIZSXZcmFnI7BrE2KHiX9TgY= 2022-09-02T05:59:28.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pdUcIZSXZcmFnI7BrE2KHiX9TgY=', 'Content-Length': '0'} 2022-09-02T05:59:28.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef065dd159e3c53-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.119 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.119 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.120 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.120 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:28.121 DEBUG:boto:Data: 2022-09-02T05:59:28.122 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.123 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.123 DEBUG:boto:Params: {} 2022-09-02T05:59:28.124 DEBUG:boto:Token: None 2022-09-02T05:59:28.125 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23 2022-09-02T05:59:28.125 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pdUcIZSXZcmFnI7BrE2KHiX9TgY= 2022-09-02T05:59:28.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pdUcIZSXZcmFnI7BrE2KHiX9TgY=', 'Content-Length': '0'} 2022-09-02T05:59:28.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df995f361e36078-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:28.129 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.130 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.131 DEBUG:boto:Method: GET 2022-09-02T05:59:28.131 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:28.132 DEBUG:boto:Data: 2022-09-02T05:59:28.133 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.134 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.134 DEBUG:boto:Params: {} 2022-09-02T05:59:28.135 DEBUG:boto:Token: None 2022-09-02T05:59:28.135 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23 2022-09-02T05:59:28.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jCQh5ZvkHRV4oAs12ecbaRxVO8Q= 2022-09-02T05:59:28.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jCQh5ZvkHRV4oAs12ecbaRxVO8Q=', 'Content-Length': '0'} 2022-09-02T05:59:28.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d79f4bf0d7ebb453-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.140 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:28.141 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:28.141 DEBUG:boto:Method: GET 2022-09-02T05:59:28.142 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:28.143 DEBUG:boto:Data: 2022-09-02T05:59:28.143 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.144 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.145 DEBUG:boto:Params: {} 2022-09-02T05:59:28.145 DEBUG:boto:Token: None 2022-09-02T05:59:28.146 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj23 2022-09-02T05:59:28.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jCQh5ZvkHRV4oAs12ecbaRxVO8Q= 2022-09-02T05:59:28.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jCQh5ZvkHRV4oAs12ecbaRxVO8Q=', 'Content-Length': '0'} 2022-09-02T05:59:28.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4aec30e286c181-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.152 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.152 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.153 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.153 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.154 DEBUG:boto:Method: GET 2022-09-02T05:59:28.154 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.155 DEBUG:boto:Data: 2022-09-02T05:59:28.155 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.156 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.157 DEBUG:boto:Params: {} 2022-09-02T05:59:28.158 DEBUG:boto:Token: None 2022-09-02T05:59:28.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe9697b627611d9-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.206 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.207 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.208 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.208 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.209 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.209 DEBUG:boto:Method: GET 2022-09-02T05:59:28.210 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:28.210 DEBUG:boto:Data: 2022-09-02T05:59:28.211 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.212 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.213 DEBUG:boto:Params: {} 2022-09-02T05:59:28.213 DEBUG:boto:Token: None 2022-09-02T05:59:28.214 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000282124428f232f2e-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.220 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:28.222 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.223 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.223 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.224 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.224 DEBUG:boto:Method: GET 2022-09-02T05:59:28.225 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.225 DEBUG:boto:Data: 2022-09-02T05:59:28.226 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.227 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.228 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.229 DEBUG:boto:Params: {} 2022-09-02T05:59:28.230 DEBUG:boto:Token: None 2022-09-02T05:59:28.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI= 2022-09-02T05:59:28.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI=', 'Content-Length': '0'} 2022-09-02T05:59:28.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82733a92ac72385-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.236 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.237 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.237 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.238 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.238 DEBUG:boto:Method: GET 2022-09-02T05:59:28.239 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.239 DEBUG:boto:Data: 2022-09-02T05:59:28.240 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.241 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.241 DEBUG:boto:Params: {} 2022-09-02T05:59:28.242 DEBUG:boto:Token: None 2022-09-02T05:59:28.243 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI= 2022-09-02T05:59:28.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI=', 'Content-Length': '0'} 2022-09-02T05:59:28.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f94de5ceed39894-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.295 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.296 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.296 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.297 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.297 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.298 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.298 DEBUG:boto:Data: 2022-09-02T05:59:28.299 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.299 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.300 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.300 DEBUG:boto:Params: {} 2022-09-02T05:59:28.301 DEBUG:boto:Token: None 2022-09-02T05:59:28.302 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.302 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kdWoO3ICkaVsUdVVQ0hkqOQd/YY= 2022-09-02T05:59:28.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kdWoO3ICkaVsUdVVQ0hkqOQd/YY=', 'Content-Length': '0'} 2022-09-02T05:59:28.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b087d3488bf499-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.306 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.306 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.307 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.307 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.308 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.308 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.309 DEBUG:boto:Data: 2022-09-02T05:59:28.309 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.310 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.311 DEBUG:boto:Params: {} 2022-09-02T05:59:28.311 DEBUG:boto:Token: None 2022-09-02T05:59:28.312 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kdWoO3ICkaVsUdVVQ0hkqOQd/YY= 2022-09-02T05:59:28.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kdWoO3ICkaVsUdVVQ0hkqOQd/YY=', 'Content-Length': '0'} 2022-09-02T05:59:28.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de2f5bac759d39c1-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:28.316 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.317 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.317 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.318 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.318 DEBUG:boto:Method: GET 2022-09-02T05:59:28.319 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.320 DEBUG:boto:Data: 2022-09-02T05:59:28.320 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.321 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.322 DEBUG:boto:Params: {} 2022-09-02T05:59:28.322 DEBUG:boto:Token: None 2022-09-02T05:59:28.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI= 2022-09-02T05:59:28.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI=', 'Content-Length': '0'} 2022-09-02T05:59:28.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c293bbdfd7e3fed-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.327 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.327 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.328 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.328 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.329 DEBUG:boto:Method: GET 2022-09-02T05:59:28.329 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.330 DEBUG:boto:Data: 2022-09-02T05:59:28.331 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.332 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.332 DEBUG:boto:Params: {} 2022-09-02T05:59:28.333 DEBUG:boto:Token: None 2022-09-02T05:59:28.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:28.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI= 2022-09-02T05:59:28.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:C4Suw5ylS7ckmNpm/CGyoCeADhI=', 'Content-Length': '0'} 2022-09-02T05:59:28.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044136bc61dc4c5a8-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.338 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.338 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.339 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.339 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.340 DEBUG:boto:Method: GET 2022-09-02T05:59:28.340 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.341 DEBUG:boto:Data: 2022-09-02T05:59:28.342 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.343 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.343 DEBUG:boto:Params: {} 2022-09-02T05:59:28.344 DEBUG:boto:Token: None 2022-09-02T05:59:28.344 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc818c7e685aba53-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.349 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.349 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.350 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.351 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.351 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.352 DEBUG:boto:Method: GET 2022-09-02T05:59:28.352 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.353 DEBUG:boto:Data: 2022-09-02T05:59:28.353 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.354 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.355 DEBUG:boto:Params: {} 2022-09-02T05:59:28.355 DEBUG:boto:Token: None 2022-09-02T05:59:28.356 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f6dbe7cbe3b1a7-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.362 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.363 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.364 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.364 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.365 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:28.365 DEBUG:boto:Data: 2022-09-02T05:59:28.366 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.367 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.367 DEBUG:boto:Params: {} 2022-09-02T05:59:28.368 DEBUG:boto:Token: None 2022-09-02T05:59:28.368 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29 2022-09-02T05:59:28.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PWb2rQI7tlrit9/OzUyfP66WBeM= 2022-09-02T05:59:28.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PWb2rQI7tlrit9/OzUyfP66WBeM=', 'Content-Length': '0'} 2022-09-02T05:59:28.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e9df128ac24542-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.372 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.373 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.373 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.374 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:28.374 DEBUG:boto:Data: 2022-09-02T05:59:28.375 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.376 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.376 DEBUG:boto:Params: {} 2022-09-02T05:59:28.377 DEBUG:boto:Token: None 2022-09-02T05:59:28.378 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29 2022-09-02T05:59:28.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PWb2rQI7tlrit9/OzUyfP66WBeM= 2022-09-02T05:59:28.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PWb2rQI7tlrit9/OzUyfP66WBeM=', 'Content-Length': '0'} 2022-09-02T05:59:28.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d99e3320de2aa1-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:28.382 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.383 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.383 DEBUG:boto:Method: GET 2022-09-02T05:59:28.384 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:28.384 DEBUG:boto:Data: 2022-09-02T05:59:28.385 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.385 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.386 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.386 DEBUG:boto:Params: {} 2022-09-02T05:59:28.387 DEBUG:boto:Token: None 2022-09-02T05:59:28.388 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29 2022-09-02T05:59:28.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pxD97ZcpWguDqZycUmTD5IQcxn8= 2022-09-02T05:59:28.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pxD97ZcpWguDqZycUmTD5IQcxn8=', 'Content-Length': '0'} 2022-09-02T05:59:28.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de554b25ea22300-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.392 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:28.392 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:28.393 DEBUG:boto:Method: GET 2022-09-02T05:59:28.393 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:28.394 DEBUG:boto:Data: 2022-09-02T05:59:28.394 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.396 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.396 DEBUG:boto:Params: {} 2022-09-02T05:59:28.397 DEBUG:boto:Token: None 2022-09-02T05:59:28.397 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj29 2022-09-02T05:59:28.398 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pxD97ZcpWguDqZycUmTD5IQcxn8= 2022-09-02T05:59:28.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pxD97ZcpWguDqZycUmTD5IQcxn8=', 'Content-Length': '0'} 2022-09-02T05:59:28.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022771e546af466ed-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.401 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.402 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.403 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.403 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.404 DEBUG:boto:Method: GET 2022-09-02T05:59:28.404 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.405 DEBUG:boto:Data: 2022-09-02T05:59:28.405 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.406 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.406 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.407 DEBUG:boto:Params: {} 2022-09-02T05:59:28.408 DEBUG:boto:Token: None 2022-09-02T05:59:28.408 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a9f6cd8c6c8deaf-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.454 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.455 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.456 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.456 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.457 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.457 DEBUG:boto:Method: GET 2022-09-02T05:59:28.458 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:28.458 DEBUG:boto:Data: 2022-09-02T05:59:28.459 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.460 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.460 DEBUG:boto:Params: {} 2022-09-02T05:59:28.461 DEBUG:boto:Token: None 2022-09-02T05:59:28.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c12bb84b361af14-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.468 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:28.469 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.470 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.470 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.471 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.471 DEBUG:boto:Method: GET 2022-09-02T05:59:28.472 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.472 DEBUG:boto:Data: 2022-09-02T05:59:28.473 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.473 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.474 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.474 DEBUG:boto:Params: {} 2022-09-02T05:59:28.475 DEBUG:boto:Token: None 2022-09-02T05:59:28.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk= 2022-09-02T05:59:28.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk=', 'Content-Length': '0'} 2022-09-02T05:59:28.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f11c33432243a2-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.480 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.481 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.481 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.482 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.482 DEBUG:boto:Method: GET 2022-09-02T05:59:28.483 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.483 DEBUG:boto:Data: 2022-09-02T05:59:28.484 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.485 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.486 DEBUG:boto:Params: {} 2022-09-02T05:59:28.486 DEBUG:boto:Token: None 2022-09-02T05:59:28.487 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk= 2022-09-02T05:59:28.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk=', 'Content-Length': '0'} 2022-09-02T05:59:28.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887a3af3ff823e10-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.491 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.492 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.492 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.493 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.494 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.494 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.495 DEBUG:boto:Data: 2022-09-02T05:59:28.495 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.496 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.496 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.497 DEBUG:boto:Params: {} 2022-09-02T05:59:28.497 DEBUG:boto:Token: None 2022-09-02T05:59:28.498 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.499 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VhbcHCQQTvIBWCrWVnDh8x267sc= 2022-09-02T05:59:28.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VhbcHCQQTvIBWCrWVnDh8x267sc=', 'Content-Length': '0'} 2022-09-02T05:59:28.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64c01d716ed5beb-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.502 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.503 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.503 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.504 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.504 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.505 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.505 DEBUG:boto:Data: 2022-09-02T05:59:28.506 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.507 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.507 DEBUG:boto:Params: {} 2022-09-02T05:59:28.508 DEBUG:boto:Token: None 2022-09-02T05:59:28.509 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VhbcHCQQTvIBWCrWVnDh8x267sc= 2022-09-02T05:59:28.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VhbcHCQQTvIBWCrWVnDh8x267sc=', 'Content-Length': '0'} 2022-09-02T05:59:28.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304353aee7497263-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:28.513 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.514 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.514 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.515 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.515 DEBUG:boto:Method: GET 2022-09-02T05:59:28.516 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.516 DEBUG:boto:Data: 2022-09-02T05:59:28.517 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.518 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.518 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.519 DEBUG:boto:Params: {} 2022-09-02T05:59:28.519 DEBUG:boto:Token: None 2022-09-02T05:59:28.520 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.521 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk= 2022-09-02T05:59:28.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk=', 'Content-Length': '0'} 2022-09-02T05:59:28.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b53224b389493d6f-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.524 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.525 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.526 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.526 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.527 DEBUG:boto:Method: GET 2022-09-02T05:59:28.527 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.528 DEBUG:boto:Data: 2022-09-02T05:59:28.528 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.529 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.530 DEBUG:boto:Params: {} 2022-09-02T05:59:28.531 DEBUG:boto:Token: None 2022-09-02T05:59:28.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:28.532 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk= 2022-09-02T05:59:28.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j7mdQF4rMY8vlscIrqdiWZG83Rk=', 'Content-Length': '0'} 2022-09-02T05:59:28.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004efd2f17e44a3c8e-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.578 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.580 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.581 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.582 DEBUG:boto:Method: GET 2022-09-02T05:59:28.583 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.584 DEBUG:boto:Data: 2022-09-02T05:59:28.585 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.586 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.587 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.588 DEBUG:boto:Params: {} 2022-09-02T05:59:28.589 DEBUG:boto:Token: None 2022-09-02T05:59:28.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b20a151c9fba738c-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.595 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.595 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.597 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.597 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.598 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.598 DEBUG:boto:Method: GET 2022-09-02T05:59:28.599 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.599 DEBUG:boto:Data: 2022-09-02T05:59:28.600 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.601 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.601 DEBUG:boto:Params: {} 2022-09-02T05:59:28.602 DEBUG:boto:Token: None 2022-09-02T05:59:28.603 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe2b7997380007e-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.609 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.610 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.610 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.611 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.611 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:28.612 DEBUG:boto:Data: 2022-09-02T05:59:28.612 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.613 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.613 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.614 DEBUG:boto:Params: {} 2022-09-02T05:59:28.615 DEBUG:boto:Token: None 2022-09-02T05:59:28.615 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31 2022-09-02T05:59:28.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cQ5nOjP8nEWkAFPJvCxb6tcxEvA= 2022-09-02T05:59:28.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cQ5nOjP8nEWkAFPJvCxb6tcxEvA=', 'Content-Length': '0'} 2022-09-02T05:59:28.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032f8d4fc10ecc65-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.619 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.620 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.620 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.621 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:28.621 DEBUG:boto:Data: 2022-09-02T05:59:28.622 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.623 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.623 DEBUG:boto:Params: {} 2022-09-02T05:59:28.624 DEBUG:boto:Token: None 2022-09-02T05:59:28.624 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31 2022-09-02T05:59:28.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cQ5nOjP8nEWkAFPJvCxb6tcxEvA= 2022-09-02T05:59:28.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cQ5nOjP8nEWkAFPJvCxb6tcxEvA=', 'Content-Length': '0'} 2022-09-02T05:59:28.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac11ebcb68da0903-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:28.631 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.632 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.633 DEBUG:boto:Method: GET 2022-09-02T05:59:28.635 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:28.644 DEBUG:boto:Data: 2022-09-02T05:59:28.645 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.645 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.646 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.646 DEBUG:boto:Params: {} 2022-09-02T05:59:28.647 DEBUG:boto:Token: None 2022-09-02T05:59:28.647 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31 2022-09-02T05:59:28.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWeaMrwM5TVAKp3LtwUUtDRx4wA= 2022-09-02T05:59:28.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWeaMrwM5TVAKp3LtwUUtDRx4wA=', 'Content-Length': '0'} 2022-09-02T05:59:28.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb539877ed5c565-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.694 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:28.695 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:28.695 DEBUG:boto:Method: GET 2022-09-02T05:59:28.696 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:28.696 DEBUG:boto:Data: 2022-09-02T05:59:28.697 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.698 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.699 DEBUG:boto:Params: {} 2022-09-02T05:59:28.699 DEBUG:boto:Token: None 2022-09-02T05:59:28.700 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj31 2022-09-02T05:59:28.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WWeaMrwM5TVAKp3LtwUUtDRx4wA= 2022-09-02T05:59:28.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WWeaMrwM5TVAKp3LtwUUtDRx4wA=', 'Content-Length': '0'} 2022-09-02T05:59:28.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d607500b9c2699b-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.704 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.705 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.705 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.706 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.707 DEBUG:boto:Method: GET 2022-09-02T05:59:28.707 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.708 DEBUG:boto:Data: 2022-09-02T05:59:28.708 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.709 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.709 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.710 DEBUG:boto:Params: {} 2022-09-02T05:59:28.711 DEBUG:boto:Token: None 2022-09-02T05:59:28.711 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f65df6f94eeea2-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.716 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.717 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.718 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.718 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.719 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.719 DEBUG:boto:Method: GET 2022-09-02T05:59:28.720 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:28.721 DEBUG:boto:Data: 2022-09-02T05:59:28.721 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.722 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.723 DEBUG:boto:Params: {} 2022-09-02T05:59:28.724 DEBUG:boto:Token: None 2022-09-02T05:59:28.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b90608399213925-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.730 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:28.732 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.732 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.733 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.733 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.734 DEBUG:boto:Method: GET 2022-09-02T05:59:28.734 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.735 DEBUG:boto:Data: 2022-09-02T05:59:28.736 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.737 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.737 DEBUG:boto:Params: {} 2022-09-02T05:59:28.738 DEBUG:boto:Token: None 2022-09-02T05:59:28.739 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA= 2022-09-02T05:59:28.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA=', 'Content-Length': '0'} 2022-09-02T05:59:28.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48ff3ea7b70db65-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.743 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.743 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.744 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.744 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.745 DEBUG:boto:Method: GET 2022-09-02T05:59:28.745 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.746 DEBUG:boto:Data: 2022-09-02T05:59:28.747 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.748 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.748 DEBUG:boto:Params: {} 2022-09-02T05:59:28.749 DEBUG:boto:Token: None 2022-09-02T05:59:28.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA= 2022-09-02T05:59:28.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA=', 'Content-Length': '0'} 2022-09-02T05:59:28.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084047af8743b084a-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.794 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.795 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.795 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.796 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.796 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.797 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.797 DEBUG:boto:Data: 2022-09-02T05:59:28.798 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.798 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.799 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.799 DEBUG:boto:Params: {} 2022-09-02T05:59:28.800 DEBUG:boto:Token: None 2022-09-02T05:59:28.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.801 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A6xaW/yzzRmVWeH/FzMk/rhpc4o= 2022-09-02T05:59:28.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A6xaW/yzzRmVWeH/FzMk/rhpc4o=', 'Content-Length': '0'} 2022-09-02T05:59:28.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac76f0b2b7906e2-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.804 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.805 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.805 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.806 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.806 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.807 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.807 DEBUG:boto:Data: 2022-09-02T05:59:28.808 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.809 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.809 DEBUG:boto:Params: {} 2022-09-02T05:59:28.810 DEBUG:boto:Token: None 2022-09-02T05:59:28.811 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.811 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:A6xaW/yzzRmVWeH/FzMk/rhpc4o= 2022-09-02T05:59:28.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:A6xaW/yzzRmVWeH/FzMk/rhpc4o=', 'Content-Length': '0'} 2022-09-02T05:59:28.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5878e36d712306-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:28.815 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.816 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.816 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.817 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.817 DEBUG:boto:Method: GET 2022-09-02T05:59:28.818 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.818 DEBUG:boto:Data: 2022-09-02T05:59:28.819 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.819 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.820 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.821 DEBUG:boto:Params: {} 2022-09-02T05:59:28.821 DEBUG:boto:Token: None 2022-09-02T05:59:28.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.822 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA= 2022-09-02T05:59:28.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA=', 'Content-Length': '0'} 2022-09-02T05:59:28.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c2e0c4635c9ffe-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.826 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.827 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.827 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.828 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.828 DEBUG:boto:Method: GET 2022-09-02T05:59:28.829 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.829 DEBUG:boto:Data: 2022-09-02T05:59:28.830 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.831 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.831 DEBUG:boto:Params: {} 2022-09-02T05:59:28.832 DEBUG:boto:Token: None 2022-09-02T05:59:28.833 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:28.833 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA= 2022-09-02T05:59:28.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MVkKuo2l1O8+TOjGNn7C1bQ2aMA=', 'Content-Length': '0'} 2022-09-02T05:59:28.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7e5502e2088e86-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.837 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.838 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.838 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.839 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.839 DEBUG:boto:Method: GET 2022-09-02T05:59:28.840 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.840 DEBUG:boto:Data: 2022-09-02T05:59:28.841 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.841 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.842 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.842 DEBUG:boto:Params: {} 2022-09-02T05:59:28.843 DEBUG:boto:Token: None 2022-09-02T05:59:28.844 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.844 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa7ae0bc28b401e-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.848 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.849 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.850 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.850 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.851 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.851 DEBUG:boto:Method: GET 2022-09-02T05:59:28.852 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.852 DEBUG:boto:Data: 2022-09-02T05:59:28.853 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.854 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.854 DEBUG:boto:Params: {} 2022-09-02T05:59:28.855 DEBUG:boto:Token: None 2022-09-02T05:59:28.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf9899dc18db7305-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.862 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.862 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.863 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.863 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.864 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:28.864 DEBUG:boto:Data: 2022-09-02T05:59:28.865 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.865 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.866 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.866 DEBUG:boto:Params: {} 2022-09-02T05:59:28.867 DEBUG:boto:Token: None 2022-09-02T05:59:28.868 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33 2022-09-02T05:59:28.868 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lncrxozfjNdYTblV518boPeGAbA= 2022-09-02T05:59:28.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lncrxozfjNdYTblV518boPeGAbA=', 'Content-Length': '0'} 2022-09-02T05:59:28.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc9e71c2da071e27-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.872 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.872 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.873 DEBUG:boto:Method: HEAD 2022-09-02T05:59:28.873 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:28.874 DEBUG:boto:Data: 2022-09-02T05:59:28.874 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.875 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.876 DEBUG:boto:Params: {} 2022-09-02T05:59:28.876 DEBUG:boto:Token: None 2022-09-02T05:59:28.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33 2022-09-02T05:59:28.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lncrxozfjNdYTblV518boPeGAbA= 2022-09-02T05:59:28.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lncrxozfjNdYTblV518boPeGAbA=', 'Content-Length': '0'} 2022-09-02T05:59:28.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c461ea445f31dd94-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:28.882 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.882 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.883 DEBUG:boto:Method: GET 2022-09-02T05:59:28.883 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:28.884 DEBUG:boto:Data: 2022-09-02T05:59:28.884 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.885 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.885 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.886 DEBUG:boto:Params: {} 2022-09-02T05:59:28.887 DEBUG:boto:Token: None 2022-09-02T05:59:28.887 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33 2022-09-02T05:59:28.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yWvNDqHWoLC+b/I9usK0vyRegOA= 2022-09-02T05:59:28.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yWvNDqHWoLC+b/I9usK0vyRegOA=', 'Content-Length': '0'} 2022-09-02T05:59:28.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000078711dfcb30d279-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.938 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:28.939 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:28.939 DEBUG:boto:Method: GET 2022-09-02T05:59:28.940 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:28.940 DEBUG:boto:Data: 2022-09-02T05:59:28.941 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.942 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.943 DEBUG:boto:Params: {} 2022-09-02T05:59:28.945 DEBUG:boto:Token: None 2022-09-02T05:59:28.946 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj33 2022-09-02T05:59:28.947 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yWvNDqHWoLC+b/I9usK0vyRegOA= 2022-09-02T05:59:28.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yWvNDqHWoLC+b/I9usK0vyRegOA=', 'Content-Length': '0'} 2022-09-02T05:59:28.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb0bb33bee1d4693-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.952 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.953 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.953 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.954 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.954 DEBUG:boto:Method: GET 2022-09-02T05:59:28.955 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.956 DEBUG:boto:Data: 2022-09-02T05:59:28.956 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.957 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.957 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.958 DEBUG:boto:Params: {} 2022-09-02T05:59:28.958 DEBUG:boto:Token: None 2022-09-02T05:59:28.959 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.960 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5beab027ac9265-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.964 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.965 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:28.965 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:28.966 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.966 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.967 DEBUG:boto:Method: GET 2022-09-02T05:59:28.968 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:28.969 DEBUG:boto:Data: 2022-09-02T05:59:28.970 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:28.971 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.972 DEBUG:boto:Params: {} 2022-09-02T05:59:28.972 DEBUG:boto:Token: None 2022-09-02T05:59:28.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/ 2022-09-02T05:59:28.974 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw= 2022-09-02T05:59:28.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jByvH34golRQpg9bWu+LmZKgriw=', 'Content-Length': '0'} 2022-09-02T05:59:28.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355862bca613ba00-0063119bc0-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.979 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:28.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:28.981 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:28.981 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:28.982 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.982 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.983 DEBUG:boto:Method: GET 2022-09-02T05:59:28.984 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.984 DEBUG:boto:Data: 2022-09-02T05:59:28.985 DEBUG:boto:Headers: {} 2022-09-02T05:59:28.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:28.987 DEBUG:boto:Port: 8000 2022-09-02T05:59:28.988 DEBUG:boto:Params: {} 2022-09-02T05:59:28.989 DEBUG:boto:Token: None 2022-09-02T05:59:28.990 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:28 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.991 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8+XKTI/0ugAorw2/xWBrd5r4Ucc= 2022-09-02T05:59:28.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:28 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8+XKTI/0ugAorw2/xWBrd5r4Ucc=', 'Content-Length': '0'} 2022-09-02T05:59:28.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c588418ad2f02ca3-0063119bc0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:28 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:28.995 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:28.996 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:28.997 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.997 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.998 DEBUG:boto:Method: GET 2022-09-02T05:59:28.998 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:28.999 DEBUG:boto:Data: 2022-09-02T05:59:28.999 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.001 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.002 DEBUG:boto:Params: {} 2022-09-02T05:59:29.003 DEBUG:boto:Token: None 2022-09-02T05:59:29.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cCFFnoLA2xJFmsTFaTI0HUYQjT4= 2022-09-02T05:59:29.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cCFFnoLA2xJFmsTFaTI0HUYQjT4=', 'Content-Length': '0'} 2022-09-02T05:59:29.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f93e5d4cf5dd734-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.010 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.011 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.011 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.012 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.012 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.013 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.013 DEBUG:boto:Data: 2022-09-02T05:59:29.014 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.015 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.015 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.015 DEBUG:boto:Params: {} 2022-09-02T05:59:29.016 DEBUG:boto:Token: None 2022-09-02T05:59:29.017 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0ZfN9C7CVuVXYBZfLBsTKGBTs7Y= 2022-09-02T05:59:29.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0ZfN9C7CVuVXYBZfLBsTKGBTs7Y=', 'Content-Length': '0'} 2022-09-02T05:59:29.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061fcd83ab5be1d29-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.021 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.021 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.022 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.022 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.023 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.023 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.024 DEBUG:boto:Data: 2022-09-02T05:59:29.024 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.025 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.026 DEBUG:boto:Params: {} 2022-09-02T05:59:29.026 DEBUG:boto:Token: None 2022-09-02T05:59:29.027 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0ZfN9C7CVuVXYBZfLBsTKGBTs7Y= 2022-09-02T05:59:29.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0ZfN9C7CVuVXYBZfLBsTKGBTs7Y=', 'Content-Length': '0'} 2022-09-02T05:59:29.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be9d94f5bcd2fd2-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:29.032 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.032 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.033 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.033 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.034 DEBUG:boto:Method: GET 2022-09-02T05:59:29.034 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.035 DEBUG:boto:Data: 2022-09-02T05:59:29.035 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.036 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.036 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.037 DEBUG:boto:Params: {} 2022-09-02T05:59:29.037 DEBUG:boto:Token: None 2022-09-02T05:59:29.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cCFFnoLA2xJFmsTFaTI0HUYQjT4= 2022-09-02T05:59:29.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cCFFnoLA2xJFmsTFaTI0HUYQjT4=', 'Content-Length': '0'} 2022-09-02T05:59:29.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af3802cc4fa1f3e-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.082 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.083 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.083 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.084 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.085 DEBUG:boto:Method: GET 2022-09-02T05:59:29.086 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.087 DEBUG:boto:Data: 2022-09-02T05:59:29.088 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.090 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.091 DEBUG:boto:Params: {} 2022-09-02T05:59:29.092 DEBUG:boto:Token: None 2022-09-02T05:59:29.093 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:29.094 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cCFFnoLA2xJFmsTFaTI0HUYQjT4= 2022-09-02T05:59:29.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cCFFnoLA2xJFmsTFaTI0HUYQjT4=', 'Content-Length': '0'} 2022-09-02T05:59:29.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002356a9e08ffba3f0-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.100 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.101 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.102 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.103 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.104 DEBUG:boto:Method: GET 2022-09-02T05:59:29.105 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.106 DEBUG:boto:Data: 2022-09-02T05:59:29.107 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.108 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.109 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.110 DEBUG:boto:Params: {} 2022-09-02T05:59:29.111 DEBUG:boto:Token: None 2022-09-02T05:59:29.112 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ffbe4cda4ed1e2-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.119 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.120 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.121 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.122 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.123 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.124 DEBUG:boto:Method: GET 2022-09-02T05:59:29.125 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.126 DEBUG:boto:Data: 2022-09-02T05:59:29.127 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.128 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.129 DEBUG:boto:Params: {} 2022-09-02T05:59:29.129 DEBUG:boto:Token: None 2022-09-02T05:59:29.130 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.131 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008703983c5df30b5f-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.178 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.179 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.180 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.180 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.181 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:29.182 DEBUG:boto:Data: 2022-09-02T05:59:29.183 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.183 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.184 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.184 DEBUG:boto:Params: {} 2022-09-02T05:59:29.185 DEBUG:boto:Token: None 2022-09-02T05:59:29.185 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34 2022-09-02T05:59:29.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aA6O08T14gKN1r0wMjnnGvVEsK0= 2022-09-02T05:59:29.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aA6O08T14gKN1r0wMjnnGvVEsK0=', 'Content-Length': '0'} 2022-09-02T05:59:29.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d8925e9380bd1f1-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.190 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.190 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.191 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.191 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:29.192 DEBUG:boto:Data: 2022-09-02T05:59:29.192 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.193 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.194 DEBUG:boto:Params: {} 2022-09-02T05:59:29.195 DEBUG:boto:Token: None 2022-09-02T05:59:29.195 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34 2022-09-02T05:59:29.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aA6O08T14gKN1r0wMjnnGvVEsK0= 2022-09-02T05:59:29.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aA6O08T14gKN1r0wMjnnGvVEsK0=', 'Content-Length': '0'} 2022-09-02T05:59:29.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f456391bc029a2db-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:29.201 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.202 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.203 DEBUG:boto:Method: GET 2022-09-02T05:59:29.204 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:29.205 DEBUG:boto:Data: 2022-09-02T05:59:29.206 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.208 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.209 DEBUG:boto:Params: {} 2022-09-02T05:59:29.210 DEBUG:boto:Token: None 2022-09-02T05:59:29.211 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34 2022-09-02T05:59:29.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TIgMXBZT2Dv1Y73uwO4lfHUiv9U= 2022-09-02T05:59:29.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TIgMXBZT2Dv1Y73uwO4lfHUiv9U=', 'Content-Length': '0'} 2022-09-02T05:59:29.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000736770e74aaed809-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.217 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:29.218 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:29.219 DEBUG:boto:Method: GET 2022-09-02T05:59:29.220 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:29.221 DEBUG:boto:Data: 2022-09-02T05:59:29.222 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.224 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.225 DEBUG:boto:Params: {} 2022-09-02T05:59:29.226 DEBUG:boto:Token: None 2022-09-02T05:59:29.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj34 2022-09-02T05:59:29.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TIgMXBZT2Dv1Y73uwO4lfHUiv9U= 2022-09-02T05:59:29.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TIgMXBZT2Dv1Y73uwO4lfHUiv9U=', 'Content-Length': '0'} 2022-09-02T05:59:29.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6ba5cbeaf61551-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.234 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.235 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.235 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.236 DEBUG:boto:Method: GET 2022-09-02T05:59:29.237 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.237 DEBUG:boto:Data: 2022-09-02T05:59:29.238 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.239 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.239 DEBUG:boto:Params: {} 2022-09-02T05:59:29.240 DEBUG:boto:Token: None 2022-09-02T05:59:29.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ab77ee79f05c46-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.245 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.246 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.247 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.248 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.248 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.249 DEBUG:boto:Method: GET 2022-09-02T05:59:29.249 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:29.250 DEBUG:boto:Data: 2022-09-02T05:59:29.250 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.252 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.252 DEBUG:boto:Params: {} 2022-09-02T05:59:29.253 DEBUG:boto:Token: None 2022-09-02T05:59:29.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcdfd0fd20a6f0b4-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.260 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:29.261 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.262 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.262 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.263 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.263 DEBUG:boto:Method: GET 2022-09-02T05:59:29.264 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.264 DEBUG:boto:Data: 2022-09-02T05:59:29.265 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.266 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.266 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.267 DEBUG:boto:Params: {} 2022-09-02T05:59:29.267 DEBUG:boto:Token: None 2022-09-02T05:59:29.268 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.269 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs= 2022-09-02T05:59:29.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs=', 'Content-Length': '0'} 2022-09-02T05:59:29.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d60904447bdd59e-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.272 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.273 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.273 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.274 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.275 DEBUG:boto:Method: GET 2022-09-02T05:59:29.275 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.276 DEBUG:boto:Data: 2022-09-02T05:59:29.276 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.277 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.278 DEBUG:boto:Params: {} 2022-09-02T05:59:29.278 DEBUG:boto:Token: None 2022-09-02T05:59:29.279 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs= 2022-09-02T05:59:29.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs=', 'Content-Length': '0'} 2022-09-02T05:59:29.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b02ad820d91011d-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.326 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.327 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.327 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.328 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.328 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.329 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.329 DEBUG:boto:Data: 2022-09-02T05:59:29.330 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.330 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.331 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.331 DEBUG:boto:Params: {} 2022-09-02T05:59:29.332 DEBUG:boto:Token: None 2022-09-02T05:59:29.333 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.333 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:63BeGpYhyxxFWH1H7PfVEZO84XI= 2022-09-02T05:59:29.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:63BeGpYhyxxFWH1H7PfVEZO84XI=', 'Content-Length': '0'} 2022-09-02T05:59:29.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf5b5afa0ff94d62-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.337 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.337 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.338 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.338 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.339 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.340 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.340 DEBUG:boto:Data: 2022-09-02T05:59:29.341 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.342 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.343 DEBUG:boto:Params: {} 2022-09-02T05:59:29.343 DEBUG:boto:Token: None 2022-09-02T05:59:29.344 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.344 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:63BeGpYhyxxFWH1H7PfVEZO84XI= 2022-09-02T05:59:29.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:63BeGpYhyxxFWH1H7PfVEZO84XI=', 'Content-Length': '0'} 2022-09-02T05:59:29.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7b55d26e91d3c6-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:29.348 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.349 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.349 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.350 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.350 DEBUG:boto:Method: GET 2022-09-02T05:59:29.351 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.352 DEBUG:boto:Data: 2022-09-02T05:59:29.352 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.353 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.353 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.354 DEBUG:boto:Params: {} 2022-09-02T05:59:29.354 DEBUG:boto:Token: None 2022-09-02T05:59:29.355 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs= 2022-09-02T05:59:29.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs=', 'Content-Length': '0'} 2022-09-02T05:59:29.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9864d4b3a6338b-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.360 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.361 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.362 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.363 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.363 DEBUG:boto:Method: GET 2022-09-02T05:59:29.364 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.364 DEBUG:boto:Data: 2022-09-02T05:59:29.365 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.366 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.366 DEBUG:boto:Params: {} 2022-09-02T05:59:29.367 DEBUG:boto:Token: None 2022-09-02T05:59:29.368 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:29.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs= 2022-09-02T05:59:29.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VPXnX2mBuSc6m3PWnCn7ReV8zJs=', 'Content-Length': '0'} 2022-09-02T05:59:29.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b89a83946e9d79-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.373 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.374 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.375 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.376 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.377 DEBUG:boto:Method: GET 2022-09-02T05:59:29.378 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.379 DEBUG:boto:Data: 2022-09-02T05:59:29.380 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.382 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.383 DEBUG:boto:Params: {} 2022-09-02T05:59:29.384 DEBUG:boto:Token: None 2022-09-02T05:59:29.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed329205715acb87-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.391 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.393 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.394 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.395 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.395 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.396 DEBUG:boto:Method: GET 2022-09-02T05:59:29.397 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.398 DEBUG:boto:Data: 2022-09-02T05:59:29.399 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.401 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.402 DEBUG:boto:Params: {} 2022-09-02T05:59:29.403 DEBUG:boto:Token: None 2022-09-02T05:59:29.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018e1584f815490a6-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.413 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.415 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.416 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.417 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.418 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:29.419 DEBUG:boto:Data: 2022-09-02T05:59:29.420 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.421 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.422 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.423 DEBUG:boto:Params: {} 2022-09-02T05:59:29.424 DEBUG:boto:Token: None 2022-09-02T05:59:29.425 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37 2022-09-02T05:59:29.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+YURxUSBAAVR9Uwj4jXg9pRYwds= 2022-09-02T05:59:29.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+YURxUSBAAVR9Uwj4jXg9pRYwds=', 'Content-Length': '0'} 2022-09-02T05:59:29.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfd8f280b8e92a0-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.431 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.432 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.433 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.434 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:29.435 DEBUG:boto:Data: 2022-09-02T05:59:29.436 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.438 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.439 DEBUG:boto:Params: {} 2022-09-02T05:59:29.440 DEBUG:boto:Token: None 2022-09-02T05:59:29.441 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37 2022-09-02T05:59:29.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+YURxUSBAAVR9Uwj4jXg9pRYwds= 2022-09-02T05:59:29.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+YURxUSBAAVR9Uwj4jXg9pRYwds=', 'Content-Length': '0'} 2022-09-02T05:59:29.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23d518f467873f0-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:29.446 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.446 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.447 DEBUG:boto:Method: GET 2022-09-02T05:59:29.448 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:29.448 DEBUG:boto:Data: 2022-09-02T05:59:29.449 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.450 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.450 DEBUG:boto:Params: {} 2022-09-02T05:59:29.451 DEBUG:boto:Token: None 2022-09-02T05:59:29.451 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37 2022-09-02T05:59:29.452 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a/dFmSvhMp2cWOkgIiMihETJyvI= 2022-09-02T05:59:29.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a/dFmSvhMp2cWOkgIiMihETJyvI=', 'Content-Length': '0'} 2022-09-02T05:59:29.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6cacb71dcc3360-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.498 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:29.499 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:29.500 DEBUG:boto:Method: GET 2022-09-02T05:59:29.500 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:29.501 DEBUG:boto:Data: 2022-09-02T05:59:29.501 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.503 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.504 DEBUG:boto:Params: {} 2022-09-02T05:59:29.504 DEBUG:boto:Token: None 2022-09-02T05:59:29.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj37 2022-09-02T05:59:29.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:a/dFmSvhMp2cWOkgIiMihETJyvI= 2022-09-02T05:59:29.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:a/dFmSvhMp2cWOkgIiMihETJyvI=', 'Content-Length': '0'} 2022-09-02T05:59:29.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a8352f09da2c57-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.510 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.511 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.511 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.512 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.513 DEBUG:boto:Method: GET 2022-09-02T05:59:29.513 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.514 DEBUG:boto:Data: 2022-09-02T05:59:29.514 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.515 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.516 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.516 DEBUG:boto:Params: {} 2022-09-02T05:59:29.517 DEBUG:boto:Token: None 2022-09-02T05:59:29.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.518 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810548caffc93b90-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.523 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.523 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.524 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.525 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.525 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.526 DEBUG:boto:Method: GET 2022-09-02T05:59:29.526 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:29.527 DEBUG:boto:Data: 2022-09-02T05:59:29.527 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.529 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.529 DEBUG:boto:Params: {} 2022-09-02T05:59:29.530 DEBUG:boto:Token: None 2022-09-02T05:59:29.531 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.531 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea34e9d8c700529-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.537 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:29.539 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.539 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.540 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.540 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.541 DEBUG:boto:Method: GET 2022-09-02T05:59:29.541 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.542 DEBUG:boto:Data: 2022-09-02T05:59:29.542 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.543 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.544 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.544 DEBUG:boto:Params: {} 2022-09-02T05:59:29.545 DEBUG:boto:Token: None 2022-09-02T05:59:29.545 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo= 2022-09-02T05:59:29.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo=', 'Content-Length': '0'} 2022-09-02T05:59:29.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5aaefbcdc54f78-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.550 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.550 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.551 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.552 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.552 DEBUG:boto:Method: GET 2022-09-02T05:59:29.553 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.553 DEBUG:boto:Data: 2022-09-02T05:59:29.554 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.555 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.556 DEBUG:boto:Params: {} 2022-09-02T05:59:29.556 DEBUG:boto:Token: None 2022-09-02T05:59:29.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo= 2022-09-02T05:59:29.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo=', 'Content-Length': '0'} 2022-09-02T05:59:29.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9902d8aeb3b60a-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.562 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.562 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.563 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.563 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.564 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.565 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.565 DEBUG:boto:Data: 2022-09-02T05:59:29.566 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.566 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.567 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.567 DEBUG:boto:Params: {} 2022-09-02T05:59:29.568 DEBUG:boto:Token: None 2022-09-02T05:59:29.568 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oIJb/5DE86AXM26huUZSS/i5mQU= 2022-09-02T05:59:29.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oIJb/5DE86AXM26huUZSS/i5mQU=', 'Content-Length': '0'} 2022-09-02T05:59:29.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d34ed95c707a8b-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.573 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.573 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.574 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.574 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.575 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.575 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.576 DEBUG:boto:Data: 2022-09-02T05:59:29.576 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.577 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.578 DEBUG:boto:Params: {} 2022-09-02T05:59:29.579 DEBUG:boto:Token: None 2022-09-02T05:59:29.579 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oIJb/5DE86AXM26huUZSS/i5mQU= 2022-09-02T05:59:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oIJb/5DE86AXM26huUZSS/i5mQU=', 'Content-Length': '0'} 2022-09-02T05:59:29.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000269b390c9711d64-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:29.584 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.585 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.585 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.586 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.586 DEBUG:boto:Method: GET 2022-09-02T05:59:29.587 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.588 DEBUG:boto:Data: 2022-09-02T05:59:29.588 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.589 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.589 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.590 DEBUG:boto:Params: {} 2022-09-02T05:59:29.591 DEBUG:boto:Token: None 2022-09-02T05:59:29.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.592 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo= 2022-09-02T05:59:29.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo=', 'Content-Length': '0'} 2022-09-02T05:59:29.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000577fcd0730dd7b04-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.596 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.596 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.597 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.597 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.598 DEBUG:boto:Method: GET 2022-09-02T05:59:29.599 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.599 DEBUG:boto:Data: 2022-09-02T05:59:29.600 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.601 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.601 DEBUG:boto:Params: {} 2022-09-02T05:59:29.602 DEBUG:boto:Token: None 2022-09-02T05:59:29.603 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:29.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo= 2022-09-02T05:59:29.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hDSnnIYVGr+ZKcMh5L76bw1fYdo=', 'Content-Length': '0'} 2022-09-02T05:59:29.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ebbf651717c678-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.608 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.608 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.609 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.609 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.610 DEBUG:boto:Method: GET 2022-09-02T05:59:29.610 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.611 DEBUG:boto:Data: 2022-09-02T05:59:29.611 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.612 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.613 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.613 DEBUG:boto:Params: {} 2022-09-02T05:59:29.614 DEBUG:boto:Token: None 2022-09-02T05:59:29.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000593d55f02eb64b27-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.666 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.667 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.668 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.669 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.670 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.670 DEBUG:boto:Method: GET 2022-09-02T05:59:29.671 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.671 DEBUG:boto:Data: 2022-09-02T05:59:29.672 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.673 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.673 DEBUG:boto:Params: {} 2022-09-02T05:59:29.674 DEBUG:boto:Token: None 2022-09-02T05:59:29.675 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.676 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080271ab58dfae7f5-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.681 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.682 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.682 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.683 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.684 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:29.684 DEBUG:boto:Data: 2022-09-02T05:59:29.685 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.685 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.686 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.686 DEBUG:boto:Params: {} 2022-09-02T05:59:29.687 DEBUG:boto:Token: None 2022-09-02T05:59:29.688 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38 2022-09-02T05:59:29.688 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sLC7bzTn5cihF6ZLey48DEu1Zs8= 2022-09-02T05:59:29.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sLC7bzTn5cihF6ZLey48DEu1Zs8=', 'Content-Length': '0'} 2022-09-02T05:59:29.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000807771f641da226c-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.692 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.692 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.693 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.693 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:29.694 DEBUG:boto:Data: 2022-09-02T05:59:29.694 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.695 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.696 DEBUG:boto:Params: {} 2022-09-02T05:59:29.697 DEBUG:boto:Token: None 2022-09-02T05:59:29.697 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38 2022-09-02T05:59:29.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sLC7bzTn5cihF6ZLey48DEu1Zs8= 2022-09-02T05:59:29.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sLC7bzTn5cihF6ZLey48DEu1Zs8=', 'Content-Length': '0'} 2022-09-02T05:59:29.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086e5c46591ed0bc0-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:29.703 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.703 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.704 DEBUG:boto:Method: GET 2022-09-02T05:59:29.704 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:29.705 DEBUG:boto:Data: 2022-09-02T05:59:29.705 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.706 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.706 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.707 DEBUG:boto:Params: {} 2022-09-02T05:59:29.708 DEBUG:boto:Token: None 2022-09-02T05:59:29.708 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38 2022-09-02T05:59:29.709 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SbCciKf44lLZHyxzA9DlxwntDM4= 2022-09-02T05:59:29.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SbCciKf44lLZHyxzA9DlxwntDM4=', 'Content-Length': '0'} 2022-09-02T05:59:29.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9f4a6429200f48-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.713 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:29.713 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:29.714 DEBUG:boto:Method: GET 2022-09-02T05:59:29.715 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:29.715 DEBUG:boto:Data: 2022-09-02T05:59:29.716 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.717 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.717 DEBUG:boto:Params: {} 2022-09-02T05:59:29.718 DEBUG:boto:Token: None 2022-09-02T05:59:29.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj38 2022-09-02T05:59:29.719 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SbCciKf44lLZHyxzA9DlxwntDM4= 2022-09-02T05:59:29.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SbCciKf44lLZHyxzA9DlxwntDM4=', 'Content-Length': '0'} 2022-09-02T05:59:29.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b9eaba4ade6311-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.766 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.767 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.768 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.768 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.769 DEBUG:boto:Method: GET 2022-09-02T05:59:29.770 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.771 DEBUG:boto:Data: 2022-09-02T05:59:29.772 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.774 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.775 DEBUG:boto:Params: {} 2022-09-02T05:59:29.776 DEBUG:boto:Token: None 2022-09-02T05:59:29.777 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006866ab16b73bab55-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.783 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.785 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.786 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.787 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.788 DEBUG:boto:Method: GET 2022-09-02T05:59:29.789 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:29.790 DEBUG:boto:Data: 2022-09-02T05:59:29.791 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.793 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.794 DEBUG:boto:Params: {} 2022-09-02T05:59:29.795 DEBUG:boto:Token: None 2022-09-02T05:59:29.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423810b8d4c48959-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.804 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:29.807 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.808 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.809 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.809 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.810 DEBUG:boto:Method: GET 2022-09-02T05:59:29.811 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.812 DEBUG:boto:Data: 2022-09-02T05:59:29.813 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.814 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.815 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.816 DEBUG:boto:Params: {} 2022-09-02T05:59:29.816 DEBUG:boto:Token: None 2022-09-02T05:59:29.817 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg= 2022-09-02T05:59:29.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg=', 'Content-Length': '0'} 2022-09-02T05:59:29.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd78d6663ac1b6d-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.862 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.863 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.863 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.864 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.864 DEBUG:boto:Method: GET 2022-09-02T05:59:29.865 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.865 DEBUG:boto:Data: 2022-09-02T05:59:29.866 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.867 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.868 DEBUG:boto:Params: {} 2022-09-02T05:59:29.868 DEBUG:boto:Token: None 2022-09-02T05:59:29.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.869 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg= 2022-09-02T05:59:29.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg=', 'Content-Length': '0'} 2022-09-02T05:59:29.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d79291f8537557-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.873 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.874 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.874 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.875 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.875 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.876 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.876 DEBUG:boto:Data: 2022-09-02T05:59:29.877 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.877 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.878 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.878 DEBUG:boto:Params: {} 2022-09-02T05:59:29.879 DEBUG:boto:Token: None 2022-09-02T05:59:29.880 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GOWWQPJK1yLdcbT40CF7wpYZe5Q= 2022-09-02T05:59:29.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GOWWQPJK1yLdcbT40CF7wpYZe5Q=', 'Content-Length': '0'} 2022-09-02T05:59:29.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fdda29c40489014-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.884 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.884 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.885 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.885 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.886 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.886 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.887 DEBUG:boto:Data: 2022-09-02T05:59:29.887 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.888 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.889 DEBUG:boto:Params: {} 2022-09-02T05:59:29.889 DEBUG:boto:Token: None 2022-09-02T05:59:29.890 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GOWWQPJK1yLdcbT40CF7wpYZe5Q= 2022-09-02T05:59:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GOWWQPJK1yLdcbT40CF7wpYZe5Q=', 'Content-Length': '0'} 2022-09-02T05:59:29.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef6490d8a9346fc5-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:29.894 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.895 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.896 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.896 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.897 DEBUG:boto:Method: GET 2022-09-02T05:59:29.897 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.898 DEBUG:boto:Data: 2022-09-02T05:59:29.898 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.899 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.899 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.900 DEBUG:boto:Params: {} 2022-09-02T05:59:29.900 DEBUG:boto:Token: None 2022-09-02T05:59:29.901 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg= 2022-09-02T05:59:29.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg=', 'Content-Length': '0'} 2022-09-02T05:59:29.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba323df58871e79-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.906 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.906 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.907 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.907 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.908 DEBUG:boto:Method: GET 2022-09-02T05:59:29.908 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.909 DEBUG:boto:Data: 2022-09-02T05:59:29.909 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.911 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.911 DEBUG:boto:Params: {} 2022-09-02T05:59:29.912 DEBUG:boto:Token: None 2022-09-02T05:59:29.913 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:29.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg= 2022-09-02T05:59:29.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DAK3fPvMukF3JwlHW50x26dgasg=', 'Content-Length': '0'} 2022-09-02T05:59:29.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b48b358b6f6275-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.917 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.917 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.918 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:29.918 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:29.919 DEBUG:boto:Method: GET 2022-09-02T05:59:29.920 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:29.920 DEBUG:boto:Data: 2022-09-02T05:59:29.921 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.922 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.922 DEBUG:boto:Params: {} 2022-09-02T05:59:29.923 DEBUG:boto:Token: None 2022-09-02T05:59:29.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74ffe7eac5dfb0e-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.929 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.930 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:29.931 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:29.932 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:29.933 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:29.934 DEBUG:boto:Method: GET 2022-09-02T05:59:29.935 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:29.936 DEBUG:boto:Data: 2022-09-02T05:59:29.937 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.939 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.940 DEBUG:boto:Params: {} 2022-09-02T05:59:29.941 DEBUG:boto:Token: None 2022-09-02T05:59:29.942 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/ 2022-09-02T05:59:29.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc= 2022-09-02T05:59:29.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dm8UTZL5TYtcpqhw40vxzH6HZCc=', 'Content-Length': '0'} 2022-09-02T05:59:29.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09b76fd7ad794a7-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.949 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:29.950 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.951 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.951 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.952 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:29.952 DEBUG:boto:Data: 2022-09-02T05:59:29.953 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.953 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.954 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.954 DEBUG:boto:Params: {} 2022-09-02T05:59:29.955 DEBUG:boto:Token: None 2022-09-02T05:59:29.955 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44 2022-09-02T05:59:29.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ru3IgTNByquoJ4J+KZ+EkfQ8wBE= 2022-09-02T05:59:29.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ru3IgTNByquoJ4J+KZ+EkfQ8wBE=', 'Content-Length': '0'} 2022-09-02T05:59:29.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008df42754f8b5f91-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.959 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.960 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.960 DEBUG:boto:Method: HEAD 2022-09-02T05:59:29.961 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:29.961 DEBUG:boto:Data: 2022-09-02T05:59:29.962 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:29.963 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.964 DEBUG:boto:Params: {} 2022-09-02T05:59:29.964 DEBUG:boto:Token: None 2022-09-02T05:59:29.965 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44 2022-09-02T05:59:29.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ru3IgTNByquoJ4J+KZ+EkfQ8wBE= 2022-09-02T05:59:29.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ru3IgTNByquoJ4J+KZ+EkfQ8wBE=', 'Content-Length': '0'} 2022-09-02T05:59:29.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eafadaa64a725f8-0063119bc1-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:29.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:29.970 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:29.971 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:29.971 DEBUG:boto:Method: GET 2022-09-02T05:59:29.972 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:29.973 DEBUG:boto:Data: 2022-09-02T05:59:29.973 DEBUG:boto:Headers: {} 2022-09-02T05:59:29.974 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:29.974 DEBUG:boto:Port: 8000 2022-09-02T05:59:29.975 DEBUG:boto:Params: {} 2022-09-02T05:59:29.975 DEBUG:boto:Token: None 2022-09-02T05:59:29.976 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:29 GMT /xljbkf-74/obj44 2022-09-02T05:59:29.977 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/sYf02sFQIvAqA9/kWTNl5hxKyk= 2022-09-02T05:59:29.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:29 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/sYf02sFQIvAqA9/kWTNl5hxKyk=', 'Content-Length': '0'} 2022-09-02T05:59:29.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d52f7ea6ef372c-0063119bc1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:29 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.022 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:30.023 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:30.023 DEBUG:boto:Method: GET 2022-09-02T05:59:30.024 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:30.024 DEBUG:boto:Data: 2022-09-02T05:59:30.025 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.026 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.027 DEBUG:boto:Params: {} 2022-09-02T05:59:30.027 DEBUG:boto:Token: None 2022-09-02T05:59:30.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj44 2022-09-02T05:59:30.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QTId10A8ok+rSsXrzHlsBpxFxGE= 2022-09-02T05:59:30.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QTId10A8ok+rSsXrzHlsBpxFxGE=', 'Content-Length': '0'} 2022-09-02T05:59:30.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdc680616f16d720-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.033 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.033 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.034 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:30.034 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:30.035 DEBUG:boto:Method: GET 2022-09-02T05:59:30.035 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:30.036 DEBUG:boto:Data: 2022-09-02T05:59:30.036 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.037 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.037 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.038 DEBUG:boto:Params: {} 2022-09-02T05:59:30.038 DEBUG:boto:Token: None 2022-09-02T05:59:30.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000282af34a96ea1f88-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.044 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.044 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.045 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.046 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:30.046 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:30.047 DEBUG:boto:Method: GET 2022-09-02T05:59:30.047 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:30.048 DEBUG:boto:Data: 2022-09-02T05:59:30.048 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.049 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.050 DEBUG:boto:Params: {} 2022-09-02T05:59:30.051 DEBUG:boto:Token: None 2022-09-02T05:59:30.051 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce17265f6b6053a-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.058 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:30.059 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.060 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.060 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.061 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.061 DEBUG:boto:Method: GET 2022-09-02T05:59:30.062 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.063 DEBUG:boto:Data: 2022-09-02T05:59:30.063 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.064 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.064 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.065 DEBUG:boto:Params: {} 2022-09-02T05:59:30.065 DEBUG:boto:Token: None 2022-09-02T05:59:30.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0= 2022-09-02T05:59:30.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0=', 'Content-Length': '0'} 2022-09-02T05:59:30.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000373d4994d45f4007-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.070 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.071 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.071 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.072 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.072 DEBUG:boto:Method: GET 2022-09-02T05:59:30.073 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.074 DEBUG:boto:Data: 2022-09-02T05:59:30.075 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.076 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.076 DEBUG:boto:Params: {} 2022-09-02T05:59:30.077 DEBUG:boto:Token: None 2022-09-02T05:59:30.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0= 2022-09-02T05:59:30.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0=', 'Content-Length': '0'} 2022-09-02T05:59:30.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008048fd0bee9856f3-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.122 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.123 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.123 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.124 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.124 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.125 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.125 DEBUG:boto:Data: 2022-09-02T05:59:30.126 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.126 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.127 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.127 DEBUG:boto:Params: {} 2022-09-02T05:59:30.128 DEBUG:boto:Token: None 2022-09-02T05:59:30.129 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vP206VnTWHC2aqGCbCE0213/3cc= 2022-09-02T05:59:30.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vP206VnTWHC2aqGCbCE0213/3cc=', 'Content-Length': '0'} 2022-09-02T05:59:30.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3db897d54d3bc74-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.133 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.133 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.134 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.134 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.135 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.135 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.136 DEBUG:boto:Data: 2022-09-02T05:59:30.136 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.137 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.138 DEBUG:boto:Params: {} 2022-09-02T05:59:30.139 DEBUG:boto:Token: None 2022-09-02T05:59:30.139 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vP206VnTWHC2aqGCbCE0213/3cc= 2022-09-02T05:59:30.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vP206VnTWHC2aqGCbCE0213/3cc=', 'Content-Length': '0'} 2022-09-02T05:59:30.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cd6ac90105ffad-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:30.144 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.145 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.145 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.146 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.146 DEBUG:boto:Method: GET 2022-09-02T05:59:30.147 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.147 DEBUG:boto:Data: 2022-09-02T05:59:30.148 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.149 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.149 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.150 DEBUG:boto:Params: {} 2022-09-02T05:59:30.150 DEBUG:boto:Token: None 2022-09-02T05:59:30.151 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.152 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0= 2022-09-02T05:59:30.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0=', 'Content-Length': '0'} 2022-09-02T05:59:30.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000804f6f1bbcd37f9a-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.155 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.156 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.156 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.157 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.157 DEBUG:boto:Method: GET 2022-09-02T05:59:30.158 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.159 DEBUG:boto:Data: 2022-09-02T05:59:30.159 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.160 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.161 DEBUG:boto:Params: {} 2022-09-02T05:59:30.161 DEBUG:boto:Token: None 2022-09-02T05:59:30.162 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:30.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0= 2022-09-02T05:59:30.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TfO+H365ABfpXePQibnkUUNKGX0=', 'Content-Length': '0'} 2022-09-02T05:59:30.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5105ee5b7626b4-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.166 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.167 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.167 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.168 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.168 DEBUG:boto:Method: GET 2022-09-02T05:59:30.169 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.170 DEBUG:boto:Data: 2022-09-02T05:59:30.170 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.171 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.171 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.172 DEBUG:boto:Params: {} 2022-09-02T05:59:30.172 DEBUG:boto:Token: None 2022-09-02T05:59:30.173 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.173 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e59c51d45d12e0ef-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.222 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.224 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.224 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.225 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.225 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.226 DEBUG:boto:Method: GET 2022-09-02T05:59:30.226 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.227 DEBUG:boto:Data: 2022-09-02T05:59:30.227 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.228 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.229 DEBUG:boto:Params: {} 2022-09-02T05:59:30.230 DEBUG:boto:Token: None 2022-09-02T05:59:30.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb7618877eddbc5-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.238 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.239 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.239 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.240 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.240 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:30.241 DEBUG:boto:Data: 2022-09-02T05:59:30.241 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.242 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.243 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.243 DEBUG:boto:Params: {} 2022-09-02T05:59:30.244 DEBUG:boto:Token: None 2022-09-02T05:59:30.244 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46 2022-09-02T05:59:30.245 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FWY4UP7vvPz0+db53cVNf8ssxoQ= 2022-09-02T05:59:30.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FWY4UP7vvPz0+db53cVNf8ssxoQ=', 'Content-Length': '0'} 2022-09-02T05:59:30.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000853b4c965b57398f-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.248 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.249 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.249 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.250 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:30.250 DEBUG:boto:Data: 2022-09-02T05:59:30.251 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.252 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.252 DEBUG:boto:Params: {} 2022-09-02T05:59:30.253 DEBUG:boto:Token: None 2022-09-02T05:59:30.254 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46 2022-09-02T05:59:30.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FWY4UP7vvPz0+db53cVNf8ssxoQ= 2022-09-02T05:59:30.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FWY4UP7vvPz0+db53cVNf8ssxoQ=', 'Content-Length': '0'} 2022-09-02T05:59:30.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010bf9e99e58cf958-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:30.258 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.259 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.260 DEBUG:boto:Method: GET 2022-09-02T05:59:30.260 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:30.261 DEBUG:boto:Data: 2022-09-02T05:59:30.261 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.262 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.263 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.263 DEBUG:boto:Params: {} 2022-09-02T05:59:30.264 DEBUG:boto:Token: None 2022-09-02T05:59:30.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46 2022-09-02T05:59:30.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1hGluyA33D57idg4SLQOMagVCRI= 2022-09-02T05:59:30.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1hGluyA33D57idg4SLQOMagVCRI=', 'Content-Length': '0'} 2022-09-02T05:59:30.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619f6e28cd763d95-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.269 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:30.270 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:30.270 DEBUG:boto:Method: GET 2022-09-02T05:59:30.271 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:30.272 DEBUG:boto:Data: 2022-09-02T05:59:30.272 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.274 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.274 DEBUG:boto:Params: {} 2022-09-02T05:59:30.275 DEBUG:boto:Token: None 2022-09-02T05:59:30.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj46 2022-09-02T05:59:30.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1hGluyA33D57idg4SLQOMagVCRI= 2022-09-02T05:59:30.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1hGluyA33D57idg4SLQOMagVCRI=', 'Content-Length': '0'} 2022-09-02T05:59:30.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a2bee28eb942c1f-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.322 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.323 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.323 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.324 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.324 DEBUG:boto:Method: GET 2022-09-02T05:59:30.325 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.326 DEBUG:boto:Data: 2022-09-02T05:59:30.326 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.327 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.327 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.328 DEBUG:boto:Params: {} 2022-09-02T05:59:30.328 DEBUG:boto:Token: None 2022-09-02T05:59:30.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187612ebfdd096f5-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.334 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.335 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.336 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.336 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.337 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.338 DEBUG:boto:Method: GET 2022-09-02T05:59:30.338 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:30.339 DEBUG:boto:Data: 2022-09-02T05:59:30.339 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.340 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.341 DEBUG:boto:Params: {} 2022-09-02T05:59:30.342 DEBUG:boto:Token: None 2022-09-02T05:59:30.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002840787fa04cbff0-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.350 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:30.351 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.352 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.353 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.353 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.354 DEBUG:boto:Method: GET 2022-09-02T05:59:30.355 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.356 DEBUG:boto:Data: 2022-09-02T05:59:30.356 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.357 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.358 DEBUG:boto:Params: {} 2022-09-02T05:59:30.359 DEBUG:boto:Token: None 2022-09-02T05:59:30.359 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI= 2022-09-02T05:59:30.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI=', 'Content-Length': '0'} 2022-09-02T05:59:30.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e042876e5e8c27f6-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.364 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.365 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.366 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.366 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.367 DEBUG:boto:Method: GET 2022-09-02T05:59:30.368 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.370 DEBUG:boto:Data: 2022-09-02T05:59:30.371 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.373 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.374 DEBUG:boto:Params: {} 2022-09-02T05:59:30.375 DEBUG:boto:Token: None 2022-09-02T05:59:30.377 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.378 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI= 2022-09-02T05:59:30.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI=', 'Content-Length': '0'} 2022-09-02T05:59:30.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e71936bab46ab29-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.382 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.383 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.384 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.385 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.385 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.386 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.387 DEBUG:boto:Data: 2022-09-02T05:59:30.388 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.388 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.389 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.389 DEBUG:boto:Params: {} 2022-09-02T05:59:30.390 DEBUG:boto:Token: None 2022-09-02T05:59:30.391 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqfJaXclYUXYB3wM9Buh6wVMJ30= 2022-09-02T05:59:30.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqfJaXclYUXYB3wM9Buh6wVMJ30=', 'Content-Length': '0'} 2022-09-02T05:59:30.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a557145e5a9ab3f4-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.396 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.396 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.397 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.397 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.398 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.399 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.400 DEBUG:boto:Data: 2022-09-02T05:59:30.401 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.403 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.404 DEBUG:boto:Params: {} 2022-09-02T05:59:30.404 DEBUG:boto:Token: None 2022-09-02T05:59:30.405 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.406 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cqfJaXclYUXYB3wM9Buh6wVMJ30= 2022-09-02T05:59:30.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cqfJaXclYUXYB3wM9Buh6wVMJ30=', 'Content-Length': '0'} 2022-09-02T05:59:30.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d737d8cfca1a895f-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:30.410 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.411 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.412 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.412 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.413 DEBUG:boto:Method: GET 2022-09-02T05:59:30.414 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.414 DEBUG:boto:Data: 2022-09-02T05:59:30.415 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.416 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.416 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.417 DEBUG:boto:Params: {} 2022-09-02T05:59:30.418 DEBUG:boto:Token: None 2022-09-02T05:59:30.419 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI= 2022-09-02T05:59:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI=', 'Content-Length': '0'} 2022-09-02T05:59:30.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad56ef2d979be99-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.466 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.468 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.469 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.470 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.471 DEBUG:boto:Method: GET 2022-09-02T05:59:30.472 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.473 DEBUG:boto:Data: 2022-09-02T05:59:30.474 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.476 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.477 DEBUG:boto:Params: {} 2022-09-02T05:59:30.478 DEBUG:boto:Token: None 2022-09-02T05:59:30.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:30.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI= 2022-09-02T05:59:30.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9U3O1ZUowlc4hiZJG8Is3ldHmPI=', 'Content-Length': '0'} 2022-09-02T05:59:30.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82eab21ff139e1f-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.484 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.484 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.485 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.486 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.486 DEBUG:boto:Method: GET 2022-09-02T05:59:30.487 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.487 DEBUG:boto:Data: 2022-09-02T05:59:30.488 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.488 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.489 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.489 DEBUG:boto:Params: {} 2022-09-02T05:59:30.490 DEBUG:boto:Token: None 2022-09-02T05:59:30.491 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002e28743c0b2e89e-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.495 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.496 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.497 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.497 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.498 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.498 DEBUG:boto:Method: GET 2022-09-02T05:59:30.499 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.500 DEBUG:boto:Data: 2022-09-02T05:59:30.500 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.501 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.502 DEBUG:boto:Params: {} 2022-09-02T05:59:30.503 DEBUG:boto:Token: None 2022-09-02T05:59:30.503 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d13f1d8bc8609597-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.550 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.551 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.552 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.552 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.553 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:30.553 DEBUG:boto:Data: 2022-09-02T05:59:30.554 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.554 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.555 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.555 DEBUG:boto:Params: {} 2022-09-02T05:59:30.556 DEBUG:boto:Token: None 2022-09-02T05:59:30.556 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51 2022-09-02T05:59:30.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yyhrb5P2tqtkL0uMwbOEdBhdFfI= 2022-09-02T05:59:30.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yyhrb5P2tqtkL0uMwbOEdBhdFfI=', 'Content-Length': '0'} 2022-09-02T05:59:30.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f70190b334b5ae-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.560 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.561 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.561 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.562 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:30.562 DEBUG:boto:Data: 2022-09-02T05:59:30.563 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.564 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.564 DEBUG:boto:Params: {} 2022-09-02T05:59:30.565 DEBUG:boto:Token: None 2022-09-02T05:59:30.566 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51 2022-09-02T05:59:30.566 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Yyhrb5P2tqtkL0uMwbOEdBhdFfI= 2022-09-02T05:59:30.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Yyhrb5P2tqtkL0uMwbOEdBhdFfI=', 'Content-Length': '0'} 2022-09-02T05:59:30.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030512380fc9ffb25-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:30.570 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.571 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.571 DEBUG:boto:Method: GET 2022-09-02T05:59:30.572 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:30.572 DEBUG:boto:Data: 2022-09-02T05:59:30.573 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.573 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.574 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.574 DEBUG:boto:Params: {} 2022-09-02T05:59:30.575 DEBUG:boto:Token: None 2022-09-02T05:59:30.576 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51 2022-09-02T05:59:30.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E+iUq0Mwi2bTd0zriJEazrbsmAE= 2022-09-02T05:59:30.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E+iUq0Mwi2bTd0zriJEazrbsmAE=', 'Content-Length': '0'} 2022-09-02T05:59:30.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2f77786ddef9cf-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.580 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:30.580 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:30.581 DEBUG:boto:Method: GET 2022-09-02T05:59:30.581 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:30.582 DEBUG:boto:Data: 2022-09-02T05:59:30.583 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.584 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.584 DEBUG:boto:Params: {} 2022-09-02T05:59:30.585 DEBUG:boto:Token: None 2022-09-02T05:59:30.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj51 2022-09-02T05:59:30.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E+iUq0Mwi2bTd0zriJEazrbsmAE= 2022-09-02T05:59:30.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E+iUq0Mwi2bTd0zriJEazrbsmAE=', 'Content-Length': '0'} 2022-09-02T05:59:30.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe040e31009b884-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.589 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.590 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.591 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.591 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.592 DEBUG:boto:Method: GET 2022-09-02T05:59:30.592 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.593 DEBUG:boto:Data: 2022-09-02T05:59:30.593 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.594 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.594 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.595 DEBUG:boto:Params: {} 2022-09-02T05:59:30.595 DEBUG:boto:Token: None 2022-09-02T05:59:30.596 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.597 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011041c8e85cd7fcb-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.601 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.601 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.602 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.603 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.603 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.604 DEBUG:boto:Method: GET 2022-09-02T05:59:30.604 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:30.605 DEBUG:boto:Data: 2022-09-02T05:59:30.605 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.607 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.608 DEBUG:boto:Params: {} 2022-09-02T05:59:30.609 DEBUG:boto:Token: None 2022-09-02T05:59:30.610 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.610 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aed230d617dd0bc-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.617 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:30.619 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.619 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.620 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.621 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.622 DEBUG:boto:Method: GET 2022-09-02T05:59:30.623 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.624 DEBUG:boto:Data: 2022-09-02T05:59:30.625 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.626 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.627 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.628 DEBUG:boto:Params: {} 2022-09-02T05:59:30.628 DEBUG:boto:Token: None 2022-09-02T05:59:30.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak= 2022-09-02T05:59:30.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak=', 'Content-Length': '0'} 2022-09-02T05:59:30.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003b1759a607a11f3-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.634 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.635 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.635 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.636 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.637 DEBUG:boto:Method: GET 2022-09-02T05:59:30.637 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.638 DEBUG:boto:Data: 2022-09-02T05:59:30.638 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.639 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.640 DEBUG:boto:Params: {} 2022-09-02T05:59:30.641 DEBUG:boto:Token: None 2022-09-02T05:59:30.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak= 2022-09-02T05:59:30.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak=', 'Content-Length': '0'} 2022-09-02T05:59:30.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1ed6613ea15e58-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.645 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.646 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.647 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.647 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.648 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.648 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.649 DEBUG:boto:Data: 2022-09-02T05:59:30.649 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.650 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.651 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.660 DEBUG:boto:Params: {} 2022-09-02T05:59:30.661 DEBUG:boto:Token: None 2022-09-02T05:59:30.661 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.662 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XHBKPm+WEqANn+MnWK9Dz0puiiw= 2022-09-02T05:59:30.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XHBKPm+WEqANn+MnWK9Dz0puiiw=', 'Content-Length': '0'} 2022-09-02T05:59:30.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c851aaa3b59797-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.665 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.666 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.667 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.667 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.668 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.669 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.669 DEBUG:boto:Data: 2022-09-02T05:59:30.670 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.671 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.671 DEBUG:boto:Params: {} 2022-09-02T05:59:30.672 DEBUG:boto:Token: None 2022-09-02T05:59:30.672 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XHBKPm+WEqANn+MnWK9Dz0puiiw= 2022-09-02T05:59:30.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XHBKPm+WEqANn+MnWK9Dz0puiiw=', 'Content-Length': '0'} 2022-09-02T05:59:30.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656987ed511141c5-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:30.677 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.677 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.678 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.678 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.679 DEBUG:boto:Method: GET 2022-09-02T05:59:30.679 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.680 DEBUG:boto:Data: 2022-09-02T05:59:30.680 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.681 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.682 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.682 DEBUG:boto:Params: {} 2022-09-02T05:59:30.683 DEBUG:boto:Token: None 2022-09-02T05:59:30.683 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.684 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak= 2022-09-02T05:59:30.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak=', 'Content-Length': '0'} 2022-09-02T05:59:30.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000272d8c51a1f8ef8a-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.688 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.688 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.689 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.689 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.690 DEBUG:boto:Method: GET 2022-09-02T05:59:30.690 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.691 DEBUG:boto:Data: 2022-09-02T05:59:30.691 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.692 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.693 DEBUG:boto:Params: {} 2022-09-02T05:59:30.693 DEBUG:boto:Token: None 2022-09-02T05:59:30.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:30.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak= 2022-09-02T05:59:30.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NgsToBx9Q/hqr4r1ltCZUc895ak=', 'Content-Length': '0'} 2022-09-02T05:59:30.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f4eeae4cf107a9-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.698 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.699 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.700 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.700 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.701 DEBUG:boto:Method: GET 2022-09-02T05:59:30.701 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.702 DEBUG:boto:Data: 2022-09-02T05:59:30.702 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.703 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.703 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.704 DEBUG:boto:Params: {} 2022-09-02T05:59:30.704 DEBUG:boto:Token: None 2022-09-02T05:59:30.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493dafe13cd64bc2-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.754 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.755 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.756 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.756 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.757 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.757 DEBUG:boto:Method: GET 2022-09-02T05:59:30.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.758 DEBUG:boto:Data: 2022-09-02T05:59:30.759 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.760 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.760 DEBUG:boto:Params: {} 2022-09-02T05:59:30.761 DEBUG:boto:Token: None 2022-09-02T05:59:30.762 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6533b5f0b7432ca-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.768 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.769 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.770 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.770 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.771 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:30.771 DEBUG:boto:Data: 2022-09-02T05:59:30.772 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.773 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.773 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.773 DEBUG:boto:Params: {} 2022-09-02T05:59:30.774 DEBUG:boto:Token: None 2022-09-02T05:59:30.775 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55 2022-09-02T05:59:30.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n5yPXrBQeBMI+coJfIBIj2NWHiI= 2022-09-02T05:59:30.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n5yPXrBQeBMI+coJfIBIj2NWHiI=', 'Content-Length': '0'} 2022-09-02T05:59:30.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e9526ff0a48f41-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.779 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.779 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.780 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.780 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:30.781 DEBUG:boto:Data: 2022-09-02T05:59:30.781 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.782 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.782 DEBUG:boto:Params: {} 2022-09-02T05:59:30.783 DEBUG:boto:Token: None 2022-09-02T05:59:30.784 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55 2022-09-02T05:59:30.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:n5yPXrBQeBMI+coJfIBIj2NWHiI= 2022-09-02T05:59:30.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:n5yPXrBQeBMI+coJfIBIj2NWHiI=', 'Content-Length': '0'} 2022-09-02T05:59:30.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab9677bb98dd1e9-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:30.788 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.789 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.789 DEBUG:boto:Method: GET 2022-09-02T05:59:30.790 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:30.790 DEBUG:boto:Data: 2022-09-02T05:59:30.791 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.792 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.792 DEBUG:boto:Params: {} 2022-09-02T05:59:30.793 DEBUG:boto:Token: None 2022-09-02T05:59:30.793 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55 2022-09-02T05:59:30.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EQlzM3trhwNGjm5vCtWYhDPiD6I= 2022-09-02T05:59:30.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EQlzM3trhwNGjm5vCtWYhDPiD6I=', 'Content-Length': '0'} 2022-09-02T05:59:30.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5996a27ffc793d-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.797 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:30.798 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:30.798 DEBUG:boto:Method: GET 2022-09-02T05:59:30.799 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:30.800 DEBUG:boto:Data: 2022-09-02T05:59:30.800 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.801 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.802 DEBUG:boto:Params: {} 2022-09-02T05:59:30.802 DEBUG:boto:Token: None 2022-09-02T05:59:30.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj55 2022-09-02T05:59:30.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EQlzM3trhwNGjm5vCtWYhDPiD6I= 2022-09-02T05:59:30.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EQlzM3trhwNGjm5vCtWYhDPiD6I=', 'Content-Length': '0'} 2022-09-02T05:59:30.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df88670eea2ded1c-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.854 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.855 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.855 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.856 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.856 DEBUG:boto:Method: GET 2022-09-02T05:59:30.857 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.858 DEBUG:boto:Data: 2022-09-02T05:59:30.858 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.859 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.860 DEBUG:boto:Params: {} 2022-09-02T05:59:30.860 DEBUG:boto:Token: None 2022-09-02T05:59:30.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bcee565c9339ac9-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.866 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.867 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.867 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.868 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.868 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.869 DEBUG:boto:Method: GET 2022-09-02T05:59:30.870 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:30.870 DEBUG:boto:Data: 2022-09-02T05:59:30.871 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.872 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.872 DEBUG:boto:Params: {} 2022-09-02T05:59:30.873 DEBUG:boto:Token: None 2022-09-02T05:59:30.874 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.874 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000112ceb48f362b15e-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.880 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:30.881 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.882 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.883 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.884 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.885 DEBUG:boto:Method: GET 2022-09-02T05:59:30.885 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.886 DEBUG:boto:Data: 2022-09-02T05:59:30.887 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.887 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.888 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.888 DEBUG:boto:Params: {} 2022-09-02T05:59:30.889 DEBUG:boto:Token: None 2022-09-02T05:59:30.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8= 2022-09-02T05:59:30.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8=', 'Content-Length': '0'} 2022-09-02T05:59:30.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce89625c810f2894-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.895 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.896 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.897 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.898 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.899 DEBUG:boto:Method: GET 2022-09-02T05:59:30.900 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.901 DEBUG:boto:Data: 2022-09-02T05:59:30.902 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.903 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.904 DEBUG:boto:Params: {} 2022-09-02T05:59:30.904 DEBUG:boto:Token: None 2022-09-02T05:59:30.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8= 2022-09-02T05:59:30.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8=', 'Content-Length': '0'} 2022-09-02T05:59:30.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da84fadd36dc46d-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.910 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.910 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.911 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.911 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.912 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.912 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.913 DEBUG:boto:Data: 2022-09-02T05:59:30.913 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.914 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.915 DEBUG:boto:Params: {} 2022-09-02T05:59:30.915 DEBUG:boto:Token: None 2022-09-02T05:59:30.916 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dlj9uq086duVsSbEYhCPoSeavuc= 2022-09-02T05:59:30.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dlj9uq086duVsSbEYhCPoSeavuc=', 'Content-Length': '0'} 2022-09-02T05:59:30.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d646ad141b22ffc-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.920 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.920 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.921 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.921 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.922 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.922 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.923 DEBUG:boto:Data: 2022-09-02T05:59:30.923 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.924 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.925 DEBUG:boto:Params: {} 2022-09-02T05:59:30.926 DEBUG:boto:Token: None 2022-09-02T05:59:30.926 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dlj9uq086duVsSbEYhCPoSeavuc= 2022-09-02T05:59:30.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dlj9uq086duVsSbEYhCPoSeavuc=', 'Content-Length': '0'} 2022-09-02T05:59:30.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084283325c5cd4c70-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:30.931 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.931 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.932 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.932 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.933 DEBUG:boto:Method: GET 2022-09-02T05:59:30.933 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.934 DEBUG:boto:Data: 2022-09-02T05:59:30.934 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.935 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.935 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.936 DEBUG:boto:Params: {} 2022-09-02T05:59:30.937 DEBUG:boto:Token: None 2022-09-02T05:59:30.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8= 2022-09-02T05:59:30.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8=', 'Content-Length': '0'} 2022-09-02T05:59:30.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833b7b105f2d14c6-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.941 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.942 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.942 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.943 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.943 DEBUG:boto:Method: GET 2022-09-02T05:59:30.944 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.944 DEBUG:boto:Data: 2022-09-02T05:59:30.945 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.946 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.947 DEBUG:boto:Params: {} 2022-09-02T05:59:30.947 DEBUG:boto:Token: None 2022-09-02T05:59:30.948 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:30.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8= 2022-09-02T05:59:30.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PiUrPVbcu4ndeTSrrUUArz8KBy8=', 'Content-Length': '0'} 2022-09-02T05:59:30.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001662aa9475cfa509-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.952 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.953 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.953 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:30.954 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:30.954 DEBUG:boto:Method: GET 2022-09-02T05:59:30.955 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:30.955 DEBUG:boto:Data: 2022-09-02T05:59:30.956 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.957 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.957 DEBUG:boto:Params: {} 2022-09-02T05:59:30.958 DEBUG:boto:Token: None 2022-09-02T05:59:30.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.959 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e6e5785a2c601e-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.963 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.964 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:30.965 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:30.965 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:30.966 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:30.966 DEBUG:boto:Method: GET 2022-09-02T05:59:30.967 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:30.967 DEBUG:boto:Data: 2022-09-02T05:59:30.968 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.969 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.970 DEBUG:boto:Params: {} 2022-09-02T05:59:30.970 DEBUG:boto:Token: None 2022-09-02T05:59:30.971 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/ 2022-09-02T05:59:30.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y= 2022-09-02T05:59:30.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2a2JcOzVDquGrv9Ku7F1yIR4/4Y=', 'Content-Length': '0'} 2022-09-02T05:59:30.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354acab82d5bbccb-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.977 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:30.978 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.978 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.979 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.979 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:30.980 DEBUG:boto:Data: 2022-09-02T05:59:30.980 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.981 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:30.981 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.982 DEBUG:boto:Params: {} 2022-09-02T05:59:30.982 DEBUG:boto:Token: None 2022-09-02T05:59:30.983 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57 2022-09-02T05:59:30.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K3FCLvCrWwvk+cJ2H8mTt8xQIis= 2022-09-02T05:59:30.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K3FCLvCrWwvk+cJ2H8mTt8xQIis=', 'Content-Length': '0'} 2022-09-02T05:59:30.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed05845c56619f6-0063119bc2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.987 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.988 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.988 DEBUG:boto:Method: HEAD 2022-09-02T05:59:30.989 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:30.989 DEBUG:boto:Data: 2022-09-02T05:59:30.990 DEBUG:boto:Headers: {} 2022-09-02T05:59:30.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:30.991 DEBUG:boto:Port: 8000 2022-09-02T05:59:30.992 DEBUG:boto:Params: {} 2022-09-02T05:59:30.992 DEBUG:boto:Token: None 2022-09-02T05:59:30.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:30 GMT /xljbkf-74/obj57 2022-09-02T05:59:30.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K3FCLvCrWwvk+cJ2H8mTt8xQIis= 2022-09-02T05:59:30.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:30 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K3FCLvCrWwvk+cJ2H8mTt8xQIis=', 'Content-Length': '0'} 2022-09-02T05:59:30.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b902efda7b98ef9f-0063119bc2-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:30 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:30.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:30.997 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:30.998 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:30.999 DEBUG:boto:Method: GET 2022-09-02T05:59:30.999 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:31.000 DEBUG:boto:Data: 2022-09-02T05:59:31.000 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.001 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.002 DEBUG:boto:Params: {} 2022-09-02T05:59:31.002 DEBUG:boto:Token: None 2022-09-02T05:59:31.003 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj57 2022-09-02T05:59:31.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LBcnXBCjf7JX5fDBPqMgOLDWS+o= 2022-09-02T05:59:31.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LBcnXBCjf7JX5fDBPqMgOLDWS+o=', 'Content-Length': '0'} 2022-09-02T05:59:31.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e2121bee9eecbb-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.050 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:31.051 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:31.051 DEBUG:boto:Method: GET 2022-09-02T05:59:31.052 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:31.052 DEBUG:boto:Data: 2022-09-02T05:59:31.053 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.054 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.055 DEBUG:boto:Params: {} 2022-09-02T05:59:31.055 DEBUG:boto:Token: None 2022-09-02T05:59:31.056 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj57 2022-09-02T05:59:31.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LBcnXBCjf7JX5fDBPqMgOLDWS+o= 2022-09-02T05:59:31.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LBcnXBCjf7JX5fDBPqMgOLDWS+o=', 'Content-Length': '0'} 2022-09-02T05:59:31.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24a5cbd76f67aef-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.061 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.061 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.062 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:31.062 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:31.063 DEBUG:boto:Method: GET 2022-09-02T05:59:31.064 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:31.065 DEBUG:boto:Data: 2022-09-02T05:59:31.066 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.067 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.068 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.069 DEBUG:boto:Params: {} 2022-09-02T05:59:31.071 DEBUG:boto:Token: None 2022-09-02T05:59:31.072 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.073 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198327387cfb9ba6-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.078 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.080 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.081 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.082 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:31.083 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:31.084 DEBUG:boto:Method: GET 2022-09-02T05:59:31.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:31.085 DEBUG:boto:Data: 2022-09-02T05:59:31.086 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.087 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.088 DEBUG:boto:Params: {} 2022-09-02T05:59:31.088 DEBUG:boto:Token: None 2022-09-02T05:59:31.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f1f31993567ad82-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.096 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:31.097 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.097 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.098 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.099 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.099 DEBUG:boto:Method: GET 2022-09-02T05:59:31.100 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.101 DEBUG:boto:Data: 2022-09-02T05:59:31.101 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.102 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.102 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.103 DEBUG:boto:Params: {} 2022-09-02T05:59:31.104 DEBUG:boto:Token: None 2022-09-02T05:59:31.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc= 2022-09-02T05:59:31.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc=', 'Content-Length': '0'} 2022-09-02T05:59:31.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052930cbe9e0885dc-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.110 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.111 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.112 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.113 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.114 DEBUG:boto:Method: GET 2022-09-02T05:59:31.115 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.116 DEBUG:boto:Data: 2022-09-02T05:59:31.117 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.118 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.119 DEBUG:boto:Params: {} 2022-09-02T05:59:31.120 DEBUG:boto:Token: None 2022-09-02T05:59:31.120 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.121 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc= 2022-09-02T05:59:31.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc=', 'Content-Length': '0'} 2022-09-02T05:59:31.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d21eeb128f3dd2dd-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.125 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.125 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.126 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.126 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.127 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.127 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.128 DEBUG:boto:Data: 2022-09-02T05:59:31.128 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.129 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.130 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.130 DEBUG:boto:Params: {} 2022-09-02T05:59:31.131 DEBUG:boto:Token: None 2022-09-02T05:59:31.132 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.133 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zo1otc+CZtqVP5e6ieja7wEIPRg= 2022-09-02T05:59:31.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zo1otc+CZtqVP5e6ieja7wEIPRg=', 'Content-Length': '0'} 2022-09-02T05:59:31.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4a0ad070f333ce-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.136 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.137 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.137 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.138 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.139 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.140 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.141 DEBUG:boto:Data: 2022-09-02T05:59:31.142 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.143 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.144 DEBUG:boto:Params: {} 2022-09-02T05:59:31.145 DEBUG:boto:Token: None 2022-09-02T05:59:31.146 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.147 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zo1otc+CZtqVP5e6ieja7wEIPRg= 2022-09-02T05:59:31.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zo1otc+CZtqVP5e6ieja7wEIPRg=', 'Content-Length': '0'} 2022-09-02T05:59:31.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007373e5032db27a88-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:31.152 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.152 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.153 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.153 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.154 DEBUG:boto:Method: GET 2022-09-02T05:59:31.154 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.155 DEBUG:boto:Data: 2022-09-02T05:59:31.155 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.156 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.157 DEBUG:boto:Params: {} 2022-09-02T05:59:31.158 DEBUG:boto:Token: None 2022-09-02T05:59:31.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc= 2022-09-02T05:59:31.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc=', 'Content-Length': '0'} 2022-09-02T05:59:31.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f397753aead3e6e-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.163 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.163 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.164 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.164 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.165 DEBUG:boto:Method: GET 2022-09-02T05:59:31.165 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.166 DEBUG:boto:Data: 2022-09-02T05:59:31.167 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.168 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.168 DEBUG:boto:Params: {} 2022-09-02T05:59:31.169 DEBUG:boto:Token: None 2022-09-02T05:59:31.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:31.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc= 2022-09-02T05:59:31.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:w+cBaNXgGoMxvNbtJR6wYSR4ADc=', 'Content-Length': '0'} 2022-09-02T05:59:31.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33e179bb60abe34-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.174 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.174 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.175 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.175 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.176 DEBUG:boto:Method: GET 2022-09-02T05:59:31.176 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.177 DEBUG:boto:Data: 2022-09-02T05:59:31.177 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.178 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.179 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.179 DEBUG:boto:Params: {} 2022-09-02T05:59:31.180 DEBUG:boto:Token: None 2022-09-02T05:59:31.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024bbc756b549cd2-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.186 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.187 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.187 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.188 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.188 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.189 DEBUG:boto:Method: GET 2022-09-02T05:59:31.189 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.190 DEBUG:boto:Data: 2022-09-02T05:59:31.191 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.192 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.192 DEBUG:boto:Params: {} 2022-09-02T05:59:31.193 DEBUG:boto:Token: None 2022-09-02T05:59:31.193 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ce3b4605f21f1dd-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.200 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.201 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.201 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.202 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.203 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:31.204 DEBUG:boto:Data: 2022-09-02T05:59:31.205 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.205 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.206 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.207 DEBUG:boto:Params: {} 2022-09-02T05:59:31.208 DEBUG:boto:Token: None 2022-09-02T05:59:31.208 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58 2022-09-02T05:59:31.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WEQYopakRZEi680wUeWjglBqT5U= 2022-09-02T05:59:31.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WEQYopakRZEi680wUeWjglBqT5U=', 'Content-Length': '0'} 2022-09-02T05:59:31.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a885ade79ea47577-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.213 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.213 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.214 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.214 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:31.215 DEBUG:boto:Data: 2022-09-02T05:59:31.215 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.217 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.217 DEBUG:boto:Params: {} 2022-09-02T05:59:31.218 DEBUG:boto:Token: None 2022-09-02T05:59:31.219 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58 2022-09-02T05:59:31.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WEQYopakRZEi680wUeWjglBqT5U= 2022-09-02T05:59:31.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WEQYopakRZEi680wUeWjglBqT5U=', 'Content-Length': '0'} 2022-09-02T05:59:31.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a36e4ca8af1f1708-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:31.225 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.225 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.226 DEBUG:boto:Method: GET 2022-09-02T05:59:31.227 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:31.227 DEBUG:boto:Data: 2022-09-02T05:59:31.228 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.229 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.229 DEBUG:boto:Params: {} 2022-09-02T05:59:31.230 DEBUG:boto:Token: None 2022-09-02T05:59:31.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58 2022-09-02T05:59:31.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NNrWatkJs0AO3sVF7i8vwYtTcR4= 2022-09-02T05:59:31.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NNrWatkJs0AO3sVF7i8vwYtTcR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5d51c6dfd2d537-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.235 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:31.235 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:31.236 DEBUG:boto:Method: GET 2022-09-02T05:59:31.236 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:31.237 DEBUG:boto:Data: 2022-09-02T05:59:31.237 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.238 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.239 DEBUG:boto:Params: {} 2022-09-02T05:59:31.240 DEBUG:boto:Token: None 2022-09-02T05:59:31.240 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj58 2022-09-02T05:59:31.241 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NNrWatkJs0AO3sVF7i8vwYtTcR4= 2022-09-02T05:59:31.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NNrWatkJs0AO3sVF7i8vwYtTcR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636979bc35ca128e-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.290 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.291 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.292 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.293 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.294 DEBUG:boto:Method: GET 2022-09-02T05:59:31.294 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.295 DEBUG:boto:Data: 2022-09-02T05:59:31.295 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.296 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.297 DEBUG:boto:Params: {} 2022-09-02T05:59:31.298 DEBUG:boto:Token: None 2022-09-02T05:59:31.298 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.299 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193daf4ddc89f216-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.303 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.304 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.305 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.305 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.306 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.306 DEBUG:boto:Method: GET 2022-09-02T05:59:31.307 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:31.307 DEBUG:boto:Data: 2022-09-02T05:59:31.308 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.309 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.310 DEBUG:boto:Params: {} 2022-09-02T05:59:31.311 DEBUG:boto:Token: None 2022-09-02T05:59:31.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a6bd5e531d3579-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.317 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:31.319 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.319 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.320 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.320 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.321 DEBUG:boto:Method: GET 2022-09-02T05:59:31.321 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.322 DEBUG:boto:Data: 2022-09-02T05:59:31.322 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.323 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.323 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.324 DEBUG:boto:Params: {} 2022-09-02T05:59:31.325 DEBUG:boto:Token: None 2022-09-02T05:59:31.325 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.326 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I= 2022-09-02T05:59:31.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I=', 'Content-Length': '0'} 2022-09-02T05:59:31.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1e34a749aea81d-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.330 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.330 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.331 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.331 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.332 DEBUG:boto:Method: GET 2022-09-02T05:59:31.332 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.333 DEBUG:boto:Data: 2022-09-02T05:59:31.333 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.334 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.335 DEBUG:boto:Params: {} 2022-09-02T05:59:31.336 DEBUG:boto:Token: None 2022-09-02T05:59:31.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I= 2022-09-02T05:59:31.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I=', 'Content-Length': '0'} 2022-09-02T05:59:31.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d11bfd8bc4e41aa-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.340 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.341 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.341 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.342 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.342 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.343 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.343 DEBUG:boto:Data: 2022-09-02T05:59:31.344 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.344 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.345 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.345 DEBUG:boto:Params: {} 2022-09-02T05:59:31.346 DEBUG:boto:Token: None 2022-09-02T05:59:31.346 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.347 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qga5UceIowf+UtC0BPxDOr7kNp8= 2022-09-02T05:59:31.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qga5UceIowf+UtC0BPxDOr7kNp8=', 'Content-Length': '0'} 2022-09-02T05:59:31.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d29a9bc1fe8a0ae-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.350 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.351 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.351 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.352 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.352 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.353 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.353 DEBUG:boto:Data: 2022-09-02T05:59:31.354 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.355 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.355 DEBUG:boto:Params: {} 2022-09-02T05:59:31.356 DEBUG:boto:Token: None 2022-09-02T05:59:31.357 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Qga5UceIowf+UtC0BPxDOr7kNp8= 2022-09-02T05:59:31.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Qga5UceIowf+UtC0BPxDOr7kNp8=', 'Content-Length': '0'} 2022-09-02T05:59:31.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0c27bbb5d8c1d52-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:31.362 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.362 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.363 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.363 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.364 DEBUG:boto:Method: GET 2022-09-02T05:59:31.364 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.365 DEBUG:boto:Data: 2022-09-02T05:59:31.365 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.366 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.366 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.367 DEBUG:boto:Params: {} 2022-09-02T05:59:31.367 DEBUG:boto:Token: None 2022-09-02T05:59:31.368 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.369 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I= 2022-09-02T05:59:31.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I=', 'Content-Length': '0'} 2022-09-02T05:59:31.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024a532ada0b01789-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.372 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.373 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.374 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.374 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.375 DEBUG:boto:Method: GET 2022-09-02T05:59:31.375 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.376 DEBUG:boto:Data: 2022-09-02T05:59:31.376 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.377 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.378 DEBUG:boto:Params: {} 2022-09-02T05:59:31.378 DEBUG:boto:Token: None 2022-09-02T05:59:31.379 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:31.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I= 2022-09-02T05:59:31.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZXc2FFfDOX/0gMMQjKxZInmPs0I=', 'Content-Length': '0'} 2022-09-02T05:59:31.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a672874bb859763e-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.383 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.384 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.384 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.385 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.385 DEBUG:boto:Method: GET 2022-09-02T05:59:31.386 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.386 DEBUG:boto:Data: 2022-09-02T05:59:31.387 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.388 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.389 DEBUG:boto:Params: {} 2022-09-02T05:59:31.389 DEBUG:boto:Token: None 2022-09-02T05:59:31.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b0b6db08a89bd4-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.394 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.395 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.396 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.396 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.397 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.397 DEBUG:boto:Method: GET 2022-09-02T05:59:31.398 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.399 DEBUG:boto:Data: 2022-09-02T05:59:31.399 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.400 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.401 DEBUG:boto:Params: {} 2022-09-02T05:59:31.401 DEBUG:boto:Token: None 2022-09-02T05:59:31.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a678d5897435ce31-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.454 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.455 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.456 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.456 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.457 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:31.457 DEBUG:boto:Data: 2022-09-02T05:59:31.458 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.459 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.459 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.460 DEBUG:boto:Params: {} 2022-09-02T05:59:31.460 DEBUG:boto:Token: None 2022-09-02T05:59:31.461 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62 2022-09-02T05:59:31.461 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:csiitM6Vh8FAV2aSFr8c2v/nTpI= 2022-09-02T05:59:31.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:csiitM6Vh8FAV2aSFr8c2v/nTpI=', 'Content-Length': '0'} 2022-09-02T05:59:31.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d1ae8d38f2aec3-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.465 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.466 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.466 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.467 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:31.467 DEBUG:boto:Data: 2022-09-02T05:59:31.468 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.469 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.469 DEBUG:boto:Params: {} 2022-09-02T05:59:31.470 DEBUG:boto:Token: None 2022-09-02T05:59:31.471 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62 2022-09-02T05:59:31.471 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:csiitM6Vh8FAV2aSFr8c2v/nTpI= 2022-09-02T05:59:31.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:csiitM6Vh8FAV2aSFr8c2v/nTpI=', 'Content-Length': '0'} 2022-09-02T05:59:31.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da03f6890c675fc-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:31.476 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.476 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.477 DEBUG:boto:Method: GET 2022-09-02T05:59:31.477 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:31.478 DEBUG:boto:Data: 2022-09-02T05:59:31.478 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.479 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.480 DEBUG:boto:Params: {} 2022-09-02T05:59:31.480 DEBUG:boto:Token: None 2022-09-02T05:59:31.481 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62 2022-09-02T05:59:31.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qrbpbdIJRFvY0I5RMWW6ESLLxVY= 2022-09-02T05:59:31.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qrbpbdIJRFvY0I5RMWW6ESLLxVY=', 'Content-Length': '0'} 2022-09-02T05:59:31.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a7d71ecd99098ec-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.495 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:31.496 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:31.497 DEBUG:boto:Method: GET 2022-09-02T05:59:31.497 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:31.498 DEBUG:boto:Data: 2022-09-02T05:59:31.498 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.499 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.500 DEBUG:boto:Params: {} 2022-09-02T05:59:31.500 DEBUG:boto:Token: None 2022-09-02T05:59:31.501 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj62 2022-09-02T05:59:31.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qrbpbdIJRFvY0I5RMWW6ESLLxVY= 2022-09-02T05:59:31.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qrbpbdIJRFvY0I5RMWW6ESLLxVY=', 'Content-Length': '0'} 2022-09-02T05:59:31.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c62b6abf83c80e-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.506 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.506 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.507 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.507 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.508 DEBUG:boto:Method: GET 2022-09-02T05:59:31.508 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.509 DEBUG:boto:Data: 2022-09-02T05:59:31.509 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.510 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.511 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.511 DEBUG:boto:Params: {} 2022-09-02T05:59:31.512 DEBUG:boto:Token: None 2022-09-02T05:59:31.512 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.513 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0abee176c8c503c-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.518 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.519 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.519 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.520 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.520 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.521 DEBUG:boto:Method: GET 2022-09-02T05:59:31.521 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:31.522 DEBUG:boto:Data: 2022-09-02T05:59:31.523 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.524 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.524 DEBUG:boto:Params: {} 2022-09-02T05:59:31.525 DEBUG:boto:Token: None 2022-09-02T05:59:31.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.526 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038b4683d357e57bd-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.531 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:31.533 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.533 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.534 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.534 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.535 DEBUG:boto:Method: GET 2022-09-02T05:59:31.536 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.536 DEBUG:boto:Data: 2022-09-02T05:59:31.537 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.537 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.538 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.538 DEBUG:boto:Params: {} 2022-09-02T05:59:31.539 DEBUG:boto:Token: None 2022-09-02T05:59:31.539 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.540 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM= 2022-09-02T05:59:31.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM=', 'Content-Length': '0'} 2022-09-02T05:59:31.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240ade227ad085bd-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.544 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.545 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.546 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.546 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.547 DEBUG:boto:Method: GET 2022-09-02T05:59:31.548 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.548 DEBUG:boto:Data: 2022-09-02T05:59:31.549 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.550 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.550 DEBUG:boto:Params: {} 2022-09-02T05:59:31.551 DEBUG:boto:Token: None 2022-09-02T05:59:31.551 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM= 2022-09-02T05:59:31.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM=', 'Content-Length': '0'} 2022-09-02T05:59:31.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036eb6fbd5a3d00e8-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.598 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.599 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.600 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.600 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.601 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.601 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.602 DEBUG:boto:Data: 2022-09-02T05:59:31.603 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.604 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.605 DEBUG:boto:Params: {} 2022-09-02T05:59:31.605 DEBUG:boto:Token: None 2022-09-02T05:59:31.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xOBTmikMZLCOAz4NQlhNE2yZ4no= 2022-09-02T05:59:31.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xOBTmikMZLCOAz4NQlhNE2yZ4no=', 'Content-Length': '0'} 2022-09-02T05:59:31.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0232dcd2a9e8a3d-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.610 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.611 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.612 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.612 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.613 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.614 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.615 DEBUG:boto:Data: 2022-09-02T05:59:31.615 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.616 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.617 DEBUG:boto:Params: {} 2022-09-02T05:59:31.617 DEBUG:boto:Token: None 2022-09-02T05:59:31.618 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xOBTmikMZLCOAz4NQlhNE2yZ4no= 2022-09-02T05:59:31.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xOBTmikMZLCOAz4NQlhNE2yZ4no=', 'Content-Length': '0'} 2022-09-02T05:59:31.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6793447a4cc6b1d-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:31.623 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.623 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.624 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.624 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.625 DEBUG:boto:Method: GET 2022-09-02T05:59:31.625 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.626 DEBUG:boto:Data: 2022-09-02T05:59:31.627 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.628 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.628 DEBUG:boto:Params: {} 2022-09-02T05:59:31.629 DEBUG:boto:Token: None 2022-09-02T05:59:31.629 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM= 2022-09-02T05:59:31.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM=', 'Content-Length': '0'} 2022-09-02T05:59:31.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008944c041dc43b16c-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.634 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.635 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.635 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.636 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.636 DEBUG:boto:Method: GET 2022-09-02T05:59:31.637 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.637 DEBUG:boto:Data: 2022-09-02T05:59:31.638 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.639 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.639 DEBUG:boto:Params: {} 2022-09-02T05:59:31.640 DEBUG:boto:Token: None 2022-09-02T05:59:31.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:31.642 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM= 2022-09-02T05:59:31.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SQbYrcdZsPYlGk6LczohBMT0EfM=', 'Content-Length': '0'} 2022-09-02T05:59:31.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aac874fdf1f66e9-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.646 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.647 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.647 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.648 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.649 DEBUG:boto:Method: GET 2022-09-02T05:59:31.650 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.650 DEBUG:boto:Data: 2022-09-02T05:59:31.651 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.652 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.652 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.653 DEBUG:boto:Params: {} 2022-09-02T05:59:31.653 DEBUG:boto:Token: None 2022-09-02T05:59:31.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ca7e8cf54e8b09-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.659 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.660 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.661 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.661 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.662 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.662 DEBUG:boto:Method: GET 2022-09-02T05:59:31.663 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.663 DEBUG:boto:Data: 2022-09-02T05:59:31.664 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.665 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.666 DEBUG:boto:Params: {} 2022-09-02T05:59:31.667 DEBUG:boto:Token: None 2022-09-02T05:59:31.668 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b84d3f80cdddd9f-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.674 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.675 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.676 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.677 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.677 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:31.678 DEBUG:boto:Data: 2022-09-02T05:59:31.678 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.679 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.679 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.680 DEBUG:boto:Params: {} 2022-09-02T05:59:31.681 DEBUG:boto:Token: None 2022-09-02T05:59:31.681 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68 2022-09-02T05:59:31.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ahudOR9FKo2XkTxKsEvAOzOoUSQ= 2022-09-02T05:59:31.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ahudOR9FKo2XkTxKsEvAOzOoUSQ=', 'Content-Length': '0'} 2022-09-02T05:59:31.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc590630ffe9ab8c-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.686 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.686 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.687 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.687 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:31.688 DEBUG:boto:Data: 2022-09-02T05:59:31.688 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.689 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.690 DEBUG:boto:Params: {} 2022-09-02T05:59:31.690 DEBUG:boto:Token: None 2022-09-02T05:59:31.691 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68 2022-09-02T05:59:31.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ahudOR9FKo2XkTxKsEvAOzOoUSQ= 2022-09-02T05:59:31.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ahudOR9FKo2XkTxKsEvAOzOoUSQ=', 'Content-Length': '0'} 2022-09-02T05:59:31.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e9f233ee90b26ec-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:31.696 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.696 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.697 DEBUG:boto:Method: GET 2022-09-02T05:59:31.697 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:31.698 DEBUG:boto:Data: 2022-09-02T05:59:31.699 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.699 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.700 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.700 DEBUG:boto:Params: {} 2022-09-02T05:59:31.701 DEBUG:boto:Token: None 2022-09-02T05:59:31.701 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68 2022-09-02T05:59:31.702 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q8nVk4cIAP/5DPYxhSKenZRKdf0= 2022-09-02T05:59:31.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q8nVk4cIAP/5DPYxhSKenZRKdf0=', 'Content-Length': '0'} 2022-09-02T05:59:31.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067633044a460b397-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.746 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:31.747 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:31.747 DEBUG:boto:Method: GET 2022-09-02T05:59:31.748 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:31.749 DEBUG:boto:Data: 2022-09-02T05:59:31.750 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.752 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.753 DEBUG:boto:Params: {} 2022-09-02T05:59:31.754 DEBUG:boto:Token: None 2022-09-02T05:59:31.755 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj68 2022-09-02T05:59:31.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q8nVk4cIAP/5DPYxhSKenZRKdf0= 2022-09-02T05:59:31.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q8nVk4cIAP/5DPYxhSKenZRKdf0=', 'Content-Length': '0'} 2022-09-02T05:59:31.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d962bbe65cc8728-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.759 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.760 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.760 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.761 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.761 DEBUG:boto:Method: GET 2022-09-02T05:59:31.762 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.762 DEBUG:boto:Data: 2022-09-02T05:59:31.763 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.764 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.764 DEBUG:boto:Params: {} 2022-09-02T05:59:31.765 DEBUG:boto:Token: None 2022-09-02T05:59:31.766 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d9789285912e73-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.771 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.771 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.772 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.772 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.773 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.773 DEBUG:boto:Method: GET 2022-09-02T05:59:31.774 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:31.775 DEBUG:boto:Data: 2022-09-02T05:59:31.775 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.776 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.777 DEBUG:boto:Params: {} 2022-09-02T05:59:31.777 DEBUG:boto:Token: None 2022-09-02T05:59:31.778 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000398bfb43c40b2bfb-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.784 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:31.786 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.786 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.787 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.787 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.788 DEBUG:boto:Method: GET 2022-09-02T05:59:31.788 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.789 DEBUG:boto:Data: 2022-09-02T05:59:31.789 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.790 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.791 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.791 DEBUG:boto:Params: {} 2022-09-02T05:59:31.792 DEBUG:boto:Token: None 2022-09-02T05:59:31.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc= 2022-09-02T05:59:31.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc=', 'Content-Length': '0'} 2022-09-02T05:59:31.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d9faa39444407c0-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.797 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.797 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.798 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.798 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.799 DEBUG:boto:Method: GET 2022-09-02T05:59:31.800 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.800 DEBUG:boto:Data: 2022-09-02T05:59:31.801 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.802 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.802 DEBUG:boto:Params: {} 2022-09-02T05:59:31.803 DEBUG:boto:Token: None 2022-09-02T05:59:31.804 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc= 2022-09-02T05:59:31.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc=', 'Content-Length': '0'} 2022-09-02T05:59:31.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c748d29e5ed71892-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.854 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.855 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.855 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.856 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.857 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.857 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.858 DEBUG:boto:Data: 2022-09-02T05:59:31.859 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.861 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.862 DEBUG:boto:Params: {} 2022-09-02T05:59:31.863 DEBUG:boto:Token: None 2022-09-02T05:59:31.864 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gp4whaCBGzKE9stPttBPxDen5ME= 2022-09-02T05:59:31.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gp4whaCBGzKE9stPttBPxDen5ME=', 'Content-Length': '0'} 2022-09-02T05:59:31.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5490eab69cf34c-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.869 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.870 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.871 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.872 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.872 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.873 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.874 DEBUG:boto:Data: 2022-09-02T05:59:31.874 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.875 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.876 DEBUG:boto:Params: {} 2022-09-02T05:59:31.876 DEBUG:boto:Token: None 2022-09-02T05:59:31.877 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gp4whaCBGzKE9stPttBPxDen5ME= 2022-09-02T05:59:31.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gp4whaCBGzKE9stPttBPxDen5ME=', 'Content-Length': '0'} 2022-09-02T05:59:31.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bb6ec90fdf3c1e-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:31.882 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.883 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.883 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.884 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.884 DEBUG:boto:Method: GET 2022-09-02T05:59:31.885 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.885 DEBUG:boto:Data: 2022-09-02T05:59:31.886 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.886 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.887 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.888 DEBUG:boto:Params: {} 2022-09-02T05:59:31.888 DEBUG:boto:Token: None 2022-09-02T05:59:31.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.890 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc= 2022-09-02T05:59:31.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc=', 'Content-Length': '0'} 2022-09-02T05:59:31.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d9c57342fe3768-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.893 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.894 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.894 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.895 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.895 DEBUG:boto:Method: GET 2022-09-02T05:59:31.896 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.896 DEBUG:boto:Data: 2022-09-02T05:59:31.897 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.898 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.899 DEBUG:boto:Params: {} 2022-09-02T05:59:31.899 DEBUG:boto:Token: None 2022-09-02T05:59:31.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:31.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc= 2022-09-02T05:59:31.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QZA3uEO2AA8HHKqX7JW11vjkrZc=', 'Content-Length': '0'} 2022-09-02T05:59:31.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9b9b50b978291e-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.904 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.905 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.905 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:31.906 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:31.907 DEBUG:boto:Method: GET 2022-09-02T05:59:31.907 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:31.908 DEBUG:boto:Data: 2022-09-02T05:59:31.908 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.909 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.910 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.910 DEBUG:boto:Params: {} 2022-09-02T05:59:31.911 DEBUG:boto:Token: None 2022-09-02T05:59:31.912 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.912 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfd738c456466c64-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.916 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.917 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:31.918 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:31.918 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:31.919 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:31.919 DEBUG:boto:Method: GET 2022-09-02T05:59:31.920 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:31.920 DEBUG:boto:Data: 2022-09-02T05:59:31.921 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.922 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.923 DEBUG:boto:Params: {} 2022-09-02T05:59:31.924 DEBUG:boto:Token: None 2022-09-02T05:59:31.925 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/ 2022-09-02T05:59:31.926 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4= 2022-09-02T05:59:31.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0E6gAAtBarC4hELLSZs/or5sfR4=', 'Content-Length': '0'} 2022-09-02T05:59:31.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba322de78004dd0-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.933 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:31.934 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.934 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.935 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.935 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:31.936 DEBUG:boto:Data: 2022-09-02T05:59:31.936 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.937 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.937 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.938 DEBUG:boto:Params: {} 2022-09-02T05:59:31.938 DEBUG:boto:Token: None 2022-09-02T05:59:31.939 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69 2022-09-02T05:59:31.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wgjl6dcb+Eo/ak2m7RyCpaG/hWo= 2022-09-02T05:59:31.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wgjl6dcb+Eo/ak2m7RyCpaG/hWo=', 'Content-Length': '0'} 2022-09-02T05:59:31.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd59a44c5b001726-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.943 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.943 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.944 DEBUG:boto:Method: HEAD 2022-09-02T05:59:31.944 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:31.945 DEBUG:boto:Data: 2022-09-02T05:59:31.945 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:31.946 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.947 DEBUG:boto:Params: {} 2022-09-02T05:59:31.947 DEBUG:boto:Token: None 2022-09-02T05:59:31.948 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69 2022-09-02T05:59:31.949 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wgjl6dcb+Eo/ak2m7RyCpaG/hWo= 2022-09-02T05:59:31.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wgjl6dcb+Eo/ak2m7RyCpaG/hWo=', 'Content-Length': '0'} 2022-09-02T05:59:31.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d4649108db54201-0063119bc3-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:31.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:31.952 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:31.953 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:31.953 DEBUG:boto:Method: GET 2022-09-02T05:59:31.954 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:31.955 DEBUG:boto:Data: 2022-09-02T05:59:31.955 DEBUG:boto:Headers: {} 2022-09-02T05:59:31.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:31.956 DEBUG:boto:Port: 8000 2022-09-02T05:59:31.957 DEBUG:boto:Params: {} 2022-09-02T05:59:31.957 DEBUG:boto:Token: None 2022-09-02T05:59:31.958 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:31 GMT /xljbkf-74/obj69 2022-09-02T05:59:31.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fu2YluaSxDheesCXGGZk9LuGGVo= 2022-09-02T05:59:31.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:31 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fu2YluaSxDheesCXGGZk9LuGGVo=', 'Content-Length': '0'} 2022-09-02T05:59:31.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000550ea482946bb310-0063119bc3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:31 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.006 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:32.007 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:32.008 DEBUG:boto:Method: GET 2022-09-02T05:59:32.008 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:32.009 DEBUG:boto:Data: 2022-09-02T05:59:32.009 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.011 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.011 DEBUG:boto:Params: {} 2022-09-02T05:59:32.012 DEBUG:boto:Token: None 2022-09-02T05:59:32.013 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj69 2022-09-02T05:59:32.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mYO2IN2MxOg40Bsb30nb8mH6xcE= 2022-09-02T05:59:32.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mYO2IN2MxOg40Bsb30nb8mH6xcE=', 'Content-Length': '0'} 2022-09-02T05:59:32.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31268ebef068136-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.017 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.018 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.019 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:32.019 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:32.020 DEBUG:boto:Method: GET 2022-09-02T05:59:32.021 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:32.021 DEBUG:boto:Data: 2022-09-02T05:59:32.022 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.023 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.024 DEBUG:boto:Params: {} 2022-09-02T05:59:32.024 DEBUG:boto:Token: None 2022-09-02T05:59:32.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcab377226e173e8-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.030 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.031 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.031 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.032 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:32.032 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:32.033 DEBUG:boto:Method: GET 2022-09-02T05:59:32.033 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:32.034 DEBUG:boto:Data: 2022-09-02T05:59:32.035 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.036 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.037 DEBUG:boto:Params: {} 2022-09-02T05:59:32.037 DEBUG:boto:Token: None 2022-09-02T05:59:32.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.039 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e79f9e65a60eb6-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.045 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:32.046 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.047 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.047 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.048 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.048 DEBUG:boto:Method: GET 2022-09-02T05:59:32.049 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.049 DEBUG:boto:Data: 2022-09-02T05:59:32.050 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.051 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.052 DEBUG:boto:Params: {} 2022-09-02T05:59:32.053 DEBUG:boto:Token: None 2022-09-02T05:59:32.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4= 2022-09-02T05:59:32.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4=', 'Content-Length': '0'} 2022-09-02T05:59:32.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b77eb01b5ebfc3db-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.057 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.058 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.059 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.059 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.060 DEBUG:boto:Method: GET 2022-09-02T05:59:32.061 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.061 DEBUG:boto:Data: 2022-09-02T05:59:32.062 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.064 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.065 DEBUG:boto:Params: {} 2022-09-02T05:59:32.066 DEBUG:boto:Token: None 2022-09-02T05:59:32.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4= 2022-09-02T05:59:32.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4=', 'Content-Length': '0'} 2022-09-02T05:59:32.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0576b743549903-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.114 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.115 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.116 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.117 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.118 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.119 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.119 DEBUG:boto:Data: 2022-09-02T05:59:32.120 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.120 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.121 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.121 DEBUG:boto:Params: {} 2022-09-02T05:59:32.122 DEBUG:boto:Token: None 2022-09-02T05:59:32.123 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRmIoaNyfSIzvONljY0tZ9ahorg= 2022-09-02T05:59:32.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRmIoaNyfSIzvONljY0tZ9ahorg=', 'Content-Length': '0'} 2022-09-02T05:59:32.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070aab42aeb8bd13d-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.127 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.127 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.128 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.128 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.129 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.129 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.130 DEBUG:boto:Data: 2022-09-02T05:59:32.131 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.132 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.132 DEBUG:boto:Params: {} 2022-09-02T05:59:32.133 DEBUG:boto:Token: None 2022-09-02T05:59:32.133 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:IRmIoaNyfSIzvONljY0tZ9ahorg= 2022-09-02T05:59:32.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:IRmIoaNyfSIzvONljY0tZ9ahorg=', 'Content-Length': '0'} 2022-09-02T05:59:32.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b42ce1210c42b68-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:32.139 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.139 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.140 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.140 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.141 DEBUG:boto:Method: GET 2022-09-02T05:59:32.141 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.142 DEBUG:boto:Data: 2022-09-02T05:59:32.143 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.143 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.144 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.144 DEBUG:boto:Params: {} 2022-09-02T05:59:32.145 DEBUG:boto:Token: None 2022-09-02T05:59:32.145 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4= 2022-09-02T05:59:32.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4=', 'Content-Length': '0'} 2022-09-02T05:59:32.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243f0397b90e38c4-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.150 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.151 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.151 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.152 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.152 DEBUG:boto:Method: GET 2022-09-02T05:59:32.153 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.153 DEBUG:boto:Data: 2022-09-02T05:59:32.154 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.155 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.156 DEBUG:boto:Params: {} 2022-09-02T05:59:32.156 DEBUG:boto:Token: None 2022-09-02T05:59:32.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:32.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4= 2022-09-02T05:59:32.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2uIjFvJDkNE935/CSTuQpUG+YR4=', 'Content-Length': '0'} 2022-09-02T05:59:32.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748fd025c9175cca-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.162 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.162 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.163 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.163 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.164 DEBUG:boto:Method: GET 2022-09-02T05:59:32.165 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.165 DEBUG:boto:Data: 2022-09-02T05:59:32.166 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.166 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.167 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.167 DEBUG:boto:Params: {} 2022-09-02T05:59:32.168 DEBUG:boto:Token: None 2022-09-02T05:59:32.168 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2876f53c921a321-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.173 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.174 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.175 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.175 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.176 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.176 DEBUG:boto:Method: GET 2022-09-02T05:59:32.177 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.177 DEBUG:boto:Data: 2022-09-02T05:59:32.178 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.179 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.179 DEBUG:boto:Params: {} 2022-09-02T05:59:32.180 DEBUG:boto:Token: None 2022-09-02T05:59:32.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf1217f4fd801cff-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.187 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.188 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.189 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.189 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.190 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:32.190 DEBUG:boto:Data: 2022-09-02T05:59:32.191 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.192 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.192 DEBUG:boto:Params: {} 2022-09-02T05:59:32.193 DEBUG:boto:Token: None 2022-09-02T05:59:32.194 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74 2022-09-02T05:59:32.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8axuMEqUIp+mvEuqcXjzOYCezI= 2022-09-02T05:59:32.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8axuMEqUIp+mvEuqcXjzOYCezI=', 'Content-Length': '0'} 2022-09-02T05:59:32.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005daa496036c6fe11-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.198 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.198 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.199 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.199 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:32.200 DEBUG:boto:Data: 2022-09-02T05:59:32.200 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.201 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.202 DEBUG:boto:Params: {} 2022-09-02T05:59:32.203 DEBUG:boto:Token: None 2022-09-02T05:59:32.203 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74 2022-09-02T05:59:32.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:h8axuMEqUIp+mvEuqcXjzOYCezI= 2022-09-02T05:59:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:h8axuMEqUIp+mvEuqcXjzOYCezI=', 'Content-Length': '0'} 2022-09-02T05:59:32.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8d65ca9bf64e3fd-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:32.208 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.208 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.209 DEBUG:boto:Method: GET 2022-09-02T05:59:32.210 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:32.210 DEBUG:boto:Data: 2022-09-02T05:59:32.211 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.211 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.212 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.212 DEBUG:boto:Params: {} 2022-09-02T05:59:32.213 DEBUG:boto:Token: None 2022-09-02T05:59:32.213 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74 2022-09-02T05:59:32.214 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s9NHySkvbd+xtfMUkZjMCcMHBjU= 2022-09-02T05:59:32.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s9NHySkvbd+xtfMUkZjMCcMHBjU=', 'Content-Length': '0'} 2022-09-02T05:59:32.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47daebcdcc2b47e-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.218 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:32.219 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:32.219 DEBUG:boto:Method: GET 2022-09-02T05:59:32.220 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:32.220 DEBUG:boto:Data: 2022-09-02T05:59:32.221 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.222 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.222 DEBUG:boto:Params: {} 2022-09-02T05:59:32.223 DEBUG:boto:Token: None 2022-09-02T05:59:32.224 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj74 2022-09-02T05:59:32.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s9NHySkvbd+xtfMUkZjMCcMHBjU= 2022-09-02T05:59:32.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s9NHySkvbd+xtfMUkZjMCcMHBjU=', 'Content-Length': '0'} 2022-09-02T05:59:32.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000588835694ab5ce67-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.270 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.271 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.271 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.272 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.272 DEBUG:boto:Method: GET 2022-09-02T05:59:32.273 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.273 DEBUG:boto:Data: 2022-09-02T05:59:32.274 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.275 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.276 DEBUG:boto:Params: {} 2022-09-02T05:59:32.276 DEBUG:boto:Token: None 2022-09-02T05:59:32.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5de512015ca3f1c-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.282 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.283 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.283 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.284 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.284 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.285 DEBUG:boto:Method: GET 2022-09-02T05:59:32.285 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:32.286 DEBUG:boto:Data: 2022-09-02T05:59:32.286 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.287 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.288 DEBUG:boto:Params: {} 2022-09-02T05:59:32.289 DEBUG:boto:Token: None 2022-09-02T05:59:32.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bff5a2a02f44efa-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.296 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:32.297 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.298 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.298 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.299 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.299 DEBUG:boto:Method: GET 2022-09-02T05:59:32.300 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.300 DEBUG:boto:Data: 2022-09-02T05:59:32.301 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.301 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.302 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.303 DEBUG:boto:Params: {} 2022-09-02T05:59:32.303 DEBUG:boto:Token: None 2022-09-02T05:59:32.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE= 2022-09-02T05:59:32.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE=', 'Content-Length': '0'} 2022-09-02T05:59:32.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe2195de2592cab-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.308 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.309 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.309 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.310 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.311 DEBUG:boto:Method: GET 2022-09-02T05:59:32.311 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.312 DEBUG:boto:Data: 2022-09-02T05:59:32.313 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.314 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.314 DEBUG:boto:Params: {} 2022-09-02T05:59:32.315 DEBUG:boto:Token: None 2022-09-02T05:59:32.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE= 2022-09-02T05:59:32.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE=', 'Content-Length': '0'} 2022-09-02T05:59:32.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fffadb275cbe5c0-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.319 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.320 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.321 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.321 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.322 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.322 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.323 DEBUG:boto:Data: 2022-09-02T05:59:32.323 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.324 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.325 DEBUG:boto:Params: {} 2022-09-02T05:59:32.325 DEBUG:boto:Token: None 2022-09-02T05:59:32.326 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4LOXg4mlTf+Gk+ZqOcL3oVo8ZXE= 2022-09-02T05:59:32.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4LOXg4mlTf+Gk+ZqOcL3oVo8ZXE=', 'Content-Length': '0'} 2022-09-02T05:59:32.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec7b700f860ba19-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.330 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.330 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.331 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.331 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.332 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.332 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.333 DEBUG:boto:Data: 2022-09-02T05:59:32.333 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.334 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.335 DEBUG:boto:Params: {} 2022-09-02T05:59:32.336 DEBUG:boto:Token: None 2022-09-02T05:59:32.336 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4LOXg4mlTf+Gk+ZqOcL3oVo8ZXE= 2022-09-02T05:59:32.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4LOXg4mlTf+Gk+ZqOcL3oVo8ZXE=', 'Content-Length': '0'} 2022-09-02T05:59:32.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd7dc6da2347d9b-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:32.341 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.341 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.342 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.342 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.343 DEBUG:boto:Method: GET 2022-09-02T05:59:32.343 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.344 DEBUG:boto:Data: 2022-09-02T05:59:32.345 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.346 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.346 DEBUG:boto:Params: {} 2022-09-02T05:59:32.347 DEBUG:boto:Token: None 2022-09-02T05:59:32.347 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE= 2022-09-02T05:59:32.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE=', 'Content-Length': '0'} 2022-09-02T05:59:32.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682d5363cec7b85c-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.352 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.352 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.353 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.353 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.354 DEBUG:boto:Method: GET 2022-09-02T05:59:32.354 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.355 DEBUG:boto:Data: 2022-09-02T05:59:32.355 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.357 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.357 DEBUG:boto:Params: {} 2022-09-02T05:59:32.358 DEBUG:boto:Token: None 2022-09-02T05:59:32.359 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:32.359 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE= 2022-09-02T05:59:32.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2pyq+sewZQKrbxF8XAhj8WjF8PE=', 'Content-Length': '0'} 2022-09-02T05:59:32.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a2791deef52bca-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.406 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.407 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.407 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.408 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.409 DEBUG:boto:Method: GET 2022-09-02T05:59:32.409 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.410 DEBUG:boto:Data: 2022-09-02T05:59:32.410 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.411 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.411 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.412 DEBUG:boto:Params: {} 2022-09-02T05:59:32.412 DEBUG:boto:Token: None 2022-09-02T05:59:32.413 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.414 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000697e06e9fbfb3da-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.417 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.418 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.419 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.419 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.420 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.420 DEBUG:boto:Method: GET 2022-09-02T05:59:32.421 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.421 DEBUG:boto:Data: 2022-09-02T05:59:32.422 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.423 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.424 DEBUG:boto:Params: {} 2022-09-02T05:59:32.424 DEBUG:boto:Token: None 2022-09-02T05:59:32.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.425 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee280b4f55893e5-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.431 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.432 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.433 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.433 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.434 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:32.434 DEBUG:boto:Data: 2022-09-02T05:59:32.435 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.435 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.436 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.436 DEBUG:boto:Params: {} 2022-09-02T05:59:32.437 DEBUG:boto:Token: None 2022-09-02T05:59:32.437 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76 2022-09-02T05:59:32.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yGwkB4SRLdUzQyG0ro5klp6JU1I= 2022-09-02T05:59:32.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yGwkB4SRLdUzQyG0ro5klp6JU1I=', 'Content-Length': '0'} 2022-09-02T05:59:32.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d799b2f679054c64-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.442 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.443 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.443 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.444 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:32.444 DEBUG:boto:Data: 2022-09-02T05:59:32.445 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.446 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.446 DEBUG:boto:Params: {} 2022-09-02T05:59:32.447 DEBUG:boto:Token: None 2022-09-02T05:59:32.448 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76 2022-09-02T05:59:32.448 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yGwkB4SRLdUzQyG0ro5klp6JU1I= 2022-09-02T05:59:32.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yGwkB4SRLdUzQyG0ro5klp6JU1I=', 'Content-Length': '0'} 2022-09-02T05:59:32.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081062d82f5153f1b-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:32.453 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.453 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.454 DEBUG:boto:Method: GET 2022-09-02T05:59:32.455 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:32.455 DEBUG:boto:Data: 2022-09-02T05:59:32.456 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.457 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.458 DEBUG:boto:Params: {} 2022-09-02T05:59:32.458 DEBUG:boto:Token: None 2022-09-02T05:59:32.459 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76 2022-09-02T05:59:32.460 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+KShXqt69HacS/RCNL/9kA1UIiU= 2022-09-02T05:59:32.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+KShXqt69HacS/RCNL/9kA1UIiU=', 'Content-Length': '0'} 2022-09-02T05:59:32.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad2b8b615723933-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.464 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:32.465 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:32.466 DEBUG:boto:Method: GET 2022-09-02T05:59:32.467 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:32.467 DEBUG:boto:Data: 2022-09-02T05:59:32.468 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.469 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.470 DEBUG:boto:Params: {} 2022-09-02T05:59:32.471 DEBUG:boto:Token: None 2022-09-02T05:59:32.472 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj76 2022-09-02T05:59:32.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+KShXqt69HacS/RCNL/9kA1UIiU= 2022-09-02T05:59:32.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+KShXqt69HacS/RCNL/9kA1UIiU=', 'Content-Length': '0'} 2022-09-02T05:59:32.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005acc5c3ed2484af1-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.518 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.519 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.520 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.521 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.523 DEBUG:boto:Method: GET 2022-09-02T05:59:32.524 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.525 DEBUG:boto:Data: 2022-09-02T05:59:32.526 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.527 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.528 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.529 DEBUG:boto:Params: {} 2022-09-02T05:59:32.531 DEBUG:boto:Token: None 2022-09-02T05:59:32.532 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009826bbf7cd66f42-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.539 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.540 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.541 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.543 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.544 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.545 DEBUG:boto:Method: GET 2022-09-02T05:59:32.546 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:32.547 DEBUG:boto:Data: 2022-09-02T05:59:32.549 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.551 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.553 DEBUG:boto:Params: {} 2022-09-02T05:59:32.554 DEBUG:boto:Token: None 2022-09-02T05:59:32.556 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.557 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097fb72d99a7870b-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.563 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:32.564 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.565 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.566 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.567 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.567 DEBUG:boto:Method: GET 2022-09-02T05:59:32.568 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.569 DEBUG:boto:Data: 2022-09-02T05:59:32.569 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.570 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.571 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.571 DEBUG:boto:Params: {} 2022-09-02T05:59:32.572 DEBUG:boto:Token: None 2022-09-02T05:59:32.573 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.574 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU= 2022-09-02T05:59:32.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU=', 'Content-Length': '0'} 2022-09-02T05:59:32.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce318df8f623d38-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.578 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.579 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.580 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.581 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.582 DEBUG:boto:Method: GET 2022-09-02T05:59:32.583 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.584 DEBUG:boto:Data: 2022-09-02T05:59:32.585 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.587 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.587 DEBUG:boto:Params: {} 2022-09-02T05:59:32.588 DEBUG:boto:Token: None 2022-09-02T05:59:32.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU= 2022-09-02T05:59:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU=', 'Content-Length': '0'} 2022-09-02T05:59:32.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0afb3273b7a24d1-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.596 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.597 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.599 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.601 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.602 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.604 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.605 DEBUG:boto:Data: 2022-09-02T05:59:32.606 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.608 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.610 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.611 DEBUG:boto:Params: {} 2022-09-02T05:59:32.613 DEBUG:boto:Token: None 2022-09-02T05:59:32.614 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u7bPJhX89aVrLn+MeYH4fiKYq/0= 2022-09-02T05:59:32.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u7bPJhX89aVrLn+MeYH4fiKYq/0=', 'Content-Length': '0'} 2022-09-02T05:59:32.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ef173c7c02ffdc-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.620 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.621 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.622 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.623 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.624 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.624 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.625 DEBUG:boto:Data: 2022-09-02T05:59:32.626 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.628 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.628 DEBUG:boto:Params: {} 2022-09-02T05:59:32.629 DEBUG:boto:Token: None 2022-09-02T05:59:32.630 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:u7bPJhX89aVrLn+MeYH4fiKYq/0= 2022-09-02T05:59:32.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:u7bPJhX89aVrLn+MeYH4fiKYq/0=', 'Content-Length': '0'} 2022-09-02T05:59:32.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba507bd10adb77a-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:32.635 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.636 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.636 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.637 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.638 DEBUG:boto:Method: GET 2022-09-02T05:59:32.639 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.639 DEBUG:boto:Data: 2022-09-02T05:59:32.640 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.641 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.642 DEBUG:boto:Params: {} 2022-09-02T05:59:32.643 DEBUG:boto:Token: None 2022-09-02T05:59:32.644 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU= 2022-09-02T05:59:32.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU=', 'Content-Length': '0'} 2022-09-02T05:59:32.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003222d70a521d3f42-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.649 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.649 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.650 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.650 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.651 DEBUG:boto:Method: GET 2022-09-02T05:59:32.651 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.652 DEBUG:boto:Data: 2022-09-02T05:59:32.653 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.654 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.654 DEBUG:boto:Params: {} 2022-09-02T05:59:32.655 DEBUG:boto:Token: None 2022-09-02T05:59:32.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:32.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU= 2022-09-02T05:59:32.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WsgnDb8YEvLgq/N+k+iLDkP7zGU=', 'Content-Length': '0'} 2022-09-02T05:59:32.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d52121828fbce253-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.660 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.660 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.661 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.662 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.662 DEBUG:boto:Method: GET 2022-09-02T05:59:32.663 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.663 DEBUG:boto:Data: 2022-09-02T05:59:32.664 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.664 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.665 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.665 DEBUG:boto:Params: {} 2022-09-02T05:59:32.666 DEBUG:boto:Token: None 2022-09-02T05:59:32.666 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.667 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8c144366dc8f9aa-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.682 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.683 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.683 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.684 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.684 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.685 DEBUG:boto:Method: GET 2022-09-02T05:59:32.686 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.686 DEBUG:boto:Data: 2022-09-02T05:59:32.687 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.688 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.688 DEBUG:boto:Params: {} 2022-09-02T05:59:32.689 DEBUG:boto:Token: None 2022-09-02T05:59:32.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ee900b58ae9a90-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.738 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.739 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.740 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.740 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.741 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:32.741 DEBUG:boto:Data: 2022-09-02T05:59:32.742 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.742 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.743 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.743 DEBUG:boto:Params: {} 2022-09-02T05:59:32.744 DEBUG:boto:Token: None 2022-09-02T05:59:32.744 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85 2022-09-02T05:59:32.745 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9twOv5dhwC5/4T80XyB1bl6I+Ac= 2022-09-02T05:59:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9twOv5dhwC5/4T80XyB1bl6I+Ac=', 'Content-Length': '0'} 2022-09-02T05:59:32.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea3f91c43dfb3111-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.748 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.749 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.750 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.750 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:32.751 DEBUG:boto:Data: 2022-09-02T05:59:32.751 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.752 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.753 DEBUG:boto:Params: {} 2022-09-02T05:59:32.754 DEBUG:boto:Token: None 2022-09-02T05:59:32.754 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85 2022-09-02T05:59:32.755 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9twOv5dhwC5/4T80XyB1bl6I+Ac= 2022-09-02T05:59:32.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9twOv5dhwC5/4T80XyB1bl6I+Ac=', 'Content-Length': '0'} 2022-09-02T05:59:32.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2fc0d3437a17a0d-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:32.759 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.760 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.760 DEBUG:boto:Method: GET 2022-09-02T05:59:32.761 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:32.761 DEBUG:boto:Data: 2022-09-02T05:59:32.762 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.762 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.763 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.763 DEBUG:boto:Params: {} 2022-09-02T05:59:32.764 DEBUG:boto:Token: None 2022-09-02T05:59:32.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85 2022-09-02T05:59:32.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:781ctaL3L0hKixSRAa4RL0aQPqw= 2022-09-02T05:59:32.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:781ctaL3L0hKixSRAa4RL0aQPqw=', 'Content-Length': '0'} 2022-09-02T05:59:32.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053c37dbd3fdbef7-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.769 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:32.769 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:32.770 DEBUG:boto:Method: GET 2022-09-02T05:59:32.771 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:32.771 DEBUG:boto:Data: 2022-09-02T05:59:32.772 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.773 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.773 DEBUG:boto:Params: {} 2022-09-02T05:59:32.774 DEBUG:boto:Token: None 2022-09-02T05:59:32.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj85 2022-09-02T05:59:32.775 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:781ctaL3L0hKixSRAa4RL0aQPqw= 2022-09-02T05:59:32.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:781ctaL3L0hKixSRAa4RL0aQPqw=', 'Content-Length': '0'} 2022-09-02T05:59:32.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c3e19a071369ad-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.779 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.780 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.780 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.781 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.781 DEBUG:boto:Method: GET 2022-09-02T05:59:32.782 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.783 DEBUG:boto:Data: 2022-09-02T05:59:32.783 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.784 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.784 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.785 DEBUG:boto:Params: {} 2022-09-02T05:59:32.785 DEBUG:boto:Token: None 2022-09-02T05:59:32.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b6bfc39ef63c0c-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.791 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.792 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.792 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.793 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.793 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.794 DEBUG:boto:Method: GET 2022-09-02T05:59:32.794 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:32.795 DEBUG:boto:Data: 2022-09-02T05:59:32.795 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.796 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.797 DEBUG:boto:Params: {} 2022-09-02T05:59:32.798 DEBUG:boto:Token: None 2022-09-02T05:59:32.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008148d6dca5034164-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.804 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:32.806 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.806 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.807 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.807 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.808 DEBUG:boto:Method: GET 2022-09-02T05:59:32.808 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.809 DEBUG:boto:Data: 2022-09-02T05:59:32.809 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.810 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.810 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.811 DEBUG:boto:Params: {} 2022-09-02T05:59:32.812 DEBUG:boto:Token: None 2022-09-02T05:59:32.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw= 2022-09-02T05:59:32.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw=', 'Content-Length': '0'} 2022-09-02T05:59:32.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ee9f53ca8208296-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.862 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.863 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.863 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.864 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.865 DEBUG:boto:Method: GET 2022-09-02T05:59:32.866 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.866 DEBUG:boto:Data: 2022-09-02T05:59:32.867 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.868 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.869 DEBUG:boto:Params: {} 2022-09-02T05:59:32.870 DEBUG:boto:Token: None 2022-09-02T05:59:32.870 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw= 2022-09-02T05:59:32.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw=', 'Content-Length': '0'} 2022-09-02T05:59:32.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1f938e7da5527c-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.875 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.875 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.876 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.876 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.877 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.877 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.878 DEBUG:boto:Data: 2022-09-02T05:59:32.878 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.879 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.880 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.880 DEBUG:boto:Params: {} 2022-09-02T05:59:32.881 DEBUG:boto:Token: None 2022-09-02T05:59:32.882 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.883 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:StMFt4UQ6hxmQ33+YQgvC9cJuyQ= 2022-09-02T05:59:32.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:StMFt4UQ6hxmQ33+YQgvC9cJuyQ=', 'Content-Length': '0'} 2022-09-02T05:59:32.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103a7e001f62be4b-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.886 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.887 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.887 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.888 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.888 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.889 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.889 DEBUG:boto:Data: 2022-09-02T05:59:32.890 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.891 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.892 DEBUG:boto:Params: {} 2022-09-02T05:59:32.893 DEBUG:boto:Token: None 2022-09-02T05:59:32.894 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:StMFt4UQ6hxmQ33+YQgvC9cJuyQ= 2022-09-02T05:59:32.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:StMFt4UQ6hxmQ33+YQgvC9cJuyQ=', 'Content-Length': '0'} 2022-09-02T05:59:32.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be289158ba1d5ef8-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:32.899 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.899 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.900 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.900 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.901 DEBUG:boto:Method: GET 2022-09-02T05:59:32.902 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.902 DEBUG:boto:Data: 2022-09-02T05:59:32.903 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.903 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.904 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.904 DEBUG:boto:Params: {} 2022-09-02T05:59:32.905 DEBUG:boto:Token: None 2022-09-02T05:59:32.906 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw= 2022-09-02T05:59:32.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw=', 'Content-Length': '0'} 2022-09-02T05:59:32.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a3b6b71dd4c97e-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.910 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.911 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.911 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.912 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.912 DEBUG:boto:Method: GET 2022-09-02T05:59:32.913 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.913 DEBUG:boto:Data: 2022-09-02T05:59:32.914 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.915 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.916 DEBUG:boto:Params: {} 2022-09-02T05:59:32.916 DEBUG:boto:Token: None 2022-09-02T05:59:32.917 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:32.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw= 2022-09-02T05:59:32.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BF0w99V5Ny5WQgR85F5ZQvzBsNw=', 'Content-Length': '0'} 2022-09-02T05:59:32.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f249b50740f4f554-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.921 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.922 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.922 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:32.923 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:32.923 DEBUG:boto:Method: GET 2022-09-02T05:59:32.924 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:32.924 DEBUG:boto:Data: 2022-09-02T05:59:32.925 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.925 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.926 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.927 DEBUG:boto:Params: {} 2022-09-02T05:59:32.927 DEBUG:boto:Token: None 2022-09-02T05:59:32.928 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.928 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9fe0202fe778df5-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.933 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.933 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:32.934 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:32.935 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:32.935 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:32.936 DEBUG:boto:Method: GET 2022-09-02T05:59:32.936 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:32.937 DEBUG:boto:Data: 2022-09-02T05:59:32.937 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.938 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.939 DEBUG:boto:Params: {} 2022-09-02T05:59:32.940 DEBUG:boto:Token: None 2022-09-02T05:59:32.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/ 2022-09-02T05:59:32.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A= 2022-09-02T05:59:32.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GALYHPZ+DGzmX5eUXhvsX1Oc22A=', 'Content-Length': '0'} 2022-09-02T05:59:32.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7d5eb99db695665-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.947 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:32.948 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.948 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.949 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.949 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:32.950 DEBUG:boto:Data: 2022-09-02T05:59:32.950 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.951 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.951 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.952 DEBUG:boto:Params: {} 2022-09-02T05:59:32.953 DEBUG:boto:Token: None 2022-09-02T05:59:32.953 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87 2022-09-02T05:59:32.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DESz90HMLLL7oXbu7IEmHKy4io4= 2022-09-02T05:59:32.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DESz90HMLLL7oXbu7IEmHKy4io4=', 'Content-Length': '0'} 2022-09-02T05:59:32.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd93bbc605f1601-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.957 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.958 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.958 DEBUG:boto:Method: HEAD 2022-09-02T05:59:32.959 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:32.959 DEBUG:boto:Data: 2022-09-02T05:59:32.960 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.961 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.961 DEBUG:boto:Params: {} 2022-09-02T05:59:32.962 DEBUG:boto:Token: None 2022-09-02T05:59:32.963 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87 2022-09-02T05:59:32.963 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DESz90HMLLL7oXbu7IEmHKy4io4= 2022-09-02T05:59:32.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DESz90HMLLL7oXbu7IEmHKy4io4=', 'Content-Length': '0'} 2022-09-02T05:59:32.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eed85e25c2cf287-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:32.967 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.968 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.968 DEBUG:boto:Method: GET 2022-09-02T05:59:32.969 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:32.969 DEBUG:boto:Data: 2022-09-02T05:59:32.970 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:32.971 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.972 DEBUG:boto:Params: {} 2022-09-02T05:59:32.972 DEBUG:boto:Token: None 2022-09-02T05:59:32.973 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87 2022-09-02T05:59:32.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:epNFui7T5JRdtM8spAA3+2+XNX4= 2022-09-02T05:59:32.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:epNFui7T5JRdtM8spAA3+2+XNX4=', 'Content-Length': '0'} 2022-09-02T05:59:32.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000610d17eac1079513-0063119bc4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:32.977 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:32.977 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:32.978 DEBUG:boto:Method: GET 2022-09-02T05:59:32.979 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:32.979 DEBUG:boto:Data: 2022-09-02T05:59:32.980 DEBUG:boto:Headers: {} 2022-09-02T05:59:32.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:32.981 DEBUG:boto:Port: 8000 2022-09-02T05:59:32.981 DEBUG:boto:Params: {} 2022-09-02T05:59:32.982 DEBUG:boto:Token: None 2022-09-02T05:59:32.983 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:32 GMT /xljbkf-74/obj87 2022-09-02T05:59:32.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:epNFui7T5JRdtM8spAA3+2+XNX4= 2022-09-02T05:59:32.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:32 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:epNFui7T5JRdtM8spAA3+2+XNX4=', 'Content-Length': '0'} 2022-09-02T05:59:32.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4203b3e4fa585a4-0063119bc4-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:32 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.030 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.031 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.031 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:33.032 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:33.032 DEBUG:boto:Method: GET 2022-09-02T05:59:33.033 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:33.033 DEBUG:boto:Data: 2022-09-02T05:59:33.034 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.035 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.035 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.036 DEBUG:boto:Params: {} 2022-09-02T05:59:33.036 DEBUG:boto:Token: None 2022-09-02T05:59:33.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.037 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077a2e541f068ce09-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.041 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.042 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.043 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.043 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:33.044 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:33.045 DEBUG:boto:Method: GET 2022-09-02T05:59:33.045 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:33.046 DEBUG:boto:Data: 2022-09-02T05:59:33.046 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.047 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.048 DEBUG:boto:Params: {} 2022-09-02T05:59:33.049 DEBUG:boto:Token: None 2022-09-02T05:59:33.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006497884a1e8cf2d0-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.056 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:33.057 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.058 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.058 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.059 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.059 DEBUG:boto:Method: GET 2022-09-02T05:59:33.060 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.060 DEBUG:boto:Data: 2022-09-02T05:59:33.061 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.061 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.062 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.062 DEBUG:boto:Params: {} 2022-09-02T05:59:33.063 DEBUG:boto:Token: None 2022-09-02T05:59:33.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4= 2022-09-02T05:59:33.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4=', 'Content-Length': '0'} 2022-09-02T05:59:33.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598fa52d650d6a1d-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.068 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.068 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.069 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.069 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.070 DEBUG:boto:Method: GET 2022-09-02T05:59:33.070 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.071 DEBUG:boto:Data: 2022-09-02T05:59:33.071 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.072 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.073 DEBUG:boto:Params: {} 2022-09-02T05:59:33.073 DEBUG:boto:Token: None 2022-09-02T05:59:33.074 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4= 2022-09-02T05:59:33.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4=', 'Content-Length': '0'} 2022-09-02T05:59:33.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78fc2e9a04fa410-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.078 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.079 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.079 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.080 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.080 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.081 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.081 DEBUG:boto:Data: 2022-09-02T05:59:33.082 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.082 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.083 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.083 DEBUG:boto:Params: {} 2022-09-02T05:59:33.084 DEBUG:boto:Token: None 2022-09-02T05:59:33.084 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.085 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YD3WoSCYd1F9As4UTNApL+Efs9Y= 2022-09-02T05:59:33.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YD3WoSCYd1F9As4UTNApL+Efs9Y=', 'Content-Length': '0'} 2022-09-02T05:59:33.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332659d96910fd22-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.089 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.089 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.090 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.090 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.091 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.091 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.092 DEBUG:boto:Data: 2022-09-02T05:59:33.092 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.093 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.094 DEBUG:boto:Params: {} 2022-09-02T05:59:33.095 DEBUG:boto:Token: None 2022-09-02T05:59:33.095 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YD3WoSCYd1F9As4UTNApL+Efs9Y= 2022-09-02T05:59:33.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YD3WoSCYd1F9As4UTNApL+Efs9Y=', 'Content-Length': '0'} 2022-09-02T05:59:33.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ed09856e64151e-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:33.100 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.100 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.101 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.101 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.102 DEBUG:boto:Method: GET 2022-09-02T05:59:33.102 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.103 DEBUG:boto:Data: 2022-09-02T05:59:33.104 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.105 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.106 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.107 DEBUG:boto:Params: {} 2022-09-02T05:59:33.108 DEBUG:boto:Token: None 2022-09-02T05:59:33.110 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4= 2022-09-02T05:59:33.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4=', 'Content-Length': '0'} 2022-09-02T05:59:33.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7bda44027e24c66-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.115 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.115 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.116 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.116 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.117 DEBUG:boto:Method: GET 2022-09-02T05:59:33.117 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.118 DEBUG:boto:Data: 2022-09-02T05:59:33.119 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.120 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.120 DEBUG:boto:Params: {} 2022-09-02T05:59:33.121 DEBUG:boto:Token: None 2022-09-02T05:59:33.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:33.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4= 2022-09-02T05:59:33.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PU0vqJyg4TteAubvWvwuRCygAk4=', 'Content-Length': '0'} 2022-09-02T05:59:33.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566869a2e83e5158-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.170 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.171 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.171 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.172 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.172 DEBUG:boto:Method: GET 2022-09-02T05:59:33.173 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.174 DEBUG:boto:Data: 2022-09-02T05:59:33.174 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.175 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.175 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.176 DEBUG:boto:Params: {} 2022-09-02T05:59:33.177 DEBUG:boto:Token: None 2022-09-02T05:59:33.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a387106e6275beb8-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.182 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.183 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.184 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.184 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.185 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.185 DEBUG:boto:Method: GET 2022-09-02T05:59:33.186 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.187 DEBUG:boto:Data: 2022-09-02T05:59:33.187 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.189 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.189 DEBUG:boto:Params: {} 2022-09-02T05:59:33.190 DEBUG:boto:Token: None 2022-09-02T05:59:33.191 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035589edac4e8b652-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.197 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.198 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.199 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.199 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.200 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:33.201 DEBUG:boto:Data: 2022-09-02T05:59:33.201 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.202 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.203 DEBUG:boto:Params: {} 2022-09-02T05:59:33.203 DEBUG:boto:Token: None 2022-09-02T05:59:33.204 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92 2022-09-02T05:59:33.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxaSQ3yZiD11D0ztwl/X6D4y8/Q= 2022-09-02T05:59:33.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxaSQ3yZiD11D0ztwl/X6D4y8/Q=', 'Content-Length': '0'} 2022-09-02T05:59:33.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bc5e056e9000e58-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.208 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.209 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.210 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.210 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:33.211 DEBUG:boto:Data: 2022-09-02T05:59:33.211 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.212 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.213 DEBUG:boto:Params: {} 2022-09-02T05:59:33.213 DEBUG:boto:Token: None 2022-09-02T05:59:33.214 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92 2022-09-02T05:59:33.215 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zxaSQ3yZiD11D0ztwl/X6D4y8/Q= 2022-09-02T05:59:33.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zxaSQ3yZiD11D0ztwl/X6D4y8/Q=', 'Content-Length': '0'} 2022-09-02T05:59:33.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f52ced375c63c62a-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:33.219 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.220 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.221 DEBUG:boto:Method: GET 2022-09-02T05:59:33.222 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:33.222 DEBUG:boto:Data: 2022-09-02T05:59:33.223 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.224 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.224 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.225 DEBUG:boto:Params: {} 2022-09-02T05:59:33.225 DEBUG:boto:Token: None 2022-09-02T05:59:33.226 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92 2022-09-02T05:59:33.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mrrqLmhFf3lGf1hq9RhejoLre8w= 2022-09-02T05:59:33.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mrrqLmhFf3lGf1hq9RhejoLre8w=', 'Content-Length': '0'} 2022-09-02T05:59:33.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008537e03219e2b8a4-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.231 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:33.231 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:33.232 DEBUG:boto:Method: GET 2022-09-02T05:59:33.232 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:33.233 DEBUG:boto:Data: 2022-09-02T05:59:33.234 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.235 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.236 DEBUG:boto:Params: {} 2022-09-02T05:59:33.237 DEBUG:boto:Token: None 2022-09-02T05:59:33.237 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj92 2022-09-02T05:59:33.238 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mrrqLmhFf3lGf1hq9RhejoLre8w= 2022-09-02T05:59:33.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mrrqLmhFf3lGf1hq9RhejoLre8w=', 'Content-Length': '0'} 2022-09-02T05:59:33.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b108ebfcbf08c5-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.241 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.242 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.243 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.243 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.244 DEBUG:boto:Method: GET 2022-09-02T05:59:33.244 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.245 DEBUG:boto:Data: 2022-09-02T05:59:33.245 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.246 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.247 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.247 DEBUG:boto:Params: {} 2022-09-02T05:59:33.248 DEBUG:boto:Token: None 2022-09-02T05:59:33.249 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.249 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000575a7c2c8db4cf8-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.254 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.254 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.255 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.256 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.257 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.257 DEBUG:boto:Method: GET 2022-09-02T05:59:33.258 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:33.258 DEBUG:boto:Data: 2022-09-02T05:59:33.259 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.260 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.261 DEBUG:boto:Params: {} 2022-09-02T05:59:33.261 DEBUG:boto:Token: None 2022-09-02T05:59:33.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.262 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a8ab61fc776877-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.268 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:33.270 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.270 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.271 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.272 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.272 DEBUG:boto:Method: GET 2022-09-02T05:59:33.273 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.274 DEBUG:boto:Data: 2022-09-02T05:59:33.275 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.276 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.277 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.278 DEBUG:boto:Params: {} 2022-09-02T05:59:33.279 DEBUG:boto:Token: None 2022-09-02T05:59:33.280 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.281 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068= 2022-09-02T05:59:33.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068=', 'Content-Length': '0'} 2022-09-02T05:59:33.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002076ecf3bda1bd7b-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.285 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.286 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.286 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.287 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.287 DEBUG:boto:Method: GET 2022-09-02T05:59:33.288 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.288 DEBUG:boto:Data: 2022-09-02T05:59:33.289 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.290 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.291 DEBUG:boto:Params: {} 2022-09-02T05:59:33.291 DEBUG:boto:Token: None 2022-09-02T05:59:33.292 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.292 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068= 2022-09-02T05:59:33.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068=', 'Content-Length': '0'} 2022-09-02T05:59:33.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fba257474f9aa56a-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.338 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.339 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.339 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.340 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.340 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.341 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.341 DEBUG:boto:Data: 2022-09-02T05:59:33.342 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.342 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.343 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.343 DEBUG:boto:Params: {} 2022-09-02T05:59:33.344 DEBUG:boto:Token: None 2022-09-02T05:59:33.345 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.345 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N5J0qEBKxuRGhYwzm10qTgnDYTc= 2022-09-02T05:59:33.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N5J0qEBKxuRGhYwzm10qTgnDYTc=', 'Content-Length': '0'} 2022-09-02T05:59:33.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5dcb6bf03545e89-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.349 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.350 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.350 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.351 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.351 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.352 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.352 DEBUG:boto:Data: 2022-09-02T05:59:33.353 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.354 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.354 DEBUG:boto:Params: {} 2022-09-02T05:59:33.355 DEBUG:boto:Token: None 2022-09-02T05:59:33.356 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:N5J0qEBKxuRGhYwzm10qTgnDYTc= 2022-09-02T05:59:33.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:N5J0qEBKxuRGhYwzm10qTgnDYTc=', 'Content-Length': '0'} 2022-09-02T05:59:33.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f478b0ce6d609417-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:33.361 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.361 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.362 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.362 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.363 DEBUG:boto:Method: GET 2022-09-02T05:59:33.363 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.364 DEBUG:boto:Data: 2022-09-02T05:59:33.364 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.365 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.365 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.366 DEBUG:boto:Params: {} 2022-09-02T05:59:33.367 DEBUG:boto:Token: None 2022-09-02T05:59:33.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068= 2022-09-02T05:59:33.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068=', 'Content-Length': '0'} 2022-09-02T05:59:33.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068ac7f7f071c8b52-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.371 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.372 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.373 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.373 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.374 DEBUG:boto:Method: GET 2022-09-02T05:59:33.374 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.375 DEBUG:boto:Data: 2022-09-02T05:59:33.375 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.376 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.377 DEBUG:boto:Params: {} 2022-09-02T05:59:33.378 DEBUG:boto:Token: None 2022-09-02T05:59:33.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:33.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068= 2022-09-02T05:59:33.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QhqsrcAmiOFy2u6IRprD4DxM068=', 'Content-Length': '0'} 2022-09-02T05:59:33.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53102007434a348-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.383 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.383 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.384 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.384 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.385 DEBUG:boto:Method: GET 2022-09-02T05:59:33.386 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.386 DEBUG:boto:Data: 2022-09-02T05:59:33.387 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.387 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.388 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.388 DEBUG:boto:Params: {} 2022-09-02T05:59:33.389 DEBUG:boto:Token: None 2022-09-02T05:59:33.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.390 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61639e2020e294a-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.434 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.435 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.436 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.436 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.437 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.437 DEBUG:boto:Method: GET 2022-09-02T05:59:33.438 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.438 DEBUG:boto:Data: 2022-09-02T05:59:33.439 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.440 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.440 DEBUG:boto:Params: {} 2022-09-02T05:59:33.441 DEBUG:boto:Token: None 2022-09-02T05:59:33.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006a17ead8e81af1a-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.448 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.449 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.449 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.450 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.450 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:33.451 DEBUG:boto:Data: 2022-09-02T05:59:33.451 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.452 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.452 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.453 DEBUG:boto:Params: {} 2022-09-02T05:59:33.453 DEBUG:boto:Token: None 2022-09-02T05:59:33.454 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93 2022-09-02T05:59:33.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dHaeouyGjbWEs51Y4OFFyKA4u50= 2022-09-02T05:59:33.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dHaeouyGjbWEs51Y4OFFyKA4u50=', 'Content-Length': '0'} 2022-09-02T05:59:33.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059139dc549690d8e-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.458 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.458 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.459 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.459 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:33.460 DEBUG:boto:Data: 2022-09-02T05:59:33.460 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.461 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.462 DEBUG:boto:Params: {} 2022-09-02T05:59:33.463 DEBUG:boto:Token: None 2022-09-02T05:59:33.463 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93 2022-09-02T05:59:33.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dHaeouyGjbWEs51Y4OFFyKA4u50= 2022-09-02T05:59:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dHaeouyGjbWEs51Y4OFFyKA4u50=', 'Content-Length': '0'} 2022-09-02T05:59:33.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005075c465858c0ef7-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:33.467 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.468 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.468 DEBUG:boto:Method: GET 2022-09-02T05:59:33.469 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:33.470 DEBUG:boto:Data: 2022-09-02T05:59:33.470 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.471 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.471 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.472 DEBUG:boto:Params: {} 2022-09-02T05:59:33.472 DEBUG:boto:Token: None 2022-09-02T05:59:33.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93 2022-09-02T05:59:33.473 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QFpX8HLskUpNW19xGU8RerlNpGs= 2022-09-02T05:59:33.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QFpX8HLskUpNW19xGU8RerlNpGs=', 'Content-Length': '0'} 2022-09-02T05:59:33.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006772e9403494deda-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.477 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:33.478 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:33.478 DEBUG:boto:Method: GET 2022-09-02T05:59:33.479 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:33.479 DEBUG:boto:Data: 2022-09-02T05:59:33.480 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.481 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.481 DEBUG:boto:Params: {} 2022-09-02T05:59:33.482 DEBUG:boto:Token: None 2022-09-02T05:59:33.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj93 2022-09-02T05:59:33.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QFpX8HLskUpNW19xGU8RerlNpGs= 2022-09-02T05:59:33.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QFpX8HLskUpNW19xGU8RerlNpGs=', 'Content-Length': '0'} 2022-09-02T05:59:33.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003702d80edb62dd0f-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.526 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.527 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.527 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.528 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.529 DEBUG:boto:Method: GET 2022-09-02T05:59:33.529 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.530 DEBUG:boto:Data: 2022-09-02T05:59:33.530 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.531 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.531 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.532 DEBUG:boto:Params: {} 2022-09-02T05:59:33.533 DEBUG:boto:Token: None 2022-09-02T05:59:33.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003f4ca7be85fffe7-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.538 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.539 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.540 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.540 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.541 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.541 DEBUG:boto:Method: GET 2022-09-02T05:59:33.542 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:33.542 DEBUG:boto:Data: 2022-09-02T05:59:33.543 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.544 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.544 DEBUG:boto:Params: {} 2022-09-02T05:59:33.545 DEBUG:boto:Token: None 2022-09-02T05:59:33.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054aea3f65863e636-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.552 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:33.554 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.554 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.555 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.555 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.556 DEBUG:boto:Method: GET 2022-09-02T05:59:33.556 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.557 DEBUG:boto:Data: 2022-09-02T05:59:33.557 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.558 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.559 DEBUG:boto:Params: {} 2022-09-02T05:59:33.560 DEBUG:boto:Token: None 2022-09-02T05:59:33.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8= 2022-09-02T05:59:33.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8=', 'Content-Length': '0'} 2022-09-02T05:59:33.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef7a0b3cad14381-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.564 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.565 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.565 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.566 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.567 DEBUG:boto:Method: GET 2022-09-02T05:59:33.567 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.568 DEBUG:boto:Data: 2022-09-02T05:59:33.568 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.569 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.570 DEBUG:boto:Params: {} 2022-09-02T05:59:33.570 DEBUG:boto:Token: None 2022-09-02T05:59:33.571 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8= 2022-09-02T05:59:33.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8=', 'Content-Length': '0'} 2022-09-02T05:59:33.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8279d9f611ee847-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.576 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.576 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.577 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.577 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.578 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.578 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.579 DEBUG:boto:Data: 2022-09-02T05:59:33.579 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.580 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.580 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.581 DEBUG:boto:Params: {} 2022-09-02T05:59:33.582 DEBUG:boto:Token: None 2022-09-02T05:59:33.582 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.583 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bkqQVYbhJ9bwUz+pQ+TjAD2Ksd4= 2022-09-02T05:59:33.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bkqQVYbhJ9bwUz+pQ+TjAD2Ksd4=', 'Content-Length': '0'} 2022-09-02T05:59:33.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a58187c32013359-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.586 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.587 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.588 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.588 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.589 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.589 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.590 DEBUG:boto:Data: 2022-09-02T05:59:33.590 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.591 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.592 DEBUG:boto:Params: {} 2022-09-02T05:59:33.592 DEBUG:boto:Token: None 2022-09-02T05:59:33.593 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bkqQVYbhJ9bwUz+pQ+TjAD2Ksd4= 2022-09-02T05:59:33.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bkqQVYbhJ9bwUz+pQ+TjAD2Ksd4=', 'Content-Length': '0'} 2022-09-02T05:59:33.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931efadee15703e9-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:33.598 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.599 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.599 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.600 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.600 DEBUG:boto:Method: GET 2022-09-02T05:59:33.601 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.601 DEBUG:boto:Data: 2022-09-02T05:59:33.602 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.603 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.603 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.604 DEBUG:boto:Params: {} 2022-09-02T05:59:33.604 DEBUG:boto:Token: None 2022-09-02T05:59:33.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8= 2022-09-02T05:59:33.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8=', 'Content-Length': '0'} 2022-09-02T05:59:33.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ceb43e93fe7ac5-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.609 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.610 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.610 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.611 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.611 DEBUG:boto:Method: GET 2022-09-02T05:59:33.612 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.613 DEBUG:boto:Data: 2022-09-02T05:59:33.613 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.614 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.615 DEBUG:boto:Params: {} 2022-09-02T05:59:33.615 DEBUG:boto:Token: None 2022-09-02T05:59:33.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:33.617 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8= 2022-09-02T05:59:33.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7iyrI+T3YlncjPyglCzTIk31JB8=', 'Content-Length': '0'} 2022-09-02T05:59:33.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f62e0b54b938f76-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.662 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.663 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.663 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.664 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.665 DEBUG:boto:Method: GET 2022-09-02T05:59:33.665 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.666 DEBUG:boto:Data: 2022-09-02T05:59:33.666 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.667 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.668 DEBUG:boto:Params: {} 2022-09-02T05:59:33.668 DEBUG:boto:Token: None 2022-09-02T05:59:33.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c54bc2e6cca7f3d-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.674 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.675 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.675 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.676 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.676 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.677 DEBUG:boto:Method: GET 2022-09-02T05:59:33.677 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.678 DEBUG:boto:Data: 2022-09-02T05:59:33.679 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.680 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.680 DEBUG:boto:Params: {} 2022-09-02T05:59:33.681 DEBUG:boto:Token: None 2022-09-02T05:59:33.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003787a0fff4984641-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.688 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.689 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.689 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.690 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.691 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:33.691 DEBUG:boto:Data: 2022-09-02T05:59:33.692 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.692 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.693 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.693 DEBUG:boto:Params: {} 2022-09-02T05:59:33.694 DEBUG:boto:Token: None 2022-09-02T05:59:33.695 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95 2022-09-02T05:59:33.695 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WcjoLaHmbg32hwQOWBrL84X64x4= 2022-09-02T05:59:33.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WcjoLaHmbg32hwQOWBrL84X64x4=', 'Content-Length': '0'} 2022-09-02T05:59:33.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7ff5356d79ad02-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.699 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.699 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.700 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.700 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:33.701 DEBUG:boto:Data: 2022-09-02T05:59:33.701 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.703 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.703 DEBUG:boto:Params: {} 2022-09-02T05:59:33.704 DEBUG:boto:Token: None 2022-09-02T05:59:33.704 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95 2022-09-02T05:59:33.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WcjoLaHmbg32hwQOWBrL84X64x4= 2022-09-02T05:59:33.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WcjoLaHmbg32hwQOWBrL84X64x4=', 'Content-Length': '0'} 2022-09-02T05:59:33.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1b203e415a1bb4-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:33.709 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.709 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.710 DEBUG:boto:Method: GET 2022-09-02T05:59:33.711 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:33.711 DEBUG:boto:Data: 2022-09-02T05:59:33.712 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.713 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.713 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.714 DEBUG:boto:Params: {} 2022-09-02T05:59:33.715 DEBUG:boto:Token: None 2022-09-02T05:59:33.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95 2022-09-02T05:59:33.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2aueGvPd0Xqx06X18s3khSkVaU= 2022-09-02T05:59:33.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2aueGvPd0Xqx06X18s3khSkVaU=', 'Content-Length': '0'} 2022-09-02T05:59:33.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff261f0fc2ee027a-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.719 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:33.720 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:33.721 DEBUG:boto:Method: GET 2022-09-02T05:59:33.721 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:33.722 DEBUG:boto:Data: 2022-09-02T05:59:33.722 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.723 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.724 DEBUG:boto:Params: {} 2022-09-02T05:59:33.725 DEBUG:boto:Token: None 2022-09-02T05:59:33.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj95 2022-09-02T05:59:33.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:I2aueGvPd0Xqx06X18s3khSkVaU= 2022-09-02T05:59:33.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:I2aueGvPd0Xqx06X18s3khSkVaU=', 'Content-Length': '0'} 2022-09-02T05:59:33.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bab6d2c986ba1fb-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.730 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.730 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.731 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.732 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.732 DEBUG:boto:Method: GET 2022-09-02T05:59:33.733 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.733 DEBUG:boto:Data: 2022-09-02T05:59:33.734 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.734 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:33.735 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.735 DEBUG:boto:Params: {} 2022-09-02T05:59:33.736 DEBUG:boto:Token: None 2022-09-02T05:59:33.736 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023697fe538f1ced0-0063119bc5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.782 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.783 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.784 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.784 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.785 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.786 DEBUG:boto:Method: GET 2022-09-02T05:59:33.786 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:33.787 DEBUG:boto:Data: 2022-09-02T05:59:33.787 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.788 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.789 DEBUG:boto:Params: {} 2022-09-02T05:59:33.790 DEBUG:boto:Token: None 2022-09-02T05:59:33.790 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA= 2022-09-02T05:59:33.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dms8vf0e7zjwtoZbWM01nGoiimA=', 'Content-Length': '0'} 2022-09-02T05:59:33.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ba31859d29f3e5-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.797 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.798 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone2, test-zone1} 2022-09-02T05:59:33.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:59:33.800 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.800 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.801 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.802 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:59:33.802 DEBUG:boto:Data: 2022-09-02T05:59:33.803 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:33.804 DEBUG:boto:Port: 8001 2022-09-02T05:59:33.804 DEBUG:boto:Params: {} 2022-09-02T05:59:33.805 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T05:59:33.806 DEBUG:boto:Token: None 2022-09-02T05:59:33.806 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.807 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DKslaLQKWKr/rzH3W+xs6j19h6I= 2022-09-02T05:59:33.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DKslaLQKWKr/rzH3W+xs6j19h6I=', 'Content-Length': '0'} 2022-09-02T05:59:33.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af74e8a20ad8e5fd-0063119bc5-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.812 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.812 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.813 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.814 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:59:33.814 DEBUG:boto:Data: 2022-09-02T05:59:33.815 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.816 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.816 DEBUG:boto:Params: {} 2022-09-02T05:59:33.817 DEBUG:boto:Token: None 2022-09-02T05:59:33.818 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/ 2022-09-02T05:59:33.818 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DKslaLQKWKr/rzH3W+xs6j19h6I= 2022-09-02T05:59:33.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DKslaLQKWKr/rzH3W+xs6j19h6I=', 'Content-Length': '0'} 2022-09-02T05:59:33.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092e19fcd6a00bc1b-0063119bc5-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.824 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:59:33.824 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.825 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.825 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.826 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.826 DEBUG:boto:Method: GET 2022-09-02T05:59:33.827 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.827 DEBUG:boto:Data: 2022-09-02T05:59:33.828 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:33.829 DEBUG:boto:Port: 8001 2022-09-02T05:59:33.830 DEBUG:boto:Params: {} 2022-09-02T05:59:33.830 DEBUG:boto:Token: None 2022-09-02T05:59:33.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/?versions 2022-09-02T05:59:33.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI= 2022-09-02T05:59:33.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI=', 'Content-Length': '0'} 2022-09-02T05:59:33.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ca573a952384db-0063119bc5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.836 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:59:33.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-02T05:59:33.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:59:33.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:59:33.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:59:33.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:59:33.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-02T05:59:33.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:59:33.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:59:33.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-02T05:59:33.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:59:33.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:59:33.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:59:33.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:59:33.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:59:33.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:59:33.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:59:33.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-02T05:59:33.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:59:33.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:59:33.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:59:33.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:59:33.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:59:33.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:59:33.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:59:33.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:59:33.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:59:33.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:59:33.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-02T05:59:33.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:59:33.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:59:33.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:59:33.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:59:33.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:59:33.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:59:33.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:59:33.856 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.857 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.857 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.858 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.858 DEBUG:boto:Method: GET 2022-09-02T05:59:33.859 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.859 DEBUG:boto:Data: 2022-09-02T05:59:33.860 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.861 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.861 DEBUG:boto:Params: {} 2022-09-02T05:59:33.862 DEBUG:boto:Token: None 2022-09-02T05:59:33.863 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/?versions 2022-09-02T05:59:33.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI= 2022-09-02T05:59:33.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI=', 'Content-Length': '0'} 2022-09-02T05:59:33.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff326d40127f6e0c-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.870 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:59:33.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-02T05:59:33.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:59:33.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:59:33.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:59:33.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:59:33.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-02T05:59:33.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:59:33.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:59:33.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-02T05:59:33.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:59:33.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:59:33.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:59:33.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:59:33.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:59:33.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:59:33.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:59:33.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-02T05:59:33.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:59:33.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:59:33.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:59:33.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:59:33.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:59:33.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:59:33.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:59:33.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:59:33.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:59:33.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:59:33.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-02T05:59:33.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:59:33.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:59:33.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:59:33.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:59:33.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:59:33.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:59:33.889 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.890 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.890 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.891 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.891 DEBUG:boto:Method: GET 2022-09-02T05:59:33.892 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.892 DEBUG:boto:Data: 2022-09-02T05:59:33.893 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:33.894 DEBUG:boto:Port: 8001 2022-09-02T05:59:33.895 DEBUG:boto:Params: {} 2022-09-02T05:59:33.895 DEBUG:boto:Token: None 2022-09-02T05:59:33.896 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/?versions 2022-09-02T05:59:33.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI= 2022-09-02T05:59:33.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI=', 'Content-Length': '0'} 2022-09-02T05:59:33.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069de476758cc4f41-0063119bc5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.901 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.903 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:33.904 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:33.904 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.905 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.905 DEBUG:boto:Method: GET 2022-09-02T05:59:33.906 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:33.906 DEBUG:boto:Data: 2022-09-02T05:59:33.907 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.908 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.908 DEBUG:boto:Params: {} 2022-09-02T05:59:33.909 DEBUG:boto:Token: None 2022-09-02T05:59:33.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/?versions 2022-09-02T05:59:33.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI= 2022-09-02T05:59:33.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dTECpEPrxQsQk++jKKtkAYbttuI=', 'Content-Length': '0'} 2022-09-02T05:59:33.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91f3bda7eaae7bc-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.917 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:33.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:33.919 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:33.920 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:33.921 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.921 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.922 DEBUG:boto:Method: GET 2022-09-02T05:59:33.922 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.923 DEBUG:boto:Data: 2022-09-02T05:59:33.923 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:33.924 DEBUG:boto:Port: 8001 2022-09-02T05:59:33.925 DEBUG:boto:Params: {} 2022-09-02T05:59:33.926 DEBUG:boto:Token: None 2022-09-02T05:59:33.926 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.927 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ISr0z97eDz1LvfyD/hZ6Cp/XOsg= 2022-09-02T05:59:33.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ISr0z97eDz1LvfyD/hZ6Cp/XOsg=', 'Content-Length': '0'} 2022-09-02T05:59:33.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5fab84786c936bb-0063119bc5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.931 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:33.931 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:33.932 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.932 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.933 DEBUG:boto:Method: GET 2022-09-02T05:59:33.933 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.934 DEBUG:boto:Data: 2022-09-02T05:59:33.934 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.936 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.936 DEBUG:boto:Params: {} 2022-09-02T05:59:33.937 DEBUG:boto:Token: None 2022-09-02T05:59:33.937 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ISr0z97eDz1LvfyD/hZ6Cp/XOsg= 2022-09-02T05:59:33.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ISr0z97eDz1LvfyD/hZ6Cp/XOsg=', 'Content-Length': '0'} 2022-09-02T05:59:33.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efae0c2f58a7b72-0063119bc5-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.982 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:33.983 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:33.983 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.984 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.984 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.985 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.985 DEBUG:boto:Data: 2022-09-02T05:59:33.986 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:33.987 DEBUG:boto:Port: 8001 2022-09-02T05:59:33.987 DEBUG:boto:Params: {} 2022-09-02T05:59:33.988 DEBUG:boto:Token: None 2022-09-02T05:59:33.989 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qSCHMKAYV5S1zIqttxoi6DgJeKU= 2022-09-02T05:59:33.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qSCHMKAYV5S1zIqttxoi6DgJeKU=', 'Content-Length': '0'} 2022-09-02T05:59:33.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd58c4392821854-0063119bc5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:33 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:33.993 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:33.993 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:33.994 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.994 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.995 DEBUG:boto:Method: HEAD 2022-09-02T05:59:33.995 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:33.996 DEBUG:boto:Data: 2022-09-02T05:59:33.996 DEBUG:boto:Headers: {} 2022-09-02T05:59:33.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:33.997 DEBUG:boto:Port: 8000 2022-09-02T05:59:33.998 DEBUG:boto:Params: {} 2022-09-02T05:59:33.999 DEBUG:boto:Token: None 2022-09-02T05:59:33.999 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:33 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qSCHMKAYV5S1zIqttxoi6DgJeKU= 2022-09-02T05:59:34.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:33 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qSCHMKAYV5S1zIqttxoi6DgJeKU=', 'Content-Length': '0'} 2022-09-02T05:59:34.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a33c93932f20aa7-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:34.005 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:34.006 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:34.007 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.008 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.009 DEBUG:boto:Method: GET 2022-09-02T05:59:34.010 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.011 DEBUG:boto:Data: 2022-09-02T05:59:34.012 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.014 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.015 DEBUG:boto:Params: {} 2022-09-02T05:59:34.016 DEBUG:boto:Token: None 2022-09-02T05:59:34.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QIAReFHaIHYy1/L95CdVr5VuX8M= 2022-09-02T05:59:34.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QIAReFHaIHYy1/L95CdVr5VuX8M=', 'Content-Length': '0'} 2022-09-02T05:59:34.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000069bc5609f07b7d2-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.021 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:34.022 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:34.022 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.023 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.024 DEBUG:boto:Method: GET 2022-09-02T05:59:34.024 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.025 DEBUG:boto:Data: 2022-09-02T05:59:34.025 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.027 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.027 DEBUG:boto:Params: {} 2022-09-02T05:59:34.028 DEBUG:boto:Token: None 2022-09-02T05:59:34.028 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:34.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QIAReFHaIHYy1/L95CdVr5VuX8M= 2022-09-02T05:59:34.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QIAReFHaIHYy1/L95CdVr5VuX8M=', 'Content-Length': '0'} 2022-09-02T05:59:34.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7cdcdced204f12-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.033 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.034 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.035 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.036 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.037 DEBUG:boto:Method: GET 2022-09-02T05:59:34.037 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.038 DEBUG:boto:Data: 2022-09-02T05:59:34.039 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.041 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.042 DEBUG:boto:Params: {} 2022-09-02T05:59:34.042 DEBUG:boto:Token: None 2022-09-02T05:59:34.043 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a84475fb2c2c4557-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.047 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.048 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.049 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.049 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.050 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.051 DEBUG:boto:Method: GET 2022-09-02T05:59:34.051 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.052 DEBUG:boto:Data: 2022-09-02T05:59:34.052 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.053 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.054 DEBUG:boto:Params: {} 2022-09-02T05:59:34.054 DEBUG:boto:Token: None 2022-09-02T05:59:34.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001c5b335adda716e-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.061 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.062 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:34.063 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:34.063 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.064 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:34.064 DEBUG:boto:Data: 2022-09-02T05:59:34.065 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.066 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.066 DEBUG:boto:Params: {} 2022-09-02T05:59:34.067 DEBUG:boto:Token: None 2022-09-02T05:59:34.067 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj11 2022-09-02T05:59:34.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ibZMRrWeheM52j5W8gXb0BP3vw8= 2022-09-02T05:59:34.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ibZMRrWeheM52j5W8gXb0BP3vw8=', 'Content-Length': '0'} 2022-09-02T05:59:34.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03e9f6e3db2c5e3-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.071 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:34.072 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:34.072 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.073 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:34.073 DEBUG:boto:Data: 2022-09-02T05:59:34.074 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.075 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.075 DEBUG:boto:Params: {} 2022-09-02T05:59:34.076 DEBUG:boto:Token: None 2022-09-02T05:59:34.076 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj11 2022-09-02T05:59:34.077 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ibZMRrWeheM52j5W8gXb0BP3vw8= 2022-09-02T05:59:34.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ibZMRrWeheM52j5W8gXb0BP3vw8=', 'Content-Length': '0'} 2022-09-02T05:59:34.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099111a317f5b4f5b-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:34.081 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:34.081 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:34.082 DEBUG:boto:Method: GET 2022-09-02T05:59:34.083 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:34.083 DEBUG:boto:Data: 2022-09-02T05:59:34.084 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.085 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.085 DEBUG:boto:Params: {} 2022-09-02T05:59:34.086 DEBUG:boto:Token: None 2022-09-02T05:59:34.086 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj11 2022-09-02T05:59:34.087 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4M4ovI26Y0bUpOUCGQ0xfrbCnKo= 2022-09-02T05:59:34.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4M4ovI26Y0bUpOUCGQ0xfrbCnKo=', 'Content-Length': '0'} 2022-09-02T05:59:34.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042c811e9af00f517-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.130 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:34.131 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:34.132 DEBUG:boto:Method: GET 2022-09-02T05:59:34.132 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:34.133 DEBUG:boto:Data: 2022-09-02T05:59:34.133 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.134 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.135 DEBUG:boto:Params: {} 2022-09-02T05:59:34.135 DEBUG:boto:Token: None 2022-09-02T05:59:34.136 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj11 2022-09-02T05:59:34.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4M4ovI26Y0bUpOUCGQ0xfrbCnKo= 2022-09-02T05:59:34.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4M4ovI26Y0bUpOUCGQ0xfrbCnKo=', 'Content-Length': '0'} 2022-09-02T05:59:34.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c31103fe18ae85-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.141 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.141 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.142 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.142 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.143 DEBUG:boto:Method: GET 2022-09-02T05:59:34.143 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.144 DEBUG:boto:Data: 2022-09-02T05:59:34.144 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.145 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.146 DEBUG:boto:Params: {} 2022-09-02T05:59:34.147 DEBUG:boto:Token: None 2022-09-02T05:59:34.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50bff7319330ad9-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.152 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.153 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.154 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.154 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.155 DEBUG:boto:Method: GET 2022-09-02T05:59:34.155 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:34.156 DEBUG:boto:Data: 2022-09-02T05:59:34.156 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.157 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.158 DEBUG:boto:Params: {} 2022-09-02T05:59:34.159 DEBUG:boto:Token: None 2022-09-02T05:59:34.159 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.160 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef3a0983d9a913ad-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.166 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:34.168 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.168 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.169 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.169 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.170 DEBUG:boto:Method: GET 2022-09-02T05:59:34.170 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.171 DEBUG:boto:Data: 2022-09-02T05:59:34.171 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.172 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.173 DEBUG:boto:Params: {} 2022-09-02T05:59:34.174 DEBUG:boto:Token: None 2022-09-02T05:59:34.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U= 2022-09-02T05:59:34.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U=', 'Content-Length': '0'} 2022-09-02T05:59:34.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002520f66a19a02205-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.179 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.179 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.180 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.180 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.181 DEBUG:boto:Method: GET 2022-09-02T05:59:34.182 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.182 DEBUG:boto:Data: 2022-09-02T05:59:34.183 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.184 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.184 DEBUG:boto:Params: {} 2022-09-02T05:59:34.185 DEBUG:boto:Token: None 2022-09-02T05:59:34.186 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.186 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U= 2022-09-02T05:59:34.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U=', 'Content-Length': '0'} 2022-09-02T05:59:34.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1513ab8d28b6e24-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.190 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.190 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.191 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.191 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.192 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.192 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.193 DEBUG:boto:Data: 2022-09-02T05:59:34.193 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.194 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.195 DEBUG:boto:Params: {} 2022-09-02T05:59:34.195 DEBUG:boto:Token: None 2022-09-02T05:59:34.196 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xhfm+RiNiQqmObFRpPDvy2LB4J8= 2022-09-02T05:59:34.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xhfm+RiNiQqmObFRpPDvy2LB4J8=', 'Content-Length': '0'} 2022-09-02T05:59:34.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e89fcd5f06bf36-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.199 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.200 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.200 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.201 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.201 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.202 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.202 DEBUG:boto:Data: 2022-09-02T05:59:34.203 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.204 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.204 DEBUG:boto:Params: {} 2022-09-02T05:59:34.205 DEBUG:boto:Token: None 2022-09-02T05:59:34.206 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Xhfm+RiNiQqmObFRpPDvy2LB4J8= 2022-09-02T05:59:34.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Xhfm+RiNiQqmObFRpPDvy2LB4J8=', 'Content-Length': '0'} 2022-09-02T05:59:34.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef4a4efa31edec7-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:34.210 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.210 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.211 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.211 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.212 DEBUG:boto:Method: GET 2022-09-02T05:59:34.212 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.213 DEBUG:boto:Data: 2022-09-02T05:59:34.214 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.215 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.215 DEBUG:boto:Params: {} 2022-09-02T05:59:34.216 DEBUG:boto:Token: None 2022-09-02T05:59:34.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U= 2022-09-02T05:59:34.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U=', 'Content-Length': '0'} 2022-09-02T05:59:34.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a522b5d27f2c4c5-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.262 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.263 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.264 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.264 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.265 DEBUG:boto:Method: GET 2022-09-02T05:59:34.265 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.266 DEBUG:boto:Data: 2022-09-02T05:59:34.266 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.267 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.268 DEBUG:boto:Params: {} 2022-09-02T05:59:34.268 DEBUG:boto:Token: None 2022-09-02T05:59:34.269 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:34.270 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U= 2022-09-02T05:59:34.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YaDpEjo81NYdFOFWUZvCUbyFP3U=', 'Content-Length': '0'} 2022-09-02T05:59:34.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a373aba97049ceb2-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.274 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.274 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.275 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.276 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.277 DEBUG:boto:Method: GET 2022-09-02T05:59:34.277 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.278 DEBUG:boto:Data: 2022-09-02T05:59:34.278 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.280 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.280 DEBUG:boto:Params: {} 2022-09-02T05:59:34.281 DEBUG:boto:Token: None 2022-09-02T05:59:34.282 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6cacfef1f0a30c5-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.287 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.288 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.288 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.289 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.289 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.290 DEBUG:boto:Method: GET 2022-09-02T05:59:34.290 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.291 DEBUG:boto:Data: 2022-09-02T05:59:34.292 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.293 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.293 DEBUG:boto:Params: {} 2022-09-02T05:59:34.294 DEBUG:boto:Token: None 2022-09-02T05:59:34.294 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.295 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e01a212a29cd05-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.301 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.301 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.302 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.303 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.303 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:34.304 DEBUG:boto:Data: 2022-09-02T05:59:34.304 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.305 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.306 DEBUG:boto:Params: {} 2022-09-02T05:59:34.306 DEBUG:boto:Token: None 2022-09-02T05:59:34.307 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111 2022-09-02T05:59:34.308 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wob27C4jObhPFZaPZ8MVThp89qY= 2022-09-02T05:59:34.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wob27C4jObhPFZaPZ8MVThp89qY=', 'Content-Length': '0'} 2022-09-02T05:59:34.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391818bed5a1a212-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.311 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.312 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.312 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.313 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:34.313 DEBUG:boto:Data: 2022-09-02T05:59:34.314 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.315 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.316 DEBUG:boto:Params: {} 2022-09-02T05:59:34.316 DEBUG:boto:Token: None 2022-09-02T05:59:34.317 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111 2022-09-02T05:59:34.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Wob27C4jObhPFZaPZ8MVThp89qY= 2022-09-02T05:59:34.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Wob27C4jObhPFZaPZ8MVThp89qY=', 'Content-Length': '0'} 2022-09-02T05:59:34.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065abd38450e960af-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:34.321 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.322 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.322 DEBUG:boto:Method: GET 2022-09-02T05:59:34.323 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:34.324 DEBUG:boto:Data: 2022-09-02T05:59:34.324 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.325 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.326 DEBUG:boto:Params: {} 2022-09-02T05:59:34.326 DEBUG:boto:Token: None 2022-09-02T05:59:34.327 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111 2022-09-02T05:59:34.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s7vEv0Ab30PsoNAQ8IkyqeoP60o= 2022-09-02T05:59:34.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s7vEv0Ab30PsoNAQ8IkyqeoP60o=', 'Content-Length': '0'} 2022-09-02T05:59:34.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0689a7345492626-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.374 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:34.374 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:34.375 DEBUG:boto:Method: GET 2022-09-02T05:59:34.376 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:34.376 DEBUG:boto:Data: 2022-09-02T05:59:34.377 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.378 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.378 DEBUG:boto:Params: {} 2022-09-02T05:59:34.379 DEBUG:boto:Token: None 2022-09-02T05:59:34.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj111 2022-09-02T05:59:34.380 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:s7vEv0Ab30PsoNAQ8IkyqeoP60o= 2022-09-02T05:59:34.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:s7vEv0Ab30PsoNAQ8IkyqeoP60o=', 'Content-Length': '0'} 2022-09-02T05:59:34.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c92dddc8a2a4da-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.384 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.384 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.385 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.385 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.386 DEBUG:boto:Method: GET 2022-09-02T05:59:34.387 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.387 DEBUG:boto:Data: 2022-09-02T05:59:34.388 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.389 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.389 DEBUG:boto:Params: {} 2022-09-02T05:59:34.390 DEBUG:boto:Token: None 2022-09-02T05:59:34.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5998ae750b3389-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.395 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.396 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.396 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.397 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.397 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.398 DEBUG:boto:Method: GET 2022-09-02T05:59:34.399 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:34.399 DEBUG:boto:Data: 2022-09-02T05:59:34.400 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.401 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.401 DEBUG:boto:Params: {} 2022-09-02T05:59:34.402 DEBUG:boto:Token: None 2022-09-02T05:59:34.403 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063e36a233f0e45d1-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.409 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:34.411 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.411 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.412 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.412 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.413 DEBUG:boto:Method: GET 2022-09-02T05:59:34.413 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.414 DEBUG:boto:Data: 2022-09-02T05:59:34.414 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.416 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.416 DEBUG:boto:Params: {} 2022-09-02T05:59:34.417 DEBUG:boto:Token: None 2022-09-02T05:59:34.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU= 2022-09-02T05:59:34.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU=', 'Content-Length': '0'} 2022-09-02T05:59:34.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000090107eca15a9abf-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.421 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.422 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.422 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.423 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.424 DEBUG:boto:Method: GET 2022-09-02T05:59:34.424 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.425 DEBUG:boto:Data: 2022-09-02T05:59:34.425 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.426 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.427 DEBUG:boto:Params: {} 2022-09-02T05:59:34.428 DEBUG:boto:Token: None 2022-09-02T05:59:34.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU= 2022-09-02T05:59:34.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU=', 'Content-Length': '0'} 2022-09-02T05:59:34.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000604dc90c245eec7e-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.474 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.475 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.475 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.476 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.476 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.477 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.477 DEBUG:boto:Data: 2022-09-02T05:59:34.478 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.479 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.479 DEBUG:boto:Params: {} 2022-09-02T05:59:34.480 DEBUG:boto:Token: None 2022-09-02T05:59:34.481 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4cCRYbYX0KvcM08N9iFlktOVF6s= 2022-09-02T05:59:34.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4cCRYbYX0KvcM08N9iFlktOVF6s=', 'Content-Length': '0'} 2022-09-02T05:59:34.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003183b2f4b4a58e42-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.485 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.485 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.486 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.486 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.487 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.487 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.488 DEBUG:boto:Data: 2022-09-02T05:59:34.489 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.490 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.490 DEBUG:boto:Params: {} 2022-09-02T05:59:34.491 DEBUG:boto:Token: None 2022-09-02T05:59:34.491 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4cCRYbYX0KvcM08N9iFlktOVF6s= 2022-09-02T05:59:34.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4cCRYbYX0KvcM08N9iFlktOVF6s=', 'Content-Length': '0'} 2022-09-02T05:59:34.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b72a9443bc36984-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:34.496 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.496 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.497 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.497 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.498 DEBUG:boto:Method: GET 2022-09-02T05:59:34.498 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.499 DEBUG:boto:Data: 2022-09-02T05:59:34.500 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.501 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.501 DEBUG:boto:Params: {} 2022-09-02T05:59:34.502 DEBUG:boto:Token: None 2022-09-02T05:59:34.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU= 2022-09-02T05:59:34.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU=', 'Content-Length': '0'} 2022-09-02T05:59:34.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d94ef29a3836d0-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.507 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.507 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.508 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.508 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.509 DEBUG:boto:Method: GET 2022-09-02T05:59:34.509 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.510 DEBUG:boto:Data: 2022-09-02T05:59:34.510 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.512 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.512 DEBUG:boto:Params: {} 2022-09-02T05:59:34.513 DEBUG:boto:Token: None 2022-09-02T05:59:34.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:34.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU= 2022-09-02T05:59:34.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WQdRSQPNwmSuit9RLsqeQ+5fpGU=', 'Content-Length': '0'} 2022-09-02T05:59:34.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091eb96083be2192d-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.518 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.519 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.519 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.520 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.520 DEBUG:boto:Method: GET 2022-09-02T05:59:34.521 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.521 DEBUG:boto:Data: 2022-09-02T05:59:34.522 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.523 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.524 DEBUG:boto:Params: {} 2022-09-02T05:59:34.524 DEBUG:boto:Token: None 2022-09-02T05:59:34.525 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070c655c7173b5c71-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.574 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.575 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.576 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.576 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.577 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.577 DEBUG:boto:Method: GET 2022-09-02T05:59:34.578 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.578 DEBUG:boto:Data: 2022-09-02T05:59:34.579 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.580 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.581 DEBUG:boto:Params: {} 2022-09-02T05:59:34.581 DEBUG:boto:Token: None 2022-09-02T05:59:34.582 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.582 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c135901dc23e17b2-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.589 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.589 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.590 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.591 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.591 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:34.592 DEBUG:boto:Data: 2022-09-02T05:59:34.592 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.593 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.594 DEBUG:boto:Params: {} 2022-09-02T05:59:34.595 DEBUG:boto:Token: None 2022-09-02T05:59:34.595 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115 2022-09-02T05:59:34.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kQfuruKDWnIw9ivskl5D0Mplo9g= 2022-09-02T05:59:34.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kQfuruKDWnIw9ivskl5D0Mplo9g=', 'Content-Length': '0'} 2022-09-02T05:59:34.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f9642922581aee7-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.600 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.600 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.601 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.601 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:34.602 DEBUG:boto:Data: 2022-09-02T05:59:34.602 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.603 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.604 DEBUG:boto:Params: {} 2022-09-02T05:59:34.604 DEBUG:boto:Token: None 2022-09-02T05:59:34.605 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115 2022-09-02T05:59:34.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kQfuruKDWnIw9ivskl5D0Mplo9g= 2022-09-02T05:59:34.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kQfuruKDWnIw9ivskl5D0Mplo9g=', 'Content-Length': '0'} 2022-09-02T05:59:34.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697d285b89d63165-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:34.611 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.611 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.612 DEBUG:boto:Method: GET 2022-09-02T05:59:34.613 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:34.613 DEBUG:boto:Data: 2022-09-02T05:59:34.614 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.616 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.616 DEBUG:boto:Params: {} 2022-09-02T05:59:34.617 DEBUG:boto:Token: None 2022-09-02T05:59:34.617 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115 2022-09-02T05:59:34.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7juVt1EEdCsRUWsMQP9BXNRNVxY= 2022-09-02T05:59:34.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7juVt1EEdCsRUWsMQP9BXNRNVxY=', 'Content-Length': '0'} 2022-09-02T05:59:34.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e89370e02d650172-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.622 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:34.623 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:34.623 DEBUG:boto:Method: GET 2022-09-02T05:59:34.624 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:34.624 DEBUG:boto:Data: 2022-09-02T05:59:34.625 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.626 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.627 DEBUG:boto:Params: {} 2022-09-02T05:59:34.627 DEBUG:boto:Token: None 2022-09-02T05:59:34.628 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj115 2022-09-02T05:59:34.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7juVt1EEdCsRUWsMQP9BXNRNVxY= 2022-09-02T05:59:34.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7juVt1EEdCsRUWsMQP9BXNRNVxY=', 'Content-Length': '0'} 2022-09-02T05:59:34.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441b63c9b3fc4133-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.678 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.679 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.680 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.681 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.688 DEBUG:boto:Method: GET 2022-09-02T05:59:34.690 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.691 DEBUG:boto:Data: 2022-09-02T05:59:34.691 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.693 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.694 DEBUG:boto:Params: {} 2022-09-02T05:59:34.696 DEBUG:boto:Token: None 2022-09-02T05:59:34.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5a5be28523a78b8-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.703 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.705 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.706 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.707 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.708 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.709 DEBUG:boto:Method: GET 2022-09-02T05:59:34.710 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:34.711 DEBUG:boto:Data: 2022-09-02T05:59:34.712 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.714 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.715 DEBUG:boto:Params: {} 2022-09-02T05:59:34.716 DEBUG:boto:Token: None 2022-09-02T05:59:34.717 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000538c00064a7cb1ac-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.725 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:34.727 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.728 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.728 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.729 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.729 DEBUG:boto:Method: GET 2022-09-02T05:59:34.730 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.731 DEBUG:boto:Data: 2022-09-02T05:59:34.731 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.732 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.733 DEBUG:boto:Params: {} 2022-09-02T05:59:34.733 DEBUG:boto:Token: None 2022-09-02T05:59:34.734 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs= 2022-09-02T05:59:34.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs=', 'Content-Length': '0'} 2022-09-02T05:59:34.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8f2411e4d5be727-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.782 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.783 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.783 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.784 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.784 DEBUG:boto:Method: GET 2022-09-02T05:59:34.785 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.786 DEBUG:boto:Data: 2022-09-02T05:59:34.786 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.787 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.788 DEBUG:boto:Params: {} 2022-09-02T05:59:34.788 DEBUG:boto:Token: None 2022-09-02T05:59:34.789 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.790 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs= 2022-09-02T05:59:34.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs=', 'Content-Length': '0'} 2022-09-02T05:59:34.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008237520272f70baf-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.794 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.795 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.795 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.796 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.797 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.798 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.798 DEBUG:boto:Data: 2022-09-02T05:59:34.799 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.800 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.800 DEBUG:boto:Params: {} 2022-09-02T05:59:34.801 DEBUG:boto:Token: None 2022-09-02T05:59:34.801 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mJy5lZDrwda6/BpUx/j+XxPAGPs= 2022-09-02T05:59:34.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mJy5lZDrwda6/BpUx/j+XxPAGPs=', 'Content-Length': '0'} 2022-09-02T05:59:34.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01543da27e0dc68-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.806 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.806 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.807 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.808 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.808 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.809 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.809 DEBUG:boto:Data: 2022-09-02T05:59:34.810 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.811 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.811 DEBUG:boto:Params: {} 2022-09-02T05:59:34.812 DEBUG:boto:Token: None 2022-09-02T05:59:34.812 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mJy5lZDrwda6/BpUx/j+XxPAGPs= 2022-09-02T05:59:34.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mJy5lZDrwda6/BpUx/j+XxPAGPs=', 'Content-Length': '0'} 2022-09-02T05:59:34.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ee6f636fec63f8-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:34.817 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.818 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.819 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.819 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.820 DEBUG:boto:Method: GET 2022-09-02T05:59:34.820 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.821 DEBUG:boto:Data: 2022-09-02T05:59:34.821 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.822 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.823 DEBUG:boto:Params: {} 2022-09-02T05:59:34.824 DEBUG:boto:Token: None 2022-09-02T05:59:34.824 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.825 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs= 2022-09-02T05:59:34.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs=', 'Content-Length': '0'} 2022-09-02T05:59:34.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a66d443c858366e3-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.828 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.829 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.830 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.830 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.831 DEBUG:boto:Method: GET 2022-09-02T05:59:34.831 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.832 DEBUG:boto:Data: 2022-09-02T05:59:34.832 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.833 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.834 DEBUG:boto:Params: {} 2022-09-02T05:59:34.835 DEBUG:boto:Token: None 2022-09-02T05:59:34.835 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:34.836 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs= 2022-09-02T05:59:34.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zeiSSFQyvv+9apxaZJyivzfvgUs=', 'Content-Length': '0'} 2022-09-02T05:59:34.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ebfa58e951e79f0-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.840 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.840 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.841 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.841 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.842 DEBUG:boto:Method: GET 2022-09-02T05:59:34.842 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.843 DEBUG:boto:Data: 2022-09-02T05:59:34.843 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.844 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.845 DEBUG:boto:Params: {} 2022-09-02T05:59:34.845 DEBUG:boto:Token: None 2022-09-02T05:59:34.846 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.847 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000520f31a5fecda6c0-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.851 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.852 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.853 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.853 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.854 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.854 DEBUG:boto:Method: GET 2022-09-02T05:59:34.855 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.855 DEBUG:boto:Data: 2022-09-02T05:59:34.856 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.857 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.858 DEBUG:boto:Params: {} 2022-09-02T05:59:34.858 DEBUG:boto:Token: None 2022-09-02T05:59:34.859 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.859 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08751ab25ebd4df-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.865 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.866 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.867 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.867 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.868 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:34.868 DEBUG:boto:Data: 2022-09-02T05:59:34.869 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.870 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.870 DEBUG:boto:Params: {} 2022-09-02T05:59:34.871 DEBUG:boto:Token: None 2022-09-02T05:59:34.871 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116 2022-09-02T05:59:34.872 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/BLFo0/wY9+Pqd4czMBWVXUkK2Q= 2022-09-02T05:59:34.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/BLFo0/wY9+Pqd4czMBWVXUkK2Q=', 'Content-Length': '0'} 2022-09-02T05:59:34.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7dc191566a3717-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.877 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.877 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.878 DEBUG:boto:Method: HEAD 2022-09-02T05:59:34.879 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:34.880 DEBUG:boto:Data: 2022-09-02T05:59:34.881 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.882 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.882 DEBUG:boto:Params: {} 2022-09-02T05:59:34.883 DEBUG:boto:Token: None 2022-09-02T05:59:34.884 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116 2022-09-02T05:59:34.884 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/BLFo0/wY9+Pqd4czMBWVXUkK2Q= 2022-09-02T05:59:34.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/BLFo0/wY9+Pqd4czMBWVXUkK2Q=', 'Content-Length': '0'} 2022-09-02T05:59:34.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e0f94c4a9b9032-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:34.888 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.889 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.889 DEBUG:boto:Method: GET 2022-09-02T05:59:34.890 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:34.891 DEBUG:boto:Data: 2022-09-02T05:59:34.891 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.892 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.893 DEBUG:boto:Params: {} 2022-09-02T05:59:34.894 DEBUG:boto:Token: None 2022-09-02T05:59:34.894 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116 2022-09-02T05:59:34.895 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DHwqe42PZ+pCI36g3jKkpA0MI9o= 2022-09-02T05:59:34.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DHwqe42PZ+pCI36g3jKkpA0MI9o=', 'Content-Length': '0'} 2022-09-02T05:59:34.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008566e51db6fea767-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.942 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:34.943 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:34.944 DEBUG:boto:Method: GET 2022-09-02T05:59:34.944 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:34.945 DEBUG:boto:Data: 2022-09-02T05:59:34.946 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.947 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.948 DEBUG:boto:Params: {} 2022-09-02T05:59:34.950 DEBUG:boto:Token: None 2022-09-02T05:59:34.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/obj116 2022-09-02T05:59:34.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DHwqe42PZ+pCI36g3jKkpA0MI9o= 2022-09-02T05:59:34.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DHwqe42PZ+pCI36g3jKkpA0MI9o=', 'Content-Length': '0'} 2022-09-02T05:59:34.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d4927cb26846c14-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.956 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.957 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.957 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.958 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.959 DEBUG:boto:Method: GET 2022-09-02T05:59:34.959 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.960 DEBUG:boto:Data: 2022-09-02T05:59:34.960 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:34.961 DEBUG:boto:Port: 8001 2022-09-02T05:59:34.962 DEBUG:boto:Params: {} 2022-09-02T05:59:34.963 DEBUG:boto:Token: None 2022-09-02T05:59:34.964 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.965 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076486da1c10b5866-0063119bc6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:34.970 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:34.970 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:34.971 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:34.972 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.972 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.973 DEBUG:boto:Method: GET 2022-09-02T05:59:34.973 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:34.974 DEBUG:boto:Data: 2022-09-02T05:59:34.975 DEBUG:boto:Headers: {} 2022-09-02T05:59:34.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:34.976 DEBUG:boto:Port: 8000 2022-09-02T05:59:34.977 DEBUG:boto:Params: {} 2022-09-02T05:59:34.977 DEBUG:boto:Token: None 2022-09-02T05:59:34.978 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:34 GMT /xljbkf-74/ 2022-09-02T05:59:34.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8= 2022-09-02T05:59:34.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:34 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E7Be+X/u4nyLPV5zJ+xt7/nCDz8=', 'Content-Length': '0'} 2022-09-02T05:59:34.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6ba0f8367d14ad-0063119bc6-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:34 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.030 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:35.032 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.032 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.033 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.033 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.034 DEBUG:boto:Method: GET 2022-09-02T05:59:35.035 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.035 DEBUG:boto:Data: 2022-09-02T05:59:35.036 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.037 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.038 DEBUG:boto:Params: {} 2022-09-02T05:59:35.038 DEBUG:boto:Token: None 2022-09-02T05:59:35.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0= 2022-09-02T05:59:35.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0=', 'Content-Length': '0'} 2022-09-02T05:59:35.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8790717a1a0d3e2-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.044 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.045 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.046 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.046 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.047 DEBUG:boto:Method: GET 2022-09-02T05:59:35.048 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.048 DEBUG:boto:Data: 2022-09-02T05:59:35.049 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.050 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.051 DEBUG:boto:Params: {} 2022-09-02T05:59:35.052 DEBUG:boto:Token: None 2022-09-02T05:59:35.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.054 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0= 2022-09-02T05:59:35.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0=', 'Content-Length': '0'} 2022-09-02T05:59:35.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b51148cc2f6308e-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.058 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.059 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.059 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.060 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.061 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.061 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.062 DEBUG:boto:Data: 2022-09-02T05:59:35.063 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.065 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.065 DEBUG:boto:Params: {} 2022-09-02T05:59:35.066 DEBUG:boto:Token: None 2022-09-02T05:59:35.067 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+MlT0iKGzBKeKpPxoE3n/Jo+CVc= 2022-09-02T05:59:35.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+MlT0iKGzBKeKpPxoE3n/Jo+CVc=', 'Content-Length': '0'} 2022-09-02T05:59:35.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720af047901fc863-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.072 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.073 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.073 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.074 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.075 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.076 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.076 DEBUG:boto:Data: 2022-09-02T05:59:35.077 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.078 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.079 DEBUG:boto:Params: {} 2022-09-02T05:59:35.080 DEBUG:boto:Token: None 2022-09-02T05:59:35.081 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.082 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+MlT0iKGzBKeKpPxoE3n/Jo+CVc= 2022-09-02T05:59:35.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+MlT0iKGzBKeKpPxoE3n/Jo+CVc=', 'Content-Length': '0'} 2022-09-02T05:59:35.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005949e22c9e19ca31-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:35.088 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.088 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.089 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.089 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.090 DEBUG:boto:Method: GET 2022-09-02T05:59:35.091 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.091 DEBUG:boto:Data: 2022-09-02T05:59:35.092 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.093 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.094 DEBUG:boto:Params: {} 2022-09-02T05:59:35.095 DEBUG:boto:Token: None 2022-09-02T05:59:35.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.096 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0= 2022-09-02T05:59:35.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0=', 'Content-Length': '0'} 2022-09-02T05:59:35.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cd187960d2bbd3-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.142 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.143 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.145 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.145 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.146 DEBUG:boto:Method: GET 2022-09-02T05:59:35.147 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.149 DEBUG:boto:Data: 2022-09-02T05:59:35.150 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.152 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.153 DEBUG:boto:Params: {} 2022-09-02T05:59:35.155 DEBUG:boto:Token: None 2022-09-02T05:59:35.156 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:35.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0= 2022-09-02T05:59:35.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:71vvV7UsoGp7X/FNdrVNPfePVi0=', 'Content-Length': '0'} 2022-09-02T05:59:35.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1a9d309222462a-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.162 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.163 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.163 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.164 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.164 DEBUG:boto:Method: GET 2022-09-02T05:59:35.165 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.165 DEBUG:boto:Data: 2022-09-02T05:59:35.166 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.167 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.168 DEBUG:boto:Params: {} 2022-09-02T05:59:35.168 DEBUG:boto:Token: None 2022-09-02T05:59:35.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.169 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e13bb97aab51d91-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.174 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.175 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.175 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.176 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.176 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.177 DEBUG:boto:Method: GET 2022-09-02T05:59:35.177 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.178 DEBUG:boto:Data: 2022-09-02T05:59:35.179 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.180 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.180 DEBUG:boto:Params: {} 2022-09-02T05:59:35.181 DEBUG:boto:Token: None 2022-09-02T05:59:35.181 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b17a7f9c7cb8bbf9-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.188 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.189 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.190 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.190 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.191 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:35.191 DEBUG:boto:Data: 2022-09-02T05:59:35.192 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.193 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.193 DEBUG:boto:Params: {} 2022-09-02T05:59:35.194 DEBUG:boto:Token: None 2022-09-02T05:59:35.195 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124 2022-09-02T05:59:35.195 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fRTuhNGcpHOE9kG84h5PKmjIlag= 2022-09-02T05:59:35.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fRTuhNGcpHOE9kG84h5PKmjIlag=', 'Content-Length': '0'} 2022-09-02T05:59:35.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a973ece74dd264c-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.200 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.200 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.201 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.202 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:35.202 DEBUG:boto:Data: 2022-09-02T05:59:35.203 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.205 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.205 DEBUG:boto:Params: {} 2022-09-02T05:59:35.206 DEBUG:boto:Token: None 2022-09-02T05:59:35.207 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124 2022-09-02T05:59:35.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fRTuhNGcpHOE9kG84h5PKmjIlag= 2022-09-02T05:59:35.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fRTuhNGcpHOE9kG84h5PKmjIlag=', 'Content-Length': '0'} 2022-09-02T05:59:35.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300ac26fe49c38f3-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:35.212 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.213 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.213 DEBUG:boto:Method: GET 2022-09-02T05:59:35.214 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:35.215 DEBUG:boto:Data: 2022-09-02T05:59:35.215 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.216 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.217 DEBUG:boto:Params: {} 2022-09-02T05:59:35.218 DEBUG:boto:Token: None 2022-09-02T05:59:35.218 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124 2022-09-02T05:59:35.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ceJnErl0nR5tTLg+3NZN2Uj1Sn0= 2022-09-02T05:59:35.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ceJnErl0nR5tTLg+3NZN2Uj1Sn0=', 'Content-Length': '0'} 2022-09-02T05:59:35.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725a6585d016a846-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.223 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:35.223 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:35.224 DEBUG:boto:Method: GET 2022-09-02T05:59:35.224 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:35.225 DEBUG:boto:Data: 2022-09-02T05:59:35.225 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.226 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.227 DEBUG:boto:Params: {} 2022-09-02T05:59:35.228 DEBUG:boto:Token: None 2022-09-02T05:59:35.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj124 2022-09-02T05:59:35.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ceJnErl0nR5tTLg+3NZN2Uj1Sn0= 2022-09-02T05:59:35.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ceJnErl0nR5tTLg+3NZN2Uj1Sn0=', 'Content-Length': '0'} 2022-09-02T05:59:35.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8c6914ce5dd8b5-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.274 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.275 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.275 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.276 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.277 DEBUG:boto:Method: GET 2022-09-02T05:59:35.277 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.278 DEBUG:boto:Data: 2022-09-02T05:59:35.279 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.280 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.280 DEBUG:boto:Params: {} 2022-09-02T05:59:35.281 DEBUG:boto:Token: None 2022-09-02T05:59:35.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db6b516057ad58f3-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.286 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.287 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.288 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.289 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.290 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.290 DEBUG:boto:Method: GET 2022-09-02T05:59:35.291 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:35.292 DEBUG:boto:Data: 2022-09-02T05:59:35.292 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.293 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.294 DEBUG:boto:Params: {} 2022-09-02T05:59:35.295 DEBUG:boto:Token: None 2022-09-02T05:59:35.295 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.296 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d90ce4e6d6dfcee6-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.302 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:35.304 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.305 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.306 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.307 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.308 DEBUG:boto:Method: GET 2022-09-02T05:59:35.309 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.310 DEBUG:boto:Data: 2022-09-02T05:59:35.311 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.313 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.314 DEBUG:boto:Params: {} 2022-09-02T05:59:35.315 DEBUG:boto:Token: None 2022-09-02T05:59:35.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0= 2022-09-02T05:59:35.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0=', 'Content-Length': '0'} 2022-09-02T05:59:35.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cefcd0d833f64b3d-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.319 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.320 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.321 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.321 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.322 DEBUG:boto:Method: GET 2022-09-02T05:59:35.323 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.323 DEBUG:boto:Data: 2022-09-02T05:59:35.324 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.326 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.327 DEBUG:boto:Params: {} 2022-09-02T05:59:35.328 DEBUG:boto:Token: None 2022-09-02T05:59:35.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.330 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0= 2022-09-02T05:59:35.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0=', 'Content-Length': '0'} 2022-09-02T05:59:35.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e7418943661883-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.335 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.336 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.337 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.338 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.338 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.339 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.339 DEBUG:boto:Data: 2022-09-02T05:59:35.340 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.341 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.341 DEBUG:boto:Params: {} 2022-09-02T05:59:35.342 DEBUG:boto:Token: None 2022-09-02T05:59:35.343 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LN7HiLLL6xkZhPH4tcyZkPwWwxE= 2022-09-02T05:59:35.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LN7HiLLL6xkZhPH4tcyZkPwWwxE=', 'Content-Length': '0'} 2022-09-02T05:59:35.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000822a4fc1c70d637d-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.347 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.347 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.348 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.349 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.349 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.350 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.350 DEBUG:boto:Data: 2022-09-02T05:59:35.351 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.352 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.352 DEBUG:boto:Params: {} 2022-09-02T05:59:35.353 DEBUG:boto:Token: None 2022-09-02T05:59:35.353 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LN7HiLLL6xkZhPH4tcyZkPwWwxE= 2022-09-02T05:59:35.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LN7HiLLL6xkZhPH4tcyZkPwWwxE=', 'Content-Length': '0'} 2022-09-02T05:59:35.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e3b2b3c04482b5-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:35.358 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.359 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.359 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.360 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.360 DEBUG:boto:Method: GET 2022-09-02T05:59:35.361 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.361 DEBUG:boto:Data: 2022-09-02T05:59:35.362 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.363 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.364 DEBUG:boto:Params: {} 2022-09-02T05:59:35.365 DEBUG:boto:Token: None 2022-09-02T05:59:35.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0= 2022-09-02T05:59:35.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0=', 'Content-Length': '0'} 2022-09-02T05:59:35.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9ffc68586c369c-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.410 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.411 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.412 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.412 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.413 DEBUG:boto:Method: GET 2022-09-02T05:59:35.413 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.414 DEBUG:boto:Data: 2022-09-02T05:59:35.414 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.415 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.416 DEBUG:boto:Params: {} 2022-09-02T05:59:35.417 DEBUG:boto:Token: None 2022-09-02T05:59:35.417 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:35.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0= 2022-09-02T05:59:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:U7wsezulw+bS/qrgh6mI0dG8IP0=', 'Content-Length': '0'} 2022-09-02T05:59:35.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e74b7d247cfa44d-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.422 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.422 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.423 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.423 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.424 DEBUG:boto:Method: GET 2022-09-02T05:59:35.425 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.425 DEBUG:boto:Data: 2022-09-02T05:59:35.426 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.427 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.427 DEBUG:boto:Params: {} 2022-09-02T05:59:35.428 DEBUG:boto:Token: None 2022-09-02T05:59:35.429 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3cc24364f8ccecf-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.433 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.435 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.436 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.436 DEBUG:boto:Method: GET 2022-09-02T05:59:35.437 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.438 DEBUG:boto:Data: 2022-09-02T05:59:35.438 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.439 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.440 DEBUG:boto:Params: {} 2022-09-02T05:59:35.441 DEBUG:boto:Token: None 2022-09-02T05:59:35.441 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.442 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a252a4733613c0c-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.448 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.449 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.450 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.450 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.451 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:35.451 DEBUG:boto:Data: 2022-09-02T05:59:35.452 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.453 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.453 DEBUG:boto:Params: {} 2022-09-02T05:59:35.454 DEBUG:boto:Token: None 2022-09-02T05:59:35.455 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145 2022-09-02T05:59:35.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yMF6PW99ffgCVScRoIvalOA3Vu4= 2022-09-02T05:59:35.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yMF6PW99ffgCVScRoIvalOA3Vu4=', 'Content-Length': '0'} 2022-09-02T05:59:35.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8aaaf27392e6f8-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.459 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.459 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.460 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.460 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:35.461 DEBUG:boto:Data: 2022-09-02T05:59:35.461 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.462 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.463 DEBUG:boto:Params: {} 2022-09-02T05:59:35.464 DEBUG:boto:Token: None 2022-09-02T05:59:35.464 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145 2022-09-02T05:59:35.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yMF6PW99ffgCVScRoIvalOA3Vu4= 2022-09-02T05:59:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yMF6PW99ffgCVScRoIvalOA3Vu4=', 'Content-Length': '0'} 2022-09-02T05:59:35.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000056fe0b95255e7b3-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:35.470 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.471 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.472 DEBUG:boto:Method: GET 2022-09-02T05:59:35.473 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:35.474 DEBUG:boto:Data: 2022-09-02T05:59:35.475 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.477 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.478 DEBUG:boto:Params: {} 2022-09-02T05:59:35.479 DEBUG:boto:Token: None 2022-09-02T05:59:35.480 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145 2022-09-02T05:59:35.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v+WxJnhwqCytQx72TBZ4sFc+W6k= 2022-09-02T05:59:35.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v+WxJnhwqCytQx72TBZ4sFc+W6k=', 'Content-Length': '0'} 2022-09-02T05:59:35.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2f4e061099fd44-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.486 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:35.487 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:35.488 DEBUG:boto:Method: GET 2022-09-02T05:59:35.489 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:35.490 DEBUG:boto:Data: 2022-09-02T05:59:35.491 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.493 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.494 DEBUG:boto:Params: {} 2022-09-02T05:59:35.495 DEBUG:boto:Token: None 2022-09-02T05:59:35.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj145 2022-09-02T05:59:35.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v+WxJnhwqCytQx72TBZ4sFc+W6k= 2022-09-02T05:59:35.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v+WxJnhwqCytQx72TBZ4sFc+W6k=', 'Content-Length': '0'} 2022-09-02T05:59:35.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce0905722c68d01-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.502 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.502 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.503 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.504 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.504 DEBUG:boto:Method: GET 2022-09-02T05:59:35.505 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.505 DEBUG:boto:Data: 2022-09-02T05:59:35.506 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.507 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.508 DEBUG:boto:Params: {} 2022-09-02T05:59:35.508 DEBUG:boto:Token: None 2022-09-02T05:59:35.509 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b550c1c3bc8215d4-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.515 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.516 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.516 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.517 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.517 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.518 DEBUG:boto:Method: GET 2022-09-02T05:59:35.519 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:35.519 DEBUG:boto:Data: 2022-09-02T05:59:35.520 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.521 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.521 DEBUG:boto:Params: {} 2022-09-02T05:59:35.522 DEBUG:boto:Token: None 2022-09-02T05:59:35.523 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b57fb34a4fabba65-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.529 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:35.531 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.531 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.532 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.533 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.533 DEBUG:boto:Method: GET 2022-09-02T05:59:35.534 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.534 DEBUG:boto:Data: 2022-09-02T05:59:35.535 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.536 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.536 DEBUG:boto:Params: {} 2022-09-02T05:59:35.537 DEBUG:boto:Token: None 2022-09-02T05:59:35.538 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg= 2022-09-02T05:59:35.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg=', 'Content-Length': '0'} 2022-09-02T05:59:35.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed872c06d4a70e7-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.542 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.543 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.544 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.544 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.545 DEBUG:boto:Method: GET 2022-09-02T05:59:35.545 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.546 DEBUG:boto:Data: 2022-09-02T05:59:35.547 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.548 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.549 DEBUG:boto:Params: {} 2022-09-02T05:59:35.549 DEBUG:boto:Token: None 2022-09-02T05:59:35.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg= 2022-09-02T05:59:35.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg=', 'Content-Length': '0'} 2022-09-02T05:59:35.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f75400c27aaf8f8d-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.557 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.558 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.558 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.559 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.559 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.560 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.561 DEBUG:boto:Data: 2022-09-02T05:59:35.562 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.564 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.564 DEBUG:boto:Params: {} 2022-09-02T05:59:35.566 DEBUG:boto:Token: None 2022-09-02T05:59:35.566 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DP4pFUwvpn/fGFx8E0OLBcccYxY= 2022-09-02T05:59:35.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DP4pFUwvpn/fGFx8E0OLBcccYxY=', 'Content-Length': '0'} 2022-09-02T05:59:35.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d8e705caeacd452-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.571 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.571 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.572 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.573 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.573 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.574 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.574 DEBUG:boto:Data: 2022-09-02T05:59:35.575 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.576 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.576 DEBUG:boto:Params: {} 2022-09-02T05:59:35.577 DEBUG:boto:Token: None 2022-09-02T05:59:35.577 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.578 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DP4pFUwvpn/fGFx8E0OLBcccYxY= 2022-09-02T05:59:35.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DP4pFUwvpn/fGFx8E0OLBcccYxY=', 'Content-Length': '0'} 2022-09-02T05:59:35.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b38359981ad37342-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:35.582 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.583 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.583 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.584 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.584 DEBUG:boto:Method: GET 2022-09-02T05:59:35.585 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.585 DEBUG:boto:Data: 2022-09-02T05:59:35.586 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.587 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.587 DEBUG:boto:Params: {} 2022-09-02T05:59:35.588 DEBUG:boto:Token: None 2022-09-02T05:59:35.589 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg= 2022-09-02T05:59:35.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg=', 'Content-Length': '0'} 2022-09-02T05:59:35.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566158944878f7a0-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.634 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.635 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.635 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.636 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.636 DEBUG:boto:Method: GET 2022-09-02T05:59:35.637 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.637 DEBUG:boto:Data: 2022-09-02T05:59:35.638 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.639 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.640 DEBUG:boto:Params: {} 2022-09-02T05:59:35.640 DEBUG:boto:Token: None 2022-09-02T05:59:35.641 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:35.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg= 2022-09-02T05:59:35.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fI3isZ2NNxK+tIEPZ+DXjxImRyg=', 'Content-Length': '0'} 2022-09-02T05:59:35.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008461811ca9433b11-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.645 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.646 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.646 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.647 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.647 DEBUG:boto:Method: GET 2022-09-02T05:59:35.648 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.648 DEBUG:boto:Data: 2022-09-02T05:59:35.649 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.650 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.651 DEBUG:boto:Params: {} 2022-09-02T05:59:35.652 DEBUG:boto:Token: None 2022-09-02T05:59:35.653 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b5662275b335614-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.659 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.660 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.660 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.661 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.662 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.663 DEBUG:boto:Method: GET 2022-09-02T05:59:35.664 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.665 DEBUG:boto:Data: 2022-09-02T05:59:35.666 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.668 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.669 DEBUG:boto:Params: {} 2022-09-02T05:59:35.670 DEBUG:boto:Token: None 2022-09-02T05:59:35.671 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.672 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e775799e0548802-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.680 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.681 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.682 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.683 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.684 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:35.685 DEBUG:boto:Data: 2022-09-02T05:59:35.686 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.688 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.689 DEBUG:boto:Params: {} 2022-09-02T05:59:35.690 DEBUG:boto:Token: None 2022-09-02T05:59:35.691 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148 2022-09-02T05:59:35.692 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E+DB+pB+v5m4YfvK+RNtPE7LmuM= 2022-09-02T05:59:35.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E+DB+pB+v5m4YfvK+RNtPE7LmuM=', 'Content-Length': '0'} 2022-09-02T05:59:35.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2cfc9afe482ca8-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.697 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.698 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.699 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.700 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:35.701 DEBUG:boto:Data: 2022-09-02T05:59:35.702 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.703 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.704 DEBUG:boto:Params: {} 2022-09-02T05:59:35.705 DEBUG:boto:Token: None 2022-09-02T05:59:35.705 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148 2022-09-02T05:59:35.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E+DB+pB+v5m4YfvK+RNtPE7LmuM= 2022-09-02T05:59:35.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E+DB+pB+v5m4YfvK+RNtPE7LmuM=', 'Content-Length': '0'} 2022-09-02T05:59:35.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090241e8926134dfe-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:35.710 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.710 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.711 DEBUG:boto:Method: GET 2022-09-02T05:59:35.712 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:35.712 DEBUG:boto:Data: 2022-09-02T05:59:35.713 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.714 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.714 DEBUG:boto:Params: {} 2022-09-02T05:59:35.715 DEBUG:boto:Token: None 2022-09-02T05:59:35.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148 2022-09-02T05:59:35.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bZLjZTKRcuX/+B9sldKkG4/7n4g= 2022-09-02T05:59:35.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bZLjZTKRcuX/+B9sldKkG4/7n4g=', 'Content-Length': '0'} 2022-09-02T05:59:35.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046064aff7633bac7-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.720 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:35.721 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:35.722 DEBUG:boto:Method: GET 2022-09-02T05:59:35.722 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:35.723 DEBUG:boto:Data: 2022-09-02T05:59:35.723 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.725 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.725 DEBUG:boto:Params: {} 2022-09-02T05:59:35.726 DEBUG:boto:Token: None 2022-09-02T05:59:35.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj148 2022-09-02T05:59:35.727 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bZLjZTKRcuX/+B9sldKkG4/7n4g= 2022-09-02T05:59:35.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bZLjZTKRcuX/+B9sldKkG4/7n4g=', 'Content-Length': '0'} 2022-09-02T05:59:35.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1e9f43d85b8ac0-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.774 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.775 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.775 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.776 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.776 DEBUG:boto:Method: GET 2022-09-02T05:59:35.777 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.777 DEBUG:boto:Data: 2022-09-02T05:59:35.778 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.779 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.779 DEBUG:boto:Params: {} 2022-09-02T05:59:35.780 DEBUG:boto:Token: None 2022-09-02T05:59:35.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c6feaa9dd702824-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.786 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.786 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.788 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.788 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.789 DEBUG:boto:Method: GET 2022-09-02T05:59:35.789 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:35.790 DEBUG:boto:Data: 2022-09-02T05:59:35.790 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.791 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.792 DEBUG:boto:Params: {} 2022-09-02T05:59:35.792 DEBUG:boto:Token: None 2022-09-02T05:59:35.793 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ffb7b908eed5dc8-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.799 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:35.800 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.801 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.801 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.802 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.803 DEBUG:boto:Method: GET 2022-09-02T05:59:35.803 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.804 DEBUG:boto:Data: 2022-09-02T05:59:35.804 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.805 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.806 DEBUG:boto:Params: {} 2022-09-02T05:59:35.806 DEBUG:boto:Token: None 2022-09-02T05:59:35.807 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.808 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg= 2022-09-02T05:59:35.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg=', 'Content-Length': '0'} 2022-09-02T05:59:35.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071db431c9cbab8c4-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.812 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.812 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.813 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.813 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.814 DEBUG:boto:Method: GET 2022-09-02T05:59:35.815 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.815 DEBUG:boto:Data: 2022-09-02T05:59:35.816 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.817 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.817 DEBUG:boto:Params: {} 2022-09-02T05:59:35.818 DEBUG:boto:Token: None 2022-09-02T05:59:35.819 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg= 2022-09-02T05:59:35.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg=', 'Content-Length': '0'} 2022-09-02T05:59:35.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096661c4c3a01b96d-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.823 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.824 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.825 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.825 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.826 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.827 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.827 DEBUG:boto:Data: 2022-09-02T05:59:35.828 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.829 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.830 DEBUG:boto:Params: {} 2022-09-02T05:59:35.831 DEBUG:boto:Token: None 2022-09-02T05:59:35.831 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x7ECHEo87Nmvn5Ecv14hoCGgs6A= 2022-09-02T05:59:35.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x7ECHEo87Nmvn5Ecv14hoCGgs6A=', 'Content-Length': '0'} 2022-09-02T05:59:35.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0f37dacffdf502-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.835 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.836 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.836 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.837 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.837 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.838 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.838 DEBUG:boto:Data: 2022-09-02T05:59:35.839 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.840 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.841 DEBUG:boto:Params: {} 2022-09-02T05:59:35.842 DEBUG:boto:Token: None 2022-09-02T05:59:35.843 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:x7ECHEo87Nmvn5Ecv14hoCGgs6A= 2022-09-02T05:59:35.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:x7ECHEo87Nmvn5Ecv14hoCGgs6A=', 'Content-Length': '0'} 2022-09-02T05:59:35.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006cd492f03ca7d36-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:35.848 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.848 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.849 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.849 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.850 DEBUG:boto:Method: GET 2022-09-02T05:59:35.850 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.851 DEBUG:boto:Data: 2022-09-02T05:59:35.852 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.854 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.854 DEBUG:boto:Params: {} 2022-09-02T05:59:35.855 DEBUG:boto:Token: None 2022-09-02T05:59:35.856 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg= 2022-09-02T05:59:35.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg=', 'Content-Length': '0'} 2022-09-02T05:59:35.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36ec4e08cc7963a-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.902 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.903 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.904 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.904 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.905 DEBUG:boto:Method: GET 2022-09-02T05:59:35.905 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.906 DEBUG:boto:Data: 2022-09-02T05:59:35.906 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.908 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.908 DEBUG:boto:Params: {} 2022-09-02T05:59:35.909 DEBUG:boto:Token: None 2022-09-02T05:59:35.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:35.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg= 2022-09-02T05:59:35.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bTgeLwjDyE5uV6tG9VKyqBBYnOg=', 'Content-Length': '0'} 2022-09-02T05:59:35.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d45af37bef36a0-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.914 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.915 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.916 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.916 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.917 DEBUG:boto:Method: GET 2022-09-02T05:59:35.917 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.918 DEBUG:boto:Data: 2022-09-02T05:59:35.918 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.919 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.920 DEBUG:boto:Params: {} 2022-09-02T05:59:35.920 DEBUG:boto:Token: None 2022-09-02T05:59:35.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5e602c7a7eca21e-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.926 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.927 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.928 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.928 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.929 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.929 DEBUG:boto:Method: GET 2022-09-02T05:59:35.930 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.931 DEBUG:boto:Data: 2022-09-02T05:59:35.931 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.932 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.933 DEBUG:boto:Params: {} 2022-09-02T05:59:35.934 DEBUG:boto:Token: None 2022-09-02T05:59:35.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000f531da3f6339a5-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.941 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.942 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.942 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.943 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.943 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:35.944 DEBUG:boto:Data: 2022-09-02T05:59:35.944 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.945 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.946 DEBUG:boto:Params: {} 2022-09-02T05:59:35.947 DEBUG:boto:Token: None 2022-09-02T05:59:35.947 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155 2022-09-02T05:59:35.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+aB7veqnIsTxleJqrFi/I+COFw= 2022-09-02T05:59:35.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+aB7veqnIsTxleJqrFi/I+COFw=', 'Content-Length': '0'} 2022-09-02T05:59:35.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9548ddfb716da9-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.952 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.952 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.953 DEBUG:boto:Method: HEAD 2022-09-02T05:59:35.953 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:35.954 DEBUG:boto:Data: 2022-09-02T05:59:35.954 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.955 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.956 DEBUG:boto:Params: {} 2022-09-02T05:59:35.957 DEBUG:boto:Token: None 2022-09-02T05:59:35.957 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155 2022-09-02T05:59:35.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+aB7veqnIsTxleJqrFi/I+COFw= 2022-09-02T05:59:35.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+aB7veqnIsTxleJqrFi/I+COFw=', 'Content-Length': '0'} 2022-09-02T05:59:35.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007974be5359258376-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:35.962 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.963 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.963 DEBUG:boto:Method: GET 2022-09-02T05:59:35.964 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:35.965 DEBUG:boto:Data: 2022-09-02T05:59:35.965 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.966 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.967 DEBUG:boto:Params: {} 2022-09-02T05:59:35.968 DEBUG:boto:Token: None 2022-09-02T05:59:35.968 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155 2022-09-02T05:59:35.969 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+o4LV59npUPL4ZCM7oRBSWK4w4= 2022-09-02T05:59:35.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+o4LV59npUPL4ZCM7oRBSWK4w4=', 'Content-Length': '0'} 2022-09-02T05:59:35.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3b782541ae061a-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.972 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:35.973 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:35.973 DEBUG:boto:Method: GET 2022-09-02T05:59:35.974 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:35.975 DEBUG:boto:Data: 2022-09-02T05:59:35.975 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.976 DEBUG:boto:Port: 8000 2022-09-02T05:59:35.977 DEBUG:boto:Params: {} 2022-09-02T05:59:35.977 DEBUG:boto:Token: None 2022-09-02T05:59:35.978 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/obj155 2022-09-02T05:59:35.979 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0+o4LV59npUPL4ZCM7oRBSWK4w4= 2022-09-02T05:59:35.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0+o4LV59npUPL4ZCM7oRBSWK4w4=', 'Content-Length': '0'} 2022-09-02T05:59:35.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016c0e8844abf5cca-0063119bc7-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.982 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.983 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.984 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.984 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.985 DEBUG:boto:Method: GET 2022-09-02T05:59:35.985 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.986 DEBUG:boto:Data: 2022-09-02T05:59:35.986 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:35.987 DEBUG:boto:Port: 8001 2022-09-02T05:59:35.988 DEBUG:boto:Params: {} 2022-09-02T05:59:35.988 DEBUG:boto:Token: None 2022-09-02T05:59:35.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:35 GMT /xljbkf-74/ 2022-09-02T05:59:35.990 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4= 2022-09-02T05:59:35.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:35 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HOFJQRsJPRWIixpM1csYJl0n2m4=', 'Content-Length': '0'} 2022-09-02T05:59:35.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3c7906856b07ab-0063119bc7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:35.994 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:35.995 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:35.995 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:35.996 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.996 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.997 DEBUG:boto:Method: GET 2022-09-02T05:59:35.997 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:35.998 DEBUG:boto:Data: 2022-09-02T05:59:35.998 DEBUG:boto:Headers: {} 2022-09-02T05:59:35.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:35.999 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.000 DEBUG:boto:Params: {} 2022-09-02T05:59:36.000 DEBUG:boto:Token: None 2022-09-02T05:59:36.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015337a4af8761213-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.008 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:36.009 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.010 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.011 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.011 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.012 DEBUG:boto:Method: GET 2022-09-02T05:59:36.012 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.013 DEBUG:boto:Data: 2022-09-02T05:59:36.013 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.014 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.015 DEBUG:boto:Params: {} 2022-09-02T05:59:36.016 DEBUG:boto:Token: None 2022-09-02T05:59:36.016 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M= 2022-09-02T05:59:36.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M=', 'Content-Length': '0'} 2022-09-02T05:59:36.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f271fad568518d09-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.021 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.021 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.022 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.022 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.023 DEBUG:boto:Method: GET 2022-09-02T05:59:36.023 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.024 DEBUG:boto:Data: 2022-09-02T05:59:36.024 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.025 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.026 DEBUG:boto:Params: {} 2022-09-02T05:59:36.027 DEBUG:boto:Token: None 2022-09-02T05:59:36.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M= 2022-09-02T05:59:36.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M=', 'Content-Length': '0'} 2022-09-02T05:59:36.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f5540feeefe9c0-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.032 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.032 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.033 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.033 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.033 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.034 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.034 DEBUG:boto:Data: 2022-09-02T05:59:36.035 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.036 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.036 DEBUG:boto:Params: {} 2022-09-02T05:59:36.037 DEBUG:boto:Token: None 2022-09-02T05:59:36.038 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxmSKk6SksrUvbbV3TTiBB+S1g= 2022-09-02T05:59:36.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxmSKk6SksrUvbbV3TTiBB+S1g=', 'Content-Length': '0'} 2022-09-02T05:59:36.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8d2ed595c717b6-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.042 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.042 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.043 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.043 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.044 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.044 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.045 DEBUG:boto:Data: 2022-09-02T05:59:36.045 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.046 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.047 DEBUG:boto:Params: {} 2022-09-02T05:59:36.047 DEBUG:boto:Token: None 2022-09-02T05:59:36.048 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vaxmSKk6SksrUvbbV3TTiBB+S1g= 2022-09-02T05:59:36.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vaxmSKk6SksrUvbbV3TTiBB+S1g=', 'Content-Length': '0'} 2022-09-02T05:59:36.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e36c2e6219058b6-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:36.052 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.053 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.053 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.054 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.054 DEBUG:boto:Method: GET 2022-09-02T05:59:36.055 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.056 DEBUG:boto:Data: 2022-09-02T05:59:36.056 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.057 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.058 DEBUG:boto:Params: {} 2022-09-02T05:59:36.058 DEBUG:boto:Token: None 2022-09-02T05:59:36.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M= 2022-09-02T05:59:36.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M=', 'Content-Length': '0'} 2022-09-02T05:59:36.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f8f29b5a1baf51a-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.063 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.064 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.064 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.065 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.065 DEBUG:boto:Method: GET 2022-09-02T05:59:36.066 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.066 DEBUG:boto:Data: 2022-09-02T05:59:36.067 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.068 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.068 DEBUG:boto:Params: {} 2022-09-02T05:59:36.069 DEBUG:boto:Token: None 2022-09-02T05:59:36.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:36.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M= 2022-09-02T05:59:36.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MRFSkj6SnkrOd5I4QVELSbWxJ9M=', 'Content-Length': '0'} 2022-09-02T05:59:36.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77261c5af1bb0a8-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.114 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.115 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.115 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.116 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.116 DEBUG:boto:Method: GET 2022-09-02T05:59:36.117 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.117 DEBUG:boto:Data: 2022-09-02T05:59:36.118 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.119 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.120 DEBUG:boto:Params: {} 2022-09-02T05:59:36.120 DEBUG:boto:Token: None 2022-09-02T05:59:36.121 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.122 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccad277dc5460535-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.126 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.127 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.127 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.128 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.128 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.129 DEBUG:boto:Method: GET 2022-09-02T05:59:36.129 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.130 DEBUG:boto:Data: 2022-09-02T05:59:36.130 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.131 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.132 DEBUG:boto:Params: {} 2022-09-02T05:59:36.132 DEBUG:boto:Token: None 2022-09-02T05:59:36.133 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080800186a5ca93ba-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.139 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.140 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.140 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.141 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.141 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:36.142 DEBUG:boto:Data: 2022-09-02T05:59:36.142 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.143 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.144 DEBUG:boto:Params: {} 2022-09-02T05:59:36.144 DEBUG:boto:Token: None 2022-09-02T05:59:36.145 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17 2022-09-02T05:59:36.146 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rdFFQVH3vNEpZMrlyi/FcstrkqM= 2022-09-02T05:59:36.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rdFFQVH3vNEpZMrlyi/FcstrkqM=', 'Content-Length': '0'} 2022-09-02T05:59:36.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999ab7e912d7cd73-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.149 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.150 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.150 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.151 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:36.151 DEBUG:boto:Data: 2022-09-02T05:59:36.152 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.153 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.153 DEBUG:boto:Params: {} 2022-09-02T05:59:36.154 DEBUG:boto:Token: None 2022-09-02T05:59:36.154 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17 2022-09-02T05:59:36.155 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rdFFQVH3vNEpZMrlyi/FcstrkqM= 2022-09-02T05:59:36.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rdFFQVH3vNEpZMrlyi/FcstrkqM=', 'Content-Length': '0'} 2022-09-02T05:59:36.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009399ba30d4b9481b-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:36.159 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.160 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.160 DEBUG:boto:Method: GET 2022-09-02T05:59:36.161 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:36.161 DEBUG:boto:Data: 2022-09-02T05:59:36.162 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.163 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.163 DEBUG:boto:Params: {} 2022-09-02T05:59:36.164 DEBUG:boto:Token: None 2022-09-02T05:59:36.165 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17 2022-09-02T05:59:36.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DGTpIlWSKxgfDITD4t37tQPDP7Q= 2022-09-02T05:59:36.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DGTpIlWSKxgfDITD4t37tQPDP7Q=', 'Content-Length': '0'} 2022-09-02T05:59:36.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef9b13239db781d-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.169 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:36.169 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:36.170 DEBUG:boto:Method: GET 2022-09-02T05:59:36.170 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:36.171 DEBUG:boto:Data: 2022-09-02T05:59:36.171 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.172 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.173 DEBUG:boto:Params: {} 2022-09-02T05:59:36.173 DEBUG:boto:Token: None 2022-09-02T05:59:36.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj17 2022-09-02T05:59:36.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DGTpIlWSKxgfDITD4t37tQPDP7Q= 2022-09-02T05:59:36.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DGTpIlWSKxgfDITD4t37tQPDP7Q=', 'Content-Length': '0'} 2022-09-02T05:59:36.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f90582c14d84a200-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.178 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.179 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.179 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.180 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.180 DEBUG:boto:Method: GET 2022-09-02T05:59:36.181 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.181 DEBUG:boto:Data: 2022-09-02T05:59:36.182 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.183 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.184 DEBUG:boto:Params: {} 2022-09-02T05:59:36.184 DEBUG:boto:Token: None 2022-09-02T05:59:36.185 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e36931fe110128d2-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.189 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.190 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.191 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.191 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.192 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.192 DEBUG:boto:Method: GET 2022-09-02T05:59:36.193 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:36.193 DEBUG:boto:Data: 2022-09-02T05:59:36.194 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.195 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.195 DEBUG:boto:Params: {} 2022-09-02T05:59:36.196 DEBUG:boto:Token: None 2022-09-02T05:59:36.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.197 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a17d08a3aa3843-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.203 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:36.204 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.205 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.206 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.206 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.207 DEBUG:boto:Method: GET 2022-09-02T05:59:36.207 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.208 DEBUG:boto:Data: 2022-09-02T05:59:36.208 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.209 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.210 DEBUG:boto:Params: {} 2022-09-02T05:59:36.210 DEBUG:boto:Token: None 2022-09-02T05:59:36.211 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.212 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M= 2022-09-02T05:59:36.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M=', 'Content-Length': '0'} 2022-09-02T05:59:36.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbfab6f6c21c6878-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.215 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.216 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.216 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.217 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.218 DEBUG:boto:Method: GET 2022-09-02T05:59:36.218 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.219 DEBUG:boto:Data: 2022-09-02T05:59:36.219 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.220 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.221 DEBUG:boto:Params: {} 2022-09-02T05:59:36.221 DEBUG:boto:Token: None 2022-09-02T05:59:36.222 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M= 2022-09-02T05:59:36.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M=', 'Content-Length': '0'} 2022-09-02T05:59:36.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f9a636e23bf308-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.270 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.271 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.271 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.272 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.272 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.273 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.274 DEBUG:boto:Data: 2022-09-02T05:59:36.274 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.275 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.276 DEBUG:boto:Params: {} 2022-09-02T05:59:36.276 DEBUG:boto:Token: None 2022-09-02T05:59:36.277 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lABgVeg1LGAuFNn2TVoZmnlC2zM= 2022-09-02T05:59:36.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lABgVeg1LGAuFNn2TVoZmnlC2zM=', 'Content-Length': '0'} 2022-09-02T05:59:36.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b0422a02cf460b-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.281 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.282 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.282 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.283 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.283 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.284 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.284 DEBUG:boto:Data: 2022-09-02T05:59:36.285 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.286 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.286 DEBUG:boto:Params: {} 2022-09-02T05:59:36.287 DEBUG:boto:Token: None 2022-09-02T05:59:36.288 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lABgVeg1LGAuFNn2TVoZmnlC2zM= 2022-09-02T05:59:36.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lABgVeg1LGAuFNn2TVoZmnlC2zM=', 'Content-Length': '0'} 2022-09-02T05:59:36.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c70f0b3e3f4dea-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:36.292 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.293 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.293 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.294 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.294 DEBUG:boto:Method: GET 2022-09-02T05:59:36.295 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.295 DEBUG:boto:Data: 2022-09-02T05:59:36.296 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.297 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.298 DEBUG:boto:Params: {} 2022-09-02T05:59:36.299 DEBUG:boto:Token: None 2022-09-02T05:59:36.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.300 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M= 2022-09-02T05:59:36.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M=', 'Content-Length': '0'} 2022-09-02T05:59:36.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81afcb9e722f563-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.304 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.305 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.305 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.306 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.306 DEBUG:boto:Method: GET 2022-09-02T05:59:36.307 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.308 DEBUG:boto:Data: 2022-09-02T05:59:36.308 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.309 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.310 DEBUG:boto:Params: {} 2022-09-02T05:59:36.311 DEBUG:boto:Token: None 2022-09-02T05:59:36.311 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:36.312 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M= 2022-09-02T05:59:36.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hvS45idkinBTXQVx2Vz46Pmf51M=', 'Content-Length': '0'} 2022-09-02T05:59:36.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d1d9d54b2137017-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.316 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.316 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.317 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.317 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.318 DEBUG:boto:Method: GET 2022-09-02T05:59:36.318 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.319 DEBUG:boto:Data: 2022-09-02T05:59:36.319 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.320 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.321 DEBUG:boto:Params: {} 2022-09-02T05:59:36.322 DEBUG:boto:Token: None 2022-09-02T05:59:36.323 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b53684e947efeb4-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.370 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.371 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.372 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.373 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.373 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.374 DEBUG:boto:Method: GET 2022-09-02T05:59:36.374 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.375 DEBUG:boto:Data: 2022-09-02T05:59:36.375 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.377 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.377 DEBUG:boto:Params: {} 2022-09-02T05:59:36.378 DEBUG:boto:Token: None 2022-09-02T05:59:36.378 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.379 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7939abfd8de691d-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.385 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.386 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.387 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.387 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.388 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:36.388 DEBUG:boto:Data: 2022-09-02T05:59:36.389 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.390 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.390 DEBUG:boto:Params: {} 2022-09-02T05:59:36.391 DEBUG:boto:Token: None 2022-09-02T05:59:36.392 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185 2022-09-02T05:59:36.392 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2GXiN9fRd9KtDhcjVm+QEDf1b6o= 2022-09-02T05:59:36.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2GXiN9fRd9KtDhcjVm+QEDf1b6o=', 'Content-Length': '0'} 2022-09-02T05:59:36.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdaf6c98394a8f67-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.396 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.396 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.397 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.398 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:36.398 DEBUG:boto:Data: 2022-09-02T05:59:36.399 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.400 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.401 DEBUG:boto:Params: {} 2022-09-02T05:59:36.402 DEBUG:boto:Token: None 2022-09-02T05:59:36.403 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185 2022-09-02T05:59:36.404 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2GXiN9fRd9KtDhcjVm+QEDf1b6o= 2022-09-02T05:59:36.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2GXiN9fRd9KtDhcjVm+QEDf1b6o=', 'Content-Length': '0'} 2022-09-02T05:59:36.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dacfb04b8748ef2-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:36.410 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.410 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.411 DEBUG:boto:Method: GET 2022-09-02T05:59:36.411 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:36.412 DEBUG:boto:Data: 2022-09-02T05:59:36.412 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.414 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.414 DEBUG:boto:Params: {} 2022-09-02T05:59:36.415 DEBUG:boto:Token: None 2022-09-02T05:59:36.416 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185 2022-09-02T05:59:36.416 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S0FskajSd3XWW2wVhfQl+wUzR68= 2022-09-02T05:59:36.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S0FskajSd3XWW2wVhfQl+wUzR68=', 'Content-Length': '0'} 2022-09-02T05:59:36.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a90cefa27deeea-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.420 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:36.420 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:36.421 DEBUG:boto:Method: GET 2022-09-02T05:59:36.421 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:36.422 DEBUG:boto:Data: 2022-09-02T05:59:36.422 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.423 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.424 DEBUG:boto:Params: {} 2022-09-02T05:59:36.425 DEBUG:boto:Token: None 2022-09-02T05:59:36.425 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj185 2022-09-02T05:59:36.426 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:S0FskajSd3XWW2wVhfQl+wUzR68= 2022-09-02T05:59:36.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:S0FskajSd3XWW2wVhfQl+wUzR68=', 'Content-Length': '0'} 2022-09-02T05:59:36.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48a7c8badc6bce9-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.470 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.471 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.472 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.472 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.473 DEBUG:boto:Method: GET 2022-09-02T05:59:36.473 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.474 DEBUG:boto:Data: 2022-09-02T05:59:36.475 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.477 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.477 DEBUG:boto:Params: {} 2022-09-02T05:59:36.479 DEBUG:boto:Token: None 2022-09-02T05:59:36.480 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.481 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8be1abdaea5bf9c-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.486 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.487 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.488 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.489 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.489 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.490 DEBUG:boto:Method: GET 2022-09-02T05:59:36.490 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:36.491 DEBUG:boto:Data: 2022-09-02T05:59:36.492 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.493 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.493 DEBUG:boto:Params: {} 2022-09-02T05:59:36.494 DEBUG:boto:Token: None 2022-09-02T05:59:36.495 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.501 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f0992289e910567-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.507 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:36.508 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.509 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.510 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.510 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.511 DEBUG:boto:Method: GET 2022-09-02T05:59:36.512 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.512 DEBUG:boto:Data: 2022-09-02T05:59:36.513 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.515 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.515 DEBUG:boto:Params: {} 2022-09-02T05:59:36.516 DEBUG:boto:Token: None 2022-09-02T05:59:36.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I= 2022-09-02T05:59:36.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I=', 'Content-Length': '0'} 2022-09-02T05:59:36.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42d93a567bd2458-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.521 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.522 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.523 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.524 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.525 DEBUG:boto:Method: GET 2022-09-02T05:59:36.526 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.527 DEBUG:boto:Data: 2022-09-02T05:59:36.528 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.530 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.531 DEBUG:boto:Params: {} 2022-09-02T05:59:36.532 DEBUG:boto:Token: None 2022-09-02T05:59:36.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I= 2022-09-02T05:59:36.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I=', 'Content-Length': '0'} 2022-09-02T05:59:36.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45928eb21b9d14a-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.539 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.539 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.540 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.540 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.541 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.541 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.542 DEBUG:boto:Data: 2022-09-02T05:59:36.543 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.544 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.544 DEBUG:boto:Params: {} 2022-09-02T05:59:36.545 DEBUG:boto:Token: None 2022-09-02T05:59:36.545 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.546 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2XhLJrePpp04CgMO8o4tlzPohcQ= 2022-09-02T05:59:36.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2XhLJrePpp04CgMO8o4tlzPohcQ=', 'Content-Length': '0'} 2022-09-02T05:59:36.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081fc715340f88bdd-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.550 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.550 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.551 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.551 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.552 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.553 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.553 DEBUG:boto:Data: 2022-09-02T05:59:36.554 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.555 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.556 DEBUG:boto:Params: {} 2022-09-02T05:59:36.556 DEBUG:boto:Token: None 2022-09-02T05:59:36.557 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2XhLJrePpp04CgMO8o4tlzPohcQ= 2022-09-02T05:59:36.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2XhLJrePpp04CgMO8o4tlzPohcQ=', 'Content-Length': '0'} 2022-09-02T05:59:36.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051975a5f090016bf-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:36.562 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.562 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.563 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.564 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.564 DEBUG:boto:Method: GET 2022-09-02T05:59:36.565 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.565 DEBUG:boto:Data: 2022-09-02T05:59:36.566 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.567 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.567 DEBUG:boto:Params: {} 2022-09-02T05:59:36.568 DEBUG:boto:Token: None 2022-09-02T05:59:36.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I= 2022-09-02T05:59:36.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I=', 'Content-Length': '0'} 2022-09-02T05:59:36.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0986d518c031063-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.573 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.574 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.574 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.575 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.575 DEBUG:boto:Method: GET 2022-09-02T05:59:36.576 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.577 DEBUG:boto:Data: 2022-09-02T05:59:36.577 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.578 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.579 DEBUG:boto:Params: {} 2022-09-02T05:59:36.580 DEBUG:boto:Token: None 2022-09-02T05:59:36.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:36.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I= 2022-09-02T05:59:36.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5L08o6L+cF68buT4S7vebkVdd+I=', 'Content-Length': '0'} 2022-09-02T05:59:36.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba6d7c586a4a077f-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.626 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.627 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.628 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.629 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.630 DEBUG:boto:Method: GET 2022-09-02T05:59:36.631 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.632 DEBUG:boto:Data: 2022-09-02T05:59:36.632 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.633 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.634 DEBUG:boto:Params: {} 2022-09-02T05:59:36.634 DEBUG:boto:Token: None 2022-09-02T05:59:36.635 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff5d08216c115641-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.640 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.641 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.641 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.642 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.642 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.643 DEBUG:boto:Method: GET 2022-09-02T05:59:36.643 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.644 DEBUG:boto:Data: 2022-09-02T05:59:36.645 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.646 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.646 DEBUG:boto:Params: {} 2022-09-02T05:59:36.647 DEBUG:boto:Token: None 2022-09-02T05:59:36.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a6e4a889405c1b7-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.654 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.655 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.655 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.656 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.657 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:36.657 DEBUG:boto:Data: 2022-09-02T05:59:36.658 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.659 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.659 DEBUG:boto:Params: {} 2022-09-02T05:59:36.660 DEBUG:boto:Token: None 2022-09-02T05:59:36.660 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19 2022-09-02T05:59:36.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o/Zall6dhnrH0YvYYOKN2ME5BbE= 2022-09-02T05:59:36.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o/Zall6dhnrH0YvYYOKN2ME5BbE=', 'Content-Length': '0'} 2022-09-02T05:59:36.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c3f14eb71f0ed32-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.667 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.668 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.669 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.670 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:36.671 DEBUG:boto:Data: 2022-09-02T05:59:36.672 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.674 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.675 DEBUG:boto:Params: {} 2022-09-02T05:59:36.677 DEBUG:boto:Token: None 2022-09-02T05:59:36.678 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19 2022-09-02T05:59:36.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o/Zall6dhnrH0YvYYOKN2ME5BbE= 2022-09-02T05:59:36.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o/Zall6dhnrH0YvYYOKN2ME5BbE=', 'Content-Length': '0'} 2022-09-02T05:59:36.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffa589319b4edd7-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:36.685 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.686 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.687 DEBUG:boto:Method: GET 2022-09-02T05:59:36.688 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:36.690 DEBUG:boto:Data: 2022-09-02T05:59:36.691 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.692 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.692 DEBUG:boto:Params: {} 2022-09-02T05:59:36.693 DEBUG:boto:Token: None 2022-09-02T05:59:36.694 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19 2022-09-02T05:59:36.694 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zfBb2Laa7Q4OUPznTENEdZCz6lQ= 2022-09-02T05:59:36.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zfBb2Laa7Q4OUPznTENEdZCz6lQ=', 'Content-Length': '0'} 2022-09-02T05:59:36.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752cef4f5d749bea-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.738 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:36.739 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:36.740 DEBUG:boto:Method: GET 2022-09-02T05:59:36.741 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:36.742 DEBUG:boto:Data: 2022-09-02T05:59:36.742 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.743 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.744 DEBUG:boto:Params: {} 2022-09-02T05:59:36.745 DEBUG:boto:Token: None 2022-09-02T05:59:36.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj19 2022-09-02T05:59:36.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zfBb2Laa7Q4OUPznTENEdZCz6lQ= 2022-09-02T05:59:36.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zfBb2Laa7Q4OUPznTENEdZCz6lQ=', 'Content-Length': '0'} 2022-09-02T05:59:36.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b75299074231dbe6-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.752 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.753 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.753 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.754 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.755 DEBUG:boto:Method: GET 2022-09-02T05:59:36.755 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.756 DEBUG:boto:Data: 2022-09-02T05:59:36.757 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.759 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.759 DEBUG:boto:Params: {} 2022-09-02T05:59:36.760 DEBUG:boto:Token: None 2022-09-02T05:59:36.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.761 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009046f5e6045a2604-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.766 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.767 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.768 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.768 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.769 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.770 DEBUG:boto:Method: GET 2022-09-02T05:59:36.771 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:36.771 DEBUG:boto:Data: 2022-09-02T05:59:36.772 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.773 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.773 DEBUG:boto:Params: {} 2022-09-02T05:59:36.774 DEBUG:boto:Token: None 2022-09-02T05:59:36.775 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.776 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac6a41a03667348-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.782 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:36.783 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.784 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.785 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.785 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.786 DEBUG:boto:Method: GET 2022-09-02T05:59:36.786 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.787 DEBUG:boto:Data: 2022-09-02T05:59:36.787 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.788 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.789 DEBUG:boto:Params: {} 2022-09-02T05:59:36.790 DEBUG:boto:Token: None 2022-09-02T05:59:36.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk= 2022-09-02T05:59:36.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk=', 'Content-Length': '0'} 2022-09-02T05:59:36.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ec4191f1d8b7759-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.795 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.796 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.797 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.797 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.798 DEBUG:boto:Method: GET 2022-09-02T05:59:36.798 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.799 DEBUG:boto:Data: 2022-09-02T05:59:36.801 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.802 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.803 DEBUG:boto:Params: {} 2022-09-02T05:59:36.804 DEBUG:boto:Token: None 2022-09-02T05:59:36.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.805 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk= 2022-09-02T05:59:36.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk=', 'Content-Length': '0'} 2022-09-02T05:59:36.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034f62bddc79c01cb-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.850 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.851 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.852 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.853 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.854 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.855 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.856 DEBUG:boto:Data: 2022-09-02T05:59:36.857 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.859 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.859 DEBUG:boto:Params: {} 2022-09-02T05:59:36.860 DEBUG:boto:Token: None 2022-09-02T05:59:36.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q3b2na9UM6z57WaLKV0fevxOXqA= 2022-09-02T05:59:36.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q3b2na9UM6z57WaLKV0fevxOXqA=', 'Content-Length': '0'} 2022-09-02T05:59:36.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3701013b6a1a707-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.868 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.869 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.870 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.871 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.872 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.872 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.873 DEBUG:boto:Data: 2022-09-02T05:59:36.874 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.876 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.877 DEBUG:boto:Params: {} 2022-09-02T05:59:36.878 DEBUG:boto:Token: None 2022-09-02T05:59:36.879 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q3b2na9UM6z57WaLKV0fevxOXqA= 2022-09-02T05:59:36.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q3b2na9UM6z57WaLKV0fevxOXqA=', 'Content-Length': '0'} 2022-09-02T05:59:36.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca3b3302c05138d-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:36.887 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.887 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.888 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.889 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.890 DEBUG:boto:Method: GET 2022-09-02T05:59:36.891 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.892 DEBUG:boto:Data: 2022-09-02T05:59:36.893 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.895 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.896 DEBUG:boto:Params: {} 2022-09-02T05:59:36.897 DEBUG:boto:Token: None 2022-09-02T05:59:36.898 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk= 2022-09-02T05:59:36.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk=', 'Content-Length': '0'} 2022-09-02T05:59:36.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005477381f6c2fb4d7-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.905 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.906 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.907 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.908 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.909 DEBUG:boto:Method: GET 2022-09-02T05:59:36.910 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.911 DEBUG:boto:Data: 2022-09-02T05:59:36.912 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.914 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.915 DEBUG:boto:Params: {} 2022-09-02T05:59:36.916 DEBUG:boto:Token: None 2022-09-02T05:59:36.916 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:36.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk= 2022-09-02T05:59:36.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WYZpOoIklTYuTbuNtL24FBX3Npk=', 'Content-Length': '0'} 2022-09-02T05:59:36.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cca78a56e43ef85-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.921 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.922 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.922 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.923 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.923 DEBUG:boto:Method: GET 2022-09-02T05:59:36.924 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.925 DEBUG:boto:Data: 2022-09-02T05:59:36.925 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.926 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.927 DEBUG:boto:Params: {} 2022-09-02T05:59:36.928 DEBUG:boto:Token: None 2022-09-02T05:59:36.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.929 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f32ed9efc5149af4-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.933 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.934 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.935 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.935 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.936 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.936 DEBUG:boto:Method: GET 2022-09-02T05:59:36.937 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.937 DEBUG:boto:Data: 2022-09-02T05:59:36.938 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.939 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.940 DEBUG:boto:Params: {} 2022-09-02T05:59:36.940 DEBUG:boto:Token: None 2022-09-02T05:59:36.941 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.942 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:36.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070d93f6bca425537-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.947 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:36.948 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.949 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.949 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.950 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:36.950 DEBUG:boto:Data: 2022-09-02T05:59:36.951 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.952 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.952 DEBUG:boto:Params: {} 2022-09-02T05:59:36.953 DEBUG:boto:Token: None 2022-09-02T05:59:36.954 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22 2022-09-02T05:59:36.954 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rg7h9G5E646kS7TDKBH5Md/ieD0= 2022-09-02T05:59:36.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rg7h9G5E646kS7TDKBH5Md/ieD0=', 'Content-Length': '0'} 2022-09-02T05:59:36.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db0328ea6cdd446c-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.958 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.958 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.959 DEBUG:boto:Method: HEAD 2022-09-02T05:59:36.960 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:36.960 DEBUG:boto:Data: 2022-09-02T05:59:36.961 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.962 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.963 DEBUG:boto:Params: {} 2022-09-02T05:59:36.963 DEBUG:boto:Token: None 2022-09-02T05:59:36.964 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22 2022-09-02T05:59:36.964 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Rg7h9G5E646kS7TDKBH5Md/ieD0= 2022-09-02T05:59:36.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Rg7h9G5E646kS7TDKBH5Md/ieD0=', 'Content-Length': '0'} 2022-09-02T05:59:36.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016622ee4b66115ce-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:36.968 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.969 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.969 DEBUG:boto:Method: GET 2022-09-02T05:59:36.970 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:36.971 DEBUG:boto:Data: 2022-09-02T05:59:36.971 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.973 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.973 DEBUG:boto:Params: {} 2022-09-02T05:59:36.974 DEBUG:boto:Token: None 2022-09-02T05:59:36.974 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22 2022-09-02T05:59:36.975 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l+AueM4OAeMf3h1a71zxbu8+Wh0= 2022-09-02T05:59:36.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l+AueM4OAeMf3h1a71zxbu8+Wh0=', 'Content-Length': '0'} 2022-09-02T05:59:36.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1eb2795634221d-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.978 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:36.979 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:36.980 DEBUG:boto:Method: GET 2022-09-02T05:59:36.980 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:36.981 DEBUG:boto:Data: 2022-09-02T05:59:36.982 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:36.983 DEBUG:boto:Port: 8000 2022-09-02T05:59:36.984 DEBUG:boto:Params: {} 2022-09-02T05:59:36.984 DEBUG:boto:Token: None 2022-09-02T05:59:36.985 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/obj22 2022-09-02T05:59:36.985 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:l+AueM4OAeMf3h1a71zxbu8+Wh0= 2022-09-02T05:59:36.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:l+AueM4OAeMf3h1a71zxbu8+Wh0=', 'Content-Length': '0'} 2022-09-02T05:59:36.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ec934023bda1e5-0063119bc8-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:36.989 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:36.990 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:36.991 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.991 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.992 DEBUG:boto:Method: GET 2022-09-02T05:59:36.992 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:36.993 DEBUG:boto:Data: 2022-09-02T05:59:36.993 DEBUG:boto:Headers: {} 2022-09-02T05:59:36.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:36.994 DEBUG:boto:Port: 8001 2022-09-02T05:59:36.995 DEBUG:boto:Params: {} 2022-09-02T05:59:36.995 DEBUG:boto:Token: None 2022-09-02T05:59:36.996 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:36 GMT /xljbkf-74/ 2022-09-02T05:59:36.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg= 2022-09-02T05:59:36.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:36 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k4atC8WQ66TkCxJqmaGEab2P0eg=', 'Content-Length': '0'} 2022-09-02T05:59:37.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f91b66db07b7504-0063119bc8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.000 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.001 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.002 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.002 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:37.003 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:37.003 DEBUG:boto:Method: GET 2022-09-02T05:59:37.004 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:37.005 DEBUG:boto:Data: 2022-09-02T05:59:37.005 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.006 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.007 DEBUG:boto:Params: {} 2022-09-02T05:59:37.008 DEBUG:boto:Token: None 2022-09-02T05:59:37.009 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.009 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c433124cd93eb66-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.015 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:37.017 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.017 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.018 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.018 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.019 DEBUG:boto:Method: GET 2022-09-02T05:59:37.019 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.020 DEBUG:boto:Data: 2022-09-02T05:59:37.020 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.021 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.022 DEBUG:boto:Params: {} 2022-09-02T05:59:37.023 DEBUG:boto:Token: None 2022-09-02T05:59:37.023 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.024 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc= 2022-09-02T05:59:37.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc=', 'Content-Length': '0'} 2022-09-02T05:59:37.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e19bd35b743662da-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.028 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.029 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.029 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.030 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.030 DEBUG:boto:Method: GET 2022-09-02T05:59:37.031 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.031 DEBUG:boto:Data: 2022-09-02T05:59:37.032 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.033 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.034 DEBUG:boto:Params: {} 2022-09-02T05:59:37.034 DEBUG:boto:Token: None 2022-09-02T05:59:37.035 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc= 2022-09-02T05:59:37.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc=', 'Content-Length': '0'} 2022-09-02T05:59:37.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb0857d89588e20-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.082 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.083 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.083 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.084 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.084 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.085 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.085 DEBUG:boto:Data: 2022-09-02T05:59:37.086 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.087 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.087 DEBUG:boto:Params: {} 2022-09-02T05:59:37.088 DEBUG:boto:Token: None 2022-09-02T05:59:37.089 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3utvPxeK5F0x8RBPq6e1iBx95yI= 2022-09-02T05:59:37.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3utvPxeK5F0x8RBPq6e1iBx95yI=', 'Content-Length': '0'} 2022-09-02T05:59:37.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c2ef3a40c78123-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.093 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.094 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.094 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.095 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.095 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.096 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.096 DEBUG:boto:Data: 2022-09-02T05:59:37.097 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.098 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.098 DEBUG:boto:Params: {} 2022-09-02T05:59:37.099 DEBUG:boto:Token: None 2022-09-02T05:59:37.100 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.100 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3utvPxeK5F0x8RBPq6e1iBx95yI= 2022-09-02T05:59:37.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3utvPxeK5F0x8RBPq6e1iBx95yI=', 'Content-Length': '0'} 2022-09-02T05:59:37.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd0d5f217d8e734a-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:37.104 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.105 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.105 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.106 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.106 DEBUG:boto:Method: GET 2022-09-02T05:59:37.107 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.107 DEBUG:boto:Data: 2022-09-02T05:59:37.108 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.109 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.109 DEBUG:boto:Params: {} 2022-09-02T05:59:37.110 DEBUG:boto:Token: None 2022-09-02T05:59:37.111 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.111 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc= 2022-09-02T05:59:37.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc=', 'Content-Length': '0'} 2022-09-02T05:59:37.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000791a12b3591e0d7-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.115 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.115 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.116 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.117 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.117 DEBUG:boto:Method: GET 2022-09-02T05:59:37.118 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.118 DEBUG:boto:Data: 2022-09-02T05:59:37.119 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.120 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.120 DEBUG:boto:Params: {} 2022-09-02T05:59:37.121 DEBUG:boto:Token: None 2022-09-02T05:59:37.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:37.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc= 2022-09-02T05:59:37.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:hY8B+WkjhaPUHNpzSRpTwzgw/Vc=', 'Content-Length': '0'} 2022-09-02T05:59:37.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b94ceaddec88f4d-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.127 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.128 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.128 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.129 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.129 DEBUG:boto:Method: GET 2022-09-02T05:59:37.130 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.130 DEBUG:boto:Data: 2022-09-02T05:59:37.131 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.132 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.132 DEBUG:boto:Params: {} 2022-09-02T05:59:37.133 DEBUG:boto:Token: None 2022-09-02T05:59:37.134 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.134 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e37483035ebc234f-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.182 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.183 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.184 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.184 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.185 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.186 DEBUG:boto:Method: GET 2022-09-02T05:59:37.186 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.187 DEBUG:boto:Data: 2022-09-02T05:59:37.187 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.188 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.189 DEBUG:boto:Params: {} 2022-09-02T05:59:37.190 DEBUG:boto:Token: None 2022-09-02T05:59:37.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.191 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ac40ba32eb84e9-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.197 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.198 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.199 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.199 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.200 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:37.200 DEBUG:boto:Data: 2022-09-02T05:59:37.201 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.202 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.203 DEBUG:boto:Params: {} 2022-09-02T05:59:37.204 DEBUG:boto:Token: None 2022-09-02T05:59:37.204 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23 2022-09-02T05:59:37.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O+GCuPTQSIbE06HsIX+K+O7hO7E= 2022-09-02T05:59:37.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O+GCuPTQSIbE06HsIX+K+O7hO7E=', 'Content-Length': '0'} 2022-09-02T05:59:37.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a91e2a06523bf264-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.209 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.209 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.210 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.211 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:37.211 DEBUG:boto:Data: 2022-09-02T05:59:37.212 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.213 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.214 DEBUG:boto:Params: {} 2022-09-02T05:59:37.214 DEBUG:boto:Token: None 2022-09-02T05:59:37.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23 2022-09-02T05:59:37.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O+GCuPTQSIbE06HsIX+K+O7hO7E= 2022-09-02T05:59:37.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O+GCuPTQSIbE06HsIX+K+O7hO7E=', 'Content-Length': '0'} 2022-09-02T05:59:37.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e74b889882c56c8-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:37.220 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.221 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.221 DEBUG:boto:Method: GET 2022-09-02T05:59:37.222 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:37.223 DEBUG:boto:Data: 2022-09-02T05:59:37.223 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.225 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.225 DEBUG:boto:Params: {} 2022-09-02T05:59:37.226 DEBUG:boto:Token: None 2022-09-02T05:59:37.227 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23 2022-09-02T05:59:37.228 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PVyU7Y0KkhR/ET2LUbbZ8VIiWhU= 2022-09-02T05:59:37.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PVyU7Y0KkhR/ET2LUbbZ8VIiWhU=', 'Content-Length': '0'} 2022-09-02T05:59:37.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c36ff1ffc7350b4f-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.232 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:37.233 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:37.234 DEBUG:boto:Method: GET 2022-09-02T05:59:37.234 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:37.235 DEBUG:boto:Data: 2022-09-02T05:59:37.236 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.237 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.238 DEBUG:boto:Params: {} 2022-09-02T05:59:37.238 DEBUG:boto:Token: None 2022-09-02T05:59:37.239 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj23 2022-09-02T05:59:37.240 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PVyU7Y0KkhR/ET2LUbbZ8VIiWhU= 2022-09-02T05:59:37.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PVyU7Y0KkhR/ET2LUbbZ8VIiWhU=', 'Content-Length': '0'} 2022-09-02T05:59:37.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c512ef41abd90a-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.286 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.287 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.288 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.289 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.290 DEBUG:boto:Method: GET 2022-09-02T05:59:37.290 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.291 DEBUG:boto:Data: 2022-09-02T05:59:37.292 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.293 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.294 DEBUG:boto:Params: {} 2022-09-02T05:59:37.295 DEBUG:boto:Token: None 2022-09-02T05:59:37.296 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.297 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2c156451b160a5-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.302 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.303 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.304 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.305 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.306 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.307 DEBUG:boto:Method: GET 2022-09-02T05:59:37.308 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:37.309 DEBUG:boto:Data: 2022-09-02T05:59:37.310 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.312 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.313 DEBUG:boto:Params: {} 2022-09-02T05:59:37.314 DEBUG:boto:Token: None 2022-09-02T05:59:37.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72014dbb9f61928-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.322 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:37.324 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.325 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.326 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.327 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.328 DEBUG:boto:Method: GET 2022-09-02T05:59:37.329 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.329 DEBUG:boto:Data: 2022-09-02T05:59:37.330 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.331 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.332 DEBUG:boto:Params: {} 2022-09-02T05:59:37.333 DEBUG:boto:Token: None 2022-09-02T05:59:37.334 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q= 2022-09-02T05:59:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q=', 'Content-Length': '0'} 2022-09-02T05:59:37.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017644e67971dfa0f-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.339 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.340 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.341 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.342 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.342 DEBUG:boto:Method: GET 2022-09-02T05:59:37.344 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.344 DEBUG:boto:Data: 2022-09-02T05:59:37.345 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.347 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.347 DEBUG:boto:Params: {} 2022-09-02T05:59:37.348 DEBUG:boto:Token: None 2022-09-02T05:59:37.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q= 2022-09-02T05:59:37.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q=', 'Content-Length': '0'} 2022-09-02T05:59:37.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b88af814767549-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.354 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.355 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.355 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.356 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.357 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.357 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.358 DEBUG:boto:Data: 2022-09-02T05:59:37.359 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.360 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.361 DEBUG:boto:Params: {} 2022-09-02T05:59:37.362 DEBUG:boto:Token: None 2022-09-02T05:59:37.363 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XxtORgdVQw/k0z0WTN5gEumbXo8= 2022-09-02T05:59:37.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XxtORgdVQw/k0z0WTN5gEumbXo8=', 'Content-Length': '0'} 2022-09-02T05:59:37.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7be37a5937923b6-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.367 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.368 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.368 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.369 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.369 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.370 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.370 DEBUG:boto:Data: 2022-09-02T05:59:37.371 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.372 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.372 DEBUG:boto:Params: {} 2022-09-02T05:59:37.373 DEBUG:boto:Token: None 2022-09-02T05:59:37.374 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XxtORgdVQw/k0z0WTN5gEumbXo8= 2022-09-02T05:59:37.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XxtORgdVQw/k0z0WTN5gEumbXo8=', 'Content-Length': '0'} 2022-09-02T05:59:37.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cecc9d4f7ba88c17-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:37.378 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.379 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.380 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.380 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.381 DEBUG:boto:Method: GET 2022-09-02T05:59:37.382 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.382 DEBUG:boto:Data: 2022-09-02T05:59:37.383 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.384 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.384 DEBUG:boto:Params: {} 2022-09-02T05:59:37.385 DEBUG:boto:Token: None 2022-09-02T05:59:37.385 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.386 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q= 2022-09-02T05:59:37.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q=', 'Content-Length': '0'} 2022-09-02T05:59:37.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089499f585bd4b172-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.434 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.435 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.435 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.436 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.437 DEBUG:boto:Method: GET 2022-09-02T05:59:37.437 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.438 DEBUG:boto:Data: 2022-09-02T05:59:37.438 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.439 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.440 DEBUG:boto:Params: {} 2022-09-02T05:59:37.441 DEBUG:boto:Token: None 2022-09-02T05:59:37.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:37.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q= 2022-09-02T05:59:37.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jxQU28QIeWPMBiBLkcX3h/PM89Q=', 'Content-Length': '0'} 2022-09-02T05:59:37.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4f392b0528da3f-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.447 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.447 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.448 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.448 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.449 DEBUG:boto:Method: GET 2022-09-02T05:59:37.449 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.450 DEBUG:boto:Data: 2022-09-02T05:59:37.450 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.452 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.452 DEBUG:boto:Params: {} 2022-09-02T05:59:37.453 DEBUG:boto:Token: None 2022-09-02T05:59:37.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059bd26be3b821136-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.458 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.459 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.460 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.460 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.461 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.462 DEBUG:boto:Method: GET 2022-09-02T05:59:37.462 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.463 DEBUG:boto:Data: 2022-09-02T05:59:37.463 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.464 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.465 DEBUG:boto:Params: {} 2022-09-02T05:59:37.465 DEBUG:boto:Token: None 2022-09-02T05:59:37.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2b722835f9c0904-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.518 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.519 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.520 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.520 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.521 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:37.521 DEBUG:boto:Data: 2022-09-02T05:59:37.522 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.523 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.524 DEBUG:boto:Params: {} 2022-09-02T05:59:37.524 DEBUG:boto:Token: None 2022-09-02T05:59:37.525 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29 2022-09-02T05:59:37.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t2Ef+HFvBFRGnw6ecjd0+dPxJgg= 2022-09-02T05:59:37.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t2Ef+HFvBFRGnw6ecjd0+dPxJgg=', 'Content-Length': '0'} 2022-09-02T05:59:37.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c81f4293d8335fb-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.529 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.529 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.530 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.531 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:37.531 DEBUG:boto:Data: 2022-09-02T05:59:37.531 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.533 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.534 DEBUG:boto:Params: {} 2022-09-02T05:59:37.535 DEBUG:boto:Token: None 2022-09-02T05:59:37.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29 2022-09-02T05:59:37.537 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:t2Ef+HFvBFRGnw6ecjd0+dPxJgg= 2022-09-02T05:59:37.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:t2Ef+HFvBFRGnw6ecjd0+dPxJgg=', 'Content-Length': '0'} 2022-09-02T05:59:37.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f409a1aaf88708e2-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:37.543 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.544 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.545 DEBUG:boto:Method: GET 2022-09-02T05:59:37.546 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:37.546 DEBUG:boto:Data: 2022-09-02T05:59:37.547 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.548 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.549 DEBUG:boto:Params: {} 2022-09-02T05:59:37.549 DEBUG:boto:Token: None 2022-09-02T05:59:37.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29 2022-09-02T05:59:37.551 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nmlxc+ncR+toR5esoAFEkH7NlA= 2022-09-02T05:59:37.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nmlxc+ncR+toR5esoAFEkH7NlA=', 'Content-Length': '0'} 2022-09-02T05:59:37.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba407d74d6fa3ec-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.554 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:37.555 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:37.555 DEBUG:boto:Method: GET 2022-09-02T05:59:37.556 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:37.556 DEBUG:boto:Data: 2022-09-02T05:59:37.557 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.558 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.559 DEBUG:boto:Params: {} 2022-09-02T05:59:37.560 DEBUG:boto:Token: None 2022-09-02T05:59:37.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj29 2022-09-02T05:59:37.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7nmlxc+ncR+toR5esoAFEkH7NlA= 2022-09-02T05:59:37.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7nmlxc+ncR+toR5esoAFEkH7NlA=', 'Content-Length': '0'} 2022-09-02T05:59:37.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c6f17e231917af-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.565 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.565 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.566 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.566 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.567 DEBUG:boto:Method: GET 2022-09-02T05:59:37.568 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.568 DEBUG:boto:Data: 2022-09-02T05:59:37.569 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.570 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.570 DEBUG:boto:Params: {} 2022-09-02T05:59:37.571 DEBUG:boto:Token: None 2022-09-02T05:59:37.572 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.572 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd89d3b5512fe292-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.576 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.578 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.579 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.580 DEBUG:boto:Method: GET 2022-09-02T05:59:37.581 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:37.581 DEBUG:boto:Data: 2022-09-02T05:59:37.582 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.583 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.584 DEBUG:boto:Params: {} 2022-09-02T05:59:37.584 DEBUG:boto:Token: None 2022-09-02T05:59:37.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5b368cbf4a059f-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.593 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:37.596 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.596 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.597 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.597 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.598 DEBUG:boto:Method: GET 2022-09-02T05:59:37.598 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.599 DEBUG:boto:Data: 2022-09-02T05:59:37.600 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.601 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.601 DEBUG:boto:Params: {} 2022-09-02T05:59:37.602 DEBUG:boto:Token: None 2022-09-02T05:59:37.603 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA= 2022-09-02T05:59:37.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA=', 'Content-Length': '0'} 2022-09-02T05:59:37.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d0fcb023b74bebf-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.607 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.608 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.608 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.609 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.610 DEBUG:boto:Method: GET 2022-09-02T05:59:37.610 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.611 DEBUG:boto:Data: 2022-09-02T05:59:37.612 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.613 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.614 DEBUG:boto:Params: {} 2022-09-02T05:59:37.615 DEBUG:boto:Token: None 2022-09-02T05:59:37.615 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.616 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA= 2022-09-02T05:59:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA=', 'Content-Length': '0'} 2022-09-02T05:59:37.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d9edec2dbea8d66-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.620 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.620 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.621 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.622 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.622 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.623 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.623 DEBUG:boto:Data: 2022-09-02T05:59:37.624 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.625 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.626 DEBUG:boto:Params: {} 2022-09-02T05:59:37.627 DEBUG:boto:Token: None 2022-09-02T05:59:37.628 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.629 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G1UNQUPzWjiNvZEGuhWQAntre64= 2022-09-02T05:59:37.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G1UNQUPzWjiNvZEGuhWQAntre64=', 'Content-Length': '0'} 2022-09-02T05:59:37.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4ea9e65bb1808b-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.633 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.634 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.635 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.635 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.636 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.636 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.637 DEBUG:boto:Data: 2022-09-02T05:59:37.637 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.638 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.639 DEBUG:boto:Params: {} 2022-09-02T05:59:37.639 DEBUG:boto:Token: None 2022-09-02T05:59:37.640 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G1UNQUPzWjiNvZEGuhWQAntre64= 2022-09-02T05:59:37.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G1UNQUPzWjiNvZEGuhWQAntre64=', 'Content-Length': '0'} 2022-09-02T05:59:37.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a693c01dc20aae8-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:37.645 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.645 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.646 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.646 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.647 DEBUG:boto:Method: GET 2022-09-02T05:59:37.648 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.648 DEBUG:boto:Data: 2022-09-02T05:59:37.649 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.650 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.650 DEBUG:boto:Params: {} 2022-09-02T05:59:37.651 DEBUG:boto:Token: None 2022-09-02T05:59:37.652 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA= 2022-09-02T05:59:37.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA=', 'Content-Length': '0'} 2022-09-02T05:59:37.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e293d191264879-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.698 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.699 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.699 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.700 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.700 DEBUG:boto:Method: GET 2022-09-02T05:59:37.701 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.702 DEBUG:boto:Data: 2022-09-02T05:59:37.702 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.703 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.704 DEBUG:boto:Params: {} 2022-09-02T05:59:37.705 DEBUG:boto:Token: None 2022-09-02T05:59:37.707 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:37.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA= 2022-09-02T05:59:37.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JPn4faAS7Fvlx13IR9cb7WbyvnA=', 'Content-Length': '0'} 2022-09-02T05:59:37.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c486f11bdc6afdd-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.713 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.714 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.716 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.716 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.717 DEBUG:boto:Method: GET 2022-09-02T05:59:37.717 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.718 DEBUG:boto:Data: 2022-09-02T05:59:37.719 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.720 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.720 DEBUG:boto:Params: {} 2022-09-02T05:59:37.721 DEBUG:boto:Token: None 2022-09-02T05:59:37.722 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.722 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d956379ff41628-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.727 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.727 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.728 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.729 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.729 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.730 DEBUG:boto:Method: GET 2022-09-02T05:59:37.731 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.731 DEBUG:boto:Data: 2022-09-02T05:59:37.732 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.733 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.733 DEBUG:boto:Params: {} 2022-09-02T05:59:37.734 DEBUG:boto:Token: None 2022-09-02T05:59:37.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246711bdc3dfc1e9-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.741 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.742 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.743 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.743 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.744 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:37.744 DEBUG:boto:Data: 2022-09-02T05:59:37.745 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.746 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.746 DEBUG:boto:Params: {} 2022-09-02T05:59:37.747 DEBUG:boto:Token: None 2022-09-02T05:59:37.748 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31 2022-09-02T05:59:37.748 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:de+CegS8VF3XLhUITNJQg5Vvvz8= 2022-09-02T05:59:37.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:de+CegS8VF3XLhUITNJQg5Vvvz8=', 'Content-Length': '0'} 2022-09-02T05:59:37.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5e14a9b7c60a6c-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.752 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.752 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.753 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.753 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:37.754 DEBUG:boto:Data: 2022-09-02T05:59:37.754 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.756 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.756 DEBUG:boto:Params: {} 2022-09-02T05:59:37.757 DEBUG:boto:Token: None 2022-09-02T05:59:37.757 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31 2022-09-02T05:59:37.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:de+CegS8VF3XLhUITNJQg5Vvvz8= 2022-09-02T05:59:37.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:de+CegS8VF3XLhUITNJQg5Vvvz8=', 'Content-Length': '0'} 2022-09-02T05:59:37.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a7c8a6b22fdc60-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:37.763 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.763 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.764 DEBUG:boto:Method: GET 2022-09-02T05:59:37.764 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:37.765 DEBUG:boto:Data: 2022-09-02T05:59:37.765 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.767 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.767 DEBUG:boto:Params: {} 2022-09-02T05:59:37.768 DEBUG:boto:Token: None 2022-09-02T05:59:37.769 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31 2022-09-02T05:59:37.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yIT/ubOUVJ/lMKvHN3lnG5QmSfs= 2022-09-02T05:59:37.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yIT/ubOUVJ/lMKvHN3lnG5QmSfs=', 'Content-Length': '0'} 2022-09-02T05:59:37.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f0e62e076cbe89-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.773 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:37.774 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:37.774 DEBUG:boto:Method: GET 2022-09-02T05:59:37.775 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:37.776 DEBUG:boto:Data: 2022-09-02T05:59:37.777 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.778 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.778 DEBUG:boto:Params: {} 2022-09-02T05:59:37.779 DEBUG:boto:Token: None 2022-09-02T05:59:37.779 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj31 2022-09-02T05:59:37.780 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yIT/ubOUVJ/lMKvHN3lnG5QmSfs= 2022-09-02T05:59:37.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yIT/ubOUVJ/lMKvHN3lnG5QmSfs=', 'Content-Length': '0'} 2022-09-02T05:59:37.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000373f9121ffa515cb-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.784 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.785 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.785 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.786 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.786 DEBUG:boto:Method: GET 2022-09-02T05:59:37.787 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.787 DEBUG:boto:Data: 2022-09-02T05:59:37.788 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.789 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.790 DEBUG:boto:Params: {} 2022-09-02T05:59:37.790 DEBUG:boto:Token: None 2022-09-02T05:59:37.791 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7bcd1a1fc0a8b6-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.795 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.796 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.797 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.797 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.798 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.798 DEBUG:boto:Method: GET 2022-09-02T05:59:37.799 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:37.799 DEBUG:boto:Data: 2022-09-02T05:59:37.800 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.802 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.802 DEBUG:boto:Params: {} 2022-09-02T05:59:37.803 DEBUG:boto:Token: None 2022-09-02T05:59:37.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000925adc5ecd0d844-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.810 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:37.811 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.812 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.812 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.813 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.813 DEBUG:boto:Method: GET 2022-09-02T05:59:37.814 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.814 DEBUG:boto:Data: 2022-09-02T05:59:37.815 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.816 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.816 DEBUG:boto:Params: {} 2022-09-02T05:59:37.817 DEBUG:boto:Token: None 2022-09-02T05:59:37.818 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.819 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc= 2022-09-02T05:59:37.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc=', 'Content-Length': '0'} 2022-09-02T05:59:37.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f20f2e9fbd5afdf-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.862 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.863 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.863 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.864 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.864 DEBUG:boto:Method: GET 2022-09-02T05:59:37.865 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.866 DEBUG:boto:Data: 2022-09-02T05:59:37.866 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.867 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.868 DEBUG:boto:Params: {} 2022-09-02T05:59:37.868 DEBUG:boto:Token: None 2022-09-02T05:59:37.869 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc= 2022-09-02T05:59:37.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc=', 'Content-Length': '0'} 2022-09-02T05:59:37.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8486ba334074683-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.873 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.874 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.874 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.875 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.875 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.876 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.876 DEBUG:boto:Data: 2022-09-02T05:59:37.877 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.878 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.878 DEBUG:boto:Params: {} 2022-09-02T05:59:37.879 DEBUG:boto:Token: None 2022-09-02T05:59:37.879 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/KRiOFytEVRXabVybduD6pvod4w= 2022-09-02T05:59:37.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/KRiOFytEVRXabVybduD6pvod4w=', 'Content-Length': '0'} 2022-09-02T05:59:37.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6eac59962fd21a0-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.884 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.884 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.885 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.885 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.886 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.886 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.887 DEBUG:boto:Data: 2022-09-02T05:59:37.887 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.889 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.889 DEBUG:boto:Params: {} 2022-09-02T05:59:37.890 DEBUG:boto:Token: None 2022-09-02T05:59:37.890 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/KRiOFytEVRXabVybduD6pvod4w= 2022-09-02T05:59:37.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/KRiOFytEVRXabVybduD6pvod4w=', 'Content-Length': '0'} 2022-09-02T05:59:37.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3fd1fe2ae7f787e-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:37.895 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.895 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.896 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.896 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.897 DEBUG:boto:Method: GET 2022-09-02T05:59:37.897 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.898 DEBUG:boto:Data: 2022-09-02T05:59:37.899 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.900 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.900 DEBUG:boto:Params: {} 2022-09-02T05:59:37.901 DEBUG:boto:Token: None 2022-09-02T05:59:37.901 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc= 2022-09-02T05:59:37.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc=', 'Content-Length': '0'} 2022-09-02T05:59:37.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a95ac7ca689eb2e-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.905 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.906 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.907 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.907 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.908 DEBUG:boto:Method: GET 2022-09-02T05:59:37.908 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.909 DEBUG:boto:Data: 2022-09-02T05:59:37.909 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.910 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.911 DEBUG:boto:Params: {} 2022-09-02T05:59:37.911 DEBUG:boto:Token: None 2022-09-02T05:59:37.912 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:37.913 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc= 2022-09-02T05:59:37.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:slTe9UAfIs/5gJazPDHVsNTWcPc=', 'Content-Length': '0'} 2022-09-02T05:59:37.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f4b3e1a6cbbe32-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.916 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.917 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.917 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:37.918 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:37.918 DEBUG:boto:Method: GET 2022-09-02T05:59:37.919 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:37.919 DEBUG:boto:Data: 2022-09-02T05:59:37.920 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.921 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.922 DEBUG:boto:Params: {} 2022-09-02T05:59:37.922 DEBUG:boto:Token: None 2022-09-02T05:59:37.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000206f3857c9fbb45b-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.927 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.928 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:37.929 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:37.929 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:37.930 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:37.930 DEBUG:boto:Method: GET 2022-09-02T05:59:37.931 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:37.931 DEBUG:boto:Data: 2022-09-02T05:59:37.932 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.933 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.934 DEBUG:boto:Params: {} 2022-09-02T05:59:37.934 DEBUG:boto:Token: None 2022-09-02T05:59:37.935 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/ 2022-09-02T05:59:37.936 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk= 2022-09-02T05:59:37.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Itw+7WYjaa6p9o/+EnmrU89vLCk=', 'Content-Length': '0'} 2022-09-02T05:59:37.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb95bf97da56f78a-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.941 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:37.942 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.943 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.943 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.944 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:37.944 DEBUG:boto:Data: 2022-09-02T05:59:37.945 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.946 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.946 DEBUG:boto:Params: {} 2022-09-02T05:59:37.947 DEBUG:boto:Token: None 2022-09-02T05:59:37.947 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33 2022-09-02T05:59:37.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LAvSQdPsPfBtzRdcQogfL+9VQeo= 2022-09-02T05:59:37.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LAvSQdPsPfBtzRdcQogfL+9VQeo=', 'Content-Length': '0'} 2022-09-02T05:59:37.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a375239eb1191c2b-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.952 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.952 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.953 DEBUG:boto:Method: HEAD 2022-09-02T05:59:37.953 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:37.954 DEBUG:boto:Data: 2022-09-02T05:59:37.954 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:37.956 DEBUG:boto:Port: 8000 2022-09-02T05:59:37.956 DEBUG:boto:Params: {} 2022-09-02T05:59:37.957 DEBUG:boto:Token: None 2022-09-02T05:59:37.957 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33 2022-09-02T05:59:37.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LAvSQdPsPfBtzRdcQogfL+9VQeo= 2022-09-02T05:59:37.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LAvSQdPsPfBtzRdcQogfL+9VQeo=', 'Content-Length': '0'} 2022-09-02T05:59:37.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000733db1d761d86d3e-0063119bc9-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:37.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:37.962 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:37.962 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:37.963 DEBUG:boto:Method: GET 2022-09-02T05:59:37.963 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:37.964 DEBUG:boto:Data: 2022-09-02T05:59:37.964 DEBUG:boto:Headers: {} 2022-09-02T05:59:37.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:37.965 DEBUG:boto:Port: 8001 2022-09-02T05:59:37.966 DEBUG:boto:Params: {} 2022-09-02T05:59:37.967 DEBUG:boto:Token: None 2022-09-02T05:59:37.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:37 GMT /xljbkf-74/obj33 2022-09-02T05:59:37.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zVQRML9n51oQLki46RkmwVSKYg8= 2022-09-02T05:59:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:37 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zVQRML9n51oQLki46RkmwVSKYg8=', 'Content-Length': '0'} 2022-09-02T05:59:37.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000766de1bc7669ce-0063119bc9-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:37 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.014 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:38.016 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:38.017 DEBUG:boto:Method: GET 2022-09-02T05:59:38.018 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:38.019 DEBUG:boto:Data: 2022-09-02T05:59:38.020 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.022 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.023 DEBUG:boto:Params: {} 2022-09-02T05:59:38.024 DEBUG:boto:Token: None 2022-09-02T05:59:38.025 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj33 2022-09-02T05:59:38.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:us34SuYFegI6dkOa7K5vtVhzpxk= 2022-09-02T05:59:38.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:us34SuYFegI6dkOa7K5vtVhzpxk=', 'Content-Length': '0'} 2022-09-02T05:59:38.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4af25583ac09024-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.029 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.030 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.030 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:38.031 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:38.032 DEBUG:boto:Method: GET 2022-09-02T05:59:38.032 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:38.033 DEBUG:boto:Data: 2022-09-02T05:59:38.034 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.036 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.037 DEBUG:boto:Params: {} 2022-09-02T05:59:38.038 DEBUG:boto:Token: None 2022-09-02T05:59:38.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008227be0275ba439f-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.044 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.044 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.045 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.046 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:38.046 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:38.047 DEBUG:boto:Method: GET 2022-09-02T05:59:38.047 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:38.048 DEBUG:boto:Data: 2022-09-02T05:59:38.048 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.049 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.050 DEBUG:boto:Params: {} 2022-09-02T05:59:38.051 DEBUG:boto:Token: None 2022-09-02T05:59:38.051 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.052 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020e13cdd434153e7-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.058 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:38.059 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.060 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.060 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.061 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.061 DEBUG:boto:Method: GET 2022-09-02T05:59:38.062 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.063 DEBUG:boto:Data: 2022-09-02T05:59:38.063 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.064 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.065 DEBUG:boto:Params: {} 2022-09-02T05:59:38.065 DEBUG:boto:Token: None 2022-09-02T05:59:38.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo= 2022-09-02T05:59:38.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo=', 'Content-Length': '0'} 2022-09-02T05:59:38.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000925f524d16ad92f0-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.070 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.071 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.072 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.072 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.073 DEBUG:boto:Method: GET 2022-09-02T05:59:38.073 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.074 DEBUG:boto:Data: 2022-09-02T05:59:38.075 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.076 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.076 DEBUG:boto:Params: {} 2022-09-02T05:59:38.077 DEBUG:boto:Token: None 2022-09-02T05:59:38.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo= 2022-09-02T05:59:38.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo=', 'Content-Length': '0'} 2022-09-02T05:59:38.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b839306c84572e5-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.122 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.123 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.124 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.124 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.125 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.125 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.126 DEBUG:boto:Data: 2022-09-02T05:59:38.126 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.127 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.128 DEBUG:boto:Params: {} 2022-09-02T05:59:38.129 DEBUG:boto:Token: None 2022-09-02T05:59:38.129 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nv6998Yznr5wm7Oq9t8wSX1oIOU= 2022-09-02T05:59:38.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nv6998Yznr5wm7Oq9t8wSX1oIOU=', 'Content-Length': '0'} 2022-09-02T05:59:38.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009572c9c0706e37fb-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.134 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.134 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.135 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.135 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.136 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.136 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.137 DEBUG:boto:Data: 2022-09-02T05:59:38.137 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.139 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.140 DEBUG:boto:Params: {} 2022-09-02T05:59:38.140 DEBUG:boto:Token: None 2022-09-02T05:59:38.141 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nv6998Yznr5wm7Oq9t8wSX1oIOU= 2022-09-02T05:59:38.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nv6998Yznr5wm7Oq9t8wSX1oIOU=', 'Content-Length': '0'} 2022-09-02T05:59:38.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1416b2fb6c50d3c-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:38.145 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.146 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.147 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.147 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.148 DEBUG:boto:Method: GET 2022-09-02T05:59:38.148 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.149 DEBUG:boto:Data: 2022-09-02T05:59:38.149 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.151 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.151 DEBUG:boto:Params: {} 2022-09-02T05:59:38.152 DEBUG:boto:Token: None 2022-09-02T05:59:38.152 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.153 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo= 2022-09-02T05:59:38.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo=', 'Content-Length': '0'} 2022-09-02T05:59:38.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e747c0649248473-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.156 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.157 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.157 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.158 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.159 DEBUG:boto:Method: GET 2022-09-02T05:59:38.159 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.160 DEBUG:boto:Data: 2022-09-02T05:59:38.161 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.162 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.162 DEBUG:boto:Params: {} 2022-09-02T05:59:38.163 DEBUG:boto:Token: None 2022-09-02T05:59:38.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:38.164 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo= 2022-09-02T05:59:38.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2YIu1+wT5QSCPq6Q40GZzcbwHZo=', 'Content-Length': '0'} 2022-09-02T05:59:38.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009daa6a28c798181e-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.168 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.168 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.169 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.169 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.170 DEBUG:boto:Method: GET 2022-09-02T05:59:38.170 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.171 DEBUG:boto:Data: 2022-09-02T05:59:38.172 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.173 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.173 DEBUG:boto:Params: {} 2022-09-02T05:59:38.174 DEBUG:boto:Token: None 2022-09-02T05:59:38.174 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000370be5507fc93687-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.179 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.180 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.181 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.181 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.182 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.183 DEBUG:boto:Method: GET 2022-09-02T05:59:38.183 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.184 DEBUG:boto:Data: 2022-09-02T05:59:38.184 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.185 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.186 DEBUG:boto:Params: {} 2022-09-02T05:59:38.186 DEBUG:boto:Token: None 2022-09-02T05:59:38.187 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.188 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e21147f834a422-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.193 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.194 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.195 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.195 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.196 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:38.196 DEBUG:boto:Data: 2022-09-02T05:59:38.197 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.198 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.198 DEBUG:boto:Params: {} 2022-09-02T05:59:38.199 DEBUG:boto:Token: None 2022-09-02T05:59:38.200 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34 2022-09-02T05:59:38.200 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6uvLJhQr9zRQzG88ZkodYv1FS98= 2022-09-02T05:59:38.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6uvLJhQr9zRQzG88ZkodYv1FS98=', 'Content-Length': '0'} 2022-09-02T05:59:38.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002962e18d90014926-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.204 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.204 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.205 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.205 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:38.206 DEBUG:boto:Data: 2022-09-02T05:59:38.206 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.207 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.208 DEBUG:boto:Params: {} 2022-09-02T05:59:38.208 DEBUG:boto:Token: None 2022-09-02T05:59:38.209 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34 2022-09-02T05:59:38.209 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6uvLJhQr9zRQzG88ZkodYv1FS98= 2022-09-02T05:59:38.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6uvLJhQr9zRQzG88ZkodYv1FS98=', 'Content-Length': '0'} 2022-09-02T05:59:38.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae855554ad3d8e0-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:38.213 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.214 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.214 DEBUG:boto:Method: GET 2022-09-02T05:59:38.215 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:38.215 DEBUG:boto:Data: 2022-09-02T05:59:38.216 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.217 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.218 DEBUG:boto:Params: {} 2022-09-02T05:59:38.218 DEBUG:boto:Token: None 2022-09-02T05:59:38.219 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34 2022-09-02T05:59:38.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8dCOVUX/k8Y/5rCLPxFBDTtxXY= 2022-09-02T05:59:38.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8dCOVUX/k8Y/5rCLPxFBDTtxXY=', 'Content-Length': '0'} 2022-09-02T05:59:38.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008547b19e369f85ec-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.266 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:38.267 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:38.268 DEBUG:boto:Method: GET 2022-09-02T05:59:38.270 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:38.271 DEBUG:boto:Data: 2022-09-02T05:59:38.272 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.275 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.275 DEBUG:boto:Params: {} 2022-09-02T05:59:38.276 DEBUG:boto:Token: None 2022-09-02T05:59:38.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj34 2022-09-02T05:59:38.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b8dCOVUX/k8Y/5rCLPxFBDTtxXY= 2022-09-02T05:59:38.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b8dCOVUX/k8Y/5rCLPxFBDTtxXY=', 'Content-Length': '0'} 2022-09-02T05:59:38.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001efa4a1f0847ed08-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.281 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.282 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.282 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.283 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.283 DEBUG:boto:Method: GET 2022-09-02T05:59:38.284 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.284 DEBUG:boto:Data: 2022-09-02T05:59:38.285 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.286 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.286 DEBUG:boto:Params: {} 2022-09-02T05:59:38.287 DEBUG:boto:Token: None 2022-09-02T05:59:38.288 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002380f88942769a50-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.293 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.293 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.294 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.294 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.295 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.296 DEBUG:boto:Method: GET 2022-09-02T05:59:38.296 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:38.297 DEBUG:boto:Data: 2022-09-02T05:59:38.298 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.299 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.299 DEBUG:boto:Params: {} 2022-09-02T05:59:38.300 DEBUG:boto:Token: None 2022-09-02T05:59:38.301 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008821bd359d956924-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.307 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:38.309 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.309 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.310 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.310 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.311 DEBUG:boto:Method: GET 2022-09-02T05:59:38.311 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.312 DEBUG:boto:Data: 2022-09-02T05:59:38.312 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.313 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.314 DEBUG:boto:Params: {} 2022-09-02T05:59:38.315 DEBUG:boto:Token: None 2022-09-02T05:59:38.315 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.316 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc= 2022-09-02T05:59:38.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc=', 'Content-Length': '0'} 2022-09-02T05:59:38.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f64b3c3db1b0d92-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.320 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.320 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.321 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.321 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.322 DEBUG:boto:Method: GET 2022-09-02T05:59:38.322 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.323 DEBUG:boto:Data: 2022-09-02T05:59:38.324 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.325 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.325 DEBUG:boto:Params: {} 2022-09-02T05:59:38.326 DEBUG:boto:Token: None 2022-09-02T05:59:38.326 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.327 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc= 2022-09-02T05:59:38.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc=', 'Content-Length': '0'} 2022-09-02T05:59:38.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630807f9bc68a871-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.332 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.332 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.333 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.333 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.334 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.335 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.335 DEBUG:boto:Data: 2022-09-02T05:59:38.336 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.338 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.339 DEBUG:boto:Params: {} 2022-09-02T05:59:38.340 DEBUG:boto:Token: None 2022-09-02T05:59:38.341 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mcWdxY60MglLsewyQTaTgT3ZtsM= 2022-09-02T05:59:38.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mcWdxY60MglLsewyQTaTgT3ZtsM=', 'Content-Length': '0'} 2022-09-02T05:59:38.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002302140a2edb9018-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.348 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.348 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.350 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.351 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.352 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.353 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.354 DEBUG:boto:Data: 2022-09-02T05:59:38.355 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.357 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.358 DEBUG:boto:Params: {} 2022-09-02T05:59:38.359 DEBUG:boto:Token: None 2022-09-02T05:59:38.360 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mcWdxY60MglLsewyQTaTgT3ZtsM= 2022-09-02T05:59:38.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mcWdxY60MglLsewyQTaTgT3ZtsM=', 'Content-Length': '0'} 2022-09-02T05:59:38.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc7bb0b4673b577-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:38.368 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.369 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.370 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.371 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.372 DEBUG:boto:Method: GET 2022-09-02T05:59:38.373 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.374 DEBUG:boto:Data: 2022-09-02T05:59:38.375 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.377 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.379 DEBUG:boto:Params: {} 2022-09-02T05:59:38.380 DEBUG:boto:Token: None 2022-09-02T05:59:38.381 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.382 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc= 2022-09-02T05:59:38.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc=', 'Content-Length': '0'} 2022-09-02T05:59:38.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900d7e138b6ccb70-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.430 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.431 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.431 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.432 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.432 DEBUG:boto:Method: GET 2022-09-02T05:59:38.433 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.433 DEBUG:boto:Data: 2022-09-02T05:59:38.434 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.435 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.436 DEBUG:boto:Params: {} 2022-09-02T05:59:38.436 DEBUG:boto:Token: None 2022-09-02T05:59:38.437 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:38.438 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc= 2022-09-02T05:59:38.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EUkSp+75lvXm/s6h5SvR2zPb/cc=', 'Content-Length': '0'} 2022-09-02T05:59:38.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000462bb058a3df91a7-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.441 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.442 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.443 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.443 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.444 DEBUG:boto:Method: GET 2022-09-02T05:59:38.444 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.445 DEBUG:boto:Data: 2022-09-02T05:59:38.445 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.446 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.447 DEBUG:boto:Params: {} 2022-09-02T05:59:38.448 DEBUG:boto:Token: None 2022-09-02T05:59:38.449 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48282e676b99711-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.453 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.454 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.455 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.456 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.456 DEBUG:boto:Method: GET 2022-09-02T05:59:38.457 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.457 DEBUG:boto:Data: 2022-09-02T05:59:38.458 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.459 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.460 DEBUG:boto:Params: {} 2022-09-02T05:59:38.461 DEBUG:boto:Token: None 2022-09-02T05:59:38.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000818d3757b6261638-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.468 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.469 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.469 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.470 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.471 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:38.472 DEBUG:boto:Data: 2022-09-02T05:59:38.472 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.473 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.474 DEBUG:boto:Params: {} 2022-09-02T05:59:38.474 DEBUG:boto:Token: None 2022-09-02T05:59:38.475 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37 2022-09-02T05:59:38.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8n40TyL5CKnf5BxK8ToYQxQ3qZE= 2022-09-02T05:59:38.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8n40TyL5CKnf5BxK8ToYQxQ3qZE=', 'Content-Length': '0'} 2022-09-02T05:59:38.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccaa63b25a488b85-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.479 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.479 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.480 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.480 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:38.481 DEBUG:boto:Data: 2022-09-02T05:59:38.482 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.483 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.484 DEBUG:boto:Params: {} 2022-09-02T05:59:38.484 DEBUG:boto:Token: None 2022-09-02T05:59:38.485 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37 2022-09-02T05:59:38.486 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8n40TyL5CKnf5BxK8ToYQxQ3qZE= 2022-09-02T05:59:38.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8n40TyL5CKnf5BxK8ToYQxQ3qZE=', 'Content-Length': '0'} 2022-09-02T05:59:38.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c43df5ed6ec4b749-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:38.490 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.491 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.491 DEBUG:boto:Method: GET 2022-09-02T05:59:38.492 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:38.493 DEBUG:boto:Data: 2022-09-02T05:59:38.493 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.494 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.495 DEBUG:boto:Params: {} 2022-09-02T05:59:38.496 DEBUG:boto:Token: None 2022-09-02T05:59:38.496 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37 2022-09-02T05:59:38.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vo5Sm7H6Dyor2GQ9MtF02aGFIf0= 2022-09-02T05:59:38.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vo5Sm7H6Dyor2GQ9MtF02aGFIf0=', 'Content-Length': '0'} 2022-09-02T05:59:38.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0ad05c6fc1e960-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.501 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:38.501 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:38.502 DEBUG:boto:Method: GET 2022-09-02T05:59:38.502 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:38.503 DEBUG:boto:Data: 2022-09-02T05:59:38.503 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.504 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.505 DEBUG:boto:Params: {} 2022-09-02T05:59:38.506 DEBUG:boto:Token: None 2022-09-02T05:59:38.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj37 2022-09-02T05:59:38.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Vo5Sm7H6Dyor2GQ9MtF02aGFIf0= 2022-09-02T05:59:38.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Vo5Sm7H6Dyor2GQ9MtF02aGFIf0=', 'Content-Length': '0'} 2022-09-02T05:59:38.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7557c0c3d5e1f4-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.554 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.555 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.555 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.556 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.556 DEBUG:boto:Method: GET 2022-09-02T05:59:38.557 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.558 DEBUG:boto:Data: 2022-09-02T05:59:38.558 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.560 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.560 DEBUG:boto:Params: {} 2022-09-02T05:59:38.561 DEBUG:boto:Token: None 2022-09-02T05:59:38.561 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354f9b1ba562a26d-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.566 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.567 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.568 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.568 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.569 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.569 DEBUG:boto:Method: GET 2022-09-02T05:59:38.570 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:38.571 DEBUG:boto:Data: 2022-09-02T05:59:38.571 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.572 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.573 DEBUG:boto:Params: {} 2022-09-02T05:59:38.573 DEBUG:boto:Token: None 2022-09-02T05:59:38.574 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9ee83d8ac8f81c-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.581 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:38.584 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.585 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.586 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.587 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.588 DEBUG:boto:Method: GET 2022-09-02T05:59:38.590 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.591 DEBUG:boto:Data: 2022-09-02T05:59:38.592 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.594 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.595 DEBUG:boto:Params: {} 2022-09-02T05:59:38.597 DEBUG:boto:Token: None 2022-09-02T05:59:38.598 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.599 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M= 2022-09-02T05:59:38.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M=', 'Content-Length': '0'} 2022-09-02T05:59:38.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff25d9d01f87383-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.604 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.605 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.606 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.607 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.608 DEBUG:boto:Method: GET 2022-09-02T05:59:38.609 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.610 DEBUG:boto:Data: 2022-09-02T05:59:38.611 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.613 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.614 DEBUG:boto:Params: {} 2022-09-02T05:59:38.615 DEBUG:boto:Token: None 2022-09-02T05:59:38.616 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M= 2022-09-02T05:59:38.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M=', 'Content-Length': '0'} 2022-09-02T05:59:38.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a076d520b16beead-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.623 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.624 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.625 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.626 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.627 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.628 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.629 DEBUG:boto:Data: 2022-09-02T05:59:38.630 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.632 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.632 DEBUG:boto:Params: {} 2022-09-02T05:59:38.634 DEBUG:boto:Token: None 2022-09-02T05:59:38.635 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R9AdVMbta6lJJFk/4ctx4YxgEwY= 2022-09-02T05:59:38.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R9AdVMbta6lJJFk/4ctx4YxgEwY=', 'Content-Length': '0'} 2022-09-02T05:59:38.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005985c7dbd5f2bc04-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.641 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.642 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.643 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.644 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.645 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.646 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.646 DEBUG:boto:Data: 2022-09-02T05:59:38.647 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.649 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.650 DEBUG:boto:Params: {} 2022-09-02T05:59:38.651 DEBUG:boto:Token: None 2022-09-02T05:59:38.652 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.653 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:R9AdVMbta6lJJFk/4ctx4YxgEwY= 2022-09-02T05:59:38.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:R9AdVMbta6lJJFk/4ctx4YxgEwY=', 'Content-Length': '0'} 2022-09-02T05:59:38.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006794f7f32bf2b7bb-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:38.658 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.659 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.659 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.660 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.661 DEBUG:boto:Method: GET 2022-09-02T05:59:38.661 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.662 DEBUG:boto:Data: 2022-09-02T05:59:38.662 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.663 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.664 DEBUG:boto:Params: {} 2022-09-02T05:59:38.664 DEBUG:boto:Token: None 2022-09-02T05:59:38.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M= 2022-09-02T05:59:38.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M=', 'Content-Length': '0'} 2022-09-02T05:59:38.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2285923c54a4667-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.710 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.711 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.712 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.713 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.722 DEBUG:boto:Method: GET 2022-09-02T05:59:38.724 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.725 DEBUG:boto:Data: 2022-09-02T05:59:38.726 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.727 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.728 DEBUG:boto:Params: {} 2022-09-02T05:59:38.730 DEBUG:boto:Token: None 2022-09-02T05:59:38.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:38.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M= 2022-09-02T05:59:38.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cRVKWZjieKcTgty0IquOYoE369M=', 'Content-Length': '0'} 2022-09-02T05:59:38.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000486f49c2e2eb6c43-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.737 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.738 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.739 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.740 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.741 DEBUG:boto:Method: GET 2022-09-02T05:59:38.742 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.743 DEBUG:boto:Data: 2022-09-02T05:59:38.744 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.746 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.747 DEBUG:boto:Params: {} 2022-09-02T05:59:38.748 DEBUG:boto:Token: None 2022-09-02T05:59:38.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebb7917d129b0b4-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.756 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.758 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.759 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.760 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.761 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.762 DEBUG:boto:Method: GET 2022-09-02T05:59:38.763 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.763 DEBUG:boto:Data: 2022-09-02T05:59:38.764 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.765 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.766 DEBUG:boto:Params: {} 2022-09-02T05:59:38.767 DEBUG:boto:Token: None 2022-09-02T05:59:38.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce167e76953a95a2-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.774 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.775 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.775 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.776 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.777 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:38.777 DEBUG:boto:Data: 2022-09-02T05:59:38.778 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.779 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.780 DEBUG:boto:Params: {} 2022-09-02T05:59:38.780 DEBUG:boto:Token: None 2022-09-02T05:59:38.781 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38 2022-09-02T05:59:38.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nvxOcuPSUmDwirJE1BZLdtBBKnM= 2022-09-02T05:59:38.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nvxOcuPSUmDwirJE1BZLdtBBKnM=', 'Content-Length': '0'} 2022-09-02T05:59:38.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3ebdac2b322d1b4-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.785 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.786 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.786 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.787 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:38.788 DEBUG:boto:Data: 2022-09-02T05:59:38.788 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.789 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.790 DEBUG:boto:Params: {} 2022-09-02T05:59:38.790 DEBUG:boto:Token: None 2022-09-02T05:59:38.791 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38 2022-09-02T05:59:38.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nvxOcuPSUmDwirJE1BZLdtBBKnM= 2022-09-02T05:59:38.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nvxOcuPSUmDwirJE1BZLdtBBKnM=', 'Content-Length': '0'} 2022-09-02T05:59:38.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b1c6934c00d9a50-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:38.795 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.796 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.797 DEBUG:boto:Method: GET 2022-09-02T05:59:38.797 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:38.798 DEBUG:boto:Data: 2022-09-02T05:59:38.798 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.800 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.800 DEBUG:boto:Params: {} 2022-09-02T05:59:38.801 DEBUG:boto:Token: None 2022-09-02T05:59:38.801 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38 2022-09-02T05:59:38.802 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xWGWs/0mo9LMBpl6iJ7vuZtJ/M8= 2022-09-02T05:59:38.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xWGWs/0mo9LMBpl6iJ7vuZtJ/M8=', 'Content-Length': '0'} 2022-09-02T05:59:38.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2786875344f9a4-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.805 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:38.806 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:38.807 DEBUG:boto:Method: GET 2022-09-02T05:59:38.807 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:38.808 DEBUG:boto:Data: 2022-09-02T05:59:38.809 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.810 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.810 DEBUG:boto:Params: {} 2022-09-02T05:59:38.811 DEBUG:boto:Token: None 2022-09-02T05:59:38.811 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj38 2022-09-02T05:59:38.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xWGWs/0mo9LMBpl6iJ7vuZtJ/M8= 2022-09-02T05:59:38.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xWGWs/0mo9LMBpl6iJ7vuZtJ/M8=', 'Content-Length': '0'} 2022-09-02T05:59:38.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5f3b754cf7c613-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.816 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.816 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.817 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.818 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.818 DEBUG:boto:Method: GET 2022-09-02T05:59:38.819 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.819 DEBUG:boto:Data: 2022-09-02T05:59:38.820 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.821 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.821 DEBUG:boto:Params: {} 2022-09-02T05:59:38.822 DEBUG:boto:Token: None 2022-09-02T05:59:38.823 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb5c067a200a047-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.827 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.828 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.829 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.829 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.830 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.830 DEBUG:boto:Method: GET 2022-09-02T05:59:38.831 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:38.831 DEBUG:boto:Data: 2022-09-02T05:59:38.832 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.833 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.833 DEBUG:boto:Params: {} 2022-09-02T05:59:38.834 DEBUG:boto:Token: None 2022-09-02T05:59:38.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079cc24caabd848db-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.886 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:38.888 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:38.888 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:38.889 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.889 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.890 DEBUG:boto:Method: GET 2022-09-02T05:59:38.891 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.891 DEBUG:boto:Data: 2022-09-02T05:59:38.892 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.893 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.893 DEBUG:boto:Params: {} 2022-09-02T05:59:38.894 DEBUG:boto:Token: None 2022-09-02T05:59:38.895 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU= 2022-09-02T05:59:38.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU=', 'Content-Length': '0'} 2022-09-02T05:59:38.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e6bb505e8dc1c9-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.901 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:38.902 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:38.903 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.903 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.904 DEBUG:boto:Method: GET 2022-09-02T05:59:38.905 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.906 DEBUG:boto:Data: 2022-09-02T05:59:38.906 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.907 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.908 DEBUG:boto:Params: {} 2022-09-02T05:59:38.909 DEBUG:boto:Token: None 2022-09-02T05:59:38.909 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.910 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU= 2022-09-02T05:59:38.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU=', 'Content-Length': '0'} 2022-09-02T05:59:38.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939c8ada4650559d-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.914 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:38.914 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:38.915 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.915 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.916 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.916 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.917 DEBUG:boto:Data: 2022-09-02T05:59:38.918 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.919 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.919 DEBUG:boto:Params: {} 2022-09-02T05:59:38.920 DEBUG:boto:Token: None 2022-09-02T05:59:38.920 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.921 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wr98/tHdysMKuFceqCXVJ/0SgH4= 2022-09-02T05:59:38.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wr98/tHdysMKuFceqCXVJ/0SgH4=', 'Content-Length': '0'} 2022-09-02T05:59:38.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001432349f0770e953-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.925 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:38.925 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:38.926 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.926 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.927 DEBUG:boto:Method: HEAD 2022-09-02T05:59:38.927 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.928 DEBUG:boto:Data: 2022-09-02T05:59:38.928 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.929 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.930 DEBUG:boto:Params: {} 2022-09-02T05:59:38.930 DEBUG:boto:Token: None 2022-09-02T05:59:38.931 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wr98/tHdysMKuFceqCXVJ/0SgH4= 2022-09-02T05:59:38.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wr98/tHdysMKuFceqCXVJ/0SgH4=', 'Content-Length': '0'} 2022-09-02T05:59:38.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a617d8a03a094e-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:38.936 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:38.936 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:38.937 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.938 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.938 DEBUG:boto:Method: GET 2022-09-02T05:59:38.939 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.939 DEBUG:boto:Data: 2022-09-02T05:59:38.940 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.941 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.942 DEBUG:boto:Params: {} 2022-09-02T05:59:38.942 DEBUG:boto:Token: None 2022-09-02T05:59:38.943 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.944 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU= 2022-09-02T05:59:38.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU=', 'Content-Length': '0'} 2022-09-02T05:59:38.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb3b06f333c2b00-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.948 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:38.949 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:38.949 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.950 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.950 DEBUG:boto:Method: GET 2022-09-02T05:59:38.951 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.951 DEBUG:boto:Data: 2022-09-02T05:59:38.952 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.953 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.954 DEBUG:boto:Params: {} 2022-09-02T05:59:38.954 DEBUG:boto:Token: None 2022-09-02T05:59:38.955 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:38.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU= 2022-09-02T05:59:38.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1i0qF9PvHL6bUfNsQETw0D2v9xU=', 'Content-Length': '0'} 2022-09-02T05:59:38.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093572e3db3df7a99-0063119bca-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.960 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.960 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.961 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:38.962 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:38.962 DEBUG:boto:Method: GET 2022-09-02T05:59:38.963 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:38.963 DEBUG:boto:Data: 2022-09-02T05:59:38.964 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:38.965 DEBUG:boto:Port: 8001 2022-09-02T05:59:38.966 DEBUG:boto:Params: {} 2022-09-02T05:59:38.966 DEBUG:boto:Token: None 2022-09-02T05:59:38.967 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.968 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a7c24298e7e250-0063119bca-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:38.972 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:38.973 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:38.973 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:38.974 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:38.974 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:38.975 DEBUG:boto:Method: GET 2022-09-02T05:59:38.975 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:38.976 DEBUG:boto:Data: 2022-09-02T05:59:38.976 DEBUG:boto:Headers: {} 2022-09-02T05:59:38.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:38.977 DEBUG:boto:Port: 8000 2022-09-02T05:59:38.978 DEBUG:boto:Params: {} 2022-09-02T05:59:38.979 DEBUG:boto:Token: None 2022-09-02T05:59:38.979 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:38 GMT /xljbkf-74/ 2022-09-02T05:59:38.980 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ= 2022-09-02T05:59:38.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+WNQ5jhuP2YnKz6TSAHhFHLvnAQ=', 'Content-Length': '0'} 2022-09-02T05:59:38.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001107acffd9bf551e-0063119bca-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.030 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.032 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:39.033 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:39.033 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.034 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:39.034 DEBUG:boto:Data: 2022-09-02T05:59:39.035 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.036 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.036 DEBUG:boto:Params: {} 2022-09-02T05:59:39.037 DEBUG:boto:Token: None 2022-09-02T05:59:39.037 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj44 2022-09-02T05:59:39.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wekdssLq4G7iQatjBusALNRHgU8= 2022-09-02T05:59:39.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wekdssLq4G7iQatjBusALNRHgU8=', 'Content-Length': '0'} 2022-09-02T05:59:39.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3f00ac0790f396-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.041 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:39.042 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:39.042 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.043 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:39.043 DEBUG:boto:Data: 2022-09-02T05:59:39.044 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.045 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.045 DEBUG:boto:Params: {} 2022-09-02T05:59:39.046 DEBUG:boto:Token: None 2022-09-02T05:59:39.047 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj44 2022-09-02T05:59:39.047 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:wekdssLq4G7iQatjBusALNRHgU8= 2022-09-02T05:59:39.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:wekdssLq4G7iQatjBusALNRHgU8=', 'Content-Length': '0'} 2022-09-02T05:59:39.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a094f22fa2ff0222-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:39.051 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:39.052 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:39.052 DEBUG:boto:Method: GET 2022-09-02T05:59:39.053 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:39.053 DEBUG:boto:Data: 2022-09-02T05:59:39.054 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.055 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.055 DEBUG:boto:Params: {} 2022-09-02T05:59:39.056 DEBUG:boto:Token: None 2022-09-02T05:59:39.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj44 2022-09-02T05:59:39.057 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TUXmTTciiSDQDwN7RP6s2YsrMNk= 2022-09-02T05:59:39.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TUXmTTciiSDQDwN7RP6s2YsrMNk=', 'Content-Length': '0'} 2022-09-02T05:59:39.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001742245f91bbbb9c-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.061 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:39.061 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:39.062 DEBUG:boto:Method: GET 2022-09-02T05:59:39.063 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:39.063 DEBUG:boto:Data: 2022-09-02T05:59:39.064 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.065 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.065 DEBUG:boto:Params: {} 2022-09-02T05:59:39.066 DEBUG:boto:Token: None 2022-09-02T05:59:39.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj44 2022-09-02T05:59:39.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TUXmTTciiSDQDwN7RP6s2YsrMNk= 2022-09-02T05:59:39.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TUXmTTciiSDQDwN7RP6s2YsrMNk=', 'Content-Length': '0'} 2022-09-02T05:59:39.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c55fe32d1f97fd-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.071 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.071 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.072 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:39.072 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:39.073 DEBUG:boto:Method: GET 2022-09-02T05:59:39.073 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:39.074 DEBUG:boto:Data: 2022-09-02T05:59:39.075 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.076 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.076 DEBUG:boto:Params: {} 2022-09-02T05:59:39.077 DEBUG:boto:Token: None 2022-09-02T05:59:39.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.078 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c2ad6fdf0687abd-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.082 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.083 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.083 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.084 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:39.084 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:39.085 DEBUG:boto:Method: GET 2022-09-02T05:59:39.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:39.086 DEBUG:boto:Data: 2022-09-02T05:59:39.086 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.087 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.088 DEBUG:boto:Params: {} 2022-09-02T05:59:39.089 DEBUG:boto:Token: None 2022-09-02T05:59:39.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.090 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f8646c9b30bf181-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.096 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:39.097 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.098 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.098 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.099 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.099 DEBUG:boto:Method: GET 2022-09-02T05:59:39.100 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.100 DEBUG:boto:Data: 2022-09-02T05:59:39.101 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.102 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.103 DEBUG:boto:Params: {} 2022-09-02T05:59:39.103 DEBUG:boto:Token: None 2022-09-02T05:59:39.104 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.105 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0= 2022-09-02T05:59:39.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0=', 'Content-Length': '0'} 2022-09-02T05:59:39.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c7962ad58a27fe-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.108 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.109 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.109 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.110 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.110 DEBUG:boto:Method: GET 2022-09-02T05:59:39.111 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.111 DEBUG:boto:Data: 2022-09-02T05:59:39.112 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.113 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.113 DEBUG:boto:Params: {} 2022-09-02T05:59:39.114 DEBUG:boto:Token: None 2022-09-02T05:59:39.115 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.115 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0= 2022-09-02T05:59:39.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0=', 'Content-Length': '0'} 2022-09-02T05:59:39.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b27a2ad91b4031fc-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.162 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.163 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.163 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.164 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.164 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.165 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.165 DEBUG:boto:Data: 2022-09-02T05:59:39.166 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.167 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.167 DEBUG:boto:Params: {} 2022-09-02T05:59:39.168 DEBUG:boto:Token: None 2022-09-02T05:59:39.169 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uGY6vCiamokofdCPANd7Z9XInPM= 2022-09-02T05:59:39.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uGY6vCiamokofdCPANd7Z9XInPM=', 'Content-Length': '0'} 2022-09-02T05:59:39.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be415c82f13e9735-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.173 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.174 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.174 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.175 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.175 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.176 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.176 DEBUG:boto:Data: 2022-09-02T05:59:39.177 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.178 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.179 DEBUG:boto:Params: {} 2022-09-02T05:59:39.179 DEBUG:boto:Token: None 2022-09-02T05:59:39.180 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:uGY6vCiamokofdCPANd7Z9XInPM= 2022-09-02T05:59:39.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:uGY6vCiamokofdCPANd7Z9XInPM=', 'Content-Length': '0'} 2022-09-02T05:59:39.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000455d0c3eaa2ad34f-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:39.185 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.185 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.186 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.187 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.187 DEBUG:boto:Method: GET 2022-09-02T05:59:39.188 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.188 DEBUG:boto:Data: 2022-09-02T05:59:39.189 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.190 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.190 DEBUG:boto:Params: {} 2022-09-02T05:59:39.191 DEBUG:boto:Token: None 2022-09-02T05:59:39.192 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0= 2022-09-02T05:59:39.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0=', 'Content-Length': '0'} 2022-09-02T05:59:39.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6d78f7528df89a-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.196 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.197 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.197 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.198 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.199 DEBUG:boto:Method: GET 2022-09-02T05:59:39.199 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.200 DEBUG:boto:Data: 2022-09-02T05:59:39.200 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.201 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.202 DEBUG:boto:Params: {} 2022-09-02T05:59:39.203 DEBUG:boto:Token: None 2022-09-02T05:59:39.204 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:39.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0= 2022-09-02T05:59:39.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Jra8NHDnoJ+TQ2BIUAgBaGPBzD0=', 'Content-Length': '0'} 2022-09-02T05:59:39.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8b0fdb69514d78-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.210 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.211 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.212 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.212 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.213 DEBUG:boto:Method: GET 2022-09-02T05:59:39.214 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.214 DEBUG:boto:Data: 2022-09-02T05:59:39.215 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.216 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.216 DEBUG:boto:Params: {} 2022-09-02T05:59:39.217 DEBUG:boto:Token: None 2022-09-02T05:59:39.218 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.218 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d228e5c9ad6bf0cd-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.222 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.223 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.224 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.224 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.225 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.225 DEBUG:boto:Method: GET 2022-09-02T05:59:39.226 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.226 DEBUG:boto:Data: 2022-09-02T05:59:39.227 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.228 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.229 DEBUG:boto:Params: {} 2022-09-02T05:59:39.229 DEBUG:boto:Token: None 2022-09-02T05:59:39.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8686ff03f7b375d-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.327 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.330 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.331 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.332 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.334 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:39.334 DEBUG:boto:Data: 2022-09-02T05:59:39.335 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.336 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.337 DEBUG:boto:Params: {} 2022-09-02T05:59:39.337 DEBUG:boto:Token: None 2022-09-02T05:59:39.338 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46 2022-09-02T05:59:39.339 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4pKMyxz5+0Sur98M8buuLxtcDA= 2022-09-02T05:59:39.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4pKMyxz5+0Sur98M8buuLxtcDA=', 'Content-Length': '0'} 2022-09-02T05:59:39.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d4560b636c2b98-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.344 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.345 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.346 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.347 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:39.348 DEBUG:boto:Data: 2022-09-02T05:59:39.350 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.352 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.352 DEBUG:boto:Params: {} 2022-09-02T05:59:39.354 DEBUG:boto:Token: None 2022-09-02T05:59:39.355 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46 2022-09-02T05:59:39.356 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:E4pKMyxz5+0Sur98M8buuLxtcDA= 2022-09-02T05:59:39.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:E4pKMyxz5+0Sur98M8buuLxtcDA=', 'Content-Length': '0'} 2022-09-02T05:59:39.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057472cd226840c78-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:39.361 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.361 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.362 DEBUG:boto:Method: GET 2022-09-02T05:59:39.362 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:39.363 DEBUG:boto:Data: 2022-09-02T05:59:39.364 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.365 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.365 DEBUG:boto:Params: {} 2022-09-02T05:59:39.366 DEBUG:boto:Token: None 2022-09-02T05:59:39.367 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46 2022-09-02T05:59:39.368 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sVgr2u5JeHhBXmawf4WagjpAU6U= 2022-09-02T05:59:39.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sVgr2u5JeHhBXmawf4WagjpAU6U=', 'Content-Length': '0'} 2022-09-02T05:59:39.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c421e5500ca61b-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.375 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:39.376 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:39.376 DEBUG:boto:Method: GET 2022-09-02T05:59:39.377 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:39.377 DEBUG:boto:Data: 2022-09-02T05:59:39.378 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.380 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.380 DEBUG:boto:Params: {} 2022-09-02T05:59:39.381 DEBUG:boto:Token: None 2022-09-02T05:59:39.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj46 2022-09-02T05:59:39.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sVgr2u5JeHhBXmawf4WagjpAU6U= 2022-09-02T05:59:39.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sVgr2u5JeHhBXmawf4WagjpAU6U=', 'Content-Length': '0'} 2022-09-02T05:59:39.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e09c77c07e85d7ac-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.387 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.388 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.388 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.389 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.389 DEBUG:boto:Method: GET 2022-09-02T05:59:39.390 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.391 DEBUG:boto:Data: 2022-09-02T05:59:39.391 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.393 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.393 DEBUG:boto:Params: {} 2022-09-02T05:59:39.394 DEBUG:boto:Token: None 2022-09-02T05:59:39.395 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000302ac24a4dc8fdbd-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.442 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.444 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.444 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.445 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.445 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.446 DEBUG:boto:Method: GET 2022-09-02T05:59:39.446 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:39.447 DEBUG:boto:Data: 2022-09-02T05:59:39.448 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.449 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.449 DEBUG:boto:Params: {} 2022-09-02T05:59:39.450 DEBUG:boto:Token: None 2022-09-02T05:59:39.451 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b1c04a94c7c835d-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.457 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:39.459 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.459 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.460 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.460 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.461 DEBUG:boto:Method: GET 2022-09-02T05:59:39.461 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.462 DEBUG:boto:Data: 2022-09-02T05:59:39.463 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.464 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.464 DEBUG:boto:Params: {} 2022-09-02T05:59:39.465 DEBUG:boto:Token: None 2022-09-02T05:59:39.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0= 2022-09-02T05:59:39.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0=', 'Content-Length': '0'} 2022-09-02T05:59:39.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2dd5e195b26b278-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.472 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.472 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.473 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.473 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.474 DEBUG:boto:Method: GET 2022-09-02T05:59:39.475 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.475 DEBUG:boto:Data: 2022-09-02T05:59:39.476 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.477 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.478 DEBUG:boto:Params: {} 2022-09-02T05:59:39.478 DEBUG:boto:Token: None 2022-09-02T05:59:39.479 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.480 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0= 2022-09-02T05:59:39.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0=', 'Content-Length': '0'} 2022-09-02T05:59:39.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d793fe150bd9e09-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.484 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.485 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.486 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.486 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.487 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.488 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.488 DEBUG:boto:Data: 2022-09-02T05:59:39.489 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.490 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.490 DEBUG:boto:Params: {} 2022-09-02T05:59:39.491 DEBUG:boto:Token: None 2022-09-02T05:59:39.492 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RvLSsdbS+e7eokIjhDJ9aPgg2uo= 2022-09-02T05:59:39.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RvLSsdbS+e7eokIjhDJ9aPgg2uo=', 'Content-Length': '0'} 2022-09-02T05:59:39.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008674f1b9b5822194-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.496 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.496 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.497 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.497 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.498 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.499 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.500 DEBUG:boto:Data: 2022-09-02T05:59:39.501 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.502 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.503 DEBUG:boto:Params: {} 2022-09-02T05:59:39.504 DEBUG:boto:Token: None 2022-09-02T05:59:39.504 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.505 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RvLSsdbS+e7eokIjhDJ9aPgg2uo= 2022-09-02T05:59:39.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RvLSsdbS+e7eokIjhDJ9aPgg2uo=', 'Content-Length': '0'} 2022-09-02T05:59:39.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f029f363e3b04f63-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:39.510 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.511 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.511 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.512 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.512 DEBUG:boto:Method: GET 2022-09-02T05:59:39.513 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.513 DEBUG:boto:Data: 2022-09-02T05:59:39.514 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.515 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.516 DEBUG:boto:Params: {} 2022-09-02T05:59:39.516 DEBUG:boto:Token: None 2022-09-02T05:59:39.517 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0= 2022-09-02T05:59:39.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0=', 'Content-Length': '0'} 2022-09-02T05:59:39.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed83aa1a881e80ec-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.562 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.563 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.563 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.564 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.564 DEBUG:boto:Method: GET 2022-09-02T05:59:39.565 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.565 DEBUG:boto:Data: 2022-09-02T05:59:39.566 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.567 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.568 DEBUG:boto:Params: {} 2022-09-02T05:59:39.568 DEBUG:boto:Token: None 2022-09-02T05:59:39.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:39.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0= 2022-09-02T05:59:39.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NDK653gbSSf2NpKrF9BAAhCRuc0=', 'Content-Length': '0'} 2022-09-02T05:59:39.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4e10b06d1b439b0-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.574 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.574 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.575 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.575 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.576 DEBUG:boto:Method: GET 2022-09-02T05:59:39.576 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.577 DEBUG:boto:Data: 2022-09-02T05:59:39.577 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.578 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.579 DEBUG:boto:Params: {} 2022-09-02T05:59:39.580 DEBUG:boto:Token: None 2022-09-02T05:59:39.580 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041d84b60371933e2-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.585 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.586 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.586 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.587 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.587 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.588 DEBUG:boto:Method: GET 2022-09-02T05:59:39.588 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.589 DEBUG:boto:Data: 2022-09-02T05:59:39.589 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.590 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.591 DEBUG:boto:Params: {} 2022-09-02T05:59:39.591 DEBUG:boto:Token: None 2022-09-02T05:59:39.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000010a48440ff3f25f-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.600 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.601 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.602 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.603 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.604 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:39.605 DEBUG:boto:Data: 2022-09-02T05:59:39.606 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.608 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.609 DEBUG:boto:Params: {} 2022-09-02T05:59:39.610 DEBUG:boto:Token: None 2022-09-02T05:59:39.611 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51 2022-09-02T05:59:39.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zhhqdABnkTo8KqFJZcRQeg4dinI= 2022-09-02T05:59:39.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zhhqdABnkTo8KqFJZcRQeg4dinI=', 'Content-Length': '0'} 2022-09-02T05:59:39.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d046d02a710151e9-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.619 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.620 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.621 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.622 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:39.623 DEBUG:boto:Data: 2022-09-02T05:59:39.624 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.625 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.626 DEBUG:boto:Params: {} 2022-09-02T05:59:39.627 DEBUG:boto:Token: None 2022-09-02T05:59:39.629 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51 2022-09-02T05:59:39.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zhhqdABnkTo8KqFJZcRQeg4dinI= 2022-09-02T05:59:39.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zhhqdABnkTo8KqFJZcRQeg4dinI=', 'Content-Length': '0'} 2022-09-02T05:59:39.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725651cdb8d1cd12-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:39.636 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.637 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.639 DEBUG:boto:Method: GET 2022-09-02T05:59:39.640 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:39.641 DEBUG:boto:Data: 2022-09-02T05:59:39.642 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.645 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.646 DEBUG:boto:Params: {} 2022-09-02T05:59:39.646 DEBUG:boto:Token: None 2022-09-02T05:59:39.647 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51 2022-09-02T05:59:39.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UoODa36Bwk+s24TJHrnGFvdGY3w= 2022-09-02T05:59:39.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UoODa36Bwk+s24TJHrnGFvdGY3w=', 'Content-Length': '0'} 2022-09-02T05:59:39.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b27a6575394f38ba-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.652 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:39.652 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:39.653 DEBUG:boto:Method: GET 2022-09-02T05:59:39.654 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:39.655 DEBUG:boto:Data: 2022-09-02T05:59:39.656 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.657 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.658 DEBUG:boto:Params: {} 2022-09-02T05:59:39.658 DEBUG:boto:Token: None 2022-09-02T05:59:39.659 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj51 2022-09-02T05:59:39.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UoODa36Bwk+s24TJHrnGFvdGY3w= 2022-09-02T05:59:39.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UoODa36Bwk+s24TJHrnGFvdGY3w=', 'Content-Length': '0'} 2022-09-02T05:59:39.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ecc9ba83b017d7-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.664 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.664 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.665 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.666 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.667 DEBUG:boto:Method: GET 2022-09-02T05:59:39.667 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.668 DEBUG:boto:Data: 2022-09-02T05:59:39.669 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.671 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.671 DEBUG:boto:Params: {} 2022-09-02T05:59:39.672 DEBUG:boto:Token: None 2022-09-02T05:59:39.673 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.674 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea920781881132ad-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.679 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.680 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.681 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.681 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.682 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.683 DEBUG:boto:Method: GET 2022-09-02T05:59:39.684 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:39.684 DEBUG:boto:Data: 2022-09-02T05:59:39.685 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.687 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.687 DEBUG:boto:Params: {} 2022-09-02T05:59:39.688 DEBUG:boto:Token: None 2022-09-02T05:59:39.689 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.690 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030567b303a09e0c0-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.696 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:39.698 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.699 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.700 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.700 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.701 DEBUG:boto:Method: GET 2022-09-02T05:59:39.702 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.703 DEBUG:boto:Data: 2022-09-02T05:59:39.703 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.705 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.706 DEBUG:boto:Params: {} 2022-09-02T05:59:39.706 DEBUG:boto:Token: None 2022-09-02T05:59:39.707 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.708 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc= 2022-09-02T05:59:39.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc=', 'Content-Length': '0'} 2022-09-02T05:59:39.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d399c0e9582ca988-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.758 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.759 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.760 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.761 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.762 DEBUG:boto:Method: GET 2022-09-02T05:59:39.763 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.763 DEBUG:boto:Data: 2022-09-02T05:59:39.764 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.766 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.767 DEBUG:boto:Params: {} 2022-09-02T05:59:39.769 DEBUG:boto:Token: None 2022-09-02T05:59:39.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.771 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc= 2022-09-02T05:59:39.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc=', 'Content-Length': '0'} 2022-09-02T05:59:39.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca986d0c901c0b7-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.776 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.777 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.778 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.779 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.780 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.780 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.781 DEBUG:boto:Data: 2022-09-02T05:59:39.782 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.784 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.785 DEBUG:boto:Params: {} 2022-09-02T05:59:39.786 DEBUG:boto:Token: None 2022-09-02T05:59:39.787 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.788 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c5U7dWRg7+9zeH/4iP/s8YqaFoE= 2022-09-02T05:59:39.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c5U7dWRg7+9zeH/4iP/s8YqaFoE=', 'Content-Length': '0'} 2022-09-02T05:59:39.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ec1353f8224ef0-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.793 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.793 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.794 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.795 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.795 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.796 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.796 DEBUG:boto:Data: 2022-09-02T05:59:39.797 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.798 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.798 DEBUG:boto:Params: {} 2022-09-02T05:59:39.799 DEBUG:boto:Token: None 2022-09-02T05:59:39.799 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.800 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:c5U7dWRg7+9zeH/4iP/s8YqaFoE= 2022-09-02T05:59:39.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:c5U7dWRg7+9zeH/4iP/s8YqaFoE=', 'Content-Length': '0'} 2022-09-02T05:59:39.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098951220e3608897-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:39.804 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.805 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.806 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.806 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.807 DEBUG:boto:Method: GET 2022-09-02T05:59:39.808 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.808 DEBUG:boto:Data: 2022-09-02T05:59:39.809 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.810 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.810 DEBUG:boto:Params: {} 2022-09-02T05:59:39.811 DEBUG:boto:Token: None 2022-09-02T05:59:39.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.813 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc= 2022-09-02T05:59:39.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc=', 'Content-Length': '0'} 2022-09-02T05:59:39.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056041bd677702e54-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.825 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.826 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.827 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.827 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.828 DEBUG:boto:Method: GET 2022-09-02T05:59:39.828 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.829 DEBUG:boto:Data: 2022-09-02T05:59:39.830 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.832 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.833 DEBUG:boto:Params: {} 2022-09-02T05:59:39.835 DEBUG:boto:Token: None 2022-09-02T05:59:39.836 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:39.837 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc= 2022-09-02T05:59:39.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:00IezS9+v0+t4VOYiNqYbRjEdqc=', 'Content-Length': '0'} 2022-09-02T05:59:39.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84c348364e9d674-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.882 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.883 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.883 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:39.884 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:39.885 DEBUG:boto:Method: GET 2022-09-02T05:59:39.885 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:39.886 DEBUG:boto:Data: 2022-09-02T05:59:39.886 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.887 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.888 DEBUG:boto:Params: {} 2022-09-02T05:59:39.889 DEBUG:boto:Token: None 2022-09-02T05:59:39.890 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.891 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e901899d23e0df85-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.896 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.897 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:39.898 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:39.898 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:39.899 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:39.900 DEBUG:boto:Method: GET 2022-09-02T05:59:39.901 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:39.901 DEBUG:boto:Data: 2022-09-02T05:59:39.902 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.903 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.904 DEBUG:boto:Params: {} 2022-09-02T05:59:39.904 DEBUG:boto:Token: None 2022-09-02T05:59:39.905 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/ 2022-09-02T05:59:39.906 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I= 2022-09-02T05:59:39.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:V3yTmELtRAMjzWe+Rg4+QYxpB2I=', 'Content-Length': '0'} 2022-09-02T05:59:39.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a15d4a7d622e3f9-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.912 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:39.914 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.915 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.915 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.916 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:39.918 DEBUG:boto:Data: 2022-09-02T05:59:39.919 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.921 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.922 DEBUG:boto:Params: {} 2022-09-02T05:59:39.923 DEBUG:boto:Token: None 2022-09-02T05:59:39.924 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55 2022-09-02T05:59:39.925 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y5wztqu0TWzkX1hZI7XOCwgmvns= 2022-09-02T05:59:39.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y5wztqu0TWzkX1hZI7XOCwgmvns=', 'Content-Length': '0'} 2022-09-02T05:59:39.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb7679cf3097e67e-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.930 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.931 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.932 DEBUG:boto:Method: HEAD 2022-09-02T05:59:39.932 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:39.933 DEBUG:boto:Data: 2022-09-02T05:59:39.934 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:39.935 DEBUG:boto:Port: 8000 2022-09-02T05:59:39.936 DEBUG:boto:Params: {} 2022-09-02T05:59:39.936 DEBUG:boto:Token: None 2022-09-02T05:59:39.937 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55 2022-09-02T05:59:39.938 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Y5wztqu0TWzkX1hZI7XOCwgmvns= 2022-09-02T05:59:39.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Y5wztqu0TWzkX1hZI7XOCwgmvns=', 'Content-Length': '0'} 2022-09-02T05:59:39.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd1992ebffc768b-0063119bcb-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:39.942 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.942 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:39.943 DEBUG:boto:Method: GET 2022-09-02T05:59:39.943 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:39.944 DEBUG:boto:Data: 2022-09-02T05:59:39.944 DEBUG:boto:Headers: {} 2022-09-02T05:59:39.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:39.945 DEBUG:boto:Port: 8001 2022-09-02T05:59:39.946 DEBUG:boto:Params: {} 2022-09-02T05:59:39.947 DEBUG:boto:Token: None 2022-09-02T05:59:39.947 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:39 GMT /xljbkf-74/obj55 2022-09-02T05:59:39.948 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sKLqHVONuIwa483uR8Ta7ARjmGE= 2022-09-02T05:59:39.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:39 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sKLqHVONuIwa483uR8Ta7ARjmGE=', 'Content-Length': '0'} 2022-09-02T05:59:39.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bb55f03120e08a6-0063119bcb-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:39 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:39.998 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:39.999 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:40.000 DEBUG:boto:Method: GET 2022-09-02T05:59:40.001 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:40.001 DEBUG:boto:Data: 2022-09-02T05:59:40.002 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.003 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.004 DEBUG:boto:Params: {} 2022-09-02T05:59:40.004 DEBUG:boto:Token: None 2022-09-02T05:59:40.005 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj55 2022-09-02T05:59:40.006 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o6ML33xvrDQcEUxJ9Z638yiCNyE= 2022-09-02T05:59:40.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o6ML33xvrDQcEUxJ9Z638yiCNyE=', 'Content-Length': '0'} 2022-09-02T05:59:40.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf9461ed8f3e832-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.010 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.011 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.012 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:40.013 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:40.014 DEBUG:boto:Method: GET 2022-09-02T05:59:40.015 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:40.016 DEBUG:boto:Data: 2022-09-02T05:59:40.017 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.019 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.020 DEBUG:boto:Params: {} 2022-09-02T05:59:40.021 DEBUG:boto:Token: None 2022-09-02T05:59:40.022 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a922875d8fa468-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.028 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.030 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.031 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.032 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:40.033 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:40.034 DEBUG:boto:Method: GET 2022-09-02T05:59:40.035 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:40.036 DEBUG:boto:Data: 2022-09-02T05:59:40.037 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.039 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.040 DEBUG:boto:Params: {} 2022-09-02T05:59:40.041 DEBUG:boto:Token: None 2022-09-02T05:59:40.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3bce09bfb8b763c-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.049 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:40.051 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.051 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.052 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.053 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.053 DEBUG:boto:Method: GET 2022-09-02T05:59:40.054 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.054 DEBUG:boto:Data: 2022-09-02T05:59:40.055 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.056 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.057 DEBUG:boto:Params: {} 2022-09-02T05:59:40.057 DEBUG:boto:Token: None 2022-09-02T05:59:40.058 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.059 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU= 2022-09-02T05:59:40.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU=', 'Content-Length': '0'} 2022-09-02T05:59:40.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e711a208d2befb-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.062 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.063 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.064 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.064 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.065 DEBUG:boto:Method: GET 2022-09-02T05:59:40.065 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.066 DEBUG:boto:Data: 2022-09-02T05:59:40.067 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.068 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.068 DEBUG:boto:Params: {} 2022-09-02T05:59:40.069 DEBUG:boto:Token: None 2022-09-02T05:59:40.070 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.070 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU= 2022-09-02T05:59:40.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU=', 'Content-Length': '0'} 2022-09-02T05:59:40.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a7622ee1a2dff1-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.118 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.119 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.119 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.120 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.121 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.121 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.122 DEBUG:boto:Data: 2022-09-02T05:59:40.122 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.123 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.124 DEBUG:boto:Params: {} 2022-09-02T05:59:40.124 DEBUG:boto:Token: None 2022-09-02T05:59:40.125 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+qyCZO61c1IMxeNNXsewY4m3n8= 2022-09-02T05:59:40.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+qyCZO61c1IMxeNNXsewY4m3n8=', 'Content-Length': '0'} 2022-09-02T05:59:40.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a648186cf6c115-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.129 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.130 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.130 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.131 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.132 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.132 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.133 DEBUG:boto:Data: 2022-09-02T05:59:40.133 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.135 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.135 DEBUG:boto:Params: {} 2022-09-02T05:59:40.136 DEBUG:boto:Token: None 2022-09-02T05:59:40.136 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.137 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:y+qyCZO61c1IMxeNNXsewY4m3n8= 2022-09-02T05:59:40.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:y+qyCZO61c1IMxeNNXsewY4m3n8=', 'Content-Length': '0'} 2022-09-02T05:59:40.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bc20a3ab182616-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:40.143 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.144 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.144 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.145 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.145 DEBUG:boto:Method: GET 2022-09-02T05:59:40.146 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.146 DEBUG:boto:Data: 2022-09-02T05:59:40.147 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.148 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.149 DEBUG:boto:Params: {} 2022-09-02T05:59:40.149 DEBUG:boto:Token: None 2022-09-02T05:59:40.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU= 2022-09-02T05:59:40.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU=', 'Content-Length': '0'} 2022-09-02T05:59:40.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ca9d5abd1772a4-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.154 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.155 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.156 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.156 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.157 DEBUG:boto:Method: GET 2022-09-02T05:59:40.158 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.158 DEBUG:boto:Data: 2022-09-02T05:59:40.159 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.161 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.162 DEBUG:boto:Params: {} 2022-09-02T05:59:40.163 DEBUG:boto:Token: None 2022-09-02T05:59:40.164 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:40.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU= 2022-09-02T05:59:40.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9WeatxEz42HIt/KyKwfFbXOSLnU=', 'Content-Length': '0'} 2022-09-02T05:59:40.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b2f6bc07ff31c6-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.170 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.171 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.171 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.172 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.172 DEBUG:boto:Method: GET 2022-09-02T05:59:40.173 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.174 DEBUG:boto:Data: 2022-09-02T05:59:40.174 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.176 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.176 DEBUG:boto:Params: {} 2022-09-02T05:59:40.177 DEBUG:boto:Token: None 2022-09-02T05:59:40.178 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c874deecbfded3f7-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.183 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.185 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.186 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.187 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.187 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.188 DEBUG:boto:Method: GET 2022-09-02T05:59:40.189 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.190 DEBUG:boto:Data: 2022-09-02T05:59:40.191 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.192 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.193 DEBUG:boto:Params: {} 2022-09-02T05:59:40.193 DEBUG:boto:Token: None 2022-09-02T05:59:40.194 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000667c87a6596ad959-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.200 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.201 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.201 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.202 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.202 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:40.203 DEBUG:boto:Data: 2022-09-02T05:59:40.204 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.205 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.205 DEBUG:boto:Params: {} 2022-09-02T05:59:40.206 DEBUG:boto:Token: None 2022-09-02T05:59:40.206 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57 2022-09-02T05:59:40.207 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TwfrhRAwNaFH+5LNXVIhkjQgiMc= 2022-09-02T05:59:40.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TwfrhRAwNaFH+5LNXVIhkjQgiMc=', 'Content-Length': '0'} 2022-09-02T05:59:40.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087c07946bbb78c7-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.210 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.211 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.211 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.212 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:40.212 DEBUG:boto:Data: 2022-09-02T05:59:40.213 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.214 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.214 DEBUG:boto:Params: {} 2022-09-02T05:59:40.215 DEBUG:boto:Token: None 2022-09-02T05:59:40.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57 2022-09-02T05:59:40.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TwfrhRAwNaFH+5LNXVIhkjQgiMc= 2022-09-02T05:59:40.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TwfrhRAwNaFH+5LNXVIhkjQgiMc=', 'Content-Length': '0'} 2022-09-02T05:59:40.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c249cbd96ca84f80-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:40.220 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.221 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.221 DEBUG:boto:Method: GET 2022-09-02T05:59:40.222 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:40.223 DEBUG:boto:Data: 2022-09-02T05:59:40.223 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.224 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.225 DEBUG:boto:Params: {} 2022-09-02T05:59:40.225 DEBUG:boto:Token: None 2022-09-02T05:59:40.226 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57 2022-09-02T05:59:40.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RqR8wDE+zSQ+SjMErmN0n6wH92o= 2022-09-02T05:59:40.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RqR8wDE+zSQ+SjMErmN0n6wH92o=', 'Content-Length': '0'} 2022-09-02T05:59:40.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9775f1bb230a41e-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.270 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:40.271 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:40.272 DEBUG:boto:Method: GET 2022-09-02T05:59:40.272 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:40.273 DEBUG:boto:Data: 2022-09-02T05:59:40.273 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.274 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.275 DEBUG:boto:Params: {} 2022-09-02T05:59:40.276 DEBUG:boto:Token: None 2022-09-02T05:59:40.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj57 2022-09-02T05:59:40.277 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RqR8wDE+zSQ+SjMErmN0n6wH92o= 2022-09-02T05:59:40.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RqR8wDE+zSQ+SjMErmN0n6wH92o=', 'Content-Length': '0'} 2022-09-02T05:59:40.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000172435ceb800d439-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.281 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.282 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.282 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.283 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.283 DEBUG:boto:Method: GET 2022-09-02T05:59:40.284 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.284 DEBUG:boto:Data: 2022-09-02T05:59:40.285 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.286 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.287 DEBUG:boto:Params: {} 2022-09-02T05:59:40.287 DEBUG:boto:Token: None 2022-09-02T05:59:40.288 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006741ad6974bacf27-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.293 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.294 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.294 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.295 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.295 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.296 DEBUG:boto:Method: GET 2022-09-02T05:59:40.296 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:40.297 DEBUG:boto:Data: 2022-09-02T05:59:40.297 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.299 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.299 DEBUG:boto:Params: {} 2022-09-02T05:59:40.300 DEBUG:boto:Token: None 2022-09-02T05:59:40.300 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028738ddb52febf17-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.307 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:40.309 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.309 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.310 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.311 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.312 DEBUG:boto:Method: GET 2022-09-02T05:59:40.313 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.314 DEBUG:boto:Data: 2022-09-02T05:59:40.315 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.317 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.318 DEBUG:boto:Params: {} 2022-09-02T05:59:40.320 DEBUG:boto:Token: None 2022-09-02T05:59:40.321 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY= 2022-09-02T05:59:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY=', 'Content-Length': '0'} 2022-09-02T05:59:40.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000775fd5218bc582df-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.326 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.327 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.328 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.328 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.329 DEBUG:boto:Method: GET 2022-09-02T05:59:40.329 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.330 DEBUG:boto:Data: 2022-09-02T05:59:40.330 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.332 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.332 DEBUG:boto:Params: {} 2022-09-02T05:59:40.333 DEBUG:boto:Token: None 2022-09-02T05:59:40.333 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.334 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY= 2022-09-02T05:59:40.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY=', 'Content-Length': '0'} 2022-09-02T05:59:40.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99e4f220a440100-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.378 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.379 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.380 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.381 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.382 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.383 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.384 DEBUG:boto:Data: 2022-09-02T05:59:40.385 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.387 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.388 DEBUG:boto:Params: {} 2022-09-02T05:59:40.389 DEBUG:boto:Token: None 2022-09-02T05:59:40.390 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0/RfQqiuqgFkoyfFh8+WG20ANTk= 2022-09-02T05:59:40.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0/RfQqiuqgFkoyfFh8+WG20ANTk=', 'Content-Length': '0'} 2022-09-02T05:59:40.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b7129ded7249bda-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.396 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.397 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.398 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.399 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.400 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.401 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.402 DEBUG:boto:Data: 2022-09-02T05:59:40.403 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.405 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.406 DEBUG:boto:Params: {} 2022-09-02T05:59:40.407 DEBUG:boto:Token: None 2022-09-02T05:59:40.408 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.409 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0/RfQqiuqgFkoyfFh8+WG20ANTk= 2022-09-02T05:59:40.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0/RfQqiuqgFkoyfFh8+WG20ANTk=', 'Content-Length': '0'} 2022-09-02T05:59:40.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f2117a34ea9d0ee-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:40.415 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.416 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.417 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.418 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.419 DEBUG:boto:Method: GET 2022-09-02T05:59:40.420 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.421 DEBUG:boto:Data: 2022-09-02T05:59:40.422 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.424 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.425 DEBUG:boto:Params: {} 2022-09-02T05:59:40.426 DEBUG:boto:Token: None 2022-09-02T05:59:40.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY= 2022-09-02T05:59:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY=', 'Content-Length': '0'} 2022-09-02T05:59:40.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000420c490160e9e7a1-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.433 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.434 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.435 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.436 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.437 DEBUG:boto:Method: GET 2022-09-02T05:59:40.438 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.439 DEBUG:boto:Data: 2022-09-02T05:59:40.440 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.441 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.441 DEBUG:boto:Params: {} 2022-09-02T05:59:40.442 DEBUG:boto:Token: None 2022-09-02T05:59:40.442 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:40.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY= 2022-09-02T05:59:40.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:i3y7SgQTrRIES10bPSvFpdG9pOY=', 'Content-Length': '0'} 2022-09-02T05:59:40.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84ca5075e9d02d7-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.447 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.447 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.448 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.448 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.449 DEBUG:boto:Method: GET 2022-09-02T05:59:40.449 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.450 DEBUG:boto:Data: 2022-09-02T05:59:40.450 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.451 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.452 DEBUG:boto:Params: {} 2022-09-02T05:59:40.453 DEBUG:boto:Token: None 2022-09-02T05:59:40.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bedcaed8994666b1-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.458 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.459 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.459 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.460 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.460 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.461 DEBUG:boto:Method: GET 2022-09-02T05:59:40.461 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.462 DEBUG:boto:Data: 2022-09-02T05:59:40.462 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.463 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.464 DEBUG:boto:Params: {} 2022-09-02T05:59:40.464 DEBUG:boto:Token: None 2022-09-02T05:59:40.465 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b365ec5283674c8d-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.471 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.472 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.473 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.473 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.474 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:40.474 DEBUG:boto:Data: 2022-09-02T05:59:40.475 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.476 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.476 DEBUG:boto:Params: {} 2022-09-02T05:59:40.477 DEBUG:boto:Token: None 2022-09-02T05:59:40.477 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58 2022-09-02T05:59:40.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k5hJInOx8FcRS+vkMZNk7DWFKJo= 2022-09-02T05:59:40.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k5hJInOx8FcRS+vkMZNk7DWFKJo=', 'Content-Length': '0'} 2022-09-02T05:59:40.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d04ccb9984080d-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.481 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.482 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.482 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.483 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:40.483 DEBUG:boto:Data: 2022-09-02T05:59:40.484 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.485 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.485 DEBUG:boto:Params: {} 2022-09-02T05:59:40.486 DEBUG:boto:Token: None 2022-09-02T05:59:40.486 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58 2022-09-02T05:59:40.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k5hJInOx8FcRS+vkMZNk7DWFKJo= 2022-09-02T05:59:40.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k5hJInOx8FcRS+vkMZNk7DWFKJo=', 'Content-Length': '0'} 2022-09-02T05:59:40.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b0c1dfa650bf3e-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:40.491 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.492 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.492 DEBUG:boto:Method: GET 2022-09-02T05:59:40.493 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:40.493 DEBUG:boto:Data: 2022-09-02T05:59:40.494 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.495 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.495 DEBUG:boto:Params: {} 2022-09-02T05:59:40.496 DEBUG:boto:Token: None 2022-09-02T05:59:40.497 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58 2022-09-02T05:59:40.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6eSQIJKOAksnnsRIFs4kBjthiuU= 2022-09-02T05:59:40.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6eSQIJKOAksnnsRIFs4kBjthiuU=', 'Content-Length': '0'} 2022-09-02T05:59:40.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1a258794c37378d-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.501 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:40.501 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:40.502 DEBUG:boto:Method: GET 2022-09-02T05:59:40.502 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:40.503 DEBUG:boto:Data: 2022-09-02T05:59:40.503 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.504 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.505 DEBUG:boto:Params: {} 2022-09-02T05:59:40.505 DEBUG:boto:Token: None 2022-09-02T05:59:40.506 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj58 2022-09-02T05:59:40.507 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6eSQIJKOAksnnsRIFs4kBjthiuU= 2022-09-02T05:59:40.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6eSQIJKOAksnnsRIFs4kBjthiuU=', 'Content-Length': '0'} 2022-09-02T05:59:40.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c5c1ca3b5484016-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.550 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.551 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.552 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.552 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.553 DEBUG:boto:Method: GET 2022-09-02T05:59:40.553 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.554 DEBUG:boto:Data: 2022-09-02T05:59:40.554 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.555 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.556 DEBUG:boto:Params: {} 2022-09-02T05:59:40.557 DEBUG:boto:Token: None 2022-09-02T05:59:40.557 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.558 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cefc7f1d0a163c1-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.562 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.563 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.563 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.564 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.564 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.565 DEBUG:boto:Method: GET 2022-09-02T05:59:40.565 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:40.566 DEBUG:boto:Data: 2022-09-02T05:59:40.566 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.567 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.568 DEBUG:boto:Params: {} 2022-09-02T05:59:40.569 DEBUG:boto:Token: None 2022-09-02T05:59:40.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000073825ebd1a4ced-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.576 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:40.577 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.578 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.578 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.579 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.580 DEBUG:boto:Method: GET 2022-09-02T05:59:40.580 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.581 DEBUG:boto:Data: 2022-09-02T05:59:40.581 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.582 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.583 DEBUG:boto:Params: {} 2022-09-02T05:59:40.583 DEBUG:boto:Token: None 2022-09-02T05:59:40.584 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.585 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE= 2022-09-02T05:59:40.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE=', 'Content-Length': '0'} 2022-09-02T05:59:40.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b3f3c0d1c035689-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.589 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.589 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.590 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.590 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.591 DEBUG:boto:Method: GET 2022-09-02T05:59:40.591 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.592 DEBUG:boto:Data: 2022-09-02T05:59:40.592 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.593 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.594 DEBUG:boto:Params: {} 2022-09-02T05:59:40.594 DEBUG:boto:Token: None 2022-09-02T05:59:40.595 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.596 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE= 2022-09-02T05:59:40.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE=', 'Content-Length': '0'} 2022-09-02T05:59:40.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d419a6306f784591-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.600 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.600 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.601 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.601 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.602 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.603 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.603 DEBUG:boto:Data: 2022-09-02T05:59:40.604 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.604 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.605 DEBUG:boto:Params: {} 2022-09-02T05:59:40.606 DEBUG:boto:Token: None 2022-09-02T05:59:40.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+CFxf+hBTBv/v4n3LBCl2ToY96g= 2022-09-02T05:59:40.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+CFxf+hBTBv/v4n3LBCl2ToY96g=', 'Content-Length': '0'} 2022-09-02T05:59:40.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d1e27589906034-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.610 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.611 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.611 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.612 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.612 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.613 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.614 DEBUG:boto:Data: 2022-09-02T05:59:40.614 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.615 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.616 DEBUG:boto:Params: {} 2022-09-02T05:59:40.616 DEBUG:boto:Token: None 2022-09-02T05:59:40.617 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+CFxf+hBTBv/v4n3LBCl2ToY96g= 2022-09-02T05:59:40.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+CFxf+hBTBv/v4n3LBCl2ToY96g=', 'Content-Length': '0'} 2022-09-02T05:59:40.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7850a8a00dc62e9-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:40.623 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.624 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.625 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.626 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.627 DEBUG:boto:Method: GET 2022-09-02T05:59:40.628 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.628 DEBUG:boto:Data: 2022-09-02T05:59:40.629 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.631 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.632 DEBUG:boto:Params: {} 2022-09-02T05:59:40.633 DEBUG:boto:Token: None 2022-09-02T05:59:40.634 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE= 2022-09-02T05:59:40.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE=', 'Content-Length': '0'} 2022-09-02T05:59:40.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001add643d8e4acf7b-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.639 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.640 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.640 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.641 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.641 DEBUG:boto:Method: GET 2022-09-02T05:59:40.642 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.643 DEBUG:boto:Data: 2022-09-02T05:59:40.643 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.644 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.645 DEBUG:boto:Params: {} 2022-09-02T05:59:40.646 DEBUG:boto:Token: None 2022-09-02T05:59:40.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:40.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE= 2022-09-02T05:59:40.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sx7K4WWn7m0uEU4qhNlXzOiUKoE=', 'Content-Length': '0'} 2022-09-02T05:59:40.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d08d77f2b0b671-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.690 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.691 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.691 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.692 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.693 DEBUG:boto:Method: GET 2022-09-02T05:59:40.693 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.694 DEBUG:boto:Data: 2022-09-02T05:59:40.694 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.696 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.696 DEBUG:boto:Params: {} 2022-09-02T05:59:40.697 DEBUG:boto:Token: None 2022-09-02T05:59:40.697 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.698 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c2662cec7cf944-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.703 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.704 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.704 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.705 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.706 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.706 DEBUG:boto:Method: GET 2022-09-02T05:59:40.707 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.708 DEBUG:boto:Data: 2022-09-02T05:59:40.709 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.710 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.710 DEBUG:boto:Params: {} 2022-09-02T05:59:40.711 DEBUG:boto:Token: None 2022-09-02T05:59:40.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.712 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000173e14d07c990f42-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.720 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.721 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.722 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.723 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.724 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:40.725 DEBUG:boto:Data: 2022-09-02T05:59:40.726 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.728 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.729 DEBUG:boto:Params: {} 2022-09-02T05:59:40.739 DEBUG:boto:Token: None 2022-09-02T05:59:40.741 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62 2022-09-02T05:59:40.742 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MCNlUGrPXBDeRHiTMiOnxZFDUyM= 2022-09-02T05:59:40.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MCNlUGrPXBDeRHiTMiOnxZFDUyM=', 'Content-Length': '0'} 2022-09-02T05:59:40.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021374b8a65a41941-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.747 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.748 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.749 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.750 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:40.751 DEBUG:boto:Data: 2022-09-02T05:59:40.752 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.754 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.755 DEBUG:boto:Params: {} 2022-09-02T05:59:40.756 DEBUG:boto:Token: None 2022-09-02T05:59:40.757 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62 2022-09-02T05:59:40.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:MCNlUGrPXBDeRHiTMiOnxZFDUyM= 2022-09-02T05:59:40.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:MCNlUGrPXBDeRHiTMiOnxZFDUyM=', 'Content-Length': '0'} 2022-09-02T05:59:40.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f7ff9aed7df300-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:40.764 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.765 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.766 DEBUG:boto:Method: GET 2022-09-02T05:59:40.767 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:40.768 DEBUG:boto:Data: 2022-09-02T05:59:40.768 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.770 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.771 DEBUG:boto:Params: {} 2022-09-02T05:59:40.773 DEBUG:boto:Token: None 2022-09-02T05:59:40.773 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62 2022-09-02T05:59:40.774 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3cKZ35RlaQ91XBd2+sPLqNfuXaM= 2022-09-02T05:59:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3cKZ35RlaQ91XBd2+sPLqNfuXaM=', 'Content-Length': '0'} 2022-09-02T05:59:40.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236261b99272d3b6-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.781 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:40.781 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:40.782 DEBUG:boto:Method: GET 2022-09-02T05:59:40.782 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:40.783 DEBUG:boto:Data: 2022-09-02T05:59:40.783 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.784 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.785 DEBUG:boto:Params: {} 2022-09-02T05:59:40.786 DEBUG:boto:Token: None 2022-09-02T05:59:40.786 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj62 2022-09-02T05:59:40.787 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3cKZ35RlaQ91XBd2+sPLqNfuXaM= 2022-09-02T05:59:40.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3cKZ35RlaQ91XBd2+sPLqNfuXaM=', 'Content-Length': '0'} 2022-09-02T05:59:40.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be4a721699e99007-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.791 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.792 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.792 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.793 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.793 DEBUG:boto:Method: GET 2022-09-02T05:59:40.794 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.794 DEBUG:boto:Data: 2022-09-02T05:59:40.795 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.796 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.797 DEBUG:boto:Params: {} 2022-09-02T05:59:40.797 DEBUG:boto:Token: None 2022-09-02T05:59:40.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.798 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb3f528a678e7ef9-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.850 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.852 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.854 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.855 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.855 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.856 DEBUG:boto:Method: GET 2022-09-02T05:59:40.857 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:40.858 DEBUG:boto:Data: 2022-09-02T05:59:40.859 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.860 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.860 DEBUG:boto:Params: {} 2022-09-02T05:59:40.861 DEBUG:boto:Token: None 2022-09-02T05:59:40.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.862 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa1cf2091c6423c-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.868 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:40.870 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:40.870 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:40.871 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.871 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.872 DEBUG:boto:Method: GET 2022-09-02T05:59:40.873 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.873 DEBUG:boto:Data: 2022-09-02T05:59:40.874 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.875 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.876 DEBUG:boto:Params: {} 2022-09-02T05:59:40.876 DEBUG:boto:Token: None 2022-09-02T05:59:40.877 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.878 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs= 2022-09-02T05:59:40.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs=', 'Content-Length': '0'} 2022-09-02T05:59:40.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03cd80414233f1e-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.882 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:40.882 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:40.883 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.883 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.884 DEBUG:boto:Method: GET 2022-09-02T05:59:40.885 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.885 DEBUG:boto:Data: 2022-09-02T05:59:40.886 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.887 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.887 DEBUG:boto:Params: {} 2022-09-02T05:59:40.888 DEBUG:boto:Token: None 2022-09-02T05:59:40.889 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.889 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs= 2022-09-02T05:59:40.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs=', 'Content-Length': '0'} 2022-09-02T05:59:40.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1dfbb24c77dc1c-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.893 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:40.893 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:40.894 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.894 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.895 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.895 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.896 DEBUG:boto:Data: 2022-09-02T05:59:40.896 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.897 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.898 DEBUG:boto:Params: {} 2022-09-02T05:59:40.898 DEBUG:boto:Token: None 2022-09-02T05:59:40.899 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.900 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NRto3wvL1C7WhCfAA1MbrHTUCm0= 2022-09-02T05:59:40.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NRto3wvL1C7WhCfAA1MbrHTUCm0=', 'Content-Length': '0'} 2022-09-02T05:59:40.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeb2f88c14ef4372-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.903 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:40.904 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:40.904 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.905 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.906 DEBUG:boto:Method: HEAD 2022-09-02T05:59:40.906 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.907 DEBUG:boto:Data: 2022-09-02T05:59:40.907 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.908 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.909 DEBUG:boto:Params: {} 2022-09-02T05:59:40.909 DEBUG:boto:Token: None 2022-09-02T05:59:40.910 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NRto3wvL1C7WhCfAA1MbrHTUCm0= 2022-09-02T05:59:40.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NRto3wvL1C7WhCfAA1MbrHTUCm0=', 'Content-Length': '0'} 2022-09-02T05:59:40.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028fff76ed59bbeca-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:40.915 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:40.915 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:40.916 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.916 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.917 DEBUG:boto:Method: GET 2022-09-02T05:59:40.917 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.918 DEBUG:boto:Data: 2022-09-02T05:59:40.919 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.920 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.921 DEBUG:boto:Params: {} 2022-09-02T05:59:40.921 DEBUG:boto:Token: None 2022-09-02T05:59:40.922 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.923 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs= 2022-09-02T05:59:40.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs=', 'Content-Length': '0'} 2022-09-02T05:59:40.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005323ef9a6c00ae09-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.927 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:40.928 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:40.929 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.930 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.931 DEBUG:boto:Method: GET 2022-09-02T05:59:40.932 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.933 DEBUG:boto:Data: 2022-09-02T05:59:40.934 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:40.936 DEBUG:boto:Port: 8000 2022-09-02T05:59:40.937 DEBUG:boto:Params: {} 2022-09-02T05:59:40.937 DEBUG:boto:Token: None 2022-09-02T05:59:40.938 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:40.939 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs= 2022-09-02T05:59:40.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kLmkyjuOdlPHVS7CqqZCIyMmDrs=', 'Content-Length': '0'} 2022-09-02T05:59:40.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e991d8a926a78c-0063119bcc-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.944 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:40.945 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:40.945 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:40.946 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:40.947 DEBUG:boto:Method: GET 2022-09-02T05:59:40.947 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:40.948 DEBUG:boto:Data: 2022-09-02T05:59:40.949 DEBUG:boto:Headers: {} 2022-09-02T05:59:40.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:40.950 DEBUG:boto:Port: 8001 2022-09-02T05:59:40.950 DEBUG:boto:Params: {} 2022-09-02T05:59:40.951 DEBUG:boto:Token: None 2022-09-02T05:59:40.952 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:40 GMT /xljbkf-74/ 2022-09-02T05:59:40.953 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE= 2022-09-02T05:59:40.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:40 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9NPPsLA/hm1JIDIqmIwPzGCWHZE=', 'Content-Length': '0'} 2022-09-02T05:59:40.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256313498182d964-0063119bcc-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:40 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:40.998 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:40.999 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.000 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.000 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.001 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.002 DEBUG:boto:Method: GET 2022-09-02T05:59:41.002 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.003 DEBUG:boto:Data: 2022-09-02T05:59:41.004 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.005 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.005 DEBUG:boto:Params: {} 2022-09-02T05:59:41.006 DEBUG:boto:Token: None 2022-09-02T05:59:41.007 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.008 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ee0233b9f002bb-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.014 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.015 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:41.015 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:41.016 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.016 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:41.017 DEBUG:boto:Data: 2022-09-02T05:59:41.017 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.019 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.019 DEBUG:boto:Params: {} 2022-09-02T05:59:41.020 DEBUG:boto:Token: None 2022-09-02T05:59:41.021 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj68 2022-09-02T05:59:41.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bnjur3J+eiCy/TLHBacKOtiBXsU= 2022-09-02T05:59:41.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bnjur3J+eiCy/TLHBacKOtiBXsU=', 'Content-Length': '0'} 2022-09-02T05:59:41.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056bed8f43c3133c0-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.025 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:41.025 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:41.026 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.026 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:41.027 DEBUG:boto:Data: 2022-09-02T05:59:41.027 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.028 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.029 DEBUG:boto:Params: {} 2022-09-02T05:59:41.030 DEBUG:boto:Token: None 2022-09-02T05:59:41.030 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj68 2022-09-02T05:59:41.031 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Bnjur3J+eiCy/TLHBacKOtiBXsU= 2022-09-02T05:59:41.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Bnjur3J+eiCy/TLHBacKOtiBXsU=', 'Content-Length': '0'} 2022-09-02T05:59:41.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eded9d1f5e1ce04-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:41.035 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:41.036 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:41.036 DEBUG:boto:Method: GET 2022-09-02T05:59:41.037 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:41.037 DEBUG:boto:Data: 2022-09-02T05:59:41.038 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.039 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.039 DEBUG:boto:Params: {} 2022-09-02T05:59:41.040 DEBUG:boto:Token: None 2022-09-02T05:59:41.041 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj68 2022-09-02T05:59:41.041 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cQmWKNFn774OQvN7kwY2YlNeMao= 2022-09-02T05:59:41.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cQmWKNFn774OQvN7kwY2YlNeMao=', 'Content-Length': '0'} 2022-09-02T05:59:41.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309927d26ba6ef7e-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.045 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:41.045 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:41.046 DEBUG:boto:Method: GET 2022-09-02T05:59:41.047 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:41.047 DEBUG:boto:Data: 2022-09-02T05:59:41.048 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.049 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.049 DEBUG:boto:Params: {} 2022-09-02T05:59:41.050 DEBUG:boto:Token: None 2022-09-02T05:59:41.051 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj68 2022-09-02T05:59:41.051 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cQmWKNFn774OQvN7kwY2YlNeMao= 2022-09-02T05:59:41.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cQmWKNFn774OQvN7kwY2YlNeMao=', 'Content-Length': '0'} 2022-09-02T05:59:41.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003627c7ae02ae7d24-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.098 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.099 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.100 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.100 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.101 DEBUG:boto:Method: GET 2022-09-02T05:59:41.101 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.102 DEBUG:boto:Data: 2022-09-02T05:59:41.102 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.103 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.104 DEBUG:boto:Params: {} 2022-09-02T05:59:41.105 DEBUG:boto:Token: None 2022-09-02T05:59:41.105 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.106 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044c1060ba5e4eef1-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.110 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.111 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.112 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.112 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.113 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.113 DEBUG:boto:Method: GET 2022-09-02T05:59:41.114 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:41.114 DEBUG:boto:Data: 2022-09-02T05:59:41.115 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.116 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.116 DEBUG:boto:Params: {} 2022-09-02T05:59:41.117 DEBUG:boto:Token: None 2022-09-02T05:59:41.117 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.118 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bbe8e8ea809d773-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.125 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:41.127 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.128 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.128 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.129 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.130 DEBUG:boto:Method: GET 2022-09-02T05:59:41.131 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.132 DEBUG:boto:Data: 2022-09-02T05:59:41.133 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.135 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.136 DEBUG:boto:Params: {} 2022-09-02T05:59:41.137 DEBUG:boto:Token: None 2022-09-02T05:59:41.138 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.140 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc= 2022-09-02T05:59:41.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc=', 'Content-Length': '0'} 2022-09-02T05:59:41.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004345adde9a1e696a-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.145 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.146 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.147 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.148 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.149 DEBUG:boto:Method: GET 2022-09-02T05:59:41.150 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.151 DEBUG:boto:Data: 2022-09-02T05:59:41.152 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.154 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.155 DEBUG:boto:Params: {} 2022-09-02T05:59:41.156 DEBUG:boto:Token: None 2022-09-02T05:59:41.157 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.158 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc= 2022-09-02T05:59:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc=', 'Content-Length': '0'} 2022-09-02T05:59:41.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035456481687e0bdf-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.164 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.165 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.166 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.167 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.168 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.169 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.169 DEBUG:boto:Data: 2022-09-02T05:59:41.170 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.172 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.173 DEBUG:boto:Params: {} 2022-09-02T05:59:41.174 DEBUG:boto:Token: None 2022-09-02T05:59:41.175 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.176 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TOUXD23wvnvmWBCagPGugBFzVFc= 2022-09-02T05:59:41.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TOUXD23wvnvmWBCagPGugBFzVFc=', 'Content-Length': '0'} 2022-09-02T05:59:41.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9cef5f67dcda38-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.181 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.182 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.183 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.184 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.185 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.186 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.187 DEBUG:boto:Data: 2022-09-02T05:59:41.188 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.190 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.191 DEBUG:boto:Params: {} 2022-09-02T05:59:41.192 DEBUG:boto:Token: None 2022-09-02T05:59:41.193 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.194 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TOUXD23wvnvmWBCagPGugBFzVFc= 2022-09-02T05:59:41.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TOUXD23wvnvmWBCagPGugBFzVFc=', 'Content-Length': '0'} 2022-09-02T05:59:41.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001483a42d4b111458-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:41.200 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.201 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.202 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.202 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.203 DEBUG:boto:Method: GET 2022-09-02T05:59:41.203 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.204 DEBUG:boto:Data: 2022-09-02T05:59:41.204 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.205 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.206 DEBUG:boto:Params: {} 2022-09-02T05:59:41.207 DEBUG:boto:Token: None 2022-09-02T05:59:41.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc= 2022-09-02T05:59:41.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc=', 'Content-Length': '0'} 2022-09-02T05:59:41.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba32166174196ba-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.211 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.212 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.212 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.213 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.214 DEBUG:boto:Method: GET 2022-09-02T05:59:41.214 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.215 DEBUG:boto:Data: 2022-09-02T05:59:41.215 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.216 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.217 DEBUG:boto:Params: {} 2022-09-02T05:59:41.217 DEBUG:boto:Token: None 2022-09-02T05:59:41.218 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:41.219 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc= 2022-09-02T05:59:41.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PJElNMiqgpxEF0Wc4UqH3fH/TUc=', 'Content-Length': '0'} 2022-09-02T05:59:41.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfa448de7704fa5-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.262 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.263 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.265 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.266 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.267 DEBUG:boto:Method: GET 2022-09-02T05:59:41.268 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.268 DEBUG:boto:Data: 2022-09-02T05:59:41.269 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.271 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.272 DEBUG:boto:Params: {} 2022-09-02T05:59:41.273 DEBUG:boto:Token: None 2022-09-02T05:59:41.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.274 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0f85532bb7089e-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.279 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.280 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.280 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.281 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.281 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.282 DEBUG:boto:Method: GET 2022-09-02T05:59:41.283 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.283 DEBUG:boto:Data: 2022-09-02T05:59:41.284 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.285 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.285 DEBUG:boto:Params: {} 2022-09-02T05:59:41.286 DEBUG:boto:Token: None 2022-09-02T05:59:41.286 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.287 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5238aa9cc3e7e41-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.293 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.294 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.294 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.295 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.295 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:41.296 DEBUG:boto:Data: 2022-09-02T05:59:41.296 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.297 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.298 DEBUG:boto:Params: {} 2022-09-02T05:59:41.298 DEBUG:boto:Token: None 2022-09-02T05:59:41.299 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69 2022-09-02T05:59:41.300 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Z06Pe+fiNK3XKSUlJ+TDffOMpI= 2022-09-02T05:59:41.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Z06Pe+fiNK3XKSUlJ+TDffOMpI=', 'Content-Length': '0'} 2022-09-02T05:59:41.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089bd9a95d6eb888a-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.303 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.304 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.304 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.305 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:41.305 DEBUG:boto:Data: 2022-09-02T05:59:41.306 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.307 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.307 DEBUG:boto:Params: {} 2022-09-02T05:59:41.308 DEBUG:boto:Token: None 2022-09-02T05:59:41.308 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69 2022-09-02T05:59:41.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9Z06Pe+fiNK3XKSUlJ+TDffOMpI= 2022-09-02T05:59:41.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9Z06Pe+fiNK3XKSUlJ+TDffOMpI=', 'Content-Length': '0'} 2022-09-02T05:59:41.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000679996a08a7b5e67-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:41.313 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.314 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.314 DEBUG:boto:Method: GET 2022-09-02T05:59:41.315 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:41.315 DEBUG:boto:Data: 2022-09-02T05:59:41.316 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.317 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.318 DEBUG:boto:Params: {} 2022-09-02T05:59:41.318 DEBUG:boto:Token: None 2022-09-02T05:59:41.319 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69 2022-09-02T05:59:41.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZH/fkhnpVOXDhfDwz7/vAXQ3cws= 2022-09-02T05:59:41.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZH/fkhnpVOXDhfDwz7/vAXQ3cws=', 'Content-Length': '0'} 2022-09-02T05:59:41.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d97ca135a491633-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.366 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:41.367 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:41.368 DEBUG:boto:Method: GET 2022-09-02T05:59:41.368 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:41.369 DEBUG:boto:Data: 2022-09-02T05:59:41.369 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.371 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.372 DEBUG:boto:Params: {} 2022-09-02T05:59:41.373 DEBUG:boto:Token: None 2022-09-02T05:59:41.374 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj69 2022-09-02T05:59:41.375 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZH/fkhnpVOXDhfDwz7/vAXQ3cws= 2022-09-02T05:59:41.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZH/fkhnpVOXDhfDwz7/vAXQ3cws=', 'Content-Length': '0'} 2022-09-02T05:59:41.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7935ce6d93ff0f-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.381 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.382 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.383 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.384 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.385 DEBUG:boto:Method: GET 2022-09-02T05:59:41.385 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.386 DEBUG:boto:Data: 2022-09-02T05:59:41.387 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.389 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.390 DEBUG:boto:Params: {} 2022-09-02T05:59:41.391 DEBUG:boto:Token: None 2022-09-02T05:59:41.392 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.393 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fcbb76f6fc3842e-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.399 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.400 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.401 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.403 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.403 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.404 DEBUG:boto:Method: GET 2022-09-02T05:59:41.405 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:41.406 DEBUG:boto:Data: 2022-09-02T05:59:41.407 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.409 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.410 DEBUG:boto:Params: {} 2022-09-02T05:59:41.411 DEBUG:boto:Token: None 2022-09-02T05:59:41.412 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.413 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f68e57001badf509-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.462 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:41.465 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.466 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.467 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.467 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.468 DEBUG:boto:Method: GET 2022-09-02T05:59:41.469 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.470 DEBUG:boto:Data: 2022-09-02T05:59:41.471 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.472 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.473 DEBUG:boto:Params: {} 2022-09-02T05:59:41.474 DEBUG:boto:Token: None 2022-09-02T05:59:41.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8= 2022-09-02T05:59:41.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8=', 'Content-Length': '0'} 2022-09-02T05:59:41.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a326ab07f2bd719-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.483 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.484 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.484 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.485 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.486 DEBUG:boto:Method: GET 2022-09-02T05:59:41.487 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.488 DEBUG:boto:Data: 2022-09-02T05:59:41.489 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.491 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.492 DEBUG:boto:Params: {} 2022-09-02T05:59:41.493 DEBUG:boto:Token: None 2022-09-02T05:59:41.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8= 2022-09-02T05:59:41.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8=', 'Content-Length': '0'} 2022-09-02T05:59:41.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5bdc3f72427a42-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.498 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.499 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.500 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.500 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.510 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.510 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.511 DEBUG:boto:Data: 2022-09-02T05:59:41.512 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.513 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.514 DEBUG:boto:Params: {} 2022-09-02T05:59:41.515 DEBUG:boto:Token: None 2022-09-02T05:59:41.516 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.517 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M/wQouZmg83GStViJ88gD3ugYik= 2022-09-02T05:59:41.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M/wQouZmg83GStViJ88gD3ugYik=', 'Content-Length': '0'} 2022-09-02T05:59:41.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ecd583c5bb6118-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.522 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.523 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.524 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.524 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.525 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.525 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.526 DEBUG:boto:Data: 2022-09-02T05:59:41.527 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.528 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.528 DEBUG:boto:Params: {} 2022-09-02T05:59:41.529 DEBUG:boto:Token: None 2022-09-02T05:59:41.529 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.530 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:M/wQouZmg83GStViJ88gD3ugYik= 2022-09-02T05:59:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:M/wQouZmg83GStViJ88gD3ugYik=', 'Content-Length': '0'} 2022-09-02T05:59:41.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ecfa21aa7fa14a7-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:41.535 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.535 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.536 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.536 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.537 DEBUG:boto:Method: GET 2022-09-02T05:59:41.537 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.538 DEBUG:boto:Data: 2022-09-02T05:59:41.538 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.540 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.540 DEBUG:boto:Params: {} 2022-09-02T05:59:41.541 DEBUG:boto:Token: None 2022-09-02T05:59:41.541 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.542 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8= 2022-09-02T05:59:41.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8=', 'Content-Length': '0'} 2022-09-02T05:59:41.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d814b3cf107b39-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.546 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.547 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.547 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.548 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.548 DEBUG:boto:Method: GET 2022-09-02T05:59:41.549 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.550 DEBUG:boto:Data: 2022-09-02T05:59:41.551 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.552 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.552 DEBUG:boto:Params: {} 2022-09-02T05:59:41.553 DEBUG:boto:Token: None 2022-09-02T05:59:41.553 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:41.554 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8= 2022-09-02T05:59:41.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xtk6FiE+Sypf/bJ/RHHAtBl2SG8=', 'Content-Length': '0'} 2022-09-02T05:59:41.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3c25b1a4af79b7-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.558 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.558 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.559 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.559 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.560 DEBUG:boto:Method: GET 2022-09-02T05:59:41.561 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.561 DEBUG:boto:Data: 2022-09-02T05:59:41.562 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.563 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.563 DEBUG:boto:Params: {} 2022-09-02T05:59:41.564 DEBUG:boto:Token: None 2022-09-02T05:59:41.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000854ebb8f52beb89d-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.569 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.570 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.570 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.571 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.572 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.572 DEBUG:boto:Method: GET 2022-09-02T05:59:41.573 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.573 DEBUG:boto:Data: 2022-09-02T05:59:41.574 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.575 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.575 DEBUG:boto:Params: {} 2022-09-02T05:59:41.576 DEBUG:boto:Token: None 2022-09-02T05:59:41.577 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.577 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3a4d26da55697ca-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.583 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.584 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.584 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.585 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.585 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:41.586 DEBUG:boto:Data: 2022-09-02T05:59:41.586 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.587 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.588 DEBUG:boto:Params: {} 2022-09-02T05:59:41.589 DEBUG:boto:Token: None 2022-09-02T05:59:41.589 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74 2022-09-02T05:59:41.590 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DpAxhunfoV/BhgJx4p/w8NsY+ns= 2022-09-02T05:59:41.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DpAxhunfoV/BhgJx4p/w8NsY+ns=', 'Content-Length': '0'} 2022-09-02T05:59:41.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002596349d3a58363b-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.594 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.595 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.596 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.597 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:41.598 DEBUG:boto:Data: 2022-09-02T05:59:41.599 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.601 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.602 DEBUG:boto:Params: {} 2022-09-02T05:59:41.603 DEBUG:boto:Token: None 2022-09-02T05:59:41.603 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74 2022-09-02T05:59:41.604 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DpAxhunfoV/BhgJx4p/w8NsY+ns= 2022-09-02T05:59:41.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DpAxhunfoV/BhgJx4p/w8NsY+ns=', 'Content-Length': '0'} 2022-09-02T05:59:41.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002544e70008206f86-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:41.609 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.609 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.610 DEBUG:boto:Method: GET 2022-09-02T05:59:41.611 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:41.611 DEBUG:boto:Data: 2022-09-02T05:59:41.612 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.613 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.613 DEBUG:boto:Params: {} 2022-09-02T05:59:41.614 DEBUG:boto:Token: None 2022-09-02T05:59:41.614 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74 2022-09-02T05:59:41.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dyrvn569fUOMg93qBYraz57D6bQ= 2022-09-02T05:59:41.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dyrvn569fUOMg93qBYraz57D6bQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcff5b47cd8909dd-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.618 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:41.619 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:41.620 DEBUG:boto:Method: GET 2022-09-02T05:59:41.620 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:41.621 DEBUG:boto:Data: 2022-09-02T05:59:41.621 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.622 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.623 DEBUG:boto:Params: {} 2022-09-02T05:59:41.623 DEBUG:boto:Token: None 2022-09-02T05:59:41.624 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj74 2022-09-02T05:59:41.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Dyrvn569fUOMg93qBYraz57D6bQ= 2022-09-02T05:59:41.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Dyrvn569fUOMg93qBYraz57D6bQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aca3dce241683d99-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.629 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.629 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.630 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.630 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.631 DEBUG:boto:Method: GET 2022-09-02T05:59:41.632 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.632 DEBUG:boto:Data: 2022-09-02T05:59:41.633 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.634 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.634 DEBUG:boto:Params: {} 2022-09-02T05:59:41.635 DEBUG:boto:Token: None 2022-09-02T05:59:41.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.636 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477e7d7564c4b465-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.641 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.642 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.643 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.643 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.644 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.645 DEBUG:boto:Method: GET 2022-09-02T05:59:41.645 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:41.646 DEBUG:boto:Data: 2022-09-02T05:59:41.646 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.647 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.648 DEBUG:boto:Params: {} 2022-09-02T05:59:41.649 DEBUG:boto:Token: None 2022-09-02T05:59:41.649 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0a9514c758f0c22-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.656 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:41.658 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.658 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.659 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.659 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.660 DEBUG:boto:Method: GET 2022-09-02T05:59:41.661 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.661 DEBUG:boto:Data: 2022-09-02T05:59:41.662 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.663 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.664 DEBUG:boto:Params: {} 2022-09-02T05:59:41.664 DEBUG:boto:Token: None 2022-09-02T05:59:41.665 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.666 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds= 2022-09-02T05:59:41.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds=', 'Content-Length': '0'} 2022-09-02T05:59:41.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080fc56fd6976ebfa-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.669 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.670 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.671 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.672 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.672 DEBUG:boto:Method: GET 2022-09-02T05:59:41.673 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.673 DEBUG:boto:Data: 2022-09-02T05:59:41.674 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.676 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.677 DEBUG:boto:Params: {} 2022-09-02T05:59:41.678 DEBUG:boto:Token: None 2022-09-02T05:59:41.679 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds= 2022-09-02T05:59:41.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds=', 'Content-Length': '0'} 2022-09-02T05:59:41.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfb598cf17b3d070-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.727 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.728 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.729 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.730 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.730 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.731 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.732 DEBUG:boto:Data: 2022-09-02T05:59:41.732 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.734 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.734 DEBUG:boto:Params: {} 2022-09-02T05:59:41.735 DEBUG:boto:Token: None 2022-09-02T05:59:41.736 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.736 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WQmLpmbEipab5nLoMSgHV+d2CmU= 2022-09-02T05:59:41.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WQmLpmbEipab5nLoMSgHV+d2CmU=', 'Content-Length': '0'} 2022-09-02T05:59:41.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e753f8f856b09f1a-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.740 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.740 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.741 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.741 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.742 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.742 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.743 DEBUG:boto:Data: 2022-09-02T05:59:41.743 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.744 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.745 DEBUG:boto:Params: {} 2022-09-02T05:59:41.745 DEBUG:boto:Token: None 2022-09-02T05:59:41.746 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WQmLpmbEipab5nLoMSgHV+d2CmU= 2022-09-02T05:59:41.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WQmLpmbEipab5nLoMSgHV+d2CmU=', 'Content-Length': '0'} 2022-09-02T05:59:41.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d78df659f4287d6-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:41.751 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.751 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.752 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.752 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.753 DEBUG:boto:Method: GET 2022-09-02T05:59:41.753 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.754 DEBUG:boto:Data: 2022-09-02T05:59:41.754 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.756 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.756 DEBUG:boto:Params: {} 2022-09-02T05:59:41.757 DEBUG:boto:Token: None 2022-09-02T05:59:41.758 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.759 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds= 2022-09-02T05:59:41.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds=', 'Content-Length': '0'} 2022-09-02T05:59:41.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef5d9d0e18ab0a9-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.762 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.763 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.763 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.764 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.764 DEBUG:boto:Method: GET 2022-09-02T05:59:41.765 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.766 DEBUG:boto:Data: 2022-09-02T05:59:41.766 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.767 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.768 DEBUG:boto:Params: {} 2022-09-02T05:59:41.769 DEBUG:boto:Token: None 2022-09-02T05:59:41.770 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:41.770 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds= 2022-09-02T05:59:41.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:gsQCMsfG5CsIXXhMtC4ccJoL1Ds=', 'Content-Length': '0'} 2022-09-02T05:59:41.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba528d0c1a2867c-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.774 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.775 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.775 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.776 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.777 DEBUG:boto:Method: GET 2022-09-02T05:59:41.777 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.778 DEBUG:boto:Data: 2022-09-02T05:59:41.778 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.779 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.780 DEBUG:boto:Params: {} 2022-09-02T05:59:41.781 DEBUG:boto:Token: None 2022-09-02T05:59:41.781 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a64f88a8c5f76ba-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.786 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.787 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.787 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.788 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.789 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.790 DEBUG:boto:Method: GET 2022-09-02T05:59:41.791 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.791 DEBUG:boto:Data: 2022-09-02T05:59:41.792 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.793 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.794 DEBUG:boto:Params: {} 2022-09-02T05:59:41.794 DEBUG:boto:Token: None 2022-09-02T05:59:41.795 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e6617c607c464c-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.801 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.802 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.803 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.803 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.804 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:41.804 DEBUG:boto:Data: 2022-09-02T05:59:41.805 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.806 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.806 DEBUG:boto:Params: {} 2022-09-02T05:59:41.807 DEBUG:boto:Token: None 2022-09-02T05:59:41.808 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76 2022-09-02T05:59:41.809 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gbMqLNXAG6bzShp3q9bypAZVhM= 2022-09-02T05:59:41.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gbMqLNXAG6bzShp3q9bypAZVhM=', 'Content-Length': '0'} 2022-09-02T05:59:41.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543fe3e70c1872c9-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.813 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.813 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.814 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.815 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:41.815 DEBUG:boto:Data: 2022-09-02T05:59:41.816 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.817 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.818 DEBUG:boto:Params: {} 2022-09-02T05:59:41.818 DEBUG:boto:Token: None 2022-09-02T05:59:41.819 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76 2022-09-02T05:59:41.820 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gbMqLNXAG6bzShp3q9bypAZVhM= 2022-09-02T05:59:41.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gbMqLNXAG6bzShp3q9bypAZVhM=', 'Content-Length': '0'} 2022-09-02T05:59:41.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab01af624b3e94ec-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:41.824 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.825 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.825 DEBUG:boto:Method: GET 2022-09-02T05:59:41.826 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:41.826 DEBUG:boto:Data: 2022-09-02T05:59:41.827 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.828 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.829 DEBUG:boto:Params: {} 2022-09-02T05:59:41.829 DEBUG:boto:Token: None 2022-09-02T05:59:41.830 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76 2022-09-02T05:59:41.831 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r1vrReUkQHYu+XksISM+0rDwYzk= 2022-09-02T05:59:41.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r1vrReUkQHYu+XksISM+0rDwYzk=', 'Content-Length': '0'} 2022-09-02T05:59:41.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d811b01a023da3a2-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.835 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:41.835 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:41.836 DEBUG:boto:Method: GET 2022-09-02T05:59:41.837 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:41.837 DEBUG:boto:Data: 2022-09-02T05:59:41.838 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.839 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.840 DEBUG:boto:Params: {} 2022-09-02T05:59:41.840 DEBUG:boto:Token: None 2022-09-02T05:59:41.841 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj76 2022-09-02T05:59:41.841 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r1vrReUkQHYu+XksISM+0rDwYzk= 2022-09-02T05:59:41.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r1vrReUkQHYu+XksISM+0rDwYzk=', 'Content-Length': '0'} 2022-09-02T05:59:41.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8a5b07aa974091-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.890 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.891 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.891 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.892 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.892 DEBUG:boto:Method: GET 2022-09-02T05:59:41.893 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.893 DEBUG:boto:Data: 2022-09-02T05:59:41.894 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.895 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.896 DEBUG:boto:Params: {} 2022-09-02T05:59:41.896 DEBUG:boto:Token: None 2022-09-02T05:59:41.897 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.897 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077abfee29b2757f7-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.901 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.902 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.903 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.903 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.904 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.904 DEBUG:boto:Method: GET 2022-09-02T05:59:41.905 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:41.905 DEBUG:boto:Data: 2022-09-02T05:59:41.906 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.907 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.907 DEBUG:boto:Params: {} 2022-09-02T05:59:41.908 DEBUG:boto:Token: None 2022-09-02T05:59:41.909 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/ 2022-09-02T05:59:41.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ= 2022-09-02T05:59:41.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:GnEbVP9Oe7lO1N9yZK/hKXI5DTQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18df41a76f5e911-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.917 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:41.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:41.919 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:41.920 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:41.921 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.922 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.923 DEBUG:boto:Method: GET 2022-09-02T05:59:41.925 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.926 DEBUG:boto:Data: 2022-09-02T05:59:41.927 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.929 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.929 DEBUG:boto:Params: {} 2022-09-02T05:59:41.930 DEBUG:boto:Token: None 2022-09-02T05:59:41.931 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.932 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA= 2022-09-02T05:59:41.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA=', 'Content-Length': '0'} 2022-09-02T05:59:41.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003facc835d2664638-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.937 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:41.938 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:41.938 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.939 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.939 DEBUG:boto:Method: GET 2022-09-02T05:59:41.940 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.941 DEBUG:boto:Data: 2022-09-02T05:59:41.941 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.942 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.943 DEBUG:boto:Params: {} 2022-09-02T05:59:41.943 DEBUG:boto:Token: None 2022-09-02T05:59:41.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA= 2022-09-02T05:59:41.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA=', 'Content-Length': '0'} 2022-09-02T05:59:41.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5001b8bc02415da-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.949 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:41.949 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:41.950 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.950 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.951 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.951 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.952 DEBUG:boto:Data: 2022-09-02T05:59:41.952 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.953 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.954 DEBUG:boto:Params: {} 2022-09-02T05:59:41.955 DEBUG:boto:Token: None 2022-09-02T05:59:41.955 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zb96FzRwReP0HXGrcv5xJeBSVkQ= 2022-09-02T05:59:41.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zb96FzRwReP0HXGrcv5xJeBSVkQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c190e89b9aee267c-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.959 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:41.960 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:41.960 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.961 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.961 DEBUG:boto:Method: HEAD 2022-09-02T05:59:41.962 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.962 DEBUG:boto:Data: 2022-09-02T05:59:41.963 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.964 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.964 DEBUG:boto:Params: {} 2022-09-02T05:59:41.965 DEBUG:boto:Token: None 2022-09-02T05:59:41.965 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.966 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Zb96FzRwReP0HXGrcv5xJeBSVkQ= 2022-09-02T05:59:41.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Zb96FzRwReP0HXGrcv5xJeBSVkQ=', 'Content-Length': '0'} 2022-09-02T05:59:41.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc93f05b76c5bfb1-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:41.971 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:41.972 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:41.972 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.973 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.973 DEBUG:boto:Method: GET 2022-09-02T05:59:41.974 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.974 DEBUG:boto:Data: 2022-09-02T05:59:41.975 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.976 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.976 DEBUG:boto:Params: {} 2022-09-02T05:59:41.977 DEBUG:boto:Token: None 2022-09-02T05:59:41.978 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA= 2022-09-02T05:59:41.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA=', 'Content-Length': '0'} 2022-09-02T05:59:41.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003465f2c350b2f00e-0063119bcd-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.982 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:41.982 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:41.983 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.983 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.984 DEBUG:boto:Method: GET 2022-09-02T05:59:41.984 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.985 DEBUG:boto:Data: 2022-09-02T05:59:41.986 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:41.987 DEBUG:boto:Port: 8000 2022-09-02T05:59:41.987 DEBUG:boto:Params: {} 2022-09-02T05:59:41.988 DEBUG:boto:Token: None 2022-09-02T05:59:41.989 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:41 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:41.989 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA= 2022-09-02T05:59:41.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:41 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pIHkldFKgL7RmxtxbxJbQ0porRA=', 'Content-Length': '0'} 2022-09-02T05:59:41.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17e1a4315e747bd-0063119bcd-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:41 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:41.993 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:41.993 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:41.994 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:41.995 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:41.995 DEBUG:boto:Method: GET 2022-09-02T05:59:41.996 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:41.997 DEBUG:boto:Data: 2022-09-02T05:59:41.997 DEBUG:boto:Headers: {} 2022-09-02T05:59:41.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:41.998 DEBUG:boto:Port: 8001 2022-09-02T05:59:41.999 DEBUG:boto:Params: {} 2022-09-02T05:59:41.999 DEBUG:boto:Token: None 2022-09-02T05:59:42.000 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.001 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094459994f9b807b3-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.005 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.006 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.007 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.008 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.009 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.010 DEBUG:boto:Method: GET 2022-09-02T05:59:42.011 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.012 DEBUG:boto:Data: 2022-09-02T05:59:42.013 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.015 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.016 DEBUG:boto:Params: {} 2022-09-02T05:59:42.017 DEBUG:boto:Token: None 2022-09-02T05:59:42.018 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.019 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f66509fd6d556aa6-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.026 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.027 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:42.027 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:42.028 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.028 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:42.029 DEBUG:boto:Data: 2022-09-02T05:59:42.029 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.030 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.031 DEBUG:boto:Params: {} 2022-09-02T05:59:42.031 DEBUG:boto:Token: None 2022-09-02T05:59:42.032 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj85 2022-09-02T05:59:42.033 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LwCEU5vWO4q7I/+i0uDHfcQahnA= 2022-09-02T05:59:42.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LwCEU5vWO4q7I/+i0uDHfcQahnA=', 'Content-Length': '0'} 2022-09-02T05:59:42.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222f8bc4b6449aa9-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.037 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:42.038 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:42.038 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.039 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:42.040 DEBUG:boto:Data: 2022-09-02T05:59:42.040 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.041 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.042 DEBUG:boto:Params: {} 2022-09-02T05:59:42.043 DEBUG:boto:Token: None 2022-09-02T05:59:42.044 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj85 2022-09-02T05:59:42.045 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LwCEU5vWO4q7I/+i0uDHfcQahnA= 2022-09-02T05:59:42.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LwCEU5vWO4q7I/+i0uDHfcQahnA=', 'Content-Length': '0'} 2022-09-02T05:59:42.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e0275c86f3221b-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:42.051 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:42.052 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:42.052 DEBUG:boto:Method: GET 2022-09-02T05:59:42.053 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:42.054 DEBUG:boto:Data: 2022-09-02T05:59:42.055 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.056 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.057 DEBUG:boto:Params: {} 2022-09-02T05:59:42.058 DEBUG:boto:Token: None 2022-09-02T05:59:42.059 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj85 2022-09-02T05:59:42.060 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UeGzdCCQ4DIHjjJBPexjbwPvzLo= 2022-09-02T05:59:42.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UeGzdCCQ4DIHjjJBPexjbwPvzLo=', 'Content-Length': '0'} 2022-09-02T05:59:42.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f9fb466ddb25f4-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.064 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:42.065 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:42.066 DEBUG:boto:Method: GET 2022-09-02T05:59:42.067 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:42.069 DEBUG:boto:Data: 2022-09-02T05:59:42.070 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.073 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.074 DEBUG:boto:Params: {} 2022-09-02T05:59:42.076 DEBUG:boto:Token: None 2022-09-02T05:59:42.077 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj85 2022-09-02T05:59:42.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UeGzdCCQ4DIHjjJBPexjbwPvzLo= 2022-09-02T05:59:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UeGzdCCQ4DIHjjJBPexjbwPvzLo=', 'Content-Length': '0'} 2022-09-02T05:59:42.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b24910caecc02a9-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.126 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.128 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.129 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.130 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.131 DEBUG:boto:Method: GET 2022-09-02T05:59:42.132 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.133 DEBUG:boto:Data: 2022-09-02T05:59:42.134 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.137 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.138 DEBUG:boto:Params: {} 2022-09-02T05:59:42.139 DEBUG:boto:Token: None 2022-09-02T05:59:42.140 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.141 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a08b2c95200dd3ef-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.151 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.153 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.154 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.154 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.155 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.156 DEBUG:boto:Method: GET 2022-09-02T05:59:42.157 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:42.158 DEBUG:boto:Data: 2022-09-02T05:59:42.159 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.160 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.161 DEBUG:boto:Params: {} 2022-09-02T05:59:42.162 DEBUG:boto:Token: None 2022-09-02T05:59:42.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.165 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce2c675ad1bcd546-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.172 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:42.175 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.176 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.177 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.178 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.179 DEBUG:boto:Method: GET 2022-09-02T05:59:42.180 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.181 DEBUG:boto:Data: 2022-09-02T05:59:42.182 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.185 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.187 DEBUG:boto:Params: {} 2022-09-02T05:59:42.188 DEBUG:boto:Token: None 2022-09-02T05:59:42.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.192 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4= 2022-09-02T05:59:42.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4=', 'Content-Length': '0'} 2022-09-02T05:59:42.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffec2641d1f079d-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.197 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.198 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.199 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.199 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.200 DEBUG:boto:Method: GET 2022-09-02T05:59:42.201 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.202 DEBUG:boto:Data: 2022-09-02T05:59:42.203 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.205 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.206 DEBUG:boto:Params: {} 2022-09-02T05:59:42.207 DEBUG:boto:Token: None 2022-09-02T05:59:42.207 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.208 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4= 2022-09-02T05:59:42.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4=', 'Content-Length': '0'} 2022-09-02T05:59:42.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe847ef9df4a7978-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.212 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.213 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.214 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.215 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.216 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.217 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.218 DEBUG:boto:Data: 2022-09-02T05:59:42.218 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.220 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.220 DEBUG:boto:Params: {} 2022-09-02T05:59:42.221 DEBUG:boto:Token: None 2022-09-02T05:59:42.221 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.222 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ajeHTMKIzB4nO8Ji6Zr8mL2QFG0= 2022-09-02T05:59:42.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ajeHTMKIzB4nO8Ji6Zr8mL2QFG0=', 'Content-Length': '0'} 2022-09-02T05:59:42.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b949c9de51b73cb-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.226 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.227 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.227 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.228 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.228 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.229 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.229 DEBUG:boto:Data: 2022-09-02T05:59:42.230 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.231 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.231 DEBUG:boto:Params: {} 2022-09-02T05:59:42.232 DEBUG:boto:Token: None 2022-09-02T05:59:42.232 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.233 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ajeHTMKIzB4nO8Ji6Zr8mL2QFG0= 2022-09-02T05:59:42.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ajeHTMKIzB4nO8Ji6Zr8mL2QFG0=', 'Content-Length': '0'} 2022-09-02T05:59:42.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005681bd02653a498d-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:42.237 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.238 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.238 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.239 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.239 DEBUG:boto:Method: GET 2022-09-02T05:59:42.240 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.240 DEBUG:boto:Data: 2022-09-02T05:59:42.241 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.242 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.242 DEBUG:boto:Params: {} 2022-09-02T05:59:42.243 DEBUG:boto:Token: None 2022-09-02T05:59:42.244 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.245 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4= 2022-09-02T05:59:42.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4=', 'Content-Length': '0'} 2022-09-02T05:59:42.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be0209cbf79d090e-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.248 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.249 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.249 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.250 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.251 DEBUG:boto:Method: GET 2022-09-02T05:59:42.251 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.252 DEBUG:boto:Data: 2022-09-02T05:59:42.252 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.254 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.254 DEBUG:boto:Params: {} 2022-09-02T05:59:42.255 DEBUG:boto:Token: None 2022-09-02T05:59:42.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:42.256 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4= 2022-09-02T05:59:42.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ylOcpBuYwrJokwMJOIeSCeJXov4=', 'Content-Length': '0'} 2022-09-02T05:59:42.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8525c7fed280dd9-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.260 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.260 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.261 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.261 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.262 DEBUG:boto:Method: GET 2022-09-02T05:59:42.263 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.263 DEBUG:boto:Data: 2022-09-02T05:59:42.264 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.265 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.265 DEBUG:boto:Params: {} 2022-09-02T05:59:42.266 DEBUG:boto:Token: None 2022-09-02T05:59:42.267 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000825447d7cd225c04-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.271 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.272 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.273 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.273 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.274 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.275 DEBUG:boto:Method: GET 2022-09-02T05:59:42.275 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.276 DEBUG:boto:Data: 2022-09-02T05:59:42.276 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.278 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.279 DEBUG:boto:Params: {} 2022-09-02T05:59:42.280 DEBUG:boto:Token: None 2022-09-02T05:59:42.281 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.282 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c2f5e630301b60-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.290 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.292 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.293 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.294 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.295 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:42.295 DEBUG:boto:Data: 2022-09-02T05:59:42.296 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.298 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.298 DEBUG:boto:Params: {} 2022-09-02T05:59:42.299 DEBUG:boto:Token: None 2022-09-02T05:59:42.300 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87 2022-09-02T05:59:42.301 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mdcsRbJwo3mr37ZKAILnX5pX3EA= 2022-09-02T05:59:42.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mdcsRbJwo3mr37ZKAILnX5pX3EA=', 'Content-Length': '0'} 2022-09-02T05:59:42.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f23d90bc86e01699-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.305 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.305 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.306 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.306 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:42.307 DEBUG:boto:Data: 2022-09-02T05:59:42.307 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.308 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.309 DEBUG:boto:Params: {} 2022-09-02T05:59:42.309 DEBUG:boto:Token: None 2022-09-02T05:59:42.310 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87 2022-09-02T05:59:42.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mdcsRbJwo3mr37ZKAILnX5pX3EA= 2022-09-02T05:59:42.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mdcsRbJwo3mr37ZKAILnX5pX3EA=', 'Content-Length': '0'} 2022-09-02T05:59:42.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce559be3f388200a-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:42.315 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.316 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.317 DEBUG:boto:Method: GET 2022-09-02T05:59:42.317 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:42.318 DEBUG:boto:Data: 2022-09-02T05:59:42.318 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.319 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.320 DEBUG:boto:Params: {} 2022-09-02T05:59:42.321 DEBUG:boto:Token: None 2022-09-02T05:59:42.321 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87 2022-09-02T05:59:42.322 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mATvBHO0hTCesLMUcx9NXkITrH4= 2022-09-02T05:59:42.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mATvBHO0hTCesLMUcx9NXkITrH4=', 'Content-Length': '0'} 2022-09-02T05:59:42.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae42746860fdee4-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.366 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:42.367 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:42.367 DEBUG:boto:Method: GET 2022-09-02T05:59:42.368 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:42.368 DEBUG:boto:Data: 2022-09-02T05:59:42.369 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.370 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.371 DEBUG:boto:Params: {} 2022-09-02T05:59:42.372 DEBUG:boto:Token: None 2022-09-02T05:59:42.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj87 2022-09-02T05:59:42.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mATvBHO0hTCesLMUcx9NXkITrH4= 2022-09-02T05:59:42.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mATvBHO0hTCesLMUcx9NXkITrH4=', 'Content-Length': '0'} 2022-09-02T05:59:42.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd228b12992a5fcf-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.376 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.377 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.378 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.378 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.379 DEBUG:boto:Method: GET 2022-09-02T05:59:42.380 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.380 DEBUG:boto:Data: 2022-09-02T05:59:42.381 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.382 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.383 DEBUG:boto:Params: {} 2022-09-02T05:59:42.384 DEBUG:boto:Token: None 2022-09-02T05:59:42.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.385 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba37e0e86c73d0f-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.389 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.390 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.391 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.391 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.392 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.393 DEBUG:boto:Method: GET 2022-09-02T05:59:42.394 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:42.395 DEBUG:boto:Data: 2022-09-02T05:59:42.396 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.397 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.398 DEBUG:boto:Params: {} 2022-09-02T05:59:42.399 DEBUG:boto:Token: None 2022-09-02T05:59:42.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.401 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000badb7cc4544d6b28-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.407 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:42.408 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.409 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.409 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.410 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.410 DEBUG:boto:Method: GET 2022-09-02T05:59:42.411 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.411 DEBUG:boto:Data: 2022-09-02T05:59:42.412 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.413 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.413 DEBUG:boto:Params: {} 2022-09-02T05:59:42.414 DEBUG:boto:Token: None 2022-09-02T05:59:42.415 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.415 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw= 2022-09-02T05:59:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw=', 'Content-Length': '0'} 2022-09-02T05:59:42.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096867550d0e8c25b-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.420 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.421 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.421 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.422 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.422 DEBUG:boto:Method: GET 2022-09-02T05:59:42.423 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.423 DEBUG:boto:Data: 2022-09-02T05:59:42.424 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.425 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.426 DEBUG:boto:Params: {} 2022-09-02T05:59:42.427 DEBUG:boto:Token: None 2022-09-02T05:59:42.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw= 2022-09-02T05:59:42.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw=', 'Content-Length': '0'} 2022-09-02T05:59:42.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e6f6cb2510f5a2-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.432 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.432 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.433 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.433 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.434 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.434 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.435 DEBUG:boto:Data: 2022-09-02T05:59:42.435 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.436 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.437 DEBUG:boto:Params: {} 2022-09-02T05:59:42.437 DEBUG:boto:Token: None 2022-09-02T05:59:42.438 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EXzYS6R7sImvNA1WnYObf1MyZGY= 2022-09-02T05:59:42.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EXzYS6R7sImvNA1WnYObf1MyZGY=', 'Content-Length': '0'} 2022-09-02T05:59:42.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d037406e775ad0-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.442 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.443 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.443 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.444 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.444 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.445 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.445 DEBUG:boto:Data: 2022-09-02T05:59:42.446 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.447 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.447 DEBUG:boto:Params: {} 2022-09-02T05:59:42.448 DEBUG:boto:Token: None 2022-09-02T05:59:42.449 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.449 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EXzYS6R7sImvNA1WnYObf1MyZGY= 2022-09-02T05:59:42.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EXzYS6R7sImvNA1WnYObf1MyZGY=', 'Content-Length': '0'} 2022-09-02T05:59:42.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a99e3f9902f838-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:42.455 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.456 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.457 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.458 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.459 DEBUG:boto:Method: GET 2022-09-02T05:59:42.460 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.461 DEBUG:boto:Data: 2022-09-02T05:59:42.462 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.464 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.464 DEBUG:boto:Params: {} 2022-09-02T05:59:42.465 DEBUG:boto:Token: None 2022-09-02T05:59:42.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.466 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw= 2022-09-02T05:59:42.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw=', 'Content-Length': '0'} 2022-09-02T05:59:42.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3e2a37639452d7-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.470 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.471 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.471 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.472 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.472 DEBUG:boto:Method: GET 2022-09-02T05:59:42.473 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.473 DEBUG:boto:Data: 2022-09-02T05:59:42.474 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.475 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.476 DEBUG:boto:Params: {} 2022-09-02T05:59:42.476 DEBUG:boto:Token: None 2022-09-02T05:59:42.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:42.478 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw= 2022-09-02T05:59:42.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nSSgle6EpwyMJvCqPwu6AF9jQdw=', 'Content-Length': '0'} 2022-09-02T05:59:42.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620d01147f517a3e-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.526 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.527 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.528 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.528 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.529 DEBUG:boto:Method: GET 2022-09-02T05:59:42.530 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.530 DEBUG:boto:Data: 2022-09-02T05:59:42.531 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.532 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.532 DEBUG:boto:Params: {} 2022-09-02T05:59:42.533 DEBUG:boto:Token: None 2022-09-02T05:59:42.534 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce6f2f47115863c2-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.538 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.539 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.540 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.540 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.541 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.541 DEBUG:boto:Method: GET 2022-09-02T05:59:42.542 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.543 DEBUG:boto:Data: 2022-09-02T05:59:42.543 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.544 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.545 DEBUG:boto:Params: {} 2022-09-02T05:59:42.545 DEBUG:boto:Token: None 2022-09-02T05:59:42.546 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.547 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780230e14fa74906-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.552 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.553 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.554 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.555 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.555 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:42.556 DEBUG:boto:Data: 2022-09-02T05:59:42.556 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.557 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.558 DEBUG:boto:Params: {} 2022-09-02T05:59:42.558 DEBUG:boto:Token: None 2022-09-02T05:59:42.559 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92 2022-09-02T05:59:42.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QHqyzAdNkI3MrPZFqA+j0MWCGeQ= 2022-09-02T05:59:42.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QHqyzAdNkI3MrPZFqA+j0MWCGeQ=', 'Content-Length': '0'} 2022-09-02T05:59:42.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4e3abd0aabe1217-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.563 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.564 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.564 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.565 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:42.565 DEBUG:boto:Data: 2022-09-02T05:59:42.566 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.567 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.567 DEBUG:boto:Params: {} 2022-09-02T05:59:42.568 DEBUG:boto:Token: None 2022-09-02T05:59:42.569 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92 2022-09-02T05:59:42.569 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QHqyzAdNkI3MrPZFqA+j0MWCGeQ= 2022-09-02T05:59:42.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QHqyzAdNkI3MrPZFqA+j0MWCGeQ=', 'Content-Length': '0'} 2022-09-02T05:59:42.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f912484e5cc7d5-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:42.573 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.574 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.575 DEBUG:boto:Method: GET 2022-09-02T05:59:42.575 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:42.576 DEBUG:boto:Data: 2022-09-02T05:59:42.577 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.578 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.578 DEBUG:boto:Params: {} 2022-09-02T05:59:42.579 DEBUG:boto:Token: None 2022-09-02T05:59:42.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92 2022-09-02T05:59:42.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dp3ct+ymDM2hpRj+zDbBFgbl4FI= 2022-09-02T05:59:42.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dp3ct+ymDM2hpRj+zDbBFgbl4FI=', 'Content-Length': '0'} 2022-09-02T05:59:42.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead6cc2a4c944528-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.630 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:42.631 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:42.631 DEBUG:boto:Method: GET 2022-09-02T05:59:42.632 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:42.633 DEBUG:boto:Data: 2022-09-02T05:59:42.633 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.634 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.635 DEBUG:boto:Params: {} 2022-09-02T05:59:42.635 DEBUG:boto:Token: None 2022-09-02T05:59:42.636 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj92 2022-09-02T05:59:42.637 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dp3ct+ymDM2hpRj+zDbBFgbl4FI= 2022-09-02T05:59:42.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dp3ct+ymDM2hpRj+zDbBFgbl4FI=', 'Content-Length': '0'} 2022-09-02T05:59:42.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7ab755fd4bc7c2-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.641 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.641 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.642 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.642 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.643 DEBUG:boto:Method: GET 2022-09-02T05:59:42.644 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.644 DEBUG:boto:Data: 2022-09-02T05:59:42.645 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.646 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.646 DEBUG:boto:Params: {} 2022-09-02T05:59:42.647 DEBUG:boto:Token: None 2022-09-02T05:59:42.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.648 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f190b636e2a457-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.652 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.653 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.654 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.654 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.655 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.656 DEBUG:boto:Method: GET 2022-09-02T05:59:42.656 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:42.657 DEBUG:boto:Data: 2022-09-02T05:59:42.657 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.658 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.659 DEBUG:boto:Params: {} 2022-09-02T05:59:42.659 DEBUG:boto:Token: None 2022-09-02T05:59:42.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ff0e04d5ea69e2-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.667 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:42.668 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.669 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.669 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.670 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.670 DEBUG:boto:Method: GET 2022-09-02T05:59:42.671 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.672 DEBUG:boto:Data: 2022-09-02T05:59:42.673 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.674 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.675 DEBUG:boto:Params: {} 2022-09-02T05:59:42.676 DEBUG:boto:Token: None 2022-09-02T05:59:42.676 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.677 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU= 2022-09-02T05:59:42.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU=', 'Content-Length': '0'} 2022-09-02T05:59:42.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b21075a7e502f9b-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.683 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.683 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.684 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.685 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.685 DEBUG:boto:Method: GET 2022-09-02T05:59:42.686 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.686 DEBUG:boto:Data: 2022-09-02T05:59:42.687 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.688 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.689 DEBUG:boto:Params: {} 2022-09-02T05:59:42.690 DEBUG:boto:Token: None 2022-09-02T05:59:42.690 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.691 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU= 2022-09-02T05:59:42.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU=', 'Content-Length': '0'} 2022-09-02T05:59:42.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005acaebef15652575-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.738 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.739 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.740 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.740 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.741 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.742 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.742 DEBUG:boto:Data: 2022-09-02T05:59:42.743 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.744 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.744 DEBUG:boto:Params: {} 2022-09-02T05:59:42.745 DEBUG:boto:Token: None 2022-09-02T05:59:42.746 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2jVwTolFBUZfL01xKbQQRj1yMNc= 2022-09-02T05:59:42.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2jVwTolFBUZfL01xKbQQRj1yMNc=', 'Content-Length': '0'} 2022-09-02T05:59:42.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de93eba6085e154-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.751 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.751 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.752 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.761 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.761 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.762 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.762 DEBUG:boto:Data: 2022-09-02T05:59:42.763 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.764 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.765 DEBUG:boto:Params: {} 2022-09-02T05:59:42.765 DEBUG:boto:Token: None 2022-09-02T05:59:42.766 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2jVwTolFBUZfL01xKbQQRj1yMNc= 2022-09-02T05:59:42.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2jVwTolFBUZfL01xKbQQRj1yMNc=', 'Content-Length': '0'} 2022-09-02T05:59:42.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3592490757e4f04-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:42.772 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.773 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.774 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.774 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.775 DEBUG:boto:Method: GET 2022-09-02T05:59:42.775 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.778 DEBUG:boto:Data: 2022-09-02T05:59:42.779 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.781 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.782 DEBUG:boto:Params: {} 2022-09-02T05:59:42.783 DEBUG:boto:Token: None 2022-09-02T05:59:42.785 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.786 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU= 2022-09-02T05:59:42.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU=', 'Content-Length': '0'} 2022-09-02T05:59:42.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da4ad2e363af620-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.790 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.791 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.792 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.792 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.793 DEBUG:boto:Method: GET 2022-09-02T05:59:42.793 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.794 DEBUG:boto:Data: 2022-09-02T05:59:42.795 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.796 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.796 DEBUG:boto:Params: {} 2022-09-02T05:59:42.797 DEBUG:boto:Token: None 2022-09-02T05:59:42.798 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:42.799 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU= 2022-09-02T05:59:42.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fgPgb3Y2IKrE4vnCHdVnbD4+sqU=', 'Content-Length': '0'} 2022-09-02T05:59:42.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659329b32ae2572c-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.804 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.805 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.805 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.806 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.807 DEBUG:boto:Method: GET 2022-09-02T05:59:42.807 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.808 DEBUG:boto:Data: 2022-09-02T05:59:42.808 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.809 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.810 DEBUG:boto:Params: {} 2022-09-02T05:59:42.811 DEBUG:boto:Token: None 2022-09-02T05:59:42.812 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002666f901cb30699e-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.820 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.822 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.823 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.824 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.826 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.827 DEBUG:boto:Method: GET 2022-09-02T05:59:42.828 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.829 DEBUG:boto:Data: 2022-09-02T05:59:42.830 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.833 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.835 DEBUG:boto:Params: {} 2022-09-02T05:59:42.837 DEBUG:boto:Token: None 2022-09-02T05:59:42.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580faf927b6218c5-0063119bce-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.850 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:42.851 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.852 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.853 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.854 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:42.855 DEBUG:boto:Data: 2022-09-02T05:59:42.856 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.857 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.859 DEBUG:boto:Params: {} 2022-09-02T05:59:42.860 DEBUG:boto:Token: None 2022-09-02T05:59:42.861 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93 2022-09-02T05:59:42.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2SPjDcsA2rXwFgmrwE5oH+FQBw= 2022-09-02T05:59:42.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2SPjDcsA2rXwFgmrwE5oH+FQBw=', 'Content-Length': '0'} 2022-09-02T05:59:42.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc2703579a01fba-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.874 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.874 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.877 DEBUG:boto:Method: HEAD 2022-09-02T05:59:42.878 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:42.879 DEBUG:boto:Data: 2022-09-02T05:59:42.880 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.883 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.884 DEBUG:boto:Params: {} 2022-09-02T05:59:42.886 DEBUG:boto:Token: None 2022-09-02T05:59:42.887 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93 2022-09-02T05:59:42.888 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:r2SPjDcsA2rXwFgmrwE5oH+FQBw= 2022-09-02T05:59:42.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:r2SPjDcsA2rXwFgmrwE5oH+FQBw=', 'Content-Length': '0'} 2022-09-02T05:59:42.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057273354409a73eb-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:42.895 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.896 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.897 DEBUG:boto:Method: GET 2022-09-02T05:59:42.900 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:42.901 DEBUG:boto:Data: 2022-09-02T05:59:42.902 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.904 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.905 DEBUG:boto:Params: {} 2022-09-02T05:59:42.906 DEBUG:boto:Token: None 2022-09-02T05:59:42.907 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93 2022-09-02T05:59:42.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/UQ6i5GxH9asaChPDH2tvO2cu1k= 2022-09-02T05:59:42.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/UQ6i5GxH9asaChPDH2tvO2cu1k=', 'Content-Length': '0'} 2022-09-02T05:59:42.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4163b360ef8145-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.962 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:42.963 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:42.964 DEBUG:boto:Method: GET 2022-09-02T05:59:42.965 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:42.965 DEBUG:boto:Data: 2022-09-02T05:59:42.966 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:42.967 DEBUG:boto:Port: 8000 2022-09-02T05:59:42.968 DEBUG:boto:Params: {} 2022-09-02T05:59:42.969 DEBUG:boto:Token: None 2022-09-02T05:59:42.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/obj93 2022-09-02T05:59:42.970 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/UQ6i5GxH9asaChPDH2tvO2cu1k= 2022-09-02T05:59:42.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/UQ6i5GxH9asaChPDH2tvO2cu1k=', 'Content-Length': '0'} 2022-09-02T05:59:42.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248a9e827f2f4200-0063119bce-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.979 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:42.980 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:42.980 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.981 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.981 DEBUG:boto:Method: GET 2022-09-02T05:59:42.982 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:42.983 DEBUG:boto:Data: 2022-09-02T05:59:42.983 DEBUG:boto:Headers: {} 2022-09-02T05:59:42.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:42.984 DEBUG:boto:Port: 8001 2022-09-02T05:59:42.985 DEBUG:boto:Params: {} 2022-09-02T05:59:42.985 DEBUG:boto:Token: None 2022-09-02T05:59:42.986 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:42 GMT /xljbkf-74/ 2022-09-02T05:59:42.987 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk= 2022-09-02T05:59:42.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:42 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j2gFLoyPHnLj4ecsx8YVJ/tHULk=', 'Content-Length': '0'} 2022-09-02T05:59:42.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773612fda15af84c-0063119bce-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:42 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:42.998 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.000 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.001 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.002 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:43.003 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:43.004 DEBUG:boto:Method: GET 2022-09-02T05:59:43.005 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:43.006 DEBUG:boto:Data: 2022-09-02T05:59:43.007 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.009 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.010 DEBUG:boto:Params: {} 2022-09-02T05:59:43.011 DEBUG:boto:Token: None 2022-09-02T05:59:43.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.013 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa86d62a4a94223e-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.023 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:43.025 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.026 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.027 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.028 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.029 DEBUG:boto:Method: GET 2022-09-02T05:59:43.030 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.031 DEBUG:boto:Data: 2022-09-02T05:59:43.032 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.035 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.036 DEBUG:boto:Params: {} 2022-09-02T05:59:43.037 DEBUG:boto:Token: None 2022-09-02T05:59:43.039 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.040 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE= 2022-09-02T05:59:43.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE=', 'Content-Length': '0'} 2022-09-02T05:59:43.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3977bc2ee2154c5-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.049 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.050 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.051 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.051 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.052 DEBUG:boto:Method: GET 2022-09-02T05:59:43.053 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.053 DEBUG:boto:Data: 2022-09-02T05:59:43.054 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.055 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.056 DEBUG:boto:Params: {} 2022-09-02T05:59:43.056 DEBUG:boto:Token: None 2022-09-02T05:59:43.057 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.058 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE= 2022-09-02T05:59:43.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE=', 'Content-Length': '0'} 2022-09-02T05:59:43.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e115a541a3d60272-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.072 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.073 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.074 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.075 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.075 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.076 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.076 DEBUG:boto:Data: 2022-09-02T05:59:43.077 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.078 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.078 DEBUG:boto:Params: {} 2022-09-02T05:59:43.079 DEBUG:boto:Token: None 2022-09-02T05:59:43.080 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VxwCTlbvhPc2ZwEV4jtRQGWQOqY= 2022-09-02T05:59:43.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VxwCTlbvhPc2ZwEV4jtRQGWQOqY=', 'Content-Length': '0'} 2022-09-02T05:59:43.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b107b1919c198b55-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.087 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.088 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.088 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.089 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.089 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.090 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.091 DEBUG:boto:Data: 2022-09-02T05:59:43.091 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.092 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.093 DEBUG:boto:Params: {} 2022-09-02T05:59:43.093 DEBUG:boto:Token: None 2022-09-02T05:59:43.094 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:VxwCTlbvhPc2ZwEV4jtRQGWQOqY= 2022-09-02T05:59:43.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:VxwCTlbvhPc2ZwEV4jtRQGWQOqY=', 'Content-Length': '0'} 2022-09-02T05:59:43.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060531119d9c4a8ea-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:43.099 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.100 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.101 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.101 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.102 DEBUG:boto:Method: GET 2022-09-02T05:59:43.102 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.103 DEBUG:boto:Data: 2022-09-02T05:59:43.103 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.105 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.106 DEBUG:boto:Params: {} 2022-09-02T05:59:43.106 DEBUG:boto:Token: None 2022-09-02T05:59:43.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE= 2022-09-02T05:59:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE=', 'Content-Length': '0'} 2022-09-02T05:59:43.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7271bde54946768-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.162 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.163 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.163 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.164 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.164 DEBUG:boto:Method: GET 2022-09-02T05:59:43.165 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.166 DEBUG:boto:Data: 2022-09-02T05:59:43.166 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.167 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.168 DEBUG:boto:Params: {} 2022-09-02T05:59:43.168 DEBUG:boto:Token: None 2022-09-02T05:59:43.169 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:43.170 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE= 2022-09-02T05:59:43.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:lk0exr0JMdjcgJ99wvhAYcH1zFE=', 'Content-Length': '0'} 2022-09-02T05:59:43.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816e7e39f48a396c-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.177 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.178 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.178 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.179 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.179 DEBUG:boto:Method: GET 2022-09-02T05:59:43.180 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.181 DEBUG:boto:Data: 2022-09-02T05:59:43.181 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.182 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.183 DEBUG:boto:Params: {} 2022-09-02T05:59:43.183 DEBUG:boto:Token: None 2022-09-02T05:59:43.184 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080335d9b57c1aed8-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.190 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.191 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.192 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.192 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.193 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.194 DEBUG:boto:Method: GET 2022-09-02T05:59:43.194 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.195 DEBUG:boto:Data: 2022-09-02T05:59:43.195 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.198 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.199 DEBUG:boto:Params: {} 2022-09-02T05:59:43.199 DEBUG:boto:Token: None 2022-09-02T05:59:43.200 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.201 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d621dbfbb95ef6fa-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.207 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.208 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.209 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.209 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.210 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:43.211 DEBUG:boto:Data: 2022-09-02T05:59:43.212 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.213 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.214 DEBUG:boto:Params: {} 2022-09-02T05:59:43.214 DEBUG:boto:Token: None 2022-09-02T05:59:43.215 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95 2022-09-02T05:59:43.216 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JjuwcEPL8epKFO/gWAyyQiy9f9Y= 2022-09-02T05:59:43.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JjuwcEPL8epKFO/gWAyyQiy9f9Y=', 'Content-Length': '0'} 2022-09-02T05:59:43.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b4047b14544eb69-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.220 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.220 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.221 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.221 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:43.222 DEBUG:boto:Data: 2022-09-02T05:59:43.223 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.224 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.225 DEBUG:boto:Params: {} 2022-09-02T05:59:43.226 DEBUG:boto:Token: None 2022-09-02T05:59:43.226 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95 2022-09-02T05:59:43.227 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JjuwcEPL8epKFO/gWAyyQiy9f9Y= 2022-09-02T05:59:43.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JjuwcEPL8epKFO/gWAyyQiy9f9Y=', 'Content-Length': '0'} 2022-09-02T05:59:43.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7414a90620b4f06-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:43.232 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.232 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.233 DEBUG:boto:Method: GET 2022-09-02T05:59:43.234 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:43.234 DEBUG:boto:Data: 2022-09-02T05:59:43.235 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.236 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.237 DEBUG:boto:Params: {} 2022-09-02T05:59:43.238 DEBUG:boto:Token: None 2022-09-02T05:59:43.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95 2022-09-02T05:59:43.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oNmkUmvrBvdXmZgZ0mCEF1RQP1U= 2022-09-02T05:59:43.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oNmkUmvrBvdXmZgZ0mCEF1RQP1U=', 'Content-Length': '0'} 2022-09-02T05:59:43.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a4045ee3a5f0815-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.243 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:43.244 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:43.245 DEBUG:boto:Method: GET 2022-09-02T05:59:43.245 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:43.246 DEBUG:boto:Data: 2022-09-02T05:59:43.247 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.248 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.249 DEBUG:boto:Params: {} 2022-09-02T05:59:43.250 DEBUG:boto:Token: None 2022-09-02T05:59:43.251 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj95 2022-09-02T05:59:43.252 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oNmkUmvrBvdXmZgZ0mCEF1RQP1U= 2022-09-02T05:59:43.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oNmkUmvrBvdXmZgZ0mCEF1RQP1U=', 'Content-Length': '0'} 2022-09-02T05:59:43.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91b2c07a966417f-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.302 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.303 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.304 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.304 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.305 DEBUG:boto:Method: GET 2022-09-02T05:59:43.306 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.306 DEBUG:boto:Data: 2022-09-02T05:59:43.307 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.308 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.309 DEBUG:boto:Params: {} 2022-09-02T05:59:43.309 DEBUG:boto:Token: None 2022-09-02T05:59:43.310 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.311 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae9b9ca61a5c512-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.315 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.316 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.316 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.317 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.317 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.318 DEBUG:boto:Method: GET 2022-09-02T05:59:43.319 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:43.319 DEBUG:boto:Data: 2022-09-02T05:59:43.320 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T05:59:43.321 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.321 DEBUG:boto:Params: {} 2022-09-02T05:59:43.322 DEBUG:boto:Token: None 2022-09-02T05:59:43.322 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.323 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000709711eb59d86c4e-0063119bcf-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.329 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone1} 2022-09-02T05:59:43.331 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:59:43.331 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.332 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.332 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.333 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:59:43.333 DEBUG:boto:Data: 2022-09-02T05:59:43.334 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.335 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.336 DEBUG:boto:Params: {} 2022-09-02T05:59:43.336 DEBUG:boto:Token: None 2022-09-02T05:59:43.337 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fxgi5EXz3ZRAp+8e1Hsvzema/P4= 2022-09-02T05:59:43.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fxgi5EXz3ZRAp+8e1Hsvzema/P4=', 'Content-Length': '0'} 2022-09-02T05:59:43.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab1c734b053b7162-0063119bcf-16ef-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.342 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.342 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.343 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.343 DEBUG:boto:Path: /xljbkf-74/ 2022-09-02T05:59:43.344 DEBUG:boto:Data: 2022-09-02T05:59:43.344 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.345 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.345 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.346 DEBUG:boto:Params: {} 2022-09-02T05:59:43.347 DEBUG:boto:Token: None 2022-09-02T05:59:43.347 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.348 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Fxgi5EXz3ZRAp+8e1Hsvzema/P4= 2022-09-02T05:59:43.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Fxgi5EXz3ZRAp+8e1Hsvzema/P4=', 'Content-Length': '0'} 2022-09-02T05:59:43.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a00014d6cab27acf-0063119bcf-186e-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-09-02T05:59:43.355 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.355 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.356 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.356 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.357 DEBUG:boto:Method: GET 2022-09-02T05:59:43.357 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.358 DEBUG:boto:Data: 2022-09-02T05:59:43.358 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.360 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.360 DEBUG:boto:Params: {} 2022-09-02T05:59:43.361 DEBUG:boto:Token: None 2022-09-02T05:59:43.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/?versions 2022-09-02T05:59:43.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38= 2022-09-02T05:59:43.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38=', 'Content-Length': '0'} 2022-09-02T05:59:43.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e35981b0854089db-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.366 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:59:43.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-02T05:59:43.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:59:43.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:59:43.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:59:43.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:59:43.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-02T05:59:43.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:59:43.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:59:43.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-02T05:59:43.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:59:43.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:59:43.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:59:43.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:59:43.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:59:43.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:59:43.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:59:43.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-02T05:59:43.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:59:43.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:59:43.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:59:43.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:59:43.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:59:43.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:59:43.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:59:43.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:59:43.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:59:43.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:59:43.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-02T05:59:43.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:59:43.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:59:43.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:59:43.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:59:43.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:59:43.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:59:43.386 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-09-02T05:59:43.387 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.388 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.388 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.389 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.389 DEBUG:boto:Method: GET 2022-09-02T05:59:43.390 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.390 DEBUG:boto:Data: 2022-09-02T05:59:43.391 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.392 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.393 DEBUG:boto:Params: {} 2022-09-02T05:59:43.393 DEBUG:boto:Token: None 2022-09-02T05:59:43.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/?versions 2022-09-02T05:59:43.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38= 2022-09-02T05:59:43.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38=', 'Content-Length': '0'} 2022-09-02T05:59:43.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004351c445cd13983c-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.400 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2022-09-02T05:59:43.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2022-09-02T05:59:43.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2022-09-02T05:59:43.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2022-09-02T05:59:43.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2022-09-02T05:59:43.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2022-09-02T05:59:43.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2022-09-02T05:59:43.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2022-09-02T05:59:43.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2022-09-02T05:59:43.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2022-09-02T05:59:43.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2022-09-02T05:59:43.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2022-09-02T05:59:43.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2022-09-02T05:59:43.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2022-09-02T05:59:43.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2022-09-02T05:59:43.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2022-09-02T05:59:43.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2022-09-02T05:59:43.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2022-09-02T05:59:43.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2022-09-02T05:59:43.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2022-09-02T05:59:43.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2022-09-02T05:59:43.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2022-09-02T05:59:43.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2022-09-02T05:59:43.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2022-09-02T05:59:43.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2022-09-02T05:59:43.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2022-09-02T05:59:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2022-09-02T05:59:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2022-09-02T05:59:43.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2022-09-02T05:59:43.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2022-09-02T05:59:43.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2022-09-02T05:59:43.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2022-09-02T05:59:43.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2022-09-02T05:59:43.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2022-09-02T05:59:43.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2022-09-02T05:59:43.421 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.422 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.422 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.423 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.423 DEBUG:boto:Method: GET 2022-09-02T05:59:43.424 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.424 DEBUG:boto:Data: 2022-09-02T05:59:43.425 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.426 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.427 DEBUG:boto:Params: {} 2022-09-02T05:59:43.427 DEBUG:boto:Token: None 2022-09-02T05:59:43.428 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/?versions 2022-09-02T05:59:43.429 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38= 2022-09-02T05:59:43.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38=', 'Content-Length': '0'} 2022-09-02T05:59:43.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0391ae4e9485b44-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.434 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.437 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.438 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.439 DEBUG:boto:path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.439 DEBUG:boto:auth_path=/xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.440 DEBUG:boto:Method: GET 2022-09-02T05:59:43.440 DEBUG:boto:Path: /xljbkf-74/?versions&max-keys=999 2022-09-02T05:59:43.441 DEBUG:boto:Data: 2022-09-02T05:59:43.441 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.442 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.442 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.443 DEBUG:boto:Params: {} 2022-09-02T05:59:43.443 DEBUG:boto:Token: None 2022-09-02T05:59:43.444 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/?versions 2022-09-02T05:59:43.445 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38= 2022-09-02T05:59:43.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ObUSnKO/oDAs+JlaqgXuoaMGd38=', 'Content-Length': '0'} 2022-09-02T05:59:43.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736f7050e4b1a910-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.453 DEBUG:boto:b'xljbkf-74999falseobj11nulltrue2022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:43.457 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.458 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.459 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.459 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.460 DEBUG:boto:Method: GET 2022-09-02T05:59:43.461 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.462 DEBUG:boto:Data: 2022-09-02T05:59:43.462 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.464 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.465 DEBUG:boto:Params: {} 2022-09-02T05:59:43.465 DEBUG:boto:Token: None 2022-09-02T05:59:43.466 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.467 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A= 2022-09-02T05:59:43.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A=', 'Content-Length': '0'} 2022-09-02T05:59:43.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7cfb8956d83ac0a-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.475 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.476 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.477 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.478 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.480 DEBUG:boto:Method: GET 2022-09-02T05:59:43.481 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.482 DEBUG:boto:Data: 2022-09-02T05:59:43.483 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.484 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.485 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.486 DEBUG:boto:Params: {} 2022-09-02T05:59:43.487 DEBUG:boto:Token: None 2022-09-02T05:59:43.488 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A= 2022-09-02T05:59:43.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A=', 'Content-Length': '0'} 2022-09-02T05:59:43.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a905558716518895-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.498 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.499 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.500 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.501 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.502 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.503 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.503 DEBUG:boto:Data: 2022-09-02T05:59:43.504 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.506 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.507 DEBUG:boto:Params: {} 2022-09-02T05:59:43.508 DEBUG:boto:Token: None 2022-09-02T05:59:43.509 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.509 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RM9yo07dmRmKZxpsfWqWo0pM/R4= 2022-09-02T05:59:43.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RM9yo07dmRmKZxpsfWqWo0pM/R4=', 'Content-Length': '0'} 2022-09-02T05:59:43.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007705af56d05ff67a-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.517 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.518 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.519 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.520 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.521 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.522 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.523 DEBUG:boto:Data: 2022-09-02T05:59:43.524 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.524 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.525 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.525 DEBUG:boto:Params: {} 2022-09-02T05:59:43.526 DEBUG:boto:Token: None 2022-09-02T05:59:43.527 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.527 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:RM9yo07dmRmKZxpsfWqWo0pM/R4= 2022-09-02T05:59:43.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:RM9yo07dmRmKZxpsfWqWo0pM/R4=', 'Content-Length': '0'} 2022-09-02T05:59:43.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15680cec6ee900d-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:43.532 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.532 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.533 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.533 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.534 DEBUG:boto:Method: GET 2022-09-02T05:59:43.534 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.535 DEBUG:boto:Data: 2022-09-02T05:59:43.535 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.536 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.537 DEBUG:boto:Params: {} 2022-09-02T05:59:43.538 DEBUG:boto:Token: None 2022-09-02T05:59:43.540 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.541 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A= 2022-09-02T05:59:43.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A=', 'Content-Length': '0'} 2022-09-02T05:59:43.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338fd9cb8408cf82-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.550 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.551 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.552 DEBUG:boto:path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.553 DEBUG:boto:auth_path=/xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.553 DEBUG:boto:Method: GET 2022-09-02T05:59:43.554 DEBUG:boto:Path: /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.554 DEBUG:boto:Data: 2022-09-02T05:59:43.555 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.556 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.557 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.558 DEBUG:boto:Params: {} 2022-09-02T05:59:43.559 DEBUG:boto:Token: None 2022-09-02T05:59:43.559 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11?versionId=null 2022-09-02T05:59:43.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A= 2022-09-02T05:59:43.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HjL9JMrgP14/zjJz2JKEbjuEm2A=', 'Content-Length': '0'} 2022-09-02T05:59:43.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5a644771734da1-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.572 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.572 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.573 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.574 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.574 DEBUG:boto:Method: GET 2022-09-02T05:59:43.575 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.575 DEBUG:boto:Data: 2022-09-02T05:59:43.576 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.577 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.578 DEBUG:boto:Params: {} 2022-09-02T05:59:43.578 DEBUG:boto:Token: None 2022-09-02T05:59:43.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd5fdab06aa0ebab-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.595 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.596 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.596 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.597 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.597 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.598 DEBUG:boto:Method: GET 2022-09-02T05:59:43.599 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.599 DEBUG:boto:Data: 2022-09-02T05:59:43.600 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.601 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.601 DEBUG:boto:Params: {} 2022-09-02T05:59:43.602 DEBUG:boto:Token: None 2022-09-02T05:59:43.603 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.603 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a74459d8543b5d-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.610 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.611 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.612 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.613 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.614 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:43.614 DEBUG:boto:Data: 2022-09-02T05:59:43.615 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.617 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.617 DEBUG:boto:Params: {} 2022-09-02T05:59:43.618 DEBUG:boto:Token: None 2022-09-02T05:59:43.619 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11 2022-09-02T05:59:43.620 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hxt6BomU2Umx8+muV2eNVytEGZA= 2022-09-02T05:59:43.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hxt6BomU2Umx8+muV2eNVytEGZA=', 'Content-Length': '0'} 2022-09-02T05:59:43.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33917a9e59f481b-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.624 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.625 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.626 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.626 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:43.627 DEBUG:boto:Data: 2022-09-02T05:59:43.628 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.628 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.629 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.630 DEBUG:boto:Params: {} 2022-09-02T05:59:43.630 DEBUG:boto:Token: None 2022-09-02T05:59:43.631 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11 2022-09-02T05:59:43.632 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Hxt6BomU2Umx8+muV2eNVytEGZA= 2022-09-02T05:59:43.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Hxt6BomU2Umx8+muV2eNVytEGZA=', 'Content-Length': '0'} 2022-09-02T05:59:43.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc24c275d24f850-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2022-09-02T05:59:43.642 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.642 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.643 DEBUG:boto:Method: GET 2022-09-02T05:59:43.644 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:43.644 DEBUG:boto:Data: 2022-09-02T05:59:43.645 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.646 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.647 DEBUG:boto:Params: {} 2022-09-02T05:59:43.648 DEBUG:boto:Token: None 2022-09-02T05:59:43.648 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11 2022-09-02T05:59:43.649 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cK5KPsVNNz2yMBp6YZLk0ZLNWsQ= 2022-09-02T05:59:43.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cK5KPsVNNz2yMBp6YZLk0ZLNWsQ=', 'Content-Length': '0'} 2022-09-02T05:59:43.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eda1041815b9f52-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.694 DEBUG:boto:path=/xljbkf-74/obj11 2022-09-02T05:59:43.697 DEBUG:boto:auth_path=/xljbkf-74/obj11 2022-09-02T05:59:43.698 DEBUG:boto:Method: GET 2022-09-02T05:59:43.698 DEBUG:boto:Path: /xljbkf-74/obj11 2022-09-02T05:59:43.699 DEBUG:boto:Data: 2022-09-02T05:59:43.700 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.701 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.702 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.703 DEBUG:boto:Params: {} 2022-09-02T05:59:43.703 DEBUG:boto:Token: None 2022-09-02T05:59:43.704 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj11 2022-09-02T05:59:43.705 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cK5KPsVNNz2yMBp6YZLk0ZLNWsQ= 2022-09-02T05:59:43.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cK5KPsVNNz2yMBp6YZLk0ZLNWsQ=', 'Content-Length': '0'} 2022-09-02T05:59:43.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006272f1b7c6ec72c4-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.709 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.709 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.710 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.711 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.712 DEBUG:boto:Method: GET 2022-09-02T05:59:43.713 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.713 DEBUG:boto:Data: 2022-09-02T05:59:43.714 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.716 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.716 DEBUG:boto:Params: {} 2022-09-02T05:59:43.717 DEBUG:boto:Token: None 2022-09-02T05:59:43.718 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.718 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c044c2be438adaa0-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.723 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.724 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.725 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.726 DEBUG:boto:path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.727 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.728 DEBUG:boto:Method: GET 2022-09-02T05:59:43.728 DEBUG:boto:Path: /xljbkf-74/?prefix=obj11 2022-09-02T05:59:43.729 DEBUG:boto:Data: 2022-09-02T05:59:43.730 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.731 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.731 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.732 DEBUG:boto:Params: {} 2022-09-02T05:59:43.733 DEBUG:boto:Token: None 2022-09-02T05:59:43.733 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.734 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4185aae65aeaf4-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.740 DEBUG:boto:b'xljbkf-74obj111000falseobj112022-09-02T05:44:47.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:43.742 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.743 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.744 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.745 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.746 DEBUG:boto:Method: GET 2022-09-02T05:59:43.746 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.747 DEBUG:boto:Data: 2022-09-02T05:59:43.748 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.749 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.750 DEBUG:boto:Params: {} 2022-09-02T05:59:43.751 DEBUG:boto:Token: None 2022-09-02T05:59:43.752 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g= 2022-09-02T05:59:43.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g=', 'Content-Length': '0'} 2022-09-02T05:59:43.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d6614d4dc2bb66f-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.759 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.759 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.760 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.760 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.761 DEBUG:boto:Method: GET 2022-09-02T05:59:43.761 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.762 DEBUG:boto:Data: 2022-09-02T05:59:43.763 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.763 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.764 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.764 DEBUG:boto:Params: {} 2022-09-02T05:59:43.765 DEBUG:boto:Token: None 2022-09-02T05:59:43.765 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.766 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g= 2022-09-02T05:59:43.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g=', 'Content-Length': '0'} 2022-09-02T05:59:43.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000204fcfc02e61f1f0-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.772 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.773 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.773 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.774 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.774 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.775 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.775 DEBUG:boto:Data: 2022-09-02T05:59:43.776 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.777 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.777 DEBUG:boto:Params: {} 2022-09-02T05:59:43.778 DEBUG:boto:Token: None 2022-09-02T05:59:43.778 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BjgZ/ZhtHo0cE/4LcDj+TSI2zk8= 2022-09-02T05:59:43.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BjgZ/ZhtHo0cE/4LcDj+TSI2zk8=', 'Content-Length': '0'} 2022-09-02T05:59:43.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002490ad6c9a914faa-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.783 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.783 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.784 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.784 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.785 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.785 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.786 DEBUG:boto:Data: 2022-09-02T05:59:43.787 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.787 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.788 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.788 DEBUG:boto:Params: {} 2022-09-02T05:59:43.789 DEBUG:boto:Token: None 2022-09-02T05:59:43.790 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.790 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BjgZ/ZhtHo0cE/4LcDj+TSI2zk8= 2022-09-02T05:59:43.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BjgZ/ZhtHo0cE/4LcDj+TSI2zk8=', 'Content-Length': '0'} 2022-09-02T05:59:43.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7254fa0688be520-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:43.795 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.796 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.796 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.797 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.797 DEBUG:boto:Method: GET 2022-09-02T05:59:43.798 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.798 DEBUG:boto:Data: 2022-09-02T05:59:43.799 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.800 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.801 DEBUG:boto:Params: {} 2022-09-02T05:59:43.802 DEBUG:boto:Token: None 2022-09-02T05:59:43.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g= 2022-09-02T05:59:43.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g=', 'Content-Length': '0'} 2022-09-02T05:59:43.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58e8df06dd9e070-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.851 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.852 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.853 DEBUG:boto:path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.854 DEBUG:boto:auth_path=/xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.855 DEBUG:boto:Method: GET 2022-09-02T05:59:43.855 DEBUG:boto:Path: /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.856 DEBUG:boto:Data: 2022-09-02T05:59:43.857 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.858 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.858 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.859 DEBUG:boto:Params: {} 2022-09-02T05:59:43.860 DEBUG:boto:Token: None 2022-09-02T05:59:43.861 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111?versionId=null 2022-09-02T05:59:43.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g= 2022-09-02T05:59:43.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:xy+rkUEAfawC7vAdxLJNajDAG5g=', 'Content-Length': '0'} 2022-09-02T05:59:43.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3828281ec85a905-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.865 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.866 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.867 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.867 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.868 DEBUG:boto:Method: GET 2022-09-02T05:59:43.869 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.869 DEBUG:boto:Data: 2022-09-02T05:59:43.871 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.873 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.874 DEBUG:boto:Params: {} 2022-09-02T05:59:43.875 DEBUG:boto:Token: None 2022-09-02T05:59:43.876 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.877 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000458dfe10440eee90-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.883 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.884 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.885 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.885 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.886 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.887 DEBUG:boto:Method: GET 2022-09-02T05:59:43.887 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.888 DEBUG:boto:Data: 2022-09-02T05:59:43.888 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.889 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.890 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.890 DEBUG:boto:Params: {} 2022-09-02T05:59:43.891 DEBUG:boto:Token: None 2022-09-02T05:59:43.892 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:43.892 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:43.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:43.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa51922ef9650a7a-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.898 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:43.900 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.901 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.901 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.902 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:43.902 DEBUG:boto:Data: 2022-09-02T05:59:43.903 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.904 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.905 DEBUG:boto:Params: {} 2022-09-02T05:59:43.905 DEBUG:boto:Token: None 2022-09-02T05:59:43.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111 2022-09-02T05:59:43.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMrXY2b6XhnEWz2g+X2G4D0qnak= 2022-09-02T05:59:43.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMrXY2b6XhnEWz2g+X2G4D0qnak=', 'Content-Length': '0'} 2022-09-02T05:59:43.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b730acdb5c89130b-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.911 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.912 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.913 DEBUG:boto:Method: HEAD 2022-09-02T05:59:43.913 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:43.914 DEBUG:boto:Data: 2022-09-02T05:59:43.914 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.915 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.915 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.916 DEBUG:boto:Params: {} 2022-09-02T05:59:43.917 DEBUG:boto:Token: None 2022-09-02T05:59:43.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111 2022-09-02T05:59:43.918 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vMrXY2b6XhnEWz2g+X2G4D0qnak= 2022-09-02T05:59:43.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vMrXY2b6XhnEWz2g+X2G4D0qnak=', 'Content-Length': '0'} 2022-09-02T05:59:43.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a421ed104e206c2b-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2022-09-02T05:59:43.923 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.923 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.924 DEBUG:boto:Method: GET 2022-09-02T05:59:43.924 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:43.925 DEBUG:boto:Data: 2022-09-02T05:59:43.926 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.927 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.927 DEBUG:boto:Params: {} 2022-09-02T05:59:43.928 DEBUG:boto:Token: None 2022-09-02T05:59:43.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111 2022-09-02T05:59:43.931 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mxdv3w0s1OPYSfbWU6v2VK59aXk= 2022-09-02T05:59:43.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mxdv3w0s1OPYSfbWU6v2VK59aXk=', 'Content-Length': '0'} 2022-09-02T05:59:43.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3e1f2d07ebb983-0063119bcf-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.981 DEBUG:boto:path=/xljbkf-74/obj111 2022-09-02T05:59:43.982 DEBUG:boto:auth_path=/xljbkf-74/obj111 2022-09-02T05:59:43.982 DEBUG:boto:Method: GET 2022-09-02T05:59:43.983 DEBUG:boto:Path: /xljbkf-74/obj111 2022-09-02T05:59:43.983 DEBUG:boto:Data: 2022-09-02T05:59:43.984 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.985 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:43.985 DEBUG:boto:Port: 8000 2022-09-02T05:59:43.986 DEBUG:boto:Params: {} 2022-09-02T05:59:43.986 DEBUG:boto:Token: None 2022-09-02T05:59:43.987 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/obj111 2022-09-02T05:59:43.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mxdv3w0s1OPYSfbWU6v2VK59aXk= 2022-09-02T05:59:43.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mxdv3w0s1OPYSfbWU6v2VK59aXk=', 'Content-Length': '0'} 2022-09-02T05:59:43.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8c84ad9e852b45-0063119bcf-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:43 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:43.993 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:43.993 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:43.994 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.994 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.995 DEBUG:boto:Method: GET 2022-09-02T05:59:43.995 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:43.996 DEBUG:boto:Data: 2022-09-02T05:59:43.996 DEBUG:boto:Headers: {} 2022-09-02T05:59:43.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:43.998 DEBUG:boto:Port: 8001 2022-09-02T05:59:43.998 DEBUG:boto:Params: {} 2022-09-02T05:59:43.999 DEBUG:boto:Token: None 2022-09-02T05:59:43.999 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:43 GMT /xljbkf-74/ 2022-09-02T05:59:44.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs= 2022-09-02T05:59:44.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:43 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FGng6dG68MgD5NjKQ7pmrW8yaQs=', 'Content-Length': '0'} 2022-09-02T05:59:44.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78c2643c61aa51e-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.012 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.013 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.014 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.015 DEBUG:boto:path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:44.015 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj111 2022-09-02T05:59:44.016 DEBUG:boto:Method: GET 2022-09-02T05:59:44.016 DEBUG:boto:Path: /xljbkf-74/?prefix=obj111 2022-09-02T05:59:44.017 DEBUG:boto:Data: 2022-09-02T05:59:44.017 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.018 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.019 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.019 DEBUG:boto:Params: {} 2022-09-02T05:59:44.020 DEBUG:boto:Token: None 2022-09-02T05:59:44.020 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.021 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a69ebd0764b912a-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.070 DEBUG:boto:b'xljbkf-74obj1111000falseobj1112022-09-02T05:58:13.848Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:44.072 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.072 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.073 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.073 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.074 DEBUG:boto:Method: GET 2022-09-02T05:59:44.075 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.075 DEBUG:boto:Data: 2022-09-02T05:59:44.076 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.077 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.077 DEBUG:boto:Params: {} 2022-09-02T05:59:44.078 DEBUG:boto:Token: None 2022-09-02T05:59:44.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI= 2022-09-02T05:59:44.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI=', 'Content-Length': '0'} 2022-09-02T05:59:44.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf298c8504edc05d-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.083 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.084 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.085 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.085 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.086 DEBUG:boto:Method: GET 2022-09-02T05:59:44.086 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.087 DEBUG:boto:Data: 2022-09-02T05:59:44.087 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.088 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.088 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.089 DEBUG:boto:Params: {} 2022-09-02T05:59:44.090 DEBUG:boto:Token: None 2022-09-02T05:59:44.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI= 2022-09-02T05:59:44.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI=', 'Content-Length': '0'} 2022-09-02T05:59:44.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b29c696bab2b3ff8-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.095 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.095 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.096 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.097 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.097 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.098 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.098 DEBUG:boto:Data: 2022-09-02T05:59:44.099 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.100 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.100 DEBUG:boto:Params: {} 2022-09-02T05:59:44.101 DEBUG:boto:Token: None 2022-09-02T05:59:44.101 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.102 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L/0ih0g6d7FtRGqSJry6UX9Gl30= 2022-09-02T05:59:44.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L/0ih0g6d7FtRGqSJry6UX9Gl30=', 'Content-Length': '0'} 2022-09-02T05:59:44.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d827ee606a89b79a-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.106 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.106 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.107 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.107 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.108 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.109 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.109 DEBUG:boto:Data: 2022-09-02T05:59:44.110 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.110 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.111 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.111 DEBUG:boto:Params: {} 2022-09-02T05:59:44.112 DEBUG:boto:Token: None 2022-09-02T05:59:44.113 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.113 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:L/0ih0g6d7FtRGqSJry6UX9Gl30= 2022-09-02T05:59:44.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:L/0ih0g6d7FtRGqSJry6UX9Gl30=', 'Content-Length': '0'} 2022-09-02T05:59:44.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9ad71af0d52c8f1-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:44.118 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.118 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.119 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.120 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.120 DEBUG:boto:Method: GET 2022-09-02T05:59:44.121 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.121 DEBUG:boto:Data: 2022-09-02T05:59:44.122 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.123 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.124 DEBUG:boto:Params: {} 2022-09-02T05:59:44.124 DEBUG:boto:Token: None 2022-09-02T05:59:44.125 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.126 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI= 2022-09-02T05:59:44.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI=', 'Content-Length': '0'} 2022-09-02T05:59:44.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008828825b966cd827-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.129 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.130 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.131 DEBUG:boto:path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.131 DEBUG:boto:auth_path=/xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.132 DEBUG:boto:Method: GET 2022-09-02T05:59:44.133 DEBUG:boto:Path: /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.133 DEBUG:boto:Data: 2022-09-02T05:59:44.134 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.134 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.135 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.135 DEBUG:boto:Params: {} 2022-09-02T05:59:44.136 DEBUG:boto:Token: None 2022-09-02T05:59:44.137 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115?versionId=null 2022-09-02T05:59:44.138 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI= 2022-09-02T05:59:44.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:knMpxQx1iRKuIXSP8iBCLO/uaYI=', 'Content-Length': '0'} 2022-09-02T05:59:44.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce433dde081be03b-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.142 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.143 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.143 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.144 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.145 DEBUG:boto:Method: GET 2022-09-02T05:59:44.145 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.146 DEBUG:boto:Data: 2022-09-02T05:59:44.146 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.148 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.148 DEBUG:boto:Params: {} 2022-09-02T05:59:44.149 DEBUG:boto:Token: None 2022-09-02T05:59:44.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001b8e01c1a1f5f82-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.154 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.156 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.156 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.157 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.157 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.158 DEBUG:boto:Method: GET 2022-09-02T05:59:44.159 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.159 DEBUG:boto:Data: 2022-09-02T05:59:44.160 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.160 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.161 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.161 DEBUG:boto:Params: {} 2022-09-02T05:59:44.162 DEBUG:boto:Token: None 2022-09-02T05:59:44.163 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.163 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c963ef205a43de4f-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.168 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.169 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.169 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.170 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.170 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:44.171 DEBUG:boto:Data: 2022-09-02T05:59:44.171 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.172 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.173 DEBUG:boto:Params: {} 2022-09-02T05:59:44.173 DEBUG:boto:Token: None 2022-09-02T05:59:44.174 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115 2022-09-02T05:59:44.175 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ai2FFLcWDskI8LEQAAzGmVUxCHw= 2022-09-02T05:59:44.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ai2FFLcWDskI8LEQAAzGmVUxCHw=', 'Content-Length': '0'} 2022-09-02T05:59:44.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628a34fdb512b718-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.178 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.179 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.179 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.180 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:44.180 DEBUG:boto:Data: 2022-09-02T05:59:44.181 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.182 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.183 DEBUG:boto:Params: {} 2022-09-02T05:59:44.183 DEBUG:boto:Token: None 2022-09-02T05:59:44.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115 2022-09-02T05:59:44.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ai2FFLcWDskI8LEQAAzGmVUxCHw= 2022-09-02T05:59:44.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ai2FFLcWDskI8LEQAAzGmVUxCHw=', 'Content-Length': '0'} 2022-09-02T05:59:44.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0555d3c879af18-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2022-09-02T05:59:44.189 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.189 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.190 DEBUG:boto:Method: GET 2022-09-02T05:59:44.191 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:44.191 DEBUG:boto:Data: 2022-09-02T05:59:44.192 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.193 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.194 DEBUG:boto:Params: {} 2022-09-02T05:59:44.195 DEBUG:boto:Token: None 2022-09-02T05:59:44.195 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115 2022-09-02T05:59:44.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YuCi+BiURawUkQW/Y+nO4uUDn6U= 2022-09-02T05:59:44.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YuCi+BiURawUkQW/Y+nO4uUDn6U=', 'Content-Length': '0'} 2022-09-02T05:59:44.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e580807118ff282-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.199 DEBUG:boto:path=/xljbkf-74/obj115 2022-09-02T05:59:44.200 DEBUG:boto:auth_path=/xljbkf-74/obj115 2022-09-02T05:59:44.201 DEBUG:boto:Method: GET 2022-09-02T05:59:44.201 DEBUG:boto:Path: /xljbkf-74/obj115 2022-09-02T05:59:44.202 DEBUG:boto:Data: 2022-09-02T05:59:44.202 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.203 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.204 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.204 DEBUG:boto:Params: {} 2022-09-02T05:59:44.205 DEBUG:boto:Token: None 2022-09-02T05:59:44.206 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj115 2022-09-02T05:59:44.206 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:YuCi+BiURawUkQW/Y+nO4uUDn6U= 2022-09-02T05:59:44.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:YuCi+BiURawUkQW/Y+nO4uUDn6U=', 'Content-Length': '0'} 2022-09-02T05:59:44.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f83e1e9f8ba5d34-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.250 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.251 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.252 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.253 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.254 DEBUG:boto:Method: GET 2022-09-02T05:59:44.256 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.257 DEBUG:boto:Data: 2022-09-02T05:59:44.258 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.259 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.260 DEBUG:boto:Params: {} 2022-09-02T05:59:44.261 DEBUG:boto:Token: None 2022-09-02T05:59:44.262 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.262 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b2605886dd32283-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.267 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.267 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.268 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.269 DEBUG:boto:path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.269 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.270 DEBUG:boto:Method: GET 2022-09-02T05:59:44.271 DEBUG:boto:Path: /xljbkf-74/?prefix=obj115 2022-09-02T05:59:44.271 DEBUG:boto:Data: 2022-09-02T05:59:44.272 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.273 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.273 DEBUG:boto:Params: {} 2022-09-02T05:59:44.274 DEBUG:boto:Token: None 2022-09-02T05:59:44.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e02ffc895c1d97a-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.280 DEBUG:boto:b'xljbkf-74obj1151000falseobj1152022-09-02T05:50:15.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:44.281 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.282 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.283 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.283 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.284 DEBUG:boto:Method: GET 2022-09-02T05:59:44.284 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.285 DEBUG:boto:Data: 2022-09-02T05:59:44.285 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.287 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.287 DEBUG:boto:Params: {} 2022-09-02T05:59:44.288 DEBUG:boto:Token: None 2022-09-02T05:59:44.288 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.289 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ= 2022-09-02T05:59:44.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005081600df98e4738-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.338 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.339 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.339 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.340 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.341 DEBUG:boto:Method: GET 2022-09-02T05:59:44.341 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.342 DEBUG:boto:Data: 2022-09-02T05:59:44.342 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.343 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.343 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.344 DEBUG:boto:Params: {} 2022-09-02T05:59:44.345 DEBUG:boto:Token: None 2022-09-02T05:59:44.345 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.346 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ= 2022-09-02T05:59:44.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c95e49f576e387-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.349 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.350 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.351 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.351 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.352 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.352 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.353 DEBUG:boto:Data: 2022-09-02T05:59:44.353 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.354 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.355 DEBUG:boto:Params: {} 2022-09-02T05:59:44.355 DEBUG:boto:Token: None 2022-09-02T05:59:44.356 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.357 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aiE8xrD6FrESde4iDIMLf9seWRA= 2022-09-02T05:59:44.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aiE8xrD6FrESde4iDIMLf9seWRA=', 'Content-Length': '0'} 2022-09-02T05:59:44.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d24b20a13dbd16-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.360 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.361 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.361 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.362 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.362 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.363 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.363 DEBUG:boto:Data: 2022-09-02T05:59:44.364 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.364 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.365 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.365 DEBUG:boto:Params: {} 2022-09-02T05:59:44.366 DEBUG:boto:Token: None 2022-09-02T05:59:44.367 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.367 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aiE8xrD6FrESde4iDIMLf9seWRA= 2022-09-02T05:59:44.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aiE8xrD6FrESde4iDIMLf9seWRA=', 'Content-Length': '0'} 2022-09-02T05:59:44.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d56fbf87e3b116-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:44.372 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.372 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.373 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.373 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.374 DEBUG:boto:Method: GET 2022-09-02T05:59:44.375 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.375 DEBUG:boto:Data: 2022-09-02T05:59:44.376 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.378 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.379 DEBUG:boto:Params: {} 2022-09-02T05:59:44.381 DEBUG:boto:Token: None 2022-09-02T05:59:44.382 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.383 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ= 2022-09-02T05:59:44.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faeae3c53a77bb1a-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.388 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.389 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.390 DEBUG:boto:path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.391 DEBUG:boto:auth_path=/xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.392 DEBUG:boto:Method: GET 2022-09-02T05:59:44.393 DEBUG:boto:Path: /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.394 DEBUG:boto:Data: 2022-09-02T05:59:44.395 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.396 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.397 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.398 DEBUG:boto:Params: {} 2022-09-02T05:59:44.399 DEBUG:boto:Token: None 2022-09-02T05:59:44.400 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116?versionId=null 2022-09-02T05:59:44.402 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ= 2022-09-02T05:59:44.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qXA2QVRQKFGgwvvNL9T8oz+rNFQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006891d610dab6964-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.406 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.407 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.408 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.409 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.410 DEBUG:boto:Method: GET 2022-09-02T05:59:44.411 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.412 DEBUG:boto:Data: 2022-09-02T05:59:44.413 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.414 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.415 DEBUG:boto:Params: {} 2022-09-02T05:59:44.416 DEBUG:boto:Token: None 2022-09-02T05:59:44.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.419 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b232da1788584ba-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.424 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.426 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.427 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.428 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.429 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.430 DEBUG:boto:Method: GET 2022-09-02T05:59:44.431 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.432 DEBUG:boto:Data: 2022-09-02T05:59:44.433 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.434 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.434 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.435 DEBUG:boto:Params: {} 2022-09-02T05:59:44.436 DEBUG:boto:Token: None 2022-09-02T05:59:44.438 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.439 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2d2138b30d8c11-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.444 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.446 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.447 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.448 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.449 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:44.450 DEBUG:boto:Data: 2022-09-02T05:59:44.450 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.452 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.453 DEBUG:boto:Params: {} 2022-09-02T05:59:44.453 DEBUG:boto:Token: None 2022-09-02T05:59:44.454 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116 2022-09-02T05:59:44.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iwSxJiIVbuxBWYUULLosN/AoyjY= 2022-09-02T05:59:44.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iwSxJiIVbuxBWYUULLosN/AoyjY=', 'Content-Length': '0'} 2022-09-02T05:59:44.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a980f9ea7f2df857-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.458 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.458 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.459 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.459 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:44.460 DEBUG:boto:Data: 2022-09-02T05:59:44.461 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.462 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.462 DEBUG:boto:Params: {} 2022-09-02T05:59:44.463 DEBUG:boto:Token: None 2022-09-02T05:59:44.463 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116 2022-09-02T05:59:44.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iwSxJiIVbuxBWYUULLosN/AoyjY= 2022-09-02T05:59:44.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iwSxJiIVbuxBWYUULLosN/AoyjY=', 'Content-Length': '0'} 2022-09-02T05:59:44.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bf60c227d5794fd-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2022-09-02T05:59:44.468 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.468 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.469 DEBUG:boto:Method: GET 2022-09-02T05:59:44.469 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:44.470 DEBUG:boto:Data: 2022-09-02T05:59:44.471 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.472 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.472 DEBUG:boto:Params: {} 2022-09-02T05:59:44.473 DEBUG:boto:Token: None 2022-09-02T05:59:44.473 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116 2022-09-02T05:59:44.474 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H/qg6+3NuLpMS6tKHFjaLJ2Nsjk= 2022-09-02T05:59:44.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H/qg6+3NuLpMS6tKHFjaLJ2Nsjk=', 'Content-Length': '0'} 2022-09-02T05:59:44.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00633916e31d2d6-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.522 DEBUG:boto:path=/xljbkf-74/obj116 2022-09-02T05:59:44.523 DEBUG:boto:auth_path=/xljbkf-74/obj116 2022-09-02T05:59:44.524 DEBUG:boto:Method: GET 2022-09-02T05:59:44.525 DEBUG:boto:Path: /xljbkf-74/obj116 2022-09-02T05:59:44.526 DEBUG:boto:Data: 2022-09-02T05:59:44.527 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.528 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.529 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.530 DEBUG:boto:Params: {} 2022-09-02T05:59:44.531 DEBUG:boto:Token: None 2022-09-02T05:59:44.533 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj116 2022-09-02T05:59:44.534 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:H/qg6+3NuLpMS6tKHFjaLJ2Nsjk= 2022-09-02T05:59:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:H/qg6+3NuLpMS6tKHFjaLJ2Nsjk=', 'Content-Length': '0'} 2022-09-02T05:59:44.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47e633f3955a9b8-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.539 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.540 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.541 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.542 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.543 DEBUG:boto:Method: GET 2022-09-02T05:59:44.544 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.545 DEBUG:boto:Data: 2022-09-02T05:59:44.546 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.547 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.548 DEBUG:boto:Params: {} 2022-09-02T05:59:44.548 DEBUG:boto:Token: None 2022-09-02T05:59:44.549 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087676bab90003881-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.553 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.554 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.555 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.555 DEBUG:boto:path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.556 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.556 DEBUG:boto:Method: GET 2022-09-02T05:59:44.557 DEBUG:boto:Path: /xljbkf-74/?prefix=obj116 2022-09-02T05:59:44.557 DEBUG:boto:Data: 2022-09-02T05:59:44.558 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.559 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.559 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.560 DEBUG:boto:Params: {} 2022-09-02T05:59:44.561 DEBUG:boto:Token: None 2022-09-02T05:59:44.562 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.562 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b642c068d7e67e6-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.566 DEBUG:boto:b'xljbkf-74obj1161000falseobj1162022-09-02T05:53:22.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:44.568 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.568 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.569 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.569 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.570 DEBUG:boto:Method: GET 2022-09-02T05:59:44.571 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.571 DEBUG:boto:Data: 2022-09-02T05:59:44.572 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.573 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.573 DEBUG:boto:Params: {} 2022-09-02T05:59:44.574 DEBUG:boto:Token: None 2022-09-02T05:59:44.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.575 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ= 2022-09-02T05:59:44.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dee4a2656d4ec0b-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.579 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.580 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.581 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.582 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.583 DEBUG:boto:Method: GET 2022-09-02T05:59:44.584 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.585 DEBUG:boto:Data: 2022-09-02T05:59:44.586 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.587 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.588 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.589 DEBUG:boto:Params: {} 2022-09-02T05:59:44.589 DEBUG:boto:Token: None 2022-09-02T05:59:44.590 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.591 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ= 2022-09-02T05:59:44.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e57f8bec9ba122c-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.595 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.595 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.596 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.597 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.597 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.598 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.598 DEBUG:boto:Data: 2022-09-02T05:59:44.599 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.600 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.600 DEBUG:boto:Params: {} 2022-09-02T05:59:44.601 DEBUG:boto:Token: None 2022-09-02T05:59:44.601 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.602 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5VxZFdvOr1zFZHixSUmdStRRa/Q= 2022-09-02T05:59:44.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5VxZFdvOr1zFZHixSUmdStRRa/Q=', 'Content-Length': '0'} 2022-09-02T05:59:44.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4d8b7b9acde953-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.605 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.606 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.606 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.607 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.607 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.608 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.608 DEBUG:boto:Data: 2022-09-02T05:59:44.609 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.610 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.611 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.612 DEBUG:boto:Params: {} 2022-09-02T05:59:44.613 DEBUG:boto:Token: None 2022-09-02T05:59:44.614 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.615 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:5VxZFdvOr1zFZHixSUmdStRRa/Q= 2022-09-02T05:59:44.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:5VxZFdvOr1zFZHixSUmdStRRa/Q=', 'Content-Length': '0'} 2022-09-02T05:59:44.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c0ab3c21515e2bb-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:44.620 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.620 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.621 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.622 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.622 DEBUG:boto:Method: GET 2022-09-02T05:59:44.623 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.623 DEBUG:boto:Data: 2022-09-02T05:59:44.624 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.625 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.626 DEBUG:boto:Params: {} 2022-09-02T05:59:44.626 DEBUG:boto:Token: None 2022-09-02T05:59:44.627 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.628 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ= 2022-09-02T05:59:44.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b4509cf3616976-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.631 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.632 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.632 DEBUG:boto:path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.633 DEBUG:boto:auth_path=/xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.633 DEBUG:boto:Method: GET 2022-09-02T05:59:44.634 DEBUG:boto:Path: /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.634 DEBUG:boto:Data: 2022-09-02T05:59:44.635 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.635 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.636 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.636 DEBUG:boto:Params: {} 2022-09-02T05:59:44.637 DEBUG:boto:Token: None 2022-09-02T05:59:44.638 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124?versionId=null 2022-09-02T05:59:44.638 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ= 2022-09-02T05:59:44.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:AEtLz9b6VFk0Y+TEPQJhj22H4mQ=', 'Content-Length': '0'} 2022-09-02T05:59:44.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e084d6cce176bf-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.643 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.644 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.645 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.646 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.647 DEBUG:boto:Method: GET 2022-09-02T05:59:44.648 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.648 DEBUG:boto:Data: 2022-09-02T05:59:44.649 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.651 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.652 DEBUG:boto:Params: {} 2022-09-02T05:59:44.653 DEBUG:boto:Token: None 2022-09-02T05:59:44.654 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.655 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ffccadedae48cfc-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.702 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.704 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.705 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.706 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.707 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.707 DEBUG:boto:Method: GET 2022-09-02T05:59:44.708 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.709 DEBUG:boto:Data: 2022-09-02T05:59:44.710 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.711 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.712 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.713 DEBUG:boto:Params: {} 2022-09-02T05:59:44.714 DEBUG:boto:Token: None 2022-09-02T05:59:44.715 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.716 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1314f1793bd39f2-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.722 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.723 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.724 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.724 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.725 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:44.725 DEBUG:boto:Data: 2022-09-02T05:59:44.726 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.727 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.728 DEBUG:boto:Params: {} 2022-09-02T05:59:44.728 DEBUG:boto:Token: None 2022-09-02T05:59:44.729 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124 2022-09-02T05:59:44.730 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PfNIn4dVc5UFbacIeC3nDYXrxTc= 2022-09-02T05:59:44.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PfNIn4dVc5UFbacIeC3nDYXrxTc=', 'Content-Length': '0'} 2022-09-02T05:59:44.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d067163d7e314c88-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.733 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.734 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.734 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.735 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:44.735 DEBUG:boto:Data: 2022-09-02T05:59:44.736 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.736 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.737 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.738 DEBUG:boto:Params: {} 2022-09-02T05:59:44.738 DEBUG:boto:Token: None 2022-09-02T05:59:44.739 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124 2022-09-02T05:59:44.739 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:PfNIn4dVc5UFbacIeC3nDYXrxTc= 2022-09-02T05:59:44.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:PfNIn4dVc5UFbacIeC3nDYXrxTc=', 'Content-Length': '0'} 2022-09-02T05:59:44.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008860babbeb8685c6-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2022-09-02T05:59:44.744 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.744 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.745 DEBUG:boto:Method: GET 2022-09-02T05:59:44.746 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:44.746 DEBUG:boto:Data: 2022-09-02T05:59:44.747 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.748 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.749 DEBUG:boto:Params: {} 2022-09-02T05:59:44.750 DEBUG:boto:Token: None 2022-09-02T05:59:44.751 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124 2022-09-02T05:59:44.752 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:borINqB471/oDl4DEjR+kALwh3Q= 2022-09-02T05:59:44.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:borINqB471/oDl4DEjR+kALwh3Q=', 'Content-Length': '0'} 2022-09-02T05:59:44.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad22932f3d52575d-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.756 DEBUG:boto:path=/xljbkf-74/obj124 2022-09-02T05:59:44.756 DEBUG:boto:auth_path=/xljbkf-74/obj124 2022-09-02T05:59:44.757 DEBUG:boto:Method: GET 2022-09-02T05:59:44.758 DEBUG:boto:Path: /xljbkf-74/obj124 2022-09-02T05:59:44.758 DEBUG:boto:Data: 2022-09-02T05:59:44.759 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.760 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.760 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.761 DEBUG:boto:Params: {} 2022-09-02T05:59:44.762 DEBUG:boto:Token: None 2022-09-02T05:59:44.768 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj124 2022-09-02T05:59:44.769 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:borINqB471/oDl4DEjR+kALwh3Q= 2022-09-02T05:59:44.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:borINqB471/oDl4DEjR+kALwh3Q=', 'Content-Length': '0'} 2022-09-02T05:59:44.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8f3ce9f7938c7a1-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.818 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.819 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.819 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.820 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.821 DEBUG:boto:Method: GET 2022-09-02T05:59:44.821 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.822 DEBUG:boto:Data: 2022-09-02T05:59:44.822 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.823 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.824 DEBUG:boto:Params: {} 2022-09-02T05:59:44.825 DEBUG:boto:Token: None 2022-09-02T05:59:44.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db33e8b06d1705da-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.830 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.831 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.832 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.832 DEBUG:boto:path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.833 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.833 DEBUG:boto:Method: GET 2022-09-02T05:59:44.834 DEBUG:boto:Path: /xljbkf-74/?prefix=obj124 2022-09-02T05:59:44.834 DEBUG:boto:Data: 2022-09-02T05:59:44.835 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.836 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.836 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.837 DEBUG:boto:Params: {} 2022-09-02T05:59:44.838 DEBUG:boto:Token: None 2022-09-02T05:59:44.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.840 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5cc9d4ea1a75b72-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.845 DEBUG:boto:b'xljbkf-74obj1241000falseobj1242022-09-02T05:56:04.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:44.846 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.847 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.848 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.848 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.849 DEBUG:boto:Method: GET 2022-09-02T05:59:44.849 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.850 DEBUG:boto:Data: 2022-09-02T05:59:44.850 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.851 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.852 DEBUG:boto:Params: {} 2022-09-02T05:59:44.853 DEBUG:boto:Token: None 2022-09-02T05:59:44.853 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.854 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM= 2022-09-02T05:59:44.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM=', 'Content-Length': '0'} 2022-09-02T05:59:44.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003056dcbaec1033ef-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.858 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.859 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.859 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.860 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.860 DEBUG:boto:Method: GET 2022-09-02T05:59:44.861 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.861 DEBUG:boto:Data: 2022-09-02T05:59:44.862 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.862 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.863 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.864 DEBUG:boto:Params: {} 2022-09-02T05:59:44.864 DEBUG:boto:Token: None 2022-09-02T05:59:44.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM= 2022-09-02T05:59:44.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM=', 'Content-Length': '0'} 2022-09-02T05:59:44.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313262a9010db59c-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.872 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.873 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.874 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.875 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.876 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.876 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.877 DEBUG:boto:Data: 2022-09-02T05:59:44.878 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.879 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.879 DEBUG:boto:Params: {} 2022-09-02T05:59:44.880 DEBUG:boto:Token: None 2022-09-02T05:59:44.881 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fLlVR2dJ/5PMmjCE4mnvlTf++ro= 2022-09-02T05:59:44.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fLlVR2dJ/5PMmjCE4mnvlTf++ro=', 'Content-Length': '0'} 2022-09-02T05:59:44.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8713fd9246c4049-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.885 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.886 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.886 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.887 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.887 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.888 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.889 DEBUG:boto:Data: 2022-09-02T05:59:44.889 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.890 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.891 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.891 DEBUG:boto:Params: {} 2022-09-02T05:59:44.892 DEBUG:boto:Token: None 2022-09-02T05:59:44.893 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fLlVR2dJ/5PMmjCE4mnvlTf++ro= 2022-09-02T05:59:44.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fLlVR2dJ/5PMmjCE4mnvlTf++ro=', 'Content-Length': '0'} 2022-09-02T05:59:44.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001667a8910539458b-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:44.898 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.899 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.900 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.901 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.902 DEBUG:boto:Method: GET 2022-09-02T05:59:44.903 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.904 DEBUG:boto:Data: 2022-09-02T05:59:44.905 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.907 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.908 DEBUG:boto:Params: {} 2022-09-02T05:59:44.909 DEBUG:boto:Token: None 2022-09-02T05:59:44.910 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.911 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM= 2022-09-02T05:59:44.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM=', 'Content-Length': '0'} 2022-09-02T05:59:44.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc312ee291398b70-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.917 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.917 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.918 DEBUG:boto:path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.918 DEBUG:boto:auth_path=/xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.919 DEBUG:boto:Method: GET 2022-09-02T05:59:44.919 DEBUG:boto:Path: /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.920 DEBUG:boto:Data: 2022-09-02T05:59:44.920 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.921 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.921 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.922 DEBUG:boto:Params: {} 2022-09-02T05:59:44.923 DEBUG:boto:Token: None 2022-09-02T05:59:44.923 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/obj145?versionId=null 2022-09-02T05:59:44.924 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM= 2022-09-02T05:59:44.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0Z7TRbPs53pQ57iDQrTmktNINWM=', 'Content-Length': '0'} 2022-09-02T05:59:44.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c9810b5948462c7-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.927 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.928 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.928 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:44.929 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:44.929 DEBUG:boto:Method: GET 2022-09-02T05:59:44.930 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:44.931 DEBUG:boto:Data: 2022-09-02T05:59:44.931 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.932 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.933 DEBUG:boto:Params: {} 2022-09-02T05:59:44.933 DEBUG:boto:Token: None 2022-09-02T05:59:44.934 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.935 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810bc4c2d318b7a1-0063119bd0-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.940 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.941 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:44.942 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:44.942 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:44.943 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:44.944 DEBUG:boto:Method: GET 2022-09-02T05:59:44.945 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:44.946 DEBUG:boto:Data: 2022-09-02T05:59:44.946 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.947 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:44.947 DEBUG:boto:Port: 8000 2022-09-02T05:59:44.948 DEBUG:boto:Params: {} 2022-09-02T05:59:44.949 DEBUG:boto:Token: None 2022-09-02T05:59:44.950 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:44 GMT /xljbkf-74/ 2022-09-02T05:59:44.950 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y= 2022-09-02T05:59:44.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:44 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NqToknQJbwhoKUb1dW/iapcD2+Y=', 'Content-Length': '0'} 2022-09-02T05:59:44.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047778f9dd70e6bed-0063119bd0-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:44 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:44.994 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:44.995 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:44.996 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:44.996 DEBUG:boto:Method: HEAD 2022-09-02T05:59:44.997 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:44.997 DEBUG:boto:Data: 2022-09-02T05:59:44.998 DEBUG:boto:Headers: {} 2022-09-02T05:59:44.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:44.999 DEBUG:boto:Port: 8001 2022-09-02T05:59:44.999 DEBUG:boto:Params: {} 2022-09-02T05:59:45.000 DEBUG:boto:Token: None 2022-09-02T05:59:45.001 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj145 2022-09-02T05:59:45.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G7I5BkXKKCm4xwS37Wp+Y42G060= 2022-09-02T05:59:45.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G7I5BkXKKCm4xwS37Wp+Y42G060=', 'Content-Length': '0'} 2022-09-02T05:59:45.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004816eb2310f431a8-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.005 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:45.006 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:45.006 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.007 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:45.007 DEBUG:boto:Data: 2022-09-02T05:59:45.008 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.008 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.009 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.010 DEBUG:boto:Params: {} 2022-09-02T05:59:45.011 DEBUG:boto:Token: None 2022-09-02T05:59:45.011 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj145 2022-09-02T05:59:45.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:G7I5BkXKKCm4xwS37Wp+Y42G060= 2022-09-02T05:59:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:G7I5BkXKKCm4xwS37Wp+Y42G060=', 'Content-Length': '0'} 2022-09-02T05:59:45.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a03075391a00befa-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2022-09-02T05:59:45.016 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:45.017 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:45.018 DEBUG:boto:Method: GET 2022-09-02T05:59:45.018 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:45.019 DEBUG:boto:Data: 2022-09-02T05:59:45.019 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.020 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.021 DEBUG:boto:Params: {} 2022-09-02T05:59:45.022 DEBUG:boto:Token: None 2022-09-02T05:59:45.022 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj145 2022-09-02T05:59:45.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nEiJzdTbbTFIhvbTEnJ/Fq6VuuY= 2022-09-02T05:59:45.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nEiJzdTbbTFIhvbTEnJ/Fq6VuuY=', 'Content-Length': '0'} 2022-09-02T05:59:45.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318677a620193e14-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.027 DEBUG:boto:path=/xljbkf-74/obj145 2022-09-02T05:59:45.027 DEBUG:boto:auth_path=/xljbkf-74/obj145 2022-09-02T05:59:45.028 DEBUG:boto:Method: GET 2022-09-02T05:59:45.028 DEBUG:boto:Path: /xljbkf-74/obj145 2022-09-02T05:59:45.029 DEBUG:boto:Data: 2022-09-02T05:59:45.029 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.030 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.030 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.031 DEBUG:boto:Params: {} 2022-09-02T05:59:45.031 DEBUG:boto:Token: None 2022-09-02T05:59:45.032 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj145 2022-09-02T05:59:45.032 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nEiJzdTbbTFIhvbTEnJ/Fq6VuuY= 2022-09-02T05:59:45.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nEiJzdTbbTFIhvbTEnJ/Fq6VuuY=', 'Content-Length': '0'} 2022-09-02T05:59:45.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc19a0483c70e0b6-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.036 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.037 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.037 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:45.038 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:45.038 DEBUG:boto:Method: GET 2022-09-02T05:59:45.039 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:45.039 DEBUG:boto:Data: 2022-09-02T05:59:45.040 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.041 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.041 DEBUG:boto:Params: {} 2022-09-02T05:59:45.042 DEBUG:boto:Token: None 2022-09-02T05:59:45.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034103c10db6ec89d-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.048 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.048 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.049 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.049 DEBUG:boto:path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:45.050 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj145 2022-09-02T05:59:45.051 DEBUG:boto:Method: GET 2022-09-02T05:59:45.051 DEBUG:boto:Path: /xljbkf-74/?prefix=obj145 2022-09-02T05:59:45.052 DEBUG:boto:Data: 2022-09-02T05:59:45.052 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.053 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.053 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.054 DEBUG:boto:Params: {} 2022-09-02T05:59:45.054 DEBUG:boto:Token: None 2022-09-02T05:59:45.055 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.055 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d5c0396326f7d2e-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.060 DEBUG:boto:b'xljbkf-74obj1451000falseobj1452022-09-02T05:53:22.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:45.061 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.062 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.062 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.063 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.063 DEBUG:boto:Method: GET 2022-09-02T05:59:45.064 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.064 DEBUG:boto:Data: 2022-09-02T05:59:45.065 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.066 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.067 DEBUG:boto:Params: {} 2022-09-02T05:59:45.067 DEBUG:boto:Token: None 2022-09-02T05:59:45.068 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.069 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig= 2022-09-02T05:59:45.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig=', 'Content-Length': '0'} 2022-09-02T05:59:45.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46fb527b201505e-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.072 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.073 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.073 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.074 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.075 DEBUG:boto:Method: GET 2022-09-02T05:59:45.075 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.076 DEBUG:boto:Data: 2022-09-02T05:59:45.076 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.077 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.078 DEBUG:boto:Params: {} 2022-09-02T05:59:45.079 DEBUG:boto:Token: None 2022-09-02T05:59:45.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig= 2022-09-02T05:59:45.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig=', 'Content-Length': '0'} 2022-09-02T05:59:45.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044205ed941e5c233-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.130 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.131 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.132 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.133 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.134 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.135 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.136 DEBUG:boto:Data: 2022-09-02T05:59:45.137 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.139 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.140 DEBUG:boto:Params: {} 2022-09-02T05:59:45.141 DEBUG:boto:Token: None 2022-09-02T05:59:45.142 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.143 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b0+zlHax0rUHTn7KNXqtpBSiyvc= 2022-09-02T05:59:45.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b0+zlHax0rUHTn7KNXqtpBSiyvc=', 'Content-Length': '0'} 2022-09-02T05:59:45.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ee56c65f4cb417a-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.148 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.149 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.149 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.150 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.151 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.153 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.153 DEBUG:boto:Data: 2022-09-02T05:59:45.155 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.157 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.158 DEBUG:boto:Params: {} 2022-09-02T05:59:45.159 DEBUG:boto:Token: None 2022-09-02T05:59:45.160 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:b0+zlHax0rUHTn7KNXqtpBSiyvc= 2022-09-02T05:59:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:b0+zlHax0rUHTn7KNXqtpBSiyvc=', 'Content-Length': '0'} 2022-09-02T05:59:45.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b0665b215e627a8-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:45.167 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.168 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.169 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.170 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.171 DEBUG:boto:Method: GET 2022-09-02T05:59:45.172 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.173 DEBUG:boto:Data: 2022-09-02T05:59:45.174 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.175 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.176 DEBUG:boto:Params: {} 2022-09-02T05:59:45.176 DEBUG:boto:Token: None 2022-09-02T05:59:45.177 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.178 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig= 2022-09-02T05:59:45.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig=', 'Content-Length': '0'} 2022-09-02T05:59:45.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6d6959a88b9e01-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.226 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.227 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.228 DEBUG:boto:path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.229 DEBUG:boto:auth_path=/xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.230 DEBUG:boto:Method: GET 2022-09-02T05:59:45.231 DEBUG:boto:Path: /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.232 DEBUG:boto:Data: 2022-09-02T05:59:45.233 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.234 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.235 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.236 DEBUG:boto:Params: {} 2022-09-02T05:59:45.237 DEBUG:boto:Token: None 2022-09-02T05:59:45.238 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148?versionId=null 2022-09-02T05:59:45.239 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig= 2022-09-02T05:59:45.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Ecy00bxPxQDWmRw6RE4lsOlKig=', 'Content-Length': '0'} 2022-09-02T05:59:45.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c15a7723331f5d-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.246 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.247 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.247 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.248 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.249 DEBUG:boto:Method: GET 2022-09-02T05:59:45.250 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.250 DEBUG:boto:Data: 2022-09-02T05:59:45.252 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.256 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.257 DEBUG:boto:Params: {} 2022-09-02T05:59:45.258 DEBUG:boto:Token: None 2022-09-02T05:59:45.259 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.260 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026a40e189ea34783-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.264 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.266 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.267 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.268 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.269 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.269 DEBUG:boto:Method: GET 2022-09-02T05:59:45.270 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.270 DEBUG:boto:Data: 2022-09-02T05:59:45.271 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.272 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.272 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.273 DEBUG:boto:Params: {} 2022-09-02T05:59:45.273 DEBUG:boto:Token: None 2022-09-02T05:59:45.274 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.275 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736c3c4d4026d58a-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.279 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.280 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.280 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.281 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.281 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:45.282 DEBUG:boto:Data: 2022-09-02T05:59:45.282 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.283 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.283 DEBUG:boto:Params: {} 2022-09-02T05:59:45.284 DEBUG:boto:Token: None 2022-09-02T05:59:45.285 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148 2022-09-02T05:59:45.285 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jCApxSuwaSwAYSP8kEySdJheXTs= 2022-09-02T05:59:45.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jCApxSuwaSwAYSP8kEySdJheXTs=', 'Content-Length': '0'} 2022-09-02T05:59:45.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028aa2565d2526cd8-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.289 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.289 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.290 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.290 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:45.291 DEBUG:boto:Data: 2022-09-02T05:59:45.291 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.292 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.292 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.292 DEBUG:boto:Params: {} 2022-09-02T05:59:45.293 DEBUG:boto:Token: None 2022-09-02T05:59:45.294 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148 2022-09-02T05:59:45.294 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jCApxSuwaSwAYSP8kEySdJheXTs= 2022-09-02T05:59:45.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jCApxSuwaSwAYSP8kEySdJheXTs=', 'Content-Length': '0'} 2022-09-02T05:59:45.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007bbaae5c4b3206b-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2022-09-02T05:59:45.298 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.299 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.299 DEBUG:boto:Method: GET 2022-09-02T05:59:45.300 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:45.300 DEBUG:boto:Data: 2022-09-02T05:59:45.301 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.302 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.302 DEBUG:boto:Params: {} 2022-09-02T05:59:45.303 DEBUG:boto:Token: None 2022-09-02T05:59:45.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148 2022-09-02T05:59:45.304 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JYmy3PC7ZLGh19B9fpIo0WSUdxY= 2022-09-02T05:59:45.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JYmy3PC7ZLGh19B9fpIo0WSUdxY=', 'Content-Length': '0'} 2022-09-02T05:59:45.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065553c6ae88fa677-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.308 DEBUG:boto:path=/xljbkf-74/obj148 2022-09-02T05:59:45.309 DEBUG:boto:auth_path=/xljbkf-74/obj148 2022-09-02T05:59:45.309 DEBUG:boto:Method: GET 2022-09-02T05:59:45.310 DEBUG:boto:Path: /xljbkf-74/obj148 2022-09-02T05:59:45.310 DEBUG:boto:Data: 2022-09-02T05:59:45.311 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.311 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.312 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.312 DEBUG:boto:Params: {} 2022-09-02T05:59:45.313 DEBUG:boto:Token: None 2022-09-02T05:59:45.314 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj148 2022-09-02T05:59:45.314 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JYmy3PC7ZLGh19B9fpIo0WSUdxY= 2022-09-02T05:59:45.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JYmy3PC7ZLGh19B9fpIo0WSUdxY=', 'Content-Length': '0'} 2022-09-02T05:59:45.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c203b507fadb20b-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.318 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.318 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.319 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.319 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.320 DEBUG:boto:Method: GET 2022-09-02T05:59:45.320 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.321 DEBUG:boto:Data: 2022-09-02T05:59:45.321 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.323 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.323 DEBUG:boto:Params: {} 2022-09-02T05:59:45.324 DEBUG:boto:Token: None 2022-09-02T05:59:45.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002051b100a5311017-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.329 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.330 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.330 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.331 DEBUG:boto:path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.331 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.332 DEBUG:boto:Method: GET 2022-09-02T05:59:45.332 DEBUG:boto:Path: /xljbkf-74/?prefix=obj148 2022-09-02T05:59:45.333 DEBUG:boto:Data: 2022-09-02T05:59:45.333 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.334 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.334 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.335 DEBUG:boto:Params: {} 2022-09-02T05:59:45.335 DEBUG:boto:Token: None 2022-09-02T05:59:45.336 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.337 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4047ad4f429da1f-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.341 DEBUG:boto:b'xljbkf-74obj1481000falseobj1482022-09-02T05:58:13.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:45.343 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.343 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.344 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.344 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.345 DEBUG:boto:Method: GET 2022-09-02T05:59:45.345 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.346 DEBUG:boto:Data: 2022-09-02T05:59:45.346 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.347 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.348 DEBUG:boto:Params: {} 2022-09-02T05:59:45.349 DEBUG:boto:Token: None 2022-09-02T05:59:45.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM= 2022-09-02T05:59:45.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM=', 'Content-Length': '0'} 2022-09-02T05:59:45.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e84ba59ddf4d53-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.353 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.354 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.355 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.355 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.356 DEBUG:boto:Method: GET 2022-09-02T05:59:45.356 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.357 DEBUG:boto:Data: 2022-09-02T05:59:45.357 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.358 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.358 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.359 DEBUG:boto:Params: {} 2022-09-02T05:59:45.359 DEBUG:boto:Token: None 2022-09-02T05:59:45.360 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.361 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM= 2022-09-02T05:59:45.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM=', 'Content-Length': '0'} 2022-09-02T05:59:45.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3b2d8bbb6c3b4a-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.364 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.365 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.366 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.366 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.367 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.367 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.368 DEBUG:boto:Data: 2022-09-02T05:59:45.368 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.369 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.370 DEBUG:boto:Params: {} 2022-09-02T05:59:45.371 DEBUG:boto:Token: None 2022-09-02T05:59:45.372 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.373 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ko7Ei74nAlJa0xe18su/uK4BFsE= 2022-09-02T05:59:45.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ko7Ei74nAlJa0xe18su/uK4BFsE=', 'Content-Length': '0'} 2022-09-02T05:59:45.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401f6e401e81e6de-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.376 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.377 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.378 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.378 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.379 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.379 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.380 DEBUG:boto:Data: 2022-09-02T05:59:45.380 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.381 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.381 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.382 DEBUG:boto:Params: {} 2022-09-02T05:59:45.382 DEBUG:boto:Token: None 2022-09-02T05:59:45.383 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ko7Ei74nAlJa0xe18su/uK4BFsE= 2022-09-02T05:59:45.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ko7Ei74nAlJa0xe18su/uK4BFsE=', 'Content-Length': '0'} 2022-09-02T05:59:45.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b045032c44b0e2b-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:45.387 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.388 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.389 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.389 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.390 DEBUG:boto:Method: GET 2022-09-02T05:59:45.390 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.391 DEBUG:boto:Data: 2022-09-02T05:59:45.391 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.392 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.393 DEBUG:boto:Params: {} 2022-09-02T05:59:45.394 DEBUG:boto:Token: None 2022-09-02T05:59:45.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.395 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM= 2022-09-02T05:59:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM=', 'Content-Length': '0'} 2022-09-02T05:59:45.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446b48672d10ef8c-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.400 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.400 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.401 DEBUG:boto:path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.401 DEBUG:boto:auth_path=/xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.402 DEBUG:boto:Method: GET 2022-09-02T05:59:45.402 DEBUG:boto:Path: /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.403 DEBUG:boto:Data: 2022-09-02T05:59:45.403 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.404 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.405 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.405 DEBUG:boto:Params: {} 2022-09-02T05:59:45.406 DEBUG:boto:Token: None 2022-09-02T05:59:45.407 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155?versionId=null 2022-09-02T05:59:45.407 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM= 2022-09-02T05:59:45.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ayvICqmrXKB0EEMVU4wsh7cNcCM=', 'Content-Length': '0'} 2022-09-02T05:59:45.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008163f1440b5bd9eb-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.454 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.456 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.456 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.457 DEBUG:boto:Method: GET 2022-09-02T05:59:45.457 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.458 DEBUG:boto:Data: 2022-09-02T05:59:45.459 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.460 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.460 DEBUG:boto:Params: {} 2022-09-02T05:59:45.461 DEBUG:boto:Token: None 2022-09-02T05:59:45.461 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.462 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05490955e715fd6-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.467 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.467 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.468 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.469 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.469 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.470 DEBUG:boto:Method: GET 2022-09-02T05:59:45.470 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.471 DEBUG:boto:Data: 2022-09-02T05:59:45.471 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.472 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.473 DEBUG:boto:Params: {} 2022-09-02T05:59:45.474 DEBUG:boto:Token: None 2022-09-02T05:59:45.475 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d15890dad7297c-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.479 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.480 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.481 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.482 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.482 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:45.483 DEBUG:boto:Data: 2022-09-02T05:59:45.483 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.485 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.485 DEBUG:boto:Params: {} 2022-09-02T05:59:45.486 DEBUG:boto:Token: None 2022-09-02T05:59:45.486 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155 2022-09-02T05:59:45.487 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVnR4PvVTozRKb5/IByTOnMEGPc= 2022-09-02T05:59:45.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVnR4PvVTozRKb5/IByTOnMEGPc=', 'Content-Length': '0'} 2022-09-02T05:59:45.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008364ab87fee27fa1-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.490 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.491 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.492 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.492 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:45.493 DEBUG:boto:Data: 2022-09-02T05:59:45.493 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.494 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.494 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.495 DEBUG:boto:Params: {} 2022-09-02T05:59:45.496 DEBUG:boto:Token: None 2022-09-02T05:59:45.496 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155 2022-09-02T05:59:45.497 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mVnR4PvVTozRKb5/IByTOnMEGPc= 2022-09-02T05:59:45.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mVnR4PvVTozRKb5/IByTOnMEGPc=', 'Content-Length': '0'} 2022-09-02T05:59:45.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005085c57d3abb6652-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2022-09-02T05:59:45.502 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.502 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.503 DEBUG:boto:Method: GET 2022-09-02T05:59:45.503 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:45.504 DEBUG:boto:Data: 2022-09-02T05:59:45.504 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.505 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.506 DEBUG:boto:Params: {} 2022-09-02T05:59:45.507 DEBUG:boto:Token: None 2022-09-02T05:59:45.507 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155 2022-09-02T05:59:45.508 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NPSK6txRe7ra7ALvrui6JsshlZ0= 2022-09-02T05:59:45.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NPSK6txRe7ra7ALvrui6JsshlZ0=', 'Content-Length': '0'} 2022-09-02T05:59:45.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6815c383398fe59-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.554 DEBUG:boto:path=/xljbkf-74/obj155 2022-09-02T05:59:45.555 DEBUG:boto:auth_path=/xljbkf-74/obj155 2022-09-02T05:59:45.556 DEBUG:boto:Method: GET 2022-09-02T05:59:45.556 DEBUG:boto:Path: /xljbkf-74/obj155 2022-09-02T05:59:45.557 DEBUG:boto:Data: 2022-09-02T05:59:45.557 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.558 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.558 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.559 DEBUG:boto:Params: {} 2022-09-02T05:59:45.560 DEBUG:boto:Token: None 2022-09-02T05:59:45.560 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj155 2022-09-02T05:59:45.561 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NPSK6txRe7ra7ALvrui6JsshlZ0= 2022-09-02T05:59:45.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NPSK6txRe7ra7ALvrui6JsshlZ0=', 'Content-Length': '0'} 2022-09-02T05:59:45.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd66b29bed18d845-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.565 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.565 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.566 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.567 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.568 DEBUG:boto:Method: GET 2022-09-02T05:59:45.569 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.569 DEBUG:boto:Data: 2022-09-02T05:59:45.570 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.572 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.573 DEBUG:boto:Params: {} 2022-09-02T05:59:45.574 DEBUG:boto:Token: None 2022-09-02T05:59:45.575 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.576 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b122431dbc4cc57-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.580 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.581 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.582 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.582 DEBUG:boto:path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.583 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.583 DEBUG:boto:Method: GET 2022-09-02T05:59:45.584 DEBUG:boto:Path: /xljbkf-74/?prefix=obj155 2022-09-02T05:59:45.584 DEBUG:boto:Data: 2022-09-02T05:59:45.585 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.585 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.586 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.586 DEBUG:boto:Params: {} 2022-09-02T05:59:45.587 DEBUG:boto:Token: None 2022-09-02T05:59:45.588 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.588 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e456ff5888aa11-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.593 DEBUG:boto:b'xljbkf-74obj1551000falseobj1552022-09-02T05:56:04.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:45.594 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.595 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.595 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.596 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.596 DEBUG:boto:Method: GET 2022-09-02T05:59:45.597 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.597 DEBUG:boto:Data: 2022-09-02T05:59:45.598 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.599 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.599 DEBUG:boto:Params: {} 2022-09-02T05:59:45.600 DEBUG:boto:Token: None 2022-09-02T05:59:45.601 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.601 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU= 2022-09-02T05:59:45.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU=', 'Content-Length': '0'} 2022-09-02T05:59:45.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c911b6462f872a-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.605 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.606 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.606 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.607 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.607 DEBUG:boto:Method: GET 2022-09-02T05:59:45.608 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.608 DEBUG:boto:Data: 2022-09-02T05:59:45.609 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.609 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.610 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.610 DEBUG:boto:Params: {} 2022-09-02T05:59:45.611 DEBUG:boto:Token: None 2022-09-02T05:59:45.612 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.612 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU= 2022-09-02T05:59:45.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU=', 'Content-Length': '0'} 2022-09-02T05:59:45.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b8b0ce0fdca301f-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.616 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.616 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.617 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.617 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.618 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.619 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.620 DEBUG:boto:Data: 2022-09-02T05:59:45.620 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.622 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.623 DEBUG:boto:Params: {} 2022-09-02T05:59:45.624 DEBUG:boto:Token: None 2022-09-02T05:59:45.625 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.625 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o9O39WdihuHY9hDbTi76xzZd4PU= 2022-09-02T05:59:45.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o9O39WdihuHY9hDbTi76xzZd4PU=', 'Content-Length': '0'} 2022-09-02T05:59:45.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f7a5906b4f2159-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.629 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.629 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.630 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.630 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.631 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.631 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.632 DEBUG:boto:Data: 2022-09-02T05:59:45.632 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.633 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.633 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.634 DEBUG:boto:Params: {} 2022-09-02T05:59:45.634 DEBUG:boto:Token: None 2022-09-02T05:59:45.635 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.635 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:o9O39WdihuHY9hDbTi76xzZd4PU= 2022-09-02T05:59:45.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:o9O39WdihuHY9hDbTi76xzZd4PU=', 'Content-Length': '0'} 2022-09-02T05:59:45.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a855c3d30c024dd9-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:45.639 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.640 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.640 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.641 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.641 DEBUG:boto:Method: GET 2022-09-02T05:59:45.642 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.642 DEBUG:boto:Data: 2022-09-02T05:59:45.643 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.644 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.645 DEBUG:boto:Params: {} 2022-09-02T05:59:45.645 DEBUG:boto:Token: None 2022-09-02T05:59:45.646 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU= 2022-09-02T05:59:45.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU=', 'Content-Length': '0'} 2022-09-02T05:59:45.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff839b7e34d9f78-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.650 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.650 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.651 DEBUG:boto:path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.651 DEBUG:boto:auth_path=/xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.652 DEBUG:boto:Method: GET 2022-09-02T05:59:45.652 DEBUG:boto:Path: /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.653 DEBUG:boto:Data: 2022-09-02T05:59:45.653 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.654 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.654 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.655 DEBUG:boto:Params: {} 2022-09-02T05:59:45.655 DEBUG:boto:Token: None 2022-09-02T05:59:45.656 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17?versionId=null 2022-09-02T05:59:45.657 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU= 2022-09-02T05:59:45.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1pVjGQGIoYFeCmzCRGy7nMZURyU=', 'Content-Length': '0'} 2022-09-02T05:59:45.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017cbd689afb4fbf3-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.706 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.707 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.708 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.708 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.709 DEBUG:boto:Method: GET 2022-09-02T05:59:45.709 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.710 DEBUG:boto:Data: 2022-09-02T05:59:45.710 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.711 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.712 DEBUG:boto:Params: {} 2022-09-02T05:59:45.713 DEBUG:boto:Token: None 2022-09-02T05:59:45.713 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.714 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d483aaf84e8fffce-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.718 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.719 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.720 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.720 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.721 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.721 DEBUG:boto:Method: GET 2022-09-02T05:59:45.722 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.722 DEBUG:boto:Data: 2022-09-02T05:59:45.723 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.723 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.724 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.724 DEBUG:boto:Params: {} 2022-09-02T05:59:45.725 DEBUG:boto:Token: None 2022-09-02T05:59:45.726 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.726 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916c1611588bb644-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.730 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.731 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.732 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.732 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.733 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:45.733 DEBUG:boto:Data: 2022-09-02T05:59:45.734 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.735 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.735 DEBUG:boto:Params: {} 2022-09-02T05:59:45.736 DEBUG:boto:Token: None 2022-09-02T05:59:45.736 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17 2022-09-02T05:59:45.737 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZHhNWDhyE+jeHS3vSTUj5MhWww8= 2022-09-02T05:59:45.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZHhNWDhyE+jeHS3vSTUj5MhWww8=', 'Content-Length': '0'} 2022-09-02T05:59:45.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000885ab1a65321a363-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.740 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.741 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.741 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.742 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:45.742 DEBUG:boto:Data: 2022-09-02T05:59:45.743 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.743 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.744 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.744 DEBUG:boto:Params: {} 2022-09-02T05:59:45.745 DEBUG:boto:Token: None 2022-09-02T05:59:45.746 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17 2022-09-02T05:59:45.746 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ZHhNWDhyE+jeHS3vSTUj5MhWww8= 2022-09-02T05:59:45.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ZHhNWDhyE+jeHS3vSTUj5MhWww8=', 'Content-Length': '0'} 2022-09-02T05:59:45.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be77866eb85a79a5-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2022-09-02T05:59:45.750 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.751 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.751 DEBUG:boto:Method: GET 2022-09-02T05:59:45.752 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:45.752 DEBUG:boto:Data: 2022-09-02T05:59:45.753 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.754 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.754 DEBUG:boto:Params: {} 2022-09-02T05:59:45.755 DEBUG:boto:Token: None 2022-09-02T05:59:45.756 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17 2022-09-02T05:59:45.756 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WNuCvoDXcXwPaWAMeXXr7rJeWKI= 2022-09-02T05:59:45.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WNuCvoDXcXwPaWAMeXXr7rJeWKI=', 'Content-Length': '0'} 2022-09-02T05:59:45.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b607b8029168051-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.760 DEBUG:boto:path=/xljbkf-74/obj17 2022-09-02T05:59:45.761 DEBUG:boto:auth_path=/xljbkf-74/obj17 2022-09-02T05:59:45.761 DEBUG:boto:Method: GET 2022-09-02T05:59:45.762 DEBUG:boto:Path: /xljbkf-74/obj17 2022-09-02T05:59:45.763 DEBUG:boto:Data: 2022-09-02T05:59:45.763 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.764 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.764 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.765 DEBUG:boto:Params: {} 2022-09-02T05:59:45.766 DEBUG:boto:Token: None 2022-09-02T05:59:45.767 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj17 2022-09-02T05:59:45.767 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WNuCvoDXcXwPaWAMeXXr7rJeWKI= 2022-09-02T05:59:45.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WNuCvoDXcXwPaWAMeXXr7rJeWKI=', 'Content-Length': '0'} 2022-09-02T05:59:45.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb93079e9569b09e-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.771 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.772 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.772 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.773 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.773 DEBUG:boto:Method: GET 2022-09-02T05:59:45.774 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.774 DEBUG:boto:Data: 2022-09-02T05:59:45.775 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.776 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.777 DEBUG:boto:Params: {} 2022-09-02T05:59:45.777 DEBUG:boto:Token: None 2022-09-02T05:59:45.778 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.779 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7857bd87af8f33a-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.783 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.784 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.784 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.785 DEBUG:boto:path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.785 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.786 DEBUG:boto:Method: GET 2022-09-02T05:59:45.786 DEBUG:boto:Path: /xljbkf-74/?prefix=obj17 2022-09-02T05:59:45.787 DEBUG:boto:Data: 2022-09-02T05:59:45.787 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.788 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.788 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.789 DEBUG:boto:Params: {} 2022-09-02T05:59:45.790 DEBUG:boto:Token: None 2022-09-02T05:59:45.790 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.791 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ceebab5468d9ac3-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.795 DEBUG:boto:b'xljbkf-74obj171000falseobj172022-09-02T05:45:27.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:45.796 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.797 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.797 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.798 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.799 DEBUG:boto:Method: GET 2022-09-02T05:59:45.799 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.800 DEBUG:boto:Data: 2022-09-02T05:59:45.800 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.801 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.802 DEBUG:boto:Params: {} 2022-09-02T05:59:45.802 DEBUG:boto:Token: None 2022-09-02T05:59:45.803 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.804 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI= 2022-09-02T05:59:45.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI=', 'Content-Length': '0'} 2022-09-02T05:59:45.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432c6a91a5608e0e-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.850 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.851 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.851 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.852 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.852 DEBUG:boto:Method: GET 2022-09-02T05:59:45.853 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.853 DEBUG:boto:Data: 2022-09-02T05:59:45.854 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.855 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.855 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.856 DEBUG:boto:Params: {} 2022-09-02T05:59:45.856 DEBUG:boto:Token: None 2022-09-02T05:59:45.857 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.857 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI= 2022-09-02T05:59:45.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI=', 'Content-Length': '0'} 2022-09-02T05:59:45.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f5af532fbe2cf4-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.861 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.862 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.863 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.863 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.865 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.865 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.866 DEBUG:boto:Data: 2022-09-02T05:59:45.866 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.867 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.868 DEBUG:boto:Params: {} 2022-09-02T05:59:45.869 DEBUG:boto:Token: None 2022-09-02T05:59:45.869 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.870 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LIk9Yj+i5qTQAGOS80VmedRXOhM= 2022-09-02T05:59:45.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LIk9Yj+i5qTQAGOS80VmedRXOhM=', 'Content-Length': '0'} 2022-09-02T05:59:45.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f278a80154dd1fc8-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.874 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.875 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.875 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.876 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.876 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.877 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.877 DEBUG:boto:Data: 2022-09-02T05:59:45.878 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.878 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.879 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.879 DEBUG:boto:Params: {} 2022-09-02T05:59:45.880 DEBUG:boto:Token: None 2022-09-02T05:59:45.880 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LIk9Yj+i5qTQAGOS80VmedRXOhM= 2022-09-02T05:59:45.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LIk9Yj+i5qTQAGOS80VmedRXOhM=', 'Content-Length': '0'} 2022-09-02T05:59:45.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ec9996d5001484-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:45.886 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.886 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.887 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.887 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.888 DEBUG:boto:Method: GET 2022-09-02T05:59:45.888 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.889 DEBUG:boto:Data: 2022-09-02T05:59:45.889 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.890 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.891 DEBUG:boto:Params: {} 2022-09-02T05:59:45.892 DEBUG:boto:Token: None 2022-09-02T05:59:45.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.893 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI= 2022-09-02T05:59:45.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI=', 'Content-Length': '0'} 2022-09-02T05:59:45.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bf46ff92a21012-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.897 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.898 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.898 DEBUG:boto:path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.899 DEBUG:boto:auth_path=/xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.899 DEBUG:boto:Method: GET 2022-09-02T05:59:45.900 DEBUG:boto:Path: /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.901 DEBUG:boto:Data: 2022-09-02T05:59:45.901 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.902 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.902 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.903 DEBUG:boto:Params: {} 2022-09-02T05:59:45.904 DEBUG:boto:Token: None 2022-09-02T05:59:45.904 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185?versionId=null 2022-09-02T05:59:45.905 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI= 2022-09-02T05:59:45.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cnKZO6jOexbM3TWjEnvFJr0MSpI=', 'Content-Length': '0'} 2022-09-02T05:59:45.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097bb5758b9ddfd7-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.908 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.909 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.910 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:45.910 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:45.911 DEBUG:boto:Method: GET 2022-09-02T05:59:45.911 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:45.912 DEBUG:boto:Data: 2022-09-02T05:59:45.912 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.913 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.914 DEBUG:boto:Params: {} 2022-09-02T05:59:45.914 DEBUG:boto:Token: None 2022-09-02T05:59:45.915 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.915 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca2daf91d2157a5-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.920 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.922 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:45.923 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:45.924 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:45.925 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:45.926 DEBUG:boto:Method: GET 2022-09-02T05:59:45.927 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:45.927 DEBUG:boto:Data: 2022-09-02T05:59:45.928 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.928 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.929 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.929 DEBUG:boto:Params: {} 2022-09-02T05:59:45.930 DEBUG:boto:Token: None 2022-09-02T05:59:45.931 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/ 2022-09-02T05:59:45.931 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw= 2022-09-02T05:59:45.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:2izJbQM59PB/L1XuVkZXLigiBiw=', 'Content-Length': '0'} 2022-09-02T05:59:45.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf655e26b27dfdc9-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.936 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:45.937 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.937 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.938 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.938 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:45.939 DEBUG:boto:Data: 2022-09-02T05:59:45.939 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.940 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.941 DEBUG:boto:Params: {} 2022-09-02T05:59:45.941 DEBUG:boto:Token: None 2022-09-02T05:59:45.942 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185 2022-09-02T05:59:45.943 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Txod3NMwsYw+VHO2nPDAZBQ2nt8= 2022-09-02T05:59:45.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Txod3NMwsYw+VHO2nPDAZBQ2nt8=', 'Content-Length': '0'} 2022-09-02T05:59:45.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065200f75a32f486f-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.946 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.947 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.947 DEBUG:boto:Method: HEAD 2022-09-02T05:59:45.948 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:45.948 DEBUG:boto:Data: 2022-09-02T05:59:45.949 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.949 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.950 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.950 DEBUG:boto:Params: {} 2022-09-02T05:59:45.951 DEBUG:boto:Token: None 2022-09-02T05:59:45.952 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185 2022-09-02T05:59:45.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Txod3NMwsYw+VHO2nPDAZBQ2nt8= 2022-09-02T05:59:45.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Txod3NMwsYw+VHO2nPDAZBQ2nt8=', 'Content-Length': '0'} 2022-09-02T05:59:45.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad4b8477bcb2e85-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2022-09-02T05:59:45.956 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.957 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.957 DEBUG:boto:Method: GET 2022-09-02T05:59:45.958 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:45.959 DEBUG:boto:Data: 2022-09-02T05:59:45.959 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:45.960 DEBUG:boto:Port: 8001 2022-09-02T05:59:45.961 DEBUG:boto:Params: {} 2022-09-02T05:59:45.961 DEBUG:boto:Token: None 2022-09-02T05:59:45.962 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185 2022-09-02T05:59:45.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9A36UzXw1AEyFWkH076XqGdSuHY= 2022-09-02T05:59:45.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9A36UzXw1AEyFWkH076XqGdSuHY=', 'Content-Length': '0'} 2022-09-02T05:59:45.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c122c44d9e83bd7f-0063119bd1-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:45.966 DEBUG:boto:path=/xljbkf-74/obj185 2022-09-02T05:59:45.967 DEBUG:boto:auth_path=/xljbkf-74/obj185 2022-09-02T05:59:45.967 DEBUG:boto:Method: GET 2022-09-02T05:59:45.968 DEBUG:boto:Path: /xljbkf-74/obj185 2022-09-02T05:59:45.968 DEBUG:boto:Data: 2022-09-02T05:59:45.969 DEBUG:boto:Headers: {} 2022-09-02T05:59:45.969 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:45.970 DEBUG:boto:Port: 8000 2022-09-02T05:59:45.970 DEBUG:boto:Params: {} 2022-09-02T05:59:45.971 DEBUG:boto:Token: None 2022-09-02T05:59:45.972 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:45 GMT /xljbkf-74/obj185 2022-09-02T05:59:45.972 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9A36UzXw1AEyFWkH076XqGdSuHY= 2022-09-02T05:59:45.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:45 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9A36UzXw1AEyFWkH076XqGdSuHY=', 'Content-Length': '0'} 2022-09-02T05:59:45.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f8c55ec4ea6eca-0063119bd1-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:45 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.018 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.019 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.020 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:46.021 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:46.022 DEBUG:boto:Method: GET 2022-09-02T05:59:46.023 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:46.023 DEBUG:boto:Data: 2022-09-02T05:59:46.024 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.025 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.026 DEBUG:boto:Params: {} 2022-09-02T05:59:46.027 DEBUG:boto:Token: None 2022-09-02T05:59:46.027 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.028 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edb31454d92f2586-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.033 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.034 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.035 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.036 DEBUG:boto:path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:46.037 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj185 2022-09-02T05:59:46.038 DEBUG:boto:Method: GET 2022-09-02T05:59:46.039 DEBUG:boto:Path: /xljbkf-74/?prefix=obj185 2022-09-02T05:59:46.040 DEBUG:boto:Data: 2022-09-02T05:59:46.041 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.042 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.043 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.044 DEBUG:boto:Params: {} 2022-09-02T05:59:46.045 DEBUG:boto:Token: None 2022-09-02T05:59:46.046 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.047 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d39684833098e4-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.052 DEBUG:boto:b'xljbkf-74obj1851000falseobj1852022-09-02T05:58:13.946Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:46.055 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.056 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.057 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.057 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.058 DEBUG:boto:Method: GET 2022-09-02T05:59:46.059 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.060 DEBUG:boto:Data: 2022-09-02T05:59:46.061 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.063 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.064 DEBUG:boto:Params: {} 2022-09-02T05:59:46.065 DEBUG:boto:Token: None 2022-09-02T05:59:46.066 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.067 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo= 2022-09-02T05:59:46.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo=', 'Content-Length': '0'} 2022-09-02T05:59:46.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a72eb0f73974b8-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.072 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.072 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.073 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.073 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.074 DEBUG:boto:Method: GET 2022-09-02T05:59:46.075 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.076 DEBUG:boto:Data: 2022-09-02T05:59:46.076 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.077 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.077 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.078 DEBUG:boto:Params: {} 2022-09-02T05:59:46.079 DEBUG:boto:Token: None 2022-09-02T05:59:46.079 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.080 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo= 2022-09-02T05:59:46.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo=', 'Content-Length': '0'} 2022-09-02T05:59:46.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7182da78c6f8b1-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.085 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.086 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.087 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.088 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.088 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.089 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.091 DEBUG:boto:Data: 2022-09-02T05:59:46.092 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.094 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.095 DEBUG:boto:Params: {} 2022-09-02T05:59:46.096 DEBUG:boto:Token: None 2022-09-02T05:59:46.097 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.099 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pc7NO1h2pqoFQSAptnWiTSLd4Jc= 2022-09-02T05:59:46.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pc7NO1h2pqoFQSAptnWiTSLd4Jc=', 'Content-Length': '0'} 2022-09-02T05:59:46.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67e1e1769de1562-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.105 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.106 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.107 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.108 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.109 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.110 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.111 DEBUG:boto:Data: 2022-09-02T05:59:46.112 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.114 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.115 DEBUG:boto:Params: {} 2022-09-02T05:59:46.116 DEBUG:boto:Token: None 2022-09-02T05:59:46.116 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:pc7NO1h2pqoFQSAptnWiTSLd4Jc= 2022-09-02T05:59:46.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:pc7NO1h2pqoFQSAptnWiTSLd4Jc=', 'Content-Length': '0'} 2022-09-02T05:59:46.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191bf3df8fe03882-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:46.121 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.121 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.122 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.122 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.123 DEBUG:boto:Method: GET 2022-09-02T05:59:46.123 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.124 DEBUG:boto:Data: 2022-09-02T05:59:46.124 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.125 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.126 DEBUG:boto:Params: {} 2022-09-02T05:59:46.126 DEBUG:boto:Token: None 2022-09-02T05:59:46.128 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.129 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo= 2022-09-02T05:59:46.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo=', 'Content-Length': '0'} 2022-09-02T05:59:46.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a624c66ad8389f5-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.174 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.175 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.176 DEBUG:boto:path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.177 DEBUG:boto:auth_path=/xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.178 DEBUG:boto:Method: GET 2022-09-02T05:59:46.179 DEBUG:boto:Path: /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.180 DEBUG:boto:Data: 2022-09-02T05:59:46.180 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.181 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.181 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.182 DEBUG:boto:Params: {} 2022-09-02T05:59:46.182 DEBUG:boto:Token: None 2022-09-02T05:59:46.183 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19?versionId=null 2022-09-02T05:59:46.184 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo= 2022-09-02T05:59:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:bvdJHIM/rJn5bprqowiGjOXUWEo=', 'Content-Length': '0'} 2022-09-02T05:59:46.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e940190539628c1-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.188 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.189 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.189 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.190 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.191 DEBUG:boto:Method: GET 2022-09-02T05:59:46.191 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.192 DEBUG:boto:Data: 2022-09-02T05:59:46.192 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.194 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.195 DEBUG:boto:Params: {} 2022-09-02T05:59:46.196 DEBUG:boto:Token: None 2022-09-02T05:59:46.197 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.198 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6525e3a3a0aa926-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.202 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.203 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.204 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.204 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.205 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.205 DEBUG:boto:Method: GET 2022-09-02T05:59:46.206 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.206 DEBUG:boto:Data: 2022-09-02T05:59:46.207 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.207 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.208 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.209 DEBUG:boto:Params: {} 2022-09-02T05:59:46.209 DEBUG:boto:Token: None 2022-09-02T05:59:46.210 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.211 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7496eb42ee378ef-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.258 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.259 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.260 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.261 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.261 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:46.262 DEBUG:boto:Data: 2022-09-02T05:59:46.262 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.263 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.264 DEBUG:boto:Params: {} 2022-09-02T05:59:46.264 DEBUG:boto:Token: None 2022-09-02T05:59:46.265 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19 2022-09-02T05:59:46.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3qCpjdqX5ZFSSC/cBB+xgYVBCik= 2022-09-02T05:59:46.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3qCpjdqX5ZFSSC/cBB+xgYVBCik=', 'Content-Length': '0'} 2022-09-02T05:59:46.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b79b05c2e6c53ce3-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.269 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.271 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.272 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.272 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:46.273 DEBUG:boto:Data: 2022-09-02T05:59:46.274 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.276 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.278 DEBUG:boto:Params: {} 2022-09-02T05:59:46.278 DEBUG:boto:Token: None 2022-09-02T05:59:46.279 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19 2022-09-02T05:59:46.280 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3qCpjdqX5ZFSSC/cBB+xgYVBCik= 2022-09-02T05:59:46.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3qCpjdqX5ZFSSC/cBB+xgYVBCik=', 'Content-Length': '0'} 2022-09-02T05:59:46.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d5e930c62ad9ce-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2022-09-02T05:59:46.284 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.284 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.285 DEBUG:boto:Method: GET 2022-09-02T05:59:46.285 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:46.286 DEBUG:boto:Data: 2022-09-02T05:59:46.287 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.288 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.288 DEBUG:boto:Params: {} 2022-09-02T05:59:46.289 DEBUG:boto:Token: None 2022-09-02T05:59:46.290 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19 2022-09-02T05:59:46.291 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W7d98lTIW8cK7S1WIfGkx7oQtPs= 2022-09-02T05:59:46.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W7d98lTIW8cK7S1WIfGkx7oQtPs=', 'Content-Length': '0'} 2022-09-02T05:59:46.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d227d671250b17-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.296 DEBUG:boto:path=/xljbkf-74/obj19 2022-09-02T05:59:46.297 DEBUG:boto:auth_path=/xljbkf-74/obj19 2022-09-02T05:59:46.298 DEBUG:boto:Method: GET 2022-09-02T05:59:46.299 DEBUG:boto:Path: /xljbkf-74/obj19 2022-09-02T05:59:46.300 DEBUG:boto:Data: 2022-09-02T05:59:46.301 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.303 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.303 DEBUG:boto:Params: {} 2022-09-02T05:59:46.304 DEBUG:boto:Token: None 2022-09-02T05:59:46.305 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj19 2022-09-02T05:59:46.306 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:W7d98lTIW8cK7S1WIfGkx7oQtPs= 2022-09-02T05:59:46.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:W7d98lTIW8cK7S1WIfGkx7oQtPs=', 'Content-Length': '0'} 2022-09-02T05:59:46.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0bc3b6028ceb058-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.309 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.310 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.310 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.311 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.312 DEBUG:boto:Method: GET 2022-09-02T05:59:46.312 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.313 DEBUG:boto:Data: 2022-09-02T05:59:46.313 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.314 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.315 DEBUG:boto:Params: {} 2022-09-02T05:59:46.316 DEBUG:boto:Token: None 2022-09-02T05:59:46.316 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.317 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df5d9937767bf84-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.321 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.322 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.322 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.323 DEBUG:boto:path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.324 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.324 DEBUG:boto:Method: GET 2022-09-02T05:59:46.325 DEBUG:boto:Path: /xljbkf-74/?prefix=obj19 2022-09-02T05:59:46.325 DEBUG:boto:Data: 2022-09-02T05:59:46.326 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.326 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.327 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.327 DEBUG:boto:Params: {} 2022-09-02T05:59:46.328 DEBUG:boto:Token: None 2022-09-02T05:59:46.329 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.329 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bed957f062e8830-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.333 DEBUG:boto:b'xljbkf-74obj191000falseobj192022-09-02T05:47:57.563Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:46.335 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.336 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.336 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.337 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.337 DEBUG:boto:Method: GET 2022-09-02T05:59:46.338 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.338 DEBUG:boto:Data: 2022-09-02T05:59:46.339 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.340 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.341 DEBUG:boto:Params: {} 2022-09-02T05:59:46.341 DEBUG:boto:Token: None 2022-09-02T05:59:46.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc= 2022-09-02T05:59:46.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc=', 'Content-Length': '0'} 2022-09-02T05:59:46.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab7df91908ed415-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.346 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.347 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.347 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.348 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.349 DEBUG:boto:Method: GET 2022-09-02T05:59:46.349 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.350 DEBUG:boto:Data: 2022-09-02T05:59:46.350 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.351 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.352 DEBUG:boto:Params: {} 2022-09-02T05:59:46.352 DEBUG:boto:Token: None 2022-09-02T05:59:46.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc= 2022-09-02T05:59:46.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc=', 'Content-Length': '0'} 2022-09-02T05:59:46.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c4f9210c6c9ff6-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.357 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.358 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.359 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.359 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.360 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.360 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.361 DEBUG:boto:Data: 2022-09-02T05:59:46.361 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.362 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.363 DEBUG:boto:Params: {} 2022-09-02T05:59:46.363 DEBUG:boto:Token: None 2022-09-02T05:59:46.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:axGapa2a8dGc/fe+yRjt9Esqogw= 2022-09-02T05:59:46.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:axGapa2a8dGc/fe+yRjt9Esqogw=', 'Content-Length': '0'} 2022-09-02T05:59:46.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056b921f1cde40581-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.368 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.369 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.369 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.370 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.371 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.371 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.372 DEBUG:boto:Data: 2022-09-02T05:59:46.373 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.373 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.374 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.374 DEBUG:boto:Params: {} 2022-09-02T05:59:46.375 DEBUG:boto:Token: None 2022-09-02T05:59:46.376 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:axGapa2a8dGc/fe+yRjt9Esqogw= 2022-09-02T05:59:46.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:axGapa2a8dGc/fe+yRjt9Esqogw=', 'Content-Length': '0'} 2022-09-02T05:59:46.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1d5d707b506437-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:46.383 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.383 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.384 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.385 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.385 DEBUG:boto:Method: GET 2022-09-02T05:59:46.386 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.386 DEBUG:boto:Data: 2022-09-02T05:59:46.387 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.388 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.389 DEBUG:boto:Params: {} 2022-09-02T05:59:46.389 DEBUG:boto:Token: None 2022-09-02T05:59:46.390 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.391 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc= 2022-09-02T05:59:46.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc=', 'Content-Length': '0'} 2022-09-02T05:59:46.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006967f7592ba1bf16-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.395 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.395 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.396 DEBUG:boto:path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.396 DEBUG:boto:auth_path=/xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.397 DEBUG:boto:Method: GET 2022-09-02T05:59:46.398 DEBUG:boto:Path: /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.398 DEBUG:boto:Data: 2022-09-02T05:59:46.399 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.399 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.400 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.400 DEBUG:boto:Params: {} 2022-09-02T05:59:46.401 DEBUG:boto:Token: None 2022-09-02T05:59:46.402 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22?versionId=null 2022-09-02T05:59:46.403 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc= 2022-09-02T05:59:46.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:cwH/yPY9Dqfm07M3/s77+0B4ggc=', 'Content-Length': '0'} 2022-09-02T05:59:46.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be361a7dab34f2b8-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.408 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.409 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.410 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.411 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.412 DEBUG:boto:Method: GET 2022-09-02T05:59:46.413 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.414 DEBUG:boto:Data: 2022-09-02T05:59:46.414 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.416 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.416 DEBUG:boto:Params: {} 2022-09-02T05:59:46.417 DEBUG:boto:Token: None 2022-09-02T05:59:46.418 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.418 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b4d69b9a1b082b-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.470 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.471 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.472 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.472 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.473 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.474 DEBUG:boto:Method: GET 2022-09-02T05:59:46.474 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.475 DEBUG:boto:Data: 2022-09-02T05:59:46.475 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.476 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.476 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.477 DEBUG:boto:Params: {} 2022-09-02T05:59:46.477 DEBUG:boto:Token: None 2022-09-02T05:59:46.478 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.479 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8c3a35665263b65-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.483 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.484 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.485 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.486 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.486 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:46.488 DEBUG:boto:Data: 2022-09-02T05:59:46.488 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.489 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.490 DEBUG:boto:Params: {} 2022-09-02T05:59:46.491 DEBUG:boto:Token: None 2022-09-02T05:59:46.491 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22 2022-09-02T05:59:46.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yVfCDcsIX9FHplqoAsI79AtmqZw= 2022-09-02T05:59:46.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yVfCDcsIX9FHplqoAsI79AtmqZw=', 'Content-Length': '0'} 2022-09-02T05:59:46.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7c4e728613754e5-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.495 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.496 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.497 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.497 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:46.498 DEBUG:boto:Data: 2022-09-02T05:59:46.498 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.499 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.500 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.500 DEBUG:boto:Params: {} 2022-09-02T05:59:46.501 DEBUG:boto:Token: None 2022-09-02T05:59:46.501 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22 2022-09-02T05:59:46.502 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:yVfCDcsIX9FHplqoAsI79AtmqZw= 2022-09-02T05:59:46.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:yVfCDcsIX9FHplqoAsI79AtmqZw=', 'Content-Length': '0'} 2022-09-02T05:59:46.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6dcd3a33c24d1c-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2022-09-02T05:59:46.506 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.507 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.508 DEBUG:boto:Method: GET 2022-09-02T05:59:46.508 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:46.509 DEBUG:boto:Data: 2022-09-02T05:59:46.509 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.520 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.521 DEBUG:boto:Params: {} 2022-09-02T05:59:46.521 DEBUG:boto:Token: None 2022-09-02T05:59:46.522 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22 2022-09-02T05:59:46.523 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aqM9A7xkPAU9hIj0BKg9hh3wBck= 2022-09-02T05:59:46.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aqM9A7xkPAU9hIj0BKg9hh3wBck=', 'Content-Length': '0'} 2022-09-02T05:59:46.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beba2e9567aac88a-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.526 DEBUG:boto:path=/xljbkf-74/obj22 2022-09-02T05:59:46.527 DEBUG:boto:auth_path=/xljbkf-74/obj22 2022-09-02T05:59:46.528 DEBUG:boto:Method: GET 2022-09-02T05:59:46.528 DEBUG:boto:Path: /xljbkf-74/obj22 2022-09-02T05:59:46.529 DEBUG:boto:Data: 2022-09-02T05:59:46.529 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.530 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.530 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.531 DEBUG:boto:Params: {} 2022-09-02T05:59:46.531 DEBUG:boto:Token: None 2022-09-02T05:59:46.532 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj22 2022-09-02T05:59:46.533 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:aqM9A7xkPAU9hIj0BKg9hh3wBck= 2022-09-02T05:59:46.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:aqM9A7xkPAU9hIj0BKg9hh3wBck=', 'Content-Length': '0'} 2022-09-02T05:59:46.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035fb79ae49f6329c-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.578 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.579 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.579 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.580 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.580 DEBUG:boto:Method: GET 2022-09-02T05:59:46.581 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.581 DEBUG:boto:Data: 2022-09-02T05:59:46.582 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.583 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.584 DEBUG:boto:Params: {} 2022-09-02T05:59:46.584 DEBUG:boto:Token: None 2022-09-02T05:59:46.585 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.586 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce8a3ee272643ff-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.591 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.593 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.594 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.595 DEBUG:boto:path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.595 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.596 DEBUG:boto:Method: GET 2022-09-02T05:59:46.597 DEBUG:boto:Path: /xljbkf-74/?prefix=obj22 2022-09-02T05:59:46.598 DEBUG:boto:Data: 2022-09-02T05:59:46.599 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.600 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.601 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.602 DEBUG:boto:Params: {} 2022-09-02T05:59:46.603 DEBUG:boto:Token: None 2022-09-02T05:59:46.604 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.605 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff90df91824c2e7c-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.610 DEBUG:boto:b'xljbkf-74obj221000falseobj222022-09-02T05:44:47.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:46.612 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.612 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.613 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.613 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.614 DEBUG:boto:Method: GET 2022-09-02T05:59:46.614 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.615 DEBUG:boto:Data: 2022-09-02T05:59:46.615 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.616 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.617 DEBUG:boto:Params: {} 2022-09-02T05:59:46.618 DEBUG:boto:Token: None 2022-09-02T05:59:46.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU= 2022-09-02T05:59:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU=', 'Content-Length': '0'} 2022-09-02T05:59:46.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2ec7c5e9c0d450-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.670 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.671 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.672 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.673 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.673 DEBUG:boto:Method: GET 2022-09-02T05:59:46.674 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.674 DEBUG:boto:Data: 2022-09-02T05:59:46.675 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.675 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.676 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.676 DEBUG:boto:Params: {} 2022-09-02T05:59:46.677 DEBUG:boto:Token: None 2022-09-02T05:59:46.678 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.679 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU= 2022-09-02T05:59:46.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU=', 'Content-Length': '0'} 2022-09-02T05:59:46.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1fe752889ac4bb-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.682 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.683 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.683 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.684 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.684 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.685 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.685 DEBUG:boto:Data: 2022-09-02T05:59:46.686 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.687 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.687 DEBUG:boto:Params: {} 2022-09-02T05:59:46.688 DEBUG:boto:Token: None 2022-09-02T05:59:46.689 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.689 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8H7a6xGqyhJ4E3b5mheBHttJ6bU= 2022-09-02T05:59:46.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8H7a6xGqyhJ4E3b5mheBHttJ6bU=', 'Content-Length': '0'} 2022-09-02T05:59:46.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4e0213399e09d3-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.694 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.695 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.695 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.696 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.696 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.697 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.697 DEBUG:boto:Data: 2022-09-02T05:59:46.698 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.698 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.699 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.699 DEBUG:boto:Params: {} 2022-09-02T05:59:46.700 DEBUG:boto:Token: None 2022-09-02T05:59:46.701 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.701 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8H7a6xGqyhJ4E3b5mheBHttJ6bU= 2022-09-02T05:59:46.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8H7a6xGqyhJ4E3b5mheBHttJ6bU=', 'Content-Length': '0'} 2022-09-02T05:59:46.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f779db9ae756dd-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:46.705 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.706 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.707 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.707 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.708 DEBUG:boto:Method: GET 2022-09-02T05:59:46.708 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.709 DEBUG:boto:Data: 2022-09-02T05:59:46.709 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.710 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.711 DEBUG:boto:Params: {} 2022-09-02T05:59:46.712 DEBUG:boto:Token: None 2022-09-02T05:59:46.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU= 2022-09-02T05:59:46.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU=', 'Content-Length': '0'} 2022-09-02T05:59:46.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288ad58343646dbf-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.717 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.717 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.718 DEBUG:boto:path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.718 DEBUG:boto:auth_path=/xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.719 DEBUG:boto:Method: GET 2022-09-02T05:59:46.719 DEBUG:boto:Path: /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.720 DEBUG:boto:Data: 2022-09-02T05:59:46.720 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.721 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.722 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.722 DEBUG:boto:Params: {} 2022-09-02T05:59:46.723 DEBUG:boto:Token: None 2022-09-02T05:59:46.724 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23?versionId=null 2022-09-02T05:59:46.725 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU= 2022-09-02T05:59:46.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:162NMkKgEYytEIoKv2fY5rOP5mU=', 'Content-Length': '0'} 2022-09-02T05:59:46.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007af032727340f2f1-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.728 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.729 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.729 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.730 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.730 DEBUG:boto:Method: GET 2022-09-02T05:59:46.731 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.731 DEBUG:boto:Data: 2022-09-02T05:59:46.732 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.733 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.733 DEBUG:boto:Params: {} 2022-09-02T05:59:46.734 DEBUG:boto:Token: None 2022-09-02T05:59:46.735 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.735 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3808ce80b809fc-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.739 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.740 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.740 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.741 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.742 DEBUG:boto:Method: GET 2022-09-02T05:59:46.743 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.743 DEBUG:boto:Data: 2022-09-02T05:59:46.744 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.745 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.745 DEBUG:boto:Params: {} 2022-09-02T05:59:46.746 DEBUG:boto:Token: None 2022-09-02T05:59:46.746 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.747 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000555d7c9783c8e873-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.751 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.752 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.753 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.753 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.754 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:46.754 DEBUG:boto:Data: 2022-09-02T05:59:46.755 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.756 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.756 DEBUG:boto:Params: {} 2022-09-02T05:59:46.757 DEBUG:boto:Token: None 2022-09-02T05:59:46.758 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23 2022-09-02T05:59:46.758 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OO+0XMTmesdxZEI7z2NFFSXamCM= 2022-09-02T05:59:46.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OO+0XMTmesdxZEI7z2NFFSXamCM=', 'Content-Length': '0'} 2022-09-02T05:59:46.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004364769863bb5ba8-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.762 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.762 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.763 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.763 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:46.764 DEBUG:boto:Data: 2022-09-02T05:59:46.765 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.765 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.766 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.766 DEBUG:boto:Params: {} 2022-09-02T05:59:46.767 DEBUG:boto:Token: None 2022-09-02T05:59:46.767 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23 2022-09-02T05:59:46.768 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OO+0XMTmesdxZEI7z2NFFSXamCM= 2022-09-02T05:59:46.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OO+0XMTmesdxZEI7z2NFFSXamCM=', 'Content-Length': '0'} 2022-09-02T05:59:46.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c64dad1d1562ab-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2022-09-02T05:59:46.772 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.772 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.773 DEBUG:boto:Method: GET 2022-09-02T05:59:46.773 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:46.774 DEBUG:boto:Data: 2022-09-02T05:59:46.775 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.776 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.776 DEBUG:boto:Params: {} 2022-09-02T05:59:46.777 DEBUG:boto:Token: None 2022-09-02T05:59:46.777 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23 2022-09-02T05:59:46.778 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CqpuxbmOCoidhCucNn71UHGvFjY= 2022-09-02T05:59:46.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CqpuxbmOCoidhCucNn71UHGvFjY=', 'Content-Length': '0'} 2022-09-02T05:59:46.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da55e41b10a6bbb-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.791 DEBUG:boto:path=/xljbkf-74/obj23 2022-09-02T05:59:46.791 DEBUG:boto:auth_path=/xljbkf-74/obj23 2022-09-02T05:59:46.792 DEBUG:boto:Method: GET 2022-09-02T05:59:46.792 DEBUG:boto:Path: /xljbkf-74/obj23 2022-09-02T05:59:46.793 DEBUG:boto:Data: 2022-09-02T05:59:46.793 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.794 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.795 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.795 DEBUG:boto:Params: {} 2022-09-02T05:59:46.796 DEBUG:boto:Token: None 2022-09-02T05:59:46.796 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj23 2022-09-02T05:59:46.797 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:CqpuxbmOCoidhCucNn71UHGvFjY= 2022-09-02T05:59:46.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:CqpuxbmOCoidhCucNn71UHGvFjY=', 'Content-Length': '0'} 2022-09-02T05:59:46.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000640ab78be7071fae-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.842 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.843 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.843 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.844 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.845 DEBUG:boto:Method: GET 2022-09-02T05:59:46.845 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.846 DEBUG:boto:Data: 2022-09-02T05:59:46.846 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.847 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.848 DEBUG:boto:Params: {} 2022-09-02T05:59:46.848 DEBUG:boto:Token: None 2022-09-02T05:59:46.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002053bd1c31faa28c-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.854 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.855 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.855 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.856 DEBUG:boto:path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.856 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.857 DEBUG:boto:Method: GET 2022-09-02T05:59:46.857 DEBUG:boto:Path: /xljbkf-74/?prefix=obj23 2022-09-02T05:59:46.859 DEBUG:boto:Data: 2022-09-02T05:59:46.859 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.860 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.860 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.861 DEBUG:boto:Params: {} 2022-09-02T05:59:46.861 DEBUG:boto:Token: None 2022-09-02T05:59:46.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032499fa1b5f8800f-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.868 DEBUG:boto:b'xljbkf-74obj231000falseobj232022-09-02T05:50:15.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:46.870 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.870 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.871 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.871 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.872 DEBUG:boto:Method: GET 2022-09-02T05:59:46.873 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.874 DEBUG:boto:Data: 2022-09-02T05:59:46.875 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.877 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.878 DEBUG:boto:Params: {} 2022-09-02T05:59:46.879 DEBUG:boto:Token: None 2022-09-02T05:59:46.880 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw= 2022-09-02T05:59:46.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw=', 'Content-Length': '0'} 2022-09-02T05:59:46.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002314b368bbcafcfc-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.886 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.886 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.887 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.888 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.888 DEBUG:boto:Method: GET 2022-09-02T05:59:46.889 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.889 DEBUG:boto:Data: 2022-09-02T05:59:46.890 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.891 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.892 DEBUG:boto:Params: {} 2022-09-02T05:59:46.893 DEBUG:boto:Token: None 2022-09-02T05:59:46.893 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.894 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw= 2022-09-02T05:59:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw=', 'Content-Length': '0'} 2022-09-02T05:59:46.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd80a37b9af1167-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.898 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.899 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.899 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.900 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.900 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.901 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.901 DEBUG:boto:Data: 2022-09-02T05:59:46.902 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.904 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.904 DEBUG:boto:Params: {} 2022-09-02T05:59:46.905 DEBUG:boto:Token: None 2022-09-02T05:59:46.906 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.907 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O8zF0Uf+ie1iygDxDjBygqCv/3g= 2022-09-02T05:59:46.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O8zF0Uf+ie1iygDxDjBygqCv/3g=', 'Content-Length': '0'} 2022-09-02T05:59:46.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e325678fcce5131-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.910 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.911 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.911 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.912 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.912 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.913 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.913 DEBUG:boto:Data: 2022-09-02T05:59:46.914 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.914 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.915 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.915 DEBUG:boto:Params: {} 2022-09-02T05:59:46.916 DEBUG:boto:Token: None 2022-09-02T05:59:46.917 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.917 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O8zF0Uf+ie1iygDxDjBygqCv/3g= 2022-09-02T05:59:46.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O8zF0Uf+ie1iygDxDjBygqCv/3g=', 'Content-Length': '0'} 2022-09-02T05:59:46.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae50c188b6638af2-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:46.921 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.922 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.923 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.923 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.924 DEBUG:boto:Method: GET 2022-09-02T05:59:46.925 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.925 DEBUG:boto:Data: 2022-09-02T05:59:46.926 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.927 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.928 DEBUG:boto:Params: {} 2022-09-02T05:59:46.928 DEBUG:boto:Token: None 2022-09-02T05:59:46.929 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.930 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw= 2022-09-02T05:59:46.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw=', 'Content-Length': '0'} 2022-09-02T05:59:46.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aa23b99b7666c14-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.933 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.934 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.934 DEBUG:boto:path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.935 DEBUG:boto:auth_path=/xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.935 DEBUG:boto:Method: GET 2022-09-02T05:59:46.936 DEBUG:boto:Path: /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.936 DEBUG:boto:Data: 2022-09-02T05:59:46.937 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.938 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.938 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.939 DEBUG:boto:Params: {} 2022-09-02T05:59:46.939 DEBUG:boto:Token: None 2022-09-02T05:59:46.940 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29?versionId=null 2022-09-02T05:59:46.941 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw= 2022-09-02T05:59:46.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UJ4SW9t23tMIUlYaRY+rqSjy2Aw=', 'Content-Length': '0'} 2022-09-02T05:59:46.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000597af3f24860fdfa-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.945 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.945 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.946 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:46.947 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:46.948 DEBUG:boto:Method: GET 2022-09-02T05:59:46.949 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:46.950 DEBUG:boto:Data: 2022-09-02T05:59:46.951 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.953 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.954 DEBUG:boto:Params: {} 2022-09-02T05:59:46.955 DEBUG:boto:Token: None 2022-09-02T05:59:46.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.958 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ee2c2b4b76750d-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.964 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.965 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:46.966 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:46.966 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:46.967 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:46.968 DEBUG:boto:Method: GET 2022-09-02T05:59:46.968 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:46.969 DEBUG:boto:Data: 2022-09-02T05:59:46.969 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.970 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.970 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.971 DEBUG:boto:Params: {} 2022-09-02T05:59:46.972 DEBUG:boto:Token: None 2022-09-02T05:59:46.972 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/ 2022-09-02T05:59:46.973 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys= 2022-09-02T05:59:46.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:/z3Xlyr1YoY6NF6iFChajOtxKys=', 'Content-Length': '0'} 2022-09-02T05:59:46.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc88e0ec8ae3fad-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.977 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:46.978 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.979 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.979 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.980 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:46.980 DEBUG:boto:Data: 2022-09-02T05:59:46.981 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:46.982 DEBUG:boto:Port: 8001 2022-09-02T05:59:46.983 DEBUG:boto:Params: {} 2022-09-02T05:59:46.983 DEBUG:boto:Token: None 2022-09-02T05:59:46.984 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29 2022-09-02T05:59:46.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k2CRA6CsWWk686Z1KLvLSZWfTCk= 2022-09-02T05:59:46.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k2CRA6CsWWk686Z1KLvLSZWfTCk=', 'Content-Length': '0'} 2022-09-02T05:59:46.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001410b0fe5765bb4c-0063119bd2-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.988 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.988 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.989 DEBUG:boto:Method: HEAD 2022-09-02T05:59:46.989 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:46.990 DEBUG:boto:Data: 2022-09-02T05:59:46.991 DEBUG:boto:Headers: {} 2022-09-02T05:59:46.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:46.992 DEBUG:boto:Port: 8000 2022-09-02T05:59:46.992 DEBUG:boto:Params: {} 2022-09-02T05:59:46.993 DEBUG:boto:Token: None 2022-09-02T05:59:46.993 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:46 GMT /xljbkf-74/obj29 2022-09-02T05:59:46.994 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k2CRA6CsWWk686Z1KLvLSZWfTCk= 2022-09-02T05:59:46.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:46 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k2CRA6CsWWk686Z1KLvLSZWfTCk=', 'Content-Length': '0'} 2022-09-02T05:59:46.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f062341ebd127505-0063119bd2-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:46 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:46.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2022-09-02T05:59:46.998 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:46.999 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:46.999 DEBUG:boto:Method: GET 2022-09-02T05:59:47.000 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:47.000 DEBUG:boto:Data: 2022-09-02T05:59:47.001 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.002 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.003 DEBUG:boto:Params: {} 2022-09-02T05:59:47.003 DEBUG:boto:Token: None 2022-09-02T05:59:47.004 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj29 2022-09-02T05:59:47.005 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LyuMUwLUDzsT/chvPJMmO/GEfQM= 2022-09-02T05:59:47.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LyuMUwLUDzsT/chvPJMmO/GEfQM=', 'Content-Length': '0'} 2022-09-02T05:59:47.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000788b5337a3dd7a14-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.054 DEBUG:boto:path=/xljbkf-74/obj29 2022-09-02T05:59:47.055 DEBUG:boto:auth_path=/xljbkf-74/obj29 2022-09-02T05:59:47.056 DEBUG:boto:Method: GET 2022-09-02T05:59:47.057 DEBUG:boto:Path: /xljbkf-74/obj29 2022-09-02T05:59:47.057 DEBUG:boto:Data: 2022-09-02T05:59:47.058 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.059 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.060 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.061 DEBUG:boto:Params: {} 2022-09-02T05:59:47.062 DEBUG:boto:Token: None 2022-09-02T05:59:47.064 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj29 2022-09-02T05:59:47.065 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LyuMUwLUDzsT/chvPJMmO/GEfQM= 2022-09-02T05:59:47.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LyuMUwLUDzsT/chvPJMmO/GEfQM=', 'Content-Length': '0'} 2022-09-02T05:59:47.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137675a0be6fb927-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.072 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.073 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.073 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:47.074 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:47.075 DEBUG:boto:Method: GET 2022-09-02T05:59:47.076 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:47.076 DEBUG:boto:Data: 2022-09-02T05:59:47.077 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.078 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.079 DEBUG:boto:Params: {} 2022-09-02T05:59:47.079 DEBUG:boto:Token: None 2022-09-02T05:59:47.080 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.081 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c0acd9b0290c74-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.085 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.086 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.087 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.087 DEBUG:boto:path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:47.088 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj29 2022-09-02T05:59:47.089 DEBUG:boto:Method: GET 2022-09-02T05:59:47.089 DEBUG:boto:Path: /xljbkf-74/?prefix=obj29 2022-09-02T05:59:47.090 DEBUG:boto:Data: 2022-09-02T05:59:47.091 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.091 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.092 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.092 DEBUG:boto:Params: {} 2022-09-02T05:59:47.093 DEBUG:boto:Token: None 2022-09-02T05:59:47.094 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.095 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008345dbccb1ad80ce-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.100 DEBUG:boto:b'xljbkf-74obj291000falseobj292022-09-02T05:53:22.382Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:47.101 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.102 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.103 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.103 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.104 DEBUG:boto:Method: GET 2022-09-02T05:59:47.105 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.106 DEBUG:boto:Data: 2022-09-02T05:59:47.107 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.109 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.111 DEBUG:boto:Params: {} 2022-09-02T05:59:47.112 DEBUG:boto:Token: None 2022-09-02T05:59:47.113 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.114 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE= 2022-09-02T05:59:47.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE=', 'Content-Length': '0'} 2022-09-02T05:59:47.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a5a64269f62ae05-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.119 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.119 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.120 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.121 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.122 DEBUG:boto:Method: GET 2022-09-02T05:59:47.123 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.123 DEBUG:boto:Data: 2022-09-02T05:59:47.124 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.125 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.125 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.126 DEBUG:boto:Params: {} 2022-09-02T05:59:47.127 DEBUG:boto:Token: None 2022-09-02T05:59:47.128 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.128 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE= 2022-09-02T05:59:47.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE=', 'Content-Length': '0'} 2022-09-02T05:59:47.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000708f7047efed0883-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.174 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.175 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.176 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.177 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.178 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.178 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.179 DEBUG:boto:Data: 2022-09-02T05:59:47.180 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.182 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.183 DEBUG:boto:Params: {} 2022-09-02T05:59:47.183 DEBUG:boto:Token: None 2022-09-02T05:59:47.184 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.185 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gtiAjLm4JqCvdruNCNeKJfty54= 2022-09-02T05:59:47.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gtiAjLm4JqCvdruNCNeKJfty54=', 'Content-Length': '0'} 2022-09-02T05:59:47.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000151827ecf56a28cb-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.190 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.191 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.192 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.192 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.193 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.194 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.196 DEBUG:boto:Data: 2022-09-02T05:59:47.197 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.198 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.199 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.200 DEBUG:boto:Params: {} 2022-09-02T05:59:47.201 DEBUG:boto:Token: None 2022-09-02T05:59:47.202 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.203 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:8gtiAjLm4JqCvdruNCNeKJfty54= 2022-09-02T05:59:47.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:8gtiAjLm4JqCvdruNCNeKJfty54=', 'Content-Length': '0'} 2022-09-02T05:59:47.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c71b6474927611f-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:47.209 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.210 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.211 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.213 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.214 DEBUG:boto:Method: GET 2022-09-02T05:59:47.215 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.216 DEBUG:boto:Data: 2022-09-02T05:59:47.217 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.219 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.220 DEBUG:boto:Params: {} 2022-09-02T05:59:47.222 DEBUG:boto:Token: None 2022-09-02T05:59:47.223 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.224 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE= 2022-09-02T05:59:47.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE=', 'Content-Length': '0'} 2022-09-02T05:59:47.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a5891128fd81c2c-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.230 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.231 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.232 DEBUG:boto:path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.233 DEBUG:boto:auth_path=/xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.234 DEBUG:boto:Method: GET 2022-09-02T05:59:47.235 DEBUG:boto:Path: /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.236 DEBUG:boto:Data: 2022-09-02T05:59:47.237 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.238 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.239 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.240 DEBUG:boto:Params: {} 2022-09-02T05:59:47.241 DEBUG:boto:Token: None 2022-09-02T05:59:47.243 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31?versionId=null 2022-09-02T05:59:47.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE= 2022-09-02T05:59:47.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:sp5kXt/rtJgsrcm2avscBOLtxvE=', 'Content-Length': '0'} 2022-09-02T05:59:47.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5d4426f2d5e1106-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.250 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.251 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.252 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.253 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.254 DEBUG:boto:Method: GET 2022-09-02T05:59:47.255 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.256 DEBUG:boto:Data: 2022-09-02T05:59:47.257 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.259 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.260 DEBUG:boto:Params: {} 2022-09-02T05:59:47.261 DEBUG:boto:Token: None 2022-09-02T05:59:47.263 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bacba7c2b5ba1429-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.269 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.270 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.270 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.271 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.272 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.272 DEBUG:boto:Method: GET 2022-09-02T05:59:47.273 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.274 DEBUG:boto:Data: 2022-09-02T05:59:47.274 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.275 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.276 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.276 DEBUG:boto:Params: {} 2022-09-02T05:59:47.277 DEBUG:boto:Token: None 2022-09-02T05:59:47.278 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003329d81633acb1c8-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.290 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.291 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.292 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.293 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.293 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:47.294 DEBUG:boto:Data: 2022-09-02T05:59:47.295 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.296 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.296 DEBUG:boto:Params: {} 2022-09-02T05:59:47.297 DEBUG:boto:Token: None 2022-09-02T05:59:47.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31 2022-09-02T05:59:47.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kn0N3utxz5IR6Mj+6v4/aHQtvtU= 2022-09-02T05:59:47.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kn0N3utxz5IR6Mj+6v4/aHQtvtU=', 'Content-Length': '0'} 2022-09-02T05:59:47.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009218374b1f4dbe13-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.302 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.303 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.304 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.304 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:47.305 DEBUG:boto:Data: 2022-09-02T05:59:47.305 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.307 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.307 DEBUG:boto:Params: {} 2022-09-02T05:59:47.308 DEBUG:boto:Token: None 2022-09-02T05:59:47.308 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31 2022-09-02T05:59:47.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Kn0N3utxz5IR6Mj+6v4/aHQtvtU= 2022-09-02T05:59:47.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Kn0N3utxz5IR6Mj+6v4/aHQtvtU=', 'Content-Length': '0'} 2022-09-02T05:59:47.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000885c5aa02d1469a9-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2022-09-02T05:59:47.313 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.314 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.315 DEBUG:boto:Method: GET 2022-09-02T05:59:47.315 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:47.316 DEBUG:boto:Data: 2022-09-02T05:59:47.317 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.318 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.319 DEBUG:boto:Params: {} 2022-09-02T05:59:47.320 DEBUG:boto:Token: None 2022-09-02T05:59:47.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31 2022-09-02T05:59:47.321 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nbeqp3ruFBjnGbgKeacvdUqygt8= 2022-09-02T05:59:47.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nbeqp3ruFBjnGbgKeacvdUqygt8=', 'Content-Length': '0'} 2022-09-02T05:59:47.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fabd5e91623baba-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.325 DEBUG:boto:path=/xljbkf-74/obj31 2022-09-02T05:59:47.326 DEBUG:boto:auth_path=/xljbkf-74/obj31 2022-09-02T05:59:47.326 DEBUG:boto:Method: GET 2022-09-02T05:59:47.327 DEBUG:boto:Path: /xljbkf-74/obj31 2022-09-02T05:59:47.327 DEBUG:boto:Data: 2022-09-02T05:59:47.328 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.329 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.330 DEBUG:boto:Params: {} 2022-09-02T05:59:47.330 DEBUG:boto:Token: None 2022-09-02T05:59:47.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj31 2022-09-02T05:59:47.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:nbeqp3ruFBjnGbgKeacvdUqygt8= 2022-09-02T05:59:47.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:nbeqp3ruFBjnGbgKeacvdUqygt8=', 'Content-Length': '0'} 2022-09-02T05:59:47.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac93cd71850fba4-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.335 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.336 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.336 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.337 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.337 DEBUG:boto:Method: GET 2022-09-02T05:59:47.338 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.338 DEBUG:boto:Data: 2022-09-02T05:59:47.339 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.340 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.340 DEBUG:boto:Params: {} 2022-09-02T05:59:47.341 DEBUG:boto:Token: None 2022-09-02T05:59:47.342 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edfd8d678aaa9cab-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.346 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.347 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.348 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.348 DEBUG:boto:path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.349 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.349 DEBUG:boto:Method: GET 2022-09-02T05:59:47.350 DEBUG:boto:Path: /xljbkf-74/?prefix=obj31 2022-09-02T05:59:47.351 DEBUG:boto:Data: 2022-09-02T05:59:47.351 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.352 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.352 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.353 DEBUG:boto:Params: {} 2022-09-02T05:59:47.353 DEBUG:boto:Token: None 2022-09-02T05:59:47.354 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.355 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5a36ac260e1cf59-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.402 DEBUG:boto:b'xljbkf-74obj311000falseobj312022-09-02T05:56:04.232Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:47.404 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.404 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.405 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.405 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.406 DEBUG:boto:Method: GET 2022-09-02T05:59:47.406 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.407 DEBUG:boto:Data: 2022-09-02T05:59:47.407 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.408 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.409 DEBUG:boto:Params: {} 2022-09-02T05:59:47.410 DEBUG:boto:Token: None 2022-09-02T05:59:47.410 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.411 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw= 2022-09-02T05:59:47.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw=', 'Content-Length': '0'} 2022-09-02T05:59:47.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb95f52095cbde35-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.414 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.415 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.416 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.416 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.417 DEBUG:boto:Method: GET 2022-09-02T05:59:47.417 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.418 DEBUG:boto:Data: 2022-09-02T05:59:47.418 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.419 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.419 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.420 DEBUG:boto:Params: {} 2022-09-02T05:59:47.420 DEBUG:boto:Token: None 2022-09-02T05:59:47.421 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.421 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw= 2022-09-02T05:59:47.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw=', 'Content-Length': '0'} 2022-09-02T05:59:47.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000331665663a5d7ad4-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.425 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.425 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.426 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.426 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.427 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.427 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.428 DEBUG:boto:Data: 2022-09-02T05:59:47.428 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.429 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.430 DEBUG:boto:Params: {} 2022-09-02T05:59:47.430 DEBUG:boto:Token: None 2022-09-02T05:59:47.431 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.432 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fp01i77e4+1FkZDtIoBpe6UUWuc= 2022-09-02T05:59:47.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fp01i77e4+1FkZDtIoBpe6UUWuc=', 'Content-Length': '0'} 2022-09-02T05:59:47.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ff67bb4be9fe86-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.435 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.436 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.436 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.437 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.437 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.438 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.439 DEBUG:boto:Data: 2022-09-02T05:59:47.439 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.440 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.440 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.441 DEBUG:boto:Params: {} 2022-09-02T05:59:47.441 DEBUG:boto:Token: None 2022-09-02T05:59:47.442 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.443 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fp01i77e4+1FkZDtIoBpe6UUWuc= 2022-09-02T05:59:47.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fp01i77e4+1FkZDtIoBpe6UUWuc=', 'Content-Length': '0'} 2022-09-02T05:59:47.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967bdd0761eb2091-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:47.449 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.450 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.452 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.453 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.454 DEBUG:boto:Method: GET 2022-09-02T05:59:47.455 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.456 DEBUG:boto:Data: 2022-09-02T05:59:47.457 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.459 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.460 DEBUG:boto:Params: {} 2022-09-02T05:59:47.461 DEBUG:boto:Token: None 2022-09-02T05:59:47.462 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.463 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw= 2022-09-02T05:59:47.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw=', 'Content-Length': '0'} 2022-09-02T05:59:47.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000336066e92e8ac62-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.467 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.468 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.469 DEBUG:boto:path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.469 DEBUG:boto:auth_path=/xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.470 DEBUG:boto:Method: GET 2022-09-02T05:59:47.470 DEBUG:boto:Path: /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.471 DEBUG:boto:Data: 2022-09-02T05:59:47.472 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.472 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.473 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.474 DEBUG:boto:Params: {} 2022-09-02T05:59:47.475 DEBUG:boto:Token: None 2022-09-02T05:59:47.476 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33?versionId=null 2022-09-02T05:59:47.476 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw= 2022-09-02T05:59:47.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3Gak9KihiEWIrWAqVcAfDmcGlcw=', 'Content-Length': '0'} 2022-09-02T05:59:47.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b71479ac69824c-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.481 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.481 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.482 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.482 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.483 DEBUG:boto:Method: GET 2022-09-02T05:59:47.484 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.484 DEBUG:boto:Data: 2022-09-02T05:59:47.485 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.487 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.488 DEBUG:boto:Params: {} 2022-09-02T05:59:47.489 DEBUG:boto:Token: None 2022-09-02T05:59:47.490 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.491 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a44347c3692e3d0b-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.542 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.544 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.545 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.545 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.546 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.546 DEBUG:boto:Method: GET 2022-09-02T05:59:47.547 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.548 DEBUG:boto:Data: 2022-09-02T05:59:47.548 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.549 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.550 DEBUG:boto:Params: {} 2022-09-02T05:59:47.551 DEBUG:boto:Token: None 2022-09-02T05:59:47.551 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc577f0db78c6be-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.556 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.557 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.557 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.558 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.559 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:47.559 DEBUG:boto:Data: 2022-09-02T05:59:47.560 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.561 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.561 DEBUG:boto:Params: {} 2022-09-02T05:59:47.562 DEBUG:boto:Token: None 2022-09-02T05:59:47.562 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33 2022-09-02T05:59:47.563 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ghe91MKNZBXo9DZrLVrCsy6Wmmo= 2022-09-02T05:59:47.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ghe91MKNZBXo9DZrLVrCsy6Wmmo=', 'Content-Length': '0'} 2022-09-02T05:59:47.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000529d5ae6e701d71c-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.566 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.567 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.567 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.568 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:47.568 DEBUG:boto:Data: 2022-09-02T05:59:47.569 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.569 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.570 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.571 DEBUG:boto:Params: {} 2022-09-02T05:59:47.571 DEBUG:boto:Token: None 2022-09-02T05:59:47.572 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33 2022-09-02T05:59:47.573 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Ghe91MKNZBXo9DZrLVrCsy6Wmmo= 2022-09-02T05:59:47.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Ghe91MKNZBXo9DZrLVrCsy6Wmmo=', 'Content-Length': '0'} 2022-09-02T05:59:47.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e441cb2825fdedd4-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2022-09-02T05:59:47.577 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.578 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.578 DEBUG:boto:Method: GET 2022-09-02T05:59:47.579 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:47.580 DEBUG:boto:Data: 2022-09-02T05:59:47.580 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.581 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.582 DEBUG:boto:Params: {} 2022-09-02T05:59:47.583 DEBUG:boto:Token: None 2022-09-02T05:59:47.583 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33 2022-09-02T05:59:47.584 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j8T+rRgNiSgROX8mhtoMktwxeHM= 2022-09-02T05:59:47.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j8T+rRgNiSgROX8mhtoMktwxeHM=', 'Content-Length': '0'} 2022-09-02T05:59:47.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793735d6ca8459ba-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.587 DEBUG:boto:path=/xljbkf-74/obj33 2022-09-02T05:59:47.588 DEBUG:boto:auth_path=/xljbkf-74/obj33 2022-09-02T05:59:47.589 DEBUG:boto:Method: GET 2022-09-02T05:59:47.589 DEBUG:boto:Path: /xljbkf-74/obj33 2022-09-02T05:59:47.590 DEBUG:boto:Data: 2022-09-02T05:59:47.590 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.591 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.591 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.592 DEBUG:boto:Params: {} 2022-09-02T05:59:47.593 DEBUG:boto:Token: None 2022-09-02T05:59:47.593 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj33 2022-09-02T05:59:47.594 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:j8T+rRgNiSgROX8mhtoMktwxeHM= 2022-09-02T05:59:47.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:j8T+rRgNiSgROX8mhtoMktwxeHM=', 'Content-Length': '0'} 2022-09-02T05:59:47.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68bbdb3c67aba77-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.599 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.599 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.600 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.600 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.601 DEBUG:boto:Method: GET 2022-09-02T05:59:47.601 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.602 DEBUG:boto:Data: 2022-09-02T05:59:47.602 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.603 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.604 DEBUG:boto:Params: {} 2022-09-02T05:59:47.605 DEBUG:boto:Token: None 2022-09-02T05:59:47.605 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.606 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e419538065105f1a-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.610 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.611 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.612 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.612 DEBUG:boto:path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.613 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.613 DEBUG:boto:Method: GET 2022-09-02T05:59:47.614 DEBUG:boto:Path: /xljbkf-74/?prefix=obj33 2022-09-02T05:59:47.614 DEBUG:boto:Data: 2022-09-02T05:59:47.615 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.616 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.616 DEBUG:boto:Params: {} 2022-09-02T05:59:47.617 DEBUG:boto:Token: None 2022-09-02T05:59:47.617 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cbc7ba001c06470-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.622 DEBUG:boto:b'xljbkf-74obj331000falseobj332022-09-02T05:44:47.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:47.624 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.625 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.625 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.626 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.626 DEBUG:boto:Method: GET 2022-09-02T05:59:47.627 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.627 DEBUG:boto:Data: 2022-09-02T05:59:47.628 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.629 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.629 DEBUG:boto:Params: {} 2022-09-02T05:59:47.630 DEBUG:boto:Token: None 2022-09-02T05:59:47.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.631 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY= 2022-09-02T05:59:47.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY=', 'Content-Length': '0'} 2022-09-02T05:59:47.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9663c1a28525ea5-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.635 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.635 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.636 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.636 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.637 DEBUG:boto:Method: GET 2022-09-02T05:59:47.637 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.638 DEBUG:boto:Data: 2022-09-02T05:59:47.639 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.640 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.640 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.641 DEBUG:boto:Params: {} 2022-09-02T05:59:47.642 DEBUG:boto:Token: None 2022-09-02T05:59:47.643 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.643 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY= 2022-09-02T05:59:47.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY=', 'Content-Length': '0'} 2022-09-02T05:59:47.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ebdcd2262e4a08-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.647 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.648 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.648 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.648 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.649 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.650 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.650 DEBUG:boto:Data: 2022-09-02T05:59:47.651 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.652 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.652 DEBUG:boto:Params: {} 2022-09-02T05:59:47.653 DEBUG:boto:Token: None 2022-09-02T05:59:47.654 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.654 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FyLvJJfQdWprfLXN64aQcT2OqYI= 2022-09-02T05:59:47.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FyLvJJfQdWprfLXN64aQcT2OqYI=', 'Content-Length': '0'} 2022-09-02T05:59:47.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005328d45c6e7a1422-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.661 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.662 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.663 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.664 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.666 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.666 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.667 DEBUG:boto:Data: 2022-09-02T05:59:47.667 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.668 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.668 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.669 DEBUG:boto:Params: {} 2022-09-02T05:59:47.669 DEBUG:boto:Token: None 2022-09-02T05:59:47.670 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.671 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FyLvJJfQdWprfLXN64aQcT2OqYI= 2022-09-02T05:59:47.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FyLvJJfQdWprfLXN64aQcT2OqYI=', 'Content-Length': '0'} 2022-09-02T05:59:47.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001152ce176165ecb6-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:47.675 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.675 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.676 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.677 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.677 DEBUG:boto:Method: GET 2022-09-02T05:59:47.678 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.678 DEBUG:boto:Data: 2022-09-02T05:59:47.679 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.680 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.680 DEBUG:boto:Params: {} 2022-09-02T05:59:47.681 DEBUG:boto:Token: None 2022-09-02T05:59:47.681 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.682 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY= 2022-09-02T05:59:47.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY=', 'Content-Length': '0'} 2022-09-02T05:59:47.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d095a1fedd32f27-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.686 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.686 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.687 DEBUG:boto:path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.687 DEBUG:boto:auth_path=/xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.688 DEBUG:boto:Method: GET 2022-09-02T05:59:47.688 DEBUG:boto:Path: /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.689 DEBUG:boto:Data: 2022-09-02T05:59:47.689 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.690 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.691 DEBUG:boto:Params: {} 2022-09-02T05:59:47.692 DEBUG:boto:Token: None 2022-09-02T05:59:47.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34?versionId=null 2022-09-02T05:59:47.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY= 2022-09-02T05:59:47.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ItjEkSSohgVjjAOYUbRsTQRL3qY=', 'Content-Length': '0'} 2022-09-02T05:59:47.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007178594ee27efa78-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.742 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.743 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.743 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.744 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.744 DEBUG:boto:Method: GET 2022-09-02T05:59:47.745 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.746 DEBUG:boto:Data: 2022-09-02T05:59:47.746 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.747 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.748 DEBUG:boto:Params: {} 2022-09-02T05:59:47.748 DEBUG:boto:Token: None 2022-09-02T05:59:47.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.750 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c584d896133eb873-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.754 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.755 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.755 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.756 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.757 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.757 DEBUG:boto:Method: GET 2022-09-02T05:59:47.758 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.758 DEBUG:boto:Data: 2022-09-02T05:59:47.759 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.759 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.760 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.760 DEBUG:boto:Params: {} 2022-09-02T05:59:47.761 DEBUG:boto:Token: None 2022-09-02T05:59:47.761 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.762 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbf7d56f67df32d9-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.766 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.767 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.768 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.768 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.769 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:47.769 DEBUG:boto:Data: 2022-09-02T05:59:47.770 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.771 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.771 DEBUG:boto:Params: {} 2022-09-02T05:59:47.772 DEBUG:boto:Token: None 2022-09-02T05:59:47.772 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34 2022-09-02T05:59:47.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iOpz2fAB+U33W/dt1DFvbnBj3L4= 2022-09-02T05:59:47.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iOpz2fAB+U33W/dt1DFvbnBj3L4=', 'Content-Length': '0'} 2022-09-02T05:59:47.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a612fad937dc4543-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.776 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.777 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.777 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.778 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:47.778 DEBUG:boto:Data: 2022-09-02T05:59:47.779 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.779 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.780 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.780 DEBUG:boto:Params: {} 2022-09-02T05:59:47.781 DEBUG:boto:Token: None 2022-09-02T05:59:47.782 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34 2022-09-02T05:59:47.782 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:iOpz2fAB+U33W/dt1DFvbnBj3L4= 2022-09-02T05:59:47.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:iOpz2fAB+U33W/dt1DFvbnBj3L4=', 'Content-Length': '0'} 2022-09-02T05:59:47.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f37e4dfe082880-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2022-09-02T05:59:47.786 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.787 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.787 DEBUG:boto:Method: GET 2022-09-02T05:59:47.788 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:47.788 DEBUG:boto:Data: 2022-09-02T05:59:47.789 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.790 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.791 DEBUG:boto:Params: {} 2022-09-02T05:59:47.791 DEBUG:boto:Token: None 2022-09-02T05:59:47.792 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34 2022-09-02T05:59:47.793 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vNHQZnywJ6sJBnM8ZfF2QuWMFh0= 2022-09-02T05:59:47.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vNHQZnywJ6sJBnM8ZfF2QuWMFh0=', 'Content-Length': '0'} 2022-09-02T05:59:47.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007daf279bf5e81041-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.796 DEBUG:boto:path=/xljbkf-74/obj34 2022-09-02T05:59:47.797 DEBUG:boto:auth_path=/xljbkf-74/obj34 2022-09-02T05:59:47.797 DEBUG:boto:Method: GET 2022-09-02T05:59:47.798 DEBUG:boto:Path: /xljbkf-74/obj34 2022-09-02T05:59:47.799 DEBUG:boto:Data: 2022-09-02T05:59:47.799 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.800 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.800 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.801 DEBUG:boto:Params: {} 2022-09-02T05:59:47.801 DEBUG:boto:Token: None 2022-09-02T05:59:47.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj34 2022-09-02T05:59:47.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vNHQZnywJ6sJBnM8ZfF2QuWMFh0= 2022-09-02T05:59:47.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vNHQZnywJ6sJBnM8ZfF2QuWMFh0=', 'Content-Length': '0'} 2022-09-02T05:59:47.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3977caf6e9a9ea2-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.806 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.807 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.808 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.808 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.809 DEBUG:boto:Method: GET 2022-09-02T05:59:47.809 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.810 DEBUG:boto:Data: 2022-09-02T05:59:47.810 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.811 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.812 DEBUG:boto:Params: {} 2022-09-02T05:59:47.812 DEBUG:boto:Token: None 2022-09-02T05:59:47.813 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.814 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e01225340e4c1501-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.818 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.819 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.819 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.820 DEBUG:boto:path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.820 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.821 DEBUG:boto:Method: GET 2022-09-02T05:59:47.821 DEBUG:boto:Path: /xljbkf-74/?prefix=obj34 2022-09-02T05:59:47.822 DEBUG:boto:Data: 2022-09-02T05:59:47.822 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.823 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.823 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.824 DEBUG:boto:Params: {} 2022-09-02T05:59:47.824 DEBUG:boto:Token: None 2022-09-02T05:59:47.825 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.826 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f47d5e84d3b3709-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.829 DEBUG:boto:b'xljbkf-74obj341000falseobj342022-09-02T05:45:27.806Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:47.831 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.831 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.832 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.832 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.833 DEBUG:boto:Method: GET 2022-09-02T05:59:47.834 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.834 DEBUG:boto:Data: 2022-09-02T05:59:47.835 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.837 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.837 DEBUG:boto:Params: {} 2022-09-02T05:59:47.838 DEBUG:boto:Token: None 2022-09-02T05:59:47.839 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.839 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM= 2022-09-02T05:59:47.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM=', 'Content-Length': '0'} 2022-09-02T05:59:47.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072c642cf2180face-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.843 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.844 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.844 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.845 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.845 DEBUG:boto:Method: GET 2022-09-02T05:59:47.846 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.847 DEBUG:boto:Data: 2022-09-02T05:59:47.847 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.848 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.848 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.849 DEBUG:boto:Params: {} 2022-09-02T05:59:47.850 DEBUG:boto:Token: None 2022-09-02T05:59:47.850 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.851 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM= 2022-09-02T05:59:47.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM=', 'Content-Length': '0'} 2022-09-02T05:59:47.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac7256bedb738e49-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.854 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.855 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.855 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.856 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.856 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.857 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.857 DEBUG:boto:Data: 2022-09-02T05:59:47.858 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.859 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.859 DEBUG:boto:Params: {} 2022-09-02T05:59:47.860 DEBUG:boto:Token: None 2022-09-02T05:59:47.860 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.861 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fVgJ68CsZpZyZ1ySgu9+YrEHkqs= 2022-09-02T05:59:47.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fVgJ68CsZpZyZ1ySgu9+YrEHkqs=', 'Content-Length': '0'} 2022-09-02T05:59:47.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e6919d5b2a6ba3-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.864 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.865 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.865 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.866 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.866 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.867 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.867 DEBUG:boto:Data: 2022-09-02T05:59:47.868 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.868 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.869 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.869 DEBUG:boto:Params: {} 2022-09-02T05:59:47.870 DEBUG:boto:Token: None 2022-09-02T05:59:47.870 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.871 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fVgJ68CsZpZyZ1ySgu9+YrEHkqs= 2022-09-02T05:59:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fVgJ68CsZpZyZ1ySgu9+YrEHkqs=', 'Content-Length': '0'} 2022-09-02T05:59:47.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c320c92dae6fc84a-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:47.875 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.875 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.876 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.876 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.877 DEBUG:boto:Method: GET 2022-09-02T05:59:47.878 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.878 DEBUG:boto:Data: 2022-09-02T05:59:47.879 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.880 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.880 DEBUG:boto:Params: {} 2022-09-02T05:59:47.881 DEBUG:boto:Token: None 2022-09-02T05:59:47.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.882 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM= 2022-09-02T05:59:47.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM=', 'Content-Length': '0'} 2022-09-02T05:59:47.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4d21987c808313-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.926 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.927 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.927 DEBUG:boto:path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.928 DEBUG:boto:auth_path=/xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.929 DEBUG:boto:Method: GET 2022-09-02T05:59:47.929 DEBUG:boto:Path: /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.930 DEBUG:boto:Data: 2022-09-02T05:59:47.930 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.931 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.931 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.932 DEBUG:boto:Params: {} 2022-09-02T05:59:47.932 DEBUG:boto:Token: None 2022-09-02T05:59:47.933 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37?versionId=null 2022-09-02T05:59:47.934 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM= 2022-09-02T05:59:47.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:X+zcnhpbQ3j2ootpcK2sp2xsFBM=', 'Content-Length': '0'} 2022-09-02T05:59:47.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e339e0264d02db-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.938 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.938 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.939 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:47.939 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:47.940 DEBUG:boto:Method: GET 2022-09-02T05:59:47.940 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:47.941 DEBUG:boto:Data: 2022-09-02T05:59:47.941 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.942 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.943 DEBUG:boto:Params: {} 2022-09-02T05:59:47.943 DEBUG:boto:Token: None 2022-09-02T05:59:47.944 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097a709064fec6a4-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.948 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.949 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:47.950 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:47.951 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:47.951 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:47.952 DEBUG:boto:Method: GET 2022-09-02T05:59:47.952 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:47.953 DEBUG:boto:Data: 2022-09-02T05:59:47.953 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.954 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.954 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.955 DEBUG:boto:Params: {} 2022-09-02T05:59:47.955 DEBUG:boto:Token: None 2022-09-02T05:59:47.956 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/ 2022-09-02T05:59:47.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk= 2022-09-02T05:59:47.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fPmUQujU7YJKvBHDWXzEW2O/9Uk=', 'Content-Length': '0'} 2022-09-02T05:59:47.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb5a5a31e44edf0-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.960 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:47.961 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.962 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.963 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.963 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:47.964 DEBUG:boto:Data: 2022-09-02T05:59:47.964 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.966 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.966 DEBUG:boto:Params: {} 2022-09-02T05:59:47.967 DEBUG:boto:Token: None 2022-09-02T05:59:47.969 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37 2022-09-02T05:59:47.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:krF4NREQt/DHUMroO6W2B5OlzI4= 2022-09-02T05:59:47.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:krF4NREQt/DHUMroO6W2B5OlzI4=', 'Content-Length': '0'} 2022-09-02T05:59:47.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fec4807a3294cb5-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.975 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.976 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.977 DEBUG:boto:Method: HEAD 2022-09-02T05:59:47.977 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:47.978 DEBUG:boto:Data: 2022-09-02T05:59:47.978 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.979 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:47.979 DEBUG:boto:Port: 8000 2022-09-02T05:59:47.980 DEBUG:boto:Params: {} 2022-09-02T05:59:47.981 DEBUG:boto:Token: None 2022-09-02T05:59:47.981 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37 2022-09-02T05:59:47.982 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:krF4NREQt/DHUMroO6W2B5OlzI4= 2022-09-02T05:59:47.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:krF4NREQt/DHUMroO6W2B5OlzI4=', 'Content-Length': '0'} 2022-09-02T05:59:47.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6721687fe85d9d3-0063119bd3-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2022-09-02T05:59:47.986 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.987 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.987 DEBUG:boto:Method: GET 2022-09-02T05:59:47.988 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:47.988 DEBUG:boto:Data: 2022-09-02T05:59:47.989 DEBUG:boto:Headers: {} 2022-09-02T05:59:47.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:47.990 DEBUG:boto:Port: 8001 2022-09-02T05:59:47.991 DEBUG:boto:Params: {} 2022-09-02T05:59:47.992 DEBUG:boto:Token: None 2022-09-02T05:59:47.992 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:47 GMT /xljbkf-74/obj37 2022-09-02T05:59:47.993 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:No6Qfl0fUcQDTNo1+PZtoMcf998= 2022-09-02T05:59:47.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:47 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:No6Qfl0fUcQDTNo1+PZtoMcf998=', 'Content-Length': '0'} 2022-09-02T05:59:47.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d872dcb9d82a6a46-0063119bd3-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:47 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:47.997 DEBUG:boto:path=/xljbkf-74/obj37 2022-09-02T05:59:47.997 DEBUG:boto:auth_path=/xljbkf-74/obj37 2022-09-02T05:59:47.998 DEBUG:boto:Method: GET 2022-09-02T05:59:47.999 DEBUG:boto:Path: /xljbkf-74/obj37 2022-09-02T05:59:48.000 DEBUG:boto:Data: 2022-09-02T05:59:48.000 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.001 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.001 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.002 DEBUG:boto:Params: {} 2022-09-02T05:59:48.003 DEBUG:boto:Token: None 2022-09-02T05:59:48.003 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj37 2022-09-02T05:59:48.004 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BwYyD1Yf5vIEZa4QFDIXQCfVwvI= 2022-09-02T05:59:48.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BwYyD1Yf5vIEZa4QFDIXQCfVwvI=', 'Content-Length': '0'} 2022-09-02T05:59:48.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf647eb50010c34-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.009 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.009 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.010 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:48.010 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:48.011 DEBUG:boto:Method: GET 2022-09-02T05:59:48.011 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:48.012 DEBUG:boto:Data: 2022-09-02T05:59:48.013 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.014 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.014 DEBUG:boto:Params: {} 2022-09-02T05:59:48.015 DEBUG:boto:Token: None 2022-09-02T05:59:48.015 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.016 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a140dacccee75f-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.020 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.021 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.022 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.022 DEBUG:boto:path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:48.023 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj37 2022-09-02T05:59:48.023 DEBUG:boto:Method: GET 2022-09-02T05:59:48.024 DEBUG:boto:Path: /xljbkf-74/?prefix=obj37 2022-09-02T05:59:48.025 DEBUG:boto:Data: 2022-09-02T05:59:48.025 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.026 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.027 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.027 DEBUG:boto:Params: {} 2022-09-02T05:59:48.028 DEBUG:boto:Token: None 2022-09-02T05:59:48.029 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.029 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c27398bbdc9570a4-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.034 DEBUG:boto:b'xljbkf-74obj371000falseobj372022-09-02T05:58:13.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:48.036 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.036 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.037 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.037 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.038 DEBUG:boto:Method: GET 2022-09-02T05:59:48.038 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.039 DEBUG:boto:Data: 2022-09-02T05:59:48.039 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.040 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.041 DEBUG:boto:Params: {} 2022-09-02T05:59:48.042 DEBUG:boto:Token: None 2022-09-02T05:59:48.042 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.043 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno= 2022-09-02T05:59:48.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno=', 'Content-Length': '0'} 2022-09-02T05:59:48.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006baac188806681a3-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.046 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.047 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.047 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.048 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.048 DEBUG:boto:Method: GET 2022-09-02T05:59:48.049 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.049 DEBUG:boto:Data: 2022-09-02T05:59:48.050 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.051 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.051 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.052 DEBUG:boto:Params: {} 2022-09-02T05:59:48.052 DEBUG:boto:Token: None 2022-09-02T05:59:48.053 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.053 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno= 2022-09-02T05:59:48.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno=', 'Content-Length': '0'} 2022-09-02T05:59:48.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d21383eab6447ce-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.057 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.058 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.058 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.059 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.059 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.060 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.060 DEBUG:boto:Data: 2022-09-02T05:59:48.061 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.062 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.062 DEBUG:boto:Params: {} 2022-09-02T05:59:48.063 DEBUG:boto:Token: None 2022-09-02T05:59:48.064 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.064 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JayTYbpsK33UXTOTv04tOJqS/ZE= 2022-09-02T05:59:48.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JayTYbpsK33UXTOTv04tOJqS/ZE=', 'Content-Length': '0'} 2022-09-02T05:59:48.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15869d861bbd9b7-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.068 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.068 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.069 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.069 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.070 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.070 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.071 DEBUG:boto:Data: 2022-09-02T05:59:48.071 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.072 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.073 DEBUG:boto:Params: {} 2022-09-02T05:59:48.073 DEBUG:boto:Token: None 2022-09-02T05:59:48.074 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JayTYbpsK33UXTOTv04tOJqS/ZE= 2022-09-02T05:59:48.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JayTYbpsK33UXTOTv04tOJqS/ZE=', 'Content-Length': '0'} 2022-09-02T05:59:48.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6eb4fcd29f1f2e-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:48.079 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.079 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.080 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.080 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.081 DEBUG:boto:Method: GET 2022-09-02T05:59:48.081 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.082 DEBUG:boto:Data: 2022-09-02T05:59:48.082 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.083 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.084 DEBUG:boto:Params: {} 2022-09-02T05:59:48.085 DEBUG:boto:Token: None 2022-09-02T05:59:48.085 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.086 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno= 2022-09-02T05:59:48.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno=', 'Content-Length': '0'} 2022-09-02T05:59:48.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2874fce38f7360a-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.090 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.090 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.091 DEBUG:boto:path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.091 DEBUG:boto:auth_path=/xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.092 DEBUG:boto:Method: GET 2022-09-02T05:59:48.092 DEBUG:boto:Path: /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.093 DEBUG:boto:Data: 2022-09-02T05:59:48.093 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.094 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.094 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.095 DEBUG:boto:Params: {} 2022-09-02T05:59:48.096 DEBUG:boto:Token: None 2022-09-02T05:59:48.096 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38?versionId=null 2022-09-02T05:59:48.097 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno= 2022-09-02T05:59:48.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zy2kFV1gtyIor5YgrvYI0o2Xkno=', 'Content-Length': '0'} 2022-09-02T05:59:48.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b63c0581e42de5b1-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.100 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.101 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.102 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.102 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.103 DEBUG:boto:Method: GET 2022-09-02T05:59:48.103 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.104 DEBUG:boto:Data: 2022-09-02T05:59:48.104 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.105 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.106 DEBUG:boto:Params: {} 2022-09-02T05:59:48.107 DEBUG:boto:Token: None 2022-09-02T05:59:48.107 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.108 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96fcc418fd9fef4-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.112 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.113 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.113 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.114 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.114 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.115 DEBUG:boto:Method: GET 2022-09-02T05:59:48.115 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.116 DEBUG:boto:Data: 2022-09-02T05:59:48.116 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.117 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.117 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.118 DEBUG:boto:Params: {} 2022-09-02T05:59:48.118 DEBUG:boto:Token: None 2022-09-02T05:59:48.119 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.120 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231010a65b2a693e-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.124 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.124 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.125 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.125 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.126 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:48.126 DEBUG:boto:Data: 2022-09-02T05:59:48.127 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.128 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.128 DEBUG:boto:Params: {} 2022-09-02T05:59:48.129 DEBUG:boto:Token: None 2022-09-02T05:59:48.130 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38 2022-09-02T05:59:48.130 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rjyF98FDTL658AsbSR2ZH3vWLjM= 2022-09-02T05:59:48.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rjyF98FDTL658AsbSR2ZH3vWLjM=', 'Content-Length': '0'} 2022-09-02T05:59:48.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce6f40549a32273-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.134 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.134 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.135 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.135 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:48.136 DEBUG:boto:Data: 2022-09-02T05:59:48.136 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.137 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.137 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.138 DEBUG:boto:Params: {} 2022-09-02T05:59:48.138 DEBUG:boto:Token: None 2022-09-02T05:59:48.139 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38 2022-09-02T05:59:48.139 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rjyF98FDTL658AsbSR2ZH3vWLjM= 2022-09-02T05:59:48.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rjyF98FDTL658AsbSR2ZH3vWLjM=', 'Content-Length': '0'} 2022-09-02T05:59:48.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d12ad4ad64dbc4-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2022-09-02T05:59:48.143 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.144 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.144 DEBUG:boto:Method: GET 2022-09-02T05:59:48.145 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:48.145 DEBUG:boto:Data: 2022-09-02T05:59:48.146 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.147 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.148 DEBUG:boto:Params: {} 2022-09-02T05:59:48.148 DEBUG:boto:Token: None 2022-09-02T05:59:48.149 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38 2022-09-02T05:59:48.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k36Gtuc1WOFs2DiE4rsQwOJgUbA= 2022-09-02T05:59:48.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k36Gtuc1WOFs2DiE4rsQwOJgUbA=', 'Content-Length': '0'} 2022-09-02T05:59:48.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a84b338e21214adc-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.198 DEBUG:boto:path=/xljbkf-74/obj38 2022-09-02T05:59:48.199 DEBUG:boto:auth_path=/xljbkf-74/obj38 2022-09-02T05:59:48.199 DEBUG:boto:Method: GET 2022-09-02T05:59:48.200 DEBUG:boto:Path: /xljbkf-74/obj38 2022-09-02T05:59:48.201 DEBUG:boto:Data: 2022-09-02T05:59:48.202 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.202 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.203 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.203 DEBUG:boto:Params: {} 2022-09-02T05:59:48.204 DEBUG:boto:Token: None 2022-09-02T05:59:48.205 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj38 2022-09-02T05:59:48.205 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k36Gtuc1WOFs2DiE4rsQwOJgUbA= 2022-09-02T05:59:48.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k36Gtuc1WOFs2DiE4rsQwOJgUbA=', 'Content-Length': '0'} 2022-09-02T05:59:48.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090b3685c22459e8f-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.209 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.210 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.210 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.211 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.211 DEBUG:boto:Method: GET 2022-09-02T05:59:48.212 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.213 DEBUG:boto:Data: 2022-09-02T05:59:48.213 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.215 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.215 DEBUG:boto:Params: {} 2022-09-02T05:59:48.216 DEBUG:boto:Token: None 2022-09-02T05:59:48.217 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.217 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003210a49efd0da99c-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.221 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.223 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.223 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.224 DEBUG:boto:path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.225 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.225 DEBUG:boto:Method: GET 2022-09-02T05:59:48.226 DEBUG:boto:Path: /xljbkf-74/?prefix=obj38 2022-09-02T05:59:48.226 DEBUG:boto:Data: 2022-09-02T05:59:48.227 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.228 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.228 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.229 DEBUG:boto:Params: {} 2022-09-02T05:59:48.230 DEBUG:boto:Token: None 2022-09-02T05:59:48.230 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.231 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a85a470edc8aeb3-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.235 DEBUG:boto:b'xljbkf-74obj381000falseobj382022-09-02T05:47:57.592Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:48.237 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.237 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.238 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.238 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.239 DEBUG:boto:Method: GET 2022-09-02T05:59:48.239 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.240 DEBUG:boto:Data: 2022-09-02T05:59:48.240 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.241 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.243 DEBUG:boto:Params: {} 2022-09-02T05:59:48.243 DEBUG:boto:Token: None 2022-09-02T05:59:48.244 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs= 2022-09-02T05:59:48.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs=', 'Content-Length': '0'} 2022-09-02T05:59:48.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed1502bc48b883b-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.248 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.249 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.249 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.250 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.250 DEBUG:boto:Method: GET 2022-09-02T05:59:48.251 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.251 DEBUG:boto:Data: 2022-09-02T05:59:48.252 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.252 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.253 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.253 DEBUG:boto:Params: {} 2022-09-02T05:59:48.254 DEBUG:boto:Token: None 2022-09-02T05:59:48.255 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs= 2022-09-02T05:59:48.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs=', 'Content-Length': '0'} 2022-09-02T05:59:48.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009effc0075533c74d-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.302 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.303 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.304 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.304 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.305 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.305 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.306 DEBUG:boto:Data: 2022-09-02T05:59:48.307 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.309 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.310 DEBUG:boto:Params: {} 2022-09-02T05:59:48.311 DEBUG:boto:Token: None 2022-09-02T05:59:48.312 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k2RCxqhTf7Xbw8EWjFkN+2Io3Ic= 2022-09-02T05:59:48.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k2RCxqhTf7Xbw8EWjFkN+2Io3Ic=', 'Content-Length': '0'} 2022-09-02T05:59:48.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddcc0ef588e2a43-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.318 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.318 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.319 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.320 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.321 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.321 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.322 DEBUG:boto:Data: 2022-09-02T05:59:48.323 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.324 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.325 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.326 DEBUG:boto:Params: {} 2022-09-02T05:59:48.326 DEBUG:boto:Token: None 2022-09-02T05:59:48.327 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.328 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:k2RCxqhTf7Xbw8EWjFkN+2Io3Ic= 2022-09-02T05:59:48.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:k2RCxqhTf7Xbw8EWjFkN+2Io3Ic=', 'Content-Length': '0'} 2022-09-02T05:59:48.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f02a444b0d9853-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:48.333 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.334 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.334 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.335 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.335 DEBUG:boto:Method: GET 2022-09-02T05:59:48.336 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.336 DEBUG:boto:Data: 2022-09-02T05:59:48.337 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.338 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.339 DEBUG:boto:Params: {} 2022-09-02T05:59:48.339 DEBUG:boto:Token: None 2022-09-02T05:59:48.340 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.341 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs= 2022-09-02T05:59:48.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs=', 'Content-Length': '0'} 2022-09-02T05:59:48.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024dc8c7d03a53c3d-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.344 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.345 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.345 DEBUG:boto:path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.346 DEBUG:boto:auth_path=/xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.347 DEBUG:boto:Method: GET 2022-09-02T05:59:48.347 DEBUG:boto:Path: /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.348 DEBUG:boto:Data: 2022-09-02T05:59:48.348 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.349 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.349 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.350 DEBUG:boto:Params: {} 2022-09-02T05:59:48.350 DEBUG:boto:Token: None 2022-09-02T05:59:48.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44?versionId=null 2022-09-02T05:59:48.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs= 2022-09-02T05:59:48.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zAknpVeQRn2vvQyeasVRAAy+VPs=', 'Content-Length': '0'} 2022-09-02T05:59:48.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f301ce782b322a5-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.355 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.356 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.356 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.357 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.357 DEBUG:boto:Method: GET 2022-09-02T05:59:48.358 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.358 DEBUG:boto:Data: 2022-09-02T05:59:48.359 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.360 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.360 DEBUG:boto:Params: {} 2022-09-02T05:59:48.361 DEBUG:boto:Token: None 2022-09-02T05:59:48.361 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.362 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000176e1b699ddb42ad-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.366 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.367 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.369 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.370 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.371 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.372 DEBUG:boto:Method: GET 2022-09-02T05:59:48.373 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.374 DEBUG:boto:Data: 2022-09-02T05:59:48.375 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.376 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.377 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.378 DEBUG:boto:Params: {} 2022-09-02T05:59:48.379 DEBUG:boto:Token: None 2022-09-02T05:59:48.380 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.381 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008993502f269f933e-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.427 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.428 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.428 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.429 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.429 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:48.430 DEBUG:boto:Data: 2022-09-02T05:59:48.430 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.431 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.432 DEBUG:boto:Params: {} 2022-09-02T05:59:48.432 DEBUG:boto:Token: None 2022-09-02T05:59:48.433 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44 2022-09-02T05:59:48.433 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HX9JcPsWedCemwSsMobotq7qR84= 2022-09-02T05:59:48.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HX9JcPsWedCemwSsMobotq7qR84=', 'Content-Length': '0'} 2022-09-02T05:59:48.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e72a1f0d615321-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.437 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.438 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.439 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.439 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:48.440 DEBUG:boto:Data: 2022-09-02T05:59:48.440 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.441 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.441 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.442 DEBUG:boto:Params: {} 2022-09-02T05:59:48.443 DEBUG:boto:Token: None 2022-09-02T05:59:48.443 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44 2022-09-02T05:59:48.444 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HX9JcPsWedCemwSsMobotq7qR84= 2022-09-02T05:59:48.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HX9JcPsWedCemwSsMobotq7qR84=', 'Content-Length': '0'} 2022-09-02T05:59:48.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ec1ce1eb0a71fc-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2022-09-02T05:59:48.448 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.449 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.449 DEBUG:boto:Method: GET 2022-09-02T05:59:48.450 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:48.450 DEBUG:boto:Data: 2022-09-02T05:59:48.451 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.452 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.452 DEBUG:boto:Params: {} 2022-09-02T05:59:48.453 DEBUG:boto:Token: None 2022-09-02T05:59:48.454 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44 2022-09-02T05:59:48.455 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3/ifBWLQfvs3Og+71UsHuKGvxY= 2022-09-02T05:59:48.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3/ifBWLQfvs3Og+71UsHuKGvxY=', 'Content-Length': '0'} 2022-09-02T05:59:48.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7cc72ba3fd269b9-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.458 DEBUG:boto:path=/xljbkf-74/obj44 2022-09-02T05:59:48.459 DEBUG:boto:auth_path=/xljbkf-74/obj44 2022-09-02T05:59:48.459 DEBUG:boto:Method: GET 2022-09-02T05:59:48.460 DEBUG:boto:Path: /xljbkf-74/obj44 2022-09-02T05:59:48.461 DEBUG:boto:Data: 2022-09-02T05:59:48.461 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.462 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.462 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.463 DEBUG:boto:Params: {} 2022-09-02T05:59:48.464 DEBUG:boto:Token: None 2022-09-02T05:59:48.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj44 2022-09-02T05:59:48.465 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:g3/ifBWLQfvs3Og+71UsHuKGvxY= 2022-09-02T05:59:48.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:g3/ifBWLQfvs3Og+71UsHuKGvxY=', 'Content-Length': '0'} 2022-09-02T05:59:48.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cea522143c44816-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.469 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.470 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.470 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.471 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.472 DEBUG:boto:Method: GET 2022-09-02T05:59:48.472 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.473 DEBUG:boto:Data: 2022-09-02T05:59:48.474 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.475 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.475 DEBUG:boto:Params: {} 2022-09-02T05:59:48.476 DEBUG:boto:Token: None 2022-09-02T05:59:48.477 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.477 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b3d7fe45872311-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.482 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.483 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.483 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.484 DEBUG:boto:path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.484 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.485 DEBUG:boto:Method: GET 2022-09-02T05:59:48.485 DEBUG:boto:Path: /xljbkf-74/?prefix=obj44 2022-09-02T05:59:48.486 DEBUG:boto:Data: 2022-09-02T05:59:48.486 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.487 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.488 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.488 DEBUG:boto:Params: {} 2022-09-02T05:59:48.489 DEBUG:boto:Token: None 2022-09-02T05:59:48.489 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.490 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2d918a12a5e35c4-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.494 DEBUG:boto:b'xljbkf-74obj441000falseobj442022-09-02T05:44:47.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:48.495 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.496 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.496 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.497 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.498 DEBUG:boto:Method: GET 2022-09-02T05:59:48.498 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.499 DEBUG:boto:Data: 2022-09-02T05:59:48.499 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.500 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.501 DEBUG:boto:Params: {} 2022-09-02T05:59:48.501 DEBUG:boto:Token: None 2022-09-02T05:59:48.502 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.503 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw= 2022-09-02T05:59:48.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw=', 'Content-Length': '0'} 2022-09-02T05:59:48.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651d33612bbd2cff-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.506 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.507 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.508 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.508 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.509 DEBUG:boto:Method: GET 2022-09-02T05:59:48.509 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.510 DEBUG:boto:Data: 2022-09-02T05:59:48.510 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.511 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.511 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.512 DEBUG:boto:Params: {} 2022-09-02T05:59:48.513 DEBUG:boto:Token: None 2022-09-02T05:59:48.513 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.514 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw= 2022-09-02T05:59:48.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw=', 'Content-Length': '0'} 2022-09-02T05:59:48.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad40acd7ad26fbb1-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.518 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.518 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.519 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.519 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.520 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.520 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.521 DEBUG:boto:Data: 2022-09-02T05:59:48.522 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.523 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.523 DEBUG:boto:Params: {} 2022-09-02T05:59:48.524 DEBUG:boto:Token: None 2022-09-02T05:59:48.524 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vV1Ku5bxSU+rmLwJ3EpqSLEecBM= 2022-09-02T05:59:48.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vV1Ku5bxSU+rmLwJ3EpqSLEecBM=', 'Content-Length': '0'} 2022-09-02T05:59:48.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000991f708759b0d6c-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.529 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.529 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.530 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.530 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.531 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.532 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.532 DEBUG:boto:Data: 2022-09-02T05:59:48.533 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.533 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.534 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.534 DEBUG:boto:Params: {} 2022-09-02T05:59:48.535 DEBUG:boto:Token: None 2022-09-02T05:59:48.536 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.536 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vV1Ku5bxSU+rmLwJ3EpqSLEecBM= 2022-09-02T05:59:48.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vV1Ku5bxSU+rmLwJ3EpqSLEecBM=', 'Content-Length': '0'} 2022-09-02T05:59:48.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b997dc065b92f49-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:48.541 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.542 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.542 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.543 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.543 DEBUG:boto:Method: GET 2022-09-02T05:59:48.544 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.544 DEBUG:boto:Data: 2022-09-02T05:59:48.545 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.546 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.547 DEBUG:boto:Params: {} 2022-09-02T05:59:48.547 DEBUG:boto:Token: None 2022-09-02T05:59:48.548 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.549 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw= 2022-09-02T05:59:48.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw=', 'Content-Length': '0'} 2022-09-02T05:59:48.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088242ce9dd72cf2e-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.552 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.553 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.554 DEBUG:boto:path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.554 DEBUG:boto:auth_path=/xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.555 DEBUG:boto:Method: GET 2022-09-02T05:59:48.555 DEBUG:boto:Path: /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.556 DEBUG:boto:Data: 2022-09-02T05:59:48.556 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.557 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.557 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.558 DEBUG:boto:Params: {} 2022-09-02T05:59:48.559 DEBUG:boto:Token: None 2022-09-02T05:59:48.559 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46?versionId=null 2022-09-02T05:59:48.560 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw= 2022-09-02T05:59:48.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JCjYVMk6qNGBSjJExqmjyfQS0Jw=', 'Content-Length': '0'} 2022-09-02T05:59:48.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afc266d348f3c9c-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.610 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.611 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.611 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.612 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.613 DEBUG:boto:Method: GET 2022-09-02T05:59:48.613 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.614 DEBUG:boto:Data: 2022-09-02T05:59:48.614 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.616 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.616 DEBUG:boto:Params: {} 2022-09-02T05:59:48.617 DEBUG:boto:Token: None 2022-09-02T05:59:48.617 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.618 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a192a054bc70073-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.622 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.623 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.623 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.624 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.624 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.625 DEBUG:boto:Method: GET 2022-09-02T05:59:48.625 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.626 DEBUG:boto:Data: 2022-09-02T05:59:48.627 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.627 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.628 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.628 DEBUG:boto:Params: {} 2022-09-02T05:59:48.629 DEBUG:boto:Token: None 2022-09-02T05:59:48.630 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e3e52b962599943-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.634 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.635 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.636 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.636 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.637 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:48.637 DEBUG:boto:Data: 2022-09-02T05:59:48.638 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.639 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.639 DEBUG:boto:Params: {} 2022-09-02T05:59:48.640 DEBUG:boto:Token: None 2022-09-02T05:59:48.640 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46 2022-09-02T05:59:48.641 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ssBpignjhYasJzQ9Wva5gMAZ7MY= 2022-09-02T05:59:48.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ssBpignjhYasJzQ9Wva5gMAZ7MY=', 'Content-Length': '0'} 2022-09-02T05:59:48.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84645f8fa22134e-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.644 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.645 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.645 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.646 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:48.646 DEBUG:boto:Data: 2022-09-02T05:59:48.647 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.647 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.648 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.648 DEBUG:boto:Params: {} 2022-09-02T05:59:48.649 DEBUG:boto:Token: None 2022-09-02T05:59:48.650 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46 2022-09-02T05:59:48.650 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ssBpignjhYasJzQ9Wva5gMAZ7MY= 2022-09-02T05:59:48.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ssBpignjhYasJzQ9Wva5gMAZ7MY=', 'Content-Length': '0'} 2022-09-02T05:59:48.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba5111a357841a3-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2022-09-02T05:59:48.654 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.655 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.655 DEBUG:boto:Method: GET 2022-09-02T05:59:48.656 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:48.656 DEBUG:boto:Data: 2022-09-02T05:59:48.657 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.658 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.658 DEBUG:boto:Params: {} 2022-09-02T05:59:48.659 DEBUG:boto:Token: None 2022-09-02T05:59:48.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46 2022-09-02T05:59:48.660 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BkGF3BTw2mCZSD/Xcea3aKdUvPo= 2022-09-02T05:59:48.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BkGF3BTw2mCZSD/Xcea3aKdUvPo=', 'Content-Length': '0'} 2022-09-02T05:59:48.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842ad5cb9fcb426f-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.664 DEBUG:boto:path=/xljbkf-74/obj46 2022-09-02T05:59:48.664 DEBUG:boto:auth_path=/xljbkf-74/obj46 2022-09-02T05:59:48.665 DEBUG:boto:Method: GET 2022-09-02T05:59:48.665 DEBUG:boto:Path: /xljbkf-74/obj46 2022-09-02T05:59:48.666 DEBUG:boto:Data: 2022-09-02T05:59:48.666 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.667 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.667 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.668 DEBUG:boto:Params: {} 2022-09-02T05:59:48.669 DEBUG:boto:Token: None 2022-09-02T05:59:48.669 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj46 2022-09-02T05:59:48.670 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:BkGF3BTw2mCZSD/Xcea3aKdUvPo= 2022-09-02T05:59:48.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:BkGF3BTw2mCZSD/Xcea3aKdUvPo=', 'Content-Length': '0'} 2022-09-02T05:59:48.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069973c33e987b795-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.673 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.674 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.675 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.675 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.676 DEBUG:boto:Method: GET 2022-09-02T05:59:48.676 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.677 DEBUG:boto:Data: 2022-09-02T05:59:48.677 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.678 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.679 DEBUG:boto:Params: {} 2022-09-02T05:59:48.680 DEBUG:boto:Token: None 2022-09-02T05:59:48.680 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.681 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd88ee1c7b762fa-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.685 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.686 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.686 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.687 DEBUG:boto:path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.687 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.688 DEBUG:boto:Method: GET 2022-09-02T05:59:48.688 DEBUG:boto:Path: /xljbkf-74/?prefix=obj46 2022-09-02T05:59:48.689 DEBUG:boto:Data: 2022-09-02T05:59:48.689 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.690 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.691 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.691 DEBUG:boto:Params: {} 2022-09-02T05:59:48.692 DEBUG:boto:Token: None 2022-09-02T05:59:48.692 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.693 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb69cbbdf014749-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.697 DEBUG:boto:b'xljbkf-74obj461000falseobj462022-09-02T05:50:15.613Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:48.699 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.699 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.700 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.700 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.701 DEBUG:boto:Method: GET 2022-09-02T05:59:48.701 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.702 DEBUG:boto:Data: 2022-09-02T05:59:48.702 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.704 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.704 DEBUG:boto:Params: {} 2022-09-02T05:59:48.705 DEBUG:boto:Token: None 2022-09-02T05:59:48.705 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.706 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584= 2022-09-02T05:59:48.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584=', 'Content-Length': '0'} 2022-09-02T05:59:48.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9099bf0c796e630-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.710 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.710 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.711 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.712 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.712 DEBUG:boto:Method: GET 2022-09-02T05:59:48.713 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.713 DEBUG:boto:Data: 2022-09-02T05:59:48.714 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.714 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.715 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.715 DEBUG:boto:Params: {} 2022-09-02T05:59:48.716 DEBUG:boto:Token: None 2022-09-02T05:59:48.716 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.717 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584= 2022-09-02T05:59:48.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584=', 'Content-Length': '0'} 2022-09-02T05:59:48.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3dd8794da77360-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.721 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.721 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.722 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.722 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.723 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.723 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.724 DEBUG:boto:Data: 2022-09-02T05:59:48.724 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.725 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.726 DEBUG:boto:Params: {} 2022-09-02T05:59:48.727 DEBUG:boto:Token: None 2022-09-02T05:59:48.727 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.728 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WDIpv0iONBdqFRmyuALrr6q+euw= 2022-09-02T05:59:48.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WDIpv0iONBdqFRmyuALrr6q+euw=', 'Content-Length': '0'} 2022-09-02T05:59:48.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702ca4c6453af02f-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.731 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.732 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.732 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.733 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.733 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.734 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.734 DEBUG:boto:Data: 2022-09-02T05:59:48.735 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.735 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.736 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.736 DEBUG:boto:Params: {} 2022-09-02T05:59:48.737 DEBUG:boto:Token: None 2022-09-02T05:59:48.738 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.738 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WDIpv0iONBdqFRmyuALrr6q+euw= 2022-09-02T05:59:48.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WDIpv0iONBdqFRmyuALrr6q+euw=', 'Content-Length': '0'} 2022-09-02T05:59:48.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5181eba90b507f1-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:48.742 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.743 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.743 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.744 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.744 DEBUG:boto:Method: GET 2022-09-02T05:59:48.745 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.745 DEBUG:boto:Data: 2022-09-02T05:59:48.746 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.747 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.748 DEBUG:boto:Params: {} 2022-09-02T05:59:48.748 DEBUG:boto:Token: None 2022-09-02T05:59:48.749 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584= 2022-09-02T05:59:48.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584=', 'Content-Length': '0'} 2022-09-02T05:59:48.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d797af3d70888de8-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.798 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.805 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.806 DEBUG:boto:path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.806 DEBUG:boto:auth_path=/xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.807 DEBUG:boto:Method: GET 2022-09-02T05:59:48.807 DEBUG:boto:Path: /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.808 DEBUG:boto:Data: 2022-09-02T05:59:48.808 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.809 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.809 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.810 DEBUG:boto:Params: {} 2022-09-02T05:59:48.811 DEBUG:boto:Token: None 2022-09-02T05:59:48.811 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51?versionId=null 2022-09-02T05:59:48.812 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584= 2022-09-02T05:59:48.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0rsOAb+O1drXS2knsAjpOhKB584=', 'Content-Length': '0'} 2022-09-02T05:59:48.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066deed594fe6d87c-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.816 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.816 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.817 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.817 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.818 DEBUG:boto:Method: GET 2022-09-02T05:59:48.818 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.819 DEBUG:boto:Data: 2022-09-02T05:59:48.819 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.820 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.821 DEBUG:boto:Params: {} 2022-09-02T05:59:48.822 DEBUG:boto:Token: None 2022-09-02T05:59:48.822 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.823 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ffa9440384ca111-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.827 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.828 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.828 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.829 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.829 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.830 DEBUG:boto:Method: GET 2022-09-02T05:59:48.831 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.831 DEBUG:boto:Data: 2022-09-02T05:59:48.832 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.832 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.833 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.833 DEBUG:boto:Params: {} 2022-09-02T05:59:48.834 DEBUG:boto:Token: None 2022-09-02T05:59:48.834 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.835 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b88a5fb4270bfa0-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.839 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.840 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.840 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.841 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.841 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:48.842 DEBUG:boto:Data: 2022-09-02T05:59:48.842 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.844 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.844 DEBUG:boto:Params: {} 2022-09-02T05:59:48.845 DEBUG:boto:Token: None 2022-09-02T05:59:48.845 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51 2022-09-02T05:59:48.846 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rGSrJAcEwVeQHET9mEYHHS2Hy4I= 2022-09-02T05:59:48.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rGSrJAcEwVeQHET9mEYHHS2Hy4I=', 'Content-Length': '0'} 2022-09-02T05:59:48.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c09533b0049756e7-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.850 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.850 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.851 DEBUG:boto:Method: HEAD 2022-09-02T05:59:48.851 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:48.852 DEBUG:boto:Data: 2022-09-02T05:59:48.852 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.853 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.853 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.854 DEBUG:boto:Params: {} 2022-09-02T05:59:48.854 DEBUG:boto:Token: None 2022-09-02T05:59:48.855 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51 2022-09-02T05:59:48.856 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:rGSrJAcEwVeQHET9mEYHHS2Hy4I= 2022-09-02T05:59:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:rGSrJAcEwVeQHET9mEYHHS2Hy4I=', 'Content-Length': '0'} 2022-09-02T05:59:48.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050390880dfd7d12c-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2022-09-02T05:59:48.859 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.860 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.860 DEBUG:boto:Method: GET 2022-09-02T05:59:48.861 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:48.862 DEBUG:boto:Data: 2022-09-02T05:59:48.862 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.863 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.864 DEBUG:boto:Params: {} 2022-09-02T05:59:48.864 DEBUG:boto:Token: None 2022-09-02T05:59:48.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51 2022-09-02T05:59:48.865 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KPR2vLpeloU1YdDRSkOAdkzWtQI= 2022-09-02T05:59:48.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KPR2vLpeloU1YdDRSkOAdkzWtQI=', 'Content-Length': '0'} 2022-09-02T05:59:48.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9155d766f0372ca-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.869 DEBUG:boto:path=/xljbkf-74/obj51 2022-09-02T05:59:48.870 DEBUG:boto:auth_path=/xljbkf-74/obj51 2022-09-02T05:59:48.870 DEBUG:boto:Method: GET 2022-09-02T05:59:48.871 DEBUG:boto:Path: /xljbkf-74/obj51 2022-09-02T05:59:48.872 DEBUG:boto:Data: 2022-09-02T05:59:48.872 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.873 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.873 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.874 DEBUG:boto:Params: {} 2022-09-02T05:59:48.874 DEBUG:boto:Token: None 2022-09-02T05:59:48.875 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj51 2022-09-02T05:59:48.875 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:KPR2vLpeloU1YdDRSkOAdkzWtQI= 2022-09-02T05:59:48.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:KPR2vLpeloU1YdDRSkOAdkzWtQI=', 'Content-Length': '0'} 2022-09-02T05:59:48.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eea497bb355ce55-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.927 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.929 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.929 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.931 DEBUG:boto:Method: GET 2022-09-02T05:59:48.932 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.933 DEBUG:boto:Data: 2022-09-02T05:59:48.934 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.935 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.937 DEBUG:boto:Params: {} 2022-09-02T05:59:48.938 DEBUG:boto:Token: None 2022-09-02T05:59:48.939 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.940 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a67194b03bedc2ca-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.946 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.948 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:48.949 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:48.950 DEBUG:boto:path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.951 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.952 DEBUG:boto:Method: GET 2022-09-02T05:59:48.953 DEBUG:boto:Path: /xljbkf-74/?prefix=obj51 2022-09-02T05:59:48.954 DEBUG:boto:Data: 2022-09-02T05:59:48.955 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.956 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.957 DEBUG:boto:Port: 8000 2022-09-02T05:59:48.958 DEBUG:boto:Params: {} 2022-09-02T05:59:48.959 DEBUG:boto:Token: None 2022-09-02T05:59:48.961 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/ 2022-09-02T05:59:48.962 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo= 2022-09-02T05:59:48.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:1O6VpqiyVDVpbOa4vMhii990Zfo=', 'Content-Length': '0'} 2022-09-02T05:59:48.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003983aaef00887460-0063119bd4-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.968 DEBUG:boto:b'xljbkf-74obj511000falseobj512022-09-02T05:45:27.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:48.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:48.970 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:48.971 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:48.972 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.973 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.974 DEBUG:boto:Method: GET 2022-09-02T05:59:48.975 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.976 DEBUG:boto:Data: 2022-09-02T05:59:48.977 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:48.979 DEBUG:boto:Port: 8001 2022-09-02T05:59:48.980 DEBUG:boto:Params: {} 2022-09-02T05:59:48.981 DEBUG:boto:Token: None 2022-09-02T05:59:48.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:48 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.984 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:HhwoJWTPWy0dZcIyncxA5IdE2Q0= 2022-09-02T05:59:48.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:48 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:HhwoJWTPWy0dZcIyncxA5IdE2Q0=', 'Content-Length': '0'} 2022-09-02T05:59:48.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046b3781f4659e513-0063119bd4-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:48 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:48.989 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:48.991 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:48.991 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.992 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.993 DEBUG:boto:Method: GET 2022-09-02T05:59:48.994 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:48.996 DEBUG:boto:Data: 2022-09-02T05:59:48.997 DEBUG:boto:Headers: {} 2022-09-02T05:59:48.998 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:48.999 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.000 DEBUG:boto:Params: {} 2022-09-02T05:59:49.000 DEBUG:boto:Token: None 2022-09-02T05:59:49.001 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.002 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Eey3u8Jd+TpCEPVf8idWHRSa1Y= 2022-09-02T05:59:49.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Eey3u8Jd+TpCEPVf8idWHRSa1Y=', 'Content-Length': '0'} 2022-09-02T05:59:49.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115df8059b14d58d-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.005 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.006 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.006 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.007 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.007 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.008 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.008 DEBUG:boto:Data: 2022-09-02T05:59:49.009 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.010 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.010 DEBUG:boto:Params: {} 2022-09-02T05:59:49.011 DEBUG:boto:Token: None 2022-09-02T05:59:49.011 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bISZpeJU+jw1ti4YMsZsN0tNa4= 2022-09-02T05:59:49.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bISZpeJU+jw1ti4YMsZsN0tNa4=', 'Content-Length': '0'} 2022-09-02T05:59:49.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ad51f4290c45b6-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.015 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.016 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.016 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.017 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.017 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.018 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.018 DEBUG:boto:Data: 2022-09-02T05:59:49.019 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.019 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.020 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.021 DEBUG:boto:Params: {} 2022-09-02T05:59:49.021 DEBUG:boto:Token: None 2022-09-02T05:59:49.022 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.023 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+bISZpeJU+jw1ti4YMsZsN0tNa4= 2022-09-02T05:59:49.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+bISZpeJU+jw1ti4YMsZsN0tNa4=', 'Content-Length': '0'} 2022-09-02T05:59:49.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636408376b203316-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:49.026 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.027 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.028 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.028 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.029 DEBUG:boto:Method: GET 2022-09-02T05:59:49.029 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.030 DEBUG:boto:Data: 2022-09-02T05:59:49.030 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.031 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.032 DEBUG:boto:Params: {} 2022-09-02T05:59:49.032 DEBUG:boto:Token: None 2022-09-02T05:59:49.033 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.034 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Eey3u8Jd+TpCEPVf8idWHRSa1Y= 2022-09-02T05:59:49.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Eey3u8Jd+TpCEPVf8idWHRSa1Y=', 'Content-Length': '0'} 2022-09-02T05:59:49.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f00b00a5048a3fc-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.082 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.083 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.083 DEBUG:boto:path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.084 DEBUG:boto:auth_path=/xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.084 DEBUG:boto:Method: GET 2022-09-02T05:59:49.085 DEBUG:boto:Path: /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.086 DEBUG:boto:Data: 2022-09-02T05:59:49.086 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.087 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.088 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.088 DEBUG:boto:Params: {} 2022-09-02T05:59:49.089 DEBUG:boto:Token: None 2022-09-02T05:59:49.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55?versionId=null 2022-09-02T05:59:49.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7Eey3u8Jd+TpCEPVf8idWHRSa1Y= 2022-09-02T05:59:49.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7Eey3u8Jd+TpCEPVf8idWHRSa1Y=', 'Content-Length': '0'} 2022-09-02T05:59:49.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e721912efd1c26f-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.095 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.096 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.097 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.098 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.098 DEBUG:boto:Method: GET 2022-09-02T05:59:49.099 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.100 DEBUG:boto:Data: 2022-09-02T05:59:49.100 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.101 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.102 DEBUG:boto:Params: {} 2022-09-02T05:59:49.103 DEBUG:boto:Token: None 2022-09-02T05:59:49.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b722c36dba14ab3-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.110 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.111 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.112 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.113 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.114 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.115 DEBUG:boto:Method: GET 2022-09-02T05:59:49.116 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.116 DEBUG:boto:Data: 2022-09-02T05:59:49.117 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.118 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.119 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.120 DEBUG:boto:Params: {} 2022-09-02T05:59:49.121 DEBUG:boto:Token: None 2022-09-02T05:59:49.122 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.123 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ef769f42ebd3000-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.174 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.175 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.176 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.176 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.177 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:49.178 DEBUG:boto:Data: 2022-09-02T05:59:49.178 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.180 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.180 DEBUG:boto:Params: {} 2022-09-02T05:59:49.181 DEBUG:boto:Token: None 2022-09-02T05:59:49.182 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55 2022-09-02T05:59:49.182 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mEh9ELBQe629Pjheirod8PHyW90= 2022-09-02T05:59:49.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mEh9ELBQe629Pjheirod8PHyW90=', 'Content-Length': '0'} 2022-09-02T05:59:49.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a5f07932059bc5f-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.186 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.187 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.188 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.188 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:49.189 DEBUG:boto:Data: 2022-09-02T05:59:49.190 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.191 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.191 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.192 DEBUG:boto:Params: {} 2022-09-02T05:59:49.193 DEBUG:boto:Token: None 2022-09-02T05:59:49.195 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55 2022-09-02T05:59:49.196 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mEh9ELBQe629Pjheirod8PHyW90= 2022-09-02T05:59:49.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mEh9ELBQe629Pjheirod8PHyW90=', 'Content-Length': '0'} 2022-09-02T05:59:49.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293d9c70f2ddb943-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2022-09-02T05:59:49.205 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.205 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.206 DEBUG:boto:Method: GET 2022-09-02T05:59:49.207 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:49.208 DEBUG:boto:Data: 2022-09-02T05:59:49.208 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.210 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.211 DEBUG:boto:Params: {} 2022-09-02T05:59:49.212 DEBUG:boto:Token: None 2022-09-02T05:59:49.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55 2022-09-02T05:59:49.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OCWtejAwkNOjh5kB25wE4kX6dvM= 2022-09-02T05:59:49.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OCWtejAwkNOjh5kB25wE4kX6dvM=', 'Content-Length': '0'} 2022-09-02T05:59:49.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068deb433c6982532-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.218 DEBUG:boto:path=/xljbkf-74/obj55 2022-09-02T05:59:49.219 DEBUG:boto:auth_path=/xljbkf-74/obj55 2022-09-02T05:59:49.219 DEBUG:boto:Method: GET 2022-09-02T05:59:49.220 DEBUG:boto:Path: /xljbkf-74/obj55 2022-09-02T05:59:49.221 DEBUG:boto:Data: 2022-09-02T05:59:49.222 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.222 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.223 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.224 DEBUG:boto:Params: {} 2022-09-02T05:59:49.225 DEBUG:boto:Token: None 2022-09-02T05:59:49.225 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj55 2022-09-02T05:59:49.226 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OCWtejAwkNOjh5kB25wE4kX6dvM= 2022-09-02T05:59:49.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OCWtejAwkNOjh5kB25wE4kX6dvM=', 'Content-Length': '0'} 2022-09-02T05:59:49.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:44:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9a208441340586-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.231 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.232 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.233 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.233 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.234 DEBUG:boto:Method: GET 2022-09-02T05:59:49.235 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.236 DEBUG:boto:Data: 2022-09-02T05:59:49.237 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.238 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.239 DEBUG:boto:Params: {} 2022-09-02T05:59:49.240 DEBUG:boto:Token: None 2022-09-02T05:59:49.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f5bedcf1b6a424-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.246 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.247 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.248 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.249 DEBUG:boto:path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.249 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.250 DEBUG:boto:Method: GET 2022-09-02T05:59:49.251 DEBUG:boto:Path: /xljbkf-74/?prefix=obj55 2022-09-02T05:59:49.252 DEBUG:boto:Data: 2022-09-02T05:59:49.252 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.253 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.254 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.255 DEBUG:boto:Params: {} 2022-09-02T05:59:49.256 DEBUG:boto:Token: None 2022-09-02T05:59:49.256 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.257 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a82cd0498bb3c7a-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.262 DEBUG:boto:b'xljbkf-74obj551000falseobj552022-09-02T05:44:47.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:49.263 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.264 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.264 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.265 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.266 DEBUG:boto:Method: GET 2022-09-02T05:59:49.266 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.267 DEBUG:boto:Data: 2022-09-02T05:59:49.268 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.269 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.270 DEBUG:boto:Params: {} 2022-09-02T05:59:49.270 DEBUG:boto:Token: None 2022-09-02T05:59:49.271 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.272 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY= 2022-09-02T05:59:49.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY=', 'Content-Length': '0'} 2022-09-02T05:59:49.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db7009c24364cb4c-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.277 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.277 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.278 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.279 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.279 DEBUG:boto:Method: GET 2022-09-02T05:59:49.280 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.281 DEBUG:boto:Data: 2022-09-02T05:59:49.282 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.282 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.283 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.284 DEBUG:boto:Params: {} 2022-09-02T05:59:49.284 DEBUG:boto:Token: None 2022-09-02T05:59:49.285 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.286 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY= 2022-09-02T05:59:49.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY=', 'Content-Length': '0'} 2022-09-02T05:59:49.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fb3b333febe590-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.290 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.291 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.291 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.292 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.292 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.293 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.294 DEBUG:boto:Data: 2022-09-02T05:59:49.294 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.296 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.296 DEBUG:boto:Params: {} 2022-09-02T05:59:49.297 DEBUG:boto:Token: None 2022-09-02T05:59:49.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mmdf1NWssX5Ajleiz4S+vBk1t5I= 2022-09-02T05:59:49.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mmdf1NWssX5Ajleiz4S+vBk1t5I=', 'Content-Length': '0'} 2022-09-02T05:59:49.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d732fd794e37eb-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.302 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.303 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.303 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.304 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.304 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.305 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.305 DEBUG:boto:Data: 2022-09-02T05:59:49.306 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.306 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.307 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.307 DEBUG:boto:Params: {} 2022-09-02T05:59:49.308 DEBUG:boto:Token: None 2022-09-02T05:59:49.308 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.309 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Mmdf1NWssX5Ajleiz4S+vBk1t5I= 2022-09-02T05:59:49.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Mmdf1NWssX5Ajleiz4S+vBk1t5I=', 'Content-Length': '0'} 2022-09-02T05:59:49.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8ab2bcf78fb9d1-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:49.313 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.314 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.314 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.315 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.315 DEBUG:boto:Method: GET 2022-09-02T05:59:49.316 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.316 DEBUG:boto:Data: 2022-09-02T05:59:49.317 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.318 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.318 DEBUG:boto:Params: {} 2022-09-02T05:59:49.319 DEBUG:boto:Token: None 2022-09-02T05:59:49.320 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.320 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY= 2022-09-02T05:59:49.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY=', 'Content-Length': '0'} 2022-09-02T05:59:49.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002064a120db8ad96a-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.324 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.324 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.325 DEBUG:boto:path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.325 DEBUG:boto:auth_path=/xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.326 DEBUG:boto:Method: GET 2022-09-02T05:59:49.326 DEBUG:boto:Path: /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.327 DEBUG:boto:Data: 2022-09-02T05:59:49.327 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.329 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.329 DEBUG:boto:Params: {} 2022-09-02T05:59:49.330 DEBUG:boto:Token: None 2022-09-02T05:59:49.331 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57?versionId=null 2022-09-02T05:59:49.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY= 2022-09-02T05:59:49.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:htU6UbfNokSFRh97W2zMPbhrFFY=', 'Content-Length': '0'} 2022-09-02T05:59:49.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f0ffc1568bb19a-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.335 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.335 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.336 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.336 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.337 DEBUG:boto:Method: GET 2022-09-02T05:59:49.337 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.338 DEBUG:boto:Data: 2022-09-02T05:59:49.338 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.339 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.340 DEBUG:boto:Params: {} 2022-09-02T05:59:49.341 DEBUG:boto:Token: None 2022-09-02T05:59:49.341 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.342 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d279bb43a2071464-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.346 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.347 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.347 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.348 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.349 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.349 DEBUG:boto:Method: GET 2022-09-02T05:59:49.350 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.350 DEBUG:boto:Data: 2022-09-02T05:59:49.351 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.352 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.352 DEBUG:boto:Params: {} 2022-09-02T05:59:49.353 DEBUG:boto:Token: None 2022-09-02T05:59:49.353 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051d4a8f479ef4eb5-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.358 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.359 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.359 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.360 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.360 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:49.361 DEBUG:boto:Data: 2022-09-02T05:59:49.361 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.362 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.363 DEBUG:boto:Params: {} 2022-09-02T05:59:49.364 DEBUG:boto:Token: None 2022-09-02T05:59:49.364 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57 2022-09-02T05:59:49.365 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+wSjK+IJxMYyJiJ/ElFLmXk9+p0= 2022-09-02T05:59:49.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+wSjK+IJxMYyJiJ/ElFLmXk9+p0=', 'Content-Length': '0'} 2022-09-02T05:59:49.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8168f5d61e5a741-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.368 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.369 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.369 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.370 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:49.370 DEBUG:boto:Data: 2022-09-02T05:59:49.371 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.371 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.372 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.372 DEBUG:boto:Params: {} 2022-09-02T05:59:49.373 DEBUG:boto:Token: None 2022-09-02T05:59:49.373 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57 2022-09-02T05:59:49.374 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:+wSjK+IJxMYyJiJ/ElFLmXk9+p0= 2022-09-02T05:59:49.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:+wSjK+IJxMYyJiJ/ElFLmXk9+p0=', 'Content-Length': '0'} 2022-09-02T05:59:49.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50f2f035b75cc8b-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2022-09-02T05:59:49.378 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.379 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.379 DEBUG:boto:Method: GET 2022-09-02T05:59:49.380 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:49.380 DEBUG:boto:Data: 2022-09-02T05:59:49.381 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.382 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.382 DEBUG:boto:Params: {} 2022-09-02T05:59:49.383 DEBUG:boto:Token: None 2022-09-02T05:59:49.384 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57 2022-09-02T05:59:49.384 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQHxAz+ONSlhsv+oDHcXvOhRumU= 2022-09-02T05:59:49.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQHxAz+ONSlhsv+oDHcXvOhRumU=', 'Content-Length': '0'} 2022-09-02T05:59:49.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238f1ec5041e5bb7-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.388 DEBUG:boto:path=/xljbkf-74/obj57 2022-09-02T05:59:49.389 DEBUG:boto:auth_path=/xljbkf-74/obj57 2022-09-02T05:59:49.389 DEBUG:boto:Method: GET 2022-09-02T05:59:49.390 DEBUG:boto:Path: /xljbkf-74/obj57 2022-09-02T05:59:49.390 DEBUG:boto:Data: 2022-09-02T05:59:49.391 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.391 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.392 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.392 DEBUG:boto:Params: {} 2022-09-02T05:59:49.393 DEBUG:boto:Token: None 2022-09-02T05:59:49.394 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj57 2022-09-02T05:59:49.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TQHxAz+ONSlhsv+oDHcXvOhRumU= 2022-09-02T05:59:49.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TQHxAz+ONSlhsv+oDHcXvOhRumU=', 'Content-Length': '0'} 2022-09-02T05:59:49.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e817128565296b4a-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.398 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.398 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.399 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.399 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.400 DEBUG:boto:Method: GET 2022-09-02T05:59:49.400 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.401 DEBUG:boto:Data: 2022-09-02T05:59:49.401 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.403 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.403 DEBUG:boto:Params: {} 2022-09-02T05:59:49.404 DEBUG:boto:Token: None 2022-09-02T05:59:49.404 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.405 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed93c730cba96cfe-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.450 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.451 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.452 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.453 DEBUG:boto:path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.453 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.454 DEBUG:boto:Method: GET 2022-09-02T05:59:49.454 DEBUG:boto:Path: /xljbkf-74/?prefix=obj57 2022-09-02T05:59:49.455 DEBUG:boto:Data: 2022-09-02T05:59:49.455 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.456 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.456 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.457 DEBUG:boto:Params: {} 2022-09-02T05:59:49.457 DEBUG:boto:Token: None 2022-09-02T05:59:49.458 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.459 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000975268d0e977a62c-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.463 DEBUG:boto:b'xljbkf-74obj571000falseobj572022-09-02T05:47:57.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:49.464 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.465 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.465 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.466 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.466 DEBUG:boto:Method: GET 2022-09-02T05:59:49.467 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.467 DEBUG:boto:Data: 2022-09-02T05:59:49.468 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.469 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.470 DEBUG:boto:Params: {} 2022-09-02T05:59:49.470 DEBUG:boto:Token: None 2022-09-02T05:59:49.471 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.472 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk= 2022-09-02T05:59:49.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk=', 'Content-Length': '0'} 2022-09-02T05:59:49.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5e748152ed584d-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.475 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.476 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.476 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.477 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.477 DEBUG:boto:Method: GET 2022-09-02T05:59:49.478 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.478 DEBUG:boto:Data: 2022-09-02T05:59:49.479 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.479 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.480 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.481 DEBUG:boto:Params: {} 2022-09-02T05:59:49.481 DEBUG:boto:Token: None 2022-09-02T05:59:49.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.482 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk= 2022-09-02T05:59:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk=', 'Content-Length': '0'} 2022-09-02T05:59:49.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081065bf18ce57832-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.530 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.531 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.531 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.532 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.533 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.533 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.534 DEBUG:boto:Data: 2022-09-02T05:59:49.534 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.535 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.536 DEBUG:boto:Params: {} 2022-09-02T05:59:49.537 DEBUG:boto:Token: None 2022-09-02T05:59:49.538 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.538 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d42TWD4+7lmkNqfv/ZMW66GZJnI= 2022-09-02T05:59:49.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d42TWD4+7lmkNqfv/ZMW66GZJnI=', 'Content-Length': '0'} 2022-09-02T05:59:49.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e45119a97facf19-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.544 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.544 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.545 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.546 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.547 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.547 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.548 DEBUG:boto:Data: 2022-09-02T05:59:49.548 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.549 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.550 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.550 DEBUG:boto:Params: {} 2022-09-02T05:59:49.551 DEBUG:boto:Token: None 2022-09-02T05:59:49.552 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:d42TWD4+7lmkNqfv/ZMW66GZJnI= 2022-09-02T05:59:49.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:d42TWD4+7lmkNqfv/ZMW66GZJnI=', 'Content-Length': '0'} 2022-09-02T05:59:49.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d5cc4bf75ab9ac-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:49.556 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.557 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.557 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.558 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.559 DEBUG:boto:Method: GET 2022-09-02T05:59:49.560 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.560 DEBUG:boto:Data: 2022-09-02T05:59:49.561 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.563 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.564 DEBUG:boto:Params: {} 2022-09-02T05:59:49.565 DEBUG:boto:Token: None 2022-09-02T05:59:49.566 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.567 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk= 2022-09-02T05:59:49.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk=', 'Content-Length': '0'} 2022-09-02T05:59:49.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc4f31a395544bb-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.572 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.573 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.574 DEBUG:boto:path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.575 DEBUG:boto:auth_path=/xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.575 DEBUG:boto:Method: GET 2022-09-02T05:59:49.576 DEBUG:boto:Path: /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.577 DEBUG:boto:Data: 2022-09-02T05:59:49.578 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.578 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.579 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.579 DEBUG:boto:Params: {} 2022-09-02T05:59:49.580 DEBUG:boto:Token: None 2022-09-02T05:59:49.581 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58?versionId=null 2022-09-02T05:59:49.581 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk= 2022-09-02T05:59:49.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OfuOOdEJU/DbbRbU+wf9w1mrYTk=', 'Content-Length': '0'} 2022-09-02T05:59:49.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e93380a2c75e81-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.585 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.586 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.586 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.587 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.587 DEBUG:boto:Method: GET 2022-09-02T05:59:49.588 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.588 DEBUG:boto:Data: 2022-09-02T05:59:49.589 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.590 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.591 DEBUG:boto:Params: {} 2022-09-02T05:59:49.592 DEBUG:boto:Token: None 2022-09-02T05:59:49.592 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.593 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c3e0d3faaed6ad-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.646 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.647 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.648 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.649 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.649 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.650 DEBUG:boto:Method: GET 2022-09-02T05:59:49.650 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.651 DEBUG:boto:Data: 2022-09-02T05:59:49.652 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.653 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.653 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.654 DEBUG:boto:Params: {} 2022-09-02T05:59:49.654 DEBUG:boto:Token: None 2022-09-02T05:59:49.655 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.656 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000584e6ea4294b8a3a-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.660 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.661 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.661 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.662 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.663 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:49.663 DEBUG:boto:Data: 2022-09-02T05:59:49.664 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.665 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.666 DEBUG:boto:Params: {} 2022-09-02T05:59:49.666 DEBUG:boto:Token: None 2022-09-02T05:59:49.667 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58 2022-09-02T05:59:49.668 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D0v3nVVDwResMVtp6/DD/JUecYA= 2022-09-02T05:59:49.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D0v3nVVDwResMVtp6/DD/JUecYA=', 'Content-Length': '0'} 2022-09-02T05:59:49.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073043f91ebc3d46c-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.671 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.672 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.673 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.673 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:49.674 DEBUG:boto:Data: 2022-09-02T05:59:49.676 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.677 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.678 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.679 DEBUG:boto:Params: {} 2022-09-02T05:59:49.681 DEBUG:boto:Token: None 2022-09-02T05:59:49.682 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58 2022-09-02T05:59:49.683 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:D0v3nVVDwResMVtp6/DD/JUecYA= 2022-09-02T05:59:49.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:D0v3nVVDwResMVtp6/DD/JUecYA=', 'Content-Length': '0'} 2022-09-02T05:59:49.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b6e3407d2b6fe8-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2022-09-02T05:59:49.689 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.690 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.690 DEBUG:boto:Method: GET 2022-09-02T05:59:49.691 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:49.692 DEBUG:boto:Data: 2022-09-02T05:59:49.692 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.694 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.695 DEBUG:boto:Params: {} 2022-09-02T05:59:49.695 DEBUG:boto:Token: None 2022-09-02T05:59:49.696 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58 2022-09-02T05:59:49.697 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q4G5CNG2CujfYRMmrxY7asFYl9c= 2022-09-02T05:59:49.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q4G5CNG2CujfYRMmrxY7asFYl9c=', 'Content-Length': '0'} 2022-09-02T05:59:49.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000373f1b0543125c0e-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.701 DEBUG:boto:path=/xljbkf-74/obj58 2022-09-02T05:59:49.702 DEBUG:boto:auth_path=/xljbkf-74/obj58 2022-09-02T05:59:49.703 DEBUG:boto:Method: GET 2022-09-02T05:59:49.704 DEBUG:boto:Path: /xljbkf-74/obj58 2022-09-02T05:59:49.705 DEBUG:boto:Data: 2022-09-02T05:59:49.706 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.708 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.709 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.711 DEBUG:boto:Params: {} 2022-09-02T05:59:49.711 DEBUG:boto:Token: None 2022-09-02T05:59:49.712 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj58 2022-09-02T05:59:49.713 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:q4G5CNG2CujfYRMmrxY7asFYl9c= 2022-09-02T05:59:49.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:q4G5CNG2CujfYRMmrxY7asFYl9c=', 'Content-Length': '0'} 2022-09-02T05:59:49.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2a4335709d9384-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.766 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.768 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.769 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.770 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.771 DEBUG:boto:Method: GET 2022-09-02T05:59:49.772 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.773 DEBUG:boto:Data: 2022-09-02T05:59:49.774 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.776 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.777 DEBUG:boto:Params: {} 2022-09-02T05:59:49.779 DEBUG:boto:Token: None 2022-09-02T05:59:49.780 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.781 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a6496f17a528936-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.787 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.789 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.790 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.791 DEBUG:boto:path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.792 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.793 DEBUG:boto:Method: GET 2022-09-02T05:59:49.794 DEBUG:boto:Path: /xljbkf-74/?prefix=obj58 2022-09-02T05:59:49.795 DEBUG:boto:Data: 2022-09-02T05:59:49.796 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.797 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.798 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.799 DEBUG:boto:Params: {} 2022-09-02T05:59:49.801 DEBUG:boto:Token: None 2022-09-02T05:59:49.802 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.803 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e66763f98259af-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.810 DEBUG:boto:b'xljbkf-74obj581000falseobj582022-09-02T05:53:22.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:49.813 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.814 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.815 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.815 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.816 DEBUG:boto:Method: GET 2022-09-02T05:59:49.816 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.817 DEBUG:boto:Data: 2022-09-02T05:59:49.817 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.818 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.819 DEBUG:boto:Params: {} 2022-09-02T05:59:49.820 DEBUG:boto:Token: None 2022-09-02T05:59:49.820 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.821 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg= 2022-09-02T05:59:49.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg=', 'Content-Length': '0'} 2022-09-02T05:59:49.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16e8f6b1d5c9d3d-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.825 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.826 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.826 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.827 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.828 DEBUG:boto:Method: GET 2022-09-02T05:59:49.828 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.829 DEBUG:boto:Data: 2022-09-02T05:59:49.829 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.830 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.830 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.831 DEBUG:boto:Params: {} 2022-09-02T05:59:49.831 DEBUG:boto:Token: None 2022-09-02T05:59:49.832 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.833 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg= 2022-09-02T05:59:49.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg=', 'Content-Length': '0'} 2022-09-02T05:59:49.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71101fefef3cfdc-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.836 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.837 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.837 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.838 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.839 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.839 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.840 DEBUG:boto:Data: 2022-09-02T05:59:49.840 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.841 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.842 DEBUG:boto:Params: {} 2022-09-02T05:59:49.842 DEBUG:boto:Token: None 2022-09-02T05:59:49.843 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.843 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4+IMoaIjl0VIDzZ0sKOGHcL3Z2s= 2022-09-02T05:59:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4+IMoaIjl0VIDzZ0sKOGHcL3Z2s=', 'Content-Length': '0'} 2022-09-02T05:59:49.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000533842b5d867c69-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.847 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.848 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.848 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.849 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.849 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.850 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.851 DEBUG:boto:Data: 2022-09-02T05:59:49.852 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.852 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.853 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.853 DEBUG:boto:Params: {} 2022-09-02T05:59:49.854 DEBUG:boto:Token: None 2022-09-02T05:59:49.855 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.855 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4+IMoaIjl0VIDzZ0sKOGHcL3Z2s= 2022-09-02T05:59:49.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4+IMoaIjl0VIDzZ0sKOGHcL3Z2s=', 'Content-Length': '0'} 2022-09-02T05:59:49.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3643dc286971682-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:49.859 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.860 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.861 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.861 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.862 DEBUG:boto:Method: GET 2022-09-02T05:59:49.862 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.863 DEBUG:boto:Data: 2022-09-02T05:59:49.863 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.865 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.865 DEBUG:boto:Params: {} 2022-09-02T05:59:49.866 DEBUG:boto:Token: None 2022-09-02T05:59:49.866 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.867 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg= 2022-09-02T05:59:49.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg=', 'Content-Length': '0'} 2022-09-02T05:59:49.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e734e28265360084-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.914 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.915 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.915 DEBUG:boto:path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.916 DEBUG:boto:auth_path=/xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.916 DEBUG:boto:Method: GET 2022-09-02T05:59:49.917 DEBUG:boto:Path: /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.918 DEBUG:boto:Data: 2022-09-02T05:59:49.918 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.919 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.919 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.920 DEBUG:boto:Params: {} 2022-09-02T05:59:49.920 DEBUG:boto:Token: None 2022-09-02T05:59:49.921 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62?versionId=null 2022-09-02T05:59:49.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg= 2022-09-02T05:59:49.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:B1SGWnJejUrOEOMuGfKr16tnMbg=', 'Content-Length': '0'} 2022-09-02T05:59:49.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f68fc836d90fc70-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.926 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.926 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.927 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.927 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.928 DEBUG:boto:Method: GET 2022-09-02T05:59:49.928 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.929 DEBUG:boto:Data: 2022-09-02T05:59:49.929 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.930 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.931 DEBUG:boto:Params: {} 2022-09-02T05:59:49.931 DEBUG:boto:Token: None 2022-09-02T05:59:49.932 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dccd92a7059f330-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.937 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.938 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.938 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.939 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.939 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.940 DEBUG:boto:Method: GET 2022-09-02T05:59:49.941 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.941 DEBUG:boto:Data: 2022-09-02T05:59:49.942 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.942 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.943 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.943 DEBUG:boto:Params: {} 2022-09-02T05:59:49.944 DEBUG:boto:Token: None 2022-09-02T05:59:49.945 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:49.945 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:49.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:49.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007911e4eca75337f0-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.949 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:49.950 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.951 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.951 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.952 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:49.952 DEBUG:boto:Data: 2022-09-02T05:59:49.953 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.954 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.954 DEBUG:boto:Params: {} 2022-09-02T05:59:49.955 DEBUG:boto:Token: None 2022-09-02T05:59:49.956 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62 2022-09-02T05:59:49.956 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XJtL6gSiagVXMi5DhIVQkwehrQk= 2022-09-02T05:59:49.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XJtL6gSiagVXMi5DhIVQkwehrQk=', 'Content-Length': '0'} 2022-09-02T05:59:49.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aecb43648a40676-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.960 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.961 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.961 DEBUG:boto:Method: HEAD 2022-09-02T05:59:49.962 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:49.963 DEBUG:boto:Data: 2022-09-02T05:59:49.964 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.964 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.965 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.965 DEBUG:boto:Params: {} 2022-09-02T05:59:49.966 DEBUG:boto:Token: None 2022-09-02T05:59:49.966 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62 2022-09-02T05:59:49.967 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:XJtL6gSiagVXMi5DhIVQkwehrQk= 2022-09-02T05:59:49.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:XJtL6gSiagVXMi5DhIVQkwehrQk=', 'Content-Length': '0'} 2022-09-02T05:59:49.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67c8266f6bb2f25-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2022-09-02T05:59:49.972 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.972 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.973 DEBUG:boto:Method: GET 2022-09-02T05:59:49.973 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:49.974 DEBUG:boto:Data: 2022-09-02T05:59:49.974 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.976 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.976 DEBUG:boto:Params: {} 2022-09-02T05:59:49.977 DEBUG:boto:Token: None 2022-09-02T05:59:49.977 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62 2022-09-02T05:59:49.978 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TVgMMfVDtm3sgKJdFZrVxAFR4MU= 2022-09-02T05:59:49.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TVgMMfVDtm3sgKJdFZrVxAFR4MU=', 'Content-Length': '0'} 2022-09-02T05:59:49.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043bba7ed1b2e1d94-0063119bd5-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.982 DEBUG:boto:path=/xljbkf-74/obj62 2022-09-02T05:59:49.983 DEBUG:boto:auth_path=/xljbkf-74/obj62 2022-09-02T05:59:49.983 DEBUG:boto:Method: GET 2022-09-02T05:59:49.984 DEBUG:boto:Path: /xljbkf-74/obj62 2022-09-02T05:59:49.984 DEBUG:boto:Data: 2022-09-02T05:59:49.985 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.986 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:49.986 DEBUG:boto:Port: 8000 2022-09-02T05:59:49.987 DEBUG:boto:Params: {} 2022-09-02T05:59:49.987 DEBUG:boto:Token: None 2022-09-02T05:59:49.988 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/obj62 2022-09-02T05:59:49.988 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TVgMMfVDtm3sgKJdFZrVxAFR4MU= 2022-09-02T05:59:49.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TVgMMfVDtm3sgKJdFZrVxAFR4MU=', 'Content-Length': '0'} 2022-09-02T05:59:49.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000760b32ee5a4e85c4-0063119bd5-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:49 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:49.992 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:49.993 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:49.993 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.994 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.995 DEBUG:boto:Method: GET 2022-09-02T05:59:49.995 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:49.996 DEBUG:boto:Data: 2022-09-02T05:59:49.996 DEBUG:boto:Headers: {} 2022-09-02T05:59:49.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:49.997 DEBUG:boto:Port: 8001 2022-09-02T05:59:49.998 DEBUG:boto:Params: {} 2022-09-02T05:59:49.999 DEBUG:boto:Token: None 2022-09-02T05:59:49.999 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:49 GMT /xljbkf-74/ 2022-09-02T05:59:50.000 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8= 2022-09-02T05:59:50.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:49 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6IVTRxMjw5dON+w8hCrA0xkdNq8=', 'Content-Length': '0'} 2022-09-02T05:59:50.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004214c7fdbc59d227-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.004 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.005 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.005 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.006 DEBUG:boto:path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:50.006 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj62 2022-09-02T05:59:50.007 DEBUG:boto:Method: GET 2022-09-02T05:59:50.007 DEBUG:boto:Path: /xljbkf-74/?prefix=obj62 2022-09-02T05:59:50.008 DEBUG:boto:Data: 2022-09-02T05:59:50.009 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.009 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.010 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.010 DEBUG:boto:Params: {} 2022-09-02T05:59:50.011 DEBUG:boto:Token: None 2022-09-02T05:59:50.012 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.012 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006224526cf4af891-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.016 DEBUG:boto:b'xljbkf-74obj621000falseobj622022-09-02T05:56:04.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:50.017 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.018 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.019 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.019 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.020 DEBUG:boto:Method: GET 2022-09-02T05:59:50.020 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.021 DEBUG:boto:Data: 2022-09-02T05:59:50.021 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.023 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.023 DEBUG:boto:Params: {} 2022-09-02T05:59:50.024 DEBUG:boto:Token: None 2022-09-02T05:59:50.024 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.025 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4= 2022-09-02T05:59:50.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4=', 'Content-Length': '0'} 2022-09-02T05:59:50.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000231d536a282c927-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.029 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.029 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.030 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.030 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.031 DEBUG:boto:Method: GET 2022-09-02T05:59:50.032 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.032 DEBUG:boto:Data: 2022-09-02T05:59:50.033 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.033 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.034 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.034 DEBUG:boto:Params: {} 2022-09-02T05:59:50.035 DEBUG:boto:Token: None 2022-09-02T05:59:50.035 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.036 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4= 2022-09-02T05:59:50.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4=', 'Content-Length': '0'} 2022-09-02T05:59:50.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ecd499bd42a6d1-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.040 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.040 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.041 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.041 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.042 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.042 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.043 DEBUG:boto:Data: 2022-09-02T05:59:50.043 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.044 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.045 DEBUG:boto:Params: {} 2022-09-02T05:59:50.045 DEBUG:boto:Token: None 2022-09-02T05:59:50.046 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.046 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SyDxlnm2DaWuHqRIFBltjmiu1KQ= 2022-09-02T05:59:50.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SyDxlnm2DaWuHqRIFBltjmiu1KQ=', 'Content-Length': '0'} 2022-09-02T05:59:50.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000822e005e2b7e863e-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.050 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.050 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.051 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.051 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.052 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.052 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.053 DEBUG:boto:Data: 2022-09-02T05:59:50.053 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.054 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.054 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.055 DEBUG:boto:Params: {} 2022-09-02T05:59:50.055 DEBUG:boto:Token: None 2022-09-02T05:59:50.056 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.056 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SyDxlnm2DaWuHqRIFBltjmiu1KQ= 2022-09-02T05:59:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SyDxlnm2DaWuHqRIFBltjmiu1KQ=', 'Content-Length': '0'} 2022-09-02T05:59:50.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec77412b60d5ddc3-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:50.060 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.061 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.061 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.062 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.062 DEBUG:boto:Method: GET 2022-09-02T05:59:50.063 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.063 DEBUG:boto:Data: 2022-09-02T05:59:50.064 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.065 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.066 DEBUG:boto:Params: {} 2022-09-02T05:59:50.066 DEBUG:boto:Token: None 2022-09-02T05:59:50.067 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.068 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4= 2022-09-02T05:59:50.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4=', 'Content-Length': '0'} 2022-09-02T05:59:50.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a30b316fde584f-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.072 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.072 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.073 DEBUG:boto:path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.073 DEBUG:boto:auth_path=/xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.074 DEBUG:boto:Method: GET 2022-09-02T05:59:50.074 DEBUG:boto:Path: /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.075 DEBUG:boto:Data: 2022-09-02T05:59:50.076 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.076 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.077 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.077 DEBUG:boto:Params: {} 2022-09-02T05:59:50.078 DEBUG:boto:Token: None 2022-09-02T05:59:50.078 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68?versionId=null 2022-09-02T05:59:50.079 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4= 2022-09-02T05:59:50.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:J2rZ1dTS/DMqLv3/6eh1vo7YYK4=', 'Content-Length': '0'} 2022-09-02T05:59:50.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a4aa18f0a5529e-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.082 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.083 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.084 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.084 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.085 DEBUG:boto:Method: GET 2022-09-02T05:59:50.085 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.086 DEBUG:boto:Data: 2022-09-02T05:59:50.086 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.087 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.088 DEBUG:boto:Params: {} 2022-09-02T05:59:50.088 DEBUG:boto:Token: None 2022-09-02T05:59:50.089 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.089 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000897c90027bb540ca-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.138 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.140 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.141 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.141 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.142 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.143 DEBUG:boto:Method: GET 2022-09-02T05:59:50.144 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.145 DEBUG:boto:Data: 2022-09-02T05:59:50.146 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.146 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.147 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.147 DEBUG:boto:Params: {} 2022-09-02T05:59:50.149 DEBUG:boto:Token: None 2022-09-02T05:59:50.150 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.150 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff561af75eb44a16-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.154 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.155 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.156 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.156 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.157 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:50.158 DEBUG:boto:Data: 2022-09-02T05:59:50.158 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.159 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.160 DEBUG:boto:Params: {} 2022-09-02T05:59:50.160 DEBUG:boto:Token: None 2022-09-02T05:59:50.161 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68 2022-09-02T05:59:50.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6u+LQbSnhdiTAcTyvC2g2lr/TYs= 2022-09-02T05:59:50.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6u+LQbSnhdiTAcTyvC2g2lr/TYs=', 'Content-Length': '0'} 2022-09-02T05:59:50.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500353f1d4ec729f-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.165 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.165 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.166 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.166 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:50.167 DEBUG:boto:Data: 2022-09-02T05:59:50.167 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.168 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.168 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.169 DEBUG:boto:Params: {} 2022-09-02T05:59:50.170 DEBUG:boto:Token: None 2022-09-02T05:59:50.170 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68 2022-09-02T05:59:50.171 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:6u+LQbSnhdiTAcTyvC2g2lr/TYs= 2022-09-02T05:59:50.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:6u+LQbSnhdiTAcTyvC2g2lr/TYs=', 'Content-Length': '0'} 2022-09-02T05:59:50.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b6f553706d600e-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2022-09-02T05:59:50.175 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.175 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.176 DEBUG:boto:Method: GET 2022-09-02T05:59:50.176 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:50.177 DEBUG:boto:Data: 2022-09-02T05:59:50.177 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.178 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.179 DEBUG:boto:Params: {} 2022-09-02T05:59:50.179 DEBUG:boto:Token: None 2022-09-02T05:59:50.180 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68 2022-09-02T05:59:50.181 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v7FD5+OsjuY0PMUK+TkZXooAMNk= 2022-09-02T05:59:50.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v7FD5+OsjuY0PMUK+TkZXooAMNk=', 'Content-Length': '0'} 2022-09-02T05:59:50.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b9fb6de16c0d0ee-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.184 DEBUG:boto:path=/xljbkf-74/obj68 2022-09-02T05:59:50.185 DEBUG:boto:auth_path=/xljbkf-74/obj68 2022-09-02T05:59:50.185 DEBUG:boto:Method: GET 2022-09-02T05:59:50.186 DEBUG:boto:Path: /xljbkf-74/obj68 2022-09-02T05:59:50.186 DEBUG:boto:Data: 2022-09-02T05:59:50.187 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.188 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.188 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.189 DEBUG:boto:Params: {} 2022-09-02T05:59:50.189 DEBUG:boto:Token: None 2022-09-02T05:59:50.190 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj68 2022-09-02T05:59:50.190 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:v7FD5+OsjuY0PMUK+TkZXooAMNk= 2022-09-02T05:59:50.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:v7FD5+OsjuY0PMUK+TkZXooAMNk=', 'Content-Length': '0'} 2022-09-02T05:59:50.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaf7392a6dbb08ef-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.234 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.235 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.235 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.236 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.236 DEBUG:boto:Method: GET 2022-09-02T05:59:50.237 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.238 DEBUG:boto:Data: 2022-09-02T05:59:50.238 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.239 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.240 DEBUG:boto:Params: {} 2022-09-02T05:59:50.240 DEBUG:boto:Token: None 2022-09-02T05:59:50.241 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.242 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d8c2099bf33ae4-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.245 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.246 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.247 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.247 DEBUG:boto:path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.248 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.249 DEBUG:boto:Method: GET 2022-09-02T05:59:50.249 DEBUG:boto:Path: /xljbkf-74/?prefix=obj68 2022-09-02T05:59:50.250 DEBUG:boto:Data: 2022-09-02T05:59:50.250 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.251 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.251 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.252 DEBUG:boto:Params: {} 2022-09-02T05:59:50.252 DEBUG:boto:Token: None 2022-09-02T05:59:50.253 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.254 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a302e25f98b87f7-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.258 DEBUG:boto:b'xljbkf-74obj681000falseobj682022-09-02T05:45:27.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:50.259 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.259 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.260 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.261 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.261 DEBUG:boto:Method: GET 2022-09-02T05:59:50.262 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.262 DEBUG:boto:Data: 2022-09-02T05:59:50.263 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.264 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.264 DEBUG:boto:Params: {} 2022-09-02T05:59:50.265 DEBUG:boto:Token: None 2022-09-02T05:59:50.265 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.266 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4= 2022-09-02T05:59:50.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4=', 'Content-Length': '0'} 2022-09-02T05:59:50.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e83abd853012faf-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.270 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.270 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.271 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.272 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.272 DEBUG:boto:Method: GET 2022-09-02T05:59:50.273 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.273 DEBUG:boto:Data: 2022-09-02T05:59:50.274 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.274 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.275 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.276 DEBUG:boto:Params: {} 2022-09-02T05:59:50.276 DEBUG:boto:Token: None 2022-09-02T05:59:50.277 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.278 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4= 2022-09-02T05:59:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4=', 'Content-Length': '0'} 2022-09-02T05:59:50.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de02a3088633ee2b-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.281 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.281 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.282 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.282 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.283 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.284 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.284 DEBUG:boto:Data: 2022-09-02T05:59:50.285 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.286 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.286 DEBUG:boto:Params: {} 2022-09-02T05:59:50.287 DEBUG:boto:Token: None 2022-09-02T05:59:50.287 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.288 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nies3mcZmvHKWPPzXZHxceK0F5Q= 2022-09-02T05:59:50.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nies3mcZmvHKWPPzXZHxceK0F5Q=', 'Content-Length': '0'} 2022-09-02T05:59:50.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236a64fb2e6e492d-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.291 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.292 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.292 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.293 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.294 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.294 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.294 DEBUG:boto:Data: 2022-09-02T05:59:50.295 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.296 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.296 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.297 DEBUG:boto:Params: {} 2022-09-02T05:59:50.297 DEBUG:boto:Token: None 2022-09-02T05:59:50.298 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.298 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Nies3mcZmvHKWPPzXZHxceK0F5Q= 2022-09-02T05:59:50.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Nies3mcZmvHKWPPzXZHxceK0F5Q=', 'Content-Length': '0'} 2022-09-02T05:59:50.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cde37f16edf1c58-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:50.304 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.304 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.305 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.305 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.306 DEBUG:boto:Method: GET 2022-09-02T05:59:50.307 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.307 DEBUG:boto:Data: 2022-09-02T05:59:50.308 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.309 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.310 DEBUG:boto:Params: {} 2022-09-02T05:59:50.311 DEBUG:boto:Token: None 2022-09-02T05:59:50.312 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.313 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4= 2022-09-02T05:59:50.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4=', 'Content-Length': '0'} 2022-09-02T05:59:50.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007375989d0d18330e-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.316 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.317 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.318 DEBUG:boto:path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.318 DEBUG:boto:auth_path=/xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.319 DEBUG:boto:Method: GET 2022-09-02T05:59:50.320 DEBUG:boto:Path: /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.320 DEBUG:boto:Data: 2022-09-02T05:59:50.321 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.321 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.322 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.323 DEBUG:boto:Params: {} 2022-09-02T05:59:50.324 DEBUG:boto:Token: None 2022-09-02T05:59:50.324 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69?versionId=null 2022-09-02T05:59:50.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4= 2022-09-02T05:59:50.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:UVObUgcSLFNlwBf3axQyfkGH+C4=', 'Content-Length': '0'} 2022-09-02T05:59:50.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971e9586d55db493-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.329 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.330 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.331 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.331 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.332 DEBUG:boto:Method: GET 2022-09-02T05:59:50.333 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.333 DEBUG:boto:Data: 2022-09-02T05:59:50.334 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.335 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.336 DEBUG:boto:Params: {} 2022-09-02T05:59:50.337 DEBUG:boto:Token: None 2022-09-02T05:59:50.338 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.338 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fa2d7c1597ab41f-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.343 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.344 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.344 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.345 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.345 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.346 DEBUG:boto:Method: GET 2022-09-02T05:59:50.347 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.347 DEBUG:boto:Data: 2022-09-02T05:59:50.348 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.348 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.349 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.349 DEBUG:boto:Params: {} 2022-09-02T05:59:50.350 DEBUG:boto:Token: None 2022-09-02T05:59:50.351 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.351 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e23ddaab0ce0c00-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.356 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.357 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.357 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.358 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.358 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:50.359 DEBUG:boto:Data: 2022-09-02T05:59:50.359 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.360 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.361 DEBUG:boto:Params: {} 2022-09-02T05:59:50.361 DEBUG:boto:Token: None 2022-09-02T05:59:50.362 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69 2022-09-02T05:59:50.363 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI7r6o54CziUapnwK2YRYwaq0Ck= 2022-09-02T05:59:50.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI7r6o54CziUapnwK2YRYwaq0Ck=', 'Content-Length': '0'} 2022-09-02T05:59:50.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a1a47d89e5d3c90-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.367 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.368 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.369 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.370 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:50.371 DEBUG:boto:Data: 2022-09-02T05:59:50.371 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.372 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.373 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.374 DEBUG:boto:Params: {} 2022-09-02T05:59:50.375 DEBUG:boto:Token: None 2022-09-02T05:59:50.376 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69 2022-09-02T05:59:50.377 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:LI7r6o54CziUapnwK2YRYwaq0Ck= 2022-09-02T05:59:50.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:LI7r6o54CziUapnwK2YRYwaq0Ck=', 'Content-Length': '0'} 2022-09-02T05:59:50.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008899bf06a933e309-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2022-09-02T05:59:50.384 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.385 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.386 DEBUG:boto:Method: GET 2022-09-02T05:59:50.387 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:50.388 DEBUG:boto:Data: 2022-09-02T05:59:50.388 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.390 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.391 DEBUG:boto:Params: {} 2022-09-02T05:59:50.392 DEBUG:boto:Token: None 2022-09-02T05:59:50.393 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69 2022-09-02T05:59:50.394 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kAEL05fVRckkHuAp6J13j3VYMXs= 2022-09-02T05:59:50.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kAEL05fVRckkHuAp6J13j3VYMXs=', 'Content-Length': '0'} 2022-09-02T05:59:50.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483f739ada80cadc-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.442 DEBUG:boto:path=/xljbkf-74/obj69 2022-09-02T05:59:50.443 DEBUG:boto:auth_path=/xljbkf-74/obj69 2022-09-02T05:59:50.445 DEBUG:boto:Method: GET 2022-09-02T05:59:50.446 DEBUG:boto:Path: /xljbkf-74/obj69 2022-09-02T05:59:50.447 DEBUG:boto:Data: 2022-09-02T05:59:50.448 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.449 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.450 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.451 DEBUG:boto:Params: {} 2022-09-02T05:59:50.452 DEBUG:boto:Token: None 2022-09-02T05:59:50.453 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj69 2022-09-02T05:59:50.454 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:kAEL05fVRckkHuAp6J13j3VYMXs= 2022-09-02T05:59:50.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:kAEL05fVRckkHuAp6J13j3VYMXs=', 'Content-Length': '0'} 2022-09-02T05:59:50.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d7d7ca0dd42b40-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.460 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.460 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.461 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.461 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.462 DEBUG:boto:Method: GET 2022-09-02T05:59:50.463 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.463 DEBUG:boto:Data: 2022-09-02T05:59:50.464 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.465 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.466 DEBUG:boto:Params: {} 2022-09-02T05:59:50.467 DEBUG:boto:Token: None 2022-09-02T05:59:50.468 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.469 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196e4ab54a1cd4b3-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.473 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.474 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.475 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.475 DEBUG:boto:path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.476 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.477 DEBUG:boto:Method: GET 2022-09-02T05:59:50.478 DEBUG:boto:Path: /xljbkf-74/?prefix=obj69 2022-09-02T05:59:50.478 DEBUG:boto:Data: 2022-09-02T05:59:50.479 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.480 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.480 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.481 DEBUG:boto:Params: {} 2022-09-02T05:59:50.481 DEBUG:boto:Token: None 2022-09-02T05:59:50.482 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.483 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae2a560a8246fe5f-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.534 DEBUG:boto:b'xljbkf-74obj691000falseobj692022-09-02T05:50:15.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:50.537 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.538 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.539 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.540 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.542 DEBUG:boto:Method: GET 2022-09-02T05:59:50.543 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.544 DEBUG:boto:Data: 2022-09-02T05:59:50.545 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.547 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.548 DEBUG:boto:Params: {} 2022-09-02T05:59:50.549 DEBUG:boto:Token: None 2022-09-02T05:59:50.550 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.552 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE= 2022-09-02T05:59:50.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE=', 'Content-Length': '0'} 2022-09-02T05:59:50.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055002d43a937fdcb-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.556 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.557 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.558 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.559 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.560 DEBUG:boto:Method: GET 2022-09-02T05:59:50.561 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.562 DEBUG:boto:Data: 2022-09-02T05:59:50.563 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.565 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.566 DEBUG:boto:Params: {} 2022-09-02T05:59:50.567 DEBUG:boto:Token: None 2022-09-02T05:59:50.569 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.570 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE= 2022-09-02T05:59:50.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE=', 'Content-Length': '0'} 2022-09-02T05:59:50.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005305ced1788458d0-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.575 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.576 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.577 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.578 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.579 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.580 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.581 DEBUG:boto:Data: 2022-09-02T05:59:50.582 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.584 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.585 DEBUG:boto:Params: {} 2022-09-02T05:59:50.587 DEBUG:boto:Token: None 2022-09-02T05:59:50.588 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.589 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JOUgvm2OSwXOpDLFPNtbeN6POrs= 2022-09-02T05:59:50.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JOUgvm2OSwXOpDLFPNtbeN6POrs=', 'Content-Length': '0'} 2022-09-02T05:59:50.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507ffc34430f1dd8-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.594 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.595 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.596 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.597 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.598 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.599 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.600 DEBUG:boto:Data: 2022-09-02T05:59:50.601 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.602 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.603 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.604 DEBUG:boto:Params: {} 2022-09-02T05:59:50.605 DEBUG:boto:Token: None 2022-09-02T05:59:50.606 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.607 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:JOUgvm2OSwXOpDLFPNtbeN6POrs= 2022-09-02T05:59:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:JOUgvm2OSwXOpDLFPNtbeN6POrs=', 'Content-Length': '0'} 2022-09-02T05:59:50.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7488f495c4f750f-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:50.613 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.614 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.615 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.616 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.618 DEBUG:boto:Method: GET 2022-09-02T05:59:50.619 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.620 DEBUG:boto:Data: 2022-09-02T05:59:50.621 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.623 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.624 DEBUG:boto:Params: {} 2022-09-02T05:59:50.625 DEBUG:boto:Token: None 2022-09-02T05:59:50.626 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.627 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE= 2022-09-02T05:59:50.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE=', 'Content-Length': '0'} 2022-09-02T05:59:50.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff689fc35e031121-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.632 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.633 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.633 DEBUG:boto:path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.634 DEBUG:boto:auth_path=/xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.635 DEBUG:boto:Method: GET 2022-09-02T05:59:50.635 DEBUG:boto:Path: /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.636 DEBUG:boto:Data: 2022-09-02T05:59:50.636 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.637 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.637 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.638 DEBUG:boto:Params: {} 2022-09-02T05:59:50.639 DEBUG:boto:Token: None 2022-09-02T05:59:50.639 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74?versionId=null 2022-09-02T05:59:50.640 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE= 2022-09-02T05:59:50.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:FddgJE7vvq4CRx8XRm7LT+DEkbE=', 'Content-Length': '0'} 2022-09-02T05:59:50.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094eb5383f7c51473-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.644 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.644 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.645 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.645 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.646 DEBUG:boto:Method: GET 2022-09-02T05:59:50.646 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.647 DEBUG:boto:Data: 2022-09-02T05:59:50.647 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.648 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.649 DEBUG:boto:Params: {} 2022-09-02T05:59:50.649 DEBUG:boto:Token: None 2022-09-02T05:59:50.650 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.651 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3eec1210fda6be1-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.655 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.656 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.656 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.657 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.657 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.658 DEBUG:boto:Method: GET 2022-09-02T05:59:50.658 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.659 DEBUG:boto:Data: 2022-09-02T05:59:50.660 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.660 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.661 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.661 DEBUG:boto:Params: {} 2022-09-02T05:59:50.662 DEBUG:boto:Token: None 2022-09-02T05:59:50.663 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.663 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051fe332747a95a9e-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.714 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.715 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.716 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.716 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.717 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:50.717 DEBUG:boto:Data: 2022-09-02T05:59:50.718 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.719 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.719 DEBUG:boto:Params: {} 2022-09-02T05:59:50.720 DEBUG:boto:Token: None 2022-09-02T05:59:50.721 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74 2022-09-02T05:59:50.721 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4ZsUfpoVZa6hnqEdYgdVq27BaiU= 2022-09-02T05:59:50.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4ZsUfpoVZa6hnqEdYgdVq27BaiU=', 'Content-Length': '0'} 2022-09-02T05:59:50.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c1dfdc959adef4-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.725 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.726 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.726 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.727 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:50.728 DEBUG:boto:Data: 2022-09-02T05:59:50.729 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.730 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.730 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.731 DEBUG:boto:Params: {} 2022-09-02T05:59:50.732 DEBUG:boto:Token: None 2022-09-02T05:59:50.732 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74 2022-09-02T05:59:50.733 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:4ZsUfpoVZa6hnqEdYgdVq27BaiU= 2022-09-02T05:59:50.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:4ZsUfpoVZa6hnqEdYgdVq27BaiU=', 'Content-Length': '0'} 2022-09-02T05:59:50.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4ea4f95928bfd9-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2022-09-02T05:59:50.737 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.738 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.738 DEBUG:boto:Method: GET 2022-09-02T05:59:50.739 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:50.739 DEBUG:boto:Data: 2022-09-02T05:59:50.740 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.741 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.742 DEBUG:boto:Params: {} 2022-09-02T05:59:50.743 DEBUG:boto:Token: None 2022-09-02T05:59:50.743 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74 2022-09-02T05:59:50.744 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K6CyQRmGrcvfL7qJ5a6sOvA/ENc= 2022-09-02T05:59:50.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K6CyQRmGrcvfL7qJ5a6sOvA/ENc=', 'Content-Length': '0'} 2022-09-02T05:59:50.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f39cf7dfbea2ee-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.748 DEBUG:boto:path=/xljbkf-74/obj74 2022-09-02T05:59:50.748 DEBUG:boto:auth_path=/xljbkf-74/obj74 2022-09-02T05:59:50.749 DEBUG:boto:Method: GET 2022-09-02T05:59:50.749 DEBUG:boto:Path: /xljbkf-74/obj74 2022-09-02T05:59:50.750 DEBUG:boto:Data: 2022-09-02T05:59:50.751 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.751 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.752 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.752 DEBUG:boto:Params: {} 2022-09-02T05:59:50.753 DEBUG:boto:Token: None 2022-09-02T05:59:50.753 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj74 2022-09-02T05:59:50.754 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:K6CyQRmGrcvfL7qJ5a6sOvA/ENc= 2022-09-02T05:59:50.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:K6CyQRmGrcvfL7qJ5a6sOvA/ENc=', 'Content-Length': '0'} 2022-09-02T05:59:50.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:58:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a472e3da53e8b88-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.758 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.758 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.759 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.760 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.760 DEBUG:boto:Method: GET 2022-09-02T05:59:50.761 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.761 DEBUG:boto:Data: 2022-09-02T05:59:50.762 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.763 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.763 DEBUG:boto:Params: {} 2022-09-02T05:59:50.764 DEBUG:boto:Token: None 2022-09-02T05:59:50.764 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.765 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe6ae680bede7b64-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.819 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.821 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.822 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.823 DEBUG:boto:path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.824 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.825 DEBUG:boto:Method: GET 2022-09-02T05:59:50.825 DEBUG:boto:Path: /xljbkf-74/?prefix=obj74 2022-09-02T05:59:50.826 DEBUG:boto:Data: 2022-09-02T05:59:50.827 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.828 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.829 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.829 DEBUG:boto:Params: {} 2022-09-02T05:59:50.830 DEBUG:boto:Token: None 2022-09-02T05:59:50.831 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.832 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000796eea26dbe6be8-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.837 DEBUG:boto:b'xljbkf-74obj741000falseobj742022-09-02T05:58:13.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:50.840 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:50.841 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:50.842 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.843 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.844 DEBUG:boto:Method: GET 2022-09-02T05:59:50.845 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.845 DEBUG:boto:Data: 2022-09-02T05:59:50.846 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.847 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.848 DEBUG:boto:Params: {} 2022-09-02T05:59:50.848 DEBUG:boto:Token: None 2022-09-02T05:59:50.849 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.850 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck= 2022-09-02T05:59:50.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck=', 'Content-Length': '0'} 2022-09-02T05:59:50.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487d03e56d22e1ca-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.853 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:50.854 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:50.855 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.855 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.856 DEBUG:boto:Method: GET 2022-09-02T05:59:50.857 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.857 DEBUG:boto:Data: 2022-09-02T05:59:50.858 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.859 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.859 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.860 DEBUG:boto:Params: {} 2022-09-02T05:59:50.861 DEBUG:boto:Token: None 2022-09-02T05:59:50.862 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.863 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck= 2022-09-02T05:59:50.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck=', 'Content-Length': '0'} 2022-09-02T05:59:50.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000264c7cc935f797ae-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.867 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:50.867 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:50.868 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.868 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.869 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.871 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.872 DEBUG:boto:Data: 2022-09-02T05:59:50.873 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.875 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.876 DEBUG:boto:Params: {} 2022-09-02T05:59:50.877 DEBUG:boto:Token: None 2022-09-02T05:59:50.878 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.880 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ICzE/1U3kGPSFaaKHe/GAonXOq8= 2022-09-02T05:59:50.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ICzE/1U3kGPSFaaKHe/GAonXOq8=', 'Content-Length': '0'} 2022-09-02T05:59:50.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a4eef68ff6fbc5-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.885 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:50.886 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:50.886 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.887 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.888 DEBUG:boto:Method: HEAD 2022-09-02T05:59:50.889 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.890 DEBUG:boto:Data: 2022-09-02T05:59:50.890 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.891 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.892 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.893 DEBUG:boto:Params: {} 2022-09-02T05:59:50.894 DEBUG:boto:Token: None 2022-09-02T05:59:50.895 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.896 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ICzE/1U3kGPSFaaKHe/GAonXOq8= 2022-09-02T05:59:50.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ICzE/1U3kGPSFaaKHe/GAonXOq8=', 'Content-Length': '0'} 2022-09-02T05:59:50.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e7c8ac08b2c964-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:50.900 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:50.901 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:50.901 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.902 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.903 DEBUG:boto:Method: GET 2022-09-02T05:59:50.903 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.904 DEBUG:boto:Data: 2022-09-02T05:59:50.905 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.906 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.907 DEBUG:boto:Params: {} 2022-09-02T05:59:50.907 DEBUG:boto:Token: None 2022-09-02T05:59:50.908 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.909 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck= 2022-09-02T05:59:50.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck=', 'Content-Length': '0'} 2022-09-02T05:59:50.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5e4d0868adee58-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.913 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:50.913 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:50.914 DEBUG:boto:path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.915 DEBUG:boto:auth_path=/xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.915 DEBUG:boto:Method: GET 2022-09-02T05:59:50.916 DEBUG:boto:Path: /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.916 DEBUG:boto:Data: 2022-09-02T05:59:50.917 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.917 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.918 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.918 DEBUG:boto:Params: {} 2022-09-02T05:59:50.919 DEBUG:boto:Token: None 2022-09-02T05:59:50.919 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/obj76?versionId=null 2022-09-02T05:59:50.920 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck= 2022-09-02T05:59:50.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oMpjlO5tjyWrVR2e7CmlMJKp7ck=', 'Content-Length': '0'} 2022-09-02T05:59:50.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155a151444d3ecfa-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.971 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.972 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.973 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:50.974 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:50.975 DEBUG:boto:Method: GET 2022-09-02T05:59:50.975 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:50.976 DEBUG:boto:Data: 2022-09-02T05:59:50.976 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:50.977 DEBUG:boto:Port: 8001 2022-09-02T05:59:50.978 DEBUG:boto:Params: {} 2022-09-02T05:59:50.979 DEBUG:boto:Token: None 2022-09-02T05:59:50.980 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.981 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:50.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005539668a8211e18c-0063119bd6-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:50.985 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:50.986 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:50.987 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:50.987 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:50.988 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:50.988 DEBUG:boto:Method: GET 2022-09-02T05:59:50.989 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:50.989 DEBUG:boto:Data: 2022-09-02T05:59:50.990 DEBUG:boto:Headers: {} 2022-09-02T05:59:50.991 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:50.991 DEBUG:boto:Port: 8000 2022-09-02T05:59:50.992 DEBUG:boto:Params: {} 2022-09-02T05:59:50.993 DEBUG:boto:Token: None 2022-09-02T05:59:50.995 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:50 GMT /xljbkf-74/ 2022-09-02T05:59:50.996 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E= 2022-09-02T05:59:50.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:87u/+Ia+2DdDceAIyzy8HQhsG2E=', 'Content-Length': '0'} 2022-09-02T05:59:51.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4b4aae12fad903b-0063119bd6-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.001 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.003 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:51.004 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:51.005 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.006 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:51.007 DEBUG:boto:Data: 2022-09-02T05:59:51.008 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.010 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.011 DEBUG:boto:Params: {} 2022-09-02T05:59:51.012 DEBUG:boto:Token: None 2022-09-02T05:59:51.013 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj76 2022-09-02T05:59:51.014 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:upvHy5CZ3sKKfJTzulZNGlwojoU= 2022-09-02T05:59:51.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:upvHy5CZ3sKKfJTzulZNGlwojoU=', 'Content-Length': '0'} 2022-09-02T05:59:51.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002652e3e4fe94b9cd-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.019 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:51.020 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:51.021 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.021 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:51.022 DEBUG:boto:Data: 2022-09-02T05:59:51.022 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.023 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.023 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.024 DEBUG:boto:Params: {} 2022-09-02T05:59:51.024 DEBUG:boto:Token: None 2022-09-02T05:59:51.025 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj76 2022-09-02T05:59:51.026 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:upvHy5CZ3sKKfJTzulZNGlwojoU= 2022-09-02T05:59:51.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:upvHy5CZ3sKKfJTzulZNGlwojoU=', 'Content-Length': '0'} 2022-09-02T05:59:51.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8bc29ac8fbb1239-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2022-09-02T05:59:51.031 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:51.031 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:51.032 DEBUG:boto:Method: GET 2022-09-02T05:59:51.032 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:51.033 DEBUG:boto:Data: 2022-09-02T05:59:51.034 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.035 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.035 DEBUG:boto:Params: {} 2022-09-02T05:59:51.036 DEBUG:boto:Token: None 2022-09-02T05:59:51.037 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj76 2022-09-02T05:59:51.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QM82svSO1sGp3CtKebZh2HE6SH0= 2022-09-02T05:59:51.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QM82svSO1sGp3CtKebZh2HE6SH0=', 'Content-Length': '0'} 2022-09-02T05:59:51.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130d4cae5015cb22-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.042 DEBUG:boto:path=/xljbkf-74/obj76 2022-09-02T05:59:51.043 DEBUG:boto:auth_path=/xljbkf-74/obj76 2022-09-02T05:59:51.043 DEBUG:boto:Method: GET 2022-09-02T05:59:51.044 DEBUG:boto:Path: /xljbkf-74/obj76 2022-09-02T05:59:51.044 DEBUG:boto:Data: 2022-09-02T05:59:51.045 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.046 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.047 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.047 DEBUG:boto:Params: {} 2022-09-02T05:59:51.049 DEBUG:boto:Token: None 2022-09-02T05:59:51.049 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj76 2022-09-02T05:59:51.050 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QM82svSO1sGp3CtKebZh2HE6SH0= 2022-09-02T05:59:51.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QM82svSO1sGp3CtKebZh2HE6SH0=', 'Content-Length': '0'} 2022-09-02T05:59:51.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e98f49a9d636d6f-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.054 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.054 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.055 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:51.055 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:51.056 DEBUG:boto:Method: GET 2022-09-02T05:59:51.056 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:51.057 DEBUG:boto:Data: 2022-09-02T05:59:51.058 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.059 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.060 DEBUG:boto:Params: {} 2022-09-02T05:59:51.061 DEBUG:boto:Token: None 2022-09-02T05:59:51.061 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.062 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068065f395493646b-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.066 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.067 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.068 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.068 DEBUG:boto:path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:51.069 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj76 2022-09-02T05:59:51.070 DEBUG:boto:Method: GET 2022-09-02T05:59:51.070 DEBUG:boto:Path: /xljbkf-74/?prefix=obj76 2022-09-02T05:59:51.071 DEBUG:boto:Data: 2022-09-02T05:59:51.072 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.072 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.073 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.073 DEBUG:boto:Params: {} 2022-09-02T05:59:51.074 DEBUG:boto:Token: None 2022-09-02T05:59:51.075 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.075 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a4ece790e9a98a-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.081 DEBUG:boto:b'xljbkf-74obj761000falseobj762022-09-02T05:47:57.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:51.082 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.083 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.084 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.084 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.085 DEBUG:boto:Method: GET 2022-09-02T05:59:51.086 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.087 DEBUG:boto:Data: 2022-09-02T05:59:51.087 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.088 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.089 DEBUG:boto:Params: {} 2022-09-02T05:59:51.089 DEBUG:boto:Token: None 2022-09-02T05:59:51.090 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.091 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4= 2022-09-02T05:59:51.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4=', 'Content-Length': '0'} 2022-09-02T05:59:51.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fddfc1f4e2f2465d-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.095 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.096 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.096 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.097 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.097 DEBUG:boto:Method: GET 2022-09-02T05:59:51.098 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.098 DEBUG:boto:Data: 2022-09-02T05:59:51.099 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.100 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.100 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.101 DEBUG:boto:Params: {} 2022-09-02T05:59:51.101 DEBUG:boto:Token: None 2022-09-02T05:59:51.102 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.103 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4= 2022-09-02T05:59:51.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4=', 'Content-Length': '0'} 2022-09-02T05:59:51.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ad56b10b7b0fe1-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.107 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.108 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.109 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.110 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.111 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.112 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.113 DEBUG:boto:Data: 2022-09-02T05:59:51.114 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.116 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.117 DEBUG:boto:Params: {} 2022-09-02T05:59:51.118 DEBUG:boto:Token: None 2022-09-02T05:59:51.119 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.120 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0jHdx7gWJR9flZzEH15F7Atf/Ds= 2022-09-02T05:59:51.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0jHdx7gWJR9flZzEH15F7Atf/Ds=', 'Content-Length': '0'} 2022-09-02T05:59:51.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ed277945c2fd00-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.125 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.126 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.127 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.128 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.129 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.130 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.131 DEBUG:boto:Data: 2022-09-02T05:59:51.132 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.133 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.134 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.134 DEBUG:boto:Params: {} 2022-09-02T05:59:51.135 DEBUG:boto:Token: None 2022-09-02T05:59:51.136 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.136 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0jHdx7gWJR9flZzEH15F7Atf/Ds= 2022-09-02T05:59:51.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0jHdx7gWJR9flZzEH15F7Atf/Ds=', 'Content-Length': '0'} 2022-09-02T05:59:51.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f105e38faf29ff57-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:51.140 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.141 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.142 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.142 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.143 DEBUG:boto:Method: GET 2022-09-02T05:59:51.143 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.144 DEBUG:boto:Data: 2022-09-02T05:59:51.144 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.145 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.146 DEBUG:boto:Params: {} 2022-09-02T05:59:51.147 DEBUG:boto:Token: None 2022-09-02T05:59:51.147 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.148 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4= 2022-09-02T05:59:51.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4=', 'Content-Length': '0'} 2022-09-02T05:59:51.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc395d0f0350aa8-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.151 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.152 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.153 DEBUG:boto:path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.153 DEBUG:boto:auth_path=/xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.154 DEBUG:boto:Method: GET 2022-09-02T05:59:51.154 DEBUG:boto:Path: /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.155 DEBUG:boto:Data: 2022-09-02T05:59:51.155 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.156 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.156 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.157 DEBUG:boto:Params: {} 2022-09-02T05:59:51.158 DEBUG:boto:Token: None 2022-09-02T05:59:51.158 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85?versionId=null 2022-09-02T05:59:51.159 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4= 2022-09-02T05:59:51.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jNSL+UxyulXWCyv+dBWN9EAW5W4=', 'Content-Length': '0'} 2022-09-02T05:59:51.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005916dcb9f3905dbe-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.202 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.203 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.204 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.205 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.205 DEBUG:boto:Method: GET 2022-09-02T05:59:51.206 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.206 DEBUG:boto:Data: 2022-09-02T05:59:51.207 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.209 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.210 DEBUG:boto:Params: {} 2022-09-02T05:59:51.211 DEBUG:boto:Token: None 2022-09-02T05:59:51.212 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.213 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d4557a6d22322d-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.219 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.220 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.221 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.222 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.223 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.223 DEBUG:boto:Method: GET 2022-09-02T05:59:51.224 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.224 DEBUG:boto:Data: 2022-09-02T05:59:51.225 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.226 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.226 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.227 DEBUG:boto:Params: {} 2022-09-02T05:59:51.227 DEBUG:boto:Token: None 2022-09-02T05:59:51.228 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.229 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008ef2a45f7c4550c-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.234 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.235 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.235 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.236 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.237 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:51.237 DEBUG:boto:Data: 2022-09-02T05:59:51.238 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.240 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.241 DEBUG:boto:Params: {} 2022-09-02T05:59:51.241 DEBUG:boto:Token: None 2022-09-02T05:59:51.242 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85 2022-09-02T05:59:51.243 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiSy5Z+k6uzpwKwk0DwvMpQnlkU= 2022-09-02T05:59:51.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiSy5Z+k6uzpwKwk0DwvMpQnlkU=', 'Content-Length': '0'} 2022-09-02T05:59:51.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7163ac4ac8242e0-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.246 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.247 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.248 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.248 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:51.249 DEBUG:boto:Data: 2022-09-02T05:59:51.249 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.250 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.251 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.251 DEBUG:boto:Params: {} 2022-09-02T05:59:51.252 DEBUG:boto:Token: None 2022-09-02T05:59:51.252 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85 2022-09-02T05:59:51.253 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:eiSy5Z+k6uzpwKwk0DwvMpQnlkU= 2022-09-02T05:59:51.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:eiSy5Z+k6uzpwKwk0DwvMpQnlkU=', 'Content-Length': '0'} 2022-09-02T05:59:51.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142e84b9a35da078-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2022-09-02T05:59:51.257 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.258 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.259 DEBUG:boto:Method: GET 2022-09-02T05:59:51.259 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:51.260 DEBUG:boto:Data: 2022-09-02T05:59:51.261 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.262 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.262 DEBUG:boto:Params: {} 2022-09-02T05:59:51.263 DEBUG:boto:Token: None 2022-09-02T05:59:51.264 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85 2022-09-02T05:59:51.264 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WnUCVaPBguIgGlQtsQMP0DNbGhA= 2022-09-02T05:59:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WnUCVaPBguIgGlQtsQMP0DNbGhA=', 'Content-Length': '0'} 2022-09-02T05:59:51.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c6769fe21589b5-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.269 DEBUG:boto:path=/xljbkf-74/obj85 2022-09-02T05:59:51.269 DEBUG:boto:auth_path=/xljbkf-74/obj85 2022-09-02T05:59:51.270 DEBUG:boto:Method: GET 2022-09-02T05:59:51.271 DEBUG:boto:Path: /xljbkf-74/obj85 2022-09-02T05:59:51.272 DEBUG:boto:Data: 2022-09-02T05:59:51.272 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.273 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.273 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.274 DEBUG:boto:Params: {} 2022-09-02T05:59:51.275 DEBUG:boto:Token: None 2022-09-02T05:59:51.276 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj85 2022-09-02T05:59:51.276 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:WnUCVaPBguIgGlQtsQMP0DNbGhA= 2022-09-02T05:59:51.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:WnUCVaPBguIgGlQtsQMP0DNbGhA=', 'Content-Length': '0'} 2022-09-02T05:59:51.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:45:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca9ec58dd108dd6-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.280 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.281 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.281 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.282 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.283 DEBUG:boto:Method: GET 2022-09-02T05:59:51.283 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.284 DEBUG:boto:Data: 2022-09-02T05:59:51.284 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.286 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.287 DEBUG:boto:Params: {} 2022-09-02T05:59:51.288 DEBUG:boto:Token: None 2022-09-02T05:59:51.289 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.290 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36cb8c91fff9e0b-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.295 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.296 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.297 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.298 DEBUG:boto:path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.298 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.299 DEBUG:boto:Method: GET 2022-09-02T05:59:51.300 DEBUG:boto:Path: /xljbkf-74/?prefix=obj85 2022-09-02T05:59:51.301 DEBUG:boto:Data: 2022-09-02T05:59:51.301 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.302 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.302 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.303 DEBUG:boto:Params: {} 2022-09-02T05:59:51.304 DEBUG:boto:Token: None 2022-09-02T05:59:51.304 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.305 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dbebafb52d1716b-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.309 DEBUG:boto:b'xljbkf-74obj851000falseobj852022-09-02T05:45:27.918Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:51.311 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.312 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.313 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.313 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.314 DEBUG:boto:Method: GET 2022-09-02T05:59:51.314 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.315 DEBUG:boto:Data: 2022-09-02T05:59:51.315 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.317 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.317 DEBUG:boto:Params: {} 2022-09-02T05:59:51.318 DEBUG:boto:Token: None 2022-09-02T05:59:51.318 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.319 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU= 2022-09-02T05:59:51.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU=', 'Content-Length': '0'} 2022-09-02T05:59:51.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3b452bd38938f4-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.323 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.323 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.324 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.324 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.325 DEBUG:boto:Method: GET 2022-09-02T05:59:51.326 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.326 DEBUG:boto:Data: 2022-09-02T05:59:51.327 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.328 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.328 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.329 DEBUG:boto:Params: {} 2022-09-02T05:59:51.329 DEBUG:boto:Token: None 2022-09-02T05:59:51.330 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.331 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU= 2022-09-02T05:59:51.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU=', 'Content-Length': '0'} 2022-09-02T05:59:51.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d42513c24bf39b7-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.335 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.336 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.336 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.337 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.337 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.338 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.338 DEBUG:boto:Data: 2022-09-02T05:59:51.339 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.340 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.340 DEBUG:boto:Params: {} 2022-09-02T05:59:51.341 DEBUG:boto:Token: None 2022-09-02T05:59:51.342 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.343 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q7AdOrCM9/ozYFqtZpY+wenDalA= 2022-09-02T05:59:51.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q7AdOrCM9/ozYFqtZpY+wenDalA=', 'Content-Length': '0'} 2022-09-02T05:59:51.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e14a231c10b5cd-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.346 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.347 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.348 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.348 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.349 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.349 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.350 DEBUG:boto:Data: 2022-09-02T05:59:51.351 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.351 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.352 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.352 DEBUG:boto:Params: {} 2022-09-02T05:59:51.353 DEBUG:boto:Token: None 2022-09-02T05:59:51.353 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.354 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Q7AdOrCM9/ozYFqtZpY+wenDalA= 2022-09-02T05:59:51.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Q7AdOrCM9/ozYFqtZpY+wenDalA=', 'Content-Length': '0'} 2022-09-02T05:59:51.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9c385ed00106d5d-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:51.358 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.359 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.359 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.360 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.360 DEBUG:boto:Method: GET 2022-09-02T05:59:51.361 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.361 DEBUG:boto:Data: 2022-09-02T05:59:51.362 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.363 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.364 DEBUG:boto:Params: {} 2022-09-02T05:59:51.364 DEBUG:boto:Token: None 2022-09-02T05:59:51.365 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.366 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU= 2022-09-02T05:59:51.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU=', 'Content-Length': '0'} 2022-09-02T05:59:51.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5d538cc8bd179e-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.414 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.415 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.416 DEBUG:boto:path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.417 DEBUG:boto:auth_path=/xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.418 DEBUG:boto:Method: GET 2022-09-02T05:59:51.420 DEBUG:boto:Path: /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.421 DEBUG:boto:Data: 2022-09-02T05:59:51.422 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.423 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.424 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.425 DEBUG:boto:Params: {} 2022-09-02T05:59:51.426 DEBUG:boto:Token: None 2022-09-02T05:59:51.427 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87?versionId=null 2022-09-02T05:59:51.428 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU= 2022-09-02T05:59:51.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:oslzM8F7O04aSEvqzBej2QQLYrU=', 'Content-Length': '0'} 2022-09-02T05:59:51.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025adae5c446e2489-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.434 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.435 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.436 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.437 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.438 DEBUG:boto:Method: GET 2022-09-02T05:59:51.439 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.440 DEBUG:boto:Data: 2022-09-02T05:59:51.441 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.443 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.444 DEBUG:boto:Params: {} 2022-09-02T05:59:51.445 DEBUG:boto:Token: None 2022-09-02T05:59:51.446 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.447 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000915b5f83c64cca7-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.453 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.454 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.455 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.456 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.457 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.458 DEBUG:boto:Method: GET 2022-09-02T05:59:51.459 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.460 DEBUG:boto:Data: 2022-09-02T05:59:51.461 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.461 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.462 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.462 DEBUG:boto:Params: {} 2022-09-02T05:59:51.463 DEBUG:boto:Token: None 2022-09-02T05:59:51.464 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.464 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a053eee14b846b9-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.468 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.469 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.470 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.470 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.471 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:51.471 DEBUG:boto:Data: 2022-09-02T05:59:51.472 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.473 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.473 DEBUG:boto:Params: {} 2022-09-02T05:59:51.474 DEBUG:boto:Token: None 2022-09-02T05:59:51.474 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87 2022-09-02T05:59:51.475 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qT0fZbnw07CAs1t5UxN9/hMGAlU= 2022-09-02T05:59:51.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qT0fZbnw07CAs1t5UxN9/hMGAlU=', 'Content-Length': '0'} 2022-09-02T05:59:51.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000277bfcb7dc299f56-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.478 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.479 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.480 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.480 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:51.481 DEBUG:boto:Data: 2022-09-02T05:59:51.481 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.482 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.482 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.483 DEBUG:boto:Params: {} 2022-09-02T05:59:51.483 DEBUG:boto:Token: None 2022-09-02T05:59:51.484 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87 2022-09-02T05:59:51.484 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:qT0fZbnw07CAs1t5UxN9/hMGAlU= 2022-09-02T05:59:51.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:qT0fZbnw07CAs1t5UxN9/hMGAlU=', 'Content-Length': '0'} 2022-09-02T05:59:51.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739441cac687a47a-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2022-09-02T05:59:51.488 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.489 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.489 DEBUG:boto:Method: GET 2022-09-02T05:59:51.490 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:51.490 DEBUG:boto:Data: 2022-09-02T05:59:51.491 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.492 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.493 DEBUG:boto:Params: {} 2022-09-02T05:59:51.493 DEBUG:boto:Token: None 2022-09-02T05:59:51.494 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87 2022-09-02T05:59:51.495 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zw1JvWojd7R4deCb3KegHxaEjoo= 2022-09-02T05:59:51.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zw1JvWojd7R4deCb3KegHxaEjoo=', 'Content-Length': '0'} 2022-09-02T05:59:51.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000997c69d80822896a-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.498 DEBUG:boto:path=/xljbkf-74/obj87 2022-09-02T05:59:51.499 DEBUG:boto:auth_path=/xljbkf-74/obj87 2022-09-02T05:59:51.499 DEBUG:boto:Method: GET 2022-09-02T05:59:51.500 DEBUG:boto:Path: /xljbkf-74/obj87 2022-09-02T05:59:51.501 DEBUG:boto:Data: 2022-09-02T05:59:51.501 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.502 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.503 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.503 DEBUG:boto:Params: {} 2022-09-02T05:59:51.504 DEBUG:boto:Token: None 2022-09-02T05:59:51.505 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj87 2022-09-02T05:59:51.506 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:zw1JvWojd7R4deCb3KegHxaEjoo= 2022-09-02T05:59:51.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:zw1JvWojd7R4deCb3KegHxaEjoo=', 'Content-Length': '0'} 2022-09-02T05:59:51.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:53:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000189f987d492aa7af-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.550 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.551 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.552 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.552 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.553 DEBUG:boto:Method: GET 2022-09-02T05:59:51.553 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.554 DEBUG:boto:Data: 2022-09-02T05:59:51.554 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.562 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.563 DEBUG:boto:Params: {} 2022-09-02T05:59:51.563 DEBUG:boto:Token: None 2022-09-02T05:59:51.564 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.565 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a7bed8bf263a80f-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.570 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.571 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.571 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.572 DEBUG:boto:path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.573 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.573 DEBUG:boto:Method: GET 2022-09-02T05:59:51.574 DEBUG:boto:Path: /xljbkf-74/?prefix=obj87 2022-09-02T05:59:51.575 DEBUG:boto:Data: 2022-09-02T05:59:51.576 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.576 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.577 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.577 DEBUG:boto:Params: {} 2022-09-02T05:59:51.579 DEBUG:boto:Token: None 2022-09-02T05:59:51.579 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.580 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fef9d6958d1e5ff-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.586 DEBUG:boto:b'xljbkf-74obj871000falseobj872022-09-02T05:53:22.479Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:51.587 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.588 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.589 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.590 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.590 DEBUG:boto:Method: GET 2022-09-02T05:59:51.591 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.592 DEBUG:boto:Data: 2022-09-02T05:59:51.593 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.595 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.596 DEBUG:boto:Params: {} 2022-09-02T05:59:51.597 DEBUG:boto:Token: None 2022-09-02T05:59:51.599 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.600 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28= 2022-09-02T05:59:51.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28=', 'Content-Length': '0'} 2022-09-02T05:59:51.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9a0382c851bff7-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.607 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.609 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.609 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.610 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.611 DEBUG:boto:Method: GET 2022-09-02T05:59:51.612 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.613 DEBUG:boto:Data: 2022-09-02T05:59:51.614 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.615 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.615 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.616 DEBUG:boto:Params: {} 2022-09-02T05:59:51.617 DEBUG:boto:Token: None 2022-09-02T05:59:51.618 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.619 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28= 2022-09-02T05:59:51.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28=', 'Content-Length': '0'} 2022-09-02T05:59:51.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edd4df852ded7188-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.623 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.624 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.624 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.625 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.626 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.627 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.627 DEBUG:boto:Data: 2022-09-02T05:59:51.628 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.629 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.630 DEBUG:boto:Params: {} 2022-09-02T05:59:51.631 DEBUG:boto:Token: None 2022-09-02T05:59:51.632 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.633 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TLASoeUtT6S07xz6Vzz81b2acc0= 2022-09-02T05:59:51.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TLASoeUtT6S07xz6Vzz81b2acc0=', 'Content-Length': '0'} 2022-09-02T05:59:51.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087bb7a252bb49476-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.637 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.637 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.638 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.638 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.639 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.639 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.640 DEBUG:boto:Data: 2022-09-02T05:59:51.640 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.641 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.641 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.642 DEBUG:boto:Params: {} 2022-09-02T05:59:51.642 DEBUG:boto:Token: None 2022-09-02T05:59:51.643 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.644 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:TLASoeUtT6S07xz6Vzz81b2acc0= 2022-09-02T05:59:51.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:TLASoeUtT6S07xz6Vzz81b2acc0=', 'Content-Length': '0'} 2022-09-02T05:59:51.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084073d98de9f2a8c-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:51.648 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.649 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.650 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.651 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.652 DEBUG:boto:Method: GET 2022-09-02T05:59:51.653 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.654 DEBUG:boto:Data: 2022-09-02T05:59:51.654 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.656 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.657 DEBUG:boto:Params: {} 2022-09-02T05:59:51.658 DEBUG:boto:Token: None 2022-09-02T05:59:51.660 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.661 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28= 2022-09-02T05:59:51.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28=', 'Content-Length': '0'} 2022-09-02T05:59:51.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000781b83274c12e35c-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.665 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.666 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.666 DEBUG:boto:path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.667 DEBUG:boto:auth_path=/xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.667 DEBUG:boto:Method: GET 2022-09-02T05:59:51.668 DEBUG:boto:Path: /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.669 DEBUG:boto:Data: 2022-09-02T05:59:51.669 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.670 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.670 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.671 DEBUG:boto:Params: {} 2022-09-02T05:59:51.671 DEBUG:boto:Token: None 2022-09-02T05:59:51.672 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92?versionId=null 2022-09-02T05:59:51.673 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28= 2022-09-02T05:59:51.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:0P4CxlA9gXBTYfdGmJY0lWw8y28=', 'Content-Length': '0'} 2022-09-02T05:59:51.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b177c923989053a-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.722 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.723 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.724 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.725 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.726 DEBUG:boto:Method: GET 2022-09-02T05:59:51.726 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.727 DEBUG:boto:Data: 2022-09-02T05:59:51.728 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.729 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.729 DEBUG:boto:Params: {} 2022-09-02T05:59:51.730 DEBUG:boto:Token: None 2022-09-02T05:59:51.731 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.732 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db52226f6d60742f-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.737 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.738 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.739 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.740 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.741 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.741 DEBUG:boto:Method: GET 2022-09-02T05:59:51.742 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.742 DEBUG:boto:Data: 2022-09-02T05:59:51.743 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.744 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.745 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.746 DEBUG:boto:Params: {} 2022-09-02T05:59:51.747 DEBUG:boto:Token: None 2022-09-02T05:59:51.748 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.749 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f355fb9ad69ed78a-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.754 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.756 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.756 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.757 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.757 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:51.758 DEBUG:boto:Data: 2022-09-02T05:59:51.759 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.760 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.760 DEBUG:boto:Params: {} 2022-09-02T05:59:51.761 DEBUG:boto:Token: None 2022-09-02T05:59:51.762 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92 2022-09-02T05:59:51.763 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ozie5I9vZZUE/iSj6IM5zbhqcX8= 2022-09-02T05:59:51.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ozie5I9vZZUE/iSj6IM5zbhqcX8=', 'Content-Length': '0'} 2022-09-02T05:59:51.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000612c8b696238b344-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.767 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.767 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.768 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.768 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:51.769 DEBUG:boto:Data: 2022-09-02T05:59:51.769 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.770 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.770 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.771 DEBUG:boto:Params: {} 2022-09-02T05:59:51.772 DEBUG:boto:Token: None 2022-09-02T05:59:51.772 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92 2022-09-02T05:59:51.773 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:ozie5I9vZZUE/iSj6IM5zbhqcX8= 2022-09-02T05:59:51.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:ozie5I9vZZUE/iSj6IM5zbhqcX8=', 'Content-Length': '0'} 2022-09-02T05:59:51.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082b62f2b02cef50c-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2022-09-02T05:59:51.777 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.777 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.778 DEBUG:boto:Method: GET 2022-09-02T05:59:51.779 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:51.779 DEBUG:boto:Data: 2022-09-02T05:59:51.780 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.781 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.781 DEBUG:boto:Params: {} 2022-09-02T05:59:51.782 DEBUG:boto:Token: None 2022-09-02T05:59:51.783 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92 2022-09-02T05:59:51.784 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9gTICNOZFVp4V4Bg9JW8YNw/YKA= 2022-09-02T05:59:51.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9gTICNOZFVp4V4Bg9JW8YNw/YKA=', 'Content-Length': '0'} 2022-09-02T05:59:51.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074182b96ed4c9df9-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.788 DEBUG:boto:path=/xljbkf-74/obj92 2022-09-02T05:59:51.788 DEBUG:boto:auth_path=/xljbkf-74/obj92 2022-09-02T05:59:51.789 DEBUG:boto:Method: GET 2022-09-02T05:59:51.789 DEBUG:boto:Path: /xljbkf-74/obj92 2022-09-02T05:59:51.790 DEBUG:boto:Data: 2022-09-02T05:59:51.791 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.791 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.792 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.792 DEBUG:boto:Params: {} 2022-09-02T05:59:51.793 DEBUG:boto:Token: None 2022-09-02T05:59:51.794 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj92 2022-09-02T05:59:51.794 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:9gTICNOZFVp4V4Bg9JW8YNw/YKA= 2022-09-02T05:59:51.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:9gTICNOZFVp4V4Bg9JW8YNw/YKA=', 'Content-Length': '0'} 2022-09-02T05:59:51.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:50:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a66581ec0ae1529-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.799 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.799 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.800 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.800 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.801 DEBUG:boto:Method: GET 2022-09-02T05:59:51.801 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.802 DEBUG:boto:Data: 2022-09-02T05:59:51.803 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.804 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.804 DEBUG:boto:Params: {} 2022-09-02T05:59:51.805 DEBUG:boto:Token: None 2022-09-02T05:59:51.805 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.806 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d8f1f39c88464b-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.850 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.852 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.853 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.854 DEBUG:boto:path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.855 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.857 DEBUG:boto:Method: GET 2022-09-02T05:59:51.858 DEBUG:boto:Path: /xljbkf-74/?prefix=obj92 2022-09-02T05:59:51.859 DEBUG:boto:Data: 2022-09-02T05:59:51.860 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.861 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.862 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.863 DEBUG:boto:Params: {} 2022-09-02T05:59:51.864 DEBUG:boto:Token: None 2022-09-02T05:59:51.865 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.866 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001948f25aa31afae4-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.872 DEBUG:boto:b'xljbkf-74obj921000falseobj922022-09-02T05:50:15.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:51.873 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:51.874 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:51.874 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.875 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.875 DEBUG:boto:Method: GET 2022-09-02T05:59:51.876 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.876 DEBUG:boto:Data: 2022-09-02T05:59:51.878 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.879 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.880 DEBUG:boto:Params: {} 2022-09-02T05:59:51.880 DEBUG:boto:Token: None 2022-09-02T05:59:51.881 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.881 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais= 2022-09-02T05:59:51.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais=', 'Content-Length': '0'} 2022-09-02T05:59:51.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048db96936d885277-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.886 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:51.887 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:51.888 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.889 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.890 DEBUG:boto:Method: GET 2022-09-02T05:59:51.891 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.892 DEBUG:boto:Data: 2022-09-02T05:59:51.895 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.896 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.897 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.898 DEBUG:boto:Params: {} 2022-09-02T05:59:51.899 DEBUG:boto:Token: None 2022-09-02T05:59:51.900 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.902 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais= 2022-09-02T05:59:51.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais=', 'Content-Length': '0'} 2022-09-02T05:59:51.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c852dd1c4f3cebc-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.910 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:51.911 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:51.912 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.913 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.914 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.915 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.916 DEBUG:boto:Data: 2022-09-02T05:59:51.917 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.919 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.920 DEBUG:boto:Params: {} 2022-09-02T05:59:51.921 DEBUG:boto:Token: None 2022-09-02T05:59:51.921 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.922 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:spPumW4zu/qi2n8xie1+Hnr7sko= 2022-09-02T05:59:51.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:spPumW4zu/qi2n8xie1+Hnr7sko=', 'Content-Length': '0'} 2022-09-02T05:59:51.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e9b005809fc78a9-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.926 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:51.926 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:51.927 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.927 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.928 DEBUG:boto:Method: HEAD 2022-09-02T05:59:51.928 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.929 DEBUG:boto:Data: 2022-09-02T05:59:51.929 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.930 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.931 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.931 DEBUG:boto:Params: {} 2022-09-02T05:59:51.932 DEBUG:boto:Token: None 2022-09-02T05:59:51.932 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.933 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:spPumW4zu/qi2n8xie1+Hnr7sko= 2022-09-02T05:59:51.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:spPumW4zu/qi2n8xie1+Hnr7sko=', 'Content-Length': '0'} 2022-09-02T05:59:51.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f88483cda5c7aa0-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:51.939 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:51.940 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:51.941 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.942 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.943 DEBUG:boto:Method: GET 2022-09-02T05:59:51.944 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.945 DEBUG:boto:Data: 2022-09-02T05:59:51.946 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.948 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.949 DEBUG:boto:Params: {} 2022-09-02T05:59:51.950 DEBUG:boto:Token: None 2022-09-02T05:59:51.951 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.952 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais= 2022-09-02T05:59:51.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais=', 'Content-Length': '0'} 2022-09-02T05:59:51.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ccee82b549e1055-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.957 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:51.958 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:51.959 DEBUG:boto:path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.960 DEBUG:boto:auth_path=/xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.961 DEBUG:boto:Method: GET 2022-09-02T05:59:51.962 DEBUG:boto:Path: /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.963 DEBUG:boto:Data: 2022-09-02T05:59:51.964 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.965 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.966 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.967 DEBUG:boto:Params: {} 2022-09-02T05:59:51.968 DEBUG:boto:Token: None 2022-09-02T05:59:51.970 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/obj93?versionId=null 2022-09-02T05:59:51.971 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais= 2022-09-02T05:59:51.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:fCq5rfRf4CVcfLveLLDPgGlbais=', 'Content-Length': '0'} 2022-09-02T05:59:51.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039a4faa115f193f6-0063119bd7-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.976 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.976 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.977 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:51.977 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:51.978 DEBUG:boto:Method: GET 2022-09-02T05:59:51.978 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:51.979 DEBUG:boto:Data: 2022-09-02T05:59:51.979 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:51.980 DEBUG:boto:Port: 8001 2022-09-02T05:59:51.981 DEBUG:boto:Params: {} 2022-09-02T05:59:51.981 DEBUG:boto:Token: None 2022-09-02T05:59:51.982 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.983 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:51.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf42b2b1aaf313b-0063119bd7-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:51 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:51.990 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:51.991 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:51.991 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:51.992 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:51.993 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:51.993 DEBUG:boto:Method: GET 2022-09-02T05:59:51.994 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:51.994 DEBUG:boto:Data: 2022-09-02T05:59:51.995 DEBUG:boto:Headers: {} 2022-09-02T05:59:51.995 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:51.996 DEBUG:boto:Port: 8000 2022-09-02T05:59:51.996 DEBUG:boto:Params: {} 2022-09-02T05:59:51.997 DEBUG:boto:Token: None 2022-09-02T05:59:51.998 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:51 GMT /xljbkf-74/ 2022-09-02T05:59:51.999 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE= 2022-09-02T05:59:51.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:51 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:mV0ei6IcoqXxkIveJbNQYqET2RE=', 'Content-Length': '0'} 2022-09-02T05:59:52.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000567cb606513350b0-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.008 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.009 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:52.010 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:52.011 DEBUG:boto:Method: HEAD 2022-09-02T05:59:52.012 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:52.013 DEBUG:boto:Data: 2022-09-02T05:59:52.013 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.014 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.015 DEBUG:boto:Params: {} 2022-09-02T05:59:52.015 DEBUG:boto:Token: None 2022-09-02T05:59:52.016 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj93 2022-09-02T05:59:52.017 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QgPfdO9+b5md7x7RA1d6ok+9rXs= 2022-09-02T05:59:52.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QgPfdO9+b5md7x7RA1d6ok+9rXs=', 'Content-Length': '0'} 2022-09-02T05:59:52.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0cb573ca86def85-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.020 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:52.021 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:52.022 DEBUG:boto:Method: HEAD 2022-09-02T05:59:52.022 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:52.023 DEBUG:boto:Data: 2022-09-02T05:59:52.023 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.024 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.024 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.025 DEBUG:boto:Params: {} 2022-09-02T05:59:52.026 DEBUG:boto:Token: None 2022-09-02T05:59:52.026 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj93 2022-09-02T05:59:52.027 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:QgPfdO9+b5md7x7RA1d6ok+9rXs= 2022-09-02T05:59:52.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:QgPfdO9+b5md7x7RA1d6ok+9rXs=', 'Content-Length': '0'} 2022-09-02T05:59:52.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c361b8db7bef5d-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2022-09-02T05:59:52.031 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:52.032 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:52.033 DEBUG:boto:Method: GET 2022-09-02T05:59:52.033 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:52.034 DEBUG:boto:Data: 2022-09-02T05:59:52.035 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.036 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.036 DEBUG:boto:Params: {} 2022-09-02T05:59:52.037 DEBUG:boto:Token: None 2022-09-02T05:59:52.038 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj93 2022-09-02T05:59:52.038 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JRT3WYu9qEPS+/sbN8rD2ZAero= 2022-09-02T05:59:52.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JRT3WYu9qEPS+/sbN8rD2ZAero=', 'Content-Length': '0'} 2022-09-02T05:59:52.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fa92dda595fcb70-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.042 DEBUG:boto:path=/xljbkf-74/obj93 2022-09-02T05:59:52.042 DEBUG:boto:auth_path=/xljbkf-74/obj93 2022-09-02T05:59:52.043 DEBUG:boto:Method: GET 2022-09-02T05:59:52.043 DEBUG:boto:Path: /xljbkf-74/obj93 2022-09-02T05:59:52.044 DEBUG:boto:Data: 2022-09-02T05:59:52.044 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.045 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.045 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.046 DEBUG:boto:Params: {} 2022-09-02T05:59:52.047 DEBUG:boto:Token: None 2022-09-02T05:59:52.047 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj93 2022-09-02T05:59:52.048 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:7JRT3WYu9qEPS+/sbN8rD2ZAero= 2022-09-02T05:59:52.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:7JRT3WYu9qEPS+/sbN8rD2ZAero=', 'Content-Length': '0'} 2022-09-02T05:59:52.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4612180b83b2671-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.096 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:52.097 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:52.097 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:52.098 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:52.098 DEBUG:boto:Method: GET 2022-09-02T05:59:52.099 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:52.099 DEBUG:boto:Data: 2022-09-02T05:59:52.100 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.101 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.102 DEBUG:boto:Params: {} 2022-09-02T05:59:52.102 DEBUG:boto:Token: None 2022-09-02T05:59:52.103 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/ 2022-09-02T05:59:52.104 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw= 2022-09-02T05:59:52.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw=', 'Content-Length': '0'} 2022-09-02T05:59:52.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e27be9aee0699263-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.108 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.109 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:52.110 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:52.110 DEBUG:boto:path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:52.111 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj93 2022-09-02T05:59:52.111 DEBUG:boto:Method: GET 2022-09-02T05:59:52.112 DEBUG:boto:Path: /xljbkf-74/?prefix=obj93 2022-09-02T05:59:52.112 DEBUG:boto:Data: 2022-09-02T05:59:52.113 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.113 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.114 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.115 DEBUG:boto:Params: {} 2022-09-02T05:59:52.115 DEBUG:boto:Token: None 2022-09-02T05:59:52.116 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/ 2022-09-02T05:59:52.117 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw= 2022-09-02T05:59:52.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw=', 'Content-Length': '0'} 2022-09-02T05:59:52.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008089d88eb1abd414-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.122 DEBUG:boto:b'xljbkf-74obj931000falseobj932022-09-02T05:56:04.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:52.124 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.124 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.125 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.125 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.126 DEBUG:boto:Method: GET 2022-09-02T05:59:52.127 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.127 DEBUG:boto:Data: 2022-09-02T05:59:52.128 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.129 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.130 DEBUG:boto:Params: {} 2022-09-02T05:59:52.131 DEBUG:boto:Token: None 2022-09-02T05:59:52.131 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.132 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI= 2022-09-02T05:59:52.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI=', 'Content-Length': '0'} 2022-09-02T05:59:52.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c21cd288dc2be5d-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.178 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.179 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.180 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.182 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.182 DEBUG:boto:Method: GET 2022-09-02T05:59:52.183 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.183 DEBUG:boto:Data: 2022-09-02T05:59:52.184 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.185 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.186 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.187 DEBUG:boto:Params: {} 2022-09-02T05:59:52.187 DEBUG:boto:Token: None 2022-09-02T05:59:52.188 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.189 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI= 2022-09-02T05:59:52.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI=', 'Content-Length': '0'} 2022-09-02T05:59:52.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001356d67086d5e381-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.192 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.193 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.194 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.195 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.196 DEBUG:boto:Method: HEAD 2022-09-02T05:59:52.197 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.198 DEBUG:boto:Data: 2022-09-02T05:59:52.198 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.200 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.201 DEBUG:boto:Params: {} 2022-09-02T05:59:52.202 DEBUG:boto:Token: None 2022-09-02T05:59:52.203 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.204 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SZ1h917dmrN8FCPY4ZKvcbuCryU= 2022-09-02T05:59:52.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SZ1h917dmrN8FCPY4ZKvcbuCryU=', 'Content-Length': '0'} 2022-09-02T05:59:52.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f488d51cb7f571-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.209 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.210 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.211 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.212 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.213 DEBUG:boto:Method: HEAD 2022-09-02T05:59:52.214 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.214 DEBUG:boto:Data: 2022-09-02T05:59:52.215 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.216 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.217 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.218 DEBUG:boto:Params: {} 2022-09-02T05:59:52.219 DEBUG:boto:Token: None 2022-09-02T05:59:52.219 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.220 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:SZ1h917dmrN8FCPY4ZKvcbuCryU= 2022-09-02T05:59:52.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:SZ1h917dmrN8FCPY4ZKvcbuCryU=', 'Content-Length': '0'} 2022-09-02T05:59:52.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae78123ae00c3634-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:52.224 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.225 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.225 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.226 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.226 DEBUG:boto:Method: GET 2022-09-02T05:59:52.227 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.227 DEBUG:boto:Data: 2022-09-02T05:59:52.228 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.229 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.230 DEBUG:boto:Params: {} 2022-09-02T05:59:52.231 DEBUG:boto:Token: None 2022-09-02T05:59:52.231 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.232 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI= 2022-09-02T05:59:52.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI=', 'Content-Length': '0'} 2022-09-02T05:59:52.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e028d675b1a2d08-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.236 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.237 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.237 DEBUG:boto:path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.238 DEBUG:boto:auth_path=/xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.239 DEBUG:boto:Method: GET 2022-09-02T05:59:52.239 DEBUG:boto:Path: /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.240 DEBUG:boto:Data: 2022-09-02T05:59:52.240 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.241 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.241 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.242 DEBUG:boto:Params: {} 2022-09-02T05:59:52.242 DEBUG:boto:Token: None 2022-09-02T05:59:52.243 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95?versionId=null 2022-09-02T05:59:52.244 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI= 2022-09-02T05:59:52.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:38sbXtfWvn4fJLzlRLDd1Wuy/AI=', 'Content-Length': '0'} 2022-09-02T05:59:52.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e000245d52a9d1c8-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.248 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:52.248 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:52.249 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.249 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.250 DEBUG:boto:Method: GET 2022-09-02T05:59:52.250 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.251 DEBUG:boto:Data: 2022-09-02T05:59:52.251 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.252 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.253 DEBUG:boto:Params: {} 2022-09-02T05:59:52.253 DEBUG:boto:Token: None 2022-09-02T05:59:52.254 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/ 2022-09-02T05:59:52.255 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw= 2022-09-02T05:59:52.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw=', 'Content-Length': '0'} 2022-09-02T05:59:52.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b0d26752c7bac46-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.259 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.260 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:52.260 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:52.261 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.261 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.262 DEBUG:boto:Method: GET 2022-09-02T05:59:52.262 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.263 DEBUG:boto:Data: 2022-09-02T05:59:52.263 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.264 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.264 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.265 DEBUG:boto:Params: {} 2022-09-02T05:59:52.265 DEBUG:boto:Token: None 2022-09-02T05:59:52.266 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/ 2022-09-02T05:59:52.267 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw= 2022-09-02T05:59:52.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw=', 'Content-Length': '0'} 2022-09-02T05:59:52.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095cefd4b39753f88-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.318 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.319 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.320 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.320 DEBUG:boto:Method: HEAD 2022-09-02T05:59:52.321 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:52.321 DEBUG:boto:Data: 2022-09-02T05:59:52.322 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.323 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.323 DEBUG:boto:Params: {} 2022-09-02T05:59:52.324 DEBUG:boto:Token: None 2022-09-02T05:59:52.325 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95 2022-09-02T05:59:52.325 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLGKU3kQim/Rd5HCFMVtmPaKpWg= 2022-09-02T05:59:52.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLGKU3kQim/Rd5HCFMVtmPaKpWg=', 'Content-Length': '0'} 2022-09-02T05:59:52.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000402747e325253904-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.329 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.329 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.330 DEBUG:boto:Method: HEAD 2022-09-02T05:59:52.331 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:52.331 DEBUG:boto:Data: 2022-09-02T05:59:52.332 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.332 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.333 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.333 DEBUG:boto:Params: {} 2022-09-02T05:59:52.334 DEBUG:boto:Token: None 2022-09-02T05:59:52.334 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95 2022-09-02T05:59:52.335 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:vLGKU3kQim/Rd5HCFMVtmPaKpWg= 2022-09-02T05:59:52.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:vLGKU3kQim/Rd5HCFMVtmPaKpWg=', 'Content-Length': '0'} 2022-09-02T05:59:52.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac227a63e69b4aa9-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2022-09-02T05:59:52.341 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.342 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.343 DEBUG:boto:Method: GET 2022-09-02T05:59:52.344 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:52.345 DEBUG:boto:Data: 2022-09-02T05:59:52.346 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.347 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.348 DEBUG:boto:Params: {} 2022-09-02T05:59:52.348 DEBUG:boto:Token: None 2022-09-02T05:59:52.349 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95 2022-09-02T05:59:52.350 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dFNeP3hFSQBZWgjZc/BXV3ky8Ok= 2022-09-02T05:59:52.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dFNeP3hFSQBZWgjZc/BXV3ky8Ok=', 'Content-Length': '0'} 2022-09-02T05:59:52.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083b0c954082280e7-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.353 DEBUG:boto:path=/xljbkf-74/obj95 2022-09-02T05:59:52.354 DEBUG:boto:auth_path=/xljbkf-74/obj95 2022-09-02T05:59:52.355 DEBUG:boto:Method: GET 2022-09-02T05:59:52.355 DEBUG:boto:Path: /xljbkf-74/obj95 2022-09-02T05:59:52.356 DEBUG:boto:Data: 2022-09-02T05:59:52.357 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.357 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.358 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.358 DEBUG:boto:Params: {} 2022-09-02T05:59:52.359 DEBUG:boto:Token: None 2022-09-02T05:59:52.360 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/obj95 2022-09-02T05:59:52.360 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:dFNeP3hFSQBZWgjZc/BXV3ky8Ok= 2022-09-02T05:59:52.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:dFNeP3hFSQBZWgjZc/BXV3ky8Ok=', 'Content-Length': '0'} 2022-09-02T05:59:52.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 02 Sep 2022 05:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb419c49f4263c9-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.364 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:52.364 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:52.365 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.366 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.366 DEBUG:boto:Method: GET 2022-09-02T05:59:52.367 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.367 DEBUG:boto:Data: 2022-09-02T05:59:52.368 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2022-09-02T05:59:52.369 DEBUG:boto:Port: 8001 2022-09-02T05:59:52.370 DEBUG:boto:Params: {} 2022-09-02T05:59:52.371 DEBUG:boto:Token: None 2022-09-02T05:59:52.372 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/ 2022-09-02T05:59:52.372 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw= 2022-09-02T05:59:52.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw=', 'Content-Length': '0'} 2022-09-02T05:59:52.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f7084620a6e053-0063119bd8-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.376 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.377 DEBUG:boto:path=/xljbkf-74/ 2022-09-02T05:59:52.378 DEBUG:boto:auth_path=/xljbkf-74/ 2022-09-02T05:59:52.379 DEBUG:boto:path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.380 DEBUG:boto:auth_path=/xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.381 DEBUG:boto:Method: GET 2022-09-02T05:59:52.381 DEBUG:boto:Path: /xljbkf-74/?prefix=obj95 2022-09-02T05:59:52.382 DEBUG:boto:Data: 2022-09-02T05:59:52.383 DEBUG:boto:Headers: {} 2022-09-02T05:59:52.383 DEBUG:boto:Host: smithi160.front.sepia.ceph.com:8000 2022-09-02T05:59:52.384 DEBUG:boto:Port: 8000 2022-09-02T05:59:52.385 DEBUG:boto:Params: {} 2022-09-02T05:59:52.386 DEBUG:boto:Token: None 2022-09-02T05:59:52.387 DEBUG:boto:StringToSign: GET Fri, 02 Sep 2022 05:59:52 GMT /xljbkf-74/ 2022-09-02T05:59:52.388 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw= 2022-09-02T05:59:52.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 05:59:52 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:3jd8rRmLQ9vBKRe42rieh4QeECw=', 'Content-Length': '0'} 2022-09-02T05:59:52.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312912f8ba955b0d-0063119bd8-186e-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 02 Sep 2022 05:59:52 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T05:59:52.394 DEBUG:boto:b'xljbkf-74obj951000falseobj952022-09-02T05:47:57.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2022-09-02T05:59:52.396 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xljbkf-74 zones={test-zone3, test-zone2} 2022-09-02T05:59:52.398 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-02T05:59:52.401 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T05:59:52.403 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T05:59:52.397+0000 7fef5ff81700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 146195) UID: 0 2022-09-02T05:59:52.404 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T05:59:52.397+0000 7fef641ec600 -1 shutting down 2022-09-02T05:59:54.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-02T05:59:58.495 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-02T05:59:58.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:59:58.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'list', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--max-entries', '10000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T05:59:58.499 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket list --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --max-entries 10000 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T05:59:58.561 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:58.561 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:58.606 INFO:teuthology.orchestra.run.smithi160.stdout:[ 2022-09-02T05:59:58.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.606 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj11", 2022-09-02T05:59:58.607 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.607 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.607 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.607 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5 2022-09-02T05:59:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.608 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.609 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.609 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.609 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:44:47.815933Z", 2022-09-02T05:59:58.609 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.610 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.610 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.610 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.610 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.611 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.611 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.611 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_GeNQ397b0PMnD8IT7R8BJoZH3Kxcipx", 2022-09-02T05:59:58.612 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.612 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.612 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.613 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj111", 2022-09-02T05:59:58.613 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 10 2022-09-02T05:59:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:58:13.848085Z", 2022-09-02T05:59:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_Wy5Pv5ePgPSl0y2Kzudv9hnRiGkLjci", 2022-09-02T05:59:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj115", 2022-09-02T05:59:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 7 2022-09-02T05:59:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:50:15.725277Z", 2022-09-02T05:59:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_oOKezbSPlvyT0XjM_uVHgZL7qohHw38", 2022-09-02T05:59:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.626 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.626 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj116", 2022-09-02T05:59:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 8 2022-09-02T05:59:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:53:22.504688Z", 2022-09-02T05:59:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.631 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.631 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.631 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.632 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.632 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.632 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.633 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ijdmGVUdTsxE48FtSltQcj3gk2inmrg", 2022-09-02T05:59:58.633 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.633 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.633 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.634 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj124", 2022-09-02T05:59:58.634 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.635 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.635 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.635 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 1 2022-09-02T05:59:58.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.636 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.636 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.636 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.636 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.637 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 5, 2022-09-02T05:59:58.637 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:56:04.383611Z", 2022-09-02T05:59:58.637 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.637 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.638 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.638 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.638 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.638 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.639 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.639 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.639 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9v9PSVYwXFeYRUQqiSR02EZytr72GKg", 2022-09-02T05:59:58.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.640 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.640 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.641 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj145", 2022-09-02T05:59:58.641 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.641 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.641 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.642 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5 2022-09-02T05:59:58.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.642 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.642 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.643 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.643 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.643 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 6, 2022-09-02T05:59:58.643 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:53:22.535805Z", 2022-09-02T05:59:58.644 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.644 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.644 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.644 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.645 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.645 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.645 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.645 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bSzudvMQN9H2-GEPTQcIza9hDyppC1m", 2022-09-02T05:59:58.646 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.646 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.647 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.648 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj148", 2022-09-02T05:59:58.648 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.648 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.648 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.649 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5 2022-09-02T05:59:58.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.649 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.649 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.650 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.650 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.650 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 6, 2022-09-02T05:59:58.650 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:58:13.873171Z", 2022-09-02T05:59:58.651 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.651 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.651 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.651 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.652 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.652 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.652 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.652 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.653 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_p2AlsWN3no_uWGlwDGvDX96R4ZdK-Zx", 2022-09-02T05:59:58.653 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.653 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj155", 2022-09-02T05:59:58.655 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.655 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.655 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.655 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 7 2022-09-02T05:59:58.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.656 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.656 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.656 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.657 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.657 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 5, 2022-09-02T05:59:58.657 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:56:04.451829Z", 2022-09-02T05:59:58.657 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.658 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.658 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.658 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.658 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.659 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.659 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.659 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.660 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_wuerBi0KCncHi55qMweYpagm0a7AMub", 2022-09-02T05:59:58.660 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.660 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.660 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.661 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj17", 2022-09-02T05:59:58.661 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.662 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.662 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.662 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 6 2022-09-02T05:59:58.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.663 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.663 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.663 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.663 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.664 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.664 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:45:27.782959Z", 2022-09-02T05:59:58.664 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.664 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.665 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.665 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.665 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.665 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.666 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.666 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.666 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_b1boMGGGQcy6kv7p7b41MX5qi9RLjvK", 2022-09-02T05:59:58.667 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.667 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.667 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.668 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj185", 2022-09-02T05:59:58.668 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.668 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.669 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.669 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4 2022-09-02T05:59:58.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.669 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.670 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.670 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.670 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.670 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 5, 2022-09-02T05:59:58.671 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:58:13.946356Z", 2022-09-02T05:59:58.671 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.671 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.671 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.672 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.672 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.672 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.672 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.673 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.673 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8gO2j3By81HGZfZ5j9X_OxQPo5YM6Uy", 2022-09-02T05:59:58.673 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.674 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.674 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.675 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj19", 2022-09-02T05:59:58.675 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.675 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.675 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.676 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4 2022-09-02T05:59:58.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.676 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.676 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.677 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.677 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.677 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.677 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:47:57.563958Z", 2022-09-02T05:59:58.678 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.678 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.678 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.678 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.679 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.679 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.679 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.679 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.680 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_z8YwcSw5_YWLkvoQY-CQdqykJ8rRYGy", 2022-09-02T05:59:58.680 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.680 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.681 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.681 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj22", 2022-09-02T05:59:58.682 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.682 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.682 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.682 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 65 2022-09-02T05:59:58.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.683 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.683 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.683 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.684 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.684 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.684 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:44:47.844891Z", 2022-09-02T05:59:58.684 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.685 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.685 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.685 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.685 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.686 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.686 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.686 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.687 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_pvJPGxVr_PBpY1-k6VoEPZH8uoE_xuB", 2022-09-02T05:59:58.687 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.687 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.687 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.688 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj23", 2022-09-02T05:59:58.688 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.689 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.689 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.689 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 7 2022-09-02T05:59:58.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.690 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.690 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.690 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.690 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.691 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.691 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:50:15.588699Z", 2022-09-02T05:59:58.691 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.691 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.692 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.692 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.692 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.692 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.693 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.693 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.693 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ZGgo1yP2GCUCmumGQDk4ou-bhPehTKy", 2022-09-02T05:59:58.694 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.694 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.694 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.695 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj29", 2022-09-02T05:59:58.695 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.695 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.696 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.696 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 9 2022-09-02T05:59:58.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.696 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.697 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.697 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.697 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.697 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.698 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:53:22.382970Z", 2022-09-02T05:59:58.698 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.698 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.698 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.699 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.699 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.699 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.699 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.700 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.700 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ReoVQ--_bBIxBld869Gio-iwT5Go5s_", 2022-09-02T05:59:58.700 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.701 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.701 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.701 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj31", 2022-09-02T05:59:58.702 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.702 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.702 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.703 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.703 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.703 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.704 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.704 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.704 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 5, 2022-09-02T05:59:58.704 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:56:04.232772Z", 2022-09-02T05:59:58.705 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.705 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.705 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.705 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.706 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.706 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.706 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.706 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.707 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ggT_2_AHe7fL-uXbuqp8_0yMqp12yql", 2022-09-02T05:59:58.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.707 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj33", 2022-09-02T05:59:58.708 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.709 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.709 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.709 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 6 2022-09-02T05:59:58.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.710 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.711 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.711 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:44:47.867908Z", 2022-09-02T05:59:58.711 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.711 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.712 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.712 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.712 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.712 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.713 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_9VK-AGy9Z3QoZdvKcrr7hF-akO4dBcm", 2022-09-02T05:59:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.715 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj34", 2022-09-02T05:59:58.715 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.715 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 6 2022-09-02T05:59:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.716 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.717 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:45:27.806722Z", 2022-09-02T05:59:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.718 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.719 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_puxCrIVynfawTHsE_SkZy--C4SIVfds", 2022-09-02T05:59:58.720 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj37", 2022-09-02T05:59:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.722 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4 2022-09-02T05:59:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.723 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.724 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:58:13.750978Z", 2022-09-02T05:59:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.725 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.726 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_hXUs00uWizLJDjbH_nn7HZO1GsjZnCU", 2022-09-02T05:59:58.727 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj38", 2022-09-02T05:59:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.729 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4 2022-09-02T05:59:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.731 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:47:57.592432Z", 2022-09-02T05:59:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.732 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.733 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.734 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.734 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IOZVqVQy1s9vWBdPlmh05jcDVfaoYzM", 2022-09-02T05:59:58.734 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj44", 2022-09-02T05:59:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.736 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5 2022-09-02T05:59:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.737 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 6, 2022-09-02T05:59:58.738 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:44:47.927221Z", 2022-09-02T05:59:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.739 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_G8n3uMf4vrgbc-vk7K-FTz5x5xoYsGz", 2022-09-02T05:59:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.741 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj46", 2022-09-02T05:59:58.742 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 2 2022-09-02T05:59:58.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.744 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:50:15.613052Z", 2022-09-02T05:59:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.745 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.746 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.747 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_sqwHvymDaxtyG6fP0NLmEwSX9j7PFZN", 2022-09-02T05:59:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj51", 2022-09-02T05:59:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.749 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 9 2022-09-02T05:59:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.750 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.751 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.752 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:45:27.830050Z", 2022-09-02T05:59:58.752 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.752 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.752 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.753 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.753 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.753 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.753 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.754 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.754 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_pNSqdj7hsXBb2wM7WdaFcOS12DIGRW_", 2022-09-02T05:59:58.754 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.755 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.755 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.756 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj55", 2022-09-02T05:59:58.756 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.756 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.756 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.757 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.757 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.757 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.758 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.758 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.758 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 5, 2022-09-02T05:59:58.758 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:44:47.955141Z", 2022-09-02T05:59:58.759 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.759 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.759 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.759 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.760 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.760 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.760 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.760 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.761 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_TCqWBNDhZo7dJBGdetIQkh96Njgbd7c", 2022-09-02T05:59:58.761 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.761 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.762 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.762 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj57", 2022-09-02T05:59:58.763 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.763 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.763 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.763 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.764 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.764 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.764 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.765 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.765 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.765 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:47:57.619275Z", 2022-09-02T05:59:58.765 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.766 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.766 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.766 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.766 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.767 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.767 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.767 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.768 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_4OzFLP_h56m7ZtSEuo-Hme29EzD-dcg", 2022-09-02T05:59:58.768 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.768 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.768 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.769 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj58", 2022-09-02T05:59:58.769 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.770 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.770 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.770 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.771 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.771 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.771 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.771 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.772 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.772 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:53:22.415618Z", 2022-09-02T05:59:58.772 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.772 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.773 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.773 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.773 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.773 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.774 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.774 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.774 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_pQ5hbWrqN8JPTs-QxbgZ8WCnl-JNHrm", 2022-09-02T05:59:58.775 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.775 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.775 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.776 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj62", 2022-09-02T05:59:58.776 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.776 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.777 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.777 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.777 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.778 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.778 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.778 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.778 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.779 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:56:04.264701Z", 2022-09-02T05:59:58.779 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.779 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.779 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.780 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.780 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.780 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.780 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.780 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.781 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_IMzmPfsgFqG2vjJkR1b5Sk2uAIY-Mlv", 2022-09-02T05:59:58.781 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.782 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.782 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.783 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj68", 2022-09-02T05:59:58.783 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.783 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.783 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.784 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 1 2022-09-02T05:59:58.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.784 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.784 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.784 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 6, 2022-09-02T05:59:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:45:27.894908Z", 2022-09-02T05:59:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.786 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.786 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.786 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.787 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.787 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.787 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.787 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.788 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_3eRUAh4Ih-L3_iuo-Cou8vW4ODpG-Cg", 2022-09-02T05:59:58.788 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.788 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.789 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.789 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj69", 2022-09-02T05:59:58.789 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.790 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.790 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.790 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.791 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.791 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.791 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.791 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.792 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.792 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:50:15.636748Z", 2022-09-02T05:59:58.792 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.792 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.793 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.793 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.793 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.793 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.794 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.794 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.794 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_ctfaejP___TX4IeGbDqEiBXclisq6ML", 2022-09-02T05:59:58.795 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.795 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.795 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.796 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj74", 2022-09-02T05:59:58.796 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.796 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.797 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.797 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 6 2022-09-02T05:59:58.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.797 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.798 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.798 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.798 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.798 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.799 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:58:13.780370Z", 2022-09-02T05:59:58.799 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.799 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.799 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.800 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.800 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.800 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.800 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.801 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.801 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_iplUdryE_4ZoKH2Y4leqYUBoKgyK0xT", 2022-09-02T05:59:58.801 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.802 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.802 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.803 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj76", 2022-09-02T05:59:58.803 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.803 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.803 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.804 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 3 2022-09-02T05:59:58.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.804 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.804 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.805 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.805 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.805 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.805 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:47:57.685309Z", 2022-09-02T05:59:58.805 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.806 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.806 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.806 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.807 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.807 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.807 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.807 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.808 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_bQiXSpO_6Ks2DCPgyA5_ucdFF4Zb8wz", 2022-09-02T05:59:58.808 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.808 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.808 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.809 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj85", 2022-09-02T05:59:58.809 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.810 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.810 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.810 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 5 2022-09-02T05:59:58.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.811 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.811 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.811 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.811 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.812 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.812 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:45:27.918314Z", 2022-09-02T05:59:58.812 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.812 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.813 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.813 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.813 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.813 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.814 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.814 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.814 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_8mlEDcsIa14hwoj7QeLxdD-273duAc-", 2022-09-02T05:59:58.815 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.815 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.815 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.816 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj87", 2022-09-02T05:59:58.816 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.816 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.817 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.817 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 11 2022-09-02T05:59:58.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.817 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.818 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.818 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.819 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.820 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.820 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:53:22.479437Z", 2022-09-02T05:59:58.820 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.820 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.821 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.821 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.821 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.821 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.822 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.822 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.822 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_lTGiddei_2VdGi_QLA8bbx_SFNcQg5j", 2022-09-02T05:59:58.823 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.823 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.823 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.824 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj92", 2022-09-02T05:59:58.824 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.824 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.825 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.825 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 4 2022-09-02T05:59:58.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.825 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.826 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.826 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.827 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.827 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 3, 2022-09-02T05:59:58.827 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:50:15.700464Z", 2022-09-02T05:59:58.828 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.828 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.828 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.828 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.829 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.829 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.830 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.830 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.830 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_iQt2IOHIdA4OgONDHG6aSUiV8eFk33f", 2022-09-02T05:59:58.831 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.831 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.831 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.832 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj93", 2022-09-02T05:59:58.832 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.833 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.833 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.833 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 7 2022-09-02T05:59:58.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.834 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.834 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.835 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.835 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.835 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.835 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:56:04.343144Z", 2022-09-02T05:59:58.836 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.836 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.836 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.836 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.837 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.837 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.837 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.837 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.838 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_mN85Zappmm7PJcvodS-DzdkKzomA-lQ", 2022-09-02T05:59:58.839 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.839 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.839 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T05:59:58.840 INFO:teuthology.orchestra.run.smithi160.stdout: "name": "obj95", 2022-09-02T05:59:58.840 INFO:teuthology.orchestra.run.smithi160.stdout: "instance": "", 2022-09-02T05:59:58.840 INFO:teuthology.orchestra.run.smithi160.stdout: "ver": { 2022-09-02T05:59:58.841 INFO:teuthology.orchestra.run.smithi160.stdout: "pool": 9, 2022-09-02T05:59:58.841 INFO:teuthology.orchestra.run.smithi160.stdout: "epoch": 7 2022-09-02T05:59:58.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.841 INFO:teuthology.orchestra.run.smithi160.stdout: "locator": "", 2022-09-02T05:59:58.842 INFO:teuthology.orchestra.run.smithi160.stdout: "exists": true, 2022-09-02T05:59:58.842 INFO:teuthology.orchestra.run.smithi160.stdout: "meta": { 2022-09-02T05:59:58.842 INFO:teuthology.orchestra.run.smithi160.stdout: "category": 1, 2022-09-02T05:59:58.842 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 16, 2022-09-02T05:59:58.843 INFO:teuthology.orchestra.run.smithi160.stdout: "mtime": "2022-09-02T05:47:57.714666Z", 2022-09-02T05:59:58.843 INFO:teuthology.orchestra.run.smithi160.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2022-09-02T05:59:58.843 INFO:teuthology.orchestra.run.smithi160.stdout: "storage_class": "", 2022-09-02T05:59:58.843 INFO:teuthology.orchestra.run.smithi160.stdout: "owner": "rgw-multisite-test-user", 2022-09-02T05:59:58.844 INFO:teuthology.orchestra.run.smithi160.stdout: "owner_display_name": "Multisite Test User", 2022-09-02T05:59:58.844 INFO:teuthology.orchestra.run.smithi160.stdout: "content_type": "application/octet-stream", 2022-09-02T05:59:58.844 INFO:teuthology.orchestra.run.smithi160.stdout: "accounted_size": 3, 2022-09-02T05:59:58.845 INFO:teuthology.orchestra.run.smithi160.stdout: "user_data": "", 2022-09-02T05:59:58.845 INFO:teuthology.orchestra.run.smithi160.stdout: "appendable": false 2022-09-02T05:59:58.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T05:59:58.845 INFO:teuthology.orchestra.run.smithi160.stdout: "tag": "_oDmXi8rAMnYA0ilZvSkN_4g1_d8MC29", 2022-09-02T05:59:58.846 INFO:teuthology.orchestra.run.smithi160.stdout: "flags": 0, 2022-09-02T05:59:58.846 INFO:teuthology.orchestra.run.smithi160.stdout: "pending_map": [], 2022-09-02T05:59:58.846 INFO:teuthology.orchestra.run.smithi160.stdout: "versioned_epoch": 0 2022-09-02T05:59:58.847 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T05:59:58.847 INFO:teuthology.orchestra.run.smithi160.stdout:] 2022-09-02T05:59:58.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:58.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:58.850 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj11 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:58.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:58.905 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:59.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj111', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.006 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj111 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:59.058 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:59.059 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:59.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj115', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.179 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj115 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:59.231 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:59.232 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:59.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj116', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.362 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj116 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:59.414 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:59.414 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:59.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj124', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.529 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj124 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:59.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:59.580 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:59.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj145', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.699 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj145 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:59.750 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:59.751 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T05:59:59.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj148', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T05:59:59.884 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj148 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T05:59:59.936 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T05:59:59.936 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:00.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj155', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.042 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj155 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:00.095 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:00.096 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:00.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.230 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj17 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:00.281 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:00.282 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:00.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-02T06:00:00.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj185', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.398 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj185 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:00.452 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:00.453 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:00.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.561 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj19 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:00.611 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:00.612 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:00.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.738 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj22 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:00.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:00.791 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:00.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:00.905 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj23 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:00.958 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:00.959 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:01.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj29', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.058 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj29 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:01.109 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:01.110 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:01.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj31', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.214 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj31 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:01.266 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:01.266 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:01.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj33', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.416 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj33 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:01.467 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:01.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj34', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.594 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj34 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:01.647 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:01.648 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:01.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj37', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.770 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj37 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:01.821 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:01.822 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:01.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj38', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:01.938 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj38 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:01.990 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:01.991 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:02.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.082 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj44 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:02.134 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:02.135 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:02.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.263 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj46 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:02.315 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:02.316 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:02.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj51', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.423 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj51 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:02.475 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:02.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:02.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj55', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.593 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj55 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:02.645 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:02.646 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:02.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj57', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.755 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj57 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:02.808 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:02.808 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:02.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj58', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:02.923 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj58 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:02.975 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:02.976 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:03.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.102 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj62 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:03.154 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:03.155 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:03.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.281 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj68 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:03.333 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:03.334 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:03.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.424 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj69 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:03.476 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:03.477 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:03.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.605 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj74 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:03.657 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:03.658 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:03.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.778 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj76 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:03.830 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:03.831 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:03.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:03.942 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj85 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:03.994 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:03.995 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:04.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj87', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.104 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj87 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:04.155 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:04.156 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:04.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.267 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj92 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:04.318 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:04.319 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:04.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj93', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.453 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj93 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:04.506 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:04.507 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:04.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'object', 'rm', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--object', 'obj95', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.639 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 object rm --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --object obj95 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:04.692 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:04.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'trim', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.818 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog trim --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:04.868 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:04.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:04.935 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:04.996 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:04.997 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:05.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:05.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'run', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'xljbkf-74', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:00:05.060 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync run --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket xljbkf-74 --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:00:05.114 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:05.115 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:05.212 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.210+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2022-09-02T06:00:05.216 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.215+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: waiting on bucket lease 2022-09-02T06:00:05.217 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.216+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: status after acquiring the lock is: init 2022-09-02T06:00:05.241 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.240+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: listing bucket for full sync 2022-09-02T06:00:05.250 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj11 2022-09-02T06:00:05.251 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj11]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj11 op=0 op_state=1 2022-09-02T06:00:05.251 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj11]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj11[0] 2022-09-02T06:00:05.251 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj111 2022-09-02T06:00:05.251 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj111 op=0 op_state=1 2022-09-02T06:00:05.252 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: : injecting data sync error on key=obj111 2022-09-02T06:00:05.253 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.253 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj111 2022-09-02T06:00:05.254 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj115 2022-09-02T06:00:05.254 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj115 op=0 op_state=1 2022-09-02T06:00:05.254 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj115[0] 2022-09-02T06:00:05.255 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj116 2022-09-02T06:00:05.255 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj116 op=0 op_state=1 2022-09-02T06:00:05.255 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj116[0] 2022-09-02T06:00:05.255 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj124 2022-09-02T06:00:05.256 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj124]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj124 op=0 op_state=1 2022-09-02T06:00:05.256 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj124]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj124[0] 2022-09-02T06:00:05.256 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj145 2022-09-02T06:00:05.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj145 op=0 op_state=1 2022-09-02T06:00:05.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj145[0] 2022-09-02T06:00:05.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj148 2022-09-02T06:00:05.257 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj148 op=0 op_state=1 2022-09-02T06:00:05.258 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj148[0] 2022-09-02T06:00:05.258 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj155 2022-09-02T06:00:05.258 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj155 op=0 op_state=1 2022-09-02T06:00:05.259 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: : injecting data sync error on key=obj155 2022-09-02T06:00:05.260 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.260 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj155 2022-09-02T06:00:05.260 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj17 2022-09-02T06:00:05.261 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj17 op=0 op_state=1 2022-09-02T06:00:05.261 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj17[0] 2022-09-02T06:00:05.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj185 2022-09-02T06:00:05.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj185]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj185 op=0 op_state=1 2022-09-02T06:00:05.262 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj185]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj185[0] 2022-09-02T06:00:05.263 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj19 2022-09-02T06:00:05.263 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj19 op=0 op_state=1 2022-09-02T06:00:05.263 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj19[0] 2022-09-02T06:00:05.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj22 2022-09-02T06:00:05.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj22 op=0 op_state=1 2022-09-02T06:00:05.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: : injecting data sync error on key=obj22 2022-09-02T06:00:05.265 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.265 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj22 2022-09-02T06:00:05.265 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj23 2022-09-02T06:00:05.265 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj23]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj23 op=0 op_state=1 2022-09-02T06:00:05.266 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj23]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj23[0] 2022-09-02T06:00:05.266 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj29 2022-09-02T06:00:05.266 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj29 op=0 op_state=1 2022-09-02T06:00:05.267 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: : injecting data sync error on key=obj29 2022-09-02T06:00:05.267 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.267 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj29 2022-09-02T06:00:05.267 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj31 2022-09-02T06:00:05.268 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj31 op=0 op_state=1 2022-09-02T06:00:05.268 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj31[0] 2022-09-02T06:00:05.268 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj33 2022-09-02T06:00:05.269 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj33]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj33 op=0 op_state=1 2022-09-02T06:00:05.269 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj33]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj33[0] 2022-09-02T06:00:05.269 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj34 2022-09-02T06:00:05.269 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj34 op=0 op_state=1 2022-09-02T06:00:05.270 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: : injecting data sync error on key=obj34 2022-09-02T06:00:05.270 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.271 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj34 2022-09-02T06:00:05.271 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj37 2022-09-02T06:00:05.271 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj37 op=0 op_state=1 2022-09-02T06:00:05.272 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj37[0] 2022-09-02T06:00:05.272 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj38 2022-09-02T06:00:05.273 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj38]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj38 op=0 op_state=1 2022-09-02T06:00:05.273 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj38]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj38[0] 2022-09-02T06:00:05.273 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj44 2022-09-02T06:00:05.274 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj44 op=0 op_state=1 2022-09-02T06:00:05.274 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj44[0] 2022-09-02T06:00:05.274 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj46 2022-09-02T06:00:05.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj46]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj46 op=0 op_state=1 2022-09-02T06:00:05.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj46]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj46[0] 2022-09-02T06:00:05.276 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.250+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: ERROR: full sync failed. error: -5 2022-09-02T06:00:05.277 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.268+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: success 2022-09-02T06:00:05.277 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.274+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: success 2022-09-02T06:00:05.277 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: success 2022-09-02T06:00:05.277 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: success 2022-09-02T06:00:05.278 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: success 2022-09-02T06:00:05.279 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: success 2022-09-02T06:00:05.279 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: success 2022-09-02T06:00:05.280 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: success 2022-09-02T06:00:05.280 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: success 2022-09-02T06:00:05.280 INFO:teuthology.orchestra.run.smithi160.stderr:*** Caught signal (Segmentation fault) ** 2022-09-02T06:00:05.281 INFO:teuthology.orchestra.run.smithi160.stderr: in thread 7f913a1b1600 thread_name:radosgw-admin 2022-09-02T06:00:05.311 INFO:teuthology.orchestra.run.smithi160.stderr: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:00:05.312 INFO:teuthology.orchestra.run.smithi160.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9141977cf0] 2022-09-02T06:00:05.312 INFO:teuthology.orchestra.run.smithi160.stderr: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x1c2) [0x7f913f9757d2] 2022-09-02T06:00:05.312 INFO:teuthology.orchestra.run.smithi160.stderr: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x555a31973e49] 2022-09-02T06:00:05.313 INFO:teuthology.orchestra.run.smithi160.stderr: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x555a3199a76e] 2022-09-02T06:00:05.313 INFO:teuthology.orchestra.run.smithi160.stderr: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x555a31a9440f] 2022-09-02T06:00:05.314 INFO:teuthology.orchestra.run.smithi160.stderr: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x555a31a94be1] 2022-09-02T06:00:05.314 INFO:teuthology.orchestra.run.smithi160.stderr: 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x555a31944caa] 2022-09-02T06:00:05.315 INFO:teuthology.orchestra.run.smithi160.stderr: 8: main() 2022-09-02T06:00:05.315 INFO:teuthology.orchestra.run.smithi160.stderr: 9: __libc_start_main() 2022-09-02T06:00:05.315 INFO:teuthology.orchestra.run.smithi160.stderr: 10: _start() 2022-09-02T06:00:05.316 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:05.280+0000 7f913a1b1600 -1 *** Caught signal (Segmentation fault) ** 2022-09-02T06:00:05.316 INFO:teuthology.orchestra.run.smithi160.stderr: in thread 7f913a1b1600 thread_name:radosgw-admin 2022-09-02T06:00:05.316 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.317 INFO:teuthology.orchestra.run.smithi160.stderr: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:00:05.317 INFO:teuthology.orchestra.run.smithi160.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9141977cf0] 2022-09-02T06:00:05.317 INFO:teuthology.orchestra.run.smithi160.stderr: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x1c2) [0x7f913f9757d2] 2022-09-02T06:00:05.317 INFO:teuthology.orchestra.run.smithi160.stderr: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x555a31973e49] 2022-09-02T06:00:05.318 INFO:teuthology.orchestra.run.smithi160.stderr: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x555a3199a76e] 2022-09-02T06:00:05.318 INFO:teuthology.orchestra.run.smithi160.stderr: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x555a31a9440f] 2022-09-02T06:00:05.318 INFO:teuthology.orchestra.run.smithi160.stderr: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x555a31a94be1] 2022-09-02T06:00:05.319 INFO:teuthology.orchestra.run.smithi160.stderr: 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x555a31944caa] 2022-09-02T06:00:05.319 INFO:teuthology.orchestra.run.smithi160.stderr: 8: main() 2022-09-02T06:00:05.319 INFO:teuthology.orchestra.run.smithi160.stderr: 9: __libc_start_main() 2022-09-02T06:00:05.320 INFO:teuthology.orchestra.run.smithi160.stderr: 10: _start() 2022-09-02T06:00:05.320 INFO:teuthology.orchestra.run.smithi160.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-02T06:00:05.320 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.321 INFO:teuthology.orchestra.run.smithi160.stderr:--- begin dump of recent events --- 2022-09-02T06:00:05.321 INFO:teuthology.orchestra.run.smithi160.stderr: -499> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.321 INFO:teuthology.orchestra.run.smithi160.stderr: -498> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.322 INFO:teuthology.orchestra.run.smithi160.stderr: -497> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.322 INFO:teuthology.orchestra.run.smithi160.stderr: -496> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.323 INFO:teuthology.orchestra.run.smithi160.stderr: -495> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.323 INFO:teuthology.orchestra.run.smithi160.stderr: -494> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.324 INFO:teuthology.orchestra.run.smithi160.stderr: -493> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.324 INFO:teuthology.orchestra.run.smithi160.stderr: -492> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.324 INFO:teuthology.orchestra.run.smithi160.stderr: -491> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.324 INFO:teuthology.orchestra.run.smithi160.stderr: -490> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.325 INFO:teuthology.orchestra.run.smithi160.stderr: -489> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.325 INFO:teuthology.orchestra.run.smithi160.stderr: -488> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.325 INFO:teuthology.orchestra.run.smithi160.stderr: -487> 2022-09-02T06:00:05.114+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.326 INFO:teuthology.orchestra.run.smithi160.stderr: -486> 2022-09-02T06:00:05.114+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.326 INFO:teuthology.orchestra.run.smithi160.stderr: -485> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.326 INFO:teuthology.orchestra.run.smithi160.stderr: -484> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient: _add_conns ranks=[1,0] 2022-09-02T06:00:05.327 INFO:teuthology.orchestra.run.smithi160.stderr: -483> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34a42800 addr v2:172.21.15.160:3300/0 2022-09-02T06:00:05.327 INFO:teuthology.orchestra.run.smithi160.stderr: -482> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a-legacy con 0x555a34a42c00 addr v1:172.21.15.160:6789/0 2022-09-02T06:00:05.327 INFO:teuthology.orchestra.run.smithi160.stderr: -481> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.327 INFO:teuthology.orchestra.run.smithi160.stderr: -480> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.328 INFO:teuthology.orchestra.run.smithi160.stderr: -479> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.755205s 2022-09-02T06:00:05.328 INFO:teuthology.orchestra.run.smithi160.stderr: -478> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34a42800 auth_method 0 2022-09-02T06:00:05.328 INFO:teuthology.orchestra.run.smithi160.stderr: -477> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.329 INFO:teuthology.orchestra.run.smithi160.stderr: -476> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.329 INFO:teuthology.orchestra.run.smithi160.stderr: -475> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.329 INFO:teuthology.orchestra.run.smithi160.stderr: -474> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.329 INFO:teuthology.orchestra.run.smithi160.stderr: -473> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.330 INFO:teuthology.orchestra.run.smithi160.stderr: -472> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.330 INFO:teuthology.orchestra.run.smithi160.stderr: -471> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6679 payload 274 2022-09-02T06:00:05.330 INFO:teuthology.orchestra.run.smithi160.stderr: -470> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.331 INFO:teuthology.orchestra.run.smithi160.stderr: -469> 2022-09-02T06:00:05.114+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.331 INFO:teuthology.orchestra.run.smithi160.stderr: -468> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.331 INFO:teuthology.orchestra.run.smithi160.stderr: -467> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.331 INFO:teuthology.orchestra.run.smithi160.stderr: -466> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.332 INFO:teuthology.orchestra.run.smithi160.stderr: -465> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: dump: 2022-09-02T06:00:05.332 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.332 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.333 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.333 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.333 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.333 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.334 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.334 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.334 INFO:teuthology.orchestra.run.smithi160.stderr: -464> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.335 INFO:teuthology.orchestra.run.smithi160.stderr: -463> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.335 INFO:teuthology.orchestra.run.smithi160.stderr: -462> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6679 2022-09-02T06:00:05.335 INFO:teuthology.orchestra.run.smithi160.stderr: -461> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.335 INFO:teuthology.orchestra.run.smithi160.stderr: -460> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.336 INFO:teuthology.orchestra.run.smithi160.stderr: -459> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.336 INFO:teuthology.orchestra.run.smithi160.stderr: -458> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: dump: 2022-09-02T06:00:05.336 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.336 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.337 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.337 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.337 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.338 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.338 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.338 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.338 INFO:teuthology.orchestra.run.smithi160.stderr: -457> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient: get_monmap_and_config success 2022-09-02T06:00:05.339 INFO:teuthology.orchestra.run.smithi160.stderr: -456> 2022-09-02T06:00:05.114+0000 7f913a1b1600 4 set_mon_vals no callback set 2022-09-02T06:00:05.339 INFO:teuthology.orchestra.run.smithi160.stderr: -455> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: shutdown 2022-09-02T06:00:05.339 INFO:teuthology.orchestra.run.smithi160.stderr: -454> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) init /var/run/ceph/c2-client.0.153981.asok 2022-09-02T06:00:05.340 INFO:teuthology.orchestra.run.smithi160.stderr: -453> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) bind_and_listen /var/run/ceph/c2-client.0.153981.asok 2022-09-02T06:00:05.340 INFO:teuthology.orchestra.run.smithi160.stderr: -452> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command 0 hook 0x555a34a93378 2022-09-02T06:00:05.340 INFO:teuthology.orchestra.run.smithi160.stderr: -451> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command version hook 0x555a34a93378 2022-09-02T06:00:05.340 INFO:teuthology.orchestra.run.smithi160.stderr: -450> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command git_version hook 0x555a34a93378 2022-09-02T06:00:05.341 INFO:teuthology.orchestra.run.smithi160.stderr: -449> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command help hook 0x555a34a36bf0 2022-09-02T06:00:05.341 INFO:teuthology.orchestra.run.smithi160.stderr: -448> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command get_command_descriptions hook 0x555a34a36b50 2022-09-02T06:00:05.341 INFO:teuthology.orchestra.run.smithi160.stderr: -447> 2022-09-02T06:00:05.115+0000 7f913715b700 5 asok(0x555a34ad6000) entry start 2022-09-02T06:00:05.342 INFO:teuthology.orchestra.run.smithi160.stderr: -446> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: get_monmap_and_config 2022-09-02T06:00:05.342 INFO:teuthology.orchestra.run.smithi160.stderr: -445> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.343 INFO:teuthology.orchestra.run.smithi160.stderr: -444> 2022-09-02T06:00:05.115+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.343 INFO:teuthology.orchestra.run.smithi160.stderr: -443> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.343 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.343 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.344 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.344 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.344 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.344 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.345 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.345 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.345 INFO:teuthology.orchestra.run.smithi160.stderr: -442> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.345 INFO:teuthology.orchestra.run.smithi160.stderr: -441> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.346 INFO:teuthology.orchestra.run.smithi160.stderr: -440> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.346 INFO:teuthology.orchestra.run.smithi160.stderr: -439> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.346 INFO:teuthology.orchestra.run.smithi160.stderr: -438> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.347 INFO:teuthology.orchestra.run.smithi160.stderr: -437> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.347 INFO:teuthology.orchestra.run.smithi160.stderr: -436> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.347 INFO:teuthology.orchestra.run.smithi160.stderr: -435> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.347 INFO:teuthology.orchestra.run.smithi160.stderr: -434> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.348 INFO:teuthology.orchestra.run.smithi160.stderr: -433> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.348 INFO:teuthology.orchestra.run.smithi160.stderr: -432> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.348 INFO:teuthology.orchestra.run.smithi160.stderr: -431> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.348 INFO:teuthology.orchestra.run.smithi160.stderr: -430> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.349 INFO:teuthology.orchestra.run.smithi160.stderr: -429> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.349 INFO:teuthology.orchestra.run.smithi160.stderr: -428> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.349 INFO:teuthology.orchestra.run.smithi160.stderr: -427> 2022-09-02T06:00:05.115+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.350 INFO:teuthology.orchestra.run.smithi160.stderr: -426> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.350 INFO:teuthology.orchestra.run.smithi160.stderr: -425> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding auth protocol: cephx 2022-09-02T06:00:05.350 INFO:teuthology.orchestra.run.smithi160.stderr: -424> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding auth protocol: cephx 2022-09-02T06:00:05.350 INFO:teuthology.orchestra.run.smithi160.stderr: -423> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding auth protocol: cephx 2022-09-02T06:00:05.351 INFO:teuthology.orchestra.run.smithi160.stderr: -422> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.351 INFO:teuthology.orchestra.run.smithi160.stderr: -421> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.351 INFO:teuthology.orchestra.run.smithi160.stderr: -420> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.351 INFO:teuthology.orchestra.run.smithi160.stderr: -419> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.352 INFO:teuthology.orchestra.run.smithi160.stderr: -418> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.352 INFO:teuthology.orchestra.run.smithi160.stderr: -417> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.352 INFO:teuthology.orchestra.run.smithi160.stderr: -416> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.352 INFO:teuthology.orchestra.run.smithi160.stderr: -415> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.353 INFO:teuthology.orchestra.run.smithi160.stderr: -414> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.353 INFO:teuthology.orchestra.run.smithi160.stderr: -413> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.354 INFO:teuthology.orchestra.run.smithi160.stderr: -412> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.354 INFO:teuthology.orchestra.run.smithi160.stderr: -411> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.354 INFO:teuthology.orchestra.run.smithi160.stderr: -410> 2022-09-02T06:00:05.116+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.354 INFO:teuthology.orchestra.run.smithi160.stderr: -409> 2022-09-02T06:00:05.116+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.355 INFO:teuthology.orchestra.run.smithi160.stderr: -408> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.355 INFO:teuthology.orchestra.run.smithi160.stderr: -407> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.355 INFO:teuthology.orchestra.run.smithi160.stderr: -406> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34a42800 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.355 INFO:teuthology.orchestra.run.smithi160.stderr: -405> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.356 INFO:teuthology.orchestra.run.smithi160.stderr: -404> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.356 INFO:teuthology.orchestra.run.smithi160.stderr: -403> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.757161s 2022-09-02T06:00:05.356 INFO:teuthology.orchestra.run.smithi160.stderr: -402> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34a42800 auth_method 0 2022-09-02T06:00:05.357 INFO:teuthology.orchestra.run.smithi160.stderr: -401> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.357 INFO:teuthology.orchestra.run.smithi160.stderr: -400> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.358 INFO:teuthology.orchestra.run.smithi160.stderr: -399> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.358 INFO:teuthology.orchestra.run.smithi160.stderr: -398> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.359 INFO:teuthology.orchestra.run.smithi160.stderr: -397> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.359 INFO:teuthology.orchestra.run.smithi160.stderr: -396> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.359 INFO:teuthology.orchestra.run.smithi160.stderr: -395> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6681 payload 274 2022-09-02T06:00:05.360 INFO:teuthology.orchestra.run.smithi160.stderr: -394> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.360 INFO:teuthology.orchestra.run.smithi160.stderr: -393> 2022-09-02T06:00:05.116+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.360 INFO:teuthology.orchestra.run.smithi160.stderr: -392> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.361 INFO:teuthology.orchestra.run.smithi160.stderr: -391> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.361 INFO:teuthology.orchestra.run.smithi160.stderr: -390> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.361 INFO:teuthology.orchestra.run.smithi160.stderr: -389> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: dump: 2022-09-02T06:00:05.362 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.362 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.363 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.363 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.363 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.364 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.364 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.365 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.365 INFO:teuthology.orchestra.run.smithi160.stderr: -388> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.365 INFO:teuthology.orchestra.run.smithi160.stderr: -387> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.366 INFO:teuthology.orchestra.run.smithi160.stderr: -386> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6681 2022-09-02T06:00:05.366 INFO:teuthology.orchestra.run.smithi160.stderr: -385> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.367 INFO:teuthology.orchestra.run.smithi160.stderr: -384> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.367 INFO:teuthology.orchestra.run.smithi160.stderr: -383> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.367 INFO:teuthology.orchestra.run.smithi160.stderr: -382> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: dump: 2022-09-02T06:00:05.368 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.368 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.369 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.369 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.370 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.370 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.370 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.371 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.371 INFO:teuthology.orchestra.run.smithi160.stderr: -381> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: get_monmap_and_config success 2022-09-02T06:00:05.372 INFO:teuthology.orchestra.run.smithi160.stderr: -380> 2022-09-02T06:00:05.116+0000 7f913a1b1600 4 set_mon_vals no callback set 2022-09-02T06:00:05.372 INFO:teuthology.orchestra.run.smithi160.stderr: -379> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: shutdown 2022-09-02T06:00:05.372 INFO:teuthology.orchestra.run.smithi160.stderr: -378> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.373 INFO:teuthology.orchestra.run.smithi160.stderr: -377> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.373 INFO:teuthology.orchestra.run.smithi160.stderr: -376> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.374 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.374 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.374 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.375 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.375 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.375 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.375 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.376 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.376 INFO:teuthology.orchestra.run.smithi160.stderr: -375> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.376 INFO:teuthology.orchestra.run.smithi160.stderr: -374> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.377 INFO:teuthology.orchestra.run.smithi160.stderr: -373> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.377 INFO:teuthology.orchestra.run.smithi160.stderr: -372> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.377 INFO:teuthology.orchestra.run.smithi160.stderr: -371> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.378 INFO:teuthology.orchestra.run.smithi160.stderr: -370> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.378 INFO:teuthology.orchestra.run.smithi160.stderr: -369> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.378 INFO:teuthology.orchestra.run.smithi160.stderr: -368> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.379 INFO:teuthology.orchestra.run.smithi160.stderr: -367> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.379 INFO:teuthology.orchestra.run.smithi160.stderr: -366> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.380 INFO:teuthology.orchestra.run.smithi160.stderr: -365> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.380 INFO:teuthology.orchestra.run.smithi160.stderr: -364> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.381 INFO:teuthology.orchestra.run.smithi160.stderr: -363> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.381 INFO:teuthology.orchestra.run.smithi160.stderr: -362> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.381 INFO:teuthology.orchestra.run.smithi160.stderr: -361> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.382 INFO:teuthology.orchestra.run.smithi160.stderr: -360> 2022-09-02T06:00:05.117+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.382 INFO:teuthology.orchestra.run.smithi160.stderr: -359> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: starting msgr at 2022-09-02T06:00:05.383 INFO:teuthology.orchestra.run.smithi160.stderr: -358> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: starting objecter 2022-09-02T06:00:05.383 INFO:teuthology.orchestra.run.smithi160.stderr: -357> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command objecter_requests hook 0x555a34a36b90 2022-09-02T06:00:05.384 INFO:teuthology.orchestra.run.smithi160.stderr: -356> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: setting wanted keys 2022-09-02T06:00:05.384 INFO:teuthology.orchestra.run.smithi160.stderr: -355> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: calling monclient init 2022-09-02T06:00:05.385 INFO:teuthology.orchestra.run.smithi160.stderr: -354> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.385 INFO:teuthology.orchestra.run.smithi160.stderr: -353> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding auth protocol: cephx 2022-09-02T06:00:05.386 INFO:teuthology.orchestra.run.smithi160.stderr: -352> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding auth protocol: cephx 2022-09-02T06:00:05.386 INFO:teuthology.orchestra.run.smithi160.stderr: -351> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding auth protocol: cephx 2022-09-02T06:00:05.387 INFO:teuthology.orchestra.run.smithi160.stderr: -350> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.387 INFO:teuthology.orchestra.run.smithi160.stderr: -349> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.388 INFO:teuthology.orchestra.run.smithi160.stderr: -348> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.388 INFO:teuthology.orchestra.run.smithi160.stderr: -347> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.389 INFO:teuthology.orchestra.run.smithi160.stderr: -346> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.389 INFO:teuthology.orchestra.run.smithi160.stderr: -345> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.390 INFO:teuthology.orchestra.run.smithi160.stderr: -344> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.390 INFO:teuthology.orchestra.run.smithi160.stderr: -343> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.391 INFO:teuthology.orchestra.run.smithi160.stderr: -342> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.391 INFO:teuthology.orchestra.run.smithi160.stderr: -341> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.391 INFO:teuthology.orchestra.run.smithi160.stderr: -340> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.392 INFO:teuthology.orchestra.run.smithi160.stderr: -339> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.392 INFO:teuthology.orchestra.run.smithi160.stderr: -338> 2022-09-02T06:00:05.117+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.392 INFO:teuthology.orchestra.run.smithi160.stderr: -337> 2022-09-02T06:00:05.117+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.392 INFO:teuthology.orchestra.run.smithi160.stderr: -336> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.393 INFO:teuthology.orchestra.run.smithi160.stderr: -335> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.393 INFO:teuthology.orchestra.run.smithi160.stderr: -334> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34a42800 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.394 INFO:teuthology.orchestra.run.smithi160.stderr: -333> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.394 INFO:teuthology.orchestra.run.smithi160.stderr: -332> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.394 INFO:teuthology.orchestra.run.smithi160.stderr: -331> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.758652s 2022-09-02T06:00:05.394 INFO:teuthology.orchestra.run.smithi160.stderr: -330> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34a42800 auth_method 0 2022-09-02T06:00:05.395 INFO:teuthology.orchestra.run.smithi160.stderr: -329> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.395 INFO:teuthology.orchestra.run.smithi160.stderr: -328> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.396 INFO:teuthology.orchestra.run.smithi160.stderr: -327> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.396 INFO:teuthology.orchestra.run.smithi160.stderr: -326> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.396 INFO:teuthology.orchestra.run.smithi160.stderr: -325> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.396 INFO:teuthology.orchestra.run.smithi160.stderr: -324> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.397 INFO:teuthology.orchestra.run.smithi160.stderr: -323> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6682 payload 824 2022-09-02T06:00:05.397 INFO:teuthology.orchestra.run.smithi160.stderr: -322> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.398 INFO:teuthology.orchestra.run.smithi160.stderr: -321> 2022-09-02T06:00:05.117+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.399 INFO:teuthology.orchestra.run.smithi160.stderr: -320> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.399 INFO:teuthology.orchestra.run.smithi160.stderr: -319> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.400 INFO:teuthology.orchestra.run.smithi160.stderr: -318> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.400 INFO:teuthology.orchestra.run.smithi160.stderr: -317> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: dump: 2022-09-02T06:00:05.400 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.401 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.401 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.402 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.402 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.402 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.403 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.404 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.404 INFO:teuthology.orchestra.run.smithi160.stderr: -316> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.405 INFO:teuthology.orchestra.run.smithi160.stderr: -315> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.406 INFO:teuthology.orchestra.run.smithi160.stderr: -314> 2022-09-02T06:00:05.118+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6682 2022-09-02T06:00:05.406 INFO:teuthology.orchestra.run.smithi160.stderr: -313> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.407 INFO:teuthology.orchestra.run.smithi160.stderr: -312> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.408 INFO:teuthology.orchestra.run.smithi160.stderr: -311> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.408 INFO:teuthology.orchestra.run.smithi160.stderr: -310> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.409 INFO:teuthology.orchestra.run.smithi160.stderr: -309> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.409 INFO:teuthology.orchestra.run.smithi160.stderr: -308> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: dump: 2022-09-02T06:00:05.410 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.410 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.411 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.412 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.412 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.412 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.413 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.413 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.414 INFO:teuthology.orchestra.run.smithi160.stderr: -307> 2022-09-02T06:00:05.118+0000 7f9136159700 4 set_mon_vals no callback set 2022-09-02T06:00:05.414 INFO:teuthology.orchestra.run.smithi160.stderr: -306> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.415 INFO:teuthology.orchestra.run.smithi160.stderr: -305> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.415 INFO:teuthology.orchestra.run.smithi160.stderr: -304> 2022-09-02T06:00:05.118+0000 7f913a1b1600 1 librados: init done 2022-09-02T06:00:05.416 INFO:teuthology.orchestra.run.smithi160.stderr: -303> 2022-09-02T06:00:05.119+0000 7f9134956700 4 mgrc handle_mgr_map Got map version 5 2022-09-02T06:00:05.417 INFO:teuthology.orchestra.run.smithi160.stderr: -302> 2022-09-02T06:00:05.119+0000 7f9134956700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.417 INFO:teuthology.orchestra.run.smithi160.stderr: -301> 2022-09-02T06:00:05.119+0000 7f9134956700 4 mgrc reconnect Starting new session with [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.417 INFO:teuthology.orchestra.run.smithi160.stderr: -300> 2022-09-02T06:00:05.119+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34a43400 auth_method 0 2022-09-02T06:00:05.418 INFO:teuthology.orchestra.run.smithi160.stderr: -299> 2022-09-02T06:00:05.119+0000 7f913895e700 10 monclient: get_auth_request con 0x555a34a42c00 auth_method 0 2022-09-02T06:00:05.418 INFO:teuthology.orchestra.run.smithi160.stderr: -298> 2022-09-02T06:00:05.120+0000 7f913815d700 10 monclient: get_auth_request con 0x555a34a43c00 auth_method 0 2022-09-02T06:00:05.419 INFO:teuthology.orchestra.run.smithi160.stderr: -297> 2022-09-02T06:00:05.121+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34d60800 auth_method 0 2022-09-02T06:00:05.419 INFO:teuthology.orchestra.run.smithi160.stderr: -296> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding auth protocol: cephx 2022-09-02T06:00:05.419 INFO:teuthology.orchestra.run.smithi160.stderr: -295> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding auth protocol: cephx 2022-09-02T06:00:05.420 INFO:teuthology.orchestra.run.smithi160.stderr: -294> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding auth protocol: cephx 2022-09-02T06:00:05.420 INFO:teuthology.orchestra.run.smithi160.stderr: -293> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.420 INFO:teuthology.orchestra.run.smithi160.stderr: -292> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.420 INFO:teuthology.orchestra.run.smithi160.stderr: -291> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.421 INFO:teuthology.orchestra.run.smithi160.stderr: -290> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.421 INFO:teuthology.orchestra.run.smithi160.stderr: -289> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.421 INFO:teuthology.orchestra.run.smithi160.stderr: -288> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.422 INFO:teuthology.orchestra.run.smithi160.stderr: -287> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.422 INFO:teuthology.orchestra.run.smithi160.stderr: -286> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.423 INFO:teuthology.orchestra.run.smithi160.stderr: -285> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.423 INFO:teuthology.orchestra.run.smithi160.stderr: -284> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.423 INFO:teuthology.orchestra.run.smithi160.stderr: -283> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.423 INFO:teuthology.orchestra.run.smithi160.stderr: -282> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.424 INFO:teuthology.orchestra.run.smithi160.stderr: -281> 2022-09-02T06:00:05.136+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.424 INFO:teuthology.orchestra.run.smithi160.stderr: -280> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache list hook 0x555a34a36b60 2022-09-02T06:00:05.424 INFO:teuthology.orchestra.run.smithi160.stderr: -279> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache inspect hook 0x555a34a36b60 2022-09-02T06:00:05.424 INFO:teuthology.orchestra.run.smithi160.stderr: -278> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache erase hook 0x555a34a36b60 2022-09-02T06:00:05.425 INFO:teuthology.orchestra.run.smithi160.stderr: -277> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache zap hook 0x555a34a36b60 2022-09-02T06:00:05.425 INFO:teuthology.orchestra.run.smithi160.stderr: -276> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: get_monmap_and_config 2022-09-02T06:00:05.426 INFO:teuthology.orchestra.run.smithi160.stderr: -275> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.426 INFO:teuthology.orchestra.run.smithi160.stderr: -274> 2022-09-02T06:00:05.137+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.426 INFO:teuthology.orchestra.run.smithi160.stderr: -273> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.427 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.427 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.427 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.428 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.428 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.428 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.428 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.429 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.429 INFO:teuthology.orchestra.run.smithi160.stderr: -272> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.429 INFO:teuthology.orchestra.run.smithi160.stderr: -271> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.429 INFO:teuthology.orchestra.run.smithi160.stderr: -270> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.430 INFO:teuthology.orchestra.run.smithi160.stderr: -269> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.430 INFO:teuthology.orchestra.run.smithi160.stderr: -268> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.430 INFO:teuthology.orchestra.run.smithi160.stderr: -267> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.431 INFO:teuthology.orchestra.run.smithi160.stderr: -266> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.431 INFO:teuthology.orchestra.run.smithi160.stderr: -265> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.431 INFO:teuthology.orchestra.run.smithi160.stderr: -264> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.431 INFO:teuthology.orchestra.run.smithi160.stderr: -263> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.432 INFO:teuthology.orchestra.run.smithi160.stderr: -262> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.432 INFO:teuthology.orchestra.run.smithi160.stderr: -261> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.433 INFO:teuthology.orchestra.run.smithi160.stderr: -260> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.433 INFO:teuthology.orchestra.run.smithi160.stderr: -259> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.433 INFO:teuthology.orchestra.run.smithi160.stderr: -258> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.434 INFO:teuthology.orchestra.run.smithi160.stderr: -257> 2022-09-02T06:00:05.137+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.434 INFO:teuthology.orchestra.run.smithi160.stderr: -256> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.434 INFO:teuthology.orchestra.run.smithi160.stderr: -255> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding auth protocol: cephx 2022-09-02T06:00:05.435 INFO:teuthology.orchestra.run.smithi160.stderr: -254> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding auth protocol: cephx 2022-09-02T06:00:05.435 INFO:teuthology.orchestra.run.smithi160.stderr: -253> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding auth protocol: cephx 2022-09-02T06:00:05.435 INFO:teuthology.orchestra.run.smithi160.stderr: -252> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.436 INFO:teuthology.orchestra.run.smithi160.stderr: -251> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.436 INFO:teuthology.orchestra.run.smithi160.stderr: -250> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.436 INFO:teuthology.orchestra.run.smithi160.stderr: -249> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.436 INFO:teuthology.orchestra.run.smithi160.stderr: -248> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.437 INFO:teuthology.orchestra.run.smithi160.stderr: -247> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.437 INFO:teuthology.orchestra.run.smithi160.stderr: -246> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.437 INFO:teuthology.orchestra.run.smithi160.stderr: -245> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.437 INFO:teuthology.orchestra.run.smithi160.stderr: -244> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.438 INFO:teuthology.orchestra.run.smithi160.stderr: -243> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.438 INFO:teuthology.orchestra.run.smithi160.stderr: -242> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.438 INFO:teuthology.orchestra.run.smithi160.stderr: -241> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.439 INFO:teuthology.orchestra.run.smithi160.stderr: -240> 2022-09-02T06:00:05.137+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.439 INFO:teuthology.orchestra.run.smithi160.stderr: -239> 2022-09-02T06:00:05.137+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.440 INFO:teuthology.orchestra.run.smithi160.stderr: -238> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.440 INFO:teuthology.orchestra.run.smithi160.stderr: -237> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.441 INFO:teuthology.orchestra.run.smithi160.stderr: -236> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34d95000 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.441 INFO:teuthology.orchestra.run.smithi160.stderr: -235> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.442 INFO:teuthology.orchestra.run.smithi160.stderr: -234> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.442 INFO:teuthology.orchestra.run.smithi160.stderr: -233> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.778340s 2022-09-02T06:00:05.443 INFO:teuthology.orchestra.run.smithi160.stderr: -232> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34d95000 auth_method 0 2022-09-02T06:00:05.443 INFO:teuthology.orchestra.run.smithi160.stderr: -231> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.444 INFO:teuthology.orchestra.run.smithi160.stderr: -230> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.444 INFO:teuthology.orchestra.run.smithi160.stderr: -229> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.445 INFO:teuthology.orchestra.run.smithi160.stderr: -228> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.445 INFO:teuthology.orchestra.run.smithi160.stderr: -227> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.446 INFO:teuthology.orchestra.run.smithi160.stderr: -226> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.446 INFO:teuthology.orchestra.run.smithi160.stderr: -225> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6683 payload 274 2022-09-02T06:00:05.447 INFO:teuthology.orchestra.run.smithi160.stderr: -224> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.447 INFO:teuthology.orchestra.run.smithi160.stderr: -223> 2022-09-02T06:00:05.137+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.448 INFO:teuthology.orchestra.run.smithi160.stderr: -222> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.449 INFO:teuthology.orchestra.run.smithi160.stderr: -221> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.449 INFO:teuthology.orchestra.run.smithi160.stderr: -220> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.450 INFO:teuthology.orchestra.run.smithi160.stderr: -219> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: dump: 2022-09-02T06:00:05.450 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.450 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.451 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.451 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.452 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.453 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.454 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.455 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.455 INFO:teuthology.orchestra.run.smithi160.stderr: -218> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.456 INFO:teuthology.orchestra.run.smithi160.stderr: -217> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.456 INFO:teuthology.orchestra.run.smithi160.stderr: -216> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.457 INFO:teuthology.orchestra.run.smithi160.stderr: -215> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.457 INFO:teuthology.orchestra.run.smithi160.stderr: -214> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.458 INFO:teuthology.orchestra.run.smithi160.stderr: -213> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: dump: 2022-09-02T06:00:05.458 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.459 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.459 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.460 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.460 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.461 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.461 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.461 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.463 INFO:teuthology.orchestra.run.smithi160.stderr: -212> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6683 2022-09-02T06:00:05.463 INFO:teuthology.orchestra.run.smithi160.stderr: -211> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: get_monmap_and_config success 2022-09-02T06:00:05.464 INFO:teuthology.orchestra.run.smithi160.stderr: -210> 2022-09-02T06:00:05.137+0000 7f913a1b1600 4 set_mon_vals no callback set 2022-09-02T06:00:05.464 INFO:teuthology.orchestra.run.smithi160.stderr: -209> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: shutdown 2022-09-02T06:00:05.464 INFO:teuthology.orchestra.run.smithi160.stderr: -208> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.465 INFO:teuthology.orchestra.run.smithi160.stderr: -207> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.465 INFO:teuthology.orchestra.run.smithi160.stderr: -206> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.465 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.465 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.466 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.466 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.466 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.467 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.467 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.468 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.468 INFO:teuthology.orchestra.run.smithi160.stderr: -205> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.468 INFO:teuthology.orchestra.run.smithi160.stderr: -204> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.469 INFO:teuthology.orchestra.run.smithi160.stderr: -203> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.469 INFO:teuthology.orchestra.run.smithi160.stderr: -202> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.470 INFO:teuthology.orchestra.run.smithi160.stderr: -201> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.470 INFO:teuthology.orchestra.run.smithi160.stderr: -200> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.470 INFO:teuthology.orchestra.run.smithi160.stderr: -199> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.471 INFO:teuthology.orchestra.run.smithi160.stderr: -198> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.471 INFO:teuthology.orchestra.run.smithi160.stderr: -197> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.471 INFO:teuthology.orchestra.run.smithi160.stderr: -196> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.471 INFO:teuthology.orchestra.run.smithi160.stderr: -195> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.472 INFO:teuthology.orchestra.run.smithi160.stderr: -194> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.472 INFO:teuthology.orchestra.run.smithi160.stderr: -193> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.472 INFO:teuthology.orchestra.run.smithi160.stderr: -192> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.473 INFO:teuthology.orchestra.run.smithi160.stderr: -191> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.473 INFO:teuthology.orchestra.run.smithi160.stderr: -190> 2022-09-02T06:00:05.138+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.473 INFO:teuthology.orchestra.run.smithi160.stderr: -189> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: starting msgr at 2022-09-02T06:00:05.473 INFO:teuthology.orchestra.run.smithi160.stderr: -188> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: starting objecter 2022-09-02T06:00:05.474 INFO:teuthology.orchestra.run.smithi160.stderr: -187> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command objecter_requests cmddesc objecter_requests hook 0x555a34a36730 EEXIST 2022-09-02T06:00:05.474 INFO:teuthology.orchestra.run.smithi160.stderr: -186> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: setting wanted keys 2022-09-02T06:00:05.474 INFO:teuthology.orchestra.run.smithi160.stderr: -185> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: calling monclient init 2022-09-02T06:00:05.474 INFO:teuthology.orchestra.run.smithi160.stderr: -184> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.475 INFO:teuthology.orchestra.run.smithi160.stderr: -183> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding auth protocol: cephx 2022-09-02T06:00:05.475 INFO:teuthology.orchestra.run.smithi160.stderr: -182> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding auth protocol: cephx 2022-09-02T06:00:05.475 INFO:teuthology.orchestra.run.smithi160.stderr: -181> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding auth protocol: cephx 2022-09-02T06:00:05.475 INFO:teuthology.orchestra.run.smithi160.stderr: -180> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.476 INFO:teuthology.orchestra.run.smithi160.stderr: -179> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.476 INFO:teuthology.orchestra.run.smithi160.stderr: -178> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.476 INFO:teuthology.orchestra.run.smithi160.stderr: -177> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.477 INFO:teuthology.orchestra.run.smithi160.stderr: -176> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.477 INFO:teuthology.orchestra.run.smithi160.stderr: -175> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.477 INFO:teuthology.orchestra.run.smithi160.stderr: -174> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.477 INFO:teuthology.orchestra.run.smithi160.stderr: -173> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.478 INFO:teuthology.orchestra.run.smithi160.stderr: -172> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.478 INFO:teuthology.orchestra.run.smithi160.stderr: -171> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.478 INFO:teuthology.orchestra.run.smithi160.stderr: -170> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.478 INFO:teuthology.orchestra.run.smithi160.stderr: -169> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.479 INFO:teuthology.orchestra.run.smithi160.stderr: -168> 2022-09-02T06:00:05.138+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.479 INFO:teuthology.orchestra.run.smithi160.stderr: -167> 2022-09-02T06:00:05.138+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.479 INFO:teuthology.orchestra.run.smithi160.stderr: -166> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.480 INFO:teuthology.orchestra.run.smithi160.stderr: -165> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.480 INFO:teuthology.orchestra.run.smithi160.stderr: -164> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34d95000 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.480 INFO:teuthology.orchestra.run.smithi160.stderr: -163> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.481 INFO:teuthology.orchestra.run.smithi160.stderr: -162> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.481 INFO:teuthology.orchestra.run.smithi160.stderr: -161> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.780058s 2022-09-02T06:00:05.481 INFO:teuthology.orchestra.run.smithi160.stderr: -160> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34d95000 auth_method 0 2022-09-02T06:00:05.481 INFO:teuthology.orchestra.run.smithi160.stderr: -159> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.482 INFO:teuthology.orchestra.run.smithi160.stderr: -158> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.482 INFO:teuthology.orchestra.run.smithi160.stderr: -157> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.482 INFO:teuthology.orchestra.run.smithi160.stderr: -156> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.483 INFO:teuthology.orchestra.run.smithi160.stderr: -155> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.483 INFO:teuthology.orchestra.run.smithi160.stderr: -154> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.483 INFO:teuthology.orchestra.run.smithi160.stderr: -153> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6684 payload 824 2022-09-02T06:00:05.483 INFO:teuthology.orchestra.run.smithi160.stderr: -152> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.484 INFO:teuthology.orchestra.run.smithi160.stderr: -151> 2022-09-02T06:00:05.139+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.484 INFO:teuthology.orchestra.run.smithi160.stderr: -150> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.484 INFO:teuthology.orchestra.run.smithi160.stderr: -149> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.484 INFO:teuthology.orchestra.run.smithi160.stderr: -148> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.485 INFO:teuthology.orchestra.run.smithi160.stderr: -147> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: dump: 2022-09-02T06:00:05.485 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.485 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.485 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.486 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.486 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.486 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.486 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.487 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.487 INFO:teuthology.orchestra.run.smithi160.stderr: -146> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.487 INFO:teuthology.orchestra.run.smithi160.stderr: -145> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.488 INFO:teuthology.orchestra.run.smithi160.stderr: -144> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.488 INFO:teuthology.orchestra.run.smithi160.stderr: -143> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.488 INFO:teuthology.orchestra.run.smithi160.stderr: -142> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.489 INFO:teuthology.orchestra.run.smithi160.stderr: -141> 2022-09-02T06:00:05.139+0000 7f9122131700 4 set_mon_vals no callback set 2022-09-02T06:00:05.489 INFO:teuthology.orchestra.run.smithi160.stderr: -140> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: dump: 2022-09-02T06:00:05.489 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.489 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.490 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.490 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.490 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.490 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.491 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.491 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.491 INFO:teuthology.orchestra.run.smithi160.stderr: -139> 2022-09-02T06:00:05.139+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6684 2022-09-02T06:00:05.491 INFO:teuthology.orchestra.run.smithi160.stderr: -138> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.492 INFO:teuthology.orchestra.run.smithi160.stderr: -137> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.492 INFO:teuthology.orchestra.run.smithi160.stderr: -136> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.492 INFO:teuthology.orchestra.run.smithi160.stderr: -135> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.492 INFO:teuthology.orchestra.run.smithi160.stderr: -134> 2022-09-02T06:00:05.139+0000 7f913a1b1600 1 librados: init done 2022-09-02T06:00:05.493 INFO:teuthology.orchestra.run.smithi160.stderr: -133> 2022-09-02T06:00:05.139+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cr dump hook 0x555a34bb2dd8 2022-09-02T06:00:05.493 INFO:teuthology.orchestra.run.smithi160.stderr: -132> 2022-09-02T06:00:05.140+0000 7f912092e700 4 mgrc handle_mgr_map Got map version 5 2022-09-02T06:00:05.493 INFO:teuthology.orchestra.run.smithi160.stderr: -131> 2022-09-02T06:00:05.140+0000 7f912092e700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.494 INFO:teuthology.orchestra.run.smithi160.stderr: -130> 2022-09-02T06:00:05.140+0000 7f912092e700 4 mgrc reconnect Starting new session with [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.494 INFO:teuthology.orchestra.run.smithi160.stderr: -129> 2022-09-02T06:00:05.140+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34de7400 auth_method 0 2022-09-02T06:00:05.494 INFO:teuthology.orchestra.run.smithi160.stderr: -128> 2022-09-02T06:00:05.140+0000 7f913895e700 10 monclient: get_auth_request con 0x555a34d94000 auth_method 0 2022-09-02T06:00:05.494 INFO:teuthology.orchestra.run.smithi160.stderr: -127> 2022-09-02T06:00:05.142+0000 7f913815d700 10 monclient: get_auth_request con 0x555a34a43800 auth_method 0 2022-09-02T06:00:05.495 INFO:teuthology.orchestra.run.smithi160.stderr: -126> 2022-09-02T06:00:05.145+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34d60400 auth_method 0 2022-09-02T06:00:05.495 INFO:teuthology.orchestra.run.smithi160.stderr: -125> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace show hook 0x555a34a6b420 2022-09-02T06:00:05.495 INFO:teuthology.orchestra.run.smithi160.stderr: -124> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace history hook 0x555a34a6b420 2022-09-02T06:00:05.496 INFO:teuthology.orchestra.run.smithi160.stderr: -123> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace active hook 0x555a34a6b420 2022-09-02T06:00:05.496 INFO:teuthology.orchestra.run.smithi160.stderr: -122> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace active_short hook 0x555a34a6b420 2022-09-02T06:00:05.496 INFO:teuthology.orchestra.run.smithi160.stderr: -121> 2022-09-02T06:00:05.210+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2022-09-02T06:00:05.497 INFO:teuthology.orchestra.run.smithi160.stderr: -120> 2022-09-02T06:00:05.215+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: waiting on bucket lease 2022-09-02T06:00:05.497 INFO:teuthology.orchestra.run.smithi160.stderr: -119> 2022-09-02T06:00:05.216+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: status after acquiring the lock is: init 2022-09-02T06:00:05.497 INFO:teuthology.orchestra.run.smithi160.stderr: -118> 2022-09-02T06:00:05.240+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: listing bucket for full sync 2022-09-02T06:00:05.498 INFO:teuthology.orchestra.run.smithi160.stderr: -117> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj11 2022-09-02T06:00:05.498 INFO:teuthology.orchestra.run.smithi160.stderr: -116> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj11]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj11 op=0 op_state=1 2022-09-02T06:00:05.498 INFO:teuthology.orchestra.run.smithi160.stderr: -115> 2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj11]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj11[0] 2022-09-02T06:00:05.499 INFO:teuthology.orchestra.run.smithi160.stderr: -114> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj111 2022-09-02T06:00:05.499 INFO:teuthology.orchestra.run.smithi160.stderr: -113> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj111 op=0 op_state=1 2022-09-02T06:00:05.499 INFO:teuthology.orchestra.run.smithi160.stderr: -112> 2022-09-02T06:00:05.246+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: : injecting data sync error on key=obj111 2022-09-02T06:00:05.500 INFO:teuthology.orchestra.run.smithi160.stderr: -111> 2022-09-02T06:00:05.246+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.500 INFO:teuthology.orchestra.run.smithi160.stderr: -110> 2022-09-02T06:00:05.246+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj111 2022-09-02T06:00:05.500 INFO:teuthology.orchestra.run.smithi160.stderr: -109> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj115 2022-09-02T06:00:05.501 INFO:teuthology.orchestra.run.smithi160.stderr: -108> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj115 op=0 op_state=1 2022-09-02T06:00:05.501 INFO:teuthology.orchestra.run.smithi160.stderr: -107> 2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj115[0] 2022-09-02T06:00:05.501 INFO:teuthology.orchestra.run.smithi160.stderr: -106> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj116 2022-09-02T06:00:05.502 INFO:teuthology.orchestra.run.smithi160.stderr: -105> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj116 op=0 op_state=1 2022-09-02T06:00:05.502 INFO:teuthology.orchestra.run.smithi160.stderr: -104> 2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj116[0] 2022-09-02T06:00:05.502 INFO:teuthology.orchestra.run.smithi160.stderr: -103> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj124 2022-09-02T06:00:05.502 INFO:teuthology.orchestra.run.smithi160.stderr: -102> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj124]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj124 op=0 op_state=1 2022-09-02T06:00:05.503 INFO:teuthology.orchestra.run.smithi160.stderr: -101> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj124]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj124[0] 2022-09-02T06:00:05.503 INFO:teuthology.orchestra.run.smithi160.stderr: -100> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj145 2022-09-02T06:00:05.504 INFO:teuthology.orchestra.run.smithi160.stderr: -99> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj145 op=0 op_state=1 2022-09-02T06:00:05.504 INFO:teuthology.orchestra.run.smithi160.stderr: -98> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj145[0] 2022-09-02T06:00:05.504 INFO:teuthology.orchestra.run.smithi160.stderr: -97> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj148 2022-09-02T06:00:05.504 INFO:teuthology.orchestra.run.smithi160.stderr: -96> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj148 op=0 op_state=1 2022-09-02T06:00:05.505 INFO:teuthology.orchestra.run.smithi160.stderr: -95> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj148[0] 2022-09-02T06:00:05.505 INFO:teuthology.orchestra.run.smithi160.stderr: -94> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj155 2022-09-02T06:00:05.506 INFO:teuthology.orchestra.run.smithi160.stderr: -93> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj155 op=0 op_state=1 2022-09-02T06:00:05.506 INFO:teuthology.orchestra.run.smithi160.stderr: -92> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: : injecting data sync error on key=obj155 2022-09-02T06:00:05.506 INFO:teuthology.orchestra.run.smithi160.stderr: -91> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.506 INFO:teuthology.orchestra.run.smithi160.stderr: -90> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj155 2022-09-02T06:00:05.507 INFO:teuthology.orchestra.run.smithi160.stderr: -89> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj17 2022-09-02T06:00:05.507 INFO:teuthology.orchestra.run.smithi160.stderr: -88> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj17 op=0 op_state=1 2022-09-02T06:00:05.507 INFO:teuthology.orchestra.run.smithi160.stderr: -87> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj17[0] 2022-09-02T06:00:05.508 INFO:teuthology.orchestra.run.smithi160.stderr: -86> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj185 2022-09-02T06:00:05.508 INFO:teuthology.orchestra.run.smithi160.stderr: -85> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj185]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj185 op=0 op_state=1 2022-09-02T06:00:05.508 INFO:teuthology.orchestra.run.smithi160.stderr: -84> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj185]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj185[0] 2022-09-02T06:00:05.508 INFO:teuthology.orchestra.run.smithi160.stderr: -83> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj19 2022-09-02T06:00:05.509 INFO:teuthology.orchestra.run.smithi160.stderr: -82> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj19 op=0 op_state=1 2022-09-02T06:00:05.509 INFO:teuthology.orchestra.run.smithi160.stderr: -81> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj19[0] 2022-09-02T06:00:05.509 INFO:teuthology.orchestra.run.smithi160.stderr: -80> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj22 2022-09-02T06:00:05.509 INFO:teuthology.orchestra.run.smithi160.stderr: -79> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj22 op=0 op_state=1 2022-09-02T06:00:05.510 INFO:teuthology.orchestra.run.smithi160.stderr: -78> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: : injecting data sync error on key=obj22 2022-09-02T06:00:05.510 INFO:teuthology.orchestra.run.smithi160.stderr: -77> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.510 INFO:teuthology.orchestra.run.smithi160.stderr: -76> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj22 2022-09-02T06:00:05.511 INFO:teuthology.orchestra.run.smithi160.stderr: -75> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj23 2022-09-02T06:00:05.511 INFO:teuthology.orchestra.run.smithi160.stderr: -74> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj23]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj23 op=0 op_state=1 2022-09-02T06:00:05.511 INFO:teuthology.orchestra.run.smithi160.stderr: -73> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj23]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj23[0] 2022-09-02T06:00:05.511 INFO:teuthology.orchestra.run.smithi160.stderr: -72> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj29 2022-09-02T06:00:05.512 INFO:teuthology.orchestra.run.smithi160.stderr: -71> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj29 op=0 op_state=1 2022-09-02T06:00:05.512 INFO:teuthology.orchestra.run.smithi160.stderr: -70> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: : injecting data sync error on key=obj29 2022-09-02T06:00:05.512 INFO:teuthology.orchestra.run.smithi160.stderr: -69> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.513 INFO:teuthology.orchestra.run.smithi160.stderr: -68> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj29 2022-09-02T06:00:05.513 INFO:teuthology.orchestra.run.smithi160.stderr: -67> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj31 2022-09-02T06:00:05.513 INFO:teuthology.orchestra.run.smithi160.stderr: -66> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj31 op=0 op_state=1 2022-09-02T06:00:05.513 INFO:teuthology.orchestra.run.smithi160.stderr: -65> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj31[0] 2022-09-02T06:00:05.514 INFO:teuthology.orchestra.run.smithi160.stderr: -64> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj33 2022-09-02T06:00:05.514 INFO:teuthology.orchestra.run.smithi160.stderr: -63> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj33]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj33 op=0 op_state=1 2022-09-02T06:00:05.515 INFO:teuthology.orchestra.run.smithi160.stderr: -62> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj33]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj33[0] 2022-09-02T06:00:05.515 INFO:teuthology.orchestra.run.smithi160.stderr: -61> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj34 2022-09-02T06:00:05.515 INFO:teuthology.orchestra.run.smithi160.stderr: -60> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj34 op=0 op_state=1 2022-09-02T06:00:05.515 INFO:teuthology.orchestra.run.smithi160.stderr: -59> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: : injecting data sync error on key=obj34 2022-09-02T06:00:05.516 INFO:teuthology.orchestra.run.smithi160.stderr: -58> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.516 INFO:teuthology.orchestra.run.smithi160.stderr: -57> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj34 2022-09-02T06:00:05.516 INFO:teuthology.orchestra.run.smithi160.stderr: -56> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj37 2022-09-02T06:00:05.516 INFO:teuthology.orchestra.run.smithi160.stderr: -55> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj37 op=0 op_state=1 2022-09-02T06:00:05.517 INFO:teuthology.orchestra.run.smithi160.stderr: -54> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj37[0] 2022-09-02T06:00:05.517 INFO:teuthology.orchestra.run.smithi160.stderr: -53> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj38 2022-09-02T06:00:05.517 INFO:teuthology.orchestra.run.smithi160.stderr: -52> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj38]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj38 op=0 op_state=1 2022-09-02T06:00:05.517 INFO:teuthology.orchestra.run.smithi160.stderr: -51> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj38]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj38[0] 2022-09-02T06:00:05.518 INFO:teuthology.orchestra.run.smithi160.stderr: -50> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj44 2022-09-02T06:00:05.518 INFO:teuthology.orchestra.run.smithi160.stderr: -49> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj44 op=0 op_state=1 2022-09-02T06:00:05.518 INFO:teuthology.orchestra.run.smithi160.stderr: -48> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj44[0] 2022-09-02T06:00:05.519 INFO:teuthology.orchestra.run.smithi160.stderr: -47> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj46 2022-09-02T06:00:05.519 INFO:teuthology.orchestra.run.smithi160.stderr: -46> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj46]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj46 op=0 op_state=1 2022-09-02T06:00:05.519 INFO:teuthology.orchestra.run.smithi160.stderr: -45> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj46]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj46[0] 2022-09-02T06:00:05.519 INFO:teuthology.orchestra.run.smithi160.stderr: -44> 2022-09-02T06:00:05.250+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: ERROR: full sync failed. error: -5 2022-09-02T06:00:05.520 INFO:teuthology.orchestra.run.smithi160.stderr: -43> 2022-09-02T06:00:05.251+0000 7f911d106700 1 get compressor lz4 = 0 2022-09-02T06:00:05.520 INFO:teuthology.orchestra.run.smithi160.stderr: -42> 2022-09-02T06:00:05.251+0000 7f911d106700 1 load compressor lz4 2022-09-02T06:00:05.520 INFO:teuthology.orchestra.run.smithi160.stderr: -41> 2022-09-02T06:00:05.251+0000 7f911d106700 1 add compressor lz4 0x555a35784570 2022-09-02T06:00:05.521 INFO:teuthology.orchestra.run.smithi160.stderr: -40> 2022-09-02T06:00:05.251+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.521 INFO:teuthology.orchestra.run.smithi160.stderr: -39> 2022-09-02T06:00:05.251+0000 7f911d106700 1 load: compressor lz4 loaded and registered 2022-09-02T06:00:05.521 INFO:teuthology.orchestra.run.smithi160.stderr: -38> 2022-09-02T06:00:05.251+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.521 INFO:teuthology.orchestra.run.smithi160.stderr: -37> 2022-09-02T06:00:05.252+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.522 INFO:teuthology.orchestra.run.smithi160.stderr: -36> 2022-09-02T06:00:05.252+0000 7f911d106700 1 get compressor zlib = 0 2022-09-02T06:00:05.522 INFO:teuthology.orchestra.run.smithi160.stderr: -35> 2022-09-02T06:00:05.252+0000 7f911d106700 1 load compressor zlib 2022-09-02T06:00:05.523 INFO:teuthology.orchestra.run.smithi160.stderr: -34> 2022-09-02T06:00:05.253+0000 7f911d106700 1 add compressor zlib 0x555a357853e0 2022-09-02T06:00:05.523 INFO:teuthology.orchestra.run.smithi160.stderr: -33> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor zlib = 0x555a357853e0 2022-09-02T06:00:05.523 INFO:teuthology.orchestra.run.smithi160.stderr: -32> 2022-09-02T06:00:05.253+0000 7f911d106700 1 load: compressor zlib loaded and registered 2022-09-02T06:00:05.524 INFO:teuthology.orchestra.run.smithi160.stderr: -31> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor zlib = 0x555a357853e0 2022-09-02T06:00:05.524 INFO:teuthology.orchestra.run.smithi160.stderr: -30> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor snappy = 0 2022-09-02T06:00:05.524 INFO:teuthology.orchestra.run.smithi160.stderr: -29> 2022-09-02T06:00:05.253+0000 7f911d106700 1 load compressor snappy 2022-09-02T06:00:05.524 INFO:teuthology.orchestra.run.smithi160.stderr: -28> 2022-09-02T06:00:05.253+0000 7f911d106700 1 add compressor snappy 0x555a357846c0 2022-09-02T06:00:05.525 INFO:teuthology.orchestra.run.smithi160.stderr: -27> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.525 INFO:teuthology.orchestra.run.smithi160.stderr: -26> 2022-09-02T06:00:05.253+0000 7f911d106700 1 load: compressor snappy loaded and registered 2022-09-02T06:00:05.526 INFO:teuthology.orchestra.run.smithi160.stderr: -25> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.526 INFO:teuthology.orchestra.run.smithi160.stderr: -24> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.526 INFO:teuthology.orchestra.run.smithi160.stderr: -23> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.527 INFO:teuthology.orchestra.run.smithi160.stderr: -22> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.527 INFO:teuthology.orchestra.run.smithi160.stderr: -21> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor zstd = 0 2022-09-02T06:00:05.527 INFO:teuthology.orchestra.run.smithi160.stderr: -20> 2022-09-02T06:00:05.254+0000 7f911d106700 1 load compressor zstd 2022-09-02T06:00:05.527 INFO:teuthology.orchestra.run.smithi160.stderr: -19> 2022-09-02T06:00:05.255+0000 7f911d106700 1 add compressor zstd 0x555a35785b90 2022-09-02T06:00:05.528 INFO:teuthology.orchestra.run.smithi160.stderr: -18> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.528 INFO:teuthology.orchestra.run.smithi160.stderr: -17> 2022-09-02T06:00:05.255+0000 7f911d106700 1 load: compressor zstd loaded and registered 2022-09-02T06:00:05.528 INFO:teuthology.orchestra.run.smithi160.stderr: -16> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.528 INFO:teuthology.orchestra.run.smithi160.stderr: -15> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.529 INFO:teuthology.orchestra.run.smithi160.stderr: -14> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.529 INFO:teuthology.orchestra.run.smithi160.stderr: -13> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.529 INFO:teuthology.orchestra.run.smithi160.stderr: -12> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.529 INFO:teuthology.orchestra.run.smithi160.stderr: -11> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.530 INFO:teuthology.orchestra.run.smithi160.stderr: -10> 2022-09-02T06:00:05.256+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.530 INFO:teuthology.orchestra.run.smithi160.stderr: -9> 2022-09-02T06:00:05.268+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: success 2022-09-02T06:00:05.530 INFO:teuthology.orchestra.run.smithi160.stderr: -8> 2022-09-02T06:00:05.274+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: success 2022-09-02T06:00:05.531 INFO:teuthology.orchestra.run.smithi160.stderr: -7> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: success 2022-09-02T06:00:05.531 INFO:teuthology.orchestra.run.smithi160.stderr: -6> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: success 2022-09-02T06:00:05.531 INFO:teuthology.orchestra.run.smithi160.stderr: -5> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: success 2022-09-02T06:00:05.532 INFO:teuthology.orchestra.run.smithi160.stderr: -4> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: success 2022-09-02T06:00:05.532 INFO:teuthology.orchestra.run.smithi160.stderr: -3> 2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: success 2022-09-02T06:00:05.532 INFO:teuthology.orchestra.run.smithi160.stderr: -2> 2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: success 2022-09-02T06:00:05.533 INFO:teuthology.orchestra.run.smithi160.stderr: -1> 2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: success 2022-09-02T06:00:05.533 INFO:teuthology.orchestra.run.smithi160.stderr: 0> 2022-09-02T06:00:05.280+0000 7f913a1b1600 -1 *** Caught signal (Segmentation fault) ** 2022-09-02T06:00:05.533 INFO:teuthology.orchestra.run.smithi160.stderr: in thread 7f913a1b1600 thread_name:radosgw-admin 2022-09-02T06:00:05.533 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.534 INFO:teuthology.orchestra.run.smithi160.stderr: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:00:05.534 INFO:teuthology.orchestra.run.smithi160.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9141977cf0] 2022-09-02T06:00:05.534 INFO:teuthology.orchestra.run.smithi160.stderr: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x1c2) [0x7f913f9757d2] 2022-09-02T06:00:05.534 INFO:teuthology.orchestra.run.smithi160.stderr: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x555a31973e49] 2022-09-02T06:00:05.535 INFO:teuthology.orchestra.run.smithi160.stderr: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x555a3199a76e] 2022-09-02T06:00:05.535 INFO:teuthology.orchestra.run.smithi160.stderr: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x555a31a9440f] 2022-09-02T06:00:05.535 INFO:teuthology.orchestra.run.smithi160.stderr: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x555a31a94be1] 2022-09-02T06:00:05.535 INFO:teuthology.orchestra.run.smithi160.stderr: 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x555a31944caa] 2022-09-02T06:00:05.536 INFO:teuthology.orchestra.run.smithi160.stderr: 8: main() 2022-09-02T06:00:05.536 INFO:teuthology.orchestra.run.smithi160.stderr: 9: __libc_start_main() 2022-09-02T06:00:05.536 INFO:teuthology.orchestra.run.smithi160.stderr: 10: _start() 2022-09-02T06:00:05.536 INFO:teuthology.orchestra.run.smithi160.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-02T06:00:05.537 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.537 INFO:teuthology.orchestra.run.smithi160.stderr:--- logging levels --- 2022-09-02T06:00:05.537 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 none 2022-09-02T06:00:05.537 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 lockdep 2022-09-02T06:00:05.538 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 context 2022-09-02T06:00:05.538 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 crush 2022-09-02T06:00:05.538 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds 2022-09-02T06:00:05.539 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_balancer 2022-09-02T06:00:05.539 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_locker 2022-09-02T06:00:05.539 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_log 2022-09-02T06:00:05.539 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_log_expire 2022-09-02T06:00:05.540 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_migrator 2022-09-02T06:00:05.540 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 buffer 2022-09-02T06:00:05.540 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 timer 2022-09-02T06:00:05.541 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 filer 2022-09-02T06:00:05.541 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 striper 2022-09-02T06:00:05.541 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 objecter 2022-09-02T06:00:05.541 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rados 2022-09-02T06:00:05.542 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd 2022-09-02T06:00:05.542 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd_mirror 2022-09-02T06:00:05.543 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd_replay 2022-09-02T06:00:05.543 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd_pwl 2022-09-02T06:00:05.543 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 journaler 2022-09-02T06:00:05.543 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 objectcacher 2022-09-02T06:00:05.544 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 immutable_obj_cache 2022-09-02T06:00:05.544 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 client 2022-09-02T06:00:05.544 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 osd 2022-09-02T06:00:05.544 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 optracker 2022-09-02T06:00:05.545 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 objclass 2022-09-02T06:00:05.545 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 3 filestore 2022-09-02T06:00:05.545 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 3 journal 2022-09-02T06:00:05.545 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 0 ms 2022-09-02T06:00:05.546 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mon 2022-09-02T06:00:05.546 INFO:teuthology.orchestra.run.smithi160.stderr: 0/10 monc 2022-09-02T06:00:05.546 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 paxos 2022-09-02T06:00:05.547 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 tp 2022-09-02T06:00:05.547 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 auth 2022-09-02T06:00:05.547 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 crypto 2022-09-02T06:00:05.547 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 finisher 2022-09-02T06:00:05.548 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 reserver 2022-09-02T06:00:05.548 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 heartbeatmap 2022-09-02T06:00:05.548 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 perfcounter 2022-09-02T06:00:05.548 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 0 rgw 2022-09-02T06:00:05.549 INFO:teuthology.orchestra.run.smithi160.stderr: 20/20 rgw_sync 2022-09-02T06:00:05.549 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 rgw_datacache 2022-09-02T06:00:05.549 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 rgw_access 2022-09-02T06:00:05.550 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 javaclient 2022-09-02T06:00:05.550 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 asok 2022-09-02T06:00:05.550 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 throttle 2022-09-02T06:00:05.551 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 0 refs 2022-09-02T06:00:05.551 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 compressor 2022-09-02T06:00:05.551 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 bluestore 2022-09-02T06:00:05.552 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 bluefs 2022-09-02T06:00:05.552 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 3 bdev 2022-09-02T06:00:05.552 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 kstore 2022-09-02T06:00:05.552 INFO:teuthology.orchestra.run.smithi160.stderr: 4/ 5 rocksdb 2022-09-02T06:00:05.553 INFO:teuthology.orchestra.run.smithi160.stderr: 4/ 5 leveldb 2022-09-02T06:00:05.553 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 fuse 2022-09-02T06:00:05.553 INFO:teuthology.orchestra.run.smithi160.stderr: 2/ 5 mgr 2022-09-02T06:00:05.554 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mgrc 2022-09-02T06:00:05.554 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 dpdk 2022-09-02T06:00:05.554 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 eventtrace 2022-09-02T06:00:05.554 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 prioritycache 2022-09-02T06:00:05.555 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 test 2022-09-02T06:00:05.555 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 cephfs_mirror 2022-09-02T06:00:05.555 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 cephsqlite 2022-09-02T06:00:05.555 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore 2022-09-02T06:00:05.556 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_onode 2022-09-02T06:00:05.556 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_odata 2022-09-02T06:00:05.556 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_omap 2022-09-02T06:00:05.556 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_tm 2022-09-02T06:00:05.557 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_t 2022-09-02T06:00:05.557 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_cleaner 2022-09-02T06:00:05.557 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_lba 2022-09-02T06:00:05.558 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_fixedkv_tree 2022-09-02T06:00:05.558 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_cache 2022-09-02T06:00:05.558 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_journal 2022-09-02T06:00:05.559 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_device 2022-09-02T06:00:05.559 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_backref 2022-09-02T06:00:05.559 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 alienstore 2022-09-02T06:00:05.559 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mclock 2022-09-02T06:00:05.560 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 cyanstore 2022-09-02T06:00:05.560 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 ceph_exporter 2022-09-02T06:00:05.560 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 memstore 2022-09-02T06:00:05.560 INFO:teuthology.orchestra.run.smithi160.stderr: -2/-2 (syslog threshold) 2022-09-02T06:00:05.561 INFO:teuthology.orchestra.run.smithi160.stderr: 99/99 (stderr threshold) 2022-09-02T06:00:05.561 INFO:teuthology.orchestra.run.smithi160.stderr:--- pthread ID / name mapping for recent threads --- 2022-09-02T06:00:05.561 INFO:teuthology.orchestra.run.smithi160.stderr: 7f911d106700 / http_manager 2022-09-02T06:00:05.561 INFO:teuthology.orchestra.run.smithi160.stderr: 7f912092e700 / ms_dispatch 2022-09-02T06:00:05.562 INFO:teuthology.orchestra.run.smithi160.stderr: 7f9122131700 / io_context_pool 2022-09-02T06:00:05.562 INFO:teuthology.orchestra.run.smithi160.stderr: 7f9134956700 / ms_dispatch 2022-09-02T06:00:05.562 INFO:teuthology.orchestra.run.smithi160.stderr: 7f9136159700 / io_context_pool 2022-09-02T06:00:05.562 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913715b700 / admin_socket 2022-09-02T06:00:05.563 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913795c700 / msgr-worker-2 2022-09-02T06:00:05.563 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913815d700 / msgr-worker-1 2022-09-02T06:00:05.563 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913895e700 / msgr-worker-0 2022-09-02T06:00:05.563 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913a1b1600 / radosgw-admin 2022-09-02T06:00:05.564 INFO:teuthology.orchestra.run.smithi160.stderr: max_recent 500 2022-09-02T06:00:05.564 INFO:teuthology.orchestra.run.smithi160.stderr: max_new 1000 2022-09-02T06:00:05.564 INFO:teuthology.orchestra.run.smithi160.stderr: log_file /var/log/ceph/c2-client.0.153981.log 2022-09-02T06:00:05.564 INFO:teuthology.orchestra.run.smithi160.stderr:--- end dump of recent events --- 2022-09-02T06:00:05.565 INFO:teuthology.orchestra.run.smithi160.stderr:--- begin dump of recent events --- 2022-09-02T06:00:05.565 INFO:teuthology.orchestra.run.smithi160.stderr: -499> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.565 INFO:teuthology.orchestra.run.smithi160.stderr: -498> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.566 INFO:teuthology.orchestra.run.smithi160.stderr: -497> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.566 INFO:teuthology.orchestra.run.smithi160.stderr: -496> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.566 INFO:teuthology.orchestra.run.smithi160.stderr: -495> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.567 INFO:teuthology.orchestra.run.smithi160.stderr: -494> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.567 INFO:teuthology.orchestra.run.smithi160.stderr: -493> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.567 INFO:teuthology.orchestra.run.smithi160.stderr: -492> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.567 INFO:teuthology.orchestra.run.smithi160.stderr: -491> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.568 INFO:teuthology.orchestra.run.smithi160.stderr: -490> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.568 INFO:teuthology.orchestra.run.smithi160.stderr: -489> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: crc 2022-09-02T06:00:05.568 INFO:teuthology.orchestra.run.smithi160.stderr: -488> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb284e0) adding con mode: secure 2022-09-02T06:00:05.569 INFO:teuthology.orchestra.run.smithi160.stderr: -487> 2022-09-02T06:00:05.114+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.569 INFO:teuthology.orchestra.run.smithi160.stderr: -486> 2022-09-02T06:00:05.114+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.569 INFO:teuthology.orchestra.run.smithi160.stderr: -485> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.569 INFO:teuthology.orchestra.run.smithi160.stderr: -484> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient: _add_conns ranks=[1,0] 2022-09-02T06:00:05.570 INFO:teuthology.orchestra.run.smithi160.stderr: -483> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34a42800 addr v2:172.21.15.160:3300/0 2022-09-02T06:00:05.570 INFO:teuthology.orchestra.run.smithi160.stderr: -482> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a-legacy con 0x555a34a42c00 addr v1:172.21.15.160:6789/0 2022-09-02T06:00:05.570 INFO:teuthology.orchestra.run.smithi160.stderr: -481> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.571 INFO:teuthology.orchestra.run.smithi160.stderr: -480> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.571 INFO:teuthology.orchestra.run.smithi160.stderr: -479> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.755205s 2022-09-02T06:00:05.571 INFO:teuthology.orchestra.run.smithi160.stderr: -478> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34a42800 auth_method 0 2022-09-02T06:00:05.571 INFO:teuthology.orchestra.run.smithi160.stderr: -477> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.572 INFO:teuthology.orchestra.run.smithi160.stderr: -476> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.572 INFO:teuthology.orchestra.run.smithi160.stderr: -475> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.572 INFO:teuthology.orchestra.run.smithi160.stderr: -474> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.572 INFO:teuthology.orchestra.run.smithi160.stderr: -473> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.573 INFO:teuthology.orchestra.run.smithi160.stderr: -472> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.573 INFO:teuthology.orchestra.run.smithi160.stderr: -471> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6679 payload 274 2022-09-02T06:00:05.573 INFO:teuthology.orchestra.run.smithi160.stderr: -470> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.573 INFO:teuthology.orchestra.run.smithi160.stderr: -469> 2022-09-02T06:00:05.114+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.574 INFO:teuthology.orchestra.run.smithi160.stderr: -468> 2022-09-02T06:00:05.114+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.574 INFO:teuthology.orchestra.run.smithi160.stderr: -467> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.574 INFO:teuthology.orchestra.run.smithi160.stderr: -466> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.575 INFO:teuthology.orchestra.run.smithi160.stderr: -465> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: dump: 2022-09-02T06:00:05.575 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.575 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.575 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.576 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.576 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.576 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.577 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.577 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.577 INFO:teuthology.orchestra.run.smithi160.stderr: -464> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.577 INFO:teuthology.orchestra.run.smithi160.stderr: -463> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.578 INFO:teuthology.orchestra.run.smithi160.stderr: -462> 2022-09-02T06:00:05.114+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6679 2022-09-02T06:00:05.578 INFO:teuthology.orchestra.run.smithi160.stderr: -461> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.578 INFO:teuthology.orchestra.run.smithi160.stderr: -460> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.578 INFO:teuthology.orchestra.run.smithi160.stderr: -459> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.579 INFO:teuthology.orchestra.run.smithi160.stderr: -458> 2022-09-02T06:00:05.114+0000 7f913715b700 10 monclient: dump: 2022-09-02T06:00:05.579 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.579 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.579 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.580 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.580 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.580 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.581 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.581 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.581 INFO:teuthology.orchestra.run.smithi160.stderr: -457> 2022-09-02T06:00:05.114+0000 7f913a1b1600 10 monclient: get_monmap_and_config success 2022-09-02T06:00:05.581 INFO:teuthology.orchestra.run.smithi160.stderr: -456> 2022-09-02T06:00:05.114+0000 7f913a1b1600 4 set_mon_vals no callback set 2022-09-02T06:00:05.582 INFO:teuthology.orchestra.run.smithi160.stderr: -455> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: shutdown 2022-09-02T06:00:05.582 INFO:teuthology.orchestra.run.smithi160.stderr: -454> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) init /var/run/ceph/c2-client.0.153981.asok 2022-09-02T06:00:05.582 INFO:teuthology.orchestra.run.smithi160.stderr: -453> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) bind_and_listen /var/run/ceph/c2-client.0.153981.asok 2022-09-02T06:00:05.583 INFO:teuthology.orchestra.run.smithi160.stderr: -452> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command 0 hook 0x555a34a93378 2022-09-02T06:00:05.583 INFO:teuthology.orchestra.run.smithi160.stderr: -451> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command version hook 0x555a34a93378 2022-09-02T06:00:05.583 INFO:teuthology.orchestra.run.smithi160.stderr: -450> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command git_version hook 0x555a34a93378 2022-09-02T06:00:05.583 INFO:teuthology.orchestra.run.smithi160.stderr: -449> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command help hook 0x555a34a36bf0 2022-09-02T06:00:05.584 INFO:teuthology.orchestra.run.smithi160.stderr: -448> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command get_command_descriptions hook 0x555a34a36b50 2022-09-02T06:00:05.584 INFO:teuthology.orchestra.run.smithi160.stderr: -447> 2022-09-02T06:00:05.115+0000 7f913715b700 5 asok(0x555a34ad6000) entry start 2022-09-02T06:00:05.584 INFO:teuthology.orchestra.run.smithi160.stderr: -446> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: get_monmap_and_config 2022-09-02T06:00:05.585 INFO:teuthology.orchestra.run.smithi160.stderr: -445> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.585 INFO:teuthology.orchestra.run.smithi160.stderr: -444> 2022-09-02T06:00:05.115+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.585 INFO:teuthology.orchestra.run.smithi160.stderr: -443> 2022-09-02T06:00:05.115+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.585 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.586 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.586 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.586 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.586 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.587 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.587 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.587 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.587 INFO:teuthology.orchestra.run.smithi160.stderr: -442> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.588 INFO:teuthology.orchestra.run.smithi160.stderr: -441> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.588 INFO:teuthology.orchestra.run.smithi160.stderr: -440> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.588 INFO:teuthology.orchestra.run.smithi160.stderr: -439> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.588 INFO:teuthology.orchestra.run.smithi160.stderr: -438> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.589 INFO:teuthology.orchestra.run.smithi160.stderr: -437> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.589 INFO:teuthology.orchestra.run.smithi160.stderr: -436> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.589 INFO:teuthology.orchestra.run.smithi160.stderr: -435> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.589 INFO:teuthology.orchestra.run.smithi160.stderr: -434> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.590 INFO:teuthology.orchestra.run.smithi160.stderr: -433> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.590 INFO:teuthology.orchestra.run.smithi160.stderr: -432> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.590 INFO:teuthology.orchestra.run.smithi160.stderr: -431> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.591 INFO:teuthology.orchestra.run.smithi160.stderr: -430> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.591 INFO:teuthology.orchestra.run.smithi160.stderr: -429> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.591 INFO:teuthology.orchestra.run.smithi160.stderr: -428> 2022-09-02T06:00:05.115+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.591 INFO:teuthology.orchestra.run.smithi160.stderr: -427> 2022-09-02T06:00:05.115+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.592 INFO:teuthology.orchestra.run.smithi160.stderr: -426> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.592 INFO:teuthology.orchestra.run.smithi160.stderr: -425> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding auth protocol: cephx 2022-09-02T06:00:05.592 INFO:teuthology.orchestra.run.smithi160.stderr: -424> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding auth protocol: cephx 2022-09-02T06:00:05.592 INFO:teuthology.orchestra.run.smithi160.stderr: -423> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding auth protocol: cephx 2022-09-02T06:00:05.593 INFO:teuthology.orchestra.run.smithi160.stderr: -422> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.593 INFO:teuthology.orchestra.run.smithi160.stderr: -421> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.593 INFO:teuthology.orchestra.run.smithi160.stderr: -420> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.593 INFO:teuthology.orchestra.run.smithi160.stderr: -419> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.594 INFO:teuthology.orchestra.run.smithi160.stderr: -418> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.594 INFO:teuthology.orchestra.run.smithi160.stderr: -417> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.594 INFO:teuthology.orchestra.run.smithi160.stderr: -416> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.595 INFO:teuthology.orchestra.run.smithi160.stderr: -415> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.595 INFO:teuthology.orchestra.run.smithi160.stderr: -414> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.595 INFO:teuthology.orchestra.run.smithi160.stderr: -413> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.595 INFO:teuthology.orchestra.run.smithi160.stderr: -412> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: crc 2022-09-02T06:00:05.596 INFO:teuthology.orchestra.run.smithi160.stderr: -411> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb28e20) adding con mode: secure 2022-09-02T06:00:05.596 INFO:teuthology.orchestra.run.smithi160.stderr: -410> 2022-09-02T06:00:05.116+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.596 INFO:teuthology.orchestra.run.smithi160.stderr: -409> 2022-09-02T06:00:05.116+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.596 INFO:teuthology.orchestra.run.smithi160.stderr: -408> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.597 INFO:teuthology.orchestra.run.smithi160.stderr: -407> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.597 INFO:teuthology.orchestra.run.smithi160.stderr: -406> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34a42800 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.597 INFO:teuthology.orchestra.run.smithi160.stderr: -405> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.597 INFO:teuthology.orchestra.run.smithi160.stderr: -404> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.598 INFO:teuthology.orchestra.run.smithi160.stderr: -403> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.757161s 2022-09-02T06:00:05.598 INFO:teuthology.orchestra.run.smithi160.stderr: -402> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34a42800 auth_method 0 2022-09-02T06:00:05.598 INFO:teuthology.orchestra.run.smithi160.stderr: -401> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.599 INFO:teuthology.orchestra.run.smithi160.stderr: -400> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.599 INFO:teuthology.orchestra.run.smithi160.stderr: -399> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.599 INFO:teuthology.orchestra.run.smithi160.stderr: -398> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.599 INFO:teuthology.orchestra.run.smithi160.stderr: -397> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.600 INFO:teuthology.orchestra.run.smithi160.stderr: -396> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.600 INFO:teuthology.orchestra.run.smithi160.stderr: -395> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6681 payload 274 2022-09-02T06:00:05.600 INFO:teuthology.orchestra.run.smithi160.stderr: -394> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.601 INFO:teuthology.orchestra.run.smithi160.stderr: -393> 2022-09-02T06:00:05.116+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.601 INFO:teuthology.orchestra.run.smithi160.stderr: -392> 2022-09-02T06:00:05.116+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.601 INFO:teuthology.orchestra.run.smithi160.stderr: -391> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.601 INFO:teuthology.orchestra.run.smithi160.stderr: -390> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.602 INFO:teuthology.orchestra.run.smithi160.stderr: -389> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: dump: 2022-09-02T06:00:05.602 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.602 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.602 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.603 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.603 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.603 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.603 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.604 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.604 INFO:teuthology.orchestra.run.smithi160.stderr: -388> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.604 INFO:teuthology.orchestra.run.smithi160.stderr: -387> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.604 INFO:teuthology.orchestra.run.smithi160.stderr: -386> 2022-09-02T06:00:05.116+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6681 2022-09-02T06:00:05.605 INFO:teuthology.orchestra.run.smithi160.stderr: -385> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.605 INFO:teuthology.orchestra.run.smithi160.stderr: -384> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.605 INFO:teuthology.orchestra.run.smithi160.stderr: -383> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.605 INFO:teuthology.orchestra.run.smithi160.stderr: -382> 2022-09-02T06:00:05.116+0000 7f9136159700 10 monclient: dump: 2022-09-02T06:00:05.606 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.606 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.606 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.606 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.607 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.607 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.607 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.607 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.608 INFO:teuthology.orchestra.run.smithi160.stderr: -381> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: get_monmap_and_config success 2022-09-02T06:00:05.608 INFO:teuthology.orchestra.run.smithi160.stderr: -380> 2022-09-02T06:00:05.116+0000 7f913a1b1600 4 set_mon_vals no callback set 2022-09-02T06:00:05.608 INFO:teuthology.orchestra.run.smithi160.stderr: -379> 2022-09-02T06:00:05.116+0000 7f913a1b1600 10 monclient: shutdown 2022-09-02T06:00:05.609 INFO:teuthology.orchestra.run.smithi160.stderr: -378> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.609 INFO:teuthology.orchestra.run.smithi160.stderr: -377> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.609 INFO:teuthology.orchestra.run.smithi160.stderr: -376> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.610 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.610 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.610 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.610 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.611 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.611 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.611 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.612 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.612 INFO:teuthology.orchestra.run.smithi160.stderr: -375> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.612 INFO:teuthology.orchestra.run.smithi160.stderr: -374> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.612 INFO:teuthology.orchestra.run.smithi160.stderr: -373> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding auth protocol: cephx 2022-09-02T06:00:05.613 INFO:teuthology.orchestra.run.smithi160.stderr: -372> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.613 INFO:teuthology.orchestra.run.smithi160.stderr: -371> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.613 INFO:teuthology.orchestra.run.smithi160.stderr: -370> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.613 INFO:teuthology.orchestra.run.smithi160.stderr: -369> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.614 INFO:teuthology.orchestra.run.smithi160.stderr: -368> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.614 INFO:teuthology.orchestra.run.smithi160.stderr: -367> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.614 INFO:teuthology.orchestra.run.smithi160.stderr: -366> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.614 INFO:teuthology.orchestra.run.smithi160.stderr: -365> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.615 INFO:teuthology.orchestra.run.smithi160.stderr: -364> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.615 INFO:teuthology.orchestra.run.smithi160.stderr: -363> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.615 INFO:teuthology.orchestra.run.smithi160.stderr: -362> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: crc 2022-09-02T06:00:05.616 INFO:teuthology.orchestra.run.smithi160.stderr: -361> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2a40) adding con mode: secure 2022-09-02T06:00:05.616 INFO:teuthology.orchestra.run.smithi160.stderr: -360> 2022-09-02T06:00:05.117+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.616 INFO:teuthology.orchestra.run.smithi160.stderr: -359> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: starting msgr at 2022-09-02T06:00:05.617 INFO:teuthology.orchestra.run.smithi160.stderr: -358> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: starting objecter 2022-09-02T06:00:05.617 INFO:teuthology.orchestra.run.smithi160.stderr: -357> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command objecter_requests hook 0x555a34a36b90 2022-09-02T06:00:05.617 INFO:teuthology.orchestra.run.smithi160.stderr: -356> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: setting wanted keys 2022-09-02T06:00:05.618 INFO:teuthology.orchestra.run.smithi160.stderr: -355> 2022-09-02T06:00:05.117+0000 7f913a1b1600 1 librados: calling monclient init 2022-09-02T06:00:05.618 INFO:teuthology.orchestra.run.smithi160.stderr: -354> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.618 INFO:teuthology.orchestra.run.smithi160.stderr: -353> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding auth protocol: cephx 2022-09-02T06:00:05.618 INFO:teuthology.orchestra.run.smithi160.stderr: -352> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding auth protocol: cephx 2022-09-02T06:00:05.619 INFO:teuthology.orchestra.run.smithi160.stderr: -351> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding auth protocol: cephx 2022-09-02T06:00:05.619 INFO:teuthology.orchestra.run.smithi160.stderr: -350> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.619 INFO:teuthology.orchestra.run.smithi160.stderr: -349> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.619 INFO:teuthology.orchestra.run.smithi160.stderr: -348> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.620 INFO:teuthology.orchestra.run.smithi160.stderr: -347> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.620 INFO:teuthology.orchestra.run.smithi160.stderr: -346> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.620 INFO:teuthology.orchestra.run.smithi160.stderr: -345> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.621 INFO:teuthology.orchestra.run.smithi160.stderr: -344> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.621 INFO:teuthology.orchestra.run.smithi160.stderr: -343> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.621 INFO:teuthology.orchestra.run.smithi160.stderr: -342> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.621 INFO:teuthology.orchestra.run.smithi160.stderr: -341> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.622 INFO:teuthology.orchestra.run.smithi160.stderr: -340> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: crc 2022-09-02T06:00:05.622 INFO:teuthology.orchestra.run.smithi160.stderr: -339> 2022-09-02T06:00:05.117+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9e0d8) adding con mode: secure 2022-09-02T06:00:05.622 INFO:teuthology.orchestra.run.smithi160.stderr: -338> 2022-09-02T06:00:05.117+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.623 INFO:teuthology.orchestra.run.smithi160.stderr: -337> 2022-09-02T06:00:05.117+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.623 INFO:teuthology.orchestra.run.smithi160.stderr: -336> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.623 INFO:teuthology.orchestra.run.smithi160.stderr: -335> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.623 INFO:teuthology.orchestra.run.smithi160.stderr: -334> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34a42800 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.624 INFO:teuthology.orchestra.run.smithi160.stderr: -333> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.624 INFO:teuthology.orchestra.run.smithi160.stderr: -332> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.624 INFO:teuthology.orchestra.run.smithi160.stderr: -331> 2022-09-02T06:00:05.117+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.758652s 2022-09-02T06:00:05.624 INFO:teuthology.orchestra.run.smithi160.stderr: -330> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34a42800 auth_method 0 2022-09-02T06:00:05.625 INFO:teuthology.orchestra.run.smithi160.stderr: -329> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.625 INFO:teuthology.orchestra.run.smithi160.stderr: -328> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.625 INFO:teuthology.orchestra.run.smithi160.stderr: -327> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.626 INFO:teuthology.orchestra.run.smithi160.stderr: -326> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.626 INFO:teuthology.orchestra.run.smithi160.stderr: -325> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.626 INFO:teuthology.orchestra.run.smithi160.stderr: -324> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.627 INFO:teuthology.orchestra.run.smithi160.stderr: -323> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6682 payload 824 2022-09-02T06:00:05.627 INFO:teuthology.orchestra.run.smithi160.stderr: -322> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.627 INFO:teuthology.orchestra.run.smithi160.stderr: -321> 2022-09-02T06:00:05.117+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.627 INFO:teuthology.orchestra.run.smithi160.stderr: -320> 2022-09-02T06:00:05.117+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.628 INFO:teuthology.orchestra.run.smithi160.stderr: -319> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.628 INFO:teuthology.orchestra.run.smithi160.stderr: -318> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.628 INFO:teuthology.orchestra.run.smithi160.stderr: -317> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: dump: 2022-09-02T06:00:05.628 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.629 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.629 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.629 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.629 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.630 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.630 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.630 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.631 INFO:teuthology.orchestra.run.smithi160.stderr: -316> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.631 INFO:teuthology.orchestra.run.smithi160.stderr: -315> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.631 INFO:teuthology.orchestra.run.smithi160.stderr: -314> 2022-09-02T06:00:05.118+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6682 2022-09-02T06:00:05.631 INFO:teuthology.orchestra.run.smithi160.stderr: -313> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.632 INFO:teuthology.orchestra.run.smithi160.stderr: -312> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.632 INFO:teuthology.orchestra.run.smithi160.stderr: -311> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.632 INFO:teuthology.orchestra.run.smithi160.stderr: -310> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.632 INFO:teuthology.orchestra.run.smithi160.stderr: -309> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.633 INFO:teuthology.orchestra.run.smithi160.stderr: -308> 2022-09-02T06:00:05.118+0000 7f9134956700 10 monclient: dump: 2022-09-02T06:00:05.633 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.633 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.633 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.634 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.634 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.634 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.634 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.635 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.635 INFO:teuthology.orchestra.run.smithi160.stderr: -307> 2022-09-02T06:00:05.118+0000 7f9136159700 4 set_mon_vals no callback set 2022-09-02T06:00:05.635 INFO:teuthology.orchestra.run.smithi160.stderr: -306> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.635 INFO:teuthology.orchestra.run.smithi160.stderr: -305> 2022-09-02T06:00:05.118+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.636 INFO:teuthology.orchestra.run.smithi160.stderr: -304> 2022-09-02T06:00:05.118+0000 7f913a1b1600 1 librados: init done 2022-09-02T06:00:05.636 INFO:teuthology.orchestra.run.smithi160.stderr: -303> 2022-09-02T06:00:05.119+0000 7f9134956700 4 mgrc handle_mgr_map Got map version 5 2022-09-02T06:00:05.636 INFO:teuthology.orchestra.run.smithi160.stderr: -302> 2022-09-02T06:00:05.119+0000 7f9134956700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.637 INFO:teuthology.orchestra.run.smithi160.stderr: -301> 2022-09-02T06:00:05.119+0000 7f9134956700 4 mgrc reconnect Starting new session with [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.637 INFO:teuthology.orchestra.run.smithi160.stderr: -300> 2022-09-02T06:00:05.119+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34a43400 auth_method 0 2022-09-02T06:00:05.637 INFO:teuthology.orchestra.run.smithi160.stderr: -299> 2022-09-02T06:00:05.119+0000 7f913895e700 10 monclient: get_auth_request con 0x555a34a42c00 auth_method 0 2022-09-02T06:00:05.637 INFO:teuthology.orchestra.run.smithi160.stderr: -298> 2022-09-02T06:00:05.120+0000 7f913815d700 10 monclient: get_auth_request con 0x555a34a43c00 auth_method 0 2022-09-02T06:00:05.638 INFO:teuthology.orchestra.run.smithi160.stderr: -297> 2022-09-02T06:00:05.121+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34d60800 auth_method 0 2022-09-02T06:00:05.638 INFO:teuthology.orchestra.run.smithi160.stderr: -296> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding auth protocol: cephx 2022-09-02T06:00:05.638 INFO:teuthology.orchestra.run.smithi160.stderr: -295> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding auth protocol: cephx 2022-09-02T06:00:05.638 INFO:teuthology.orchestra.run.smithi160.stderr: -294> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding auth protocol: cephx 2022-09-02T06:00:05.639 INFO:teuthology.orchestra.run.smithi160.stderr: -293> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.639 INFO:teuthology.orchestra.run.smithi160.stderr: -292> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.639 INFO:teuthology.orchestra.run.smithi160.stderr: -291> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.640 INFO:teuthology.orchestra.run.smithi160.stderr: -290> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.640 INFO:teuthology.orchestra.run.smithi160.stderr: -289> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.640 INFO:teuthology.orchestra.run.smithi160.stderr: -288> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.640 INFO:teuthology.orchestra.run.smithi160.stderr: -287> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.641 INFO:teuthology.orchestra.run.smithi160.stderr: -286> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.641 INFO:teuthology.orchestra.run.smithi160.stderr: -285> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.641 INFO:teuthology.orchestra.run.smithi160.stderr: -284> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.641 INFO:teuthology.orchestra.run.smithi160.stderr: -283> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: crc 2022-09-02T06:00:05.642 INFO:teuthology.orchestra.run.smithi160.stderr: -282> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29830) adding con mode: secure 2022-09-02T06:00:05.642 INFO:teuthology.orchestra.run.smithi160.stderr: -281> 2022-09-02T06:00:05.136+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.643 INFO:teuthology.orchestra.run.smithi160.stderr: -280> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache list hook 0x555a34a36b60 2022-09-02T06:00:05.643 INFO:teuthology.orchestra.run.smithi160.stderr: -279> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache inspect hook 0x555a34a36b60 2022-09-02T06:00:05.643 INFO:teuthology.orchestra.run.smithi160.stderr: -278> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache erase hook 0x555a34a36b60 2022-09-02T06:00:05.643 INFO:teuthology.orchestra.run.smithi160.stderr: -277> 2022-09-02T06:00:05.136+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cache zap hook 0x555a34a36b60 2022-09-02T06:00:05.644 INFO:teuthology.orchestra.run.smithi160.stderr: -276> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: get_monmap_and_config 2022-09-02T06:00:05.644 INFO:teuthology.orchestra.run.smithi160.stderr: -275> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.644 INFO:teuthology.orchestra.run.smithi160.stderr: -274> 2022-09-02T06:00:05.137+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.644 INFO:teuthology.orchestra.run.smithi160.stderr: -273> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.645 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.645 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.645 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.645 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.646 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.646 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.646 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.646 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.647 INFO:teuthology.orchestra.run.smithi160.stderr: -272> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.647 INFO:teuthology.orchestra.run.smithi160.stderr: -271> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.647 INFO:teuthology.orchestra.run.smithi160.stderr: -270> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.647 INFO:teuthology.orchestra.run.smithi160.stderr: -269> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.648 INFO:teuthology.orchestra.run.smithi160.stderr: -268> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.648 INFO:teuthology.orchestra.run.smithi160.stderr: -267> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.648 INFO:teuthology.orchestra.run.smithi160.stderr: -266> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.648 INFO:teuthology.orchestra.run.smithi160.stderr: -265> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.649 INFO:teuthology.orchestra.run.smithi160.stderr: -264> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.649 INFO:teuthology.orchestra.run.smithi160.stderr: -263> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.649 INFO:teuthology.orchestra.run.smithi160.stderr: -262> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.649 INFO:teuthology.orchestra.run.smithi160.stderr: -261> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.650 INFO:teuthology.orchestra.run.smithi160.stderr: -260> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.650 INFO:teuthology.orchestra.run.smithi160.stderr: -259> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.650 INFO:teuthology.orchestra.run.smithi160.stderr: -258> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.651 INFO:teuthology.orchestra.run.smithi160.stderr: -257> 2022-09-02T06:00:05.137+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.651 INFO:teuthology.orchestra.run.smithi160.stderr: -256> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.651 INFO:teuthology.orchestra.run.smithi160.stderr: -255> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding auth protocol: cephx 2022-09-02T06:00:05.652 INFO:teuthology.orchestra.run.smithi160.stderr: -254> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding auth protocol: cephx 2022-09-02T06:00:05.652 INFO:teuthology.orchestra.run.smithi160.stderr: -253> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding auth protocol: cephx 2022-09-02T06:00:05.652 INFO:teuthology.orchestra.run.smithi160.stderr: -252> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.652 INFO:teuthology.orchestra.run.smithi160.stderr: -251> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.653 INFO:teuthology.orchestra.run.smithi160.stderr: -250> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.653 INFO:teuthology.orchestra.run.smithi160.stderr: -249> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.653 INFO:teuthology.orchestra.run.smithi160.stderr: -248> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.653 INFO:teuthology.orchestra.run.smithi160.stderr: -247> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.654 INFO:teuthology.orchestra.run.smithi160.stderr: -246> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.654 INFO:teuthology.orchestra.run.smithi160.stderr: -245> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.654 INFO:teuthology.orchestra.run.smithi160.stderr: -244> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.654 INFO:teuthology.orchestra.run.smithi160.stderr: -243> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.655 INFO:teuthology.orchestra.run.smithi160.stderr: -242> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: crc 2022-09-02T06:00:05.655 INFO:teuthology.orchestra.run.smithi160.stderr: -241> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 AuthRegistry(0x7ffd5fb29460) adding con mode: secure 2022-09-02T06:00:05.655 INFO:teuthology.orchestra.run.smithi160.stderr: -240> 2022-09-02T06:00:05.137+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.655 INFO:teuthology.orchestra.run.smithi160.stderr: -239> 2022-09-02T06:00:05.137+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.656 INFO:teuthology.orchestra.run.smithi160.stderr: -238> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.656 INFO:teuthology.orchestra.run.smithi160.stderr: -237> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.656 INFO:teuthology.orchestra.run.smithi160.stderr: -236> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34d95000 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.656 INFO:teuthology.orchestra.run.smithi160.stderr: -235> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.657 INFO:teuthology.orchestra.run.smithi160.stderr: -234> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.657 INFO:teuthology.orchestra.run.smithi160.stderr: -233> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.778340s 2022-09-02T06:00:05.657 INFO:teuthology.orchestra.run.smithi160.stderr: -232> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34d95000 auth_method 0 2022-09-02T06:00:05.657 INFO:teuthology.orchestra.run.smithi160.stderr: -231> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.658 INFO:teuthology.orchestra.run.smithi160.stderr: -230> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.658 INFO:teuthology.orchestra.run.smithi160.stderr: -229> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.658 INFO:teuthology.orchestra.run.smithi160.stderr: -228> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.659 INFO:teuthology.orchestra.run.smithi160.stderr: -227> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.659 INFO:teuthology.orchestra.run.smithi160.stderr: -226> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.659 INFO:teuthology.orchestra.run.smithi160.stderr: -225> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6683 payload 274 2022-09-02T06:00:05.659 INFO:teuthology.orchestra.run.smithi160.stderr: -224> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.660 INFO:teuthology.orchestra.run.smithi160.stderr: -223> 2022-09-02T06:00:05.137+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.660 INFO:teuthology.orchestra.run.smithi160.stderr: -222> 2022-09-02T06:00:05.137+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.660 INFO:teuthology.orchestra.run.smithi160.stderr: -221> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.661 INFO:teuthology.orchestra.run.smithi160.stderr: -220> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.661 INFO:teuthology.orchestra.run.smithi160.stderr: -219> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: dump: 2022-09-02T06:00:05.661 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.661 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.662 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.662 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.662 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.662 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.663 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.663 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.663 INFO:teuthology.orchestra.run.smithi160.stderr: -218> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.663 INFO:teuthology.orchestra.run.smithi160.stderr: -217> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.664 INFO:teuthology.orchestra.run.smithi160.stderr: -216> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.664 INFO:teuthology.orchestra.run.smithi160.stderr: -215> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.664 INFO:teuthology.orchestra.run.smithi160.stderr: -214> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.664 INFO:teuthology.orchestra.run.smithi160.stderr: -213> 2022-09-02T06:00:05.137+0000 7f9122131700 10 monclient: dump: 2022-09-02T06:00:05.665 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.665 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.665 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.665 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.666 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.666 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.666 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.667 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.667 INFO:teuthology.orchestra.run.smithi160.stderr: -212> 2022-09-02T06:00:05.137+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6683 2022-09-02T06:00:05.667 INFO:teuthology.orchestra.run.smithi160.stderr: -211> 2022-09-02T06:00:05.137+0000 7f913a1b1600 10 monclient: get_monmap_and_config success 2022-09-02T06:00:05.667 INFO:teuthology.orchestra.run.smithi160.stderr: -210> 2022-09-02T06:00:05.137+0000 7f913a1b1600 4 set_mon_vals no callback set 2022-09-02T06:00:05.668 INFO:teuthology.orchestra.run.smithi160.stderr: -209> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: shutdown 2022-09-02T06:00:05.668 INFO:teuthology.orchestra.run.smithi160.stderr: -208> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: build_initial_monmap 2022-09-02T06:00:05.668 INFO:teuthology.orchestra.run.smithi160.stderr: -207> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 build_initial for_mkfs: 0 2022-09-02T06:00:05.668 INFO:teuthology.orchestra.run.smithi160.stderr: -206> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: monmap: 2022-09-02T06:00:05.669 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 0 2022-09-02T06:00:05.669 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 00000000-0000-0000-0000-000000000000 2022-09-02T06:00:05.669 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 0.000000 2022-09-02T06:00:05.669 INFO:teuthology.orchestra.run.smithi160.stderr:created 0.000000 2022-09-02T06:00:05.670 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 0 (unknown) 2022-09-02T06:00:05.670 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.670 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.noname-a 2022-09-02T06:00:05.671 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.671 INFO:teuthology.orchestra.run.smithi160.stderr: -205> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.671 INFO:teuthology.orchestra.run.smithi160.stderr: -204> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.672 INFO:teuthology.orchestra.run.smithi160.stderr: -203> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding auth protocol: cephx 2022-09-02T06:00:05.672 INFO:teuthology.orchestra.run.smithi160.stderr: -202> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.672 INFO:teuthology.orchestra.run.smithi160.stderr: -201> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.672 INFO:teuthology.orchestra.run.smithi160.stderr: -200> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.673 INFO:teuthology.orchestra.run.smithi160.stderr: -199> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.673 INFO:teuthology.orchestra.run.smithi160.stderr: -198> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.673 INFO:teuthology.orchestra.run.smithi160.stderr: -197> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.673 INFO:teuthology.orchestra.run.smithi160.stderr: -196> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.674 INFO:teuthology.orchestra.run.smithi160.stderr: -195> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.674 INFO:teuthology.orchestra.run.smithi160.stderr: -194> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.674 INFO:teuthology.orchestra.run.smithi160.stderr: -193> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.674 INFO:teuthology.orchestra.run.smithi160.stderr: -192> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: crc 2022-09-02T06:00:05.675 INFO:teuthology.orchestra.run.smithi160.stderr: -191> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a357a2140) adding con mode: secure 2022-09-02T06:00:05.675 INFO:teuthology.orchestra.run.smithi160.stderr: -190> 2022-09-02T06:00:05.138+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.675 INFO:teuthology.orchestra.run.smithi160.stderr: -189> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: starting msgr at 2022-09-02T06:00:05.675 INFO:teuthology.orchestra.run.smithi160.stderr: -188> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: starting objecter 2022-09-02T06:00:05.676 INFO:teuthology.orchestra.run.smithi160.stderr: -187> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command objecter_requests cmddesc objecter_requests hook 0x555a34a36730 EEXIST 2022-09-02T06:00:05.676 INFO:teuthology.orchestra.run.smithi160.stderr: -186> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: setting wanted keys 2022-09-02T06:00:05.676 INFO:teuthology.orchestra.run.smithi160.stderr: -185> 2022-09-02T06:00:05.138+0000 7f913a1b1600 1 librados: calling monclient init 2022-09-02T06:00:05.676 INFO:teuthology.orchestra.run.smithi160.stderr: -184> 2022-09-02T06:00:05.138+0000 7f913a1b1600 10 monclient: init 2022-09-02T06:00:05.677 INFO:teuthology.orchestra.run.smithi160.stderr: -183> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding auth protocol: cephx 2022-09-02T06:00:05.677 INFO:teuthology.orchestra.run.smithi160.stderr: -182> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding auth protocol: cephx 2022-09-02T06:00:05.677 INFO:teuthology.orchestra.run.smithi160.stderr: -181> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding auth protocol: cephx 2022-09-02T06:00:05.677 INFO:teuthology.orchestra.run.smithi160.stderr: -180> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.678 INFO:teuthology.orchestra.run.smithi160.stderr: -179> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.678 INFO:teuthology.orchestra.run.smithi160.stderr: -178> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.678 INFO:teuthology.orchestra.run.smithi160.stderr: -177> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.678 INFO:teuthology.orchestra.run.smithi160.stderr: -176> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.679 INFO:teuthology.orchestra.run.smithi160.stderr: -175> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.679 INFO:teuthology.orchestra.run.smithi160.stderr: -174> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.679 INFO:teuthology.orchestra.run.smithi160.stderr: -173> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.680 INFO:teuthology.orchestra.run.smithi160.stderr: -172> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.680 INFO:teuthology.orchestra.run.smithi160.stderr: -171> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.680 INFO:teuthology.orchestra.run.smithi160.stderr: -170> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: crc 2022-09-02T06:00:05.681 INFO:teuthology.orchestra.run.smithi160.stderr: -169> 2022-09-02T06:00:05.138+0000 7f913a1b1600 5 AuthRegistry(0x555a34a9f0d8) adding con mode: secure 2022-09-02T06:00:05.681 INFO:teuthology.orchestra.run.smithi160.stderr: -168> 2022-09-02T06:00:05.138+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.681 INFO:teuthology.orchestra.run.smithi160.stderr: -167> 2022-09-02T06:00:05.138+0000 7f913a1b1600 2 auth: KeyRing::load: loaded key file /etc/ceph/c2.client.0.keyring 2022-09-02T06:00:05.681 INFO:teuthology.orchestra.run.smithi160.stderr: -166> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _reopen_session rank -1 2022-09-02T06:00:05.682 INFO:teuthology.orchestra.run.smithi160.stderr: -165> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _add_conns ranks=[0] 2022-09-02T06:00:05.682 INFO:teuthology.orchestra.run.smithi160.stderr: -164> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): picked mon.noname-a con 0x555a34d95000 addr [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2022-09-02T06:00:05.682 INFO:teuthology.orchestra.run.smithi160.stderr: -163> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): start opening mon connection 2022-09-02T06:00:05.683 INFO:teuthology.orchestra.run.smithi160.stderr: -162> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): _renew_subs 2022-09-02T06:00:05.683 INFO:teuthology.orchestra.run.smithi160.stderr: -161> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient(hunting): authenticate will time out at 7252.780058s 2022-09-02T06:00:05.683 INFO:teuthology.orchestra.run.smithi160.stderr: -160> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): get_auth_request con 0x555a34d95000 auth_method 0 2022-09-02T06:00:05.683 INFO:teuthology.orchestra.run.smithi160.stderr: -159> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2022-09-02T06:00:05.684 INFO:teuthology.orchestra.run.smithi160.stderr: -158> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): _init_auth method 2 2022-09-02T06:00:05.684 INFO:teuthology.orchestra.run.smithi160.stderr: -157> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): _init_auth creating new auth 2022-09-02T06:00:05.684 INFO:teuthology.orchestra.run.smithi160.stderr: -156> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload 9 2022-09-02T06:00:05.684 INFO:teuthology.orchestra.run.smithi160.stderr: -155> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2022-09-02T06:00:05.685 INFO:teuthology.orchestra.run.smithi160.stderr: -154> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2022-09-02T06:00:05.685 INFO:teuthology.orchestra.run.smithi160.stderr: -153> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient(hunting): handle_auth_done global_id 6684 payload 824 2022-09-02T06:00:05.685 INFO:teuthology.orchestra.run.smithi160.stderr: -152> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient: _finish_hunting 0 2022-09-02T06:00:05.685 INFO:teuthology.orchestra.run.smithi160.stderr: -151> 2022-09-02T06:00:05.139+0000 7f913815d700 1 monclient: found mon.noname-a 2022-09-02T06:00:05.686 INFO:teuthology.orchestra.run.smithi160.stderr: -150> 2022-09-02T06:00:05.139+0000 7f913815d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.686 INFO:teuthology.orchestra.run.smithi160.stderr: -149> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.686 INFO:teuthology.orchestra.run.smithi160.stderr: -148> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2022-09-02T06:00:05.687 INFO:teuthology.orchestra.run.smithi160.stderr: -147> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: dump: 2022-09-02T06:00:05.687 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.687 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.687 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.688 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.688 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.688 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.688 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.689 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.689 INFO:teuthology.orchestra.run.smithi160.stderr: -146> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: _finish_auth 0 2022-09-02T06:00:05.689 INFO:teuthology.orchestra.run.smithi160.stderr: -145> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: _check_auth_tickets 2022-09-02T06:00:05.690 INFO:teuthology.orchestra.run.smithi160.stderr: -144> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: handle_config config(0 keys) v1 2022-09-02T06:00:05.690 INFO:teuthology.orchestra.run.smithi160.stderr: -143> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: handle_monmap mon_map magic: 0 v1 2022-09-02T06:00:05.690 INFO:teuthology.orchestra.run.smithi160.stderr: -142> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2022-09-02T06:00:05.690 INFO:teuthology.orchestra.run.smithi160.stderr: -141> 2022-09-02T06:00:05.139+0000 7f9122131700 4 set_mon_vals no callback set 2022-09-02T06:00:05.691 INFO:teuthology.orchestra.run.smithi160.stderr: -140> 2022-09-02T06:00:05.139+0000 7f912092e700 10 monclient: dump: 2022-09-02T06:00:05.691 INFO:teuthology.orchestra.run.smithi160.stderr:epoch 1 2022-09-02T06:00:05.691 INFO:teuthology.orchestra.run.smithi160.stderr:fsid 4a69b944-3256-4419-904f-7378d4120c1b 2022-09-02T06:00:05.691 INFO:teuthology.orchestra.run.smithi160.stderr:last_changed 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.692 INFO:teuthology.orchestra.run.smithi160.stderr:created 2022-09-02T04:15:43.374867+0000 2022-09-02T06:00:05.692 INFO:teuthology.orchestra.run.smithi160.stderr:min_mon_release 17 (quincy) 2022-09-02T06:00:05.692 INFO:teuthology.orchestra.run.smithi160.stderr:election_strategy: 1 2022-09-02T06:00:05.692 INFO:teuthology.orchestra.run.smithi160.stderr:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2022-09-02T06:00:05.693 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.693 INFO:teuthology.orchestra.run.smithi160.stderr: -139> 2022-09-02T06:00:05.139+0000 7f913a1b1600 5 monclient: authenticate success, global_id 6684 2022-09-02T06:00:05.694 INFO:teuthology.orchestra.run.smithi160.stderr: -138> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.694 INFO:teuthology.orchestra.run.smithi160.stderr: -137> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.695 INFO:teuthology.orchestra.run.smithi160.stderr: -136> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _renew_subs 2022-09-02T06:00:05.695 INFO:teuthology.orchestra.run.smithi160.stderr: -135> 2022-09-02T06:00:05.139+0000 7f913a1b1600 10 monclient: _send_mon_message to mon.a at v2:172.21.15.160:3300/0 2022-09-02T06:00:05.696 INFO:teuthology.orchestra.run.smithi160.stderr: -134> 2022-09-02T06:00:05.139+0000 7f913a1b1600 1 librados: init done 2022-09-02T06:00:05.696 INFO:teuthology.orchestra.run.smithi160.stderr: -133> 2022-09-02T06:00:05.139+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command cr dump hook 0x555a34bb2dd8 2022-09-02T06:00:05.697 INFO:teuthology.orchestra.run.smithi160.stderr: -132> 2022-09-02T06:00:05.140+0000 7f912092e700 4 mgrc handle_mgr_map Got map version 5 2022-09-02T06:00:05.697 INFO:teuthology.orchestra.run.smithi160.stderr: -131> 2022-09-02T06:00:05.140+0000 7f912092e700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.698 INFO:teuthology.orchestra.run.smithi160.stderr: -130> 2022-09-02T06:00:05.140+0000 7f912092e700 4 mgrc reconnect Starting new session with [v2:172.21.15.160:6824/103922,v1:172.21.15.160:6825/103922] 2022-09-02T06:00:05.699 INFO:teuthology.orchestra.run.smithi160.stderr: -129> 2022-09-02T06:00:05.140+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34de7400 auth_method 0 2022-09-02T06:00:05.699 INFO:teuthology.orchestra.run.smithi160.stderr: -128> 2022-09-02T06:00:05.140+0000 7f913895e700 10 monclient: get_auth_request con 0x555a34d94000 auth_method 0 2022-09-02T06:00:05.700 INFO:teuthology.orchestra.run.smithi160.stderr: -127> 2022-09-02T06:00:05.142+0000 7f913815d700 10 monclient: get_auth_request con 0x555a34a43800 auth_method 0 2022-09-02T06:00:05.700 INFO:teuthology.orchestra.run.smithi160.stderr: -126> 2022-09-02T06:00:05.145+0000 7f913795c700 10 monclient: get_auth_request con 0x555a34d60400 auth_method 0 2022-09-02T06:00:05.701 INFO:teuthology.orchestra.run.smithi160.stderr: -125> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace show hook 0x555a34a6b420 2022-09-02T06:00:05.701 INFO:teuthology.orchestra.run.smithi160.stderr: -124> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace history hook 0x555a34a6b420 2022-09-02T06:00:05.702 INFO:teuthology.orchestra.run.smithi160.stderr: -123> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace active hook 0x555a34a6b420 2022-09-02T06:00:05.702 INFO:teuthology.orchestra.run.smithi160.stderr: -122> 2022-09-02T06:00:05.206+0000 7f913a1b1600 5 asok(0x555a34ad6000) register_command sync trace active_short hook 0x555a34a6b420 2022-09-02T06:00:05.703 INFO:teuthology.orchestra.run.smithi160.stderr: -121> 2022-09-02T06:00:05.210+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2022-09-02T06:00:05.703 INFO:teuthology.orchestra.run.smithi160.stderr: -120> 2022-09-02T06:00:05.215+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: waiting on bucket lease 2022-09-02T06:00:05.704 INFO:teuthology.orchestra.run.smithi160.stderr: -119> 2022-09-02T06:00:05.216+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: status after acquiring the lock is: init 2022-09-02T06:00:05.704 INFO:teuthology.orchestra.run.smithi160.stderr: -118> 2022-09-02T06:00:05.240+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: listing bucket for full sync 2022-09-02T06:00:05.705 INFO:teuthology.orchestra.run.smithi160.stderr: -117> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj11 2022-09-02T06:00:05.705 INFO:teuthology.orchestra.run.smithi160.stderr: -116> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj11]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj11 op=0 op_state=1 2022-09-02T06:00:05.706 INFO:teuthology.orchestra.run.smithi160.stderr: -115> 2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj11]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj11[0] 2022-09-02T06:00:05.707 INFO:teuthology.orchestra.run.smithi160.stderr: -114> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj111 2022-09-02T06:00:05.707 INFO:teuthology.orchestra.run.smithi160.stderr: -113> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj111 op=0 op_state=1 2022-09-02T06:00:05.708 INFO:teuthology.orchestra.run.smithi160.stderr: -112> 2022-09-02T06:00:05.246+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: : injecting data sync error on key=obj111 2022-09-02T06:00:05.708 INFO:teuthology.orchestra.run.smithi160.stderr: -111> 2022-09-02T06:00:05.246+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.709 INFO:teuthology.orchestra.run.smithi160.stderr: -110> 2022-09-02T06:00:05.246+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj111]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj111 2022-09-02T06:00:05.709 INFO:teuthology.orchestra.run.smithi160.stderr: -109> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj115 2022-09-02T06:00:05.710 INFO:teuthology.orchestra.run.smithi160.stderr: -108> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj115 op=0 op_state=1 2022-09-02T06:00:05.710 INFO:teuthology.orchestra.run.smithi160.stderr: -107> 2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj115[0] 2022-09-02T06:00:05.711 INFO:teuthology.orchestra.run.smithi160.stderr: -106> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj116 2022-09-02T06:00:05.712 INFO:teuthology.orchestra.run.smithi160.stderr: -105> 2022-09-02T06:00:05.246+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj116 op=0 op_state=1 2022-09-02T06:00:05.712 INFO:teuthology.orchestra.run.smithi160.stderr: -104> 2022-09-02T06:00:05.246+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj116[0] 2022-09-02T06:00:05.712 INFO:teuthology.orchestra.run.smithi160.stderr: -103> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj124 2022-09-02T06:00:05.712 INFO:teuthology.orchestra.run.smithi160.stderr: -102> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj124]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj124 op=0 op_state=1 2022-09-02T06:00:05.713 INFO:teuthology.orchestra.run.smithi160.stderr: -101> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj124]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj124[0] 2022-09-02T06:00:05.713 INFO:teuthology.orchestra.run.smithi160.stderr: -100> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj145 2022-09-02T06:00:05.713 INFO:teuthology.orchestra.run.smithi160.stderr: -99> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj145 op=0 op_state=1 2022-09-02T06:00:05.713 INFO:teuthology.orchestra.run.smithi160.stderr: -98> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj145[0] 2022-09-02T06:00:05.714 INFO:teuthology.orchestra.run.smithi160.stderr: -97> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj148 2022-09-02T06:00:05.714 INFO:teuthology.orchestra.run.smithi160.stderr: -96> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj148 op=0 op_state=1 2022-09-02T06:00:05.714 INFO:teuthology.orchestra.run.smithi160.stderr: -95> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj148[0] 2022-09-02T06:00:05.715 INFO:teuthology.orchestra.run.smithi160.stderr: -94> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj155 2022-09-02T06:00:05.715 INFO:teuthology.orchestra.run.smithi160.stderr: -93> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj155 op=0 op_state=1 2022-09-02T06:00:05.715 INFO:teuthology.orchestra.run.smithi160.stderr: -92> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: : injecting data sync error on key=obj155 2022-09-02T06:00:05.716 INFO:teuthology.orchestra.run.smithi160.stderr: -91> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.716 INFO:teuthology.orchestra.run.smithi160.stderr: -90> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj155]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj155 2022-09-02T06:00:05.716 INFO:teuthology.orchestra.run.smithi160.stderr: -89> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj17 2022-09-02T06:00:05.717 INFO:teuthology.orchestra.run.smithi160.stderr: -88> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj17 op=0 op_state=1 2022-09-02T06:00:05.717 INFO:teuthology.orchestra.run.smithi160.stderr: -87> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj17[0] 2022-09-02T06:00:05.717 INFO:teuthology.orchestra.run.smithi160.stderr: -86> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj185 2022-09-02T06:00:05.718 INFO:teuthology.orchestra.run.smithi160.stderr: -85> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj185]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj185 op=0 op_state=1 2022-09-02T06:00:05.718 INFO:teuthology.orchestra.run.smithi160.stderr: -84> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj185]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj185[0] 2022-09-02T06:00:05.718 INFO:teuthology.orchestra.run.smithi160.stderr: -83> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj19 2022-09-02T06:00:05.718 INFO:teuthology.orchestra.run.smithi160.stderr: -82> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj19 op=0 op_state=1 2022-09-02T06:00:05.719 INFO:teuthology.orchestra.run.smithi160.stderr: -81> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj19[0] 2022-09-02T06:00:05.719 INFO:teuthology.orchestra.run.smithi160.stderr: -80> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj22 2022-09-02T06:00:05.719 INFO:teuthology.orchestra.run.smithi160.stderr: -79> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj22 op=0 op_state=1 2022-09-02T06:00:05.720 INFO:teuthology.orchestra.run.smithi160.stderr: -78> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: : injecting data sync error on key=obj22 2022-09-02T06:00:05.720 INFO:teuthology.orchestra.run.smithi160.stderr: -77> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.720 INFO:teuthology.orchestra.run.smithi160.stderr: -76> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj22]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj22 2022-09-02T06:00:05.721 INFO:teuthology.orchestra.run.smithi160.stderr: -75> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj23 2022-09-02T06:00:05.721 INFO:teuthology.orchestra.run.smithi160.stderr: -74> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj23]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj23 op=0 op_state=1 2022-09-02T06:00:05.721 INFO:teuthology.orchestra.run.smithi160.stderr: -73> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj23]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj23[0] 2022-09-02T06:00:05.721 INFO:teuthology.orchestra.run.smithi160.stderr: -72> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj29 2022-09-02T06:00:05.722 INFO:teuthology.orchestra.run.smithi160.stderr: -71> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj29 op=0 op_state=1 2022-09-02T06:00:05.722 INFO:teuthology.orchestra.run.smithi160.stderr: -70> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: : injecting data sync error on key=obj29 2022-09-02T06:00:05.722 INFO:teuthology.orchestra.run.smithi160.stderr: -69> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.722 INFO:teuthology.orchestra.run.smithi160.stderr: -68> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj29]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj29 2022-09-02T06:00:05.723 INFO:teuthology.orchestra.run.smithi160.stderr: -67> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj31 2022-09-02T06:00:05.723 INFO:teuthology.orchestra.run.smithi160.stderr: -66> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj31 op=0 op_state=1 2022-09-02T06:00:05.723 INFO:teuthology.orchestra.run.smithi160.stderr: -65> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj31[0] 2022-09-02T06:00:05.724 INFO:teuthology.orchestra.run.smithi160.stderr: -64> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj33 2022-09-02T06:00:05.724 INFO:teuthology.orchestra.run.smithi160.stderr: -63> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj33]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj33 op=0 op_state=1 2022-09-02T06:00:05.724 INFO:teuthology.orchestra.run.smithi160.stderr: -62> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj33]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj33[0] 2022-09-02T06:00:05.725 INFO:teuthology.orchestra.run.smithi160.stderr: -61> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj34 2022-09-02T06:00:05.725 INFO:teuthology.orchestra.run.smithi160.stderr: -60> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj34 op=0 op_state=1 2022-09-02T06:00:05.725 INFO:teuthology.orchestra.run.smithi160.stderr: -59> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: : injecting data sync error on key=obj34 2022-09-02T06:00:05.725 INFO:teuthology.orchestra.run.smithi160.stderr: -58> 2022-09-02T06:00:05.247+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2022-09-02T06:00:05.726 INFO:teuthology.orchestra.run.smithi160.stderr: -57> 2022-09-02T06:00:05.247+0000 7f913a1b1600 0 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj34]: ERROR: failed to sync object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj34 2022-09-02T06:00:05.726 INFO:teuthology.orchestra.run.smithi160.stderr: -56> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj37 2022-09-02T06:00:05.727 INFO:teuthology.orchestra.run.smithi160.stderr: -55> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj37 op=0 op_state=1 2022-09-02T06:00:05.727 INFO:teuthology.orchestra.run.smithi160.stderr: -54> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj37[0] 2022-09-02T06:00:05.727 INFO:teuthology.orchestra.run.smithi160.stderr: -53> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj38 2022-09-02T06:00:05.727 INFO:teuthology.orchestra.run.smithi160.stderr: -52> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj38]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj38 op=0 op_state=1 2022-09-02T06:00:05.728 INFO:teuthology.orchestra.run.smithi160.stderr: -51> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj38]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj38[0] 2022-09-02T06:00:05.728 INFO:teuthology.orchestra.run.smithi160.stderr: -50> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj44 2022-09-02T06:00:05.728 INFO:teuthology.orchestra.run.smithi160.stderr: -49> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj44 op=0 op_state=1 2022-09-02T06:00:05.729 INFO:teuthology.orchestra.run.smithi160.stderr: -48> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj44[0] 2022-09-02T06:00:05.729 INFO:teuthology.orchestra.run.smithi160.stderr: -47> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: [full sync] syncing object: xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0/obj46 2022-09-02T06:00:05.729 INFO:teuthology.orchestra.run.smithi160.stderr: -46> 2022-09-02T06:00:05.247+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj46]: bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b=bucket sync single entry (source_zone=b210c8c8-c433-420b-87c1-57ebfc433073) b= log_entry=obj46 op=0 op_state=1 2022-09-02T06:00:05.729 INFO:teuthology.orchestra.run.smithi160.stderr: -45> 2022-09-02T06:00:05.247+0000 7f913a1b1600 5 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj46]: bucket sync: sync obj: b210c8c8-c433-420b-87c1-57ebfc433073/:xljbkf-74[b210c8c8-c433-420b-87c1-57ebfc433073.5527.25])/obj46[0] 2022-09-02T06:00:05.730 INFO:teuthology.orchestra.run.smithi160.stderr: -44> 2022-09-02T06:00:05.250+0000 7f913a1b1600 20 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]: ERROR: full sync failed. error: -5 2022-09-02T06:00:05.730 INFO:teuthology.orchestra.run.smithi160.stderr: -43> 2022-09-02T06:00:05.251+0000 7f911d106700 1 get compressor lz4 = 0 2022-09-02T06:00:05.730 INFO:teuthology.orchestra.run.smithi160.stderr: -42> 2022-09-02T06:00:05.251+0000 7f911d106700 1 load compressor lz4 2022-09-02T06:00:05.730 INFO:teuthology.orchestra.run.smithi160.stderr: -41> 2022-09-02T06:00:05.251+0000 7f911d106700 1 add compressor lz4 0x555a35784570 2022-09-02T06:00:05.731 INFO:teuthology.orchestra.run.smithi160.stderr: -40> 2022-09-02T06:00:05.251+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.731 INFO:teuthology.orchestra.run.smithi160.stderr: -39> 2022-09-02T06:00:05.251+0000 7f911d106700 1 load: compressor lz4 loaded and registered 2022-09-02T06:00:05.731 INFO:teuthology.orchestra.run.smithi160.stderr: -38> 2022-09-02T06:00:05.251+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.732 INFO:teuthology.orchestra.run.smithi160.stderr: -37> 2022-09-02T06:00:05.252+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.732 INFO:teuthology.orchestra.run.smithi160.stderr: -36> 2022-09-02T06:00:05.252+0000 7f911d106700 1 get compressor zlib = 0 2022-09-02T06:00:05.732 INFO:teuthology.orchestra.run.smithi160.stderr: -35> 2022-09-02T06:00:05.252+0000 7f911d106700 1 load compressor zlib 2022-09-02T06:00:05.732 INFO:teuthology.orchestra.run.smithi160.stderr: -34> 2022-09-02T06:00:05.253+0000 7f911d106700 1 add compressor zlib 0x555a357853e0 2022-09-02T06:00:05.733 INFO:teuthology.orchestra.run.smithi160.stderr: -33> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor zlib = 0x555a357853e0 2022-09-02T06:00:05.733 INFO:teuthology.orchestra.run.smithi160.stderr: -32> 2022-09-02T06:00:05.253+0000 7f911d106700 1 load: compressor zlib loaded and registered 2022-09-02T06:00:05.733 INFO:teuthology.orchestra.run.smithi160.stderr: -31> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor zlib = 0x555a357853e0 2022-09-02T06:00:05.734 INFO:teuthology.orchestra.run.smithi160.stderr: -30> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor snappy = 0 2022-09-02T06:00:05.734 INFO:teuthology.orchestra.run.smithi160.stderr: -29> 2022-09-02T06:00:05.253+0000 7f911d106700 1 load compressor snappy 2022-09-02T06:00:05.734 INFO:teuthology.orchestra.run.smithi160.stderr: -28> 2022-09-02T06:00:05.253+0000 7f911d106700 1 add compressor snappy 0x555a357846c0 2022-09-02T06:00:05.735 INFO:teuthology.orchestra.run.smithi160.stderr: -27> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.735 INFO:teuthology.orchestra.run.smithi160.stderr: -26> 2022-09-02T06:00:05.253+0000 7f911d106700 1 load: compressor snappy loaded and registered 2022-09-02T06:00:05.735 INFO:teuthology.orchestra.run.smithi160.stderr: -25> 2022-09-02T06:00:05.253+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.735 INFO:teuthology.orchestra.run.smithi160.stderr: -24> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.736 INFO:teuthology.orchestra.run.smithi160.stderr: -23> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.736 INFO:teuthology.orchestra.run.smithi160.stderr: -22> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.736 INFO:teuthology.orchestra.run.smithi160.stderr: -21> 2022-09-02T06:00:05.254+0000 7f911d106700 1 get compressor zstd = 0 2022-09-02T06:00:05.736 INFO:teuthology.orchestra.run.smithi160.stderr: -20> 2022-09-02T06:00:05.254+0000 7f911d106700 1 load compressor zstd 2022-09-02T06:00:05.737 INFO:teuthology.orchestra.run.smithi160.stderr: -19> 2022-09-02T06:00:05.255+0000 7f911d106700 1 add compressor zstd 0x555a35785b90 2022-09-02T06:00:05.737 INFO:teuthology.orchestra.run.smithi160.stderr: -18> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.737 INFO:teuthology.orchestra.run.smithi160.stderr: -17> 2022-09-02T06:00:05.255+0000 7f911d106700 1 load: compressor zstd loaded and registered 2022-09-02T06:00:05.737 INFO:teuthology.orchestra.run.smithi160.stderr: -16> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.738 INFO:teuthology.orchestra.run.smithi160.stderr: -15> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.738 INFO:teuthology.orchestra.run.smithi160.stderr: -14> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.738 INFO:teuthology.orchestra.run.smithi160.stderr: -13> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.739 INFO:teuthology.orchestra.run.smithi160.stderr: -12> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor lz4 = 0x555a35784570 2022-09-02T06:00:05.739 INFO:teuthology.orchestra.run.smithi160.stderr: -11> 2022-09-02T06:00:05.255+0000 7f911d106700 1 get compressor zstd = 0x555a35785b90 2022-09-02T06:00:05.739 INFO:teuthology.orchestra.run.smithi160.stderr: -10> 2022-09-02T06:00:05.256+0000 7f911d106700 1 get compressor snappy = 0x555a357846c0 2022-09-02T06:00:05.739 INFO:teuthology.orchestra.run.smithi160.stderr: -9> 2022-09-02T06:00:05.268+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj31]: success 2022-09-02T06:00:05.740 INFO:teuthology.orchestra.run.smithi160.stderr: -8> 2022-09-02T06:00:05.274+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj145]: success 2022-09-02T06:00:05.740 INFO:teuthology.orchestra.run.smithi160.stderr: -7> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj116]: success 2022-09-02T06:00:05.740 INFO:teuthology.orchestra.run.smithi160.stderr: -6> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj17]: success 2022-09-02T06:00:05.740 INFO:teuthology.orchestra.run.smithi160.stderr: -5> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj44]: success 2022-09-02T06:00:05.741 INFO:teuthology.orchestra.run.smithi160.stderr: -4> 2022-09-02T06:00:05.275+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj148]: success 2022-09-02T06:00:05.741 INFO:teuthology.orchestra.run.smithi160.stderr: -3> 2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj37]: success 2022-09-02T06:00:05.741 INFO:teuthology.orchestra.run.smithi160.stderr: -2> 2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj19]: success 2022-09-02T06:00:05.742 INFO:teuthology.orchestra.run.smithi160.stderr: -1> 2022-09-02T06:00:05.278+0000 7f913a1b1600 10 RGW-SYNC:bucket[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25<-xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:full_sync[xljbkf-74:b210c8c8-c433-420b-87c1-57ebfc433073.5527.25:0]:entry[obj115]: success 2022-09-02T06:00:05.742 INFO:teuthology.orchestra.run.smithi160.stderr: 0> 2022-09-02T06:00:05.280+0000 7f913a1b1600 -1 *** Caught signal (Segmentation fault) ** 2022-09-02T06:00:05.742 INFO:teuthology.orchestra.run.smithi160.stderr: in thread 7f913a1b1600 thread_name:radosgw-admin 2022-09-02T06:00:05.743 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.743 INFO:teuthology.orchestra.run.smithi160.stderr: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:00:05.743 INFO:teuthology.orchestra.run.smithi160.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f9141977cf0] 2022-09-02T06:00:05.743 INFO:teuthology.orchestra.run.smithi160.stderr: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x1c2) [0x7f913f9757d2] 2022-09-02T06:00:05.744 INFO:teuthology.orchestra.run.smithi160.stderr: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x555a31973e49] 2022-09-02T06:00:05.744 INFO:teuthology.orchestra.run.smithi160.stderr: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x555a3199a76e] 2022-09-02T06:00:05.744 INFO:teuthology.orchestra.run.smithi160.stderr: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x555a31a9440f] 2022-09-02T06:00:05.744 INFO:teuthology.orchestra.run.smithi160.stderr: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x555a31a94be1] 2022-09-02T06:00:05.745 INFO:teuthology.orchestra.run.smithi160.stderr: 7: (RGWBucketPipeSyncStatusManager::run(DoutPrefixProvider const*)+0xaca) [0x555a31944caa] 2022-09-02T06:00:05.745 INFO:teuthology.orchestra.run.smithi160.stderr: 8: main() 2022-09-02T06:00:05.745 INFO:teuthology.orchestra.run.smithi160.stderr: 9: __libc_start_main() 2022-09-02T06:00:05.745 INFO:teuthology.orchestra.run.smithi160.stderr: 10: _start() 2022-09-02T06:00:05.746 INFO:teuthology.orchestra.run.smithi160.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-02T06:00:05.746 INFO:teuthology.orchestra.run.smithi160.stderr: 2022-09-02T06:00:05.746 INFO:teuthology.orchestra.run.smithi160.stderr:--- logging levels --- 2022-09-02T06:00:05.746 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 none 2022-09-02T06:00:05.747 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 lockdep 2022-09-02T06:00:05.747 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 context 2022-09-02T06:00:05.747 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 crush 2022-09-02T06:00:05.748 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds 2022-09-02T06:00:05.748 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_balancer 2022-09-02T06:00:05.748 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_locker 2022-09-02T06:00:05.748 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_log 2022-09-02T06:00:05.749 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_log_expire 2022-09-02T06:00:05.749 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mds_migrator 2022-09-02T06:00:05.749 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 buffer 2022-09-02T06:00:05.749 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 timer 2022-09-02T06:00:05.750 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 filer 2022-09-02T06:00:05.750 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 striper 2022-09-02T06:00:05.750 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 1 objecter 2022-09-02T06:00:05.751 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rados 2022-09-02T06:00:05.751 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd 2022-09-02T06:00:05.752 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd_mirror 2022-09-02T06:00:05.752 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd_replay 2022-09-02T06:00:05.752 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 rbd_pwl 2022-09-02T06:00:05.752 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 journaler 2022-09-02T06:00:05.753 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 objectcacher 2022-09-02T06:00:05.753 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 immutable_obj_cache 2022-09-02T06:00:05.753 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 client 2022-09-02T06:00:05.753 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 osd 2022-09-02T06:00:05.754 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 optracker 2022-09-02T06:00:05.754 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 objclass 2022-09-02T06:00:05.754 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 3 filestore 2022-09-02T06:00:05.754 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 3 journal 2022-09-02T06:00:05.755 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 0 ms 2022-09-02T06:00:05.755 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mon 2022-09-02T06:00:05.755 INFO:teuthology.orchestra.run.smithi160.stderr: 0/10 monc 2022-09-02T06:00:05.756 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 paxos 2022-09-02T06:00:05.756 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 tp 2022-09-02T06:00:05.756 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 auth 2022-09-02T06:00:05.756 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 crypto 2022-09-02T06:00:05.757 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 finisher 2022-09-02T06:00:05.757 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 reserver 2022-09-02T06:00:05.757 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 heartbeatmap 2022-09-02T06:00:05.757 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 perfcounter 2022-09-02T06:00:05.758 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 0 rgw 2022-09-02T06:00:05.758 INFO:teuthology.orchestra.run.smithi160.stderr: 20/20 rgw_sync 2022-09-02T06:00:05.758 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 rgw_datacache 2022-09-02T06:00:05.759 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 rgw_access 2022-09-02T06:00:05.759 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 javaclient 2022-09-02T06:00:05.759 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 asok 2022-09-02T06:00:05.760 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 1 throttle 2022-09-02T06:00:05.760 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 0 refs 2022-09-02T06:00:05.760 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 compressor 2022-09-02T06:00:05.760 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 bluestore 2022-09-02T06:00:05.761 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 bluefs 2022-09-02T06:00:05.761 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 3 bdev 2022-09-02T06:00:05.761 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 kstore 2022-09-02T06:00:05.761 INFO:teuthology.orchestra.run.smithi160.stderr: 4/ 5 rocksdb 2022-09-02T06:00:05.762 INFO:teuthology.orchestra.run.smithi160.stderr: 4/ 5 leveldb 2022-09-02T06:00:05.762 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 fuse 2022-09-02T06:00:05.762 INFO:teuthology.orchestra.run.smithi160.stderr: 2/ 5 mgr 2022-09-02T06:00:05.762 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mgrc 2022-09-02T06:00:05.763 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 dpdk 2022-09-02T06:00:05.763 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 eventtrace 2022-09-02T06:00:05.763 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 prioritycache 2022-09-02T06:00:05.764 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 test 2022-09-02T06:00:05.764 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 cephfs_mirror 2022-09-02T06:00:05.764 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 cephsqlite 2022-09-02T06:00:05.764 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore 2022-09-02T06:00:05.765 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_onode 2022-09-02T06:00:05.765 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_odata 2022-09-02T06:00:05.765 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_omap 2022-09-02T06:00:05.765 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_tm 2022-09-02T06:00:05.766 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_t 2022-09-02T06:00:05.766 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_cleaner 2022-09-02T06:00:05.766 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_lba 2022-09-02T06:00:05.766 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_fixedkv_tree 2022-09-02T06:00:05.767 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_cache 2022-09-02T06:00:05.767 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_journal 2022-09-02T06:00:05.767 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_device 2022-09-02T06:00:05.768 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 seastore_backref 2022-09-02T06:00:05.768 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 alienstore 2022-09-02T06:00:05.768 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 mclock 2022-09-02T06:00:05.768 INFO:teuthology.orchestra.run.smithi160.stderr: 0/ 5 cyanstore 2022-09-02T06:00:05.769 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 ceph_exporter 2022-09-02T06:00:05.769 INFO:teuthology.orchestra.run.smithi160.stderr: 1/ 5 memstore 2022-09-02T06:00:05.769 INFO:teuthology.orchestra.run.smithi160.stderr: -2/-2 (syslog threshold) 2022-09-02T06:00:05.769 INFO:teuthology.orchestra.run.smithi160.stderr: 99/99 (stderr threshold) 2022-09-02T06:00:05.770 INFO:teuthology.orchestra.run.smithi160.stderr:--- pthread ID / name mapping for recent threads --- 2022-09-02T06:00:05.770 INFO:teuthology.orchestra.run.smithi160.stderr: 7f911d106700 / http_manager 2022-09-02T06:00:05.770 INFO:teuthology.orchestra.run.smithi160.stderr: 7f912092e700 / ms_dispatch 2022-09-02T06:00:05.770 INFO:teuthology.orchestra.run.smithi160.stderr: 7f9122131700 / io_context_pool 2022-09-02T06:00:05.771 INFO:teuthology.orchestra.run.smithi160.stderr: 7f9134956700 / ms_dispatch 2022-09-02T06:00:05.771 INFO:teuthology.orchestra.run.smithi160.stderr: 7f9136159700 / io_context_pool 2022-09-02T06:00:05.771 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913715b700 / admin_socket 2022-09-02T06:00:05.772 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913795c700 / msgr-worker-2 2022-09-02T06:00:05.772 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913815d700 / msgr-worker-1 2022-09-02T06:00:05.772 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913895e700 / msgr-worker-0 2022-09-02T06:00:05.773 INFO:teuthology.orchestra.run.smithi160.stderr: 7f913a1b1600 / radosgw-admin 2022-09-02T06:00:05.773 INFO:teuthology.orchestra.run.smithi160.stderr: max_recent 500 2022-09-02T06:00:05.773 INFO:teuthology.orchestra.run.smithi160.stderr: max_new 1000 2022-09-02T06:00:05.773 INFO:teuthology.orchestra.run.smithi160.stderr: log_file /var/lib/ceph/crash/2022-09-02T06:00:05.281819Z_d5cb8faf-566c-4ede-a32e-600c72f98dcd/log 2022-09-02T06:00:05.774 INFO:teuthology.orchestra.run.smithi160.stderr:--- end dump of recent events --- 2022-09-02T06:00:05.775 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2022-09-02T06:00:05.396+0000 7fb73c7d2700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16) 2022-09-02T06:00:05.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2022-09-02T06:00:05.777 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-02T06:00:05.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-02T06:00:05.781 INFO:tasks.rgw.c2.client.0:Started 2022-09-02T06:00:05.782 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:00:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T06:00:05.805 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:00:05.806 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:00:05.806 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T06:00:06.808 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:00:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T06:00:06.837 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:00:06.837 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:00:06.837 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T06:00:08.174 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2022-09-02T06:00:08.172+0000 7f7dd0f15700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16)ERROR: collect() returned error (ret=-16) 2022-09-02T06:00:08.839 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:00:08.866 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:00:08.867 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:00:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T06:00:08.869 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T06:00:12.871 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:00:12.900 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:00:12.901 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:00:12.901 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T06:00:12.905 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2022-09-02T06:00:13.078 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2022-09-02T06:00:13.083 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-xljbkf-1 2022-09-02T06:00:13.085 DEBUG:boto:Method: POST 2022-09-02T06:00:13.090 DEBUG:boto:Path: / 2022-09-02T06:00:13.091 DEBUG:boto:Data: 2022-09-02T06:00:13.092 DEBUG:boto:Headers: {} 2022-09-02T06:00:13.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:00:13.093 DEBUG:boto:Port: 8000 2022-09-02T06:00:13.094 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-02T06:00:13.095 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:00:13.096 DEBUG:boto:Token: None 2022-09-02T06:00:13.097 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20220902T060013Z host;x-amz-date 653c0e4fce41fe75a7f83ee442d6ac3bee6da2fd50a76aa96901c7169daa4609 2022-09-02T06:00:13.097 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060013Z 20220902/front/smithi099/aws4_request 92a39e2b0a88591f1c9b28e33eb3bbb35d3f88a3cc050cd7821babfdd3e26bf7 2022-09-02T06:00:13.098 DEBUG:boto:Signature: b3b3fbb6cca4560b0e1a495e5d273c97ec1be167cee00047bd17b0db0a63cf50 2022-09-02T06:00:13.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060013Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=b3b3fbb6cca4560b0e1a495e5d273c97ec1be167cee00047bd17b0db0a63cf50', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2022-09-02T06:00:13.124 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b72ce7c46d975561-0063119bed-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Fri, 02 Sep 2022 06:00:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:00:13.126 DEBUG:boto:b'045ff5f5-ed23-499b-a50c-10f72d907a15roles-xljbkf-1/arn:aws:iam:::role/roles-xljbkf-12022-09-02T06:00:13.110Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000b72ce7c46d975561-0063119bed-1595-test-zone1' 2022-09-02T06:00:13.129 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-xljbkf-2 2022-09-02T06:00:13.131 DEBUG:boto:Method: POST 2022-09-02T06:00:13.132 DEBUG:boto:Path: / 2022-09-02T06:00:13.134 DEBUG:boto:Data: 2022-09-02T06:00:13.135 DEBUG:boto:Headers: {} 2022-09-02T06:00:13.136 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2022-09-02T06:00:13.138 DEBUG:boto:Port: 8000 2022-09-02T06:00:13.140 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-02T06:00:13.141 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:00:13.142 DEBUG:boto:Token: None 2022-09-02T06:00:13.144 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20220902T060013Z host;x-amz-date 9531108313406e1242fc41165b6c43b045c8c67b4da4457f4e0851b3c3e5b23f 2022-09-02T06:00:13.145 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060013Z 20220902/front/smithi160/aws4_request 8f9f3022d4797a75c4243640625f0ba5ea980f96c0a0cc723c500d67e0ba4060 2022-09-02T06:00:13.147 DEBUG:boto:Signature: 98953fff02e290691be46b9fc330a328cd80046a86a23094a266d8cea3961cf8 2022-09-02T06:00:13.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060013Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=98953fff02e290691be46b9fc330a328cd80046a86a23094a266d8cea3961cf8', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2022-09-02T06:00:35.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000017a46b41f86a20fa-0063119bed-1a20-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Fri, 02 Sep 2022 06:00:35 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:00:35.144 DEBUG:boto:b'56f9b8c1-0946-4dc1-929d-0abccccde199roles-xljbkf-2/arn:aws:iam:::role/roles-xljbkf-22022-09-02T06:00:13.173Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000017a46b41f86a20fa-0063119bed-1a20-test-zone2' 2022-09-02T06:00:35.146 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-xljbkf-3 2022-09-02T06:00:35.147 DEBUG:boto:Method: POST 2022-09-02T06:00:35.148 DEBUG:boto:Path: / 2022-09-02T06:00:35.149 DEBUG:boto:Data: 2022-09-02T06:00:35.150 DEBUG:boto:Headers: {} 2022-09-02T06:00:35.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:00:35.152 DEBUG:boto:Port: 8001 2022-09-02T06:00:35.153 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2022-09-02T06:00:35.154 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T06:00:35.155 DEBUG:boto:Token: None 2022-09-02T06:00:35.155 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20220902T060035Z host;x-amz-date f077c4f2d39185040064c6b051d7f92fec908230b70f921ea2fcbe863faf0111 2022-09-02T06:00:35.156 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060035Z 20220902/front/smithi099/aws4_request 18de17dbc76fc81d0a0e3a83eeb21e488a64fbc7dd5ab34e181f817da9fc94de 2022-09-02T06:00:35.157 DEBUG:boto:Signature: e4b682f5c756658e19f65e76f1a69529590f603f1bf5fdf694220558a306263d 2022-09-02T06:00:35.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060035Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=e4b682f5c756658e19f65e76f1a69529590f603f1bf5fdf694220558a306263d', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2022-09-02T06:00:36.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c4c46803e20335d4-0063119c03-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Fri, 02 Sep 2022 06:00:36 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:00:36.182 DEBUG:boto:b'ea4daf4c-88ce-4aac-8e59-41ea3874cbf8roles-xljbkf-3/arn:aws:iam:::role/roles-xljbkf-32022-09-02T06:00:36.173Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000c4c46803e20335d4-0063119c03-16ef-test-zone3' 2022-09-02T06:00:36.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:36.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:36.184 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:00:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:36.286 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:36.298 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T06:00:36.321 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T06:00:36.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:00:36.322 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:00:36.323 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:00:36.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:00:36.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:00:36.324 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:00:36.324 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T06:00:36.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:00:36.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:00:36.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098436.167588_443.1", 2022-09-02T06:00:36.325 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:36.167588Z" 2022-09-02T06:00:36.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:00:36.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:00:36.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:00:36.327 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:00:36.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:00:36.328 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T06:00:36.331 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:00:36.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T06:00:36.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:36.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:36.336 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:00:36.401 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:36.402 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:36.435 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:36.433+0000 7f0e167f3600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:36.436 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:36.434+0000 7f0e167f3600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:36.450 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:00:36.451 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:00:36.451 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:00:36.451 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:00:36.452 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:00:36.452 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:00:36.452 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:36.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:36.453 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:00:36.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:36.453 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:00:36.454 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:36.454 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:36.454 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:00:36.454 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:36.455 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:00:36.455 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:36.455 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:00:36.455 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:36.456 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:36.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:36.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:36.457 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:00:36.458 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:36.458 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:36.459 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:00:36.459 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:36.460 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:36.460 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:36.460 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:00:36.461 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:36.461 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:36.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:36.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:36.463 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:00:36.463 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:36.464 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:36.464 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:00:36.464 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:36.464 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:36.465 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:36.465 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:00:36.465 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:36.466 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:36.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:36.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:36.467 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:00:36.467 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:36.467 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:36.468 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:00:36.468 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:36.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:36.469 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:36.469 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:00:36.470 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:36.471 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:36.471 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:36.472 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:00:36.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:36.473 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:00:36.473 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:00:36.474 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:00:36.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:36.475 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:00:36Z" 2022-09-02T06:00:36.475 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:00:36.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:00:36Z" } 2022-09-02T06:00:36.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:00:36.480 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:00:36.480 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:00:36.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:00:41.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:41.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:41.486 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:00:41.543 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:41.544 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:41.579 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:41.575+0000 7f3cb31ba600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:41.580 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:41.576+0000 7f3cb31ba600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:41.595 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:00:41.595 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:00:41.596 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:00:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:00:41.597 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:00:41.598 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:00:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:41.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:41.600 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:00:41.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:41.601 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:00:41.602 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:41.603 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:00:41.604 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:41.605 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:00:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:41.606 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:00:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:41.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:00:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:41.608 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:00:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:41.609 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:00:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:00:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:00:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:41.612 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:00:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:41.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:41.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:41.626 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:00:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:41.627 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:00:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:41.628 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:00:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:41.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:41.630 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:00:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:00:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:00:41.631 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:00:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:41.632 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:00:41Z" 2022-09-02T06:00:41.632 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:00:41.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:00:41Z" } 2022-09-02T06:00:41.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:00:41.635 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:00:41.636 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:00:41.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:00:46.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:46.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:46.641 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:00:46.699 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:46.700 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:46.743 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:46.742+0000 7f19cf95a600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:46.744 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:46.743+0000 7f19cf95a600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:46.761 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:00:46.762 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:00:46.762 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:00:46.762 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:00:46.762 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:00:46.763 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:00:46.763 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:46.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:46.763 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:00:46.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:46.764 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:00:46.764 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:46.764 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:46.765 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:00:46.765 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:46.765 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:00:46.765 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:46.766 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:00:46.766 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:46.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:46.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:46.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:46.767 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:00:46.768 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:46.768 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:46.768 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:00:46.768 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:46.769 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:46.769 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:46.769 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:00:46.770 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:46.770 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:46.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:46.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:46.771 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:00:46.771 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:46.771 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:46.772 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:00:46.772 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:46.772 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:46.772 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:46.773 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:00:46.773 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:46.773 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:46.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:46.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:46.774 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:00:46.775 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:46.775 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:46.775 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:00:46.775 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:46.776 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:46.776 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:46.776 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:00:46.776 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:46.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:46.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:46.777 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:00:46.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:46.778 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:00:46.778 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:00:46.778 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:00:46.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:46.779 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:00:46Z" 2022-09-02T06:00:46.779 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:00:46.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:00:46Z" } 2022-09-02T06:00:46.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:00:46.782 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:00:46.783 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:00:46.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:00:51.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:51.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:51.786 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:00:51.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:51.844 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:51.877 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:51.875+0000 7f62c1ba7600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:51.878 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:51.876+0000 7f62c1ba7600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:51.899 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:00:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:00:51.900 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:00:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:00:51.901 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:00:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:00:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:51.902 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:00:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:00:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:51.903 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:00:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:51.904 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:00:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:00:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:51.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:00:51.906 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:00:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:51.907 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:00:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:51.908 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:51.909 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:00:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:00:51.910 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:00:51.911 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:51.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:00:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:51.913 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:00:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:51.914 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:00:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:51.915 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:00:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:00:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:00:51.916 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:00:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:51.917 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:00:51Z" 2022-09-02T06:00:51.917 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:00:51.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:00:51Z" } 2022-09-02T06:00:51.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:00:51.920 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:00:51.920 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:00:51.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:00:56.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:56.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:00:56.923 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:00:56.981 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:00:56.981 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:00:57.036 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:57.034+0000 7f540bb7c600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:57.037 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:00:57.036+0000 7f540bb7c600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:00:57.060 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:00:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:00:57.060 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:00:57.061 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:00:57.061 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:00:57.062 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:00:57.062 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:57.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:57.063 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:00:57.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:57.064 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:00:57.065 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:57.065 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:57.066 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:00:57.066 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:57.067 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:00:57.068 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:57.068 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:00:57.069 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:57.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:57.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:57.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:57.071 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:00:57.071 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:57.071 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:57.072 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:00:57.072 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:57.073 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:57.073 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:57.074 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:00:57.074 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:57.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:57.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:57.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:57.075 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:00:57.076 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:57.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:57.076 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:00:57.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:57.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:57.077 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:57.077 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:00:57.078 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:00:57.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:57.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:57.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:00:57.079 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:00:57.079 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:00:57.079 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:00:57.079 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:00:57.080 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:00:57.080 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:00:57.080 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:00:57.081 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:00:57.081 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:00:57.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:57.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:00:57.082 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:00:57.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:57.082 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:00:57.083 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:00:57.083 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:00:57.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:00:57.083 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:00:57Z" 2022-09-02T06:00:57.084 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:00:57.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:00:57Z" } 2022-09-02T06:00:57.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:00:57.086 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:00:57.087 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:00:57.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:02.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:02.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:02.090 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:02.149 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:02.150 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:02.183 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:02.181+0000 7f530ad20600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:02.184 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:02.182+0000 7f530ad20600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:02.198 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:02.199 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:02.199 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:02.199 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:02.200 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:02.200 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:02.200 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:02.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:02.201 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:02.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:02.201 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:02.202 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:02.202 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:02.202 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:02.202 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:02.203 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:02.203 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:02.203 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:02.203 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:02.204 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:02.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:02.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:02.204 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:02.205 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:02.205 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:02.205 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:02.206 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:02.206 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:02.206 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:02.207 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:02.207 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:02.207 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:02.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:02.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:02.208 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:02.208 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:02.208 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:02.209 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:02.209 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:02.209 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:02.209 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:02.210 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:02.210 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:02.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:02.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:02.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:02.211 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:02.211 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:02.211 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:02.212 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:02.212 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:02.212 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:02.212 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:02.213 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:02.213 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:02.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:02.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:02.214 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:02.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:02.214 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:02.215 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:02.215 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:02.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:02.216 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:02Z" 2022-09-02T06:01:02.216 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:02.217 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:02Z" } 2022-09-02T06:01:02.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:02.219 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:02.220 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:02.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:07.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:07.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:07.224 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:07.281 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:07.282 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:07.324 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:07.322+0000 7f3f29b34600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:07.324 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:07.323+0000 7f3f29b34600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:07.344 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:07.344 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:07.344 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:07.345 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:07.345 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:07.345 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:07.345 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:07.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:07.346 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:07.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:07.347 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:07.347 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:07.347 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:07.347 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:07.348 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:07.348 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:07.348 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:07.348 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:07.348 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:07.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:07.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:07.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:07.349 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:07.350 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:07.350 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:07.351 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:07.351 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:07.351 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:07.351 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:07.352 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:07.352 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:07.352 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:07.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:07.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:07.353 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:07.353 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:07.353 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:07.354 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:07.354 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:07.354 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:07.354 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:07.355 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:07.355 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:07.355 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:07.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:07.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:07.356 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:07.356 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:07.356 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:07.356 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:07.357 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:07.357 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:07.357 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:07.357 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:07.358 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:07.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:07.358 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:07.358 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:07.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:07.359 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:07.359 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:07.360 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:07.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:07.360 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:07Z" 2022-09-02T06:01:07.360 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:07.362 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:07Z" } 2022-09-02T06:01:07.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:07.363 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:07.363 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:07.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:12.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:12.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:12.366 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:12.426 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:12.427 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:12.483 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:12.482+0000 7f717c8c5600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:12.484 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:12.483+0000 7f717c8c5600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:12.498 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:12.499 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:12.499 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:12.499 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:12.500 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:12.500 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:12.500 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:12.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:12.501 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:12.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:12.501 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:12.501 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:12.502 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:12.502 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:12.502 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:12.503 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:12.503 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:12.503 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:12.504 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:12.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:12.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:12.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:12.505 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:12.505 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:12.506 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:12.506 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:12.506 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:12.506 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:12.507 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:12.507 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:12.508 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:12.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:12.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:12.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:12.510 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:12.510 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:12.511 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:12.511 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:12.511 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:12.511 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:12.512 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:12.512 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:12.512 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:12.512 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:12.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:12.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:12.513 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:12.514 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:12.514 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:12.514 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:12.514 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:12.515 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:12.515 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:12.515 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:12.515 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:12.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:12.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:12.516 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:12.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:12.517 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:12.517 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:12.517 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:12.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:12.519 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:12Z" 2022-09-02T06:01:12.519 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:12.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:12Z" } 2022-09-02T06:01:12.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:12.523 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:12.524 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:12.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:17.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:17.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:17.530 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:17.587 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:17.588 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:17.628 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:17.627+0000 7f56e3350600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:17.629 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:17.628+0000 7f56e3350600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:17.645 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:17.646 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:17.646 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:17.647 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:17.647 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:17.647 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:17.648 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:17.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:17.648 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:17.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:17.649 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:17.649 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:17.649 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:17.649 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:17.650 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:17.650 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:17.651 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:17.651 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:17.651 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:17.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:17.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:17.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:17.652 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:17.652 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:17.653 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:17.653 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:17.653 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:17.653 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:17.654 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:17.654 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:17.654 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:17.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:17.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:17.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:17.656 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:17.656 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:17.657 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:17.657 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:17.657 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:17.657 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:17.658 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:17.658 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:17.658 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:17.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:17.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:17.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:17.659 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:17.660 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:17.660 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:17.660 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:17.661 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:17.661 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:17.661 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:17.661 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:17.662 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:17.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:17.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:17.662 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:17.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:17.663 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:17.663 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:17.664 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:17.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:17.664 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:17Z" 2022-09-02T06:01:17.664 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:17.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:17Z" } 2022-09-02T06:01:17.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:17.667 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:17.668 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:17.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:22.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:22.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:22.672 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:22.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:22.730 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:22.762 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:22.761+0000 7f77385fb600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:22.763 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:22.762+0000 7f77385fb600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:22.777 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:22.778 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:22.778 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:22.778 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:22.779 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:22.779 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:22.779 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:22.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:22.780 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:22.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:22.781 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:22.781 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:22.782 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:22.782 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:22.782 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:22.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:22.783 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:22.783 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:22.783 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:22.784 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:22.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:22.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:22.785 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:22.785 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:22.785 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:22.785 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:22.786 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:22.786 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:22.786 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:22.787 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:22.787 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:22.787 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:22.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:22.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:22.788 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:22.788 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:22.788 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:22.789 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:22.789 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:22.789 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:22.789 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:22.790 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:22.790 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:22.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:22.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:22.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:22.791 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:22.791 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:22.791 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:22.792 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:22.792 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:22.792 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:22.793 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:22.793 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:22.793 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:22.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:22.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:22.794 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:22.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:22.794 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:22.795 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:22.795 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:22.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:22.795 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:22Z" 2022-09-02T06:01:22.796 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:22.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:22Z" } 2022-09-02T06:01:22.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:22.798 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:22.799 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:22.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:27.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:27.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:27.802 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:27.862 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:27.862 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:27.913 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:27.913+0000 7ff05d85b600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:27.914 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:27.914+0000 7ff05d85b600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:27.931 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:27.931 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:27.932 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:27.932 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:27.932 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:27.933 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:27.933 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:27.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:27.933 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:27.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:27.934 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:27.934 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:27.935 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:27.935 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:27.935 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:27.936 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:27.936 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:27.936 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:27.936 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:27.937 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:27.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:27.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:27.938 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:27.938 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:27.938 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:27.938 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:27.939 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:27.939 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:27.939 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:27.940 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:27.940 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:27.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:27.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:27.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:27.941 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:27.941 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:27.941 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:27.942 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:27.942 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:27.942 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:27.942 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:27.943 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:27.943 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:27.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:27.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:27.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:27.944 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:27.944 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:27.945 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:27.945 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:27.945 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:27.945 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:27.946 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:27.946 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:27.946 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:27.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:27.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:27.947 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:27.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:27.948 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:27.948 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:27.948 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:27.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:27.949 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:27Z" 2022-09-02T06:01:27.949 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:27.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:27Z" } 2022-09-02T06:01:27.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:27.952 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:27.952 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:27.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:32.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:32.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:32.955 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:33.014 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:33.015 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:33.048 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:33.047+0000 7f4e41c40600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:33.049 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:33.048+0000 7f4e41c40600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:33.064 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:33.065 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:33.065 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:33.066 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:33.066 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:33.067 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:33.067 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:33.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:33.068 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:33.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:33.069 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:33.070 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:33.070 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:33.071 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:33.071 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:33.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:33.072 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:33.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:33.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:33.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:33.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:33.075 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:33.076 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "", 2022-09-02T06:01:33.077 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:33.077 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:33.078 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:33.078 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "0.000000", 2022-09-02T06:01:33.078 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:33.079 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:33.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:33.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:33.080 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:33.080 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:33.080 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:33.080 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098293.625010_438.1", 2022-09-02T06:01:33.081 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:33.081 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:33.081 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:33.081 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T05:58:13.625010Z", 2022-09-02T06:01:33.082 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:33.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:33.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:33.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:33.083 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:33.083 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:33.083 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:33.083 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:33.084 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:33.084 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:33.084 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:33.084 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:33.085 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:33.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:33.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:33.085 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:33.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:33.086 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:33.086 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:33.086 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:33.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:33.087 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:33Z" 2022-09-02T06:01:33.087 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:33.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098293.625010_438.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T05:58:13.625010Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:33Z" } 2022-09-02T06:01:33.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098293.625010_438.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T05:58:13.625010Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:33.090 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:33.091 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '1_1662098293.625010_438.1', 3: ''} 2022-09-02T06:01:33.091 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1662098413.164890_2143.1 target=, shard=2 master=1_1662098436.167588_443.1 target=1_1662098293.625010_438.1 2022-09-02T06:01:38.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.095 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:38.152 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:38.153 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:38.194 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:38.194+0000 7f5bf05f6600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:38.195+0000 7f5bf05f6600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.217 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:38.217 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:38.218 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:38.218 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:38.218 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:38.219 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:38.219 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.219 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:38.220 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.220 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:38.220 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.221 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.221 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:38.221 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.221 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:38.222 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.222 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:38.222 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:38.222 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.223 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:38.224 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.224 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.225 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:38.225 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.225 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.225 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.226 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:38.226 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.226 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.227 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:38.227 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.228 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.228 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098436.167588_443.1", 2022-09-02T06:01:38.228 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.228 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.229 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.229 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:36.167588Z", 2022-09-02T06:01:38.229 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.229 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.230 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:38.230 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.231 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.231 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:38.231 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.232 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.232 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.232 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:38.233 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:38.233 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.233 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.233 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:38.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.234 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:38.234 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:38.234 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:38.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.235 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:38Z" 2022-09-02T06:01:38.235 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:38.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098436.167588_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:36.167588Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:38Z" } 2022-09-02T06:01:38.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098436.167588_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:36.167588Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:38.238 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:38.238 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:38.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T06:01:38.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T06:01:38.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.241 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:38.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:38.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:38.332 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T06:01:38.330+0000 7fb14191c600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.332 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T06:01:38.331+0000 7fb14191c600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.346 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T06:01:38.346 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T06:01:38.346 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T06:01:38.347 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T06:01:38.347 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T06:01:38.347 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:38.348 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T06:01:38.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.348 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T06:01:38.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T06:01:38.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:38.349 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:38.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:38.350 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:38.350 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T06:01:38.350 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:38.351 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:38.351 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T06:01:38.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:38.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.352 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T06:01:38.352 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:38.352 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:38.353 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:38.353 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:38.353 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T06:01:38.353 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:38.354 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:38.354 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T06:01:38.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:38.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.355 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T06:01:38.355 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:38.355 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:38.356 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098436.167588_443.1", 2022-09-02T06:01:38.356 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:38.356 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T06:01:38.356 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:38.357 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T06:00:36.167588Z", 2022-09-02T06:01:38.357 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T06:01:38.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:38.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.358 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T06:01:38.358 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:38.359 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:38.359 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:38.359 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:38.359 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T06:01:38.360 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:38.360 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:38.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T06:01:38.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:38.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:38.361 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T06:01:38.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.361 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T06:01:38.362 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T06:01:38.362 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T06:01:38.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.362 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T06:01:38Z" 2022-09-02T06:01:38.363 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T06:01:38.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098436.167588_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:36.167588Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:38Z" } 2022-09-02T06:01:38.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098436.167588_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:36.167588Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:38.365 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:38.366 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:38.366 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T06:01:38.366 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T06:01:38.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.368 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:38.419 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:38.420 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:38.458 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:38.456+0000 7f398eb97600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.458 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:38.457+0000 7f398eb97600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.471 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:38.472 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:38.472 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:38.472 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:38.473 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:38.473 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:38.473 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.474 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:38.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.474 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:38.475 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:38.476 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.477 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:38.479 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.480 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098436.167588_443.1", 2022-09-02T06:01:38.481 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:36.167588Z", 2022-09-02T06:01:38.482 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.483 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:38.484 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:38.485 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:38.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:38.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.488 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:38Z" 2022-09-02T06:01:38.488 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:38.489 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098436.167588_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:36.167588Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:38Z" } 2022-09-02T06:01:38.490 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098436.167588_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:36.167588Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:38.490 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:38.491 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098436.167588_443.1', 3: ''} 2022-09-02T06:01:38.491 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T06:01:38.492 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xljbkf-1 zones={test-zone2, test-zone1} 2022-09-02T06:01:38.493 DEBUG:boto:Method: POST 2022-09-02T06:01:38.493 DEBUG:boto:Path: / 2022-09-02T06:01:38.494 DEBUG:boto:Data: 2022-09-02T06:01:38.494 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.495 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2022-09-02T06:01:38.495 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.496 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.497 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:01:38.498 DEBUG:boto:Token: None 2022-09-02T06:01:38.498 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date ed230a6aadbf870a833e03715f6b596c826fb73e6286256f8479144a7be8343b 2022-09-02T06:01:38.499 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi160/aws4_request 3331feb631fb1991c8f100d7480aa9642bf75a98d911dc8a90aa68fe1f703131 2022-09-02T06:01:38.500 DEBUG:boto:Signature: ee26cfeba444c680922843f4b94e81d264cc49ffc1ee8d6eb814107b5a17a4d8 2022-09-02T06:01:38.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=ee26cfeba444c680922843f4b94e81d264cc49ffc1ee8d6eb814107b5a17a4d8', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.505 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027e2acf586084bac-0063119c42-1a20-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.505 DEBUG:boto:b'tx0000027e2acf586084bac-0063119c42-1a20-test-zone2045ff5f5-ed23-499b-a50c-10f72d907a15roles-xljbkf-1/arn:aws:iam:::role/roles-xljbkf-12022-09-02T06:00:13.110Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.507 DEBUG:boto:Method: POST 2022-09-02T06:01:38.507 DEBUG:boto:Path: / 2022-09-02T06:01:38.508 DEBUG:boto:Data: 2022-09-02T06:01:38.508 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.509 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.510 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.511 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:01:38.512 DEBUG:boto:Token: None 2022-09-02T06:01:38.512 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date ed230a6aadbf870a833e03715f6b596c826fb73e6286256f8479144a7be8343b 2022-09-02T06:01:38.513 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 76fd6624ac9462b9f3b63d7e7907d5ee98aee994da74f41453814191aaa66f3c 2022-09-02T06:01:38.514 DEBUG:boto:Signature: 3d0d9821a80fbbb57e3ec032185a7408ddb8377a086743008f187f68e73ab010 2022-09-02T06:01:38.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=3d0d9821a80fbbb57e3ec032185a7408ddb8377a086743008f187f68e73ab010', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.517 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009dfc269632a49899-0063119c42-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.517 DEBUG:boto:b'tx000009dfc269632a49899-0063119c42-1595-test-zone1045ff5f5-ed23-499b-a50c-10f72d907a15roles-xljbkf-1/arn:aws:iam:::role/roles-xljbkf-12022-09-02T06:00:13.110Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.518 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xljbkf-1 2022-09-02T06:01:38.519 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xljbkf-1 zones={test-zone2, test-zone1} 2022-09-02T06:01:38.520 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xljbkf-1 zones={test-zone3, test-zone1} 2022-09-02T06:01:38.520 DEBUG:boto:Method: POST 2022-09-02T06:01:38.521 DEBUG:boto:Path: / 2022-09-02T06:01:38.522 DEBUG:boto:Data: 2022-09-02T06:01:38.523 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.526 DEBUG:boto:Port: 8001 2022-09-02T06:01:38.527 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.528 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2022-09-02T06:01:38.529 DEBUG:boto:Token: None 2022-09-02T06:01:38.530 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20220902T060138Z host;x-amz-date ed230a6aadbf870a833e03715f6b596c826fb73e6286256f8479144a7be8343b 2022-09-02T06:01:38.531 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 0c7975054cef79a953068a3e0db829e891260d76bef67f013120f3ec87df0a27 2022-09-02T06:01:38.532 DEBUG:boto:Signature: aebc7abffd0f658a4a33435feceb430a0d2ebe0b4beedea10b5142b74719f604 2022-09-02T06:01:38.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=aebc7abffd0f658a4a33435feceb430a0d2ebe0b4beedea10b5142b74719f604', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2022-09-02T06:01:38.537 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a585f197635ad19e-0063119c42-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.538 DEBUG:boto:b'tx00000a585f197635ad19e-0063119c42-16ef-test-zone3045ff5f5-ed23-499b-a50c-10f72d907a15roles-xljbkf-1/arn:aws:iam:::role/roles-xljbkf-12022-09-02T06:00:13.194Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.539 DEBUG:boto:Method: POST 2022-09-02T06:01:38.540 DEBUG:boto:Path: / 2022-09-02T06:01:38.540 DEBUG:boto:Data: 2022-09-02T06:01:38.541 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.542 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.542 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.543 DEBUG:boto:Token: None 2022-09-02T06:01:38.544 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date ed230a6aadbf870a833e03715f6b596c826fb73e6286256f8479144a7be8343b 2022-09-02T06:01:38.544 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 76fd6624ac9462b9f3b63d7e7907d5ee98aee994da74f41453814191aaa66f3c 2022-09-02T06:01:38.545 DEBUG:boto:Signature: 3d0d9821a80fbbb57e3ec032185a7408ddb8377a086743008f187f68e73ab010 2022-09-02T06:01:38.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=3d0d9821a80fbbb57e3ec032185a7408ddb8377a086743008f187f68e73ab010', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.547 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000099a992ea3134e072-0063119c42-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.547 DEBUG:boto:b'tx0000099a992ea3134e072-0063119c42-1595-test-zone1045ff5f5-ed23-499b-a50c-10f72d907a15roles-xljbkf-1/arn:aws:iam:::role/roles-xljbkf-12022-09-02T06:00:13.110Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.549 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xljbkf-1 2022-09-02T06:01:38.549 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xljbkf-1 zones={test-zone3, test-zone1} 2022-09-02T06:01:38.550 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xljbkf-2 zones={test-zone1, test-zone2} 2022-09-02T06:01:38.550 DEBUG:boto:Method: POST 2022-09-02T06:01:38.551 DEBUG:boto:Path: / 2022-09-02T06:01:38.551 DEBUG:boto:Data: 2022-09-02T06:01:38.552 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.554 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.554 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.556 DEBUG:boto:Token: None 2022-09-02T06:01:38.557 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date 0d7a87d7d2ae430d855a90a9f2cd30208ccfc6c50d7c9340a1f96e6f040f6577 2022-09-02T06:01:38.558 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 17964fdf8677cfd60eff97cd1f3be794baea919a0a455f8dcb4e8b8ab7e2636d 2022-09-02T06:01:38.558 DEBUG:boto:Signature: 4637687747145ce9b6def6765081a37f96e5ffc3b392f28806378dd1a7773307 2022-09-02T06:01:38.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=4637687747145ce9b6def6765081a37f96e5ffc3b392f28806378dd1a7773307', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.561 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033d0af95c42d3bce-0063119c42-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.561 DEBUG:boto:b'tx0000033d0af95c42d3bce-0063119c42-1595-test-zone156f9b8c1-0946-4dc1-929d-0abccccde199roles-xljbkf-2/arn:aws:iam:::role/roles-xljbkf-22022-09-02T06:00:13.161Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.562 DEBUG:boto:Method: POST 2022-09-02T06:01:38.563 DEBUG:boto:Path: / 2022-09-02T06:01:38.563 DEBUG:boto:Data: 2022-09-02T06:01:38.564 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.564 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2022-09-02T06:01:38.565 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.566 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.566 DEBUG:boto:Token: None 2022-09-02T06:01:38.567 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date 0d7a87d7d2ae430d855a90a9f2cd30208ccfc6c50d7c9340a1f96e6f040f6577 2022-09-02T06:01:38.568 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi160/aws4_request 18b577250069dd6467f4ddc2267ff024ac5433cb49356ded0bbd9d0de360ec38 2022-09-02T06:01:38.568 DEBUG:boto:Signature: b8536c40c2ee971d5596c31f76db13f4f0636daed0421dbba85f748fc9b4a57a 2022-09-02T06:01:38.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=b8536c40c2ee971d5596c31f76db13f4f0636daed0421dbba85f748fc9b4a57a', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.571 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b8dd63dfba1d4990-0063119c42-1a20-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.572 DEBUG:boto:b'tx00000b8dd63dfba1d4990-0063119c42-1a20-test-zone256f9b8c1-0946-4dc1-929d-0abccccde199roles-xljbkf-2/arn:aws:iam:::role/roles-xljbkf-22022-09-02T06:00:13.161Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.573 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xljbkf-2 2022-09-02T06:01:38.574 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xljbkf-2 zones={test-zone1, test-zone2} 2022-09-02T06:01:38.574 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xljbkf-2 zones={test-zone3, test-zone2} 2022-09-02T06:01:38.575 DEBUG:boto:Method: POST 2022-09-02T06:01:38.576 DEBUG:boto:Path: / 2022-09-02T06:01:38.576 DEBUG:boto:Data: 2022-09-02T06:01:38.576 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.577 DEBUG:boto:Port: 8001 2022-09-02T06:01:38.578 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.579 DEBUG:boto:Token: None 2022-09-02T06:01:38.579 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20220902T060138Z host;x-amz-date 0d7a87d7d2ae430d855a90a9f2cd30208ccfc6c50d7c9340a1f96e6f040f6577 2022-09-02T06:01:38.580 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 28a4656859160682adbd8b2747dd47683be6f42795bc765ada5905e176f1c545 2022-09-02T06:01:38.581 DEBUG:boto:Signature: 7ceb7db4167984b0b413d68769a7ff95226863b81de1269207b96b0517c00494 2022-09-02T06:01:38.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=7ceb7db4167984b0b413d68769a7ff95226863b81de1269207b96b0517c00494', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2022-09-02T06:01:38.583 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000147edcac57986609-0063119c42-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.584 DEBUG:boto:b'tx00000147edcac57986609-0063119c42-16ef-test-zone356f9b8c1-0946-4dc1-929d-0abccccde199roles-xljbkf-2/arn:aws:iam:::role/roles-xljbkf-22022-09-02T06:00:13.392Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.585 DEBUG:boto:Method: POST 2022-09-02T06:01:38.586 DEBUG:boto:Path: / 2022-09-02T06:01:38.586 DEBUG:boto:Data: 2022-09-02T06:01:38.587 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.588 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2022-09-02T06:01:38.589 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.589 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.590 DEBUG:boto:Token: None 2022-09-02T06:01:38.591 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date 0d7a87d7d2ae430d855a90a9f2cd30208ccfc6c50d7c9340a1f96e6f040f6577 2022-09-02T06:01:38.591 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi160/aws4_request 18b577250069dd6467f4ddc2267ff024ac5433cb49356ded0bbd9d0de360ec38 2022-09-02T06:01:38.592 DEBUG:boto:Signature: b8536c40c2ee971d5596c31f76db13f4f0636daed0421dbba85f748fc9b4a57a 2022-09-02T06:01:38.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=b8536c40c2ee971d5596c31f76db13f4f0636daed0421dbba85f748fc9b4a57a', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.594 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000220eeb4afab80ae9-0063119c42-1a20-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.595 DEBUG:boto:b'tx00000220eeb4afab80ae9-0063119c42-1a20-test-zone256f9b8c1-0946-4dc1-929d-0abccccde199roles-xljbkf-2/arn:aws:iam:::role/roles-xljbkf-22022-09-02T06:00:13.161Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.596 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xljbkf-2 2022-09-02T06:01:38.596 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xljbkf-2 zones={test-zone3, test-zone2} 2022-09-02T06:01:38.597 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xljbkf-3 zones={test-zone1, test-zone3} 2022-09-02T06:01:38.597 DEBUG:boto:Method: POST 2022-09-02T06:01:38.598 DEBUG:boto:Path: / 2022-09-02T06:01:38.599 DEBUG:boto:Data: 2022-09-02T06:01:38.599 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.600 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.601 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.601 DEBUG:boto:Token: None 2022-09-02T06:01:38.602 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date bf37901586ba50fb2bf5a48fde05d7a4a480ea4cd66b2faf346968ae47533070 2022-09-02T06:01:38.602 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 5cb5387e23e3370e477bb61ec9b812e348f9ba45f9a7028076afe24f7328d34b 2022-09-02T06:01:38.603 DEBUG:boto:Signature: e514cedbebc44cbfdb8b2160b08eb192fe5df8720c83085d5ebd9ff84ee98398 2022-09-02T06:01:38.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=e514cedbebc44cbfdb8b2160b08eb192fe5df8720c83085d5ebd9ff84ee98398', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.605 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d12161048854226-0063119c42-1595-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.605 DEBUG:boto:b'tx000002d12161048854226-0063119c42-1595-test-zone1ea4daf4c-88ce-4aac-8e59-41ea3874cbf8roles-xljbkf-3/arn:aws:iam:::role/roles-xljbkf-32022-09-02T06:00:36.163Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.607 DEBUG:boto:Method: POST 2022-09-02T06:01:38.607 DEBUG:boto:Path: / 2022-09-02T06:01:38.608 DEBUG:boto:Data: 2022-09-02T06:01:38.608 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.609 DEBUG:boto:Port: 8001 2022-09-02T06:01:38.610 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.610 DEBUG:boto:Token: None 2022-09-02T06:01:38.611 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20220902T060138Z host;x-amz-date bf37901586ba50fb2bf5a48fde05d7a4a480ea4cd66b2faf346968ae47533070 2022-09-02T06:01:38.612 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 5d815dbeb80fcac6dc4af58da53c8046a777921f60218d80173b5c1f720b57e8 2022-09-02T06:01:38.612 DEBUG:boto:Signature: 802f55073e3a502fa4320ea3b7b8d69159553e4cb1e975d803c4c4c1d6386db4 2022-09-02T06:01:38.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=802f55073e3a502fa4320ea3b7b8d69159553e4cb1e975d803c4c4c1d6386db4', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2022-09-02T06:01:38.615 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079efd70dfa62195d-0063119c42-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.615 DEBUG:boto:b'tx0000079efd70dfa62195d-0063119c42-16ef-test-zone3ea4daf4c-88ce-4aac-8e59-41ea3874cbf8roles-xljbkf-3/arn:aws:iam:::role/roles-xljbkf-32022-09-02T06:00:36.163Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.616 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xljbkf-3 2022-09-02T06:01:38.617 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xljbkf-3 zones={test-zone1, test-zone3} 2022-09-02T06:01:38.617 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xljbkf-3 zones={test-zone2, test-zone3} 2022-09-02T06:01:38.618 DEBUG:boto:Method: POST 2022-09-02T06:01:38.618 DEBUG:boto:Path: / 2022-09-02T06:01:38.619 DEBUG:boto:Data: 2022-09-02T06:01:38.619 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.620 DEBUG:boto:Host: smithi160.front.sepia.ceph.com 2022-09-02T06:01:38.620 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.621 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.621 DEBUG:boto:Token: None 2022-09-02T06:01:38.622 DEBUG:boto:CanonicalRequest: POST / host:smithi160.front.sepia.ceph.com:8000 x-amz-date:20220902T060138Z host;x-amz-date bf37901586ba50fb2bf5a48fde05d7a4a480ea4cd66b2faf346968ae47533070 2022-09-02T06:01:38.623 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi160/aws4_request c495e7e9a63c13f8e108d90d113051d00fd84e49920f4a267c7c8ab2e09d6b60 2022-09-02T06:01:38.623 DEBUG:boto:Signature: 46ebf6070d458d460200b8c3e927e692c389909e7d3cbe22cb50c996ae2b333e 2022-09-02T06:01:38.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi160/aws4_request,SignedHeaders=host;x-amz-date,Signature=46ebf6070d458d460200b8c3e927e692c389909e7d3cbe22cb50c996ae2b333e', 'Host': 'smithi160.front.sepia.ceph.com:8000'} 2022-09-02T06:01:38.626 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000bf2545b753782da-0063119c42-1a20-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.626 DEBUG:boto:b'tx000000bf2545b753782da-0063119c42-1a20-test-zone2ea4daf4c-88ce-4aac-8e59-41ea3874cbf8roles-xljbkf-3/arn:aws:iam:::role/roles-xljbkf-32022-09-02T06:00:36.163Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.627 DEBUG:boto:Method: POST 2022-09-02T06:01:38.628 DEBUG:boto:Path: / 2022-09-02T06:01:38.628 DEBUG:boto:Data: 2022-09-02T06:01:38.629 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2022-09-02T06:01:38.630 DEBUG:boto:Port: 8001 2022-09-02T06:01:38.630 DEBUG:boto:Params: {'RoleName': 'roles-xljbkf-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2022-09-02T06:01:38.631 DEBUG:boto:Token: None 2022-09-02T06:01:38.632 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20220902T060138Z host;x-amz-date bf37901586ba50fb2bf5a48fde05d7a4a480ea4cd66b2faf346968ae47533070 2022-09-02T06:01:38.632 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20220902T060138Z 20220902/front/smithi099/aws4_request 5d815dbeb80fcac6dc4af58da53c8046a777921f60218d80173b5c1f720b57e8 2022-09-02T06:01:38.633 DEBUG:boto:Signature: 802f55073e3a502fa4320ea3b7b8d69159553e4cb1e975d803c4c4c1d6386db4 2022-09-02T06:01:38.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'X-Amz-Date': '20220902T060138Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=K2P3ZGZ9QHARN05S5UWB/20220902/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=802f55073e3a502fa4320ea3b7b8d69159553e4cb1e975d803c4c4c1d6386db4', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2022-09-02T06:01:38.635 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e71b71fdd949fb7-0063119c42-16ef-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.636 DEBUG:boto:b'tx000000e71b71fdd949fb7-0063119c42-16ef-test-zone3ea4daf4c-88ce-4aac-8e59-41ea3874cbf8roles-xljbkf-3/arn:aws:iam:::role/roles-xljbkf-32022-09-02T06:00:36.163Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2022-09-02T06:01:38.637 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xljbkf-3 2022-09-02T06:01:38.637 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xljbkf-3 zones={test-zone2, test-zone3} 2022-09-02T06:01:38.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2022-09-02T06:01:38.640 DEBUG:boto:path=/xljbkf-75/ 2022-09-02T06:01:38.641 DEBUG:boto:auth_path=/xljbkf-75/ 2022-09-02T06:01:38.641 DEBUG:boto:Method: PUT 2022-09-02T06:01:38.642 DEBUG:boto:Path: /xljbkf-75/ 2022-09-02T06:01:38.642 DEBUG:boto:Data: 2022-09-02T06:01:38.643 DEBUG:boto:Headers: {} 2022-09-02T06:01:38.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:38.644 DEBUG:boto:Port: 8000 2022-09-02T06:01:38.644 DEBUG:boto:Params: {} 2022-09-02T06:01:38.645 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:01:38.645 DEBUG:boto:Token: None 2022-09-02T06:01:38.646 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 06:01:38 GMT /xljbkf-75/ 2022-09-02T06:01:38.647 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:NYqcy3hktLPozedBj0yGtaf5q5E= 2022-09-02T06:01:38.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:01:38 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:NYqcy3hktLPozedBj0yGtaf5q5E=', 'Content-Length': '0'} 2022-09-02T06:01:38.680 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008eb85ef219e3a5e3-0063119c42-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:01:38 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:38.681 DEBUG:rgw_multi.tests:created bucket=xljbkf-75 2022-09-02T06:01:38.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:38.732 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:38.733 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:38.779 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T06:01:38.807 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T06:01:38.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.808 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:01:38.809 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:01:38.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.810 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:38.810 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T06:01:38.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.811 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:01:38.811 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:01:38.675176Z" 2022-09-02T06:01:38.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:38.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:38.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:01:38.812 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:01:38.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:38.813 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T06:01:38.814 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:38.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T06:01:38.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.816 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:38.869 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:38.870 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:38.903 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:38.902+0000 7f1e35670600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.904 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:38.903+0000 7f1e35670600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:38.918 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:38.918 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:38.918 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:38.919 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.920 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:38.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.922 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:38.923 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.923 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.923 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.924 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:38.925 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:38.925 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.926 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:38.927 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.928 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.929 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.931 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:38.931 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.932 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:01:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.933 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.934 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:01:38.935 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:38.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:38.937 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:38.938 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:38.939 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:38.940 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:38.941 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:38.942 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:38.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.943 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:38.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:38.945 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:38Z" 2022-09-02T06:01:38.945 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:38.946 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:38Z" } 2022-09-02T06:01:38.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:38.948 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:38.948 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:38.949 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-09-02T06:01:38.949 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-09-02T06:01:38.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:38.952 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:39.001 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:39.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T06:01:39.031+0000 7f1070230600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:39.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-09-02T06:01:39.032+0000 7f1070230600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:39.051 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2022-09-02T06:01:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2022-09-02T06:01:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2022-09-02T06:01:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2022-09-02T06:01:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2022-09-02T06:01:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T06:01:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2022-09-02T06:01:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2022-09-02T06:01:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2022-09-02T06:01:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T06:01:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2022-09-02T06:01:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T06:01:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T06:01:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2022-09-02T06:01:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:01:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T06:01:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:01:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 4 2022-09-02T06:01:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:01:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2022-09-02T06:01:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2022-09-02T06:01:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2022-09-02T06:01:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2022-09-02T06:01:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2022-09-02T06:01:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2022-09-02T06:01:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2022-09-02T06:01:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:01:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2022-09-02T06:01:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2022-09-02T06:01:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2022-09-02T06:01:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2022-09-02T06:01:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:01:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2022-09-02T06:01:39Z" 2022-09-02T06:01:39.073 INFO:teuthology.orchestra.run.smithi099.stdout:} 2022-09-02T06:01:39.075 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:39Z" } 2022-09-02T06:01:39.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:39.076 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:39.076 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:39.077 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-09-02T06:01:39.077 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-09-02T06:01:39.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:39.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:39.079 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:39.129 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:39.129 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:39.159 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:39.158+0000 7f1521744600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:39.160 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:39.159+0000 7f1521744600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:39.173 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:39.173 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:39.174 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:39.174 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:39.174 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:39.174 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:39.175 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:39.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:39.175 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:39.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:39.176 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:39.176 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:39.176 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:39.177 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:39.177 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:39.177 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:39.177 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:39.178 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:39.178 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:39.178 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:39.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:39.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:39.179 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:39.179 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:39.179 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:39.180 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:39.180 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:39.180 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:39.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:39.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:39.182 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:39.183 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:39.183 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:39.183 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.660929_444.1", 2022-09-02T06:01:39.184 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:39.184 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:39.184 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:39.184 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.660929Z", 2022-09-02T06:01:39.185 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:39.185 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:39.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:39.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:39.186 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:39.186 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:39.186 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:39.187 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:39.187 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:39.187 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:39.187 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:39.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:39.188 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:39.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:39.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:39.189 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:39.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:39.189 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:39.190 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:39.190 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:39.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:39.191 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:39Z" 2022-09-02T06:01:39.191 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:39.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.660929_444.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.660929Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:39Z" } 2022-09-02T06:01:39.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.660929_444.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.660929Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:39.194 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:39.194 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.660929_444.1', 3: ''} 2022-09-02T06:01:39.195 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1662098498.675176_447.1 target=1_1662098498.660929_444.1 2022-09-02T06:01:44.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:44.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:01:44.198 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:01:44.256 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:44.257 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:44.305 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:44.304+0000 7f1a38d07600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:44.306 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:44.305+0000 7f1a38d07600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:01:44.319 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:01:44.319 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:01:44.320 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:01:44.320 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:01:44.320 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:01:44.320 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:01:44.321 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:44.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:44.321 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:01:44.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:44.322 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:01:44.322 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:44.322 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:44.322 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:01:44.323 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:44.323 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:01:44.323 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:44.323 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:01:44.324 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:44.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:44.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:44.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:44.325 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:01:44.325 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:44.325 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:44.325 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:01:44.326 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:44.326 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:44.326 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:44.327 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:01:44.327 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:44.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:44.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:44.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:44.328 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:01:44.328 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:44.329 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:44.329 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:01:44.329 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:44.329 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:44.330 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:44.330 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:01:44.330 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:01:44.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:44.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:44.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:01:44.331 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:01:44.331 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:01:44.332 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:01:44.332 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:01:44.332 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:01:44.332 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:01:44.333 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:01:44.333 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:01:44.333 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:01:44.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:44.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:01:44.334 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:01:44.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:44.335 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:01:44.335 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:01:44.335 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:01:44.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:01:44.336 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:01:44Z" 2022-09-02T06:01:44.336 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:01:44.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:01:44Z" } 2022-09-02T06:01:44.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:01:44.339 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:44.339 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:01:44.340 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-09-02T06:01:44.341 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-09-02T06:01:44.341 INFO:teuthology.orchestra.run:waiting for 300 2022-09-02T06:01:44.342 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T06:01:44.340+0000 7fa8edf4b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 154135) UID: 0 2022-09-02T06:01:44.343 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T06:01:44.340+0000 7fa8f21b6600 -1 shutting down 2022-09-02T06:01:49.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-02T06:01:50.444 INFO:tasks.rgw.c2.client.0:Stopped 2022-09-02T06:01:50.445 DEBUG:boto:path=/xljbkf-75/ 2022-09-02T06:01:50.446 DEBUG:boto:auth_path=/xljbkf-75/ 2022-09-02T06:01:50.447 DEBUG:boto:Method: HEAD 2022-09-02T06:01:50.448 DEBUG:boto:Path: /xljbkf-75/ 2022-09-02T06:01:50.449 DEBUG:boto:Data: 2022-09-02T06:01:50.450 DEBUG:boto:Headers: {} 2022-09-02T06:01:50.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.453 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.453 DEBUG:boto:Params: {} 2022-09-02T06:01:50.455 DEBUG:boto:Token: None 2022-09-02T06:01:50.456 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/ 2022-09-02T06:01:50.457 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8= 2022-09-02T06:01:50.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8=', 'Content-Length': '0'} 2022-09-02T06:01:50.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042ea2028a6235070-0063119c4e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.463 DEBUG:boto:path=/xljbkf-75/obj11 2022-09-02T06:01:50.464 DEBUG:boto:auth_path=/xljbkf-75/obj11 2022-09-02T06:01:50.464 DEBUG:boto:Method: PUT 2022-09-02T06:01:50.465 DEBUG:boto:Path: /xljbkf-75/obj11 2022-09-02T06:01:50.465 DEBUG:boto:Data: 2022-09-02T06:01:50.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T06:01:50.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.467 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.468 DEBUG:boto:Params: {} 2022-09-02T06:01:50.468 DEBUG:boto:Token: None 2022-09-02T06:01:50.469 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/obj11 2022-09-02T06:01:50.470 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:19bos1rL5hVwKcyJZmkEn0qmN6I= 2022-09-02T06:01:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:19bos1rL5hVwKcyJZmkEn0qmN6I='} 2022-09-02T06:01:50.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe380f8e2c2e4197-0063119c4e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.484 DEBUG:boto:path=/xljbkf-75/ 2022-09-02T06:01:50.484 DEBUG:boto:auth_path=/xljbkf-75/ 2022-09-02T06:01:50.485 DEBUG:boto:Method: HEAD 2022-09-02T06:01:50.486 DEBUG:boto:Path: /xljbkf-75/ 2022-09-02T06:01:50.486 DEBUG:boto:Data: 2022-09-02T06:01:50.487 DEBUG:boto:Headers: {} 2022-09-02T06:01:50.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.488 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.489 DEBUG:boto:Params: {} 2022-09-02T06:01:50.489 DEBUG:boto:Token: None 2022-09-02T06:01:50.490 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/ 2022-09-02T06:01:50.492 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8= 2022-09-02T06:01:50.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8=', 'Content-Length': '0'} 2022-09-02T06:01:50.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccfee6d3b1d05665-0063119c4e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.497 DEBUG:boto:path=/xljbkf-75/obj22 2022-09-02T06:01:50.498 DEBUG:boto:auth_path=/xljbkf-75/obj22 2022-09-02T06:01:50.499 DEBUG:boto:Method: PUT 2022-09-02T06:01:50.499 DEBUG:boto:Path: /xljbkf-75/obj22 2022-09-02T06:01:50.500 DEBUG:boto:Data: 2022-09-02T06:01:50.500 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T06:01:50.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.502 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.502 DEBUG:boto:Params: {} 2022-09-02T06:01:50.503 DEBUG:boto:Token: None 2022-09-02T06:01:50.504 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/obj22 2022-09-02T06:01:50.504 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:tqJRx+77tEf5Ui2pzYEyJoHaLI8= 2022-09-02T06:01:50.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:tqJRx+77tEf5Ui2pzYEyJoHaLI8='} 2022-09-02T06:01:50.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048e672b57f515a56-0063119c4e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.513 DEBUG:boto:path=/xljbkf-75/ 2022-09-02T06:01:50.514 DEBUG:boto:auth_path=/xljbkf-75/ 2022-09-02T06:01:50.515 DEBUG:boto:Method: HEAD 2022-09-02T06:01:50.516 DEBUG:boto:Path: /xljbkf-75/ 2022-09-02T06:01:50.517 DEBUG:boto:Data: 2022-09-02T06:01:50.517 DEBUG:boto:Headers: {} 2022-09-02T06:01:50.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.520 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.520 DEBUG:boto:Params: {} 2022-09-02T06:01:50.522 DEBUG:boto:Token: None 2022-09-02T06:01:50.524 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/ 2022-09-02T06:01:50.525 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8= 2022-09-02T06:01:50.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8=', 'Content-Length': '0'} 2022-09-02T06:01:50.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f35410125a9a521-0063119c4e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.532 DEBUG:boto:path=/xljbkf-75/obj33 2022-09-02T06:01:50.533 DEBUG:boto:auth_path=/xljbkf-75/obj33 2022-09-02T06:01:50.534 DEBUG:boto:Method: PUT 2022-09-02T06:01:50.535 DEBUG:boto:Path: /xljbkf-75/obj33 2022-09-02T06:01:50.536 DEBUG:boto:Data: 2022-09-02T06:01:50.538 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T06:01:50.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.539 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.540 DEBUG:boto:Params: {} 2022-09-02T06:01:50.542 DEBUG:boto:Token: None 2022-09-02T06:01:50.543 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/obj33 2022-09-02T06:01:50.544 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:DbdbFsJgRwhqykbWlTLMVFrPwiI= 2022-09-02T06:01:50.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:DbdbFsJgRwhqykbWlTLMVFrPwiI='} 2022-09-02T06:01:50.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a5045ce0ef3d04-0063119c4e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.591 DEBUG:boto:path=/xljbkf-75/ 2022-09-02T06:01:50.592 DEBUG:boto:auth_path=/xljbkf-75/ 2022-09-02T06:01:50.592 DEBUG:boto:Method: HEAD 2022-09-02T06:01:50.593 DEBUG:boto:Path: /xljbkf-75/ 2022-09-02T06:01:50.593 DEBUG:boto:Data: 2022-09-02T06:01:50.594 DEBUG:boto:Headers: {} 2022-09-02T06:01:50.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.595 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.595 DEBUG:boto:Params: {} 2022-09-02T06:01:50.597 DEBUG:boto:Token: None 2022-09-02T06:01:50.597 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/ 2022-09-02T06:01:50.598 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8= 2022-09-02T06:01:50.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8=', 'Content-Length': '0'} 2022-09-02T06:01:50.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001678f0a6621b1c54-0063119c4e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.602 DEBUG:boto:path=/xljbkf-75/obj44 2022-09-02T06:01:50.603 DEBUG:boto:auth_path=/xljbkf-75/obj44 2022-09-02T06:01:50.603 DEBUG:boto:Method: PUT 2022-09-02T06:01:50.604 DEBUG:boto:Path: /xljbkf-75/obj44 2022-09-02T06:01:50.604 DEBUG:boto:Data: 2022-09-02T06:01:50.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T06:01:50.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.607 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.607 DEBUG:boto:Params: {} 2022-09-02T06:01:50.608 DEBUG:boto:Token: None 2022-09-02T06:01:50.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/obj44 2022-09-02T06:01:50.609 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:EiObFAeHxHYOOK4UJ2HfYDPtkwA= 2022-09-02T06:01:50.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:EiObFAeHxHYOOK4UJ2HfYDPtkwA='} 2022-09-02T06:01:50.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc2e8d5057c86cb2-0063119c4e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.619 DEBUG:boto:path=/xljbkf-75/ 2022-09-02T06:01:50.620 DEBUG:boto:auth_path=/xljbkf-75/ 2022-09-02T06:01:50.621 DEBUG:boto:Method: HEAD 2022-09-02T06:01:50.622 DEBUG:boto:Path: /xljbkf-75/ 2022-09-02T06:01:50.623 DEBUG:boto:Data: 2022-09-02T06:01:50.624 DEBUG:boto:Headers: {} 2022-09-02T06:01:50.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.626 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.627 DEBUG:boto:Params: {} 2022-09-02T06:01:50.628 DEBUG:boto:Token: None 2022-09-02T06:01:50.629 DEBUG:boto:StringToSign: HEAD Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/ 2022-09-02T06:01:50.630 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8= 2022-09-02T06:01:50.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:O69UJltCYhI/OFM/6csBIlyZRD8=', 'Content-Length': '0'} 2022-09-02T06:01:50.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b87754df62efb182-0063119c4e-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.635 DEBUG:boto:path=/xljbkf-75/obj55 2022-09-02T06:01:50.636 DEBUG:boto:auth_path=/xljbkf-75/obj55 2022-09-02T06:01:50.637 DEBUG:boto:Method: PUT 2022-09-02T06:01:50.637 DEBUG:boto:Path: /xljbkf-75/obj55 2022-09-02T06:01:50.638 DEBUG:boto:Data: 2022-09-02T06:01:50.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2022-09-02T06:01:50.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:01:50.642 DEBUG:boto:Port: 8000 2022-09-02T06:01:50.643 DEBUG:boto:Params: {} 2022-09-02T06:01:50.644 DEBUG:boto:Token: None 2022-09-02T06:01:50.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 02 Sep 2022 06:01:50 GMT /xljbkf-75/obj55 2022-09-02T06:01:50.646 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:deEWoC9N8qRhtCU1tUig2cgrllU= 2022-09-02T06:01:50.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 02 Sep 2022 06:01:50 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:deEWoC9N8qRhtCU1tUig2cgrllU='} 2022-09-02T06:01:50.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8ee812181084310-0063119c4e-1595-test-zone1'), ('Date', 'Fri, 02 Sep 2022 06:01:50 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:01:50.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:01:50.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-09-02T06:01:50.657 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-09-02T06:01:50.717 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:50.718 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:50.779 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:50.777+0000 7fef6fd95600 10 RGW-SYNC:data:init_data_sync_status: took lease 2022-09-02T06:01:50.807 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-09-02T06:01:50.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-09-02T06:01:50.812 INFO:tasks.rgw.c2.client.0:Started 2022-09-02T06:01:50.813 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:01:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T06:01:50.840 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:01:50.841 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:01:50.841 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T06:01:51.843 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:01:51.867 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:01:51.867 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:01:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T06:01:51.869 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T06:01:53.871 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:01:53.897 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:01:53.897 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:01:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-09-02T06:01:53.901 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi160.front.sepia.ceph.com port 8000: Connection refused 2022-09-02T06:01:54.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-09-02T06:01:57.902 DEBUG:teuthology.orchestra.run.smithi160:> curl http://smithi160.front.sepia.ceph.com:8000 2022-09-02T06:01:57.928 INFO:teuthology.orchestra.run.smithi160.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-09-02T06:01:57.928 INFO:teuthology.orchestra.run.smithi160.stderr: Dload Upload Total Spent Left Speed 2022-09-02T06:01:57.928 INFO:teuthology.orchestra.run.smithi160.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-09-02T06:01:57.929 INFO:teuthology.orchestra.run.smithi160.stdout:anonymous 2022-09-02T06:01:57.930 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=xljbkf-75 2022-09-02T06:01:57.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T06:01:57.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xljbkf-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-09-02T06:01:57.932 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xljbkf-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-09-02T06:01:58.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:01:58.028 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:01:58.072 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:01:58.071+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:03.073 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:03.071+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:08.073 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:08.072+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:13.074 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:13.072+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:18.074 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:18.073+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:23.075 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:23.074+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:28.076 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:28.075+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:33.076 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:33.076+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:38.077 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:38.076+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:43.077 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:43.077+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:48.076+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:53.078 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:53.077+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:02:58.079 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:02:58.077+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:03.079 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:03.078+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:08.080 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:08.078+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:13.080 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:13.079+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:18.082 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:18.079+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:23.081 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:23.080+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:28.082 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:28.080+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:33.083 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:33.081+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:38.083 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:38.081+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:43.084 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:43.083+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:48.085 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:48.083+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:53.085 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:53.083+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:03:58.086 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:03:58.084+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:01.132 INFO:tasks.rgw.c2.client.0.smithi160.stdout:*** Caught signal (Segmentation fault) ** 2022-09-02T06:04:01.132 INFO:tasks.rgw.c2.client.0.smithi160.stdout: in thread 7f7a45277700 thread_name:data-sync 2022-09-02T06:04:01.135 INFO:tasks.rgw.c2.client.0.smithi160.stdout: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:04:01.136 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7a6ef87cf0] 2022-09-02T06:04:01.136 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x15b) [0x7f7a6e6ec76b] 2022-09-02T06:04:01.137 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x56496aa39879] 2022-09-02T06:04:01.137 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x56496aa5eb6e] 2022-09-02T06:04:01.138 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x56496ab48e2f] 2022-09-02T06:04:01.138 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x56496ab49601] 2022-09-02T06:04:01.138 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x56496ab4a611] 2022-09-02T06:04:01.138 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x56496aa02aca] 2022-09-02T06:04:01.139 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x56496ac3c528] 2022-09-02T06:04:01.140 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x56496ac068c3] 2022-09-02T06:04:01.140 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7a6ef7d1ca] 2022-09-02T06:04:01.141 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 12: clone() 2022-09-02T06:04:01.141 INFO:tasks.rgw.c2.client.0.smithi160.stdout:2022-09-02T06:04:01.133+0000 7f7a45277700 -1 *** Caught signal (Segmentation fault) ** 2022-09-02T06:04:01.141 INFO:tasks.rgw.c2.client.0.smithi160.stdout: in thread 7f7a45277700 thread_name:data-sync 2022-09-02T06:04:01.142 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2022-09-02T06:04:01.142 INFO:tasks.rgw.c2.client.0.smithi160.stdout: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:04:01.142 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7a6ef87cf0] 2022-09-02T06:04:01.143 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x15b) [0x7f7a6e6ec76b] 2022-09-02T06:04:01.143 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x56496aa39879] 2022-09-02T06:04:01.143 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x56496aa5eb6e] 2022-09-02T06:04:01.143 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x56496ab48e2f] 2022-09-02T06:04:01.144 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x56496ab49601] 2022-09-02T06:04:01.144 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x56496ab4a611] 2022-09-02T06:04:01.144 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x56496aa02aca] 2022-09-02T06:04:01.144 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x56496ac3c528] 2022-09-02T06:04:01.145 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x56496ac068c3] 2022-09-02T06:04:01.145 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7a6ef7d1ca] 2022-09-02T06:04:01.145 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 12: clone() 2022-09-02T06:04:01.146 INFO:tasks.rgw.c2.client.0.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-02T06:04:01.146 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2022-09-02T06:04:01.152 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 0> 2022-09-02T06:04:01.133+0000 7f7a45277700 -1 *** Caught signal (Segmentation fault) ** 2022-09-02T06:04:01.153 INFO:tasks.rgw.c2.client.0.smithi160.stdout: in thread 7f7a45277700 thread_name:data-sync 2022-09-02T06:04:01.153 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2022-09-02T06:04:01.153 INFO:tasks.rgw.c2.client.0.smithi160.stdout: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:04:01.154 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7a6ef87cf0] 2022-09-02T06:04:01.154 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x15b) [0x7f7a6e6ec76b] 2022-09-02T06:04:01.154 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x56496aa39879] 2022-09-02T06:04:01.155 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x56496aa5eb6e] 2022-09-02T06:04:01.155 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x56496ab48e2f] 2022-09-02T06:04:01.155 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x56496ab49601] 2022-09-02T06:04:01.155 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x56496ab4a611] 2022-09-02T06:04:01.156 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x56496aa02aca] 2022-09-02T06:04:01.156 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x56496ac3c528] 2022-09-02T06:04:01.156 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x56496ac068c3] 2022-09-02T06:04:01.157 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7a6ef7d1ca] 2022-09-02T06:04:01.157 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 12: clone() 2022-09-02T06:04:01.157 INFO:tasks.rgw.c2.client.0.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-02T06:04:01.158 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2022-09-02T06:04:01.164 INFO:tasks.rgw.c2.client.0.smithi160.stdout: -122> 2022-09-02T06:04:01.133+0000 7f7a45277700 -1 *** Caught signal (Segmentation fault) ** 2022-09-02T06:04:01.165 INFO:tasks.rgw.c2.client.0.smithi160.stdout: in thread 7f7a45277700 thread_name:data-sync 2022-09-02T06:04:01.166 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2022-09-02T06:04:01.166 INFO:tasks.rgw.c2.client.0.smithi160.stdout: ceph version 17.0.0-14689-gafd42238 (afd422385fab8119a539d289dd1aeaad4c056fe4) quincy (dev) 2022-09-02T06:04:01.167 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7a6ef87cf0] 2022-09-02T06:04:01.167 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2: (std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)+0x15b) [0x7f7a6e6ec76b] 2022-09-02T06:04:01.167 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 3: (std::_Rb_tree::marker_entry>, std::_Select1st::marker_entry> >, std::less, std::allocator::marker_entry> > >::_M_erase_aux(std::_Rb_tree_const_iterator::marker_entry> >, std::_Rb_tree_const_iterator::marker_entry> >)+0x49) [0x56496aa39879] 2022-09-02T06:04:01.168 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 4: (RGWBucketSyncSingleEntryCR::operate(DoutPrefixProvider const*)+0x8fe) [0x56496aa5eb6e] 2022-09-02T06:04:01.168 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x18f) [0x56496ab48e2f] 2022-09-02T06:04:01.168 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x521) [0x56496ab49601] 2022-09-02T06:04:01.169 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x56496ab4a611] 2022-09-02T06:04:01.169 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x20a) [0x56496aa02aca] 2022-09-02T06:04:01.169 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x56496ac3c528] 2022-09-02T06:04:01.169 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 10: (RGWRadosThread::Worker::entry()+0xb3) [0x56496ac068c3] 2022-09-02T06:04:01.170 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 11: /lib64/libpthread.so.0(+0x81ca) [0x7f7a6ef7d1ca] 2022-09-02T06:04:01.170 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 12: clone() 2022-09-02T06:04:01.170 INFO:tasks.rgw.c2.client.0.smithi160.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-02T06:04:01.171 INFO:tasks.rgw.c2.client.0.smithi160.stdout: 2022-09-02T06:04:01.252 INFO:tasks.rgw.c2.client.0.smithi160.stderr:daemon-helper: command crashed with signal 11 2022-09-02T06:04:03.086 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:03.084+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:05.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-09-02T06:04:08.087 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:08.085+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:11.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2022-09-02T06:04:13.087 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:13.086+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:17.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2022-09-02T06:04:18.088 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:18.086+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:22.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2022-09-02T06:04:23.088 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:23.087+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:28.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:28.087+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:28.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2022-09-02T06:04:33.089 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:33.088+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:33.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2022-09-02T06:04:38.090 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:38.089+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:39.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2022-09-02T06:04:43.091 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:43.089+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:45.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2022-09-02T06:04:48.092 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:48.091+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:50.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2022-09-02T06:04:53.092 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:53.091+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:04:56.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2022-09-02T06:04:58.092 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:04:58.091+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:02.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2022-09-02T06:05:03.093 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:03.091+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:07.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2022-09-02T06:05:08.094 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:08.092+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:13.094 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:13.093+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:13.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2022-09-02T06:05:18.095 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:18.093+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:18.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2022-09-02T06:05:23.096 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:23.094+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:24.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~78s 2022-09-02T06:05:28.099 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:28.095+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:30.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2022-09-02T06:05:33.097 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:33.095+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:35.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2022-09-02T06:05:38.097 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:38.096+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:41.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2022-09-02T06:05:43.098 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:43.097+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:46.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2022-09-02T06:05:48.099 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:48.097+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:52.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~107s 2022-09-02T06:05:53.099 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:53.099+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:05:58.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2022-09-02T06:05:58.100 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:05:58.099+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:03.101 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:03.100+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:03.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2022-09-02T06:06:08.102 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:08.100+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:09.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2022-09-02T06:06:13.102 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:13.100+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:14.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2022-09-02T06:06:18.102 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:18.101+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:20.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2022-09-02T06:06:23.103 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:23.101+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:26.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2022-09-02T06:06:28.103 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:28.102+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:31.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2022-09-02T06:06:33.104 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:33.103+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:37.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2022-09-02T06:06:38.105 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:38.103+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:42.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2022-09-02T06:06:43.106 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:43.104+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:48.107 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:48.105+0000 7fec64e88600 1 waiting to reach incremental sync.. 2022-09-02T06:06:48.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2022-09-02T06:06:53.107 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:53.106+0000 7fec64e88600 -1 bucket checkpoint timed out waiting to reach incremental sync 2022-09-02T06:06:53.107 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:53.106+0000 7fec64e88600 0 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T06:06:53.108 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:53.106+0000 7fec64e88600 -1 bucket sync checkpoint failed: (110) Connection timed out 2022-09-02T06:06:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 110 2022-09-02T06:06:53.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2022-09-02T06:06:53.148 DEBUG:boto:path=/xljbkf-76/ 2022-09-02T06:06:53.152 DEBUG:boto:auth_path=/xljbkf-76/ 2022-09-02T06:06:53.153 DEBUG:boto:Method: PUT 2022-09-02T06:06:53.154 DEBUG:boto:Path: /xljbkf-76/ 2022-09-02T06:06:53.154 DEBUG:boto:Data: 2022-09-02T06:06:53.155 DEBUG:boto:Headers: {} 2022-09-02T06:06:53.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:06:53.156 DEBUG:boto:Port: 8000 2022-09-02T06:06:53.157 DEBUG:boto:Params: {} 2022-09-02T06:06:53.158 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:06:53.159 DEBUG:boto:Token: None 2022-09-02T06:06:53.160 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 06:06:53 GMT /xljbkf-76/ 2022-09-02T06:06:53.161 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:OFtMCUUkaoRsyM56Lvyic/EbaZE= 2022-09-02T06:06:53.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:06:53 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:OFtMCUUkaoRsyM56Lvyic/EbaZE=', 'Content-Length': '0'} 2022-09-02T06:06:53.194 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000529b30694aa35653-0063119d7d-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:06:53 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:06:53.196 DEBUG:rgw_multi.tests:created bucket=xljbkf-76 2022-09-02T06:06:53.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:06:53.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:06:53.199 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:06:53.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:06:53.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:06:53.304 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T06:06:53.329 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T06:06:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:06:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:06:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:06:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:06:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:06:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:06:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T06:06:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:06:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:06:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098813.188335_451.1", 2022-09-02T06:06:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:06:53.188335Z" 2022-09-02T06:06:53.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:06:53.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:06:53.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:06:53.333 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:06:53.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:06:53.334 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T06:06:53.335 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:06:53.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T06:06:53.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:06:53.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:06:53.338 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:06:53.396 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:06:53.397 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:06:53.438 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:53.437+0000 7f9441bca600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:06:53.439 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:53.438+0000 7f9441bca600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:06:53.456 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:06:53.457 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:06:53.457 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:06:53.457 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:06:53.457 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:06:53.458 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:06:53.458 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:06:53.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:53.459 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:06:53.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:53.460 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:06:53.460 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:53.461 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:53.461 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:06:53.462 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:53.462 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:06:53.463 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:53.463 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:06:53.464 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:06:53.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:53.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:53.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:53.466 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:06:53.466 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:53.467 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:53.467 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:06:53.468 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:53.468 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:06:53.469 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:53.469 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:06:53.470 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:06:53.470 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:53.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:53.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:53.472 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:06:53.472 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:53.472 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:53.473 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:06:53.473 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:53.473 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:06:53.473 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:53.474 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:06:53.474 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:06:53.474 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:53.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:53.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:53.475 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:06:53.476 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:53.476 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:53.476 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:06:53.476 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:53.477 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:06:53.477 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:53.477 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:06:53.477 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:06:53.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:53.478 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:53.478 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:06:53.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:53.479 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:06:53.479 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:06:53.479 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:06:53.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:53.480 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:06:53Z" 2022-09-02T06:06:53.480 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:06:53.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:06:53Z" } 2022-09-02T06:06:53.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:06:53.483 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:06:53.483 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:06:53.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:06:54.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~168s 2022-09-02T06:06:58.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:06:58.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:06:58.487 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:06:58.543 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:06:58.544 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:06:58.598 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:58.596+0000 7f7d8c106600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:06:58.599 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:06:58.598+0000 7f7d8c106600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:06:58.613 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:06:58.613 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:06:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:06:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:06:58.614 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:06:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:06:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:06:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:58.615 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:06:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:06:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:58.616 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:06:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:06:58.617 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:06:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:06:58.618 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:06:58.619 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:06:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:58.620 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:06:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:06:58.621 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:06:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:58.622 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:06:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:06:58.623 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:06:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:58.624 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:06:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:06:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:58.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:06:58.626 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:06:58.626 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:06:58.626 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:06:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:06:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:06:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:06:58.627 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:06:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:06:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:06:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:58.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:06:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:06:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:06:58.629 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:06:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:06:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:06:58.630 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:06:58Z" 2022-09-02T06:06:58.631 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:06:58.632 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:06:58Z" } 2022-09-02T06:06:58.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:06:58.634 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:06:58.634 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:06:58.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:06:59.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2022-09-02T06:07:03.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:03.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:03.637 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:03.695 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:03.696 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:03.733 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:03.733+0000 7faa48c3d600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:03.734 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:03.734+0000 7faa48c3d600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:03.753 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:03.754 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:03.754 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:03.754 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:03.755 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:03.755 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:03.755 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:03.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:03.756 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:03.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:03.756 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:03.756 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:03.757 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:03.757 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:03.757 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:03.757 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:03.758 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:03.758 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:03.758 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:03.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:03.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:03.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:03.759 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:03.759 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:03.760 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:03.760 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:03.760 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:03.760 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:03.761 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:03.761 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:03.761 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:03.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:03.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:03.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:03.762 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:03.762 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:03.763 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:03.763 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:03.763 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:03.763 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:03.764 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:03.764 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:03.764 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:03.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:03.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:03.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:03.765 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:03.765 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:03.766 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:03.766 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:03.766 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:03.766 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:03.767 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:03.767 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:03.767 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:03.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:03.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:03.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:03.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:03.768 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:03.769 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:03.769 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:03.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:03.769 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:03Z" 2022-09-02T06:07:03.770 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:03.771 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:03Z" } 2022-09-02T06:07:03.772 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:03.773 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:03.773 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:03.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:05.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~179s 2022-09-02T06:07:08.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:08.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:08.777 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:08.833 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:08.834 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:08.874 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:08.874+0000 7f351aaa5600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:08.875 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:08.875+0000 7f351aaa5600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:08.893 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:08Z" 2022-09-02T06:07:08.914 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:08.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:08Z" } 2022-09-02T06:07:08.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:08.917 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:08.918 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:08.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:10.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2022-09-02T06:07:13.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:13.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:13.922 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:13.986 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:14.015 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:14.014+0000 7fd70205f600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:14.016 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:14.015+0000 7fd70205f600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:14Z" 2022-09-02T06:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:14.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:14Z" } 2022-09-02T06:07:14.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:14.058 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:14.059 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:14.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:16.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2022-09-02T06:07:19.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:19.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:19.061 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:19.119 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:19.119 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:19.156 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:19.155+0000 7f9ebd706600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:19.157 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:19.156+0000 7f9ebd706600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:19.181 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:19.181 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:19.182 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:19.182 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:19.182 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:19.182 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:19.183 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:19.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:19.183 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:19.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:19.184 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:19.184 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:19.184 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:19.184 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:19.185 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:19.185 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:19.185 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:19.185 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:19.186 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:19.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:19.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:19.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:19.187 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:19.187 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:19.187 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:19.187 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:19.188 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:19.188 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:19.188 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:19.188 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:19.189 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:19.189 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:19.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:19.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:19.190 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:19.190 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:19.190 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:19.191 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:19.191 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:19.191 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:19.191 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:19.192 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:19.192 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:19.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:19.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:19.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:19.193 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:19.194 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:19.194 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:19.194 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:19.194 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:19.195 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:19.195 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:19.195 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:19.195 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:19.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:19.196 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:19.196 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:19.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:19.197 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:19.197 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:19.197 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:19.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:19.198 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:19Z" 2022-09-02T06:07:19.198 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:19.199 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:19Z" } 2022-09-02T06:07:19.200 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:19.200 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:19.201 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:19.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:22.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2022-09-02T06:07:24.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:24.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:24.204 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:24.262 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:24.263 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:24.318 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:24.317+0000 7ff7b9e42600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:24.319 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:24.318+0000 7ff7b9e42600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:24.335 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:24.335 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:24.335 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:24.336 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:24.336 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:24.336 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:24.336 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:24.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:24.337 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:24.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:24.337 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:24.338 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:24.338 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:24.338 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:24.339 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:24.339 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:24.339 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:24.340 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:24.340 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:24.341 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:24.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:24.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:24.341 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:24.342 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:24.342 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:24.343 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:24.343 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:24.343 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:24.344 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:24.344 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:24.345 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:24.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:24.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:24.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:24.346 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:24.346 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:24.346 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:24.347 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:24.347 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:24.347 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:24.348 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:24.348 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:24.348 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:24.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:24.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:24.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:24.351 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:24.351 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:24.352 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:24.352 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:24.352 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:24.352 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:24.353 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:24.353 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:24.353 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:24.353 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:24.354 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:24.354 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:24.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:24.354 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:24.355 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:24.355 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:24.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:24.356 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:24Z" 2022-09-02T06:07:24.356 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:24.358 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:24Z" } 2022-09-02T06:07:24.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:24.359 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:24.360 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:24.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:27.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~202s 2022-09-02T06:07:29.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:29.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:29.364 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:29.422 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:29.423 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:29.459 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:29.458+0000 7f0c1966f600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:29.460 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:29.459+0000 7f0c1966f600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:29.476 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:29.476 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:29.477 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:29.477 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:29.477 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:29.478 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:29.478 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:29.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:29.478 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:29.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:29.479 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:29.479 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:29.480 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:29.480 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:29.480 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:29.480 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:29.481 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:29.481 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:29.481 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:29.481 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:29.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:29.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:29.482 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:29.483 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:29.483 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:29.483 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:29.483 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:29.484 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:29.484 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:29.484 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:29.484 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:29.485 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:29.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:29.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:29.486 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:29.486 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:29.486 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:29.486 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:29.487 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:29.487 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:29.487 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:29.488 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:29.488 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:29.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:29.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:29.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:29.489 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:29.489 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:29.490 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:29.490 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:29.490 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:29.491 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:29.491 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:29.491 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:29.491 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:29.492 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:29.492 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:29.492 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:29.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:29.493 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:29.493 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:29.494 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:29.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:29.495 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:29Z" 2022-09-02T06:07:29.496 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:29.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:29Z" } 2022-09-02T06:07:29.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:29.500 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:29.501 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:29.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:33.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2022-09-02T06:07:34.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:34.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:34.505 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:34.564 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:34.564 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:34.608 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:34.607+0000 7fde042bd600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:34.609 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:34.608+0000 7fde042bd600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:34.629 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:34.630 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:34.630 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:34.630 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:34.630 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:34.631 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:34.631 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:34.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:34.632 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:34.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:34.632 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:34.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:34.633 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:34.633 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:34.633 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:34.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:34.634 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:34.634 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:34.634 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:34.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:34.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:34.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:34.635 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:34.635 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:34.636 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:34.636 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:34.636 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:34.636 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:34.637 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:34.637 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:34.637 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:34.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:34.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:34.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:34.638 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:34.639 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:34.639 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:34.639 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:34.639 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:34.640 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:34.640 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:34.640 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:34.640 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:34.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:34.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:34.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:34.641 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:34.642 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:34.642 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:34.642 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:34.643 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:34.643 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:34.643 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:34.643 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:34.644 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:34.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:34.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:34.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:34.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:34.645 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:34.645 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:34.645 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:34.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:34.646 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:34Z" 2022-09-02T06:07:34.646 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:34.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:34Z" } 2022-09-02T06:07:34.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:34.649 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:34.649 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:34.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:39.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2022-09-02T06:07:39.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:39.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:39.654 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:39.713 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:39.714 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:39.747 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:39.745+0000 7f345843d600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:39.748 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:39.747+0000 7f345843d600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:39.766 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:39.766 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:39.767 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:39.767 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:39.768 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:39.768 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:39.769 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:39.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:39.769 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:39.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:39.770 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:39.770 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:39.771 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:39.771 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:39.771 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:39.771 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:39.772 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:39.772 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:39.772 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:39.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:39.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:39.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:39.773 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:39.773 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:39.774 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:39.774 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:39.774 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:39.775 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:39.775 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:39.775 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:39.775 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:39.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:39.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:39.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:39.776 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:39.777 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:39.777 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:39.777 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:39.777 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:39.778 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:39.778 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:39.778 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:39.778 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:39.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:39.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:39.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:39.779 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:39.780 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:39.780 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:39.780 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:39.780 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:39.781 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:39.781 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:39.781 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:39.781 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:39.782 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:39.783 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:39.784 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:39Z" 2022-09-02T06:07:39.784 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:39.786 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:39Z" } 2022-09-02T06:07:39.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:39.787 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:39.787 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:39.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:44.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2022-09-02T06:07:44.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:44.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:44.791 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:44.846 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:44.846 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:44.881 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:44.879+0000 7f3c54176600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:44.882 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:44.881+0000 7f3c54176600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:44.902 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:44.902 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:44.902 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:44.903 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:44.903 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:44.903 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:44.903 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:44.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:44.904 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:44.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:44.904 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:44.905 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:44.905 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:44.905 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:44.905 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:44.906 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:44.906 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:44.907 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:44.907 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:44.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:44.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:44.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:44.908 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:44.908 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:44.908 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:44.909 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:44.909 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:44.909 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:44.909 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:44.910 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:44.910 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:44.910 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:44.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:44.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:44.911 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:44.912 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:44.912 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:44.912 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:44.912 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:44.913 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:44.913 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:44.913 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:44.913 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:44.914 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:44.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:44.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:44.914 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:44.915 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:44.915 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:44.915 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:44.915 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:44.916 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:44.916 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:44.916 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:44.916 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:44.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:44.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:44.917 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:44.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:44.918 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:44.918 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:44.918 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:44.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:44.919 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:44Z" 2022-09-02T06:07:44.919 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:44.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:44Z" } 2022-09-02T06:07:44.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:44.922 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:44.922 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:44.923 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:49.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:49.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:49.926 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:49.984 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:49.984 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:50.018 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:50.016+0000 7fa7cf116600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:50.018 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:50.017+0000 7fa7cf116600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:50.034 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:50.034 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:50.035 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:50.035 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:50.035 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:50.035 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:50.036 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:50.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:50.037 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:50.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:50.037 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:50.038 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:50.038 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:50.038 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:50.039 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:50.039 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:50.039 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:50.040 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:50.040 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:50.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:50.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:50.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:50.041 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:50.041 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:50.042 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:50.042 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:50.042 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:50.043 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:50.043 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:50.043 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:50.043 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:50.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:50.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:50.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:50.044 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:50.045 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:50.045 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:50.045 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:50.046 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:50.046 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:50.047 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:50.047 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:50.047 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:50.048 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:50.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:50.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:50.050 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:50.050 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:50.050 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:50.052 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:50.052 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:50.053 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:50.053 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:50.054 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:50.054 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:50.055 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:50.055 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:50.056 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:50.057 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:50.057 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:50.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:50.058 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:50Z" 2022-09-02T06:07:50.059 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:50.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:50Z" } 2022-09-02T06:07:50.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:50.062 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:50.063 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:50.063 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:50.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2022-09-02T06:07:55.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:55.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:07:55.067 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:07:55.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:07:55.124 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:07:55.178 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:55.176+0000 7f5c60753600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:55.179 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:07:55.178+0000 7f5c60753600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:07:55.203 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:07:55.203 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:07:55.204 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:07:55.204 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:07:55.205 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:07:55.205 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:07:55.206 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:55.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:55.207 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:07:55.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:55.207 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:07:55.208 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:55.208 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:55.208 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:07:55.209 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:55.209 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:07:55.209 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:55.210 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:07:55.210 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:55.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:55.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:55.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:55.211 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:07:55.211 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:55.212 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:55.212 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:07:55.212 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:55.213 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:55.213 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:55.213 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:07:55.214 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:55.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:55.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:55.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:07:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:55.215 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:07:55.216 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:55.216 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:55.216 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:07:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:07:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:55.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:55.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:07:55.218 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:07:55.218 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:07:55.219 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:07:55.219 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:07:55.219 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:07:55.219 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:07:55.220 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:07:55.220 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:07:55.220 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:07:55.220 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:07:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:07:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:07:55.222 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:07:55.222 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:07:55.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:07:55.223 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:07:55Z" 2022-09-02T06:07:55.223 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:07:55.224 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:07:55Z" } 2022-09-02T06:07:55.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:07:55.225 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:07:55.226 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:07:55.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:07:55.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2022-09-02T06:08:00.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:00.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:00.229 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:00.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:00.286 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:00.326 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:00.324+0000 7fd54b298600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:00.327 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:00.325+0000 7fd54b298600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:00.341 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:00.342 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:00.342 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:00.342 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:00.342 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:00.343 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:00.343 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:00.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:00.343 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:00.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:00.344 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:00.344 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:00.345 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:00.345 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:00.345 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:00.346 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:00.346 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:00.346 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:00.346 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:00.347 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:00.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:00.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:00.347 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:00.348 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:00.348 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:00.348 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:00.348 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:00.349 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:00.349 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:00.349 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:00.349 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:00.350 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:00.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:00.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:00.351 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:00.351 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:00.351 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:00.352 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:00.352 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:00.352 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:00.353 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:00.354 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:00.355 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:00.356 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:00.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:00.358 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:00.358 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:00.358 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:00.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:00.359 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:00Z" 2022-09-02T06:08:00.359 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:00.360 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:00Z" } 2022-09-02T06:08:00.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:00.363 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:00.363 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:00.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:01.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2022-09-02T06:08:05.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:05.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:05.367 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:05.424 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:05.425 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:05.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:05.468+0000 7f68de155600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:05.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:05.469+0000 7f68de155600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:05.489 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:05.490 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:05.490 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:05.491 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:05.491 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:05.491 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:05.492 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:05.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:05.492 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:05.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:05.493 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:05.493 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:05.493 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:05.494 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:05.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:05.496 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:05.497 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:05.497 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:05.497 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:05.498 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:05.498 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:05.498 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:05.499 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:05.499 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:05.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:05.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:05.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:05.500 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:05.500 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:05.500 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:05.501 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:05.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:05.503 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:05.504 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:05.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:05.506 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:05.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:05.508 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:05Z" 2022-09-02T06:08:05.508 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:05.509 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:05Z" } 2022-09-02T06:08:05.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:05.511 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:05.511 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:05.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:07.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2022-09-02T06:08:10.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:10.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:10.516 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:10.574 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:10.574 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:10.608 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:10.607+0000 7fb9d0bcd600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:10.608 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:10.608+0000 7fb9d0bcd600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:10.623 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:10.623 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:10.624 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:10.624 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:10.624 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:10.624 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:10.625 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:10.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:10.625 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:10.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:10.626 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:10.626 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:10.626 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:10.627 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:10.627 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:10.627 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:10.627 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:10.628 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:10.628 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:10.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:10.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:10.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:10.629 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:10.629 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:10.630 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:10.630 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:10.630 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:10.630 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:10.631 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:10.631 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:10.631 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:10.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:10.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:10.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:10.632 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:10.632 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:10.633 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:10.633 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:10.633 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:10.633 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:10.634 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:10.634 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:10.634 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:10.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:10.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:10.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:10.635 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:10.636 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:10.636 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:10.636 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:10.636 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:10.637 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:10.637 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:10.637 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:10.637 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:10.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:10.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:10.638 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:10.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:10.639 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:10.639 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:10.639 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:10.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:10.640 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:10Z" 2022-09-02T06:08:10.640 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:10.642 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:10Z" } 2022-09-02T06:08:10.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:10.644 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:10.645 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:10.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:12.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2022-09-02T06:08:15.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:15.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:15.650 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:15.707 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:15.707 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:15.739 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:15.738+0000 7ff2dd5e1600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:15.740 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:15.740+0000 7ff2dd5e1600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:15.759 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:15.759 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:15.760 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:15.761 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:15.762 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:15.763 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:15.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:15.765 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:15.766 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:15.767 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:15.768 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:15.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:15.770 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:15.771 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:15.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:15.773 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:15.774 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:15.775 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:15.776 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:15.777 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:15.778 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:15.779 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:15Z" 2022-09-02T06:08:15.779 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:15.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:15Z" } 2022-09-02T06:08:15.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:15.782 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:15.783 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:15.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:18.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~252s 2022-09-02T06:08:20.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:20.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:20.786 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:20.843 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:20.844 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:20.887 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:20.885+0000 7f1f1a271600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:20.888 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:20.886+0000 7f1f1a271600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:20.904 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:20.905 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:20.905 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:20.906 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:20.907 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:20.908 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:20.908 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:20.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:20.909 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:20.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:20.910 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:20.911 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:20.912 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:20.913 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:20.913 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:20.914 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:20.914 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:20.915 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:20.915 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:20.916 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:20.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:20.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:20.917 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:20.918 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:20.918 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:20.919 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:20.919 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:20.920 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:20.920 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:20.921 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:20.921 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:20.921 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:20.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:20.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:20.922 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:20.923 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:20.923 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:20.923 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:20.924 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:20.924 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:20.924 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:20.925 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:20.925 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:20.925 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:20.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:20.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:20.926 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:20.927 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:20.927 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:20.927 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:20.928 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:20.928 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:20.929 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:20.929 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:20.929 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:20.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:20.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:20.930 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:20.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:20.931 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:20.931 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:20.931 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:20.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:20.932 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:20Z" 2022-09-02T06:08:20.932 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:20.934 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:20Z" } 2022-09-02T06:08:20.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:20.936 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:20.937 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:20.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:23.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~258s 2022-09-02T06:08:25.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:25.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:25.940 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:25.999 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:26.000 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:26.050 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:26.048+0000 7f3e89c1b600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:26.050 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:26.049+0000 7f3e89c1b600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:26.068 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:26.069 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:26.069 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:26.069 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:26.070 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:26.070 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:26.070 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:26.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:26.071 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:26.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:26.071 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:26.071 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:26.072 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:26.072 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:26.072 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:26.072 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:26.073 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:26.073 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:26.073 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:26.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:26.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:26.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:26.074 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:26.074 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:26.075 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:26.075 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:26.076 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:26.076 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:26.076 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:26.077 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:26.077 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:26.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:26.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:26.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:26.078 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:26.078 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:26.078 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:26.079 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:26.079 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:26.079 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:26.079 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:26.080 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:26.080 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:26.080 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:26.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:26.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:26.081 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:26.081 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:26.081 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:26.082 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:26.082 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:26.082 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:26.082 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:26.083 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:26.083 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:26.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:26.083 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:26.084 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:26.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:26.084 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:26.084 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:26.085 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:26.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:26.085 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:26Z" 2022-09-02T06:08:26.086 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:26.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:26Z" } 2022-09-02T06:08:26.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:26.089 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:26.089 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:26.090 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:29.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~264s 2022-09-02T06:08:31.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:31.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:31.092 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:31.151 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:31.151 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:31.194+0000 7f7d5a097600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:31.196 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:31.195+0000 7f7d5a097600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:31.214 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:31.214 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:31.215 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:31.215 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:31.215 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:31.215 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:31.216 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:31.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:31.217 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:31.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:31.217 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:31.217 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:31.218 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:31.218 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:31.218 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:31.219 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:31.219 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:31.219 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:31.219 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:31.220 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:31.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:31.220 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:31.220 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:31.221 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:31.221 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:31.221 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:31.221 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:31.222 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:31.222 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:31.222 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:31.223 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:31.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:31.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:31.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:31.224 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:31.224 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:31.224 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:31.225 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:31.225 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:31.225 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:31.225 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:31.226 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:31.226 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:31.226 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:31.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:31.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:31.227 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:31.228 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:31.228 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:31.228 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:31.228 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:31.229 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:31.229 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:31.229 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:31.229 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:31.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:31.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:31.230 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:31.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:31.231 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:31.231 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:31.232 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:31.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:31.232 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:31Z" 2022-09-02T06:08:31.232 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:31.234 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:31Z" } 2022-09-02T06:08:31.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:31.235 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:31.235 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:31.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:35.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~269s 2022-09-02T06:08:36.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:36.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:36.239 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:36.296 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:36.296 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:36.336 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:36.334+0000 7f55be3e2600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:36.337 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:36.335+0000 7f55be3e2600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:36.352 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:36.353 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:36.353 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:36.354 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:36.354 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:36.355 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:36.356 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:36.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:36.357 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:36.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:36.358 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:36.358 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:36.359 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:36.359 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:36.359 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:36.360 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:36.360 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:36.361 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:36.361 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:36.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:36.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:36.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:36.363 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:36.364 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:36.365 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:36.365 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:36.366 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:36.366 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:36.367 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:36.367 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:36.367 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:36.368 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:36.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:36.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:36.369 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:36.370 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:36.370 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:36.371 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:36.371 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:36.372 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:36.372 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:36.373 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:36.373 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:36.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:36.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:36.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:36.375 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:36.376 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:36.376 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:36.377 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:36.377 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:36.378 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:36.378 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:36.379 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:36.379 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:36.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:36.380 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:36.381 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:36.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:36.382 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:36.382 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:36.383 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:36.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:36.384 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:36Z" 2022-09-02T06:08:36.384 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:36.386 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:36Z" } 2022-09-02T06:08:36.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:36.389 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:36.389 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:36.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:40.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2022-09-02T06:08:41.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:41.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:41.394 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:41.451 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:41.452 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:41.504 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:41.503+0000 7f14ea854600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:41.505 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:41.504+0000 7f14ea854600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:41.529 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:41.529 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:41.530 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:41.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:41.532 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:41.533 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:41.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:41.535 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:41.536 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:41.537 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:41.538 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:41.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:41.540 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:41.541 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:41.542 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:41.543 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:41.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:41.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:41.546 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:41.547 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:41.548 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:41.549 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:41.550 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:41.551 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:41.552 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:41.553 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:41.554 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:41.555 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:41.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:41.557 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:41Z" 2022-09-02T06:08:41.557 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:41.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:41Z" } 2022-09-02T06:08:41.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:41.560 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:41.561 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:41.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:46.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2022-09-02T06:08:46.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:46.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:46.564 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:46.620 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:46.621 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:46.662 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:46.660+0000 7f8519f84600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:46.662 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:46.661+0000 7f8519f84600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:46.678 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:46.678 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:46.678 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:46.679 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:46.679 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:46.679 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:46.680 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:46.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:46.680 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:46.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:46.681 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:46.681 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:46.681 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:46.681 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:46.682 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:46.682 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:46.682 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:46.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:46.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:46.684 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:46.685 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:46.686 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:46.686 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:46.686 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:46.686 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:46.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:46.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:46.688 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:46.688 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:46.689 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:46.690 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:46.690 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:46.691 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:46.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:46.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:46.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:46.693 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:46.693 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:46.694 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:46.694 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:46.695 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:46.695 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:46.696 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:46.696 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:46.697 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:46.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:46.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:46.699 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:46.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:46.700 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:46.700 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:46.701 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:46.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:46.702 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:46Z" 2022-09-02T06:08:46.702 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:46.704 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:46Z" } 2022-09-02T06:08:46.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:46.706 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:46.707 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:46.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:51.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:51.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:51.712 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:51.769 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:51.769 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:51.809 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:51.807+0000 7faeecf9b600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:51.809 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:51.808+0000 7faeecf9b600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:51.824 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:51.825 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:51.826 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:51.828 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:51.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:51.830 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:51.831 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:51.832 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:51.833 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:51.834 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:51.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:51.836 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:51.837 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:51.838 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:51.839 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:51.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:51.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:51.842 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:51.843 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:51.844 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:51.845 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:51.846 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:51.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:51.848 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:51.849 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:51.850 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:51.851 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:51.852 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:51.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:51.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:51.855 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:51.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:51.857 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:51Z" 2022-09-02T06:08:51.857 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:51.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:51Z" } 2022-09-02T06:08:51.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:51.862 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:51.863 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:51.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:51.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2022-09-02T06:08:56.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:56.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:08:56.867 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:08:56.925 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:08:56.925 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:08:56.978 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:56.976+0000 7fc844acc600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:56.979 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:08:56.977+0000 7fc844acc600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:08:56.996 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:08:56.996 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:08:56.997 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:08:56.997 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:08:56.997 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:08:56.998 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:08:56.998 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:56.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:56.999 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:08:57.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:57.000 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:08:57.001 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:57.001 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:57.001 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:08:57.001 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:57.002 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:08:57.002 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:57.002 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:08:57.003 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:57.003 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:57.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:57.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:57.004 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:08:57.004 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:57.004 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:57.004 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:08:57.005 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:57.005 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:57.005 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:57.005 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:08:57.006 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:57.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:57.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:57.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:57.007 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:08:57.007 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:57.007 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:57.007 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:08:57.008 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:57.008 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:57.008 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:57.008 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:08:57.009 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:08:57.009 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:57.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:57.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:08:57.009 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:08:57.010 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:08:57.010 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:08:57.010 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:08:57.010 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:08:57.011 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:08:57.011 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:08:57.011 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:08:57.011 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:08:57.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:57.012 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:08:57.012 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:08:57.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:57.013 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:08:57.013 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:08:57.013 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:08:57.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:08:57.014 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:08:56Z" 2022-09-02T06:08:57.014 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:08:57.015 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:08:56Z" } 2022-09-02T06:08:57.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:08:57.016 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:08:57.017 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:08:57.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:08:57.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2022-09-02T06:09:02.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:09:02.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:09:02.021 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:09:02.081 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:09:02.082 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:09:02.114 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:09:02.113+0000 7fdefa2df600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:09:02.115 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:09:02.114+0000 7fdefa2df600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:09:02.139 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:09:02.140 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:09:02.140 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:09:02.141 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:09:02.141 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:09:02.141 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:09:02.142 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:09:02.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:02.142 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:09:02.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:02.143 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:09:02.143 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:02.143 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:02.144 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:09:02.144 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:02.144 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:09:02.144 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:02.145 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:09:02.145 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:09:02.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:02.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:02.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:02.146 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:09:02.146 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:02.147 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:02.147 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:09:02.147 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:02.147 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:09:02.148 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:02.148 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:09:02.148 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:09:02.148 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:02.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:02.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:02.149 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:09:02.149 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:02.150 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:02.150 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:09:02.150 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:02.151 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:09:02.151 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:02.151 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:09:02.151 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:09:02.152 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:02.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:02.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:02.152 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:09:02.153 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:02.153 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:02.153 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:09:02.153 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:02.154 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:09:02.154 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:02.154 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:09:02.155 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:09:02.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:02.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:02.155 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:09:02.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:02.156 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:09:02.156 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:09:02.157 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:09:02.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:02.157 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:09:02Z" 2022-09-02T06:09:02.158 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:09:02.159 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:09:02Z" } 2022-09-02T06:09:02.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:09:02.160 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:09:02.161 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:09:02.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:09:03.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2022-09-02T06:09:07.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:09:07.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:09:07.164 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:09:07.220 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:09:07.221 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:09:07.263 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:09:07.262+0000 7f31252a3600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:09:07.264 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:09:07.263+0000 7f31252a3600 20 RGW-SYNC:meta: read sync status 2022-09-02T06:09:07.278 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2022-09-02T06:09:07.279 INFO:teuthology.orchestra.run.smithi160.stdout: "sync_status": { 2022-09-02T06:09:07.279 INFO:teuthology.orchestra.run.smithi160.stdout: "info": { 2022-09-02T06:09:07.280 INFO:teuthology.orchestra.run.smithi160.stdout: "status": "sync", 2022-09-02T06:09:07.280 INFO:teuthology.orchestra.run.smithi160.stdout: "num_shards": 4, 2022-09-02T06:09:07.280 INFO:teuthology.orchestra.run.smithi160.stdout: "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", 2022-09-02T06:09:07.281 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:09:07.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:07.281 INFO:teuthology.orchestra.run.smithi160.stdout: "markers": [ 2022-09-02T06:09:07.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:07.282 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 0, 2022-09-02T06:09:07.282 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:07.282 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:07.282 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092292.280197_71.1", 2022-09-02T06:09:07.283 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:07.283 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 1, 2022-09-02T06:09:07.283 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:07.283 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:18:12.280197Z", 2022-09-02T06:09:07.284 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:09:07.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:07.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:07.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:07.285 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 1, 2022-09-02T06:09:07.285 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:07.285 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:07.285 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:09:07.286 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:07.286 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:09:07.286 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:07.287 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:00:13.164890Z", 2022-09-02T06:09:07.287 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:09:07.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:07.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:07.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:07.288 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 2, 2022-09-02T06:09:07.289 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:07.289 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:07.289 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662098498.675176_447.1", 2022-09-02T06:09:07.289 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:07.289 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:09:07.290 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:07.290 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T06:01:38.675176Z", 2022-09-02T06:09:07.290 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 4 2022-09-02T06:09:07.291 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:07.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:07.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-09-02T06:09:07.291 INFO:teuthology.orchestra.run.smithi160.stdout: "key": 3, 2022-09-02T06:09:07.292 INFO:teuthology.orchestra.run.smithi160.stdout: "val": { 2022-09-02T06:09:07.292 INFO:teuthology.orchestra.run.smithi160.stdout: "state": 1, 2022-09-02T06:09:07.292 INFO:teuthology.orchestra.run.smithi160.stdout: "marker": "1_1662092421.771218_140.1", 2022-09-02T06:09:07.292 INFO:teuthology.orchestra.run.smithi160.stdout: "next_step_marker": "", 2022-09-02T06:09:07.293 INFO:teuthology.orchestra.run.smithi160.stdout: "total_entries": 0, 2022-09-02T06:09:07.293 INFO:teuthology.orchestra.run.smithi160.stdout: "pos": 0, 2022-09-02T06:09:07.293 INFO:teuthology.orchestra.run.smithi160.stdout: "timestamp": "2022-09-02T04:20:21.771218Z", 2022-09-02T06:09:07.293 INFO:teuthology.orchestra.run.smithi160.stdout: "realm_epoch": 2 2022-09-02T06:09:07.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:07.294 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-09-02T06:09:07.294 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-09-02T06:09:07.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:07.295 INFO:teuthology.orchestra.run.smithi160.stdout: "full_sync": { 2022-09-02T06:09:07.295 INFO:teuthology.orchestra.run.smithi160.stdout: "total": 1, 2022-09-02T06:09:07.295 INFO:teuthology.orchestra.run.smithi160.stdout: "complete": 1 2022-09-02T06:09:07.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-09-02T06:09:07.296 INFO:teuthology.orchestra.run.smithi160.stdout: "current_time": "2022-09-02T06:09:07Z" 2022-09-02T06:09:07.296 INFO:teuthology.orchestra.run.smithi160.stdout:} 2022-09-02T06:09:07.297 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b798a9cd-aa0e-480c-bb07-05f02abf3f41", "realm_epoch": 4 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1662092292.280197_71.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2022-09-02T04:18:12.280197Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1662098413.164890_2143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:00:13.164890Z", "realm_epoch": 4 } }, { "key": 2, "val": { "state": 1, "marker": "1_1662098498.675176_447.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T06:01:38.675176Z", "realm_epoch": 4 } }, { "key": 3, "val": { "state": 1, "marker": "1_1662092421.771218_140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2022-09-02T04:20:21.771218Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2022-09-02T06:09:07Z" } 2022-09-02T06:09:07.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1662092292.280197_71.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2022-09-02T04:18:12.280197Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1662098413.164890_2143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:00:13.164890Z', 'realm_epoch': 4}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1662098498.675176_447.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T06:01:38.675176Z', 'realm_epoch': 4}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1662092421.771218_140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2022-09-02T04:20:21.771218Z', 'realm_epoch': 2}}] 2022-09-02T06:09:07.299 DEBUG:rgw_multi.tests:log_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098813.188335_451.1', 3: ''} 2022-09-02T06:09:07.299 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662098498.675176_447.1', 3: ''} 2022-09-02T06:09:07.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1662098813.188335_451.1 target=1_1662098498.675176_447.1 2022-09-02T06:09:08.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2022-09-02T06:09:08.730 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-09-02T06:09:09.335 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2022-09-02T06:09:09.338 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2022-09-02T06:09:09.339 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2022-09-02T06:09:09.339 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2022-09-02T06:09:09.340 INFO:tasks.rgw.c2.client.1:Sent signal 15 2022-09-02T06:09:09.340 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2022-09-02T06:09:09.342 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T06:09:09.335+0000 7f7cb9b0b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 104274) UID: 0 2022-09-02T06:09:09.342 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T06:09:09.335+0000 7f7cb9b0b700 -1 osd.0 63 *** Got signal Terminated *** 2022-09-02T06:09:09.342 INFO:tasks.ceph.c2.osd.0.smithi160.stderr:2022-09-02T06:09:09.335+0000 7f7cb9b0b700 -1 osd.0 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-02T06:09:09.343 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T06:09:09.338+0000 7fa575af4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 104282) UID: 0 2022-09-02T06:09:09.343 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T06:09:09.338+0000 7fa575af4700 -1 osd.1 63 *** Got signal Terminated *** 2022-09-02T06:09:09.344 INFO:tasks.ceph.c2.osd.1.smithi160.stderr:2022-09-02T06:09:09.338+0000 7fa575af4700 -1 osd.1 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-02T06:09:09.344 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T06:09:09.338+0000 7f8ea2e41700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 104324) UID: 0 2022-09-02T06:09:09.345 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T06:09:09.338+0000 7f8ea2e41700 -1 osd.2 63 *** Got signal Terminated *** 2022-09-02T06:09:09.345 INFO:tasks.ceph.c2.osd.2.smithi160.stderr:2022-09-02T06:09:09.338+0000 7f8ea2e41700 -1 osd.2 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-09-02T06:09:09.345 INFO:tasks.ceph.c2.mon.a.smithi160.stderr:2022-09-02T06:09:09.339+0000 7f3d85228700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 103910) UID: 0 2022-09-02T06:09:09.346 INFO:tasks.ceph.c2.mon.a.smithi160.stderr:2022-09-02T06:09:09.339+0000 7f3d85228700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-02T06:09:09.347 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2022-09-02T06:09:09.339+0000 7fd2a96d3700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 112978) UID: 0 2022-09-02T06:09:09.347 INFO:tasks.rgw.c2.client.1.smithi160.stdout:2022-09-02T06:09:09.339+0000 7fd2ad93e600 -1 shutting down 2022-09-02T06:09:12.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:09:12.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:09:12.303 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:09:12.361 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:09:12.361 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:14:12.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:14:12.361+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:19:12.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:19:12.362+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:24:12.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:24:12.362+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:29:12.364 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:29:12.363+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:34:12.367 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:34:12.363+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:39:12.365 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:39:12.362+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:44:12.366 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:44:12.364+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:49:12.366 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:49:12.364+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:54:12.367 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:54:12.364+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:59:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T06:59:12.426 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T06:59:12.365+0000 7f64edd5b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T06:59:12.427 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T06:59:12.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2022-09-02T06:59:12.437 DEBUG:boto:path=/xljbkf-77/ 2022-09-02T06:59:12.441 DEBUG:boto:auth_path=/xljbkf-77/ 2022-09-02T06:59:12.443 DEBUG:boto:Method: PUT 2022-09-02T06:59:12.443 DEBUG:boto:Path: /xljbkf-77/ 2022-09-02T06:59:12.444 DEBUG:boto:Data: 2022-09-02T06:59:12.444 DEBUG:boto:Headers: {} 2022-09-02T06:59:12.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T06:59:12.445 DEBUG:boto:Port: 8000 2022-09-02T06:59:12.446 DEBUG:boto:Params: {} 2022-09-02T06:59:12.447 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T06:59:12.448 DEBUG:boto:Token: None 2022-09-02T06:59:12.449 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 06:59:12 GMT /xljbkf-77/ 2022-09-02T06:59:12.451 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:Tk4lWsac/vtl8sfB94grLghixhA= 2022-09-02T06:59:12.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 06:59:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:Tk4lWsac/vtl8sfB94grLghixhA=', 'Content-Length': '0'} 2022-09-02T06:59:12.487 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b09f4bea91f3c11-006311a9c0-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 06:59:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T06:59:12.489 DEBUG:rgw_multi.tests:created bucket=xljbkf-77 2022-09-02T06:59:12.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:59:12.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:59:12.494 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:59:12.556 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:59:12.557 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T06:59:12.604 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T06:59:12.633 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T06:59:12.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:59:12.634 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:59:12.634 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:59:12.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:59:12.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:59:12.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T06:59:12.636 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T06:59:12.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:59:12.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:59:12.637 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662101952.484782_469.1", 2022-09-02T06:59:12.637 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:59:12.484782Z" 2022-09-02T06:59:12.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T06:59:12.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T06:59:12.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T06:59:12.639 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T06:59:12.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T06:59:12.639 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T06:59:12.640 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662101952.484782_469.1', 3: ''} 2022-09-02T06:59:12.641 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T06:59:12.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:59:12.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T06:59:12.645 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T06:59:12.706 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T06:59:12.707 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T07:04:12.706 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:04:12.705+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:09:12.707 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:09:12.705+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:14:12.708 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:14:12.705+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:19:12.708 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:19:12.706+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:24:12.708 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:24:12.706+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:29:12.711 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:29:12.706+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:34:12.722 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:34:12.707+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:39:12.711 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:39:12.707+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:44:12.713 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:44:12.707+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:49:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T07:49:12.767 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:49:12.708+0000 7f5abde2c600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:49:12.768 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T07:49:12.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2022-09-02T07:49:12.777 DEBUG:boto:path=/xljbkf-78/ 2022-09-02T07:49:12.782 DEBUG:boto:auth_path=/xljbkf-78/ 2022-09-02T07:49:12.783 DEBUG:boto:Method: PUT 2022-09-02T07:49:12.785 DEBUG:boto:Path: /xljbkf-78/ 2022-09-02T07:49:12.786 DEBUG:boto:Data: 2022-09-02T07:49:12.786 DEBUG:boto:Headers: {} 2022-09-02T07:49:12.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T07:49:12.788 DEBUG:boto:Port: 8000 2022-09-02T07:49:12.789 DEBUG:boto:Params: {} 2022-09-02T07:49:12.791 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T07:49:12.793 DEBUG:boto:Token: None 2022-09-02T07:49:12.794 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 07:49:12 GMT /xljbkf-78/ 2022-09-02T07:49:12.796 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:txMmNR0p/Rlr2UdycbWWB4uTnS0= 2022-09-02T07:49:12.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 07:49:12 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:txMmNR0p/Rlr2UdycbWWB4uTnS0=', 'Content-Length': '0'} 2022-09-02T07:49:12.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002538fd15ebfc13d6-006311b578-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 07:49:12 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T07:49:12.840 DEBUG:rgw_multi.tests:created bucket=xljbkf-78 2022-09-02T07:49:12.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T07:49:12.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T07:49:12.844 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T07:49:12.899 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T07:49:12.900 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T07:49:12.944 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T07:49:12.963 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T07:49:12.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T07:49:12.964 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T07:49:12.965 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T07:49:12.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T07:49:12.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T07:49:12.965 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T07:49:12.966 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T07:49:12.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T07:49:12.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T07:49:12.966 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662104952.830477_483.1", 2022-09-02T07:49:12.967 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T07:49:12.830477Z" 2022-09-02T07:49:12.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T07:49:12.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T07:49:12.968 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T07:49:12.968 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T07:49:12.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T07:49:12.968 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T07:49:12.970 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662104952.830477_483.1', 3: ''} 2022-09-02T07:49:12.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T07:49:12.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T07:49:12.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T07:49:12.974 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T07:49:13.030 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T07:49:13.031 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T07:54:13.034 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:54:13.030+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T07:59:13.036 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T07:59:13.031+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:04:13.036 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:04:13.031+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:09:13.038 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:09:13.033+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:14:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:14:13.033+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:19:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:19:13.034+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:24:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:24:13.035+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:29:13.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:29:13.036+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:34:13.039 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:34:13.036+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:39:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T08:39:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:39:13.036+0000 7f2ad69d0600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:39:13.105 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T08:39:13.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2022-09-02T08:39:13.111 DEBUG:boto:path=/xljbkf-79/ 2022-09-02T08:39:13.115 DEBUG:boto:auth_path=/xljbkf-79/ 2022-09-02T08:39:13.116 DEBUG:boto:Method: PUT 2022-09-02T08:39:13.117 DEBUG:boto:Path: /xljbkf-79/ 2022-09-02T08:39:13.118 DEBUG:boto:Data: 2022-09-02T08:39:13.118 DEBUG:boto:Headers: {} 2022-09-02T08:39:13.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2022-09-02T08:39:13.119 DEBUG:boto:Port: 8000 2022-09-02T08:39:13.120 DEBUG:boto:Params: {} 2022-09-02T08:39:13.121 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2022-09-02T08:39:13.122 DEBUG:boto:Token: None 2022-09-02T08:39:13.123 DEBUG:boto:StringToSign: PUT Fri, 02 Sep 2022 08:39:13 GMT /xljbkf-79/ 2022-09-02T08:39:13.124 DEBUG:boto:Signature: AWS K2P3ZGZ9QHARN05S5UWB:jHG+CPvQNdlGv0EOB4MVXrHeEgo= 2022-09-02T08:39:13.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.9 Linux/5.4.0-113-generic', 'Date': 'Fri, 02 Sep 2022 08:39:13 GMT', 'Authorization': 'AWS K2P3ZGZ9QHARN05S5UWB:jHG+CPvQNdlGv0EOB4MVXrHeEgo=', 'Content-Length': '0'} 2022-09-02T08:39:13.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020211a20b9f0d325-006311c131-1595-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 02 Sep 2022 08:39:13 GMT'), ('Connection', 'Keep-Alive')] 2022-09-02T08:39:13.164 DEBUG:rgw_multi.tests:created bucket=xljbkf-79 2022-09-02T08:39:13.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T08:39:13.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T08:39:13.168 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T08:39:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T08:39:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T08:39:13.491 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T08:39:13.511 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T08:39:13.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T08:39:13.513 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T08:39:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T08:39:13.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T08:39:13.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T08:39:13.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T08:39:13.516 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T08:39:13.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T08:39:13.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T08:39:13.517 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T08:39:13.517 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T08:39:13.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T08:39:13.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T08:39:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T08:39:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T08:39:13.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T08:39:13.522 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T08:39:13.580 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T08:39:13.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T08:39:13.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T08:39:13.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T08:39:13.583 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T08:39:13.643 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T08:39:13.644 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T08:44:13.645 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:44:13.642+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:49:13.647 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:49:13.644+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:54:13.645 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:54:13.643+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T08:59:13.646 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T08:59:13.644+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:04:13.650 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:04:13.644+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:09:13.648 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:09:13.646+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:14:13.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:14:13.646+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:19:13.650 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:19:13.647+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:24:13.652 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:24:13.646+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:29:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T09:29:13.723 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:29:13.647+0000 7fb64c7e7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:29:13.724 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T09:29:13.727 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2022-09-02T09:29:13.729 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2022-09-02T09:29:13.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T09:29:13.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T09:29:13.736 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T09:29:13.817 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T09:29:13.818 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T09:29:13.863 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T09:29:13.876 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T09:29:13.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T09:29:13.877 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T09:29:13.877 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T09:29:13.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T09:29:13.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T09:29:13.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T09:29:13.879 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T09:29:13.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T09:29:13.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T09:29:13.880 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T09:29:13.880 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T09:29:13.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T09:29:13.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T09:29:13.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T09:29:13.881 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T09:29:13.881 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T09:29:13.882 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T09:29:13.883 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T09:29:13.884 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T09:29:13.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T09:29:13.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T09:29:13.886 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T09:29:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T09:29:13.939 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T09:34:13.942 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:34:13.939+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:39:13.949 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:39:13.939+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:44:13.945 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:44:13.940+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:49:13.944 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:49:13.940+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:54:13.952 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:54:13.941+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T09:59:13.945 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T09:59:13.942+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:04:13.946 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:04:13.942+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:09:13.947 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:09:13.942+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:14:13.948 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:14:13.943+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:19:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T10:19:14.032 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:19:13.945+0000 7fdabbe84600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:19:14.033 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T10:19:14.037 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2022-09-02T10:19:14.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T10:19:14.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T10:19:14.044 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T10:19:14.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T10:19:14.127 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T10:19:14.181 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T10:19:14.203 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T10:19:14.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T10:19:14.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T10:19:14.204 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T10:19:14.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T10:19:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T10:19:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T10:19:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T10:19:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T10:19:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T10:19:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T10:19:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T10:19:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T10:19:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T10:19:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T10:19:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T10:19:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T10:19:14.208 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T10:19:14.210 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T10:19:14.212 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T10:19:14.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T10:19:14.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T10:19:14.213 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T10:19:14.269 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T10:19:14.270 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T10:24:14.273 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:24:14.269+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:29:14.276 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:29:14.269+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:34:14.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:34:14.269+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:39:14.274 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:39:14.270+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:44:14.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:44:14.270+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:49:14.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:49:14.271+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:54:14.279 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:54:14.272+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T10:59:14.274 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T10:59:14.271+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:04:14.275 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:04:14.272+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:09:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T11:09:14.351 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:09:14.273+0000 7fada351b600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:09:14.351 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T11:09:14.355 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2022-09-02T11:09:14.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:09:14.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:09:14.359 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T11:09:14.610 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T11:09:14.611 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T11:09:14.664 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T11:09:14.681 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T11:09:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:09:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T11:09:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T11:09:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T11:09:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:09:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T11:09:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T11:09:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T11:09:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:09:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T11:09:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T11:09:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T11:09:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:09:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T11:09:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T11:09:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T11:09:14.686 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T11:09:14.689 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T11:09:14.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T11:09:14.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:09:14.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:09:14.692 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T11:09:14.747 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T11:09:14.747 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T11:14:14.750 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:14:14.748+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:19:14.753 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:19:14.749+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:24:14.754 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:24:14.749+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:29:14.754 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:29:14.750+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:34:14.754 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:34:14.751+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:39:14.755 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:39:14.750+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:44:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:44:14.752+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:49:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:49:14.752+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:54:14.758 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:54:14.752+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:59:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T11:59:14.822 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T11:59:14.753+0000 7f7ac2b14600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T11:59:14.823 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T11:59:14.827 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2022-09-02T11:59:14.830 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2022-09-02T11:59:14.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2022-09-02T11:59:14.832 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2022-09-02T11:59:14.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:59:14.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:59:14.837 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T11:59:15.003 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T11:59:15.003 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T11:59:15.035 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T11:59:15.051 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T11:59:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:59:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T11:59:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T11:59:15.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T11:59:15.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:59:15.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T11:59:15.053 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T11:59:15.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T11:59:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:59:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T11:59:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T11:59:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T11:59:15.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T11:59:15.056 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T11:59:15.056 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T11:59:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T11:59:15.057 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T11:59:15.059 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T11:59:15.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T11:59:15.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:59:15.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T11:59:15.061 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T11:59:15.115 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T11:59:15.117 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T12:04:15.119 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:04:15.115+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:09:15.121 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:09:15.115+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:14:15.123 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:14:15.117+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:19:15.122 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:19:15.118+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:24:15.123 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:24:15.117+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:29:15.124 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:29:15.119+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:34:15.124 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:34:15.118+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:39:15.124 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:39:15.119+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:44:15.134 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:44:15.119+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:49:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T12:49:15.183 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:49:15.120+0000 7f7202dcc600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:49:15.183 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T12:49:15.186 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2022-09-02T12:49:15.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T12:49:15.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T12:49:15.191 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T12:49:15.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T12:49:15.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T12:49:15.561 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T12:49:15.575 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T12:49:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T12:49:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T12:49:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T12:49:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T12:49:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T12:49:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T12:49:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T12:49:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T12:49:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T12:49:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T12:49:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T12:49:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T12:49:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T12:49:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T12:49:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T12:49:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T12:49:15.581 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T12:49:15.583 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T12:49:15.583 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T12:49:15.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T12:49:15.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T12:49:15.585 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T12:49:15.645 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T12:49:15.646 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T12:54:15.648 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:54:15.644+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T12:59:15.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T12:59:15.644+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:04:15.650 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:04:15.644+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:09:15.649 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:09:15.645+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:14:15.657 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:14:15.645+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:19:15.650 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:19:15.646+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:24:15.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:24:15.646+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:29:15.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:29:15.646+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:34:15.651 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:34:15.646+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:39:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T13:39:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:39:15.647+0000 7fd2a81aa600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:39:15.712 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T13:39:15.715 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2022-09-02T13:39:15.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T13:39:15.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T13:39:15.719 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T13:39:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T13:39:15.892 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T13:39:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T13:39:15.971 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T13:39:15.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T13:39:15.972 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T13:39:15.972 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T13:39:15.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T13:39:15.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T13:39:15.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T13:39:15.973 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T13:39:15.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T13:39:15.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T13:39:15.974 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T13:39:15.974 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T13:39:15.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T13:39:15.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T13:39:15.975 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T13:39:15.975 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T13:39:15.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T13:39:15.976 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T13:39:15.979 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T13:39:15.980 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T13:39:15.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T13:39:15.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T13:39:15.982 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T13:39:16.035 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T13:39:16.036 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T13:44:16.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:44:16.036+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:49:16.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:49:16.035+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:54:16.040 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:54:16.036+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T13:59:16.041 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T13:59:16.036+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:04:16.041 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:04:16.037+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:09:16.043 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:09:16.037+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:14:16.044 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:14:16.038+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:19:16.044 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:19:16.040+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:24:16.046 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:24:16.040+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:29:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T14:29:16.066 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:29:16.041+0000 7fd5b0ff7600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:29:16.066 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T14:29:16.069 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2022-09-02T14:29:16.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T14:29:16.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T14:29:16.075 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T14:29:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T14:29:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T14:29:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T14:29:16.399 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T14:29:16.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T14:29:16.400 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T14:29:16.400 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T14:29:16.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T14:29:16.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T14:29:16.401 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T14:29:16.401 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T14:29:16.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T14:29:16.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T14:29:16.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T14:29:16.402 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T14:29:16.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T14:29:16.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T14:29:16.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T14:29:16.403 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T14:29:16.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T14:29:16.404 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T14:29:16.405 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T14:29:16.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T14:29:16.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T14:29:16.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T14:29:16.408 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T14:29:16.463 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T14:29:16.465 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T14:34:16.467 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:34:16.461+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:39:16.468 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:39:16.461+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:44:16.469 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:44:16.462+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:49:16.469 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:49:16.462+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:54:16.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:54:16.463+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T14:59:16.470 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T14:59:16.464+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:04:16.471 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:04:16.464+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:09:16.472 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:09:16.464+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:14:16.472 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:14:16.464+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:19:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-02T15:19:16.509 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:19:16.465+0000 7fabef0e3600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:19:16.510 INFO:teuthology.orchestra.run.smithi160.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-09-02T15:19:16.514 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2022-09-02T15:19:16.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T15:19:16.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T15:19:16.519 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T15:19:16.674 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2022-09-02T15:19:16.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T15:19:16.726 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=b798a9cd-aa0e-480c-bb07-05f02abf3f41 2022-09-02T15:19:16.745 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-09-02T15:19:16.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T15:19:16.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T15:19:16.746 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T15:19:16.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T15:19:16.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T15:19:16.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662098413.164890_2143.1", 2022-09-02T15:19:16.748 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T06:00:13.164890Z" 2022-09-02T15:19:16.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T15:19:16.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T15:19:16.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1662107953.158334_499.1", 2022-09-02T15:19:16.749 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2022-09-02T08:39:13.158334Z" 2022-09-02T15:19:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-09-02T15:19:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-09-02T15:19:16.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2022-09-02T15:19:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2022-09-02T15:19:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-09-02T15:19:16.753 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-09-02T15:19:16.755 DEBUG:rgw_multi.tests:master meta markers={0: '', 1: '1_1662098413.164890_2143.1', 2: '1_1662107953.158334_499.1', 3: ''} 2022-09-02T15:19:16.756 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-09-02T15:19:16.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T15:19:16.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-09-02T15:19:16.760 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-09-02T15:19:16.820 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setuser ceph since I am not root 2022-09-02T15:19:16.821 INFO:teuthology.orchestra.run.smithi160.stderr:ignoring --setgroup ceph since I am not root 2022-09-02T15:24:16.824 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:24:16.816+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:29:16.824 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:29:16.816+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:34:16.825 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:34:16.818+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:39:16.825 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:39:16.818+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:44:16.825 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:44:16.819+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:49:16.825 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:49:16.818+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:54:16.826 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:54:16.819+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T15:59:16.826 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T15:59:16.820+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T16:04:16.827 INFO:teuthology.orchestra.run.smithi160.stderr:2022-09-02T16:04:16.820+0000 7f4690544600 0 monclient(hunting): authenticate timed out after 300 2022-09-02T16:06:04.141 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-09-02T16:06:04.148 DEBUG:teuthology.task.console_log:Killing console logger for smithi099 2022-09-02T16:06:04.150 DEBUG:teuthology.task.console_log:Killing console logger for smithi160 2022-09-02T16:06:04.151 DEBUG:teuthology.exit:Finished running handlers